Skip to content

[Brook] Added RedisStreamPlugin over the linked temporalio-streams crate. - #330

Closed
moetemp wants to merge 7 commits into
moe/brook/v3/p7-streams-facadefrom
moe/brook/v3/p8-redis-plugin
Closed

moetemp wants to merge 7 commits into
moe/brook/v3/p7-streams-facadefrom
moe/brook/v3/p8-redis-plugin

Conversation

@moetemp

@moetemp moetemp commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR adds temporalio.contrib.streams.redis.RedisStreamPlugin, and links the temporalio-streams crate into the bridge behind a streams feature. The feature is off by default and on in the official builds (poe build-develop, the cibuildwheel wheels and fips). The bridge builds it as the temporalio-streams-redis worker extension.

The plugin:

  • puts the extension on the Worker's extension option and gives the facade a StreamBackend;
  • takes the Workflow's CreateStream, AppendRecords and CloseStream in a System Nexus outbound interceptor and returns a done handle;
  • converts Record values in the sandbox, runs the codec with no serialization context in on_workflow_activation_complete, merges adjacent appends, and calls BufferWorkflowCalls;
  • always closes the default Stream when the run ends, and the other auto_close Streams it wrote, with chain_continues for a retry (attempts and time left), a cron run, or nothing for continue-as-new;
  • sets withdraw_run on every call of a retry run;
  • puts the default Stream and a per-run counter key into each Activity's and Local Activity's headers, so the user's workflow.random() stays the same, and builds the Activity, Local Activity and Standalone Activity Producers and epochs;
  • withdraws earlier attempts at the start of each attempt above 1 with no create, and closes every writer when the attempt ends;
  • passes temporalio.contrib.streams through the sandbox.

Why?

One plugin serves Workflow code, Activities and the client, as in the proposal. The crate must be in the same native library as Core, so it links into the bridge.

How did you test it?

pytest tests/contrib/streams/test_redis.py tests/contrib/streams/test_chain.py against the dev server and a local redis-server on a free port. It covers the default Stream, scoped Streams, a client Stream that a Workflow writes to, Activity retries, a failed Workflow Task, the Workflow's random values, a retried Activity with no Streams, a close after continue-as-new, a cancelled with block, a sync Activity, temporal-only sandbox passthrough, and each chain_continues case.

Please note that an offline build of the sdist still fails with the feature off, since Cargo resolves the optional git dependencies.

  • Unit Tests
  • Staging
  • End to End Tests

The bridge links the crate behind a streams feature that is on by default, so the plugin can build the Redis worker extension by name. The plugin buffers each Workflow Activation's Stream calls and hands them to the crate at the end of the Activation, and it gives Activities and clients their writers.
Activity keys come from a per-run counter, so the plugin leaves the Workflow's random values alone. A retried attempt withdraws without a create, the last run always closes the default Stream, and chain_continues covers retries, cron and the execution timeout. The streams feature is now off by default and on in the official builds.
The crate hardened how it tracks closes in flight, which the Worker shutdown drain depends on.
Python closes an evicted run's coroutine outside the Workflow's loop. The interceptor then called the runtime, and each eviction printed an Exception ignored traceback.
@moetemp

moetemp commented Oct 25, 2026

Copy link
Copy Markdown
Owner Author

Closing as stale. This layer is folded into the v5 tip, #365, and the upstream review uses a fresh series instead. Plan: https://www.notion.so/4048fc567738814990b7c796ac28fe3f

@moetemp moetemp closed this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant