Skip to content

[Brook] Added the Redis and memory stream store plugins. - #117

Closed
moetemp wants to merge 20 commits into
moe/brook/p2d-topics-refsfrom
moe/brook/p2e-provider-plugin
Closed

moetemp wants to merge 20 commits into
moe/brook/p2d-topics-refsfrom
moe/brook/p2e-provider-plugin

Conversation

@moetemp

@moetemp moetemp commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR adds RedisStreams and MemoryStreams, the client plugins that name the store Core runs.

  • StreamStorePlugin is a SimplePlugin that connects the store in connect_service_client. Every Worker built from that client hands it to Core as stream_store.
  • RedisStreams takes URLs and Core's settings. TLS comes from a rediss:// URL and checks the server against the system roots only. MemoryStreams() is for tests.
  • A store on a Worker alone raises ValueError, and so does a second store on one client or a second connection on one store.
  • delete_workflow_streams(namespace, workflow_id) goes through Core's DeleteOwner.

Why?

Core runs the store, so the provider classes become configuration (DD-6, DD-90). Core checks a stream's owner over the connection that connected the store, so the store comes from the client.

How did you test it?

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

tests/contrib/streams/test_store_plugin.py runs on the dev server.

moetemp added 16 commits October 9, 2026 14:57
A provider hands out handles and producers, and registers once on a client
or Worker, so the accessors never pick between two stores silently.
Providers must serialize calls on one producer, and a caller needs to know
what to do after a cancelled or unknown append.
The read contract promised every record once, which a resumed read keeps only for stored records. A second producer object for one session is refused, and registering the plugin twice warns.
Core runs the store, so the provider classes become client plugins that name one. Connecting the client connects the store, and every Worker built from that client hands it to Core.
Core asks about owners over the connection that connected the store, so a second connection got the first one's answers. A Worker now needs a client made with Client.connect whatever connected the store first, and the delete test checks a stream that held a record.
@moetemp
moetemp marked this pull request as draft October 12, 2026 06:58
@moetemp moetemp closed this Oct 13, 2026
@moetemp moetemp reopened this Oct 13, 2026
@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 13, 2026
@moetemp

moetemp commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Replaced by Brook #153 and Ford #163. Brook was split in two: Brook now has streams with Activity and client publish and outside readers, and Ford has the Workflow Task publish path. The code is the same, re-cut into new branches.

@moetemp moetemp closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant