Skip to content

[Brook] Reached Core's stream store from Python. - #113

Closed
moetemp wants to merge 14 commits into
mainfrom
moe/brook/p1-core-stream-store
Closed

moetemp wants to merge 14 commits into
mainfrom
moe/brook/p1-core-stream-store

Conversation

@moetemp

@moetemp moetemp commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR pins Core to the in-process stream service and reaches it from Python through a generated stub.

  • temporalio/bridge/sdk-core points at Core 5ae0ae71. It has the output commit, the streams crate with its Redis store, Worker staging and promotion, and the StreamService.
  • temporalio/bridge/src/streams.rs is about 60 lines of pyo3. Each connect_stream_store call builds a store from a serialized StreamStoreConfig on the client's connection.
  • scripts/gen_bridge_client.py generates a typed StreamService into streams_generated.py, so no stream call is hand-written. A failed call raises StreamCallFailure with the parsed StreamFailure.
  • The bridge WorkerConfig gains stream_store. The payload visitor covers OutputRecord.body and OutputClose.result.

Why?

The store runtime lives in Core now, so every Core-based SDK binds one copy of it (DD-86). Lang reaches it through one generated stub and one call (DD-88).

How did you test it?

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

tests/bridge/test_stream_store.py runs on the dev server with Core's memory store.

Core now holds the stream runtime, so Python takes its stream protos, the record envelope among them, and the commit command that carries records. The visitor runs the codec on each record's body.
One bridge call carries each serialized request, so the typed StreamService stub is generated from Core's protos and a Worker takes the same store. A failed call crosses as Core's StreamFailure.
Stores now compare the batch digest lang takes before the codec, so an append must carry one. The pin also brings the topic closes a Workflow's output commit can carry.
Core now closes each topic a Workflow's commit closes once the commit's records are promoted, so a closed topic refuses appends after the task.
Core now holds a Python script for its cross-SDK tests, and mypy and basedpyright reached it through the submodule and failed on code that isn't this SDK's.
Core now closes a run chain only after the final task's topic closes land, so a closed topic's result is never overtaken by the chain's close.
The import check in poe lint reached Core's cross-SDK test script through the submodule, like mypy and basedpyright did.
Each connect builds its own service and store, and the tests rely on that, so the comment shouldn't call it process-wide.
The Core review fixes add the topic flag to the append script, keep stages on a NotFound completion and cap reads. Only proto comments and the envelope's Ruby package changed, so the regenerated files differ in docstrings.
C5b 5ae0ae71 only explains why a refused promotion is retried, so the final pin matches the Brook Core top.
@moetemp
moetemp marked this pull request as draft October 12, 2026 06:58
@moetemp moetemp closed this Oct 13, 2026
@moetemp moetemp reopened this Oct 13, 2026
Core now reports a call-input mistake apart from a store refusal, so lang can raise it as ValueError. The pin also takes upstream Core main.
Core's changelog lost the start of one entry in a merge. The code is the same.
@moetemp moetemp closed this Oct 13, 2026
@moetemp moetemp reopened this Oct 13, 2026
The Core pin moved by hand, so update-core never imported the entries from main's pin to the new one.
@moetemp

moetemp commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Replaced by Brook #149 and Ford #163. Brook was split in two: Brook now has streams with Activity and client publish and outside readers, and Ford has the Workflow Task publish path. The code is the same, re-cut into new branches.

@moetemp moetemp closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant