Skip to content

[Brook] Added stream records and the hashes lang owes Core. - #114

Closed
moetemp wants to merge 22 commits into
moe/brook/p1-core-stream-storefrom
moe/brook/p2b-records-wire
Closed

moetemp wants to merge 22 commits into
moe/brook/p1-core-stream-storefrom
moe/brook/p2b-records-wire

Conversation

@moetemp

@moetemp moetemp commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What changed?

This PR adds StreamRecord and the hashes lang takes before Core stores the records.

  • StreamRecord[T], RecordKind, Supersession and Cursor go in temporalio.contrib.streams.
  • _body.py takes the plaintext content_hash and the batch digest, both before the codec. Core stores them unchanged.
  • Only the writer that made the digest compares it. Deterministic protobuf bytes differ across implementations.
  • _wire.py turns a record of Core's read answer into a StreamRecord. A supersession that Core synthesized becomes a SUPERSEDED record.
  • The envelope proto is Core's, so this PR has no proto of its own.

Why?

Only lang has the plaintext. With both hashes taken before the codec, a retry through a codec with a fresh nonce still dedupes (DD-11).

How did you test it?

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

tests/contrib/streams/test_records_wire.py also checks the digest against the vector Core's tests pin.

moetemp added 18 commits October 9, 2026 14:54
Streams store records as temporal.sdk.streams.v1.StreamRecord. Only the
Python SDK reads it, so it is generated here rather than in the API repo.
Records map to the envelope proto in one place, and the body helpers hash
the converted payload before the codec so a retry deduplicates.
The generated v1 modules are rewritten on every proto run, so the warning
goes on the hand-written package above them.
Only provider code uses them, so exporting them would commit the package
to an interface no application needs.
A stored kind at 3 would decode as SUPERSEDED, which readers refuse. The format test now pins field types and labels too, since a retyped field breaks stored records.
A resumed read started with no attempts seen, so a retry right after the cursor reached the reader as plain DATA with no SUPERSEDED.
Core now owns the envelope proto and builds every stored record, so lang keeps conversion and the two hashes that need the plaintext. The content hash is raw bytes for Core, and the batch digest feeds the store's dedupe.
Deterministic serialization differs across protobuf implementations, so a retry through another SDK wouldn't dedupe. A publishing task does add one marker event, so the package shouldn't promise none.
@moetemp
moetemp marked this pull request as draft October 12, 2026 06:58
@moetemp moetemp closed this Oct 13, 2026
@moetemp moetemp reopened this Oct 13, 2026
@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 13, 2026
@moetemp

moetemp commented Oct 13, 2026

Copy link
Copy Markdown
Owner Author

Replaced by Brook #150. Brook was split in two: Brook now has streams with Activity and client publish and outside readers, and Ford has the Workflow Task publish path. The code is the same, re-cut into new branches.

@moetemp moetemp closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant