Repository navigation
feat: preserve native citations during record and replay - #594
Merged
Merged
Conversation
commit: |
… path The cited Responses JSON branch of buildFixtureResponse added its own captureUsage call, which broke the response-adoption guard (one nonstream capture per family). Reuse the usage the citation reader already captured, sanitized with sanitizeRecordedUsage, so the raw envelope usage never reaches the fixture unsanitized.
The post-collapse block that re-adds ordered blocks ran for every streamed recording, so an uncited Responses stream in responsesTools extended mode gained a blocks key next to responsesBlocks. Run it only when a text block carries native citation metadata and the turn is not already ordered by responsesBlocks, and sanitize its tool calls like the branches above.
The citation recorder keyed text by output item and content part, but replay opened a new message item for every text block, so one upstream message with two output_text parts replayed as two messages. Mark later parts of one recorded message with continuesMessage, and let Responses replay group such blocks into one message item with a content part each. Fixtures without the marker replay exactly as before.
…citations CitationRecording no longer builds tool calls, content or usage of its own. It reads only the text segments and their native metadata, and orders them with the collapser's own finalized tool calls. The result is the uncited collapse plus ordered blocks: - JSON-captured Responses usage is sanitized once at capture, and the second non-stream pass that overwrote it with raw upstream usage is gone. - Anthropic server_tool_use / mcp_tool_use input no longer becomes a nameless tool call; index-less tool_use calls are kept. - Tool arguments and namespaces follow the uncited normalization; empty uncited text blocks, `toolCalls: []` and stream usage are no longer added. - One predicate (hasNativeCitations) decides whether a turn is cited, and a cited turn always keeps `blocks`, so text/tool order survives custom calls when extended Responses tools are off. - Null output/summary entries no longer break a cited Responses JSON body, and a cited empty-text stream no longer logs the empty-content warning.
…served prepareClaudeMisbehavior already dropped portable citations for Bedrock InvokeModel (`citations: false`) but kept a text block's native `citations`, so an applied misbehavior served them. Apply the same flag to native block citations, and remove the unused `citations` field from the request-side ClaudeContentBlock type.
`annotations` / `citations` must be arrays of plain objects and `continuesMessage` a boolean, at load (validateFixtures) and when a block is resolved for serving, where a bad value now raises the usual "Invalid fixture block" error instead of a bare DataCloneError or a silent drop. The empty-text exemption applies only to a block with valid, non-empty citation data.
…itations - afterEach runs every cleanup even when one fails, and recorded() surfaces the real recording error instead of a fetch timeout. - The tool-mode tests stream real Responses events and use real nested usage, asserting the exact sanitized JSON usage and no usage on a stream. - A turn whose terminal metadata clears every citation records as uncited. - Regression tests for server-tool input, empty text blocks, whitespace arguments, empty namespaces, index-less tool_use, text-only turns, null output entries, the empty-content warning, custom-call ordering in both modes, the public collapse results, Bedrock misbehavior citations and native citation field validation.
… sparse arrays The provider SDKs use citations: null for an uncited text block. Fixtures that carried it failed validation and served HTTP 500, although main served them. Null now reads as absent at load and serve time. Array holes are not citation objects, so a sparse annotations or citations array is rejected.
…eld is empty An explicit empty annotations or citations array suppressed the portable response-level citations for that block. Empty now reads as absent, as on main.
… calls collapseAnthropicSSE and CitationRecording minted synthetic keys from two counters, so an index-less cited text block was recorded after the tool call that followed it. The collapser now keys every block start and hands the key to the citation recording.
An annotation or citations_delta event addressed to a tool item recorded an extra empty cited text block that upstream never sent. Text metadata on a known non-text index is now dropped.
A JSON body whose only content is an empty text block with citations fell through to a proxy_error fixture. It now records content "" plus its cited blocks, as the streaming path does.
…ng and SDK replay Regression tests for each round-2 fix, a parity test that compares the cited blocks, a specific empty-block error, and record-to-replay checks read through the installed openai and @anthropic-ai/sdk clients.
…stead of failing Earlier releases ignored annotations, citations and continuesMessage on a text block, so a code-built fixture that carries a malformed one (a string, null, a sparse array, a non-cloneable value) served 200. Read such a value as absent at serve time for blocks and responsesBlocks; validateFixtures still reports it at load.
A `data: null` frame threw while keying the citation recording and aborted the collapse; main ignores it.
… cited turn An index-less text delta before the first content_block_start had no block key, so the citation recording dropped it and the cited content lost it. Key it below every real index so it stays first.
The cited blocks replaced the harmony-parsed content with the raw channel tokens and left the harmony tool calls out of blocks. Rebuild the cited blocks from the parsed content and tool calls, keeping the annotations.
A cited turn's blocks are in index order while the flat toolCalls kept arrival order, so the two disagreed (CopilotKit#274) when the orders differ. Derive the flat list from the blocks, as an uncited interleaved turn does.
…nstead of rebuilding it A cited stream turn was collapsed twice: once by the base collapser and once by CitationRecording, which rebuilt its own text, index keys and tool calls. The two kept drifting (text lost without content_index, harmony offsets into raw tokens, frames without event lines, block-start text, index vs arrival order). The base collapse is now the one source of truth. It tells the recording which text part each piece of content belongs to, and the recording only attaches citation data: it splits the base text blocks at part boundaries and puts each part's annotations or citations on its pieces. Content, reasoning, tool calls, block order, harmony parsing, dropped chunks and usage come unchanged from the base collapse. - Harmony turns move annotation offsets into the parsed text, or drop and report the ones that point outside it. - Annotations sent as response.output_text_annotation.added (openai@4) are accepted on record; replay keeps the API's dotted name. - Non-object citation entries are dropped and reported (citationNote, logged by the recorder); a Responses JSON turn that carried only such entries records its text instead of an error fixture. - A Responses JSON body is collapsed as the stream of the same turn.
… citation data A property test runs every cited stream shape the suite and the review rounds built (and three Responses JSON bodies) through the collapser and checks that, with its citation data taken away, the result deep-equals the collapse of the same stream with its citations removed. Regression tests cover the round-4 findings: a Responses delta without content_index, harmony annotation offsets, Anthropic frames without event lines, Anthropic block-start text, the openai@4 annotation event name, and a Responses JSON turn whose annotations are all not objects.
Adds a "Recording citations" section next to the CopilotKit#578 citations docs: what is recorded, that it is on by default, and per-provider notes. The lone docs/native-citation-replay.md, which no page linked and the site does not publish, is removed.
…dy itself collapseResponsesJSON streamed each output_text twice (delta and content_part.done) and capped the re-encoded events, so a complete body near the limit was marked truncated. The text is now streamed once and the limit is checked against the body.
…age everywhere The JSON collapse and the citation recording now share isResponsesMessage, so a typeless item records its text with its annotations instead of annotations on an empty text.
A non-object entry replaced by a later snapshot, or addressed to a tool call, is no longer counted, and one entry reads "entry that is not an object".
…-of-range offsets A point index at the end of a split piece stays on that piece (the replay rule in responses.ts). An offset outside its text is dropped and noted whether or not the part is split; a part that streamed no text keeps its entries as sent.
…ool call Tool calls are recorded in arrival order next to the text parts, so an empty cited part goes after what arrived before it and keeps continuesMessage.
…xt block The parsed text is laid out over the streamed text body by body (preamble, several finals, interleaved parts), each text block around custom calls is replaced by its parsed text, and an annotation that cannot be placed is reported with the real reason.
…dropped as uncited The JSON branch is taken only for a turn that keeps a citation; the note is still logged.
The JSON path now logs the truncated, dropped-chunk, harmony and citation warnings the stream path logs, through one shared helper.
…corpus Includes two separate cited messages, which record and replay as two messages.
…ited turn plus citations A cited gpt-oss harmony turn with a custom tool call replaced the streamed text in its blocks with the harmony-parsed text, and built them from responsesBlocks, which hold no harmony tool call. Replay of the cited recording then lost the harmony function call that the uncited recording serves. blocks now hold the parsed text in the order the flat fields imply, with every tool call, and responsesBlocks keep the streamed text exactly as the uncited turn's do, with each annotation at the offsets it was sent with. Both shapes are back in the cited-equals-uncited property corpus.
… block Earlier releases ignored annotations, citations and continuesMessage on a text block, so a fixture with a malformed value loaded and served. Load validation reported it as an error, which made addFixturesFromJSON throw, the control-API fixture POST answer 400 and --validate-on-load exit 1. Report it as a warning that the value is ignored when served, the way a malformed response-level citations value is reported. The fixture loads on every path and serves as before.
jpr5
added
public-api-addition
Contributor
|
Thanks for this. Recording citations from real provider turns fills a real gap, and it's on by default now. We pushed fixes on top of your branch so recordings without citations stay byte-identical to before: a cited turn is now the normal recording plus its citation data, built from one collapse path instead of two. We also kept existing fixtures loading as they did, matched Bedrock's citation handling, tightened validation, and added tests that replay recordings through the OpenAI and Anthropic SDKs. Shout if anything looks off. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The existing portable response-level citations cannot preserve all provider-native fields or which text segment owns each citation when recording and replaying a multi-part response. This adds native
annotationsandcitationsarrays to text blocks for OpenAI Responses and Anthropic Messages respectively.JSON and SSE capture retain text ownership, indexed events, terminal snapshots, unknown metadata fields and repeated entries. Replay emits the corresponding annotation events or citation deltas and completed JSON metadata. Empty cited text is supported. Existing portable citations remain available; native block metadata takes precedence for its provider. Current extended-tool opt-in, namespaces/custom calls, reasoning and usage are preserved. Uncited recordings keep their existing representation.
Validation: 1,011 tests pass across eight affected suites, including 55 native-citation cases and mixed extended/default tool-mode recording. Type checks, build, ESLint, formatting, generated API reports and package export checks pass. Documentation includes a fixture example.
This extends the public text-block types; the updated
etc/reports are included. Please applypublic-api-additionfor compatibility review.