Skip to content

feat(tracing): add failure-check span attribute helper - #555

Open
mohammadatallah-scale wants to merge 9 commits into
mainfrom
mohammad/ove-1375-span-attributes
Open

mohammadatallah-scale wants to merge 9 commits into
mainfrom
mohammad/ove-1375-span-attributes

Conversation

@mohammadatallah-scale

@mohammadatallah-scale mohammadatallah-scale commented Oct 27, 2026 •

Copy link
Copy Markdown

Important

Pending Mohammad's self-review. This note is removed by a human, not automation.

The problem. Offline failure checks read attributes that an agent declares on its spans. AgentEx agents have to hand-build every key and nothing checks the values (OVE-1375).

The fix. A helper that writes and validates those attributes from one spec. The scale-gp-beta port is scaleapi/sgp-dev-python-staging#10.

  • Spec: one validated object writes the root span's declarations as flat attribute keys, with the output schema as JSON text that is checked again when it renders.
  • Per-span helpers: one call each for a required output, a verification result and an escalation.
  • Attach: pass the result as a span's data and the SGP processor sends it as span metadata. The tracing tutorial shows how.

Test plan

  • Same results and error messages as the scale-gp-beta port over 160,000 generated inputs, including schema edits made after construction and random non-string keys.
  • gap: not exported to a live SGP spans API from an AgentEx agent, the span request tests cover the payload and the port's dev-sgp round trip covers the store.
  • gap: a traced example whose spans pass the failure checks is not shown here, the end-to-end harness in OVE-1400 proves it.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issue was found.

Summary

Adds FailureCheckSpec, VerificationRule, and per-span helpers through adk.

  • A checked spec writes failure-check declarations onto root spans.
  • Output, verification, and escalation spans carry matching failure-check markers.

No new actionable issues found. Tests were inspected, not run.

Reviews (4) · Last reviewed commit: "fix(tracing): reject non-string keys in ..." · Reviewed by Greptile

mohammadatallah-scale and others added 3 commits October 27, 2026 10:41
Offline failure checks read sgp.agent.* attributes the agent declares on its
spans. FailureCheckSpec validates the root span declarations on construction
and renders them as flat dotted keys holding native JSON values, which the SGP
processor ships verbatim as span metadata. output_name, verification_result
and escalates write the per-span attributes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The span store keeps metadata in a ClickHouse JSON column, where a schema
object spreads into many dynamic paths whose keys can collide after
dot-splitting. FailureCheckSpec still takes a JSON object and validates it,
then renders sgp.agent.output_schema as a JSON string. Every other attribute
stays a native value.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
output_schema is a mutable dict on a frozen spec, so an edit after
construction skipped the checks construction applies and could write NaN or
invalid JSON text. attributes() now runs the same check before writing the
schema, so an edited schema fails with the error construction raises.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread tests/lib/core/tracing/test_failure_attributes.py Outdated
Comment thread examples/tutorials/10_async/00_base/030_tracing/README.md Outdated
@mohammadatallah-scale

Copy link
Copy Markdown
Author

P3 tests/lib/core/tracing/test_failure_attributes.py:154 - the "error names the attribute" contract is pinned only for step_budget, stop_reason and prompt_version. Relabelling escalation_channels as quote_arguments in _NORMALIZERS, or renaming the required_outputs, verification and VerificationRule labels, leaves all 118 tests green.
P3 failure_attributes.py:89 - json.dumps coerces non-string keys, so output_schema={1: "a", "1": "b"} is accepted and silently stored as {"1": "b"} instead of being rejected.

Refuted: __agent_version__ and __commit_sha__ README claims hold at this head (_add_source_to_span, code_revision), and the port's logic matches line for line apart from 3.9 syntax.

mohammadatallah-scale and others added 2 commits October 27, 2026 10:54
adk.tracing.span yields None when trace_id is empty or the span fails to
start, so the example reading root.data raised AttributeError on that path.
It now guards the span the way the tutorial's acp.py does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only step_budget, stop_reason and prompt_version errors were checked for
the attribute they name, so relabelling escalation_channels as
quote_arguments still passed. Every root field and VerificationRule now
asserts the name its error starts with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mohammadatallah-scale

Copy link
Copy Markdown
Author

P3 Follow-up: the attribute-naming pins landed in 815dad0, and each label swap above now fails 2 to 11 tests. The non-string key coercion is left open because fixing it changes behavior the scale-gp-beta port must match, so both helpers need it together.

mohammadatallah-scale and others added 4 commits October 27, 2026 11:07
The depth was an inline number, so a reader could not tell what it controls.
It is now a named class constant.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A path like send_email.to.cc was rejected with a message that did not say
why. Tool names cannot contain dots, so an entry is a tool and one top-level
argument, and the error now says nested argument paths are not supported.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
escalates("") failed with a message about a verification span id, which did
not name the attribute being written. The message now starts with escalates,
as the other validation errors start with their attribute.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
JSON writes every object key as a string, so the encoder quietly turned an
int, bool, float or None key into text and let {1: "a", "1": "b"} collapse
into one entry. Any non-string mapping key in the schema is now rejected, at
construction and again when attributes render.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant