Repository navigation
fix(aio): keep content out of ai events in privacy mode - #1105
Conversation
![posthog[bot]](https://avatars.githubusercontent.com/in/1203049?s=60&v=4){kind=small}
| _CONTENT_ATTRIBUTE_ROOTS = ( | ||
| "gen_ai.prompt", | ||
| "gen_ai.completion", | ||
| "llm.prompts", | ||
| "llm.input_messages", | ||
| "llm.output_messages", | ||
| ) |
There was a problem hiding this comment.
P1 · Redact OpenInference function calls and prompt-template content
With OpenInference LangChain instrumentation, spans carry model-generated arguments in llm.function_call and user inputs in llm.prompt_template.variables. Neither matches _is_content_attribute, so both OTel integrations export this content with privacy_mode=True, even after removing the message attributes. Extend the filter to cover these content fields.
| attributes=attributes, | ||
| events=events, | ||
| links=span.links, |
There was a problem hiding this comment.
P2 · Preserve collection types supported by older OTel encoders
With supported OpenTelemetry SDK 1.20.0, privacy-mode spans retaining any attribute fail OTLP serialization: ReadableSpan.dropped_attributes reads .dropped, which this plain dict lacks. Nonempty event lists and link tuples fail similarly. Both integrations lose the export batch; use compatible bounded collections or raise the minimum OTel version.
posthog-python Compliance ReportDate: 2026-10-23T14:02:30.419574+00:00 ✅ All Tests Passed!121/121 tests passed Capture_V1 Tests✅ 95/95 tests passed View Details
Capture_Ai Tests✅ 5/5 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
Feature_Flags_Local_Evaluation Tests✅ 4/4 tests passed View Details
|
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
| "pydantic_ai.all_messages", | ||
| "final_result", | ||
| "tool_arguments", | ||
| "tool_response", |
There was a problem hiding this comment.
P1 · Redact Pydantic AI model request parameters under privacy mode
With Pydantic AI 1.107.7, model_request_parameters contains instruction_parts[].content and prompted_output_template. _CONTENT_ATTRIBUTES omits this attribute, so both OTel integrations export prompt text with privacy mode enabled. A canary survives _redact_span and OTLP serialization; drop this attribute or redact its content fields.
![stamphog[bot]](https://avatars.githubusercontent.com/in/3000332?s=60&v=4){kind=small}
There was a problem hiding this comment.
Not approved — this change needs a human reviewer.
Re-adding the stamphog label gives the same result unless the changed files change.
stamphog can't approve this pull request, because two gates refused it. The deny-list gate matched the public_api category: the new privacy_mode and client kwargs on PostHogSpanProcessor and PostHogTraceExporter change the public API, and references/public_api_snapshot.txt is modified. The tier gate classified it as T2-never: a cross-cutting fix of 1206 lines across 19 files that touches many AI integrations at once.
Please ask a human reviewer to look at it. Splitting it up may also help, for example by moving the OTel constructor changes into their own pull request, separate from the privacy fixes in the other integrations.
- posthog[bot] reviewed the current head.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✗ | matches: public_api |
| size | ✓ | 549L, 13F substantive, 1206L/19F incl. docs/generated/snapshots — within ceiling |
| tier | ✗ | classified as T2-never: T2-never (1206L, 19F, cross-cutting, fix) |
| stamphog 2.6.1 | .stamphog/policy.yml @ unknown · reviewed head 157ad38 |
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 1fec9fd: 2 Must fix.
This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
| if metadata is not None: | ||
| kept[key] = metadata | ||
| continue | ||
| kept[key] = value |
There was a problem hiding this comment.
P1 · Mask MCP credentials in retained OTel tool definitions
Vercel AI SDK telemetry serializes remote MCP tools into ai.prompt.tools, including args.authorization and args.headers. _without_content retains these definitions verbatim, so both OTel integrations export access tokens with privacy_mode=True; a credential canary survives OTLP encoding. Parse retained tool definitions and mask MCP credentials.
| for source in sources | ||
| for key in _ERROR_CODE_FIELDS | ||
| if ( | ||
| code := _matching( | ||
| _read_error_field(source, key), _ERROR_PROVIDER_CODE_RE | ||
| ) |
There was a problem hiding this comment.
P1 · Restrict privacy summaries to trusted diagnostic values
With privacy enabled, a LangChain tool exception with message request failed and code='sk-proj-secret-token-123' now exports that token in $ai_error; previously, _stringify_exception emitted only the message. _summarize_error_for_privacy accepts arbitrary exception attributes based on character syntax, which also accepts secrets. Allowlist known diagnostic values and omit unknown fields.
There was a problem hiding this comment.
PostHog Review · Standard · reviewed f3df40e: 1 Must fix.
This is review round 4 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
| "ai.prompt", | ||
| "ai.prompt.messages", | ||
| "ai.request.text", | ||
| "ai.documents", |
There was a problem hiding this comment.
P1 · Strip document content from Vercel reranking attributes
Vercel AI SDK reranking spans store the original documents in ai.ranking, whose serialized entries contain document, originalIndex, and score. _without_content removes ai.documents but retains ai.ranking, so both OTel integrations still export the documents with privacy_mode=True. Drop this attribute or remove each entry's document content.
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 0621831: 1 Should fix.
This is review round 5 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
| for span in ai_spans: | ||
| warn_if_posthog_ai_gateway_otel_attributes(span.attributes) | ||
| privacy_mode = _otel_privacy_mode_enabled(self._privacy_mode, self._client) | ||
| ai_spans = [_redact_span(span, privacy_mode) for span in ai_spans] |
There was a problem hiding this comment.
P2 · Isolate redaction failures to the affected span
A deeply nested JSON string in ai.prompt.tools raises RecursionError inside _redact_span, with privacy enabled or disabled. This unguarded comprehension aborts export before forwarding any spans, so BatchSpanProcessor loses healthy spans in the same batch. Catch redaction errors per span and drop only the failing span, matching PostHogSpanProcessor.
There was a problem hiding this comment.
PostHog Review · Standard · reviewed ff9058f: 1 Must fix.
This is review round 6 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
| if metadata is not None: | ||
| kept[key] = metadata | ||
| continue | ||
| kept[key] = value |
There was a problem hiding this comment.
P1 · Redact credentials from Vercel request-header attributes
When Vercel AI SDK callers supply an Authorization header, its telemetry records ai.request.headers.Authorization. _without_content retains this attribute, so both OTel integrations export the credential with privacy_mode=True; a canary survives OTLP encoding. Mask sensitive request-header attributes before export.
There was a problem hiding this comment.
PostHog Review · Standard · reviewed ad27041: 1 Must fix.
This is review round 7 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
marandaneto
left a comment
marandaneto
left a comment
There was a problem hiding this comment.
The code looks good; no qualifying findings. Verdict: correct.
|
some bots pr open tho |
…ode-leaks # Conflicts: # posthog/ai/gemini/_shared.py
💡 Motivation and Context
privacy mode doesn't keep content out of AI events:
$ai_errorisstr(exc), never redacted..parse()validation errors quote model output, gemini client-side validation quotes the prompt, langchainOutputParserExceptionincludes the full output, and openai-compatible servers echo the request in 4xx bodies.$exceptionautocapture ignores privacy.headers/authorizationare sent verbatim in$ai_tools, privacy or not.posthog_privacy_mode=Falseundid a client default ofTrue.model_configprompt/instructions and$ai_trace_metadatasent under privacy.context: https://posthog.slack.com/archives/C0ACMS2BNTU/p1792232645298019
Changes
$ai_errorunder privacy (client or per-call/handler flag): built only from structured fields that pass a strict grammar, never message/traceback/attributes. format, shared with posthog-js:{Name}[: {status}][ {code}…][ param={param}][ request_id={id}][ (caused by A, B)]e.g.
BadRequestError: 400 invalid_request_error param=messages[3].content request_id=req_123. applied to openai / anthropic / gemini (incl. embeddings), langchain generation/span/trace, openai agents. the middleware's fixed "Tool returned an error" text is kept.$exceptionskipped under privacy.headers/authorizationvalues ontype == "mcp"tools become[REDACTED], always, without mutating the caller's objects. tool definitions otherwise unchanged and still sent under privacy.model_configprompt/instructionsand$ai_trace_metadataset toNone.privacy_modeandclientkwargs onPostHogSpanProcessor/PostHogTraceExporter; also follows the module-level client. content attributes and exception messages stripped, list derived from the server's otel →$ai_*mapping. tool definitions andexception.typekept. if redaction fails the span is dropped, not sent.Notes for reviewers
capture-aionly defines privacy as stripping content properties; this widens it to errors and OTel. spec update to follow.references/public_api_snapshot.txt).💚 How did you test it?
posthog/test/ai/test_privacy_mode.py: summarizer against real openai / anthropic / google-genai / pydantic errors and grammar rejections; MCP masking across providers and tool shapes; canary tests with a realClient(privacy_mode=True, enable_exception_autocapture=True)asserting a canary string appears nowhere in any captured event, incl.$exception, across every integration and OTel.pytest posthog/test/ai1288 passed; ruff, mypy (baseline),make public_api_checkclean.📝 Checklist
If releasing new changes
sampo addto generate a changeset file🤖 Agent context
Autonomy: Human-driven (agent-assisted)
$ai_errorover text scrubbing: regex/PII detection can't guarantee anything. rejected a full schema allowlist for$ai_tools(would silently drop new tool types); only the documented MCP credential slots are masked.fix(aio)/ai-privacy-mode-leaks.🤖 Generated with Claude Code