[pcbeingused333]

💡 Motivation and Context

The posthog-js LangChain handler records $ai_web_search_count, and so do the Python OpenAI, Anthropic and Gemini wrappers, but the Python LangChain callback handler never did. When Anthropic reports usage.server_tool_use.web_search_requests (as a dict or an object), _parse_usage_model now reads it into a new web_search_count field of ModelUsage, and the handler adds $ai_web_search_count to the event, including a count of 0. Nothing is added when the provider reports no count. langchain-anthropic passes the raw Anthropic usage (with server_tool_use) in llm_output for non-streamed calls, which is where the handler reads it; streamed calls only carry the standard usage_metadata, so no count is recorded for them. Only this exact count is handled; posthog-js also infers a single search from citations or grounding metadata for other providers, which is not ported here.

💚 How did you test it?

• TestWebSearchCount in posthog/test/ai/langchain/test_callbacks.py: a response with two web search requests, one with zero, and one without server_tool_use. The first two fail on master and pass with the change; the third guards the omitted case.
• posthog/test/ai/langchain/test_callbacks.py: 83 passed, 8 skipped. ruff format --check passes on the touched directories. The rest of the suite and a live LangChain call with the Anthropic web search tool were not run.

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• I updated the docs if needed. (n/a)
• No breaking change or entry added to the changelog.

If releasing new changes

• Ran sampo add to generate a changeset file (added .sampo/changesets/langchain-web-search-count.md)

🤖 Agent context

Autonomy: Fully autonomous

• The gap was found and the change, tests and description were written by Claude Code (Claude Sonnet 5.5) by comparing the Python and JS LangChain handlers. Agent-authored, needs human review.

🤖 Generated with Claude Code

🤖 Generated with Claude Code

[marandaneto]

Automated advisory code review.

[marandaneto]

@PostHog/team-ai-observability do we need any backend/frontend change for this?

[carlos-marchal-ph]

This emits $ai_web_search_count even when it's 0, but the other Python wrappers only emit it when it's > 0 (posthog/ai/utils.py), so a LangChain Anthropic event would differ from a direct Anthropic wrapper event for the same call. Can we gate on > 0 here too?

[carlos-marchal-ph]

This never fires on real ChatAnthropic output. langchain-anthropic always sets AIMessage.usage_metadata, so the generations loop in _parse_usage runs after the llm_output["usage"] parse and overwrites it with usage that has no server_tool_use, resetting web_search_count to None. Same llm_output gives 2 without usage_metadata on the message and None with it. Could _parse_usage read server_tool_use from llm_output["usage"] separately and merge it into whichever ModelUsage wins for the tokens?

[carlos-marchal-ph]

Could this follow the rest of the file? The mock_client fixture and the module-level imports already cover CallbackHandler, uuid, AIMessage and MagicMock, and the three tests could be one @pytest.mark.parametrize test like test_stop_reason_resolution just above.

[carlos-marchal-ph]

ChatAnthropic always sets usage_metadata on the message, and this fixture leaves it out, which is why the tests pass while the feature does nothing on real output. With usage_metadata={"input_tokens": 10, "output_tokens": 5, "total_tokens": 15} they cover the real path, and should fail until the override above is fixed.

[carlos-marchal-ph]

@PostHog/team-ai-observability do we need any backend/frontend change for this?

No, but there were some issues with the implementation that I surfaced on my review

[pcbeingused333]

Thanks, all four points addressed in the latest commit:

• $ai_web_search_count is now emitted only when the count is greater than 0, like posthog/ai/utils.py.
• server_tool_use is read from llm_output["usage"] separately in _parse_usage and merged into whichever ModelUsage wins for tokens, so it is no longer lost when usage_metadata is set on the message.
• The three tests are now one parametrized test_web_search_count using the module-level imports and the mock_client fixture.
• The fixture sets usage_metadata={"input_tokens": 10, "output_tokens": 5, "total_tokens": 15}; with the callbacks.py change reverted the two-searches case fails, and with it the test passes.