Skip to content

fix(test): drain queued connections before HTTP observation stops - #4562

Open
ScriptedAlchemy wants to merge 1 commit into
masterfrom
fix/http-observation-drain
Open

ScriptedAlchemy wants to merge 1 commit into
masterfrom
fix/http-observation-drain

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4561

Problem

cli_non_interactive_test::http_observation_counts_connections_and_releases_its_listener_on_unwind intermittently saw 0 accepted connections instead of 1 (PR #4534, run 40275017040).

Root cause

The accept loop in observe_http_connections read the stop flag after accept() returned WouldBlock. A connection the action completed, followed by the stop signal, between that accept() and the flag load was never counted.

Fix

Read the stop flag before accept(). A WouldBlock that follows an already-set flag proves the backlog was empty after the action returned, so every completed connection is counted. The listener is still dropped when the observer thread returns, before observation returns, and a panicking action still unwinds through the StopHttpObservation guard. All four callers of the helper share the one loop. No sleeps or timeouts were added or changed.

Verification

The race does not fire naturally on this macOS host. 200 runs under 54 busy loops (--runs_per_test=200) passed 200/200 both before and after the fix. To prove the mechanism I injected a temporary 20 ms delay between accept() and the flag decision (not committed):

  • Before fix, with delay: 30/30 runs fail with a completed connection must be observed, left: 0, right: 1 (the CI failure verbatim).
  • After fix, with the same delay: 30/30 pass.

After removing the delay:

  • 200 runs of the test under load: 200/200 pass.
  • Full //crates/tracedecay-cli:core_cli_suite: 198 pass, 4 fail. The 4 failures are remote_cli_test::remote_* (error sending request for url (https://127.0.0.1:.../remote/enrollment)) and fail identically on unmodified origin/master on this host.
  • bazel build --config=clippy //crates/tracedecay-cli/... is clean.

🤖 Generated with Claude Code


Devin Review

The observer loop read the stop flag after accept() returned
WouldBlock. A connection the action completed between that accept and
the flag load was never counted, so a loaded runner saw 0 instead of 1.
Read the flag before accept(): a WouldBlock after a set flag proves the
backlog was already empty once the action returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 790eeba

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test flake: http_observation_counts_connections stops accepting before a completed connection is counted

1 participant