Skip to content

test(cloudlink): QUIC fails after its dial in the demotion test - #1839

Merged
frahlg merged 1 commit into
masterfrom
fix/cloudlink-test-flakes
Oct 25, 2026
Merged

frahlg merged 1 commit into
masterfrom
fix/cloudlink-test-flakes

Conversation

@frahlg

@frahlg frahlg commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Test-only. TestAQUICSessionThatEndsBeforeHelloOKDemotesQUIC failed CI twice: run 39557013162 (master) and run 40178374052 attempt 3 (PR #1827).

--- FAIL: TestAQUICSessionThatEndsBeforeHelloOKDemotesQUIC (0.02s)
    quic_test.go:481: a QUIC win moved "quic" first within the day

Cause

The test's QUIC listener closes each session right after the handshake. The box's QUIC dial does more after the handshake: it opens its control stream and writes the target. When the close beat those steps, the dial failed, as it would on a path that blocks UDP. The WebSocket then won the race normally, nothing was demoted, and the later rememberWinner(QUIC) moved QUIC first. The code is right in both cases; the test only set up the case it meant some of the time.

Fixing that showed a second timing assumption. Under load the QUIC handshake can take longer than cloudwire.RaceDelay (250 ms), so the WebSocket won before any QUIC session reached the listener (QUIC was never tried, 2 of 800 runs).

Fix

  • The test's listener closes a session only once the box has opened its control stream. The box's dial has then succeeded, and the session ends before hello_ok, which is the case under test.
  • A test-only hook on the fake service, beforeWebSocket, holds each WebSocket upgrade until the link has put the WebSocket first on this network (bounded at 20 s). QUIC therefore always has its failed session first, however slow the handshake.

Evidence

Race detector, 8 parallel copies × -count=100 of this test (800 runs):

master this branch
a QUIC win moved "quic" first 1 in 200 (first batch) 0 in 800
QUIC was never tried 0 0 in 800 (2 in 800 before the hold)
listen udp …: bind: address already in use 2 in 800 1 in 800

The bind failure is a separate, local-only issue: the test listens for UDP on the same port number as the fake service's TCP listener, and another process's UDP socket can hold it. CI has never shown it, and this PR leaves it alone.

Whole internal/cloudlink package under the race detector passes (73 s). go vet passes.

Review

An independent review (Sonnet) found no bug. It delayed the server's first UDP read by 600 ms: with the hold, 10 of 10 runs passed under -race; without it, 6 of 6 failed with QUIC was never tried. It also broke the code on purpose: when rememberWinner ignores the demoted flag, the test fails 5 of 5; when demote is never called, it fails after 20 s without hanging. Small risk it noted: after the demotion, a WebSocket slower than 250 ms lets QUIC start second and fail again, which doubles the backoff; several in a row could pass the 20 s wait. Not seen.

Not changed

TestTheBoxFollowsAWrongShardCloseOnceAndKeepsTheHost failed once in CI (run 39908891473, no roster within 5s after the restart). It did not fail in 1,000 local runs under the same load, and I found no cause I could show. I did not change it.

🤖 Generated with Claude Code

TestAQUICSessionThatEndsBeforeHelloOKDemotesQUIC closed each QUIC
session right after the handshake. When that close beat the end of the
box's dial, the dial failed as on a path that blocks UDP, the WebSocket
won the race, nothing was demoted, and the later QUIC win moved QUIC
first. The service now closes once the box has opened its control
stream, so the dial has succeeded and the session ends before hello_ok.

Under load the QUIC handshake can also outlast the 250 ms race delay,
so the WebSocket won before QUIC was tried. A test-only hook holds the
fake service's WebSocket until QUIC is demoted on this network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frahlg frahlg added the no-changeset PR intentionally exempt from the changeset requirement (dev tooling / non-shipping) label Oct 25, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T11:36:09.306874Z 5278dd8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@frahlg
frahlg merged commit e723ced into master Oct 25, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changeset PR intentionally exempt from the changeset requirement (dev tooling / non-shipping)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant