Skip to content

fix(gate): take the host validation lock for heavy gates - #3900

Merged
thymikee merged 4 commits into
mainfrom
fix/unit-ci-host-lock
Oct 22, 2026
Merged

thymikee merged 4 commits into
mainfrom
fix/unit-ci-host-lock

Conversation

@thymikee

@thymikee thymikee commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Summary

pnpm gate unit-ci (and other whole-repo gates) did not take the host validation lock. Four worktrees ran it at once on a 12-core Mac; load reached ~1000 and every run hit false 5 s test timeouts.

pnpm gate <id> now runs heavy gates under the same lock that check:affected --run and check hold: same owner, waiter output, dead-owner reclaim, and 2-hour timeout. No second lock.

  • Locked (heavyGate in CHECK_CATALOG, hostLocked): unit, unit-ci, provider-integration (whole-repo suites), fuzz-parsers, mutation, mutation-affected, concurrency-torture (many workers/processes). Cheap gates stay unlocked.
  • Re-entry: a gate run inside a locked validation sees the inherited AGENT_DEVICE_VALIDATION_PARENT and runs directly, so nothing deadlocks.
  • CI: GITHUB_ACTIONS=true runs the gate unlocked (single-tenant runners). Unsupported hosts (Windows) also run unlocked.
  • acquireValidationLock takes an optional env, so nested-owner detection reads the same environment as the gate runner.
  • Docs: AGENTS.md and docs/agents/testing.md name the locked gates. To stay inside the guidance byte budgets, I removed a few duplicate sentences in testing.md/pull-requests.md.

The lock protocol (publish, activate, finish) and the signal-to-exit-code mapping have one owner in scripts/check-affected: runLockedCommand and abortOnTerminationSignal. Validation runs and gates both use them.

No linked issue.

Validation

  • Tested SHA: 1628711.
  • pnpm check:affected --run --head 1628711576 exits 0 with exactCommit: true (all runnable checks passed, including lint on the merged origin/main).
  • scripts/gate/host-locked.test.ts (7 tests) and host-lock.test.ts (4 tests) pass. They cover waiting, the exit code, nested no-deadlock, unlocked cheap and GitHub gates, a signal while waiting (143), a signal while the locked command runs (143, command group gone, lock and active.json removed), and runLockedCommand publishing and clearing ownership.
  • pnpm gate unit-ci waited behind another owner, then ran through the shared runLockedCommand path: exit 0, 16977 tests passed, lock released.
  • CI owns full coverage, integration, and native lanes.

🤖 Generated with Claude Code

View guided diff Turn on auto-fix

thymikee and others added 2 commits October 22, 2026 08:32
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Pass the environment to acquireValidationLock so nested-owner detection and
the gate runner read the same variables, skip the lock on unsupported hosts,
cover signal cancellation, and tighten the lock documentation.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.53 MB 5.53 MB 0 B
Package (unpacked) 5.53 MB 5.53 MB 0 B
Package (download) 1.68 MB 1.68 MB -2 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 42.2 ms 41.9 ms -0.3 ms
CLI --help 86.1 ms 88.8 ms +2.7 ms

@thymikee

Copy link
Copy Markdown
Member Author

The locked branch of executeGate copies the lock-activation protocol from executeValidationCommand almost line for line (https://github.com/callstack/agent-device/blob/cdc78ec/scripts/gate/host-locked.ts#L62), plus the SIGHUP/SIGINT/SIGTERM to 129/130/143 mapping from openValidationRun. Orphan reclaim in readOrphanCommand depends on one exact order: publish ownership, then activate, then finishCommand in finally. Two copies of that protocol can drift apart. If one changes the handshake, the kill grace or the active.json contents, the gate path can leave orphans that cannot be reclaimed, or start a check before ownership is published. The rule should be that every command run under the host validation lock goes through one function in scripts/check-affected that owns publish-then-activate-then-finish. Please extract runLockedCommand(lock, receipt, command, { cwd, env, signal, onStdoutChunk, onStderrChunk }) from executeValidationCommand, keep its log, heartbeat and receipt bookkeeping around it, and have executeGate call it. Move the signal-to-exit-code mapping into one shared helper that openValidationRun and executeGate both use.

Not blocking, and you can take or leave these: docs/agents/testing.md now drops unrelated rules to fit the byte budget (no lock wrappers or lock-file deletion, never call a failing gate passed, read required-check status on the PR, initial/final checks do not sandbox the checkout, and the authorized incomplete draft line in pull-requests.md), so shortening the new sentences or rewording the existing "Both broad commands acquire the same host lock" line would keep them; and cleanEnv in scripts/gate/host-locked.test.ts spreads extra twice, where only the second spread has any effect.

Would it be simpler to keep the hostLocked trait and requiresHostLock, replace the roughly 50-line locked branch with a call to the extracted runLockedCommand and signal helper, and let host-locked.ts shrink to the predicate, acquire, run and release? The lock protocol would then have one owner, and no ADR is needed.

This review covers cdc78ec. I read the change but did not run the new tests or your locked pnpm gate unit-ci run. No test covers a signal that arrives while the locked command is running, as opposed to while waiting. The Lint, Typecheck & Package failure is in oxlint on packages/platform-apple/src/snapshot-route.test.ts:1003, the 1000-line cap tracked in #3890 on main. It is likely unrelated, since this PR only touches scripts/ and docs/. There are no conflicts. Before merge, the locked run needs to go through the shared helpers, and CI lint stays red until #3890 lands.

thymikee and others added 2 commits October 22, 2026 10:09
* origin/main:
  feat(test-app): add a Bluesky-depth heavy feed screen (#3884)
  fix(android): read a child that vanished mid-read as gone (#3901)
  feat(ios): route iPad Simulator snapshots through the host AX bridge (#3895)
Move the publish, activate, finish protocol and the signal-to-exit-code
mapping into scripts/check-affected so validation runs and heavy gates
share one owner. Restore the testing and pull-request rules that the
previous revision dropped, and cover a signal that arrives while the
locked command is running.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member Author

Addressed in 1628711 (also merged origin/main, which carries the snapshot-route line-cap fix).

Blocking: one owner for the lock protocol

  • runLockedCommand(lock, receipt, command, { cwd, env, signal, onStdoutChunk, onStderrChunk }) now lives in scripts/check-affected/host-lock.ts. It owns publish → activate (stdin: 'run\n') → finishCommand in finally, plus the detached/kill grace/AGENT_DEVICE_VALIDATION_PARENT env.
  • executeValidationCommand keeps its log, heartbeat and receipt bookkeeping around it. finishCommand now runs inside the helper, before the report collection.
  • Signal mapping moved into abortOnTerminationSignal(controller) (scripts/check-affected/termination.ts), used by openValidationRun and executeGate.
  • host-locked.ts keeps the hostLocked trait and requiresHostLock, and shrinks to predicate / acquire / run / release.
  • New tests: a signal while the locked command runs (host-locked.test.ts: command stopped, exit 143, lock and active.json gone). A direct runLockedCommand test checks that ownership is published with the receipt and pid while it runs and cleared after (host-lock.test.ts).

Non-blocking

  • docs/agents/testing.md: the lock-wrapper / lock-file rule, "never call a failing gate passed", required-check status, and the initial/final checks line are restored. The new text is a single reworded line: "Both broad commands and heavy pnpm gate ids acquire the same host lock." pull-requests.md regains the authorized incomplete draft line. AGENTS.md keeps its original lock sentence, and its parallel-worker bullet now reads "...and heavy pnpm gate ids share one lock."
  • cleanEnv no longer spreads extra twice.

@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at 1628711. The findings from the earlier review at cdc78ec (#3900 (comment)) are fixed, and I found nothing new in the code. I read the new host-lock and gate tests but did not run them locally. CI shows 18 checks with none failing at this head. There are no conflicts. Not blocking, and you can take or leave it: the PR body Validation section still cites cdc78ec, 8 files, 6 tests and removed sentences in testing.md and pull-requests.md, which the later commits reverted. It would help to update it to 1628711, 11 files, 7 gate tests plus the runLockedCommand test, and to re-run pnpm gate unit-ci through the extracted helper if you cite a locked run, since the cited run predates runLockedCommand. Nothing else must happen before merge.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Oct 22, 2026
@thymikee
thymikee merged commit 1778e24 into main Oct 22, 2026
18 checks passed
@thymikee
thymikee deleted the fix/unit-ci-host-lock branch October 22, 2026 09:45
@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-22 09:46 UTC

thymikee added a commit that referenced this pull request Oct 22, 2026
Brings in #3892, which fixes the iOS simulator logs stop hang.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* origin/main:
  perf(ios): read each resolved text input through one snapshot (#3882)
  fix(gate): take the host validation lock for heavy gates (#3900)
  fix(logs): keep an early iOS simulator logs stop from hanging the session (#3892)
  fix(apple): clean the runner derived path the runner uses (#3897)
  test(provider-scenarios): open deadline scenarios without the test IME (#3903)
  test(host-kit): stub os.homedir for the exec-trace late-close test (#3898)
thymikee added a commit that referenced this pull request Oct 22, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ps/ios-suite:
  fix(perf): stop the log stream right after it starts
  perf(ios): read each resolved text input through one snapshot (#3882)
  fix(gate): take the host validation lock for heavy gates (#3900)
  fix(logs): keep an early iOS simulator logs stop from hanging the session (#3892)
  fix(apple): clean the runner derived path the runner uses (#3897)
  test(provider-scenarios): open deadline scenarios without the test IME (#3903)
  test(host-kit): stub os.homedir for the exec-trace late-close test (#3898)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant