Skip to content

simplify(diagnostics): share daemon compiler admission and bounds - #4546

Open
ScriptedAlchemy wants to merge 19 commits into
simplify/diagnostics-project-compiler-statusfrom
simplify/diagnostics-compiler-admission
Open

ScriptedAlchemy wants to merge 19 commits into
simplify/diagnostics-project-compiler-statusfrom
simplify/diagnostics-compiler-admission

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Why

Each producer had its own four-child compiler pool and effectively unlimited subprocess capture. Concurrent project producers did not share their child capacity.

What changed

  • Mount one compiler admission in the daemon generation and reuse the installed background CPU authority. Keep both permits until the existing child supervisor settles.
  • Replace eager per-run semaphore tasks with a bounded lazy stream. Close queued and running work with daemon admission.
  • Refuse checks after 20 minutes, 64 MiB stdout, or 1 MiB stderr. Preserve typed failures, partial findings, generation reuse, and cancellation on drop.

Scope

Fixes #4439. Parent link is #4545 on simplify/diagnostics-project-compiler-status.
The project-owned status cutover remains intact. Compiler-owned configuration evidence in #4440 and operation retirement in #4403 remain separate links. This change does not modify #4524.

Tradeoffs

The existing background CPU authority meters active compiler work. A daemon-owned semaphore additionally bounds live compiler children without another scheduler or a process-global registry. No RSS reduction or production budget calibration is claimed.

Blast Radius

Automatic project-pinned TypeScript diagnostics now share daemon admission. A stalled or excessive-output compiler fails rather than publishing clean evidence. Retrieval still reads generation-bound publications without running a compiler.

Verification

  • Bazel LSP compiler tests passed 25, including wait/run cancellation, child settlement, stdout and stderr refusal, and 1,000 valid findings. Formatting passed for all affected crates.
  • Bazel application producer/publication/store/availability tests passed 37. Three isolated producers performed nine checks under the shared four-child limit and retained partial findings.
  • Bazel supervisor and CPU tests passed 16. Daemon shutdown tests passed 3. Scoped Clippy passed for all three affected unit targets with warnings denied. The macOS test link reported existing unwind-size warnings.

Devin Review

Held ASK

Devin thread on scheduler.rs stale-lock admission (stale_lock_secs vs unverifiable owner) is a concurrency/lock-admission question. Left open for the operator: remove the obsolete setting, or carry the configured bound into the legacy-owner decision. Not implemented here.

Current-head CI failure repairs

The Linux fleet run 40437266717 completed 169 of 171 targets successfully. This follow-up repairs the two failed journeys without changing their budgets or dropping existing assertions.

The retained-route journey reopened a project after a sibling changed configuration. The old Work runtime rejected the same store authority before the compiler producer could register. Registration now refreshes policy, topology, routing, and evidence admission together, fenced by the captured incumbent and the canonical current configuration. Exact store, actor, grant, scope, and Work authority remain required. The retained fan-out recovery view receives the same admission update without replacing recovery owners or creating an owner cycle. The real journey now also checks stable grant identity, the current registration, refusal of stale or sibling registrations, and preservation of the current authority after refusal.

The pending-compiler fixture held its real compiler until the test could release it, but setup waited for lexical publication first. Source inspection establishes a reachable CPU admission/readiness cycle consistent with the small four-file CI fixture stalling after index build; this is not a measured occupancy claim. The fixture now uses the existing production opener, proves compiler startup before the unchanged typed pending assertions, releases the compiler, then requires graph readiness and the published TS4023 finding. Workload, polling bounds, and existing result checks are preserved.

Both focused commits passed pinned formatting, whitespace checks, normal pre-commit/commit-msg hooks, and independent source review. The complete native journeys were not run locally because of disk and memory constraints; the new head must pass hosted CI.

Configuration publication race follow-up

A changed retained Work runtime now verifies and publishes its configuration under the configuration store's existing canonical writer lane. The store reads durable current state through a registered immediate transaction, revalidates its retained authority, and invokes a synchronous registry transition before releasing the lane. The captured-incumbent comparison remains in place to prevent a delayed registrar from replacing a newer installed runtime.

The registry transition only reconciles an existing component; missing, reserved, closed, or fenced slots are refused. It performs no new owner build or asynchronous wait. Complete routing and recovery admission are refreshed together. The read-only transaction is dropped synchronously after the callback, using the existing rollback-before-writer-release contract, with no post-publication await.

Initial and unchanged retained pins preserve their existing lifetime. Work settings retain their daemon-restart semantics: a durable commit after successful publication is allowed, while a superseded candidate cannot claim to be current at publication.

The existing real route-eviction/compiler-status journey now pauses candidate B, commits C through the actual configuration surface, resumes B and requires its typed refusal with installed A unchanged, then installs and verifies C. Existing assertions and budgets remain intact. Pinned formatting, whitespace checks, normal commit hooks, and independent source/authority/lock-order review passed. The complete native journey was not executed locally and must pass hosted CI.

Replace the token-file task-lock protocol (random ownership token,
hard-link publication, staging files, PID/age reclamation, retrying
destructor unlink) with a nonblocking exclusive FileLease on a stable
`<key>.lease` inode held for the task and retained-settlement lifetime.
The lease file is never unlinked, so exclusion cannot split across inodes.

A live pre-lease holder (`<key>.lock` naming a live pid) still keeps the
key; dead or pid-reused owners' artifacts are retired and unverifiable
ownership is a typed error. The stale-lock timeout no longer applies.

Refs #4426

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 999c46d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

Independent source review at exact head e38c25d19a780067d4712268900854d7b7eaadd5 found two lifecycle gaps. This is not a runtime reproduction or a CI failure attribution.

  1. Windows discovery selects tsc.cmd. The existing bounded runner kills/reaps only the direct wrapper, so its Node descendant can outlive cancellation, deadline or output refusal. Releasing compiler/CPU permits after that return does not establish actual compiler termination.
  2. Dropping the async producer cancels its child token, but the detached blocking runner has no terminal join in project shutdown ownership. A project can report clean while queued or running compiler cleanup remains unsettled.

A separate correction worker is now implementing actual compiler supervision and project-owned terminal settlement with deterministic regressions. The dependent #4440 publication is held until the correction is reviewed and verified. No timeout increase, higher capacity, clean empty result, or branch rewrite is proposed. The existing local test report is preserved, but does not discharge these untested lifetime boundaries.

devin-ai-integration[bot]

This comment was marked as resolved.

ScriptedAlchemy and others added 3 commits October 25, 2026 22:46
Fleet logs show five complete 48,000-function generations take 53–65 seconds each. The default medium deadline kills the final refresh near 300 seconds despite steady successful publication. Declare the measured aggregate workload through the existing Cargo-to-Bazel size metadata, preserving every refresh, memory bound, and assertion.

The current #4542 run independently confirmed the same limit: 170/171 tests passed; complete generations published at 62, 123, 184, and 245 seconds before the final refresh was interrupted at 300.1 seconds with nominal memory usage.

Failure evidence: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/40248084891/job/121393230765
Resolve .github/workflows/ci.yml to master's ci-windows configuration,
which already carries this branch's --jobs=2 Windows concurrency bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy force-pushed the simplify/diagnostics-compiler-admission branch from 64b873e to 7f0d09f Compare October 26, 2026 07:16
ScriptedAlchemy and others added 4 commits October 26, 2026 02:45
The observer loop read the stop flag after accept() returned
WouldBlock. A connection the action completed between that accept and
the flag load was never counted, so a loaded runner saw 0 instead of 1.
Read the flag before accept(): a WouldBlock after a set flag proves the
backlog was already empty once the action returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(test): drain queued connections before HTTP observation stops
…k-lease

simplify(automation): hold one lifetime file lease per task key
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

ScriptedAlchemy added a commit that referenced this pull request Oct 27, 2026
@ScriptedAlchemy
ScriptedAlchemy force-pushed the simplify/diagnostics-compiler-admission branch 2 times, most recently from a449b1d to 5a46a14 Compare October 27, 2026 00:58
Route eviction can retain a project's Work runtime while a sibling commits
new configuration. Reopening the route previously refused that same store
authority before the compiler producer could register.

Refresh the policy, topology, routing and evidence admission together under
the captured runtime and canonical current-configuration fence. Keep exact
store, actor, scope, grant and Work authority checks. Refresh the retained
fan-out recovery view without replacing recovery owners or their signal.

Extend the real compiler route journey to verify stable grant identity,
current Work authority, stale and sibling rejection, and retained authority
after refusals. Preserve all existing assertions and budgets.

Validation: pinned rustfmt and diff checks passed; independently reviewed.
Native tests were not run locally because of disk and memory constraints.
Hosted CI must verify the repaired route journey.
The pending-producer fixture deliberately holds a compiler CPU admission
until the test releases it. Waiting for lexical readiness before that
release can leave the sorter waiting for the same process-wide CPU width.
The source-proven cycle is consistent with the failing CI snapshot: four
files were built in 60 ms before setup stalled at lexical publication.
This is not an occupancy measurement or a reason to raise a deadline.

Use the existing seeded production opener, prove that the held compiler
actually started, and preserve the typed pending-state assertions. Release
the process before requiring graph readiness, then require its published
TS4023 finding. Keep the fixture workload and all budgets unchanged.

Validation: pinned rustfmt and diff checks passed; independently reviewed.
Native tests were not run locally because of disk and memory constraints.
Hosted CI must verify the complete pending-to-published journey.
devin-ai-integration[bot]

This comment was marked as resolved.

Hold the canonical configuration writer lane while validating and
publishing a changed retained Work runtime. Refuse stale candidates
without replacing the incumbent or waiting for a new owner.

Exercise the real route journey with a newer durable configuration
committed while refresh is paused, then admit the current revision.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant