Repository navigation
fix(writer-lock): reap a live writer stuck at ready:false past the timeout (fixes #2918) - #3041
Closed
johnhldiver-beep wants to merge 1 commit into
Closed
johnhldiver-beep wants to merge 1 commit into
johnhldiver-beep wants to merge 1 commit into
Conversation
…meout A daemon that acquires writer.pid and then never finishes its first catch-up holds the project with `ready: false` forever: the pid is alive, so nothing clears the lock, and every query times out until the process is killed by hand (colbymchenry#2918). Treat an explicit `ready: false` record older than ten minutes as stale, clear it pid-verified (re-read, confirm the same pid), and retry once, the same way a dead pid is cleared. Override the threshold with CODEGRAPH_WRITER_READY_TIMEOUT_MS. The held-lock message now names the start time and says the writer has not reported ready. The steal is deliberately narrow: only readiness-publishing MCP writers (daemon/direct/fallback) can go stale this way, a ready writer is never stolen however old it is (colbymchenry#1978), and the holder is never signalled (colbymchenry#2819) — only the lock is cleared, so a wedged daemon is stopped through the existing `codegraph daemon` controls and its eventual exit cannot delete the successor's record. Tests cover age classification, the env override, stealing a live stale writer, refusing a fresh one, and the held message.
Owner
|
Thank you @johnhldiver-beep for working on this! We retested the report on current main: a dead writer lock is reclaimed on its own, and with a live writer stuck at |
Author
|
Hi @colbymchenry — thanks for retesting this and for the clear
explanation. Understood: #2795 already covers the stuck-writer case,
and taking the lock from a running writer would leave two writers on
one project, so keeping the current behavior makes sense. I won't
re-file #2918. Thanks also for landing the #2885 and #2896 work
yourselves with credit — appreciated.
…On Fri, 30 Oct 2026 03:41:30 -0700, Colby Mchenry ***@***.***> wrote:
colbymchenry left a comment (colbymchenry/codegraph#3041)
Thank you @johnhldiver-beep for working on this! We retested the report on current main: a dead writer lock is reclaimed on its own, and with a live writer stuck at `ready:false`, new sessions answer within a few seconds, read-only, and say which process holds the lock. Most of that comes from #2795, which caps the wait on a daemon that never answers at 6 s and ships in the next release. Taking the lock from a writer that is still running would leave two writers watching one project, so we're keeping the current behavior and closing this one.
--
Reply to this email directly or view it on GitHub:
#3041 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
A writer records
ready: falseinwriter.pidwhen it acquires the lock, flipped totruebymarkWriterReady()once its first catch-up finishes. A daemon that acquires the lock but never finishes catch-up keeps a live pid inwriter.pidwithready: falseforever. Acquisition only treated a record as stale when the pid was dead, so a live-but-never-ready writer was indistinguishable from a healthy one: every query waited on it until the process was killed by hand (#2918, observed for ~28 h).Fix
tryAcquireWriterLock()now also treats a live holder whose record is explicitlyready: falseas stale once it is older thanWRITER_READY_TIMEOUT_MS(10 minutes). It is cleared exactly like a dead pid: re-read the file, confirm the same pid, unlink, retry once. One stderr line names the pid, mode, lock path, and how many seconds it went without reporting ready.CODEGRAPH_WRITER_READY_TIMEOUT_MSoverrides the timeout (finite, > 0; otherwise the default).writerLockHeldMessage()now names the start time (ISO) and says the writer has not reported ready, so the failure names the lock and pid instead of a bare timeout.ready: true, a missing/zerostartedAt, and a futurestartedAtare never stale, so legacy records without areadyfield are not reaped.No SIGTERM; #1978 and #2819 stay intact
codegraph daemoncontrols. BecausereleaseWriterLock()is pid-checked, the old daemon exiting later cannot delete the successor's record.ready: falsepast the timeout, and a holder still within the timeout is never touched.Scope note (slightly narrower than the sketch)
tryAcquireWriterLock()is also used forrebuild.pid, and forwriter.pidholders inrebuild/cleanup/handovermodes. Those never callmarkWriterReady(), soready: falseis their normal state and age is not evidence of a wedge — a full re-index is legitimately not-ready for as long as it runs. Reaping is therefore limited to the readiness-publishing agent modes (daemon,direct,fallback). That keeps a longcodegraph indexfrom being stolen while covering every case in #2918.Tests
Added 5 tests in
__tests__/writer-lock.test.ts:ready: falsepast the timeout; not stale when fresh, whenready: true, whenstartedAtis 0, or when null;CODEGRAPH_WRITER_READY_TIMEOUT_MS=1000makesresolveWriterReadyTimeoutMs()return 1000 and a 2 s-old not-ready record stale (env restored afterwards);ready: falserecord is stolen (the new pid owns the lock);taken, child pid);Verification
npm ci,npx tsc --noEmit,npm run build: all pass.__tests__/writer-lock.test.ts: 12/12.distwith real processes: a stale live writer is reclaimed; a fresh writer is not stolen; a dead pid is still cleared by the old path; a pid-checked release leaves a successor's record intact.react-router.test.tsamidtoo many open filesfrom parallel watcher tests; that file passes 70/70 alone and the clean rerun passed.)Fixes #2918