Skip to content

fix(writer-lock): reap a live writer stuck at ready:false past the timeout (fixes #2918) - #3041

Closed
johnhldiver-beep wants to merge 1 commit into
colbymchenry:mainfrom
johnhldiver-beep:fix/2918-daemon-ready-stale-lock
Closed

johnhldiver-beep wants to merge 1 commit into
colbymchenry:mainfrom
johnhldiver-beep:fix/2918-daemon-ready-stale-lock

Conversation

@johnhldiver-beep

Copy link
Copy Markdown

Root cause

A writer records ready: false in writer.pid when it acquires the lock, flipped to true by markWriterReady() once its first catch-up finishes. A daemon that acquires the lock but never finishes catch-up keeps a live pid in writer.pid with ready: false forever. Acquisition only treated a record as stale when the pid was dead, so a live-but-never-ready writer was indistinguishable from a healthy one: every query waited on it until the process was killed by hand (#2918, observed for ~28 h).

Fix

  • tryAcquireWriterLock() now also treats a live holder whose record is explicitly ready: false as stale once it is older than WRITER_READY_TIMEOUT_MS (10 minutes). It is cleared exactly like a dead pid: re-read the file, confirm the same pid, unlink, retry once. One stderr line names the pid, mode, lock path, and how many seconds it went without reporting ready.
  • CODEGRAPH_WRITER_READY_TIMEOUT_MS overrides the timeout (finite, > 0; otherwise the default).
  • writerLockHeldMessage() now names the start time (ISO) and says the writer has not reported ready, so the failure names the lock and pid instead of a bare timeout.
  • ready: true, a missing/zero startedAt, and a future startedAt are never stale, so legacy records without a ready field are not reaped.

No SIGTERM; #1978 and #2819 stay intact

Scope note (slightly narrower than the sketch)

tryAcquireWriterLock() is also used for rebuild.pid, and for writer.pid holders in rebuild/cleanup/handover modes. Those never call markWriterReady(), so ready: false is their normal state and age is not evidence of a wedge — a full re-index is legitimately not-ready for as long as it runs. Reaping is therefore limited to the readiness-publishing agent modes (daemon, direct, fallback). That keeps a long codegraph index from being stolen while covering every case in #2918.

Tests

Added 5 tests in __tests__/writer-lock.test.ts:

  1. age classification: stale at ready: false past the timeout; not stale when fresh, when ready: true, when startedAt is 0, or when null;
  2. CODEGRAPH_WRITER_READY_TIMEOUT_MS=1000 makes resolveWriterReadyTimeoutMs() return 1000 and a 2 s-old not-ready record stale (env restored afterwards);
  3. a live child process with an old ready: false record is stolen (the new pid owns the lock);
  4. a live child process within the timeout is not stolen (taken, child pid);
  5. the held message contains the pid, the lock path, and "has not reported ready".

Verification

  • npm ci, npx tsc --noEmit, npm run build: all pass.
  • __tests__/writer-lock.test.ts: 12/12.
  • E2E on the built dist with real processes: a stale live writer is reclaimed; a fresh writer is not stolen; a dead pid is still cleared by the old path; a pid-checked release leaves a successor's record intact.
  • Full suite: 774 test files passed / 18 skipped; 8884 tests passed / 418 skipped, exit 0. (An initial run hit an environmental 10 s hook timeout in react-router.test.ts amid too many open files from parallel watcher tests; that file passes 70/70 alone and the clean rerun passed.)
  • CHANGELOG (Unreleased → Fixes) and the README writer-lock FAQ updated.

Fixes #2918

…meout

A daemon that acquires writer.pid and then never finishes its first
catch-up holds the project with `ready: false` forever: the pid is alive,
so nothing clears the lock, and every query times out until the process
is killed by hand (colbymchenry#2918).

Treat an explicit `ready: false` record older than ten minutes as stale,
clear it pid-verified (re-read, confirm the same pid), and retry once, the
same way a dead pid is cleared. Override the threshold with
CODEGRAPH_WRITER_READY_TIMEOUT_MS. The held-lock message now names the
start time and says the writer has not reported ready.

The steal is deliberately narrow: only readiness-publishing MCP writers
(daemon/direct/fallback) can go stale this way, a ready writer is never
stolen however old it is (colbymchenry#1978), and the holder is never signalled
(colbymchenry#2819) — only the lock is cleared, so a wedged daemon is stopped through
the existing `codegraph daemon` controls and its eventual exit cannot
delete the successor's record.

Tests cover age classification, the env override, stealing a live stale
writer, refusing a fresh one, and the held message.
@colbymchenry

Copy link
Copy Markdown
Owner

Thank you @johnhldiver-beep for working on this! We retested the report on current main: a dead writer lock is reclaimed on its own, and with a live writer stuck at ready:false, new sessions answer within a few seconds, read-only, and say which process holds the lock. Most of that comes from #2795, which caps the wait on a daemon that never answers at 6 s and ships in the next release. Taking the lock from a writer that is still running would leave two writers watching one project, so we're keeping the current behavior and closing this one.

@johnhldiver-beep

johnhldiver-beep commented Oct 30, 2026 via email

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daemon can keep a dead writer lock, then run for hours with ready:false while every query times out

2 participants