Skip to content

fix(code-index): bound cold builds and resume refused publication - #4510

Merged
ScriptedAlchemy merged 12 commits into
fix/lcm-retry-transient-summarizer-failuresfrom
fix/cold-pass-memory-retry-loop
Oct 25, 2026
Merged

ScriptedAlchemy merged 12 commits into
fix/lcm-retry-transient-summarizer-failuresfrom
fix/cold-pass-memory-retry-loop

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Merge #4478 first, followed by #4526, #4510, #4524, #4489, #4485, #4527, #4528, #4530, #4531, #4529, #4534, and #4541.

This PR is number 3 of 13, based on #4526. Merge #4524 next.

#4516 has already merged into this branch.

Use merge commits. After merging a parent PR, change its child's base to master before merging the child. Auto-merge is disabled.

A cold build that exceeds the resident-memory limit now waits until enough memory is available for the full reconstruction. Cold publication also avoids holding a decoded repository alongside its encoded output.

The existing memory authority tracks allocations owned by the operation. It retains the full memory requirement after a refused build unwinds. A prepared publication can resume without repeating completed extraction. Worker admission measures current pressure and reserves memory for the installed worker count. Before activation, publication verifies every referenced digest and exact extent, including inherited segments.

Cold builds now reserve memory before replacement allocations and compact copies. They limit temporary carry data to one file, reuse authenticated unchanged segments, release resolution facts before building the graph, and defer projection rows. Symbol-name lookup and resolution-index page construction request memory in batches. Digest serialization streams explicitly ordered objects and preserves the exact bytes.

Fixes #4505.

Validation

Before the final integration of ancestor commits, the following tests passed:

  • Runtime-core passed 542 tests.
  • Index unit tests passed 357 tests.
  • Index integration passed 216 tests.
  • Scheduler runtime passed 664 tests, with 2 existing ignored tests.
  • The reporting integration regression passed separately.

Additional checks passed 96 retention tests, 4 resident-accounting tests, 75 query integration tests, and 27 evaluator tests. Changed benchmark callers compiled. Scoped Clippy and formatting checks passed.

The ignored scale fixture with 6,600 files and 422,400 symbols published under the unchanged 1 GiB limit in 241.64 seconds. Peak admission occupancy was 805,367,808 bytes. Peak RSS was 850,829,312 bytes. The run did not decode the full generation eagerly.

At stacked head 3e10b5fbb0a5d49b59c7fbe7216e43c6f10b0a2d, with nightly-2026-10-25 and inherited dependencies, fresh execution passed 542 runtime-core, 357 index unit, 223 index integration, and 664 scheduler runtime tests. The 2 existing ignored tests remained ignored. All three affected formatting targets passed. Bazel invocation 27ebabd4-4bd2-4576-83c9-64be264208cc exited with code 0.

The scale result covers cold publication. It does not establish a bound for retained-text serving or every allocator. Existing limits in decoder preallocation and measurement of native, parser, and Go internal workspaces remain. No budget or timeout increased.

Publication cancellation correction

A source epoch changing inside final durable commit surfaced as CompareAndSwap and could park the project. The scheduler now restores cancellation only when the attempt control confirms it, preserving genuine pointer conflicts and memory-refusal precedence. Fix: dab12b4.

Focused regression BEFORE: 0 passed / 1 failed (fa27d3a7-0aa4-4164-a350-6f943e556f4c). AFTER publication-store module: 45 passed plus formatting (836e4ba5-46ab-4200-aa71-af891b3acd38). Full daemon suite: 79 passed, including all three CI failures (711bbf0f-9197-456f-be73-3746d23f04bb). Scoped Clippy passed (bfa5119d-e483-4697-9846-3d10e9ff3330). These ran on the #4510 correction before stack propagation. The separate refresh-memory journey passed with existing phase tracing in 158.23s (a7c28c80-c892-4279-ba35-5d4beed34574), then passed again after removing all investigative edits: 2 tests, 0 failures, 157.87s (fc944727-2cd0-4620-b7bc-86cea6ea5047). Both completed cold indexing and all four refreshes under the unchanged 6 GiB limit and 300-second target timeout. The slower CI execution was not reproduced; no performance fix or timeout increase is claimed.

@changeset-bot

changeset-bot Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: dab12b4

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

ScriptedAlchemy and others added 2 commits October 24, 2026 21:46
A cold build the resident-memory checkpoint stops gives back what it
built. The latch clearing behind that release must not wake its own
retry; memory another holder gives back must.

Refs #4505

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cold reconcile the resident-memory checkpoint stops drops what it
built. RSS falls under the low watermark, the latch clears, and the
admission headroom epoch advances. Both the worker loop and the
headroom watcher read that epoch as memory given back, so the pass
rebuilt and was refused again on every maintenance sample.

Record where measured RSS stood when such a pass began. Admission
headroom now retries it only once the process measures below that
point, and the bound registers with the resident-memory authority so a
later sample under it still advances the epoch. Ledger refusals and
passes admitted while already over the watermark keep retrying on the
latch as before; owner releases by others still wake the retry.

Fixes #4505

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy force-pushed the fix/cold-pass-memory-retry-loop branch from f083b0f to 3434b35 Compare October 24, 2026 21:46
ScriptedAlchemy and others added 5 commits October 25, 2026 01:13
JSON and TOML documents whose keys read as high-entropy tokens are
withheld whole today. Pin the target behavior: the key becomes a
placeholder unique within its object, the values under it are redacted,
and passphrase-shaped secrets stay redacted in every position.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A JSON or TOML object key that trips a credential or entropy detector
withheld the whole document, because the sanitizer had no way to rewrite
a key. Long identifiers used as keys (schema $defs type names, benchmark
target names, path keys) score above the entropy floor, so files such as
dashboard-contracts.schema.json were never indexed.

The span-reporting JSON and TOML parsers already give each key's byte
range. Replace a flagged key with a placeholder unique within its object
(TraceDecay-redacted-key-N), redact every scalar under it, and keep the
rest of the document byte-exact. Detection is unchanged: no token is
exempted. A document is still withheld when the key has no span (YAML)
or the rewritten text no longer parses as its format.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy changed the base branch from fix/privacy-redact-keys-in-place to fix/lcm-retry-transient-summarizer-failures October 25, 2026 01:32
@ScriptedAlchemy ScriptedAlchemy changed the title fix(code-index): stop a refused cold pass from waking its own retry fix(code-index): bound cold construction and retain refused publication Oct 25, 2026
@ScriptedAlchemy ScriptedAlchemy changed the title fix(code-index): bound cold construction and retain refused publication fix(code-index): bound cold builds and resume refused publication Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

code-index: a cold pass refused for memory retries forever and never indexes

1 participant