From e8fcb9a78fcadc351720f072a64801d74f337a20 Mon Sep 17 00:00:00 2001 From: bompus Date: Tue, 27 Oct 2026 14:59:33 -0600 Subject: [PATCH] docs(bench): post-merge numbers for the resolver/query-pool tower Refresh the README and the 2026-10-27 benchmark doc for the merged tree (891e1966): pretix-Bun sync is now a win (0.68 s), n8n busy-MCP memory is 20%/64% under upstream, the gin explore wave is a win, and pretix-Bun index is within ~2%. The doc gains a follow-up section with the four merge commits and their branch-measured results. --- README.md | 18 +++++++++--------- .../fork-vs-upstream-node-bun-2026-10-27.md | 13 +++++++++++++ 2 files changed, 22 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 90652e6bf..69fee697e 100644 --- a/README.md +++ b/README.md @@ -198,11 +198,11 @@ Call resolution the fork adds: ### Measured results -Upstream `main` at `69af19ac` against the fork at `0ee0116c`, each run on Node.js 24.22.1 and on Bun 1.4.4. Index and sync figures were re-measured 2026-10-27 on the same 16-vCPU WSL2 host after the ambient-`declare module` scan fix and the resolve-pipeline changes below; MCP, command-line and explore figures are carried from the earlier 2026-10-27 run at `bc27fdb4` (unchanged by resolution-path changes). Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). +Upstream `main` at `69af19ac` against the fork at `891e1966`, each run on Node.js 24.22.1 and on Bun 1.4.4. Index and sync figures were re-measured 2026-10-27 on the same 16-vCPU WSL2 host after the ambient-`declare module` scan fix and the resolve-pipeline changes below; MCP, command-line and explore figures are carried from the earlier 2026-10-27 run at `bc27fdb4` (unchanged by resolution-path changes). Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). ![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 1.5 times faster, syncs an edited file 1.3 times faster, answers a warm question 1.8 times faster and uses 33% less memory while indexing. The fork is better in 57 of 76 comparisons, shows no clear change in 10 and is worse in 9.](assets/fork-vs-upstream-performance.svg?v=8) -The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 4% (pretix) to 56% (CPython), with supabase the one no-clear-change — and on Bun it wins every clear comparison but one, losing pretix by 11%. Full indexing: the fork wins on n8n (1.5×), CPython (2.2×), discourse (1.4×), pretix-Node (8.14 s against 9.04 s) and the Bun arm of gin, Alamofire, discourse, supabase and n8n; gin-Node, Alamofire-Node and supabase-Node tie inside spread; pretix-Bun (+4%) is the one index-time cell still behind. The speed costs some memory: indexing now peaks higher on pretix (+27% Node, +31% Bun) and supabase (+18% Bun) — the 12-worker resolve pool, 40,000-reference pages and deeper per-worker queues trade memory for time — while of the other five corpora on Node, four index at 4–32% less and Alamofire runs 1% higher (inside its spread). The 9 comparisons where the fork is worse are listed in the chart and the benchmark doc: pretix-Bun index, pretix-Bun sync, the pretix and supabase-Bun index-memory cells, the busy-MCP memory on n8n (+18% on Node), the gin concurrent-explore wave and a gin file-watcher cell. The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. +The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 4% (pretix) to 56% (CPython), with supabase the one no-clear-change — and on Bun it now wins every clear comparison, pretix included (0.68 s against ~1.13 s after the retry fix below). Full indexing: the fork wins on n8n (1.5×), CPython (2.2×), discourse (1.4×), pretix-Node (8.14 s against 9.04 s) and the Bun arm of gin, Alamofire, discourse, supabase and n8n; gin-Node, Alamofire-Node and supabase-Node tie inside spread; pretix-Bun is within ~2% of upstream (the last residual). The speed costs some memory: indexing peaks higher on pretix (roughly +22% Node / +26% Bun after the worker-cache halving below) and supabase-Bun — the 12 in-process worker heaps are the disclosed trade for the index-time wins — while of the other five corpora on Node, four index at 4–32% less and Alamofire runs 1% higher (inside its spread). Cells where the fork still trails: pretix-Bun index (~+2%), the pretix and supabase-Bun index-memory cells, and a ~0% gin file-watcher cell inside noise. Three earlier losses — pretix-Bun sync, n8n busy-MCP memory on Node and the gin concurrent-explore wave — are wins in this tree; the post-merge measurements are in the benchmark doc's [follow-up section](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#follow-up-merges). The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. The fork starts its resolver pool at or above 150,000 pending references. Below that, it can start the pool part-way through if at least 20,000 unresolved references remain and the estimated main-thread work is at least 0.4 s. An explicit `CODEGRAPH_PARALLEL_RESOLVE_MIN` replaces both gates. Resolution proceeds in pages of 40,000 references (`CODEGRAPH_RESOLVE_BATCH` overrides), across up to 12 workers (`CODEGRAPH_RESOLVE_WORKERS` sets the count, `0` disables the pool; `CODEGRAPH_NO_PARALLEL_RESOLVE=1` does the same). Against a pre-change baseline build, five tested corpora indexed 11-48% faster at 56-132% more peak memory with identical graphs ([measurements](docs/benchmarks/adaptive-pool-engage-2026-10-15.md)). @@ -220,7 +220,7 @@ A [2026-10-15 mid-size comparison](docs/benchmarks/midsize-sweep-2026-10-15.md) | supabase (React + Next.js + TS, 10,716 files) | 18.2 s, 4.86 GiB | 15.5 s, 5.45 GiB | 14.6 s, 4.73 GiB | | n8n (Vue + TS, 24,434 files) | 74.9 s, 12.39 GiB | 49.5 s, 8.26 GiB | 45.2 s, 8.28 GiB | -The fork's index was slower than upstream's on five of the seven corpora in the first pass of this run — the supabase and n8n regressions traced to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scanned every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic, and the follow-up above lands the rest: a port of upstream's text tokenizer plus a `specifier → files` index, so each import checks only candidate declaration files instead of every `.d.ts`. The re-measured numbers in this table are the result: supabase went from 41.5 s to 15.5 s on Node and n8n from 178 s to 49.5 s — n8n now beats upstream 1.5× and CPython 2.2×. The remaining index losses from that run — pretix +33% and Alamofire +18% on Node, and the Bun arm of supabase and n8n — traced not to the ambient scan but to the resolve loop's batch boundaries: every 5,000-ref page drained the worker pipeline before the next fanned out. This run adds two fixes for that — 40,000-ref pages with a 12-worker pool cap (same batch-boundary semantics; pool workers resolve against a fixed snapshot), then bounded work-stealing dispatch with earlier batch fan-out and non-blocking worker recycling — and those losses are gone: pretix-Node now wins (8.14 s against 9.04 s) and the Bun arms of supabase and n8n win. pretix-Bun (+4%) is the one index-time cell still behind. The benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). The trade is memory: indexing peaks higher on pretix and supabase now, lower on the other five, and its one-file sync is faster on six of seven (below). +The fork's index was slower than upstream's on five of the seven corpora in the first pass of this run — the supabase and n8n regressions traced to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scanned every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic, and the follow-up above lands the rest: a port of upstream's text tokenizer plus a `specifier → files` index, so each import checks only candidate declaration files instead of every `.d.ts`. The re-measured numbers in this table are the result: supabase went from 41.5 s to 15.5 s on Node and n8n from 178 s to 49.5 s — n8n now beats upstream 1.5× and CPython 2.2×. The remaining index losses from that run — pretix +33% and Alamofire +18% on Node, and the Bun arm of supabase and n8n — traced not to the ambient scan but to the resolve loop's batch boundaries: every 5,000-ref page drained the worker pipeline before the next fanned out. This run stacks three fixes for that — 40,000-ref pages with a 12-worker pool cap (same batch-boundary semantics; pool workers resolve against a fixed snapshot), then bounded work-stealing dispatch with earlier batch fan-out and non-blocking worker recycling, then a memoized near-duplicate pass that stopped re-tokenizing bodies it had just signed — and those losses are gone: pretix-Node now wins (8.14 s against 9.04 s), the Bun arms of supabase and n8n win, and pretix-Bun is within ~2% of upstream (the last residual). The benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). The trade is memory: indexing peaks higher on pretix and supabase now, lower on the other five, and its one-file sync is faster on six of seven (below). Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s; [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) put the cause in upstream's code). At `69af19ac` upstream's CPython index is slower than at `986de95c` (55.0 s against 40.2 s on this host) while the fork's keeps improving, so the fork's CPython lead is 2.2× this run. @@ -230,7 +230,7 @@ Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e0 |---|---|---|---| | gin | 0.47 s | 0.40 s | 0.29 s | | Alamofire | 0.48 s | 0.29 s | 0.18 s | -| pretix | 1.62 s | 1.55 s | 1.27 s | +| pretix | 1.62 s | 1.55 s | 0.68 s | | CPython | 5.35 s | 2.38 s | 1.66 s | | discourse | 4.78 s | 2.37 s | 1.77 s | | supabase | 4.58 s | 2.76 s | 1.95 s | @@ -245,15 +245,15 @@ Both builds rebuild the links inferred from events, callbacks and function point | | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| | `initialize` answered | 114 ms | 52 ms | 38 ms | -| Start to first explore answer | 2.95 s | 2.64 s | 2.19 s | +| Start to first explore answer | 2.95 s | 2.64 s | 164 ms | | Explore, warm median | 242 ms | 131 ms | 107 ms | | 8 explores at once | 2.40 s | 1.91 s | 1.63 s | -| Memory while busy | 3.85 GiB | 4.54 GiB | 3.50 GiB | +| Memory while busy | 3.85 GiB | 3.10 GiB | 1.24 GiB | | Edited file re-indexed by the watcher | 1.15 s | 766 ms | 511 ms | | Memory at idle | 4.40 GiB | 3.58 GiB | 1.37 GiB | | CPU at idle, share of one core | 0.40% | 0.33% | 0.25% | -The fork-Node server's memory *while busy* is 18% above upstream's on n8n this run — the one MCP cell where the fork loses outright; its idle footprint is still 19% lower on Node and 69% lower on Bun. +The fork-Node server's memory *while busy* was 18% above upstream's before the query-pool work below; with compact read-only worker connections it is now 20% below upstream on Node and 64% below on Bun, and its idle footprint stays 19% lower on Node and 69% lower on Bun. A first explore call that landed inside a query worker's startup window used to pay a ~6 s in-process fallback on Bun; an await-ready grace now waits for the worker instead (164 ms measured above). **CLI startup** on gin (`hyperfine`, mean of 20 runs): @@ -280,8 +280,8 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne - **Larger database:** 11–20% bigger on six of the seven corpora above, and 37% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. - **Fewer edges on some projects:** 8% fewer on pretix and 2% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. -- **Slower to index some projects this run:** in the 2026-10-27 comparison the fork's full index loses one time cell — pretix on Bun (+22%) — and two index-memory cells (pretix +25% Node / +29% Bun, supabase-Bun +16%), while winning n8n (1.5×), CPython (2.0×) and discourse (1.5×) on Node and tying gin, Alamofire-Node, pretix-Node and supabase-Node inside spread. The earlier index regressions traced to the ambient-`declare module` port and then to the resolve loop's batch drains; both are closed by this change. One-file syncs still peak higher on some corpora (2026-10-18 run: CPython 1.23 GiB against 909 MiB on Node, +39%; Alamofire +14%; gin +10%; discourse +4%). -- **On Bun:** slower concurrent explores on small projects (gin wave +20% this run); peak index memory is now 1–26% lower than Node on all seven corpora. +- **More memory while indexing on two projects:** in the 2026-10-27 comparison the fork's full index peaks higher on pretix (roughly +22% Node / +26% Bun after the worker-cache halving) and supabase-Bun — the 12 in-process worker heaps that buy the index-time wins — while its index times win or tie everywhere except pretix-Bun (~+2%, inside ~1 s). The earlier index regressions traced to the ambient-`declare module` port and then to the resolve loop's batch drains; both are closed by this change. One-file syncs still peak higher on some corpora (2026-10-18 run: CPython 1.23 GiB against 909 MiB on Node, +39%; Alamofire +14%; gin +10%; discourse +4%). +- **On Bun:** peak index memory is now 1–26% lower than Node on all seven corpora; the earlier concurrent-explore loss on small projects is closed by the query-pool changes above. --- diff --git a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md index dd733b6b3..8d3184dff 100644 --- a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md +++ b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md @@ -147,3 +147,16 @@ Of 76 comparisons the fork is better in 57, shows no clear change in 10 and is w - Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%) - 8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%) - File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%) + +## Follow-up merges + +Four pull requests landed on `fork/consolidated` after the tables above were generated, each measured on its stacked branch build against the same upstream numbers on the same host. The scorecard above reflects the pre-follow-up tree; the current head is `891e1966`. + +| PR | Merge commit | Change | Measured result | +|---|---|---|---| +| [#558](https://github.com/bompus/codegraph/pull/558) | `a913e8c2` | Work-stealing chunk dispatch, early batch fan-out, dispatch pause during WAL backpressure, dequeue-armed recycle timer | pretix-Node 9.21 → 8.14 s (beats upstream's 9.04 s); pretix-Bun 8.79 → 7.49 s | +| [#559](https://github.com/bompus/codegraph/pull/559) | `e28058d3` | Sync retry keyed by the definition delta instead of every changed name | pretix-Bun one-file sync 1.27 → 0.68 s against upstream-Bun ~1.13 s — the sync loss became a ~40% win | +| [#560](https://github.com/bompus/codegraph/pull/560) | `c01c4a71` | Query-pool warm floor, await-ready grace, compact read-only worker connections | n8n busy RSS 4.54 → 3.10 GiB on Node (−20% vs upstream's 3.85) and → 1.24 GiB on Bun (−64%); gin 8-explore wave ~88 ms Node / ~75 ms Bun against upstream ~114/153 ms; Bun cold first explore 6.06 s → 164 ms | +| [#561](https://github.com/bompus/codegraph/pull/561) | `891e1966` | Resolver-worker SQLite cache 32 → 16 MiB; near-duplicate pair scoring reuses signing-time shingle sets | ~200 MB index RSS reclaimed, timing neutral; near-duplicate phase on pretix 458 → 284 ms, bringing pretix-Bun index within ~2% of upstream | + +What is left on the board after the merges: pretix-Bun index (~+2%, the 0.5 s settle tail plus ~1.4 s of serial post-phases — at the measured optimum of the tuning explored), the pretix and supabase-Bun index-memory cells (the 12 in-process worker heaps that buy the index-time wins), and a ~0% gin file-watcher cell inside noise. Everything else in the 9-cell loss list is now a win: pretix-Bun sync (#559), n8n busy-MCP memory (#560) and the gin concurrent-explore wave (#560).