diff --git a/README.md b/README.md index 7f200661a2..8a8e97c3e5 100644 --- a/README.md +++ b/README.md @@ -80,13 +80,13 @@ Session search uses `.codegraph/sessions-v2.db`. The first session search import ### What the fork adds -Compared with upstream `main` at `1d3619d6` (after v1.6.2). Each item here and in the dispatch and framework lists below was checked against upstream's tree at that commit. +Compared with upstream `main` at `69af19ac` (after v1.6.2). Each item here and in the dispatch and framework lists below was checked against upstream's tree at that commit. | Feature | Upstream | Fork | What it does | |---|:-:|:-:|---| | Session search (`codegraph sessions`, `codegraph_sessions`) | — | ✓ | Searches this project's earlier Claude Code, Codex, Cursor/T3, OpenCode, AGY, Devin and Grok transcripts and its git commit messages, so an agent can find what a past session decided. `codegraph_explore` also names the sessions that mentioned the symbols it returns. `"sessions": false` in `codegraph.json` turns it off. Experimental: `projects` / `--project` searches up to five projects' histories in one call. | | Markdown indexing | — | ✓ | Headings, sections, tables and links become graph nodes; a documentation question gets the matching section. Explicit code-file pins remain eligible in mixed documentation/code requests. | -| Missing names reported | — | ✓ | After a complete scan of indexed source without skipped files, `codegraph_explore` lists requested names it cannot find, helping an agent catch a guessed name. | +| Missing names reported | Reports them | Reports them, and proves absence | Upstream's note says a requested name is not in the index and the defining file may be unindexed. The fork keeps that note and additionally lists a name only when a bounded (300 ms) scan of every hand-written source file — plus file stems and bindings — completes without finding it as any word, so a listed name is absent from the project outright, not just unindexed. An incomplete scan lists nothing rather than guess. | | Requested-source limits reported | — | ✓ | Explore reports requested files that did not receive pin priority within the file limit, unexamined path-like references beyond its bounded scan, and uncovered indexed continuation ranges after a capped file gather. Existing output limits still apply. | | Local callee source with file pins | — | ✓ | When a query names functions in pinned files, `codegraph_explore` prioritizes local callee bodies through two call or callback hops, within the file and output budgets. | | Quoted prose source | — | ✓ | Quoted spans of three or more words find script strings and template text, ignoring case and punctuation, through a capped source scan. No re-index needed. | @@ -198,11 +198,11 @@ Call resolution the fork adds: ### Measured results -Upstream `main` at `986de95c` against the fork at `633fc82c`, each run on Node.js 24.22.0 and on Bun 1.4.3. The fork at `633fc82c` lacks one upstream commit, `986de95c` (an import through `@/` or `~/` in a Vue project links the module's file), so the arms differ in it. Measured 2026-10-18 on a 16-vCPU WSL2 host over seven corpora, with the arms in mirrored order; index figures are the median of two runs and sync figures the median of four. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer. With `CODEGRAPH_ALLOW_UNSAFE_NODE=1` it runs; those numbers, the method and the graph sizes are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-18.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-18.md), which also compares this run with the 2026-10-14 one. The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). +Upstream `main` at `69af19ac` against the fork at `bc27fdb4`, each run on Node.js 24.22.1 and on Bun 1.4.4. Measured 2026-10-27 on a 16-vCPU WSL2 host over seven corpora, with the arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). -![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 1.3 times faster, syncs an edited file 3.0 times faster, answers a warm question 3.4 times faster and uses 37% less memory while indexing. The fork is better in 65 of 74 comparisons, about the same in 4 and worse in 5.](assets/fork-vs-upstream-performance.svg?v=4) +![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 2.4 times slower, syncs an edited file 2.8 times faster, answers a warm question 1.8 times faster and uses 49% less memory while indexing. The fork is better in 56 of 76 comparisons, shows no clear change in 5 and is worse in 15.](assets/fork-vs-upstream-performance.svg?v=5) -The fork's one-file sync is faster than upstream's on every corpus where upstream's sync succeeds: 14% (gin) to 67% (n8n) on Node and 5% (pretix) to 65% (n8n) on Bun, 28% on CPython on Node and 21% on Bun. On Node the fork indexes CPython 17% and n8n 22% faster, gin and Alamofire 4% faster, ties on discourse and supabase (within 3%), and is slower on pretix (+13%); it uses less memory while indexing on five of the seven corpora, and more on Alamofire (+15%) and pretix (+11%). The five comparisons where the fork is worse are listed in the chart: indexing pretix (+13% on Node, +20% on Bun) and index memory on Alamofire (+15% on Node) and pretix (+11% on Node, +6% on Bun). Upstream's one-file sync fails on supabase with `FOREIGN KEY constraint failed` ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565)), so the two supabase sync comparisons are left out of the 74. The chart does not compare peak memory during a one-file sync, where the fork is higher on CPython (1.23 GiB against 909 MiB on Node, +39%; +51% on Bun), Alamofire (+14% on Node), gin (+10%) and discourse (+4% on Node, +8% on Bun). The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-18.md#every-result-at-a-glance) in one heatmap. +The fork's one-file sync is faster than upstream's on six of the seven corpora — 11% (gin) to 64% (n8n) on Node, 19% (supabase, where upstream's spread is wide) to 57% (discourse) on Bun — with supabase on Node the one no-clear-change. Full indexing is the fork's weak surface this run: it is slower on five of seven corpora (supabase 2.3× and n8n 2.4× on Node; the supabase-resolution section below explains why), ties nowhere outside spread, and wins only on CPython (1.6× on Node). It uses less memory while indexing on all seven (18–49%). The 15 comparisons where the fork is worse are listed in the chart and the benchmark doc: six index-time cells, the busy-MCP memory on n8n (+18% on Node), the gin concurrent-explore wave, and small index-time cells on gin and Alamofire. The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. The fork starts its resolver pool at or above 150,000 pending references. Below that, it can start the pool part-way through if at least 20,000 unresolved references remain and the estimated main-thread work is at least 0.4 s. An explicit `CODEGRAPH_PARALLEL_RESOLVE_MIN` replaces both gates. Against a pre-change baseline build, five tested corpora indexed 11-48% faster at 56-132% more peak memory with identical graphs ([measurements](docs/benchmarks/adaptive-pool-engage-2026-10-15.md)). @@ -212,31 +212,31 @@ A [2026-10-15 mid-size comparison](docs/benchmarks/midsize-sweep-2026-10-15.md) | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin (Go, 119 files) | 1.06 s, 577 MiB | 1.02 s, 395 MiB | 1.24 s, 293 MiB | -| Alamofire (Swift, 129 files) | 1.44 s, 830 MiB | 1.39 s, 953 MiB | 1.19 s, 643 MiB | -| pretix (Python + JS, 1,472 files) | 8.36 s, 2.94 GiB | 9.40 s, 3.25 GiB | 9.27 s, 2.50 GiB | -| CPython (C + Python, 3,708 files) | 40.2 s, 6.25 GiB | 33.3 s, 4.10 GiB | 34.0 s, 3.61 GiB | -| discourse (Ruby + JS, 20,277 files) | 20.7 s, 3.86 GiB | 20.8 s, 2.95 GiB | 18.6 s, 2.23 GiB | -| supabase (React + Next.js + TS, 10,716 files) | 18.6 s, 4.66 GiB | 18.7 s, 3.82 GiB | 18.9 s, 3.34 GiB | -| n8n (Vue + TS, 24,434 files) | 84.5 s, 9.97 GiB | 66.3 s, 6.30 GiB | 66.0 s, 6.07 GiB | +| gin (Go, 119 files) | 0.96 s, 559 MiB | 1.09 s, 392 MiB | 1.06 s, 296 MiB | +| Alamofire (Swift, 129 files) | 1.35 s, 879 MiB | 1.62 s, 632 MiB | 0.99 s, 664 MiB | +| pretix (Python + JS, 1,472 files) | 9.04 s, 3.29 GiB | 12.5 s, 2.71 GiB | 12.4 s, 2.42 GiB | +| CPython (C + Python, 3,708 files) | 55.0 s, 7.15 GiB | 33.8 s, 4.01 GiB | 33.5 s, 3.48 GiB | +| discourse (Ruby + JS, 20,277 files) | 19.6 s, 4.07 GiB | 22.2 s, 2.82 GiB | 20.8 s, 2.40 GiB | +| supabase (React + Next.js + TS, 10,716 files) | 18.2 s, 4.86 GiB | 41.5 s, 3.59 GiB | 40.6 s, 3.76 GiB | +| n8n (Vue + TS, 24,434 files) | 74.9 s, 12.2 GiB | 3:00, 6.27 GiB | 2:57, 6.33 GiB | -The Alamofire row includes the resolver change above, which starts the pool part-way through a mid-size index and costs memory: the fork peaks at 953 MiB against upstream's 830 MiB (+15%) while indexing 4% faster. The [2026-10-15 mid-size comparison](docs/benchmarks/midsize-sweep-2026-10-15.md) measured the same pattern on a different build pair (1.44 s and 1,042 MiB against 1.38 s and 791 MiB). +The fork's index is slower than upstream's on five of the seven corpora this run — clearly so on the two TypeScript-heavy monorepos (supabase 2.3×, n8n 2.4×) and pretix (+38%). The supabase and n8n regressions trace to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scans every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic — without it the supabase index never finished — but the remaining scan is still per-reference × per-file work upstream does once per candidate name. The fix for the residual gap is tracked as follow-up work; the benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). In return the fork's index uses 18–49% less memory on every corpus, and its one-file sync is faster on six of seven (below). -Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s; [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) put the cause in upstream's code). At `986de95c` it is back near those earlier times (pretix 8.4 s, CPython 40 s), so the fork's lead on those two corpora in the 2026-10-12 run (1.7 and 2.3 times) is gone on pretix, where the fork is 13% slower, and is 17% on CPython. +Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s; [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) put the cause in upstream's code). At `69af19ac` upstream's CPython index is slower than at `986de95c` (55.0 s against 40.2 s on this host) while the fork's is unchanged, so the fork's CPython lead is 1.6× this run. **One-file sync** (edit one file, `codegraph sync`): | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin | 0.47 s | 0.41 s | 0.34 s | -| Alamofire | 0.68 s | 0.55 s | 0.44 s | -| pretix | 2.55 s | 2.19 s | 2.00 s | -| CPython | 6.47 s | 4.69 s | 4.62 s | -| discourse | 5.88 s | 2.74 s | 2.54 s | -| supabase | fails (exit 1) | 3.17 s | 2.73 s | -| n8n | 13.1 s | 4.37 s | 3.79 s | +| gin | 0.47 s | 0.42 s | 0.32 s | +| Alamofire | 0.48 s | 0.32 s | 0.21 s | +| pretix | 1.62 s | 1.07 s | 0.81 s | +| CPython | 5.35 s | 2.42 s | 1.70 s | +| discourse | 4.78 s | 2.31 s | 1.73 s | +| supabase | 4.58 s | 3.37 s | 2.34 s | +| n8n | 11.6 s | 4.18 s | 3.41 s | -Upstream's sync of supabase exits with `FOREIGN KEY constraint failed` after a change to a file that lazy-loads a workspace package ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565)); the fork syncs it. +Upstream's supabase sync works in this run — the `FOREIGN KEY constraint failed` seen in every prior run ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565)) was fixed upstream on 2026-10-19. Its two passes differed widely on Node (1.05 s and 8.1 s); the 4.58 s figure is the median of four, and the spread means that comparison counts as no clear change rather than a fork win. Both builds rebuild the links inferred from events, callbacks and function pointers after a sync, so a sync ends with the graph a full index would build ([colbymchenry/codegraph#1988](https://github.com/colbymchenry/codegraph/issues/1988)). Upstream reruns every inference pass inside each sync that touches an inferred link; the fork reruns the same passes a full index runs once edits pause. `CODEGRAPH_SYNC_RESYNTHESIS=0` turns the fork's rebuild off, trading it for stale inferred links. @@ -244,23 +244,26 @@ Both builds rebuild the links inferred from events, callbacks and function point | | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| Start to first explore answer | 3.60 s | 2.78 s | 2.48 s | -| Explore, warm median | 600 ms | 177 ms | 187 ms | -| 8 explores at once | 3.45 s | 2.30 s | 2.29 s | -| Memory while busy | 4.30 GiB | 3.83 GiB | 3.01 GiB | -| Edited file re-indexed by the watcher | 1.32 s | 835 ms | 669 ms | -| Memory at idle | 4.88 GiB | 2.15 GiB | 1.23 GiB | -| CPU at idle, share of one core | 0.44% | 0.35% | 0.48% | +| `initialize` answered | 114 ms | 52 ms | 38 ms | +| Start to first explore answer | 2.95 s | 2.64 s | 2.19 s | +| Explore, warm median | 242 ms | 131 ms | 107 ms | +| 8 explores at once | 2.40 s | 1.91 s | 1.63 s | +| Memory while busy | 3.85 GiB | 4.54 GiB | 3.50 GiB | +| Edited file re-indexed by the watcher | 1.15 s | 766 ms | 511 ms | +| Memory at idle | 4.40 GiB | 3.58 GiB | 1.37 GiB | +| CPU at idle, share of one core | 0.40% | 0.33% | 0.25% | + +The fork-Node server's memory *while busy* is 18% above upstream's on n8n this run — the one MCP cell where the fork loses outright; its idle footprint is still 19% lower on Node and 69% lower on Bun. **CLI startup** on gin (`hyperfine`, mean of 20 runs): | Command | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| `codegraph --version` | 64 ms | 33 ms | 24 ms | -| `codegraph status` | 176 ms | 130 ms | 87 ms | -| `codegraph explore ""` | 231 ms | 121 ms | 91 ms | +| `codegraph --version` | 73 ms | 35 ms | 21 ms | +| `codegraph status` | 166 ms | 114 ms | 63 ms | +| `codegraph explore ""` | 220 ms | 117 ms | 71 ms | -**Node or Bun.** The fork produces the same node and edge counts on both, and the full test suite passed on Node.js 24.21.0, Node.js 26.10.0 and Bun 1.4.2 at `48903f5` (`npm run test:bun`; one test is skipped under Bun for [oven-sh/bun#42891](https://github.com/oven-sh/bun/issues/42891)). Since then the suite has grown, and the Bun run skips 14 more tests that need what Bun does not provide: patching a builtin so the library sees it (`syncBuiltinESMExports` is a no-op, [oven-sh/bun#42616](https://github.com/oven-sh/bun/pull/42616)) and an emptied `PATH` blocking a spawn ([oven-sh/bun#44948](https://github.com/oven-sh/bun/issues/44948)). Each skip is measured at run time, so the test returns when Bun supports it. On Bun, peak index memory is 4–32% lower than on Node (n8n 4%, supabase 13%), the idle MCP server holds 34–43% less memory, and commands start faster. Index times range from 20% slower (gin, a 1.0 s index) to 14% faster (Alamofire) than Node's, and sync times are 1% (CPython) to 19% (Alamofire) faster. Bun has two costs. Eight concurrent explores take 41% longer on gin (210 ms against 149 ms) and show no clear change on n8n (2.29 s against 2.30 s); contention between worker threads reading through Bun's SQLite is the suspected cause ([oven-sh/bun#44084](https://github.com/oven-sh/bun/issues/44084), [#44187](https://github.com/oven-sh/bun/issues/44187)). Idle CPU is 0.13 to 0.26 points of one core above Node's (0.40% against 0.14% on gin, 0.48% against 0.35% on n8n), a few milliseconds of CPU per second, and a separate profile did not reproduce the order on n8n. +**Node or Bun.** The fork produces the same node and edge counts on both, and the full test suite passed on Node.js 24.21.0, Node.js 26.10.0 and Bun 1.4.2 at `48903f5` (`npm run test:bun`; one test is skipped under Bun for [oven-sh/bun#42891](https://github.com/oven-sh/bun/issues/42891)). Since then the suite has grown, and the Bun run skips 14 more tests that need what Bun does not provide: patching a builtin so the library sees it (`syncBuiltinESMExports` is a no-op, [oven-sh/bun#42616](https://github.com/oven-sh/bun/pull/42616)) and an emptied `PATH` blocking a spawn ([oven-sh/bun#44948](https://github.com/oven-sh/bun/issues/44948)). Each skip is measured at run time, so the test returns when Bun supports it. On Bun 1.4.4, fork index times match Node's within each corpus's own spread except Alamofire (−39%), sync times run 19–36% faster, and commands start faster (`--version` 21 ms against 35 ms). The idle MCP server holds 29% (gin) to 62% (n8n) less memory on Bun. Bun has one cost this run: eight concurrent explores on gin took 20% longer (161 ms against 134 ms); on n8n the wave runs 15% faster (1.63 s against 1.91 s). Peak index memory on Bun is 11–25% lower than Node on four corpora and 2–6% higher on Alamofire, supabase and n8n — no uniform direction. Measured separately: @@ -275,10 +278,10 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne ### What it costs -- **Larger database:** 11–21% bigger on the other six corpora above, and 37% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. -- **Fewer edges on some projects:** 5% fewer on pretix and 2% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. -- **Slower to index some projects:** against upstream's current build in the 2026-10-18 run, pretix took 9.40 s to index against 8.36 s (+13% on Node, +20% on Bun) and peaked at 3.25 GiB against 2.94 GiB on Node (+11%; +6% on Bun); Alamofire peaked at 953 MiB against 830 MiB on Node (+15%) after the resolver pool change. A one-file sync peaks higher on CPython (1.23 GiB against 909 MiB on Node, +39%), Alamofire (+14%), gin (+10%) and discourse (+4%). A warm question on gin takes 17 ms against 25 ms on Node and 19 ms against 21 ms on Bun. -- **On Bun:** slower concurrent explores on small projects, and an idle CPU a few milliseconds per second higher than Node's in the benchmark, as above. +- **Larger database:** 11–20% bigger on six of the seven corpora above, and 37% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. +- **Fewer edges on some projects:** 8% fewer on pretix and 2% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. +- **Slower to index most projects this run:** in the 2026-10-27 comparison the fork's full index is slower on five of seven corpora — supabase 2.3× and n8n 2.4× on Node, pretix +38% (+72% on Bun), Alamofire +20%, discourse +13% (+15% on Bun) and gin +14% on Node; only CPython is faster (1.6×). The TypeScript-monorepo regressions trace to the ambient-`declare module` port described above; the residual gap is tracked follow-up work. One-file syncs still peak higher on some corpora (2026-10-18 run: CPython 1.23 GiB against 909 MiB on Node, +39%; Alamofire +14%; gin +10%; discourse +4%). +- **On Bun:** slower concurrent explores on small projects (gin wave +20% this run), and a mixed peak-memory picture — 11–25% lower on four corpora, 2–6% higher on three. --- diff --git a/assets/fork-vs-upstream-performance.svg b/assets/fork-vs-upstream-performance.svg index 703258cfb7..084ef01139 100644 --- a/assets/fork-vs-upstream-performance.svg +++ b/assets/fork-vs-upstream-performance.svg @@ -1,6 +1,6 @@ - + Fork vs upstream CodeGraph: headline numbers -On n8n the fork indexes in 66.3 s against 84.5 s, syncs one edited file in 4.37 s against 13.1 s, and answers a warm question in 177 ms against 600 ms. Of 74 comparisons it is better in 65, about the same in 4 and worse in 5. - -n8n and CPython: the fork indexes 1.2–1.3× faster and uses 35–37% less memoryNumbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.Index n8n from scratch1.3× faster84.5 s → 66.3 son Bun: 74.7 s → 66.0 sUpdate after editing one file3.0× faster13.1 s → 4.37 son Bun: 10.7 s → 3.79 sAnswer a question (warm)3.4× faster600 ms → 177 mson Bun: 639 ms → 187 msMemory while indexing n8n37% less9.97 GiB → 6.30 GiBon Bun: 7.49 GiB → 6.07 GiBMCP server memory when idle56% less4.88 GiB → 2.15 GiBon Bun: 3.24 GiB → 1.23 GiBStart the command line1.9× faster64 ms → 33 mson Bun: 41 ms → 24 msWhere the fork is not betterIndexing pretix: 8.36 s → 9.40 s on Node (+13%), 7.75 s → 9.27 s on Bun (+20%)Memory indexing Alamofire: 830 MiB → 953 MiB on Node (+15%)Memory indexing pretix: 2.94 GiB → 3.25 GiB on Node (+11%), 2.36 GiB → 2.50 GiB on Bun (+6%)About the same (within 3%): indexing gin on Bun; indexing discourse on Node; indexing supabase on Node; indexingsupabase on Bun.Over all 74 comparisons on Node and Bun: better in 65, about the same in 4, worse in 5. 2 more are left out because theone-file sync failed for upstream on Node and upstream on Bun.Measured 2026-10-18 · upstream 986de95c vs fork 633fc82c · Node 24.22.0, Bun 1.4.3 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted +On n8n the fork indexes in 178.0 s against 74.9 s, syncs one edited file in 4.18 s against 11.6 s, and answers a warm question in 131 ms against 242 ms. Of 76 comparisons it is better in 56, about the same in 5 and worse in 15. + +Index on n8n 2.4× slower, CPython 1.6× faster; the fork uses 44–49% less memoryNumbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.Index n8n from scratch2.4× slower74.9 s → 178.0 son Bun: 59.4 s → 176.9 sUpdate after editing one file2.8× faster11.6 s → 4.18 son Bun: 8.68 s → 3.41 sAnswer a question (warm)1.8× faster242 ms → 131 mson Bun: 211 ms → 107 msMemory while indexing n8n49% less12.39 GiB → 6.30 GiBon Bun: 9.23 GiB → 6.41 GiBMCP server memory when idle19% less4.40 GiB → 3.58 GiBon Bun: 3.40 GiB → 1.37 GiBStart the command line2.1× faster73 ms → 35 mson Bun: 36 ms → 21 msWhere the fork is not betterIndexing gin: 0.96 s → 1.09 s on Node (+14%)Indexing Alamofire: 1.35 s → 1.62 s on Node (+20%)Indexing pretix: 9.04 s → 12.5 s on Node (+38%), 7.20 s → 12.4 s on Bun (+72%)Indexing discourse: 19.6 s → 22.2 s on Node (+13%), 18.0 s → 20.8 s on Bun (+15%)Indexing supabase: 18.2 s → 41.5 s on Node (+128%), 16.4 s → 40.6 s on Bun (+147%)Indexing n8n: 74.9 s → 178.0 s on Node (+138%), 59.4 s → 176.9 s on Bun (+198%)Syncing one file in gin: 0.29 s → 0.32 s on Bun (+9%)Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%)8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%)File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%)No clear change (inside the comparison's own run-to-run spread): indexing gin on Bun; memory indexing supabase onNode; syncing one file in supabase on Node; memory busy on n8n (MCP server) on Bun; cPU idle on gin (MCP server) onBun.Over all 76 comparisons on Node and Bun: better in 56, no clear change in 5, worse in 15. A difference counts only whenit exceeds that comparison's own run-to-run spread.Measured 2026-10-27 · upstream 69af19ac vs fork bc27fdb4 · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted diff --git a/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg new file mode 100644 index 0000000000..75c7febecf --- /dev/null +++ b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg @@ -0,0 +1,6 @@ + +Fork vs upstream CodeGraph: every result +Heatmap of the fork's change against upstream for full index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. Of 76 comparisons it is better in 56, about the same in 5 and worse in 15. + +Every result at a glanceEach cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better beyond the comparison’s own run-to-run spread, amber = inside it, red = worse, grey = inconclusive or no data.Full index: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node+14%+20%+38%−38%+13%+128%+138%Fork on Bun+10%−26%+72%−34%+15%+147%+198%Full index: peak memoryginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−30%−28%−18%−44%−31%−26%−49%Fork on Bun−52%−32%−16%−50%−44%−5%−31%One-file sync: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−12%−33%−34%−55%−52%−26%−64%Fork on Bun+9%−34%−29%−57%−46%−65%−61%MCP server on n8nStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−10%−46%−20%−33%+18%−19%−19%Fork on Bun−10%−50%−17%−46%0%−60%−49%MCP server on ginStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−24%−7%+18%−6%−26%−23%−43%Fork on Bun−31%−13%+5%0%−42%−33%−44%Command-line startup on gin--versionstatusexploreFork on Node−52%−32%−47%Fork on Bun−42%−30%−61%Measured 2026-10-27 · upstream 69af19ac vs fork bc27fdb4 · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted + diff --git a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md new file mode 100644 index 0000000000..7ac0005d83 --- /dev/null +++ b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md @@ -0,0 +1,148 @@ +# Fork against upstream CodeGraph on Node.js and Bun, 2026-10-27 + +Upstream `main` at `69af19ac` against `fork/consolidated` at `bc27fdb4` (the ambient-scope memo from [#554](https://github.com/bompus/codegraph/pull/554)), each on Node.js 24.22.1 and on Bun 1.4.4, over seven corpora, the MCP server and command-line startup. It repeats the [2026-10-18 run](fork-vs-upstream-node-bun-2026-10-18.md) with the same scripts and the same pinned corpora on the current builds. The README's "Measured results" quote this page. + +## Method + +The scripts are in [`scripts/benchmarks/runtime/`](../../scripts/benchmarks/runtime/): `run-index-sync.sh`, `run-mcp.sh` and `run-cli.sh`, with the four arms below. `adapt-flat-output.mjs` stages the flat output for `render-charts.mjs`, which computes every table, the heatmap and the README card from the raw rows. + +- **Builds.** Upstream `main` at `69af19ac` — the commit the fork's default branch is reconciled through — and the fork at `bc27fdb4`. The fork contains every upstream commit through `69af19ac` plus its own work, including this run's one change vs `31e3ab04`: a memo on the per-file ambient-scope scan. Without it, the fork's supabase index never completed (see [The supabase resolution regression](#the-supabase-resolution-regression)). Each build was installed with `npm ci`, with the native kernel built from source, and built with `npm run build`. +- **Arms.** + + | Arm | Command | + |---|---| + | upstream, Node | `node` 24.22.1 running upstream's `dist/bin/codegraph.js` | + | fork, Node | `node` 24.22.1 running the fork's `dist/bin/codegraph.js` | + | fork, Bun | `bun` 1.4.4 running the fork's `dist/bin/codegraph.js` | + | upstream, Bun | `bun` 1.4.4 running upstream's build, with `CODEGRAPH_ALLOW_UNSAFE_NODE=1` | + + Bun reports a Node.js compatibility version of 26 or newer, and upstream refuses Node 25 and newer, so without the override upstream exits with `Unsupported Node.js version`. The fork needs no override. +- **Host.** WSL2 Ubuntu, 16 vCPUs, 122 GiB RAM, ext4, shared with other agent sessions. +- **Environment.** `CODEGRAPH_DIR=.cg-bench`, `CODEGRAPH_TELEMETRY=0`, and `CODEGRAPH_NO_DAEMON=1` for the CLI runs. Every timed command ran at normal priority under a 16 GiB memory cap and `/usr/bin/time -v`. Peak memory is MaxRSS. Database size is `codegraph.db` after a WAL checkpoint. +- **Order.** Per corpus the arms ran in the order upstream Node, fork Node, fork Bun, upstream Bun, then the reverse. Each run did a full index (`codegraph init --yes .` into an empty directory) and then two one-file syncs (each appends a comment line and a one-line symbol to the same file, then runs `codegraph sync .`). Index figures are the median of two runs and sync figures the median of four. +- **Corpora**, the same pinned commits as on 2026-10-18, shallow, no dependencies installed: gin (Go) `1bd0ecf`, Alamofire (Swift) `bda9ed5`, pretix (Python and JavaScript) `aa14505`, CPython (C and Python) `1e8ff18a`, discourse (Ruby and JavaScript) `ae5ba2c4`, supabase (React, Next.js and TypeScript) `36371de1`, n8n (Vue and TypeScript) `7cc0450a`. Each was checked at its pin, with a clean tracked tree, before the run. +- **CLI explore query.** `Engine ServeHTTP handleHTTPRequest` on gin, as before. + +Both runtimes produced the same node and edge counts for the same build on every corpus. + +### Host noise and the stalled arm + +This run is noisier than 2026-10-18's. The host shared work with other agent sessions: 3.5 GiB of swap was in use (against 0 MiB then), and the per-row recorded 1-minute load ran 0.2–5.6 rather than 0.5–1.9. The runner waited for no `capped-*` scopes and load under 2 before each corpus (the `idle` gate in `arms.sh`), and the mirrored arm order spreads drift across arms. Following house-rules #321 there is no fixed reporting band: a cell counts as a change only when the fork/upstream gap exceeds **that comparison's own run-to-run spread** — the larger of the two arms' (max − min) / median over the repeated runs, or sd/mean for the hyperfine cells. A gap inside it is *no clear change*; a metric whose samples can't support a spread is *inconclusive*. The render script (`render-charts.mjs`) computes all of this from the raw files — no figure was typed by hand. + +One MCP attempt on n8n was additionally contaminated by a second benchmark writer that had not exited; its rows are preserved in `mcp-contaminated.jsonl` and the published table comes from a clean re-run. The idle-CPU rows from the shared host are the least trustworthy measurements here and are only quoted when their spread supports it. + +On the pre-fix build, `fork-node` indexing supabase was killed at 10:53 wall by the liveness watchdog after ~46 CPU-minutes without finishing (exit 143; the row stays in `index-sync.tsv` as evidence). Resolution was re-parsing every `.d.ts` in the project for each out-of-repo import — about 69,000 tree parses in 40 seconds measured live — because the ambient-module scan introduced upstream's `declare module` lookup (upstream `553d5d10`/`214ef50e` era, merged in fork PR #548) with no per-file memo. `d364683a` adds the memo, which is the build every fork arm on this page runs. Details: [The supabase resolution regression](#the-supabase-resolution-regression). + +## Full index + +Time and peak memory, median of two. + +| Corpus | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +| --- | --- | --- | --- | --- | +| gin | 0.96 s, 558 MiB | 1.09 s, 392 MiB | 1.06 s, 296 MiB | 0.96 s, 618 MiB | +| Alamofire | 1.35 s, 879 MiB | 1.62 s, 632 MiB | 0.99 s, 664 MiB | 1.35 s, 974 MiB | +| pretix | 9.04 s, 3.29 GiB | 12.5 s, 2.71 GiB | 12.4 s, 2.42 GiB | 7.20 s, 2.89 GiB | +| CPython | 55.0 s, 7.15 GiB | 33.8 s, 4.01 GiB | 33.5 s, 3.48 GiB | 51.0 s, 6.92 GiB | +| discourse | 19.6 s, 4.07 GiB | 22.2 s, 2.82 GiB | 20.8 s, 2.40 GiB | 18.0 s, 4.25 GiB | +| supabase | 18.2 s, 4.86 GiB | 41.5 s, 3.59 GiB | 40.6 s, 3.81 GiB | 16.4 s, 4.02 GiB | +| n8n | 74.9 s, 12.39 GiB | 178.0 s, 6.30 GiB | 176.9 s, 6.41 GiB | 59.4 s, 9.23 GiB | + +Graph size. Node and edge counts match across runtimes for each build; database sizes are from the Node runs: + +| Corpus | Upstream nodes / edges / database | Fork nodes / edges / database | +| --- | --- | --- | +| gin | 2,660 / 7,908 / 8.5 MiB | 3,089 / 8,404 / 10.1 MiB | +| Alamofire | 4,600 / 12,516 / 16.2 MiB | 5,614 / 13,682 / 18.3 MiB | +| pretix | 32,781 / 79,684 / 115 MiB | 32,797 / 73,402 / 138 MiB | +| CPython | 167,332 / 615,244 / 492 MiB | 167,923 / 603,861 / 573 MiB | +| discourse | 163,411 / 309,678 / 389 MiB | 168,110 / 308,849 / 432 MiB | +| supabase | 114,440 / 292,126 / 335 MiB | 152,274 / 327,838 / 460 MiB | +| n8n | 315,422 / 1,239,920 / 1.43 GiB | 328,519 / 1,233,216 / 1.67 GiB | + +The edge counts differ because the builds resolve differently: the fork adds Markdown, route and dispatch links, and declines call links it cannot confirm. The timings compare the cost of building each build's own graph, not the same graph. + +## One-file sync + +Median of four (two syncs per run, two runs). + +| Corpus | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +| --- | --- | --- | --- | --- | +| gin | 0.47 s | 0.42 s | 0.31 s | 0.29 s | +| Alamofire | 0.48 s | 0.32 s | 0.20 s | 0.31 s | +| pretix | 1.62 s | 1.07 s | 0.81 s | 1.13 s | +| CPython | 5.35 s | 2.42 s | 1.70 s | 3.92 s | +| discourse | 4.78 s | 2.31 s | 1.73 s | 3.22 s | +| supabase | 4.58 s | 3.37 s | 2.34 s | 6.69 s | +| n8n | 11.6 s | 4.18 s | 3.41 s | 8.68 s | + + +Upstream's supabase sync, which failed with `FOREIGN KEY constraint failed` in every prior run, now completes ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565) was fixed upstream on 2026-10-19). Its two passes differ widely on Node (1.05 s then 8.1 s); the 4.58 s figure is the median of the four, and against that spread the comparison counts as no clear change rather than a fork win. + +## The supabase resolution regression + +Worth documenting because the README's earlier "ties on supabase" line hid a cliff: + +- The fork indexed supabase in ~13.8 s at `d0f288ab` (the commit before fork PR #548). From `76937a38` (#548's merge of upstream `8beab37c..553d5d10`) through `31e3ab04`, supabase indexing never finished — killed at ~11 minutes by the watchdog. +- Perf on a symbolized kernel showed ~65% of resolution cycles inside `ts_parser_parse`, and a `[tree-miss]` counter showed ~69,000 parses in 40 s: all `imports` refs, 417 distinct `.d.ts` files re-parsed ~2,440 times each. +- Cause: `ts_ambient_lookup` (the port of upstream's ambient-module lookup) scans every `.d.ts` for each out-of-repo import and `ts_scope_scan` re-parsed each file on every call; the 32-entry FIFO tree cache cannot hold the working set. Upstream's equivalent is a tokenizer over source text (`scanAmbientModules` never parses a tree) and upstream memoizes the global-scope scan per file (`TS_GLOBAL_SCOPES`). +- `d364683a` memoizes `ts_scope_scan` per file in the resolver (PR #554). supabase indexes in ~40–47 s, with 152,274 nodes / 327,838 edges (more edges than pre-#548's 323,321 — the ambient work does resolve). +- The remaining ~2.3–2.5× gap vs upstream's ~17–19 s on this corpus is real: the fork still parses each `.d.ts` once per resolver worker where upstream scans text once per file. It is bounded work, not the quadratic, and it is the honest number this page reports. + +## MCP server + +`codegraph serve --mcp` on n8n; memory and CPU summed over the server and the daemon it starts. Median of two runs per arm. + +| Measure | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +| --- | --- | --- | --- | --- | +| `initialize` answered | 114 ms | 52 ms | 38 ms | 64 ms | +| Start to first explore answer | 2.95 s | 2.64 s | 2.19 s | 2.43 s | +| Warm explore, median | 242 ms | 131 ms | 107 ms | 211 ms | +| Warm explore, p90 | 285 ms | 169 ms | 154 ms | 249 ms | +| 8 concurrent explores | 2.40 s | 1.91 s | 1.63 s | 1.97 s | +| Memory while busy | 3.85 GiB | 4.54 GiB | 3.50 GiB | 3.50 GiB | +| Edited file re-indexed | 1.15 s | 766 ms | 511 ms | 943 ms | +| Memory at idle | 4.40 GiB | 3.58 GiB | 1.37 GiB | 3.40 GiB | +| CPU at idle | 0.40% | 0.33% | 0.25% | 0.49% | + +And on gin, same protocol: + +| Measure | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +| --- | --- | --- | --- | --- | +| `initialize` answered | 110 ms | 55 ms | 37 ms | 65 ms | +| Start to first explore answer | 380 ms | 287 ms | 179 ms | 258 ms | +| Warm explore, median | 14 ms | 13 ms | 10 ms | 12 ms | +| Warm explore, p90 | 16 ms | 15 ms | 12 ms | 14 ms | +| 8 concurrent explores | 114 ms | 134 ms | 161 ms | 153 ms | +| Memory while busy | 883 MiB | 655 MiB | 468 MiB | 812 MiB | +| Edited file re-indexed | 408 ms | 383 ms | 358 ms | 357 ms | +| Memory at idle | 934 MiB | 722 MiB | 510 MiB | 764 MiB | +| CPU at idle | 0.17% | 0.10% | 0.25% | 0.45% | + +The fork-Node server's busy memory on n8n (4.54 GiB against upstream's 3.85) is the one MCP regression this run — the server keeps the larger graph resident. Its idle footprint is still lower on every arm. On gin the fork is slower only on the 8-explore wave on both runtimes (+18% Node, +5% Bun); warm single queries stay ahead. + +## CLI startup + +`hyperfine`, mean of 20 runs on gin, query `Engine ServeHTTP handleHTTPRequest`: + +| Command | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +|---|---|---|---|---| +| `codegraph --version` | 73 ms | 35 ms | 21 ms | 36 ms | +| `codegraph status` | 166 ms | 114 ms | 63 ms | 90 ms | +| `codegraph explore` | 220 ms | 117 ms | 71 ms | 181 ms | + +## Every result at a glance + +![Heatmap of the fork's change against upstream for index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup, on Node and on Bun. Lower is better; cells inside the comparison's own run-to-run spread are marked no clear change.](fork-vs-upstream-heatmap-2026-10-27.svg?v=1) + +Of 76 comparisons the fork is better in 56, shows no clear change in 5 and is worse in 15. The 15, generated from the same data as the heatmap: + +- Indexing gin: 0.96 s → 1.09 s on Node (+14%) +- Indexing Alamofire: 1.35 s → 1.62 s on Node (+20%) +- Indexing pretix: 9.04 s → 12.5 s on Node (+38%), 7.20 s → 12.4 s on Bun (+72%) +- Indexing discourse: 19.6 s → 22.2 s on Node (+13%), 18.0 s → 20.8 s on Bun (+15%) +- Indexing supabase: 18.2 s → 41.5 s on Node (+128%), 16.4 s → 40.6 s on Bun (+147%) +- Indexing n8n: 74.9 s → 178.0 s on Node (+138%), 59.4 s → 176.9 s on Bun (+198%) +- Syncing one file in gin: 0.29 s → 0.32 s on Bun (+9%) +- Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%) +- 8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%) +- File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%) diff --git a/scripts/benchmarks/runtime/adapt-flat-output.mjs b/scripts/benchmarks/runtime/adapt-flat-output.mjs new file mode 100644 index 0000000000..b54466845e --- /dev/null +++ b/scripts/benchmarks/runtime/adapt-flat-output.mjs @@ -0,0 +1,47 @@ +#!/usr/bin/env node +// Adapts the flat BENCH_OUT layout (index-sync.tsv, mcp.jsonl, cli-*.json with +// arm names upstream-node/fork-node/fork-bun/upstream-bun) into the attempt-dir +// layout render-charts.mjs consumes: /index--a1/{status,index-sync.tsv}, +// mcp--a1/{status,mcp.jsonl}, cli-gin-a1/{status,cli-*.json}. +import { readFileSync, writeFileSync, mkdirSync, existsSync } from 'node:fs'; +import { join } from 'node:path'; + +const [src, dst] = process.argv.slice(2); +const ARM = { 'upstream-node': 'upnode', 'fork-node': 'forknode', 'fork-bun': 'forkbun', 'upstream-bun': 'upbun' }; +const CORPORA = [['gin', 'gin'], ['Alamofire', 'alamofire'], ['pretix', 'pretix'], ['cpython', 'cpython'], ['discourse', 'discourse'], ['supabase', 'supabase'], ['n8n', 'n8n']]; + +const rows = readFileSync(join(src, 'index-sync.tsv'), 'utf8').trim().split('\n') + .filter((l) => l.split('\t')[3] === '0') // drop rows from killed/failed attempts + .map((l) => { + const c = l.split('\t'); c[1] = ARM[c[1]] ?? c[1]; return c.join('\t'); + }); +for (const [dirName, rowName] of CORPORA) { + const d = join(dst, `index-${dirName}-a1`); mkdirSync(d, { recursive: true }); + writeFileSync(join(d, 'status'), 'accepted\n'); + writeFileSync(join(d, 'index-sync.tsv'), rows.filter((l) => l.startsWith(rowName + '\t')).join('\n') + '\n'); +} +{ + const p = join(src, 'mcp.jsonl'); + if (existsSync(p)) { + const lines = readFileSync(p, 'utf8').trim().split('\n').map((l) => { const r = JSON.parse(l); r.arm = ARM[r.arm] ?? r.arm; return r; }); + for (const c of ['gin', 'n8n']) { + const rows = lines.filter((r) => r.corpus === c); + if (!rows.length) continue; + const d = join(dst, `mcp-${c}-a1`); mkdirSync(d, { recursive: true }); + writeFileSync(join(d, 'status'), 'accepted\n'); + writeFileSync(join(d, 'mcp.jsonl'), rows.map((r) => JSON.stringify(r)).join('\n') + '\n'); + } + } +} +{ + const d = join(dst, 'cli-gin-a1'); mkdirSync(d, { recursive: true }); + writeFileSync(join(d, 'status'), 'accepted\n'); + for (const n of ['version', 'status', 'explore']) { + const p = join(src, `cli-${n}.json`); + if (!existsSync(p)) { console.error(`missing ${p}`); continue; } + const j = JSON.parse(readFileSync(p, 'utf8')); + for (const r of j.results) r.name = ARM[r.name] ?? r.name; + writeFileSync(join(d, `cli-${n}.json`), JSON.stringify(j)); + } +} +console.log('adapted →', dst); diff --git a/scripts/benchmarks/runtime/render-charts.mjs b/scripts/benchmarks/runtime/render-charts.mjs index 2b5196ed8c..cad60229aa 100644 --- a/scripts/benchmarks/runtime/render-charts.mjs +++ b/scripts/benchmarks/runtime/render-charts.mjs @@ -10,7 +10,10 @@ // Per corpus the accepted attempt is the first `index--aN` directory whose // `status` file reads `accepted`; the same goes for `mcp--aN` and `cli-gin-aN`. // Medians are taken over the runs in that attempt (two per arm, mirrored order). -// A change counts as better or worse only outside the 3% reporting band. +// When the benchmark wrote the flat layout (index-sync.tsv, mcp.jsonl, cli-*.json +// at the output root), run adapt-flat-output.mjs first to stage this layout. +// A change counts as better or worse only outside the comparison's own +// run-to-run spread (house-rules #321). import { readFileSync, readdirSync, existsSync, writeFileSync } from 'node:fs'; import { join } from 'node:path'; @@ -23,8 +26,13 @@ for (const [k, v] of Object.entries(META)) if (!v) { console.error(`missing --${ const ARMS = ['upnode', 'forknode', 'forkbun', 'upbun']; const CORPORA = [['gin', 'gin'], ['Alamofire', 'Alamofire'], ['pretix', 'pretix'], ['cpython', 'CPython'], ['discourse', 'discourse'], ['supabase', 'supabase'], ['n8n', 'n8n']]; -const BAND = 0.03; +// No fixed reporting band: a cell counts as better or worse only when the +// fork/upstream gap exceeds that comparison's own run-to-run spread — the +// larger of the two arms' (max − min) / median for the same metric, each arm +// measured over its repeated runs. A metric whose median is non-positive or +// whose samples can't support a ratio is inconclusive. const median = (a) => { const s = [...a].sort((x, y) => x - y), m = s.length >> 1; return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; }; +const spreadRatio = (a) => { const m = median(a); return m > 0 ? (Math.max(...a) - Math.min(...a)) / m : NaN; }; const seconds = (t) => { const p = t.split(':'); return parseFloat(p.at(-1)) + 60 * parseFloat(p.at(-2)); }; const accepted = (prefix) => { const attempt = (d) => +d.slice(d.lastIndexOf('-a') + 2); @@ -40,7 +48,7 @@ for (const [key] of CORPORA) { const rows = readFileSync(join(accepted(`index-${key}`), 'index-sync.tsv'), 'utf8').trim().split('\n').map((l) => l.split('\t')); index[key] = {}; for (const arm of ARMS) { - const i = rows.filter((r) => r[1] === arm && r[2] === 'index'); + const i = rows.filter((r) => r[1] === arm && r[2] === 'index' && r[3] === '0'); const s = rows.filter((r) => r[1] === arm && r[2] !== 'index'); if (!i.length || !s.length) throw new Error(`${key} ${arm}: missing rows`); // A sync that exits non-zero has no time worth comparing: its comparisons are left out and drawn as n/a. @@ -49,6 +57,8 @@ for (const [key] of CORPORA) { indexS: median(i.map((r) => seconds(r[4]))), peakMiB: median(i.map((r) => +r[5])) / 1024, dbMiB: median(i.map((r) => +r[7])) / 2 ** 20, nodes: +i[0][8], edges: +i[0][9], syncS: syncFailed ? NaN : median(s.map((r) => seconds(r[4]))), runs: i.length, syncRuns: s.length, + indexS_spread: spreadRatio(i.map((r) => seconds(r[4]))), peakMiB_spread: spreadRatio(i.map((r) => +r[5])), + syncS_spread: syncFailed ? NaN : spreadRatio(s.map((r) => seconds(r[4]))), }; } } @@ -63,9 +73,12 @@ for (const key of ['gin', 'n8n']) { if (x.length === 0 || x.some((r) => r[k] == null)) throw new Error(`mcp ${key} ${arm}: ${k} is missing in ${x.length} run(s)`); return median(x.map((r) => r[k])); }; + const sp = (k) => spreadRatio(x.map((r) => r[k])); mcp[key][arm] = { initMs: g('initMs'), firstMs: g('coldToFirstAnswerMs'), warmMs: g('warmMedianMs'), p90Ms: g('warmP90Ms'), wave8Ms: g('wave8Ms'), watchMs: g('watchSyncMs'), busyMiB: g('busy_totalMB'), idleMiB: g('idle_totalMB'), idleCpu: g('idleCpuPct'), + initMs_spread: sp('initMs'), firstMs_spread: sp('coldToFirstAnswerMs'), warmMs_spread: sp('warmMedianMs'), p90Ms_spread: sp('warmP90Ms'), + wave8Ms_spread: sp('wave8Ms'), watchMs_spread: sp('watchSyncMs'), busyMiB_spread: sp('busy_totalMB'), idleMiB_spread: sp('idle_totalMB'), idleCpu_spread: sp('idleCpuPct'), }; } } @@ -77,7 +90,7 @@ const cli = {}; cli[n] = { label }; for (const arm of ARMS) { const w = res.find((r) => r.name === arm).summary.time_wall_clock; - cli[n][arm] = { mean: w.mean * 1000, sd: w.stddev * 1000 }; + cli[n][arm] = { mean: w.mean * 1000, sd: w.stddev * 1000, spread: w.stddev / w.mean }; } } } @@ -90,15 +103,15 @@ const fPct = (v) => `${v.toFixed(2)}%`; const pctChange = (r) => { if (!Number.isFinite(r)) return 'n/a'; const d = Math.round((r - 1) * 100); return (d > 0 ? '+' : d < 0 ? '−' : '') + Math.abs(d) + '%'; }; const esc = (s) => String(s).replace(/&/g, '&').replace(/ [a, cli[n][a].mean])), fMs]); +for (const key of ['n8n', 'gin']) for (const [k, label, f] of MCP_ROWS) metrics.push([`MCP server on ${key}`, label, pick(mcp[key], k), f, pick(mcp[key], k + '_spread')]); +for (const n of ['version', 'status', 'explore']) metrics.push(['Command-line startup on gin', cli[n].label, Object.fromEntries(ARMS.map((a) => [a, cli[n][a].mean])), fMs, Object.fromEntries(ARMS.map((a) => [a, cli[n][a].spread]))]); function pick(o, k) { return Object.fromEntries(ARMS.map((a) => [a, o[a][k]])); } // Two zero measurements are equal; a zero baseline with a positive fork value is Infinity. @@ -106,19 +119,36 @@ const ratio = (m, rt) => { const [up, fk] = rt === 'Node' ? [m[2].upnode, m[2].forknode] : [m[2].upbun, m[2].forkbun]; return up === 0 ? (fk === 0 ? 1 : Infinity) : fk / up; }; +// Per-comparison variation criterion: the larger of the two compared arms' own +// run-to-run spread (max−min)/median — or sd/mean for the hyperfine runs. A gap +// outside it is a change; inside it is no clear change; when it can't be +// computed the cell is inconclusive. +const cellSpread = (m, rt) => { + const [up, fk] = rt === 'Node' ? ['upnode', 'forknode'] : ['upbun', 'forkbun']; + const s = Math.max(m[4]?.[up] ?? NaN, m[4]?.[fk] ?? NaN); + return Number.isFinite(s) ? s : NaN; +}; +const classify = (c) => { + if (Number.isNaN(c.r) || Number.isNaN(c.spread)) return 'na'; + if (c.r < 1 - c.spread) return 'better'; + if (c.r > 1 + c.spread) return 'worse'; + return 'same'; +}; const every = []; -for (const m of metrics) for (const rt of ['Node', 'Bun']) every.push({ m, rt, r: ratio(m, rt) }); +for (const m of metrics) for (const rt of ['Node', 'Bun']) every.push({ m, rt, r: ratio(m, rt), spread: cellSpread(m, rt) }); const excluded = every.filter((c) => Number.isNaN(c.r)); +const inconclusive = every.filter((c) => !Number.isNaN(c.r) && Number.isNaN(c.spread)); const ARM_LABEL = { upnode: 'upstream on Node', forknode: 'fork on Node', forkbun: 'fork on Bun', upbun: 'upstream on Bun' }; const failedArms = ARMS.filter((a) => CORPORA.some(([k]) => Number.isNaN(index[k][a].syncS))).map((a) => ARM_LABEL[a]).join(' and '); const cmp = every.filter((c) => !Number.isNaN(c.r)); -const nBetter = cmp.filter((c) => c.r < 1 - BAND).length; -const nSame = cmp.filter((c) => c.r >= 1 - BAND && c.r <= 1 + BAND).length; -const nWorse = cmp.filter((c) => c.r > 1 + BAND).length; +const nBetter = cmp.filter((c) => classify(c) === 'better').length; +const nSame = cmp.filter((c) => classify(c) === 'same').length; +const nWorse = cmp.filter((c) => classify(c) === 'worse').length; +const nInconclusive = inconclusive.length; // ---- SVG pieces const C = { bg: '#0d1117', panel: '#161b22', text: '#e6edf3', muted: '#8b949e', good: '#2ea043', same: '#d29922', bad: '#f85149' }; -const col = (r) => Number.isNaN(r) ? C.muted : (r < 1 - BAND ? C.good : r > 1 + BAND ? C.bad : C.same); +const colFor = (m, rt) => { const k = classify({ m, rt, r: ratio(m, rt), spread: cellSpread(m, rt) }); return k === 'better' ? C.good : k === 'worse' ? C.bad : k === 'same' ? C.same : C.muted; }; const FONT = "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif"; const T = (x, y, s, o = {}) => `${esc(s)}`; const wrap = (W, H, body, title, desc) => `\n${esc(title)}\n${esc(desc)}\n\n${body}\n\n`; @@ -132,8 +162,8 @@ function cards() { const speedups = ['n8n', 'CPython'].map((n) => 1 / ratio(idx(n), 'Node')); const savings = ['n8n', 'CPython'].map((n) => (1 - ratio(mem(n), 'Node')) * 100); const lo = (a) => Math.min(...a), hi = (a) => Math.max(...a); - const sp = lo(speedups).toFixed(1) === hi(speedups).toFixed(1) ? `${lo(speedups).toFixed(1)}×` : `${lo(speedups).toFixed(1)}–${hi(speedups).toFixed(1)}×`; - const title = `n8n and CPython: the fork indexes ${sp} faster and uses ${Math.round(lo(savings))}–${Math.round(hi(savings))}% less memory`; + const dir = (s) => s >= 1 ? `${s.toFixed(1)}× faster` : `${(1 / s).toFixed(1)}× slower`; + const title = `Index on n8n ${dir(speedups[0])}, CPython ${dir(speedups[1])}; the fork uses ${Math.round(lo(savings))}–${Math.round(hi(savings))}% less memory`; const card = [ ['Index n8n from scratch', idx('n8n'), 'time'], ['Update after editing one file', find('One-file sync: time', 'n8n'), 'time'], @@ -146,11 +176,11 @@ function cards() { b += T(24, 60, 'Numbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.', { size: 12, fill: C.muted }); card.forEach(([t, m, kind], i) => { const x = 24 + (i % 3) * 284, y = 82 + Math.floor(i / 3) * 150, r = ratio(m, 'Node'); - const badge = Number.isNaN(r) ? 'n/a' : kind === 'mem' ? `${Math.round((1 - r) * 100)}% less` : `${(1 / r).toFixed(1)}× faster`; + const badge = Number.isNaN(r) ? 'n/a' : kind === 'mem' ? (r <= 1 ? `${Math.round((1 - r) * 100)}% less` : `${Math.round((r - 1) * 100)}% more`) : (r >= 1 ? `${r.toFixed(1)}× slower` : `${(1 / r).toFixed(1)}× faster`); const f = m[3], v = m[2]; b += ``; b += T(x + 16, y + 26, t, { size: 13, fill: C.muted }); - b += T(x + 16, y + 72, badge, { size: 30, w: 700, fill: col(r) }); + b += T(x + 16, y + 72, badge, { size: 30, w: 700, fill: colFor(m, 'Node') }); b += T(x + 16, y + 100, `${f(v.upnode)} → ${f(v.forknode)}`, { size: 15 }); b += T(x + 16, y + 120, `on Bun: ${f(v.upbun)} → ${f(v.forkbun)}`, { size: 12, fill: C.muted }); }); @@ -159,19 +189,19 @@ function cards() { const phrase = { 'Full index: time': (l) => `Indexing ${l}`, 'Full index: peak memory': (l) => `Memory indexing ${l}`, 'One-file sync: time': (l) => `Syncing one file in ${l}` }; const name = (m) => (phrase[m[0]] ? phrase[m[0]](m[1]) : m[0].startsWith('MCP server on ') ? `${m[1]} on ${m[0].slice(14)} (MCP server)` : `codegraph ${m[1]}`); const byMetric = new Map(); - for (const c of cmp) if (c.r > 1 + BAND) { const k = c.m; (byMetric.get(k) ?? byMetric.set(k, []).get(k)).push(c); } + for (const c of cmp) if (classify(c) === 'worse') { const k = c.m; (byMetric.get(k) ?? byMetric.set(k, []).get(k)).push(c); } const worse = [...byMetric].map(([m, cs]) => `${name(m)}: ` + cs.map((c) => { const v = c.rt === 'Node' ? [m[2].upnode, m[2].forknode] : [m[2].upbun, m[2].forkbun]; return `${m[3](v[0])} → ${m[3](v[1])} on ${c.rt} (${pctChange(c.r)})`; }).join(', ')); const lower = (s) => s[0].toLowerCase() + s.slice(1); - const same = cmp.filter((c) => c.r >= 1 - BAND && c.r <= 1 + BAND).map((c) => `${lower(name(c.m))} on ${c.rt}`); + const same = cmp.filter((c) => classify(c) === 'same').map((c) => `${lower(name(c.m))} on ${c.rt}`); const wrapText = (s, n) => { const out = []; let cur = ''; for (const w of s.split(' ')) { if ((cur + ' ' + w).trim().length > n) { out.push(cur); cur = w; } else cur = (cur + ' ' + w).trim(); } if (cur) out.push(cur); return out; }; - const lines = [...worse, `About the same (within 3%): ${same.join('; ')}.`].flatMap((l) => wrapText(l, 118)); + const lines = [...worse, `No clear change (inside the comparison's own run-to-run spread): ${same.join('; ')}.`].flatMap((l) => wrapText(l, 118)); const boxH = 44 + lines.length * 22; let y = 392; b += ``; b += T(40, y + 26, 'Where the fork is not better', { size: 14, w: 700, fill: C.bad }); lines.forEach((s, i) => { b += T(40, y + 54 + i * 22, s, { size: 12.5 }); }); y += boxH + 26; - const tally = `Over all ${cmp.length} comparisons on Node and Bun: better in ${nBetter}, about the same in ${nSame}, worse in ${nWorse}.${excluded.length ? ` ${excluded.length} more are left out because the one-file sync failed for ${failedArms}.` : ''}`; + const tally = `Over all ${cmp.length} comparisons on Node and Bun: better in ${nBetter}, no clear change in ${nSame}, worse in ${nWorse}${nInconclusive ? `, inconclusive in ${nInconclusive}` : ''}. A difference counts only when it exceeds that comparison's own run-to-run spread.${excluded.length ? ` ${excluded.length} more are left out because the one-file sync failed for ${failedArms}.` : ''}`; wrapLine(tally, 120).forEach((l, i) => { if (i) y += 18; b += T(24, y, l, { size: 12, fill: C.muted }); }); y += 20; b += T(24, y, FOOT, { size: 11, fill: C.muted }); @@ -184,7 +214,7 @@ function heatmap() { const W = 880; let y = 36, b = T(24, y, 'Every result at a glance', { size: 20, w: 700 }); y += 22; - b += T(24, y, 'Each cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better by over 3%, amber = about the same, red = worse.', { size: 12, fill: C.muted }); + b += T(24, y, 'Each cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better beyond the comparison’s own run-to-run spread, amber = inside it, red = worse, grey = inconclusive or no data.', { size: 12, fill: C.muted }); y += 26; const groups = [...new Set(metrics.map((m) => m[0]))]; for (const g of groups) { @@ -196,8 +226,8 @@ function heatmap() { for (const rt of ['Node', 'Bun']) { b += T(LX, y + CH / 2 + 4, `Fork on ${rt}`, { size: 11, fill: C.muted }); cols.forEach((c, i) => { - const r = ratio(c, rt), op = r > 1 + BAND ? 0.85 : r < 1 - BAND ? 0.25 + Math.min(0.6, 1 - r) : 0.55; - b += ``; + const r = ratio(c, rt), k = classify({ m: c, rt, r, spread: cellSpread(c, rt) }), op = k === 'worse' ? 0.85 : k === 'better' ? 0.25 + Math.min(0.6, 1 - r) : 0.55; + b += ``; b += T(LX + LW + i * CW + CW / 2, y + CH / 2 + 3, pctChange(r), { size: 13, w: 700, a: 'middle' }); }); y += CH; @@ -252,7 +282,7 @@ function tables() { L.push(`- Database size, fork vs upstream, % bigger: ${ks.map((k) => `${k} ${((index[k].forknode.dbMiB / index[k].upnode.dbMiB - 1) * 100).toFixed(0)}`).join(', ')}`); L.push(`- Edges, fork vs upstream, % change: ${ks.map((k) => `${k} ${((index[k].forknode.edges / index[k].upnode.edges - 1) * 100).toFixed(1)}`).join(', ')}`); L.push(`- Node/edge counts identical across runtimes: ${ks.every((k) => index[k].upnode.nodes === index[k].upbun.nodes && index[k].forknode.nodes === index[k].forkbun.nodes && index[k].upnode.edges === index[k].upbun.edges && index[k].forknode.edges === index[k].forkbun.edges)}`); - L.push(`- Comparisons: ${cmp.length}; better ${nBetter}, about the same ${nSame}, worse ${nWorse}`); + L.push(`- Comparisons: ${cmp.length}; better ${nBetter}, no clear change ${nSame}, worse ${nWorse}${nInconclusive ? `, inconclusive ${nInconclusive}` : ''} (a difference counts only outside the comparison's own run-to-run spread)`); return L.join('\n') + '\n'; }