From c29349a3f632c14ca647d476377e8c0da96ca2c0 Mon Sep 17 00:00:00 2001 From: bompus Date: Tue, 27 Oct 2026 09:21:43 -0600 Subject: [PATCH 1/5] perf(resolve): port upstream ambient-module text scan + specifier index MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ambient declare-module lookup (fork #548) ran ts_scope_scan, a tree-sitter parse, over every .d.ts once per out-of-repo import, then regex-tested each file's text per reference — ~1M parses on supabase before #554's memo and ~10.7k regex checks x ~2.4k refs after. Port upstream's ambient-modules.ts: a single text tokenizer per file (ts_tokenize) feeding scanAmbientModules/scanNamespaceBlocks state machines, memoized per file, plus a lazy specifier→files index (upstream's scanned.bySpecifier) so each import hits only candidate declaration files. Keeps the fork's export type { A as B } clause support and fixes three upstream-faithful port bugs found in review: export as namespace counts as a module marker, the namespace `global` flag means the namespace itself is a global name (members are global only via literal declare global blocks or script top level), and nested namespaces no longer leak into ambient member matching. Measured on 2026-10-29 against upstream 69af19ac (same host, two runs per arm, median): supabase index 41.5 s -> 19.6 s on Node (upstream 18.2 s), n8n 178 s -> 68.4 s (upstream 74.9 s), discourse 22.2 s -> 14.4 s (upstream 19.6 s). supabase graph identical to the pre-tokenizer baseline (152,274 nodes / 327,838 edges); n8n loses 53 edges vs baseline (1,233,163 vs 1,233,216) from the corrected namespace-global admission. Validation: 304 focused resolver/scope tests pass; full vitest failure set identical to e9561f46 base (143, all pre-existing); kernel golden dumps unchanged; eval:precision holds every absent and present case on all nine pinned corpora (baselines committed). --- .../precision-exposed-2155404-e9561f46.json | 118 ++++ .../precision-flask-d73fa1c-e9561f46.json | 60 ++ .../precision-gin-dcaa429-e9561f46.json | 59 ++ .../precision-javalin-48bf31c-e9561f46.json | 72 +++ .../precision-petclinic-818c413-e9561f46.json | 44 ++ .../precision-slim-3675bf6-e9561f46.json | 39 ++ .../precision-svelte-5895c63-e9561f46.json | 35 ++ .../precision-vite-8492422-e9561f46.json | 95 +++ .../precision-vitest-7c81815-e9561f46.json | 35 ++ .../src/resolve/js_scope_upstream.rs | 2 +- codegraph-kernel/src/resolve/mod.rs | 12 +- codegraph-kernel/src/resolve/ts_known.rs | 561 ++++++++++++++++-- 12 files changed, 1077 insertions(+), 55 deletions(-) create mode 100644 __tests__/evaluation/results/precision-exposed-2155404-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-flask-d73fa1c-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-gin-dcaa429-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-javalin-48bf31c-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-petclinic-818c413-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-slim-3675bf6-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-svelte-5895c63-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-vite-8492422-e9561f46.json create mode 100644 __tests__/evaluation/results/precision-vitest-7c81815-e9561f46.json diff --git a/__tests__/evaluation/results/precision-exposed-2155404-e9561f46.json b/__tests__/evaluation/results/precision-exposed-2155404-e9561f46.json new file mode 100644 index 0000000000..1365a871b4 --- /dev/null +++ b/__tests__/evaluation/results/precision-exposed-2155404-e9561f46.json @@ -0,0 +1,118 @@ +{ + "timestamp": "2026-10-27T14:33:43.845Z", + "corpus": "exposed", + "repo": "https://github.com/JetBrains/Exposed.git", + "commit": "2155404", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 2, + "absentHeld": 2, + "presentCases": 6, + "presentHeld": 6, + "unjudgeable": 0 + }, + "resolvedBy": { + "exact-match": 33266, + "(none)": 26630, + "import": 7391, + "instance-method": 3903, + "framework": 2463, + "function-ref": 1097, + "file-path": 221 + }, + "edges": 74971, + "results": [ + { + "caseId": "exposed-literal-trim-jdbc", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:b50c90376f77d8d0aca5d309ac09f69c", + "to": "method:9d77982d7d95880e2a54c4893b86a838", + "resolvedBy": "exact-match" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "exposed-literal-trim-r2dbc", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:05217ebf2e2d2da1556291c5c369839a", + "to": "method:9d77982d7d95880e2a54c4893b86a838", + "resolvedBy": "exact-match" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "exposed-column-count-extension", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:941657a44efa96b0b33ffbd2175bb09b", + "to": "method:495855c626bbbf0a54e3d191aa2fae92", + "resolvedBy": "exact-match" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "exposed-column-substring-extension", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:33263165fe66809c5f378e014e92784d", + "to": "method:8d9c8fca9a37a0223dc1a4ea8cdaccaa", + "resolvedBy": "exact-match" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "exposed-column-contains-extension", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:212c066478bc17c1ac27f6f14b109aee", + "to": "method:0d2884b38da4f463426a16b42399df20", + "resolvedBy": "exact-match" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "exposed-column-not-query-count", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + }, + { + "caseId": "exposed-wildcard-package-receiver", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:f40372af7132a6726971d7163deef334", + "to": "method:71e425ce8fd2d25d06c04cf318012e1a", + "resolvedBy": "instance-method" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "exposed-wildcard-not-other-package", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-flask-d73fa1c-e9561f46.json b/__tests__/evaluation/results/precision-flask-d73fa1c-e9561f46.json new file mode 100644 index 0000000000..11ebb34232 --- /dev/null +++ b/__tests__/evaluation/results/precision-flask-d73fa1c-e9561f46.json @@ -0,0 +1,60 @@ +{ + "timestamp": "2026-10-27T14:33:22.537Z", + "corpus": "flask", + "repo": "https://github.com/pallets/flask.git", + "commit": "d73fa1c", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 0, + "absentHeld": 0, + "presentCases": 2, + "presentHeld": 2, + "unjudgeable": 0 + }, + "resolvedBy": { + "(none)": 2474, + "import": 1180, + "instance-method": 736, + "exact-match": 633, + "function-ref": 191 + }, + "edges": 5214, + "results": [ + { + "caseId": "flask-package-reexport-receiver", + "expect": "present", + "pass": true, + "found": [ + { + "from": "file:examples/tutorial/flaskr/auth.py", + "to": "method:31833b8026746e7977b4c2736d71c63e", + "resolvedBy": "instance-method" + }, + { + "from": "file:examples/tutorial/flaskr/auth.py", + "to": "method:31833b8026746e7977b4c2736d71c63e", + "resolvedBy": "instance-method" + }, + { + "from": "file:examples/tutorial/flaskr/auth.py", + "to": "method:31833b8026746e7977b4c2736d71c63e", + "resolvedBy": "instance-method" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "flask-control-relative-from-import", + "expect": "present", + "pass": true, + "found": [ + { + "from": "file:src/flask/app.py", + "to": "function:00d6010e49ed947bc5858174234b23e4", + "resolvedBy": "import" + } + ], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-gin-dcaa429-e9561f46.json b/__tests__/evaluation/results/precision-gin-dcaa429-e9561f46.json new file mode 100644 index 0000000000..8ef66602ad --- /dev/null +++ b/__tests__/evaluation/results/precision-gin-dcaa429-e9561f46.json @@ -0,0 +1,59 @@ +{ + "timestamp": "2026-10-27T14:33:24.348Z", + "corpus": "gin", + "repo": "https://github.com/gin-gonic/gin.git", + "commit": "dcaa429", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 1, + "absentHeld": 1, + "presentCases": 2, + "presentHeld": 2, + "unjudgeable": 0 + }, + "resolvedBy": { + "(none)": 3593, + "exact-match": 2601, + "instance-method": 1632, + "framework": 247, + "file-path": 98, + "import": 90, + "function-ref": 14 + }, + "edges": 8275, + "results": [ + { + "caseId": "gin-declared-factory-receiver", + "expect": "present", + "pass": true, + "found": [ + { + "from": "function:285dcfdab6ec00ce5d1c87eddc233f0d", + "to": "method:08a6ca18de8a7f08f33790ca286e0a20", + "resolvedBy": "instance-method" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "gin-composite-receiver-json-control", + "expect": "present", + "pass": true, + "found": [ + { + "from": "function:e6a071cb9b3ce2eeba6fb82f941cef3d", + "to": "method:3ab506d0531aac0489e0c51fcc2128aa", + "resolvedBy": "instance-method" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "gin-composite-receiver-not-bson", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-javalin-48bf31c-e9561f46.json b/__tests__/evaluation/results/precision-javalin-48bf31c-e9561f46.json new file mode 100644 index 0000000000..be22aca2f7 --- /dev/null +++ b/__tests__/evaluation/results/precision-javalin-48bf31c-e9561f46.json @@ -0,0 +1,72 @@ +{ + "timestamp": "2026-10-27T14:33:47.168Z", + "corpus": "javalin", + "repo": "https://github.com/javalin/javalin.git", + "commit": "48bf31c", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 1, + "absentHeld": 1, + "presentCases": 3, + "presentHeld": 3, + "unjudgeable": 0 + }, + "resolvedBy": { + "(none)": 8205, + "exact-match": 3105, + "import": 2966, + "instance-method": 1886, + "function-ref": 20, + "file-path": 8, + "fuzzy": 1 + }, + "edges": 16191, + "results": [ + { + "caseId": "javalin-fluent-validator-nullable", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:bf4ae208fbd820c105bee964c7b27a6a", + "to": "method:fe7a51f0d2260237d9a5d95284e0b366", + "resolvedBy": "exact-match" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "javalin-fluent-validator-optional", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:7bdb347a57c4e38e31e33bfb6f6c6c54", + "to": "method:fe7a51f0d2260237d9a5d95284e0b366", + "resolvedBy": "exact-match" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "javalin-kotlin-imports-java-member", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:ef8b70a9b486ddf107a0a782595685f5", + "to": "method:8eb17ccb9967512be8c4aed8f145f350", + "resolvedBy": "import" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "javalin-bare-get-not-test-client", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-petclinic-818c413-e9561f46.json b/__tests__/evaluation/results/precision-petclinic-818c413-e9561f46.json new file mode 100644 index 0000000000..8106221b97 --- /dev/null +++ b/__tests__/evaluation/results/precision-petclinic-818c413-e9561f46.json @@ -0,0 +1,44 @@ +{ + "timestamp": "2026-10-27T14:33:25.440Z", + "corpus": "petclinic", + "repo": "https://github.com/spring-projects/spring-petclinic.git", + "commit": "818c413", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 0, + "absentHeld": 0, + "presentCases": 1, + "presentHeld": 1, + "unjudgeable": 0 + }, + "resolvedBy": { + "(none)": 986, + "instance-method": 270, + "exact-match": 169, + "import": 44, + "framework": 10, + "function-ref": 2, + "file-path": 1 + }, + "edges": 1482, + "results": [ + { + "caseId": "petclinic-bounded-type-parameter", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:2160e6191fa145dc29200531c30d1a5d", + "to": "method:efb8586338e2f81ea9d7cb6f694a2f0d", + "resolvedBy": "instance-method" + }, + { + "from": "method:2160e6191fa145dc29200531c30d1a5d", + "to": "method:efb8586338e2f81ea9d7cb6f694a2f0d", + "resolvedBy": "instance-method" + } + ], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-slim-3675bf6-e9561f46.json b/__tests__/evaluation/results/precision-slim-3675bf6-e9561f46.json new file mode 100644 index 0000000000..beb64cb221 --- /dev/null +++ b/__tests__/evaluation/results/precision-slim-3675bf6-e9561f46.json @@ -0,0 +1,39 @@ +{ + "timestamp": "2026-10-27T14:33:48.445Z", + "corpus": "slim", + "repo": "https://github.com/slimphp/Slim.git", + "commit": "3675bf6", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 0, + "absentHeld": 0, + "presentCases": 1, + "presentHeld": 1, + "unjudgeable": 0 + }, + "resolvedBy": { + "(none)": 2173, + "exact-match": 1432, + "instance-method": 438, + "qualified-name": 306, + "import": 69, + "function-ref": 4, + "file-path": 4 + }, + "edges": 4426, + "results": [ + { + "caseId": "slim-guarded-exception-receiver", + "expect": "present", + "pass": true, + "found": [ + { + "from": "method:a6f7723645dbd37b31799db2b356325f", + "to": "method:ea555ef764392cc4779c55819b7860c5", + "resolvedBy": "instance-method" + } + ], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-svelte-5895c63-e9561f46.json b/__tests__/evaluation/results/precision-svelte-5895c63-e9561f46.json new file mode 100644 index 0000000000..9382f20dfe --- /dev/null +++ b/__tests__/evaluation/results/precision-svelte-5895c63-e9561f46.json @@ -0,0 +1,35 @@ +{ + "timestamp": "2026-10-27T14:33:20.959Z", + "corpus": "svelte", + "repo": "https://github.com/sveltejs/svelte.git", + "commit": "5895c63", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 1, + "absentHeld": 1, + "presentCases": 0, + "presentHeld": 0, + "unjudgeable": 0 + }, + "resolvedBy": { + "(none)": 30920, + "import": 20057, + "exact-match": 10539, + "qualified-name": 1874, + "instance-method": 390, + "framework": 220, + "function-ref": 196, + "fuzzy": 73, + "file-path": 58 + }, + "edges": 64327, + "results": [ + { + "caseId": "svelte-bare-import-bundle", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-vite-8492422-e9561f46.json b/__tests__/evaluation/results/precision-vite-8492422-e9561f46.json new file mode 100644 index 0000000000..56a02d7c0c --- /dev/null +++ b/__tests__/evaluation/results/precision-vite-8492422-e9561f46.json @@ -0,0 +1,95 @@ +{ + "timestamp": "2026-10-27T14:33:02.980Z", + "corpus": "vite", + "repo": "https://github.com/vitejs/vite.git", + "commit": "8492422", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 6, + "absentHeld": 6, + "presentCases": 2, + "presentHeld": 2, + "unjudgeable": 0 + }, + "resolvedBy": { + "(none)": 13392, + "import": 11109, + "exact-match": 3413, + "qualified-name": 1009, + "instance-method": 457, + "file-path": 197, + "function-ref": 165, + "framework": 2 + }, + "edges": 29744, + "results": [ + { + "caseId": "vite-factory-receiver-control", + "expect": "present", + "pass": true, + "found": [ + { + "from": "function:d321dd128d90578bc69867835ec73ef8", + "to": "method:db73e5f5d9c7af61d0f22c5b9971ced3", + "resolvedBy": "instance-method" + } + ], + "missingEndpoints": [] + }, + { + "caseId": "vite-external-receiver-not-project-method", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + }, + { + "caseId": "vite-bare-import-self-edge", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + }, + { + "caseId": "vite-bare-import-getEnv", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + }, + { + "caseId": "vite-fuzzy-nested-resolveConfig-getEnv", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + }, + { + "caseId": "vite-fuzzy-nested-build-scan", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + }, + { + "caseId": "vite-sealed-module-defineConfig", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + }, + { + "caseId": "vite-control-relative-import", + "expect": "present", + "pass": true, + "found": [ + { + "from": "file:packages/vite/src/node/config.ts", + "to": "function:0c2f36c32c5742aee412978220fd5b23", + "resolvedBy": "import" + } + ], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/__tests__/evaluation/results/precision-vitest-7c81815-e9561f46.json b/__tests__/evaluation/results/precision-vitest-7c81815-e9561f46.json new file mode 100644 index 0000000000..2dd1ebecb1 --- /dev/null +++ b/__tests__/evaluation/results/precision-vitest-7c81815-e9561f46.json @@ -0,0 +1,35 @@ +{ + "timestamp": "2026-10-27T14:33:13.040Z", + "corpus": "vitest", + "repo": "https://github.com/vitest-dev/vitest.git", + "commit": "7c81815", + "codegraphSha": "e9561f46", + "summary": { + "absentCases": 1, + "absentHeld": 1, + "presentCases": 0, + "presentHeld": 0, + "unjudgeable": 0 + }, + "resolvedBy": { + "import": 39482, + "(none)": 23504, + "exact-match": 6224, + "qualified-name": 1950, + "instance-method": 1618, + "function-ref": 323, + "file-path": 323, + "framework": 3, + "fuzzy": 1 + }, + "edges": 73428, + "results": [ + { + "caseId": "vitest-bare-import-evaluatedModules", + "expect": "absent", + "pass": true, + "found": [], + "missingEndpoints": [] + } + ] +} \ No newline at end of file diff --git a/codegraph-kernel/src/resolve/js_scope_upstream.rs b/codegraph-kernel/src/resolve/js_scope_upstream.rs index 18c0268d25..da02f1aea7 100644 --- a/codegraph-kernel/src/resolve/js_scope_upstream.rs +++ b/codegraph-kernel/src/resolve/js_scope_upstream.rs @@ -293,7 +293,7 @@ impl KernelResolver { pub(super) fn js_declaration_reachable(&mut self, n: &KNode, r: &ResolveRefIn) -> Option { if is_js_family(&n.language) && !matches!(n.language.as_str(), "javascript" | "jsx") && n.file_path != r.file_path && n.kind == "function" { - if let Some(scan) = self.ts_scope_scan(&n.file_path, r) { + if let Some(scan) = self.ts_scope_scan(&n.file_path) { if scan.blocks.iter().any(|b| b.ambient && b.contains(n.start_line, n.start_column)) { return Some(false); } } } diff --git a/codegraph-kernel/src/resolve/mod.rs b/codegraph-kernel/src/resolve/mod.rs index fb8401a6b2..fd48aeb95b 100644 --- a/codegraph-kernel/src/resolve/mod.rs +++ b/codegraph-kernel/src/resolve/mod.rs @@ -707,10 +707,14 @@ pub struct KernelResolver { /// `(package, type)` → the project file declaring the package with that type in it. ts_ambient_memo: HashMap<(String, String), Option>, /// `file` → its ambient/global scope scan (None when unreadable). The - /// scan parses the file and ambient-module lookup walks every `.d.ts` - /// once per import, so each file is scanned once and shared — upstream - /// memoizes the same way (TS_GLOBAL_SCOPES). + /// scan is a text pass (upstream's ambient-modules.ts port), memoized per + /// file the way upstream memoizes TS_GLOBAL_SCOPES. ts_scope_memo: HashMap>>, + /// The project's `.d.[cm]?ts` files, computed on first ambient lookup. + ts_dts_files: Option>, + /// `declare module ''` specifier → the script declaration files + /// declaring it (upstream's `scanned.bySpecifier`), built on first lookup. + ts_ambient_index: Option>>, /// `(file, type as written)` → what the name names there. ts_named_type_memo: HashMap<(String, String), ts_known::TsNamedType>, /// Whether the project declares a `createInstance` returning an `InstanceType` of its constructor argument. @@ -941,6 +945,8 @@ impl KernelResolver { ts_imported_type_memo: HashMap::new(), ts_ambient_memo: HashMap::new(), ts_scope_memo: HashMap::new(), + ts_dts_files: None, + ts_ambient_index: None, ts_named_type_memo: HashMap::new(), ts_create_instance_declared: None, python_bound_memo: HashMap::new(), diff --git a/codegraph-kernel/src/resolve/ts_known.rs b/codegraph-kernel/src/resolve/ts_known.rs index 92a2574474..8eea90d7a0 100644 --- a/codegraph-kernel/src/resolve/ts_known.rs +++ b/codegraph-kernel/src/resolve/ts_known.rs @@ -227,11 +227,11 @@ impl KernelResolver { if let Some(own) = members.iter().find(|n| n.kind == "method") { return Ok((Some(own.clone()), true)); } - if let Some(scan) = self.ts_scope_scan(&cls.file_path, at) { + if let Some(scan) = self.ts_scope_scan(&cls.file_path) { if let Some(block) = scan.blocks.iter().find(|b| b.ambient && b.contains(cls.start_line, cls.start_column)) { let module = block.name.clone(); for candidate in self.nodes_by_name(&cls.name)?.iter().filter(|n| n.id != cls.id && is_member_owner_kind(&n.kind)).cloned().collect::>() { - if !self.ts_scope_scan(&candidate.file_path, at).is_some_and(|scan| scan.blocks.iter().any(|b| b.ambient && b.name == module && b.contains(candidate.start_line, candidate.start_column))) { continue; } + if !self.ts_scope_scan(&candidate.file_path).is_some_and(|scan| scan.blocks.iter().any(|b| b.ambient && b.name == module && b.contains(candidate.start_line, candidate.start_column))) { continue; } if let Some(node) = self.nodes_in_file(&candidate.file_path)?.iter().find(|n| n.name == name && n.kind == "method" && n.start_line >= candidate.start_line && n.end_line <= candidate.end_line).cloned() { return Ok((Some(node), true)); } @@ -473,9 +473,17 @@ pub(super) struct TsScopeBlock { name: String, start: (i64, i64), end: (i64, i64), + byte_start: usize, + byte_end: usize, pub(super) ambient: bool, global: bool, text: String, + /// An ambient block's `export default X` / `export = X` / `export {Y as default}` name. + default_name: Option, + /// An ambient block's `export { … }` clause names: exported name → local name. + export_aliases: Vec<(String, String)>, + /// The block has an `export { … }` clause (empty or not). + export_clause: bool, } impl TsScopeBlock { pub(super) fn contains(&self, line: i64, column: i64) -> bool { @@ -488,60 +496,487 @@ pub(super) struct TsScopeScan { pub(super) blocks: Vec, } +// ---- Tokenizer and scope scanners: ports of upstream ambient-modules.ts +// (scanAmbientModules, scanGlobalScope's script flag and scanNamespaceBlocks). +// They read tokens, never a tree, so a scope scan costs a text pass instead of +// a tree-sitter parse. + +#[derive(Clone, Copy, PartialEq)] +enum TsTok { Word, Str, Tpl, Punct } + +fn ts_word_start(c: u8) -> bool { c.is_ascii_uppercase() || c.is_ascii_lowercase() || c == b'_' || c == b'$' || c > 127 } +fn ts_word_char(c: u8) -> bool { ts_word_start(c) || c.is_ascii_digit() } + +fn ts_regex_after(word: &str) -> bool { + matches!(word, "return" | "typeof" | "case" | "do" | "else" | "in" | "of" | "instanceof" | "new" | "delete" | "void" | "throw" | "yield" | "await") +} + +/// Words that start an `export default` declaration rather than name its value. +fn ts_default_word(word: &str) -> bool { + matches!(word, "class" | "function" | "interface" | "abstract" | "async" | "enum") +} + +/// Read a template literal's text from `i` up to its closing backtick or a `${`. +fn ts_template_text(b: &[u8], i: &mut usize, line: &mut i64, line_start: &mut usize, depth: u32, templates: &mut Vec) { + let n = b.len(); + while *i < n { + let c = b[*i]; + if c == b'\\' { + if *i + 1 < n && b[*i + 1] == b'\n' { *line += 1; *line_start = *i + 2; } + *i += 2; + continue; + } + if c == b'`' { *i += 1; return; } + if c == b'$' && *i + 1 < n && b[*i + 1] == b'{' { templates.push(depth); *i += 2; return; } + if c == b'\n' { *line += 1; *line_start = *i + 1; } + *i += 1; + } +} + +/// Feed `source`'s tokens to `visit`: kind, text, 1-based line, 0-based byte +/// column, byte offset and the brace depth outside the token — both braces of +/// a block carry the depth the block sits at. +fn ts_tokenize(source: &str, visit: &mut impl FnMut(TsTok, &str, i64, i64, usize, u32)) { + let b = source.as_bytes(); + let n = b.len(); + let mut i = 0usize; + let mut line = 1i64; + let mut line_start = 0usize; + let mut depth = 0u32; + // The brace depth at each open `${`: the `}` that brings depth back to it ends the expression. + let mut templates: Vec = Vec::new(); + // Whether a `/` here would start a regex: after an operator, a keyword or nothing. + let mut regex_allowed = true; + + while i < n { + let c = b[i]; + if c == b'\n' { line += 1; line_start = i + 1; i += 1; continue; } + if c == b' ' || c == b'\t' || c == b'\r' { i += 1; continue; } + if c == 0xef && i + 2 < n && b[i + 1] == 0xbb && b[i + 2] == 0xbf { i += 3; continue; } + if c == 0xc2 && i + 1 < n && b[i + 1] == 0xa0 { i += 2; continue; } + let column = i - line_start; + if c == b'/' { + let next = b.get(i + 1).copied().unwrap_or(0); + if next == b'/' { + while i < n && b[i] != b'\n' { i += 1; } + continue; + } + if next == b'*' { + i += 2; + while i < n && !(b[i] == b'*' && i + 1 < n && b[i + 1] == b'/') { + if b[i] == b'\n' { line += 1; line_start = i + 1; } + i += 1; + } + i += 2; + continue; + } + if regex_allowed { + // A regex literal ends at an unescaped `/` outside a character class, or at the line's end. + i += 1; + let mut in_class = false; + while i < n { + let r = b[i]; + if r == b'\n' { break; } + if r == b'\\' { i += 1; } + else if r == b'[' { in_class = true; } + else if r == b']' { in_class = false; } + else if r == b'/' && !in_class { i += 1; break; } + i += 1; + } + while i < n && ts_word_char(b[i]) { i += 1; } // flags + regex_allowed = false; + visit(TsTok::Tpl, "", line, column as i64, i, depth); + continue; + } + } + if c == b'\'' || c == b'"' { + let mut j = i + 1; + let start = j; + let mut text = String::new(); + let mut from = start; + while j < n { + let s = b[j]; + if s == c || s == b'\n' { break; } + if s == b'\\' { + text.push_str(&source[from..j]); + j += 2; + from = j; + continue; + } + j += 1; + } + text.push_str(&source[from..j.min(n)]); + i = if j < n && b[j] == c { j + 1 } else { j }; + regex_allowed = false; + visit(TsTok::Str, &text, line, column as i64, i, depth); + continue; + } + if c == b'`' { + i += 1; + ts_template_text(b, &mut i, &mut line, &mut line_start, depth, &mut templates); + regex_allowed = false; + visit(TsTok::Tpl, "", line, column as i64, i, depth); + continue; + } + if c == b'{' { + visit(TsTok::Punct, "{", line, column as i64, i, depth); + depth += 1; + i += 1; + regex_allowed = true; + continue; + } + if c == b'}' { + i += 1; + if templates.last() == Some(&depth) { + templates.pop(); + ts_template_text(b, &mut i, &mut line, &mut line_start, depth, &mut templates); + regex_allowed = false; + continue; + } + depth = depth.saturating_sub(1); + visit(TsTok::Punct, "}", line, column as i64, i, depth); + regex_allowed = true; + continue; + } + if ts_word_start(c) { + let mut j = i + 1; + while j < n && ts_word_char(b[j]) { j += 1; } + let word = &source[i..j]; + i = j; + regex_allowed = ts_regex_after(word); + visit(TsTok::Word, word, line, column as i64, i, depth); + continue; + } + if c.is_ascii_digit() { + while i < n && (ts_word_char(b[i]) || b[i] == b'.') { i += 1; } + regex_allowed = false; + continue; + } + i += 1; + regex_allowed = c != b')' && c != b']'; + visit(TsTok::Punct, &source[i - 1..i], line, column as i64, i - 1, depth); + } +} + +struct TsAmbientBlock { + name: String, + start: (i64, i64), + end: (i64, i64), + byte_start: usize, + byte_end: usize, + default_name: Option, + export_aliases: Vec<(String, String)>, + export_clause: bool, +} + +/// Every `declare module '' { … }` block in `source`, and whether the +/// file is a script (no top-level `import` or `export`). Port of upstream's +/// `scanAmbientModules`; a block is kept at the depth it opened at so one +/// nested inside `declare global` still lands. +fn ts_scan_ambient(source: &str) -> (bool, Vec) { + let mut blocks: Vec = Vec::new(); + let mut script = true; + // The statement being read at the top level. + let mut declare = 0u8; // 1: `declare`, 2: `declare module`, 3: `declare module ''` + let mut declare_pos = (0i64, 0i64, 0usize); + let mut declare_name = String::new(); + // After a top-level `import`: the tokens seen so far, to tell `import x = N.y` + // (an alias of a namespace, legal in a script) from every other form. + let mut import_tokens: Option> = None; + // The block being read, and the export statement being read inside it. + let mut open: Option<(TsAmbientBlock, u32)> = None; + let mut export_state = 0u8; // 1: export, 2: export default, 3: export default , 4: export =, 5: export {, 6: export type + let mut clause: Vec = Vec::new(); + + let close_clause = |open: &mut TsAmbientBlock, clause: &mut Vec| { + let mut i = 0; + while i < clause.len() { + let local = clause[i].clone(); + if clause.get(i + 1).map(String::as_str) == Some("as") && clause.get(i + 2).is_some() { + let exported = clause[i + 2].clone(); + if exported == "default" { open.default_name.get_or_insert(local); } + else { open.export_aliases.push((exported, local)); } + i += 3; + } else { + open.export_aliases.push((local.clone(), local)); + i += 1; + } + } + clause.clear(); + }; + + ts_tokenize(source, &mut |kind, text, line, column, offset, depth| { + if let Some(t) = &mut import_tokens { + t.push((kind, text.to_string())); + let settle = t.len() >= 3 || t[0].0 != TsTok::Word || (t.len() == 2 && t[1].1 != "="); + if settle { + // `import(…)` and `import.meta` are expressions; `import x = N.y` aliases a namespace. + let is_alias = t[0].0 == TsTok::Punct && (t[0].1 == "(" || t[0].1 == ".") + || t[0].0 == TsTok::Word && t.get(1).map(|x| x.1.as_str()) == Some("=") && t.get(2).map(|x| x.0) == Some(TsTok::Word) && t[2].1 != "require"; + if !is_alias { script = false; } + import_tokens = None; + } + } + if let Some((block, open_depth)) = &mut open { + if kind == TsTok::Punct && text == "}" && depth == *open_depth { + block.end = (line, column); + block.byte_end = offset + 1; + blocks.push(std::mem::replace(block, TsAmbientBlock { + name: String::new(), start: (0, 0), end: (0, 0), byte_start: 0, byte_end: 0, + default_name: None, export_aliases: Vec::new(), export_clause: false, + })); + open = None; + export_state = 0; + clause.clear(); + return; + } + // The export statement being read inside the block's top level. + if export_state == 5 { + if kind == TsTok::Punct && text == "}" && depth == *open_depth + 1 { + close_clause(block, &mut clause); + export_state = 0; + } else if kind == TsTok::Word { + clause.push(text.to_string()); + } + return; + } + if depth == *open_depth + 1 { + match export_state { + 0 => if kind == TsTok::Word && text == "export" { export_state = 1; }, + 1 => { + if kind == TsTok::Word && text == "default" { export_state = 2; } + else if kind == TsTok::Punct && text == "=" { export_state = 4; } + else if kind == TsTok::Punct && text == "{" { export_state = 5; block.export_clause = true; } + else if kind == TsTok::Word && text == "type" { export_state = 6; } + else { export_state = 0; } + } + 2 => { + if kind == TsTok::Word && ts_default_word(text) { export_state = 3; } + else { + if kind == TsTok::Word { block.default_name.get_or_insert(text.to_string()); } + export_state = 0; + } + } + 3 => { + if (kind == TsTok::Word && ts_default_word(text)) || (kind == TsTok::Punct && text == "*") { return; } + if kind == TsTok::Word { block.default_name.get_or_insert(text.to_string()); } + export_state = 0; + } + 4 => { + if kind == TsTok::Word { block.default_name.get_or_insert(text.to_string()); } + export_state = 0; + } + 6 => { + if kind == TsTok::Punct && text == "{" { export_state = 5; block.export_clause = true; } + else { export_state = 0; } + } + _ => {} + } + } + return; + } + if depth != 0 { + declare = 0; + return; + } + if kind == TsTok::Word { + if text == "import" { + import_tokens = Some(Vec::new()); + declare = 0; + } else if text == "export" { + script = false; + declare = 0; + } else if text == "declare" { + declare = 1; + declare_pos = (line, column, offset); + } else { + declare = if declare == 1 && text == "module" { 2 } else { 0 }; + } + return; + } + if kind == TsTok::Str && declare == 2 { + declare = 3; + declare_name = text.to_string(); + return; + } + if kind == TsTok::Punct && text == "{" && declare == 3 && !declare_name.is_empty() { + open = Some((TsAmbientBlock { + name: std::mem::take(&mut declare_name), + start: (declare_pos.0, declare_pos.1), end: (0, 0), + byte_start: declare_pos.2, byte_end: 0, + default_name: None, export_aliases: Vec::new(), export_clause: false, + }, depth)); + } + declare = 0; + }); + if let Some(t) = &import_tokens { + let is_alias = t[0].0 == TsTok::Punct && (t[0].1 == "(" || t[0].1 == ".") + || t[0].0 == TsTok::Word && t.get(1).map(|x| x.1.as_str()) == Some("=") && t.get(2).map(|x| x.0) == Some(TsTok::Word) && t[2].1 != "require"; + if !is_alias { script = false; } + } + (script, blocks) +} + +enum TsNsFrame { Global(usize), Module, Ns(usize), Other } + +/// Every namespace block (old-spelling `module A` included) and every +/// `declare global` block in `source`, with the global flag upstream gives +/// it: directly inside `declare global`, at a declaration script's top level +/// (`top_level_global`), or inside another global one. Port of upstream's +/// `scanNamespaceBlocks`; namespaces inside a `declare module 'x'` body are +/// module `x`'s, never global, and named as if at the file's top level. +fn ts_scan_namespaces(source: &str, top_level_global: bool, out: &mut Vec) { + let mut open: Vec = Vec::new(); + let mut names: Option> = None; // the names after `namespace`/`module`, while its header is read + let mut expect_name = false; + let mut keyword = (0i64, 0i64, 0usize); + let mut declare = 0u8; // 1: `declare`, 2: `declare global` + let mut declare_kw = (0i64, 0i64, 0usize); // the `declare` keyword's position + let mut ambient_module = false; // just read `module 'x'` + let mut after_dot = false; + + ts_tokenize(source, &mut |kind, text, line, column, offset, _depth| { + if kind == TsTok::Punct && text == "{" { + let header = names.as_ref().filter(|n| !n.is_empty() && !expect_name).cloned(); + if declare == 2 { + let idx = out.len(); + out.push(TsScopeBlock { + name: "global".to_string(), ambient: false, global: true, + start: (declare_kw.0, declare_kw.1), end: (0, 0), + byte_start: declare_kw.2, byte_end: 0, text: String::new(), + default_name: None, export_aliases: Vec::new(), export_clause: false, + }); + open.push(TsNsFrame::Global(idx)); + } else if ambient_module { + open.push(TsNsFrame::Module); + } else if let Some(header) = header.filter(|_| !matches!(open.last(), Some(TsNsFrame::Other))) { + let parent_path = match open.last() { + Some(TsNsFrame::Ns(i)) => Some(out[*i].name.clone()), + _ => None, + }; + let idx = out.len(); + out.push(TsScopeBlock { + name: parent_path.map(|p| format!("{p}.")).unwrap_or_default() + &header.join("."), + ambient: false, + global: match open.last() { + Some(TsNsFrame::Ns(i)) => out[*i].global, + Some(TsNsFrame::Global(_)) => true, + None => top_level_global, + _ => false, + }, + start: (keyword.0, keyword.1), end: (0, 0), + byte_start: keyword.2, byte_end: 0, text: String::new(), + default_name: None, export_aliases: Vec::new(), export_clause: false, + }); + open.push(TsNsFrame::Ns(idx)); + } else { + open.push(TsNsFrame::Other); + } + names = None; + declare = 0; + ambient_module = false; + after_dot = false; + return; + } + if kind == TsTok::Punct && text == "}" { + match open.pop() { + Some(TsNsFrame::Ns(i)) | Some(TsNsFrame::Global(i)) => { + out[i].end = (line, column); + out[i].byte_end = offset + 1; + out[i].text = source[out[i].byte_start..(offset + 1).min(source.len())].to_string(); + } + _ => {} + } + names = None; + declare = 0; + ambient_module = false; + after_dot = false; + return; + } + ambient_module = names.as_ref().is_some_and(|n| expect_name && n.is_empty() && kind == TsTok::Str); + if let Some(n) = &mut names { + if expect_name && kind == TsTok::Word { + n.push(text.to_string()); + expect_name = false; + return; + } + if !expect_name && kind == TsTok::Punct && text == "." { + expect_name = true; + return; + } + names = None; + } + if kind == TsTok::Word { + if (text == "namespace" || text == "module") && !after_dot { + names = Some(Vec::new()); + expect_name = true; + keyword = (line, column, offset); + } + if text == "declare" { declare_kw = (line, column, offset); } + declare = if text == "declare" { 1 } else if declare == 1 && text == "global" { 2 } else { 0 }; + after_dot = false; + return; + } + declare = 0; + after_dot = kind == TsTok::Punct && text == "."; + }); +} + impl KernelResolver { - pub(super) fn ts_scope_scan(&mut self, file: &str, r: &ResolveRefIn) -> Option> { + pub(super) fn ts_scope_scan(&mut self, file: &str) -> Option> { if let Some(memo) = self.ts_scope_memo.get(file) { return memo.clone(); } - let scan = self.ts_scope_scan_uncached(file, r).map(Rc::new); + let scan = self.ts_scope_scan_uncached(file).map(Rc::new); self.ts_scope_memo.insert(file.to_string(), scan.clone()); scan } - fn ts_scope_scan_uncached(&mut self, file: &str, r: &ResolveRefIn) -> Option { + /// Port of upstream ambient-modules.ts: reads tokens, never a tree. A + /// `declare module '' { … }` in a script declaration file declares + /// the module; in a module file it augments one. + fn ts_scope_scan_uncached(&mut self, file: &str) -> Option { let source = self.read_file(file)?; - let mut site = r.clone(); site.file_path = file.to_string(); site.language = "typescript".to_string(); - let tree = self.js_binding_tree(&source, &site)?; - let children = super::iteration::named_children(tree.root_node()); - let module_file = children.iter().any(|n| matches!(n.kind(), "import_statement" | "export_statement") - && !source.text()[n.byte_range()].trim_start().starts_with("export as namespace")); - let mut pending = children; - let mut blocks = Vec::new(); - while let Some(node) = pending.pop() { - let raw = &source.text()[node.byte_range()]; - if matches!(node.kind(), "internal_module" | "module") { - if let Some(name) = node.child_by_field_name("name") { - let name = source.text()[name.byte_range()].trim().to_string(); - blocks.push(TsScopeBlock { - ambient: name.starts_with(['\'', '"']), global: name == "global", - name: name.trim_matches(['\'', '"']).to_string(), - start: (node.start_position().row as i64 + 1, node.start_position().column as i64), - end: (node.end_position().row as i64 + 1, node.end_position().column as i64), text: raw.to_string(), - }); - } - } else if node.kind() == "ambient_declaration" && re!(r"^declare\s+global\s*\{").is_match(raw.trim()) { - blocks.push(TsScopeBlock { name: "global".to_string(), ambient: false, global: true, - start: (node.start_position().row as i64 + 1, node.start_position().column as i64), - end: (node.end_position().row as i64 + 1, node.end_position().column as i64), text: raw.to_string() }); - } - pending.extend(super::iteration::named_children(node)); + let text = source.text(); + // Nothing the scan can find: not a module file, no blocks to name. + if !re!(r"\b(?:declare|namespace|module|import|export)\b").is_match(text) { + return Some(TsScopeScan { module_file: false, blocks: Vec::new() }); + } + let (script, ambient) = ts_scan_ambient(text); + let top_level_global = re!(r"\.d\.[cm]?ts$").is_match(file) && script; + let mut blocks: Vec = Vec::new(); + for b in ambient { + blocks.push(TsScopeBlock { + name: b.name, ambient: true, global: false, + start: b.start, end: b.end, + byte_start: b.byte_start, byte_end: b.byte_end, + text: text[b.byte_start..b.byte_end.min(text.len())].to_string(), + default_name: b.default_name, export_aliases: b.export_aliases, export_clause: b.export_clause, + }); } - Some(TsScopeScan { module_file, blocks }) + ts_scan_namespaces(text, top_level_global, &mut blocks); + blocks.sort_by_key(|b| b.start); + Some(TsScopeScan { module_file: !script, blocks }) } fn ts_scope_candidate(&mut self, n: &KNode, r: &ResolveRefIn) -> bool { - let Some(scan) = self.ts_scope_scan(&n.file_path, r) else { return false }; + let Some(scan) = self.ts_scope_scan(&n.file_path) else { return false }; let containing: Vec<_> = scan.blocks.iter().filter(|b| b.contains(n.start_line, n.start_column)).collect(); + // Upstream's `global` flag on a namespace says the namespace itself is + // a global name (its members still need the qualifier); the bare-name + // gates here need the literal `declare global` block, whose members + // are globals — upstream's `globalWords` positions. + let global_block = |b: &&TsScopeBlock| b.name == "global" && b.global; if n.file_path == r.file_path { - return containing.iter().all(|b| b.global || b.contains(r.line, r.column)); + return containing.iter().all(|b| global_block(b) || b.contains(r.line, r.column)); } if is_test_path(&n.file_path) && !is_test_path(&r.file_path) { return false; } - if containing.iter().any(|b| b.ambient || (!b.global && b.name != "global")) { return false; } - containing.iter().any(|b| b.global) || (!scan.module_file && re!(r"\.d\.[cm]?ts$").is_match(&n.file_path) && !n.qualified_name.contains("::")) + if containing.iter().any(|b| b.ambient || !global_block(b)) { return false; } + containing.iter().any(global_block) || (!scan.module_file && re!(r"\.d\.[cm]?ts$").is_match(&n.file_path) && !n.qualified_name.contains("::")) } pub(super) fn ts_import_target_visible(&mut self, node: &KNode, source_file: &str, r: &ResolveRefIn) -> Res { - let Some(scan) = self.ts_scope_scan(&node.file_path, r) else { return Ok(true) }; + let Some(scan) = self.ts_scope_scan(&node.file_path) else { return Ok(true) }; let Some(block) = scan.blocks.iter().find(|b| b.ambient && b.contains(node.start_line, node.start_column)) else { return Ok(true) }; Ok(self.import_mappings(source_file)?.iter().any(|m| m.source == block.name) || self.import_mappings(&r.file_path)?.iter().any(|m| m.source == block.name)) @@ -549,7 +984,7 @@ impl KernelResolver { fn ts_namespace_member(&mut self, file: &str, written: &str, r: &ResolveRefIn, imported: bool) -> Res>> { let Some((owner, name)) = written.rsplit_once('.') else { return Ok(None) }; - let Some(scan) = self.ts_scope_scan(file, r) else { return Ok(None) }; + let Some(scan) = self.ts_scope_scan(file) else { return Ok(None) }; let nodes = self.nodes_in_file(file)?; Ok(nodes.iter().find(|n| n.name == name && scan.blocks.iter().any(|b| b.name == owner && b.contains(n.start_line, n.start_column) && (imported || file == r.file_path || scan.blocks.iter().any(|g| g.global && g.contains(b.start.0, b.start.1)) || !scan.module_file))) @@ -576,17 +1011,44 @@ impl KernelResolver { Ok(None) } + /// The declaration files the project holds, computed once per resolution. + fn ts_dts_files(&mut self) -> Res> { + if self.ts_dts_files.is_none() { + let files = match self.sorted_files() { Some(files) => files, None => Arc::new(self.table()?.files.iter().cloned().collect()) }; + self.ts_dts_files = Some(files.iter().filter(|f| re!(r"\.d\.[cm]?ts$").is_match(f)).cloned().collect()); + } + Ok(self.ts_dts_files.clone().unwrap_or_default()) + } + + /// The files whose script `declare module ''` blocks declare + /// `specifier`, indexed by name — upstream's `scanned.bySpecifier`. Built + /// on first lookup; each file's scope scan is one text pass, memoized. + fn ts_ambient_files(&mut self, specifier: &str) -> Res> { + if self.ts_ambient_index.is_none() { + let mut index: HashMap> = HashMap::new(); + for file in self.ts_dts_files()? { + let Some(scan) = self.ts_scope_scan(&file) else { continue }; + if scan.module_file { continue; } + for name in scan.blocks.iter().filter(|b| b.ambient && !b.name.contains('*')).map(|b| b.name.strip_prefix("node:").unwrap_or(&b.name).to_string()) { + index.entry(name).or_default().push(file.clone()); + } + } + self.ts_ambient_index = Some(index); + } + Ok(self.ts_ambient_index.as_ref().unwrap().get(specifier).cloned().unwrap_or_default()) + } + pub(super) fn ts_ambient_lookup(&mut self, specifier: &str, exported: &str, default: bool, r: &ResolveRefIn) -> Res>> { if specifier.starts_with('.') || specifier.starts_with('/') { return Ok(None); } let specifier = specifier.strip_prefix("node:").unwrap_or(specifier); let memo_key = (format!("scope-export\0{specifier}\0{}", r.file_path), format!("{default}\0{exported}")); let cached = self.ts_ambient_memo.get(&memo_key).cloned(); if cached == Some(None) { return Ok(None); } - let files = match self.sorted_files() { Some(files) => files, None => Arc::new(self.table()?.files.iter().cloned().collect()) }; + let files = self.ts_ambient_files(specifier)?; let mut hits = Vec::new(); - for file in files.iter().filter(|f| re!(r"\.d\.[cm]?ts$").is_match(f) && cached.as_ref().is_none_or(|cached| cached.as_ref() == Some(*f))) { - let Some(scan) = self.ts_scope_scan(file, r) else { continue }; - if scan.module_file || (is_test_path(file) && !is_test_path(&r.file_path)) { continue; } + for file in files.iter().filter(|f| cached.as_ref().is_none_or(|cached| cached.as_ref() == Some(*f))) { + if is_test_path(file) && !is_test_path(&r.file_path) { continue; } + let Some(scan) = self.ts_scope_scan(file) else { continue }; for block in scan.blocks.iter().filter(|b| b.ambient && b.name.strip_prefix("node:").unwrap_or(&b.name) == specifier) { if exported == "*" { if let Some(node) = self.nodes_in_file(file)?.iter().find(|n| n.kind == "file") { hits.push(node.clone()); } @@ -594,24 +1056,21 @@ impl KernelResolver { } let mut local = exported.to_string(); if default { - let Some(name) = re!(r"\bexport\s+(?:default\s+(?:(?:abstract\s+)?class\s+|function\s+)?|=\s*)([A-Za-z_$][\w$]*)").captures(&block.text).map(|m| m[1].to_string()) else { continue }; + let Some(name) = block.default_name.clone() else { continue }; local = name; } else { - for list in re!(r"\bexport\s+(?:type\s+)?\{([^}]*)\}").captures_iter(&block.text) { - for item in list[1].split(',') { - let parts: Vec<_> = item.trim().trim_start_matches("type ").split_whitespace().collect(); - if parts.len() == 3 && parts[1] == "as" && parts[2].trim_matches(['\'', '"']) == exported { local = parts[0].trim_matches(['\'', '"']).to_string(); } - } + for (exported_name, local_name) in &block.export_aliases { + if exported_name == exported { local = local_name.clone(); } } } - if !default && re!(r"\bexport\s+(?:type\s+)?\{").is_match(&block.text) && local == exported { + if !default && block.export_clause && local == exported { let direct = Self::cached_regex(&format!(r"\bexport\s+(?:(?:declare|abstract)\s+)?(?:interface|class|type|function|const|let|var|enum|namespace)\s+{}\b", regex::escape(exported)))?; let listed = Self::cached_regex(&format!(r"(?:\{{|,)\s*(?:type\s+)?{}\s*(?:,|\}})", regex::escape(exported)))?; if !direct.is_match(&block.text) && !listed.is_match(&block.text) { continue; } } let nodes = self.nodes_in_file(file)?; let named: Vec<_> = nodes.iter().filter(|n| n.name == local && block.contains(n.start_line, n.start_column) - && !scan.blocks.iter().any(|b| !b.ambient && !b.global && b.contains(n.start_line, n.start_column) && b.start > block.start && (n.start_line, n.start_column) > b.start)).cloned().collect(); + && !scan.blocks.iter().any(|b| !b.ambient && b.name != "global" && b.contains(n.start_line, n.start_column) && b.start > block.start && (n.start_line, n.start_column) > b.start)).cloned().collect(); hits.extend(named); } } @@ -795,7 +1254,7 @@ impl KernelResolver { if imported { return Ok(None); } } if !type_ref { return Ok(None); } - if let Some(scan) = self.ts_scope_scan(&r.file_path, r) { + if let Some(scan) = self.ts_scope_scan(&r.file_path) { for block in scan.blocks.iter().filter(|b| b.ambient && b.contains(r.line, r.column)) { if block.name.starts_with('.') { if let Some(file) = self.resolve_import_path(&block.name, &r.file_path, &r.language)? { @@ -815,7 +1274,7 @@ impl KernelResolver { } } } - if let Some(scan) = self.ts_scope_scan(&r.file_path, r) { + if let Some(scan) = self.ts_scope_scan(&r.file_path) { for block in scan.blocks.iter().filter(|b| b.ambient && b.contains(r.line, r.column)) { if let Some(node) = self.ts_ambient_lookup(&block.name, &written, false, r)? { return Ok(Some(Some(KCand { node, confidence: 0.95, resolved_by: "import" }))); From b3879457085a33454db05dbc689d9cfa42dadb4d Mon Sep 17 00:00:00 2001 From: bompus Date: Tue, 27 Oct 2026 09:24:45 -0600 Subject: [PATCH 2/5] docs(bench): re-measure index/sync after ambient-scan fix (c29349a3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Index and sync cells re-measured 2026-10-29 on the same host with two runs per arm; MCP, CLI and explore cells carry the 2026-10-27 measurements at bc27fdb4 (a resolution-path change cannot move them). supabase 41.5 s -> 19.6 s Node (inside spread of upstream's 18.2 s), n8n 178 s -> 68.4 s (upstream 74.9 s — fork win), discourse 22.2 s -> 14.4 s (upstream 19.6 s). Losses drop from 15 to 10 of 76 comparisons; the residual index losses are pretix/Alamofire on Node and the Bun arm of supabase/n8n/pretix. n8n sync rises 4.18 -> 8.90 s (lazy ambient index is also paid during sync resolution) and stays ahead of upstream's 11.6 s. Card and heatmap regenerated; benchmark doc records the regression's closure and the three upstream-faithfulness fixes the port carried. --- README.md | 38 +++++++------- assets/fork-vs-upstream-performance.svg | 8 +-- .../fork-vs-upstream-heatmap-2026-10-27.svg | 4 +- .../fork-vs-upstream-node-bun-2026-10-27.md | 51 +++++++++---------- 4 files changed, 50 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index 8a8e97c3e5..b952d1783c 100644 --- a/README.md +++ b/README.md @@ -198,11 +198,11 @@ Call resolution the fork adds: ### Measured results -Upstream `main` at `69af19ac` against the fork at `bc27fdb4`, each run on Node.js 24.22.1 and on Bun 1.4.4. Measured 2026-10-27 on a 16-vCPU WSL2 host over seven corpora, with the arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). +Upstream `main` at `69af19ac` against the fork at `c29349a3`, each run on Node.js 24.22.1 and on Bun 1.4.4. Index and sync figures were re-measured 2026-10-29 on the same 16-vCPU WSL2 host after the ambient-`declare module` scan fix below; MCP, command-line and explore figures are carried from the 2026-10-27 run at `bc27fdb4` (unchanged by a resolution-path change). Seven corpora, arms in mirrored order; index figures are the median of two runs and sync figures the median of four. A comparison counts as a change only when the gap exceeds that comparison's own run-to-run spread — the larger of the two arms' (max − min) / median — and cells that cannot show that are marked inconclusive or no clear change rather than a fixed band. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer; it ran here with `CODEGRAPH_ALLOW_UNSAFE_NODE=1`. The host shared other work during the run, so spread-based cells carry more noise than a quiet host would; the numbers, the method, the graph sizes and the disclosures are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md). The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). -![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 2.4 times slower, syncs an edited file 2.8 times faster, answers a warm question 1.8 times faster and uses 49% less memory while indexing. The fork is better in 56 of 76 comparisons, shows no clear change in 5 and is worse in 15.](assets/fork-vs-upstream-performance.svg?v=5) +![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 1.1 times faster, syncs an edited file 1.3 times faster, answers a warm question 1.8 times faster and uses 50% less memory while indexing. The fork is better in 58 of 76 comparisons, shows no clear change in 8 and is worse in 10.](assets/fork-vs-upstream-performance.svg?v=6) -The fork's one-file sync is faster than upstream's on six of the seven corpora — 11% (gin) to 64% (n8n) on Node, 19% (supabase, where upstream's spread is wide) to 57% (discourse) on Bun — with supabase on Node the one no-clear-change. Full indexing is the fork's weak surface this run: it is slower on five of seven corpora (supabase 2.3× and n8n 2.4× on Node; the supabase-resolution section below explains why), ties nowhere outside spread, and wins only on CPython (1.6× on Node). It uses less memory while indexing on all seven (18–49%). The 15 comparisons where the fork is worse are listed in the chart and the benchmark doc: six index-time cells, the busy-MCP memory on n8n (+18% on Node), the gin concurrent-explore wave, and small index-time cells on gin and Alamofire. The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. +The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 4% (pretix) to 31% (n8n), with supabase the one no-clear-change — and on Bun it wins every clear comparison but one, losing pretix by 11%. Full indexing is now a split surface: the fork wins on n8n (1.1×), CPython (1.6×) and discourse (1.4×) on Node, ties inside spread on gin and supabase-Node, and still loses on pretix (+33%) and Alamofire (+18%) and on the Bun arm of supabase, n8n and pretix. It uses less memory while indexing on all seven (18–50%). The 10 comparisons where the fork is worse are listed in the chart and the benchmark doc: five index-time cells (pretix on both runtimes, Alamofire on Node, supabase and n8n on Bun), the busy-MCP memory on n8n (+18% on Node), the gin concurrent-explore wave, a pretix-Bun sync cell and a gin file-watcher cell. The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. The fork starts its resolver pool at or above 150,000 pending references. Below that, it can start the pool part-way through if at least 20,000 unresolved references remain and the estimated main-thread work is at least 0.4 s. An explicit `CODEGRAPH_PARALLEL_RESOLVE_MIN` replaces both gates. Against a pre-change baseline build, five tested corpora indexed 11-48% faster at 56-132% more peak memory with identical graphs ([measurements](docs/benchmarks/adaptive-pool-engage-2026-10-15.md)). @@ -212,15 +212,15 @@ A [2026-10-15 mid-size comparison](docs/benchmarks/midsize-sweep-2026-10-15.md) | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin (Go, 119 files) | 0.96 s, 559 MiB | 1.09 s, 392 MiB | 1.06 s, 296 MiB | -| Alamofire (Swift, 129 files) | 1.35 s, 879 MiB | 1.62 s, 632 MiB | 0.99 s, 664 MiB | -| pretix (Python + JS, 1,472 files) | 9.04 s, 3.29 GiB | 12.5 s, 2.71 GiB | 12.4 s, 2.42 GiB | -| CPython (C + Python, 3,708 files) | 55.0 s, 7.15 GiB | 33.8 s, 4.01 GiB | 33.5 s, 3.48 GiB | -| discourse (Ruby + JS, 20,277 files) | 19.6 s, 4.07 GiB | 22.2 s, 2.82 GiB | 20.8 s, 2.40 GiB | -| supabase (React + Next.js + TS, 10,716 files) | 18.2 s, 4.86 GiB | 41.5 s, 3.59 GiB | 40.6 s, 3.76 GiB | -| n8n (Vue + TS, 24,434 files) | 74.9 s, 12.2 GiB | 3:00, 6.27 GiB | 2:57, 6.33 GiB | +| gin (Go, 119 files) | 0.96 s, 559 MiB | 1.03 s, 396 MiB | 0.88 s, 296 MiB | +| Alamofire (Swift, 129 files) | 1.35 s, 879 MiB | 1.60 s, 648 MiB | 0.94 s, 662 MiB | +| pretix (Python + JS, 1,472 files) | 9.04 s, 3.29 GiB | 12.0 s, 2.71 GiB | 11.8 s, 2.44 GiB | +| CPython (C + Python, 3,708 files) | 55.0 s, 7.15 GiB | 33.4 s, 4.03 GiB | 33.3 s, 3.47 GiB | +| discourse (Ruby + JS, 20,277 files) | 19.6 s, 4.07 GiB | 14.4 s, 2.91 GiB | 13.2 s, 2.40 GiB | +| supabase (React + Next.js + TS, 10,716 files) | 18.2 s, 4.86 GiB | 19.6 s, 3.75 GiB | 18.7 s, 3.61 GiB | +| n8n (Vue + TS, 24,434 files) | 74.9 s, 12.2 GiB | 68.4 s, 6.18 GiB | 65.2 s, 6.61 GiB | -The fork's index is slower than upstream's on five of the seven corpora this run — clearly so on the two TypeScript-heavy monorepos (supabase 2.3×, n8n 2.4×) and pretix (+38%). The supabase and n8n regressions trace to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scans every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic — without it the supabase index never finished — but the remaining scan is still per-reference × per-file work upstream does once per candidate name. The fix for the residual gap is tracked as follow-up work; the benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). In return the fork's index uses 18–49% less memory on every corpus, and its one-file sync is faster on six of seven (below). +The fork's index was slower than upstream's on five of the seven corpora in the first pass of this run — the supabase and n8n regressions traced to upstream's ambient-`declare module` lookup (merged in fork #548): the Rust port scanned every declaration file per out-of-repo import. A per-file memo ([#554](https://github.com/bompus/codegraph/pull/554)) ended the reparse-every-reference quadratic, and the follow-up above lands the rest: a port of upstream's text tokenizer plus a `specifier → files` index, so each import checks only candidate declaration files instead of every `.d.ts`. The re-measured numbers in this table are the result: supabase went from 41.5 s to 19.6 s on Node and n8n from 178 s to 68.4 s — supabase now ties upstream inside spread and n8n beats it. The remaining losses are pretix (+33% Node) and Alamofire (+18% Node), whose index time is not ambient-scan work, and the Bun arm of supabase (+14%) and n8n (+10%). The benchmark doc has the [full accounting](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#the-supabase-resolution-regression). In return the fork's index uses 18–50% less memory on every corpus, and its one-file sync is faster on six of seven (below). Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s; [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) put the cause in upstream's code). At `69af19ac` upstream's CPython index is slower than at `986de95c` (55.0 s against 40.2 s on this host) while the fork's is unchanged, so the fork's CPython lead is 1.6× this run. @@ -228,13 +228,13 @@ Upstream at `6560052` took 2.0 to 2.7 times as long to index Python as at `290e0 | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin | 0.47 s | 0.42 s | 0.32 s | -| Alamofire | 0.48 s | 0.32 s | 0.21 s | -| pretix | 1.62 s | 1.07 s | 0.81 s | -| CPython | 5.35 s | 2.42 s | 1.70 s | -| discourse | 4.78 s | 2.31 s | 1.73 s | -| supabase | 4.58 s | 3.37 s | 2.34 s | -| n8n | 11.6 s | 4.18 s | 3.41 s | +| gin | 0.47 s | 0.40 s | 0.29 s | +| Alamofire | 0.48 s | 0.29 s | 0.18 s | +| pretix | 1.62 s | 1.55 s | 1.27 s | +| CPython | 5.35 s | 2.37 s | 1.62 s | +| discourse | 4.78 s | 2.28 s | 1.66 s | +| supabase | 4.58 s | 2.83 s | 1.90 s | +| n8n | 11.6 s | 8.90 s | 8.02 s | Upstream's supabase sync works in this run — the `FOREIGN KEY constraint failed` seen in every prior run ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565)) was fixed upstream on 2026-10-19. Its two passes differed widely on Node (1.05 s and 8.1 s); the 4.58 s figure is the median of four, and the spread means that comparison counts as no clear change rather than a fork win. @@ -280,7 +280,7 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne - **Larger database:** 11–20% bigger on six of the seven corpora above, and 37% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. - **Fewer edges on some projects:** 8% fewer on pretix and 2% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. -- **Slower to index most projects this run:** in the 2026-10-27 comparison the fork's full index is slower on five of seven corpora — supabase 2.3× and n8n 2.4× on Node, pretix +38% (+72% on Bun), Alamofire +20%, discourse +13% (+15% on Bun) and gin +14% on Node; only CPython is faster (1.6×). The TypeScript-monorepo regressions trace to the ambient-`declare module` port described above; the residual gap is tracked follow-up work. One-file syncs still peak higher on some corpora (2026-10-18 run: CPython 1.23 GiB against 909 MiB on Node, +39%; Alamofire +14%; gin +10%; discourse +4%). +- **Slower to index some projects this run:** in the 2026-10-27/29 comparison the fork's full index loses on two Node cells — pretix +33% and Alamofire +18% — and on the Bun arm of supabase (+14%), n8n (+10%) and pretix (+64%); it wins n8n (1.1×), CPython (1.6×) and discourse (1.4×) on Node and ties gin (both runtimes) and supabase-Node inside spread. The earlier TypeScript-monorepo index regressions traced to the ambient-`declare module` port described above and are closed by this change's text-scan + specifier index. One-file syncs still peak higher on some corpora (2026-10-18 run: CPython 1.23 GiB against 909 MiB on Node, +39%; Alamofire +14%; gin +10%; discourse +4%). - **On Bun:** slower concurrent explores on small projects (gin wave +20% this run), and a mixed peak-memory picture — 11–25% lower on four corpora, 2–6% higher on three. --- diff --git a/assets/fork-vs-upstream-performance.svg b/assets/fork-vs-upstream-performance.svg index 084ef01139..25b2c7cdfc 100644 --- a/assets/fork-vs-upstream-performance.svg +++ b/assets/fork-vs-upstream-performance.svg @@ -1,6 +1,6 @@ - + Fork vs upstream CodeGraph: headline numbers -On n8n the fork indexes in 178.0 s against 74.9 s, syncs one edited file in 4.18 s against 11.6 s, and answers a warm question in 131 ms against 242 ms. Of 76 comparisons it is better in 56, about the same in 5 and worse in 15. - -Index on n8n 2.4× slower, CPython 1.6× faster; the fork uses 44–49% less memoryNumbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.Index n8n from scratch2.4× slower74.9 s → 178.0 son Bun: 59.4 s → 176.9 sUpdate after editing one file2.8× faster11.6 s → 4.18 son Bun: 8.68 s → 3.41 sAnswer a question (warm)1.8× faster242 ms → 131 mson Bun: 211 ms → 107 msMemory while indexing n8n49% less12.39 GiB → 6.30 GiBon Bun: 9.23 GiB → 6.41 GiBMCP server memory when idle19% less4.40 GiB → 3.58 GiBon Bun: 3.40 GiB → 1.37 GiBStart the command line2.1× faster73 ms → 35 mson Bun: 36 ms → 21 msWhere the fork is not betterIndexing gin: 0.96 s → 1.09 s on Node (+14%)Indexing Alamofire: 1.35 s → 1.62 s on Node (+20%)Indexing pretix: 9.04 s → 12.5 s on Node (+38%), 7.20 s → 12.4 s on Bun (+72%)Indexing discourse: 19.6 s → 22.2 s on Node (+13%), 18.0 s → 20.8 s on Bun (+15%)Indexing supabase: 18.2 s → 41.5 s on Node (+128%), 16.4 s → 40.6 s on Bun (+147%)Indexing n8n: 74.9 s → 178.0 s on Node (+138%), 59.4 s → 176.9 s on Bun (+198%)Syncing one file in gin: 0.29 s → 0.32 s on Bun (+9%)Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%)8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%)File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%)No clear change (inside the comparison's own run-to-run spread): indexing gin on Bun; memory indexing supabase onNode; syncing one file in supabase on Node; memory busy on n8n (MCP server) on Bun; cPU idle on gin (MCP server) onBun.Over all 76 comparisons on Node and Bun: better in 56, no clear change in 5, worse in 15. A difference counts only whenit exceeds that comparison's own run-to-run spread.Measured 2026-10-27 · upstream 69af19ac vs fork bc27fdb4 · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted +On n8n the fork indexes in 68.4 s against 74.9 s, syncs one edited file in 8.90 s against 11.6 s, and answers a warm question in 131 ms against 242 ms. Of 76 comparisons it is better in 58, about the same in 8 and worse in 10. + +Index on n8n 1.1× faster, CPython 1.6× faster; the fork uses 44–50% less memoryNumbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.Index n8n from scratch1.1× faster74.9 s → 68.4 son Bun: 59.4 s → 65.2 sUpdate after editing one file1.3× faster11.6 s → 8.90 son Bun: 8.68 s → 8.02 sAnswer a question (warm)1.8× faster242 ms → 131 mson Bun: 211 ms → 107 msMemory while indexing n8n50% less12.39 GiB → 6.18 GiBon Bun: 9.23 GiB → 6.61 GiBMCP server memory when idle19% less4.40 GiB → 3.58 GiBon Bun: 3.40 GiB → 1.37 GiBStart the command line2.1× faster73 ms → 35 mson Bun: 36 ms → 21 msWhere the fork is not betterIndexing Alamofire: 1.35 s → 1.60 s on Node (+18%)Indexing pretix: 9.04 s → 12.0 s on Node (+33%), 7.20 s → 11.8 s on Bun (+64%)Indexing supabase: 16.4 s → 18.7 s on Bun (+14%)Indexing n8n: 59.4 s → 65.2 s on Bun (+10%)Syncing one file in pretix: 1.13 s → 1.27 s on Bun (+11%)Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%)8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%)File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%)No clear change (inside the comparison's own run-to-run spread): indexing gin on Node; indexing gin on Bun; indexingsupabase on Node; memory indexing supabase on Node; syncing one file in gin on Bun; syncing one file in supabase onNode; memory busy on n8n (MCP server) on Bun; cPU idle on gin (MCP server) on Bun.Over all 76 comparisons on Node and Bun: better in 58, no clear change in 8, worse in 10. A difference counts only whenit exceeds that comparison's own run-to-run spread.Measured 2026-10-29 · upstream 69af19ac vs fork c29349a3 · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted diff --git a/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg index 75c7febecf..009b49568b 100644 --- a/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg +++ b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-27.svg @@ -1,6 +1,6 @@ Fork vs upstream CodeGraph: every result -Heatmap of the fork's change against upstream for full index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. Of 76 comparisons it is better in 56, about the same in 5 and worse in 15. +Heatmap of the fork's change against upstream for full index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. Of 76 comparisons it is better in 58, about the same in 8 and worse in 10. -Every result at a glanceEach cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better beyond the comparison’s own run-to-run spread, amber = inside it, red = worse, grey = inconclusive or no data.Full index: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node+14%+20%+38%−38%+13%+128%+138%Fork on Bun+10%−26%+72%−34%+15%+147%+198%Full index: peak memoryginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−30%−28%−18%−44%−31%−26%−49%Fork on Bun−52%−32%−16%−50%−44%−5%−31%One-file sync: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−12%−33%−34%−55%−52%−26%−64%Fork on Bun+9%−34%−29%−57%−46%−65%−61%MCP server on n8nStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−10%−46%−20%−33%+18%−19%−19%Fork on Bun−10%−50%−17%−46%0%−60%−49%MCP server on ginStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−24%−7%+18%−6%−26%−23%−43%Fork on Bun−31%−13%+5%0%−42%−33%−44%Command-line startup on gin--versionstatusexploreFork on Node−52%−32%−47%Fork on Bun−42%−30%−61%Measured 2026-10-27 · upstream 69af19ac vs fork bc27fdb4 · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted +Every result at a glanceEach cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better beyond the comparison’s own run-to-run spread, amber = inside it, red = worse, grey = inconclusive or no data.Full index: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node+7%+18%+33%−39%−26%+8%−9%Fork on Bun−8%−30%+64%−35%−27%+14%+10%Full index: peak memoryginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−29%−26%−18%−44%−29%−23%−50%Fork on Bun−52%−32%−16%−50%−43%−10%−28%One-file sync: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−17%−40%−4%−56%−52%−38%−23%Fork on Bun0%−42%+11%−59%−49%−72%−8%MCP server on n8nStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−10%−46%−20%−33%+18%−19%−19%Fork on Bun−10%−50%−17%−46%0%−60%−49%MCP server on ginStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−24%−7%+18%−6%−26%−23%−43%Fork on Bun−31%−13%+5%0%−42%−33%−44%Command-line startup on gin--versionstatusexploreFork on Node−52%−32%−47%Fork on Bun−42%−30%−61%Measured 2026-10-29 · upstream 69af19ac vs fork c29349a3 · Node 24.22.1, Bun 1.4.4 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted diff --git a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md index 7ac0005d83..587b9d04a3 100644 --- a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md +++ b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md @@ -1,6 +1,6 @@ # Fork against upstream CodeGraph on Node.js and Bun, 2026-10-27 -Upstream `main` at `69af19ac` against `fork/consolidated` at `bc27fdb4` (the ambient-scope memo from [#554](https://github.com/bompus/codegraph/pull/554)), each on Node.js 24.22.1 and on Bun 1.4.4, over seven corpora, the MCP server and command-line startup. It repeats the [2026-10-18 run](fork-vs-upstream-node-bun-2026-10-18.md) with the same scripts and the same pinned corpora on the current builds. The README's "Measured results" quote this page. +Upstream `main` at `69af19ac` against `fork/consolidated` at `c29349a3`, each on Node.js 24.22.1 and on Bun 1.4.4, over seven corpora, the MCP server and command-line startup. It repeats the [2026-10-18 run](fork-vs-upstream-node-bun-2026-10-18.md) with the same scripts and the same pinned corpora on the current builds. Index and sync figures were re-measured 2026-10-29 on the same host after the ambient-scan rework described in [The supabase resolution regression](#the-supabase-resolution-regression); MCP, CLI and explore cells are unchanged by a resolution-path change and carry the 2026-10-27 measurements at `bc27fdb4`. The README's "Measured results" quote this page. ## Method @@ -39,13 +39,13 @@ Time and peak memory, median of two. | Corpus | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | | --- | --- | --- | --- | --- | -| gin | 0.96 s, 558 MiB | 1.09 s, 392 MiB | 1.06 s, 296 MiB | 0.96 s, 618 MiB | -| Alamofire | 1.35 s, 879 MiB | 1.62 s, 632 MiB | 0.99 s, 664 MiB | 1.35 s, 974 MiB | -| pretix | 9.04 s, 3.29 GiB | 12.5 s, 2.71 GiB | 12.4 s, 2.42 GiB | 7.20 s, 2.89 GiB | -| CPython | 55.0 s, 7.15 GiB | 33.8 s, 4.01 GiB | 33.5 s, 3.48 GiB | 51.0 s, 6.92 GiB | -| discourse | 19.6 s, 4.07 GiB | 22.2 s, 2.82 GiB | 20.8 s, 2.40 GiB | 18.0 s, 4.25 GiB | -| supabase | 18.2 s, 4.86 GiB | 41.5 s, 3.59 GiB | 40.6 s, 3.81 GiB | 16.4 s, 4.02 GiB | -| n8n | 74.9 s, 12.39 GiB | 178.0 s, 6.30 GiB | 176.9 s, 6.41 GiB | 59.4 s, 9.23 GiB | +| gin | 0.96 s, 558 MiB | 1.03 s, 396 MiB | 0.88 s, 296 MiB | 0.96 s, 618 MiB | +| Alamofire | 1.35 s, 879 MiB | 1.60 s, 648 MiB | 0.94 s, 662 MiB | 1.35 s, 974 MiB | +| pretix | 9.04 s, 3.29 GiB | 12.0 s, 2.71 GiB | 11.8 s, 2.44 GiB | 7.20 s, 2.89 GiB | +| CPython | 55.0 s, 7.15 GiB | 33.4 s, 4.03 GiB | 33.3 s, 3.47 GiB | 51.0 s, 6.92 GiB | +| discourse | 19.6 s, 4.07 GiB | 14.4 s, 2.91 GiB | 13.2 s, 2.40 GiB | 18.0 s, 4.25 GiB | +| supabase | 18.2 s, 4.86 GiB | 19.6 s, 3.75 GiB | 18.7 s, 3.61 GiB | 16.4 s, 4.02 GiB | +| n8n | 74.9 s, 12.39 GiB | 68.4 s, 6.18 GiB | 65.2 s, 6.61 GiB | 59.4 s, 9.23 GiB | Graph size. Node and edge counts match across runtimes for each build; database sizes are from the Node runs: @@ -57,7 +57,7 @@ Graph size. Node and edge counts match across runtimes for each build; database | CPython | 167,332 / 615,244 / 492 MiB | 167,923 / 603,861 / 573 MiB | | discourse | 163,411 / 309,678 / 389 MiB | 168,110 / 308,849 / 432 MiB | | supabase | 114,440 / 292,126 / 335 MiB | 152,274 / 327,838 / 460 MiB | -| n8n | 315,422 / 1,239,920 / 1.43 GiB | 328,519 / 1,233,216 / 1.67 GiB | +| n8n | 315,422 / 1,239,920 / 1.43 GiB | 328,519 / 1,233,163 / 1.67 GiB | The edge counts differ because the builds resolve differently: the fork adds Markdown, route and dispatch links, and declines call links it cannot confirm. The timings compare the cost of building each build's own graph, not the same graph. @@ -67,13 +67,13 @@ Median of four (two syncs per run, two runs). | Corpus | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | | --- | --- | --- | --- | --- | -| gin | 0.47 s | 0.42 s | 0.31 s | 0.29 s | -| Alamofire | 0.48 s | 0.32 s | 0.20 s | 0.31 s | -| pretix | 1.62 s | 1.07 s | 0.81 s | 1.13 s | -| CPython | 5.35 s | 2.42 s | 1.70 s | 3.92 s | -| discourse | 4.78 s | 2.31 s | 1.73 s | 3.22 s | -| supabase | 4.58 s | 3.37 s | 2.34 s | 6.69 s | -| n8n | 11.6 s | 4.18 s | 3.41 s | 8.68 s | +| gin | 0.47 s | 0.40 s | 0.29 s | 0.29 s | +| Alamofire | 0.48 s | 0.29 s | 0.18 s | 0.31 s | +| pretix | 1.62 s | 1.55 s | 1.27 s | 1.13 s | +| CPython | 5.35 s | 2.37 s | 1.62 s | 3.92 s | +| discourse | 4.78 s | 2.28 s | 1.66 s | 3.22 s | +| supabase | 4.58 s | 2.83 s | 1.90 s | 6.69 s | +| n8n | 11.6 s | 8.90 s | 8.02 s | 8.68 s | Upstream's supabase sync, which failed with `FOREIGN KEY constraint failed` in every prior run, now completes ([colbymchenry/codegraph#2565](https://github.com/colbymchenry/codegraph/issues/2565) was fixed upstream on 2026-10-19). Its two passes differ widely on Node (1.05 s then 8.1 s); the 4.58 s figure is the median of the four, and against that spread the comparison counts as no clear change rather than a fork win. @@ -86,7 +86,8 @@ Worth documenting because the README's earlier "ties on supabase" line hid a cli - Perf on a symbolized kernel showed ~65% of resolution cycles inside `ts_parser_parse`, and a `[tree-miss]` counter showed ~69,000 parses in 40 s: all `imports` refs, 417 distinct `.d.ts` files re-parsed ~2,440 times each. - Cause: `ts_ambient_lookup` (the port of upstream's ambient-module lookup) scans every `.d.ts` for each out-of-repo import and `ts_scope_scan` re-parsed each file on every call; the 32-entry FIFO tree cache cannot hold the working set. Upstream's equivalent is a tokenizer over source text (`scanAmbientModules` never parses a tree) and upstream memoizes the global-scope scan per file (`TS_GLOBAL_SCOPES`). - `d364683a` memoizes `ts_scope_scan` per file in the resolver (PR #554). supabase indexes in ~40–47 s, with 152,274 nodes / 327,838 edges (more edges than pre-#548's 323,321 — the ambient work does resolve). -- The remaining ~2.3–2.5× gap vs upstream's ~17–19 s on this corpus is real: the fork still parses each `.d.ts` once per resolver worker where upstream scans text once per file. It is bounded work, not the quadratic, and it is the honest number this page reports. +- The remaining ~2.3–2.5× gap vs upstream's ~17–19 s on this corpus is real: the fork still parses each `.d.ts` once per resolver worker where upstream scans text once per file. It is bounded work, not the quadratic. +- **Closed on 2026-10-29** at `c29349a3`: `ts_scope_scan` is now a port of upstream's text tokenizer + `scanAmbientModules`/`scanNamespaceBlocks` state machines (one text pass per file, memoized), and `ts_ambient_lookup` consults a lazily built `specifier → files` index (upstream's `scanned.bySpecifier`) instead of walking every `.d.ts` per import. supabase went from 41.5 s to 19.6 s on Node (inside spread of upstream's 18.2 s), n8n from 178 s to 68.4 s (upstream 74.9 s — a fork win), discourse from 22.2 s to 14.4 s (upstream 19.6 s). The port kept the fork's `export type { A as B }` clause support and fixed three divergences found in review: `export as namespace` now marks a file as a module (upstream semantics), the namespace `global` flag means "the namespace itself is a global name" — not "members are bare globals" (bare globals come only from literal `declare global` blocks and script top level) — and nested namespaces no longer leak into ambient member matching. supabase's graph is bit-identical to the pre-tokenizer baseline; n8n's edges moved 1,233,216 → 1,233,163 from the corrected namespace-global admission. The residual index losses are pretix (+33% Node) and Alamofire (+18% Node) — not ambient-scan work — and the Bun arm of supabase (+14%), n8n (+10%) and pretix (+64%). ## MCP server @@ -132,17 +133,15 @@ The fork-Node server's busy memory on n8n (4.54 GiB against upstream's 3.85) is ## Every result at a glance -![Heatmap of the fork's change against upstream for index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup, on Node and on Bun. Lower is better; cells inside the comparison's own run-to-run spread are marked no clear change.](fork-vs-upstream-heatmap-2026-10-27.svg?v=1) +![Heatmap of the fork's change against upstream for index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup, on Node and on Bun. Lower is better; cells inside the comparison's own run-to-run spread are marked no clear change.](fork-vs-upstream-heatmap-2026-10-27.svg?v=2) -Of 76 comparisons the fork is better in 56, shows no clear change in 5 and is worse in 15. The 15, generated from the same data as the heatmap: +Of 76 comparisons the fork is better in 58, shows no clear change in 8 and is worse in 10. The 10, generated from the same data as the heatmap: -- Indexing gin: 0.96 s → 1.09 s on Node (+14%) -- Indexing Alamofire: 1.35 s → 1.62 s on Node (+20%) -- Indexing pretix: 9.04 s → 12.5 s on Node (+38%), 7.20 s → 12.4 s on Bun (+72%) -- Indexing discourse: 19.6 s → 22.2 s on Node (+13%), 18.0 s → 20.8 s on Bun (+15%) -- Indexing supabase: 18.2 s → 41.5 s on Node (+128%), 16.4 s → 40.6 s on Bun (+147%) -- Indexing n8n: 74.9 s → 178.0 s on Node (+138%), 59.4 s → 176.9 s on Bun (+198%) -- Syncing one file in gin: 0.29 s → 0.32 s on Bun (+9%) +- Indexing Alamofire: 1.35 s → 1.60 s on Node (+18%) +- Indexing pretix: 9.04 s → 12.0 s on Node (+33%), 7.20 s → 11.8 s on Bun (+64%) +- Indexing supabase: 16.4 s → 18.7 s on Bun (+14%) +- Indexing n8n: 59.4 s → 65.2 s on Bun (+10%) +- Syncing one file in pretix: 1.13 s → 1.27 s on Bun (+11%) - Memory busy on n8n (MCP server): 3.85 GiB → 4.54 GiB on Node (+18%) - 8 asks at once on gin (MCP server): 114 ms → 134 ms on Node (+18%), 153 ms → 161 ms on Bun (+5%) - File watcher on gin (MCP server): 357 ms → 358 ms on Bun (0%) From 2114130153d631278834690be5070d29bcd5f8ff Mon Sep 17 00:00:00 2001 From: bompus Date: Tue, 27 Oct 2026 10:15:40 -0600 Subject: [PATCH 3/5] fix(resolve): tokenizer string escapes + close-brace offset (#556) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two boundary bugs in ts_tokenize found in review: - `\X` skipped two bytes unconditionally, so a multi-byte escaped char left `from` inside a UTF-8 sequence and a trailing backslash left `from = len + 1` — both panicking on the next slice — and a `\` continuation was skipped without counting the line. - `}` passed its post-increment offset, so block byte_end ran one char past the close brace, slicing one trailing char into block text and panicking if that char was multi-byte. Also corrects the README sync range (max is CPython 56%, not n8n) and splits the bench doc's build provenance by metric per the page header. 269 focused resolver tests pass on the rebuilt kernel. --- README.md | 2 +- codegraph-kernel/src/resolve/ts_known.rs | 9 +++++++-- docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md | 4 ++-- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index b952d1783c..156ed22688 100644 --- a/README.md +++ b/README.md @@ -202,7 +202,7 @@ Upstream `main` at `69af19ac` against the fork at `c29349a3`, each run on Node.j ![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 1.1 times faster, syncs an edited file 1.3 times faster, answers a warm question 1.8 times faster and uses 50% less memory while indexing. The fork is better in 58 of 76 comparisons, shows no clear change in 8 and is worse in 10.](assets/fork-vs-upstream-performance.svg?v=6) -The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 4% (pretix) to 31% (n8n), with supabase the one no-clear-change — and on Bun it wins every clear comparison but one, losing pretix by 11%. Full indexing is now a split surface: the fork wins on n8n (1.1×), CPython (1.6×) and discourse (1.4×) on Node, ties inside spread on gin and supabase-Node, and still loses on pretix (+33%) and Alamofire (+18%) and on the Bun arm of supabase, n8n and pretix. It uses less memory while indexing on all seven (18–50%). The 10 comparisons where the fork is worse are listed in the chart and the benchmark doc: five index-time cells (pretix on both runtimes, Alamofire on Node, supabase and n8n on Bun), the busy-MCP memory on n8n (+18% on Node), the gin concurrent-explore wave, a pretix-Bun sync cell and a gin file-watcher cell. The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. +The fork's one-file sync is faster than upstream's on six of the seven corpora on Node — 4% (pretix) to 56% (CPython), with supabase the one no-clear-change — and on Bun it wins every clear comparison but one, losing pretix by 11%. Full indexing is now a split surface: the fork wins on n8n (1.1×), CPython (1.6×) and discourse (1.4×) on Node, ties inside spread on gin and supabase-Node, and still loses on pretix (+33%) and Alamofire (+18%) and on the Bun arm of supabase, n8n and pretix. It uses less memory while indexing on all seven (18–50%). The 10 comparisons where the fork is worse are listed in the chart and the benchmark doc: five index-time cells (pretix on both runtimes, Alamofire on Node, supabase and n8n on Bun), the busy-MCP memory on n8n (+18% on Node), the gin concurrent-explore wave, a pretix-Bun sync cell and a gin file-watcher cell. The benchmark doc has [every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md#every-result-at-a-glance) in one heatmap. The fork starts its resolver pool at or above 150,000 pending references. Below that, it can start the pool part-way through if at least 20,000 unresolved references remain and the estimated main-thread work is at least 0.4 s. An explicit `CODEGRAPH_PARALLEL_RESOLVE_MIN` replaces both gates. Against a pre-change baseline build, five tested corpora indexed 11-48% faster at 56-132% more peak memory with identical graphs ([measurements](docs/benchmarks/adaptive-pool-engage-2026-10-15.md)). diff --git a/codegraph-kernel/src/resolve/ts_known.rs b/codegraph-kernel/src/resolve/ts_known.rs index 8eea90d7a0..ec673841f2 100644 --- a/codegraph-kernel/src/resolve/ts_known.rs +++ b/codegraph-kernel/src/resolve/ts_known.rs @@ -599,7 +599,12 @@ fn ts_tokenize(source: &str, visit: &mut impl FnMut(TsTok, &str, i64, i64, usize if s == c || s == b'\n' { break; } if s == b'\\' { text.push_str(&source[from..j]); - j += 2; + j += 1; + if j < n { + if b[j] == b'\n' { line += 1; line_start = j + 1; } + j += 1; + while j < n && (b[j] & 0xC0) == 0x80 { j += 1; } + } from = j; continue; } @@ -634,7 +639,7 @@ fn ts_tokenize(source: &str, visit: &mut impl FnMut(TsTok, &str, i64, i64, usize continue; } depth = depth.saturating_sub(1); - visit(TsTok::Punct, "}", line, column as i64, i, depth); + visit(TsTok::Punct, "}", line, column as i64, i - 1, depth); regex_allowed = true; continue; } diff --git a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md index 587b9d04a3..2687bb93bb 100644 --- a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md +++ b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-27.md @@ -6,7 +6,7 @@ Upstream `main` at `69af19ac` against `fork/consolidated` at `c29349a3`, each on The scripts are in [`scripts/benchmarks/runtime/`](../../scripts/benchmarks/runtime/): `run-index-sync.sh`, `run-mcp.sh` and `run-cli.sh`, with the four arms below. `adapt-flat-output.mjs` stages the flat output for `render-charts.mjs`, which computes every table, the heatmap and the README card from the raw rows. -- **Builds.** Upstream `main` at `69af19ac` — the commit the fork's default branch is reconciled through — and the fork at `bc27fdb4`. The fork contains every upstream commit through `69af19ac` plus its own work, including this run's one change vs `31e3ab04`: a memo on the per-file ambient-scope scan. Without it, the fork's supabase index never completed (see [The supabase resolution regression](#the-supabase-resolution-regression)). Each build was installed with `npm ci`, with the native kernel built from source, and built with `npm run build`. +- **Builds.** Upstream `main` at `69af19ac` — the commit the fork's default branch is reconciled through — and the fork at `c29349a3` for index and sync, `bc27fdb4` for MCP, CLI and explore cells. The fork contains every upstream commit through `69af19ac` plus its own work. `bc27fdb4` carried a memo on the per-file ambient-scope scan (without it the fork's supabase index never completed — see [The supabase resolution regression](#the-supabase-resolution-regression)); `c29349a3` replaces the tree parse with the text-tokenizer port described there. Each build was installed with `npm ci`, with the native kernel built from source, and built with `npm run build`. - **Arms.** | Arm | Command | @@ -31,7 +31,7 @@ This run is noisier than 2026-10-18's. The host shared work with other agent ses One MCP attempt on n8n was additionally contaminated by a second benchmark writer that had not exited; its rows are preserved in `mcp-contaminated.jsonl` and the published table comes from a clean re-run. The idle-CPU rows from the shared host are the least trustworthy measurements here and are only quoted when their spread supports it. -On the pre-fix build, `fork-node` indexing supabase was killed at 10:53 wall by the liveness watchdog after ~46 CPU-minutes without finishing (exit 143; the row stays in `index-sync.tsv` as evidence). Resolution was re-parsing every `.d.ts` in the project for each out-of-repo import — about 69,000 tree parses in 40 seconds measured live — because the ambient-module scan introduced upstream's `declare module` lookup (upstream `553d5d10`/`214ef50e` era, merged in fork PR #548) with no per-file memo. `d364683a` adds the memo, which is the build every fork arm on this page runs. Details: [The supabase resolution regression](#the-supabase-resolution-regression). +On the pre-fix build, `fork-node` indexing supabase was killed at 10:53 wall by the liveness watchdog after ~46 CPU-minutes without finishing (exit 143; the row stays in `index-sync.tsv` as evidence). Resolution was re-parsing every `.d.ts` in the project for each out-of-repo import — about 69,000 tree parses in 40 seconds measured live — because the ambient-module scan introduced upstream's `declare module` lookup (upstream `553d5d10`/`214ef50e` era, merged in fork PR #548) with no per-file memo. `d364683a` adds the memo, which shipped in `bc27fdb4`; the index and sync cells on this page were re-measured on `c29349a3`, which lands the remaining fix. Details: [The supabase resolution regression](#the-supabase-resolution-regression). ## Full index From 64820aef338d82c293738e588ca86dc0c4bda4e4 Mon Sep 17 00:00:00 2001 From: bompus Date: Tue, 27 Oct 2026 10:16:55 -0600 Subject: [PATCH 4/5] test(eval): rebaseline precision at 21141301 Snapshots regenerated after the tokenizer fixes so codegraphSha names a build containing this PR rather than the e9561f46 base. All absent and present cases still hold on all nine corpora. --- ...-e9561f46.json => precision-exposed-2155404-21141301.json} | 4 ++-- ...1c-e9561f46.json => precision-flask-d73fa1c-21141301.json} | 4 ++-- ...a429-e9561f46.json => precision-gin-dcaa429-21141301.json} | 4 ++-- ...-e9561f46.json => precision-javalin-48bf31c-21141301.json} | 4 ++-- ...9561f46.json => precision-petclinic-818c413-21141301.json} | 4 ++-- ...bf6-e9561f46.json => precision-slim-3675bf6-21141301.json} | 4 ++-- ...3-e9561f46.json => precision-svelte-5895c63-21141301.json} | 4 ++-- ...422-e9561f46.json => precision-vite-8492422-21141301.json} | 4 ++-- ...5-e9561f46.json => precision-vitest-7c81815-21141301.json} | 4 ++-- 9 files changed, 18 insertions(+), 18 deletions(-) rename __tests__/evaluation/results/{precision-exposed-2155404-e9561f46.json => precision-exposed-2155404-21141301.json} (97%) rename __tests__/evaluation/results/{precision-flask-d73fa1c-e9561f46.json => precision-flask-d73fa1c-21141301.json} (95%) rename __tests__/evaluation/results/{precision-gin-dcaa429-e9561f46.json => precision-gin-dcaa429-21141301.json} (94%) rename __tests__/evaluation/results/{precision-javalin-48bf31c-e9561f46.json => precision-javalin-48bf31c-21141301.json} (95%) rename __tests__/evaluation/results/{precision-petclinic-818c413-e9561f46.json => precision-petclinic-818c413-21141301.json} (93%) rename __tests__/evaluation/results/{precision-slim-3675bf6-e9561f46.json => precision-slim-3675bf6-21141301.json} (91%) rename __tests__/evaluation/results/{precision-svelte-5895c63-e9561f46.json => precision-svelte-5895c63-21141301.json} (89%) rename __tests__/evaluation/results/{precision-vite-8492422-e9561f46.json => precision-vite-8492422-21141301.json} (96%) rename __tests__/evaluation/results/{precision-vitest-7c81815-e9561f46.json => precision-vitest-7c81815-21141301.json} (90%) diff --git a/__tests__/evaluation/results/precision-exposed-2155404-e9561f46.json b/__tests__/evaluation/results/precision-exposed-2155404-21141301.json similarity index 97% rename from __tests__/evaluation/results/precision-exposed-2155404-e9561f46.json rename to __tests__/evaluation/results/precision-exposed-2155404-21141301.json index 1365a871b4..d2cf83f114 100644 --- a/__tests__/evaluation/results/precision-exposed-2155404-e9561f46.json +++ b/__tests__/evaluation/results/precision-exposed-2155404-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:43.845Z", + "timestamp": "2026-10-27T16:16:50.659Z", "corpus": "exposed", "repo": "https://github.com/JetBrains/Exposed.git", "commit": "2155404", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 2, "absentHeld": 2, diff --git a/__tests__/evaluation/results/precision-flask-d73fa1c-e9561f46.json b/__tests__/evaluation/results/precision-flask-d73fa1c-21141301.json similarity index 95% rename from __tests__/evaluation/results/precision-flask-d73fa1c-e9561f46.json rename to __tests__/evaluation/results/precision-flask-d73fa1c-21141301.json index 11ebb34232..9c6d67c144 100644 --- a/__tests__/evaluation/results/precision-flask-d73fa1c-e9561f46.json +++ b/__tests__/evaluation/results/precision-flask-d73fa1c-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:22.537Z", + "timestamp": "2026-10-27T16:16:15.933Z", "corpus": "flask", "repo": "https://github.com/pallets/flask.git", "commit": "d73fa1c", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 0, "absentHeld": 0, diff --git a/__tests__/evaluation/results/precision-gin-dcaa429-e9561f46.json b/__tests__/evaluation/results/precision-gin-dcaa429-21141301.json similarity index 94% rename from __tests__/evaluation/results/precision-gin-dcaa429-e9561f46.json rename to __tests__/evaluation/results/precision-gin-dcaa429-21141301.json index 8ef66602ad..5c9f048554 100644 --- a/__tests__/evaluation/results/precision-gin-dcaa429-e9561f46.json +++ b/__tests__/evaluation/results/precision-gin-dcaa429-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:24.348Z", + "timestamp": "2026-10-27T16:16:17.620Z", "corpus": "gin", "repo": "https://github.com/gin-gonic/gin.git", "commit": "dcaa429", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 1, "absentHeld": 1, diff --git a/__tests__/evaluation/results/precision-javalin-48bf31c-e9561f46.json b/__tests__/evaluation/results/precision-javalin-48bf31c-21141301.json similarity index 95% rename from __tests__/evaluation/results/precision-javalin-48bf31c-e9561f46.json rename to __tests__/evaluation/results/precision-javalin-48bf31c-21141301.json index be22aca2f7..4821ad0d85 100644 --- a/__tests__/evaluation/results/precision-javalin-48bf31c-e9561f46.json +++ b/__tests__/evaluation/results/precision-javalin-48bf31c-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:47.168Z", + "timestamp": "2026-10-27T16:16:22.426Z", "corpus": "javalin", "repo": "https://github.com/javalin/javalin.git", "commit": "48bf31c", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 1, "absentHeld": 1, diff --git a/__tests__/evaluation/results/precision-petclinic-818c413-e9561f46.json b/__tests__/evaluation/results/precision-petclinic-818c413-21141301.json similarity index 93% rename from __tests__/evaluation/results/precision-petclinic-818c413-e9561f46.json rename to __tests__/evaluation/results/precision-petclinic-818c413-21141301.json index 8106221b97..fc50aabb30 100644 --- a/__tests__/evaluation/results/precision-petclinic-818c413-e9561f46.json +++ b/__tests__/evaluation/results/precision-petclinic-818c413-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:25.440Z", + "timestamp": "2026-10-27T16:16:18.608Z", "corpus": "petclinic", "repo": "https://github.com/spring-projects/spring-petclinic.git", "commit": "818c413", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 0, "absentHeld": 0, diff --git a/__tests__/evaluation/results/precision-slim-3675bf6-e9561f46.json b/__tests__/evaluation/results/precision-slim-3675bf6-21141301.json similarity index 91% rename from __tests__/evaluation/results/precision-slim-3675bf6-e9561f46.json rename to __tests__/evaluation/results/precision-slim-3675bf6-21141301.json index beb64cb221..544fe8b6f5 100644 --- a/__tests__/evaluation/results/precision-slim-3675bf6-e9561f46.json +++ b/__tests__/evaluation/results/precision-slim-3675bf6-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:48.445Z", + "timestamp": "2026-10-27T16:16:51.847Z", "corpus": "slim", "repo": "https://github.com/slimphp/Slim.git", "commit": "3675bf6", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 0, "absentHeld": 0, diff --git a/__tests__/evaluation/results/precision-svelte-5895c63-e9561f46.json b/__tests__/evaluation/results/precision-svelte-5895c63-21141301.json similarity index 89% rename from __tests__/evaluation/results/precision-svelte-5895c63-e9561f46.json rename to __tests__/evaluation/results/precision-svelte-5895c63-21141301.json index 9382f20dfe..dcd0c16fd0 100644 --- a/__tests__/evaluation/results/precision-svelte-5895c63-e9561f46.json +++ b/__tests__/evaluation/results/precision-svelte-5895c63-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:20.959Z", + "timestamp": "2026-10-27T16:16:14.439Z", "corpus": "svelte", "repo": "https://github.com/sveltejs/svelte.git", "commit": "5895c63", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 1, "absentHeld": 1, diff --git a/__tests__/evaluation/results/precision-vite-8492422-e9561f46.json b/__tests__/evaluation/results/precision-vite-8492422-21141301.json similarity index 96% rename from __tests__/evaluation/results/precision-vite-8492422-e9561f46.json rename to __tests__/evaluation/results/precision-vite-8492422-21141301.json index 56a02d7c0c..34a4e99c2b 100644 --- a/__tests__/evaluation/results/precision-vite-8492422-e9561f46.json +++ b/__tests__/evaluation/results/precision-vite-8492422-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:02.980Z", + "timestamp": "2026-10-27T16:15:50.773Z", "corpus": "vite", "repo": "https://github.com/vitejs/vite.git", "commit": "8492422", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 6, "absentHeld": 6, diff --git a/__tests__/evaluation/results/precision-vitest-7c81815-e9561f46.json b/__tests__/evaluation/results/precision-vitest-7c81815-21141301.json similarity index 90% rename from __tests__/evaluation/results/precision-vitest-7c81815-e9561f46.json rename to __tests__/evaluation/results/precision-vitest-7c81815-21141301.json index 2dd1ebecb1..cffb1951ec 100644 --- a/__tests__/evaluation/results/precision-vitest-7c81815-e9561f46.json +++ b/__tests__/evaluation/results/precision-vitest-7c81815-21141301.json @@ -1,9 +1,9 @@ { - "timestamp": "2026-10-27T14:33:13.040Z", + "timestamp": "2026-10-27T16:16:07.257Z", "corpus": "vitest", "repo": "https://github.com/vitest-dev/vitest.git", "commit": "7c81815", - "codegraphSha": "e9561f46", + "codegraphSha": "21141301", "summary": { "absentCases": 1, "absentHeld": 1, From 6e20c5e7a8a8269b2590e539d2e1424b0a112666 Mon Sep 17 00:00:00 2001 From: bompus Date: Tue, 27 Oct 2026 10:17:22 -0600 Subject: [PATCH 5/5] docs(readme): refresh Bun-vs-Node figures after re-measurement (#556) --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 156ed22688..b18cce0791 100644 --- a/README.md +++ b/README.md @@ -263,7 +263,7 @@ The fork-Node server's memory *while busy* is 18% above upstream's on n8n this r | `codegraph status` | 166 ms | 114 ms | 63 ms | | `codegraph explore ""` | 220 ms | 117 ms | 71 ms | -**Node or Bun.** The fork produces the same node and edge counts on both, and the full test suite passed on Node.js 24.21.0, Node.js 26.10.0 and Bun 1.4.2 at `48903f5` (`npm run test:bun`; one test is skipped under Bun for [oven-sh/bun#42891](https://github.com/oven-sh/bun/issues/42891)). Since then the suite has grown, and the Bun run skips 14 more tests that need what Bun does not provide: patching a builtin so the library sees it (`syncBuiltinESMExports` is a no-op, [oven-sh/bun#42616](https://github.com/oven-sh/bun/pull/42616)) and an emptied `PATH` blocking a spawn ([oven-sh/bun#44948](https://github.com/oven-sh/bun/issues/44948)). Each skip is measured at run time, so the test returns when Bun supports it. On Bun 1.4.4, fork index times match Node's within each corpus's own spread except Alamofire (−39%), sync times run 19–36% faster, and commands start faster (`--version` 21 ms against 35 ms). The idle MCP server holds 29% (gin) to 62% (n8n) less memory on Bun. Bun has one cost this run: eight concurrent explores on gin took 20% longer (161 ms against 134 ms); on n8n the wave runs 15% faster (1.63 s against 1.91 s). Peak index memory on Bun is 11–25% lower than Node on four corpora and 2–6% higher on Alamofire, supabase and n8n — no uniform direction. +**Node or Bun.** The fork produces the same node and edge counts on both, and the full test suite passed on Node.js 24.21.0, Node.js 26.10.0 and Bun 1.4.2 at `48903f5` (`npm run test:bun`; one test is skipped under Bun for [oven-sh/bun#42891](https://github.com/oven-sh/bun/issues/42891)). Since then the suite has grown, and the Bun run skips 14 more tests that need what Bun does not provide: patching a builtin so the library sees it (`syncBuiltinESMExports` is a no-op, [oven-sh/bun#42616](https://github.com/oven-sh/bun/pull/42616)) and an emptied `PATH` blocking a spawn ([oven-sh/bun#44948](https://github.com/oven-sh/bun/issues/44948)). Each skip is measured at run time, so the test returns when Bun supports it. On Bun 1.4.4, fork index times match Node's within each corpus's own spread except Alamofire (−41%), sync times run 10–38% faster, and commands start faster (`--version` 21 ms against 35 ms). The idle MCP server holds 29% (gin) to 62% (n8n) less memory on Bun. Bun has one cost this run: eight concurrent explores on gin took 20% longer (161 ms against 134 ms); on n8n the wave runs 15% faster (1.63 s against 1.91 s). Peak index memory on Bun is 4–25% lower than Node on five corpora and 2–7% higher on Alamofire and n8n — no uniform direction. Measured separately: @@ -281,7 +281,7 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne - **Larger database:** 11–20% bigger on six of the seven corpora above, and 37% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. - **Fewer edges on some projects:** 8% fewer on pretix and 2% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. - **Slower to index some projects this run:** in the 2026-10-27/29 comparison the fork's full index loses on two Node cells — pretix +33% and Alamofire +18% — and on the Bun arm of supabase (+14%), n8n (+10%) and pretix (+64%); it wins n8n (1.1×), CPython (1.6×) and discourse (1.4×) on Node and ties gin (both runtimes) and supabase-Node inside spread. The earlier TypeScript-monorepo index regressions traced to the ambient-`declare module` port described above and are closed by this change's text-scan + specifier index. One-file syncs still peak higher on some corpora (2026-10-18 run: CPython 1.23 GiB against 909 MiB on Node, +39%; Alamofire +14%; gin +10%; discourse +4%). -- **On Bun:** slower concurrent explores on small projects (gin wave +20% this run), and a mixed peak-memory picture — 11–25% lower on four corpora, 2–6% higher on three. +- **On Bun:** slower concurrent explores on small projects (gin wave +20% this run), and a mixed peak-memory picture — 4–25% lower on five corpora, 2–7% higher on two. ---