From a5fe6e1a278d68c76a8816125f67b29b5b5b8666 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 21 Oct 2026 16:56:19 +0000 Subject: [PATCH 01/95] fix(go): a call on an indexed element reaches its element type's method (#2755) A Go call on an indexed element (kubectl's `schemas[ix].ValidateBytes(data)`, deepcopy-gen's `(*in)[i].DeepCopyInto(&(*out)[i])`, go-to-protobuf's `m.Items[len(m.Items)-1].Unmarshal(...)`) reaches the resolver by its bare name, at the column where the receiver starts, and nothing typed the receiver: a method one type declares was guessed by name, often onto a namesake, and a common one like `DeepCopyInto` got nothing. matchGoIndexedCall, an early hook beside matchGoAssertedCall, reads the receiver chain with goChainLinks and types it as Go does: the head from the declaration in scope (goLocalDecl; readGoScopes now records where each `:=` or `var` value starts and which name it is) or a package-level variable; values (composite literals, make, new, append, conversions, fields, elements, addresses, range elements); struct fields, promoted ones too; and the element type of a slice, array, array pointer, map or named slice type, with `(*p)` dereferenced. The method is the element type's own, its interface's or one embedding promotes, and a type parameter's comes from its constraint. An element type from outside the project, a predeclared one, a type literal or an outside alias links nothing, and so does a method the type declares in a file of its package the index couldn't read (kubernetes' core/v1 generated.pb.go), which is read from that file's text. What can't be typed (a call's result) is left to the other strategies. On kubernetes 4c05e667 the indexed-element calls go from 760 resolved and 2,284 unresolved to 2,666 resolved and 378 unresolved; every changed site in kubernetes, prometheus and etcd is such a call, nodes are identical, and the express and flask controls are byte-identical. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/go-indexed-receiver.test.ts | 415 ++++++++++++++ src/resolution/index.ts | 6 +- src/resolution/name-matcher.ts | 774 +++++++++++++++++++++++++- 4 files changed, 1187 insertions(+), 9 deletions(-) create mode 100644 __tests__/go-indexed-receiver.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ff9e3ba31..0ddc05cd37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,6 +45,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, calls through a method receiver or parameter of an unexported type, the usual shape of gRPC and HTTP handlers (`s.service.AddItem()` inside `func (s *server) Create()`), now resolve, and always within that type's own package: another package's `server` with a same-named method no longer takes the call, and a method the type gets from a struct or interface it embeds is found too. Re-index Go projects after upgrading. Thanks @GoDiao for the report and the fix. (#2323) - In Go, a call made through a type assertion, like `srv.(KVServer).Range(ctx, in)` in generated gRPC code, `c.Reader.(*pipe).Close()` or `v.(storage.Store).Fetch(key)`, now links to that method of the asserted type, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, it linked to whichever method shared the name, so every gRPC handler pointed at the `Unimplemented…Server` stub instead of the server interface. Re-index Go projects after upgrading. - In Go, a call through a variable that holds the result of a type assertion, like `st.Fetch(key)` after `st := v.(storage.Store)` or `rc.Read(p)` after `rc, ok := v.(ReadCloser)`, now links to that method of the asserted type, as a call made through the assertion itself does, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, such a call linked to whichever method shared the name or to nothing, and a variable named like a standard-library package, like grpc-go's `parser, ok := builder.(balancer.ConfigParser)`, was taken for that package. Re-index Go projects after upgrading. +- In Go, a method called on an element of a slice, array or map, like kubectl's `schemas[ix].ValidateBytes(data)`, `pods[i].DeepCopy()`, `m.Items[len(m.Items)-1].Unmarshal(…)` in generated protobuf code or `(*in)[i].DeepCopyInto(&(*out)[i])` in every deepcopy file, now links to that method of the element's type. The type is read from what is indexed: a parameter, a variable, a struct field or a named slice type like `type ConjunctiveSchema []Schema`, and an element held by pointer, like `[]*Item`, reaches `Item`'s methods. Before, such a call linked to whichever method shared its name, often one in an unrelated package, or to nothing when many types declare it, as with Kubernetes' `DeepCopyInto`. An element whose type comes from outside your project, like `[]*net.IPNet`, or is a built-in one like `[]error`, now links to nothing rather than to a project method of that name. Re-index Go projects after upgrading. - In Go, calls into a module whose `go.mod` is not at the project root now resolve, whether it is a `server/` backend next to a `web/` frontend or one of several modules side by side as in etcd, so calls like `store.New()` and `s.db.CreateItem()` find their targets instead of being treated as calls into a third-party package. A name written through a package, like a `job.OPCommand` result type or a field of type `artifact.Manager`, now links to that package's symbol rather than a same-named one elsewhere, which also corrects links in projects with a single `go.mod`. Re-index Go projects after upgrading. Thanks @GoDiao for the report and @danusha2345 for the fix. (#2322) - In Go, an exported type written without a package, like `Node` in `func Walk(v Visitor, node Node)`, now links to the type of that name in its own package, whether it is a struct, an interface or another kind of type. Before, it could link to a struct of the same name in an unrelated package, so prometheus's PromQL parser functions pointed at the Kubernetes discovery `Node` struct instead of the parser's own `Node` interface, or, for a result type like `Appender` in `func (f *fanout) Appender(…) Appender`, to the method itself. A type from a package outside your project, like `apiv1.Node` or `http.Handler`, no longer links to a struct of the same name in the file that uses it; a variadic parameter like `...storage.Filter` now links to that package's type; a route handler written as a method value, like `h.Follow` in an Echo or Gin app, links to that method rather than a model struct named `Follow`; and a method called on another call's result or on an indexed value, like `err[i].Error()` or a chained `.String()`, is no longer recorded as creating a struct of the same name. Re-index Go projects after upgrading. - In Go, a function or method passed as a value, like `sync.Pool{New: wm.new}`, now shows the function that passes it among its callers and impact even when it is named `new`, `nil`, `None`, `self` or a handful of similar words, or when a line break or comment sits beside the dot. Before, such a value was skipped, so a method used only that way looked unused. Re-index Go projects after upgrading. diff --git a/__tests__/go-indexed-receiver.test.ts b/__tests__/go-indexed-receiver.test.ts new file mode 100644 index 0000000000..e4ba2e3b96 --- /dev/null +++ b/__tests__/go-indexed-receiver.test.ts @@ -0,0 +1,415 @@ +/** + * Go calls on an indexed element: `xs[i].M()`, `m[k].M()`, + * `(*in)[i].DeepCopyInto(…)`, `m.Items[len(m.Items)-1].Unmarshal(…)`. + * + * Such a call reaches the resolver by its bare method name, at the column + * where the receiver expression starts, and nothing typed the receiver: a + * method name only one type declares got a guess by name, a common one + * (every API type's `DeepCopyInto`) nothing at all. The receiver's element + * type is now read from the declaration of what is indexed: a slice, array, + * pointer to a slice through `(*x)`, map, or a named slice type, through a + * parameter, receiver, local, package-level variable or field. An element type + * from outside the project's modules, or a predeclared one, has no project + * method: no edge, as before for any receiver typed outside the project. + * + * Every method a shape test expects is also declared by a type of `alpha`, which + * sorts before the real package, so a guess by name can't land on the right + * one by luck. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-go-indexed-recv-')); + const files: Record = { + 'go.mod': 'module example.com/app\n\ngo 1.22\n', + 'alpha/decoy.go': `package alpha + +type Decoy struct{} + +func (d *Decoy) Validate() error { return nil } + +func (d Decoy) Key() string { return "" } + +func (d *Decoy) Unmarshal(dAtA []byte) error { return nil } + +func (d *Decoy) DeepCopyInto(out *Decoy) {} + +func (d Decoy) ValidateBytes(data []byte) error { return nil } + +type Addr struct{} + +func (a *Addr) Contains(ip []byte) bool { return false } +`, + 'model/item.go': `package model + +type Item struct { + Meta + Name string +} + +func (i *Item) Validate() error { return nil } + +func (i Item) Key() string { return i.Name } + +func (i *Item) Rename(name string) { i.Name = name } + +// Archive is the only method of its name in the project. +func (i *Item) Archive() {} + +type Validator interface { + Validate() error +} + +// Items is a named slice type. +type Items []*Item + +// Catalog holds a list of items and an index of them by name. +type Catalog struct { + List []Item + ByName map[string]*Item +} +`, + 'model/meta.go': `package model + +type Meta struct{} + +func (m *Meta) Unmarshal(dAtA []byte) error { return nil } + +func (m *Meta) Describe() string { return "" } +`, + // Over the index's 1 MB limit, so none of it is read: Item's own + // Unmarshal is missing from the graph. + 'model/generated.pb.go': `package model + +func (i *Item) Unmarshal(dAtA []byte) error { return nil } +${'// generated padding, never read by the indexer\n'.repeat(24_000)}`, + 'caller/caller.go': `package caller + +import ( + "net" + "sync" + + "example.com/app/alpha" + "example.com/app/model" +) + +func params(xs []*model.Item) error { + return xs[0].Validate() +} + +func array() string { + var ring [4]model.Item + return ring[1].Key() +} + +func maps(m map[string]*model.Item) error { + return m["a"].Validate() +} + +func deref(p *[]model.Item) string { + return (*p)[0].Key() +} + +func named(items model.Items) error { + return items[0].Validate() +} + +func fields(c *model.Catalog) (string, error) { + return c.List[len(c.List)-1].Key(), c.ByName["a"].Validate() +} + +func literals() (string, error) { + keys, decoys := []model.Item{{Name: "a"}}, []alpha.Decoy{} + byName := make(map[string]*model.Item) + _ = decoys + return keys[0].Key(), byName["a"].Validate() +} + +func sliced(items model.Items) error { + return items[1:][0].Validate() +} + +var defaults = []model.Item{{Name: "a"}} + +func packageVar() string { + return defaults[0].Key() +} + +func shadowed(xs []*model.Item) error { + { + xs := []alpha.Decoy{} + _ = xs + } + return xs[0].Validate() +} + +// A namesake of the call's element type, beside the call: the type the +// method was guessed on by name. +type local struct{} + +func (l *local) Rename(name string) {} + +func retarget(xs []model.Item) { + xs[0].Rename("b") +} + +// IPNet is the project's namesake of net.IPNet, and the one project type +// with a String, Contains or Error method. +type IPNet struct{} + +func (n *IPNet) String() string { return "" } + +func (n *IPNet) Contains(ip net.IP) bool { return false } + +func (n *IPNet) Error() string { return "" } + +func unread(xs []model.Item, dAtA []byte) error { + return xs[0].Unmarshal(dAtA) +} + +func promoted(xs []model.Item) string { + return xs[0].Describe() +} + +func outside(cidrs []*net.IPNet, errs []error) string { + return cidrs[0].String() + errs[0].Error() +} + +func outsideGuessed(cidrs []*net.IPNet, ip net.IP) bool { + return cidrs[0].Contains(ip) +} + +// Addr is net.IPNet under another name; alpha declares a struct Addr. +type Addr = net.IPNet + +func aliasOutside(addrs []Addr, ip net.IP) bool { + return addrs[0].Contains(ip) +} + +// Holder embeds a type from outside the project, and hands out its items. +type Holder struct { + sync.Mutex +} + +func (h *Holder) All() []*model.Item { return nil } + +func callResult(h *Holder) { + h.All()[0].Archive() +} + +// T is a type of this package, and the type parameter of First. +type T struct{} + +func (t *T) Validate() error { return nil } + +func First[T interface{ Validate() error }](xs []T) error { + return xs[0].Validate() +} + +func FirstNamed[T model.Validator](xs []T) error { + return xs[0].Validate() +} +`, + // kubectl's ConjunctiveSchema (kubernetes 4c05e667, + // staging/src/k8s.io/kubectl/pkg/validation/schema.go). + 'validation/schema.go': `package validation + +type Schema interface { + ValidateBytes(data []byte) error +} + +type NullSchema struct{} + +func (NullSchema) ValidateBytes(data []byte) error { return nil } + +type ConjunctiveSchema []Schema + +func (c ConjunctiveSchema) ValidateBytes(data []byte) error { + schemas := []Schema(c) + for ix := range schemas { + if err := schemas[ix].ValidateBytes(data); err != nil { + return err + } + } + return nil +} + +func (c ConjunctiveSchema) First(data []byte) error { + return c[0].ValidateBytes(data) +} +`, + // What deepcopy-gen and go-to-protobuf generate for a list type. + 'api/types.go': `package api + +type Item struct { + Tags []string +} + +type ItemList struct { + Items []Item +} + +func (in *Item) DeepCopyInto(out *Item) { + *out = *in +} + +func (in *ItemList) DeepCopyInto(out *ItemList) { + *out = *in + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]Item, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +func (m *Item) Unmarshal(dAtA []byte) error { return nil } + +func (m *ItemList) Unmarshal(dAtA []byte) error { + iNdEx, postIndex := 0, len(dAtA) + m.Items = append(m.Items, Item{}) + if err := m.Items[len(m.Items)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + return nil +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `file::qualifiedName` of every symbol the function or method `name` in `file` calls. */ +function callTargets(file: string, name: string, owner?: string): string[] { + const fn = cg.getNodesInFile(file).find((n) => + n.name === name && (owner === undefined || n.qualifiedName === `${owner}::${name}`)); + expect(fn, `${owner ? `${owner}::` : ''}${name} in ${file}`).toBeDefined(); + return cg + .getOutgoingEdges(fn!.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.filePath.replace(/\\/g, '/')}::${n.qualifiedName}`) + .sort(); +} + +describe('Go calls on an indexed element', () => { + it('a slice parameter of pointers reaches the element type\'s method', () => { + expect(callTargets('caller/caller.go', 'params')).toEqual(['model/item.go::Item::Validate']); + }); + + it('an array local', () => { + expect(callTargets('caller/caller.go', 'array')).toEqual(['model/item.go::Item::Key']); + }); + + it('a map parameter', () => { + expect(callTargets('caller/caller.go', 'maps')).toEqual(['model/item.go::Item::Validate']); + }); + + it('a pointer to a slice, dereferenced', () => { + expect(callTargets('caller/caller.go', 'deref')).toEqual(['model/item.go::Item::Key']); + }); + + it('a named slice type of another package', () => { + expect(callTargets('caller/caller.go', 'named')).toEqual(['model/item.go::Item::Validate']); + }); + + it('a slice field and a map field of a parameter', () => { + expect(callTargets('caller/caller.go', 'fields')).toEqual([ + 'model/item.go::Item::Key', + 'model/item.go::Item::Validate', + ]); + }); + + it('a slice of a named slice type keeps that type', () => { + expect(callTargets('caller/caller.go', 'sliced')).toEqual(['model/item.go::Item::Validate']); + }); + + it('locals bound to a composite literal, each its own of several, and to make', () => { + expect(callTargets('caller/caller.go', 'literals')).toEqual([ + 'model/item.go::Item::Key', + 'model/item.go::Item::Validate', + ]); + }); + + it('a package-level variable', () => { + expect(callTargets('caller/caller.go', 'packageVar')).toEqual(['model/item.go::Item::Key']); + }); + + it('a declaration in a block that has closed does not type the receiver', () => { + expect(callTargets('caller/caller.go', 'shadowed')).toEqual(['model/item.go::Item::Validate']); + }); + + it('a conversion to a slice of an interface, and a receiver of a named slice type', () => { + expect(callTargets('validation/schema.go', 'ValidateBytes', 'ConjunctiveSchema')).toEqual([ + 'validation/schema.go::Schema::ValidateBytes', + ]); + expect(callTargets('validation/schema.go', 'First', 'ConjunctiveSchema')).toEqual([ + 'validation/schema.go::Schema::ValidateBytes', + ]); + }); + + it("deepcopy's `(*in)[i]` after `in, out := &in.Items, &out.Items`", () => { + expect(callTargets('api/types.go', 'DeepCopyInto', 'ItemList')).toEqual(['api/types.go::Item::DeepCopyInto']); + }); + + it("protobuf's `m.Items[len(m.Items)-1].Unmarshal(…)` on the receiver's slice field", () => { + expect(callTargets('api/types.go', 'Unmarshal', 'ItemList')).toEqual(['api/types.go::Item::Unmarshal']); + }); + + it("a namesake method beside the call is not the element type's", () => { + expect(callTargets('caller/caller.go', 'retarget')).toEqual(['model/item.go::Item::Rename']); + }); + + // A control: String and Error need a typed receiver to be linked at all, + // so these had no edge before either. Neither is `IPNet` the project's + // namesake of the outside type. + it('an element type from outside the project, or a predeclared one, links nothing', () => { + expect(callTargets('caller/caller.go', 'outside')).toEqual([]); + }); + + it("a method an embedded type has is not taken for one the index couldn't read", () => { + expect(callTargets('caller/caller.go', 'unread')).toEqual([]); + }); + + it("a method an embedded type has is the element type's when no file of its package declares it", () => { + expect(callTargets('caller/caller.go', 'promoted')).toEqual(['model/meta.go::Meta::Describe']); + }); + + it("an outside element type's method is not guessed by its name", () => { + expect(callTargets('caller/caller.go', 'outsideGuessed')).toEqual([]); + }); + + it('an alias of an outside type is that type, not a namesake of the alias', () => { + expect(callTargets('caller/caller.go', 'aliasOutside')).toEqual([]); + }); + + // A control: what a call returns isn't read here, so the call goes on to + // name matching, which finds the one method of its name. + it("an element of a call's result is left to the other strategies", () => { + expect(callTargets('caller/caller.go', 'callResult')).toEqual([ + 'caller/caller.go::Holder::All', + 'model/item.go::Item::Archive', + ]); + }); + + it("a type parameter's element has its constraint's methods, not those of the type its package names alike", () => { + expect(callTargets('caller/caller.go', 'First')).toEqual([]); + expect(callTargets('caller/caller.go', 'FirstNamed')).toEqual(['model/item.go::Validator::Validate']); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 6dfccfb190..24872d9e70 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, GO_TYPE_KINDS, matchGoAssertedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -1418,6 +1418,10 @@ export class ReferenceResolver { // to nothing the project declares. const asserted = matchGoAssertedCall(ref, this.context); if (asserted !== undefined) return asserted; + // So is one on an indexed element (`schemas[ix].ValidateBytes(data)`, + // deepcopy's `(*in)[i].DeepCopyInto(…)`): to a method of the element type. + const indexed = matchGoIndexedCall(ref, this.context); + if (indexed !== undefined) return indexed; // A section or snippet a Shopify theme's file names is that theme's own, // or nothing: Shopify never looks in another theme (see ./shopify-themes). diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e76a5b6cca..1dac5a1de5 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -2667,6 +2667,8 @@ interface GoScopeIndex { decls: Map; /** Where each declared name is bound, read on demand: [start, end, declaration]. */ scopes: Map>; + /** Whether the file declares a generic function or method, read on demand. */ + generic?: boolean; } interface GoDecl { @@ -2684,6 +2686,12 @@ interface GoDecl { from?: string; /** The function of the file's own package a `:=` takes its value from: `newStore` in `store := newStore(…)`. */ callee?: string; + /** + * Where the values a `:=`, or a `var` that writes no type, gives its names + * start (past the `:=` or `=`), and this name's place `k` among the `n` it + * declares: `out` is 1 of 2 in `in, out := &in.Items, &out.Items`. + */ + value?: { at: number; k: number; n: number }; } const GO_SCOPE_INDEXES = new WeakMap>(); @@ -2706,7 +2714,11 @@ const GO_PREDECLARED_CALLS: ReadonlySet = new Set([ function goLocalDecl(name: string, ref: Pick, context: ResolutionContext): GoDecl | undefined { const index = goScopeIndex(ref.filePath, context); if (!index?.decls.has(name)) return undefined; - const at = (index.lineStarts[ref.line - 1] ?? index.code.length) + Math.max(0, ref.column); + return goLocalDeclAt(index, name, (index.lineStarts[ref.line - 1] ?? index.code.length) + Math.max(0, ref.column)); +} + +/** {@link goLocalDecl}, at an offset of the file `index` reads. */ +function goLocalDeclAt(index: GoScopeIndex, name: string, at: number): GoDecl | undefined { let found: [number, number, GoDecl] | undefined; for (const scope of goScopesOf(index, name)) { if (at > scope[0] && at < scope[1] && (!found || scope[0] > found[0])) found = scope; @@ -2819,7 +2831,8 @@ function readGoScopes(code: string): GoScopeIndex { const from = /^[ \t]*&?[ \t]*([A-Za-z_]\w*)\.[A-Za-z_]\w*[ \t]*[({]/.exec(rhs)?.[1]; const call = from ? undefined : /^[ \t]*([A-Za-z_]\w*)[ \t]*(?:\[(?:[^[\]\n]|\[[^[\]\n]*\])*\][ \t]*)?\(/.exec(rhs)?.[1]; const callee = call && !GO_PREDECLARED_CALLS.has(call) ? call : undefined; - for (const [name, at] of names) add(name, { at, kind, type, from, callee }); + // The names were read backwards from the `:=`. + names.forEach(([name, at], r) => add(name, { at, kind, type, from, callee, value: { at: i + 2, k: names.length - 1 - r, n: names.length } })); } // `var x T`, `var a, b = …`, `const (…)`: names before a type, an `=` or // the line's end — in a group, on lines outside the brackets of a value @@ -2830,12 +2843,16 @@ function readGoScopes(code: string): GoScopeIndex { const s = spec.exec(code); if (!s) return; const type = /^[ \t]+([^\s=;][^=;\n]*?)[ \t]*(?:[=;\n]|$)/.exec(code.slice(at + s[0].length, at + s[0].length + 160))?.[1]; + // `var a, b = x, y`: the values that type them. + const eq = type === undefined ? /^[ \t]*=(?!=)/.exec(code.slice(at + s[0].length, at + s[0].length + 40)) : null; + const parts = s[1]!.split(','); let p = at + s[0].length - s[1]!.length; - for (const part of s[1]!.split(',')) { + parts.forEach((part, k) => { const name = part.trim(); - add(name, { at: p + part.indexOf(name), kind: 'local', type }); + const value = eq ? { at: at + s[0].length + eq[0].length, k, n: parts.length } : undefined; + add(name, { at: p + part.indexOf(name), kind: 'local', type, value }); p += part.length + 1; - } + }); }; for (const m of code.matchAll(/(?; +} + +/** + * A type, or null for a value of a type from outside the project or a + * predeclared one (no field or method the project declares), or undefined + * when that can't be read. + */ +type GoTyped = GoTypeText | null | undefined; + +/** A Go type declaration as written: `type A = B`, its type parameters, and the type after its name. */ +interface GoTypeSpec { + alias: boolean; + params: Map; + /** `[]Schema`, `struct { … }`, `model.Item`. */ + underlying: string; +} + +/** A struct's fields, each name's type as written, and the types it embeds. */ +interface GoStructMembers { + fields: Map; + embedded: string[]; + params: Map; +} + +const GO_TYPE_SPECS = new WeakMap>(); +const GO_STRUCT_MEMBERS = new WeakMap>(); +const GO_FUNC_TYPE_PARAMS = new WeakMap>>(); + +/** The keywords a type literal starts with: it declares no method of its own. */ +const GO_TYPE_LITERAL = /^(?:\[|map\b|chan\b|<-|func\b|struct\b|interface\b)/; + +/** How many values, fields and types one receiver is followed through. */ +const GO_VALUE_DEPTH = 12; + +/** + * A Go method call on an indexed element — kubectl's `schemas[ix].ValidateBytes(data)`, + * go-to-protobuf's `m.Items[len(m.Items)-1].Unmarshal(dAtA[iNdEx:postIndex])`, + * deepcopy-gen's `(*in)[i].DeepCopyInto(&(*out)[i])` — calls a method of the + * element type of what is indexed. It reaches the resolver by its bare name, + * at the column where its receiver expression starts, like a call through a + * type assertion, and nothing typed the receiver: a method name one type + * declares went to it, another a namesake in the package that sorts first, + * and a common one (every API type's `DeepCopyInto`) to nothing. + * + * The receiver is typed as Go types it: + * - its head is the parameter, receiver, named result or local the code + * around the call declares there (goLocalDecl), else a package-level + * variable of the call's package or, through an import, of another project + * package; + * - a declaration gives the type it writes (`xs []*T`, `var ring [4]T`) or + * the type of its value: a composite literal, `make`, `new`, `append`, a + * conversion (`[]Schema(c)`), another variable, a field, an element, an + * address (`in, out := &in.Items, &out.Items`), a `range` clause's element; + * - a field's type is read from its struct, a promoted one's from the struct + * embedded, the shallowest as Go's selector picks it; + * - an index takes the element type of a slice, an array, a pointer to an + * array or a map, through a named type's underlying type + * (`type ConjunctiveSchema []Schema`), and `(*p)` dereferences. + * The method is then the element type's own, its interface's, or one + * embedding promotes into it; a pointer element (`[]*T`) reaches T's + * methods. An element type from outside the project's modules + * (`[]*net.IPNet`), a predeclared one (`[]error`), a type literal or an alias + * of one of those declares no project method, so the call links nothing. + * Undefined when the call is not on an indexed element, or what is indexed + * can't be typed — a call's result, a type parameter — which leaves the call + * to name matching as before. + */ +export function matchGoIndexedCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + if (ref.language !== 'go' || ref.referenceKind !== 'calls' || !/^[A-Za-z_]\w*$/.test(ref.referenceName)) return undefined; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); + const line = lines?.[ref.line - 1]; + // The receiver is indexed on the call's own line. + if (!lines || line === undefined || ref.column < 0 || line.indexOf('].', ref.column) < 0) return undefined; + // A column inside a name is no expression's start, and a call written at + // its own name calls a function. + if (ref.column > 0 && /\w/.test(line[ref.column - 1]!)) return undefined; + if (line.startsWith(ref.referenceName, ref.column) && /^\s*\(/.test(line.slice(ref.column + ref.referenceName.length))) return undefined; + // The call is the chain's first `.name(…)` link, and ours when an index + // comes right before it. Two calls of one name in a chain + // (`a[0].Get().b[1].Get()`) share the column; the first is there either way. + const callAfterIndex = (links: GoChainLink[]): number => { + const k = links.findIndex((l, i) => i > 0 && l.kind === 'select' && l.text === ref.referenceName && links[i + 1]?.kind === 'call'); + return k > 0 && links[k - 1]!.kind === 'index' ? k : -1; + }; + if (callAfterIndex(goChainLinks(lines.slice(ref.line - 1, ref.line + GO_CHAIN_LINES).join('\n'), ref.column)) < 0) return undefined; + // Read again from the file with its comments and strings blanked, where + // the receiver's names are scoped. + const index = goScopeIndex(ref.filePath, context); + if (!index) return undefined; + const at = (index.lineStarts[ref.line - 1] ?? index.code.length) + ref.column; + const links = goChainLinks(index.code, at); + const k = callAfterIndex(links); + if (k < 0) return undefined; + const receiver = goReceiverChainType(links.slice(0, k), index.code[at] === '(', at, ref.filePath, index, context, 0); + if (receiver == null) return receiver; + return goMethodOnElement(receiver, ref, context); +} + +/** + * The method `ref` names on the type of its receiver. A pointer's is the + * type pointed to's. + */ +function goMethodOnElement(t: GoTypeText, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + let text = t.text.replace(/^\*/, ''); + // A pointer to a pointer, or a type literal, has no method of its own. + if (text.startsWith('*') || GO_TYPE_LITERAL.test(text)) return null; + // A type parameter has the methods its constraint declares: a named + // interface's (`[T Validator]`), or none of the project's (`[T any]`, + // `[T interface{ Validate() error }]`, a union). + const constraint = t.params?.get(text); + if (constraint !== undefined) { + if (!constraint) return undefined; + text = goNormalizeType(constraint); + if (!/^[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)?(?:\[[\s\S]*\])?$/.test(text) || text === 'comparable') return null; + } + const named = goNamedTypeDecls({ ...t, text }, context); + if (named == null) return named; + // An alias of a type from outside the project (`type Ctx = context.Context`) + // has that type's methods, none the project declares. + if (named.nodes.every((n) => goAliasTarget(n, context) === null)) return null; + const result = resolveMethodOnType(named.name, ref.referenceName, ref, context, 0.9, 'instance-method', named.dir); + // A type's own method may sit in a file of its package the index couldn't + // read: kubernetes' core/v1 generated.pb.go is too large to index, so + // `PersistentVolumeClaim`'s own `Unmarshal` is missing, and the one its + // embedded `TypeMeta` has would take the call. What such a file declares + // is read from its text; the call then links nothing. + const target = result ? context.getNodeById?.(result.targetNodeId) : null; + if (!target || goPackageDir(target.filePath) !== named.dir || target.qualifiedName !== `${named.name}::${target.name}`) { + for (const f of goUnreadPackageFiles(named.dir, context)) { + const methods = goUnreadFileMethods(f, context); + if (methods === null || methods.has(`${named.name}::${ref.referenceName}`)) return null; + } + } + return result; +} + +const GO_PACKAGE_FILES = new WeakMap>(); +const GO_UNREAD_FILES = new WeakMap>(); +const GO_UNREAD_METHODS = new WeakMap | null>>(); + +/** + * The files of a Go package the index read nothing from — one over the size + * limit, like kubernetes' 1.7 MB core/v1 generated.pb.go, or one that would + * not parse — whose types' methods are missing from the graph. A test file's + * methods are no other code's to call. + */ +function goUnreadPackageFiles(dir: string, context: ResolutionContext): string[] { + let memo = GO_UNREAD_FILES.get(context); + if (!memo) GO_UNREAD_FILES.set(context, (memo = new Map())); + const hit = memo.get(dir); + if (hit) return hit; + let files = GO_PACKAGE_FILES.get(context); + if (!files) { + files = new Map(); + for (const f of context.getAllFiles()) { + if (!f.endsWith('.go') || f.endsWith('_test.go')) continue; + const d = goPackageDir(f); + const list = files.get(d); + if (list) list.push(f); + else files.set(d, [f]); + } + GO_PACKAGE_FILES.set(context, files); + } + // A file the index read has a node of its own at least. + const unread = (files.get(dir) ?? []).filter((f) => context.getNodesInFile(f).length === 0); + memo.set(dir, unread); + return unread; +} + +/** How large an unread Go file may be for its method declarations to be read from its text. */ +const GO_UNREAD_SCAN_BYTES = 32 * 1024 * 1024; + +/** + * Each `Type::Method` a Go file the index didn't read declares, from its + * text: `func (m *PersistentVolumeClaim) Unmarshal(`. Null when the file is + * too large to read, or can't be. + */ +function goUnreadFileMethods(filePath: string, context: ResolutionContext): Set | null { + let memo = GO_UNREAD_METHODS.get(context); + if (!memo) GO_UNREAD_METHODS.set(context, (memo = new Map())); + const hit = memo.get(filePath); + if (hit !== undefined) return hit; + let methods: Set | null = null; + try { + const abs = path.join(context.getProjectRoot(), filePath); + if (fs.statSync(abs).size <= GO_UNREAD_SCAN_BYTES) { + methods = new Set(); + const decl = /^func\s*\(\s*(?:[A-Za-z_]\w*\s+)?\*?\s*([A-Za-z_]\w*)\s*(?:\[[^\]\n]*\])?\s*\)\s*([A-Za-z_]\w*)\s*[[(]/gm; + for (const m of fs.readFileSync(abs, 'utf8').matchAll(decl)) methods.add(`${m[1]}::${m[2]}`); + } + } catch { + methods = null; + } + memo.set(filePath, methods); + return methods; +} + +/** + * The project declarations a named Go type means — `Item` in the package of + * the file it is written in or one it dot-imports, `model.Item` in the + * package the import names — with its name and package directory. Null for a + * predeclared type or one of a package outside the project's modules; + * undefined when that can't be told: a type parameter, a name no package + * declares (a type declared in a function, a file the index lacks), an + * import the index doesn't know. + */ +function goNamedTypeDecls(t: GoTypeText, context: ResolutionContext): { name: string; dir: string; nodes: Node[] } | null | undefined { + const m = /^([A-Za-z_]\w*)(?:\.([A-Za-z_]\w*))?(?:\[[\s\S]*\])?$/.exec(t.text); + if (!m) return undefined; + const [, first, second] = m; + let name = first!; + let dir: string | null | undefined; + if (second === undefined) { + if (t.params?.has(name)) return undefined; + if (GO_BUILTIN_FIELD_TYPES.has(name)) return null; + dir = [goPackageDir(t.file), ...goDotImportDirs(t.file, context)].find((d) => goPackageTypes(name, d, context).length > 0); + if (dir === undefined) return undefined; + } else { + dir = goImportPackageDir(first!, t.file, context); + if (dir === undefined) return undefined; + if (dir === null) return isGoOutsidePackage(first!, t.file, context) ? null : undefined; + name = second; + } + const nodes = goPackageTypes(name, dir, context); + return nodes.length > 0 ? { name, dir, nodes } : undefined; +} + +/** + * The type of the Go expression a receiver chain's links spell — `in`, + * `(*in)[i]`, `m.Items[k]`, `v.(T)[0]` — at offset `at` of the file `index` + * reads. `paren` says the first link is a parenthesized expression. + */ +function goReceiverChainType( + links: GoChainLink[], paren: boolean, at: number, filePath: string, index: GoScopeIndex, + context: ResolutionContext, depth: number, +): GoTyped { + const head = links[0]; + if (!head || head.kind !== 'operand' || depth > GO_VALUE_DEPTH) return undefined; + // A call's result (`x.Items()[0]`) isn't typed here. + if (links.some((l) => l.kind === 'call')) return undefined; + let t: GoTyped; + let i = 1; + if (paren) { + t = goExprType(head.text, at, filePath, index, context, depth + 1); + } else { + const decl = goLocalDeclAt(index, head.text, at); + if (decl) { + t = goDeclaredValueType(decl, filePath, index, context, depth + 1); + } else if (context.getImportMappings(filePath, 'go').some((m) => m.localName === head.text)) { + // `pkg.Var`: a package-level variable of the package imported. + if (links[1]?.kind !== 'select') return undefined; + const dir = goImportPackageDir(head.text, filePath, context); + if (dir == null) return dir === null && isGoOutsidePackage(head.text, filePath, context) ? null : undefined; + t = goPackageVarTypeText(links[1].text, dir, filePath, true, context, depth + 1); + i = 2; + } else { + t = goPackageVarTypeText(head.text, goPackageDir(filePath), filePath, false, context, depth + 1); + } + } + for (; i < links.length && t; i++) { + const link = links[i]!; + if (link.kind === 'select') t = goFieldTypeOf(t, link.text, context, depth + 1); + else if (link.kind === 'index') t = goIsSliceIndex(link.text) ? goSlicedType(t, context, depth + 1) : goElementOf(t, context, depth + 1); + // `.(T)` is a T, written where the expression is. + else if (link.kind === 'assert' && link.text !== 'type') t = goTypeTextAt(link.text, filePath, at, index, context); + else return undefined; + } + return t; +} + +/** A type written at offset `at` of a file, or at package level for no `index`. */ +function goTypeTextAt(text: string, filePath: string, at: number, index: GoScopeIndex | null, context: ResolutionContext): GoTypeText { + return { text: goNormalizeType(text), file: filePath, params: index ? goTypeParamsAt(index, filePath, at, context) : undefined }; +} + +/** + * The type of a Go expression written in `filePath`: at offset `at` of the + * file `index` reads, or at package level for no `index`, where no local is + * seen. + */ +function goExprType( + expr: string, at: number, filePath: string, index: GoScopeIndex | null, context: ResolutionContext, depth: number, +): GoTyped { + const s = expr.trim(); + if (!s || depth > GO_VALUE_DEPTH) return undefined; + if (s.startsWith('&')) { + const t = goExprType(s.slice(1), at, filePath, index, context, depth + 1); + return t ? { ...t, text: `*${t.text}` } : t; + } + if (s.startsWith('*')) { + const t = goExprType(s.slice(1), at, filePath, index, context, depth + 1); + if (!t) return t; + return t.text.startsWith('*') ? { ...t, text: t.text.slice(1) } : undefined; + } + // `make(T, …)`, `new(T)`, `append(xs, …)`. + const builtin = /^(make|new|append)\s*\(/.exec(s); + if (builtin) { + const open = builtin[0].length - 1; + if (goClosingBracket(s, open) !== s.length - 1) return undefined; + const first = goSplitTopLevel(s.slice(open + 1, -1), ',')[0]?.trim(); + if (!first) return undefined; + if (builtin[1] === 'make') return goTypeTextAt(first, filePath, at, index, context); + if (builtin[1] === 'new') return goTypeTextAt(`*${first}`, filePath, at, index, context); + return goExprType(first, at, filePath, index, context, depth + 1); + } + // A composite literal `T{…}`, or a conversion `T(x)`. + const typeEnd = goTypeTextEnd(s, 0); + if (typeEnd > 0) { + let j = typeEnd; + while (s[j] === ' ' || s[j] === '\t') j++; + if ((s[j] === '{' || s[j] === '(') && goClosingBracket(s, j) === s.length - 1) { + const type = goTypeTextAt(s.slice(0, typeEnd), filePath, at, index, context); + // `T(x)` is a conversion only when T is a type, not a function. + return s[j] === '{' || GO_TYPE_LITERAL.test(type.text) || goNamedTypeDecls(type, context) ? type : undefined; + } + } + if (!index) return undefined; + // A name, a field, an element, an assertion. + const end = { at: 0 }; + const links = goChainLinks(s, 0, end); + if (end.at !== s.length) return undefined; + return goReceiverChainType(links, s.startsWith('('), at, filePath, index, context, depth + 1); +} + +/** + * The type a declaration gives the name it declares: the type of the value + * it is given, or else the type it writes, read where the declaration is. + */ +function goDeclaredValueType(decl: GoDecl, filePath: string, index: GoScopeIndex, context: ResolutionContext, depth: number): GoTyped { + if (decl.value) { + const { at, k, n } = decl.value; + const code = index.code; + const rhs = code.slice(at, decl.kind === 'header' ? goHeaderValueEnd(code, at) : goStatementEnd(code, at)); + // `for k, v := range xs`: v is an element of xs. + const ranged = /^\s*range\b/.exec(rhs); + if (ranged) { + if (k !== 1 || n !== 2) return undefined; + return goElementOf(goExprType(rhs.slice(ranged[0].length), decl.at, filePath, index, context, depth + 1), context, depth + 1); + } + const values = goSplitTopLevel(rhs, ','); + // `v, ok := m[key]`, `v, ok := x.(T)`. + const value = values.length === n ? values[k] : values.length === 1 && n === 2 && k === 0 && /[\])]\s*$/.test(rhs) ? rhs : undefined; + return value === undefined ? undefined : goExprType(value, decl.at, filePath, index, context, depth + 1); + } + if (decl.type === undefined) return undefined; + // A variadic `xs ...T` is a `[]T`. + return goTypeTextAt(decl.type.replace(/^\.\.\./, '[]'), filePath, decl.at, index, context); +} + +/** + * The type of the package-level variable or constant `name` of the package + * in directory `dir`, as `fromFile` sees it — its own package's, where a + * `foo_test` package shares the directory and only a test file sees a test + * file's, or an imported package's, whose test files no importer sees — read + * from its declaration: the type it writes, or the type of its value. + */ +function goPackageVarTypeText( + name: string, dir: string, fromFile: string, imported: boolean, context: ResolutionContext, depth: number, +): GoTyped { + if (depth > GO_VALUE_DEPTH) return undefined; + const clause = imported ? undefined : goPackageClause(fromFile, context); + const testFile = (f: string): boolean => /_test\.go$/.test(f); + const vars = context.getNodesByName(name).filter((n) => + n.language === 'go' && (n.kind === 'variable' || n.kind === 'constant') && goPackageDir(n.filePath) === dir && + (imported ? !testFile(n.filePath) : goPackageClause(n.filePath, context) === clause && (testFile(fromFile) || !testFile(n.filePath)))); + let found: GoTyped; + for (const v of vars) { + const t = goPackageVarSpecType(v, name, context, depth + 1); + // Files for other build targets may declare it differently. + if (t === undefined || (found !== undefined && (found?.text ?? null) !== (t?.text ?? null))) return undefined; + found = t; + } + return found; +} + +/** The type of one package-level declaration of `name`: the type its spec writes, or its value's. */ +function goPackageVarSpecType(v: Node, name: string, context: ResolutionContext, depth: number): GoTyped { + const declared = goPackageValueType(v, context); + if (declared === null) return null; + if (declared) return goTypeTextAt(declared.written, declared.file, -1, null, context); + const lines = context.getFileLines?.(v.filePath) ?? context.readFile(v.filePath)?.split(/\r?\n/); + if (!lines) return undefined; + // `var name = value`, `name = value` in a group, `var a, name = x, y`: the + // spec from its line's start, where its first name is, to its end. + const code = blankGoCode(lines.slice(v.startLine - 1, Math.max(v.startLine, v.endLine)).join('\n')); + const spec = /^\s*(?:(?:var|const)\s+)?([A-Za-z_]\w*(?:\s*,\s*[A-Za-z_]\w*)*)\s*=(?!=)\s*([\s\S]*)$/ + .exec(code.slice(0, goStatementEnd(code, v.startColumn))); + if (!spec) return undefined; + const names = spec[1]!.split(',').map((x) => x.trim()); + const values = goSplitTopLevel(spec[2]!, ','); + const k = names.indexOf(name); + return k >= 0 && values.length === names.length ? goExprType(values[k]!, -1, v.filePath, null, context, depth + 1) : undefined; +} + +/** + * The type of field `name` of a value of type t: one its struct declares, or + * one a struct it embeds promotes, the shallowest as Go's selector picks it. + * A pointer is dereferenced first. + */ +function goFieldTypeOf(t: GoTypeText, name: string, context: ResolutionContext, depth: number): GoTyped { + const text = t.text.replace(/^\*/, ''); + if (text.startsWith('*') || GO_TYPE_LITERAL.test(text) || depth > GO_VALUE_DEPTH) return undefined; + let level: GoTypeText[] = [{ ...t, text }]; + const seen = new Set(); + for (let d = 0; d < 4 && level.length > 0; d++) { + const found: GoTypeText[] = []; + const next: GoTypeText[] = []; + let outside = false; + for (const type of level) { + const struct = goStructOf(type, context, depth + 1); + if (struct === null) outside = true; + if (!struct) continue; + const own = struct.members.fields.get(name); + if (own !== undefined) found.push({ text: goNormalizeType(own), file: struct.file, params: struct.members.params }); + for (const e of struct.members.embedded) { + if (seen.has(`${struct.file}\0${e}`)) continue; + seen.add(`${struct.file}\0${e}`); + next.push({ text: goNormalizeType(e), file: struct.file, params: struct.members.params }); + } + } + if (found.length === 1) return found[0]; + if (found.length > 1) return undefined; + // A field no project struct at this depth declares, beside an embedded + // type from outside the project, is that type's, of an outside type too. + if (outside && next.length === 0) return null; + level = next; + } + return undefined; +} + +/** + * The struct a named type is, with the file that declares it: its own + * declaration, or the struct an alias or a defined type (`type B A`) names. + * Null for a type from outside the project. + */ +function goStructOf(t: GoTypeText, context: ResolutionContext, depth: number): { members: GoStructMembers; file: string } | null | undefined { + if (depth > GO_VALUE_DEPTH) return undefined; + const named = goNamedTypeDecls({ ...t, text: t.text.replace(/^\*/, '') }, context); + if (named == null) return named; + for (const n of named.nodes) { + if (n.kind === 'struct') { + const members = goStructMembers(n, context); + if (members) return { members, file: n.filePath }; + } else if (n.kind === 'type_alias') { + const spec = goTypeSpec(n, context); + if (!spec) continue; + const via = goStructOf({ text: goNormalizeType(spec.underlying), file: n.filePath, params: spec.params }, context, depth + 1); + if (via !== undefined) return via; + } + } + return undefined; +} + +/** + * What `x[i]` is for an x of type t: a slice's, an array's or an array + * pointer's element, a map's value, the same of a named type's underlying + * type. A string's bytes have no method. + */ +function goElementOf(t: GoTyped, context: ResolutionContext, depth: number): GoTyped { + if (!t || depth > GO_VALUE_DEPTH) return t; + let s = t.text; + // A pointer to an array is indexed as the array; any other pointer only + // once dereferenced, `(*p)[i]`. + if (s.startsWith('*')) { + s = s.slice(1); + if (!/^\[[^\]]/.test(s)) return undefined; + } + if (s.startsWith('[') || s.startsWith('map[')) { + const close = goClosingBracket(s, s.indexOf('[')); + const element = close < 0 ? '' : s.slice(close + 1).trim(); + return element ? { ...t, text: element } : undefined; + } + if (s === 'string') return null; + if (GO_TYPE_LITERAL.test(s)) return undefined; + const named = goNamedTypeDecls({ ...t, text: s }, context); + if (named == null) return named; + for (const n of named.nodes) { + const spec = n.kind === 'type_alias' ? goTypeSpec(n, context) : null; + if (!spec) continue; + const element = goElementOf({ text: goNormalizeType(spec.underlying), file: n.filePath, params: spec.params }, context, depth + 1); + if (element !== undefined) return element; + } + return undefined; +} + +/** + * What `x[lo:hi]` is for an x of type t: a slice of the element of an array + * or an array pointer, else (a slice, a named slice type, a string) t. + */ +function goSlicedType(t: GoTypeText, context: ResolutionContext, depth: number): GoTyped { + const s = t.text.replace(/^\*/, ''); + const array = /^\[[^\]]/.test(s) ? { ...t, text: s } : null; + if (!array && t.text.startsWith('*')) return undefined; + let underlying = array; + if (!underlying && s !== 'string' && !s.startsWith('[')) { + const named = goNamedTypeDecls(t, context); + if (named === undefined) return undefined; + for (const n of named?.nodes ?? []) { + const spec = n.kind === 'type_alias' ? goTypeSpec(n, context) : null; + if (spec && /^\[[^\]]/.test(spec.underlying)) underlying = { text: goNormalizeType(spec.underlying), file: n.filePath, params: spec.params }; + } + } + if (!underlying) return t; + const element = goElementOf(underlying, context, depth + 1); + return element ? { ...element, text: `[]${element.text}` } : element; +} + +/** Whether an index's brackets hold a slice expression's bounds, `[lo:hi]`. */ +function goIsSliceIndex(inner: string): boolean { + return goSplitTopLevel(inner, ':').length > 1; +} + +/** A type's text with its whitespace collapsed: `map[string]*model.Item`. */ +function goNormalizeType(text: string): string { + return text.trim().replace(/\s+/g, ' ').replace(/([*\]])\s+/g, '$1').replace(/\s*\.\s*/g, '.'); +} + +/** + * A Go type declaration read from its source, where its node starts (at the + * type's name): `type A = B`, its type parameters, and the type after them. + */ +function goTypeSpec(node: Node, context: ResolutionContext): GoTypeSpec | null { + let memo = GO_TYPE_SPECS.get(context); + if (!memo) GO_TYPE_SPECS.set(context, (memo = new Map())); + const hit = memo.get(node.id); + if (hit !== undefined) return hit; + const lines = context.getFileLines?.(node.filePath) ?? context.readFile(node.filePath)?.split(/\r?\n/) ?? []; + const code = blankGoCode(lines.slice(node.startLine - 1, Math.max(node.startLine, node.endLine)).join('\n')); + let spec: GoTypeSpec | null = null; + const name = /^[A-Za-z_]\w*/.exec(code.slice(node.startColumn)); + if (name) { + let i = node.startColumn + name[0].length; + const skip = (): void => { while (code[i] === ' ' || code[i] === '\t') i++; }; + skip(); + let params = new Map(); + // `Name[T any, U comparable]` declares type parameters; `Name []T` and + // `Name [4]T` are a slice and an array. + if (code[i] === '[') { + const close = goClosingBracket(code, i); + const inner = close > i ? code.slice(i + 1, close) : ''; + if (/^\s*[A-Za-z_]\w*(?:\s*,\s*[A-Za-z_]\w*)*\s+[^\s,\]]/.test(inner)) { + params = goTypeParamList(inner); + i = close + 1; + skip(); + } + } + const alias = code[i] === '='; + if (alias) i++; + const underlying = code.slice(i, goStatementEnd(code, i)).trim(); + if (underlying) spec = { alias, params, underlying }; + } + memo.set(node.id, spec); + return spec; +} + +/** + * The fields a Go struct declares, by name, with the type each is written + * with, and the types it embeds (`Base`, `pkg.Base`, `Base[T]`, a pointer's + * without its `*`). + */ +function goStructMembers(node: Node, context: ResolutionContext): GoStructMembers | null { + let memo = GO_STRUCT_MEMBERS.get(context); + if (!memo) GO_STRUCT_MEMBERS.set(context, (memo = new Map())); + const hit = memo.get(node.id); + if (hit !== undefined) return hit; + let members: GoStructMembers | null = null; + const spec = goTypeSpec(node, context); + const open = spec ? /^struct\s*\{/.exec(spec.underlying) : null; + const close = spec && open ? goClosingBracket(spec.underlying, open[0].length - 1) : -1; + if (spec && open && close > 0) { + members = { fields: new Map(), embedded: [], params: spec.params }; + for (const raw of goSplitTopLevel(spec.underlying.slice(open[0].length, close), '\n;')) { + // A tag is a string: its quotes stay, its contents are blank. + const member = raw.replace(/(?:`[^`]*`|"[^"]*")\s*$/, '').trim(); + if (!member) continue; + const embedded = /^\*?\s*([A-Za-z_]\w*(?:\s*\.\s*[A-Za-z_]\w*)?(?:\s*\[[\s\S]*\])?)$/.exec(member); + if (embedded) { + members.embedded.push(embedded[1]!); + continue; + } + const named = /^([A-Za-z_]\w*(?:\s*,\s*[A-Za-z_]\w*)*)\s+(\S[\s\S]*)$/.exec(member); + if (!named) continue; + for (const n of named[1]!.split(',')) members.fields.set(n.trim(), named[2]!); + } + } + memo.set(node.id, members); + return members; +} + +/** + * The type parameters in scope at offset `at` of a Go file: those its + * enclosing function declares, or a method's generic receiver + * (`func (s *Set[T]) Add(v T)`). + */ +function goTypeParamsAt(index: GoScopeIndex, filePath: string, at: number, context: ResolutionContext): ReadonlyMap | undefined { + // Most files declare no generic function: nothing to look up. + index.generic ??= /(?> 1; + if (index.lineStarts[mid]! <= at) lo = mid; else hi = mid - 1; + } + const line = lo + 1; + const fn = context.getNodesInFile(filePath).find((n) => + (n.kind === 'function' || n.kind === 'method') && n.startLine <= line && n.endLine >= line); + if (!fn) return undefined; + let memo = GO_FUNC_TYPE_PARAMS.get(context); + if (!memo) GO_FUNC_TYPE_PARAMS.set(context, (memo = new Map())); + let params = memo.get(fn.id); + if (!params) { + params = new Map(); + const lineStart = index.lineStarts[fn.startLine - 1] ?? 0; + const head = index.code.slice(lineStart, index.lineStarts[Math.min(fn.endLine, fn.startLine + 40) - 1] ?? index.code.length); + // `func Name[T any](`, or `func (s *Set[T, U]) Name(`, whose type + // arguments name its type's parameters, constrained where the type is. + const generic = /^\s*func\s*(?:\(\s*(?:[A-Za-z_]\w*\s+)?\*?\s*([A-Za-z_]\w*)\s*(\[)|[A-Za-z_]\w*\s*(\[))/.exec(head); + if (generic) { + const open = generic.index + generic[0].length - 1; + const close = goClosingBracket(head, open); + const inner = close > open ? head.slice(open + 1, close) : ''; + if (generic[1] === undefined) { + params = goTypeParamList(inner); + } else { + const declared = goPackageTypes(generic[1], goPackageDir(filePath), context) + .map((n) => goTypeSpec(n, context)?.params).find((p) => p !== undefined && p.size > 0); + const constraints = declared ? [...declared.values()] : []; + goSplitTopLevel(inner, ',').forEach((p, k) => { + const ident = /^\s*([A-Za-z_]\w*)\s*$/.exec(p); + if (ident) params!.set(ident[1]!, constraints[k] ?? ''); + }); + } + } + memo.set(fn.id, params); + } + return params.size > 0 ? params : undefined; +} + +/** + * The type parameters a `[…]` list declares, each with its constraint as + * written: `T any`, `K comparable, V Validator`, `A, B Number` (both are + * `Number`s). + */ +function goTypeParamList(inner: string): Map { + const params = new Map(); + let pending: string[] = []; + for (const part of goSplitTopLevel(inner, ',')) { + const m = /^\s*([A-Za-z_]\w*)\s*([\s\S]*?)\s*$/.exec(part); + if (!m) continue; + pending.push(m[1]!); + if (!m[2]) continue; + for (const p of pending) params.set(p, m[2]); + pending = []; + } + for (const p of pending) params.set(p, ''); + return params; +} + +/** + * The end of the Go type written at offset `i` of `s` — `[]*model.Item`, + * `map[string]T`, `pkg.T[int]`, `struct{…}` — or -1 when none starts there. + * A function or channel type isn't read. + */ +function goTypeTextEnd(s: string, i: number): number { + while (s[i] === ' ' || s[i] === '\t') i++; + if (s[i] === '*') return goTypeTextEnd(s, i + 1); + if (s[i] === '[') { + const close = goClosingBracket(s, i); + return close < 0 ? -1 : goTypeTextEnd(s, close + 1); + } + const map = /^map\s*\[/.exec(s.slice(i, i + 8)); + if (map) { + const close = goClosingBracket(s, i + map[0].length - 1); + return close < 0 ? -1 : goTypeTextEnd(s, close + 1); + } + const literal = /^(?:struct|interface)\s*\{/.exec(s.slice(i, i + 16)); + if (literal) { + const close = goClosingBracket(s, i + literal[0].length - 1); + return close < 0 ? -1 : close + 1; + } + const name = /^[A-Za-z_]\w*(?:\s*\.\s*[A-Za-z_]\w*)?/.exec(s.slice(i)); + if (!name || /^(?:func|chan|range|make|new|append)$/.test(name[0])) return -1; + let end = i + name[0].length; + // Type arguments: `List[T]`. + if (s[end] === '[') { + const close = goClosingBracket(s, end); + if (close > end && !goIsSliceIndex(s.slice(end + 1, close))) end = close + 1; + } + return end; +} + +/** `s` split at each of the characters `seps` that sits outside brackets. */ +function goSplitTopLevel(s: string, seps: string): string[] { + const parts: string[] = []; + let depth = 0; + let from = 0; + for (let i = 0; i < s.length; i++) { + const c = s[i]!; + if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') depth--; + else if (depth === 0 && seps.includes(c)) { + parts.push(s.slice(from, i)); + from = i + 1; + } + } + parts.push(s.slice(from)); + return parts; +} + +/** Where an `if` / `for` / `switch` header's simple statement ends: its `;`, or the `{` its block opens with. */ +function goHeaderValueEnd(code: string, at: number): number { + let depth = 0; + for (let i = at; i < code.length; i++) { + const c = code[i]; + if (c === '(' || c === '[') depth++; + else if (c === ')' || c === ']') { if (--depth < 0) return i; } + else if (depth === 0 && (c === ';' || c === '{' || c === '\n')) return i; + } + return code.length; +} + /** Python types whose methods are the runtime's, never a project symbol's. */ const PYTHON_RUNTIME_TYPES: ReadonlySet = new Set([ 'list', 'dict', 'set', 'frozenset', 'tuple', 'str', 'bytes', 'bytearray', From 1f3a274c53dc34e8c776f708bbd82bf771c657d8 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 21 Oct 2026 17:44:33 +0000 Subject: [PATCH 02/95] fix(go): a method a type declares in a file the index couldn't read is not its embedded type's (#2757) kubernetes' core/v1 generated.pb.go is 1.7 MB, over the index's 1 MB limit, so every API type's own String, Marshal, Unmarshal, Size and MarshalToSizedBuffer is missing from the graph. Resolving `pod.String()` on a `*v1.Pod`, or `m.Template.Size()` on a `v1.PodTemplateSpec`, the walk over embedded types found the method the embedded TypeMeta or ObjectMeta has and linked that, a method the call never runs. resolveGoMethodInPackage, which every typed Go receiver goes through (parameters, locals, fields, assertions, package-level values, indexed elements), now reads the method declarations of the files of the type's package that the index couldn't read, from their text, and links nothing when the type declares the method there. matchMethodCall's typed branch and matchGoPackageValueCall keep that null instead of falling through to name matching, which found the same promoted method. A method the type really gets from an embedded type, like GetName from ObjectMeta, still links. The check that #2755 added for indexed elements moves into the shared path. On kubernetes 4c05e667, 63 calls lose that link and nothing else changes; etcd, prometheus, express and flask are unchanged. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/go-unread-file-methods.test.ts | 150 +++++++++++++++++++++++ src/resolution/name-matcher.ts | 55 ++++++--- 3 files changed, 187 insertions(+), 19 deletions(-) create mode 100644 __tests__/go-unread-file-methods.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ddc05cd37..69ded831bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a call made through a type assertion, like `srv.(KVServer).Range(ctx, in)` in generated gRPC code, `c.Reader.(*pipe).Close()` or `v.(storage.Store).Fetch(key)`, now links to that method of the asserted type, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, it linked to whichever method shared the name, so every gRPC handler pointed at the `Unimplemented…Server` stub instead of the server interface. Re-index Go projects after upgrading. - In Go, a call through a variable that holds the result of a type assertion, like `st.Fetch(key)` after `st := v.(storage.Store)` or `rc.Read(p)` after `rc, ok := v.(ReadCloser)`, now links to that method of the asserted type, as a call made through the assertion itself does, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, such a call linked to whichever method shared the name or to nothing, and a variable named like a standard-library package, like grpc-go's `parser, ok := builder.(balancer.ConfigParser)`, was taken for that package. Re-index Go projects after upgrading. - In Go, a method called on an element of a slice, array or map, like kubectl's `schemas[ix].ValidateBytes(data)`, `pods[i].DeepCopy()`, `m.Items[len(m.Items)-1].Unmarshal(…)` in generated protobuf code or `(*in)[i].DeepCopyInto(&(*out)[i])` in every deepcopy file, now links to that method of the element's type. The type is read from what is indexed: a parameter, a variable, a struct field or a named slice type like `type ConjunctiveSchema []Schema`, and an element held by pointer, like `[]*Item`, reaches `Item`'s methods. Before, such a call linked to whichever method shared its name, often one in an unrelated package, or to nothing when many types declare it, as with Kubernetes' `DeepCopyInto`. An element whose type comes from outside your project, like `[]*net.IPNet`, or is a built-in one like `[]error`, now links to nothing rather than to a project method of that name. Re-index Go projects after upgrading. +- In Go, a method a type declares in a file too large to index, like the `String`, `Marshal` and `Unmarshal` every Kubernetes API type has in `core/v1/generated.pb.go`, is no longer replaced by the method of that name a type it embeds has. Before, `pod.String()` on a `*v1.Pod` linked to `TypeMeta`'s `String`, a method the call never runs; it now links to nothing, since the pod's own method isn't in the graph. A method the type really gets from a type it embeds, like `GetName` from `ObjectMeta`, still links. Re-index Go projects after upgrading. - In Go, calls into a module whose `go.mod` is not at the project root now resolve, whether it is a `server/` backend next to a `web/` frontend or one of several modules side by side as in etcd, so calls like `store.New()` and `s.db.CreateItem()` find their targets instead of being treated as calls into a third-party package. A name written through a package, like a `job.OPCommand` result type or a field of type `artifact.Manager`, now links to that package's symbol rather than a same-named one elsewhere, which also corrects links in projects with a single `go.mod`. Re-index Go projects after upgrading. Thanks @GoDiao for the report and @danusha2345 for the fix. (#2322) - In Go, an exported type written without a package, like `Node` in `func Walk(v Visitor, node Node)`, now links to the type of that name in its own package, whether it is a struct, an interface or another kind of type. Before, it could link to a struct of the same name in an unrelated package, so prometheus's PromQL parser functions pointed at the Kubernetes discovery `Node` struct instead of the parser's own `Node` interface, or, for a result type like `Appender` in `func (f *fanout) Appender(…) Appender`, to the method itself. A type from a package outside your project, like `apiv1.Node` or `http.Handler`, no longer links to a struct of the same name in the file that uses it; a variadic parameter like `...storage.Filter` now links to that package's type; a route handler written as a method value, like `h.Follow` in an Echo or Gin app, links to that method rather than a model struct named `Follow`; and a method called on another call's result or on an indexed value, like `err[i].Error()` or a chained `.String()`, is no longer recorded as creating a struct of the same name. Re-index Go projects after upgrading. - In Go, a function or method passed as a value, like `sync.Pool{New: wm.new}`, now shows the function that passes it among its callers and impact even when it is named `new`, `nil`, `None`, `self` or a handful of similar words, or when a line break or comment sits beside the dot. Before, such a value was skipped, so a method used only that way looked unused. Re-index Go projects after upgrading. diff --git a/__tests__/go-unread-file-methods.test.ts b/__tests__/go-unread-file-methods.test.ts new file mode 100644 index 0000000000..6c859b122d --- /dev/null +++ b/__tests__/go-unread-file-methods.test.ts @@ -0,0 +1,150 @@ +/** + * A Go method a type declares in a file the index couldn't read. + * + * kubernetes' core/v1 generated.pb.go is 1.7 MB, over the index's 1 MB + * limit, so `Pod`'s own `String`, `Marshal` and `Unmarshal` are not in the + * graph. Resolving `pod.String()` on a `*v1.Pod`, the walk over the types + * `Pod` embeds found the `String` its `TypeMeta` has and linked that, a + * method the call never runs. Every typed Go receiver took that walk: a + * parameter, a local, a field, an assertion, a package-level variable. + * + * The method declarations of such a file are read from its text: when the + * type declares the method there, the call links nothing. A method the type + * really gets from an embedded type, like `GetName` from `ObjectMeta`, still + * links. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-go-unread-methods-')); + const files: Record = { + 'go.mod': 'module example.com/app\n\ngo 1.22\n', + 'meta/meta.go': `package meta + +type TypeMeta struct { + Kind string +} + +func (m *TypeMeta) String() string { return m.Kind } + +func (m *TypeMeta) Unmarshal(dAtA []byte) error { return nil } + +type ObjectMeta struct { + Name string +} + +func (m *ObjectMeta) GetName() string { return m.Name } +`, + 'api/types.go': `package api + +import "example.com/app/meta" + +type Pod struct { + meta.TypeMeta + meta.ObjectMeta + Spec string +} +`, + // Over the index's 1 MB limit, so none of it is read. + 'api/generated.pb.go': `package api + +func (m *Pod) String() string { return "" } + +func (m *Pod) Unmarshal(dAtA []byte) error { return nil } +${'// generated padding, never read by the indexer\n'.repeat(24_000)}`, + 'caller/caller.go': `package caller + +import "example.com/app/api" + +func param(pod *api.Pod) string { + return pod.String() +} + +func local() error { + pod := &api.Pod{} + return pod.Unmarshal(nil) +} + +type holder struct { + pod *api.Pod +} + +func (h *holder) field() string { + return h.pod.String() +} + +func asserted(x any) string { + return x.(*api.Pod).String() +} + +var shared api.Pod + +func packageVar() error { + return shared.Unmarshal(nil) +} + +func promoted(pod *api.Pod) string { + return pod.GetName() +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `file::qualifiedName` of every symbol the function or method `name` in `file` calls. */ +function callTargets(file: string, name: string): string[] { + const fn = cg.getNodesInFile(file).find((n) => n.name === name); + expect(fn, `${name} in ${file}`).toBeDefined(); + return cg + .getOutgoingEdges(fn!.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.filePath.replace(/\\/g, '/')}::${n.qualifiedName}`) + .sort(); +} + +describe("Go methods declared in a file the index couldn't read", () => { + it('the file is indexed without a node', () => { + expect(cg.getNodesInFile('api/generated.pb.go')).toEqual([]); + }); + + it('a parameter of the type links nothing, not the method an embedded type has', () => { + expect(callTargets('caller/caller.go', 'param')).toEqual([]); + }); + + it('a local bound to a composite literal', () => { + expect(callTargets('caller/caller.go', 'local')).toEqual([]); + }); + + it('a field of the type', () => { + expect(callTargets('caller/caller.go', 'field')).toEqual([]); + }); + + it('a type assertion', () => { + expect(callTargets('caller/caller.go', 'asserted')).toEqual([]); + }); + + it('a package-level variable', () => { + expect(callTargets('caller/caller.go', 'packageVar')).toEqual([]); + }); + + it('a method an embedded type gives the type still links', () => { + expect(callTargets('caller/caller.go', 'promoted')).toEqual(['meta/meta.go::ObjectMeta::GetName']); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 1dac5a1de5..cb992fb721 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -19275,6 +19275,10 @@ export function matchMethodCall( // defined over an interface (`type Codec Serializer`) has methods the // index doesn't give it. if (goStd && goDeclared !== undefined) return null; + // Nor one the type declares in a file the index couldn't read: in + // kubernetes, `pod.Unmarshal(…)` on a `*v1.Pod` is the pod's own, in + // core/v1's generated.pb.go, never the `TypeMeta` one a name finds. + if (ref.language === 'go' && importedFqn !== undefined && goUnreadFileDeclares(inferredType, methodName!, importedFqn, context)) return null; // The receiver's declared type is one the project doesn't declare — // `List list`, `String s`, `val sb = StringBuilder()` — so the // method is that outside type's. gson's `list.add(…)` went to a project @@ -19931,8 +19935,10 @@ export const GO_TYPE_KINDS: ReadonlySet = new Set(['struct', 'interface' * method the type doesn't declare is promoted from a type it embeds, which * its declaration names (`BaseAPI`, `*cached.BaseManager`, `ReadView`); the * name-based supertype walk would mix in every same-named type's embeddings. - * Undefined when the package declares no such type (a dot import, a file - * that isn't indexed): the caller then resolves by name as before. + * One the type declares in a file of its package the index couldn't read is + * missing from the graph, and nothing stands for it. Undefined when the + * package declares no such type (a dot import, a file that isn't indexed): + * the caller then resolves by name as before. */ function resolveGoMethodInPackage( typeName: string, @@ -19952,6 +19958,10 @@ function resolveGoMethodInPackage( } const types = goPackageTypes(typeName, pkgDir, context); if (types.length === 0) return undefined; + // kubernetes' core/v1 generated.pb.go is too large to index, so `Pod`'s + // own `String` is missing, and `pod.String()` went to the one its embedded + // `TypeMeta` has. + if (goUnreadFileDeclares(typeName, methodName, pkgDir, context)) return null; if (depth >= 4) return null; let unplaced = 0; for (const t of types) { @@ -20328,8 +20338,10 @@ function matchGoPackageValueCall(receiver: string, method: string, ref: Unresolv // A project type whose methods aren't found is left to the strategies // below: kubernetes' `var UnstructuredJSONScheme runtime.Codec = // unstructuredJSONScheme{}` is a `Codec`, defined over an interface whose - // methods its node doesn't carry. - return resolveMethodOnType(name, method, ref, context, 0.9, 'instance-method', pkgDir) ?? undefined; + // methods its node doesn't carry. Not when the type declares the method in + // a file the index couldn't read: that one is the type's own. + return resolveMethodOnType(name, method, ref, context, 0.9, 'instance-method', pkgDir) ?? + (goUnreadFileDeclares(name, method, pkgDir, context) ? null : undefined); } /** A Go package-level value's type as its declaration writes it, with the declaring file; null for one with no project method. */ @@ -20697,20 +20709,24 @@ function goMethodOnElement(t: GoTypeText, ref: UnresolvedRef, context: Resolutio // An alias of a type from outside the project (`type Ctx = context.Context`) // has that type's methods, none the project declares. if (named.nodes.every((n) => goAliasTarget(n, context) === null)) return null; - const result = resolveMethodOnType(named.name, ref.referenceName, ref, context, 0.9, 'instance-method', named.dir); - // A type's own method may sit in a file of its package the index couldn't - // read: kubernetes' core/v1 generated.pb.go is too large to index, so - // `PersistentVolumeClaim`'s own `Unmarshal` is missing, and the one its - // embedded `TypeMeta` has would take the call. What such a file declares - // is read from its text; the call then links nothing. - const target = result ? context.getNodeById?.(result.targetNodeId) : null; - if (!target || goPackageDir(target.filePath) !== named.dir || target.qualifiedName !== `${named.name}::${target.name}`) { - for (const f of goUnreadPackageFiles(named.dir, context)) { - const methods = goUnreadFileMethods(f, context); - if (methods === null || methods.has(`${named.name}::${ref.referenceName}`)) return null; - } + return resolveMethodOnType(named.name, ref.referenceName, ref, context, 0.9, 'instance-method', named.dir); +} + +/** + * Whether a file of the Go package in `dir` that the index couldn't read + * declares `typeName`'s method `methodName` (or can't be read either): the + * method is then missing from the graph, and the one an embedded type has + * is not it. kubernetes' 1.7 MB core/v1 generated.pb.go holds `Pod`'s and + * `PersistentVolumeClaim`'s own `String`, `Marshal` and `Unmarshal`. Read + * from the files' text, so a method that really is promoted, like the + * `GetName` `ObjectMeta` gives `Pod`, still links. + */ +function goUnreadFileDeclares(typeName: string, methodName: string, dir: string, context: ResolutionContext): boolean { + for (const f of goUnreadPackageFiles(dir, context)) { + const methods = goUnreadFileMethods(f, context); + if (methods === null || methods.has(`${typeName}::${methodName}`)) return true; } - return result; + return false; } const GO_PACKAGE_FILES = new WeakMap>(); @@ -20740,8 +20756,9 @@ function goUnreadPackageFiles(dir: string, context: ResolutionContext): string[] } GO_PACKAGE_FILES.set(context, files); } - // A file the index read has a node of its own at least. - const unread = (files.get(dir) ?? []).filter((f) => context.getNodesInFile(f).length === 0); + // A file the index read has a node of its own, named after it. + const unread = (files.get(dir) ?? []).filter((f) => + (context.getNodesInFileNamed?.(f, path.posix.basename(f)) ?? context.getNodesInFile(f)).length === 0); memo.set(dir, unread); return unread; } From 847f1df77deb372a7148004afbd046418ce3f4b6 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 21 Oct 2026 18:13:54 +0000 Subject: [PATCH 03/95] fix(go): an interface or struct embedding a standard-library type counts its methods (#2759) goImplementsEdges (#584) follows declared embeddings since #2402, but an embedding of a type outside the index resolves to nothing, so it added no methods: prometheus's TBRun (testing.TB plus Run) went to a struct with only a Run, QueryLogger (slog.Handler plus io.Closer) had no implementations, kapacitor's Diagnostic (a third-party Logger plus Log) took every struct with a Log, and the predeclared `error` was not recorded at all. - go-stdlib-methods.ts lists the methods of 130 standard-library types, each as `name:params/results`, generated from go1.27.3's source and counted the way goSignatureArity counts a stored signature. - Both extractors record an embedded `error`, which stays unresolved. The predeclared-type set is unchanged, so alias and defined-type references still skip it. - goImplementsEdges reads each Go type's failed `extends` rows and reads the package back off the line (goRefQualification, now exported). A standard-library type's methods join the method set, keyed as goMethodKey keys the project's (`std:` for an unexported one), and the arity gate holds them to the table's counts on both sides: a `sync.Mutex` makes a struct hugo's `Locker`, not etcd's gRPC `LockClient`. Any other outside type is met by a type embedding the same one, or by the only type with every other method the interface names. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/go-implements-embedding.test.ts | 15 +- __tests__/go-implements-stdlib.test.ts | 491 ++++++++++++++++++++++ __tests__/go-interface-embedding.test.ts | 9 +- __tests__/go-struct-field-refs.test.ts | 2 + codegraph-kernel/src/go.rs | 7 +- src/db/queries.ts | 36 ++ src/extraction/languages/go.ts | 6 +- src/resolution/callback-synthesizer.ts | 143 +++++-- src/resolution/go-stdlib-methods.ts | 253 +++++++++++ src/resolution/name-matcher.ts | 9 +- 11 files changed, 933 insertions(+), 39 deletions(-) create mode 100644 __tests__/go-implements-stdlib.test.ts create mode 100644 src/resolution/go-stdlib-methods.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 69ded831bf..0619badc1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -159,6 +159,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C and C++, a call written from the global scope, like `::leveldb::RepairDB(…)` or `::hpb::CreateMessage(…)`, now links to the function of exactly that name, and one through a namespace alias, like protobuf's `::_pbi::…` calls, to the function in the aliased namespace. Before, these calls were dropped unless the repository also held Swift and Objective-C files, and even then a name spelled out in full, like `::google::protobuf::internal::TSanWrite(…)`, linked to nothing, while `::operator delete(…)` linked to one class's own `operator delete`. A global call never links to a function or method of the same name inside a namespace or class, such as a wrapper `open()` that calls the C library's `::open(…)`, and `::std::…` and `::testing::…` calls stay unlinked unless that library's source is part of your project. Re-index C and C++ projects after upgrading. - In Go, an interface that embeds another, like gin's `IRouter` embedding `IRoutes`, now links to it, so `codegraph_explore` and impact show the embedded interface as its base. Before, every embedded interface was dropped. An embedded field written through another package of your project, like `*refresh.Discovery` or `storage.Appender`, or with type arguments, like `List[int]`, now links too. One from the standard library or another module, like `io.Closer` or `sync.Mutex`, links to nothing rather than to a project type that shares its name, and a type-set constraint like `~int | ~float64` is not an embedding. An embedded type written without a package now always links to its own package's type: before, etcd's `Client`, which embeds its own `Lease` interface, was linked to a `Lease` struct in another package. Re-index Go projects after upgrading. - In Go, working out which structs implement an interface now counts the methods of the interfaces it embeds, and the methods a struct gets from the types it embeds. Before, only each type's own methods counted: etcd's `AuthReadTx`, which embeds `UnsafeAuthReader` and adds `RLock` and `RUnlock`, was listed as implemented by every read-write lock in the project, an interface that only embeds others had no implementations at all, and a struct that gets its methods from an embedded base, like prometheus's service discoveries embedding `refresh.Discovery` or gin's `Engine` embedding `RouterGroup`, was missing. `codegraph_explore`, impact and the type hierarchy now list the right implementations, and a call through the interface reaches a method the embedding struct overrides, like gin's `Engine.Use`. Re-index Go projects after upgrading. +- In Go, the methods a type gets from an embedded standard-library type, like `io.Closer`, `http.ResponseWriter`, `testing.TB`, `*os.File` or `sync.Mutex`, or from the built-in `error`, now count when working out which types implement an interface, each with its parameters and results. Before, they were left out: prometheus's `TBRun`, which is all of `testing.TB` plus `Run`, was listed as implemented by a type that only has a `Run`, an interface made only of standard-library ones, like `slog.Handler` plus `io.Closer`, had no implementations at all, and a struct embedding `sync.Mutex` didn't implement an interface of your own that asks for `Lock` and `Unlock`. An interface that embeds a type from another module, whose methods can't be seen, is now implemented only by a type that embeds the same one or by the only type with all its other methods, not by every type that shares a method name with it. Re-index Go projects after upgrading. - In Go, a call through an interface now reaches a method that the implementing struct gets from a type it embeds: prometheus's `targetScraper` gets `Report` from the `*Target` it embeds, so a call through the `scraper` interface now reaches `Target.Report`, and etcd's `Hasher.MemberID` reaches `EtcdServer.MemberID` through `hasherAdapter`. Before, the call reached only structs that declare the method themselves, often just a test double, so `codegraph_explore`'s call path stopped at the interface. The method is picked the way Go picks it, from the nearest embedded type, and none is linked when two embedded types at the same depth both have it or it comes from an embedded interface. Re-index Go projects after upgrading. - In Go, a type defined over a slice, map, function or basic type now counts as implementing the interfaces its methods satisfy, just as a struct does: prometheus's `type staticDiscoverer []*targetgroup.Group` is a `Discoverer`, gin's `type formSource map[string][]string` a `setter`, etcd's `type Int64Comparable int64` a `Comparable`, and an adapter like `type HandlerFunc func(…)` a `Handler`. Before, only structs were considered, so these types were missing from an interface's implementations in `codegraph_explore`, impact and the type hierarchy, and a call through the interface, like `Discoverer.Run` or `setter.TrySet`, stopped there instead of reaching their methods. Re-index Go projects after upgrading. - In Go, a type alias like `type Event = mvccpb.Event`, alone or inside a `type ( … )` group, is now indexed and links to the type it names, so impact on that type reaches the code that uses the alias. Before, aliases were missing: a use written through the alias's package, like `clientv3.Event` or prometheus's `storage.AOptions{…}`, linked to nothing, and one inside the package, like `&PrometheusVersion{}` in prometheus's `web` package, linked to whichever type of that name the project had, there a test helper's struct. A method called on a value of an alias's type now reaches the aliased type's method rather than a namesake's, and a generic alias like `type Items[T any] = List[T]` links to `List`, so `xs.Len()` on one reaches `List.Len` instead of nothing. Re-index Go projects after upgrading. diff --git a/__tests__/go-implements-embedding.test.ts b/__tests__/go-implements-embedding.test.ts index 702a52f634..ca317090c8 100644 --- a/__tests__/go-implements-embedding.test.ts +++ b/__tests__/go-implements-embedding.test.ts @@ -217,11 +217,18 @@ func (both) R() {} `, 'empty/empty.go': `package empty -import "io" +import ( + "io" + + "go.uber.org/zap/zapcore" +) type Any interface{} -// Embeds only what the project does not declare: nothing to match. +// Embeds only a type nothing here names the methods of: nothing to match. +type Core interface{ zapcore.Core } + +// What io.Closer asks for: a Close. type Closer interface{ io.Closer } `, // More implementers than the cap: every unit through the base they all @@ -294,8 +301,10 @@ describe('Go implicit satisfaction counts embedded methods on both sides', () => 'gin.RouterGroup -> gin.IRouter', 'gin.RouterGroup -> gin.IRoutes', 'gin.routeInfo -> gin.Chain', + 'storage.BaseQuerier -> empty.Closer', 'storage.BaseQuerier -> storage.ChunkQuerier', 'storage.BaseQuerier -> storage.LabelQuerier', + 'tsdb.blockQuerier -> empty.Closer', 'tsdb.blockQuerier -> storage.ChunkQuerier', 'tsdb.blockQuerier -> storage.LabelQuerier', 'tsdb.blockQuerier -> storage.Querier', @@ -337,7 +346,7 @@ describe('Go implicit satisfaction counts embedded methods on both sides', () => const into = (name: string, file: string) => cg.getIncomingEdgesTo([one(name, file).id], ['implements']).filter((e) => meta(e).synthesizedBy === 'go-implements'); expect(into('Any', 'empty/empty.go')).toEqual([]); - expect(into('Closer', 'empty/empty.go')).toEqual([]); + expect(into('Core', 'empty/empty.go')).toEqual([]); }); it('caps an interface at 40 implementers, keeping those that declare its methods', () => { diff --git a/__tests__/go-implements-stdlib.test.ts b/__tests__/go-implements-stdlib.test.ts new file mode 100644 index 0000000000..35255c1d69 --- /dev/null +++ b/__tests__/go-implements-stdlib.test.ts @@ -0,0 +1,491 @@ +/** + * Go's implicit interface satisfaction (#584) compares method sets, following + * the embeddings that resolve (#2402). An embedding of a type outside the + * index resolves to nothing, so it added nothing: + * + * type TBRun interface { // prometheus's promql/promqltest + * testing.TB // two dozen methods, one of them unexported + * Run(string, func(*testing.T)) bool + * } + * + * asked only for `Run`, and 40 service discoverers "implemented" it. The + * embedded `error` was not even recorded. + * + * The methods of the standard-library types projects embed are known now, + * with their parameter and result counts, and `error` is recorded as the + * embedding it is. Such a method counts like any other, at its counts: an + * embedded `sync.Mutex` makes a struct hugo's `Locker`, but not etcd's gRPC + * `LockClient`, whose `Lock` takes a context and a request, and a `Write` that + * takes a record is no `io.Writer`'s. + * An unexported one is its own package's, as Go has it: `testing.TB`'s + * `private`, which no project type can declare. Embedding any other outside + * type (`zapcore.Core`, `grpc.ClientStream`) asks for methods nothing names: + * a struct embedding the same type has them, and otherwise only the one struct + * with every method the interface does name is taken to implement it. + */ +import { describe, it, expect, beforeAll, afterAll, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { buildTypeHierarchy } from '../src/graph/type-hierarchy'; +import { goStdlibMethods } from '../src/resolution/go-stdlib-methods'; +import type { Edge, Node } from '../src/types'; + +/** + * A project struct declaring every method `testing.TB` has, at its counts, + * its unexported `private` included, and a TBRun's `Run`: in a package whose + * directory is itself named `testing`, its `private` is still not testing's. + */ +function fakeTestingTB(): string { + const lines = ['package testing', '', 'type fakeTB struct{}', '']; + for (const [name, arity] of [...goStdlibMethods('testing', 'TB')!, ['Run', '2/1'] as const]) { + const [params, results] = arity.split('/').map(Number) as [number, number]; + const list = Array.from({ length: params }, (_, i) => `a${i} int`).join(', '); + const result = results === 0 ? '' : results === 1 ? ' int' : ` (${Array(results).fill('int').join(', ')})`; + lines.push(`func (f *fakeTB) ${name}(${list})${result} { panic(0) }`); + } + return `${lines.join('\n')}\n`; +} + +const FILES: Record = { + 'go.mod': 'module example.com/app\n\ngo 1.22\n', + // prometheus. + 'promql/promql.go': `package promql + +import ( + "context" + "io" + "log/slog" + "os" + "testing" +) + +// All of testing.TB, which only an embedding can have, and Run. +type TBRun interface { + testing.TB + Run(string, func(*testing.T)) bool +} + +// A Run, and nothing of testing.TB. +type Discovery struct{} + +func (d *Discovery) Run(name string, f func(*testing.T)) bool { return false } + +// testing.TB through *testing.T, and a Run of its own. +type tRunner struct{ *testing.T } + +func (r tRunner) Run(name string, f func(*testing.T)) bool { return r.T.Run(name, f) } + +// discovery/file's test helper: all of TBRun through *testing.T, Run included. +type testRunner struct { + *testing.T + dir string +} + +type syncWriter interface { + io.Writer + Sync() error +} + +// Sync, and no Write. +type scrapePool struct{} + +func (sp *scrapePool) Sync() error { return nil } + +type queryLogFile struct{} + +func (f *queryLogFile) Write(p []byte) (int, error) { return len(p), nil } +func (f *queryLogFile) Sync() error { return nil } + +// Its Write takes a record and returns only an error: no io.Writer's. +type walLog struct{} + +func (w *walLog) Write(rec *Record) error { return nil } +func (w *walLog) Sync() error { return nil } + +type SegmentFile interface { + Stat() (os.FileInfo, error) + Sync() error + io.Writer + io.Reader + io.Closer +} + +// Write through the SegmentFile it embeds, which has it from io.Writer. +type Segment struct { + SegmentFile + dir string +} + +// No method of its own: what slog.Handler and io.Closer ask for. +type QueryLogger interface { + slog.Handler + io.Closer +} + +type JSONFileLogger struct{} + +func (l *JSONFileLogger) Enabled(ctx context.Context, level slog.Level) bool { return true } +func (l *JSONFileLogger) Handle(ctx context.Context, r slog.Record) error { return nil } +func (l *JSONFileLogger) WithAttrs(attrs []slog.Attr) slog.Handler { return l } +func (l *JSONFileLogger) WithGroup(name string) slog.Handler { return l } +func (l *JSONFileLogger) Close() error { return nil } +`, + // etcd. + 'fileutil/fileutil.go': `package fileutil + +import ( + "bufio" + "context" + "io" + "io/fs" + "os" + "sync" +) + +type FileReader interface { + io.Reader + FileInfo() (fs.FileInfo, error) +} + +// Read is promoted from *os.File. +type fileReader struct{ *os.File } + +func (fr *fileReader) FileInfo() (fs.FileInfo, error) { return fr.Stat() } + +// Read is promoted from *bufio.Reader. +type FileBufReader struct { + *bufio.Reader + fi fs.FileInfo +} + +func (fb *FileBufReader) FileInfo() (fs.FileInfo, error) { return fb.fi, nil } + +// FileInfo, and nothing to Read with. +type fileStat struct{} + +func (s *fileStat) FileInfo() (fs.FileInfo, error) { return nil, nil } + +// v3lockpb's client: Lock and Unlock are gRPC calls. +type LockClient interface { + Lock(ctx context.Context, name string) error + Unlock(ctx context.Context, key string) error +} + +// hugo's: what a mutex has. +type Locker interface { + Lock() + Unlock() +} + +// A mutex's Lock and Unlock: a Locker's, not a LockClient's. +type waitRecorder struct { + sync.Mutex + actions []string +} + +type lockProxy struct{} + +func (p *lockProxy) Lock(ctx context.Context, name string) error { return nil } +func (p *lockProxy) Unlock(ctx context.Context, key string) error { return nil } +`, + // gin. + 'gin/response_writer.go': `package gin + +import ( + "bufio" + "net" + "net/http" +) + +type ResponseWriter interface { + http.ResponseWriter + http.Hijacker + http.Flusher + http.CloseNotifier + Status() int + Size() int + WriteString(string) (int, error) + Written() bool + WriteHeaderNow() + Pusher() http.Pusher +} + +// Header is promoted from the http.ResponseWriter it wraps. +type responseWriter struct { + http.ResponseWriter + size int + status int +} + +func (w *responseWriter) WriteHeader(code int) {} +func (w *responseWriter) WriteHeaderNow() {} +func (w *responseWriter) Write(data []byte) (int, error) { return len(data), nil } +func (w *responseWriter) WriteString(s string) (int, error) { return len(s), nil } +func (w *responseWriter) Status() int { return w.status } +func (w *responseWriter) Size() int { return w.size } +func (w *responseWriter) Written() bool { return w.size >= 0 } +func (w *responseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { return nil, nil, nil } +func (w *responseWriter) CloseNotify() <-chan bool { return nil } +func (w *responseWriter) Flush() {} +func (w *responseWriter) Pusher() http.Pusher { return nil } + +// The methods ResponseWriter declares, none that http asks for. +type sizeTracker struct{ size, status int } + +func (t *sizeTracker) Status() int { return t.status } +func (t *sizeTracker) Size() int { return t.size } +func (t *sizeTracker) WriteString(s string) (int, error) { return len(s), nil } +func (t *sizeTracker) Written() bool { return t.size >= 0 } +func (t *sizeTracker) WriteHeaderNow() {} +func (t *sizeTracker) Pusher() http.Pusher { return nil } +`, + 'errs/errs.go': `package errs + +type coded interface { + error + Code() int +} + +// Error is promoted from the error it wraps. +type statusErr struct { + error + code int +} + +func (e statusErr) Code() int { return e.code } + +// A Code, and no Error. +type counter struct{ n int } + +func (c *counter) Code() int { return c.n } +`, + // Interfaces embedding types nothing here describes. + 'diag/diag.go': `package diag + +import ( + "github.com/libdns/libdns" + plog "github.com/prometheus/common/log" + "go.uber.org/zap" + "go.uber.org/zap/zapcore" + "google.golang.org/grpc" +) + +// kapacitor's scraper.Diagnostic: Log, and a Logger. +type Diagnostic interface { + plog.Logger + Log(keyvals ...interface{}) error +} + +type ScraperHandler struct{} + +func (h *ScraperHandler) Debug(args ...interface{}) {} +func (h *ScraperHandler) Log(keyvals ...interface{}) error { return nil } + +type AlertNode struct{} + +func (n *AlertNode) Log(keyvals ...interface{}) error { return nil } + +// caddy's: nothing else has a FlushTo. +type LogBufferCoreInterface interface { + zapcore.Core + FlushTo(*zap.Logger) +} + +type LogBufferCore struct{} + +func (c *LogBufferCore) FlushTo(l *zap.Logger) {} + +type ClientStream interface { + grpc.ClientStream + Send(m any) error + Recv(m any) error +} + +// Has the gRPC stream's methods by embedding it. +type wrappedStream struct{ grpc.ClientStream } + +func (s *wrappedStream) Send(m any) error { return nil } +func (s *wrappedStream) Recv(m any) error { return nil } + +// Sends and receives, as a server stream. +type serverStream struct{} + +func (s *serverStream) Send(m any) error { return nil } +func (s *serverStream) Recv(m any) error { return nil } + +// Nothing it asks for has a name here. +type DNSProvider interface { + libdns.RecordGetter + libdns.RecordSetter +} +`, + 'testing/fake.go': fakeTestingTB(), +}; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-go-implements-stdlib-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}, 120_000); + +afterAll(() => { + cg?.destroy(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const meta = (e: Edge): Record => (e.metadata ?? {}) as Record; +const pkg = (n: Node | null): string => `${path.posix.dirname(n?.filePath ?? '?')}.${n?.name}`; + +/** The one non-import node with this name in this file. */ +function one(name: string, file: string): Node { + const found = cg.getNodesByName(name).filter((n) => n.filePath === file && n.kind !== 'import'); + expect(found, `${name} in ${file}`).toHaveLength(1); + return found[0]!; +} + +/** Every synthesized struct → interface edge, as `pkg.Struct -> pkg.Interface`. */ +function goImplements(): string[] { + const structs = cg.getNodesByKind('struct').filter((n) => n.language === 'go'); + return cg + .getOutgoingEdgesFrom(structs.map((n) => n.id), ['implements']) + .filter((e) => meta(e).synthesizedBy === 'go-implements') + .map((e) => `${pkg(cg.getNode(e.source))} -> ${pkg(cg.getNode(e.target))}`) + .sort(); +} + +describe('Go implicit satisfaction knows what standard-library embeddings bring', () => { + it('links exactly the structs that have every method, embedded ones included', () => { + expect(goImplements()).toEqual([ + 'diag.LogBufferCore -> diag.LogBufferCoreInterface', + 'diag.wrappedStream -> diag.ClientStream', + 'errs.statusErr -> errs.coded', + 'fileutil.FileBufReader -> fileutil.FileReader', + 'fileutil.fileReader -> fileutil.FileReader', + // An *os.File is what prometheus's SegmentFile and syncWriter describe. + 'fileutil.fileReader -> promql.SegmentFile', + 'fileutil.fileReader -> promql.syncWriter', + 'fileutil.lockProxy -> fileutil.LockClient', + 'fileutil.waitRecorder -> fileutil.Locker', + 'gin.responseWriter -> gin.ResponseWriter', + 'promql.JSONFileLogger -> promql.QueryLogger', + 'promql.Segment -> promql.syncWriter', + 'promql.queryLogFile -> promql.syncWriter', + 'promql.tRunner -> promql.TBRun', + 'promql.testRunner -> promql.TBRun', + ]); + }); + + it('asks for every method of testing.TB and io.Writer, testing\'s own private and io\'s counts included', () => { + const implementers = (name: string, file: string) => + (buildTypeHierarchy(cg, one(name, file))?.descendants ?? []).map((d) => `${d.relation} ${pkg(d.node)}`).sort(); + // Not the discoverer with a Run, nor fakeTB, whose private is its own package's. + expect(implementers('TBRun', 'promql/promql.go')).toEqual(['implements promql.tRunner', 'implements promql.testRunner']); + // Not walLog, whose Write takes a record and returns only an error. + expect(implementers('syncWriter', 'promql/promql.go')).toEqual([ + 'implements fileutil.fileReader', + 'implements promql.Segment', + 'implements promql.queryLogFile', + ]); + // A call through TBRun reaches tRunner's Run, not the discoverer's; + // testRunner's is testing's, outside the graph. + const run = cg + .getOutgoingEdgesFrom([one('TBRun', 'promql/promql.go').id], ['contains']) + .map((e) => cg.getNode(e.target)) + .find((n) => n?.name === 'Run')!; + expect( + cg + .getOutgoingEdgesFrom([run.id], ['calls']) + .filter((e) => meta(e).synthesizedBy === 'interface-impl') + .map((e) => cg.getNode(e.target)?.qualifiedName) + ).toEqual(['tRunner::Run']); + }); + + it('records an embedded error, which stays outside the graph', () => { + for (const [name, file] of [['statusErr', 'errs/errs.go'], ['coded', 'errs/errs.go']] as const) { + const node = one(name, file); + expect( + cg.getUnresolvedReferencesFrom(node.id).filter((r) => r.referenceKind === 'extends').map((r) => r.referenceName) + ).toEqual(['error']); + expect(cg.getOutgoingEdgesFrom([node.id], ['extends', 'implements']).filter((e) => e.provenance !== 'heuristic')).toEqual([]); + } + }); + + it('holds a method a standard-library embedding brings to the counts of the one it answers', () => { + const into = (name: string, file: string) => + cg + .getIncomingEdgesTo([one(name, file).id], ['implements']) + .filter((e) => meta(e).synthesizedBy === 'go-implements') + .map((e) => pkg(cg.getNode(e.source))) + .sort(); + // waitRecorder's Lock and Unlock are sync.Mutex's: they take nothing. + expect(into('Locker', 'fileutil/fileutil.go')).toEqual(['fileutil.waitRecorder']); + expect(into('LockClient', 'fileutil/fileutil.go')).toEqual(['fileutil.lockProxy']); + // responseWriter's Header is http.ResponseWriter's, which ResponseWriter asks for. + expect(into('ResponseWriter', 'gin/response_writer.go')).toEqual(['gin.responseWriter']); + }); + + it('links an interface embedding an undescribed type to the same embedding, or to its one candidate', () => { + const into = (name: string) => + cg + .getIncomingEdgesTo([one(name, 'diag/diag.go').id], ['implements']) + .filter((e) => meta(e).synthesizedBy === 'go-implements') + .map((e) => pkg(cg.getNode(e.source))) + .sort(); + // Two structs have a Log: neither is known to be a plog.Logger. + expect(into('Diagnostic')).toEqual([]); + expect(into('LogBufferCoreInterface')).toEqual(['diag.LogBufferCore']); + expect(into('ClientStream')).toEqual(['diag.wrappedStream']); + expect(into('DNSProvider')).toEqual([]); + }); +}); + +describe('a sync that changes what an interface embeds', () => { + let dir = ''; + let graph: CodeGraph | undefined; + + afterEach(() => { + graph?.destroy(); + graph = undefined; + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + dir = ''; + }); + + it('re-reads the standard-library type it names', async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-go-implements-stdlib-sync-')); + const write = (rel: string, content: string) => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + const api = 'package api\n\nimport stdio "io"\n\ntype SyncWriter interface {\n\tstdio.Writer\n\tSync() error\n}\n'; + write('go.mod', 'module example.com/app\n\ngo 1.22\n'); + write('api/api.go', api); + write( + 'log/file.go', + 'package log\n\ntype file struct{}\n\nfunc (f *file) Write(p []byte) (int, error) { return len(p), nil }\nfunc (f *file) Sync() error { return nil }\n' + ); + graph = await CodeGraph.init(dir, { index: true }); + const g = graph; + const linked = () => + g + .getOutgoingEdgesFrom(g.getNodesByKind('struct').map((n) => n.id), ['implements']) + .filter((e) => meta(e).synthesizedBy === 'go-implements') + .map((e) => `${g.getNode(e.source)?.name} -> ${g.getNode(e.target)?.name}`); + + expect(linked()).toEqual(['file -> SyncWriter']); + // An io.Reader has to Read, which file can't. + write('api/api.go', api.replace('stdio.Writer', 'stdio.Reader')); + await g.sync({ paths: ['api/api.go'] }); + expect(linked()).toEqual([]); + write('api/api.go', api); + await g.sync({ paths: ['api/api.go'] }); + expect(linked()).toEqual(['file -> SyncWriter']); + }, 60_000); +}); diff --git a/__tests__/go-interface-embedding.test.ts b/__tests__/go-interface-embedding.test.ts index 2883c6f50b..f1d1736c3d 100644 --- a/__tests__/go-interface-embedding.test.ts +++ b/__tests__/go-interface-embedding.test.ts @@ -18,9 +18,10 @@ * resolution reads the package back from the source. `storage.LabelQuerier` * reaches that package's interface; `io.Closer` and `sync.Mutex` stay * unresolved rather than landing on a project type that shares the name. - * Go's predeclared types are never recorded: `int64` is a constraint's term, - * and `error`, `any` and `comparable` exist in no graph. Runs against the - * native kernel (when built) and the wasm extractor, which must agree. + * Go's predeclared types are not recorded: `int64` is a constraint's term, + * and `any` and `comparable` have no methods. `error` has one, so embedding it + * is recorded, and stays unresolved like `io.Closer`. Runs against the native + * kernel (when built) and the wasm extractor, which must agree. * * Resolution keeps an embedded type in its package. A bare name is its own * package's type, not another package's struct of that name, which the Go @@ -204,6 +205,7 @@ describe('Go embedded types are supertypes', () => { 'Querier extends Closer', 'Querier extends LabelQuerier', 'Querier extends Queryable', + 'Timeout extends error', ]); expectRefsOnTheirNames(result, source); // The interfaces' own methods are still its members, the embeddings are not. @@ -228,6 +230,7 @@ describe('Go embedded types are supertypes', () => { 'Store extends List', 'Store extends Mutex', 'Store extends Reader', + 'Store extends error', ]); expectRefsOnTheirNames(result, source); }); diff --git a/__tests__/go-struct-field-refs.test.ts b/__tests__/go-struct-field-refs.test.ts index 351f10841e..385d686051 100644 --- a/__tests__/go-struct-field-refs.test.ts +++ b/__tests__/go-struct-field-refs.test.ts @@ -260,6 +260,7 @@ describe('a Go struct references the types its fields are declared with', () => .map((r) => `${r.referenceKind} ${r.referenceName}`) .sort(); }; + // An embedded `error` is an embedding too: the one predeclared type with a method. expect(refsOf('Embedder')).toEqual([ 'extends Base', 'extends ClientWithList', @@ -267,6 +268,7 @@ describe('a Go struct references the types its fields are declared with', () => 'extends List', 'extends Pointer', 'extends Queryable', + 'extends error', 'references Pod', 'references PodList', ]); diff --git a/codegraph-kernel/src/go.rs b/codegraph-kernel/src/go.rs index 9b52e97ae7..71d7ba58db 100644 --- a/codegraph-kernel/src/go.rs +++ b/codegraph-kernel/src/go.rs @@ -936,7 +936,7 @@ impl<'t> Walker<'t> { /// embedding names — `T`, `*T` (the `*` is a sibling token), `pkg.T` and /// `T[X]` all embed `T`. A qualified type yields its name; resolution reads /// the package back from the source at that position. None for any other - /// type and for a predeclared one. + /// type and for a predeclared one but `error`, an interface with a method. fn embedded_type_name(&self, ty: Option>) -> Option> { let mut ty = ty?; if ty.kind() == "generic_type" { @@ -944,7 +944,10 @@ impl<'t> Walker<'t> { } match ty.kind() { "qualified_type" => ty.child_by_field_name("name"), - "type_identifier" if !is_go_predeclared_type(self.text(ty)) => Some(ty), + "type_identifier" => { + let name = self.text(ty); + (name == "error" || !is_go_predeclared_type(name)).then_some(ty) + } _ => None, } } diff --git a/src/db/queries.ts b/src/db/queries.ts index 331f9f4869..5b02ba255d 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -3027,6 +3027,42 @@ export class QueryBuilder { })); } + /** + * {@link getUnresolvedReferencesFrom} for many nodes at once, limited to + * `kinds`: a Go type's embeddings of types outside the index, say. Chunked + * probe over `idx_unresolved_from_node`. + */ + getUnresolvedReferencesFromNodes(nodeIds: Iterable, kinds: readonly string[]): UnresolvedReference[] { + const unique = [...new Set(nodeIds)]; + const found: UnresolvedReference[] = []; + if (unique.length === 0 || kinds.length === 0) return found; + const kindPlaceholders = kinds.map(() => '?').join(','); + for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) { + const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); + const placeholders = chunk.map(() => '?').join(','); + const rows = this.db + .prepare( + `SELECT * FROM unresolved_refs + WHERE from_node_id IN (${placeholders}) + AND reference_kind IN (${kindPlaceholders})` + ) + .all(...chunk, ...kinds) as UnresolvedRefRow[]; + for (const row of rows) { + found.push({ + fromNodeId: row.from_node_id, + referenceName: row.reference_name, + referenceKind: row.reference_kind as EdgeKind, + line: row.line, + column: row.col, + filePath: row.file_path, + language: row.language as Language, + rowId: row.id, + }); + } + } + return found; + } + /** * Find all edges where both source and target are in the given node set. * Useful for recovering inter-node connectivity after BFS. diff --git a/src/extraction/languages/go.ts b/src/extraction/languages/go.ts index 7dae97d16e..d3a5144481 100644 --- a/src/extraction/languages/go.ts +++ b/src/extraction/languages/go.ts @@ -55,13 +55,15 @@ const GO_PREDECLARED_TYPES: ReadonlySet = new Set([ * embed `Base`. A qualified type yields its name, not its package: the package * stays in the source text, and resolution reads it back from the reference's * position (#2322). Undefined for any other type (a literal, `~T`, a union's - * term) and for a predeclared one. + * term) and for a predeclared one but `error`: an interface with a method, + * which an embedding brings in like any type's from outside the index. */ export function goEmbeddedTypeName(type: SyntaxNode | null | undefined, source: string): SyntaxNode | undefined { if (type?.type === 'generic_type') type = getChildByField(type, 'type'); if (type?.type === 'qualified_type') return getChildByField(type, 'name') ?? undefined; if (type?.type !== 'type_identifier') return undefined; - return GO_PREDECLARED_TYPES.has(getNodeText(type, source)) ? undefined : type; + const name = getNodeText(type, source); + return name === 'error' || !GO_PREDECLARED_TYPES.has(name) ? type : undefined; } /** diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index f54ce3c44f..fa7ed1da70 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -40,7 +40,8 @@ import { createYielder, type MaybeYield } from './cooperative-yield'; import { crossTierEdges, hasCrossTierPattern, hasTestRequestPattern, testRequestEdges } from './tier-synthesizer'; import { enclosingFn, makeLineAt } from './synth-utils'; import { resolveImportPath } from './import-resolver'; -import { cppSpecializationArguments, crossesCodeBoundary, jsCodeBindsName } from './name-matcher'; +import { cppSpecializationArguments, crossesCodeBoundary, goRefQualification, jsCodeBindsName } from './name-matcher'; +import { goStdlibMethods } from './go-stdlib-methods'; import { declaredComponent, exportedComponent, lazyValueModules, wrappedComponentName } from './frameworks/react'; import { jsxCode } from './jsx-code'; @@ -843,6 +844,25 @@ function goMethodKey(m: Node): string { const file = m.filePath.replace(/\\/g, '/'); return `${file.slice(0, Math.max(file.lastIndexOf('/'), 0))}\0${m.name}`; } +/** + * {@link goMethodKey} for a method of a standard-library type: its name, and + * for an unexported one (`testing.TB`'s `private`) the declaring package, + * spelled so it is no directory of the project's. + */ +function goStdlibMethodKey(importPath: string | null, name: string): string { + return GO_EXPORTED_NAME.test(name) ? name : `std:${importPath ?? ''}\0${name}`; +} +/** What a Go type has, or an interface asks for. */ +interface GoMethodNames { + /** + * Its methods by {@link goMethodKey}: its own, the project types' it + * embeds, and those the standard-library types it embeds bring + * (./go-stdlib-methods), `error`'s included. + */ + readonly methods: Set; + /** Embedded types from outside the index whose methods nothing names, by package and name. */ + readonly opaque: ReadonlySet; +} /** * The top-level items of the Go bracketed list opening at `text[open]` — the * parameters of `(ctx context.Context, a, b int)` are `ctx context.Context`, `a` @@ -958,8 +978,18 @@ function goGrpcServerArity(signature: string | undefined): string | null { * has the methods promoted from the structs, interfaces and defined types it * embeds, at any depth: counting embedded interfaces alone would drop every * struct that satisfies one through promoted methods. Synthesized edges are - * never followed (this pass writes them), and an embedding that resolved to - * nothing (`io.Closer`) adds nothing. + * never followed (this pass writes them). + * + * A type embedded from outside the index leaves only the resolver's failed + * `extends` row. The methods of a standard-library type, and of `error`, are + * known (./go-stdlib-methods), with their counts: prometheus's `TBRun` is all + * of `testing.TB` plus `Run`, not any struct with a `Run`, and a struct + * embedding `sync.Mutex` is hugo's `Locker` but not etcd's gRPC `LockClient`, + * whose `Lock` takes a context and a request. Any other outside type's + * methods are unknown, so a struct embedding the same type has them, and + * otherwise only the one type with every method the interface does name is + * taken to implement it — kapacitor's `Diagnostic` (a third-party `Logger` + * plus `Log`) has no more claim on every struct with a `Log` than on none. * * Methods match by name, and an unexported name by its package too * ({@link goMethodKey}). prometheus has a `refresher { refresh(…) }` in each of @@ -973,10 +1003,12 @@ function goGrpcServerArity(signature: string | undefined): string | null { * Each wanted method also needs a declaration with its parameter and result * counts, read off the stored signatures: Go wants the signatures identical, * and the counts are what the text settles for certain — a cache's `Get(ctx, - * key, opts ...OpOption)` is no `Get(id types.ID) Peer`. A signature that - * doesn't read rules nothing out. A gRPC client interface also takes the - * structs serving its service (see `wantedArities`), the bridge a client's - * call crosses to reach its handler. + * key, opts ...OpOption)` is no `Get(id types.ID) Peer`. A standard-library + * method carries its counts in the table, so an `io.Writer`'s `Write` asks for + * one parameter and two results. A signature that doesn't read rules nothing + * out. A gRPC client interface also takes the structs serving its service + * (see `wantedArities`), the bridge a client's call crosses to reach its + * handler. * * Types are not compared, and a promoted name counts even where Go finds it * ambiguous or hidden by a field — over-approximation accepted, in line with @@ -984,7 +1016,7 @@ function goGrpcServerArity(signature: string | undefined): string | null { * (`any`) are skipped so they don't match every type, and a struct that * embeds the interface keeps the edge its embedding already is. */ -async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Promise { +async function goImplementsEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise { let scanned255 = 0; const edges: Edge[] = []; const seen = new Set(); @@ -1073,13 +1105,42 @@ async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Pr return [...reached]; }; - // Own methods plus every embedded type's. - const methodSet = (id: string): Set => { + // The types each one embeds from outside the index, as the resolver's failed + // `extends` rows: the methods of a standard-library type or of `error`, by + // key with their arities, or else the embedded type itself, named by the + // package it is written through so that the same embedding elsewhere is + // recognized. + const outside = new Map[]; opaque: string[] }>(); + for (const ref of queries.getUnresolvedReferencesFromNodes(typeIds, ['extends'])) { + if (ref.language !== 'go' || !ref.filePath) continue; + const name = ref.referenceName; + const { written, imported } = goRefQualification({ ...ref, filePath: ref.filePath, language: 'go' }, ctx); + const importPath = imported?.source ?? null; + const known = imported ? goStdlibMethods(importPath, name) : written ? undefined : goStdlibMethods(null, name); + let found = outside.get(ref.fromNodeId); + if (!found) outside.set(ref.fromNodeId, (found = { stdlib: [], opaque: [] })); + if (known) found.stdlib.push(new Map([...known].map(([m, arity]) => [goStdlibMethodKey(importPath, m), arity]))); + else if (imported) found.opaque.push(`${imported.source}.${name}`); + else if (written) found.opaque.push(`${written}.${name}`); + else found.opaque.push(`${ref.filePath.replace(/\\/g, '/').replace(/\/?[^/]*$/, '')}#${name}`); + } + await onYield(); + + // Own methods plus every embedded type's, a standard-library one's too. + const NONE_OPAQUE: ReadonlySet = new Set(); + const methodNames = (id: string): GoMethodNames => { const types = embedClosure(id); - if (types.length === 1) return ownMethods(id); - const names = new Set(); - for (const t of types) for (const m of ownMethods(t)) names.add(m); - return names; + if (types.length === 1 && !outside.has(id)) return { methods: ownMethods(id), opaque: NONE_OPAQUE }; + const methods = new Set(); + const opaque = new Set(); + for (const t of types) { + for (const m of ownMethods(t)) methods.add(m); + const out = outside.get(t); + if (!out) continue; + for (const known of out.stdlib) for (const key of known.keys()) methods.add(key); + for (const type of out.opaque) opaque.add(type); + } + return { methods, opaque }; }; const covers = (have: Set, want: Set): boolean => { @@ -1089,11 +1150,18 @@ async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Pr } return true; }; + const coversOpaque = (have: GoMethodNames, want: GoMethodNames): boolean => { + for (const type of want.opaque) { + if (!have.opaque.has(type)) return false; + } + return true; + }; // Each method in a type's method set, by its goMethodKey, → the arities // (`params/results`) its declarations there have, or null when one of their - // signatures doesn't read (then the name rules nothing out). Built only for - // the pairs whose names already match. + // signatures doesn't read (then the name rules nothing out). A method a + // standard-library embedding brings has the table's. Built only for the + // pairs whose names already match. const arityMemo = new Map | null>>(); const methodArities = (id: string): Map | null> => { let arities = arityMemo.get(id); @@ -1114,6 +1182,14 @@ async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Pr known.add(arity); } } + for (const methods of outside.get(t)?.stdlib ?? []) { + for (const [name, arity] of methods) { + let known = arities.get(name); + if (known === null) continue; + if (!known) arities.set(name, (known = new Set())); + known.add(arity); + } + } } arityMemo.set(id, arities); return arities; @@ -1137,6 +1213,8 @@ async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Pr } } } + // What only a standard-library embedding asks for keeps the table's arity. + for (const [name, arities] of own) if (!wanted.has(name)) wanted.set(name, arities); return wanted; }; @@ -1161,21 +1239,22 @@ async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Pr return true; }; - // A type without a method satisfies no interface this pass looks at. + // A type without a method, or an embedded type nothing describes, satisfies + // no interface this pass looks at. const candidates: Node[] = []; - const candidateMethods = new Map>(); + const candidateMethods = new Map(); for (const s of goImplementers) { if ((++scanned255 & 63) === 0) await onYield(); - const have = methodSet(s.id); - if (have.size === 0) continue; + const have = methodNames(s.id); + if (have.methods.size + have.opaque.size === 0) continue; candidates.push(s); candidateMethods.set(s.id, have); } for (const iface of goInterfaces) { if ((++scanned255 & 63) === 0) await onYield(); - const want = methodSet(iface.id); - if (want.size === 0) continue; // empty interface (`any`) — would match everything + const want = methodNames(iface.id); + if (want.methods.size + want.opaque.size === 0) continue; // empty interface (`any`) — would match everything let added = 0; const link = (s: Node): void => { const key = `${s.id}>${iface.id}`; @@ -1195,20 +1274,30 @@ async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Pr // of them, every defined type among them: under the cap, those are what // the interface-dispatch bridge links a call through the interface to. A // namesake it declares of an unexported method from another package is - // not one of them: the method it needs is promoted. + // not one of them: the method it needs is promoted. One that lacks only + // the embedded types nothing names the methods of counts when no other + // type has the rest. const throughEmbedding: Node[] = []; let wanted: Map | null> | undefined; + let covering = 0; + let sole: Node | undefined; for (const s of candidates) { if (added >= MAX_CALLBACKS_PER_CHANNEL) break; const have = candidateMethods.get(s.id)!; - if (!covers(have, want)) continue; - if (embeds(s.id).includes(iface.id)) continue; // declared by embedding it + if (!covers(have.methods, want.methods)) continue; if (!wanted) wanted = wantedArities(iface); if (!aritiesFit(s.id, wanted)) continue; + covering++; + if (embeds(s.id).includes(iface.id)) continue; // declared by embedding it + if (!coversOpaque(have, want)) { + if (covering === 1) sole = s; + continue; + } const own = ownMethods(s.id); - if (own !== have && !covers(own, want)) throughEmbedding.push(s); + if (want.opaque.size > 0 || (own !== have.methods && !covers(own, want.methods))) throughEmbedding.push(s); else link(s); } + if (sole && covering === 1 && want.methods.size > 0) throughEmbedding.push(sole); for (const s of throughEmbedding) { if (added >= MAX_CALLBACKS_PER_CHANNEL) break; link(s); @@ -5181,7 +5270,7 @@ export async function synthesizeCallbackEdges( // interface-dispatch bridge below reads `implements` edges from the DB, and // Go has none statically. (Other languages already have static implements // edges from extraction, so they don't need this pre-pass.) - const goImpl = has('go') ? await goImplementsEdges(queries, yieldToLoop) : NONE; + const goImpl = has('go') ? await goImplementsEdges(queries, ctx, yieldToLoop) : NONE; for (let i = 0; i < goImpl.length; i += 2000) { queries.insertEdges(goImpl.slice(i, i + 2000)); await yieldToLoop(); diff --git a/src/resolution/go-stdlib-methods.ts b/src/resolution/go-stdlib-methods.ts new file mode 100644 index 0000000000..a2c85387ad --- /dev/null +++ b/src/resolution/go-stdlib-methods.ts @@ -0,0 +1,253 @@ +/** + * The methods of the standard-library types Go code embeds, for Go's implicit + * interface satisfaction (#584). An embedded type from outside the index + * (`io.Closer`, `http.ResponseWriter`, `*os.File`) resolves to nothing, so + * without this an interface embedding `testing.TB` asked only for its own + * `Run`, and a struct embedding `*os.File` had no `Read`. + * + * Keyed by import path, then type name; each value lists the type's methods, + * space-separated, as `name:params/results`: the parameter and result counts + * goSignatureArity reads off a stored signature, so the arity gate holds a + * project method answering one to the same counts. An interface lists every + * method, embedded ones included, and so does a struct with those its embedded + * fields promote. Unexported names are kept where an interface requires one: + * `testing.TB`'s `private` and `reflect.Type`'s `common` and `uncommon` are + * why only an embedding satisfies those outside their own package. + * + * Read from Go's own source (go1.27.3). The interfaces are the ones projects + * embed; the structs are the commonly embedded ones that implement one of + * them. A struct whose methods would answer one of these interfaces' names + * with another signature (`http.Header.Write`, `sync.WaitGroup.Done`) is left + * out. + */ +const GO_STDLIB_METHODS: Readonly>>> = { + io: { + Reader: 'Read:1/2', + Writer: 'Write:1/2', + Closer: 'Close:0/1', + Seeker: 'Seek:2/2', + ReadWriter: 'Read:1/2 Write:1/2', + ReadCloser: 'Close:0/1 Read:1/2', + WriteCloser: 'Close:0/1 Write:1/2', + ReadWriteCloser: 'Close:0/1 Read:1/2 Write:1/2', + ReadSeeker: 'Read:1/2 Seek:2/2', + ReadSeekCloser: 'Close:0/1 Read:1/2 Seek:2/2', + WriteSeeker: 'Seek:2/2 Write:1/2', + ReadWriteSeeker: 'Read:1/2 Seek:2/2 Write:1/2', + ReaderFrom: 'ReadFrom:1/2', + WriterTo: 'WriteTo:1/2', + ReaderAt: 'ReadAt:2/2', + WriterAt: 'WriteAt:2/2', + ByteReader: 'ReadByte:0/2', + ByteScanner: 'ReadByte:0/2 UnreadByte:0/1', + ByteWriter: 'WriteByte:1/1', + RuneReader: 'ReadRune:0/3', + RuneScanner: 'ReadRune:0/3 UnreadRune:0/1', + StringWriter: 'WriteString:1/2', + PipeReader: 'Close:0/1 CloseWithError:1/1 Read:1/2', + PipeWriter: 'Close:0/1 CloseWithError:1/1 Write:1/2', + SectionReader: 'Outer:0/3 Read:1/2 ReadAt:2/2 Seek:2/2 Size:0/1', + }, + 'io/fs': { + FS: 'Open:1/2', + File: 'Close:0/1 Read:1/2 Stat:0/2', + DirEntry: 'Info:0/2 IsDir:0/1 Name:0/1 Type:0/1', + FileInfo: 'IsDir:0/1 ModTime:0/1 Mode:0/1 Name:0/1 Size:0/1 Sys:0/1', + ReadDirFile: 'Close:0/1 Read:1/2 ReadDir:1/2 Stat:0/2', + ReadDirFS: 'Open:1/2 ReadDir:1/2', + ReadFileFS: 'Open:1/2 ReadFile:1/2', + StatFS: 'Open:1/2 Stat:1/2', + SubFS: 'Open:1/2 Sub:1/2', + GlobFS: 'Glob:1/2 Open:1/2', + }, + os: { + File: 'Chdir:0/1 Chmod:1/1 Chown:2/1 Close:0/1 Fd:0/1 Name:0/1 PollFD:0/1 Read:1/2 ReadAt:2/2 ReadDir:1/2 ReadFrom:1/2 Readdir:1/2 Readdirnames:1/2 Seek:2/2 SetDeadline:1/1 SetReadDeadline:1/1 SetWriteDeadline:1/1 Stat:0/2 Sync:0/1 SyscallConn:0/2 Truncate:1/1 Write:1/2 WriteAt:2/2 WriteString:1/2 WriteTo:1/2', + FileInfo: 'IsDir:0/1 ModTime:0/1 Mode:0/1 Name:0/1 Size:0/1 Sys:0/1', + DirEntry: 'Info:0/2 IsDir:0/1 Name:0/1 Type:0/1', + Signal: 'Signal:0/0 String:0/1', + }, + fmt: { + Stringer: 'String:0/1', + GoStringer: 'GoString:0/1', + Formatter: 'Format:2/0', + State: 'Flag:1/1 Precision:0/2 Width:0/2 Write:1/2', + Scanner: 'Scan:2/1', + ScanState: 'Read:1/2 ReadRune:0/3 SkipSpace:0/0 Token:2/2 UnreadRune:0/1 Width:0/2', + }, + sort: { + Interface: 'Len:0/1 Less:2/1 Swap:2/0', + }, + 'container/heap': { + Interface: 'Len:0/1 Less:2/1 Pop:0/1 Push:1/0 Swap:2/0', + }, + context: { + Context: 'Deadline:0/2 Done:0/1 Err:0/1 Value:1/1', + }, + sync: { + Locker: 'Lock:0/0 Unlock:0/0', + Mutex: 'Lock:0/0 TryLock:0/1 Unlock:0/0', + RWMutex: 'Lock:0/0 RLock:0/0 RLocker:0/1 RUnlock:0/0 TryLock:0/1 TryRLock:0/1 Unlock:0/0', + }, + net: { + Conn: 'Close:0/1 LocalAddr:0/1 Read:1/2 RemoteAddr:0/1 SetDeadline:1/1 SetReadDeadline:1/1 SetWriteDeadline:1/1 Write:1/2', + PacketConn: 'Close:0/1 LocalAddr:0/1 ReadFrom:1/3 SetDeadline:1/1 SetReadDeadline:1/1 SetWriteDeadline:1/1 WriteTo:2/2', + Listener: 'Accept:0/2 Addr:0/1 Close:0/1', + Addr: 'Network:0/1 String:0/1', + Error: 'Error:0/1 Temporary:0/1 Timeout:0/1', + TCPConn: 'Close:0/1 CloseRead:0/1 CloseWrite:0/1 File:0/2 LocalAddr:0/1 MultipathTCP:0/2 Read:1/2 ReadFrom:1/2 RemoteAddr:0/1 SetDeadline:1/1 SetKeepAlive:1/1 SetKeepAliveConfig:1/1 SetKeepAlivePeriod:1/1 SetLinger:1/1 SetNoDelay:1/1 SetReadBuffer:1/1 SetReadDeadline:1/1 SetWriteBuffer:1/1 SetWriteDeadline:1/1 SyscallConn:0/2 Write:1/2 WriteTo:1/2', + UDPConn: 'Close:0/1 File:0/2 LocalAddr:0/1 Read:1/2 ReadFrom:1/3 ReadFromUDP:1/3 ReadFromUDPAddrPort:1/3 ReadMsgUDP:2/5 ReadMsgUDPAddrPort:2/5 RemoteAddr:0/1 SetDeadline:1/1 SetReadBuffer:1/1 SetReadDeadline:1/1 SetWriteBuffer:1/1 SetWriteDeadline:1/1 SyscallConn:0/2 Write:1/2 WriteMsgUDP:3/3 WriteMsgUDPAddrPort:3/3 WriteTo:2/2 WriteToUDP:2/2 WriteToUDPAddrPort:2/2', + UnixConn: 'Close:0/1 CloseRead:0/1 CloseWrite:0/1 File:0/2 LocalAddr:0/1 Read:1/2 ReadFrom:1/3 ReadFromUnix:1/3 ReadMsgUnix:2/5 RemoteAddr:0/1 SetDeadline:1/1 SetReadBuffer:1/1 SetReadDeadline:1/1 SetWriteBuffer:1/1 SetWriteDeadline:1/1 SyscallConn:0/2 Write:1/2 WriteMsgUnix:3/3 WriteTo:2/2 WriteToUnix:2/2', + TCPListener: 'Accept:0/2 AcceptTCP:0/2 Addr:0/1 Close:0/1 File:0/2 SetDeadline:1/1 SyscallConn:0/2', + UnixListener: 'Accept:0/2 AcceptUnix:0/2 Addr:0/1 Close:0/1 File:0/2 SetDeadline:1/1 SetUnlinkOnClose:1/0 SyscallConn:0/2', + }, + 'net/http': { + ResponseWriter: 'Header:0/1 Write:1/2 WriteHeader:1/0', + Hijacker: 'Hijack:0/3', + Flusher: 'Flush:0/0', + CloseNotifier: 'CloseNotify:0/1', + Pusher: 'Push:2/1', + Handler: 'ServeHTTP:2/0', + RoundTripper: 'RoundTrip:1/2', + File: 'Close:0/1 Read:1/2 Readdir:1/2 Seek:2/2 Stat:0/2', + FileSystem: 'Open:1/2', + CookieJar: 'Cookies:1/1 SetCookies:2/0', + ServeMux: 'Handle:2/0 HandleFunc:2/0 Handler:1/2 ServeHTTP:2/0', + Transport: 'CancelRequest:1/0 Clone:0/1 CloseIdleConnections:0/0 NewClientConn:3/2 RegisterProtocol:2/0 RoundTrip:1/2', + }, + 'net/http/httptest': { + ResponseRecorder: 'Flush:0/0 Header:0/1 Result:0/1 Write:1/2 WriteHeader:1/0 WriteString:1/2', + }, + bufio: { + Reader: 'Buffered:0/1 Discard:1/2 Peek:1/2 Read:1/2 ReadByte:0/2 ReadBytes:1/2 ReadLine:0/3 ReadRune:0/3 ReadSlice:1/2 ReadString:1/2 Reset:1/0 Size:0/1 UnreadByte:0/1 UnreadRune:0/1 WriteTo:1/2', + Writer: 'Available:0/1 AvailableBuffer:0/1 Buffered:0/1 Flush:0/1 ReadFrom:1/2 Reset:1/0 Size:0/1 Write:1/2 WriteByte:1/1 WriteRune:1/2 WriteString:1/2', + ReadWriter: 'Available:0/1 AvailableBuffer:0/1 Buffered:0/1 Discard:1/2 Flush:0/1 Peek:1/2 Read:1/2 ReadByte:0/2 ReadBytes:1/2 ReadFrom:1/2 ReadLine:0/3 ReadRune:0/3 ReadSlice:1/2 ReadString:1/2 Reset:1/0 Size:0/1 UnreadByte:0/1 UnreadRune:0/1 Write:1/2 WriteByte:1/1 WriteRune:1/2 WriteString:1/2 WriteTo:1/2', + }, + bytes: { + Buffer: 'Available:0/1 AvailableBuffer:0/1 Bytes:0/1 Cap:0/1 Grow:1/0 Len:0/1 Next:1/1 Peek:1/2 Read:1/2 ReadByte:0/2 ReadBytes:1/2 ReadFrom:1/2 ReadRune:0/3 ReadString:1/2 Reset:0/0 String:0/1 Truncate:1/0 UnreadByte:0/1 UnreadRune:0/1 Write:1/2 WriteByte:1/1 WriteRune:1/2 WriteString:1/2 WriteTo:1/2', + Reader: 'Len:0/1 Read:1/2 ReadAt:2/2 ReadByte:0/2 ReadRune:0/3 Reset:1/0 Seek:2/2 Size:0/1 UnreadByte:0/1 UnreadRune:0/1 WriteTo:1/2', + }, + strings: { + Reader: 'Len:0/1 Read:1/2 ReadAt:2/2 ReadByte:0/2 ReadRune:0/3 Reset:1/0 Seek:2/2 Size:0/1 UnreadByte:0/1 UnreadRune:0/1 WriteTo:1/2', + Builder: 'Cap:0/1 Grow:1/0 Len:0/1 Reset:0/0 String:0/1 Write:1/2 WriteByte:1/1 WriteRune:1/2 WriteString:1/2', + }, + testing: { + TB: 'ArtifactDir:0/1 Attr:2/0 Chdir:1/0 Cleanup:1/0 Context:0/1 Error:1/0 Errorf:2/0 Fail:0/0 FailNow:0/0 Failed:0/1 Fatal:1/0 Fatalf:2/0 Helper:0/0 Log:1/0 Logf:2/0 Name:0/1 Output:0/1 Setenv:2/0 Skip:1/0 SkipNow:0/0 Skipf:2/0 Skipped:0/1 TempDir:0/1 private:0/0', + T: 'ArtifactDir:0/1 Attr:2/0 Chdir:1/0 Cleanup:1/0 Context:0/1 Deadline:0/2 Error:1/0 Errorf:2/0 Fail:0/0 FailNow:0/0 Failed:0/1 Fatal:1/0 Fatalf:2/0 Helper:0/0 Log:1/0 Logf:2/0 Name:0/1 Output:0/1 Parallel:0/0 Run:2/1 Setenv:2/0 Skip:1/0 SkipNow:0/0 Skipf:2/0 Skipped:0/1 TempDir:0/1 private:0/0', + B: 'ArtifactDir:0/1 Attr:2/0 Chdir:1/0 Cleanup:1/0 Context:0/1 Elapsed:0/1 Error:1/0 Errorf:2/0 Fail:0/0 FailNow:0/0 Failed:0/1 Fatal:1/0 Fatalf:2/0 Helper:0/0 Log:1/0 Logf:2/0 Loop:0/1 Name:0/1 Output:0/1 ReportAllocs:0/0 ReportMetric:2/0 ResetTimer:0/0 Run:2/1 RunParallel:1/0 SetBytes:1/0 SetParallelism:1/0 Setenv:2/0 Skip:1/0 SkipNow:0/0 Skipf:2/0 Skipped:0/1 StartTimer:0/0 StopTimer:0/0 TempDir:0/1 private:0/0', + F: 'Add:1/0 ArtifactDir:0/1 Attr:2/0 Chdir:1/0 Cleanup:1/0 Context:0/1 Error:1/0 Errorf:2/0 Fail:0/0 FailNow:0/0 Failed:0/1 Fatal:1/0 Fatalf:2/0 Fuzz:1/0 Helper:0/0 Log:1/0 Logf:2/0 Name:0/1 Output:0/1 Setenv:2/0 Skip:1/0 SkipNow:0/0 Skipf:2/0 Skipped:0/1 TempDir:0/1 private:0/0', + }, + 'log/slog': { + Handler: 'Enabled:2/1 Handle:2/1 WithAttrs:1/1 WithGroup:1/1', + JSONHandler: 'Enabled:2/1 Handle:2/1 WithAttrs:1/1 WithGroup:1/1', + TextHandler: 'Enabled:2/1 Handle:2/1 WithAttrs:1/1 WithGroup:1/1', + }, + encoding: { + BinaryMarshaler: 'MarshalBinary:0/2', + BinaryUnmarshaler: 'UnmarshalBinary:1/1', + TextMarshaler: 'MarshalText:0/2', + TextUnmarshaler: 'UnmarshalText:1/1', + }, + 'encoding/json': { + Marshaler: 'MarshalJSON:0/2', + Unmarshaler: 'UnmarshalJSON:1/1', + RawMessage: 'MarshalJSON:0/2 UnmarshalJSON:1/1', + }, + 'encoding/xml': { + Marshaler: 'MarshalXML:2/1', + Unmarshaler: 'UnmarshalXML:2/1', + }, + 'encoding/gob': { + GobEncoder: 'GobEncode:0/2', + GobDecoder: 'GobDecode:1/1', + }, + expvar: { + Var: 'String:0/1', + }, + hash: { + Hash: 'BlockSize:0/1 Reset:0/0 Size:0/1 Sum:1/1 Write:1/2', + Hash32: 'BlockSize:0/1 Reset:0/0 Size:0/1 Sum:1/1 Sum32:0/1 Write:1/2', + Hash64: 'BlockSize:0/1 Reset:0/0 Size:0/1 Sum:1/1 Sum64:0/1 Write:1/2', + }, + crypto: { + Signer: 'Public:0/1 Sign:3/2', + Decrypter: 'Decrypt:3/2 Public:0/1', + }, + 'crypto/elliptic': { + Curve: 'Add:4/2 Double:2/2 IsOnCurve:2/1 Params:0/1 ScalarBaseMult:1/2 ScalarMult:3/2', + }, + 'crypto/cipher': { + Block: 'BlockSize:0/1 Decrypt:2/0 Encrypt:2/0', + Stream: 'XORKeyStream:2/0', + AEAD: 'NonceSize:0/1 Open:4/2 Overhead:0/1 Seal:4/1', + BlockMode: 'BlockSize:0/1 CryptBlocks:2/0', + }, + 'crypto/tls': { + Conn: 'Close:0/1 CloseWrite:0/1 ConnectionState:0/1 Handshake:0/1 HandshakeContext:1/1 LocalAddr:0/1 NetConn:0/1 OCSPResponse:0/1 Read:1/2 RemoteAddr:0/1 SetDeadline:1/1 SetReadDeadline:1/1 SetWriteDeadline:1/1 VerifyHostname:1/1 Write:1/2', + }, + image: { + Image: 'At:2/1 Bounds:0/1 ColorModel:0/1', + PalettedImage: 'At:2/1 Bounds:0/1 ColorIndexAt:2/1 ColorModel:0/1', + }, + 'image/color': { + Color: 'RGBA:0/4', + Model: 'Convert:1/1', + }, + 'database/sql': { + Scanner: 'Scan:1/1', + }, + 'database/sql/driver': { + Valuer: 'Value:0/2', + Driver: 'Open:1/2', + Conn: 'Begin:0/2 Close:0/1 Prepare:1/2', + Stmt: 'Close:0/1 Exec:1/2 NumInput:0/1 Query:1/2', + Tx: 'Commit:0/1 Rollback:0/1', + Rows: 'Close:0/1 Columns:0/1 Next:1/1', + Result: 'LastInsertId:0/2 RowsAffected:0/2', + }, + flag: { + Value: 'Set:1/1 String:0/1', + Getter: 'Get:0/1 Set:1/1 String:0/1', + }, + runtime: { + Error: 'Error:0/1 RuntimeError:0/0', + }, + reflect: { + Type: 'Align:0/1 AssignableTo:1/1 Bits:0/1 CanSeq:0/1 CanSeq2:0/1 ChanDir:0/1 Comparable:0/1 ConvertibleTo:1/1 Elem:0/1 Field:1/1 FieldAlign:0/1 FieldByIndex:1/1 FieldByName:1/2 FieldByNameFunc:1/2 Fields:0/1 Implements:1/1 In:1/1 Ins:0/1 IsVariadic:0/1 Key:0/1 Kind:0/1 Len:0/1 Method:1/1 MethodByName:1/2 Methods:0/1 Name:0/1 NumField:0/1 NumIn:0/1 NumMethod:0/1 NumOut:0/1 Out:1/1 Outs:0/1 OverflowComplex:1/1 OverflowFloat:1/1 OverflowInt:1/1 OverflowUint:1/1 PkgPath:0/1 Size:0/1 String:0/1 common:0/1 uncommon:0/1', + }, + time: { + Time: 'Add:1/1 AddDate:3/1 After:1/1 AppendBinary:1/2 AppendFormat:2/1 AppendText:1/2 Before:1/1 Clock:0/3 Compare:1/1 Date:0/3 Day:0/1 Equal:1/1 Format:1/1 GoString:0/1 GobDecode:1/1 GobEncode:0/2 Hour:0/1 ISOWeek:0/2 In:1/1 IsDST:0/1 IsZero:0/1 Local:0/1 Location:0/1 MarshalBinary:0/2 MarshalJSON:0/2 MarshalText:0/2 Minute:0/1 Month:0/1 Nanosecond:0/1 Round:1/1 Second:0/1 String:0/1 Sub:1/1 Truncate:1/1 UTC:0/1 Unix:0/1 UnixMicro:0/1 UnixMilli:0/1 UnixNano:0/1 UnmarshalBinary:1/1 UnmarshalJSON:1/1 UnmarshalText:1/1 Weekday:0/1 Year:0/1 YearDay:0/1 Zone:0/2 ZoneBounds:0/2', + }, + 'compress/gzip': { + Reader: 'Close:0/1 Multistream:1/0 Read:1/2 Reset:1/1', + Writer: 'Close:0/1 Flush:0/1 Reset:1/0 Write:1/2', + }, +}; + +/** A standard-library type's methods: each name and its `params/results`. */ +export type GoStdlibMethods = ReadonlyMap; + +/** `error`, the one predeclared type with a method. */ +const GO_ERROR_METHODS: GoStdlibMethods = new Map([['Error', '0/1']]); + +const parsed = new Map(); + +/** + * The methods of the standard-library type `typeName` that the package at + * `importPath` declares, or of the predeclared `error` when `importPath` is + * null. Undefined for a type this table does not know. + */ +export function goStdlibMethods(importPath: string | null, typeName: string): GoStdlibMethods | undefined { + if (importPath === null) return typeName === 'error' ? GO_ERROR_METHODS : undefined; + const key = `${importPath}.${typeName}`; + let methods = parsed.get(key); + if (methods === undefined) { + const listed = Object.prototype.hasOwnProperty.call(GO_STDLIB_METHODS, importPath) + ? GO_STDLIB_METHODS[importPath]![typeName] + : undefined; + methods = typeof listed === 'string' + ? new Map(listed.split(' ').map((entry) => entry.split(':') as [string, string])) + : null; + parsed.set(key, methods); + } + return methods ?? undefined; +} diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index cb992fb721..215a06c2a3 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -2435,7 +2435,7 @@ export const GO_STDLIB_PACKAGES: ReadonlySet = new Set([ 'utilruntime', 'utilwait', 'utilnet', ]); -interface GoQualification { +export interface GoQualification { /** The package name written before the reference's name, as spelled. */ written?: string; /** The file's import that name is. */ @@ -2452,7 +2452,12 @@ interface GoQualification { const GO_QUALIFIERS = new WeakMap>(); -function goRefQualification(ref: UnresolvedRef, context: ResolutionContext): GoQualification { +/** + * The package a Go reference is written through, read from its line (the + * index keeps only the name): `io` in `io.Closer`, and the file's import of + * it. Empty for a name written bare. + */ +export function goRefQualification(ref: UnresolvedRef, context: ResolutionContext): GoQualification { if (ref.referenceKind === 'imports') return {}; const name = ref.referenceName.split('.').pop()!; if (!/^[A-Za-z_]\w*$/.test(name)) return {}; From 8b5d1d543dc7d28ca33287f9a72af5645f98e560 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 21 Oct 2026 18:27:03 +0000 Subject: [PATCH 04/95] chore(scripts): regenerate the Go standard-library method table from Go's source (#2760) go-stdlib-methods.ts lists the methods of 130 standard-library types with their parameter and result counts. Its generator lived in a validation kit. scripts/gen-go-stdlib-methods.mjs reads a Go source tree with the engine's Go grammar, counts with the engine's own goSignatureArity, and prints (or --write's) the table for the types it lists, so a later Go release's methods are one command away. Against go1.27.3 it reproduces the table byte for byte. Co-Authored-By: Claude Opus 5.5 --- scripts/gen-go-stdlib-methods.mjs | 188 ++++++++++++++++++++++++++++ src/resolution/go-stdlib-methods.ts | 11 +- 2 files changed, 194 insertions(+), 5 deletions(-) create mode 100644 scripts/gen-go-stdlib-methods.mjs diff --git a/scripts/gen-go-stdlib-methods.mjs b/scripts/gen-go-stdlib-methods.mjs new file mode 100644 index 0000000000..88f672b7a6 --- /dev/null +++ b/scripts/gen-go-stdlib-methods.mjs @@ -0,0 +1,188 @@ +#!/usr/bin/env node +/** + * Regenerate the table in src/resolution/go-stdlib-methods.ts from Go's own + * source: + * + * npm run build + * node scripts/gen-go-stdlib-methods.mjs [--write] + * + * is a Go source tree (a release's checkout of golang/go, say). For + * every type the table lists now — add or drop one by editing the table, then + * rerun — the script reads /src//*.go, tests skipped, + * with the engine's Go grammar, and lists each method as + * `name:params/results`: + * - an interface's methods, and those of the interfaces it embeds; + * - a struct's or other named type's (receiver T or *T), and those its + * embedded fields promote; + * - an alias's, its target's. + * The counts come from the engine's own goSignatureArity, so a table method + * is held to what the arity gate reads off a project method's stored + * signature. Exported names are kept, and an unexported one only where a + * listed interface requires it (`testing.TB`'s `private`). + * + * Prints the table body; --write puts it in the module instead. Exits 1 when + * a listed type isn't found or a signature doesn't read. + */ + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { createRequire } from 'node:module'; +import { fileURLToPath, pathToFileURL } from 'node:url'; + +const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); +const MODULE = path.join(ROOT, 'src', 'resolution', 'go-stdlib-methods.ts'); + +const goRoot = process.argv[2]; +const write = process.argv.includes('--write'); +if (!goRoot || !fs.existsSync(path.join(goRoot, 'src', 'io'))) { + console.error('usage: gen-go-stdlib-methods.mjs [--write] (a Go source tree, with src/io)'); + process.exit(2); +} +const wasm = path.join(ROOT, 'dist', 'extraction', 'wasm', 'tree-sitter-go.wasm'); +if (!fs.existsSync(wasm)) { + console.error('dist/ is missing: run `npm run build` first'); + process.exit(2); +} +const { goSignatureArity } = await import(pathToFileURL(path.join(ROOT, 'dist', 'resolution', 'callback-synthesizer.js')).href); +const { Parser, Language } = createRequire(path.join(ROOT, 'package.json'))('web-tree-sitter'); + +// The types the table lists, in its order. +const moduleText = fs.readFileSync(MODULE, 'utf8'); +const open = moduleText.indexOf('{', moduleText.indexOf('const GO_STDLIB_METHODS')); +const close = moduleText.indexOf('\n};', open); +const listed = new Function(`return ${moduleText.slice(open, close + 2)}`)(); + +await Parser.init(); +const parser = new Parser(); +parser.setLanguage(await Language.load(wasm)); + +const signature = (node) => { + const params = node.childForFieldName('parameters')?.text; + const result = node.childForFieldName('result')?.text; + return params && result ? `${params} ${result}` : params; +}; + +/** A package's top-level type declarations and the methods declared on each. */ +const packages = new Map(); +function pkg(importPath) { + let found = packages.get(importPath); + if (found) return found; + found = { types: new Map(), methods: new Map() }; + packages.set(importPath, found); + const dir = path.join(goRoot, 'src', importPath); + const files = fs.existsSync(dir) ? fs.readdirSync(dir).filter((f) => f.endsWith('.go') && !f.endsWith('_test.go')) : []; + for (const file of files) { + const tree = parser.parse(fs.readFileSync(path.join(dir, file), 'utf8')); + const imports = new Map(); + for (const spec of tree.rootNode.descendantsOfType('import_spec')) { + const ip = spec.childForFieldName('path')?.text.replace(/^"|"$/g, ''); + if (ip) imports.set(spec.childForFieldName('name')?.text ?? ip.split('/').pop(), ip); + } + for (const spec of tree.rootNode.descendantsOfType(['type_spec', 'type_alias'])) { + if (spec.parent?.type !== 'type_declaration' || spec.parent.parent?.type !== 'source_file') continue; + const name = spec.childForFieldName('name')?.text; + if (name && !found.types.has(name)) { + found.types.set(name, { type: spec.childForFieldName('type'), imports, alias: spec.type === 'type_alias' }); + } + } + for (const m of tree.rootNode.namedChildren.filter((c) => c.type === 'method_declaration')) { + const receiver = /\(\s*(?:\w+\s+)?\*?\s*(\w+)/.exec(m.childForFieldName('receiver')?.text ?? '')?.[1]; + const name = m.childForFieldName('name')?.text; + if (!receiver || !name) continue; + const methods = found.methods.get(receiver) ?? found.methods.set(receiver, new Map()).get(receiver); + if (!methods.has(name)) methods.set(name, goSignatureArity(signature(m))); + } + } + return found; +} + +/** Method name → `params/results` for the type `name` the package at `importPath` declares. */ +function methodsOf(importPath, name, seen = new Set()) { + const methods = new Map(); + const key = `${importPath}.${name}`; + if (seen.has(key)) return methods; + seen.add(key); + const p = pkg(importPath); + for (const [m, arity] of p.methods.get(name) ?? []) methods.set(m, arity); + const declared = p.types.get(name); + if (!declared) return methods; + const take = (more) => { + for (const [m, arity] of more) if (!methods.has(m)) methods.set(m, arity); + }; + const embedded = (type, imports) => { + if (type?.type === 'pointer_type') type = type.namedChildren[0]; + if (type?.type === 'generic_type') type = type.childForFieldName('type'); + if (type?.type === 'qualified_type') { + take(methodsOf(imports.get(type.childForFieldName('package').text), type.childForFieldName('name').text, seen)); + } else if (type?.type === 'type_identifier') { + if (type.text === 'error') take(new Map([['Error', '0/1']])); + else take(methodsOf(importPath, type.text, seen)); + } + }; + const type = declared.type; + if (type?.type === 'interface_type') { + for (const element of type.namedChildren) { + if (element.type === 'method_elem') methods.set(element.childForFieldName('name').text, goSignatureArity(signature(element))); + else if (element.type === 'type_elem') { + const terms = element.namedChildren.filter((c) => c.type !== 'comment'); + if (terms.length === 1) embedded(terms[0], declared.imports); + } + } + } else if (type?.type === 'struct_type') { + const fields = type.namedChildren.find((c) => c.type === 'field_declaration_list'); + for (const field of fields?.namedChildren.filter((c) => c.type === 'field_declaration') ?? []) { + if (!field.namedChildren.some((c) => c.type === 'field_identifier')) embedded(field.childForFieldName('type'), declared.imports); + } + } else if (declared.alias) { + embedded(type, declared.imports); + } + return methods; +} + +const EXPORTED = /^\p{Lu}/u; +const generated = []; +const required = new Set(); +let failed = false; +for (const [importPath, types] of Object.entries(listed)) { + for (const name of Object.keys(types)) { + const declared = pkg(importPath).types.get(name); + if (!declared) { + console.error(`not found: ${importPath}.${name}`); + failed = true; + continue; + } + const methods = methodsOf(importPath, name); + generated.push({ importPath, name, methods }); + if (declared.type?.type === 'interface_type') for (const m of methods.keys()) if (!EXPORTED.test(m)) required.add(m); + } +} + +const lines = []; +let current = null; +for (const { importPath, name, methods } of generated) { + if (importPath !== current) { + if (current !== null) lines.push(' },'); + lines.push(` ${/^[a-z]+$/.test(importPath) ? importPath : `'${importPath}'`}: {`); + current = importPath; + } + const kept = [...methods].filter(([m]) => EXPORTED.test(m) || required.has(m)).sort(([a], [b]) => (a < b ? -1 : 1)); + for (const [m, arity] of kept) { + if (arity === null) { + console.error(`signature doesn't read: ${importPath}.${name}.${m}`); + failed = true; + } + } + lines.push(` ${name}: '${kept.map(([m, arity]) => `${m}:${arity}`).join(' ')}',`); +} +lines.push(' },'); +if (failed) process.exit(1); + +if (!write) { + console.log(lines.join('\n')); +} else { + const eol = moduleText.includes('\r\n') ? '\r\n' : '\n'; + const start = moduleText.indexOf('\n', open) + 1; + const end = close + 1; + fs.writeFileSync(MODULE, moduleText.slice(0, start) + lines.join(eol) + eol + moduleText.slice(end)); + console.error(`wrote ${generated.length} types to ${path.relative(ROOT, MODULE)}`); +} diff --git a/src/resolution/go-stdlib-methods.ts b/src/resolution/go-stdlib-methods.ts index a2c85387ad..8e9f10e530 100644 --- a/src/resolution/go-stdlib-methods.ts +++ b/src/resolution/go-stdlib-methods.ts @@ -14,11 +14,12 @@ * `testing.TB`'s `private` and `reflect.Type`'s `common` and `uncommon` are * why only an embedding satisfies those outside their own package. * - * Read from Go's own source (go1.27.3). The interfaces are the ones projects - * embed; the structs are the commonly embedded ones that implement one of - * them. A struct whose methods would answer one of these interfaces' names - * with another signature (`http.Header.Write`, `sync.WaitGroup.Done`) is left - * out. + * Read from Go's own source (go1.27.3) by scripts/gen-go-stdlib-methods.mjs, + * which regenerates the table for the types it lists. The interfaces are the + * ones projects embed; the structs are the commonly embedded ones that + * implement one of them. A struct whose methods would answer one of these + * interfaces' names with another signature (`http.Header.Write`, + * `sync.WaitGroup.Done`) is left out. */ const GO_STDLIB_METHODS: Readonly>>> = { io: { From dbb6aa9ba9679d61d269444b376be63524970ed3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 21 Oct 2026 19:46:36 +0000 Subject: [PATCH 05/95] fix(vbnet): link reads through a namespace or a bare member name, and run the parameterless overload (#2762) Three kinds of VB.NET member read that #2355 and #2377 left unlinked: - A method named without parentheses (`Me.Load`, `Calc.GetTargetDAR`, a constructor's `Me.New` / `MyBase.New`) went to the first overload declared, so `Me.Load` reached `Load(path)` instead of `Load(Optional force = False)`, and a `Me.New` whose first overload was the calling constructor itself linked nothing. - A read whose first name is a namespace (`Billing.Invoice.MaxLines`, `StaxRip.UI.ButtonEx.ButtonSymbol.Delete` with the root namespace in front, a Module's `RedGifs.RedGifsSiteKey`) linked nothing: the receiver was looked up as a value or a type only. - A field, property or Module variable named bare before the dot (`Button1.Text`, `Settings.Plugins`, staxrip's global `p.TargetWidth`) was not counted as read: a bare name is no member access, so only what is read through it could link, and nothing when that is .NET's (`Text`). Fix (src/resolution/vbnet-receivers.ts; resolution only, extraction output unchanged): - A method a reference runs without parentheses resolves among its type's overloads to one that takes no arguments (none, or only Optional and ParamArray parameters, read from the declaration), never the member the reference is in. AddressOf / NameOf keep the first overload, as before. - A receiver name that is neither a value nor a type is looked up as a namespace the way VB.NET does: nested in a namespace around the read, outward to the root namespaces, then in the file's and project's Imports. Each further link is a type declared in it, a nested namespace, or a member of one of its Modules. The read links the type (`Billing.Invoice`) or the member (`Assets.Logo`), as #2355's `Outer.Mode` reads do. - A bare name before the dot that VB.NET binds to a field, property or Module variable (not a local or parameter, which hide it) is linked as a read alongside what is read through it. Not for a `With` block's `.Text`, a method run through the name (the call is the method's alone, as with parentheses), another project's Module, or a member declared with an outside type of its own name (`Responser As Responser`), whose Shared members and nested types VB.NET reads through the type; a project type of its own name follows #2377's Color Color rule. - membersNamed looked a type's members up in every project's same-named type and memoised the list by qualified name alone, so SCrawler's two plugin `UserData` classes shared one list ordered by whichever was asked first. It now keeps to the type's own project. Alone this leaves main's graph unchanged on SCrawler and staxrip; without it, 17 of the new bare-name reads in one plugin linked the other plugin's members. Dropped from the 2026-10-04 test set: a method named through its class also linking the class (#2355 treats it as a call, and its test pins that); `AppSession.SessionId` through a property typed OtherSession linking nothing (#2377 links OtherSession's field, pinned the same way); `SCrawler.My.Resources...` reads (#2355/#2377 send no path that starts with `My` or `Global`); and same-named classes picked by namespace, Imports and project, which main already does once the fixture declares its RootNamespace. Verification: new vbnet-member-access-scope.test.ts (9 tests) fails 6/9 on main and passes; a mutation that restores main's membersNamed fails the own-project test. The other VB.NET, C#, extraction, resolution, sync and framework suites pass; tsc and build clean. Validation, main vs this branch (indexes from each arm's own build; nodes unchanged; self-loops unchanged; serilog and gin dumps byte-identical): - SCrawler 31,201 -> 36,378 edges. 1 removed: `COLOR_PICKER.ColorsGet` re-pointed from the two-argument overload to `ColorsGet() As DataColor`. Added: 4,973 bare-name reads (3,731 own class, 957 Module variables, 285 inherited), 192 namespace reads, 12 reads through a Module member reached by namespace, and that 1 overload. - staxrip 70,864 -> 83,909 edges, 0 removed. Added: 12,983 bare-name reads (8,991 own class, 3,820 Module variables, 172 inherited), 61 reads with the root namespace in front, and `box.Show` reaching `Function Show() As DialogResult` from its Shared overload. - Checked against source: every bare-name read's name starts the access, has no local of that name, and binds to its class, an ancestor (23 through a qualified `Inherits Instagram.UserData` the graph's own extends edge misses) or a Module of its project; every namespace read's path ends its target's full name and the target's project is the reader's or one it references; no edge crosses between staxrip and its AutoCrop tool; 48 sampled edges hand-checked, all correct. - Cost (5 interleaved rounds, medians, quiet machine): SCrawler wall 5.60 -> 5.75 s, CPU 19.5 -> 20.3 s, DB 43.4 -> 45.1 MB; staxrip wall 7.23 -> 7.11 s (noise), CPU 25.7 -> 26.3 s, DB 59.0 -> 64.4 MB, the growth being the new edges (unresolved refs fall 47,074 -> 43,607 and 38,051 -> 36,165). Not covered: a call through a bare name (`Button1.Focus()`) or a function named before the dot, reads in a `With Button1` block, and `My.` / `Global.` paths. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/vbnet-member-access-scope.test.ts | 483 ++++++++++++++++++++ src/resolution/vbnet-receivers.ts | 293 +++++++++++- 3 files changed, 757 insertions(+), 21 deletions(-) create mode 100644 __tests__/vbnet-member-access-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0619badc1a..dd39604439 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -69,6 +69,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Dart, what a top-level variable's or a field's initializer calls, creates and reads is now linked. In Riverpod apps, where most wiring lives in provider closures like `final repoProvider = Provider((ref) => Repository(ref.watch(dioProvider)));`, `callers`, impact and `codegraph_explore` now follow a provider to the code it builds instead of stopping at the provider. A `final` or `const` initializer belongs to its constant, and a `var`'s or a field's to its file or class. A call through a type's static constant, like `FutureProvider.autoDispose(…)`, now links to that constant instead of an unrelated method that shares its name. Re-index Dart projects after upgrading. - In Dart, a call further down a chain, like Riverpod's `FutureProvider.autoDispose.family(…)` or `events.map(mapper).transform(…)` on a stream, now links to the method of the type the chain before it produces, worked out from what each step is declared to return, and to nothing when that type can't be worked out. Before, it linked to whichever project method shared its name, so `X.autoDispose.family(…)` reached the same provider builder whatever `X` was, and a stream's `transform` reached an unrelated `transform` method. `super.` calls now reach the superclass's method, `context.read().add(…)` reaches the bloc's own `add`, and the members of an extension marked `@internal` are found too. Re-index Dart projects after upgrading. (#750) - In VB.NET, reading or setting a field or property through a variable, parameter, field, `Me` or `MyBase`, like `x.Normal = 3`, `Me._h.Title`, `.Value` inside a `With` block or `.Switch` in `New BoolParam With {…}`, now counts as a use of that member, and so do calls inside a `With` block, functions called without parentheses and reads through a class whose name starts in lower case. Before, `codegraph callers` on an instance field or property missed most of its uses. Only a member that the value's declared type has or inherits is linked, nothing when that type comes from outside your project, and a value named after its class, like `theme As Theme`, counts as a use of the class only when it reads a `Shared` member. Re-index VB.NET projects after upgrading. +- In VB.NET, a type or member read through a namespace, like `Billing.Invoice.MaxLines`, `StaxRip.UI.ButtonEx.ButtonSymbol.Delete` with the project's root namespace in front, or a Module's member read through its namespace like `RedGifs.RedGifsSiteKey`, now links to what it names, and a method called without parentheses, like `Me.Load` or a constructor's `Me.New`, now links to the overload that takes no arguments instead of the first one declared or nothing. The namespace is looked up as VB.NET does, so a namespace of the same name in another project is not mistaken for it. Thanks @serkanince for the report. (#2305) +- In VB.NET, a field, property or Module variable written by its bare name in front of a member, like `Button1` in `Button1.Text = "x"` or `Settings` in `Settings.Plugins.Count`, now counts as used by that code, so `codegraph callers` on a form's control or a shared settings object lists the code that reads through it. Nothing is linked for a local or parameter of that name, for the member of a same-named class in another project, or for a member declared with an outside type of its own name, like `Responser As Responser`, whose `Shared` members VB.NET reads through the type; a call through the name, like `Button1.Focus()`, still links only the method. Re-index VB.NET projects after upgrading. (#2305) - In JavaScript and TypeScript, an import whose name contains a `$`, like `import items$ from './store'`, `import * as api$ from './api'` or `import { load as load$ } from './data'`, now links the calls made through it, such as `items$.getState().inc()`, to the code it imports, and so does a `$` name in a `require(…)` destructuring or an `export { … } from` barrel. Before, such an import was skipped or bound under a cut-off name (`load` for `load$`), so those calls reached nothing and the functions they call could look unused. A local variable or parameter that reuses an imported `$` name, like `const items$ = …` inside a function, is no longer mistaken for the import. Re-index JavaScript and TypeScript projects after upgrading. - In Dart, calling a parameter or a local variable, like `create(ref, arg)` inside `overrideWith(… create)`, a closure's parameter, a loop variable or a test's mock `listener(0, 1)`, no longer links the call to an unrelated function or constant of the same name in another file, or to a method the class inherits: a Bloc event handler's `emit(…)` is the handler's own `emit` parameter, not `Bloc.emit`. So `codegraph callers` and impact no longer list code that never calls them, while calls the parameter or variable doesn't cover, and calls to a local function, link as before. Re-index Dart projects after upgrading. - In Dart, `const` constructors and redirecting factories, like flutter_bloc's `const BlocProvider.value(…)` or freezed's `const factory LoginState.loading() = _Loading;`, are now indexed, so a call such as `BlocProvider.value(…)` or `BlocProvider.value(…)` links to that constructor instead of another class's member with the same name. A call through a class name now reaches only that class's own constructors and static members, never a look-alike class, a parent class, an instance getter of the same name, or a project method that merely shares its name with an SDK call like `Uri.parse(…)`, and a constructor is no longer linked from a bare name, the end of a call chain or an annotation like `@internal`. Re-index Dart projects after upgrading. diff --git a/__tests__/vbnet-member-access-scope.test.ts b/__tests__/vbnet-member-access-scope.test.ts new file mode 100644 index 0000000000..b912f7ef1b --- /dev/null +++ b/__tests__/vbnet-member-access-scope.test.ts @@ -0,0 +1,483 @@ +/** + * VB.NET member reads that #2355 and #2377 left unlinked (#2305 follow-up): + * + * - A method named without parentheses runs: VB.NET calls the overload that + * takes no arguments (`Me.Load` is `Load(Optional force = False)`, not + * `Load(path)`), and a constructor's `Me.New` / `MyBase.New` is another + * constructor, never itself. + * - A name before the dot can be a namespace: `Billing.Invoice.MaxLines`, + * the project's root namespace in front (`Acme.Billing.Invoice.MaxLines`), + * and a Module's member through its namespace (`Assets.Logo`). The namespace + * is looked up as VB.NET does — in the namespaces around the read, outward to + * the root namespaces, then in the file's `Imports` — so another project's + * namespace of that name is not read. + * - A bare name before the dot that is a field or property of the class around + * the read (or of one it inherits, or a Module's variable) is a read of that + * member too: `Button1.Text` reads the form's own `Button1` — never another + * form's, nor a same-named class's in another project — although `Text` is + * .NET's. A parameter or local of that name hides it; a value named like its + * type (`Property Mode As Mode`) is read only when the member read through it + * is an instance one, and not at all when that type is from outside + * (`Responser As Responser`); a method run through the name is the call's + * alone. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const files: Record = { + 'App.vbproj': '\n \n Acme\n \n\n', + 'Plugin/Plugin.vbproj': '\n \n Plugin\n \n\n', + 'AppSession.vb': `Public Class AppSession + Public Shared Property CurrentUser As String +End Class +`, + 'OtherSession.vb': `Public Class OtherSession + Public Shared SessionId As Guid +End Class +`, + // Overloads of a method VB.NET runs without parentheses. + 'Chain.vb': `Public Class Chain + Public Sub New(ByVal name As String) + Me.New + Me.Load + Dim n = Me.Count + End Sub + + Public Sub New() + End Sub + + Public Sub Load(ByVal path As String) + End Sub + + Public Sub Load(Optional ByVal force As Boolean = False) + End Sub + + Public Function Count(ByVal items As Integer()) As Integer + Return 0 + End Function + + Public Function Count(ParamArray items As Integer()) As Integer + Return 0 + End Function +End Class + +Public Class LinkedChain + Inherits Chain + + Public Sub New() + MyBase.New + End Sub +End Class + +Public Class Calc + Public Shared Function GetTargetDAR(ByVal width As Integer) As Double + Return 0 + End Function + + Public Shared Function GetTargetDAR() As Double + Return 0 + End Function +End Class + +Public Class Measurer + Private Tracker As Chain + + Public Function Ratio(ByVal c As Chain) As Double + Dim n = c.Count + Return Calc.GetTargetDAR + End Function + + Public Sub Track() + Tracker.Load + End Sub +End Class +`, + // Types and a Module in namespaces of the Acme project. + 'Billing/Invoice.vb': `Namespace Billing + Public Class Invoice + Public Const MaxLines As Integer = 100 + Public Shared Function Create() As Invoice + Return Nothing + End Function + End Class + + Public Enum Status + Open + Paid + End Enum +End Namespace +`, + 'Assets/Images.vb': `Namespace Assets + Public Module Images + Public Logo As String = "logo.png" + Public Function Pick() As String + Return Logo + End Function + End Module +End Namespace +`, + 'Forms/MainForm.Designer.vb': ` +Partial Class MainForm + Inherits System.Windows.Forms.Form + + Private Sub InitializeComponent() + Me.Button1 = New System.Windows.Forms.Button() + Me.Button1.Size = New System.Drawing.Size(75, 23) + Me.Size = New System.Drawing.Size(300, 200) + End Sub + + Friend WithEvents Button1 As System.Windows.Forms.Button +End Class +`, + 'Forms/MainForm.vb': `Public Class MainForm + Private Clicks As Integer + + Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click + Me.Clicks = Me.Clicks + 1 + Button1.Text = AppSession.CurrentUser + With Button1 + .Enabled = False + End With + Me.Text = Console.Out.ToString() + Dim n = Billing.Invoice.MaxLines + Dim m = Acme.Billing.Invoice.MaxLines + Dim s = Billing.Status.Paid + Dim l = Assets.Logo + Dim p = Assets.Pick + Dim i = Billing.Invoice.Create + Dim r = System.Drawing.Color.Red + End Sub +End Class +`, + 'Forms/OtherForm.Designer.vb': `Partial Class OtherForm + Inherits System.Windows.Forms.Form + + Friend WithEvents Button1 As System.Windows.Forms.Button +End Class +`, + 'Forms/OtherForm.vb': `Public Class OtherForm + Private Sub Reload() + Button1.Text = OtherSession.SessionId.ToString() + End Sub +End Class +`, + // The Plugin project sees no \`Billing\` namespace: Acme's is another project's. + 'Plugin/Report.vb': `Public Class Report + Public Sub Run() + Dim n = Billing.Invoice.MaxLines + End Sub +End Class +`, + 'Model.vb': `Public Enum Mode + Fast + Slow +End Enum + +Public Class Settings + Public Property Level As Integer +End Class + +Public Class Widget + Public Property Length As Integer +End Class +`, + 'Globals.vb': `Public Module Globals + Public Current As Settings +End Module +`, + // What the name before the dot binds to: a parameter or local, then a member + // of the class around it (or one it inherits, or a Module's), then a type. + 'Scopes.vb': `Public Class Reporter + Public Sub Report(ByVal Mode As String) + Dim Widget = "x" + Dim n = Mode.Length + Widget.Length + End Sub +End Class + +Public Class SessionView + Private ReadOnly Property AppSession As OtherSession + + Public Sub Show() + Dim id = AppSession.SessionId + End Sub +End Class + +Public Class SettingsUser + Public Property Settings As Settings + Public Property Mode As Mode + + Public Sub Apply() + Dim t = Settings.Level + Dim f = Mode.Fast + End Sub +End Class + +Public Class BaseView + Protected Title As String +End Class + +Public Class DerivedView + Inherits BaseView + + Public Sub Render() + Dim n = Title.Length + End Sub +End Class + +Public Class Viewer + Private Button1 As Settings + + Public Sub Show() + Dim lv = Current.Level + End Sub + + Public Sub Hide() + Dim Button1 As New Settings() + Dim x = Button1.Level + End Sub +End Class + +Public Class Detector + Public Class Data + Public Shared Empty As Integer + End Class + + Public Function Probe() As Integer + Return Data.Empty + End Function +End Class + +Public Class Failure + Inherits System.Exception + + Public Function Describe() As Integer + Return Data.Empty + End Function +End Class +`, + // CRLF line endings, and multibyte text — naming the field — before the read. + 'Weird.vb': [ + 'Public Class Weird', + ' Private Label1 As Settings', + ' Public Sub Go()', + ' Dim s = "Label1 éééé 😀" & Label1.Level & Billing.Invoice.MaxLines', + ' End Sub', + 'End Class', + '', + ].join('\r\n'), + // One class name in two namespaces and in another project: the file's + // \`Imports\` says which \`UserData\` it means. A type in the file's own + // namespace (the root one) comes before an imported one: \`Helper.Name\`. + 'Plugin/UserData.vb': `Public Class UserData + Public Shared ChannelsLabel As String + Private Label As String + + Public Function Size() As Integer + Return Label.Length + End Function +End Class +`, + // Another plugin project declares a \`UserData\` of its own: its members are not Plugin's. + 'Plugin2/Plugin2.vbproj': '\n \n Plugin2\n \n\n', + 'Plugin2/UserData.vb': `Public Class UserData + Private Label As String + + Public Function Size() As Integer + Return Label.Length + End Function +End Class +`, + // A member declared As an outside type of its own name: VB.NET reads that + // type's Shared members and nested types through the type ("Color Color"). + 'Client.vb': `Public Class Client + Protected Responser As Responser + + Public Function Code() As Integer + Dim m = Responser.Modes.Curl + Return Responser.StatusCode + End Function +End Class +`, + 'Helper.vb': `Public Class Helper + Public Shared Name As String +End Class +`, + 'Lib/Helper.vb': `Namespace Lib + Public Class Helper + Public Shared Name As String + End Class +End Namespace +`, + 'Reddit/UserData.vb': `Namespace API.Reddit + Public Class UserData + Public Const ChannelsLabel As String = "Channels" + End Class +End Namespace +`, + 'Twitter/UserData.vb': `Namespace API.Twitter + Public Class UserData + Public Const ChannelsLabel As String = "Lists" + End Class +End Namespace +`, + 'ChannelView.vb': `Imports Acme.API.Reddit +Imports Acme.Lib + +Public Class ChannelView + Public Sub Add() + Dim l = UserData.ChannelsLabel + Dim h = Helper.Name + End Sub +End Class +`, +}; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-scope-')); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** The node named `qualifiedName` — at `line`, or in `file`, when the name alone leaves several. */ +function nodeId(qualifiedName: string, at?: number | string): string { + const found = cg.getNodesInFiles(Object.keys(files)).filter((n) => n.qualifiedName === qualifiedName && + (at === undefined || (typeof at === 'number' ? n.startLine === at : n.filePath === at))); + if (found.length !== 1) throw new Error(`${found.length} nodes named ${qualifiedName}${at === undefined ? '' : ` at ${at}`}`); + return found[0]!.id; +} + +/** What links to `qualifiedName` with an edge of `kind`, one per edge: `file:qualifiedName` when `withFiles`. */ +function linkedFrom(qualifiedName: string, kind: 'references' | 'calls', at?: number | string, withFiles = false): string[] { + return cg + .getIncomingEdgesTo([nodeId(qualifiedName, at)], [kind]) + .map((e) => cg.getNode(e.source)!) + .map((n) => (withFiles ? `${n.filePath}:${n.qualifiedName}` : n.qualifiedName)) + .sort(); +} + +describe('VB.NET member reads through a namespace, a member of the class around them, or a method without parentheses (#2305)', () => { + it('calls the overload that takes no arguments, and a constructor never itself', () => { + // Chain.vb: New(name) at 2, New() at 8, Load(path) at 11, Load(Optional) at 14, + // Count(items) at 17, Count(ParamArray) at 21. + expect(linkedFrom('Chain::New', 'calls', 8)).toEqual(['Chain::New', 'LinkedChain::New']); + expect(linkedFrom('Chain::New', 'calls', 2)).toEqual([]); + // `Me.Load`, and `Tracker.Load` through a field, whose read is the call's alone. + expect(linkedFrom('Chain::Load', 'calls', 14)).toEqual(['Chain::New', 'Measurer::Track']); + expect(linkedFrom('Chain::Load', 'calls', 11)).toEqual([]); + expect(linkedFrom('Measurer::Tracker', 'calls')).toEqual([]); + expect(linkedFrom('Chain::Count', 'calls', 21)).toEqual(['Chain::New', 'Measurer::Ratio']); + expect(linkedFrom('Chain::Count', 'calls', 17)).toEqual([]); + // Calc's GetTargetDAR(width) at 35, GetTargetDAR() at 39. + expect(linkedFrom('Calc::GetTargetDAR', 'calls', 39)).toEqual(['Measurer::Ratio']); + expect(linkedFrom('Calc::GetTargetDAR', 'calls', 35)).toEqual([]); + }); + + it('reads a type, a constant, an Enum case and a Module member through their namespace, the root namespace in front or not', () => { + // `Billing.Invoice.MaxLines` and `Acme.Billing.Invoice.MaxLines`. + expect(linkedFrom('Billing::Invoice::MaxLines', 'references')) + .toEqual(['MainForm::Button1_Click', 'MainForm::Button1_Click', 'Weird::Go']); + // The type each of them, and `Billing.Invoice.Create`, names. + expect(linkedFrom('Billing::Invoice', 'references')) + .toEqual(['MainForm::Button1_Click', 'MainForm::Button1_Click', 'MainForm::Button1_Click', 'Weird::Go']); + expect(linkedFrom('Billing::Invoice::Create', 'calls')).toEqual(['MainForm::Button1_Click']); + expect(linkedFrom('Billing::Status::Paid', 'references')).toEqual(['MainForm::Button1_Click']); + expect(linkedFrom('Billing::Status', 'references')).toEqual(['MainForm::Button1_Click']); + // A Module's members are its namespace's; the Module itself is not written. + expect(linkedFrom('Assets::Images::Logo', 'references')).toEqual(['MainForm::Button1_Click']); + expect(linkedFrom('Assets::Images::Pick', 'calls')).toEqual(['MainForm::Button1_Click']); + expect(linkedFrom('Assets::Images', 'references')).toEqual([]); + }); + + it('reads no namespace another project declares', () => { + for (const qn of ['Billing::Invoice::MaxLines', 'Billing::Invoice']) { + expect(linkedFrom(qn, 'references')).not.toContain('Report::Run'); + } + }); + + it('reads a field or property of the class around the read through its bare name', () => { + // `Button1.Text` in Button1_Click and the designer's two `Me.Button1`; the + // `With Button1` block's `.Enabled` is no second read of it. + expect(linkedFrom('MainForm::Button1', 'references')) + .toEqual(['MainForm::Button1_Click', 'MainForm::InitializeComponent', 'MainForm::InitializeComponent']); + expect(linkedFrom('OtherForm::Button1', 'references')).toEqual(['OtherForm::Reload']); + expect(linkedFrom('MainForm::Clicks', 'references')).toEqual(['MainForm::Button1_Click', 'MainForm::Button1_Click']); + // An inherited field, a Module's variable, and a field read after multibyte text. + expect(linkedFrom('BaseView::Title', 'references')).toEqual(['DerivedView::Render']); + expect(linkedFrom('Globals::Current', 'references')).toEqual(['Viewer::Show']); + expect(linkedFrom('Weird::Label1', 'references')).toEqual(['Weird::Go']); + expect(linkedFrom('Settings::Level', 'references')).toEqual(['SettingsUser::Apply', 'Viewer::Hide', 'Viewer::Show', 'Weird::Go']); + }); + + it('binds the name before the dot as VB does: parameter or local, own member, then type', () => { + // `Mode` is a parameter and `Widget` a local of Report; `Button1` a local of Hide. + expect(linkedFrom('Mode', 'references')).toEqual(['SettingsUser::Apply']); + expect(linkedFrom('Widget', 'references')).toEqual([]); + expect(linkedFrom('Widget::Length', 'references')).toEqual([]); + expect(linkedFrom('Viewer::Button1', 'references')).toEqual([]); + // SessionView's own `AppSession` property holds an OtherSession. + expect(linkedFrom('SessionView::AppSession', 'references')).toEqual(['SessionView::Show']); + expect(linkedFrom('OtherSession::SessionId', 'references')).toEqual(['OtherForm::Reload', 'SessionView::Show']); + // "Color Color": `Settings.Level` reads the property, and an instance + // member of its type; `Mode.Fast` reads an Enum case through the type. + expect(linkedFrom('SettingsUser::Settings', 'references')).toEqual(['SettingsUser::Apply']); + expect(linkedFrom('SettingsUser::Mode', 'references')).toEqual([]); + expect(linkedFrom('Mode::Fast', 'references')).toEqual(['SettingsUser::Apply']); + // A type nested in a class is a bare name only inside it: Failure's `Data` is Exception's. + expect(linkedFrom('Detector::Data::Empty', 'references')).toEqual(['Detector::Probe']); + expect(linkedFrom('Detector::Data', 'references')).toEqual(['Detector::Probe']); + // `Responser As Responser`, a type from outside: `Responser.Modes` may name + // the type's nested enum, which nothing here can tell, so no read is linked. + expect(linkedFrom('Client::Responser', 'references')).toEqual([]); + }); + + it("reads a member of the class's own project, not of a same-named class in another", () => { + expect(linkedFrom('UserData::Label', 'references', 'Plugin/UserData.vb', true)).toEqual(['Plugin/UserData.vb:UserData::Size']); + expect(linkedFrom('UserData::Label', 'references', 'Plugin2/UserData.vb', true)).toEqual(['Plugin2/UserData.vb:UserData::Size']); + }); + + it('picks among same-named classes the one VB finds first, for the member and the class', () => { + expect(linkedFrom('API.Reddit::UserData::ChannelsLabel', 'references')).toEqual(['ChannelView::Add']); + expect(linkedFrom('API.Reddit::UserData', 'references')).toEqual(['ChannelView::Add']); + expect(linkedFrom('API.Twitter::UserData::ChannelsLabel', 'references')).toEqual([]); + expect(linkedFrom('UserData::ChannelsLabel', 'references')).toEqual([]); + expect(linkedFrom('Helper::Name', 'references')).toEqual(['ChannelView::Add']); + expect(linkedFrom('Lib::Helper::Name', 'references')).toEqual([]); + }); + + it('does not bind a member of another class that only shares the name', () => { + // `Me.Size` / `Me.Text` on a Form, `Console.Out`, `System.Drawing.Color.Red`. + expect(linkedFrom('MainForm::Button1', 'references')).not.toContain('OtherForm::Reload'); + expect(linkedFrom('OtherForm::Button1', 'references')).not.toContain('MainForm::Button1_Click'); + expect(linkedFrom('Widget::Length', 'references')).toEqual([]); + }); + + it('keeps the edges through an incremental sync', async () => { + for (const rel of ['Forms/MainForm.vb', 'Billing/Invoice.vb', 'Chain.vb', 'Scopes.vb']) { + fs.writeFileSync(path.join(root, rel), `' touched\n${files[rel]}`); + } + await cg.sync(); + expect(linkedFrom('Billing::Invoice::MaxLines', 'references')) + .toEqual(['MainForm::Button1_Click', 'MainForm::Button1_Click', 'Weird::Go']); + expect(linkedFrom('Assets::Images::Logo', 'references')).toEqual(['MainForm::Button1_Click']); + expect(linkedFrom('MainForm::Button1', 'references')) + .toEqual(['MainForm::Button1_Click', 'MainForm::InitializeComponent', 'MainForm::InitializeComponent']); + expect(linkedFrom('SessionView::AppSession', 'references')).toEqual(['SessionView::Show']); + // The touched Chain.vb moved down a line. + expect(linkedFrom('Chain::Load', 'calls', 15)).toEqual(['Chain::New', 'Measurer::Track']); + expect(linkedFrom('Chain::New', 'calls', 9)).toEqual(['Chain::New', 'LinkedChain::New']); + }); +}); diff --git a/src/resolution/vbnet-receivers.ts b/src/resolution/vbnet-receivers.ts index b78704a85a..884f1530d9 100644 --- a/src/resolution/vbnet-receivers.ts +++ b/src/resolution/vbnet-receivers.ts @@ -466,21 +466,27 @@ const VB_MEMBERS = new WeakMap>(); * Members named `name` declared directly in `type` (any partial part of it), * case aside. A member is declared inside a part of its type, so it is looked * up in those parts' files: one lookup per type name, not one per member name. + * The parts are in the type's own project: SCrawler's plugins each declare a + * `UserData`, whose members are not one another's. */ function membersNamed(type: Node, name: string, context: ResolutionContext): Node[] { let memo = VB_MEMBERS.get(context); if (!memo) VB_MEMBERS.set(context, (memo = new Map())); + const project = projectOf(type.filePath, context); const qn = `${type.qualifiedName}::${name}`.toLowerCase(); - const hit = memo.get(qn); + const key = `${project}|${qn}`; + const hit = memo.get(key); if (hit) return hit; const own = type.qualifiedName.toLowerCase(); const files = new Set([type.filePath]); for (const part of context.getNodesByLowerName(type.name.toLowerCase())) { - if (part.language === 'vbnet' && part.qualifiedName.toLowerCase() === own) files.add(part.filePath); + if (part.language === 'vbnet' && part.qualifiedName.toLowerCase() === own && projectOf(part.filePath, context) === project) { + files.add(part.filePath); + } } const members = [...files].flatMap((file) => nodesByQualifiedName(file, context).get(qn) ?? []); if (memo.size >= 65536) memo.delete(memo.keys().next().value!); - memo.set(qn, members); + memo.set(key, members); return members; } @@ -848,6 +854,83 @@ function typesNamedAt( return { owners: [...tier.filter((c) => c.filePath === file), ...tier.filter((c) => c.filePath !== file)], ambiguous: distinct.size > 1 }; } +const VB_NAMESPACES = new WeakMap>(); + +/** + * Every namespace the project's VB.NET code is declared in, by its full name + * (lowercased, dot-joined, root namespace first), and each namespace around + * one: `acme`, `acme.billing`, `scrawler.api.reddit`. + */ +function knownNamespaces(context: ResolutionContext): Set { + const hit = VB_NAMESPACES.get(context); + if (hit) return hit; + const known = new Set(); + const add = (segments: string[]) => { + for (let i = 1; i <= segments.length; i++) known.add(segments.slice(0, i).join('.')); + }; + for (const n of context.getNodesByKind('namespace')) if (n.language === 'vbnet') add(fullSegments(n, context)); + for (const file of context.getAllFiles()) if (/\.vb$/i.test(file)) add(projectInfo(file, context).root); + VB_NAMESPACES.set(context, known); + return known; +} + +/** The namespace a line of a file is in, as full segments: its project's root namespace, then the `Namespace` block around it. */ +function namespaceAround(file: string, line: number, context: ResolutionContext): string[] { + const root = projectInfo(file, context).root; + const around = typesAround(file, line, context); + const outer = around[around.length - 1]; + const cut = outer ? outer.qualifiedName.indexOf('::') : -1; + if (cut < 0) return root; + const head = outer!.qualifiedName.slice(0, cut).toLowerCase(); + return nodesByQualifiedName(file, context).get(head)?.some((n) => n.kind === 'namespace') ? [...root, ...head.split('.')] : root; +} + +/** + * The namespace a name written first in a path means where it is written, + * as full segments — as VB.NET looks one up: nested in a namespace around the + * read, the innermost first and the root namespaces last, else in one the + * file or its project imports. Null when no namespace of that name is there: + * `Billing` is Acme's `Billing` in Acme's code, and nothing in a project + * whose own namespaces don't have one. + */ +function namespaceNamed(name: string, ref: UnresolvedRef, context: ResolutionContext): string[] | null { + const known = knownNamespaces(context); + const key = name.toLowerCase(); + const around = namespaceAround(ref.filePath, ref.line, context); + for (let i = around.length; i >= 0; i--) { + const candidate = [...around.slice(0, i), key]; + if (known.has(candidate.join('.'))) return candidate; + } + for (const imported of importedNamespaces(ref.filePath, context)) { + const candidate = [...imported.split('.'), key]; + if (known.has(candidate.join('.'))) return candidate; + } + return null; +} + +/** + * The project types named `name` declared directly in namespace `ns` (full + * segments): the caller's project's, if it has one — SCrawler's projects + * share a root namespace — and null when that leaves two different types. + */ +function typesInNamespace(ns: string[], name: string, ref: UnresolvedRef, context: ResolutionContext): Node[] | null { + const key = ns.join('.'); + let found = projectTypesNamed(name, context, VB_READ_TYPE_KINDS) + .filter((c) => fullSegments(c, context).slice(0, -1).join('.') === key); + const own = found.filter((c) => sameVbProject(c.filePath, ref.filePath, context)); + if (own.length > 0) found = own; + const distinct = new Set(found.map((c) => `${projectOf(c.filePath, context)}|${c.qualifiedName.toLowerCase()}`)); + return distinct.size > 1 ? null + : [...found.filter((c) => c.filePath === ref.filePath), ...found.filter((c) => c.filePath !== ref.filePath)]; +} + +/** A member named `name` of a `Module` declared directly in namespace `ns`, which VB.NET reads through the namespace: `Assets.Logo`. */ +function moduleMemberIn(ns: string[], name: string, ref: UnresolvedRef, context: ResolutionContext): Node | undefined { + const key = ns.join('.'); + return moduleMembersNamed(name, ref, context).find((n) => + (VB_VALUE_KINDS.has(n.kind) || n.kind === 'method') && fullSegments(n, context).slice(0, -2).join('.') === key); +} + /** The call site's own file first, then its project's, the rest after, each in its given order. */ export function preferVbProject(nodes: Node[], ref: UnresolvedRef, context: ResolutionContext): Node[] { if (nodes.length < 2) return nodes; @@ -941,20 +1024,30 @@ function valueMemberType(member: Node, context: ResolutionContext, args: Readonl } /** - * The type of what a receiver-less `name` reads where a call is: a member of - * a class around the call or of one it inherits, else of a `Module`. Null when - * it declares no type the source shows, undefined when nothing has the name. + * The member a receiver-less `name` reads where a call is: a field, property + * or function of a class around the call or of one it inherits (with what + * that class's type parameters stand for there), else of a `Module`. */ -function memberType(name: string, ref: UnresolvedRef, context: ResolutionContext): VbType | null | undefined { +function memberAround(name: string, ref: UnresolvedRef, context: ResolutionContext): { node: Node; args: Map } | undefined { const fits = (n: Node) => VB_VALUE_KINDS.has(n.kind) || n.kind === 'method'; for (const around of typesAround(ref.filePath, ref.line, context)) { for (const { node, args } of ancestry(around, context)) { const member = membersNamed(node, name, context).find(fits); - if (member) return valueMemberType(member, context, args); + if (member) return { node: member, args }; } } const global = moduleMembersNamed(name, ref, context).find(fits); - return global ? valueMemberType(global, context) : undefined; + return global ? { node: global, args: new Map() } : undefined; +} + +/** + * The type of what a receiver-less `name` reads where a call is (see + * memberAround). Null when it declares no type the source shows, undefined + * when nothing has the name. + */ +function memberType(name: string, ref: UnresolvedRef, context: ResolutionContext): VbType | null | undefined { + const member = memberAround(name, ref, context); + return member ? valueMemberType(member.node, context, member.args) : undefined; } /** The type a method, function or property returns, read from its declaration; null for a `Sub` or none written. */ @@ -1378,6 +1471,11 @@ interface VbReceiver { throughType: boolean; /** The name is a value whose type has the same name (`theme As Theme`): VB.NET's "Color Color" rule. */ colorColor?: boolean; + /** + * The receiver names a namespace (`Billing`, `Acme.Billing`), whose types and + * Modules' members are read through it: its full segments. No owners. + */ + namespace?: string[]; } /** `Name` for an escaped `[Name]`. */ @@ -1398,7 +1496,8 @@ function valueReceiver(t: VbType | null | undefined, context: ResolutionContext) * inherits, `{T}` a value of `T`. A name holds a value of its declared type * when it is a local, a parameter, or a member of a type around the read or * of a Module; any other name means the class, module, structure, interface - * or enum it names there. A value whose type has its name (`Public Property + * or enum it names there, else a namespace (`Billing`, the root namespace + * `Acme`). A value whose type has its name (`Public Property * Settings As Settings`, `theme As Theme`) may mean either, by VB.NET's * "Color Color" rule: the member read through it decides (matchVbMemberRead). */ @@ -1419,17 +1518,29 @@ function receiverAt(head: string, ref: UnresolvedRef, context: ResolutionContext : bound && !bound.array && bound.name.toLowerCase() === name.toLowerCase() ? bound : null; if (!written) return valueReceiver(bound, context); const found = typesNamedAt(written, context, VB_READ_TYPE_KINDS); - return found.ambiguous || found.owners.length === 0 ? null - : { owners: found.owners, throughType: true, ...(bound ? { colorColor: true } : {}) }; + if (found.ambiguous) return null; + if (found.owners.length > 0) return { owners: found.owners, throughType: true, ...(bound ? { colorColor: true } : {}) }; + const namespace = bound === undefined ? namespaceNamed(name, ref, context) : null; + return namespace ? { owners: [], throughType: false, namespace } : null; } /** * What `name`, read through `at`, is as a receiver in turn: a value of the * type a field, property or function declares, or a type nested in a type - * named for it (`Outer.Mode`). Null where the member, or its type, is not - * known here. + * named for it (`Outer.Mode`); through a namespace, a type declared in it, a + * namespace nested in it, or the value one of its Modules' members holds. + * Null where the member, or its type, is not known here. */ function memberAt(at: VbReceiver, name: string, ref: UnresolvedRef, context: ResolutionContext): VbReceiver | null { + if (at.namespace) { + const types = typesInNamespace(at.namespace, name, ref, context); + if (types === null) return null; + if (types.length > 0) return { owners: types, throughType: true }; + const nested = [...at.namespace, name.toLowerCase()]; + if (knownNamespaces(context).has(nested.join('.'))) return { owners: [], throughType: false, namespace: nested }; + const member = moduleMemberIn(at.namespace, name, ref, context); + return member ? valueReceiver(valueMemberType(member, context), context) : null; + } if (at.owners.length === 0) return null; const fits = (n: Node) => VB_INSTANCE_READ_KINDS.has(n.kind) || (at.throughType && VB_READ_TYPE_KINDS.has(n.kind)); const found = memberOn(at.owners, name, ref, context, true, at.typed, fits) ?? @@ -1482,6 +1593,116 @@ function namesWithoutRunning(ref: UnresolvedRef, context: ResolutionContext): bo return /\b(?:AddressOf\s+|NameOf\s*\(\s*)$/i.test(line.slice(0, Math.max(0, ref.column))); } +const VB_NO_ARGUMENTS = new WeakMap>(); + +/** + * Whether VB.NET can call a method with no arguments: it declares no + * parameter, or only `Optional` ones and a `ParamArray`. Read from its + * declaration; true when that can't be read. + */ +function takesNoArguments(n: Node, context: ResolutionContext): boolean { + let memo = VB_NO_ARGUMENTS.get(context); + if (!memo) VB_NO_ARGUMENTS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit !== undefined) return hit; + const code = codeLines(n.filePath, context).slice(n.startLine - 1, n.startLine + 6).join(' '); + const head = new RegExp(`\\b(?:Sub|Function)\\s+\\[?${escapeRegex(unescaped(n.name))}\\]?(?![\\w$%&!#@])`, 'i').exec(code); + let none = true; + if (head) { + let at = head.index + head[0].length; + // `Function Parse(Of T)(text As String)`: the type parameters come first. + if (/^\s*\(\s*Of\b/i.test(code.slice(at))) at = closeParen(code, code.indexOf('(', at)) + 1; + const open = at > 0 ? /^\s*\(/.exec(code.slice(at)) : null; + const params = open ? splitArgs(code, at + open[0].length - 1) : null; + none = !params || params.every((p) => { + const param = p.replace(/<[^<>]*>/g, '').replace(/^[\s_]+/, ''); + return param === '' || /^(?:(?:ByVal|ByRef)\s+)*(?:Optional|ParamArray)\b/i.test(param); + }); + } + memo.set(n.id, none); + return none; +} + +/** + * The overload of `member` (a method) that VB.NET runs where a reference + * names it without parentheses: one that takes no arguments — `Me.Load` is + * `Load(Optional force = False)`, not `Load(path)` — and never the member + * the reference is in: a constructor's `Me.New` is another constructor. + * `member` itself when its overloads don't tell; null when the only one is + * the member the reference is in. + */ +function runnableOverload(member: Node, ref: UnresolvedRef, context: ResolutionContext): Node | null { + const cut = member.qualifiedName.lastIndexOf('::'); + const owner = cut > 0 ? context.getNodesByQualifiedName(member.qualifiedName.slice(0, cut)) + .find((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)) : undefined; + const project = projectOf(member.filePath, context); + const overloads = owner ? membersNamed(owner, member.name, context) + .filter((n) => n.kind === 'method' && projectOf(n.filePath, context) === project) : []; + const others = overloads.filter((n) => n.id !== ref.fromNodeId); + if (others.length === 0) return member.id === ref.fromNodeId ? null : member; + const runnable = others.filter((n) => takesNoArguments(n, context)); + const pool = runnable.length > 0 ? runnable : others; + return pool.find((n) => n.id === member.id) ?? pool[0]!; +} + +/** + * The field, property or Module variable a bare name before the dot reads as + * well, where VB.NET binds the name to one: `Button1` in `Button1.Text` is + * the form's own control, `Title` a field its class inherits, `Current` a + * variable of one of its project's Modules — whatever the member read through + * it. Null for a parameter or local of that name, which hides the member; + * for `Me`, `MyBase` and a `{T}` value; for a `With` block's `.Text`, whose + * `With` names the value; and for a member declared as a type of its own name + * from outside the project (`Responser As Responser`): VB.NET reads that + * type's Shared members and nested types through the type, not the member, + * and nothing here shows which these are. + */ +function headMemberOf(head: string, ref: UnresolvedRef, context: ResolutionContext): Node | null { + if (head.startsWith('{') || /^(?:Me|MyClass|MyBase)$/i.test(head)) return null; + const name = unescaped(head); + // The name itself starts the access where the reference is: `Button1` or `[Button1]`, then the dot. + const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1] ?? ''; + const bracketed = line[ref.column] === '['; + let at = ref.column + (bracketed ? 1 : 0); + if (line.slice(at, at + name.length).toLowerCase() !== name.toLowerCase()) return null; + at += name.length; + if (bracketed && line[at++] !== ']') return null; + while (line[at] === ' ' || line[at] === '\t') at++; + if (line[at] !== '.') return null; + if (localBinding(name, ref, context)) return null; + const found = memberAround(name, ref, context); + const member = found?.node; + if (!member || !VB_VALUE_KINDS.has(member.kind) || member.id === ref.fromNodeId) return null; + // Another project's Module is one this code can't name without a reference to it. + const cut = member.qualifiedName.lastIndexOf('::'); + const foreign = projectOf(member.filePath, context) !== projectOf(ref.filePath, context); + if (foreign && cut > 0 && isModule(member.qualifiedName.slice(0, cut), context)) return null; + return isOutsideNamesake(member, found!.args, context) ? null : member; +} + +const VB_NAMESAKES = new WeakMap>(); + +/** + * Whether a member is declared as a type of its own name that isn't one of + * the project's (`Responser As Responser`), or names two of them: a read + * through its name may be one through that type, which nothing here can tell. + */ +function isOutsideNamesake(member: Node, args: ReadonlyMap, context: ResolutionContext): boolean { + let memo = VB_NAMESAKES.get(context); + if (!memo) VB_NAMESAKES.set(context, (memo = new Map())); + // What a type parameter stands for depends on where the member is reached from. + const hit = args.size === 0 ? memo.get(member.id) : undefined; + if (hit !== undefined) return hit; + const declared = valueMemberType(member, context, args); + let namesake = false; + if (declared && !declared.array && declared.name.toLowerCase() === unescaped(member.name).toLowerCase()) { + const types = typesNamedAt(declared, context, VB_READ_TYPE_KINDS); + namesake = types.ambiguous || types.owners.length === 0; + } + if (args.size === 0) memo.set(member.id, namesake); + return namesake; +} + /** * Resolve a VB.NET member read or write to the member the receiver's type * declares or inherits. Through a type's name (#2305) — `AppSession.SessionId`, @@ -1494,7 +1715,12 @@ function namesWithoutRunning(ref: UnresolvedRef, context: ResolutionContext): bo * mean. A value named like its type (`theme As Theme`) reads an instance * member as a value does and a Shared one as the type does, by VB.NET's * "Color Color" rule. A method named either way is called, as without - * parentheses VB.NET runs it, unless `AddressOf` or `NameOf` only names it. + * parentheses VB.NET runs it — the overload that takes no arguments — unless + * `AddressOf` or `NameOf` only names it. Through a namespace — `Billing.Invoice`, + * `Acme.Billing.Invoice.MaxLines`, a Module's `Assets.Logo` — it links the type + * or the Module's member the namespace declares. A bare name before the dot + * that is a field, property or Module variable (`Button1` of `Button1.Text`) + * is read too, whatever the member read through it. * Null when a link of the path is not typed here (an outside type: * `Color.Red`, `str.Length`), is late-bound (`Object`), or names two of the * project's types. @@ -1504,10 +1730,11 @@ export function matchVbMemberRead(ref: UnresolvedRef, context: ResolutionContext if (!m) return null; const links = m[2] ? m[2].slice(1).split('.') : []; const at = pathReceiver(m[1]!, links, ref, context); - if (!at || at.owners.length === 0) return null; // `MySettings.Default` reads the property declared `[Default]`, and back. const name = unescaped(m[3]!); const calls = (member: Node) => member.kind === 'method' && !namesWithoutRunning(ref, context); + // A method VB.NET runs here is the overload it can call without arguments. + const run = (member: Node | undefined): Node | null => (!member ? null : calls(member) ? runnableOverload(member, ref, context) : member); const throughValue = (member: Node): ResolvedRef => ({ original: ref, targetNodeId: member.id, @@ -1515,13 +1742,36 @@ export function matchVbMemberRead(ref: UnresolvedRef, context: ResolutionContext resolvedBy: 'instance-method', ...(calls(member) ? { edgeKind: 'calls' as const } : {}), }); + // The member a bare name before the dot reads, beside what is read through + // it. A method run through the name (`Button1.PerformClick`) is linked alone, + // as a call written with parentheses is. + const withHead = (read: ResolvedRef | null): ResolvedRef | null => { + const head = links.length === 0 ? headMemberOf(m[1]!, ref, context) : null; + if (!head) return read; + if (!read) return { original: ref, targetNodeId: head.id, confidence: 0.9, resolvedBy: 'instance-method' }; + return read.edgeKind === 'calls' ? read : { ...read, alsoTargets: [...(read.alsoTargets ?? []), { targetNodeId: head.id }] }; + }; + if (at?.namespace) { + // A type, or a Module's member, read through its namespace. + const types = typesInNamespace(at.namespace, name, ref, context); + const target = types?.[0] ?? (types ? run(moduleMemberIn(at.namespace, name, ref, context)) : null); + if (!target || target.id === ref.fromNodeId) return null; + return { + original: ref, + targetNodeId: target.id, + confidence: target.kind === 'method' ? 0.85 : 0.9, + resolvedBy: 'qualified-name', + ...(calls(target) ? { edgeKind: 'calls' as const } : {}), + }; + } + if (!at || at.owners.length === 0) return withHead(null); if (!at.throughType) { // The member the name means: the nearest the type declares or inherits. // The read's own member (`Return Me.Count` in `Count`) runs itself. const fits = (n: Node) => VB_INSTANCE_READ_KINDS.has(n.kind); - const member = (memberOn(at.owners, name, ref, context, true, at.typed, fits) ?? - memberOn(at.owners, `[${name}]`, ref, context, true, at.typed, fits))?.node; - return member && member.id !== ref.fromNodeId ? throughValue(member) : null; + const member = run((memberOn(at.owners, name, ref, context, true, at.typed, fits) ?? + memberOn(at.owners, `[${name}]`, ref, context, true, at.typed, fits))?.node); + return withHead(member && member.id !== ref.fromNodeId ? throughValue(member) : null); } // Not the member the read is written in: staxrip's `Overrides ReadOnly // Property Package` returns the class's `Shared ReadOnly Property Package`. @@ -1534,9 +1784,10 @@ export function matchVbMemberRead(ref: UnresolvedRef, context: ResolutionContext const found = member; member = context.getNodesByQualifiedName(found.qualifiedName).find((n) => n.isStatic && n.filePath === found.filePath && fits(n)) ?? found; } + member = run(member) ?? member; // "Color Color": a value named like its type (`theme As Theme`) reads an // instance member through itself, and only a Shared one through the type. - if (at.colorColor && !isSharedMember(member, context)) return throughValue(member); + if (at.colorColor && !isSharedMember(member, context)) return withHead(throughValue(member)); if (member.kind === 'method') { return { original: ref, @@ -1649,7 +1900,7 @@ export function isVbMemberInScope(n: Node, ref: UnresolvedRef, context: Resoluti export function clearVbnetReceiverMemos(context: ResolutionContext): void { for (const memo of [VB_CODE_LINES, VB_WORD_LINES, VB_BINDINGS, VB_FILE_TYPES, VB_FILE_QNS, VB_MEMBERS, VB_MODULES, VB_SUPERS, VB_ANCESTRIES, VB_PROJECTS, VB_PROJECT_INFO, VB_FILE_IMPORTS, VB_ALIASES, VB_BASE_NAMES, VB_TYPE_PARAMS, VB_EXTENSIONS, VB_TYPE_IMPORTS, - VB_SCOPES] as Array>) { + VB_SCOPES, VB_NAMESPACES, VB_NO_ARGUMENTS, VB_NAMESAKES] as Array>) { memo.delete(context); } } From bf6f1fa67e7a69c2371ed2f765e21d0749cfa993 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 21 Oct 2026 20:46:20 +0000 Subject: [PATCH 06/95] fix(dart): a typedef references the types it aliases (#2763) In class Report { final int score; const Report(this.score); } typedef ReportCallback = void Function(Report r); `codegraph callers Report` did not list ReportCallback. Both extractors minted the typedef's node and nothing else. tree-sitter-dart gives `type_alias` no fields, so extractTypeAlias found no `value` to read and had no Dart branch, and the kernel's extract_type_alias mirrored that. Impact on a model type missed every callback or collection type built on it. The legacy form with a return type, `typedef Report LegacyMaker(int n);`, was also named after that return type: with no `name` field the name fell back to the first type name in the declaration, so a `type_alias Report` (or `Widget`, `Future`, `bool`) stood beside the real type and took references meant for it. Both extractors now name a typedef by the last type name before its `=`, its type parameters or its parameter list, and add a `references` ref from the typedef, on each type's name, for every type its aliased type names: a function type's return and parameter types, generic arguments, record fields, a type parameter's bound, and the legacy form's return and parameter types. Annotations are not part of it, an import prefix names no type, and a type parameter (the typedef's own, or one a generic function type or an old-style generic function-typed parameter declares) is skipped unless written through a prefix. Kernel and wasm output are identical; a new parity fixture, TortureTypedefs.dart, pins every form in LF and CRLF. Validation, kernel path, arms built from dbb6aa9b and from this change, each with its own kernel: - felangel/bloc b9be1e25: 17,875 -> 17,884 edges (+9, 0 removed). - rrousselGit/riverpod 4ded4f01: 84,726 -> 84,761 (+35, 0 removed). - jonataslaw/getx e3c8fa04: 8,436 -> 8,456 (+20, 0 removed). - alibaba/flutter-go a41b2124 (9 legacy typedefs, 6 with a return type): 7,994 -> 7,968; 39 wrong links removed (34 `Widget` and 5 `Future` uses that had gone to typedefs misnamed `Widget` and `Future`), 13 added (4 from typedefs, 9 fields typed by the renamed ones). - peng8350/flutter_pulltorefresh 0be096c1 (8 legacy, 5 with a return type): 3,186 -> 2,976; 221 wrong `Widget` links removed, 11 added. All 88 added non-contains edges were checked against the source; every removed edge pointed at a typedef misnamed after its return type. The other new typedef refs name SDK or Flutter types and stay unresolved. gin and zustand dumps are byte-identical across the arms, kernel and wasm dumps are byte-identical on bloc, riverpod and pulltorefresh, and the parity sweep over the five Dart repos is 2,330/2,633 files at parity, 0 diffs, 303 deferred on both arms. Index time is unchanged within noise (median of 5 interleaved runs: riverpod 12.6 s vs 11.9 s, bloc 5.0 s vs 5.0 s, getx 2.8 s vs 2.8 s), and the DB grows by 8 to 24 KB (pulltorefresh shrinks by 64 KB). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/dart-typedef-refs.test.ts | 312 ++++++++++++++++++ .../kernel-parity/TortureTypedefs.dart | 78 +++++ __tests__/kernel-dart-parity.test.ts | 11 +- codegraph-kernel/src/dart.rs | 95 +++++- docs/design/dart-kernel-port-checklist.md | 14 +- src/extraction/languages/dart.ts | 79 +++++ src/extraction/tree-sitter.ts | 8 +- 8 files changed, 588 insertions(+), 10 deletions(-) create mode 100644 __tests__/dart-typedef-refs.test.ts create mode 100644 __tests__/fixtures/kernel-parity/TortureTypedefs.dart diff --git a/CHANGELOG.md b/CHANGELOG.md index dd39604439..5f93154ed2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In COBOL, asking `codegraph_explore` (or `codegraph explore` and `codegraph context`) about a copybook by name, such as `CVACT01Y` or a member brought in with `EXEC SQL INCLUDE`, now answers with the copybook's own source when it is indexed and every `COPY` or `EXEC SQL INCLUDE` statement that includes it, and says so when the copybook's source isn't part of the project. Before, it answered "No relevant code found", or showed an unrelated program whose name was a letter or two away. Thanks @popolusiak for the report. (#2342) - In Dart, reading a getter such as `box.area` or `status.label` now counts as calling it, so `codegraph callers` and impact list the code that reads it — whether the getter is declared on the class, inherited, or added by an extension — and a method an extension adds to an enum, like `shape.shout()`, now links as well. A plain field, a getter of a type from outside your project, or a value whose type isn't written down still links nothing. Re-index Dart projects after upgrading. Thanks @brookly255-lv for the report. (#2338) - In Dart, a type is now linked from more of the places it is written, not only from parameters and return types: the type an `extension … on` targets, a field's type, a top-level variable's type, generic arguments such as `Future.value(null)` or Riverpod's `final reportProvider = Family()`, and a local variable's type, a cast or a type check inside a function. So `callers` and impact for a model class now reach the extensions, models and providers that use it. Thanks @mg-mg-mg for the report. (#2327) +- In Dart, a `typedef` now links to the types it is built from, like `Report` in `typedef ReportCallback = void Function(Report r);` or `typedef Reports = List;`, so `callers` and impact for a model class also reach the callback and collection types defined on it. An old-style typedef with a return type, like `typedef Widget ItemBuilder(BuildContext context);`, is now indexed under its own name instead of its return type's, so code that only uses `Widget` no longer links to it. Thanks @mg-mg-mg for the report. (#2327) - In Go, calls through a method receiver or parameter of an unexported type, the usual shape of gRPC and HTTP handlers (`s.service.AddItem()` inside `func (s *server) Create()`), now resolve, and always within that type's own package: another package's `server` with a same-named method no longer takes the call, and a method the type gets from a struct or interface it embeds is found too. Re-index Go projects after upgrading. Thanks @GoDiao for the report and the fix. (#2323) - In Go, a call made through a type assertion, like `srv.(KVServer).Range(ctx, in)` in generated gRPC code, `c.Reader.(*pipe).Close()` or `v.(storage.Store).Fetch(key)`, now links to that method of the asserted type, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, it linked to whichever method shared the name, so every gRPC handler pointed at the `Unimplemented…Server` stub instead of the server interface. Re-index Go projects after upgrading. - In Go, a call through a variable that holds the result of a type assertion, like `st.Fetch(key)` after `st := v.(storage.Store)` or `rc.Read(p)` after `rc, ok := v.(ReadCloser)`, now links to that method of the asserted type, as a call made through the assertion itself does, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, such a call linked to whichever method shared the name or to nothing, and a variable named like a standard-library package, like grpc-go's `parser, ok := builder.(balancer.ConfigParser)`, was taken for that package. Re-index Go projects after upgrading. diff --git a/__tests__/dart-typedef-refs.test.ts b/__tests__/dart-typedef-refs.test.ts new file mode 100644 index 0000000000..6b8c7eeb92 --- /dev/null +++ b/__tests__/dart-typedef-refs.test.ts @@ -0,0 +1,312 @@ +/** + * A Dart typedef references the types it aliases (#2327). + * + * In + * + * class Report { final int score; const Report(this.score); } + * typedef ReportCallback = void Function(Report r); + * + * `callers Report` did not list `ReportCallback`. Both extractors minted the + * typedef's node and nothing else: the grammar gives `type_alias` no fields, + * so the aliased type was never read, and impact on a model type missed every + * callback type built on it. The legacy form with a return type, `typedef + * Report LegacyMaker(int n);`, was named after that return type, the first + * type name in it, so a `type_alias Report` stood beside the class. + * + * A typedef now references every type its aliased type names, from the + * typedef, on each type's name: a function type's return and parameter types, + * generic arguments, record fields, a type parameter's bound, and the legacy + * form's return and parameter types. A type parameter, the typedef's own or a + * generic function type's, means that parameter and never a project type of + * its name; an import prefix names no type; and the legacy form is named by + * its own name. + * + * Runs against the native kernel (when built) and the wasm extractor, which + * must agree. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import type { ExtractionResult, Node } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +/** Every typedef shape: new style and legacy, generic, prefixed, annotated, wrapped. */ +const SOURCE = `import 'model.dart' as m; + +class Report { + final int score; + const Report(this.score); +} + +typedef ReportCallback = void Function(Report r); +typedef ReportMaker = Report Function(); +typedef Reports = List; +typedef ReportsByName = Map; +typedef Results = Future?> Function(int page); +typedef void LegacyCallback(Report r); +typedef Report LegacyMaker(int n); +typedef List LegacyLoader([Report? seed]); +typedef m.Report PrefixedLegacy(m.Report r); +typedef Mapper = T Function(Report r); +typedef Picker = T Function(T a, Report r); +typedef T LegacyPicker(T a, Report r); +typedef Bounded = List; +typedef Sdk = Future; +typedef Dynamic = dynamic Function(int n); +typedef Prefixed = m.Report; +typedef Named = void Function({required Report report, int? n}); +typedef Pair = (Report, {m.Report other}); +typedef Nested = void Function(Report Function(int) make); +/// Documented. +@Deprecated('Use ReportCallback') +typedef OldCallback = void Function(Report r); +typedef Formatted = + Map Function( + Report first, + Report second, + ); +`; + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +describe('a Dart typedef references the types it aliases (#2327)', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['dart']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', source: string): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource('lib/typedefs.dart', source, 'dart'); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + const result = tryKernelExtract('lib/typedefs.dart', source, 'dart'); + expect(result, 'kernel extraction').not.toBeNull(); + return result!; + } + + /** ` ` for every `references` ref out of a typedef, sorted. */ + function typedefRefs(result: ExtractionResult): string[] { + const byId = new Map(result.nodes.map((n) => [n.id, n])); + return result.unresolvedReferences + .filter((r) => r.referenceKind === 'references' && byId.get(r.fromNodeId)?.kind === 'type_alias') + .map((r) => `${byId.get(r.fromNodeId)!.name} ${r.referenceName}`) + .sort(); + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + + for (const crlf of [false, true]) { + const eol = (s: string) => (crlf ? s.replace(/\n/g, '\r\n') : s); + const label = crlf ? ' (CRLF)' : ''; + + it.each(backends)(`a legacy typedef is named by its own name, not its return type: %s${label}`, (backend) => { + const result = extract(backend, eol(SOURCE)); + expect(result.nodes.filter((n) => n.kind === 'type_alias').map((n) => n.name)).toEqual([ + 'ReportCallback', + 'ReportMaker', + 'Reports', + 'ReportsByName', + 'Results', + 'LegacyCallback', + 'LegacyMaker', + 'LegacyLoader', + 'PrefixedLegacy', + 'Mapper', + 'Picker', + 'LegacyPicker', + 'Bounded', + 'Sdk', + 'Dynamic', + 'Prefixed', + 'Named', + 'Pair', + 'Nested', + 'OldCallback', + 'Formatted', + ]); + // An annotated typedef keeps the dartdoc above its annotation. + expect(result.nodes.find((n) => n.name === 'OldCallback')!.docstring).toBe('Documented.'); + }); + + it.each(backends)(`every type the alias names is referenced from the typedef: %s${label}`, (backend) => { + const source = eol(SOURCE); + const result = extract(backend, source); + // Built-ins (`void`, `int`, `String`, `dynamic`) are no references, an + // import prefix (`m`) is none, and neither is a type parameter (`T`, + // `X`), declared by the typedef or by a generic function type. + expect(typedefRefs(result)).toEqual([ + 'Bounded List', + 'Bounded Report', + 'Formatted Map', + 'Formatted Report', + 'Formatted Report', + 'Formatted Report', + 'LegacyCallback Report', + 'LegacyLoader List', + 'LegacyLoader Report', + 'LegacyLoader Report', + 'LegacyMaker Report', + 'LegacyPicker Report', + 'Mapper Report', + 'Named Report', + 'Nested Report', + 'OldCallback Report', + 'Pair Report', + 'Pair Report', + 'Picker Report', + 'Prefixed Report', + 'PrefixedLegacy Report', + 'PrefixedLegacy Report', + 'ReportCallback Report', + 'ReportMaker Report', + 'Reports List', + 'Reports Report', + 'ReportsByName Map', + 'ReportsByName Report', + 'Results Future', + 'Results List', + 'Results Report', + 'Sdk Future', + ]); + // Each sits on the type's name, where resolution reads an import prefix back. + const lines = source.split(/\r?\n/); + const byId = new Map(result.nodes.map((n) => [n.id, n])); + for (const r of result.unresolvedReferences.filter((x) => byId.get(x.fromNodeId)?.kind === 'type_alias')) { + expect(lines[r.line - 1]!.startsWith(r.referenceName, r.column), `${r.referenceName} at ${r.line}:${r.column}`).toBe(true); + } + // The file takes none of them: no type is credited twice. + const file = result.nodes.find((n) => n.kind === 'file')!; + expect(result.unresolvedReferences.filter((r) => r.fromNodeId === file.id && r.referenceKind === 'references')).toEqual([]); + }); + } +}); + +describe('an indexed Dart library links a typedef to the types it aliases (#2327)', () => { + const FILES: Record = { + 'pubspec.yaml': 'name: repro\nenvironment:\n sdk: ">=3.0.0 <4.0.0"\n', + // The issue's repro. + 'lib/report.dart': 'class Report {\n final int score;\n const Report(this.score);\n}\n', + 'lib/callback.dart': "import 'report.dart';\n\ntypedef ReportCallback = void Function(Report r);\n", + // A namesake reached only through the `m.` prefix. + 'lib/model.dart': 'class Report {}\n', + // Project types named like the type parameters, imported where those are declared. + 'lib/params.dart': 'class T {}\n\nclass X {}\n', + // A project `Future` the typedefs' library does not import. + 'lib/async.dart': 'class Future {}\n', + 'lib/typedefs.dart': `import 'model.dart' as m; +import 'params.dart'; +import 'report.dart'; + +typedef ReportMaker = Report Function(); +typedef Reports = List; +typedef Report LegacyMaker(int n); +typedef Mapper = T Function(Report r); +typedef Picker = T Function(T a, Report r); +typedef Bounded = List; +typedef Sdk = Future; +typedef Prefixed = m.Report; +typedef Pair = (Report, {m.Report other}); +`, + }; + + let root = ''; + let cg: CodeGraph | undefined; + let kernel: string | undefined; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-dart-typedef-refs-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + kernel = process.env.CODEGRAPH_KERNEL; + }); + + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(root, { recursive: true, force: true }); + if (kernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernel; + }); + + it.each(['default', 'wasm'])('callers of a type list the typedefs built on it (%s)', async (backend) => { + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + cg = await CodeGraph.init(root, { index: true }); + const graph = cg; + const node = (file: string, name: string): Node => { + const found = graph.getNodesInFile(file).filter((n) => n.name === name && n.kind !== 'import'); + expect(found, `${name} in ${file}`).toHaveLength(1); + return found[0]!; + }; + /** `kind file:qualifiedName` of each `references` target of the typedef `name`, sorted. */ + const referenced = (name: string): string[] => + [...new Set(graph + .getOutgoingEdgesFrom([node('lib/typedefs.dart', name).id], ['references']) + .map((e) => graph.getNode(e.target)!) + .map((n) => `${n.kind} ${n.filePath}:${n.qualifiedName}`))] + .sort(); + + // `callers Report` lists every typedef that names it, the issue's first. + const callers = graph.getCallers(node('lib/report.dart', 'Report').id).map(({ node: n }) => `${n.kind} ${n.filePath}:${n.name}`); + expect(callers.filter((c) => c.startsWith('type_alias')).sort()).toEqual([ + 'type_alias lib/callback.dart:ReportCallback', + 'type_alias lib/typedefs.dart:Bounded', + 'type_alias lib/typedefs.dart:LegacyMaker', + 'type_alias lib/typedefs.dart:Mapper', + 'type_alias lib/typedefs.dart:Pair', + 'type_alias lib/typedefs.dart:Picker', + 'type_alias lib/typedefs.dart:ReportMaker', + 'type_alias lib/typedefs.dart:Reports', + ]); + + // A prefixed type is the one its import names. + expect(referenced('Prefixed')).toEqual(['class lib/model.dart:Report']); + expect(referenced('Pair')).toEqual(['class lib/model.dart:Report', 'class lib/report.dart:Report']); + // A type parameter is not the project's `T` or `X`, though both are imported. + expect(referenced('Mapper')).toEqual(['class lib/report.dart:Report']); + expect(referenced('Picker')).toEqual(['class lib/report.dart:Report']); + expect(referenced('Bounded')).toEqual(['class lib/report.dart:Report']); + expect(graph.getCallers(node('lib/params.dart', 'T').id)).toEqual([]); + expect(graph.getCallers(node('lib/params.dart', 'X').id)).toEqual([]); + // The SDK's Future is not a project class the library does not import. + expect(referenced('Sdk')).toEqual([]); + expect(graph.getCallers(node('lib/async.dart', 'Future').id)).toEqual([]); + // The legacy typedef is named LegacyMaker; nothing else is named Report there. + expect(graph.getNodesInFile('lib/typedefs.dart').filter((n) => n.name === 'Report')).toEqual([]); + expect(referenced('LegacyMaker')).toEqual(['class lib/report.dart:Report']); + }, 60_000); +}); diff --git a/__tests__/fixtures/kernel-parity/TortureTypedefs.dart b/__tests__/fixtures/kernel-parity/TortureTypedefs.dart new file mode 100644 index 0000000000..f152ad8f35 --- /dev/null +++ b/__tests__/fixtures/kernel-parity/TortureTypedefs.dart @@ -0,0 +1,78 @@ +// Typedefs reference the types they alias (#2327): every new-style and legacy +// shape, type parameters (the typedef's own and a generic function type's), +// import prefixes, annotations, comments and wrapped declarations. +import 'package:meta/meta.dart'; +import 'model.dart' as m; +import 'model.dart' as M; + +class Report { + final int score; + const Report(this.score); +} + +class T {} + +void tag(Object o) {} + +// New style. +typedef ReportCallback = void Function(Report r); +typedef ReportMaker = Report Function(); +typedef Reports = List; +typedef ReportsByName = Map; +typedef Results = Future?> Function(int page); +typedef Opt = Report?; +typedef MaybeMaker = Report Function()?; +typedef Pair = (Report, int); +typedef NamedPair = ({Report first, m.Report second}); +typedef Named = void Function({required Report report, int? n}); +typedef Positional = void Function([Report? report]); +typedef Unnamed = void Function(Report, m.Report); +typedef Nested = void Function(void Function(Report) inner); +typedef Maker = void Function(Report Function() make); +typedef Dynamic = dynamic Function(Object? o); +typedef Void = void; +typedef Fn = Function; +typedef Prefixed = m.Report; +typedef UpperPrefixed = M.Report; +typedef PrefixedGeneric = m.Box; + +// Type parameters: the typedef's own, a bound, a generic function type's. +typedef Mapper = T Function(Report r); +typedef Bounded = List; +typedef SelfBounded> = int Function(C a, C b); +typedef Picker = T Function(T a, Report r); +typedef Outer = S Function(U u, S s, T t); +typedef Shadow = m.T Function(T t); +typedef TwoParams = Map>; + +// Legacy. +typedef void LegacyCallback(Report r); +typedef Report LegacyMaker(int n); +typedef List LegacyLoader([Report? seed]); +typedef m.Report PrefixedLegacy(m.Report r); +typedef Report? LegacyOptional({Report? fallback}); +typedef void Function(Report) LegacyFnReturn(int n); +typedef T LegacyPicker(T a, Report r); +typedef void LegacyFnParam(Report make(int n)); +typedef void LegacyGenericFnParam(S make(S s, Report r)); +typedef void LegacyCovariant(covariant Report r, final Report s); + +// Annotated, documented, commented, wrapped. +/// Documented. +@Deprecated('Use ReportCallback') +typedef OldCallback = void Function(Report r); +@visibleForTesting +@Tag(tag) +typedef Tagged = Report Function(/* the report */ Report r); +typedef Commented = // why + Report Function(Report r); +typedef Wrapped = + Map Function( + Report first, + Report second, + ); + +class Tag { + const Tag(this.f); + final Object f; +} diff --git a/__tests__/kernel-dart-parity.test.ts b/__tests__/kernel-dart-parity.test.ts index 487fcfd17d..c1f868a1b1 100644 --- a/__tests__/kernel-dart-parity.test.ts +++ b/__tests__/kernel-dart-parity.test.ts @@ -37,8 +37,10 @@ * TortureAnnotatedComments.dart: the comments between annotations and their * declaration, stepped over by the decorator scan; TortureDirectives.dart / * TorturePartOfName.dart: `part` as an import of the part's file, `part of` - * as nothing, and the arguments of annotated directives) and their CRLF - * variants (derived in-memory — #1329), plus defer and generated-file pins. + * as nothing, and the arguments of annotated directives; + * TortureTypedefs.dart: the types a typedef aliases, in every form) and + * their CRLF variants (derived in-memory — #1329), plus defer and + * generated-file pins. * * The full-repo sweeps live in scripts/kernel-parity.mjs (shelf/bloc/flutter * with --max-deferral 0.3); this suite keeps the invariant alive in @@ -167,6 +169,11 @@ describe.skipIf(!kernelBuilt)('kernel Dart extraction parity', () => { ['TortureDirectives.dart', 20], // `part of` a library named by its library name records nothing. ['TorturePartOfName.dart', 3], + // Typedefs reference the types they alias (#2327), new style and legacy + // (named by its own name, not its return type), with type parameters — + // the typedef's own and a generic function type's — skipped, prefixed, + // annotated, commented and wrapped declarations. + ['TortureTypedefs.dart', 40], ] as const; for (const [file, minNodes] of FIXTURES) { diff --git a/codegraph-kernel/src/dart.rs b/codegraph-kernel/src/dart.rs index 78bc8a5db9..7dcc5ef1bf 100644 --- a/codegraph-kernel/src/dart.rs +++ b/codegraph-kernel/src/dart.rs @@ -156,6 +156,23 @@ fn next_named<'t>(node: Node<'t>) -> Option> { next } +/// dartTypeAliasName (dart.ts) — the name a typedef declares: the last type +/// name before the `=`, the type parameters or the parameter list. The +/// legacy form writes its return type first (`typedef Report LegacyMaker(int +/// n);`), so the first type name is not always it. +fn type_alias_name<'t>(node: Node<'t>) -> Option> { + let mut name = None; + let mut cursor = node.walk(); + for child in node.children(&mut cursor) { + match child.kind() { + "=" | "type_parameters" | "formal_parameter_list" => break, + "type_identifier" => name = Some(child), + _ => {} + } + } + name +} + struct Scope { row: u32, kind: &'static str, @@ -668,6 +685,14 @@ impl<'t> Walker<'t> { /// name field → the method_signature inner unwrap → identifier-ish /// child → `` (operators land here). fn extract_name(&self, node: Node<'t>) -> String { + // resolveName hook (dart.ts): a typedef has no `name` field, and a + // legacy one writes its return type before its name (`typedef Report + // LegacyMaker(int n);`). + if node.kind() == "type_alias" { + if let Some(name) = type_alias_name(node) { + return self.text(name).to_string(); + } + } // resolveName hook (dart.ts:244-260): named ctor/factory → ctor name. if let Some((class_name, ctor_name)) = self.ctor_info(node) { if ctor_name != class_name { @@ -1034,12 +1059,76 @@ impl<'t> Walker<'t> { return false; } let docstring = self.docstring_of(node); - // `value` field is null (type_alias has no fields) → no refs from - // the aliased type; returns false → children re-visited. - self.create_node("type_alias", &name, node, Extra { docstring, ..Default::default() }); + // type_alias has no fields (no `value`): the dart branch reads the + // aliased type from the rest of the declaration (#2327). Returns + // false → children re-visited (they match nothing type-related). + let row = self.create_node("type_alias", &name, node, Extra { docstring, ..Default::default() }); + if let Some(row) = row { + self.type_alias_refs(node, row); + } false } + /// pushDartTypeAliasRefs (dart.ts) — a `references` ref from the typedef + /// for every type its aliased type names; its annotations are not its + /// type, and a type parameter is no type of the project's. + fn type_alias_refs(&mut self, node: Node<'t>, from_row: u32) { + let name = type_alias_name(node); + let scope = self.type_parameter_scope(node, &[]).unwrap_or_default(); + let mut cursor = node.walk(); + let kids: Vec> = node.named_children(&mut cursor).collect(); + for child in kids { + if child.kind() == "annotation" || Some(child) == name { + continue; + } + self.aliased_type_refs(child, from_row, &scope); + } + } + + /// dartTypeParameterScope (dart.ts) — `scope` and the type parameters + /// `node` declares (`Mapper`, `T Function(T a)`), or None when it + /// declares none. + fn type_parameter_scope(&self, node: Node<'t>, scope: &[&'t str]) -> Option> { + let mut cursor = node.walk(); + let parameters = node.named_children(&mut cursor).find(|c| c.kind() == "type_parameters")?; + let mut inner = scope.to_vec(); + let mut pc = parameters.walk(); + for parameter in parameters.named_children(&mut pc) { + if parameter.kind() != "type_parameter" { + continue; + } + let mut ic = parameter.walk(); + let name = parameter.named_children(&mut ic).find(|c| c.kind() == "type_identifier"); + if let Some(name) = name { + inner.push(self.text(name)); + } + } + Some(inner) + } + + /// pushDartAliasedTypeRefs (dart.ts) — type_refs_dart for a typedef's + /// aliased type, where a type parameter in `scope` is no type of its own + /// (but `p.T` is the type an import names). + fn aliased_type_refs(&mut self, node: Node<'t>, from_row: u32, scope: &[&'t str]) { + stack_guard!(); + if node.kind() == "type_identifier" { + let text = self.text(node); + let parameter = + scope.contains(&text) && !node.prev_sibling().map(|s| s.kind() == ".").unwrap_or(false); + if !parameter && self.is_type_name_dart(node) { + self.push_ref_at(from_row, text, "references", node); + } + return; + } + let inner = self.type_parameter_scope(node, scope); + let scope = inner.as_deref().unwrap_or(scope); + let mut cursor = node.walk(); + let kids: Vec> = node.named_children(&mut cursor).collect(); + for child in kids { + self.aliased_type_refs(child, from_row, scope); + } + } + // --- extractImport (:3170; hook dart.ts extractImport) ----------------- fn extract_import(&mut self, node: Node<'t>) { diff --git a/docs/design/dart-kernel-port-checklist.md b/docs/design/dart-kernel-port-checklist.md index 0386b419a2..d130ad1126 100644 --- a/docs/design/dart-kernel-port-checklist.md +++ b/docs/design/dart-kernel-port-checklist.md @@ -226,7 +226,12 @@ Grammar-shape facts the config leans on (all probed, `mini-cst.txt` / keyword node, identifier, on-types as bare type_identifiers, class_body). `enum_declaration`: `name:` + `body:` (enum_body); `enum_constant.name` ✓. `type_alias` has NO fields (both modern `typedef X = T` and legacy - `typedef void X(int)` are node `type_alias`). + `typedef void X(int)` are node `type_alias`). Its name is the last direct + `type_identifier` before the `=`, `type_parameters` or + `formal_parameter_list` child: a legacy return type comes first (`typedef + Report X(int n)`), and until 2026-10-21 the first-identifier fallback + named such a typedef after its return type (`dartTypeAliasName` / + `type_alias_name` now, via resolveName). `initialized_variable_definition`: `name:` + one-or-more `value:` fields (the RHS postfix chain is FLATTENED into multiple value-fielded siblings). - **In-class member shapes:** methods/getters/setters/operators/factories @@ -472,7 +477,7 @@ Registration: `EXTRACTORS.dart` (languages/index.ts:56), `FN_REF_SPECS.dart` | `mixin_declaration`, `extension_declaration` | extraClassNodeTypes:1022 | extractClass(node) → kind **`class`** | | `method_signature`, `constructor_signature` | methodTypes:1027 | **NOT class-gated at the ladder** — extractMethod:1737 runs anywhere; its own gate :1747 (not class-like, no methodsAreTopLevel, no receiver) falls back: parent `object`/`object_expression`? (never in dart) else **extractFunction** — this is how extension-TYPE members and any stray non-class method_signature become plain `function` nodes (pinned: `extension type MetersT` → function `km`, bare QN) | | `enum_declaration` | enumTypes:1064 → extractEnum:1914 | §Class family | -| `type_alias` | typeAliasTypes:1071 → extractTypeAlias:2890 | plain type_alias node; `getChildByField(node,'value')` → **null** (no fields) → **NO refs from the aliased type** (`typedef MapAlias = Map` emits nothing); returns false → children re-visited (function_type/formal_parameter_list children match nothing) | +| `type_alias` | typeAliasTypes:1071 → extractTypeAlias:2890 | type_alias node named by the resolveName hook (§Extractor config); `getChildByField(node,'value')` → **null** (no fields), so the dart branch — `pushDartTypeAliasRefs` / `type_alias_refs`, since 2026-10-21 (#2327; nothing before) — emits a `references` ref from the node for each type named in every named child but the name and `annotation`s, through the isDartTypeName gate (`typedef MapAlias = Map` → `Map`, `WidgetT`). A type parameter in scope is skipped unless written `p.T`: the typedef's own, and those of any node that declares `type_parameters` (a generic `function_type`, an old-style generic function-typed `formal_parameter`) within it. Returns false → children re-visited (they match nothing type-related; an annotated typedef's arguments are fn-ref capture points) | | `import_or_export`, `part_directive` | importTypes:1209 → extractImport:3170 | §Extractor config. No skipChildren: the directive's children are still recursed, so an annotated directive's arguments are fn-ref capture points (`@Tag(f) import 'a.dart';` captures `f`). The kernel arm recurses too — since 2026-10-06; before, its `import_or_export` arm returned and missed those (TortureDirectives.dart pins both) | | `new_expression` | INSTANTIATION_KINDS:1255 (`new_expression` ∈ :354-361) | extractInstantiation:4610 → ctor field lookups null → namedChild(0) = type_identifier → `instantiates` ref from stack top; `<`-strip + last-`.`-segment apply (`new p.Foo()` → `Foo`). findAnonymousClassBody → always null for dart. Children still recursed | | `function_body` (sibling of a consumed signature) | **no branch** | recursed → THE DOUBLE-WALK (§below) | @@ -1299,8 +1304,9 @@ NO unwrap, NO ungatedModes, NO addressOfOnly. Pins: sealed/base modifiers (plain classes); enum (simple + enhanced: ctor'd constants at constant positions, args unwalked, `with` GAP vs `implements` ✓, members after `;`); typedef ×3 (modern/legacy/generic — - zero refs); extension type (container invisible, **members leak as bare - `function`s**); the FULL call matrix (§Calls rows verbatim — incl. + zero refs until 2026-10-21, the aliased types since; every form is in + TortureTypedefs.dart); extension type (container invisible, **members + leak as bare `function`s**); the FULL call matrix (§Calls rows verbatim — incl. cascade invisibility, `?.`, const-object args, chain re-encodes, interpolation calls, `$x` vs `${x}`); static reads (`X.member`, `X.method()` double-emission, 3-segment nothing, case-pattern GAP); diff --git a/src/extraction/languages/dart.ts b/src/extraction/languages/dart.ts index 6f68fb234e..6fe24dccd1 100644 --- a/src/extraction/languages/dart.ts +++ b/src/extraction/languages/dart.ts @@ -43,6 +43,82 @@ export function pushDartTypeRefs( for (const child of node.namedChildren) pushDartTypeRefs(child, fromNodeId, push); } +/** + * The name a typedef declares. The grammar gives `type_alias` no fields, and + * the first type name is not always it: the legacy form writes its return + * type first (`typedef Report LegacyMaker(int n);`). The name is the last type + * name before the `=`, the type parameters or the parameter list. + */ +export function dartTypeAliasName(node: SyntaxNode): SyntaxNode | undefined { + let name: SyntaxNode | undefined; + for (let i = 0; i < node.childCount; i++) { + const child = node.child(i); + if (!child || child.type === '=' || child.type === 'type_parameters' || child.type === 'formal_parameter_list') break; + if (child.type === 'type_identifier') name = child; + } + return name; +} + +/** `scope` and the type parameters `node` declares, when it declares any (`Mapper`, `T Function(T a)`). */ +function dartTypeParameterScope(node: SyntaxNode, scope: ReadonlySet): ReadonlySet { + const parameters = node.namedChildren.find((c: SyntaxNode) => c.type === 'type_parameters'); + if (!parameters) return scope; + const inner = new Set(scope); + for (const parameter of parameters.namedChildren) { + const name = parameter.type === 'type_parameter' + ? parameter.namedChildren.find((c: SyntaxNode) => c.type === 'type_identifier') + : undefined; + if (name) inner.add(name.text); + } + return inner; +} + +/** pushDartTypeRefs for a typedef's aliased type: a type parameter in `scope` is no type of its own. */ +function pushDartAliasedTypeRefs( + node: SyntaxNode, + fromNodeId: string, + scope: ReadonlySet, + push: (ref: UnresolvedReference) => void, +): void { + if (node.type === 'type_identifier') { + // `p.T` is the type an import names, whatever is declared as `T` here. + const parameter = scope.has(node.text) && node.previousSibling?.type !== '.'; + if (!parameter && isDartTypeName(node)) { + push({ + fromNodeId, + referenceName: node.text, + referenceKind: 'references', + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + } + return; + } + const inner = dartTypeParameterScope(node, scope); + for (const child of node.namedChildren) pushDartAliasedTypeRefs(child, fromNodeId, inner, push); +} + +/** + * A `references` ref from the typedef `node` for every type its aliased type + * names (#2327): a function type's return and parameter types, generic + * arguments, record fields, a type parameter's bound, and the legacy form's + * return and parameter types. A type parameter, the typedef's own or a + * generic function type's (`T Function(T a)`), means that parameter and no + * type of the project's. The typedef's annotations are not its type. + */ +export function pushDartTypeAliasRefs( + node: SyntaxNode, + fromNodeId: string, + push: (ref: UnresolvedReference) => void, +): void { + const name = dartTypeAliasName(node); + const scope = dartTypeParameterScope(node, new Set()); + for (const child of node.namedChildren) { + if (child.type === 'annotation' || (name && child.equals(name))) continue; + pushDartAliasedTypeRefs(child, fromNodeId, scope, push); + } +} + /** The bodies a Dart field `declaration` can sit in: a class's or mixin's, an extension's, an enum's. */ const DART_MEMBER_BODIES: ReadonlySet = new Set(['class_body', 'extension_body', 'enum_body']); @@ -646,6 +722,9 @@ export const dartExtractor: LanguageExtractor = { return false; }, resolveName: (node) => { + // A typedef has no `name` field, and a legacy one writes its return type + // before its name (`typedef Report LegacyMaker(int n);`). + if (node.type === 'type_alias') return dartTypeAliasName(node)?.text; // `class A = B with C;` — a mixin application — names its class inside // the `mixin_application_class`, not in a `name` field. if (node.type === 'class_definition') { diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index ee757ab08d..2cffc3a495 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -26,7 +26,7 @@ import { NestedIntervals, scanCppBraceScopes, type CppBraceScopes } from './lang import { appendCppMacroMethods } from './languages/cpp-macro-methods'; import { rustImplItemOwner, rustImplTypeName } from './languages/rust'; import { goAliasTypeNames, goEmbeddedTypeName, goInterfaceMethodTypeNames, goStructFieldTypeNames } from './languages/go'; -import { dartMisparsedGenericCall, dartReceiverOf, isDartTypeName, pushDartTypeRefs } from './languages/dart'; +import { dartMisparsedGenericCall, dartReceiverOf, isDartTypeName, pushDartTypeAliasRefs, pushDartTypeRefs } from './languages/dart'; import { LiquidExtractor } from './liquid-extractor'; import { RazorExtractor } from './razor-extractor'; import { SvelteExtractor } from './svelte-extractor'; @@ -4246,6 +4246,12 @@ export class TreeSitterExtractor { column: type.startPosition.column, }); } + } else if (this.language === 'dart') { + // A Dart typedef has no fields either: the type it aliases is the + // rest of the declaration, `void Function(Report r)` in `typedef + // ReportCallback = void Function(Report r);`, or a legacy one's + // return and parameter types (#2327). + pushDartTypeAliasRefs(node, typeAliasNode.id, (ref) => this.unresolvedReferences.push(ref)); } } return false; From efff745783191ebd8edb6db0dd1ed410cf1a9bbb Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 21 Oct 2026 21:46:24 +0000 Subject: [PATCH 07/95] fix(go): a field typed by a dot-imported package resolves in that package (#2764) A Go type written without a package qualifier was looked up only in the file's own package. In a file that dot-imports a project package, like harbor's config tests (`import . ".../controller/config"`, then a `controller Controller` field), the type isn't there, so the call `c.controller.AllConfigs()` fell back to name matching. Name matching took the same-named type of a package the file doesn't import, the one that sorts first: matchGoFieldChainCall used the struct's own directory for a bare field type, resolveGoMethodInPackage found no such type there, and resolveMethodOnType picked the first of every `Controller::AllConfigs`. The same gap affected calls through parameters, locals and package-level variables of such a type (one set by a conversion), through an alias of it, and through methods promoted from an embedded field. It also affected method values, type references, and the embedding edge itself where the namesake was as near. The field chain, the typed-receiver and method-value package lookup, embedded types, alias targets, package-level values and Go type positions now ask goBareTypeDir. It reads a bare name as Go does: the file's own package first, then a project package the file dot-imports that declares it. An exported name that only a dot import of a package outside the project's modules can supply (`. "github.com/onsi/gomega"`) names none of the project's types. The exception is a type that one of the file's functions declares, which the index holds no node for: name matching still handles it, as before. A bare embedding is retargeted only when a dot import decides, and files without dot imports resolve exactly as before. Tests: go-dot-import-field-type.test.ts has 10 cases on both the kernel and wasm backends. 18 tests fail on main and 2 guards pass. All go-*, goframe, gin-*, frameworks*, resolution*, sync* and dead-code test files pass (65 files, 1,140 tests). tsc and the build are clean. Validation compares main bf6f1fa6 with this commit, each built with a kernel from its own sources. Each repo was indexed from scratch and diffed by call site: - gomega 820ad4d5: 30 calls retargeted from internal.Gomega's Expect to types.Gomega's, the type the dot-imported root package's `Gomega` alias names. All 30 were checked against source. - ginkgo 7ff96d7d: 135 calls added and 1,022 confirmed (same target, 0.7 to 0.9), all through package-level vars typed by the dot-imported internal/test_helpers. 9 retargeted (8 composite literals to the root package's SemVerConstraints aliases, 1 Status call to the fake handler). None removed. - harbor src/ 4e07c69c and kubernetes d2c4b85b: byte-identical dumps. Their dot-imported types' methods have no namesakes, so main already linked them. - Controls gin 0f09c3a9 and zustand d7a5583c (TypeScript): byte-identical. - Index time and DB size are within noise. On kubernetes, the median of 3 interleaved rounds was 151.5 s on main and 148.7 s with the fix. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/go-dot-import-field-type.test.ts | 398 +++++++++++++++++++++ src/resolution/index.ts | 5 +- src/resolution/name-matcher.ts | 142 ++++++-- 4 files changed, 511 insertions(+), 35 deletions(-) create mode 100644 __tests__/go-dot-import-field-type.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f93154ed2..92bbf5895b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Dart, a type is now linked from more of the places it is written, not only from parameters and return types: the type an `extension … on` targets, a field's type, a top-level variable's type, generic arguments such as `Future.value(null)` or Riverpod's `final reportProvider = Family()`, and a local variable's type, a cast or a type check inside a function. So `callers` and impact for a model class now reach the extensions, models and providers that use it. Thanks @mg-mg-mg for the report. (#2327) - In Dart, a `typedef` now links to the types it is built from, like `Report` in `typedef ReportCallback = void Function(Report r);` or `typedef Reports = List;`, so `callers` and impact for a model class also reach the callback and collection types defined on it. An old-style typedef with a return type, like `typedef Widget ItemBuilder(BuildContext context);`, is now indexed under its own name instead of its return type's, so code that only uses `Widget` no longer links to it. Thanks @mg-mg-mg for the report. (#2327) - In Go, calls through a method receiver or parameter of an unexported type, the usual shape of gRPC and HTTP handlers (`s.service.AddItem()` inside `func (s *server) Create()`), now resolve, and always within that type's own package: another package's `server` with a same-named method no longer takes the call, and a method the type gets from a struct or interface it embeds is found too. Re-index Go projects after upgrading. Thanks @GoDiao for the report and the fix. (#2323) +- In Go, a type a file gets from a dot import, like `Controller` in harbor's config tests after `import . "…/controller/config"`, is now looked up in that package, so `c.controller.AllConfigs()` on a `controller Controller` field, a call through a parameter, variable, alias or embedded field of that type, and the type's own links reach that package's `Controller` instead of a same-named type in a package the file doesn't import. A name the file can only get from a dot import of a package outside your project, like `. "github.com/onsi/gomega"`, no longer links to a project type that shares it. Re-index Go projects after upgrading. Thanks @GoDiao. (#2323) - In Go, a call made through a type assertion, like `srv.(KVServer).Range(ctx, in)` in generated gRPC code, `c.Reader.(*pipe).Close()` or `v.(storage.Store).Fetch(key)`, now links to that method of the asserted type, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, it linked to whichever method shared the name, so every gRPC handler pointed at the `Unimplemented…Server` stub instead of the server interface. Re-index Go projects after upgrading. - In Go, a call through a variable that holds the result of a type assertion, like `st.Fetch(key)` after `st := v.(storage.Store)` or `rc.Read(p)` after `rc, ok := v.(ReadCloser)`, now links to that method of the asserted type, as a call made through the assertion itself does, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, such a call linked to whichever method shared the name or to nothing, and a variable named like a standard-library package, like grpc-go's `parser, ok := builder.(balancer.ConfigParser)`, was taken for that package. Re-index Go projects after upgrading. - In Go, a method called on an element of a slice, array or map, like kubectl's `schemas[ix].ValidateBytes(data)`, `pods[i].DeepCopy()`, `m.Items[len(m.Items)-1].Unmarshal(…)` in generated protobuf code or `(*in)[i].DeepCopyInto(&(*out)[i])` in every deepcopy file, now links to that method of the element's type. The type is read from what is indexed: a parameter, a variable, a struct field or a named slice type like `type ConjunctiveSchema []Schema`, and an element held by pointer, like `[]*Item`, reaches `Item`'s methods. Before, such a call linked to whichever method shared its name, often one in an unrelated package, or to nothing when many types declare it, as with Kubernetes' `DeepCopyInto`. An element whose type comes from outside your project, like `[]*net.IPNet`, or is a built-in one like `[]error`, now links to nothing rather than to a project method of that name. Re-index Go projects after upgrading. diff --git a/__tests__/go-dot-import-field-type.test.ts b/__tests__/go-dot-import-field-type.test.ts new file mode 100644 index 0000000000..69e0626465 --- /dev/null +++ b/__tests__/go-dot-import-field-type.test.ts @@ -0,0 +1,398 @@ +/** + * A Go type written without a package qualifier is the file's own package's, + * or one of a package the file dot-imports (#2323): + * + * import . "example.com/app/config" + * + * type controllerSuite struct { + * controller Controller // config.Controller + * } + * + * func (c *controllerSuite) GetAll() { + * c.controller.AllConfigs() // config.Controller's, not aaa.Controller's + * } + * + * The type was looked up in the file's own package alone. Not found there, + * the call went to name matching, which took the same-named type of a + * package the file doesn't import — the one that sorts first. harbor's config + * tests have this shape. The same held for a parameter, a local or a + * package-level variable of such a type, an alias of it, a method promoted + * from an embedded one and the embedding itself. A type of the file's own + * package of the name comes first; a name the file can only have from a + * package outside the project's modules (`. "github.com/onsi/gomega"`) has + * none of the project's methods, unless a function of the file declares a + * type of that name, which hides the dot import's. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +/** + * The module the tests index. `aaa` declares a namesake of every type and + * sorts ahead of `config`, where name matching looked first. + */ +const FILES: Record = { + 'go.mod': 'module example.com/app\n\ngo 1.22\n', + 'aaa/aaa.go': `package aaa + +type Controller interface { + AllConfigs() map[string]any + UserConfigs() map[string]any +} + +type Manager struct{} + +func (m *Manager) Save() error { return nil } + +type Level int + +func (l Level) Label() string { return "" } + +type Gomega interface { + Expect(actual any) bool +} +`, + 'config/controller.go': `package config + +type Controller interface { + AllConfigs() map[string]any + UserConfigs() map[string]any +} + +type Manager struct{} + +func (m *Manager) Save() error { return nil } + +type Level int + +func (l Level) Label() string { return "" } +`, + // The issue's file, and the other ways a file can use a dot-imported type. + 'config/test/suite.go': `package test + +import ( + "testing" + + . "example.com/app/config" +) + +type controllerSuite struct { + t *testing.T + controller Controller + mgr *Manager +} + +func (c *controllerSuite) GetAll() { + c.controller.AllConfigs() +} + +func (c *controllerSuite) Persist() error { + return c.mgr.Save() +} + +func ViaParam(s *controllerSuite) { + s.controller.UserConfigs() +} + +func Param(ctl Controller) { + ctl.AllConfigs() +} + +func Local() { + var ctl Controller + ctl.UserConfigs() + var m *Manager + m.Save() +} + +func Indexed(ctls []Controller) { + ctls[0].AllConfigs() +} + +type embedded struct { + Controller + *Manager +} + +func (e *embedded) Run() { + e.AllConfigs() + e.Save() +} + +var shared Controller + +var level = Level(2) + +func Shared() string { + shared.AllConfigs() + return level.Label() +} + +type Ctl = Controller + +func Aliased(c Ctl) { + c.UserConfigs() +} + +func register(fn any) {} + +func Values(ctl Controller, c *controllerSuite) { + register(ctl.AllConfigs) + register(c.controller.UserConfigs) +} +`, + // A file whose own package declares the name too: Go rejects the clash, + // but the package's own type is the one the index reads first. Its + // directory is as far from aaa as from config. + 'own/own.go': `package own + +import . "example.com/app/config" + +type Controller interface { + AllConfigs() map[string]any +} + +type suite struct { + controller Controller +} + +func (s *suite) Get() { + s.controller.AllConfigs() +} + +func Param(ctl Controller) { + ctl.AllConfigs() +} + +func Uses(m *Manager) error { + return m.Save() +} + +type wrapper struct { + *Manager +} + +func (w *wrapper) Keep() error { + return w.Save() +} +`, + // A dot import of a package from outside the project's modules: a name + // its own package doesn't declare may be that package's. + 'ext/ext.go': `package ext + +import ( + . "github.com/onsi/gomega" +) + +type matcherSuite struct { + g Gomega +} + +func (m *matcherSuite) Check() { + m.g.Expect(1) +} + +func Param(g Gomega) { + g.Expect(2) +} + +func Many(gs []Gomega) { + gs[0].Expect(3) +} +`, + // ginkgo's spec context test: a type a function declares hides the dot + // import's names, and the index holds no node for it. + 'ext/local.go': `package ext + +import ( + . "github.com/onsi/gomega" +) + +type progressManager struct{} + +func (p *progressManager) QueryProgressReporters() []string { return nil } + +func Attached(v any) { + type CompleteSpecContext interface { + QueryProgressReporters() []string + } + ctx := v.(CompleteSpecContext) + ctx.QueryProgressReporters() +} +`, +}; + +describe.each(['default', 'wasm'])('Go types written bare are read in the packages the file dot-imports too (%s)', (backend) => { + let root = ''; + let cg: CodeGraph | undefined; + let kernel: string | undefined; + + beforeAll(async () => { + kernel = process.env.CODEGRAPH_KERNEL; + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-go-dot-import-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + // One file with a Windows checkout's line endings. + fs.writeFileSync(path.join(root, rel), rel === 'own/own.go' ? content.replace(/\n/g, '\r\n') : content); + } + cg = await CodeGraph.init(root, { index: true }); + }, 60_000); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + if (kernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernel; + }); + + /** The node `qualifiedName` in `file`. */ + function nodeOf(file: string, qualifiedName: string) { + const node = cg!.getNodesInFile(file).find((n) => n.qualifiedName === qualifiedName && n.kind !== 'parameter'); + expect(node, `${qualifiedName} in ${file}`).toBeDefined(); + return node!; + } + + /** + * ` -> ::` for every edge + * of `kind` the function or method `name` in `file` has to a function or + * method. + */ + function links(file: string, name: string, kind: 'calls' | 'references' = 'calls'): string[] { + const lines = FILES[file]!.split('\n'); + return cg! + .getOutgoingEdges(nodeOf(file, name).id) + .filter((e) => e.kind === kind) + .map((e) => ({ e, target: cg!.getNode(e.target)! })) + .filter(({ target }) => target.kind === 'method' || target.kind === 'function') + .map(({ e, target }) => `${lines[e.line! - 1]!.trim()} -> ${target.filePath}::${target.qualifiedName}`) + .sort(); + } + + /** ` ::` for each edge of `kinds` from `name` in `file` to a type, as resolved (no synthesized edge). */ + function typeLinks(file: string, name: string, kinds: string[]): string[] { + return cg! + .getOutgoingEdges(nodeOf(file, name).id) + .filter((e) => kinds.includes(e.kind) && e.provenance !== 'heuristic') + .map((e) => ({ e, target: cg!.getNode(e.target)! })) + .filter(({ target }) => ['interface', 'struct', 'type_alias'].includes(target.kind)) + .map(({ e, target }) => `${e.kind} ${target.filePath}::${target.qualifiedName}`) + .sort(); + } + + it("resolves a call through a field typed by a dot-imported package's type", () => { + // The issue: name matching took aaa's Controller, which sorts first. + expect(links('config/test/suite.go', 'controllerSuite::GetAll')).toEqual([ + 'c.controller.AllConfigs() -> config/controller.go::Controller::AllConfigs', + ]); + expect(links('config/test/suite.go', 'controllerSuite::Persist')).toEqual([ + 'return c.mgr.Save() -> config/controller.go::Manager::Save', + ]); + expect(links('config/test/suite.go', 'ViaParam')).toEqual([ + 's.controller.UserConfigs() -> config/controller.go::Controller::UserConfigs', + ]); + }); + + it('resolves a call through a parameter, a local or a package-level variable of such a type', () => { + expect(links('config/test/suite.go', 'Param')).toEqual([ + 'ctl.AllConfigs() -> config/controller.go::Controller::AllConfigs', + ]); + expect(links('config/test/suite.go', 'Local')).toEqual([ + 'ctl.UserConfigs() -> config/controller.go::Controller::UserConfigs', + 'm.Save() -> config/controller.go::Manager::Save', + ]); + expect(links('config/test/suite.go', 'Indexed')).toEqual([ + 'ctls[0].AllConfigs() -> config/controller.go::Controller::AllConfigs', + ]); + // `var level = Level(2)` converts to the dot-imported type. + expect(links('config/test/suite.go', 'Shared')).toEqual([ + 'return level.Label() -> config/controller.go::Level::Label', + 'shared.AllConfigs() -> config/controller.go::Controller::AllConfigs', + ]); + }); + + it('follows an alias of such a type to it', () => { + expect(links('config/test/suite.go', 'Aliased')).toEqual([ + 'c.UserConfigs() -> config/controller.go::Controller::UserConfigs', + ]); + }); + + it('resolves a method promoted from an embedded field of such a type, and links the embedding to it', () => { + expect(links('config/test/suite.go', 'embedded::Run')).toEqual([ + 'e.AllConfigs() -> config/controller.go::Controller::AllConfigs', + 'e.Save() -> config/controller.go::Manager::Save', + ]); + expect(links('own/own.go', 'wrapper::Keep')).toEqual(['return w.Save() -> config/controller.go::Manager::Save']); + expect(typeLinks('config/test/suite.go', 'embedded', ['extends', 'implements'])).toEqual([ + 'extends config/controller.go::Manager', + 'implements config/controller.go::Controller', + ]); + expect(typeLinks('own/own.go', 'wrapper', ['extends', 'implements'])).toEqual(['extends config/controller.go::Manager']); + }); + + it('links a method value taken through such a receiver as the call would', () => { + expect(links('config/test/suite.go', 'Values', 'references')).toEqual([ + 'register(c.controller.UserConfigs) -> config/controller.go::Controller::UserConfigs', + 'register(ctl.AllConfigs) -> config/controller.go::Controller::AllConfigs', + ]); + }); + + it("links a type written bare to the dot-imported package's type", () => { + // A struct's field types, a parameter's type: the dot import's even + // where another package's namesake is as near. + expect(typeLinks('config/test/suite.go', 'controllerSuite', ['references'])).toEqual([ + 'references config/controller.go::Controller', + 'references config/controller.go::Manager', + ]); + expect(typeLinks('config/test/suite.go', 'Param', ['references'])).toEqual(['references config/controller.go::Controller']); + expect(typeLinks('own/own.go', 'Uses', ['references'])).toEqual(['references config/controller.go::Manager']); + }); + + it("reads the file's own package's type of the name first", () => { + expect(links('own/own.go', 'suite::Get')).toEqual(['s.controller.AllConfigs() -> own/own.go::Controller::AllConfigs']); + expect(links('own/own.go', 'Param')).toEqual(['ctl.AllConfigs() -> own/own.go::Controller::AllConfigs']); + expect(typeLinks('own/own.go', 'suite', ['references'])).toEqual(['references own/own.go::Controller']); + // A name its own package lacks is still the dot import's. + expect(links('own/own.go', 'Uses')).toEqual(['return m.Save() -> config/controller.go::Manager::Save']); + }); + + it('makes no guess onto a project type through a dot import from outside the project', () => { + expect(links('ext/ext.go', 'matcherSuite::Check')).toEqual([]); + expect(links('ext/ext.go', 'Param')).toEqual([]); + expect(links('ext/ext.go', 'Many')).toEqual([]); + expect(typeLinks('ext/ext.go', 'matcherSuite', ['references'])).toEqual([]); + expect(typeLinks('ext/ext.go', 'Param', ['references'])).toEqual([]); + }); + + it('leaves a type a function declares to name matching, as before, beside such a dot import', () => { + expect(links('ext/local.go', 'Attached')).toEqual([ + 'ctx.QueryProgressReporters() -> ext/local.go::progressManager::QueryProgressReporters', + ]); + }); + + it('never links a same-named type of a package the file does not import', () => { + const decoys = cg!.getNodesInFile('aaa/aaa.go').filter((n) => n.kind !== 'file'); + expect(decoys.map((n) => n.qualifiedName).sort()).toEqual([ + 'Controller', + 'Controller::AllConfigs', + 'Controller::UserConfigs', + 'Gomega', + 'Gomega::Expect', + 'Level', + 'Level::Label', + 'Manager', + 'Manager::Save', + ]); + for (const decoy of decoys) { + // A struct whose promoted methods satisfy an interface does implement + // it, whichever package declares it: those edges are synthesized. + expect( + cg!.getIncomingEdges(decoy.id).filter((e) => e.kind !== 'contains' && e.provenance !== 'heuristic'), + decoy.qualifiedName, + ).toEqual([]); + } + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 24872d9e70..8c5a5899b7 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -3653,6 +3653,9 @@ export class ReferenceResolver { // imported unaliased as `go.etcd.io/etcd/client/v3`), a type found by its // bare name is some other package's namesake. if (ref.language === 'go' && isGoUnknownQualified(ref, this.context)) return null; + // One embedded bare that only a package the file dot-imports declares is + // that package's, not a namesake as near the file. + if (ref.language === 'go') return goDotImportedSupertype(result, ref, this.context); return result; } diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 215a06c2a3..ee8e44eeef 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -2538,9 +2538,11 @@ export function isGoUnknownQualified(ref: UnresolvedRef, context: ResolutionCont * literal's type — names: a type, which Go reads from one package. A method * or a function is never it; Go reaches those only through a value or a * package. Whichever strategy found a declaration of the name, the type is - * the one of that name in the reference's own package for a bare name, or in - * the imported project package for `pkg.T`. Without one there, a method or - * function of the name is nothing the reference means. etcd's + * the one of that name in the reference's own package for a bare name (or in + * a package its file dot-imports, goBareTypeDir), or in the imported project + * package for `pkg.T`. Without one there, a method or function of the name is + * nothing the reference means, nor is any type a bare name only a package + * from outside the project the file dot-imports can give it. etcd's * `func (ti *treeIndex) KeyIndex(keyi *keyIndex) *keyIndex` linked both * `keyIndex` types to the method `treeIndex.keyIndex` beside it, * prometheus's `(ec2Client, error)` result to the method the line declares, @@ -2566,8 +2568,10 @@ export function goTypePositionTarget(result: ResolvedRef, ref: UnresolvedRef, co } else { const { written, imported } = goRefQualification(ref, context); bare = written === undefined; - if (bare) pkgDir = goPackageDir(ref.filePath); - else if (imported) pkgDir = context.getGoPackageDir?.(imported.source, ref.filePath) ?? null; + if (bare) { + pkgDir = goBareTypeDir(name, ref.filePath, context); + if (pkgDir === null) return null; + } else if (imported) pkgDir = context.getGoPackageDir?.(imported.source, ref.filePath) ?? null; } if (isType && (!bare || goPackageDir(target.filePath) === pkgDir)) return result; const types = pkgDir ? goPackageTypes(name, pkgDir, context) : []; @@ -19916,15 +19920,19 @@ function goImportPackageDir(qualifier: string, filePath: string, context: Resolu /** * The package directory that declares a Go type spelled `raw` in `filePath`: - * the file's own package for a bare name (`server`, `*Store`), the imported - * package for a qualified one (`store.Manager`) when it is a project package. + * the file's own package for a bare name (`server`, `*Store`), or a package + * the file dot-imports that declares it (goBareTypeDir), the imported package + * for a qualified one (`store.Manager`) when it is a project package. * Undefined when that can't be told, which leaves resolution as it was. */ function goDeclaredTypePackage(raw: string | undefined, filePath: string, context: ResolutionContext): string | undefined { if (!raw) return undefined; const name = raw.replace(/[*&\s]/g, ''); const dot = name.indexOf('.'); - if (dot < 0) return goPackageDir(filePath); + if (dot < 0) { + const bare = /^[A-Za-z_]\w*/.exec(name)?.[0]; + return bare ? goBareTypeDir(bare, filePath, context) ?? undefined : goPackageDir(filePath); + } return goImportPackageDir(name.slice(0, dot), filePath, context) ?? undefined; } @@ -20021,7 +20029,7 @@ function goAliasTarget(typeNode: Node, context: ResolutionContext): { name: stri const m = head && /^\s*(?:\*\s*)?([A-Za-z_]\w*)(?:\s*\.\s*([A-Za-z_]\w*))?\s*(?:\[.*\])?\s*;?\s*$/.exec(decl.slice(head[0].length)); let target: { name: string; pkgDir: string } | null | undefined; if (m) { - const pkgDir = m[2] ? goImportPackageDir(m[1]!, typeNode.filePath, context) : goPackageDir(typeNode.filePath); + const pkgDir = m[2] ? goImportPackageDir(m[1]!, typeNode.filePath, context) : goBareTypeDir(m[1]!, typeNode.filePath, context); const name = m[2] ?? m[1]!; target = pkgDir == null || (!m[2] && GO_BUILTIN_FIELD_TYPES.has(name)) ? null : { name, pkgDir }; } else { @@ -20069,7 +20077,7 @@ function goEmbeddedTypes(typeNode: Node, context: ResolutionContext): Array<{ na for (const member of body.split(/[\n;]/)) { const m = /^\s*\*?\s*([A-Za-z_]\w*)(?:\.([A-Za-z_]\w*))?\s*(?:\[[^\]]*\])?\s*(?:`[^`]*`|"[^"]*")?\s*$/.exec(member); if (!m) continue; - const pkgDir = m[2] ? goImportPackageDir(m[1]!, typeNode.filePath, context) : goPackageDir(typeNode.filePath); + const pkgDir = m[2] ? goImportPackageDir(m[1]!, typeNode.filePath, context) : goBareTypeDir(m[1]!, typeNode.filePath, context); if (pkgDir == null) continue; embedded.push({ name: m[2] ?? m[1]!, pkgDir }); } @@ -20125,11 +20133,12 @@ export function matchGoAssertedCall(ref: UnresolvedRef, context: ResolutionConte * declares it, found where Go finds it: in the file's own package or a * package it dot-imports for a bare name, in the imported package for a * qualified one. Null for a type found in none — one from outside the - * project (`http.Flusher`), a predeclared one (`error`) or an alias of such a - * type: none has a method the project declares. Undefined for one it can't - * tell: a type literal (`interface{ Flush() }`), a name no package of the - * project declares (a type declared inside a function, a type parameter), a - * type the imported project package doesn't hold. + * project (`http.Flusher`, or a bare name only a package from outside it + * that the file dot-imports can give it), a predeclared one (`error`) or an + * alias of such a type: none has a method the project declares. Undefined + * for one it can't tell: a type literal (`interface{ Flush() }`), a name no + * package of the project declares (a type declared inside a function, a type + * parameter), a type the imported project package doesn't hold. */ function goWrittenType(written: string, ref: UnresolvedRef, context: ResolutionContext): GoProjectType | null | undefined { // `KVServer`, `*pipe`, `storage.Store`, `*List[int]`: anything else is a type literal. @@ -20143,8 +20152,9 @@ function goWrittenType(written: string, ref: UnresolvedRef, context: ResolutionC if (!dir) return null; if (!declares(dir)) return undefined; } else { - dir = [goPackageDir(ref.filePath), ...goDotImportDirs(ref.filePath, context)].find(declares); - if (!dir) return GO_BUILTIN_FIELD_TYPES.has(name!) ? null : undefined; + dir = goBareTypeDir(name!, ref.filePath, context); + if (dir === null) return null; + if (!declares(dir)) return GO_BUILTIN_FIELD_TYPES.has(name!) ? null : undefined; } // An alias of a type from outside the project (`type Ctx = context.Context`) // has that type's methods, none the project declares. @@ -20338,7 +20348,8 @@ function matchGoPackageValueCall(receiver: string, method: string, ref: Unresolv const dot = named.indexOf('.'); const name = named.slice(dot + 1); if (dot < 0 && GO_BUILTIN_FIELD_TYPES.has(name)) return null; - const pkgDir = dot < 0 ? goPackageDir(type.file) : goImportPackageDir(named.slice(0, dot), type.file, context); + const pkgDir = dot < 0 ? goBareTypeDir(name, type.file, context) : goImportPackageDir(named.slice(0, dot), type.file, context); + if (pkgDir === null && dot < 0) return null; if (!pkgDir || goPackageTypes(name, pkgDir, context).length === 0) return undefined; // A project type whose methods aren't found is left to the strategies // below: kubernetes' `var UnstructuredJSONScheme runtime.Codec = @@ -20402,7 +20413,7 @@ function goValueType(value: string, filePath: string, context: ResolutionContext const written = qualifier !== undefined ? `${qualifier}.${name}` : name!; if (open === '{') return GO_TYPE_KEYWORDS.has(name!) ? null : { written, file: filePath }; // A call is a conversion when it names a type of its package. - const dir = qualifier !== undefined ? goImportPackageDir(qualifier, filePath, context) : goPackageDir(filePath); + const dir = qualifier !== undefined ? goImportPackageDir(qualifier, filePath, context) : goBareTypeDir(name!, filePath, context); return dir && goPackageTypes(name!, dir, context).length > 0 ? { written, file: filePath } : undefined; } @@ -20564,24 +20575,83 @@ function goClosingBracket(text: string, open: number): number { return -1; } -const GO_DOT_IMPORTS = new WeakMap>(); +/** The packages a Go file dot-imports. */ +interface GoDotImports { + /** The project's, by directory: their exported names the file reads bare. */ + dirs: string[]; + /** Whether one is a package from outside the project's modules (`. "github.com/onsi/gomega"`). */ + outside: boolean; + /** The type names the file's text declares, inside its functions too: read when first needed. */ + types?: Set; +} -/** The project packages a Go file dot-imports (`import . "example.com/app/storage"`), by directory: their names it reads bare. */ -function goDotImportDirs(filePath: string, context: ResolutionContext): string[] { +const GO_DOT_IMPORTS = new WeakMap>(); + +/** The packages a Go file dot-imports (`import . "example.com/app/storage"`). */ +function goDotImports(filePath: string, context: ResolutionContext): GoDotImports { let memo = GO_DOT_IMPORTS.get(context); if (!memo) GO_DOT_IMPORTS.set(context, (memo = new Map())); - let dirs = memo.get(filePath); - if (dirs) return dirs; - dirs = []; + let found = memo.get(filePath); + if (found) return found; + found = { dirs: [], outside: false }; for (const line of context.getFileLines?.(filePath) ?? context.readFile(filePath)?.split(/\r?\n/) ?? []) { // The imports come before any other declaration. if (/^(?:func|type|var|const)\b/.test(line)) break; const dot = /^\s*(?:import\s+)?\.\s+"([^"]+)"/.exec(line); - const dir = dot ? context.getGoPackageDir?.(dot[1]!, filePath) : null; - if (dir) dirs.push(dir); + if (!dot || !context.getGoPackageDir) continue; + const dir = context.getGoPackageDir(dot[1]!, filePath); + if (dir) found.dirs.push(dir); + // A relative or `/internal/` path is the project's own, mapped or not. + else if (!dot[1]!.startsWith('.') && !dot[1]!.includes('/internal/')) found.outside = true; + } + memo.set(filePath, found); + return found; +} + +/** + * The directory of the package that declares a Go type written bare in + * `filePath` (`Controller`, `*Manager`), as Go reads the name: the file's own + * package, else a package of the project the file dot-imports that declares + * it (harbor's config tests read `Controller` from `. ".../controller/config"`). + * Null for an exported name that only a package from outside the project the + * file dot-imports can give it (`. "github.com/onsi/gomega"`): no type of the + * project's. The file's own package when no package declares the name, as + * before: a type the index doesn't hold there, like one a function declares + * (ginkgo's `type CompleteSpecContext interface {…}` inside a test), which + * hides a dot import's. + */ +function goBareTypeDir(name: string, filePath: string, context: ResolutionContext): string | null { + const own = goPackageDir(filePath); + const dots = goDotImports(filePath, context); + if ((dots.dirs.length === 0 && !dots.outside) || goPackageTypes(name, own, context).length > 0) return own; + const dir = dots.dirs.find((d) => goPackageTypes(name, d, context).length > 0); + if (dir !== undefined) return dir; + if (!dots.outside || !/^[A-Z]\w*$/.test(name)) return own; + if (!dots.types) { + dots.types = new Set(); + for (const line of context.getFileLines?.(filePath) ?? context.readFile(filePath)?.split(/\r?\n/) ?? []) { + for (const m of line.replace(/\/\/.*$/, '').matchAll(/(?:^|[^\w.])type\s+([A-Za-z_]\w*)/g)) dots.types.add(m[1]!); + } } - memo.set(filePath, dirs); - return dirs; + return dots.types.has(name) ? own : null; +} + +/** + * What a Go embedding written bare (`*Manager`) names when the file's own + * package doesn't declare the name: the type of the package the file + * dot-imports that does (goBareTypeDir), not another package's namesake name + * matching found as near the file, or nothing for a name only a package from + * outside the project can give the file. Any other is left as it is. + */ +export function goDotImportedSupertype(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const name = ref.referenceName; + if (!/^[A-Za-z_]\w*$/.test(name)) return result; + const dir = goBareTypeDir(name, ref.filePath, context); + if (dir === goPackageDir(ref.filePath) || goRefQualification(ref, context).written !== undefined) return result; + if (dir === null) return null; + const target = context.getNodeById?.(result.targetNodeId); + if (target && goPackageDir(target.filePath) === dir) return result; + return { ...result, targetNodeId: preferCallSiteFile(goPackageTypes(name, dir, context), ref.filePath)[0]!.id }; } /** @@ -20814,8 +20884,8 @@ function goNamedTypeDecls(t: GoTypeText, context: ResolutionContext): { name: st if (second === undefined) { if (t.params?.has(name)) return undefined; if (GO_BUILTIN_FIELD_TYPES.has(name)) return null; - dir = [goPackageDir(t.file), ...goDotImportDirs(t.file, context)].find((d) => goPackageTypes(name, d, context).length > 0); - if (dir === undefined) return undefined; + dir = goBareTypeDir(name, t.file, context); + if (dir === null) return null; } else { dir = goImportPackageDir(first!, t.file, context); if (dir === undefined) return undefined; @@ -22481,9 +22551,13 @@ function matchGoFieldChainCall( const m = line.match(fieldTypeRe); if (!m || !m[1]) continue; const rawType = m[1]; - // The package that declares the field's type: the struct's own for a - // bare type name, the imported one for a package-qualified name. - let pkgDir = goPackageDir(s.filePath); + // The package that declares the field's type: for a bare type name the + // struct's own, or one its file dot-imports (harbor's config tests read + // `controller Controller` from `. ".../controller/config"`) — none of + // the project's for a name only a package from outside it can give the + // file — the imported one for a package-qualified name. + let pkgDir = rawType.includes('.') ? undefined : goBareTypeDir(rawType, s.filePath, context); + if (pkgDir === null) continue; // A package-qualified field type (`http.Handler`, `sql.DB`) is only // followed when the package is one of the project's: stripping the // qualifier and matching the bare name would conflate a stdlib or From d6e8c91e0a9c23cba3b549fa2b3facad65541aaa Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 00:33:16 +0000 Subject: [PATCH 08/95] fix(installer): upgrade keeps everything around CodeGraph's section in instructions files (#2765) Root cause: the write that refreshes CodeGraph's marker-fenced section in an agent's instructions file (CLAUDE.md, AGENTS.md, GEMINI.md) located the section with two independent searches: the first start marker anywhere in the file and the first end marker anywhere in the file. Everything between them was treated as CodeGraph's and replaced. A stray start marker above the real section (what is left when part of an older section is deleted by hand, or the marker quoted in the user's own notes) therefore stretched the "section" over the user's text, and when that marker was the first line the file came back as just the block, which is the report. The `codegraph upgrade` refresh (`codegraph install --refresh`) and plain `codegraph install` both go through this write, for the Claude Code, Codex, Gemini and opencode targets. The same pairing made a lone end marker look like "no section", so every run appended another block, and left a second block stale. Uninstall (and the npm preuninstall hook) and the Cursor rules-file cleanup used the same pairing to delete the same span. Fix: one parser finds the well-formed sections. Each end marker closes the nearest start marker before it, and a marker counts only when it is alone on its line, as every block codegraph has written puts it. A stray start or end marker is left as plain text. The refresh rewrites only the bytes between the first section's markers, using the line breaks that section already has, so a CRLF file stays CRLF and a current CRLF block is unchanged. Further sections are duplicates and are removed, keeping the text around them. With no section yet the block is appended after the existing bytes, which are no longer trimmed. Uninstall and the Cursor cleanup strip only well-formed sections. Validation: - 18 new tests (17 in installer-targets.test.ts: exact bytes for a stray start marker on line 1, a stray start marker with no section after it, a marker quoted in prose, a lone end marker, two sections, CRLF, no final newline, appending, a BOM, uninstall; a refresh plus uninstall per target for Claude Code, Codex, Gemini and opencode; the Cursor rules file. 1 in upgrade.test.ts: an npm upgrade whose refresh runs the built CLI against a private home). 16 fail on main's sources (the other 2 are guards that hold on both), all pass with the fix. - The real CLI on this Windows machine against private temp homes: `codegraph install --refresh`, 14 file shapes x 4 targets x 2 runs. Main had 28 bad results of 112 (the user's text lost in 3 shapes, a block added on every run after a lone end marker, a stale second block kept), the fix 0. With the fix, the bytes outside the section are identical in every single-section shape, and the input stays a byte prefix in every append. - `codegraph install --yes` and `codegraph uninstall --yes --keep-cli` on the reported shape: main kept none of the notes (install left only the 802-byte block, uninstall left 19 bytes); the fix kept them all. Fixes #2506 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/installer-targets.test.ts | 201 ++++++++++++++++++++++++++++ __tests__/upgrade.test.ts | 71 ++++++++++ src/installer/targets/cursor.ts | 20 +-- src/installer/targets/shared.ts | 179 ++++++++++++++++++++----- 5 files changed, 430 insertions(+), 42 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 92bbf5895b..57a82ae6e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - VB.NET designer code such as `New System.Drawing.Point(…)` or `Me.Size = New System.Drawing.Size(…)` is no longer linked to a project type or member that only shares the name, and a class nested inside another is matched by its bare name only from inside that class or one that inherits it. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) +- `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) - In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) diff --git a/__tests__/installer-targets.test.ts b/__tests__/installer-targets.test.ts index ad12e8c706..7609912598 100644 --- a/__tests__/installer-targets.test.ts +++ b/__tests__/installer-targets.test.ts @@ -24,6 +24,12 @@ import { ALL_TARGETS, getTarget, resolveTargetFlag } from '../src/installer/targ import { uninstallTargets, refreshTargets } from '../src/installer'; import { upsertTomlTable, removeTomlTable, buildTomlTable } from '../src/installer/targets/toml'; import { cleanupLegacyHooks, writePromptHookEntry, removePromptHookEntry } from '../src/installer/targets/claude'; +import { replaceOrAppendMarkedSection, removeMarkedSection } from '../src/installer/targets/shared'; +import { + CODEGRAPH_INSTRUCTIONS_BLOCK, + CODEGRAPH_SECTION_END, + CODEGRAPH_SECTION_START, +} from '../src/installer/instructions-template'; function mkTmpDir(label: string): string { return fs.mkdtempSync(path.join(os.tmpdir(), `cg-targets-${label}-`)); @@ -1798,6 +1804,190 @@ describe('Installer — refreshTargets sweep (codegraph install --refresh)', () }); }); +// --------------------------------------------------------------------------- +// The instructions block never touches anything outside its markers (#2506) +// +// `codegraph upgrade` runs `install --refresh`, which rewrites the block in +// every configured agent's instructions file. The write paired the FIRST +// start marker with the FIRST end marker anywhere in the file, so one stray +// start marker above the real block — an older block whose end marker was +// deleted by hand, or the marker quoted in the user's own notes — made the +// refresh replace everything between the two. With the stray marker on the +// first line, the reported global AGENTS.md came back as just the block. +// --------------------------------------------------------------------------- +describe('Installer — the instructions block keeps everything outside its markers (#2506)', () => { + const START = CODEGRAPH_SECTION_START; + const END = CODEGRAPH_SECTION_END; + const BLOCK = CODEGRAPH_INSTRUCTIONS_BLOCK; + const BOM = String.fromCharCode(0xfeff); + const crlf = (s: string) => s.replace(/\n/g, '\r\n'); + // A maintained guidance file like the reporter's, with text below the block too. + const NOTES = [ + '# Global agent guidance', + '', + '## Toolset inventory', + '- ripgrep, fd, jq', + '', + '## Commit policy', + '- conventional commits', + ].join('\n'); + const MORE = ['## Runtime notes', '', '- prefer the sandbox'].join('\n'); + // What is left of an older block whose end marker was deleted by hand. + const STRAY = `${START}\n## CodeGraph\n\n`; + + let dir: string; + let file: string; + beforeEach(() => { + dir = mkTmpDir('marked'); + file = path.join(dir, 'AGENTS.md'); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + function upsert(content: string): { action: string; after: string } { + fs.writeFileSync(file, content); + const action = replaceOrAppendMarkedSection(file, BLOCK, START, END); + return { action, after: fs.readFileSync(file, 'utf-8') }; + } + function upsertAgain(): { action: string; after: string } { + const action = replaceOrAppendMarkedSection(file, BLOCK, START, END); + return { action, after: fs.readFileSync(file, 'utf-8') }; + } + function remove(content: string): { action: string; after: string | null } { + fs.writeFileSync(file, content); + const action = removeMarkedSection(file, START, END); + return { action, after: fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : null }; + } + + it('a stray start marker on the first line no longer reduces the file to the block (the report)', () => { + const { action, after } = upsert(`${STRAY}${NOTES}\n\n${LEGACY_BLOCK}\n`); + expect(action).toBe('updated'); + expect(after).toBe(`${STRAY}${NOTES}\n\n${BLOCK}\n`); + expect(upsertAgain()).toEqual({ action: 'unchanged', after }); + }); + + it('a stray start marker with no block after it: the block is appended, and the next run keeps the text below the stray marker', () => { + const content = `${NOTES}\n\n${STRAY}old text, end marker deleted\n\n${MORE}\n`; + const first = upsert(content); + expect(first.action).toBe('appended'); + expect(first.after).toBe(`${content}\n${BLOCK}\n`); + expect(upsertAgain()).toEqual({ action: 'unchanged', after: first.after }); + }); + + it('the start marker quoted in a sentence of the user\'s is their text, not the edge of a block', () => { + const prose = `Keep your own notes above the ${START} line.\n\n`; + const { after } = upsert(`${prose}${NOTES}\n\n${LEGACY_BLOCK}\n`); + expect(after).toBe(`${prose}${NOTES}\n\n${BLOCK}\n`); + }); + + it('a lone end marker no longer makes every run append another block', () => { + const content = `${NOTES}\n\nold text, start marker deleted\n${END}\n\n${MORE}\n`; + const first = upsert(content); + expect(first.after).toBe(`${content}\n${BLOCK}\n`); + const second = upsertAgain(); + expect(second).toEqual({ action: 'unchanged', after: first.after }); + }); + + it('two blocks: the first is refreshed, the duplicate removed, and the text between them kept', () => { + const { action, after } = upsert(`${NOTES}\n\n${LEGACY_BLOCK}\n\n${MORE}\n\n${LEGACY_BLOCK}\n`); + expect(action).toBe('updated'); + expect(after).toBe(`${NOTES}\n\n${BLOCK}\n\n${MORE}\n`); + expect(upsertAgain()).toEqual({ action: 'unchanged', after }); + }); + + it('a CRLF file stays CRLF: the text around the block is untouched and the block takes the file\'s line breaks', () => { + const { action, after } = upsert(crlf(`${NOTES}\n\n${LEGACY_BLOCK}\n\n${MORE}\n`)); + expect(action).toBe('updated'); + expect(after).toBe(crlf(`${NOTES}\n\n${BLOCK}\n\n${MORE}\n`)); + expect(upsertAgain()).toEqual({ action: 'unchanged', after }); + }); + + it('a CRLF file that already holds the current block is left alone', () => { + const content = crlf(`${NOTES}\n\n${BLOCK}\n`); + expect(upsert(content)).toEqual({ action: 'unchanged', after: content }); + }); + + it('a file without a trailing newline keeps that shape', () => { + expect(upsert(`${NOTES}\n\n${LEGACY_BLOCK}`).after).toBe(`${NOTES}\n\n${BLOCK}`); + }); + + it('appending keeps every existing byte: nothing is trimmed, only line breaks are added', () => { + expect(upsert(NOTES).after).toBe(`${NOTES}\n\n${BLOCK}\n`); + expect(upsert(`${NOTES}\n`).after).toBe(`${NOTES}\n\n${BLOCK}\n`); + expect(upsert(`${NOTES}\n\n\n`).after).toBe(`${NOTES}\n\n\n${BLOCK}\n`); + expect(upsert(crlf(`${NOTES}\n`)).after).toBe(crlf(`${NOTES}\n\n${BLOCK}\n`)); + }); + + it('a byte-order mark survives', () => { + expect(upsert(`${BOM}${NOTES}\n\n${LEGACY_BLOCK}\n`).after).toBe(`${BOM}${NOTES}\n\n${BLOCK}\n`); + expect(upsert(`${BOM}${LEGACY_BLOCK}\n\n${NOTES}\n`).after).toBe(`${BOM}${BLOCK}\n\n${NOTES}\n`); + }); + + it('uninstall removes only the block, never the text after a stray start marker', () => { + const { action, after } = remove(`${STRAY}${NOTES}\n\n${LEGACY_BLOCK}\n\n${MORE}\n`); + expect(action).toBe('removed'); + expect(after).toBe(`${STRAY}${NOTES}\n\n${MORE}\n`); + }); + + it('uninstall removes every block, keeps CRLF, and leaves a file holding only stray markers alone', () => { + expect(remove(crlf(`${NOTES}\n\n${LEGACY_BLOCK}\n\n${MORE}\n\n${LEGACY_BLOCK}\n`)).after).toBe(crlf(`${NOTES}\n\n${MORE}\n`)); + const strays = `${NOTES}\n${START}\n${MORE}\n${END}x\n`; + expect(remove(strays)).toEqual({ action: 'not-found', after: strays }); + // A file that held nothing but the block is deleted, as before. + expect(remove(`${LEGACY_BLOCK}\n`)).toEqual({ action: 'removed', after: null }); + }); +}); + +describe('Installer — install --refresh keeps each agent\'s instructions file (#2506)', () => { + let tmpHome: string; + let tmpCwd: string; + let origCwd: string; + let homeRestore: { restore: () => void }; + + beforeEach(() => { + tmpHome = mkTmpDir('rf2506-home'); + tmpCwd = mkTmpDir('rf2506-cwd'); + origCwd = process.cwd(); + process.chdir(tmpCwd); + homeRestore = setHome(tmpHome); + }); + + afterEach(() => { + homeRestore.restore(); + process.chdir(origCwd); + fs.rmSync(tmpHome, { recursive: true, force: true }); + fs.rmSync(tmpCwd, { recursive: true, force: true }); + }); + + // Every target that writes an instructions file, at its global path. + const INSTRUCTIONS: Array<[string, (home: string) => string]> = [ + ['claude', (home) => path.join(home, '.claude', 'CLAUDE.md')], + ['codex', (home) => path.join(home, '.codex', 'AGENTS.md')], + ['gemini', (home) => path.join(home, '.gemini', 'GEMINI.md')], + ['opencode', (home) => path.join(home, '.config', 'opencode', 'AGENTS.md')], + ]; + const USER = `${CODEGRAPH_SECTION_START}\n## CodeGraph\n\n# My notes\n\nAlways run the tests first.\n\n`; + + for (const [id, instructionsFile] of INSTRUCTIONS) { + it(`${id}: the refresh rewrites only the block, and uninstall removes only the block`, () => { + const target = getTarget(id)!; + target.install('global', { autoAllow: false }); + const file = instructionsFile(tmpHome); + fs.writeFileSync(file, `${USER}${LEGACY_BLOCK}\n`); + + const report = refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === id)!; + expect(report.status).toBe('refreshed'); + expect(report.changedPaths).toContain(file); + expect(fs.readFileSync(file, 'utf-8')).toBe(`${USER}${CODEGRAPH_INSTRUCTIONS_BLOCK}\n`); + expect(refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === id)!.status).toBe('unchanged'); + + target.uninstall('global'); + expect(fs.readFileSync(file, 'utf-8')).toBe(USER.trimEnd() + '\n'); + }); + } +}); + describe('Installer — detection never writes (#1870)', () => { let tmpHome: string; let tmpCwd: string; @@ -1946,6 +2136,17 @@ describe('Installer — Cursor rules file cleanup on uninstall', () => { expect(after).not.toContain('codegraph_search'); expect(after).not.toContain('CODEGRAPH_START'); }); + + it('a stray start marker above the block never takes the user\'s rules with it (#2506)', () => { + fs.mkdirSync(path.dirname(rulesFile()), { recursive: true }); + const userRules = `${CODEGRAPH_SECTION_START}\n## My own rule\nkeep me\n`; + fs.writeFileSync(rulesFile(), `${MDC_FRONTMATTER}${userRules}\n${LEGACY_BLOCK}\n`); + + cursor.uninstall('local'); + + expect(fs.existsSync(rulesFile())).toBe(true); + expect(fs.readFileSync(rulesFile(), 'utf-8')).toBe(`${MDC_FRONTMATTER}${userRules}`); + }); }); function listAllFiles(dir: string): string[] { diff --git a/__tests__/upgrade.test.ts b/__tests__/upgrade.test.ts index ad46a17ad4..53f6a89586 100644 --- a/__tests__/upgrade.test.ts +++ b/__tests__/upgrade.test.ts @@ -2,6 +2,8 @@ import { describe, it, expect, beforeEach, afterEach } from 'vitest'; import * as fs from 'node:fs'; import * as path from 'node:path'; import * as os from 'node:os'; +import { spawnSync } from 'node:child_process'; +import { CODEGRAPH_INSTRUCTIONS_BLOCK } from '../src/installer/instructions-template'; import { detectInstallMethod, deriveInstallDir, @@ -824,6 +826,75 @@ describe('post-upgrade refresh of installed agent surfaces', () => { }); }); +// --------------------------------------------------------------------------- +// The refresh an upgrade runs keeps the user's instructions files (#2506) +// --------------------------------------------------------------------------- + +describe('post-upgrade refresh keeps the user\'s instructions files (#2506)', () => { + const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + let home: string; + + beforeEach(() => { + home = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-upgrade-2506-')); + }); + + afterEach(() => { + fs.rmSync(home, { recursive: true, force: true }); + }); + + it('an npm upgrade refreshes the block in the global opencode AGENTS.md and keeps every line around it', async () => { + // opencode configured globally, as a previous `codegraph install` left it. + const dir = path.join(home, '.config', 'opencode'); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(path.join(dir, 'opencode.jsonc'), JSON.stringify({ + $schema: 'https://opencode.ai/config.json', + mcp: { servers: { codegraph: { type: 'local', command: ['codegraph', 'serve', '--mcp'], disabled: false, codemode: false } } }, + }, null, 2) + '\n'); + // The user's maintained guidance, opening with what is left of an older + // block whose end marker was deleted, and an old block at the end. + const agentsMd = path.join(dir, 'AGENTS.md'); + const notes = '\n## CodeGraph\n\n# Toolset\n\n- ripgrep, fd\n\n## Commit policy\n\n- conventional commits\n\n'; + const oldBlock = '\n## CodeGraph\n\nPrefer `codegraph_search` over grep.\n\n'; + fs.writeFileSync(agentsMd, notes + oldBlock); + + const work = path.join(home, 'work'); + fs.mkdirSync(work); + const env: NodeJS.ProcessEnv = { + ...process.env, + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: path.join(home, '.config'), + APPDATA: path.join(home, 'AppData', 'Roaming'), + LOCALAPPDATA: path.join(home, 'AppData', 'Local'), + CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_TELEMETRY: '0', + CODEGRAPH_NO_UPDATE_CHECK: '1', + DO_NOT_TRACK: '1', + }; + for (const key of ['CODEX_HOME', 'COPILOT_HOME', 'HERMES_HOME', 'CLAUDE_CONFIG_DIR', 'CODEGRAPH_NO_INSTALL_REFRESH']) delete env[key]; + + const { deps, calls } = makeDeps({ + method: { kind: 'npm', scope: 'global' }, + currentVersion: '0.9.8', + hasCommand: (cmd) => cmd === 'codegraph', + }); + // `npm install -g` succeeds without running; the refresh it is followed + // by runs the real CLI against the private home. + deps.run = (cmd, args, runEnv) => { + calls.runs.push({ cmd, args, env: runEnv }); + if (cmd !== 'codegraph') return 0; + const r = spawnSync(process.execPath, [BIN, ...args], { cwd: work, env, encoding: 'utf-8', timeout: 60_000 }); + return r.status ?? -1; + }; + + expect(await runUpgrade({}, deps)).toBe(0); + const refresh = calls.runs[calls.runs.length - 1]; + expect(refresh?.cmd).toBe('codegraph'); + expect(refresh?.args).toEqual(['install', '--refresh']); + expect(fs.readFileSync(agentsMd, 'utf-8')).toBe(notes + CODEGRAPH_INSTRUCTIONS_BLOCK + '\n'); + }, 90_000); +}); + // --------------------------------------------------------------------------- // Post-upgrade version probe — does the PATH-resolved `codegraph` serve the // version we just installed, in THIS terminal? diff --git a/src/installer/targets/cursor.ts b/src/installer/targets/cursor.ts index c2d3595cad..07db056d5c 100644 --- a/src/installer/targets/cursor.ts +++ b/src/installer/targets/cursor.ts @@ -46,6 +46,7 @@ import { getMcpServerConfig, jsonDeepEqual, readJsonFile, + stripMarkedSections, writeJsonFile, } from './shared'; import { @@ -217,18 +218,17 @@ function removeRulesEntry(): WriteResult['files'][number] { } const ourFrontmatter = MDC_FRONTMATTER.trim(); - const startIdx = content.indexOf(CODEGRAPH_SECTION_START); - const endIdx = content.indexOf(CODEGRAPH_SECTION_END); - - // Our marked block is present — strip it, then decide what's left. - if (startIdx !== -1 && endIdx > startIdx) { - const before = content.substring(0, startIdx).trimEnd(); - const after = content.substring(endIdx + CODEGRAPH_SECTION_END.length).trimStart(); - const remainder = (before + (before && after ? '\n\n' : '') + after).trim(); - if (remainder === '' || remainder === ourFrontmatter) { + + // Our marked block is present — strip it, then decide what's left. Only + // well-formed blocks are stripped, so a stray marker never takes the + // user's rules with it (#2506). + const remainder = stripMarkedSections(content, CODEGRAPH_SECTION_START, CODEGRAPH_SECTION_END); + if (remainder !== null) { + const rest = remainder.trim(); + if (rest === '' || rest === ourFrontmatter) { try { fs.unlinkSync(file); } catch { /* ignore */ } } else { - atomicWriteFileSync(file, remainder + '\n'); + atomicWriteFileSync(file, remainder); } return { path: file, action: 'removed' }; } diff --git a/src/installer/targets/shared.ts b/src/installer/targets/shared.ts index 52f2c25bd3..dca78989c9 100644 --- a/src/installer/targets/shared.ts +++ b/src/installer/targets/shared.ts @@ -154,17 +154,111 @@ export function jsonDeepEqual(a: unknown, b: unknown): boolean { return ak.every((k) => jsonDeepEqual(ao[k], bo[k])); } +/** One marker-delimited block: `start` is the start marker's offset, `end` is just past the end marker. */ +interface MarkedSection { + start: number; + end: number; +} + +/** + * True when the marker at `at` is alone on its line: only spaces or tabs + * around it (plus a file-leading BOM, or the `\r` of a CRLF). Every block + * codegraph has ever written puts its markers on lines of their own, so a + * marker the user quotes inside a sentence is their text, not a block edge. + */ +function markerOnOwnLine(content: string, at: number, length: number): boolean { + for (let i = at - 1; i >= 0 && content[i] !== '\n'; i--) { + if (content[i] !== ' ' && content[i] !== '\t' && content[i] !== '\uFEFF') return false; + } + for (let i = at + length; i < content.length && content[i] !== '\n'; i++) { + if (content[i] !== ' ' && content[i] !== '\t' && content[i] !== '\r') return false; + } + return true; +} + +/** + * Every well-formed marker block in `content`, in file order. Each end + * marker closes the nearest start marker before it, so a block never + * contains another start marker. A start marker left without its end (half + * of a block deleted by hand) and an end marker without its start are plain + * text — neither can stretch a block over the user's own content. + * + * Issue #2506: the first start marker was paired with the first end marker + * anywhere in the file, so one stray start marker above the real block made + * an upgrade's refresh replace everything between them — the whole file, + * when the stray marker was its first line. + */ +function findMarkedSections(content: string, startMarker: string, endMarker: string): MarkedSection[] { + const marks: Array<{ at: number; isStart: boolean }> = []; + for (const [marker, isStart] of [[startMarker, true], [endMarker, false]] as const) { + for (let at = content.indexOf(marker); at !== -1; at = content.indexOf(marker, at + marker.length)) { + if (markerOnOwnLine(content, at, marker.length)) marks.push({ at, isStart }); + } + } + marks.sort((a, b) => a.at - b.at); + + const sections: MarkedSection[] = []; + let open = -1; + for (const mark of marks) { + if (mark.isStart) { + open = mark.at; + } else if (open !== -1) { + sections.push({ start: open, end: mark.at + endMarker.length }); + open = -1; + } + } + return sections; +} + +/** The line break `text` uses (its first one); `\n` when it has none. */ +function lineBreakOf(text: string): '\n' | '\r\n' { + const nl = text.indexOf('\n'); + return nl > 0 && text[nl - 1] === '\r' ? '\r\n' : '\n'; +} + +function withLineBreaks(text: string, eol: string): string { + return text.replace(/\r?\n/g, eol); +} + +/** True when `text` holds nothing but whitespace (and a BOM). */ +function isBlank(text: string): boolean { + return text.replace(/^\uFEFF/, '').trim() === ''; +} + +/** + * `content` without the given blocks. Only the whitespace touching a removed + * block changes: the text on either side is rejoined with one blank line, or + * with nothing at the start or end of the file. + */ +function dropMarkedSections(content: string, sections: readonly MarkedSection[], eol: string): string { + let out = content.slice(0, sections[0]!.start); + for (let i = 0; i < sections.length; i++) { + const nextStart = i + 1 < sections.length ? sections[i + 1]!.start : content.length; + const left = out.replace(/[ \t\r\n]+$/, ''); + const right = content.slice(sections[i]!.end, nextStart).replace(/^[ \t\r\n]+/, ''); + if (right === '') out = isBlank(left) ? left : left + eol; + else if (isBlank(left)) out = left + right; + else out = left + eol + eol + right; + } + return out; +} + /** * Replace or append a marker-delimited section in a markdown-ish file. * - * Used by Claude / Codex for the ` ... ` block. Preserves all content outside the - * markers verbatim. + * Used for the ` ... ` block + * every target writes into its instructions file. Only the bytes between the + * first block's markers change: everything outside them — the user's text, + * their line endings, a missing final newline, a BOM, a stray marker — is + * kept byte-for-byte. The block is written with the line breaks it already + * had, so a CRLF file stays CRLF. Further blocks (duplicates) are removed, + * keeping the text around them. With no block yet, the section is appended + * after the existing bytes, which are left exactly as they were. * - * Returns `created` when the file didn't exist; `updated` when - * markers were found and content swapped; `appended` when markers - * weren't found and section was added at end. `unchanged` when the - * existing block already matches `body`. + * Returns `created` when the file didn't exist; `updated` when a block was + * found and rewritten; `appended` when none was found and the section was + * added at the end; `unchanged` when the file already holds exactly one + * block matching `body`. */ export function replaceOrAppendMarkedSection( filePath: string, @@ -178,34 +272,49 @@ export function replaceOrAppendMarkedSection( } const content = fs.readFileSync(filePath, 'utf-8'); - const startIdx = content.indexOf(startMarker); - const endIdx = content.indexOf(endMarker); + const sections = findMarkedSections(content, startMarker, endMarker); - if (startIdx !== -1 && endIdx > startIdx) { - const existingBlock = content.substring(startIdx, endIdx + endMarker.length); - if (existingBlock === body) { + if (sections.length > 0) { + const [first, ...duplicates] = sections; + const block = withLineBreaks(body, lineBreakOf(content.slice(first!.start, first!.end))); + let next = content.slice(0, first!.start) + block + content.slice(first!.end); + if (duplicates.length > 0) { + const shift = block.length - (first!.end - first!.start); + next = dropMarkedSections( + next, + duplicates.map((s) => ({ start: s.start + shift, end: s.end + shift })), + lineBreakOf(content), + ); + } + if (next === content) { return 'unchanged'; } - const before = content.substring(0, startIdx); - const after = content.substring(endIdx + endMarker.length); - atomicWriteFileSync(filePath, before + body + after); + atomicWriteFileSync(filePath, next); return 'updated'; } - // No markers — append. Preserve existing content with a separating - // blank line. - const trimmed = content.trimEnd(); - const sep = trimmed.length > 0 ? '\n\n' : ''; - atomicWriteFileSync(filePath, trimmed + sep + body + '\n'); + // No block yet — append one, separated by a blank line. Nothing already in + // the file is trimmed or rewritten; only line breaks are added, in the + // file's own style. + const eol = lineBreakOf(content); + const block = withLineBreaks(body, eol); + if (isBlank(content)) { + atomicWriteFileSync(filePath, (content.startsWith('\uFEFF') ? '\uFEFF' : '') + block + eol); + return 'appended'; + } + const sep = /\n[ \t]*\r?\n[ \t\r]*$/.test(content) ? '' : /\n[ \t\r]*$/.test(content) ? eol : eol + eol; + atomicWriteFileSync(filePath, content + sep + block + eol); return 'appended'; } /** * Upsert the CodeGraph instructions block into an agent instructions * file (CLAUDE.md / AGENTS.md / GEMINI.md). The one write shared by - * every target: self-heals a stale pre-#529 long block (markers match → - * replaced by the current short one), appends after existing user - * content otherwise, and reports `unchanged` on byte-equal re-runs so + * every target — `install`, and the `install --refresh` that + * `codegraph upgrade` runs: self-heals a stale pre-#529 long block + * (markers match → replaced by the current short one), appends after + * existing user content otherwise, never changes a byte outside the + * block (#2506), and reports `unchanged` on byte-equal re-runs so * install stays idempotent. See `instructions-template.ts` for why this * block exists (#704: subagents + non-MCP harnesses never see the MCP * initialize instructions). @@ -220,6 +329,17 @@ export function upsertInstructionsEntry(file: string): { path: string; action: ' return { path: file, action: action === 'appended' ? 'updated' : action }; } +/** + * `content` with every well-formed marker block removed, or `null` when it + * has none. A stray start or end marker is left alone (see + * `findMarkedSections`), and only the whitespace touching a removed block + * changes. Shared by `removeMarkedSection` and the Cursor rules-file cleanup. + */ +export function stripMarkedSections(content: string, startMarker: string, endMarker: string): string | null { + const sections = findMarkedSections(content, startMarker, endMarker); + return sections.length === 0 ? null : dropMarkedSections(content, sections, lineBreakOf(content)); +} + /** * Inverse of `replaceOrAppendMarkedSection`. Strips the marker * block from `filePath` if present. If the file becomes empty after @@ -243,18 +363,13 @@ export function removeMarkedSection( return 'kept'; } - const startIdx = content.indexOf(startMarker); - const endIdx = content.indexOf(endMarker); - if (startIdx === -1 || endIdx <= startIdx) return 'not-found'; - - const before = content.substring(0, startIdx).trimEnd(); - const after = content.substring(endIdx + endMarker.length).trimStart(); - const joined = before + (before && after ? '\n\n' : '') + after; + const rest = stripMarkedSections(content, startMarker, endMarker); + if (rest === null) return 'not-found'; - if (joined.trim() === '') { + if (isBlank(rest)) { try { fs.unlinkSync(filePath); } catch { /* ignore */ } } else { - atomicWriteFileSync(filePath, joined.trim() + '\n'); + atomicWriteFileSync(filePath, rest); } return 'removed'; } From 0abae119592281b22a925c58ae1df3754bb4f2dc Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 01:06:12 +0000 Subject: [PATCH 09/95] fix(installer): a symlinked instructions or config file is written through, not replaced (#2768) Root cause: every installer write goes through one helper that writes `.tmp.` and renames it onto the path. A rename replaces a symbolic link itself, not the file it points to. So a `~/.codex/AGENTS.md` or `~/.claude.json` linked from a dotfiles repo became a detached regular copy on install: the shared file never got CodeGraph's settings, and later edits to it no longer reached the agent. This held for every target (Claude Code, Codex, opencode, Cursor, Gemini, Antigravity, Kiro, Hermes and the Copilot family) and for install, the refresh that `codegraph upgrade` runs, and uninstall. Uninstall also deleted a link whose file it had emptied, leaving CodeGraph's entry behind in the shared file. Fix: - The write helper resolves a link to the file at the end of its chain and writes and renames there, so the link stays exactly as it was, relative or absolute. - A link to a file that doesn't exist yet gets that file created, but only in a folder that already exists, as writing through it in a shell would. No folder is ever created where a link points. - A dangling chain is followed hop by hop from the real folder holding each link, so a relative link under a linked folder lands where the OS would put it. - When the linked file can't be written (a read-only file, a folder that doesn't exist), the error names the link and its target and nothing changes. - Where uninstall used to delete a user's file it had emptied (an instructions file, Codex's config.toml, Copilot CLI's mcp-config.json, the legacy project .claude.json), a link is kept and the emptied content is written through it. - The refresh sweep catches a failure per agent, reports it, still refreshes the others, and exits non-zero, so the upgrade warns. Decided: links to CodeGraph's own legacy files (Cursor's codegraph.mdc, Kiro's steering doc), which install only ever deletes, are still removed. Validation: - 28 new tests. 25 are in installer-targets.test.ts: - every target and location: install and uninstall keep each of its files that is a link (18 cases); - the shared files' content through install and uninstall for Claude Code, Codex, opencode (two agents sharing one AGENTS.md) and Cursor; - relative links under a linked folder; - a link into a missing folder; - a link to a read-only file; - uninstall emptying through links; - a failing link in a refresh; - a folder link guard. The other 3: - 1 in upgrade.test.ts, where the refresh runs the built CLI; - 2 in a new installer-links-simulated.test.ts, which simulates the link with spied lstat/realpath/readlink, so it runs on machines that can't create file symlinks. With main's sources put back, 27 fail; the folder-link guard passes on both. All pass with the fix. The real-link tests are skipped, with the reason in their names, where file symlinks can't be created (Windows without Developer Mode or admin). - This Windows machine has Developer Mode on, so the real-link tests ran for real. - The real CLI against a private home (`install --yes`, `install --refresh`, `uninstall --yes --keep-cli`), 10 linked files for five agents plus relative, dangling and read-only links: - on main, all 29 link checks found the link replaced by a copy; - with the fix, none did, every link's text was unchanged, the shared files got CodeGraph's settings and lost them again on uninstall, and the missing-folder and read-only cases exited 1 with the new message and changed nothing. Fixes #2721 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/installer-links-simulated.test.ts | 76 +++++ __tests__/installer-targets.test.ts | 289 ++++++++++++++++++++ __tests__/upgrade.test.ts | 84 ++++++ src/bin/codegraph.ts | 10 +- src/installer/index.ts | 21 +- src/installer/targets/claude.ts | 3 +- src/installer/targets/codex.ts | 3 +- src/installer/targets/copilot-cli.ts | 5 +- src/installer/targets/shared.ts | 91 +++++- 10 files changed, 571 insertions(+), 12 deletions(-) create mode 100644 __tests__/installer-links-simulated.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 57a82ae6e3..b3546c451f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) +- `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) - In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) diff --git a/__tests__/installer-links-simulated.test.ts b/__tests__/installer-links-simulated.test.ts new file mode 100644 index 0000000000..d63ccedb24 --- /dev/null +++ b/__tests__/installer-links-simulated.test.ts @@ -0,0 +1,76 @@ +/** + * The installer writes through a symbolic link instead of replacing it + * (#2721) — checked here with the link SIMULATED, so it runs on every machine. + * + * The real-link tests in `installer-targets.test.ts` need to create file + * symlinks, which Windows only allows with Developer Mode or admin rights; + * where it can't, they are skipped. These two still run: the file system + * reports one path as a link (lstat) and says where it points (realpath / + * readlink), and the write must land there, leaving the link's own path alone. + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { atomicWriteFileSync } from '../src/installer/targets/shared'; + +// A plain copy of `fs`, so single functions can be spied on. +vi.mock('fs', async (importOriginal) => ({ ...await importOriginal() })); + +describe('Installer — writes land on the file a link points to (link simulated) (#2721)', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-link-sim-')); + }); + afterEach(() => { + vi.restoreAllMocks(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + /** Make the file system report `linkPath` as a symbolic link. */ + function reportAsLink(linkPath: string, opts: { realpath: () => string; readlink?: string }): void { + const realLstat = fs.lstatSync; + const realRealpath = fs.realpathSync; + const realReadlink = fs.readlinkSync; + vi.spyOn(fs, 'lstatSync').mockImplementation(((p: fs.PathLike, ...rest: unknown[]) => + p === linkPath ? ({ isSymbolicLink: () => true } as fs.Stats) : (realLstat as any)(p, ...rest)) as typeof fs.lstatSync); + vi.spyOn(fs, 'realpathSync').mockImplementation(((p: fs.PathLike, ...rest: unknown[]) => + p === linkPath ? opts.realpath() : (realRealpath as any)(p, ...rest)) as typeof fs.realpathSync); + vi.spyOn(fs, 'readlinkSync').mockImplementation(((p: fs.PathLike, ...rest: unknown[]) => + p === linkPath ? opts.readlink : (realReadlink as any)(p, ...rest)) as typeof fs.readlinkSync); + } + + it('a live link: the file it points to is replaced, and the link\'s own path is never written', () => { + const linkPath = path.join(dir, 'AGENTS.md'); + const target = path.join(dir, 'dotfiles', 'AGENTS.md'); + fs.mkdirSync(path.dirname(target)); + fs.writeFileSync(linkPath, 'stands in for the link'); + fs.writeFileSync(target, '# shared\n'); + reportAsLink(linkPath, { realpath: () => target }); + + atomicWriteFileSync(linkPath, '# shared\nwith codegraph\n'); + + expect(fs.readFileSync(target, 'utf-8')).toBe('# shared\nwith codegraph\n'); + expect(fs.readFileSync(linkPath, 'utf-8')).toBe('stands in for the link'); + expect(fs.readdirSync(path.dirname(target))).toEqual(['AGENTS.md']); + }); + + it('a dangling relative link: its file is created relative to the link\'s folder, and a missing folder is an error', () => { + const linkPath = path.join(dir, 'AGENTS.md'); + fs.writeFileSync(linkPath, 'stands in for the link'); + fs.mkdirSync(path.join(dir, 'dotfiles')); + const enoent = (): never => { throw Object.assign(new Error('ENOENT: no such file'), { code: 'ENOENT' }); }; + reportAsLink(linkPath, { realpath: enoent, readlink: path.join('dotfiles', 'AGENTS.md') }); + + atomicWriteFileSync(linkPath, 'block\n'); + expect(fs.readFileSync(path.join(dir, 'dotfiles', 'AGENTS.md'), 'utf-8')).toBe('block\n'); + expect(fs.readFileSync(linkPath, 'utf-8')).toBe('stands in for the link'); + + vi.restoreAllMocks(); + reportAsLink(linkPath, { realpath: enoent, readlink: path.join('not-cloned', 'AGENTS.md') }); + expect(() => atomicWriteFileSync(linkPath, 'block\n')).toThrow(/is a symbolic link to .*not-cloned.*its folder does not exist/); + expect(fs.existsSync(path.join(dir, 'not-cloned'))).toBe(false); + expect(fs.readFileSync(linkPath, 'utf-8')).toBe('stands in for the link'); + }); +}); diff --git a/__tests__/installer-targets.test.ts b/__tests__/installer-targets.test.ts index 7609912598..c303d4eca0 100644 --- a/__tests__/installer-targets.test.ts +++ b/__tests__/installer-targets.test.ts @@ -1988,6 +1988,295 @@ describe('Installer — install --refresh keeps each agent\'s instructions file } }); +// --------------------------------------------------------------------------- +// A symbolic link is written through, never replaced (#2721) +// +// Instructions and config files are often symbolic links into a dotfiles or +// team repo (`~/.codex/AGENTS.md -> ~/dotfiles/AGENTS.md`). Every installer +// write went to `.tmp.` and was renamed onto the path, and a +// rename replaces the link itself: install left a detached copy, so the +// shared file never got CodeGraph's entry and later edits to it never reached +// the agent. Uninstall deleted a link whose file it had emptied. +// --------------------------------------------------------------------------- + +/** + * Whether this machine can create a file symlink. On Windows that needs + * Developer Mode or admin rights; where it can't, the link tests are skipped, + * with the reason in their name, rather than passing without running. + */ +function canCreateFileSymlink(): boolean { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-symlink-probe-')); + try { + fs.writeFileSync(path.join(dir, 'target'), ''); + fs.symlinkSync(path.join(dir, 'target'), path.join(dir, 'link'), 'file'); + return true; + } catch { + return false; + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} +const FILE_SYMLINKS = canCreateFileSymlink(); +const NO_FILE_SYMLINKS = ' — skipped: creating a file symlink needs Developer Mode or admin rights on Windows'; + +describe.skipIf(!FILE_SYMLINKS)(`Installer — a symbolic link is written through, never replaced (#2721)${FILE_SYMLINKS ? '' : NO_FILE_SYMLINKS}`, () => { + let tmpHome: string; + let tmpCwd: string; + let origCwd: string; + let homeRestore: { restore: () => void }; + let restoreWritable: Array<() => void>; + + beforeEach(() => { + tmpHome = mkTmpDir('link-home'); + tmpCwd = mkTmpDir('link-cwd'); + origCwd = process.cwd(); + process.chdir(tmpCwd); + homeRestore = setHome(tmpHome); + restoreWritable = []; + }); + + afterEach(() => { + for (const restore of restoreWritable) restore(); + homeRestore.restore(); + process.chdir(origCwd); + fs.rmSync(tmpHome, { recursive: true, force: true }); + fs.rmSync(tmpCwd, { recursive: true, force: true }); + }); + + const isLink = (p: string): boolean => { + try { return fs.lstatSync(p).isSymbolicLink(); } catch { return false; } + }; + /** Make `linkPath` a file symlink whose text is `text`, creating the folder that holds it. */ + function link(linkPath: string, text: string): void { + fs.mkdirSync(path.dirname(linkPath), { recursive: true }); + fs.symlinkSync(text, linkPath, 'file'); + } + /** A file in the shared folder the links point into, written with `content` when given. */ + function sharedFile(name: string, content?: string): string { + const file = path.join(tmpHome, 'dotfiles', name); + fs.mkdirSync(path.dirname(file), { recursive: true }); + if (content !== undefined) fs.writeFileSync(file, content); + return file; + } + + // Every file each agent's install writes, at both locations: whatever the + // installer does, it never replaces or removes a link. + for (const target of ALL_TARGETS) { + for (const location of (['global', 'local'] as const).filter((l) => target.supportsLocation(l))) { + it(`${target.id} (${location}): install and uninstall keep every one of its files that is a link`, () => { + const links = target.describePaths(location).map((p, i) => { + const dest = sharedFile(`${i}-${path.basename(p)}`); + link(p, dest); + return { path: p, dest }; + }); + + const result = target.install(location, { autoAllow: true }); + for (const l of links) { + expect(isLink(l.path), l.path).toBe(true); + expect(fs.readlinkSync(l.path)).toBe(l.dest); + } + for (const f of result.files.filter((x) => x.action === 'created' || x.action === 'updated')) { + const written = links.find((l) => l.path === f.path); + expect(written, f.path).toBeDefined(); + expect(fs.existsSync(written!.dest), written!.dest).toBe(true); + } + expect(target.detect(location).alreadyConfigured).toBe(true); + + target.uninstall(location); + for (const l of links) { + expect(isLink(l.path), l.path).toBe(true); + expect(fs.readlinkSync(l.path)).toBe(l.dest); + } + expect(target.detect(location).alreadyConfigured).toBe(false); + }); + } + } + + it('the shared files get CodeGraph\'s entries next to the user\'s own, and uninstall takes only those out again', () => { + const files = { + claudeJson: sharedFile('claude.json', '{\n "numStartups": 3\n}\n'), + claudeMd: sharedFile('CLAUDE.md', '# my claude notes\n'), + agentsMd: sharedFile('AGENTS.md', '# shared rules\n\nBe terse.\n'), + codexToml: sharedFile('codex.toml', 'model = "o3"\n'), + opencode: sharedFile('opencode.jsonc', '{\n // my theme\n "theme": "tokyonight"\n}\n'), + cursor: sharedFile('cursor-mcp.json', '{\n "mcpServers": {\n "other": {\n "command": "x"\n }\n }\n}\n'), + }; + const original = Object.fromEntries(Object.entries(files).map(([k, f]) => [k, fs.readFileSync(f, 'utf-8')])); + link(path.join(tmpHome, '.claude.json'), files.claudeJson); + link(path.join(tmpHome, '.claude', 'CLAUDE.md'), files.claudeMd); + // Two agents sharing one instructions file. + link(path.join(tmpHome, '.codex', 'AGENTS.md'), files.agentsMd); + link(path.join(tmpHome, '.config', 'opencode', 'AGENTS.md'), files.agentsMd); + link(path.join(tmpHome, '.codex', 'config.toml'), files.codexToml); + link(path.join(tmpHome, '.config', 'opencode', 'opencode.jsonc'), files.opencode); + link(path.join(tmpHome, '.cursor', 'mcp.json'), files.cursor); + const ids = ['claude', 'codex', 'opencode', 'cursor'] as const; + + for (const id of ids) getTarget(id)!.install('global', { autoAllow: false }); + + const read = (f: string) => fs.readFileSync(f, 'utf-8'); + expect(JSON.parse(read(files.claudeJson))).toMatchObject({ numStartups: 3, mcpServers: { codegraph: {} } }); + expect(read(files.claudeMd)).toBe(`# my claude notes\n\n${CODEGRAPH_INSTRUCTIONS_BLOCK}\n`); + expect(read(files.agentsMd)).toBe(`# shared rules\n\nBe terse.\n\n${CODEGRAPH_INSTRUCTIONS_BLOCK}\n`); + expect(read(files.codexToml)).toContain('model = "o3"'); + expect(read(files.codexToml)).toContain('[mcp_servers.codegraph]'); + expect(read(files.opencode)).toContain('// my theme'); + expect(parseJsonc(read(files.opencode)).mcp.servers.codegraph).toBeDefined(); + expect(JSON.parse(read(files.cursor)).mcpServers).toMatchObject({ other: { command: 'x' }, codegraph: {} }); + // Re-running is a no-op through the links too. + for (const id of ids) { + for (const f of getTarget(id)!.install('global', { autoAllow: false }).files) expect(f.action, f.path).toBe('unchanged'); + } + + for (const id of ids) getTarget(id)!.uninstall('global'); + + expect(JSON.parse(read(files.claudeJson))).toEqual({ numStartups: 3 }); + expect(read(files.claudeMd)).toBe(original.claudeMd); + expect(read(files.agentsMd)).toBe(original.agentsMd); + expect(read(files.codexToml)).toBe(original.codexToml); + expect(parseJsonc(read(files.opencode))).toEqual({ $schema: 'https://opencode.ai/config.json', theme: 'tokyonight' }); + expect(read(files.opencode)).toContain('// my theme'); + expect(JSON.parse(read(files.cursor))).toEqual({ mcpServers: { other: { command: 'x' } } }); + }); + + it('a relative link keeps its text, and is followed from the folder that really holds it', () => { + // `~/.codex` is itself a folder link into the dotfiles repo, and the + // config inside it points one level up, relative to the real folder. + const realCodex = path.join(tmpHome, 'dotfiles', 'codex'); + fs.mkdirSync(realCodex, { recursive: true }); + fs.symlinkSync(realCodex, path.join(tmpHome, '.codex'), 'junction'); + const agentsText = path.join('..', 'AGENTS.md'); + const tomlText = path.join('..', 'codex.toml'); + link(path.join(realCodex, 'AGENTS.md'), agentsText); + link(path.join(realCodex, 'config.toml'), tomlText); + fs.writeFileSync(path.join(tmpHome, 'dotfiles', 'codex.toml'), 'model = "o3"\n'); + // Only the AGENTS.md target is missing, so it is created where the link + // really points: dotfiles/AGENTS.md, not a lexical ~/AGENTS.md. + + getTarget('codex')!.install('global', { autoAllow: false }); + + expect(fs.readlinkSync(path.join(realCodex, 'AGENTS.md'))).toBe(agentsText); + expect(fs.readlinkSync(path.join(realCodex, 'config.toml'))).toBe(tomlText); + expect(isLink(path.join(tmpHome, '.codex'))).toBe(true); + expect(fs.readFileSync(path.join(tmpHome, 'dotfiles', 'AGENTS.md'), 'utf-8')).toBe(`${CODEGRAPH_INSTRUCTIONS_BLOCK}\n`); + expect(fs.existsSync(path.join(tmpHome, 'AGENTS.md'))).toBe(false); + expect(fs.readFileSync(path.join(tmpHome, 'dotfiles', 'codex.toml'), 'utf-8')).toContain('[mcp_servers.codegraph]'); + }); + + it('a link into a folder that does not exist is an error naming the link, and nothing is created', () => { + const agentsMd = path.join(tmpHome, '.codex', 'AGENTS.md'); + const missing = path.join(tmpHome, 'dotfiles-not-cloned-yet', 'AGENTS.md'); + link(agentsMd, missing); + + expect(() => getTarget('codex')!.install('global', { autoAllow: false })).toThrow( + /AGENTS\.md is a symbolic link to .*dotfiles-not-cloned-yet.*its folder does not exist.*never replaces one with a copy/, + ); + expect(isLink(agentsMd)).toBe(true); + expect(fs.readlinkSync(agentsMd)).toBe(missing); + expect(fs.existsSync(path.dirname(missing))).toBe(false); + }); + + // POSIX refuses the write in a read-only folder, except to root; Windows + // refuses replacing a file whose read-only attribute is set. + const POSIX_ROOT = process.platform !== 'win32' && process.getuid?.() === 0; + it.skipIf(POSIX_ROOT)(`a link to a read-only file is an error naming the link; the file is unchanged and no temp file is left${POSIX_ROOT ? ' — skipped: root can write into a read-only folder' : ''}`, () => { + const shared = sharedFile(path.join('managed', 'AGENTS.md'), '# managed by home-manager\n'); + const lock = process.platform === 'win32' ? shared : path.dirname(shared); + fs.chmodSync(lock, process.platform === 'win32' ? 0o444 : 0o555); + restoreWritable.push(() => fs.chmodSync(lock, process.platform === 'win32' ? 0o666 : 0o755)); + const agentsMd = path.join(tmpHome, '.codex', 'AGENTS.md'); + link(agentsMd, shared); + + expect(() => getTarget('codex')!.install('global', { autoAllow: false })).toThrow(/AGENTS\.md is a symbolic link to .*it is read-only/); + expect(isLink(agentsMd)).toBe(true); + expect(fs.readFileSync(shared, 'utf-8')).toBe('# managed by home-manager\n'); + expect(fs.readdirSync(path.dirname(shared)).filter((n) => n.includes('.tmp.'))).toEqual([]); + }); + + it('uninstall empties a linked file that held only CodeGraph\'s part, instead of deleting the link', () => { + const agentsMd = sharedFile('AGENTS.md'); + const codexToml = sharedFile('codex.toml'); + const copilot = sharedFile('copilot.json'); + link(path.join(tmpHome, '.codex', 'AGENTS.md'), agentsMd); + link(path.join(tmpHome, '.codex', 'config.toml'), codexToml); + link(path.join(tmpHome, '.copilot', 'mcp-config.json'), copilot); + getTarget('codex')!.install('global', { autoAllow: false }); + getTarget('copilot-cli')!.install('global', { autoAllow: false }); + expect(fs.readFileSync(agentsMd, 'utf-8')).toBe(`${CODEGRAPH_INSTRUCTIONS_BLOCK}\n`); + + getTarget('codex')!.uninstall('global'); + getTarget('copilot-cli')!.uninstall('global'); + + for (const p of ['.codex/AGENTS.md', '.codex/config.toml', '.copilot/mcp-config.json']) { + expect(isLink(path.join(tmpHome, p)), p).toBe(true); + } + expect(fs.readFileSync(agentsMd, 'utf-8')).toBe(''); + expect(fs.readFileSync(codexToml, 'utf-8')).toBe(''); + expect(JSON.parse(fs.readFileSync(copilot, 'utf-8'))).toEqual({}); + }); + + it('a link that can\'t be written fails only its own agent in a refresh: the others are still refreshed', () => { + getTarget('codex')!.install('global', { autoAllow: false }); + getTarget('gemini')!.install('global', { autoAllow: false }); + // Codex's instructions file is now a link into a folder that is gone, + // and Gemini's block is stale. + const agentsMd = path.join(tmpHome, '.codex', 'AGENTS.md'); + fs.unlinkSync(agentsMd); + link(agentsMd, path.join(tmpHome, 'gone', 'AGENTS.md')); + const geminiMd = path.join(tmpHome, '.gemini', 'GEMINI.md'); + fs.writeFileSync(geminiMd, `${LEGACY_BLOCK}\n`); + + const reports = refreshTargets(ALL_TARGETS, 'global'); + + const codex = reports.find((r) => r.id === 'codex')!; + expect(codex.status).toBe('failed'); + expect(codex.error).toMatch(/AGENTS\.md is a symbolic link to .*gone.*its folder does not exist/); + const gemini = reports.find((r) => r.id === 'gemini')!; + expect(gemini.status).toBe('refreshed'); + expect(fs.readFileSync(geminiMd, 'utf-8')).toBe(`${CODEGRAPH_INSTRUCTIONS_BLOCK}\n`); + expect(isLink(agentsMd)).toBe(true); + }); +}); + +describe('Installer — a config folder that is a link keeps working (#2721)', () => { + let tmpHome: string; + let tmpCwd: string; + let origCwd: string; + let homeRestore: { restore: () => void }; + + beforeEach(() => { + tmpHome = mkTmpDir('dirlink-home'); + tmpCwd = mkTmpDir('dirlink-cwd'); + origCwd = process.cwd(); + process.chdir(tmpCwd); + homeRestore = setHome(tmpHome); + }); + + afterEach(() => { + homeRestore.restore(); + process.chdir(origCwd); + fs.rmSync(tmpHome, { recursive: true, force: true }); + fs.rmSync(tmpCwd, { recursive: true, force: true }); + }); + + // A folder link needs no special rights: a junction on Windows, a symlink elsewhere. + it('~/.codex as a folder link: files are written inside the real folder and the link stays', () => { + const real = path.join(tmpHome, 'dotfiles', 'codex'); + fs.mkdirSync(real, { recursive: true }); + const dirLink = path.join(tmpHome, '.codex'); + fs.symlinkSync(real, dirLink, 'junction'); + + getTarget('codex')!.install('global', { autoAllow: false }); + expect(fs.lstatSync(dirLink).isSymbolicLink()).toBe(true); + expect(fs.readFileSync(path.join(real, 'config.toml'), 'utf-8')).toContain('[mcp_servers.codegraph]'); + expect(fs.readFileSync(path.join(real, 'AGENTS.md'), 'utf-8')).toBe(`${CODEGRAPH_INSTRUCTIONS_BLOCK}\n`); + + getTarget('codex')!.uninstall('global'); + expect(fs.lstatSync(dirLink).isSymbolicLink()).toBe(true); + expect(fs.existsSync(path.join(real, 'AGENTS.md'))).toBe(false); + }); +}); + describe('Installer — detection never writes (#1870)', () => { let tmpHome: string; let tmpCwd: string; diff --git a/__tests__/upgrade.test.ts b/__tests__/upgrade.test.ts index 53f6a89586..403271b7fc 100644 --- a/__tests__/upgrade.test.ts +++ b/__tests__/upgrade.test.ts @@ -895,6 +895,90 @@ describe('post-upgrade refresh keeps the user\'s instructions files (#2506)', () }, 90_000); }); +// --------------------------------------------------------------------------- +// A symbolic link the refresh can't write through (#2721) +// --------------------------------------------------------------------------- + +/** File symlinks need Developer Mode or admin rights on Windows. */ +function canCreateFileSymlink(): boolean { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-symlink-probe-')); + try { + fs.writeFileSync(path.join(dir, 'target'), ''); + fs.symlinkSync(path.join(dir, 'target'), path.join(dir, 'link'), 'file'); + return true; + } catch { + return false; + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} +const FILE_SYMLINKS = canCreateFileSymlink(); + +describe.skipIf(!FILE_SYMLINKS)(`post-upgrade refresh with a link it can't write through (#2721)${FILE_SYMLINKS ? '' : ' — skipped: creating a file symlink needs Developer Mode or admin rights on Windows'}`, () => { + const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + let home: string; + + beforeEach(() => { + home = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-upgrade-2721-')); + }); + + afterEach(() => { + fs.rmSync(home, { recursive: true, force: true }); + }); + + it('warns, keeps the link, and still refreshes every other agent', async () => { + // Codex and Gemini configured globally. Codex's AGENTS.md is a link into a + // dotfiles folder that isn't there; Gemini's block is out of date. + fs.mkdirSync(path.join(home, '.codex'), { recursive: true }); + fs.writeFileSync(path.join(home, '.codex', 'config.toml'), '[mcp_servers.codegraph]\ncommand = "codegraph"\nargs = ["serve", "--mcp"]\n'); + const agentsMd = path.join(home, '.codex', 'AGENTS.md'); + const missing = path.join(home, 'dotfiles', 'AGENTS.md'); + fs.symlinkSync(missing, agentsMd, 'file'); + fs.mkdirSync(path.join(home, '.gemini'), { recursive: true }); + fs.writeFileSync(path.join(home, '.gemini', 'settings.json'), JSON.stringify({ mcpServers: { codegraph: { type: 'stdio', command: 'codegraph', args: ['serve', '--mcp'] } } }, null, 2) + '\n'); + const geminiMd = path.join(home, '.gemini', 'GEMINI.md'); + fs.writeFileSync(geminiMd, '\n## CodeGraph\n\nPrefer `codegraph_search` over grep.\n\n'); + + const work = path.join(home, 'work'); + fs.mkdirSync(work); + const env: NodeJS.ProcessEnv = { + ...process.env, + HOME: home, + USERPROFILE: home, + XDG_CONFIG_HOME: path.join(home, '.config'), + APPDATA: path.join(home, 'AppData', 'Roaming'), + LOCALAPPDATA: path.join(home, 'AppData', 'Local'), + CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_TELEMETRY: '0', + CODEGRAPH_NO_UPDATE_CHECK: '1', + DO_NOT_TRACK: '1', + NO_COLOR: '1', + }; + for (const key of ['CODEX_HOME', 'COPILOT_HOME', 'HERMES_HOME', 'CLAUDE_CONFIG_DIR', 'CODEGRAPH_NO_INSTALL_REFRESH']) delete env[key]; + + const { deps, calls } = makeDeps({ + method: { kind: 'npm', scope: 'global' }, + currentVersion: '0.9.8', + hasCommand: (cmd) => cmd === 'codegraph', + }); + let refreshOutput = ''; + deps.run = (cmd, args, runEnv) => { + calls.runs.push({ cmd, args, env: runEnv }); + if (cmd !== 'codegraph') return 0; + const r = spawnSync(process.execPath, [BIN, ...args], { cwd: work, env, encoding: 'utf-8', timeout: 60_000 }); + refreshOutput = `${r.stdout}\n${r.stderr}`; + return r.status ?? -1; + }; + + expect(await runUpgrade({}, deps)).toBe(0); + expect(calls.logs.join('\n')).toMatch(/Could not refresh the installed agent surfaces/); + expect(refreshOutput).toMatch(/Codex CLI: could not refresh — .*AGENTS\.md is a symbolic link to .*its folder does not exist/); + expect(fs.lstatSync(agentsMd).isSymbolicLink()).toBe(true); + expect(fs.existsSync(path.dirname(missing))).toBe(false); + expect(fs.readFileSync(geminiMd, 'utf-8')).toBe(CODEGRAPH_INSTRUCTIONS_BLOCK + '\n'); + }, 90_000); +}); + // --------------------------------------------------------------------------- // Post-upgrade version probe — does the PATH-resolved `codegraph` serve the // version we just installed, in THIS terminal? diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index dd0287dddb..296d950d16 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -2737,17 +2737,25 @@ program ? [opts.location as 'global' | 'local'] : ['global', 'local']; let changed = 0; + let failed = 0; for (const loc of locs) { for (const report of refreshTargets(ALL_TARGETS, loc)) { + if (report.status === 'failed') { + failed += 1; + error(`${report.displayName}: could not refresh — ${report.error}`); + } for (const p of report.changedPaths) { changed += 1; console.log(` ${report.displayName}: refreshed ${p}`); } } } - if (changed === 0) { + if (changed === 0 && failed === 0) { console.log('All configured agent surfaces are already current.'); } + // Non-zero so `codegraph upgrade` tells the user to look; every other + // agent was still refreshed. + if (failed > 0) process.exitCode = 1; return; } diff --git a/src/installer/index.ts b/src/installer/index.ts index 8930e48a70..579d595933 100644 --- a/src/installer/index.ts +++ b/src/installer/index.ts @@ -22,7 +22,7 @@ import { getTarget, resolveTargetFlag, } from './targets/registry'; -import type { AgentTarget, Location, TargetId } from './targets/types'; +import type { AgentTarget, Location, TargetId, WriteResult } from './targets/types'; // Import the lightweight submodules directly (not the ../sync barrel, which // re-exports FileWatcher and would transitively pull in ../extraction — the // installer must stay importable even when native modules can't load). @@ -380,13 +380,13 @@ export function uninstallTargets( }); } -export type RefreshStatus = 'refreshed' | 'unchanged' | 'not-configured' | 'unsupported'; +export type RefreshStatus = 'refreshed' | 'unchanged' | 'not-configured' | 'unsupported' | 'failed'; /** * Per-target outcome of a refresh sweep. `refreshed` means at least one * filesystem entry was created, updated, or removed; `unchanged` means the target was - * already current (every write reported byte-identical); the other two - * mirror `UninstallStatus`. + * already current (every write reported byte-identical); `failed` means a + * write threw (`error` says why); the other two mirror `UninstallStatus`. */ export interface RefreshReport { id: TargetId; @@ -395,6 +395,8 @@ export interface RefreshReport { status: RefreshStatus; /** Absolute paths created, updated, or removed by the refresh. */ changedPaths: string[]; + /** Why the refresh of this target failed (status `failed` only). */ + error?: string; } /** @@ -428,7 +430,16 @@ export function refreshTargets( if (!target.detect(location).alreadyConfigured) { return { ...base, status: 'not-configured' as const, changedPaths: [] }; } - const result = target.install(location, { autoAllow: false, promptHook: undefined }); + let result: WriteResult; + try { + result = target.install(location, { autoAllow: false, promptHook: undefined }); + } catch (err) { + // One agent's file that can't be written (a symbolic link to a + // read-only file, say: #2721) must not stop the rest of the sweep + // `codegraph upgrade` runs. + const error = err instanceof Error ? err.message : String(err); + return { ...base, status: 'failed' as const, changedPaths: [], error }; + } const changedPaths = result.files .filter((f) => f.action === 'created' || f.action === 'updated' || f.action === 'removed') .map((f) => f.path); diff --git a/src/installer/targets/claude.ts b/src/installer/targets/claude.ts index 3c789849d3..1f2c434897 100644 --- a/src/installer/targets/claude.ts +++ b/src/installer/targets/claude.ts @@ -35,6 +35,7 @@ import { getMcpServerConfig, jsonDeepEqual, readJsonFile, + removeFileUnlessLinked, removeMarkedSection, writeJsonFile, upsertInstructionsEntry, @@ -295,7 +296,7 @@ function cleanupLegacyLocalMcp(): WriteResult['files'][number] | null { delete config.mcpServers.codegraph; if (Object.keys(config.mcpServers).length === 0) delete config.mcpServers; if (Object.keys(config).length === 0) { - try { fs.unlinkSync(file); } catch { /* ignore */ } + removeFileUnlessLinked(file, '{}\n'); } else { writeJsonFile(file, config); } diff --git a/src/installer/targets/codex.ts b/src/installer/targets/codex.ts index 797acff1e8..d6506282ce 100644 --- a/src/installer/targets/codex.ts +++ b/src/installer/targets/codex.ts @@ -42,6 +42,7 @@ import { import { atomicWriteFileSync, getMcpServerConfig, + removeFileUnlessLinked, removeMarkedSection, upsertInstructionsEntry, } from './shared'; @@ -134,7 +135,7 @@ class CodexTarget implements AgentTarget { const { content: nextContent, action } = removeTomlTable(content, TOML_HEADER); if (action === 'removed') { if (nextContent.trim() === '') { - try { fs.unlinkSync(tomlPath); } catch { /* ignore */ } + removeFileUnlessLinked(tomlPath, ''); } else { atomicWriteFileSync(tomlPath, nextContent.trimEnd() + '\n'); } diff --git a/src/installer/targets/copilot-cli.ts b/src/installer/targets/copilot-cli.ts index c355ca29b9..bbffd0108b 100644 --- a/src/installer/targets/copilot-cli.ts +++ b/src/installer/targets/copilot-cli.ts @@ -42,6 +42,7 @@ import { getMcpServerConfig, jsonDeepEqual, readJsonFile, + removeFileUnlessLinked, writeJsonFile, } from './shared'; @@ -158,7 +159,9 @@ class CopilotCliTarget implements AgentTarget { // Nothing left but the `{}` we'd write back — delete the file so // uninstall fully reverses a from-scratch install. A leftover // empty file would keep detect() reporting the CLI as installed. - fs.unlinkSync(file); + // A symbolic link is the user's, so it is kept with `{}` written + // through it (#2721). + removeFileUnlessLinked(file, '{}\n'); } else { writeJsonFile(file, config); } diff --git a/src/installer/targets/shared.ts b/src/installer/targets/shared.ts index dca78989c9..050ba068a6 100644 --- a/src/installer/targets/shared.ts +++ b/src/installer/targets/shared.ts @@ -78,10 +78,35 @@ export function readJsonFile(filePath: string): Record { * * Prevents corruption if the process crashes mid-write. The temp * file is cleaned up on rename failure. + * + * A symbolic link is written through, never replaced (#2721). Renaming + * onto the link's own path swapped the link for a regular copy, so a + * `~/.codex/AGENTS.md -> ~/dotfiles/AGENTS.md` kept in a dotfiles repo + * stopped reaching the shared file. The write lands on the file at the end + * of the link instead, and the link itself, relative or absolute, is left + * as it was. A link to a file that doesn't exist yet gets that file + * created, but only in a folder that already exists, as writing through it + * in a shell would: no folder is ever created where a link points. When the + * linked file can't be written, the error names the link and nothing + * changes. */ export function atomicWriteFileSync(filePath: string, content: string): void { + if (!isSymbolicLink(filePath)) { + writeThenRename(filePath, content, true); + return; + } + let target = filePath; + try { + target = linkTarget(filePath); + writeThenRename(target, content, false); + } catch (err) { + throw linkWriteError(filePath, target, err); + } +} + +function writeThenRename(filePath: string, content: string, createFolder: boolean): void { const dir = path.dirname(filePath); - if (!fs.existsSync(dir)) { + if (createFolder && !fs.existsSync(dir)) { fs.mkdirSync(dir, { recursive: true }); } const tmpPath = filePath + '.tmp.' + process.pid; @@ -94,6 +119,65 @@ export function atomicWriteFileSync(filePath: string, content: string): void { } } +/** True when `p` itself is a symbolic link (on Windows, also a junction). */ +function isSymbolicLink(p: string): boolean { + try { + return fs.lstatSync(p).isSymbolicLink(); + } catch { + return false; + } +} + +/** + * The file a symbolic link ends at, through any chain of links. It may not + * exist yet: each hop of a dangling chain is resolved from the real folder + * that holds the link, as the OS resolves it, not by joining names. + */ +function linkTarget(link: string): string { + try { + return fs.realpathSync(link); + } catch { + let current = link; + for (let hop = 0; hop < 40 && isSymbolicLink(current); hop++) { + current = path.resolve(fs.realpathSync(path.dirname(current)), fs.readlinkSync(current)); + } + if (isSymbolicLink(current)) { + throw Object.assign(new Error('too many levels of symbolic links'), { code: 'ELOOP' }); + } + return current; + } +} + +function linkWriteError(link: string, target: string, cause: unknown): Error { + const code = (cause as NodeJS.ErrnoException | undefined)?.code; + const why = code === 'ENOENT' ? 'its folder does not exist' + : code === 'EACCES' || code === 'EROFS' ? 'it is read-only' + : code === 'EPERM' || code === 'EBUSY' ? 'it is read-only or in use' + : cause instanceof Error ? cause.message : String(cause); + const message = target === link + ? `${link} is a symbolic link that can't be followed (${why}).` + : `${link} is a symbolic link to ${target}, which can't be written (${why}).`; + return Object.assign( + new Error(`${message} CodeGraph writes through links and never replaces one with a copy: fix the link or make its file writable, then run this again.`, { cause }), + { code }, + ); +} + +/** + * Delete a file that codegraph's change left with nothing of the user's in + * it. A symbolic link is kept instead and `emptied` is written through it: + * the link is the user's, usually into a shared dotfiles folder, and + * deleting it would leave codegraph's entry behind in the shared file + * (#2721). + */ +export function removeFileUnlessLinked(filePath: string, emptied: string): void { + if (isSymbolicLink(filePath)) { + atomicWriteFileSync(filePath, emptied); + return; + } + try { fs.unlinkSync(filePath); } catch { /* ignore */ } +} + /** * Copy a file we're about to overwrite to `.backup` when it isn't * parseable JSON — `readJsonFile` handed the caller `{}` for it, so the @@ -344,7 +428,8 @@ export function stripMarkedSections(content: string, startMarker: string, endMar * Inverse of `replaceOrAppendMarkedSection`. Strips the marker * block from `filePath` if present. If the file becomes empty after * removal, deletes the file entirely (matches the existing Claude - * uninstall behavior). + * uninstall behavior) — unless it is a symbolic link, which is kept + * and emptied through. * * Returns `removed` when content was stripped, `not-found` when * the markers weren't present, `kept` when the file didn't exist. @@ -367,7 +452,7 @@ export function removeMarkedSection( if (rest === null) return 'not-found'; if (isBlank(rest)) { - try { fs.unlinkSync(filePath); } catch { /* ignore */ } + removeFileUnlessLinked(filePath, rest); } else { atomicWriteFileSync(filePath, rest); } From 1ef8866aae5c6d57dd8eba42dc37d3be08b36647 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 01:38:45 +0000 Subject: [PATCH 10/95] test(explore): the cross-call dedup test passes with LF and CRLF checkouts alike (#2772) explore-cross-call-dedup's "never re-sends a line it already sent" case has failed on every LF checkout (Linux, macOS, CI) since PR #2655. It passed on Windows only because core.autocrlf writes the fixture with CRLF. Root cause. Since PR #2655 the payroll fixture's RunResult struct (cycle.go lines 31-37) references Payslip through a field, so call 1 renders it. Call 2 renders the file's head, and that 7-line held run comes back inside it: dedupeRange keeps a covered run shorter than MIN_COVERED_LINES (8) in the emit set on purpose, rule 3 of the range algebra, which its own unit test pins. The handler case counted any repeated line as a duplicate. On a CRLF checkout explore's per-file character budget also counts each line's carriage return, so cycle.go rendered fewer lines and the case never met the short run. Fix, test only: - The case groups the repeated lines into consecutive runs and fails on a run of MIN_COVERED_LINES lines or more, read from EXPLORE_DEDUP. A shorter run is the algebra's deliberate re-send. What agents receive is unchanged. - The suite gives its copy of the fixture LF line endings, so every checkout indexes and renders the same bytes. Validation on Windows, with the kernel and with CODEGRAPH_KERNEL=0: - main's test fails with an LF fixture ("call 2 re-sent internal/usecase/payroll/cycle.go lines 31,32,33,34,35") and passes with the CRLF one. The new test passes 26/26 with both. - With the normalisation and main's assertion, a CRLF checkout fails the way an LF one does (lines 31-37): both checkouts now test one scenario. - Teeth: an algebra mutated to re-send held runs of 8-23 lines fails the new case ("call 2 re-sent internal/usecase/payroll/cycle.go lines 42-51 (10 lines)"). - tsc on the test file is clean. Contributor PR #2758 by @danusha2345 reached the same diagnosis. This change was written separately and also normalises the fixture. Fixes #2756 Co-Authored-By: Claude Opus 5.5 --- __tests__/explore-cross-call-dedup.test.ts | 46 ++++++++++++++++++++-- 1 file changed, 42 insertions(+), 4 deletions(-) diff --git a/__tests__/explore-cross-call-dedup.test.ts b/__tests__/explore-cross-call-dedup.test.ts index ce208dc95f..e18cc4371b 100644 --- a/__tests__/explore-cross-call-dedup.test.ts +++ b/__tests__/explore-cross-call-dedup.test.ts @@ -42,6 +42,37 @@ const FIXTURE_SRC = path.join(__dirname, 'fixtures', 'payroll-go'); const QUERY = 'how does payroll cycle create and calculate payslips?'; const POINTER = 'Already sent earlier in this conversation'; +/** + * Give every file of the copied fixture LF line endings. A Windows checkout + * with `core.autocrlf` writes the fixture with CRLF, and explore's per-file + * character budget counts each line's `\r`, so there a file renders fewer + * lines and the second call meets different held spans than on Linux or macOS + * (#2756). The suite tests one scenario on every checkout. + */ +function normalizeLineEndings(dir: string): void { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const file = path.join(dir, entry.name); + if (entry.isDirectory()) { + normalizeLineEndings(file); + continue; + } + // latin1 maps every byte to one char, so the rewrite changes nothing but the `\r`s. + const text = fs.readFileSync(file, 'latin1'); + if (text.includes('\r\n')) fs.writeFileSync(file, text.replace(/\r\n/g, '\n'), 'latin1'); + } +} + +/** Consecutive line numbers grouped into runs: [3, 4, 5, 9] → [[3, 4, 5], [9]]. */ +function consecutiveRuns(lines: Iterable): number[][] { + const runs: number[][] = []; + for (const n of [...lines].sort((a, b) => a - b)) { + const run = runs[runs.length - 1]; + if (run && n === run[run.length - 1]! + 1) run.push(n); + else runs.push([n]); + } + return runs; +} + describe('dedup configuration', () => { it('defaults off and requires an explicit truthy opt-in', () => { const previous = process.env.CODEGRAPH_EXPLORE_DEDUP; @@ -213,6 +244,7 @@ describe('a second call against a real index', () => { testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cg18-')); fs.cpSync(FIXTURE_SRC, testDir, { recursive: true }); fs.rmSync(path.join(testDir, '.codegraph'), { recursive: true, force: true }); + normalizeLineEndings(testDir); cg = CodeGraph.initSync(testDir); await cg.indexAll(); handler = new ToolHandler(cg); @@ -266,10 +298,16 @@ describe('a second call against a real index', () => { expect(second).toContain(POINTER); for (const [file, lines] of before) { const repeated = [...(after.get(file) ?? [])].filter((n) => lines.has(n)); - if (repeated.length === 0) continue; - // The only sanctioned repeat is the anti-abandonment restore, which fires - // ONLY when the call found nothing new to say — and this one did. - throw new Error(`call 2 re-sent ${file} lines ${repeated.slice(0, 5).join(',')}`); + // Two repeats are sanctioned. The anti-abandonment restore fires ONLY + // when the call found nothing new to say, and this one did. And rule 3 + // of the range algebra: a held run shorter than MIN_COVERED_LINES stays + // inside the span around it rather than becoming a pointer (cycle.go's + // 7-line `RunResult` struct, held by call 1, comes back inside call 2's + // render of the file's head, #2756). A longer run is a real duplicate. + for (const run of consecutiveRuns(repeated)) { + if (run.length < EXPLORE_DEDUP.MIN_COVERED_LINES) continue; + throw new Error(`call 2 re-sent ${file} lines ${run[0]}-${run[run.length - 1]} (${run.length} lines)`); + } } }, 120_000); From 6bd217a2630512d7a6e3d44f1a1e7204e3fa3ba0 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 01:39:08 +0000 Subject: [PATCH 11/95] fix(cpp): code that calls operator++ by its name no longer stops an index or sync (#2769) Indexing or syncing a C++ project stopped at "Resolving refs" with "Invalid regular expression: /\boperator++\s*(?:<[^<>()]*>)?\s*\(/: Nothing to repeat" as soon as its code called an overloaded operator by name, as `this->operator++();` inside `operator++(int)` does. Choosing a method's overload reads the argument list after the name, and cppParenListIn and cppOverloadParameters wrote the name into that pattern with only `~` escaped. `operator++` did not compile. `operator()` matched the name's own empty parentheses as the arguments, `operator[]` never matched, and `operator+=` matched the next `operator=`. The same helper serves every language that overloads by arity, so Scala `++` overloads and Kotlin names in backticks that hold a `(` failed the same way. The fix adds one escape helper, escapeRegExp in src/utils.ts, the same body as the private copies elsewhere. It is used at every place in src/ where a node name, a reference name, a piece of one, or a word read from source goes into a RegExp unchecked: 114 interpolations in 13 files. 79 were raw, 31 escaped only `$`, 1 `$` and `\`, 1 only `.`, and the 2 overload readers only `~`. An AST pass over all 490 dynamic RegExp constructions followed each interpolated value to its sources. The 72 it still flags were each read by hand. They are identifier-checked first, built from escaped names, constants, or the CLI/MCP glob converters. The kernel's one dynamic regex already escapes. With operator arguments now read, nlohmann/json's `p->operator[](0)` reached the argument-type scorer and picked `template operator[](T* key)`: a `size_type` parameter scored as an unknown class, a `0` as a null pointer. `size_type` and `difference_type` now count as numbers, as `size_t` does. Validation: - New __tests__/operator-method-names.test.ts (5 tests: init and sync of an operator fixture, operator() and operator+= overload choice, the json operator[](0) shape, Scala and Kotlin). With main's sources, 4 fail (3 throw "Nothing to repeat"); the json shape fails only with size_type taken out of the number types. All pass on the kernel and with CODEGRAPH_KERNEL=0. - Census with a preload that logs every throwing RegExp: main threw only in cppParenListIn (CTRE 27, folly 3); the fix threw 0 times in all 12 repos below. - A/B on main efff7457, graph dumps: - fmt, glm, rocksdb, protobuf: byte-identical. - json: 9 calls retargeted, all right (`j.at("name")` to the key overload, `v.at(i)` to size_type). - abseil-cpp: 3 retargeted, all right (operator() self-call to the two-argument overload, assign(3, 10) to the count overload). - CTRE and folly crash on main; both index now. CTRE: 10 of 12 calls retargeted against main's census run are right; 2 single-header calls reach the identical copy in include/. folly: 4 of 6 are right; 2 small_vector assign(int, int) calls link the count overload where C++ takes the unconstrained template. - Controls gin (Go), bulletproof-react (TS), ktlint (Kotlin, 86 backtick names with pattern characters) and cats (Scala symbolic methods): byte-identical. - Cost: none measurable. Medians of 3 interleaved init runs: fmt -1.5%, json +2.1%, glm +0.2%, abseil -2.5%, gin -2.1%, bulletproof-react -0.3%, rocksdb +0.4% (45.2 s vs 45.3 s). - kernel-parity on fmt + json (cpp/c): 458/669 at parity, 0 diffs, both arms. Fixes #2457 Fixes #2482 Fixes #2636 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/operator-method-names.test.ts | 285 ++++++++++++++++++++ src/graph/branch-guards.ts | 3 +- src/resolution/c-feature-variants.ts | 5 +- src/resolution/callback-synthesizer.ts | 7 +- src/resolution/frameworks/angular-router.ts | 19 +- src/resolution/frameworks/expo-modules.ts | 5 +- src/resolution/frameworks/react.ts | 11 +- src/resolution/go-func-field-synthesizer.ts | 3 +- src/resolution/name-matcher.ts | 154 +++++------ src/resolution/react-router-synthesizer.ts | 5 +- src/resolution/swift-type-visibility.ts | 3 +- src/resolution/type-parameters.ts | 3 +- src/ui-server/api/screens.ts | 3 +- src/ui-server/api/steps.ts | 3 +- src/utils.ts | 11 + 16 files changed, 416 insertions(+), 105 deletions(-) create mode 100644 __tests__/operator-method-names.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b3546c451f..90fee5a18c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -148,6 +148,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, a call made with `->` on an `absl::StatusOr`, an `absl::optional` or an `absl::NoDestructor`, like protocolbuffers/protobuf's `resolver->MergeFeatures(…)` on an `absl::StatusOr`, now links to the method of the class it holds, as one on a `std::unique_ptr` or `std::optional` does. A call written without a receiver in a class template now also finds a member of a base that depends on the template's parameters when the class brings it in with a using-declaration, like facebook/rocksdb's `using BasicTypedCacheHelperFns::DownCastValue;`, and so does a call on such a member. Re-index C++ projects after upgrading. - In C and C++, a call or a type name no longer links to something only another source file can see: a class, struct or alias a `.c` or `.cc` file declares for itself, its members, or anything inside that file's unnamed `namespace { … }`. In facebook/rocksdb, other test files' bare `Put(…)`, `Flush()` and `dbfull()` calls had linked to the methods of a test fixture declared in `column_family_test.cc`, and calls on an `Env*` had linked to a test-only environment class in `memenv.cc`. When a header and a source file each declare a class of the same name, like RocksDB's `SharedState`, a call now links to the header's. A source file's own declarations are still found from the file itself, from a file that `#include`s it, and from a header it includes below them. Re-index C and C++ projects after upgrading. - In C++, a call to an overloaded method now links to the overload its arguments fit: by how many there are, and then by their types. That covers a call on a variable or member of a known class, like facebook/rocksdb's `db->Get(ro, key, &value)` and `db->Put(write_opts, key, value)`, as well as a call written without a receiver, like `Insert(ch)` on a `char ch`. Before, a call on a variable always linked to the first overload declared, so RocksDB's convenience overloads of `DB::Get`, `DB::Put` and `DB::NewIterator` and protocolbuffers/protobuf's `io::Printer::Print`, `Emit` and `Annotate` had almost no callers, and a call without a receiver picked by argument count alone. A literal, a variable the calling function declares, `&variable`, an element read by subscript, like `container[i]` or a map's `values[key]`, and the result of a call are read for their types; a parameter's default value is read from the method's declaration in its class, and comments in argument lists are skipped. Re-index C++ projects after upgrading. +- Indexing or syncing a project no longer stops at "Resolving refs" with `Invalid regular expression … Nothing to repeat` when its code calls a method whose name is not a plain word, like a C++ `operator++` written as `this->operator++()` inside `operator++(int)`, an overloaded Scala `++`, or a Kotlin name in backticks. In C++, a call to another overload of an operator, like `this->operator()(x, 2)` or `total.operator+=(other)`, now links to the overload its arguments fit; before, `operator()` read the empty parentheses of its own name as the call's arguments and stayed linked to itself. A container's `size_type` now counts as a number when overloads are compared, so nlohmann/json's `j.at("name")` and `j.at(i)` each reach the overload that takes them. Re-index C++ projects after upgrading. Thanks @Sven-lxw, @ArronLin and @daheix for the reports and @danusha2345. (#2457, #2482, #2636) - In C++, a nested class defined outside the class it belongs to, like google/leveldb's `struct SkipList::Node { … }` or facebook/rocksdb's `struct BlockBasedTable::Rep { … }`, is now indexed under its own name, `SkipList::Node`, with its methods and members. Before, it was indexed as an unnamed class, so it was missing from search and `codegraph_explore`, its methods belonged to no class you could name, and a call like `x->Next(level)` on a `Node*` was matched by the method's name alone. Creating an object, as in `new log::Writer(…)` or `Writer w(…)`, now links to the class C++ itself finds for the type as written there: leveldb's `new log::Writer(…)` in `DBImpl` methods no longer links to `DBImpl`'s own `Writer`, RocksDB's tests' `Random rnd(301)` links to RocksDB's `Random` instead of googletest's, an object created through an alias links to the class the alias names, and one links to its class rather than to the class's constructor. An object of a template parameter's type, or of a type the function names with its own `using`, like protocolbuffers/protobuf's `using Map = std::map<…>`, no longer links to a project class that only shares the name. Re-index C++ projects after upgrading. - In C++, a call whose receiver's type can't be read no longer links to a method of a class nested in the class the receiver's name suggests, or of a class whose name only contains that name. Before, such a guess took any method of the call's name whose qualified name contained the class's name: protocolbuffers/protobuf's `descriptor->file()` in a template reached `Descriptor::ExtensionRange::file`, and facebook/rocksdb's `CompositeEnv::IsInstanceOf(name)` reached `CompositeEnvWrapper::IsInstanceOf`. A call written through a class, like `PessimisticTransaction::PopSavePoint()` in an override, now links to the method that class inherits, here `TransactionBaseImpl::PopSavePoint`. Re-index C++ projects after upgrading. - In C++, an alias that names a class through a class derived from the one that declares it, like protocolbuffers/protobuf's `using ConformanceRequestSetting = BinaryAndJsonConformanceSuite::ConformanceRequestSetting;`, now leads to that class, `ConformanceTestSuite::ConformanceRequestSetting`. Before, an alias was followed only to a class declared under exactly the name it spells, so objects created through this one linked to the alias rather than the class. Re-index C++ projects after upgrading. diff --git a/__tests__/operator-method-names.test.ts b/__tests__/operator-method-names.test.ts new file mode 100644 index 0000000000..b4ffae8fe4 --- /dev/null +++ b/__tests__/operator-method-names.test.ts @@ -0,0 +1,285 @@ +/** + * A method whose name is not a plain word: C++'s `operator++` and + * `operator()`, Scala's `++`, a Kotlin name written in backticks. The + * resolver writes names into the patterns it reads source with, and these + * went in as pattern syntax: + * + * - `operator++` is "Nothing to repeat". Choosing between a method's + * overloads reads the call's argument list after its name, so one + * `this->operator++();` inside `operator++(int)` stopped a whole index, + * and a sync after an edit, at "Resolving refs" (#2457, #2482, #2636). + * - `operator()` matched the empty `()` of its own name, so a call to the + * two-argument overload read as one with no arguments and stayed linked + * to the overload it was written in. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import type { Node } from '../src/types'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** The 1-based line of `text` that contains `needle` (exactly one must). */ +const lineOf = (text: string, needle: string): number => { + const at = text.split('\n').map((l, i) => (l.includes(needle) ? i + 1 : 0)).filter(Boolean); + if (at.length !== 1) throw new Error(`${JSON.stringify(needle)} is on ${at.length} lines`); + return at[0]!; +}; + +const COUNTER_HPP = lines( + '#pragma once', + '#include ', + '#include ', + '', + 'struct Counter {', + ' int value = 0;', + ' Counter operator++(int) {', + ' Counter before = *this;', + ' this->operator++();', + ' return before;', + ' }', + ' Counter& operator++() {', + ' ++value;', + ' return *this;', + ' }', + ' Counter& operator+=(int n) {', + ' value += n;', + ' return *this;', + ' }', + ' Counter& operator+=(const Counter& other) {', + ' value += other.value;', + ' return *this;', + ' }', + ' auto operator<=>(const Counter& other) const = default;', + ' explicit operator bool() const { return value != 0; }', + '};', + '', + 'struct Scale {', + ' int operator()(int x) {', + ' return this->operator()(x, 2);', + ' }', + ' int operator()(int x, int factor) {', + ' return x * factor;', + ' }', + '};', + '', + 'struct Table {', + ' int cells[8] = {};', + ' int& operator[](std::size_t i) { return cells[i]; }', + '};', + '', + 'struct Pointee {', + ' int field = 0;', + ' int read() const { return field; }', + '};', + '', + 'struct Handle {', + ' Pointee* target = nullptr;', + ' Pointee* operator->() const { return target; }', + ' int& operator->*(int Pointee::*m) const { return target->*m; }', + '};', + '', + 'struct Pool {', + ' static void* operator new[](std::size_t n);', + ' static void operator delete(void* p);', + ' static void operator delete[](void* p);', + ' static void* grab(std::size_t n) { return Pool::operator new[](n); }', + ' static void drop(void* p) { Pool::operator delete(p); }', + '};', + '', + 'struct Distance { long double km; };', + 'Distance operator"" _km(long double v);', +); + +const COUNTER_CPP = lines( + '#include "counter.hpp"', + '#include ', + '', + 'void* Pool::operator new[](std::size_t n) { return std::malloc(n); }', + 'void Pool::operator delete(void* p) { std::free(p); }', + 'void Pool::operator delete[](void* p) { Pool::operator delete(p); }', + 'Distance operator"" _km(long double v) { return Distance{v}; }', + '', + 'int useAll() {', + ' Counter c;', + ' c.operator++();', + ' c.operator++(0);', + ' c.operator+=(3);', + ' Counter d;', + ' d.operator+=(c);', + ' bool nonzero = c.operator bool();', + ' auto order = c.operator<=>(d);', + ' (void)order;', + ' Scale s;', + ' int scaled = s.operator()(4);', + ' Table t;', + ' t.operator[](1) = 7;', + ' Pointee pointee;', + ' Handle h{&pointee};', + ' int got = h.operator->()->read() + h.operator->*(&Pointee::field);', + ' void* arr = Pool::operator new[](32);', + ' Pool::operator delete[](arr);', + ' Distance near = operator"" _km(2.0L);', + ' return scaled + got + (nonzero ? 1 : 0) + static_cast(near.km);', + '}', +); + +describe('a method whose name is not a plain word', () => { + const roots: string[] = []; + const graphs: CodeGraph[] = []; + + afterEach(() => { + for (const cg of graphs.splice(0)) cg.close(); + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); + }); + + const project = (files: Record): string => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-operator-names-')); + roots.push(root); + for (const [rel, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), text); + } + return root; + }; + + const index = async (root: string): Promise => { + const cg = await CodeGraph.init(root, { index: true }); + graphs.push(cg); + return cg; + }; + + /** `@:` of each call from the node at `file`:`line`. */ + const callsFrom = (cg: CodeGraph, file: string, line: number): string[] => { + const from = cg.getNodesInFile(file).find((n: Node) => n.startLine === line && (n.kind === 'method' || n.kind === 'function')); + expect(from, `${file}:${line}`).toBeDefined(); + return cg.getCallees(from!.id).filter((r) => r.edge.kind === 'calls') + .map((r) => `${r.node.qualifiedName}@${path.basename(r.node.filePath)}:${r.node.startLine}`) + .sort(); + }; + + const hpp = (needle: string): string => `counter.hpp:${lineOf(COUNTER_HPP, needle)}`; + const cpp = (needle: string): string => `counter.cpp:${lineOf(COUNTER_CPP, needle)}`; + + it('C++: an index of overloaded operators that call each other by name finishes, and each call reaches its overload', async () => { + const cg = await index(project({ 'src/counter.hpp': COUNTER_HPP, 'src/counter.cpp': COUNTER_CPP })); + + // `this->operator++();` inside the postfix `operator++(int)`: the prefix one. + expect(callsFrom(cg, 'src/counter.hpp', lineOf(COUNTER_HPP, 'Counter operator++(int)'))) + .toEqual([`Counter::operator++@${hpp('Counter& operator++()')}`]); + // `this->operator()(x, 2)` inside the one-argument `operator()`: the two-argument one. + expect(callsFrom(cg, 'src/counter.hpp', lineOf(COUNTER_HPP, 'int operator()(int x) {'))) + .toEqual([`Scale::operator()@${hpp('int operator()(int x, int factor)')}`]); + expect(callsFrom(cg, 'src/counter.hpp', lineOf(COUNTER_HPP, 'static void* grab'))) + .toEqual([`Pool::operator new[]@${cpp('void* Pool::operator new[]')}`]); + expect(callsFrom(cg, 'src/counter.hpp', lineOf(COUNTER_HPP, 'static void drop'))) + .toEqual([`Pool::operator delete@${cpp('void Pool::operator delete(void* p)')}`]); + + const useAll = callsFrom(cg, 'src/counter.cpp', lineOf(COUNTER_CPP, 'int useAll()')); + expect(useAll).toEqual(expect.arrayContaining([ + `Counter::operator++@${hpp('Counter& operator++()')}`, + `Counter::operator++@${hpp('Counter operator++(int)')}`, + `Counter::operator+=@${hpp('Counter& operator+=(int n)')}`, + `Counter::operator+=@${hpp('Counter& operator+=(const Counter& other)')}`, + `Counter::operator<=>@${hpp('auto operator<=>')}`, + `Scale::operator()@${hpp('int operator()(int x) {')}`, + `Table::operator[]@${hpp('int& operator[]')}`, + `Handle::operator->@${hpp('Pointee* operator->()')}`, + `Handle::operator->*@${hpp('int& operator->*')}`, + `Pool::operator new[]@${cpp('void* Pool::operator new[]')}`, + `Pool::operator delete[]@${cpp('void Pool::operator delete[]')}`, + `operator"" _km@${cpp('Distance operator"" _km(long double v) {')}`, + ])); + }, 120_000); + + it('C++: a sync that adds such calls to an indexed project finishes, and links them', async () => { + // Indexed first without a single call of `operator++` by its name. + const strip = (text: string): string => text + .replace(' this->operator++();\n', ' ++value;\n') + .replace(' c.operator++();\n', '') + .replace(' c.operator++(0);\n', ''); + const root = project({ 'src/counter.hpp': strip(COUNTER_HPP), 'src/counter.cpp': strip(COUNTER_CPP) }); + const cg = await index(root); + expect(callsFrom(cg, 'src/counter.hpp', lineOf(COUNTER_HPP, 'Counter operator++(int)'))).toEqual([]); + + fs.writeFileSync(path.join(root, 'src/counter.hpp'), COUNTER_HPP); + fs.writeFileSync(path.join(root, 'src/counter.cpp'), COUNTER_CPP); + await cg.sync(); + + expect(callsFrom(cg, 'src/counter.hpp', lineOf(COUNTER_HPP, 'Counter operator++(int)'))) + .toEqual([`Counter::operator++@${hpp('Counter& operator++()')}`]); + expect(callsFrom(cg, 'src/counter.cpp', lineOf(COUNTER_CPP, 'int useAll()'))).toEqual(expect.arrayContaining([ + `Counter::operator++@${hpp('Counter& operator++()')}`, + `Counter::operator++@${hpp('Counter operator++(int)')}`, + ])); + }, 120_000); + + it('C++: the argument list of an `operator()` call is the one after the name, not the name\'s own parentheses', async () => { + // No `operator++` here: nothing threw before, the calls just went to the wrong overloads. + const hppText = lines( + 'struct Scale {', + ' int operator()(int x) { return this->operator()(x, 2); }', + ' int operator()(int x, int factor) { return x * factor; }', + '};', + 'struct Total {', + ' int value = 0;', + ' Total& operator+=(int n) { value += n; return *this; }', + ' Total& operator+=(const Total& other) { value += other.value; return *this; }', + '};', + ); + const cppText = lines( + '#include "scale.hpp"', + 'int addUp() {', + ' Total a;', + ' Total b;', + ' b.operator+=(a);', + ' return b.value;', + '}', + ); + const cg = await index(project({ 'src/scale.hpp': hppText, 'src/scale.cpp': cppText })); + expect(callsFrom(cg, 'src/scale.hpp', lineOf(hppText, 'int operator()(int x) {'))) + .toEqual([`Scale::operator()@scale.hpp:${lineOf(hppText, 'int operator()(int x, int factor)')}`]); + expect(callsFrom(cg, 'src/scale.cpp', lineOf(cppText, 'int addUp()'))) + .toEqual([`Total::operator+=@scale.hpp:${lineOf(hppText, 'Total& operator+=(const Total& other)')}`]); + }, 120_000); + + it('C++: `operator[](0)` still reaches the `size_type` overload, now that its arguments are read', async () => { + // nlohmann/json's `p = &p->operator[](0);`: the index overload, as before, not the C-string key one. + // A container's `size_type` is a number, which a `0` fits better than it fits a pointer. + const hppText = lines( + 'struct Json {', + ' using size_type = unsigned long;', + ' Json& operator[](size_type idx) { return *this; }', + ' template ', + ' JSON_NON_NULL(2)', + ' Json& operator[](T* key) { return *this; }', + '};', + 'Json* walk(Json* p) { return &p->operator[](0); }', + ); + const cg = await index(project({ 'src/json.hpp': hppText })); + expect(callsFrom(cg, 'src/json.hpp', lineOf(hppText, 'Json* walk('))) + .toEqual([`Json::operator[]@json.hpp:${lineOf(hppText, 'operator[](size_type idx)')}`]); + }, 120_000); + + it('Scala and Kotlin: overloads named `++` or in backticks no longer stop the index', async () => { + const scala = lines( + 'class Vec(val x: Int) {', + ' def ++(o: Vec): Vec = new Vec(x + o.x)', + ' def ++(n: Int, m: Int): Vec = this.++(new Vec(n + m))', + ' def twice: Vec = this.++(1, 2)', + '}', + ); + const kotlin = lines( + 'class Weird {', + ' fun `reads (a value`(x: Int): Int = x', + ' fun `reads (a value`(x: Int, y: Int): Int = `reads (a value`(x + y)', + ' fun use(): Int = `reads (a value`(1, 2)', + '}', + ); + const cg = await index(project({ 'src/Vec.scala': scala, 'src/Weird.kt': kotlin })); + expect(callsFrom(cg, 'src/Vec.scala', lineOf(scala, 'def twice'))).toEqual([`Vec::++@Vec.scala:${lineOf(scala, 'def ++(n: Int, m: Int)')}`]); + expect(callsFrom(cg, 'src/Weird.kt', lineOf(kotlin, 'fun use()'))).toEqual([`Weird::\`reads (a value\`@Weird.kt:${lineOf(kotlin, 'x: Int, y: Int')}`]); + }, 120_000); +}); diff --git a/src/graph/branch-guards.ts b/src/graph/branch-guards.ts index bc45ef45e7..8f86fd666a 100644 --- a/src/graph/branch-guards.ts +++ b/src/graph/branch-guards.ts @@ -34,6 +34,7 @@ import type { Node as SyntaxNode, Tree } from 'web-tree-sitter'; import type { Language } from '../types'; import { getParser, loadGrammarsForLanguages } from '../extraction/grammars'; import { parseWithinBudget } from '../extraction/parse-budget'; +import { escapeRegExp } from '../utils'; // ============================================================================= // Public shape @@ -576,7 +577,7 @@ function statusSetBefore(call: SyntaxNode, callee: string): number | null { const receiver = bare.split('.')[0]!; let statement: SyntaxNode | null = call; while (statement.parent && !STATEMENT_BLOCKS.has(statement.parent.type)) statement = statement.parent; - const re = new RegExp(`^\\s*(?:await\\s+)?${receiver}\\s*\\.\\s*(?:status|code)\\s*\\(\\s*([1-5]\\d{2})\\s*\\)\\s*;?\\s*$|^\\s*${receiver}\\s*\\.\\s*statusCode\\s*=\\s*([1-5]\\d{2})\\s*;?\\s*$`); + const re = new RegExp(`^\\s*(?:await\\s+)?${escapeRegExp(receiver)}\\s*\\.\\s*(?:status|code)\\s*\\(\\s*([1-5]\\d{2})\\s*\\)\\s*;?\\s*$|^\\s*${escapeRegExp(receiver)}\\s*\\.\\s*statusCode\\s*=\\s*([1-5]\\d{2})\\s*;?\\s*$`); let prev: SyntaxNode | null = statement.previousNamedSibling; for (let i = 0; prev && i < STATUS_LOOKBACK; i++, prev = prev.previousNamedSibling) { if (prev.type === 'comment') continue; diff --git a/src/resolution/c-feature-variants.ts b/src/resolution/c-feature-variants.ts index ae3fafa0d9..6df07881a2 100644 --- a/src/resolution/c-feature-variants.ts +++ b/src/resolution/c-feature-variants.ts @@ -26,6 +26,7 @@ import type { Node } from '../types'; import type { ResolutionContext } from './types'; import { directiveLines } from './cpp-macro-visibility'; +import { escapeRegExp } from '../utils'; /** Three-valued: true / false / undefined = the build may go either way. */ type Truth = boolean | undefined; @@ -130,7 +131,7 @@ function isOff(groups: Directive[][], file: FileDirectives, scope: string[], con if (name.startsWith('_') || PLATFORM_FLAGS.has(name) || file.defines.has(name)) return undefined; let named = memo.named.get(name); // A word of its own, or a compiler flag's `-DX` / `/DX`. - if (!named) memo.named.set(name, (named = new RegExp(`(?:^|[^\\w]|[-/]D)${name}\\b`, 'm'))); + if (!named) memo.named.set(name, (named = new RegExp(`(?:^|[^\\w]|[-/]D)${escapeRegExp(name)}\\b`, 'm'))); if (dirs.some((dir) => named!.test(buildTextOf(dir, context, memo)))) return undefined; const components = new Set(scope.map((f) => componentOf(dirOf(f), context, memo))); return [...components].some((component) => isDefinedIn(name, component, context, memo)) ? undefined : false; @@ -299,7 +300,7 @@ function isDefinedIn(name: string, component: string, context: ResolutionContext if (hit !== undefined) return hit; memo.cFiles ??= context.getAllFiles().filter((file) => C_FAMILY_FILE.test(file)); const prefix = component ? `${component}/` : ''; - const define = new RegExp(`^[ \\t]*#[ \\t]*define[ \\t]+${name}\\b`, 'm'); + const define = new RegExp(`^[ \\t]*#[ \\t]*define[ \\t]+${escapeRegExp(name)}\\b`, 'm'); const defined = memo.cFiles.some((file) => file.startsWith(prefix) && (context.fileContains ? context.fileContains(file, name) : true) && diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index fa7ed1da70..0dac3e1e1d 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -26,6 +26,7 @@ import type { QueryBuilder } from '../db/queries'; import type { ResolutionContext } from './types'; import { isGeneratedFile } from '../extraction/generated-detection'; import { stripCommentsForRegex } from './strip-comments'; +import { escapeRegExp } from '../utils'; import { cFnPointerDispatchEdges } from './c-fnptr-synthesizer'; import { goframeRouteEdges } from './goframe-synthesizer'; import { goFuncFieldEdges } from './go-func-field-synthesizer'; @@ -183,7 +184,7 @@ async function fieldChannelEdges(queries: QueryBuilder, ctx: ResolutionContext, (d) => d.node.filePath === reg.node.filePath && d.field === reg.field ); if (chDispatchers.length === 0) continue; - const argRe = new RegExp(`${reg.node.name}\\s*\\(\\s*(this\\.\\w+|\\w+)\\s*(?=[,)])`); + const argRe = new RegExp(`${escapeRegExp(reg.node.name)}\\s*\\(\\s*(this\\.\\w+|\\w+)\\s*(?=[,)])`); let added = 0; for (const e of queries.getIncomingEdges(reg.node.id, ['calls'])) { if (added >= MAX_CALLBACKS_PER_CHANNEL) break; @@ -2078,7 +2079,7 @@ function writtenBare(src: string, name: string): boolean { * a parameter or local of a function elsewhere in the file binds nothing here. */ function ownLinesBind(name: string, src: string): boolean { - const n = name.replace(/\$/g, '\\$'); + const n = escapeRegExp(name); return new RegExp(`\\b(?:const|let|var)\\s+${n}\\b(?!\\s*[,\\]}])`).test(src); } @@ -2828,7 +2829,7 @@ function constantEventName(expr: string, file: string, ctx: ResolutionContext, m const from = importedFile(owner); decls = from ? declaredIn(from) - : ctx.getNodesByName(name).filter((n) => CONSTANT_KINDS.has(n.kind) && new RegExp(`(?:^|[.:])${owner}(?:[.:]|$)`).test(n.qualifiedName.slice(0, n.qualifiedName.lastIndexOf(name)))); + : ctx.getNodesByName(name).filter((n) => CONSTANT_KINDS.has(n.kind) && new RegExp(`(?:^|[.:])${escapeRegExp(owner)}(?:[.:]|$)`).test(n.qualifiedName.slice(0, n.qualifiedName.lastIndexOf(name)))); } const values = new Set(decls.map(read).filter((v): v is string => v !== null)); value = values.size === 1 ? [...values][0]! : null; diff --git a/src/resolution/frameworks/angular-router.ts b/src/resolution/frameworks/angular-router.ts index 5d37951a99..d0ebfc65fe 100644 --- a/src/resolution/frameworks/angular-router.ts +++ b/src/resolution/frameworks/angular-router.ts @@ -46,6 +46,7 @@ import type { Node } from '../../types'; import type { FrameworkExtractionResult, FrameworkResolver, ResolutionContext, ResolvedRef, UnresolvedRef } from '../types'; import { resolveImportPath } from '../import-resolver'; import { stripCommentsForRegex } from '../strip-comments'; +import { escapeRegExp } from '../../utils'; import { matchBracket, readFields, skipString, topLevelObjects } from './object-literal'; import { dependsOn } from './package-deps'; import { @@ -516,22 +517,22 @@ export function constantText(expr: string, fromFile: string, context: Resolution } return null; }; - const decl = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*(?::[^=]+)?=\s*\{`).exec(safe); - const enumDecl = decl ? null : new RegExp(String.raw`\benum\s+${root}\s*\{`).exec(safe); - const classDecl = decl || enumDecl ? null : new RegExp(String.raw`\bclass\s+${root}\b[^{]*\{`).exec(safe); + const decl = new RegExp(String.raw`\b(?:const|let)\s+${escapeRegExp(root)}\s*(?::[^=]+)?=\s*\{`).exec(safe); + const enumDecl = decl ? null : new RegExp(String.raw`\benum\s+${escapeRegExp(root)}\s*\{`).exec(safe); + const classDecl = decl || enumDecl ? null : new RegExp(String.raw`\bclass\s+${escapeRegExp(root)}\b[^{]*\{`).exec(safe); if (decl) value = readChain(decl.index + decl[0].length - 1, chain); else if (enumDecl && chain.length === 1) { // `enum Paths { Board = 'board' }` const open = enumDecl.index + enumDecl[0].length - 1; const body = safe.slice(open, Math.max(open, matchBracket(safe, open))); - value = new RegExp(String.raw`\b${chain[0]}\s*=\s*(['"\`][^'"\`]*['"\`])`).exec(body)?.[1] ?? null; + value = new RegExp(String.raw`\b${escapeRegExp(chain[0]!)}\s*=\s*(['"\`][^'"\`]*['"\`])`).exec(body)?.[1] ?? null; } else if (classDecl) { // `class ProjectConst { static readonly IssueId = 'issueId' }`, and // `class RouterUtil { static Configuration = { Visualizer: 'visualizer' } }`. const open = classDecl.index + classDecl[0].length - 1; const close = matchBracket(safe, open); const body = close > open ? safe.slice(open, close) : ''; - const field = new RegExp(String.raw`\bstatic\s+(?:readonly\s+)?${chain[0]}\s*(?::[^=;]+)?=\s*`).exec(body); + const field = new RegExp(String.raw`\bstatic\s+(?:readonly\s+)?${escapeRegExp(chain[0]!)}\s*(?::[^=;]+)?=\s*`).exec(body); if (field) { const at = open + field.index + field[0].length; if (safe[at] === '{') value = chain.length > 1 ? readChain(at, chain.slice(1)) : null; @@ -550,12 +551,12 @@ export function constantText(expr: string, fromFile: string, context: Resolution function declarationSource(root: string, file: string, context: ResolutionContext, hops: number): string | null { const content = context.readFile(file); if (!content) return null; - if (new RegExp(String.raw`\b(?:const|let|class|enum)\s+${root}\b`).test(content)) return content; + if (new RegExp(String.raw`\b(?:const|let|class|enum)\s+${escapeRegExp(root)}\b`).test(content)) return content; if (hops >= 3) return null; const reexports = /\bexport\s+(\*|\{[^}]*\})\s*from\s+(['"])([^'"]+)\2/g; let m: RegExpExecArray | null; while ((m = reexports.exec(content)) !== null) { - if (m[1] !== '*' && !new RegExp(String.raw`\b${root}\b`).test(m[1]!)) continue; + if (m[1] !== '*' && !new RegExp(String.raw`\b${escapeRegExp(root)}\b`).test(m[1]!)) continue; const target = resolveImportPath(m[3]!, file, 'typescript', context); const found = target ? declarationSource(root, target, context, hops + 1) : null; if (found) return found; @@ -577,7 +578,7 @@ function localAlias(expr: string, fromFile: string, context: ResolutionContext): if ((local ?? prop) === root && prop) return [`${m[2]!.replace(/\s+/g, '')}.${prop}`, ...rest].join('.'); } } - const simple = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*=\s*([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)+)\s*;`).exec(content); + const simple = new RegExp(String.raw`\b(?:const|let)\s+${escapeRegExp(root)}\s*=\s*([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)+)\s*;`).exec(content); return simple ? [simple[1]!.replace(/\s+/g, ''), ...rest].join('.') : null; } @@ -868,7 +869,7 @@ export function namesSomewhere(href: HrefLiteral | null): HrefLiteral | null { /** A class property's initializer, read from its body: `routerLinkAbout = publicRoutes.about.routerLink;`. */ export function propertyInitializer(owner: Node, name: string, content: string): string | null { const body = content.split('\n').slice(owner.startLine - 1, owner.endLine).join('\n'); - const decl = new RegExp(String.raw`(?:^|[\s;{])(?:(?:public|private|protected|readonly|static|override)\s+)*${name}\s*(?::[^=;\n]+)?=\s*`).exec(body); + const decl = new RegExp(String.raw`(?:^|[\s;{])(?:(?:public|private|protected|readonly|static|override)\s+)*${escapeRegExp(name)}\s*(?::[^=;\n]+)?=\s*`).exec(body); if (!decl) return null; let i = decl.index + decl[0].length; const start = i; diff --git a/src/resolution/frameworks/expo-modules.ts b/src/resolution/frameworks/expo-modules.ts index c70d1d5acd..244ce17795 100644 --- a/src/resolution/frameworks/expo-modules.ts +++ b/src/resolution/frameworks/expo-modules.ts @@ -48,6 +48,7 @@ import { } from '../types'; import { resolveImportPath } from '../import-resolver'; import { stripCommentsForRegex } from '../strip-comments'; +import { escapeRegExp } from '../../utils'; /** * Match `Function("name")`, `AsyncFunction("name")`, or `Property("name")` @@ -234,7 +235,7 @@ function nativeBinding(name: string, file: string, context: ResolutionContext): let found: NativeBinding | null = null; const own = context.readFile(file); if (own && own.includes('NativeModule')) { - const local = new RegExp(String.raw`\b(?:const|let|var)\s+${name}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source).exec(stripCommentsForRegex(own, 'typescript')); + const local = new RegExp(String.raw`\b(?:const|let|var)\s+${escapeRegExp(name)}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source).exec(stripCommentsForRegex(own, 'typescript')); if (local) found = { type: local[1] ?? null, module: local[2]!, file }; } if (!found) { @@ -246,7 +247,7 @@ function nativeBinding(name: string, file: string, context: ResolutionContext): const safe = stripCommentsForRegex(content, 'typescript'); const pattern = binding!.isDefault ? new RegExp(String.raw`\bexport\s+default\s+` + REQUIRE_NATIVE.source) - : new RegExp(String.raw`\bexport\s+(?:const|let)\s+${binding!.exportedName}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source); + : new RegExp(String.raw`\bexport\s+(?:const|let)\s+${escapeRegExp(binding!.exportedName)}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source); const m = pattern.exec(safe); if (m) found = { type: m[1] ?? null, module: m[2]!, file: target }; } diff --git a/src/resolution/frameworks/react.ts b/src/resolution/frameworks/react.ts index 93eccf257f..b46e051f81 100644 --- a/src/resolution/frameworks/react.ts +++ b/src/resolution/frameworks/react.ts @@ -11,6 +11,7 @@ import { dependsOn } from './package-deps'; import { resolveImportPath } from '../import-resolver'; import { defaultExportBinding, extractLocalExportAliases } from '../alias-binding'; import { stripCommentsForRegex } from '../strip-comments'; +import { escapeRegExp } from '../../utils'; import { FN_KINDS, makeLineAt } from '../synth-utils'; import { isTestPath } from '../../search/query-utils'; import { matchBracket, skipString } from './object-literal'; @@ -316,7 +317,7 @@ function constantPathValue(expr: string, fromFile: string, context: ResolutionCo if (!decl) return null; const lines = context.readFile(file)?.split('\n') ?? []; const text = lines.slice(decl.startLine - 1, decl.endLine).join('\n'); - const open = text.indexOf('{', Math.max(0, text.search(new RegExp(`\\b${name}\\b`)))); + const open = text.indexOf('{', Math.max(0, text.search(new RegExp(`\\b${escapeRegExp(name)}\\b`)))); return open < 0 ? null : readObjectPath(text, open, keys); } @@ -348,7 +349,7 @@ export function configHrefExpression(expr: string, fromFile: string, context: Re if (!decl) return null; const lines = context.readFile(file)?.split('\n') ?? []; const text = lines.slice(decl.startLine - 1, decl.endLine).join('\n'); - const open = text.indexOf('{', Math.max(0, text.search(new RegExp(`\\b${name}\\b`)))); + const open = text.indexOf('{', Math.max(0, text.search(new RegExp(`\\b${escapeRegExp(name)}\\b`)))); if (open < 0) return null; let value = readObjectValue(text, open, keys); if (value === null) return null; @@ -611,7 +612,7 @@ function ownExport(source: string, name: string): string | null { if (name === 'default') { const declared = defaultExportedName(source); if (declared) return declared; - } else if (new RegExp(String.raw`\bexport\s+(?:async\s+)?(?:const|let|var|function|class)\s+${name.replace(/\$/g, '\\$')}(?![\w$])`).test(source)) { + } else if (new RegExp(String.raw`\bexport\s+(?:async\s+)?(?:const|let|var|function|class)\s+${escapeRegExp(name)}(?![\w$])`).test(source)) { return name; } return extractLocalExportAliases(source).find((a) => a.exportedName === name)?.localName ?? null; @@ -2112,7 +2113,7 @@ function exportedTable(file: string, exported: string, call: boolean, context: R } else local = /\bexport\s*\{[^}]*?\b([A-Za-z_$][\w$]*)\s+as\s+default\b/.exec(safe)?.[1] ?? ''; } else { // `export { appRoutes as routes }` - const alias = new RegExp(`\\bexport\\s*\\{[^}]*?\\b([A-Za-z_$][\\w$]*)\\s+as\\s+${exported.replace(/\$/g, '\\$')}\\b`).exec(safe); + const alias = new RegExp(`\\bexport\\s*\\{[^}]*?\\b([A-Za-z_$][\\w$]*)\\s+as\\s+${escapeRegExp(exported)}\\b`).exec(safe); if (alias) local = alias[1]!; } // What a module exports is declared at its top level. @@ -2143,7 +2144,7 @@ function declaredTable( call: boolean, visible: (keywords: number[]) => number | undefined ): TableSite[] { - const id = name.replace(/\$/g, '\\$'); + const id = escapeRegExp(name); const decls = [...safe.matchAll(new RegExp(`(?:^|[^\\w$.])((?:const|let|var)\\s+${id}\\s*(?::[^=;]*)?=\\s*)`, 'g'))]; const chosen = decls.length > 0 ? visible(decls.map((d) => d.index + d[0].length - d[1]!.length)) : undefined; const decl = chosen === undefined ? undefined : decls[chosen]; diff --git a/src/resolution/go-func-field-synthesizer.ts b/src/resolution/go-func-field-synthesizer.ts index 12732240f4..e07b6d56de 100644 --- a/src/resolution/go-func-field-synthesizer.ts +++ b/src/resolution/go-func-field-synthesizer.ts @@ -52,6 +52,7 @@ import type { QueryBuilder } from '../db/queries'; import type { ResolutionContext } from './types'; import type { MaybeYield } from './cooperative-yield'; import { enclosingFn } from './synth-utils'; +import { escapeRegExp } from '../utils'; import { blankGoCode, goLocalDeclarationAt, GO_TYPE_KINDS } from './name-matcher'; /** One field of a Go struct, in declaration order. */ @@ -686,7 +687,7 @@ export async function goFuncFieldEdges( // The literal's `{` is on its type's line: Go puts no line break between them. const lineStart = f.lineStarts[lit.line - 1] ?? 0; const lineText = f.text.slice(lineStart, f.lineStarts[lit.line] ?? f.text.length); - const typeRe = new RegExp(String.raw`(? n.filePath === cls.filePath); if (own.length) return own; return [...new Map(pythonBases(cls, ref, context).flatMap(p => pythonMembers(p, member, ref, context, seen)).map(n => [n.id, n])).values()]; @@ -563,7 +564,7 @@ function pythonLineBindings(lines: string[], index: number, statements: boolean[ return out; } if (!line.includes(name)) return out; - const word = new RegExp(`(? word.test(t))) { out.push({ kind: 'other' }); } } else { - const annotated = line.match(new RegExp(`^\\s*${name}\\s*:\\s*["']?([\\w.]+)["']?\\s*$`)); + const annotated = line.match(new RegExp(`^\\s*${escapeRegExp(name)}\\s*:\\s*["']?([\\w.]+)["']?\\s*$`)); if (annotated) out.push({ kind: 'assign', type: annotated[1]!, value: '', line: index }); else if (new RegExp(`^\\s*del\\b`).test(line)) out.push({ kind: 'other' }); } } - if (new RegExp(`\\b${name}[ \\t]*:=|\\bas[ \\t]+${name}\\b`).test(line)) out.push({ kind: 'other' }); + if (new RegExp(`\\b${escapeRegExp(name)}[ \\t]*:=|\\bas[ \\t]+${escapeRegExp(name)}\\b`).test(line)) out.push({ kind: 'other' }); for (const loop of line.matchAll(/\bfor\s+([^:]+?)\s+in\b/g)) if (word.test(loop[1]!)) out.push({ kind: 'other' }); for (const lambda of line.matchAll(/\blambda\b([^:]*):/g)) if (word.test(lambda[1]!)) out.push({ kind: 'other' }); return out; @@ -603,7 +604,7 @@ function pythonLineBindings(lines: string[], index: number, statements: boolean[ function pythonBindsLocally(name: string, ref: UnresolvedRef, context: ResolutionContext, importsBind: boolean): boolean { const lines = pythonMemberLines(ref.filePath, context); const statements = pythonStatementStarts(ref.filePath, context); - const param = new RegExp(`[(,]\\s*\\*{0,2}${name}\\s*[:=,)]`); + const param = new RegExp(`[(,]\\s*\\*{0,2}${escapeRegExp(name)}\\s*[:=,)]`); const scopes = context.getNodesInFile(ref.filePath).filter(n => (n.kind === 'function' || n.kind === 'method') && n.startLine <= ref.line && n.endLine >= ref.line) .sort((a, b) => b.startLine - a.startLine); @@ -785,7 +786,7 @@ function pythonModuleAliases(filePath: string, moduleFile: string, context: Reso if (!files.includes(moduleFile)) continue; // The mapping cannot tell `import a.b` from `import a.b as b`; the source line can. // Exactly this module (not `other.a.b`), outside string literals. - const explicit = new RegExp(`\\bimport\\s[^\\n]*(? explicit.test(blankPythonStrings(line))); (files.length === 1 ? aliases : ambiguous).add(plainDotted ? m.source : m.localName); @@ -858,9 +859,9 @@ function pythonExternalWrites(global: Node, context: ResolutionContext): { write const spell = (names: string[]) => `(?:${names.map(escape).join('|')})`; // Test files: any receiver (the double may sit on any spelling of the module). const receiver = test ? '[\\w.]+' : spell([...aliases, ...ambiguous]); - const target = new RegExp(`(? n.kind === 'method' && n.qualifiedName.startsWith(`${owner.qualifiedName}::`)); const types = new Set(); @@ -1998,7 +1999,7 @@ function luaLocalDecl(name: string, file: string, line: number, context: Resolut .sort((a, b) => (a.endLine - a.startLine) - (b.endLine - b.startLine))[0]; if (fn && (!best || best.startLine < fn.startLine)) { const lines = context.getFileLines?.(file) ?? context.readFile(file)?.split(/\r?\n/) ?? []; - const decl = new RegExp(`^\\s*local\\s+${name}\\s*=\\s*(.+?)\\s*$`); + const decl = new RegExp(`^\\s*local\\s+${escapeRegExp(name)}\\s*=\\s*(.+?)\\s*$`); for (let at = line - 1; at > fn.startLine; at--) { const m = decl.exec(lines[at - 1] ?? ''); if (!m) continue; @@ -2093,7 +2094,7 @@ function luaMemberIn(file: string, path: string[], context: ResolutionContext, d const source = (context.readFile(file) ?? '') .replace(/--\[(=*)\[[\s\S]*?\]\1\]/g, (c) => c.replace(/[^\n]/g, '')) .replace(/--[^\n]*/g, ''); - const exported = new RegExp(`(?:^|[\\s{,.])${member}\\s*=\\s*([A-Za-z_]\\w*(?:\\s*\\.\\s*[A-Za-z_]\\w*)*)\\s*(?:[,;}]|$)`, 'm').exec(source); + const exported = new RegExp(`(?:^|[\\s{,.])${escapeRegExp(member)}\\s*=\\s*([A-Za-z_]\\w*(?:\\s*\\.\\s*[A-Za-z_]\\w*)*)\\s*(?:[,;}]|$)`, 'm').exec(source); if (exported && exported[1] !== member) { const rhs = exported[1]!; const line = source.slice(0, exported.index).split('\n').length + 1; @@ -3221,8 +3222,8 @@ function isJavaTypeVisible(candidate: Node, ref: UnresolvedRef, context: Resolut const segs = candidate.qualifiedName.split('::'); const enumName = segs[segs.length - 2] ?? ''; const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1] ?? ''; - const name = ref.referenceName.replace(/\$/g, '\\$'); - if (new RegExp(`\\bcase\\b[^:;]*\\b${name}\\b`).test(line) || new RegExp(`\\b${enumName}\\s*\\.\\s*${name}\\b`).test(line)) return true; + const name = escapeRegExp(ref.referenceName); + if (new RegExp(`\\bcase\\b[^:;]*\\b${name}\\b`).test(line) || new RegExp(`\\b${escapeRegExp(enumName)}\\s*\\.\\s*${name}\\b`).test(line)) return true; const here = javaFileScope(ref.filePath, context); const enumFqn = [javaFileScope(candidate.filePath, context).pkg, ...segs.slice(1, -1)].filter((p) => p !== '').join('.'); return here.single.has(`${enumFqn}.${ref.referenceName}`) || here.demand.has(enumFqn); @@ -3238,9 +3239,9 @@ function isJavaTypeVisible(candidate: Node, ref: UnresolvedRef, context: Resolut // inner class's `outer.new Inner()` — the qualifier says which: the owner // (or the package) of this candidate. const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1] ?? ''; - if (new RegExp(`\\.\\s*new\\s+${ref.referenceName.replace(/\$/g, '\\$')}\\b`).test(line)) return true; + if (new RegExp(`\\.\\s*new\\s+${escapeRegExp(ref.referenceName)}\\b`).test(line)) return true; // (A type annotation may sit between them: jsoup's `Range.@Nullable Spans`.) - const qualifiers = [...line.matchAll(new RegExp(`([A-Za-z_$][\\w$.]*)\\s*\\.\\s*(?:@[\\w.]+(?:\\([^)]*\\))?\\s+)*${ref.referenceName.replace(/\$/g, '\\$')}\\b`, 'g'))].map((m) => m[1]!); + const qualifiers = [...line.matchAll(new RegExp(`([A-Za-z_$][\\w$.]*)\\s*\\.\\s*(?:@[\\w.]+(?:\\([^)]*\\))?\\s+)*${escapeRegExp(ref.referenceName)}\\b`, 'g'))].map((m) => m[1]!); // Nested only inside a type the file declares; a class local to a method is // the lexical rule's to judge. const ownerQn = segs.slice(0, -1).join('::'); @@ -3304,7 +3305,7 @@ function isScalaPackageObjectMemberVisible(candidate: Node, ref: UnresolvedRef, const here = scalaPackageClauses(text).join('.'); if (here === objectPkg || here.startsWith(`${objectPkg}.`)) return true; const last = objectPkg.split('.').pop()!; - return new RegExp(`^\\s*import\\s+[^\\n]*\\b${last.replace(/\$/g, '\\$')}\\b`, 'm').test(text); + return new RegExp(`^\\s*import\\s+[^\\n]*\\b${escapeRegExp(last)}\\b`, 'm').test(text); } const SFC_SCRIPT_RANGES = new WeakMap>>(); @@ -3575,7 +3576,7 @@ function isCollapsedSelfRecursion(chain: { root: string; links: string[] }, ref: if (!cls) return false; const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/) ?? []; const body = lines.slice(cls.startLine - 1, cls.endLine).join('\n'); - const f = field.replace(/\$/g, '\\$'); + const f = escapeRegExp(field); const declared = new RegExp(`(?:^|[\\s(,])#?${f}\\s*[?!]?\\s*:\\s*([A-Za-z_$][\\w$]*)`, 'm').exec(body)?.[1] ?? new RegExp(`\\bthis\\.${f}\\s*=\\s*new\\s+([A-Za-z_$][\\w$]*)`).exec(body)?.[1]; return declared === owner; @@ -3890,7 +3891,7 @@ function pythonFixtureReturnType(receiver: string, ref: UnresolvedRef, context: // The receiver must be the test's own parameter. const signature = caller && (caller.kind === 'function' || caller.kind === 'method') ? lines.slice(caller.startLine - 1, caller.startLine + 4).join(' ').split(/\)\s*(?:->[^:]*)?:/)[0] ?? '' : ''; - if (new RegExp(`[(,]\\s*${receiver}\\s*(?:[:=,)]|$)`).test(signature)) { + if (new RegExp(`[(,]\\s*${escapeRegExp(receiver)}\\s*(?:[:=,)]|$)`).test(signature)) { const fixtures = context.getNodesByName(receiver) .filter((n) => n.kind === 'function' && n.language === 'python' && isFixtureInReach(n, ref.filePath, context)) .sort((a, b) => (a.filePath === ref.filePath ? -1 : 0) - (b.filePath === ref.filePath ? -1 : 0) || b.filePath.length - a.filePath.length); @@ -3974,7 +3975,7 @@ function pythonLocalBinding(name: string, ref: UnresolvedRef, context: Resolutio PY_LOCAL_FILE.set(context, (file = { filePath: ref.filePath, lines, module: new Map() })); } const lines = file.lines; - const n = name; + const n = escapeRegExp(name); const assigns = new RegExp(`^\\s*(?:[\\w\\s,*()\\[\\]]*,\\s*)?\\(?\\*?${n}\\)?\\s*(?:,[\\w\\s,*()\\[\\]]*)?(?::[^=]+)?=(?!=)`); const targets = new RegExp(`\\bfor\\s+[\\w\\s,()]*\\b${n}\\b[\\w\\s,()]*\\s+in\\b|\\bas\\s+${n}\\b`); const params = new RegExp(`[(,]\\s*\\*{0,2}${n}\\s*(?:[:=,)]|$)`); @@ -4220,9 +4221,8 @@ function isDartConstructor(n: Node, context: ResolutionContext): boolean { const owner = cut > 0 ? n.qualifiedName.slice(0, cut).split('::').pop()! : ''; const lines = context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []; const head = (lines[n.startLine - 1] ?? '').slice(n.startColumn); - const escape = (name: string): string => name.replace(/\$/g, '\\$'); const constructor = owner !== '' && - new RegExp(String.raw`^(?:(?:const|factory|external)\s+)*${escape(owner)}\s*\.\s*${escape(n.name)}(?![\w$])`).test(head); + new RegExp(String.raw`^(?:(?:const|factory|external)\s+)*${escapeRegExp(owner)}\s*\.\s*${escapeRegExp(n.name)}(?![\w$])`).test(head); memo.set(n.id, constructor); return constructor; } @@ -4439,8 +4439,8 @@ function dartTypeCallPrefix(typeName: string, member: string, ref: UnresolvedRef if (prefixes.size === 0) return null; const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1]; if (line === undefined) return null; - const written = new RegExp(String.raw`(?]|<(?:[^<>]|<[^<>]*>)*>)*>)?\s*\.\s*${member.replace(/\$/g, '\\$')}(?![\w$])`, 'g'); + const written = new RegExp(String.raw`(?]|<(?:[^<>]|<[^<>]*>)*>)*>)?\s*\.\s*${escapeRegExp(member)}(?![\w$])`, 'g'); // The one at the reference's column, else the first: a column counted in bytes misses after non-ASCII text. const found = [...line.matchAll(written)].filter((m) => prefixes.has(m[1]!)); return (found.find((m) => m.index === ref.column) ?? found[0])?.[1] ?? null; @@ -5182,7 +5182,7 @@ function dartNameUses(code: DartCode, filePath: string, name: string, context: R const key = `${filePath}|${name}`; const hit = memo.get(key); if (hit) return hit; - const uses = [...code.text.matchAll(new RegExp(`(? m.index!); + const uses = [...code.text.matchAll(new RegExp(`(? m.index!); if (memo.size >= 4096) memo.delete(memo.keys().next().value!); memo.set(key, uses); return uses; @@ -5223,7 +5223,7 @@ function dartConstantType(constant: Node, ref: UnresolvedRef, context: Resolutio const code = dartCodeOf(constant.filePath, context); const from = code?.lineStarts[constant.startLine - 1]; if (!code || from === undefined) return null; - const head = new RegExp(`\\b(?:const|final|var)\\s+(?:([A-Za-z_$][\\w$.]*)\\s*(?:<[^;=]*>)?\\s*\\??\\s+)?${constant.name.replace(/\$/g, '\\$')}\\s*=(?![=>])`) + const head = new RegExp(`\\b(?:const|final|var)\\s+(?:([A-Za-z_$][\\w$.]*)\\s*(?:<[^;=]*>)?\\s*\\??\\s+)?${escapeRegExp(constant.name)}\\s*=(?![=>])`) .exec(code.text.slice(from, from + 600)); if (!head) return null; if (head[1] && !DART_NON_TYPE_WORDS.has(head[1])) return dartSimpleTypeName(head[1]); @@ -5244,7 +5244,7 @@ function dartFieldType(typeName: string, name: string, ref: UnresolvedRef, conte const declared = classMemberType(decl, name, context); if (declared) return dartSimpleTypeName(declared); // An abstract getter mints no node: `ProviderElement? get _listenedElement;`. - const getter = new RegExp(String.raw`(?:^|[\s;{}])([A-Za-z_$][\w$]*)\s*(?:<[^;{}=]*>)?\s*\??\s+get\s+${name.replace(/\$/g, '\\$')}\s*;`); + const getter = new RegExp(String.raw`(?:^|[\s;{}])([A-Za-z_$][\w$]*)\s*(?:<[^;{}=]*>)?\s*\??\s+get\s+${escapeRegExp(name)}\s*;`); for (const { text } of classMemberLines(decl, context)) { const m = text.includes(name) ? getter.exec(text) : null; if (m && !DART_NON_TYPE_WORDS.has(m[1]!)) return dartSimpleTypeName(m[1]!); @@ -5253,7 +5253,7 @@ function dartFieldType(typeName: string, name: string, ref: UnresolvedRef, conte const from = code?.lineStarts[decl.startLine - 1]; const to = code?.lineStarts[decl.endLine]; if (!code || from === undefined) continue; - const init = new RegExp(`\\b(?:final|var|late)\\s+${name.replace(/\$/g, '\\$')}\\s*=(?![=>])`).exec(code.text.slice(from, to)); + const init = new RegExp(`\\b(?:final|var|late)\\s+${escapeRegExp(name)}\\s*=(?![=>])`).exec(code.text.slice(from, to)); if (init) { const site: UnresolvedRef = { ...ref, filePath: decl.filePath, line: decl.startLine, column: 0 }; return dartInitializerType(code.text, from + init.index + init[0].length, site, context, depth + 1); @@ -5379,7 +5379,7 @@ function isDartGetter(n: Node, context: ResolutionContext): boolean { const lines = context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []; // The declaration from its start, and the next line for a return type that wraps. const head = `${(lines[n.startLine - 1] ?? '').slice(n.startColumn)} ${lines[n.startLine] ?? ''}`; - const getter = new RegExp(String.raw`^[^={;]*?\bget\s+${n.name.replace(/\$/g, '\\$')}(?![\w$])`).test(head); + const getter = new RegExp(String.raw`^[^={;]*?\bget\s+${escapeRegExp(n.name)}(?![\w$])`).test(head); memo.set(n.id, getter); return getter; } @@ -5394,7 +5394,7 @@ function isDartSetter(n: Node, context: ResolutionContext): boolean { const hit = memo.get(n.id); if (hit !== undefined) return hit; const line = (context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? [])[n.startLine - 1] ?? ''; - const setter = new RegExp(String.raw`^[^={;]*?\bset\s+${n.name.replace(/\$/g, '\\$')}\s*\(`).test(line.slice(n.startColumn)); + const setter = new RegExp(String.raw`^[^={;]*?\bset\s+${escapeRegExp(n.name)}\s*\(`).test(line.slice(n.startColumn)); memo.set(n.id, setter); return setter; } @@ -5678,7 +5678,7 @@ function rustGoCallShape(ref: UnresolvedRef, context: ResolutionContext): { shap let line = lines?.[ref.line - 1]; if (line === undefined) return null; const name = ref.referenceName; - const at = new RegExp(`(?= 0 && !/[.:]\s*$/.test(line.slice(0, start)); @@ -6371,7 +6371,7 @@ function swiftCallShape(ref: UnresolvedRef, context: ResolutionContext): SwiftCa const name = ref.referenceName; let start = line.startsWith(name, ref.column) && !/[\w$]/.test(line[ref.column - 1] ?? '') ? ref.column : -1; if (start < 0) { - const m = new RegExp(`(?]*>)?\\s*\\(\\s*${label}\\b`).test(head); + return new RegExp(`\\bfunc\\s+${escapeRegExp(n.name)}\\s*(?:<[^>]*>)?\\s*\\(\\s*${escapeRegExp(label)}\\b`).test(head); } /** @@ -6685,7 +6685,7 @@ function isScalaMemberInScope(n: Node, ref: UnresolvedRef, context: ResolutionCo const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split('\n')[ref.line - 1]; if (line === undefined) return true; const name = ref.referenceName; - const at = new RegExp(`(?|<-))|(?:\\(\\s*${n}\\s*(?:,[^)]*)?\\)\\s*=>)`, 'g'); if (fn) return binder.test(text) ? fn : null; // In a class body, a binder counts only inside a block still open at the site — @@ -6986,7 +6986,7 @@ function csharpSupertypesOf(typeName: string, context: ResolutionContext): strin else if (ch === '>' || ch === ')') depth = Math.max(0, depth - 1); else if (depth === 0) flat += ch; } - const bases = new RegExp(`\\b${typeName}\\s*:\\s*(.*?)(?:\\bwhere\\b|$)`).exec(flat)?.[1] ?? ''; + const bases = new RegExp(`\\b${escapeRegExp(typeName)}\\s*:\\s*(.*?)(?:\\bwhere\\b|$)`).exec(flat)?.[1] ?? ''; for (const m of bases.matchAll(/([A-Za-z_][\w.]*)/g)) names.push(m[1]!.split('.').pop()!); } memo.set(typeName, names); @@ -7233,7 +7233,7 @@ function objcCallShape(ref: UnresolvedRef, context: ResolutionContext): 'c-call' if (line === undefined) return null; const name = ref.referenceName.split(':')[0]!; if (!name) return null; - const at = new RegExp(`(?(); - const decl = new RegExp(`@property\\s*(?:\\([^)]*\\)\\s*)?([A-Z]\\w*)\\s*(?:<[^;]*>\\s*)?\\*\\s*(?:_Nullable\\s+|_Nonnull\\s+)?${name}\\b`); + const decl = new RegExp(`@property\\s*(?:\\([^)]*\\)\\s*)?([A-Z]\\w*)\\s*(?:<[^;]*>\\s*)?\\*\\s*(?:_Nullable\\s+|_Nonnull\\s+)?${escapeRegExp(name)}\\b`); for (const n of context.getNodesByName(name)) { if (n.kind !== 'property' || n.language !== 'objc') continue; const line = context.getFileLines?.(n.filePath)?.[n.startLine - 1] ?? context.readFile(n.filePath)?.split('\n')[n.startLine - 1] ?? ''; @@ -7504,7 +7504,7 @@ function objcClassFiles(name: string, ref: UnresolvedRef, context: ResolutionCon if (hit) return hit; const decls = context.getNodesByName(name).filter((n) => n.kind === 'class' && n.language === 'objc'); const primary = new Set(); - const own = new RegExp(`@interface\\s+${name}\\b(?!\\s*\\()`); + const own = new RegExp(`@interface\\s+${escapeRegExp(name)}\\b(?!\\s*\\()`); for (const decl of decls) { const lines = context.getFileLines?.(decl.filePath) ?? context.readFile(decl.filePath)?.split(/\r?\n/) ?? []; if (lines.slice(decl.startLine - 1, decl.endLine).some((l) => own.test(l))) primary.add(decl.filePath); @@ -8890,7 +8890,7 @@ export function isRustNameInScope(candidate: Node, ref: UnresolvedRef, context: // whatever path the line writes elsewhere. const pathed = line === undefined || viaSelf ? null : (line.startsWith(name, ref.column) && /::\s*$/.test(line.slice(0, ref.column)) ? /((?:[A-Za-z_]\w*\s*::\s*)*)([A-Za-z_]\w*)?\s*::\s*$/.exec(line.slice(0, ref.column)) - : !new RegExp(`(?= 0; i--) { const line = lines[i] ?? ''; if (arg === 'self' && /^\s*(?:pub(?:\([^)]*\))?\s+)?(?:unsafe\s+)?impl\b/.test(line)) { @@ -9510,7 +9510,7 @@ function isRustTypeParam(name: string, ref: UnresolvedRef, context: ResolutionCo let from = open.index + open[0].length; for (let i = from - 1; i < header.length; i++) { const ch = header[i]; - const param = (): boolean => new RegExp(`^\\s*(?:const\\s+)?${name}\\b`).test(header.slice(from, i)); + const param = (): boolean => new RegExp(`^\\s*(?:const\\s+)?${escapeRegExp(name)}\\b`).test(header.slice(from, i)); if (ch === '<') depth++; else if (ch === '>' && header[i - 1] !== '-' && --depth === 0) return param(); else if (ch === ',' && depth === 1) { @@ -10312,7 +10312,7 @@ function rustIsAsyncFn(fn: Node, context: ResolutionContext): boolean { if (!scopes) return false; const start = rustNodeOffset(fn, context); const head = scopes.code.slice(start, start + 300); - const name = head.search(new RegExp(`\\bfn\\s+(?:r#)?${fn.name.replace(/^r#/, '')}\\b`)); + const name = head.search(new RegExp(`\\bfn\\s+(?:r#)?${escapeRegExp(fn.name.replace(/^r#/, ''))}\\b`)); return name >= 0 && /\basync\b/.test(head.slice(0, name)); } @@ -10618,7 +10618,7 @@ function readRustDeclaredFields(decl: Node, context: ResolutionContext, generics if (!scopes) return null; const code = scopes.code; const start = rustNodeOffset(decl, context); - const name = new RegExp(`\\b${decl.name}\\b`).exec(code.slice(start, start + 512)); + const name = new RegExp(`\\b${escapeRegExp(decl.name)}\\b`).exec(code.slice(start, start + 512)); if (!name) return null; let p = start + name.index + decl.name.length; while (p < code.length && /\s/.test(code[p]!)) p++; @@ -10854,7 +10854,7 @@ function rustHasBody(method: Node, context: ResolutionContext): boolean { /** What `impl Deref for Name { type Target = … }` makes a value of the type `name` hand calls on to. */ function rustDerefTarget(name: string, typeFiles: Iterable, context: ResolutionContext): RustWrittenType | null { - const header = new RegExp(String.raw`\bimpl\b\s*(?:<[^{;]*?>)?\s*(?:(?:std|core)\s*::\s*ops\s*::\s*|ops\s*::\s*)?Deref\s+for\s+(?:[\w:]+::)?${name}\b[^{;]*\{`, 'g'); + const header = new RegExp(String.raw`\bimpl\b\s*(?:<[^{;]*?>)?\s*(?:(?:std|core)\s*::\s*ops\s*::\s*|ops\s*::\s*)?Deref\s+for\s+(?:[\w:]+::)?${escapeRegExp(name)}\b[^{;]*\{`, 'g'); for (const file of typeFiles) { const code = rustScopesOf(file, context)?.code ?? ''; if (!code.includes('Deref')) continue; @@ -10872,7 +10872,7 @@ function rustAliasTarget(alias: Node, context: ResolutionContext): RustWrittenTy const scopes = rustScopesOf(alias.filePath, context); if (!scopes) return null; const start = rustNodeOffset(alias, context); - const m = new RegExp(String.raw`\btype\s+${alias.name}\b\s*(?:<([^=;]*?)>)?\s*=\s*([^;]+);`).exec(scopes.code.slice(Math.max(0, start - 64), start + 2000)); + const m = new RegExp(String.raw`\btype\s+${escapeRegExp(alias.name)}\b\s*(?:<([^=;]*?)>)?\s*=\s*([^;]+);`).exec(scopes.code.slice(Math.max(0, start - 64), start + 2000)); return m ? { text: m[2]!.trim(), file: alias.filePath, at: start, generics: m[1] ?? '' } : null; } @@ -10890,7 +10890,7 @@ function rustTraitMethods(traits: Node[], method: string, context: ResolutionCon // `trait Name<…>: Super + Other where … {` const scopes = rustScopesOf(trait.filePath, context); const start = rustNodeOffset(trait, context); - const header = scopes ? new RegExp(String.raw`\btrait\s+${trait.name}\b\s*(?:<(?:[^<>]|<(?:[^<>]|<[^<>]*>)*>)*>)?\s*:([^{;]*?)(?:\bwhere\b[^{]*)?\{`).exec(scopes.code.slice(start, start + 2000)) : null; + const header = scopes ? new RegExp(String.raw`\btrait\s+${escapeRegExp(trait.name)}\b\s*(?:<(?:[^<>]|<(?:[^<>]|<[^<>]*>)*>)*>)?\s*:([^{;]*?)(?:\bwhere\b[^{]*)?\{`).exec(scopes.code.slice(start, start + 2000)) : null; if (!header) continue; for (const bound of splitRustTopLevel(header[1]!, '+')) { const head = rustPathHead(bound); @@ -10932,7 +10932,7 @@ function rustDelegatesToOwn(method: Node, typeName: string, context: ResolutionC if (!scopes) return false; const start = scopes.lineStarts[method.startLine - 1] ?? 0; const end = scopes.lineStarts[method.endLine ?? method.startLine] ?? scopes.code.length; - return new RegExp(String.raw`\b(?:Self|${typeName})\s*::\s*${method.name}\s*\(`).test(scopes.code.slice(start, end)); + return new RegExp(String.raw`\b(?:Self|${escapeRegExp(typeName)})\s*::\s*${escapeRegExp(method.name)}\s*\(`).test(scopes.code.slice(start, end)); } /** @@ -11231,7 +11231,7 @@ export function jsCodeBindsName(name: string, fn: Node, filePath: string, line: let binds = memo.get(key); if (binds === undefined) { const text = jsFunctionCodeThrough(fn, filePath, line, context); - const n = name.replace(/\$/g, '\\$'); + const n = escapeRegExp(name); // A plain declaration. Destructuring re-binds what a call returns under // the same name — `const { t } = useI18n()`, `const { getLabel } = // useProps(props)` — which is the same-named function more often than not. @@ -11283,7 +11283,7 @@ function jsBindingEnclosesReference(name: string, fn: Node, ref: UnresolvedRef, const chars = blankStringContents(cut, regexes).split(''); for (let r = 0; r + 1 < regexes.length; r += 2) chars.fill(' ', regexes[r]! + 1, regexes[r + 1]!); const code = chars.join(''); - const n = name.replace(/\$/g, '\\$'); + const n = escapeRegExp(name); const declaration = new RegExp( `\\b(?:(?:const|let|var)\\s+${n}(?![\\w$])(?!\\s*[,\\]}])(\\s*=\\s*(?:await\\s+)?(?:require|import)\\s*\\()?|(?:function|class)\\s+${n}(?![\\w$]))`, 'g' @@ -15936,6 +15936,8 @@ const CPP_CHAR_TYPES: ReadonlySet = new Set(['char', 'wchar_t', 'char8_t const CPP_NUMBER_TYPES: ReadonlySet = new Set([ 'int', 'unsigned', 'signed', 'short', 'long', 'float', 'double', 'size_t', 'ssize_t', 'ptrdiff_t', 'int8_t', 'int16_t', 'int32_t', 'int64_t', 'uint8_t', 'uint16_t', 'uint32_t', 'uint64_t', 'intptr_t', 'uintptr_t', + // A container's own integer aliases, by convention: nlohmann/json's `operator[](size_type idx)`. + 'size_type', 'difference_type', ]); const CPP_STRING_TYPES: ReadonlySet = new Set(['string', 'wstring', 'string_view', 'wstring_view', 'basic_string', 'basic_string_view']); @@ -16553,7 +16555,7 @@ function isCppConstructorThrough(namespace: string, method: Node, context: Resol n.filePath === method.filePath && n.startLine <= method.startLine && (n.endLine ?? n.startLine) >= method.startLine); const lines = holder ? context.getFileLines?.(holder.filePath) ?? context.readFile(holder.filePath)?.split(/\r?\n/) : undefined; const head = holder && lines ? lines.slice(holder.startLine - 1, holder.startLine + 1).join(' ') : ''; - return new RegExp(String.raw`\b(?:class|struct|union)\s+(?:[A-Za-z_]\w*\s+)*${method.name}\s*<`).test(head); + return new RegExp(String.raw`\b(?:class|struct|union)\s+(?:[A-Za-z_]\w*\s+)*${escapeRegExp(method.name)}\s*<`).test(head); } /** The names a C++ file makes something of itself, which the index holds no node for (see cppNamesBroughtIn). */ @@ -18090,7 +18092,7 @@ function bindsNameItself(fn: Node, name: string, context: ResolutionContext): bo if (hit !== undefined) return hit; const lines = context.getFileLines?.(fn.filePath) ?? context.readFile(fn.filePath)?.split(/\r?\n/) ?? []; const body = lines.slice(fn.startLine - 1, fn.endLine).join('\n'); - const r = name.replace(/\$/g, '\\$'); + const r = escapeRegExp(name); const binds = new RegExp(`\\b(?:var|val|let|out\\s+[\\w.<>?]+|foreach\\s*\\(\\s*[\\w.<>?,\\s]+?)\\s+${r}\\b|\\bfor\\s*\\([^;)]*\\s${r}\\s*:|\\b${r}\\s*=>|[(,]\\s*${r}\\s*(?:,[^()]*)?\\)\\s*=>|\\b${r}\\s*(?:,[^{}]*)?->`).test(body); memo.set(key, binds); return binds; @@ -18169,7 +18171,7 @@ function angleArguments(text: string): string[] { function declaredTypeParameters(decl: Node, context: ResolutionContext): string[] { const lines = context.getFileLines?.(decl.filePath) ?? context.readFile(decl.filePath)?.split(/\r?\n/) ?? []; const head = lines.slice(decl.startLine - 1, decl.startLine + 3).join(' '); - const at = new RegExp(`\\b${decl.name}\\s*<`).exec(head); + const at = new RegExp(`\\b${escapeRegExp(decl.name)}\\s*<`).exec(head); if (!at) return []; return angleArguments(head.slice(at.index)).map((p) => /([A-Za-z_]\w*)\s*(?:extends\b.*|:.*)?$/.exec(p.replace(/^(?:in|out|reified)\s+/, ''))?.[1] ?? ''); } @@ -18178,7 +18180,7 @@ function declaredTypeParameters(decl: Node, context: ResolutionContext): string[ function headTypeArguments(cls: Node, sup: string, context: ResolutionContext): string[] { const lines = context.getFileLines?.(cls.filePath) ?? context.readFile(cls.filePath)?.split(/\r?\n/) ?? []; const head = lines.slice(cls.startLine - 1, cls.startLine + 8).join(' ').split('{')[0]!; - const at = new RegExp(`(?:[:,]|\\bextends|\\bimplements)\\s*(?:[\\w.]+\\.)?${sup}\\s*<`).exec(head); + const at = new RegExp(`(?:[:,]|\\bextends|\\bimplements)\\s*(?:[\\w.]+\\.)?${escapeRegExp(sup)}\\s*<`).exec(head); if (!at) return []; return angleArguments(head.slice(at.index)).map((a) => a.replace(/<[\s\S]*$/, '').split('.').pop()!.trim()); } @@ -18318,7 +18320,7 @@ function classMemberType(cls: Node, name: string, context: ResolutionContext): s } const key = `${cls.id}|${name}`; if (memo.has(key)) return memo.get(key)!; - const r = name.replace(/\$/g, '\\$'); + const r = escapeRegExp(name); const pattern = cls.language === 'kotlin' || cls.language === 'swift' ? new RegExp(`\\b(?:val|var|let)\\s+${r}\\s*:\\s*([A-Z][\\w.]*)`) : new RegExp(`(?:^|[\\s(,])([A-Za-z_][\\w.]*)\\s*${TYPE_ARGS}\\??\\s+${r}\\s*(?:[=;,)]|\\{)`); @@ -19839,7 +19841,7 @@ function isVueComponentMethod(n: Node): boolean { function isThisCallInOwnFile(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { if (n.filePath !== ref.filePath) return false; const line = (context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/))?.[ref.line - 1] ?? ''; - return new RegExp(String.raw`\bthis\s*\??\.\s*${n.name.replace(/\$/g, '\\$')}\s*\(`).test(line); + return new RegExp(String.raw`\bthis\s*\??\.\s*${escapeRegExp(n.name)}\s*\(`).test(line); } /** Is the root of a member call's receiver (`CameraManager` in `CameraManager.x`) one of the file's imports? */ @@ -22002,7 +22004,7 @@ function pythonClassBaseNames(cls: Node, context: ResolutionContext): string[] { const lines = pythonLines(context, cls.filePath); if (!lines) return []; const header = lines.slice(cls.startLine - 1, cls.startLine + 4).join(' '); - const bases = new RegExp(`\\bclass\\s+${cls.name}\\s*\\(([^)]*)\\)`).exec(header)?.[1]; + const bases = new RegExp(`\\bclass\\s+${escapeRegExp(cls.name)}\\s*\\(([^)]*)\\)`).exec(header)?.[1]; return (bases ?? '') .split(',') .map((b) => b.trim()) @@ -22671,7 +22673,7 @@ function matchRustSelfCall( // `(self.into_route)(state)` calls what a field holds, no method: axum's // `MakeErasedHandler::into_route` went to itself. const at = (context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/))?.[ref.line - 1]?.slice(ref.column); - if (at?.startsWith('(') && new RegExp(`^\\(\\s*self\\s*\\.\\s*${methodName}\\s*\\)`).test(at)) return null; + if (at?.startsWith('(') && new RegExp(`^\\(\\s*self\\s*\\.\\s*${escapeRegExp(methodName)}\\s*\\)`).test(at)) return null; // In a trait impl, `fn wake(self) { self.wake() }` hands the call on to the // type's own `wake`: its inherent one, or for a type from outside the // project one the index doesn't hold (tokio's `impl WakerRef for Waker`) — @@ -22858,7 +22860,7 @@ function matchRustAsMutChainCall(ref: UnresolvedRef, context: ResolutionContext) // The call's expression starts at its column with the whole chain: // `self.delay.as_mut().reset(` — `?` after `as_mut()` taking what an // `Option` / `Result` holds back out (`self.client.as_mut()?.next()`). - const lent = new RegExp(`^(self|[a-z_]\\w*)(?:\\s*\\.\\s*([a-z_]\\w*))?\\s*\\.\\s*as_(?:mut|ref)\\s*\\(\\s*\\)\\s*(\\?)?\\s*\\.\\s*${name}\\s*(?:\\(|::<)`) + const lent = new RegExp(`^(self|[a-z_]\\w*)(?:\\s*\\.\\s*([a-z_]\\w*))?\\s*\\.\\s*as_(?:mut|ref)\\s*\\(\\s*\\)\\s*(\\?)?\\s*\\.\\s*${escapeRegExp(name)}\\s*(?:\\(|::<)`) .exec(line.slice(Math.max(0, ref.column))); if (!lent) return undefined; const [, base, field, question] = lent; @@ -22942,7 +22944,7 @@ function rustCallOnType(type: string, name: string, ref: UnresolvedRef, context: function rustProjectsSelf(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/) ?? []; const start = Math.max(0, enclosingScopeStartLine(ref, context) - 1); - const bound = new RegExp(`\\blet\\s+(?:mut\\s+)?${name}\\b\\s*=\\s*(.*)$`); + const bound = new RegExp(`\\blet\\s+(?:mut\\s+)?${escapeRegExp(name)}\\b\\s*=\\s*(.*)$`); for (let i = Math.min(lines.length, ref.line) - 1; i >= start; i--) { const m = bound.exec(lines[i] ?? ''); if (m) return /^self\s*(?:\.\s*as_mut\s*\(\s*\)\s*)?\.\s*project\s*\(\s*\)/.test(m[1]!); @@ -22964,7 +22966,7 @@ function rustFieldDeclaration(owner: string, field: string, ref: UnresolvedRef, .map((s) => fileLines(s.filePath).slice(s.startLine - 1, s.endLine)); if (spans.length === 0) { const lines = fileLines(ref.filePath); - const header = new RegExp(`\\bstruct\\s+${owner}\\b`); + const header = new RegExp(`\\bstruct\\s+${escapeRegExp(owner)}\\b`); const at = lines.findIndex((l) => header.test(l.replace(/\/\/.*$/, ''))); if (at >= 0) { let depth = 0; @@ -22978,7 +22980,7 @@ function rustFieldDeclaration(owner: string, field: string, ref: UnresolvedRef, spans.push(lines.slice(at, end + 1)); } } - const declared = new RegExp(`(?:^|[\\s{(,])(?:pub(?:\\([^)]*\\))?\\s+)?${field}\\s*:(?!:)\\s*(.*)$`); + const declared = new RegExp(`(?:^|[\\s{(,])(?:pub(?:\\([^)]*\\))?\\s+)?${escapeRegExp(field)}\\s*:(?!:)\\s*(.*)$`); for (const span of spans) { for (let i = 0; i < span.length; i++) { const line = (span[i] ?? '').replace(/\/\/.*$/, ''); @@ -23649,7 +23651,7 @@ function tsIndexedPropertyType( function tsClassTypeParameter(cls: Node, name: string, context: ResolutionContext): boolean { const lines = context.getFileLines?.(cls.filePath) ?? context.readFile(cls.filePath)?.split(/\r?\n/) ?? []; const head = lines.slice(cls.startLine - 1, cls.startLine + 2).join(' ').split('{')[0]!; - const open = new RegExp(`\\b${cls.name.replace(/\$/g, '\\$')}\\s*<`).exec(head); + const open = new RegExp(`\\b${escapeRegExp(cls.name)}\\s*<`).exec(head); if (!open) return false; // The parameters between the `<` after the name and its own `>`, split // at their top level: `T`, `K extends keyof T = never`. @@ -24344,7 +24346,7 @@ function destructuredKeyTarget(calleeName: string, key: string, ref: UnresolvedR if (!callee || !sameLanguageFamily(callee.language, ref.language)) return null; const calleeText = (context.getFileLines?.(callee.filePath) ?? context.readFile(callee.filePath)?.split('\n') ?? []) .slice(callee.startLine - 1, callee.endLine).join('\n'); - if (!new RegExp(`\\breturn\\s*\\{[^]*?\\b${key}\\b`).test(calleeText)) return null; + if (!new RegExp(`\\breturn\\s*\\{[^]*?\\b${escapeRegExp(key)}\\b`).test(calleeText)) return null; const callable = (n: Node) => n.kind === 'function' || n.kind === 'method' || n.kind === 'constant' || n.kind === 'variable'; const inFile = context.getNodesInFile(callee.filePath); const inner = inFile.filter((n) => n.name === key && callable(n) && n.id !== callee.id && rangeWithin(n, callee) && @@ -24943,7 +24945,7 @@ export function matchReference( function otherSupertypeNamed(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { const name = ref.referenceName.split(/::|\./).pop()!; const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1] ?? ''; - const written = new RegExp(`^\\s*((?:[\\w$]+\\.)+)${name.replace(/\$/g, '\\$')}\\b`).exec(line.slice(Math.max(0, ref.column))); + const written = new RegExp(`^\\s*((?:[\\w$]+\\.)+)${escapeRegExp(name)}\\b`).exec(line.slice(Math.max(0, ref.column))); const qualifier = written ? written[1]!.slice(0, -1) : ''; const candidates = context.getNodesByName(name).filter((n) => n.id !== ref.fromNodeId && isSupertypeTarget(n) && sameLanguageFamily(n.language, ref.language) && @@ -25570,7 +25572,7 @@ function cppParenListAfter(file: string, line: number, column: number, name: str /** The `( … )` that follows `name` in `text`, from `column` on, without its parentheses. */ function cppParenListIn(text: string, column: number, name: string): string | null { - const at = new RegExp(`\\b${name.replace(/[~]/g, '\\$&')}\\s*(?:<[^<>()]*>)?\\s*\\(`).exec(text.slice(column)); + const at = new RegExp(`\\b${escapeRegExp(name)}\\s*(?:<[^<>()]*>)?\\s*\\(`).exec(text.slice(column)); if (!at) return null; const open = column + at.index + at[0].length - 1; let depth = 0; @@ -25640,7 +25642,7 @@ function cppOverloadParameters(n: Node, name: string, context: ResolutionContext .filter((p) => p !== 'void') .map((p) => normalizeCppTypeName(cppParameterType(p)) ?? p); const want = types(own); - const word = new RegExp(`\\b${name.replace(/[~]/g, '\\$&')}\\s*\\(`); + const word = new RegExp(`\\b${escapeRegExp(name)}\\s*\\(`); search: for (const cls of classes) { const lines = context.getFileLines?.(cls.filePath) ?? context.readFile(cls.filePath)?.split(/\r?\n/) ?? []; for (let line = cls.startLine; line <= Math.min(lines.length, cls.endLine ?? cls.startLine); line++) { diff --git a/src/resolution/react-router-synthesizer.ts b/src/resolution/react-router-synthesizer.ts index ad8be602a7..666e39f5ca 100644 --- a/src/resolution/react-router-synthesizer.ts +++ b/src/resolution/react-router-synthesizer.ts @@ -33,6 +33,7 @@ import type { Edge, Language } from '../types'; import type { ResolutionContext } from './types'; import type { MaybeYield } from './cooperative-yield'; import { stripCommentsForRegex } from './strip-comments'; +import { escapeRegExp } from '../utils'; import { isTestPath } from '../search/query-utils'; import { parseHrefExpression, routesForFile, toHref, type HrefLiteral } from './frameworks/expo-router'; import { matchBracket } from './frameworks/object-literal'; @@ -50,7 +51,7 @@ const LINK_TAGS = ['Link', 'NavLink', 'Navigate', 'Redirect', 'LinkContainer', ' /** `` for any of `tags`, the attribute anywhere in the tag. */ function linkTagPattern(tags: readonly string[]): RegExp { // An attribute before `to` may hold an arrow (`onMouseEnter={() => …}`), whose `>` is not the tag's end. - return new RegExp(`<(${tags.map((t) => t.replace(/[$]/g, '\\$&')).join('|')})\\b((?:[^>]|=>)*?)\\bto\\s*=\\s*(?:"([^"]*)"|'([^']*)'|(?=\\{))`, 'g'); + return new RegExp(`<(${tags.map(escapeRegExp).join('|')})\\b((?:[^>]|=>)*?)\\bto\\s*=\\s*(?:"([^"]*)"|'([^']*)'|(?=\\{))`, 'g'); } const LINK_TAG = linkTagPattern(LINK_TAGS); @@ -131,7 +132,7 @@ export async function reactRouterLinkEdges(ctx: ResolutionContext, onYield: Mayb if (!source || !/\bto\s*=/.test(source)) continue; const tags = linkTagsFor(file, wrappers, ctx); const pattern = tags.length === LINK_TAGS.length ? LINK_TAG : linkTagPattern(tags); - if (!new RegExp(`<(?:${tags.join('|')})\\b`).test(source)) continue; + if (!new RegExp(`<(?:${tags.map(escapeRegExp).join('|')})\\b`).test(source)) continue; const safe = stripCommentsForRegex(source, 'typescript'); const nodes = ctx.getNodesInFile(file); const lineOf = makeLineAt(safe, 1); diff --git a/src/resolution/swift-type-visibility.ts b/src/resolution/swift-type-visibility.ts index 0e1446c183..3577b272ac 100644 --- a/src/resolution/swift-type-visibility.ts +++ b/src/resolution/swift-type-visibility.ts @@ -27,6 +27,7 @@ */ import type { Node } from '../types'; import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; +import { escapeRegExp } from '../utils'; const TYPE_KINDS: ReadonlySet = new Set(['class', 'struct', 'enum', 'interface', 'protocol', 'type_alias']); @@ -229,7 +230,7 @@ function writtenName(ref: UnresolvedRef, context: ResolutionContext): string { const before = QUALIFIER_BEFORE.exec(text.slice(Math.max(0, column - 240), column)); return before ? clean(before[1]!) : name; } - const chain = new RegExp(String.raw`^((?:${QUALIFIER_SEGMENT})+)${name.replace(/[$\\]/g, '\\$&')}\b`).exec(text.slice(ref.column, ref.column + 240)); + const chain = new RegExp(String.raw`^((?:${QUALIFIER_SEGMENT})+)${escapeRegExp(name)}\b`).exec(text.slice(ref.column, ref.column + 240)); return chain ? clean(chain[1]!) : name; } diff --git a/src/resolution/type-parameters.ts b/src/resolution/type-parameters.ts index c0588cba4e..42a2a63edb 100644 --- a/src/resolution/type-parameters.ts +++ b/src/resolution/type-parameters.ts @@ -16,6 +16,7 @@ */ import type { Node } from '../types'; import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; +import { escapeRegExp } from '../utils'; const GENERIC_LANGUAGES: ReadonlySet = new Set([ 'scala', 'java', 'kotlin', 'csharp', 'typescript', 'tsx', 'swift', 'rust', 'dart', 'go', 'cpp', 'vbnet', 'arkts', @@ -143,7 +144,7 @@ function scalaMemberInScope(result: ResolvedRef, ref: UnresolvedRef, context: Re // Imported: `import cats.data.FreeT.FlatMapped._`, `Owner.{A, B}`, `Owner.*`. const text = context.readFile(ref.filePath) ?? ''; const imports = new RegExp( - String.raw`\bimport\s+[\w.]*\b${owner.name}\.(?:_|\*|given\b|\{[^}]*\b(?:_|\*|${target.name})\b[^}]*\}|${target.name}\b)`, + String.raw`\bimport\s+[\w.]*\b${escapeRegExp(owner.name)}\.(?:_|\*|given\b|\{[^}]*\b(?:_|\*|${escapeRegExp(target.name)})\b[^}]*\}|${escapeRegExp(target.name)}\b)`, ); return /^[A-Za-z_]\w*$/.test(owner.name) && /^[A-Za-z_]\w*$/.test(target.name) ? imports.test(text) : true; } diff --git a/src/ui-server/api/screens.ts b/src/ui-server/api/screens.ts index 8a79ff2d82..6d15b56fbc 100644 --- a/src/ui-server/api/screens.ts +++ b/src/ui-server/api/screens.ts @@ -35,6 +35,7 @@ import { resolveProjectFile } from '../security'; import { findIndexedFile, hasDriftedOnDisk } from './source'; import { createWhenReader } from './when'; import { toNodeRef, type WireNodeRef } from './wire'; +import { escapeRegExp } from '../../utils'; // ============================================================================= // Wire shapes @@ -586,7 +587,7 @@ function mentionsOf(cg: CodeGraph, projectRoot: string, value: Node): Edge[] { .slice(0, MAX_MENTION_FILES); if (importers.length === 0) return out; const files = cg.getNodesByIds(importers); - const word = new RegExp(`(?,?. ]*?)\\s+${n.name}\\b`).exec(sig) ?? new RegExp(`\\b${n.name}\\s*:\\s*([A-Z][\\w<>,?. ]*)`).exec(sig); + const typed = new RegExp(`(?:^|\\s)([A-Z][\\w<>,?. ]*?)\\s+${escapeRegExp(n.name)}\\b`).exec(sig) ?? new RegExp(`\\b${escapeRegExp(n.name)}\\s*:\\s*([A-Z][\\w<>,?. ]*)`).exec(sig); if (typed && !types.has(n.name)) types.set(n.name, typed[1]!.trim()); } fileTypes.set(caller.filePath, types); diff --git a/src/utils.ts b/src/utils.ts index cdc46e831f..ff8428cfb1 100644 --- a/src/utils.ts +++ b/src/utils.ts @@ -231,6 +231,17 @@ export function normalizePath(filePath: string): string { return filePath.replace(/\\/g, '/'); } +/** + * Escape `text` to match itself literally inside a regular expression. + * Symbol names are not always plain words: C++'s `operator++` and + * `operator()`, Scala's `++`, a Kotlin name in backticks. Pass every name + * through this before writing it into a `new RegExp(…)` pattern, or the name + * is read as pattern syntax and may not compile at all. + */ +export function escapeRegExp(text: string): string { + return text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + /** * Cross-process file lock using a lock file with PID tracking. * From 0de94978fc47c549ab1790b6234d5bdc7dda4799 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 01:59:51 +0000 Subject: [PATCH 12/95] fix(installer): the Codex table is found however its config.toml writes it (#2771) Root cause: the TOML helpers behind Codex's install, refresh and uninstall found CodeGraph's table with two literal searches, `[mcp_servers.codegraph]` at the start of the file or right after a newline, and detection used a plain substring check. Some valid ways of writing that table went unseen: - an indented header, which formatters that indent sub-tables write; - spaces inside the brackets or around the dot; - quoted or escaped keys. Install then appended a second `[mcp_servers.codegraph]`. A table defined twice is invalid TOML, so Codex loaded none of its config. Install and refresh also appended a table after a definition written with dotted keys or as an inline table, or under an `mcp_servers` that is an inline table or an array of tables (both invalid too). The literal search also matched header-shaped text inside a multiline string and spliced into the string. Uninstall left the user's original table, a `[mcp_servers.codegraph.env]` sub-table (a server with no command), and dotted or inline definitions in place. CRLF files came back with mixed line endings. Fix: a small structure scan of the file finds every statement (table header, array-of-tables header, key/value). It decodes key paths the way TOML reads them (bare, "basic" with escapes, 'literal', spaces around dots), tracks the table each key sits in, and folds the lines of a value that spans several into the key that opened it. On top of that scan: - The `[mcp_servers.codegraph]` table is found however its header line is written. Install and refresh replace the first one in place and remove any further copies, so a file the old bug broke parses again. Sub-tables such as `.env` are kept. - A block ends at its last key, so a comment heading the next table stays with that table. - Uninstall removes the table, every copy, its sub-tables, and definitions written with dotted keys or as an inline table, as whole statements. - An entry the user wrote with dotted keys or as an inline table is left as written, with a note. - Where a table can't be added (`mcp_servers` set as an inline table, a value or an array of tables), install refuses with a clear error and changes nothing. - An entry inside an inline `mcp_servers` table is reported by uninstall, never edited. The uninstaller now prints such notes. - Everything is written in the file's own line breaks, and the existing block is compared regardless of them. Validation: - 25 new tests in installer-targets.test.ts: - 20 on the TOML helpers: 8 header forms, CRLF, the old bug's duplicate, look-alikes (other names, header-shaped text inside strings and arrays), sub-tables, comments, 4 dotted/inline/array definitions, 2 shapes where a table can't be added, an entry inside an inline table; - 5 on Codex install, re-install, refresh and uninstall end to end. With main's toml.ts, codex.ts and installer index.ts put back, all 25 fail: 24 on the behavior itself, and one, a header with a trailing comment, only on the new detection helper, since main already handled that form. The 13 existing TOML tests pass on both. - The real CLI on this Windows machine against private homes: `install --yes`, a second `install --yes`, `install --refresh` and `uninstall --yes --keep-cli`, for 23 config.toml shapes, each result parsed with Python's strict tomllib. - main (0abae119): 46 of 92 results were invalid TOML (15 of 23 right after install), the entry was still there after uninstall in 16 shapes, and 8 results had mixed line endings. - The fix: 0 invalid and no mixed line endings. The entry remains after uninstall only inside an inline `mcp_servers` table, which uninstall reports. Install refuses with exit 1 and an unchanged file for the two shapes where a table can't be added. Fixes #2562 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/installer-targets.test.ts | 233 +++++++++++++++- src/installer/index.ts | 6 +- src/installer/targets/codex.ts | 60 ++-- src/installer/targets/toml.ts | 406 +++++++++++++++++++++++----- 5 files changed, 619 insertions(+), 87 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 90fee5a18c..0861455134 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) +- `codegraph install` and `upgrade` no longer break Codex's `config.toml` when CodeGraph's `[mcp_servers.codegraph]` entry is written differently from how CodeGraph writes it, such as indented, with spaces or quotes inside the brackets, or with Windows line endings: the entry is now updated in place, where before a second copy was added and Codex could load none of the file. A file already broken that way is repaired on the next install or upgrade, an example of the entry inside a multiline setting such as `developer_instructions` is no longer mistaken for it, uninstall removes the entry however it is written, and an entry you wrote with dotted keys or as an inline table is left as you wrote it. When your `mcp_servers` settings are one inline table, CodeGraph now says it can't add itself instead of breaking the file. Thanks @dencoseca for the report, and @danusha2345 and @rudycelekli for the first fixes. (#2562) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) - In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) diff --git a/__tests__/installer-targets.test.ts b/__tests__/installer-targets.test.ts index c303d4eca0..d988d9ad50 100644 --- a/__tests__/installer-targets.test.ts +++ b/__tests__/installer-targets.test.ts @@ -22,7 +22,7 @@ import { spawnSync } from 'child_process'; import { parse as parseJsonc } from 'jsonc-parser'; import { ALL_TARGETS, getTarget, resolveTargetFlag } from '../src/installer/targets/registry'; import { uninstallTargets, refreshTargets } from '../src/installer'; -import { upsertTomlTable, removeTomlTable, buildTomlTable } from '../src/installer/targets/toml'; +import { upsertTomlTable, removeTomlTable, buildTomlTable, tomlDefinesTable } from '../src/installer/targets/toml'; import { cleanupLegacyHooks, writePromptHookEntry, removePromptHookEntry } from '../src/installer/targets/claude'; import { replaceOrAppendMarkedSection, removeMarkedSection } from '../src/installer/targets/shared'; import { @@ -1619,6 +1619,237 @@ describe('Installer targets — TOML serializer (Codex backbone)', () => { }); }); +// --------------------------------------------------------------------------- +// The Codex table is found however it is written (#2562) +// +// The header search matched only a column-0 `[mcp_servers.codegraph]`, so an +// indented one (valid TOML, and what formatters that indent sub-tables +// write) was taken for absent and a second table was appended: a table +// defined twice is invalid TOML, and Codex then loaded none of its config. +// --------------------------------------------------------------------------- +describe('Installer — the Codex table is found however it is written (#2562)', () => { + const HEADER = 'mcp_servers.codegraph'; + const BLOCK = buildTomlTable(HEADER, { command: 'codegraph', args: ['serve', '--mcp'] }); + const OLD_BODY = 'command = "codegraph"\nargs = ["serve"]'; + const BS = String.fromCharCode(92); + const crlf = (s: string) => s.replace(/\n/g, '\r\n'); + const top = 'model = "o3"'; + const other = '[mcp_servers.other]\ncommand = "other-server"'; + + it.each([ + ['indented with spaces (the report)', ' [mcp_servers.codegraph]', ' command = "codegraph"\n args = ["serve"]'], + ['indented with a tab', '\t[mcp_servers.codegraph]', OLD_BODY], + ['spaces inside the brackets', '[ mcp_servers.codegraph ]', OLD_BODY], + ['spaces around the dot', '[mcp_servers . codegraph]', OLD_BODY], + ['a quoted key', '[mcp_servers."codegraph"]', OLD_BODY], + ['a literal-quoted key', "['mcp_servers'.codegraph]", OLD_BODY], + ['an escape in a quoted key', `[mcp_servers."code${BS}u0067raph"]`, OLD_BODY], + ['a comment after it', '[mcp_servers.codegraph] # added by hand', OLD_BODY], + ])('a header written with %s is replaced in place, once, and removed', (_form, header, body) => { + const input = `${top}\n\n[mcp_servers]\n\n${header}\n${body}\n\n${other}\n`; + + const up = upsertTomlTable(input, HEADER, BLOCK); + expect(up.action).toBe('replaced'); + expect(up.content).toBe(`${top}\n\n[mcp_servers]\n\n${BLOCK}\n\n${other}\n`); + expect(upsertTomlTable(up.content, HEADER, BLOCK).action).toBe('unchanged'); + + const rm = removeTomlTable(input, HEADER); + expect(rm.action).toBe('removed'); + expect(rm.content).toBe(`${top}\n\n[mcp_servers]\n\n${other}\n`); + expect(tomlDefinesTable(input, HEADER)).toBe(true); + }); + + it('a CRLF file stays CRLF, and an up-to-date CRLF table is unchanged', () => { + const input = crlf(`${top}\n\n[mcp_servers.codegraph]\n${OLD_BODY}\n\n${other}\n`); + const up = upsertTomlTable(input, HEADER, BLOCK); + expect(up.content).toBe(crlf(`${top}\n\n${BLOCK}\n\n${other}\n`)); + expect(upsertTomlTable(up.content, HEADER, BLOCK).action).toBe('unchanged'); + expect(removeTomlTable(up.content, HEADER).content).toBe(crlf(`${top}\n\n${other}\n`)); + // Inserting into a CRLF file uses CRLF too. + expect(upsertTomlTable(crlf(`${top}\n`), HEADER, BLOCK).content).toBe(crlf(`${top}\n\n${BLOCK}\n`)); + }); + + it('a second table, as the old bug appended, is removed: the file parses again', () => { + const input = `${top}\n\n [mcp_servers.codegraph]\n command = "codegraph"\n args = ["serve"]\n\n${other}\n\n${BLOCK}\n`; + const up = upsertTomlTable(input, HEADER, BLOCK); + expect(up.action).toBe('replaced'); + expect(up.content).toBe(`${top}\n\n${BLOCK}\n\n${other}\n`); + expect(removeTomlTable(input, HEADER).content).toBe(`${top}\n\n${other}\n`); + }); + + it('look-alikes are never taken for the table: other names, and header-shaped text inside values', () => { + const lookalikes = [ + top, + '', + '[mcp_servers.codegraph2]', + 'command = "x"', + '', + '[mcp_servers.other]', + 'note = """', + '[mcp_servers.codegraph]', + '"""', + "literal = '''", + '[mcp_servers.codegraph]', + "'''", + 'args = [', + ' "[mcp_servers.codegraph]",', + ']', + '', + ].join('\n'); + const up = upsertTomlTable(lookalikes, HEADER, BLOCK); + expect(up.action).toBe('inserted'); + expect(up.content).toBe(`${lookalikes.trimEnd()}\n\n${BLOCK}\n`); + expect(removeTomlTable(lookalikes, HEADER)).toEqual({ content: lookalikes, action: 'not-found' }); + expect(tomlDefinesTable(lookalikes, HEADER)).toBe(false); + }); + + it('a sub-table such as [mcp_servers.codegraph.env] is kept by install and removed with the table', () => { + const env = '[mcp_servers.codegraph.env]\nCODEGRAPH_MCP_TOOLS = "all"'; + const input = `${top}\n\n[mcp_servers.codegraph]\n${OLD_BODY}\n\n${env}\n\n${other}\n`; + expect(upsertTomlTable(input, HEADER, BLOCK).content).toBe(`${top}\n\n${BLOCK}\n\n${env}\n\n${other}\n`); + expect(removeTomlTable(input, HEADER).content).toBe(`${top}\n\n${other}\n`); + // Only the sub-table left (a server with no command): install adds the table. + expect(tomlDefinesTable(`${top}\n\n${env}\n`, HEADER)).toBe(true); + expect(upsertTomlTable(`${top}\n\n${env}\n`, HEADER, BLOCK).content).toBe(`${top}\n\n${env}\n\n${BLOCK}\n`); + }); + + it('a comment heading the next table stays with it', () => { + const input = `${top}\n\n[mcp_servers.codegraph]\n${OLD_BODY}\n\n# the other server, set up by hand\n${other}\n`; + expect(upsertTomlTable(input, HEADER, BLOCK).content).toBe(`${top}\n\n${BLOCK}\n\n# the other server, set up by hand\n${other}\n`); + expect(removeTomlTable(input, HEADER).content).toBe(`${top}\n\n# the other server, set up by hand\n${other}\n`); + }); + + it.each([ + ['dotted keys at the top level', `mcp_servers.codegraph.command = "codegraph"\nmcp_servers.codegraph.args = ["serve", "--mcp"]\n${top}\n\n${other}\n`, 1, 'dotted keys', `${top}\n\n${other}\n`], + ['dotted keys under [mcp_servers]', `${top}\n\n[mcp_servers]\ncodegraph.command = "codegraph"\ncodegraph.args = [\n "serve",\n "--mcp",\n]\nother.command = "x"\n`, 4, 'dotted keys', `${top}\n\n[mcp_servers]\nother.command = "x"\n`], + ['an inline table under [mcp_servers]', `${top}\n\n[mcp_servers]\ncodegraph = { command = "codegraph", args = ["serve", "--mcp"] }\n`, 4, 'an inline table', `${top}\n\n[mcp_servers]\n`], + ['[[mcp_servers.codegraph]]', `${top}\n\n[[mcp_servers.codegraph]]\ncommand = "codegraph"\n`, 3, 'an array of tables', `${top}\n`], + ])('written with %s: install leaves it as is (no second table), uninstall removes it', (_form, input, line, form, removed) => { + expect(upsertTomlTable(input, HEADER, BLOCK)).toEqual({ + content: input, + action: 'conflict', + conflict: expect.objectContaining({ kind: 'defined', line, form }), + }); + const rm = removeTomlTable(input, HEADER); + expect(rm.action).toBe('removed'); + expect(rm.content).toBe(removed); + expect(tomlDefinesTable(input, HEADER)).toBe(true); + }); + + it.each([ + ['mcp_servers as an inline table', `${top}\nmcp_servers = { other = { command = "x" } }\n`, 'an inline table'], + ['mcp_servers as an array of tables', `${top}\n\n[[mcp_servers]]\nname = "other"\n`, 'an array of tables'], + ])('%s: a [mcp_servers.codegraph] table can\'t be added, so nothing is', (_form, input, form) => { + expect(upsertTomlTable(input, HEADER, BLOCK)).toEqual({ + content: input, + action: 'conflict', + conflict: expect.objectContaining({ kind: 'blocked', key: 'mcp_servers', form }), + }); + expect(removeTomlTable(input, HEADER).action).toBe('not-found'); + expect(tomlDefinesTable(input, HEADER)).toBe(false); + }); + + it('codegraph inside an inline mcp_servers table is reported, never edited', () => { + const input = `${top}\nmcp_servers = { codegraph = { command = "codegraph", args = ["serve", "--mcp"] } }\n`; + expect(upsertTomlTable(input, HEADER, BLOCK).action).toBe('conflict'); + expect(removeTomlTable(input, HEADER)).toEqual({ + content: input, + action: 'conflict', + conflict: expect.objectContaining({ key: 'mcp_servers', form: 'an inline table', line: 2 }), + }); + expect(tomlDefinesTable(input, HEADER)).toBe(true); + }); +}); + +describe('Installer — Codex install, re-install, refresh and uninstall find the table however it is written (#2562)', () => { + let tmpHome: string; + let tmpCwd: string; + let origCwd: string; + let homeRestore: { restore: () => void }; + + beforeEach(() => { + tmpHome = mkTmpDir('codex2562-home'); + tmpCwd = mkTmpDir('codex2562-cwd'); + origCwd = process.cwd(); + process.chdir(tmpCwd); + homeRestore = setHome(tmpHome); + }); + + afterEach(() => { + homeRestore.restore(); + process.chdir(origCwd); + fs.rmSync(tmpHome, { recursive: true, force: true }); + fs.rmSync(tmpCwd, { recursive: true, force: true }); + }); + + const configToml = () => path.join(tmpHome, '.codex', 'config.toml'); + function writeConfig(text: string): void { + fs.mkdirSync(path.dirname(configToml()), { recursive: true }); + fs.writeFileSync(configToml(), text); + } + const block = buildTomlTable('mcp_servers.codegraph', { command: 'codegraph', args: ['serve', '--mcp'] }); + + it('the reported indented table: detected, replaced once, unchanged after, refreshed in place, removed by uninstall', () => { + const siblings = 'model = "o3"\n\n[mcp_servers]\n\n [mcp_servers.other]\n command = "other-server"\n'; + writeConfig(`${siblings}\n [mcp_servers.codegraph]\n command = "codegraph"\n args = ["serve", "--mcp"]\n`); + const codex = getTarget('codex')!; + expect(codex.detect('global').alreadyConfigured).toBe(true); + + const first = codex.install('global', { autoAllow: false }); + expect(first.files.find((f) => f.path === configToml())!.action).toBe('updated'); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe(`${siblings}\n${block}\n`); + expect(codex.install('global', { autoAllow: false }).files.find((f) => f.path === configToml())!.action).toBe('unchanged'); + expect(refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === 'codex')!.status).toBe('unchanged'); + + codex.uninstall('global'); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe(siblings); + expect(codex.detect('global').alreadyConfigured).toBe(false); + }); + + it('a CRLF config keeps CRLF through install and uninstall', () => { + writeConfig('model = "o3"\r\n\r\n[mcp_servers.codegraph]\r\ncommand = "codegraph"\r\nargs = ["serve"]\r\n'); + const codex = getTarget('codex')!; + codex.install('global', { autoAllow: false }); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe(`model = "o3"\r\n\r\n${block.replace(/\n/g, '\r\n')}\r\n`); + codex.uninstall('global'); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe('model = "o3"\r\n'); + }); + + it('an entry written with dotted keys is left as written, with a note; uninstall removes it', () => { + const input = 'model = "o3"\n\n[mcp_servers]\ncodegraph.command = "codegraph"\ncodegraph.args = ["serve", "--mcp"]\n'; + writeConfig(input); + const codex = getTarget('codex')!; + expect(codex.detect('global').alreadyConfigured).toBe(true); + + const result = codex.install('global', { autoAllow: false }); + expect(result.files.find((f) => f.path === configToml())!.action).toBe('unchanged'); + expect(result.notes?.join('\n')).toMatch(/Left the mcp_servers\.codegraph entry in .*config\.toml as you wrote it \(dotted keys, line 4\)/); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe(input); + expect(fs.readFileSync(path.join(tmpHome, '.codex', 'AGENTS.md'), 'utf-8')).toContain('codegraph explore'); + expect(refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === 'codex')!.status).toBe('unchanged'); + + codex.uninstall('global'); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe('model = "o3"\n\n[mcp_servers]\n'); + }); + + it('mcp_servers as an inline table: install refuses with a clear error and changes nothing', () => { + const input = 'model = "o3"\nmcp_servers = { other = { command = "x" } }\n'; + writeConfig(input); + expect(() => getTarget('codex')!.install('global', { autoAllow: false })).toThrow( + /config\.toml sets mcp_servers as an inline table \(line 2\), so a \[mcp_servers\.codegraph\] table can't be added without breaking the file\. Nothing was changed/, + ); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe(input); + }); + + it('codegraph inside an inline mcp_servers table: uninstall says it can\'t remove it, and leaves the file alone', () => { + const input = 'model = "o3"\nmcp_servers = { codegraph = { command = "codegraph", args = ["serve", "--mcp"] } }\n'; + writeConfig(input); + const [report] = uninstallTargets([getTarget('codex')!], 'global'); + expect(report!.notes.join('\n')).toMatch(/sets mcp_servers\.codegraph inside mcp_servers \(an inline table, line 2\), which CodeGraph can't edit safely/); + expect(fs.readFileSync(configToml(), 'utf-8')).toBe(input); + }); +}); + describe('Installer — uninstallTargets sweep (codegraph uninstall)', () => { let tmpHome: string; let tmpCwd: string; diff --git a/src/installer/index.ts b/src/installer/index.ts index 579d595933..d995c84a60 100644 --- a/src/installer/index.ts +++ b/src/installer/index.ts @@ -517,10 +517,14 @@ export async function runUninstaller(opts: RunUninstallerOptions): Promise clack.log.success(`${r.displayName}: removed ${tildify(p)}`); } } else if (r.status === 'not-configured') { - clack.log.info(`${r.displayName}: not configured — nothing to remove`); + clack.log.info(r.notes.length > 0 ? `${r.displayName}: nothing removed` : `${r.displayName}: not configured — nothing to remove`); } else { clack.log.info(`${r.displayName}: skipped — ${r.notes[0] ?? 'unsupported location'}`); + continue; } + // What a target couldn't remove on its own (e.g. a Codex entry inside + // an inline table, #2562). + for (const note of r.notes) clack.log.warn(`${r.displayName}: ${note}`); } // Step 4: for local uninstall, the index dir is separate — point at diff --git a/src/installer/targets/codex.ts b/src/installer/targets/codex.ts index d6506282ce..2f0843f9ab 100644 --- a/src/installer/targets/codex.ts +++ b/src/installer/targets/codex.ts @@ -50,7 +50,7 @@ import { CODEGRAPH_SECTION_END, CODEGRAPH_SECTION_START, } from '../instructions-template'; -import { buildTomlTable, removeTomlTable, upsertTomlTable } from './toml'; +import { buildTomlTable, removeTomlTable, tomlDefinesTable, tomlLineBreak, upsertTomlTable } from './toml'; const TOML_HEADER = 'mcp_servers.codegraph'; @@ -103,7 +103,9 @@ class CodexTarget implements AgentTarget { if (fs.existsSync(tomlPath)) { try { const content = fs.readFileSync(tomlPath, 'utf-8'); - alreadyConfigured = content.includes(`[${TOML_HEADER}]`); + // However the table is written — an indented or quoted header, + // dotted keys, an inline table (#2562). + alreadyConfigured = tomlDefinesTable(content, TOML_HEADER); } catch { /* ignore */ } } // Global: ~/.codex/ existing means Codex has run here. Local: the @@ -115,33 +117,42 @@ class CodexTarget implements AgentTarget { install(loc: Location, _opts: InstallOptions): WriteResult { const files: WriteResult['files'] = []; + const notes: string[] = []; - files.push(writeMcpEntry(loc)); + const mcp = writeMcpEntry(loc); + files.push(mcp.file); + if (mcp.note) notes.push(mcp.note); // AGENTS.md gets the short marker-fenced CodeGraph block (#704): // subagents and non-MCP harnesses read AGENTS.md but never the MCP // initialize instructions. Upsert self-heals a stale pre-#529 block. files.push(upsertInstructionsEntry(instructionsPath(loc))); - return loc === 'local' ? { files, notes: [trustNote()] } : { files }; + if (loc === 'local') notes.push(trustNote()); + return notes.length > 0 ? { files, notes } : { files }; } uninstall(loc: Location): WriteResult { const files: WriteResult['files'] = []; + const notes: string[] = []; const tomlPath = tomlConfigPath(loc); if (fs.existsSync(tomlPath)) { const content = fs.readFileSync(tomlPath, 'utf-8'); - const { content: nextContent, action } = removeTomlTable(content, TOML_HEADER); - if (action === 'removed') { - if (nextContent.trim() === '') { + const result = removeTomlTable(content, TOML_HEADER); + if (result.action === 'removed') { + if (result.content.trim() === '') { removeFileUnlessLinked(tomlPath, ''); } else { - atomicWriteFileSync(tomlPath, nextContent.trimEnd() + '\n'); + atomicWriteFileSync(tomlPath, result.content.trimEnd() + tomlLineBreak(content)); } files.push({ path: tomlPath, action: 'removed' }); } else { files.push({ path: tomlPath, action: 'not-found' }); + if (result.action === 'conflict') { + const { conflict } = result; + notes.push(`${tomlPath} sets ${TOML_HEADER} inside ${conflict.key} (${conflict.form}, line ${conflict.line}), which CodeGraph can't edit safely. Remove it from there yourself.`); + } } } else { files.push({ path: tomlPath, action: 'not-found' }); @@ -149,7 +160,7 @@ class CodexTarget implements AgentTarget { files.push(removeInstructionsEntry(loc)); - return { files }; + return notes.length > 0 ? { files, notes } : { files }; } printConfig(loc: Location): string { @@ -170,7 +181,7 @@ function buildCodegraphBlock(): string { }); } -function writeMcpEntry(loc: Location): WriteResult['files'][number] { +function writeMcpEntry(loc: Location): { file: WriteResult['files'][number]; note?: string } { const file = tomlConfigPath(loc); const dir = path.dirname(file); if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true }); @@ -181,13 +192,30 @@ function writeMcpEntry(loc: Location): WriteResult['files'][number] { // between two `fs.existsSync` calls. const existing = fs.existsSync(file) ? fs.readFileSync(file, 'utf-8') : ''; const created = existing.length === 0; - const { content: nextContent, action } = upsertTomlTable(existing, TOML_HEADER, block); - - if (action === 'unchanged') { - return { path: file, action: 'unchanged' }; + const result = upsertTomlTable(existing, TOML_HEADER, block); + + if (result.action === 'conflict') { + // Never append a second definition: a table defined twice makes the + // whole file invalid TOML, and Codex then loads none of it (#2562). + const { conflict } = result; + if (conflict.kind === 'defined') { + // The user's own entry, in a form CodeGraph doesn't rewrite: it + // already configures Codex, so it is left exactly as written. + return { + file: { path: file, action: 'unchanged' }, + note: `Left the ${TOML_HEADER} entry in ${file} as you wrote it (${conflict.form}, line ${conflict.line}): CodeGraph only rewrites a [${TOML_HEADER}] table.`, + }; + } + throw new Error( + `${file} sets ${conflict.key} as ${conflict.form} (line ${conflict.line}), so a [${TOML_HEADER}] table can't be added without breaking the file. ` + + `Nothing was changed: add CodeGraph's server to ${conflict.key} yourself, or rewrite it as [${conflict.key}.] tables, then run this again.`, + ); + } + if (result.action === 'unchanged') { + return { file: { path: file, action: 'unchanged' } }; } - atomicWriteFileSync(file, nextContent); - return { path: file, action: created ? 'created' : 'updated' }; + atomicWriteFileSync(file, result.content); + return { file: { path: file, action: created ? 'created' : 'updated' } }; } /** diff --git a/src/installer/targets/toml.ts b/src/installer/targets/toml.ts index 1dc086bf3d..763fca0df2 100644 --- a/src/installer/targets/toml.ts +++ b/src/installer/targets/toml.ts @@ -5,11 +5,13 @@ * general TOML parser/serializer; that would mean pulling in a * dependency (~50KB) for ~6 lines of output. * - * Strategy: treat the file as text. Find the `[mcp_servers.codegraph]` - * header line, splice it (and the lines that follow it until the next - * `[...]` / `[[...]]` header or EOF) in or out. A small lexical scan keeps - * header-shaped text inside multiline values out of the boundary search. - * Everything outside that block is preserved verbatim, byte-for-byte. + * Strategy: treat the file as text. A small structural scan finds each + * statement — table header, array-of-tables header, key/value — with its + * key path decoded the way TOML reads it, and skips the lines of a value + * that spans several (a multiline string, array or inline table). The + * `[mcp_servers.codegraph]` block (its header line and the lines that follow + * until the next header or EOF) is spliced in or out. Everything outside + * that block is preserved verbatim, byte-for-byte. * * Limitations (acceptable for our narrow use): * - Only writes a top-level table header. Array-of-tables and sibling @@ -18,6 +20,8 @@ * - Doesn't validate sibling TOML — if the file is malformed * elsewhere, our injection won't fix it but won't make it worse. * - Quotes string values with double quotes; escapes `\` and `"`. + * - Never rewrites a table written in another form (dotted keys, an + * inline table): those are reported as a conflict instead (#2562). */ /** @@ -54,116 +58,380 @@ export function buildTomlTable(header: string, values: Record 0 ? '\n\n' : ''; - return { - content: trimmed + sep + block + '\n', - action: 'inserted', - }; + const sep = trimmed.length > 0 ? eol + eol : ''; + return { content: trimmed + sep + ours + eol, action: 'inserted' }; } - // Find the end of this block: next table header or EOF. - const blockEnd = findNextTableHeader(fileContent, headerIdx + headerLine.length); - const existingBlock = fileContent.substring(headerIdx, blockEnd).replace(/\n+$/, ''); - - if (existingBlock === block) { + const existingBlock = fileContent.substring(first.start, first.end).replace(/(?:\r?\n)+$/, ''); + if (duplicates.length === 0 && existingBlock === ours) { return { content: fileContent, action: 'unchanged' }; } - const before = fileContent.substring(0, headerIdx); - const after = fileContent.substring(blockEnd); - // Trim trailing blank lines from `before` (we'll re-add one) and - // leading blank lines from `after` so the file shape stays clean. - const beforeClean = before.replace(/\n+$/, ''); - const afterClean = after.replace(/^\n+/, ''); - const sepBefore = beforeClean.length > 0 ? '\n\n' : ''; - const sepAfter = afterClean.length > 0 ? '\n\n' : '\n'; - return { - content: beforeClean + sepBefore + block + sepAfter + afterClean, - action: 'replaced', - }; + let content = fileContent; + for (const dup of [...duplicates].reverse()) content = spliceBlock(content, dup, null, eol); + return { content: spliceBlock(keepFinalLineBreak(fileContent, content, eol), first, ours, eol), action: 'replaced' }; +} + +/** `content` ending with a line break when `original` did (removing the last block drops it). */ +function keepFinalLineBreak(original: string, content: string, eol: string): string { + return content.length > 0 && original.endsWith('\n') && !content.endsWith('\n') ? content + eol : content; } /** * Remove a top-level dotted-key TOML table block. Returns the * possibly-empty new content + an action flag. + * + * Removes every definition of the table: each `[header]` block however + * its line is written, its sub-tables (`[header.env]` alone would leave a + * server entry with no command), and keys that define it with dotted keys + * or an inline table. `'conflict'` (content untouched) when it is defined + * inside a parent's inline table, which can't be edited safely. */ export function removeTomlTable( fileContent: string, header: string, -): { content: string; action: 'removed' | 'not-found' } { - const headerLine = `[${header}]`; - const headerIdx = findHeaderIndex(fileContent, headerLine); - if (headerIdx === -1) return { content: fileContent, action: 'not-found' }; +): + | { content: string; action: 'removed' | 'not-found' } + | { content: string; action: 'conflict'; conflict: TomlConflict } { + const layout = layoutOf(fileContent, header.split('.')); + if (layout.inParentValue) return { content: fileContent, action: 'conflict', conflict: layout.inParentValue }; + + const eol = tomlLineBreak(fileContent); + const blocks = [...layout.tables, ...layout.subTables]; + const removals = [ + ...blocks.map((b) => ({ ...b, isBlock: true })), + ...layout.keys.map((k) => ({ start: k.start, end: k.end, isBlock: false })), + ].sort((a, b) => b.start - a.start); + if (removals.length === 0) return { content: fileContent, action: 'not-found' }; + + let content = fileContent; + for (const r of removals) { + content = r.isBlock ? spliceBlock(content, r, null, eol) : content.slice(0, r.start) + content.slice(r.end); + } + return { content: keepFinalLineBreak(fileContent, content, eol), action: 'removed' }; +} - const blockEnd = findNextTableHeader(fileContent, headerIdx + headerLine.length); - const before = fileContent.substring(0, headerIdx).replace(/\n+$/, ''); - const after = fileContent.substring(blockEnd).replace(/^\n+/, ''); - const joined = before + (before && after ? '\n\n' : '') + after; - return { content: joined, action: 'removed' }; +/** + * Whether the file defines the table in any form — a `[header]` block + * however it is written, a sub-table of it, dotted keys, or an inline + * table. What a Codex install counts as "already configured". + */ +export function tomlDefinesTable(fileContent: string, header: string): boolean { + const layout = layoutOf(fileContent, header.split('.')); + return layout.tables.length > 0 || layout.subTables.length > 0 || layout.defined !== null; +} + +/** The line break `content` uses (its first one); `\n` when it has none. */ +export function tomlLineBreak(content: string): '\n' | '\r\n' { + const nl = content.indexOf('\n'); + return nl > 0 && content[nl - 1] === '\r' ? '\r\n' : '\n'; } /** - * Locate the byte index of a header line (`[foo.bar]`) when it - * appears at the start of a line. Returns -1 if not found. + * Replace (or, with `replacement` null, remove) the block at `range`, + * rejoining what's around it with one blank line, in the file's line + * breaks — the shape the installer has always left. */ -function findHeaderIndex(content: string, headerLine: string): number { - // Search BOL or right after a newline. - if (content.startsWith(headerLine)) return 0; - const needle = '\n' + headerLine; - const idx = content.indexOf(needle); - return idx === -1 ? -1 : idx + 1; +function spliceBlock(content: string, range: { start: number; end: number }, replacement: string | null, eol: string): string { + const before = content.substring(0, range.start).replace(/(?:\r?\n)+$/, ''); + const after = content.substring(range.end).replace(/^(?:\r?\n)+/, ''); + if (replacement === null) return before + (before && after ? eol + eol : '') + after; + return before + (before ? eol + eol : '') + replacement + (after ? eol + eol : eol) + after; +} + +// --------------------------------------------------------------------------- +// Structure scan +// --------------------------------------------------------------------------- + +interface TomlStatement { + kind: 'table' | 'array-table' | 'key'; + /** The decoded key path; for a key, including the table it sits in. Null when unreadable. */ + path: string[] | null; + /** The table a key sits in (for a header, its own path). */ + table: string[] | null; + /** Offset of the statement's first line. */ + start: number; + /** Offset just past the statement's last line (its line break included). */ + end: number; + /** 1-based line of the statement's first line. */ + line: number; + /** For a key: whether its value is an inline table (`{`). */ + inlineTable?: boolean; +} + +interface TableLayout { + /** `[path]` blocks: from the header line to the next header or EOF. */ + tables: Array<{ start: number; end: number }>; + /** Blocks of headers below `path` (`[path.env]`, `[[path.x]]`). */ + subTables: Array<{ start: number; end: number }>; + /** Key statements that define `path` outside its own blocks. */ + keys: TomlStatement[]; + /** The first definition of `path` in a form other than a `[path]` block. */ + defined: TomlConflict | null; + /** The first statement that makes a `[path]` header invalid. */ + blocked: TomlConflict | null; + /** `path` set inside a parent's value (e.g. `mcp_servers = { codegraph = … }`). */ + inParentValue: TomlConflict | null; +} + +function layoutOf(content: string, path: string[]): TableLayout { + const statements = scanTomlStatements(content); + // A header's block runs to the end of its last key, so blank and comment + // lines before the next header (a comment heading the next table) stay + // with what follows. + const blockOf = (i: number) => { + let end = statements[i]!.end; + for (let j = i + 1; j < statements.length && statements[j]!.kind === 'key'; j++) end = statements[j]!.end; + return { start: statements[i]!.start, end }; + }; + const layout: TableLayout = { tables: [], subTables: [], keys: [], defined: null, blocked: null, inParentValue: null }; + const keyText = path.join('.'); + + statements.forEach((s, i) => { + if (!s.path) return; + if (s.kind !== 'key') { + if (s.kind === 'table' && samePath(s.path, path)) { + layout.tables.push(blockOf(i)); + } else if (startsWithPath(s.path, path)) { + // `[path.env]`, `[[path.x]]` — and `[[path]]`, which also defines the + // table as an array of tables we don't rewrite. + layout.subTables.push(blockOf(i)); + if (samePath(s.path, path)) { + const conflict: TomlConflict = { kind: 'defined', line: s.line, form: 'an array of tables', key: keyText }; + layout.defined ??= conflict; + layout.blocked ??= { ...conflict, kind: 'blocked' }; + } + } else if (s.kind === 'array-table' && startsWithPath(path, s.path)) { + layout.blocked ??= { kind: 'blocked', line: s.line, form: 'an array of tables', key: s.path.join('.') }; + } + return; + } + // A key inside the table's own blocks is its body, not a definition. + if (s.table && startsWithPath(s.table, path)) return; + if (startsWithPath(s.path, path)) { + layout.keys.push(s); + const form = s.path.length > path.length ? 'dotted keys' : s.inlineTable ? 'an inline table' : 'a value'; + layout.defined ??= { kind: 'defined', line: s.line, form, key: s.path.join('.') }; + } else if (startsWithPath(path, s.path)) { + // A parent set as a value: `mcp_servers = { … }`. No `[path]` header + // can follow it, and anything of ours inside it can't be edited. + const conflict: TomlConflict = { + kind: 'blocked', line: s.line, form: s.inlineTable ? 'an inline table' : 'a value', key: s.path.join('.'), + }; + layout.blocked ??= conflict; + const text = content.substring(s.start, s.end); + const last = path[path.length - 1]!; + if (new RegExp(String.raw`(^|[{,.\s])["']?${escapeRegExp(last)}["']?\s*[=.]`).test(text.slice(text.indexOf('=') + 1))) { + layout.inParentValue ??= { ...conflict, kind: 'defined' }; + layout.defined ??= { ...conflict, kind: 'defined' }; + } + } + }); + return layout; +} + +function samePath(a: readonly string[], b: readonly string[]): boolean { + return a.length === b.length && a.every((x, i) => x === b[i]); +} + +function startsWithPath(a: readonly string[], prefix: readonly string[]): boolean { + return prefix.length <= a.length && prefix.every((x, i) => x === a[i]); +} + +function escapeRegExp(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); } /** - * Find the byte index of the next `[...]` or `[[...]]` table header - * starting from `from`, or return content length when none. + * Split `content` into statements. A line counts only when it starts + * outside a multiline value; the lines of a value that spans several are + * folded into the key statement that opened it. */ -function findNextTableHeader(content: string, from: number): number { +function scanTomlStatements(content: string): TomlStatement[] { + const statements: TomlStatement[] = []; const state: TomlLexState = { multilineString: null, arrayDepth: 0, inlineTableDepth: 0 }; - let lineStart = from; - let isHeaderRemainder = true; + let table: string[] | null = []; + let open: TomlStatement | null = null; + let lineStart = 0; + let line = 1; while (lineStart < content.length) { const newlineIdx = content.indexOf('\n', lineStart); const lineEnd = newlineIdx === -1 ? content.length : newlineIdx; - const line = content.slice(lineStart, lineEnd); - - if ( - !isHeaderRemainder && - state.multilineString === null && - state.arrayDepth === 0 && - state.inlineTableDepth === 0 && - isTomlTableHeader(line) - ) { - return lineStart; + const next = newlineIdx === -1 ? content.length : newlineIdx + 1; + const text = content.slice(lineStart, lineEnd); + const clean = state.multilineString === null && state.arrayDepth === 0 && state.inlineTableDepth === 0; + + if (!clean) { + if (open) open.end = next; + } else { + open = null; + if (isTomlTableHeader(text)) { + const parsed = parseTableHeader(text); + table = parsed.path; + statements.push({ kind: parsed.array ? 'array-table' : 'table', path: parsed.path, table, start: lineStart, end: next, line }); + } else { + const key = parseKeyValueStart(text); + if (key) { + open = { + kind: 'key', + path: key.path && table ? [...table, ...key.path] : null, + table, + start: lineStart, + end: next, + line, + inlineTable: key.inlineTable, + }; + statements.push(open); + } + } } - scanTomlLine(line, state); - if (newlineIdx === -1) break; - lineStart = newlineIdx + 1; - isHeaderRemainder = false; + scanTomlLine(text, state); + lineStart = next; + line++; + } + return statements; +} + +/** `[a.b]` / `[[a.b]]` (already known to be a header line) → its decoded path. */ +function parseTableHeader(text: string): { array: boolean; path: string[] | null } { + let i = skipWs(text, 0) + 1; + const array = text[i] === '['; + if (array) i++; + const key = parseDottedKey(text, skipWs(text, i)); + return { array, path: key ? key.path : null }; +} + +/** `key = …` at the start of a line → the key's decoded path. */ +function parseKeyValueStart(text: string): { path: string[] | null; inlineTable: boolean } | null { + const start = skipWs(text, 0); + if (start >= text.length || text[start] === '#' || text[start] === '\r') return null; + const key = parseDottedKey(text, start); + if (!key) return null; + const eq = skipWs(text, key.end); + if (text[eq] !== '=') return null; + return { path: key.path, inlineTable: text[skipWs(text, eq + 1)] === '{' }; +} + +function skipWs(text: string, i: number): number { + while (i < text.length && (text[i] === ' ' || text[i] === '\t')) i++; + return i; +} + +/** + * A dotted key at `i`: bare parts, `"basic"` parts (escapes decoded) and + * `'literal'` parts, joined by dots with optional spaces around them. + * Returns the decoded parts and the offset just past the key, or null. + */ +function parseDottedKey(text: string, i: number): { path: string[]; end: number } | null { + const path: string[] = []; + for (;;) { + const part = parseKeyPart(text, i); + if (!part) return null; + path.push(part.value); + const dot = skipWs(text, part.end); + if (text[dot] !== '.') return { path, end: part.end }; + i = skipWs(text, dot + 1); } +} - return content.length; +function parseKeyPart(text: string, i: number): { value: string; end: number } | null { + const ch = text[i]; + if (ch === '"') { + let j = i + 1; + while (j < text.length && text[j] !== '"') j += text[j] === '\\' ? 2 : 1; + if (j >= text.length) return null; + const value = decodeBasicString(text.slice(i + 1, j)); + return value === null ? null : { value, end: j + 1 }; + } + if (ch === "'") { + const j = text.indexOf("'", i + 1); + return j === -1 ? null : { value: text.slice(i + 1, j), end: j + 1 }; + } + let j = i; + while (j < text.length && /[A-Za-z0-9_-]/.test(text[j]!)) j++; + return j === i ? null : { value: text.slice(i, j), end: j }; +} + +const SIMPLE_ESCAPES: Record = { + b: String.fromCharCode(8), t: String.fromCharCode(9), n: String.fromCharCode(10), f: String.fromCharCode(12), + r: String.fromCharCode(13), e: String.fromCharCode(27), '"': '"', '\\': '\\', +}; + +/** The value of a TOML basic string's body, or null for an invalid escape. */ +function decodeBasicString(raw: string): string | null { + let out = ''; + for (let i = 0; i < raw.length; i++) { + if (raw[i] !== '\\') { + out += raw[i]; + continue; + } + const esc = raw[++i] ?? ''; + if (esc in SIMPLE_ESCAPES) { + out += SIMPLE_ESCAPES[esc]; + continue; + } + const width = esc === 'x' ? 2 : esc === 'u' ? 4 : esc === 'U' ? 8 : 0; + const hex = raw.slice(i + 1, i + 1 + width); + if (width === 0 || hex.length !== width || !/^[0-9A-Fa-f]+$/.test(hex)) return null; + const codePoint = parseInt(hex, 16); + if (codePoint > 0x10ffff || (codePoint >= 0xd800 && codePoint <= 0xdfff)) return null; + out += String.fromCodePoint(codePoint); + i += width; + } + return out; } type MultilineStringDelimiter = '"""' | "'''"; From 5a6917b437505380ab05e0ba86a665a7fdded488 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 02:16:31 +0000 Subject: [PATCH 13/95] fix(c): a function whose return type is a macro call is indexed (#2773) In C, a function declared with its return type written as a macro call, like cJSON's `CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item)` or libgit2's `GIT_EXTERN(int) git_blob_dup(...);`, had no node. blankCppAnnotationMacroCalls blanks a line-leading ALL-CAPS macro call that a declaration follows, the markup shape of Unreal's `UPROPERTY(...)`. Since the C/C++ kernel port, C's pre-parse runs it as well, to set aside redis's `REDIS_NO_SANITIZE("bounds")` and similar. In C the same shape is often the return type. Blanked together with its argument, it left `cJSON_Print(...)` and a block, which parse as a call statement. The function was lost, the calls in its body went to the file, its locals and parameters became file-level variables, and a header's prototype read as a call from the header. C now passes keepCReturnTypes. With it, a macro is kept, and tree-sitter reads it as a macro_type_specifier, when all of these hold: - it is in column 0; - its one argument reads as a type; - only a function declarator follows: a name, its parameter list, `__attribute__` or ALL-CAPS attributes, then `{` or `;`; - the line right above leaves no declaration open. jemalloc writes `size_t JEMALLOC_NOTHROW` there, above `JEMALLOC_ATTR(pure)`. C++ is unchanged. Validation: - New __tests__/c-macro-return-types.test.ts (3 tests: the issue's reproduction plus GNU-style, prototype-with-attribute, ATTR_FORMAT, REDIS_NO_SANITIZE and jemalloc shapes; a libgit2-style header; the blank itself). All 3 fail with main's c-cpp.ts and pass on the kernel and with CODEGRAPH_KERNEL=0. - Pre-parse output compared file by file on 12 repos. The only changes are macro text kept where main blanked it; no file's parse-error count got worse. Better: cJSON 4 files (cJSON.c 131 -> 6 errors), expat 1, APR 69, libgit2 82. Byte-identical pre-parse, and so graphs, on jq, redis (with jemalloc), sqlite and curl, and on the C++ controls fmt, glm, json and abseil-cpp. - A/B on main 6bd217a2, graph dumps: - cJSON: +92 functions; 7 functions named `(void)` and 100 phantom variables and constants gone; edges 2,882 -> 3,750. - libgit2: +403 functions; 3 `(void)` functions and 293 phantoms gone; 37 calls that went to a phantom parameter gone; edges 81,984 -> 85,834. - APR: +1,551 functions; 7 bogus ones and 1,723 phantoms gone; edges 16,163 -> 23,278. 39 calls now reach the real APR_DECLARE definition instead of its EBCDIC stub. 283 portable calls now see the OS/2 and Windows copies beside the Unix one: 278 go to OS/2 at 0.4 (was Unix at 0.9), and 5 from win32 code go to win32. - expat: 22 phantom parameters gone. - kernel-parity: 0 diffs on both arms; deferred files fall from 33 to 31 (cJSON), 218 to 161 (APR) and 208 to 128 (libgit2). - Cost: medians of 3 interleaved init runs: cJSON -1.6%, expat -4.3%, libgit2 -6.9%, APR +18.4%. APR's graph is 44% larger. The pre-parse itself is unchanged (APR 262 vs 257 ms, libgit2 598 vs 599 ms). Fixes #2470 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/c-macro-return-types.test.ts | 174 +++++++++++++++++++++++++ src/extraction/languages/c-cpp.ts | 80 +++++++++++- 3 files changed, 248 insertions(+), 7 deletions(-) create mode 100644 __tests__/c-macro-return-types.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0861455134..2cdadfd688 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -137,6 +137,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, an explicit specialization of a function or member function template, like protocolbuffers/protobuf's `template <> bool WireFormatLite::ReadPrimitive(…)` or facebook/rocksdb's `template <> int64_t Deserialize(…)`, is now indexed under its template's name, so search finds it and calls link to it. Before, it kept its template arguments in its name, and one with a `::` among them was named after the last part, like `TYPE_INT32>`. A call that writes template arguments, like `Deserialize(src, offset)`, links to the specialization written with the same ones, and otherwise to the template itself, as fmt's `num_bits()` does; a call that writes none links to a specialization only when nothing else has that name. Re-index C++ projects after upgrading. - In C++, a call to an overloaded function outside a class now links to the overload its arguments fit, by how many it passes and the types they have where the call writes them, as a call to a method already did: protocolbuffers/protobuf's `ParseWithLimit(limit, data, result)` on an `int32_t result` reaches the overload taking an `int32_t&`, and facebook/rocksdb's `NewLRUCache(cache_opts)` the one taking an `LRUCacheOptions`. Before, the overload nearest the call was taken. Such a call is never linked to an overload only another source file can see, like another `.cc` file's `static` function: it now links to the one its own file sees, where before it often linked to nothing. A call to a name that both a function and a class have reaches the function, which hides the class, and a parameter named like the function it belongs to, like protobuf's `std::string u64(uint64_t u64)`, now has its own type. Of two overloads alike but for a `const`, like protobuf's `DynamicCastMessage(const MessageLite* from)` and `DynamicCastMessage(MessageLite* from)`, a call now reaches the one C++ picks: the non-const overload for a pointer to a non-const object, or for the object itself, and the const one for a const object or a temporary, where before the overload declared first won. An argument declared with `const` in front, like `const Buffer* b`, counts as const, and an array as a pointer to its element, so facebook/rocksdb's `Add(level, file, smallest, largest, …)` on `char` arrays reaches the overload taking `const char*`. An object is no longer matched to a pointer parameter, nor a character array to a pointer of another type, so protobuf's `printer->WithVars(EnumVars(…))` reaches `WithVars(Map&& vars)`, not `WithVars(const Map* vars)`. Re-index C++ projects after upgrading. - In C, a call now links to the function of that name its own file can see: its own, else one a header it includes defines, like a `static inline` helper, else one another source file defines without `static`. Before, the function nearest the call was taken, and when that was another source file's `static` function, which no other file can call, the call linked to nothing: in redis, the calls to `server.c`'s `mstime()` and `ustime()` from the rest of the server were lost to the `static` copies other files keep for themselves, and jemalloc's tests' calls to the `do_arena_create` their shared header defines to a neighbouring test's own. A call to a `static inline` function that several headers define reaches the copy in a header the file includes, so protocolbuffers/protobuf's upb code no longer links to the copies in its PHP extension's combined `php-upb.h`. A source file that `#include`s another, like redis's `ae.c` with its event-loop backend or hiredis's `async.c` with `dict.c`, now calls that file's `static` functions, and a header a file includes through an include directory, like jemalloc's `#include "jemalloc/internal/bit_util.h"`, counts as included. A function a header defines only for a build that turns on a flag, like lua-cjson's `fpconv_init` under `#ifdef USE_INTERNAL_FPCONV`, gives way to another source file's definition when nothing in the project's code or build files sets that flag, so redis's calls to `fpconv_init()` reach the `fpconv.c` its build compiles. Re-index C projects after upgrading. +- In C, a function whose return type is written as a macro call, like cJSON's `CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item)`, libgit2's `GIT_EXTERN(int) git_blob_dup(…);` or APR's `APR_DECLARE(apr_status_t) apr_pool_create_ex(…)`, is now indexed, and calls to it link to it. Before, the macro was taken for markup and dropped together with the type it wraps, so the function had no entry, the calls in its body counted as the file's, its locals and parameters showed up as file-level variables, and a header declaring it looked like it called it. Markup written above a declaration, like `ATTR_FORMAT(1, 2)` or redis's `REDIS_NO_SANITIZE("bounds")`, is still set aside as before. Re-index C projects after upgrading. Thanks @xhoms for the report and @danusha2345. (#2470) - In C++, text inside a string literal is no longer taken for a variable's declaration. That covers the C code protocolbuffers/protobuf's upb generator writes in a raw string (`R"cc( … upb_MiniTableField field = $field_init$; … )cc"`) and log messages like facebook/rocksdb's `"… in WriteBatch handler; …"`. Before, a later call like `field.ctype()` on the function's own `upb::FieldDefPtr field` parameter, or `handler->PutCF(…)` on a `WriteBatch::Handler* handler`, was matched against the type that text seemed to name, so it linked to nothing or to another class's method. Re-index C++ projects after upgrading. - In C++, a call on an `auto` variable now links to the method of the class C++ finds for the type its initializer gives it, as a call on a variable declared with that type does. The type is read from a cast or smart pointer written out, like facebook/rocksdb's `auto iter = std::unique_ptr(…)` or `auto* mutex = static_cast(arg)`, from `new` or `std::make_unique` of a class, and from what a function or method is declared to return, like the `Iterator*` of `txn->GetIterator(…)`, read to the end of an initializer that goes on to the next line. Before, such a variable took the type of an earlier variable of the same name, or none: in RocksDB, `iter->SeekToFirst()` reached `MemTableRep::Iterator` and `mutex->Lock()` reached `port::Mutex`. A chained call like `cfd->current()->storage_info()` now looks up the class the inner call is declared to return the same way, through a smart pointer too, so protocolbuffers/protobuf's Java generator no longer reaches the C# generator's `EnumGenerator`. A type from outside the project no longer links to a class of the project that shares its name: protobuf's `status.ok()` on an `absl::Status` reached `upb::Status::ok`, and fmtlib/fmt's `os.flush()` on a `std::ostream&` reached fmt's own `ostream::flush`. The same holds for a variable declared in a macro body, or typed by what a function in another file returns, and for a class the index doesn't hold whose name only another class shares: protobuf's `_status.ok()` in its `RETURN_IF_ERROR` macro reached `upb::Status::ok`, and `range.options()` on its generated `DescriptorProto::ExtensionRange` reached `Descriptor::ExtensionRange::options`. A namespace a macro stands for is the project's under either name: facebook/rocksdb declares its classes in `namespace ROCKSDB_NAMESPACE` with `#define ROCKSDB_NAMESPACE rocksdb`, so `db->Put(…)` on a `rocksdb::DB*` links to RocksDB's own `DB::Put`. Re-index C++ projects after upgrading. - In C++, a call on a data member of the calling method's class, like `versions_->GetColumnFamilySet()` in facebook/rocksdb's `db_impl_write.cc`, now links to the method of the member's type wherever the class is declared. Before, only the call's own file and the header of the same name were read for the member, and RocksDB declares `DBImpl` in `db_impl.h` while its methods are spread over `db_impl_write.cc`, `db_impl_compaction_flush.cc`, `db_impl_open.cc` and more, so such calls were guessed by the member's name. A member declared below an inline method that uses it, one a base class declares, and, in a gtest `TEST_F` or `TEST_P` body, one of the test's fixture, like RocksDB's `db_`, are found the same way, and a member of a class nested in the caller's class is no longer taken for the caller's own. A call on a member of a standard-library type, like `std::atomic shutting_down_`, no longer links to a project method that only shares the method's name. A call on a member reached through other members now links the same way, like RocksDB's `thread->rand.Next()` on a `Random64` or `meta->largest.user_key()` on an `InternalKey`. Before, it went to whichever class had a method of that name, like `ArenaWrappedDBIter::Next` or `ClippingIterator::user_key`, or, inside a method, to that method's own class. When the member's class has no such method, the call no longer links to another class's. The member is read from the class the calling file sees: RocksDB's `db_bench_tool.cc` and its stress tool each declare their own `ThreadState`, whose `rand` is a `Random64` in one and a `Random` in the other. It is also found in the classes that class derives from. RocksDB's `Options` inherits its members from `DBOptions` and `ColumnFamilyOptions`, so `options.statistics->getTickerCount(…)` now reaches `Statistics`, and `options.table_factory.reset(…)` on a `std::shared_ptr` no longer links to a project method of that name. Re-index C++ projects after upgrading. diff --git a/__tests__/c-macro-return-types.test.ts b/__tests__/c-macro-return-types.test.ts new file mode 100644 index 0000000000..a85b8686a9 --- /dev/null +++ b/__tests__/c-macro-return-types.test.ts @@ -0,0 +1,174 @@ +/** + * A C function whose return type is written as a macro call: cJSON's + * `CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item)`, libgit2's + * `GIT_EXTERN(int) git_blob_dup(…);`. The pre-parse blank for line-leading + * markup macros (Unreal's `UPROPERTY(…)`, redis's `REDIS_NO_SANITIZE("bounds")`) + * took the macro for markup and blanked it with the return type it wraps. What + * was left, `cJSON_Print(…)` and a block, parses as a call: the function had + * no node, its callers resolved to nothing, and a header's prototype read as + * a call from the header (#2470). + * + * Markup above a declaration is still blanked: one with a type of its own, + * and jemalloc's attribute between a type on the line above and the name. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import type { Node } from '../src/types'; +import { blankCppAnnotationMacroCalls } from '../src/extraction/languages/c-cpp'; + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** The 1-based line of `text` that contains `needle`, or is exactly it with `exact` (exactly one must). */ +const lineOf = (text: string, needle: string, exact = false): number => { + const at = text.split('\n').map((l, i) => ((exact ? l === needle : l.includes(needle)) ? i + 1 : 0)).filter(Boolean); + if (at.length !== 1) throw new Error(`${JSON.stringify(needle)} is on ${at.length} lines`); + return at[0]!; +}; + +// The issue's reproduction, then shapes around it. +const LIB_C = lines( + '#define EXPORT(type) type', + '#define ATTR_FORMAT(a, b)', + '#define ATTR(x)', + '#define REDIS_NO_SANITIZE(s)', + '', + 'EXPORT(char *) make_name(int id)', + '{', + ' return id > 0 ? "named" : "anonymous";', + '}', + '', + 'EXPORT(int) version(void)', + '{', + ' return 3;', + '}', + '', + '/* GNU style: the name on its own line. */', + 'EXPORT(const char *)', + 'label_for(int id)', + '{', + ' return id ? "x" : "y";', + '}', + '', + 'EXPORT(int) checked_sum(int a, int b) __attribute__((warn_unused_result));', + 'EXPORT(int) checked_sum(int a, int b)', + '{', + ' return a + b;', + '}', + '', + 'ATTR_FORMAT(1, 2)', + 'void log_line(const char *fmt, ...)', + '{', + '}', + '', + 'REDIS_NO_SANITIZE("bounds")', + 'int key_slot(const char *key)', + '{', + ' return key[0];', + '}', + '', + 'unsigned long', + 'ATTR(pure)', + 'usable_size(const void *ptr)', + '{', + ' return ptr != 0;', + '}', + '', + 'int caller(void)', + '{', + ' log_line("%d", version());', + ' return make_name(1) != 0 && usable_size(0) && label_for(2) && checked_sum(1, 2) && key_slot("k");', + '}', +); + +// libgit2's header shape: prototypes after a lone macro line and after a doc comment. +const API_H = lines( + '#ifndef API_H', + '#define API_H', + '#define GIT_EXTERN(type) extern type', + '#define GIT_BEGIN_DECL', + 'GIT_BEGIN_DECL', + '', + 'GIT_EXTERN(int) git_blob_dup(int **out, int *source);', + '', + '/** Looks the name up. */', + 'GIT_EXTERN(const char *) git_name(int id);', + '#endif', +); + +const API_C = lines( + '#include "api.h"', + 'int git_blob_dup(int **out, int *source) { *out = source; return 0; }', + 'const char *git_name(int id) { return id ? "a" : "b"; }', + 'int use_api(void) { int *p = 0; return git_blob_dup(&p, 0) + (git_name(1) != 0); }', +); + +describe('a C function whose return type is a macro call', () => { + const roots: string[] = []; + const graphs: CodeGraph[] = []; + + afterEach(() => { + for (const cg of graphs.splice(0)) cg.close(); + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); + }); + + const index = async (files: Record): Promise => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-c-macro-return-')); + roots.push(root); + for (const [rel, text] of Object.entries(files)) fs.writeFileSync(path.join(root, rel), text); + const cg = await CodeGraph.init(root, { index: true }); + graphs.push(cg); + return cg; + }; + + it('is indexed as a function on its line, and its callers reach it', async () => { + const cg = await index({ 'lib.c': LIB_C }); + const functions = cg.getNodesInFile('lib.c').filter((n: Node) => n.kind === 'function') + .map((n) => `${n.name}@${n.startLine}`).sort(); + expect(functions).toEqual([ + `caller@${lineOf(LIB_C, 'int caller(void)')}`, + `checked_sum@${lineOf(LIB_C, 'EXPORT(int) checked_sum(int a, int b)', true)}`, + `key_slot@${lineOf(LIB_C, 'int key_slot(')}`, + `label_for@${lineOf(LIB_C, 'EXPORT(const char *)')}`, + `log_line@${lineOf(LIB_C, 'void log_line(')}`, + `make_name@${lineOf(LIB_C, 'make_name(int id)')}`, + `usable_size@${lineOf(LIB_C, 'unsigned long')}`, + `version@${lineOf(LIB_C, 'EXPORT(int) version(void)')}`, + ].sort()); + const caller = cg.getNodesInFile('lib.c').find((n: Node) => n.name === 'caller')!; + const callees = cg.getCallees(caller.id).filter((r) => r.edge.kind === 'calls').map((r) => r.node.name).sort(); + expect(callees).toEqual(['checked_sum', 'key_slot', 'label_for', 'log_line', 'make_name', 'usable_size', 'version']); + }, 120_000); + + it('a header prototype declares it, and is no call from the header', async () => { + const cg = await index({ 'api.h': API_H, 'api.c': API_C }); + const header = cg.getNodesInFile('api.h').find((n: Node) => n.kind === 'file')!; + expect(cg.getOutgoingEdgesFrom([header.id]).filter((e) => e.kind === 'calls')).toEqual([]); + const useApi = cg.getNodesInFile('api.c').find((n: Node) => n.name === 'use_api')!; + expect(cg.getCallees(useApi.id).filter((r) => r.edge.kind === 'calls').map((r) => `${r.node.name}@${r.node.filePath}`).sort()) + .toEqual(['git_blob_dup@api.c', 'git_name@api.c']); + }, 120_000); + + it('the pre-parse blank leaves it alone, and still blanks markup', () => { + const keep = (text: string): string => blankCppAnnotationMacroCalls(text, { keepCReturnTypes: true }); + const blank = (text: string): string => text.replace(/[^\n\r]/g, ' '); + const kept = [ + 'CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item)\n{\n}\n', + 'GIT_EXTERN(int) git_blob_dup(git_blob **out, git_blob *source);\n', + 'GIT_INLINE(void) git_free(void *p)\n{\n}\n', + ]; + for (const text of kept) expect(keep(text)).toBe(text); + // Markup: not a type, a declaration with its own type, the type on the line above, a statement in a body. + expect(keep('ATTR_FORMAT(1, 2)\nvoid log_line(const char *fmt, ...);\n')) + .toBe(`${blank('ATTR_FORMAT(1, 2)')}\nvoid log_line(const char *fmt, ...);\n`); + expect(keep('ATTR(cold) void die(void);\n')).toBe(`${blank('ATTR(cold)')} void die(void);\n`); + expect(keep('size_t JEMALLOC_NOTHROW\nJEMALLOC_ATTR(pure)\nje_sallocx(const void *ptr, int flags) {\n}\n')) + .toBe(`size_t JEMALLOC_NOTHROW\n${blank('JEMALLOC_ATTR(pure)')}\nje_sallocx(const void *ptr, int flags) {\n}\n`); + expect(keep('void f(void) {\n FOREACH(node) visit(node);\n}\n')) + .toBe(`void f(void) {\n ${blank('FOREACH(node)')} visit(node);\n}\n`); + // C++ gets no exemption. + expect(blankCppAnnotationMacroCalls('GIT_EXTERN(int) git_blob_dup(void);\n')).toBe(`${blank('GIT_EXTERN(int)')} git_blob_dup(void);\n`); + }); +}); diff --git a/src/extraction/languages/c-cpp.ts b/src/extraction/languages/c-cpp.ts index 11d51d4849..52b3339345 100644 --- a/src/extraction/languages/c-cpp.ts +++ b/src/extraction/languages/c-cpp.ts @@ -1151,12 +1151,20 @@ export function maskCppRawStrings(source: string): { source: string; restore: (b * an operator (`MAKE(a) + 1`) — all rejected. String/char literals inside the * args are skipped so an embedded `)` can't mis-close the balance. * - * C++-only (wired into cppExtractor). A blanked macro inside a block comment is - * harmless (comments don't parse), and the rare line-leading no-semicolon - * ALL-CAPS call that isn't markup only loses that one annotation, never a whole - * class. + * C runs it too (redis's `REDIS_NO_SANITIZE("bounds")` above a definition), + * with `keepCReturnTypes`: there a macro that wraps the return type of the + * function declared after it is no markup, it IS the type — cJSON's + * `CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item)`, libgit2's + * `GIT_EXTERN(int) git_blob_dup(…);` — and tree-sitter-c reads it as one (a + * `macro_type_specifier`). Blanked with its argument, it left `cJSON_Print(…)` + * and a block, which parse as a call: the function had no node (#2470). See + * isCReturnTypeMacro for how such a macro is told from markup. + * + * A blanked macro inside a block comment is harmless (comments don't parse), + * and the rare line-leading no-semicolon ALL-CAPS call that isn't markup only + * loses that one annotation, never a whole class. */ -export function blankCppAnnotationMacroCalls(source: string): string { +export function blankCppAnnotationMacroCalls(source: string, options: { keepCReturnTypes?: boolean } = {}): string { if (!/^[ \t]*[A-Z][A-Z0-9_]{2,}\s*\(/m.test(source)) return source; const rawStrings = maskCppRawStrings(source); source = rawStrings.source; @@ -1165,7 +1173,8 @@ export function blankCppAnnotationMacroCalls(source: string): string { let m: RegExpExecArray | null; while ((m = re.exec(source)) !== null) { const macroStart = m.index + (m[1] ?? '').length; // skip leading indent - let i = m.index + m[0].length - 1; // index of the opening '(' + const open = m.index + m[0].length - 1; // index of the opening '(' + let i = open; let depth = 0; let end = -1; for (; i < source.length; i++) { @@ -1192,6 +1201,7 @@ export function blankCppAnnotationMacroCalls(source: string): string { // Only markup is followed by the declaration it decorates; a statement call // (`;`), init-list item (`,`/`{`), or expression fragment (operator) is not. if (!after || !/[A-Za-z_~#]/.test(after)) continue; + if (options.keepCReturnTypes && macroStart === m.index && isCReturnTypeMacro(source, m.index, open, end, j)) continue; for (let k = macroStart; k < end; k++) { if (chars[k] !== '\n' && chars[k] !== '\r') chars[k] = ' '; } @@ -1200,6 +1210,61 @@ export function blankCppAnnotationMacroCalls(source: string): string { return rawStrings.restore(chars.join('')); } +/** A macro argument that reads as a C type: `char *`, `const cJSON *`, `unsigned long`, `git_repository *`. */ +const C_RETURN_TYPE_ARG_RE = /^\s*[A-Za-z_][\w\s*]*$/; +/** A function declarator's name and the `(` of its parameter list. */ +const C_DECLARATOR_HEAD_RE = /[A-Za-z_]\w*\s*\(/y; +/** + * What may end a function declarator: attributes (`__attribute__((format(printf, 1, 2)))`, + * an ALL-CAPS attribute macro, with or without arguments), then the body's `{` or a prototype's `;`. + */ +const C_DECLARATOR_TAIL_RE = /\s*(?:(?:__attribute__\s*\(\((?:[^()]|\([^()]*\))*\)\)|[A-Z][A-Z0-9_]*(?![\w$])(?:\s*\((?:[^()]|\([^()]*\))*\))?)\s*)*[{;]/y; + +/** + * Whether the column-0 macro call `MACRO(…)` starting at `lineStart` (its + * argument list from `open` to `end`, the next code at `next`) is the return + * type of the function declared right after it, not markup: + * - its one argument reads as a type, so `ATTR_FORMAT(1, 2)` and + * `REDIS_NO_SANITIZE("bounds")` are markup; + * - only the function's declarator follows: a name, its parameter list, + * attributes, then `{` or `;`. Markup is followed by a declaration with a + * type of its own (`ATTR(x) void f(void);`); + * - the line above leaves no declaration open. jemalloc writes the type + * there: `size_t JEMALLOC_NOTHROW\nJEMALLOC_ATTR(pure)\nje_sallocx(…) {`, + * and that macro is the attribute it looks like. + * Column 0 is where a file-scope declaration starts; an indented statement + * prefix in a body (`FOREACH(n) visit(n);`) is no return type. + */ +function isCReturnTypeMacro(source: string, lineStart: number, open: number, end: number, next: number): boolean { + if (!C_RETURN_TYPE_ARG_RE.test(source.slice(open + 1, end - 1))) return false; + C_DECLARATOR_HEAD_RE.lastIndex = next; + const head = C_DECLARATOR_HEAD_RE.exec(source); + if (!head) return false; + let close = -1; + let depth = 0; + for (let i = next + head[0].length - 1; i < source.length && close < 0; i++) { + const c = source[i]; + if (c === '(') depth++; + else if (c === ')') { if (--depth === 0) close = i; } + else if (c === ';' || c === '{' || c === '}') return false; + } + if (close < 0) return false; + C_DECLARATOR_TAIL_RE.lastIndex = close + 1; + return C_DECLARATOR_TAIL_RE.test(source) && !opensCDeclaration(source, lineStart); +} + +/** + * Whether the line right above `lineStart` leaves a declaration open: its + * code ends in a type's last token (`unsigned long`, `char *`, `size_t + * JEMALLOC_NOTHROW`). A blank line, a directive, or a comment closes none. + */ +function opensCDeclaration(source: string, lineStart: number): boolean { + if (lineStart === 0) return false; + const from = source.lastIndexOf('\n', lineStart - 2) + 1; + const line = source.slice(from, lineStart - 1).replace(/\/\/.*$/, '').trim(); + return line !== '' && !line.startsWith('#') && /[\w*]$/.test(line); +} + /** * Blank a macro that is the ONLY token on its line — no parens, no semicolon: * namespace-management macros (`FMT_BEGIN_NAMESPACE`, `FMT_END_EXPORT`, @@ -2498,7 +2563,8 @@ function preParseCSource(source: string): string { ) ) ) - ) + ), + { keepCReturnTypes: true } ) ) ) From 7dfef8451fc9d937cb4ffc9f7ec2a4063f9898d2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 02:29:50 +0000 Subject: [PATCH 14/95] fix(installer): Hermes keeps your MCP entry and gets no platform_toolsets line (#2774) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause, two parts: 1. Install and refresh appended `mcp-codegraph` to `platform_toolsets.cli` in `~/.hermes/config.yaml`. It was never needed. Hermes loads every enabled MCP server on a platform whose list names none (hermes_cli/tools_config.py `_merge_mcp_servers`, a rule present back in May 2026 when the target was written). The name itself is harmful: - Hermes up to 0.22.2 rejects it at startup ("Unknown toolsets"). - Hermes 0.22.4 maps `mcp-` onto the server, so the list becomes an MCP allowlist and every other server disappears from CLI sessions. A user who deleted the line got it back on the next upgrade. 2. When the `mcp_servers.codegraph` block differed from the template, it was replaced wholesale. That reverted an absolute command path, the timeouts, `enabled: false` and `env`. The line-based YAML code also missed some shapes: - a codegraph key written on one line or quoted, which got a second codegraph key; - a 4-space-indented `mcp_servers`, which came out unparseable; - a one-line `mcp_servers: {…}`, which got a second top-level key, and Hermes's PyYAML reader then dropped the user's servers. CRLF files came back LF. Fix: - CodeGraph never writes `platform_toolsets`. - A `- mcp-codegraph` line an earlier install left in `platform_toolsets.cli` is removed, or becomes `- codegraph` when the list is an allowlist naming other MCP servers (the spelling every Hermes version reads as the server). An allowlist that leaves CodeGraph out gets a note, not an edit. - An existing `mcp_servers.codegraph` entry is the user's: nothing in it changes. A block entry missing `command` or `args` gets just those, at its own indent. - A new entry goes in at the indent its siblings use, and a codegraph key is found however it is written. `mcp_servers: {}` becomes a block. - A one-line mapping that holds other servers makes install refuse with a clear error and change nothing. One that holds codegraph is left as written, with a note, and uninstall says it can't remove it. - The file keeps its line breaks, and is written only when something was edited. - Uninstall also removes a `- codegraph` the refresh may have written. Hermes references (NousResearch/hermes-agent v0.22.4, 64dbae64): - hermes_cli/tools_config.py `_get_platform_tools` / `_merge_mcp_servers` ("mcp- names the same server as its bare alias: either spelling selects only that server"). - hermes_cli/toolset_validation.py `saved_toolset_resolver`. - website/docs/reference/mcp-config-reference.md (command, args, env, enabled, timeout, connect_timeout). - hermes_cli/tools_config.py at 1b9c539c (2026-05-17) for the May 2026 rule. Validation: - installer-targets.test.ts: 13 new Hermes tests. 3 existing tests that asserted the `mcp-codegraph` write are updated; the #456 same-indent list case now proves the old line is removed cleanly. With main's hermes.ts put back, those 16 fail, all on behavior. The 7 other Hermes tests pass on both. - The real CLI on this Windows machine against a private HERMES_HOME: `install --yes`, a second `install --yes`, `install --refresh` and `uninstall --yes --keep-cli`, for 14 config.yaml shapes. Each result was loaded with PyYAML (the reader Hermes uses), checked for duplicate keys, and run through a re-implementation of the cited selection rule. Of 42 results: - main: 2 unparseable, 10 with duplicate keys, 26 carrying `mcp-codegraph`, 10 where Hermes 0.22.4 would load only codegraph in CLI sessions, 3 converted CRLF to LF, and the customized entry reverted. - The fix: 0, 0, 0, 0, 0, the customized entry kept byte-for-byte, and 1 install refused by design (a one-line `mcp_servers` holding other servers). Fixes #2485 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/installer-targets.test.ts | 214 ++++++++++-- src/installer/targets/hermes.ts | 498 ++++++++++++++++------------ 3 files changed, 473 insertions(+), 240 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2cdadfd688..3c733f7453 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) - `codegraph install` and `upgrade` no longer break Codex's `config.toml` when CodeGraph's `[mcp_servers.codegraph]` entry is written differently from how CodeGraph writes it, such as indented, with spaces or quotes inside the brackets, or with Windows line endings: the entry is now updated in place, where before a second copy was added and Codex could load none of the file. A file already broken that way is repaired on the next install or upgrade, an example of the entry inside a multiline setting such as `developer_instructions` is no longer mistaken for it, uninstall removes the entry however it is written, and an entry you wrote with dotted keys or as an inline table is left as you wrote it. When your `mcp_servers` settings are one inline table, CodeGraph now says it can't add itself instead of breaking the file. Thanks @dencoseca for the report, and @danusha2345 and @rudycelekli for the first fixes. (#2562) +- For Hermes Agent, `codegraph install` and `upgrade` no longer add `mcp-codegraph` to `platform_toolsets.cli`. Hermes up to 0.22.2 reported that name as an unknown toolset on every start, and Hermes 0.22.4 reads it as the only MCP server to load, hiding your others from CLI sessions; an entry an earlier install added is taken out again. A refresh also no longer resets your own `mcp_servers.codegraph` settings, such as an absolute command path, timeouts or `enabled: false`, and a config indented or written differently from Hermes's default no longer gets a duplicate or unreadable entry. Thanks @Huy3ko for the report and @conifercone for confirming it. (#2485) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) - In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) diff --git a/__tests__/installer-targets.test.ts b/__tests__/installer-targets.test.ts index d988d9ad50..778c14b6a5 100644 --- a/__tests__/installer-targets.test.ts +++ b/__tests__/installer-targets.test.ts @@ -802,33 +802,28 @@ describe('Installer targets — partial-state idempotency', () => { expect(cliAfter.mcpServers.codegraph).toBeDefined(); }); - it('hermes: install adds codegraph MCP server and cli toolset, preserving existing yaml', () => { + it('hermes: install adds the codegraph MCP server and leaves platform_toolsets alone, preserving existing yaml (#2485)', () => { const hermes = getTarget('hermes')!; const config = path.join(tmpHome, '.hermes', 'config.yaml'); fs.mkdirSync(path.dirname(config), { recursive: true }); - fs.writeFileSync(config, [ - 'model:', - ' default: qwen-3.7', - 'mcp_servers:', - ' other:', - ' command: other', + const toolsets = [ 'platform_toolsets:', ' cli:', ' - hermes-cli', ' discord:', ' - hermes-discord', '', - ].join('\n')); + ].join('\n'); + fs.writeFileSync(config, ['model:', ' default: qwen-3.7', 'mcp_servers:', ' other:', ' command: other', toolsets].join('\n')); const result = hermes.install('global', { autoAllow: true }); expect(result.files[0].action).toBe('updated'); const body = fs.readFileSync(config, 'utf-8'); expect(body).toContain('model:\n default: qwen-3.7'); - expect(body).toContain('mcp_servers:\n other:\n command: other'); - expect(body).toContain(' codegraph:\n command: codegraph'); - expect(body).toContain(' - hermes-cli'); - expect(body).toContain(' - mcp-codegraph'); - expect(body).toContain(' discord:\n - hermes-discord'); + expect(body).toContain('mcp_servers:\n other:\n command: other\n codegraph:\n command: codegraph'); + // No toolset entry: a list naming no MCP server loads every enabled one. + expect(body.endsWith(toolsets)).toBe(true); + expect(body).not.toContain('mcp-codegraph'); const second = hermes.install('global', { autoAllow: true }); expect(second.files[0].action).toBe('unchanged'); @@ -852,13 +847,14 @@ describe('Installer targets — partial-state idempotency', () => { // Regression for #456: PyYAML's default block style writes list items at the // SAME indent as the parent key (`cli:` and its `- hermes-cli` are both at // indent 2). The pre-fix line-based patcher mistook that first list item for - // the next sibling key, truncated the cli block, and spliced `- mcp-codegraph` - // at indent 4 BEFORE the existing items — producing unparseable YAML. - it('hermes: install preserves PyYAML-default list-at-same-indent style (issue #456)', () => { + // the next sibling key and truncated the cli block. Install no longer adds to + // that list (#2485), but it still has to read it to take out the + // `- mcp-codegraph` line an earlier install appended there. + it('hermes: install reads the PyYAML-default list-at-same-indent style (issue #456)', () => { const hermes = getTarget('hermes')!; const config = path.join(tmpHome, '.hermes', 'config.yaml'); fs.mkdirSync(path.dirname(config), { recursive: true }); - const original = [ + const lines = [ 'model:', ' default: gpt-4o', 'platform_toolsets:', @@ -873,29 +869,20 @@ describe('Installer targets — partial-state idempotency', () => { ' discord:', ' - hermes-discord', '', - ].join('\n'); - fs.writeFileSync(config, original); + ]; + // As an earlier install left it: our entry appended at the items' indent. + fs.writeFileSync(config, [...lines.slice(0, 9), ' - mcp-codegraph', ...lines.slice(9)].join('\n')); hermes.install('global', { autoAllow: true }); const body = fs.readFileSync(config, 'utf-8'); - // mcp-codegraph appended at the same 2-space indent as existing items - expect(body).toContain('\n - mcp-codegraph\n'); - // hermes-cli preserved - expect(body).toContain('\n - hermes-cli\n'); - // Sibling sections kept their indent — `telegram:` is still a key under - // platform_toolsets, not promoted up. - expect(body).toContain('\n telegram:\n - hermes-telegram\n'); - expect(body).toContain('\n discord:\n - hermes-discord\n'); + // The old entry is gone and every other line is where it was. + expect(body).not.toContain('mcp-codegraph'); + expect(body.startsWith(lines.join('\n'))).toBe(true); // No list items leaked to the platform_toolsets level (indent 0). expect(body).not.toMatch(/^- browser/m); expect(body).not.toMatch(/^- hermes-telegram/m); - // The whole platform_toolsets block extracted by line search should - // start with `cli:` and not contain a stray 4-space `mcp-codegraph` - // appearing before the rest of the existing items. - expect(body).toContain(' cli:\n - hermes-cli\n - browser'); - // Idempotent const second = hermes.install('global', { autoAllow: true }); expect(second.files[0]?.action).toBe('unchanged'); @@ -918,15 +905,15 @@ describe('Installer targets — partial-state idempotency', () => { hermes.install('global', { autoAllow: true }); const installed = fs.readFileSync(config, 'utf-8'); - expect(installed).toContain('- mcp-codegraph'); + expect(installed).not.toContain('mcp-codegraph'); + expect(installed.startsWith(original)).toBe(true); expect(installed).toContain('codegraph:'); hermes.uninstall('global'); const body = fs.readFileSync(config, 'utf-8'); expect(body).not.toContain('mcp-codegraph'); expect(body).not.toContain('command: codegraph'); - expect(body).toContain(' cli:\n - hermes-cli\n - browser'); - expect(body).toContain(' telegram:\n - hermes-telegram'); + expect(body.startsWith(original)).toBe(true); }); it('opencode: uninstall removes only mcp.servers.codegraph, preserves comments and siblings', () => { @@ -1850,6 +1837,163 @@ describe('Installer — Codex install, re-install, refresh and uninstall find th }); }); +// --------------------------------------------------------------------------- +// Hermes: no toolset entry, and the user's MCP entry is theirs (#2485) +// +// Install and refresh appended `mcp-codegraph` to `platform_toolsets.cli`. +// Hermes up to 0.22.2 rejects that name at startup ("Unknown toolsets"), and +// 0.22.4 reads it as naming codegraph, which turns the list into an MCP +// allowlist and hides every other server from CLI sessions +// (hermes_cli/tools_config.py `_merge_mcp_servers`). No entry is needed: a +// list that names no MCP server loads every enabled one. The refresh also +// replaced the whole `mcp_servers.codegraph` block with CodeGraph's template, +// undoing an absolute command path, timeouts, `enabled: false` and env. +// --------------------------------------------------------------------------- +describe('Installer — Hermes never writes platform_toolsets and keeps the user\'s MCP entry (#2485)', () => { + let tmpHome: string; + let tmpCwd: string; + let origCwd: string; + let homeRestore: { restore: () => void }; + + beforeEach(() => { + tmpHome = mkTmpDir('hermes2485-home'); + tmpCwd = mkTmpDir('hermes2485-cwd'); + origCwd = process.cwd(); + process.chdir(tmpCwd); + homeRestore = setHome(tmpHome); + }); + + afterEach(() => { + homeRestore.restore(); + process.chdir(origCwd); + fs.rmSync(tmpHome, { recursive: true, force: true }); + fs.rmSync(tmpCwd, { recursive: true, force: true }); + }); + + const hermes = getTarget('hermes')!; + const config = () => path.join(tmpHome, '.hermes', 'config.yaml'); + function writeConfig(text: string): void { + fs.mkdirSync(path.dirname(config()), { recursive: true }); + fs.writeFileSync(config(), text); + } + const read = () => fs.readFileSync(config(), 'utf-8'); + const codegraphKeys = (text: string) => text.split(/\r?\n/).filter((l) => /^\s*["']?codegraph["']?\s*:/.test(l)).length; + const OTHERS = 'mcp_servers:\n agentmail:\n command: npx\n args:\n - -y\n - agentmail-mcp\n'; + const ENTRY = ' codegraph:\n command: codegraph\n args:\n - serve\n - --mcp\n timeout: 120\n connect_timeout: 60\n enabled: true\n'; + + it('a fresh install writes the server and no platform_toolsets', () => { + hermes.install('global', { autoAllow: false }); + expect(read()).toBe(`mcp_servers:\n${ENTRY}`); + expect(hermes.install('global', { autoAllow: false }).files[0]!.action).toBe('unchanged'); + expect(hermes.printConfig('global')).not.toContain('platform_toolsets'); + }); + + it('the mcp-codegraph line an earlier install added is taken out, so every server loads again; a second refresh is unchanged', () => { + const before = `model: big\n${OTHERS}${ENTRY}platform_toolsets:\n cli:\n - hermes-cli\n - browser\n`; + writeConfig(`model: big\n${OTHERS}${ENTRY}platform_toolsets:\n cli:\n - hermes-cli\n - browser\n - mcp-codegraph\n`); + const report = refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === 'hermes')!; + expect(report.status).toBe('refreshed'); + expect(read()).toBe(before); + expect(refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === 'hermes')!.status).toBe('unchanged'); + }); + + it('a line the user removed is not put back', () => { + const text = `model: big\n${OTHERS}${ENTRY}platform_toolsets:\n cli:\n - hermes-cli\n`; + writeConfig(text); + expect(refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === 'hermes')!.status).toBe('unchanged'); + expect(read()).toBe(text); + }); + + it('in an allowlist of MCP servers, mcp-codegraph becomes codegraph, the name every Hermes version reads', () => { + writeConfig(`${OTHERS}${ENTRY}platform_toolsets:\n cli:\n - hermes-cli\n - agentmail\n - mcp-codegraph # added by codegraph\n`); + hermes.install('global', { autoAllow: false }); + expect(read()).toBe(`${OTHERS}${ENTRY}platform_toolsets:\n cli:\n - hermes-cli\n - agentmail\n - codegraph\n`); + }); + + it('an allowlist that leaves CodeGraph out is left alone, with a note', () => { + const text = `${OTHERS}${ENTRY}platform_toolsets:\n cli:\n - hermes-cli\n - agentmail\n`; + writeConfig(text); + const result = hermes.install('global', { autoAllow: false }); + expect(result.files[0]!.action).toBe('unchanged'); + expect(result.notes?.join('\n')).toMatch(/platform_toolsets\.cli in .*config\.yaml names MCP servers, so CLI sessions load only those\. Add `codegraph`/); + expect(read()).toBe(text); + }); + + it('a customized entry is kept byte-for-byte by install and refresh', () => { + const text = `model: big\n${OTHERS} codegraph:\n command: /home/u/.local/bin/codegraph\n args: [serve, --mcp]\n timeout: 300\n connect_timeout: 90\n enabled: false\n env:\n CODEGRAPH_MCP_TOOLS: all\n`; + writeConfig(text); + expect(hermes.detect('global').alreadyConfigured).toBe(true); + expect(hermes.install('global', { autoAllow: false }).files[0]!.action).toBe('unchanged'); + expect(refreshTargets(ALL_TARGETS, 'global').find((r) => r.id === 'hermes')!.status).toBe('unchanged'); + expect(read()).toBe(text); + }); + + it('an entry missing command or args gets only those, at its own indent', () => { + writeConfig('mcp_servers:\n codegraph:\n enabled: true\n timeout: 300\n'); + hermes.install('global', { autoAllow: false }); + expect(read()).toBe('mcp_servers:\n codegraph:\n enabled: true\n timeout: 300\n command: codegraph\n args: [serve, --mcp]\n'); + }); + + it('a 4-space-indented mcp_servers gets the new entry at 4 spaces', () => { + writeConfig('mcp_servers:\n agentmail:\n command: npx\n'); + hermes.install('global', { autoAllow: false }); + expect(read()).toBe([ + 'mcp_servers:', ' agentmail:', ' command: npx', + ' codegraph:', ' command: codegraph', ' args:', ' - serve', ' - --mcp', + ' timeout: 120', ' connect_timeout: 60', ' enabled: true', '', + ].join('\n')); + }); + + it('an entry written on one line or with a quoted key is found, so no second codegraph key is added', () => { + for (const entry of [' codegraph: {command: /opt/bin/codegraph, args: [serve, --mcp]}\n', ' "codegraph":\n command: /opt/bin/codegraph\n args: [serve, --mcp]\n']) { + const text = `mcp_servers:\n${entry}`; + writeConfig(text); + expect(hermes.detect('global').alreadyConfigured).toBe(true); + hermes.install('global', { autoAllow: false }); + expect(read()).toBe(text); + hermes.uninstall('global'); + expect(codegraphKeys(read())).toBe(0); + } + }); + + it('mcp_servers written as a one-line mapping: install refuses with a clear error and changes nothing; {} becomes a block', () => { + const text = 'model: big\nmcp_servers: {agentmail: {command: npx, args: [-y, agentmail-mcp]}}\n'; + writeConfig(text); + expect(() => hermes.install('global', { autoAllow: false })).toThrow( + /config\.yaml writes mcp_servers as a one-line mapping \(line 2\), so CodeGraph can't add its server without rewriting it\. Nothing was changed/, + ); + expect(read()).toBe(text); + + writeConfig('model: big\nmcp_servers: {}\n'); + hermes.install('global', { autoAllow: false }); + expect(read()).toBe(`model: big\nmcp_servers:\n${ENTRY}`); + }); + + it('codegraph inside a one-line mcp_servers mapping is left as written; uninstall says it can\'t remove it', () => { + const text = 'mcp_servers: {codegraph: {command: codegraph, args: [serve, --mcp]}}\n'; + writeConfig(text); + expect(hermes.detect('global').alreadyConfigured).toBe(true); + const result = hermes.install('global', { autoAllow: false }); + expect(result.notes?.join('\n')).toMatch(/Left the mcp_servers\.codegraph entry in .*config\.yaml \(line 1\) as you wrote it/); + const removed = hermes.uninstall('global'); + expect(removed.notes?.join('\n')).toMatch(/inside a one-line mcp_servers mapping \(line 1\), which CodeGraph can't edit safely/); + expect(read()).toBe(text); + }); + + it('a CRLF config stays CRLF', () => { + writeConfig(`model: big\r\nplatform_toolsets:\r\n cli:\r\n - hermes-cli\r\n - mcp-codegraph\r\n`); + hermes.install('global', { autoAllow: false }); + expect(read()).toBe(`model: big\r\nplatform_toolsets:\r\n cli:\r\n - hermes-cli\r\n\r\nmcp_servers:\r\n${ENTRY.replace(/\n/g, '\r\n')}`); + }); + + it('uninstall removes the server and any codegraph toolset name, and keeps the rest', () => { + writeConfig(`${OTHERS}${ENTRY}platform_toolsets:\n cli:\n - hermes-cli\n - agentmail\n - codegraph\n telegram:\n - hermes-telegram\n`); + hermes.uninstall('global'); + expect(read()).toBe(`${OTHERS}platform_toolsets:\n cli:\n - hermes-cli\n - agentmail\n telegram:\n - hermes-telegram\n`); + expect(hermes.detect('global').alreadyConfigured).toBe(false); + }); +}); + describe('Installer — uninstallTargets sweep (codegraph uninstall)', () => { let tmpHome: string; let tmpCwd: string; diff --git a/src/installer/targets/hermes.ts b/src/installer/targets/hermes.ts index eacab6d3df..cf32f2d7c3 100644 --- a/src/installer/targets/hermes.ts +++ b/src/installer/targets/hermes.ts @@ -2,16 +2,28 @@ * Hermes Agent target. * * Hermes reads MCP servers from `$HERMES_HOME/config.yaml` under the - * top-level `mcp_servers` key, and exposes discovered MCP tools through - * dynamic toolsets named `mcp-`. We add: + * top-level `mcp_servers` key. We add one entry: * * mcp_servers.codegraph -> `codegraph serve --mcp` - * platform_toolsets.cli -> `mcp-codegraph` * - * The second entry matters because Hermes CLI profiles often enable an - * explicit `platform_toolsets.cli` list. Without `mcp-codegraph` in that - * list, the MCP server can be configured and connected but its tools may - * still be filtered out of normal CLI sessions. + * and nothing else. In particular we never write `platform_toolsets` (#2485). + * With no MCP server named in a platform's list, Hermes loads every enabled + * MCP server on that platform; naming one, as `` or `mcp-`, + * turns the list into an allowlist of the servers named + * (`hermes_cli/tools_config.py` `_merge_mcp_servers` in NousResearch/hermes-agent + * v0.22.4; the same rule, bare names only, dates back to May 2026). Earlier + * installs appended `mcp-codegraph` to `platform_toolsets.cli`: Hermes up to + * 0.22.2 reported it as an unknown toolset on every start, and 0.22.4 reads + * it as naming codegraph, which hides every other MCP server from CLI + * sessions. Install and refresh take that line out again + * (`healCliToolsetEntry`). + * + * An existing `mcp_servers.codegraph` entry belongs to the user: its values + * (an absolute command path, timeouts, `enabled: false`, env) are never + * rewritten. A refresh only adds `command` / `args` when one is missing. + * + * The file is edited as text, line by line, so comments and formatting + * survive; Hermes itself writes it with a comment-preserving round-trip. */ import * as fs from 'fs'; @@ -26,8 +38,6 @@ import { } from './types'; import { atomicWriteFileSync } from './shared'; -type LineRange = { start: number; end: number }; - class HermesTarget implements AgentTarget { readonly id = 'hermes' as const; readonly displayName = 'Hermes Agent'; @@ -46,7 +56,7 @@ class HermesTarget implements AgentTarget { const installed = fs.existsSync(hermesHome()) || fs.existsSync(file); return { installed, - alreadyConfigured: hasCodeGraphMcpServer(content), + alreadyConfigured: findCodeGraphServer(splitText(content).lines) !== null, configPath: file, }; } @@ -58,9 +68,10 @@ class HermesTarget implements AgentTarget { notes: ['Hermes Agent uses $HERMES_HOME/config.yaml; re-run with --location=global.'], }; } + const { file, notes } = writeHermesConfig(); return { - files: [writeHermesConfig()], - notes: ['Start a new Hermes session for MCP changes to take effect.'], + files: [file], + notes: [...notes, 'Start a new Hermes session for MCP changes to take effect.'], }; } @@ -71,30 +82,29 @@ class HermesTarget implements AgentTarget { return { files: [{ path: file, action: 'not-found' }] }; } - const before = readText(file); - const after = removeCodeGraphToolset(removeCodeGraphMcpServer(before)); - if (after === before) { - return { files: [{ path: file, action: 'not-found' }] }; + const text = splitText(readText(file)); + const notes: string[] = []; + const server = findCodeGraphServer(text.lines); + let removed = false; + if (server?.inFlowParent) { + notes.push(`${file} sets mcp_servers.codegraph inside a one-line mcp_servers mapping (line ${server.line + 1}), which CodeGraph can't edit safely. Remove it from there yourself.`); + } else if (server) { + text.lines.splice(server.line, server.end - server.line); + removed = true; } - atomicWriteFileSync(file, ensureTrailingNewline(after)); - return { files: [{ path: file, action: 'removed' }] }; + // The line earlier installs wrote, and `codegraph` (what a refresh made of + // it in an allowlist): with the server gone, either would be unknown. + if (removeCliToolsetEntries(text.lines, ['mcp-codegraph', 'codegraph'])) removed = true; + const action = removed ? 'removed' : 'not-found'; + if (removed) atomicWriteFileSync(file, joinText(text)); + return notes.length > 0 ? { files: [{ path: file, action }], notes } : { files: [{ path: file, action }] }; } printConfig(loc: Location): string { if (loc !== 'global') { return '# Hermes Agent uses $HERMES_HOME/config.yaml; use --location=global.\n'; } - return [ - `# Add to ${configPath()}`, - '', - renderCodeGraphMcpBlock().join('\n'), - '', - 'platform_toolsets:', - ' cli:', - ' - hermes-cli', - ' - mcp-codegraph', - '', - ].join('\n'); + return [`# Add to ${configPath()}`, '', 'mcp_servers:', ...renderCodeGraphServer(2), ''].join('\n'); } describePaths(loc: Location): string[] { @@ -120,237 +130,315 @@ function readText(file: string): string { } } -function writeHermesConfig(): WriteResult['files'][number] { +function writeHermesConfig(): { file: WriteResult['files'][number]; notes: string[] } { const file = configPath(); const existed = fs.existsSync(file); const before = readText(file); - const afterMcp = upsertCodeGraphMcpServer(before); - const after = upsertCodeGraphToolset(afterMcp); + const text = splitText(before); + const notes: string[] = []; + + const outcome = upsertCodeGraphServer(text.lines); + if (outcome === 'blocked') { + const top = findTop(text.lines, 'mcp_servers')!; + throw new Error( + `${file} writes mcp_servers as a one-line mapping (line ${top.line + 1}), so CodeGraph can't add its server without rewriting it. ` + + 'Nothing was changed: add `codegraph: {command: codegraph, args: [serve, --mcp]}` to it yourself, or write mcp_servers as a block, then run this again.', + ); + } + if (outcome === 'left-as-written') { + const server = findCodeGraphServer(text.lines)!; + notes.push(`Left the mcp_servers.codegraph entry in ${file} (line ${server.line + 1}) as you wrote it: CodeGraph only fills in a block-style entry.`); + } + + const toolsets = healCliToolsetEntry(text.lines); + if (toolsets === 'allowlist-without-codegraph') { + notes.push(`platform_toolsets.cli in ${file} names MCP servers, so CLI sessions load only those. Add \`codegraph\` to that list to use CodeGraph there.`); + } - if (after === before) { - return { path: file, action: 'unchanged' }; + // Rewrite only what was edited: a file CodeGraph had nothing to change in + // is left byte-for-byte as it is. + if (outcome !== 'changed' && toolsets !== 'changed') { + return { file: { path: file, action: 'unchanged' }, notes }; } - atomicWriteFileSync(file, ensureTrailingNewline(after)); - return { path: file, action: existed ? 'updated' : 'created' }; + atomicWriteFileSync(file, joinText(text)); + return { file: { path: file, action: existed ? 'updated' : 'created' }, notes }; } -function ensureTrailingNewline(text: string): string { - return text.endsWith('\n') ? text : text + '\n'; +// --------------------------------------------------------------------------- +// Line-level YAML: just the shapes Hermes and its users write for these keys. +// --------------------------------------------------------------------------- + +interface Text { + lines: string[]; + eol: '\n' | '\r\n'; } -function splitLines(content: string): string[] { - return content.replace(/\r\n/g, '\n').replace(/\r/g, '\n').split('\n'); +function splitText(content: string): Text { + const first = /\r\n|\r|\n/.exec(content); + const lines = content.length === 0 ? [] : content.split(/\r\n|\r|\n/); + if (lines.length > 0 && lines[lines.length - 1] === '') lines.pop(); + return { lines, eol: first?.[0] === '\r\n' ? '\r\n' : '\n' }; } -function joinLines(lines: string[]): string { - while (lines.length > 0 && lines[lines.length - 1] === '') lines.pop(); - return lines.join('\n') + '\n'; +/** The lines joined with the file's own line break, ending with one; no trailing blank lines. */ +function joinText(text: Text): string { + const lines = [...text.lines]; + while (lines.length > 0 && (lines[lines.length - 1] ?? '').trim() === '') lines.pop(); + return lines.length === 0 ? '' : lines.join(text.eol) + text.eol; } -function topLevelRange(lines: string[], key: string): LineRange | null { - const start = lines.findIndex((line) => line.trim() === `${key}:`); - if (start === -1) return null; - let end = lines.length; - for (let i = start + 1; i < lines.length; i++) { - const line = lines[i] ?? ''; - if (line.trim() === '') continue; - if (/^[A-Za-z_][A-Za-z0-9_-]*:\s*(?:#.*)?$/.test(line)) { - end = i; - break; - } - } - return { start, end }; +function indentOf(line: string): number { + return /^ */.exec(line)![0].length; } -function childRange(lines: string[], parent: LineRange, child: string): LineRange | null { - const startPattern = new RegExp(`^ ${escapeRegExp(child)}:\\s*(?:#.*)?$`); - let start = -1; - for (let i = parent.start + 1; i < parent.end; i++) { - if (startPattern.test(lines[i] ?? '')) { - start = i; - break; +function isBlankOrComment(line: string): boolean { + const t = line.trim(); + return t === '' || t.startsWith('#'); +} + +/** The value text of a `key: value` remainder with any trailing `# comment` removed. */ +function stripComment(rest: string): string { + let quote: string | null = null; + for (let i = 0; i < rest.length; i++) { + const ch = rest[i]; + if (quote) { + if (ch === quote) quote = null; + } else if (ch === '"' || ch === "'") { + quote = ch; + } else if (ch === '#' && (i === 0 || /\s/.test(rest[i - 1]!))) { + return rest.slice(0, i).trim(); } } - if (start === -1) return null; + return rest.trim(); +} - let end = parent.end; - for (let i = start + 1; i < parent.end; i++) { - const line = lines[i] ?? ''; - if (line.trim() === '') continue; - if (/^ \S/.test(line)) { - end = i; - break; - } +function unquote(key: string): string { + if (key.length >= 2 && key.startsWith('"') && key.endsWith('"')) { + return key.slice(1, -1).replace(/\\(.)/g, '$1'); } - while (end > start + 1 && (lines[end - 1] ?? '').trim() === '') { - end--; + if (key.length >= 2 && key.startsWith("'") && key.endsWith("'")) { + return key.slice(1, -1).replace(/''/g, "'"); } - return { start, end }; + return key; } -/** - * Block-range for a 2-space-indented child whose value is a YAML block list. - * - * Unlike `childRange`, this handles PyYAML's default `default_flow_style=False` - * serialization, where list items sit at the SAME indent as the parent key: - * - * cli: - * - hermes-cli # indent 2 — belongs to cli, not a sibling - * - browser - * - * `childRange`'s `^ \S` heuristic mistakes that first ` - hermes-cli` line - * for the next sibling key and truncates the block, causing inserts to land - * before the existing items at a different indent (issue #456). This helper - * recognizes a ` - ` line as part of the block instead, and reports back - * the actual indent used by existing items so the inserter matches it. - */ -function listChildBlock( - lines: string[], - parent: LineRange, - child: string, -): (LineRange & { itemIndent: string }) | null { - const startPattern = new RegExp(`^ ${escapeRegExp(child)}:\\s*(?:#.*)?$`); - let start = -1; - for (let i = parent.start + 1; i < parent.end; i++) { - if (startPattern.test(lines[i] ?? '')) { - start = i; +/** ` key: value` → its indent, decoded key and inline value ('' for a block). Not for `- item` lines. */ +function parseKeyLine(line: string): { indent: number; key: string; value: string } | null { + const m = /^( *)("(?:[^"\\]|\\.)*"|'(?:[^']|'')*'|[^\s#'"\-?:,[\]{}][^:#]*?|-[^\s:#][^:#]*?)[ \t]*:(?=[ \t]|$)(.*)$/.exec(line); + if (!m) return null; + return { indent: m[1]!.length, key: unquote(m[2]!), value: stripComment(m[3]!) }; +} + +/** A column-0 key: its line, inline value, and where the next column-0 key starts. */ +function findTop(lines: string[], key: string): { line: number; value: string; end: number } | null { + const start = lines.findIndex((l) => { + const k = parseKeyLine(l); + return k !== null && k.indent === 0 && k.key === key; + }); + if (start === -1) return null; + let end = lines.length; + for (let i = start + 1; i < lines.length; i++) { + const k = parseKeyLine(lines[i]!); + if (k && k.indent === 0) { + end = i; break; } } - if (start === -1) return null; + return { line: start, value: parseKeyLine(lines[start]!)!.value, end }; +} - let end = parent.end; - for (let i = start + 1; i < parent.end; i++) { - const line = lines[i] ?? ''; - if (line.trim() === '') continue; - const indentMatch = line.match(/^( *)/); - const indent = indentMatch?.[1]?.length ?? 0; - if (indent >= 4) continue; - if (indent === 2 && /^ - /.test(line)) continue; - end = i; - break; - } - while (end > start + 1 && (lines[end - 1] ?? '').trim() === '') { - end--; +/** One past the last indented, non-blank line of a block: where a new child goes. */ +function contentEnd(lines: string[], start: number, end: number): number { + let last = start; + for (let i = start + 1; i < end; i++) { + if ((lines[i] ?? '').trim() !== '' && indentOf(lines[i]!) > 0) last = i; } + return last + 1; +} - let itemIndent = ' '; - for (let i = start + 1; i < end; i++) { - const m = (lines[i] ?? '').match(/^( +)- /); - if (m && m[1]) { - itemIndent = m[1]; +interface Child { + key: string; + line: number; + value: string; + /** One past the child's last line. */ + end: number; +} + +/** The keys one level inside the block [start, end), and the indent they use (null when empty). */ +function childrenOf(lines: string[], start: number, end: number): { indent: number | null; children: Child[] } { + const stop = contentEnd(lines, start, end); + let indent: number | null = null; + for (let i = start + 1; i < stop; i++) { + if (!isBlankOrComment(lines[i]!)) { + indent = indentOf(lines[i]!); break; } } - return { start, end, itemIndent }; + const children: Child[] = []; + if (indent === null || indent === 0) return { indent: null, children }; + for (let i = start + 1; i < stop; i++) { + const k = parseKeyLine(lines[i]!); + if (k && k.indent === indent) { + if (children.length > 0) children[children.length - 1]!.end = i; + children.push({ key: k.key, line: i, value: k.value, end: stop }); + } + } + // A child's range ends at its last indented line, not on blank lines before the next sibling. + for (const c of children) { + while (c.end > c.line + 1 && (lines[c.end - 1] ?? '').trim() === '') c.end--; + } + return { indent, children }; } -function escapeRegExp(value: string): string { - return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); -} +// --------------------------------------------------------------------------- +// mcp_servers.codegraph +// --------------------------------------------------------------------------- -function renderCodeGraphMcpChild(): string[] { +/** The entry CodeGraph writes into a new install, with its key at `indent` spaces. */ +function renderCodeGraphServer(indent: number): string[] { + const a = ' '.repeat(indent); + const b = ' '.repeat(indent * 2); + const c = ' '.repeat(indent * 3); return [ - ' codegraph:', - ' command: codegraph', - ' args:', - ' - serve', - ' - --mcp', - ' timeout: 120', - ' connect_timeout: 60', - ' enabled: true', + `${a}codegraph:`, + `${b}command: codegraph`, + `${b}args:`, + `${c}- serve`, + `${c}- --mcp`, + `${b}timeout: 120`, + `${b}connect_timeout: 60`, + `${b}enabled: true`, ]; } -function renderCodeGraphMcpBlock(): string[] { - return ['mcp_servers:', ...renderCodeGraphMcpChild()]; -} - -function hasCodeGraphMcpServer(content: string): boolean { - const lines = splitLines(content); - const parent = topLevelRange(lines, 'mcp_servers'); - return !!parent && !!childRange(lines, parent, 'codegraph'); +/** + * Where `mcp_servers.codegraph` is written, in any form: a block or a + * one-line value under a block `mcp_servers`, at whatever indent its + * siblings use and with its key quoted or not; or inside a one-line + * `mcp_servers: {…}` mapping (`inFlowParent`). + */ +function findCodeGraphServer(lines: string[]): (Child & { inFlowParent: boolean }) | null { + const top = findTop(lines, 'mcp_servers'); + if (!top) return null; + if (top.value !== '') { + const mentions = /(^|[{,\s])["']?codegraph["']?\s*:/.test(top.value.replace(/^\{/, ' ')); + return mentions ? { key: 'codegraph', line: top.line, value: top.value, end: top.line + 1, inFlowParent: true } : null; + } + const { children } = childrenOf(lines, top.line, top.end); + const child = children.find((c) => c.key === 'codegraph'); + return child ? { ...child, inFlowParent: false } : null; } -function upsertCodeGraphMcpServer(content: string): string { - const lines = splitLines(content); - const parent = topLevelRange(lines, 'mcp_servers'); - const child = parent ? childRange(lines, parent, 'codegraph') : null; - const replacement = renderCodeGraphMcpChild(); - - if (!parent) { - if (lines.length > 0 && lines[lines.length - 1] === '') lines.pop(); +/** + * Add the codegraph server when the file has none. An existing entry is + * the user's: nothing in it is changed, except that a block-style entry + * missing `command` or `args` (it can't start without them) gets CodeGraph's. + */ +function upsertCodeGraphServer(lines: string[]): 'changed' | 'unchanged' | 'left-as-written' | 'blocked' { + const top = findTop(lines, 'mcp_servers'); + if (!top) { + while (lines.length > 0 && (lines[lines.length - 1] ?? '').trim() === '') lines.pop(); if (lines.length > 0) lines.push(''); - lines.push(...renderCodeGraphMcpBlock()); - return joinLines(lines); + lines.push('mcp_servers:', ...renderCodeGraphServer(2)); + return 'changed'; } - if (child) { - const existing = lines.slice(child.start, child.end); - if (arrayEqual(existing, replacement)) return joinLines(lines); - lines.splice(child.start, child.end - child.start, ...replacement); - return joinLines(lines); + if (top.value !== '') { + if (findCodeGraphServer(lines)) return 'left-as-written'; + // An empty one-line value (`{}`, `null`, `~`) is rewritten as a block; any + // other one-line mapping holds the user's servers and is never rewritten. + if (!/^(\{\s*\}|null|Null|NULL|~)$/.test(top.value)) return 'blocked'; + lines.splice(top.line, 1, 'mcp_servers:', ...renderCodeGraphServer(2)); + return 'changed'; } - lines.splice(parent.end, 0, ...replacement); - return joinLines(lines); -} - -function removeCodeGraphMcpServer(content: string): string { - const lines = splitLines(content); - const parent = topLevelRange(lines, 'mcp_servers'); - const child = parent ? childRange(lines, parent, 'codegraph') : null; - if (!child) return content; - lines.splice(child.start, child.end - child.start); - return joinLines(lines); + const { indent, children } = childrenOf(lines, top.line, top.end); + const existing = children.find((c) => c.key === 'codegraph'); + if (!existing) { + lines.splice(contentEnd(lines, top.line, top.end), 0, ...renderCodeGraphServer(indent ?? 2)); + return 'changed'; + } + if (existing.value !== '') return 'unchanged'; + + const own = childrenOf(lines, existing.line, existing.end); + const keyIndent = own.indent ?? (indent ?? 2) * 2; + const has = (key: string) => own.children.some((c) => c.key === key); + const missing: string[] = []; + if (!has('command')) missing.push(`${' '.repeat(keyIndent)}command: codegraph`); + if (!has('args')) missing.push(`${' '.repeat(keyIndent)}args: [serve, --mcp]`); + if (missing.length === 0) return 'unchanged'; + lines.splice(contentEnd(lines, existing.line, existing.end), 0, ...missing); + return 'changed'; } -function upsertCodeGraphToolset(content: string): string { - const lines = splitLines(content); - const parent = topLevelRange(lines, 'platform_toolsets'); - const cli = parent ? listChildBlock(lines, parent, 'cli') : null; +// --------------------------------------------------------------------------- +// platform_toolsets.cli +// --------------------------------------------------------------------------- - if (!parent) { - if (lines.length > 0 && lines[lines.length - 1] === '') lines.pop(); - if (lines.length > 0) lines.push(''); - lines.push('platform_toolsets:', ' cli:', ' - hermes-cli', ' - mcp-codegraph'); - return joinLines(lines); - } +interface CliList { + /** Each `- item` line of the cli list and its unquoted value. */ + items: Array<{ line: number; value: string }>; +} - if (!cli) { - lines.splice(parent.end, 0, ' cli:', ' - hermes-cli', ' - mcp-codegraph'); - return joinLines(lines); +/** The block-style `platform_toolsets.cli` list, or null (absent, or written on one line). */ +function cliList(lines: string[]): CliList | null { + const top = findTop(lines, 'platform_toolsets'); + if (!top || top.value !== '') return null; + const cli = childrenOf(lines, top.line, top.end).children.find((c) => c.key === 'cli'); + if (!cli || cli.value !== '') return null; + const items: CliList['items'] = []; + for (let i = cli.line + 1; i < cli.end; i++) { + const m = /^ *- +(.*)$/.exec(lines[i]!); + if (m) items.push({ line: i, value: unquote(stripComment(m[1]!)) }); } - - const hasEntry = lines - .slice(cli.start + 1, cli.end) - .some((line) => line.trim() === '- mcp-codegraph'); - if (hasEntry) return joinLines(lines); - - lines.splice(cli.end, 0, `${cli.itemIndent}- mcp-codegraph`); - return joinLines(lines); + return { items }; } -function removeCodeGraphToolset(content: string): string { - const lines = splitLines(content); - const parent = topLevelRange(lines, 'platform_toolsets'); - const cli = parent ? listChildBlock(lines, parent, 'cli') : null; - if (!cli) return content; - - const hasEntry = lines - .slice(cli.start + 1, cli.end) - .some((line) => line.trim() === '- mcp-codegraph'); - if (!hasEntry) return content; +/** The names of the servers under a block-style `mcp_servers`. */ +function serverNames(lines: string[]): Set { + const top = findTop(lines, 'mcp_servers'); + if (!top || top.value !== '') return new Set(); + return new Set(childrenOf(lines, top.line, top.end).children.map((c) => c.key)); +} - const next = lines.filter((line, idx) => { - if (idx <= cli.start || idx >= cli.end) return true; - return line.trim() !== '- mcp-codegraph'; - }); - return joinLines(next); +/** + * Take out the `- mcp-codegraph` line earlier installs appended to + * `platform_toolsets.cli`. When the list names no other MCP server, the + * line is removed, so every enabled server loads again. When it names + * others too — an allowlist the user keeps — it becomes `- codegraph`, the + * spelling every Hermes version reads as naming the server, so CodeGraph + * stays in it. Reports an allowlist that doesn't include CodeGraph. + */ +function healCliToolsetEntry(lines: string[]): 'unchanged' | 'changed' | 'allowlist-without-codegraph' { + const list = cliList(lines); + if (!list) return 'unchanged'; + const servers = serverNames(lines); + const names = (value: string) => servers.has(value) ? value + : value.startsWith('mcp-') && servers.has(value.slice(4)) ? value.slice(4) : null; + const othersNamed = list.items.some((it) => { const n = names(it.value); return n !== null && n !== 'codegraph'; }); + const ours = list.items.filter((it) => it.value === 'mcp-codegraph'); + const bare = list.items.some((it) => it.value === 'codegraph'); + + if (ours.length === 0) return othersNamed && !bare ? 'allowlist-without-codegraph' : 'unchanged'; + const [keep, ...drop] = ours; + for (const it of [...drop].reverse()) lines.splice(it.line, 1); + if (othersNamed && !bare) lines[keep!.line] = lines[keep!.line]!.replace(/- +.*$/, '- codegraph'); + else lines.splice(keep!.line, 1); + return 'changed'; } -function arrayEqual(a: string[], b: string[]): boolean { - return a.length === b.length && a.every((value, idx) => value === b[idx]); +/** Remove `- ` lines for any of `values` from `platform_toolsets.cli`; true when one was. */ +function removeCliToolsetEntries(lines: string[], values: string[]): boolean { + const list = cliList(lines); + if (!list) return false; + let removed = false; + for (const it of [...list.items].reverse()) { + if (values.includes(it.value)) { + lines.splice(it.line, 1); + removed = true; + } + } + return removed; } export const hermesTarget: AgentTarget = new HermesTarget(); From c326b6bec19a826f679a1fd8ec325888521ccde1 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 02:48:37 +0000 Subject: [PATCH 15/95] fix(go): a function value keeps its link past a local of the same name (#2776) In Go, a function used as a value, like `apply(run)`, was recorded once per function and name, at its first use. Since PR #2658 the resolver drops a use that a local or parameter of that name shadows. When a function's first use was the shadowed one and a later use, outside the local's scope, was the package function, that later use had no reference left, and the function lost its references edge to the package function (an inner-block `run := 2`, a closure's `func(run int)`, a case clause, an if header or a range variable, then `apply(run)`). Issue #2596 predicted this. Fix, in both extractors (function-ref.ts with tree-sitter.ts's flushFnRefCandidates, and the kernel's go.rs flush_fn_ref_candidates): - A Go function that binds the name itself records one use per scope instead of one per function. Bindings are a func literal's parameters, :=, var, const, range and type-switch variables and a select case's value. Scopes are Go's: a block, an if/for/switch/select statement's implicit block and each case clause. Within one scope a local declared before the first use also covers every later use, so the first use still speaks for the scope. - The function's own receiver, parameters and results keep a single reference: they cover the whole body. - A func literal at file scope is its own unit. - Every other function is unchanged. The scope and enclosing function are read from the tree only for names a function uses at least twice, after the name gate (reading them for every container made kernel extraction of prometheus 2.4x slower). Validation: - go-shadowed-function-value.test.ts: 14 cases, kernel and wasm, LF and CRLF. On main's sources and kernel 10 fail; with the fix 14 pass. Per-scope keys without the binding check, and dropping range variables, case clauses or func literal parameters, each fail 3. - Kernel parity, Go: gin 99/99, etcd 1065/1105 (40 deferred, as main), prometheus 746/746, kubernetes 17779/17952 (173 deferred, as main), 0 files with diffs. - A/B against 6bd217a2 (fresh indexes, each arm with its own kernel): gin 0f09c3a9, etcd bf9cadf0, prometheus 017fcf2e and kubernetes 006cdf87 keep identical nodes and edges. The only change is 0, 4, 13 and 163 more function_ref rows, all failed: per-scope uses of a local (all 17 on etcd and prometheus and 36 on kubernetes read by hand). express and flask dump byte-identically. - Cost: kernel extraction of prometheus 2.90 s vs 2.90 s, wasm 6.06 vs 6.07 s; kubernetes 34.9 vs 34.9 s; prometheus index medians 12.35 s / 49.1 s CPU vs 12.45 s / 48.3 s over 4 interleaved rounds. Fixes #2596 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/go-shadowed-function-value.test.ts | 275 +++++++++++++++++++ codegraph-kernel/src/go.rs | 196 +++++++++++-- src/extraction/function-ref.ts | 160 +++++++++++ src/extraction/tree-sitter.ts | 24 +- 5 files changed, 635 insertions(+), 21 deletions(-) create mode 100644 __tests__/go-shadowed-function-value.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c733f7453..248f157d17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -178,6 +178,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a type defined from other types, like etcd's `type WatchChan <-chan WatchResponse` or gin's `type HandlerFunc func(*Context)` and `type HandlersChain []HandlerFunc`, now links to the types it is built from, so impact on `WatchResponse` or gin's `Context` reaches these types and the code that uses them. Before, such a type linked to nothing. One written through another package, like etcd's `type PutResponse pb.PutResponse`, links to that package's type, and a recursive one like `type stateFn func(*Lexer) stateFn` doesn't link to itself. Re-index Go projects after upgrading. - In Go, a struct now links to the types its fields are declared with, like etcd's `PutResponse` holding a `*ResponseHeader` or gin's `Context` holding a `HandlersChain`, so impact on `ResponseHeader` or `HandlersChain` reaches every struct that holds one and the code that uses those structs, and `codegraph_node` lists a struct's field types. Before, a struct linked only to the types it embeds. A field written through another package, like `*pb.ResponseHeader`, links to that package's type, one from outside your project, like `context.Context`, links to nothing, the types an embedded generic is filled in with, like the `Pod` in client-go's `listers.ResourceIndexer[*v1.Pod]`, link too, and a struct that holds itself, like a list node's `next *node`, doesn't link to itself. `codegraph_explore` still answers a question about a flow with the code that runs it, not with the structs that merely hold one of its types. Re-index Go projects after upgrading. - In Go, an unexported name used in another file of its package, like prometheus's `memSeries` in `tsdb/head_append.go`, now links to that package's own declaration when a package nested inside it, like `tsdb/agent`, declares the same name. Before, such a parameter, result, struct field, composite literal or call linked to nothing. A call or a function value that the surrounding function binds itself, like a test's `init := func(…)` followed by `init(t)`, or a local `volumeSpec` variable passed on, no longer links to the package's declaration of that name. A name bound in one `case` of a `switch` or `select`, or a parameter named inside a function-typed result like `func() func(run string)`, is not such a binding anywhere else, so a call there still links. Thanks @inth3shadows. (#2580) Re-index Go projects after upgrading. +- In Go, when a function uses a package function as a value, like `apply(run)`, and also has a local or parameter of the same name somewhere, like a `run := 2` inside an `if` or a `case`, a loop variable, or a closure's parameter, each use is now read on its own: a use the local covers links to nothing, and a use outside it links to the package function. Before, only the function's first use was looked at, so a local used first hid every later use of the package function. Re-index Go projects after upgrading. Thanks @inth3shadows for the report. (#2596) - In Go, a method an interface declares, like etcd's `KV.Put(ctx context.Context, key, val string, opts ...OpOption) (*PutResponse, error)`, now links to the types its parameters and results name, so impact on `PutResponse` or `OpOption` reaches the interface method and every caller that goes through the interface. Before, an interface's methods linked to no types at all. Re-index Go projects after upgrading. - In Go, a package imported under an explicit name, like `v3 "go.etcd.io/etcd/client/v3"`, now owns that name even when another import's path ends in it, like `"github.com/cheggaaa/pb/v3"` (package `pb`) in etcd's benchmark tool. Before, whichever of the two came first took every `v3.` name, so `v3.Client` and `v3.Op` could link to nothing. Re-index Go projects after upgrading. - In Go, a struct or defined type now counts as implementing an interface only when its methods take as many parameters and return as many results as the interface's. Before, matching method names was enough, so etcd's watch cache, whose `Get` takes a context, a key and options, was listed as a `peerGetter`, whose `Get` takes a peer ID, and a call through an interface could be shown reaching methods it can never reach. Look-alikes could also crowd out the real implementations, so `codegraph_explore` stopped at grpc-go's xDS `TransportBuilder` instead of reaching the transport it builds. A server still counts for the gRPC client interface of the service it serves, like etcd's `kvServer` for `KVClient`, so a client's call keeps leading to the handler that serves it. Re-index Go projects after upgrading. diff --git a/__tests__/go-shadowed-function-value.test.ts b/__tests__/go-shadowed-function-value.test.ts new file mode 100644 index 0000000000..e9c9887686 --- /dev/null +++ b/__tests__/go-shadowed-function-value.test.ts @@ -0,0 +1,275 @@ +/** + * A Go function used as a value is recorded once per function, at its first + * use (#756). Since #2658 a use that a parameter or local of the same name + * shadows links to nothing, which is right for that use. But when the first + * use in a function is the shadowed one and a later use, outside the local's + * scope, is the package function, the later use had no reference left: the + * function lost its `references` edge to the package function (#2596). + * + * A function that binds the name itself now records one use per scope (a + * block, a statement's implicit block, a case clause), so the later use keeps + * its own reference. Within one scope a local declared before the first use + * covers every later use as well, so the first use still speaks for the + * scope. A function that binds no such local keeps its single reference, and + * so does one whose own parameter has the name: that covers the whole body. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import type { ExtractionResult } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +const GO_MOD = 'module example.com/app\n\ngo 1.22\n'; + +/** Line numbers are pinned by the assertions below: keep the layout. */ +const MAIN_GO = `package main + +func run(a, b int) int { return a + b } + +func apply(f any) {} + +func pick(f any) any { return f } + +func shadowThenUse(x int) { + if x > 0 { + run := 2 + _ = run + } + apply(run) +} + +func closureThenUse() { + g := func(run int) { _ = run } + g(1) + apply(run) +} + +func caseThenUse(x int) { + switch x { + case 1: + run := 3 + _ = run + case 2: + apply(run) + } +} + +func headerThenUse() { + if run := pick(nil); run != nil { + _ = run + } + apply(run) +} + +func rangeThenUse(xs []int) { + for _, run := range xs { + _ = run + } + apply(run) +} + +func useThenShadow(x int) { + apply(run) + if x > 0 { + run := 2 + _ = run + } + apply(run) +} + +func allShadowed(run func(int, int) int) { + apply(run) + if true { + apply(run) + } +} + +func noBinding(x int) { + apply(run) + if x > 0 { + apply(run) + } +} + +func paramValue(run func(int, int) int) { + apply(run) +} + +func localValue() { + run := 1 + _ = run +} + +func main() {} + +var literalThenUse = func(x int) { + if x > 0 { + run := 2 + _ = run + } + apply(run) +} +`; + +/** ` L` for every function_ref ref to `run`, in source order. */ +function runRefs(result: ExtractionResult): string[] { + const byId = new Map(result.nodes.map((n) => [n.id, n])); + return result.unresolvedReferences + .filter((r) => r.referenceKind === 'function_ref' && r.referenceName === 'run') + .map((r) => `${byId.get(r.fromNodeId)?.name} L${r.line}`); +} + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +describe('a Go function value keeps its reference past a shadowing local (#2596)', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['go']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', source: string): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource('main.go', source, 'go'); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + const result = tryKernelExtract('main.go', source, 'go'); + expect(result, 'kernel extraction of main.go').not.toBeNull(); + return result!; + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + + for (const crlf of [false, true]) { + const source = crlf ? MAIN_GO.replace(/\n/g, '\r\n') : MAIN_GO; + const label = crlf ? ' (CRLF)' : ''; + + it.each(backends)(`records a use per scope only where the function binds the name: %s${label}`, (backend) => { + expect(runRefs(extract(backend, source))).toEqual([ + // Binds `run` in a scope of its own: one use per scope. + 'shadowThenUse L12', + 'shadowThenUse L14', + 'closureThenUse L18', + 'closureThenUse L20', + 'caseThenUse L27', + 'caseThenUse L29', + 'headerThenUse L35', + 'headerThenUse L37', + 'rangeThenUse L42', + 'rangeThenUse L44', + // The function body's two uses are one scope: the first speaks for both. + 'useThenShadow L48', + 'useThenShadow L51', + // A parameter of the function itself covers its whole body: one use. + 'allShadowed L57', + // Binds no `run`: one use per function, as before. + 'noBinding L64', + 'paramValue L71', + 'localValue L76', + // A func literal at file scope is its own unit. + 'literalThenUse L84', + 'literalThenUse L86', + ]); + }); + } + + it.runIf(kernelAvailable)('the kernel and the wasm extractor record the same references', () => { + const kernel = extract('kernel', MAIN_GO); + const wasm = extract('wasm', MAIN_GO); + const refs = (r: ExtractionResult) => + r.unresolvedReferences.map((u) => `${u.referenceKind} ${u.referenceName} ${u.line}:${u.column}`); + expect(refs(kernel)).toEqual(refs(wasm)); + }); +}); + +describe('the references a function keeps to the package function (#2596)', () => { + let tempDir: string; + let cg: CodeGraph | null = null; + + beforeAll(async () => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2596-')); + fs.writeFileSync(path.join(tempDir, 'go.mod'), GO_MOD); + fs.writeFileSync(path.join(tempDir, 'main.go'), MAIN_GO); + cg = await CodeGraph.init(tempDir, { index: true }); + cg.resolveReferences(); + }, 120_000); + + afterAll(() => { + cg?.close(); + cg = null; + if (tempDir) fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + /** `L` of each `references` edge from `fromName` to the package `func run`. */ + function refsToRun(fromName: string, kind: 'function' | 'variable' = 'function'): string[] { + const from = cg!.getNodesByKind(kind).find((n) => n.name === fromName); + expect(from, fromName).toBeDefined(); + return cg! + .getOutgoingEdges(from!.id) + .filter((e) => e.kind === 'references') + .filter((e) => { + const target = cg!.getNode(e.target); + return target?.kind === 'function' && target.name === 'run'; + }) + .map((e) => `L${e.line}`) + .sort(); + } + + it.each([ + ['an inner block local', 'shadowThenUse', 'L14'], + ['a func literal parameter', 'closureThenUse', 'L20'], + ['a local in another case clause', 'caseThenUse', 'L29'], + ['an if-header local', 'headerThenUse', 'L37'], + ['a range variable', 'rangeThenUse', 'L44'], + ])('a later use outside %s links to the package function', (_label, fn, line) => { + expect(refsToRun(fn)).toEqual([line]); + }); + + it('a later use in a func literal at file scope links to the package function', () => { + expect(refsToRun('literalThenUse', 'variable')).toEqual(['L86']); + }); + + it('a function whose first use is the package function keeps that one reference', () => { + expect(refsToRun('useThenShadow')).toEqual(['L48']); + }); + + it('a function that binds no local of the name keeps its single reference', () => { + expect(refsToRun('noBinding')).toEqual(['L64']); + }); + + it('no use the local shadows links to the package function', () => { + expect(refsToRun('allShadowed')).toEqual([]); + expect(refsToRun('paramValue')).toEqual([]); + expect(refsToRun('localValue')).toEqual([]); + }); +}); diff --git a/codegraph-kernel/src/go.rs b/codegraph-kernel/src/go.rs index 71d7ba58db..a4557e13ac 100644 --- a/codegraph-kernel/src/go.rs +++ b/codegraph-kernel/src/go.rs @@ -68,12 +68,15 @@ struct ValueScope<'t> { name: String, } -struct Cand { +struct Cand<'t> { from: u32, name: String, line: u32, column_byte: usize, row: usize, + /// The value container the candidate came from: the flush reads its + /// scope from there when the function shadows the name (#2596). + node: Node<'t>, } /// Per-node metadata for the receiver-method owner lookup (mirrors the TS @@ -95,7 +98,7 @@ pub struct Walker<'t> { node_ids: Vec, defined_fn_names: HashSet, imported_names: HashSet, - fn_ref_cands: Vec, + fn_ref_cands: Vec>, fs_values: HashMap, fs_value_counts: HashMap, value_scopes: Vec>, @@ -1187,13 +1190,13 @@ impl<'t> Walker<'t> { } for v in values { - self.normalize_fn_ref_value(v, from, 0); + self.normalize_fn_ref_value(v, from, 0, node); } } /// normalizeValue with GO_SPEC's transparent layers (literal_element, /// expression_list — both fan out to named children). - fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32) { + fn normalize_fn_ref_value(&mut self, v: Node<'t>, from: u32, depth: u32, container: Node<'t>) { stack_guard!(); if depth > 4 { return; @@ -1211,6 +1214,7 @@ impl<'t> Walker<'t> { line: p.row as u32 + 1, column_byte: v.start_byte(), row: p.row, + node: container, }); } // #1820: preserve the receiver of a method value. The name is @@ -1238,12 +1242,13 @@ impl<'t> Walker<'t> { line: p.row as u32 + 1, column_byte: field.start_byte(), row: p.row, + node: container, }); } "literal_element" | "expression_list" => { for i in 0..v.named_child_count() { if let Some(c) = v.named_child(i) { - self.normalize_fn_ref_value(c, from, depth + 1); + self.normalize_fn_ref_value(c, from, depth + 1, container); } } } @@ -1282,17 +1287,24 @@ impl<'t> Walker<'t> { if cands.is_empty() || util::is_generated_file(self.file_path) { return; } - let mut seen: HashSet<(String, String)> = HashSet::new(); - for c in cands { - if !c.name.starts_with("this.") - && !c.name.contains('.') - && !c.name.contains("::") - && !self.defined_fn_names.contains(&c.name) - && !self.imported_names.contains(&c.name) - { - continue; - } - if !seen.insert((self.node_ids[c.from as usize].clone(), c.name.clone())) { + let survivors: Vec> = cands + .into_iter() + .filter(|c| { + c.name.starts_with("this.") + || c.name.contains('.') + || c.name.contains("::") + || self.defined_fn_names.contains(&c.name) + || self.imported_names.contains(&c.name) + }) + .collect(); + // One reference per function and name, at its first use; one per + // scope where the function binds the name itself (#2596). Mirrors + // flushFnRefCandidates / goScopedFnRefKeys. + let scopes = go_scoped_fn_ref_scopes(&survivors, &self.node_ids, self.src); + let mut seen: HashSet<(String, String, Option)> = HashSet::new(); + for (i, c) in survivors.into_iter().enumerate() { + let scope = scopes.get(&i).copied(); + if !seen.insert((self.node_ids[c.from as usize].clone(), c.name.clone(), scope)) { continue; } let column = util::col16(self.src, &self.line_starts, c.row, c.column_byte); @@ -1480,3 +1492,155 @@ fn opt_str(arena: &mut Arena, s: Option<&str>) -> StrRef { None => NONE_STR, } } + +// --- a local that shadows the function a value names (#2596) --------------------- + +/// Go nodes that open a scope a local can be declared in: a block, the +/// implicit block of an `if` / `for` / `switch` / `select` statement, and each +/// clause of a `switch` or `select`. Mirrors GO_SCOPE_TYPES (function-ref.ts). +const GO_SCOPE_TYPES: [&str; 10] = [ + "block", + "if_statement", + "for_statement", + "expression_switch_statement", + "type_switch_statement", + "select_statement", + "expression_case", + "default_case", + "type_case", + "communication_case", +]; + +/// Where a function-value candidate sits: the innermost scope node's id +/// (usize::MAX when there is none) and the unit it is written in: the +/// function or method declaration, or for a func literal at file scope the +/// outermost literal, else None. Mirrors goFnRefSite. +fn go_fn_ref_site<'t>(node: Node<'t>) -> (usize, Option>) { + let mut scope = usize::MAX; + let mut literal: Option> = None; + let mut cur = Some(node); + while let Some(n) = cur { + if scope == usize::MAX && GO_SCOPE_TYPES.contains(&n.kind()) { + scope = n.id(); + } + if matches!(n.kind(), "function_declaration" | "method_declaration") { + return (scope, Some(n)); + } + if n.kind() == "func_literal" { + literal = Some(n); + } + cur = n.parent(); + } + (scope, literal) +} + +/// The names among `names` that a function binds itself, as (unit id, name): +/// its func literals' parameters, `:=`, `var` and `const`, range and +/// type-switch variables, and a select case's received value. A name the +/// function's own receiver, parameters or results bind is left out: those +/// cover its whole body. Mirrors goLocalBindings. +fn go_local_bindings<'t>(units: &[Node<'t>], names: &HashSet<&str>, src: &str) -> HashSet<(usize, String)> { + let mut out: HashSet<(usize, String)> = HashSet::new(); + for unit in units { + let mut bind = |id: Option>| { + if let Some(id) = id { + if id.kind() == "identifier" { + let name = &src[id.start_byte()..id.end_byte()]; + if names.contains(name) { + out.insert((unit.id(), name.to_string())); + } + } + } + }; + let declares = |n: Node<'t>| { + (0..n.child_count()).any(|i| n.child(i).map_or(false, |c| !c.is_named() && c.kind() == ":=")) + }; + let mut stack: Vec> = vec![*unit]; + while let Some(n) = stack.pop() { + let list = match n.kind() { + "short_var_declaration" => n.child_by_field_name("left"), + "range_clause" | "receive_statement" if declares(n) => n.child_by_field_name("left"), + "type_switch_statement" => n.child_by_field_name("alias"), + _ => None, + }; + if let Some(list) = list { + for i in 0..list.named_child_count() { + bind(list.named_child(i)); + } + } + if matches!(n.kind(), "var_spec" | "const_spec" | "parameter_declaration" | "variadic_parameter_declaration") { + let mut cursor = n.walk(); + for c in n.children_by_field_name("name", &mut cursor) { + bind(Some(c)); + } + } + for i in (0..n.named_child_count()).rev() { + if let Some(c) = n.named_child(i) { + stack.push(c); + } + } + } + for field in ["receiver", "parameters", "result"] { + let Some(list) = unit.child_by_field_name(field) else { continue }; + if list.kind() != "parameter_list" { + continue; + } + for i in 0..list.named_child_count() { + let Some(p) = list.named_child(i) else { continue }; + if !matches!(p.kind(), "parameter_declaration" | "variadic_parameter_declaration") { + continue; + } + let mut cursor = p.walk(); + for c in p.children_by_field_name("name", &mut cursor) { + if c.kind() == "identifier" { + out.remove(&(unit.id(), src[c.start_byte()..c.end_byte()].to_string())); + } + } + } + } + } + out +} + +/// The candidates, by index, that get one reference per scope instead of +/// one per function, each with the id of the scope it sits in: names a +/// function uses at least twice and binds itself. Only those candidates' +/// positions are read from the tree. Mirrors goScopedFnRefScopes. +fn go_scoped_fn_ref_scopes<'t>(cands: &[Cand<'t>], node_ids: &[String], src: &str) -> HashMap { + let mut groups: HashMap<(&str, &str), Vec> = HashMap::new(); + for (i, c) in cands.iter().enumerate() { + if c.name.contains('.') { + continue; + } + groups.entry((node_ids[c.from as usize].as_str(), c.name.as_str())).or_default().push(i); + } + let mut sites: Vec<(usize, usize, Node<'t>)> = Vec::new(); + let mut units: Vec> = Vec::new(); + let mut unit_ids: HashSet = HashSet::new(); + let mut names: HashSet<&str> = HashSet::new(); + for members in groups.values() { + if members.len() < 2 { + continue; + } + for &i in members { + let (scope, unit) = go_fn_ref_site(cands[i].node); + let Some(unit) = unit else { continue }; + sites.push((i, scope, unit)); + if unit_ids.insert(unit.id()) { + units.push(unit); + } + names.insert(cands[i].name.as_str()); + } + } + let mut out: HashMap = HashMap::new(); + if units.is_empty() { + return out; + } + let bound = go_local_bindings(&units, &names, src); + for (i, scope, unit) in sites { + if bound.contains(&(unit.id(), cands[i].name.clone())) { + out.insert(i, scope); + } + } + out +} diff --git a/src/extraction/function-ref.ts b/src/extraction/function-ref.ts index 46d540c260..798b158d53 100644 --- a/src/extraction/function-ref.ts +++ b/src/extraction/function-ref.ts @@ -869,3 +869,163 @@ function rubyEnclosingCall(node: SyntaxNode): SyntaxNode | null { } return null; } + +// --------------------------------------------------------------------------- +// Go: a local that shadows the function a value names (#2596) +// --------------------------------------------------------------------------- + +/** + * Go nodes that open a scope a local can be declared in: a block, the implicit + * block of an `if` / `for` / `switch` / `select` statement, and each clause of + * a `switch` or `select`. + */ +const GO_SCOPE_TYPES = new Set([ + 'block', + 'if_statement', + 'for_statement', + 'expression_switch_statement', + 'type_switch_statement', + 'select_statement', + 'expression_case', + 'default_case', + 'type_case', + 'communication_case', +]); + +/** + * Where a Go function-value candidate sits: the innermost scope node around + * it, and the unit it is written in: the function or method declaration, or + * for a func literal at file scope (`var h = func(…) {…}`) the outermost + * literal, else null. Mirrored in the kernel (go.rs `go_fn_ref_site`). + */ +export interface GoFnRefSite { + scope: number; + unit: SyntaxNode | null; +} + +/** {@link GoFnRefSite} of the value container `node`. */ +export function goFnRefSite(node: SyntaxNode): GoFnRefSite { + let scope = -1; + let literal: SyntaxNode | null = null; + for (let cur: SyntaxNode | null = node; cur; cur = cur.parent) { + if (scope < 0 && GO_SCOPE_TYPES.has(cur.type)) scope = cur.id; + if (cur.type === 'function_declaration' || cur.type === 'method_declaration') return { scope, unit: cur }; + if (cur.type === 'func_literal') literal = cur; + } + return { scope, unit: literal }; +} + +/** + * The names among `names` that a Go function binds itself, keyed + * `|`: its func literals' parameters, `:=`, `var` and `const`, + * range and type-switch variables, and a select case's received value. Such a + * local shadows a package function of the same name for the rest of its + * scope, which the resolver reads per use (`isGoLocallyBound`). A name the + * function's own receiver, parameters or results bind is left out: those + * cover its whole body, so every use is the local and one reference is + * enough. Mirrored in the kernel (go.rs `go_local_bindings`). + */ +export function goLocalBindings( + units: Iterable, + names: ReadonlySet, + source: string +): Set { + const out = new Set(); + for (const unit of units) { + const bind = (id: SyntaxNode | null | undefined): void => { + if (id?.type !== 'identifier') return; + const name = getNodeText(id, source); + if (names.has(name)) out.add(`${unit.id}|${name}`); + }; + const declares = (n: SyntaxNode): boolean => { + for (let i = 0; i < n.childCount; i++) { + const c = n.child(i); + if (c && !c.isNamed && c.type === ':=') return true; + } + return false; + }; + const stack: SyntaxNode[] = [unit]; + while (stack.length > 0) { + const n = stack.pop()!; + switch (n.type) { + case 'short_var_declaration': + for (const c of getChildByField(n, 'left')?.namedChildren ?? []) bind(c); + break; + case 'range_clause': + case 'receive_statement': + if (declares(n)) for (const c of getChildByField(n, 'left')?.namedChildren ?? []) bind(c); + break; + case 'type_switch_statement': + for (const c of getChildByField(n, 'alias')?.namedChildren ?? []) bind(c); + break; + case 'var_spec': + case 'const_spec': + case 'parameter_declaration': + case 'variadic_parameter_declaration': + for (const c of n.childrenForFieldName('name')) bind(c); + break; + } + for (let i = n.namedChildCount - 1; i >= 0; i--) { + const c = n.namedChild(i); + if (c) stack.push(c); + } + } + for (const field of ['receiver', 'parameters', 'result']) { + const list = getChildByField(unit, field); + if (list?.type !== 'parameter_list') continue; + for (const p of list.namedChildren) { + if (p.type !== 'parameter_declaration' && p.type !== 'variadic_parameter_declaration') continue; + for (const c of p.childrenForFieldName('name')) { + if (c.type === 'identifier') out.delete(`${unit.id}|${getNodeText(c, source)}`); + } + } + } + } + return out; +} + +/** + * The Go function-value candidates that need one reference per scope instead + * of one per function (#2596), each with the id of the scope it sits in. A + * function records a value-use of a name once, at its first use; but when the + * function also binds that name itself, the first use can be the local while + * a later one, in another scope, is the package function. The resolver drops + * a use the local shadows, so the later use needs a reference of its own. One + * per scope node is enough: within one scope a local declared before the + * first use still covers every later use. Only a name a function uses at + * least twice is looked up, so a candidate's position is read from the tree + * just for those. Mirrored in the kernel (go.rs `go_scoped_fn_ref_scopes`). + */ +export function goScopedFnRefScopes( + candidates: ReadonlyArray, + source: string +): Map { + const groups = new Map(); + for (const c of candidates) { + if (!c.goNode || c.name.includes('.')) continue; + const key = `${c.fromNodeId}|${c.name}`; + const group = groups.get(key); + if (group) group.push(c); + else groups.set(key, [c]); + } + const sites = new Map(); + const units = new Map(); + const names = new Set(); + for (const group of groups.values()) { + if (group.length < 2) continue; + for (const c of group) { + const site = goFnRefSite(c.goNode!); + if (!site.unit) continue; + sites.set(c, site); + units.set(site.unit.id, site.unit); + names.add(c.name); + } + } + const out = new Map(); + if (units.size === 0) return out; + const bound = goLocalBindings(units.values(), names, source); + for (const [c, site] of sites) { + if (bound.has(`${site.unit!.id}|${c.name}`)) out.set(c, site.scope); + } + return out; +} diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 2cffc3a495..2d84db37e6 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -17,7 +17,7 @@ import { } from '../types'; import { getParser, detectLanguage, isLanguageSupported, isFileLevelOnlyLanguage } from './grammars'; import { NodeIdAllocator, getNodeText, getChildByField, getPrecedingDocstring, BUILTIN_TYPE_NAMES } from './tree-sitter-helpers'; -import { FN_REF_SPECS, captureFnRefCandidates, type FnRefSpec, type FnRefCandidate } from './function-ref'; +import { FN_REF_SPECS, captureFnRefCandidates, goScopedFnRefScopes, type FnRefSpec, type FnRefCandidate } from './function-ref'; import { isGeneratedFile, isMinifiedContent } from './generated-detection'; import type { LanguageExtractor, ExtractorContext } from './tree-sitter-types'; import { EXTRACTORS } from './languages'; @@ -732,7 +732,9 @@ export class TreeSitterExtractor { // during the walk, gated & flushed into unresolvedReferences at end-of-file // (see flushFnRefCandidates). private fnRefSpec: FnRefSpec | undefined; - private fnRefCandidates: Array = []; + // A Go candidate also keeps the value container it came from: the flush + // reads its scope from there when the function shadows the name (#2596). + private fnRefCandidates: Array = []; // Memoized "is this a Vue store file" verdict (per-extractor = per-file). private vueStoreFile: boolean | null = null; // An object literal hung on a path (`window.App = {…}`, `ns.mod = {…}`) is @@ -928,8 +930,9 @@ export class TreeSitterExtractor { if (!rule || this.nodeStack.length === 0) return; const fromNodeId = this.nodeStack[this.nodeStack.length - 1]; if (!fromNodeId) return; + const goNode = this.language === 'go' ? node : undefined; for (const cand of captureFnRefCandidates(node, rule, spec, this.source)) { - this.fnRefCandidates.push({ ...cand, fromNodeId }); + this.fnRefCandidates.push({ ...cand, fromNodeId, goNode }); } } @@ -1024,7 +1027,7 @@ export class TreeSitterExtractor { const ungated = this.fnRefSpec?.ungatedModes; const addressOfOnly = this.fnRefSpec?.addressOfOnly === true; - const seen = new Set(); + const survivors: typeof candidates = []; for (const c of candidates) { const atFileScope = c.fromNodeId.startsWith('file:'); // C++ (addressOfOnly): a BARE identifier qualifies only inside a @@ -1062,7 +1065,18 @@ export class TreeSitterExtractor { continue; } } - const key = `${c.fromNodeId}|${c.name}`; + survivors.push(c); + } + + // One reference per function and name, at its first use. In Go, a + // function that binds the name itself gets one per scope instead (#2596): + // its first use may be the local, which resolution drops, while a later + // use in another scope is the package function. + const goScopes = this.language === 'go' ? goScopedFnRefScopes(survivors, this.source) : null; + const seen = new Set(); + for (const c of survivors) { + const scope = goScopes?.get(c); + const key = scope === undefined ? `${c.fromNodeId}|${c.name}` : `${c.fromNodeId}|${c.name}|${scope}`; if (seen.has(key)) continue; seen.add(key); this.unresolvedReferences.push({ From 57ce5b9cdc7d32d50113f3ac9802ddfa3f2d6802 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 03:11:17 +0000 Subject: [PATCH 16/95] fix(upgrade): a global npm install is upgraded with -g from any folder, even home (#2777) codegraph upgrade told a global npm install from a local one by whether the running file sat under the current folder. npm's global folder is often inside the home folder (Windows %APPDATA%\npm, nvm, fnm, ~/.npm-global), so run from home every such global install was taken for a local one: npm ran without -g, the global copy stayed on the old version and a second copy landed in the home folder. The same test sent a project's own install to npm -g when upgrade ran from one of its subfolders or from outside it, and installed into the parent folder instead when it ran from one. The scope now comes from npm's layout around the outermost node_modules. A global launcher beside it (\codegraph.cmd on Windows, /bin/codegraph next to /lib on POSIX) means global. A project around it (node_modules/.bin/codegraph or a package.json) means local, and npm then runs in that project, wherever upgrade was run from; the version npm left there is read back, with a warning when it did not change. A pnpm, yarn or bun install (a .pnpm layout, or that manager's lockfile beside node_modules) is named and given that manager's command, since npm run in its folder would only add a second copy. Validation: on Windows, the real CLI was run with a private USERPROFILE, HOME, APPDATA and LOCALAPPDATA (npm config get prefix = the private AppData\Roaming\npm) against installs laid out the way npm lays them out, with a fake npm first on PATH that logs its command and folder. main ran the wrong npm command in 6 of 9 cases (a global install under %APPDATA%\npm or ~\.npm-global run from home, and project installs run from a subfolder, from home, from the drive root or from another project), the fix ran the right one in 9 of 9; an npx cache run and the bundled runtime are detected the same by both. On main's code 14 of the 91 upgrade tests fail on assertions (12 new ones, and 2 existing ones that now also expect the project folder); all pass with the fix. Fixes #2477 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/upgrade.test.ts | 188 ++++++++++++++++++++++++++++++++++++-- src/upgrade/index.ts | 142 +++++++++++++++++++++++++--- 3 files changed, 309 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 248f157d17..c5feb1e8f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) - `codegraph install` and `upgrade` no longer break Codex's `config.toml` when CodeGraph's `[mcp_servers.codegraph]` entry is written differently from how CodeGraph writes it, such as indented, with spaces or quotes inside the brackets, or with Windows line endings: the entry is now updated in place, where before a second copy was added and Codex could load none of the file. A file already broken that way is repaired on the next install or upgrade, an example of the entry inside a multiline setting such as `developer_instructions` is no longer mistaken for it, uninstall removes the entry however it is written, and an entry you wrote with dotted keys or as an inline table is left as you wrote it. When your `mcp_servers` settings are one inline table, CodeGraph now says it can't add itself instead of breaking the file. Thanks @dencoseca for the report, and @danusha2345 and @rudycelekli for the first fixes. (#2562) - For Hermes Agent, `codegraph install` and `upgrade` no longer add `mcp-codegraph` to `platform_toolsets.cli`. Hermes up to 0.22.2 reported that name as an unknown toolset on every start, and Hermes 0.22.4 reads it as the only MCP server to load, hiding your others from CLI sessions; an entry an earlier install added is taken out again. A refresh also no longer resets your own `mcp_servers.codegraph` settings, such as an absolute command path, timeouts or `enabled: false`, and a config indented or written differently from Hermes's default no longer gets a duplicate or unreadable entry. Thanks @Huy3ko for the report and @conifercone for confirming it. (#2485) +- `codegraph upgrade` now upgrades a global npm install when run from your home folder, which holds npm's global folder on Windows and with tools like nvm: before, it ran `npm install` without `-g` there, which left the global copy on the old version and put a second copy in your home folder. A copy installed in a project is now upgraded in that project wherever you run the command, with a warning if npm left it on the old version, and a copy installed with pnpm, yarn or bun gets the command that upgrades it instead of npm. Thanks @timlincool for the report and @maxmilian for the first fix. (#2477) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) - In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) diff --git a/__tests__/upgrade.test.ts b/__tests__/upgrade.test.ts index 403271b7fc..3c9b698602 100644 --- a/__tests__/upgrade.test.ts +++ b/__tests__/upgrade.test.ts @@ -85,8 +85,8 @@ describe('detectInstallMethod', () => { it('detects a local (project) npm install as local', () => { const cwd = '/home/u/project'; const filename = `${cwd}/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`; - const m = detectInstallMethod({ filename, platform: 'linux', cwd, exists: () => false }); - expect(m).toEqual({ kind: 'npm', scope: 'local' }); + const m = detectInstallMethod({ filename, platform: 'linux', cwd, exists: bundleExists(new Set([`${cwd}/package.json`])) }); + expect(m).toEqual({ kind: 'npm', scope: 'local', root: cwd }); }); it('detects an npx run from the _npx cache', () => { @@ -117,9 +117,9 @@ describe('detectInstallMethod', () => { const cwd = '/home/u/project'; const root = `${cwd}/node_modules/@colbymchenry/codegraph/node_modules/@colbymchenry/codegraph-darwin-arm64`; const filename = `${root}/lib/dist/bin/codegraph.js`; - const present = new Set([`${root}/node`, `${root}/bin/codegraph`]); + const present = new Set([`${root}/node`, `${root}/bin/codegraph`, `${cwd}/node_modules/.bin/codegraph`]); const m = detectInstallMethod({ filename, platform: 'darwin', cwd, exists: bundleExists(present) }); - expect(m).toEqual({ kind: 'npm', scope: 'local' }); + expect(m).toEqual({ kind: 'npm', scope: 'local', root: cwd }); }); it('still detects an npx run when the cached platform package has the bundle layout', () => { @@ -152,6 +152,99 @@ describe('detectInstallMethod', () => { }); expect(m.kind).toBe('unknown'); }); + + // #2477: the scope came from "is the running file under cwd", so a global + // prefix inside the home folder (Windows %APPDATA%\npm, nvm, ~/.npm-global) + // was taken for a local install whenever upgrade ran from home — npm then + // ran without -g and the global copy stayed old. It is now read from npm's + // layout: a global launcher beside the tree, or a project around it. + describe('npm scope comes from npm\'s layout, never from cwd (#2477)', () => { + const slash = (p: string | undefined) => p?.replace(/\\/g, '/'); + const winHome = 'C:/Users/X'; + const winPrefix = `${winHome}/AppData/Roaming/npm`; + const winThin = `${winPrefix}/node_modules/@colbymchenry/codegraph/node_modules/@colbymchenry/codegraph-win32-x64/lib/dist/bin/codegraph.js`; + + it.each([winHome, 'C:/Users', 'C:/', 'C:/Users/X/projects/app'])('a Windows global install under %%APPDATA%%\\npm is global with cwd %s', (cwd) => { + const present = new Set([`${winPrefix}/codegraph.cmd`, `${winHome}/package.json`]); + for (const filename of [winThin, `${winPrefix}/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`]) { + expect(detectInstallMethod({ filename, platform: 'win32', cwd, exists: bundleExists(present) })).toEqual({ kind: 'npm', scope: 'global' }); + } + }); + + it.each(['/home/u', '/', '/home'])('a POSIX global install under nvm in the home folder is global with cwd %s', (cwd) => { + const prefix = '/home/u/.nvm/versions/node/v22.11.0'; + const filename = `${prefix}/lib/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`; + const present = new Set([`${prefix}/bin/codegraph`, '/home/u/package.json']); + expect(detectInstallMethod({ filename, platform: 'linux', cwd, exists: bundleExists(present) })).toEqual({ kind: 'npm', scope: 'global' }); + }); + + it('a global install under a custom prefix is global from home (~/.npm-global, D:\\tools\\npm)', () => { + const posix = '/home/u/.npm-global'; + expect(detectInstallMethod({ + filename: `${posix}/lib/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`, + platform: 'darwin', cwd: '/home/u', exists: bundleExists(new Set([`${posix}/bin/codegraph`])), + })).toEqual({ kind: 'npm', scope: 'global' }); + expect(detectInstallMethod({ + filename: 'D:/tools/npm/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js', + platform: 'win32', cwd: 'D:/', exists: bundleExists(new Set(['D:/tools/npm/codegraph.cmd'])), + })).toEqual({ kind: 'npm', scope: 'global' }); + }); + + it('a local install whose project root is the home folder stays local and belongs to the home folder, from any cwd', () => { + const filename = `${winHome}/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`; + const present = new Set([`${winHome}/package.json`, `${winHome}/node_modules/.bin/codegraph.cmd`, `${winHome}/projects/app/package.json`]); + // ~/projects/app is a project of its own: npm run there would install into it. + for (const cwd of [winHome, 'C:/', `${winHome}/projects/app`]) { + const m = detectInstallMethod({ filename, platform: 'win32', cwd, exists: bundleExists(present) }) as Extract; + expect([m.scope, slash(m.root)]).toEqual(['local', winHome]); + } + }); + + it('a local install in a project belongs to that project, from any cwd', () => { + const project = '/home/u/projects/app'; + const filename = `${project}/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`; + const present = new Set([`${project}/package.json`, `${project}/node_modules/.bin/codegraph`]); + for (const cwd of [project, `${project}/src`, `${project}/packages/web`, '/home/u', '/', '/home/u/projects/application']) { + expect(detectInstallMethod({ filename, platform: 'linux', cwd, exists: bundleExists(present) })).toEqual({ kind: 'npm', scope: 'local', root: project }); + } + }); + + it('npx and bundled installs under the home folder are unaffected by a home cwd', () => { + expect(detectInstallMethod({ + filename: `${winHome}/AppData/Local/npm-cache/_npx/1a2b3c/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`, + platform: 'win32', cwd: winHome, exists: () => false, + })).toEqual({ kind: 'npx' }); + const winBundle = `${winHome}/AppData/Local/codegraph/current`; + const win = detectInstallMethod({ + filename: `${winBundle}/lib/dist/bin/codegraph.js`, platform: 'win32', cwd: winHome, + exists: bundleExists(new Set([`${winBundle}/node.exe`, `${winBundle}/bin/codegraph.cmd`])), + }); + expect(win.kind).toBe('bundle'); + const unixBundle = '/home/u/.codegraph/versions/v1.6.2'; + expect(detectInstallMethod({ + filename: `${unixBundle}/lib/dist/bin/codegraph.js`, platform: 'linux', cwd: '/home/u', + exists: bundleExists(new Set([`${unixBundle}/node`, `${unixBundle}/bin/codegraph`, '/home/u/.codegraph'])), + })).toEqual({ kind: 'bundle', os: 'unix', bundleRoot: unixBundle, installDir: '/home/u/.codegraph' }); + }); + + it('a pnpm, yarn or bun install is named as such, so npm never runs in its folder', () => { + const pnpm = '/home/u/.local/share/pnpm/global/5'; + expect(detectInstallMethod({ + filename: `${pnpm}/node_modules/.pnpm/@colbymchenry+codegraph@1.6.2/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`, + platform: 'linux', cwd: '/home/u', exists: bundleExists(new Set([`${pnpm}/package.json`])), + })).toEqual({ kind: 'package-manager', manager: 'pnpm', global: true }); + const yarn = '/home/u/.config/yarn/global'; + expect(detectInstallMethod({ + filename: `${yarn}/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`, + platform: 'linux', cwd: '/home/u', exists: bundleExists(new Set([`${yarn}/package.json`, `${yarn}/yarn.lock`])), + })).toEqual({ kind: 'package-manager', manager: 'yarn', global: true }); + const project = '/home/u/app'; + expect(detectInstallMethod({ + filename: `${project}/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`, + platform: 'linux', cwd: project, exists: bundleExists(new Set([`${project}/package.json`, `${project}/bun.lock`])), + })).toEqual({ kind: 'package-manager', manager: 'bun', global: false }); + }); + }); }); describe('deriveInstallDir', () => { @@ -338,7 +431,7 @@ describe('windows bundle swap script (#2185)', () => { // --------------------------------------------------------------------------- interface Calls { - runs: Array<{ cmd: string; args: string[]; env?: NodeJS.ProcessEnv }>; + runs: Array<{ cmd: string; args: string[]; env?: NodeJS.ProcessEnv; cwd?: string }>; captures: Array<{ cmd: string; args: string[] }>; logs: string[]; errors: string[]; @@ -355,8 +448,8 @@ function makeDeps( currentVersion: overrides.currentVersion, method: overrides.method, resolveLatest: overrides.resolveLatest ?? (async () => 'v0.9.9'), - run: (cmd, args, env) => { - calls.runs.push({ cmd, args, env }); + run: (cmd, args, env, cwd) => { + calls.runs.push({ cmd, args, env, cwd }); return runExit; }, // Default probe: spawn fails → 'inconclusive'. Tests that exercise the @@ -559,6 +652,87 @@ describe('runUpgrade', () => { }); }); +describe('runUpgrade picks npm\'s scope and folder from the install, not from cwd (#2477)', () => { + let tmp: string; + beforeEach(() => { + tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-upgrade-2477-')); + }); + afterEach(() => { + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + const npmRun = (calls: Calls) => calls.runs.find((r) => r.args.join(' ').includes('install')); + + it('a Windows global install run from the home folder upgrades with -g (the report)', async () => { + const home = 'C:/Users/X'; + const prefix = `${home}/AppData/Roaming/npm`; + const method = detectInstallMethod({ + filename: `${prefix}/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js`, + platform: 'win32', + cwd: home, + exists: (p) => p.replace(/\\/g, '/') === `${prefix}/codegraph.cmd`, + }); + const { deps, calls } = makeDeps({ method, currentVersion: '0.9.8', platform: 'win32' }); + expect(await runUpgrade({}, deps)).toBe(0); + const run = npmRun(calls)!; + expect(run.args).toEqual(['/d', '/s', '/c', `npm install -g ${NPM_PACKAGE}@latest`]); + expect(run.cwd).toBeUndefined(); + }); + + it('a local install is upgraded in its own project, and the version npm left there is checked', async () => { + const project = path.join(tmp, 'app'); + const manifest = path.join(project, 'node_modules', '@colbymchenry', 'codegraph', 'package.json'); + fs.mkdirSync(path.dirname(manifest), { recursive: true }); + fs.writeFileSync(path.join(project, 'package.json'), '{}\n'); + + const method: InstallMethod = { kind: 'npm', scope: 'local', root: project }; + // npm left the project on the old version: say so, don't just claim success. + fs.writeFileSync(manifest, JSON.stringify({ version: '0.9.8' })); + const stale = makeDeps({ method, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, stale.deps)).toBe(0); + expect(npmRun(stale.calls)!).toMatchObject({ cmd: 'npm', args: ['install', `${NPM_PACKAGE}@latest`], cwd: project }); + expect(stale.calls.logs.join('\n')).toContain(`Running: npm install ${NPM_PACKAGE}@latest (in ${project})`); + expect(stale.calls.logs.join('\n')).toContain(`npm finished, but ${project} still has CodeGraph v0.9.8.`); + + fs.writeFileSync(manifest, JSON.stringify({ version: '0.9.9' })); + const done = makeDeps({ method, currentVersion: '0.9.8' }); + await runUpgrade({}, done.deps); + expect(done.calls.logs.join('\n')).toContain(`✓ ${project} now has CodeGraph v0.9.9.`); + }); + + it('a local install in a home folder that is a project is upgraded there, even when run from another project inside it', async () => { + const home = path.join(tmp, 'home'); + const other = path.join(home, 'projects', 'app'); + fs.mkdirSync(path.join(home, 'node_modules', '.bin'), { recursive: true }); + fs.mkdirSync(other, { recursive: true }); + fs.writeFileSync(path.join(home, 'package.json'), '{}\n'); + fs.writeFileSync(path.join(home, 'node_modules', '.bin', 'codegraph'), ''); + fs.writeFileSync(path.join(other, 'package.json'), '{}\n'); + const method = detectInstallMethod({ + filename: path.join(home, 'node_modules', '@colbymchenry', 'codegraph', 'dist', 'bin', 'codegraph.js'), + platform: process.platform, + cwd: other, + }); + const { deps, calls } = makeDeps({ method, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, deps)).toBe(0); + expect(npmRun(calls)!.cwd).toBe(home); + }); + + it('a pnpm, yarn or bun install is not touched with npm: the user gets the command for their manager', async () => { + for (const [manager, global, command] of [ + ['pnpm', true, `pnpm add -g ${NPM_PACKAGE}@latest`], + ['yarn', true, `yarn global add ${NPM_PACKAGE}@latest`], + ['bun', false, `bun add ${NPM_PACKAGE}@latest`], + ] as const) { + const { deps, calls } = makeDeps({ method: { kind: 'package-manager', manager, global }, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, deps)).toBe(1); + expect(calls.runs).toHaveLength(0); + expect(calls.logs.join('\n')).toContain(`installed with ${manager}, so npm can't upgrade it`); + expect(calls.logs.join('\n')).toContain(command); + } + }); +}); + // --------------------------------------------------------------------------- // Beta signup offer — fires ONLY after a real, successful binary update. // (The hook itself gates on TTY + the once-per-machine stored choice; see diff --git a/src/upgrade/index.ts b/src/upgrade/index.ts index 0e088e9982..e5384f973d 100644 --- a/src/upgrade/index.ts +++ b/src/upgrade/index.ts @@ -8,8 +8,9 @@ * canonical installer script (single source of truth) so the download / * version-resolution / PATH logic never drifts between first-install and * upgrade. - * - **npm** — installed via `npm i -g @colbymchenry/codegraph`. Upgrading - * shells out to npm. + * - **npm** — installed via `npm i -g @colbymchenry/codegraph`, or into a + * project. Upgrading shells out to npm (for a project, in that project). + * A pnpm, yarn or bun install is named, with that manager's command. * - **npx** — ephemeral; nothing to upgrade (next `npx` fetches latest). * - **source** — a git checkout running its own `dist/`; `git pull` + rebuild. * @@ -43,8 +44,15 @@ export const INSTALL_PS1_URL = `${RAW_BASE}/install.ps1`; export type InstallMethod = | { kind: 'bundle'; os: 'unix' | 'windows'; bundleRoot: string; installDir: string | null } - | { kind: 'npm'; scope: 'global' | 'local' } + | { + kind: 'npm'; + scope: 'global' | 'local'; + /** Local only: the project folder whose node_modules holds this install, where npm runs. */ + root?: string; + } | { kind: 'npx' } + /** Installed by another package manager, which has to upgrade it (npm can't). */ + | { kind: 'package-manager'; manager: 'pnpm' | 'yarn' | 'bun'; global: boolean } | { kind: 'source'; root: string } | { kind: 'unknown'; reason: string }; @@ -52,7 +60,11 @@ export interface DetectInput { /** `__filename` of the running CLI module — `<…>/dist/bin/codegraph.js`. */ filename: string; platform: NodeJS.Platform; - cwd: string; + /** + * The folder `upgrade` runs from. Detection doesn't read it: where an + * install lives says how it was installed, from any folder (#2477). + */ + cwd?: string; /** Injectable existence probe (defaults to fs.existsSync) — for tests. */ exists?: (p: string) => boolean; } @@ -120,10 +132,31 @@ export function detectInstallMethod(input: DetectInput): InstallMethod { return { kind: 'npx' }; } - // npm install (global or local): lives under a node_modules tree. - if (norm.includes('/node_modules/')) { - const underCwd = norm.startsWith(toPosix(P.resolve(input.cwd)) + '/'); - return { kind: 'npm', scope: underCwd ? 'local' : 'global' }; + // npm install (global or local): lives under a node_modules tree. Which of + // the two is read from npm's own layout, never from cwd (#2477). A global + // prefix often sits inside the home folder — Windows `%APPDATA%\npm`, nvm, + // fnm, `~/.npm-global` — so "the running file is under cwd" took every such + // global install for a local one whenever `upgrade` ran from home: `npm + // install` without -g left the global copy stale and dropped a stray local + // one into the home folder. + const nodeModulesAt = norm.indexOf('/node_modules/'); + if (nodeModulesAt !== -1) { + // The folder holding the outermost node_modules (the same offset in + // `filename`, which `toPosix` only changes slash by slash): the global + // prefix on Windows, `/lib` on POSIX, or the project root. + let owner = input.filename.slice(0, nodeModulesAt); + if (owner === '' || /^[A-Za-z]:$/.test(owner)) owner += P.sep; // a tree at a filesystem/drive root + // pnpm, yarn and bun keep their global packages in a folder that looks + // like an npm project (a package.json beside node_modules); running npm + // there, or a second copy through npm, is never the upgrade. + const manager = otherPackageManager(norm, owner, P, exists); + if (manager) return manager; + if (isNpmGlobalTree(owner, P, isWin, exists)) return { kind: 'npm', scope: 'global' }; + // A project's install is upgraded in that project, whatever folder + // `upgrade` runs from: npm run anywhere else — outside the project, or in + // another project inside it, as in a home folder that is itself one — + // installs a second copy there and leaves this one as it was. + return { kind: 'npm', scope: 'local', root: P.resolve(owner) }; } // Bundle: /lib/dist/bin/codegraph.js → is up 3 from bin/. @@ -145,6 +178,50 @@ export function detectInstallMethod(input: DetectInput): InstallMethod { return { kind: 'unknown', reason: `unrecognized install layout at ${input.filename}` }; } +/** A pnpm, yarn or bun install, from its `.pnpm` layout or its lockfile beside node_modules. */ +function otherPackageManager( + norm: string, + owner: string, + P: path.PlatformPath, + exists: (p: string) => boolean, +): Extract | null { + const where = toPosix(owner).toLowerCase(); + if (norm.includes('/node_modules/.pnpm/') || exists(P.join(owner, 'pnpm-lock.yaml'))) { + return { kind: 'package-manager', manager: 'pnpm', global: /\/pnpm\/global\//.test(where + '/') }; + } + if (exists(P.join(owner, 'yarn.lock'))) { + return { kind: 'package-manager', manager: 'yarn', global: /\/yarn\/(?:data\/)?global$/.test(where) }; + } + if (exists(P.join(owner, 'bun.lockb')) || exists(P.join(owner, 'bun.lock'))) { + return { kind: 'package-manager', manager: 'bun', global: /\/\.bun\/install\/global$/.test(where) }; + } + return null; +} + +/** + * Whether the node_modules tree in `owner` is npm's global one, read from + * where npm put the launcher: beside a global tree — `\codegraph.cmd` + * on Windows, `/bin/codegraph` next to `/lib/node_modules` on + * POSIX — and in `node_modules/.bin` of a project folder (one with a + * package.json) for a local install. A tree with neither is taken for + * global, since no project owns it. + */ +function isNpmGlobalTree( + owner: string, + P: path.PlatformPath, + isWin: boolean, + exists: (p: string) => boolean, +): boolean { + const globalLauncher = isWin + ? exists(P.join(owner, 'codegraph.cmd')) + : P.basename(owner) === 'lib' && exists(P.join(P.dirname(owner), 'bin', 'codegraph')); + if (globalLauncher) return true; + const bin = P.join(owner, 'node_modules', '.bin'); + const local = exists(P.join(bin, 'codegraph')) || exists(P.join(bin, 'codegraph.cmd')) + || exists(P.join(owner, 'package.json')); + return !local; +} + // --------------------------------------------------------------------------- // Version helpers (pure) // --------------------------------------------------------------------------- @@ -291,7 +368,7 @@ export interface UpgradeDeps { method: InstallMethod; resolveLatest: (pin?: string) => Promise; /** Run a command inheriting stdio; returns its exit code (-1 = spawn failed). */ - run: (cmd: string, args: string[], env?: NodeJS.ProcessEnv) => number; + run: (cmd: string, args: string[], env?: NodeJS.ProcessEnv, cwd?: string) => number; /** Run a command capturing stdout (nothing reaches the terminal); null = spawn failed. */ capture: (cmd: string, args: string[]) => { code: number; stdout: string } | null; hasCommand: (cmd: string) => boolean; @@ -393,6 +470,17 @@ export async function runUpgrade(opts: UpgradeOptions, deps: UpgradeDeps): Promi deps.log(c.green('npx always runs the latest version on demand — nothing to upgrade.')); deps.log(c.dim(`Force a fresh fetch with: npx ${NPM_PACKAGE}@latest`)); return 0; + case 'package-manager': { + const pkg = `${NPM_PACKAGE}@${opts.version ? stripV(latest) : 'latest'}`; + const command = { + pnpm: method.global ? `pnpm add -g ${pkg}` : `pnpm add ${pkg}`, + yarn: method.global ? `yarn global add ${pkg}` : `yarn add ${pkg}`, + bun: method.global ? `bun add -g ${pkg}` : `bun add ${pkg}`, + }[method.manager]; + deps.warn(`This CodeGraph was installed with ${method.manager}, so npm can't upgrade it.`); + deps.log(c.dim(`Upgrade it with: ${command}`)); + return 1; + } case 'source': deps.warn(`Running from a source checkout at ${method.root}.`); deps.log(c.dim('Upgrade it with: git pull && npm run build')); @@ -485,8 +573,18 @@ export function verifyResolvedVersion(latest: string, deps: UpgradeDeps): Versio function reportResolvedVersion(latest: string, deps: UpgradeDeps): VersionProbe { const { method } = deps; // A project-local npm install isn't served by PATH's `codegraph` (that - // would be some other install) — a probe could only false-alarm. - if (method.kind === 'npm' && method.scope === 'local') return 'inconclusive'; + // would be some other install) — a probe could only false-alarm. Read the + // version npm left in the project instead, so a no-op install isn't + // reported as an upgrade (#2477). + if (method.kind === 'npm' && method.scope === 'local') { + const installed = method.root ? readInstalledVersion(method.root) : null; + if (installed && parseSemver(installed) && compareVersions(installed, latest) === 0) { + deps.log(c.green(`✓ ${method.root} now has CodeGraph ${latest}.`)); + } else if (installed) { + deps.warn(`npm finished, but ${method.root} still has CodeGraph ${normalizeVersion(installed)}.`); + } + return 'inconclusive'; + } const probe = verifyResolvedVersion(latest, deps); switch (probe) { case 'match': @@ -772,9 +870,12 @@ function upgradeNpm( const args = method.scope === 'global' ? ['install', '-g', `${NPM_PACKAGE}@${versionSpec}`] : ['install', `${NPM_PACKAGE}@${versionSpec}`]; - deps.log(c.dim(`Running: npm ${args.join(' ')}`)); + // A local install is upgraded in the project it belongs to: npm run from + // any other folder would install a second, stray copy there. + const where = method.scope === 'local' ? method.root : undefined; + deps.log(c.dim(`Running: npm ${args.join(' ')}${where ? ` (in ${where})` : ''}`)); const inv = npmInvocation(deps.platform, args); - const code = deps.run(inv.cmd, inv.args, process.env); + const code = deps.run(inv.cmd, inv.args, process.env, where); if (code !== 0) { deps.error(`npm exited with code ${code}.`); if (method.scope === 'global') { @@ -820,12 +921,23 @@ export function hasCommand(cmd: string): boolean { return false; } -export function defaultRun(cmd: string, args: string[], env?: NodeJS.ProcessEnv): number { - const r = spawnSync(cmd, args, { stdio: 'inherit', env: env ?? process.env, windowsHide: true }); +export function defaultRun(cmd: string, args: string[], env?: NodeJS.ProcessEnv, cwd?: string): number { + const r = spawnSync(cmd, args, { stdio: 'inherit', env: env ?? process.env, windowsHide: true, cwd }); if (r.error) return -1; return r.status ?? -1; } +/** The CodeGraph version installed in `root/node_modules`, or null. */ +function readInstalledVersion(root: string): string | null { + try { + const manifest = path.join(root, 'node_modules', ...NPM_PACKAGE.split('/'), 'package.json'); + const version: unknown = JSON.parse(fs.readFileSync(manifest, 'utf-8')).version; + return typeof version === 'string' ? version : null; + } catch { + return null; + } +} + export function defaultCapture(cmd: string, args: string[]): { code: number; stdout: string } | null { // stdio is piped (the default with `encoding`), so nothing the probed // command prints reaches the user's terminal. The timeout keeps a wedged From 929c65c83a681b4db3f705c3ad9cda03e755527c Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 03:39:38 +0000 Subject: [PATCH 17/95] fix(pascal): Delphi routines with a parameter attribute like `const [ref]` are indexed (#2778) The Pascal grammar's parameter rule takes no RTTI attribute, so a declaration like `procedure WithRef(const [ref] AValue: Integer; ...)` or DelphiMVCFramework's `function CreatePerson([MVCFromBody] const Person: TPerson)` broke the parse at the `[`. The method was lost, the parameter after the attribute became a field of the class, every member declared after it went missing until the next class, and in an implementation the routine's locals became unit variables and its calls were credited to the file node. The Pascal extractor now has an offset-preserving preParse that blanks an attribute group at the start of a routine parameter, before or after its modifier (const/var/out/constref), in any routine header, procedural type or anonymous method (`operator` counts only after `class`, since Delphi doesn't reserve it). It is a linear hand-written scanner that skips strings, comments and directives, reads generic and qualified routine names, and leaves every other bracket alone (set defaults, array types, member and method attributes, property index lists, code). A new LanguageExtractor flag, signatureAsWritten, lets Pascal signatures read the source as written, so `(const [MVCFromBody] Person: TPerson)` keeps its attribute instead of showing blank space. Only Pascal sets the flag. Validation: - New __tests__/pascal-param-attributes.test.ts (8 tests, the issue's unit included): 6 fail on main's wiring, all pass with the fix. - A/B against main 5a6917b4, both arms built from their own sources: SynEdit, HeidiSQL and DWScript (2,497 files) are byte-identical, as is mORMot2 (its two `[ref] const` signatures keep the attribute). DelphiMVCFramework (32 files changed by the preParse, 150 attributes): 4 nodes recovered (2 methods, a function and a constant), 10 removed, all misparses (9 method locals hoisted to unit variables and a parameter read as a field), 18 methods get a correct signature, end line and return type; edges -39 +90: 29 calls move from the file node or the wrong method to the method making them, 60 new edges come from bodies and declarations main lost, 10 contains edges went with the misparsed nodes. Every changed edge was triaged. - Controls leveldb (C++, kernel) and SDWebImage (Objective-C, wasm with its own preParse) are byte-identical in dump-graph. - Cost: the preParse takes about 4 ms per MB (95 ms on mORMot2's 25 MB); index time is within noise (3 interleaved rounds, medians: HeidiSQL 6.53 to 6.47 s, mORMot2 16.7 to 17.2 s, DWScript 16.3 to 16.1 s wall). - The shipped scanner matches the prototype used for triage on all 4,578 Pascal files of the five repos, with lengths and newlines kept. Fixes #2545 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/pascal-param-attributes.test.ts | 232 +++++++++++++++++ src/extraction/languages/pascal-preparse.ts | 260 ++++++++++++++++++++ src/extraction/languages/pascal.ts | 6 + src/extraction/tree-sitter-types.ts | 7 + src/extraction/tree-sitter.ts | 22 +- 6 files changed, 522 insertions(+), 6 deletions(-) create mode 100644 __tests__/pascal-param-attributes.test.ts create mode 100644 src/extraction/languages/pascal-preparse.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index c5feb1e8f3..6c4a7c0f47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In JavaScript and TypeScript, a call written as the body of an `if`, `while`, `for` or `do` without braces, like `if (e.key === "Escape") setOpen(false);`, no longer links to a same-named property or method in another file when the name is the file's own, such as a `setOpen` from `useState` (#2585). - `codegraph sync` no longer fails with "FOREIGN KEY constraint failed" in a project whose code lazy-loads a workspace package, like Supabase Studio's `dynamic(() => import('icons').then((mod) => mod.EdgeFunctions))` or vben's `() => import('@vben/layouts').then((m) => m.IFrameView)`. Before, every sync after an edit in such a project rolled back, so the index stopped following your changes until a full re-index. Thanks @bompus for the report and @inth3shadows. (#2565) - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. +- In Delphi, a routine with an attribute on one of its parameters, like `const [ref] AValue: Integer` or DelphiMVCFramework's `const [MVCFromBody] Person: TPerson`, is now indexed with its calls, and so is every class member declared after it. Before, the routine was lost or its calls were credited to the file, the parameter showed up as a field of the class, and the members after it were missing. Thanks @stalexteam for the report. (#2545) - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. diff --git a/__tests__/pascal-param-attributes.test.ts b/__tests__/pascal-param-attributes.test.ts new file mode 100644 index 0000000000..b95ce1165e --- /dev/null +++ b/__tests__/pascal-param-attributes.test.ts @@ -0,0 +1,232 @@ +/** + * An attribute on a Delphi routine parameter (#2545): `const [ref] AValue: + * Integer`, `[ref] const Source: TVarData`, DelphiMVCFramework's + * `const [MVCFromBody] Person: TPerson`. + * + * The Pascal grammar's parameter rule takes no attribute, so the parse broke + * at the `[`: the method was lost, the parameter after the attribute became a + * field of the class, and every member declared after it was missing until + * the next class. The extractor's preParse now blanks such an attribute, + * offsets kept, before the grammar parses the unit. + */ +import { describe, it, expect, beforeAll, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { preParsePascal } from '../src/extraction/languages/pascal-preparse'; + +beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['pascal']); +}); + +// The issue's own unit, with its bodies calling `Self.Before`. +const MIN_PAS = `unit Min; + +interface + +type + TMin = class + public + procedure Before; + procedure WithRef(const [ref] AValue: Integer; const AOther: string); + procedure After; + end; + + TNext = class + public + procedure Other; + end; + +const + CAfterClass = 1; + +implementation + +procedure TMin.Before; +begin +end; + +procedure TMin.WithRef(const [ref] AValue: Integer; const AOther: string); +begin + Self.Before; +end; + +procedure TMin.After; +begin + Self.Before; +end; + +procedure TNext.Other; +begin +end; + +end. +`; + +function members(code: string, file = 'Min.pas') { + const result = extractFromSource(file, code); + const of = (kind: string) => + result.nodes.filter((n) => n.kind === kind).map((n) => `${n.qualifiedName}:${n.startLine}`).sort(); + return { result, methods: of('method'), fields: of('field'), constants: of('constant') }; +} + +describe('Delphi parameter attributes (#2545)', () => { + it('keeps every class member after a `const [ref]` parameter', () => { + const { methods, fields, constants } = members(MIN_PAS); + expect(methods).toEqual([ + 'TMin::After:10', + 'TMin::Before:8', + 'TMin::WithRef:9', + 'TNext::Other:15', + ]); + // The parameter after the attribute is not a field of the class. + expect(fields).toEqual([]); + expect(constants).toEqual(['CAfterClass:19']); + }); + + it('keeps the member after a `[ref]` parameter that ends its list on its own line', () => { + const code = MIN_PAS + .replace('procedure WithRef(const [ref] AValue: Integer; const AOther: string);', 'procedure WithRef(const [ref] AValue: Integer);') + .replace('procedure TMin.WithRef(const [ref] AValue: Integer; const AOther: string);', 'procedure TMin.WithRef(const [ref] AValue: Integer);'); + const { methods } = members(code); + expect(methods).toEqual(['TMin::After:10', 'TMin::Before:8', 'TMin::WithRef:9', 'TNext::Other:15']); + }); + + it('keeps the attribute in the signature, as the source writes it', () => { + const { result } = members(MIN_PAS); + const withRef = result.nodes.find((n) => n.qualifiedName === 'TMin::WithRef'); + expect(withRef?.signature).toBe('(const [ref] AValue: Integer; const AOther: string)'); + }); + + it('reads the calls in the body of a routine whose header has an attribute', () => { + const { result } = members(MIN_PAS); + const withRef = result.nodes.find((n) => n.qualifiedName === 'TMin::WithRef'); + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls' && r.fromNodeId === withRef?.id) + .map((r) => `${r.referenceName}:${r.line}`); + expect(calls).toEqual(['Self.Before:29']); + }); + + it('takes attributes before or after the modifier, with arguments, stacked or spanning lines', () => { + const code = `unit People; + +interface + +type + TRec = record + A: Integer; + class operator Implicit(const [ref] Src: TRec): Integer; + procedure CopyFrom([ref] const Source: TRec); + procedure AfterRec; + end; + + TPeopleController = class(TMVCController) + public + [MVCPath('/people')] + [MVCHTTPMethod([httpPOST])] + function CreatePerson([MVCFromBody] const Person: TPerson): IMVCResponse; + function Search(const [MVCFromQueryString('q', '')] Query: string; + [MVCFromHeader('X-Page')] [MVCInject] Page: Integer; + const [MVCFromQueryString('order', + 'name')] Order: string): IMVCResponse; + function Find(const [ref] Key: TKey): T; + procedure Last; + end; + +implementation + +end. +`; + const { result, methods, fields } = members(code, 'People.pas'); + // A method's node starts at the attributes above it (CreatePerson's at 15). + expect(methods).toEqual([ + 'TPeopleController::CreatePerson:15', + 'TPeopleController::Find:22', + 'TPeopleController::Last:23', + 'TPeopleController::Search:18', + 'TRec::AfterRec:10', + 'TRec::CopyFrom:9', + 'TRec::Implicit:8', + ]); + expect(fields).toEqual(['TRec::A:7']); + const signatureOf = (qn: string) => result.nodes.find((n) => n.qualifiedName === qn)?.signature; + expect(signatureOf('TPeopleController::CreatePerson')).toBe('([MVCFromBody] const Person: TPerson): IMVCResponse'); + expect(signatureOf('TRec::CopyFrom')).toBe('([ref] const Source: TRec)'); + }); + + it('blanks only parameter attributes, keeping every offset', () => { + const query = "[MVCFromQueryString('q', '')]"; + const page = "[MVCFromHeader('X-Page')]"; + const code = [`procedure Search(const ${query} Query: string;`, ` ${page}`, ' Page: Integer);'].join('\r\n'); + const out = preParsePascal(code); + expect(out.length).toBe(code.length); + expect(out.split('\r\n')).toEqual([ + `procedure Search(const ${' '.repeat(query.length)} Query: string;`, + ` ${' '.repeat(page.length)}`, + ' Page: Integer);', + ]); + }); + + it('leaves brackets alone everywhere else', () => { + const code = `unit Keep; + +interface + +type + TOwner = class + private + [Weak] FOwner: TObject; + FItems: array[0..1] of Integer; + public + [MVCPath('/x')] + procedure Get(const Options: TOptions = [ooA, ooB]; Count: Integer = 1); + property Items[Index: Integer]: Integer read GetItem; + end; + +implementation + +{ procedure Old(const [ref] A: Integer); } +// function Older([ref] const A: Integer): Integer; + +procedure TOwner.Get(const Options: TOptions = [ooA, ooB]; Count: Integer = 1); +begin + Foo([1, 2]); + Bar(Format('%d', [Count]), [ooA]); + WriteLn('procedure Fake(const [ref] A: Integer)'); + FItems[0] := Ord(ooA in [ooA]); + Count := Operator([1, 2]); // Delphi doesn't reserve \`operator\` +end; + +end. +`; + expect(preParsePascal(code)).toBe(code); + }); + + it('indexes the issue unit with TMin.After resolvable and no field from the parameter', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-attr-')); + try { + fs.writeFileSync(path.join(dir, 'Min.pas'), MIN_PAS); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const after = cg.getNodesByKind('method').find((n) => n.qualifiedName === 'TMin::After'); + expect(after).toBeDefined(); + expect(cg.getNodesByKind('field').map((n) => n.name)).toEqual([]); + const before = cg.getNodesByKind('method').find((n) => n.qualifiedName === 'TMin::Before')!; + const callers = cg + .getIncomingEdges(before.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.source)?.name) + .sort(); + expect(callers).toEqual(['After', 'WithRef']); + } finally { + cg.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/src/extraction/languages/pascal-preparse.ts b/src/extraction/languages/pascal-preparse.ts new file mode 100644 index 0000000000..8b7c5fb176 --- /dev/null +++ b/src/extraction/languages/pascal-preparse.ts @@ -0,0 +1,260 @@ +/** + * Offset-preserving source fixes the Pascal grammar needs before it parses a + * Delphi unit (the extractor's `preParse`). Each one blanks syntax the grammar + * has no rule for with spaces, keeping every newline, so node positions and + * text stay where they are. + * + * - An attribute on a routine parameter, before or after its modifier: + * `const [ref] AValue: Integer`, `[ref] const Source: TVarData`, + * `const [MVCFromBody] Person: TPerson`, `[MVCFromQueryString('q', '')] Q: string`. + * The grammar's parameter rule takes no attribute, so the parse fell apart + * at the `[` and every member declared after it in the class was lost. + * The extractor reads nothing from an attribute, and a signature still + * shows it: Pascal signatures read the source as written. + */ + +const MODIFIERS: ReadonlySet = new Set(['const', 'var', 'out', 'constref']); +const ROUTINE_KEYWORDS: ReadonlySet = new Set(['procedure', 'function', 'constructor', 'destructor', 'operator']); +/** An attribute group longer than this is not one (and keeps the scan linear). */ +const MAX_ATTRIBUTE_LENGTH = 2000; + +const CH_TAB = 9, CH_LF = 10, CH_CR = 13, CH_SPACE = 32, CH_AMP = 38, CH_QUOTE = 39; +const CH_LPAREN = 40, CH_RPAREN = 41, CH_STAR = 42, CH_DOT = 46, CH_SLASH = 47; +const CH_SEMI = 59, CH_LT = 60, CH_GT = 62, CH_LBRACKET = 91, CH_RBRACKET = 93, CH_LBRACE = 123; + +/** Identifier characters, `&` (Delphi's keyword escape) and any non-ASCII letter. */ +function isIdentStart(c: number): boolean { + return (c >= 65 && c <= 90) || (c >= 97 && c <= 122) || c === 95 || c === CH_AMP || c >= 128; +} +function isIdentPart(c: number): boolean { + return (c >= 65 && c <= 90) || (c >= 97 && c <= 122) || c === 95 || (c >= 48 && c <= 57) || c >= 128; +} + +function wordEnd(s: string, i: number): number { + while (i < s.length && isIdentPart(s.charCodeAt(i))) i++; + return i; +} + +/** + * The index after the comment or string that starts at `i`, or -1 when none + * does. Pascal comments are `{…}` (compiler directives included), `(*…*)` and + * `//…`; a string is `'…'` with `''` for a quote and ends at its line's end + * when unterminated. + */ +function commentOrStringEnd(s: string, i: number): number { + const c = s.charCodeAt(i); + if (c === CH_LBRACE) { + const e = s.indexOf('}', i + 1); + return e < 0 ? s.length : e + 1; + } + if (c === CH_LPAREN && s.charCodeAt(i + 1) === CH_STAR) { + const e = s.indexOf('*)', i + 2); + return e < 0 ? s.length : e + 2; + } + if (c === CH_SLASH && s.charCodeAt(i + 1) === CH_SLASH) { + const e = s.indexOf('\n', i + 2); + return e < 0 ? s.length : e; + } + if (c === CH_QUOTE) { + let j = i + 1; + while (j < s.length) { + const d = s.charCodeAt(j); + if (d === CH_QUOTE) { + if (s.charCodeAt(j + 1) !== CH_QUOTE) return j + 1; + j += 2; + continue; + } + if (d === CH_LF) return j; + j++; + } + return j; + } + return -1; +} + +/** The index of the first character from `i` that is neither whitespace nor in a comment. */ +function skipTrivia(s: string, i: number): number { + while (i < s.length) { + const c = s.charCodeAt(i); + if (c === CH_SPACE || c === CH_TAB || c === CH_LF || c === CH_CR) { + i++; + } else if (c === CH_LBRACE || (c === CH_LPAREN && s.charCodeAt(i + 1) === CH_STAR) || (c === CH_SLASH && s.charCodeAt(i + 1) === CH_SLASH)) { + i = commentOrStringEnd(s, i); + } else { + break; + } + } + return i; +} + +/** + * The index after the `]` closing the attribute group whose `[` is at `i`, + * or -1 when the group doesn't close the way an attribute does: nested + * brackets and parentheses balance, strings and comments are skipped, and an + * attribute's arguments hold no `;`. + */ +function attributeEnd(s: string, i: number): number { + const limit = Math.min(s.length, i + MAX_ATTRIBUTE_LENGTH); + let depth = 0; + let j = i; + while (j < limit) { + const k = commentOrStringEnd(s, j); + if (k >= 0) { + j = k; + continue; + } + const c = s.charCodeAt(j); + if (c === CH_LBRACKET || c === CH_LPAREN) { + depth++; + } else if (c === CH_RBRACKET || c === CH_RPAREN) { + depth--; + if (depth === 0) return c === CH_RBRACKET ? j + 1 : -1; + } else if (c === CH_SEMI) { + return -1; + } + j++; + } + return -1; +} + +/** + * The index after a routine's name — `Name`, `TFoo.Bar`, `TFoo.Bar`, + * `Get` — read from `i`, or `i` itself (trivia + * skipped) when no name starts there (`procedure(…) of object`). + */ +function routineNameEnd(s: string, i: number): number { + for (;;) { + i = skipTrivia(s, i); + if (!isIdentStart(s.charCodeAt(i))) return i; + i = wordEnd(s, i + 1); + let j = skipTrivia(s, i); + if (s.charCodeAt(j) === CH_LT) { + let depth = 0; + for (; j < s.length; j++) { + const c = s.charCodeAt(j); + if (c === CH_LT) { + depth++; + } else if (c === CH_GT) { + if (--depth === 0) { + j++; + break; + } + } else if (c === CH_SEMI || c === CH_LPAREN || c === CH_RPAREN || c === CH_LF) { + return i; // not a generic parameter list + } + } + i = j; + j = skipTrivia(s, i); + } + if (s.charCodeAt(j) !== CH_DOT) return i; + i = j + 1; + } +} + +/** + * Walk the parameter list whose `(` is at `open`, pushing the `[from, to)` + * range of each attribute group a parameter starts with — before or after its + * modifier — onto `ranges`. Returns the index after the list's `)`. + */ +function scanParameterList(s: string, open: number, ranges: number[]): number { + let depth = 0; + let i = open; + let atParameterStart = false; + while (i < s.length) { + if (atParameterStart) { + atParameterStart = false; + for (;;) { + i = skipTrivia(s, i); + const c = s.charCodeAt(i); + if (c === CH_LBRACKET) { + const end = attributeEnd(s, i); + if (end < 0) break; + ranges.push(i, end); + i = end; + } else if (isIdentStart(c)) { + const e = wordEnd(s, i + 1); + if (!MODIFIERS.has(s.slice(i, e).toLowerCase())) break; + i = e; + } else { + break; + } + } + continue; + } + const k = commentOrStringEnd(s, i); + if (k >= 0) { + i = k; + continue; + } + const c = s.charCodeAt(i); + if (c === CH_LPAREN || c === CH_LBRACKET) { + if (++depth === 1) atParameterStart = true; + } else if (c === CH_RPAREN || c === CH_RBRACKET) { + if (--depth <= 0) return i + 1; + } else if (c === CH_SEMI && depth === 1) { + atParameterStart = true; + } + i++; + } + return i; +} + +/** `s` with every `[from, to)` range in `ranges` blanked to spaces, newlines kept. */ +function blankRanges(s: string, ranges: number[]): string { + if (ranges.length === 0) return s; + let out = ''; + let last = 0; + for (let r = 0; r < ranges.length; r += 2) { + const from = ranges[r]!; + const to = ranges[r + 1]!; + out += s.slice(last, from) + s.slice(from, to).replace(/[^\r\n]/g, ' '); + last = to; + } + return out + s.slice(last); +} + +/** Whether the word before index `i`, past whitespace, is `class`. */ +function followsClass(s: string, i: number): boolean { + let j = i - 1; + while (j >= 0 && (s.charCodeAt(j) === CH_SPACE || s.charCodeAt(j) === CH_TAB || s.charCodeAt(j) === CH_LF || s.charCodeAt(j) === CH_CR)) j--; + return j >= 4 && s.slice(j - 4, j + 1).toLowerCase() === 'class' && (j < 5 || !isIdentPart(s.charCodeAt(j - 5))); +} + +/** + * Blank what the Pascal grammar can't parse (see the file comment). Returns + * the source unchanged when there is nothing to blank. + */ +export function preParsePascal(source: string): string { + if (source.indexOf('[') < 0) return source; + const ranges: number[] = []; + const n = source.length; + let i = 0; + while (i < n) { + const k = commentOrStringEnd(source, i); + if (k >= 0) { + i = k; + continue; + } + const c = source.charCodeAt(i); + if (isIdentStart(c)) { + const e = wordEnd(source, i + 1); + const len = e - i; + const word = len >= 8 && len <= 11 ? source.slice(i, e).toLowerCase() : ''; + // `operator` is no reserved word in Delphi (a call `Operator([1, 2])` is + // code), so it starts a header only after `class`. + if (ROUTINE_KEYWORDS.has(word) && (word !== 'operator' || followsClass(source, i))) { + const open = skipTrivia(source, routineNameEnd(source, e)); + if (source.charCodeAt(open) === CH_LPAREN) { + i = scanParameterList(source, open, ranges); + continue; + } + } + i = e; + } else if (c >= 48 && c <= 57) { + i = wordEnd(source, i + 1); // a number + } else { + i++; + } + } + return blankRanges(source, ranges); +} diff --git a/src/extraction/languages/pascal.ts b/src/extraction/languages/pascal.ts index 004dadc83d..164758f815 100644 --- a/src/extraction/languages/pascal.ts +++ b/src/extraction/languages/pascal.ts @@ -1,8 +1,14 @@ import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText, getChildByField } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; +import { preParsePascal } from './pascal-preparse'; export const pascalExtractor: LanguageExtractor = { + // Blank Delphi syntax the grammar has no rule for, such as an attribute on a + // parameter (`const [ref] AValue: Integer`), offsets kept (#2545). Signatures + // still read the attribute from the source as written. + preParse: preParsePascal, + signatureAsWritten: true, functionTypes: ['declProc'], classTypes: ['declClass'], methodTypes: ['declProc'], diff --git a/src/extraction/tree-sitter-types.ts b/src/extraction/tree-sitter-types.ts index b883febc1a..ae629e5538 100644 --- a/src/extraction/tree-sitter-types.ts +++ b/src/extraction/tree-sitter-types.ts @@ -97,6 +97,13 @@ export interface LanguageExtractor { * language id serves several dialects (C++ also parses `.metal` shaders). */ preParse?: (source: string, filePath?: string) => string; + /** + * Read `getSignature` from the source as the file has it, not from + * `preParse`'s output, when the preParse blanks syntax a reader of the + * signature still wants: a Delphi parameter's attribute, `const [ref] AValue` + * or `const [MVCFromBody] Person` (#2545). Offsets are the same in both. + */ + signatureAsWritten?: boolean; // --- Node type mappings --- diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 2d84db37e6..0d10db3aa1 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -747,6 +747,9 @@ export class TreeSitterExtractor { // point (this instance is the wasm fallback for a kernel-deferred file) — // don't blank it a second time. private sourceIsPreParsed = false; + // The source as the file has it, kept when this instance's preParse changed + // it, for an extractor whose signatures read it (`signatureAsWritten`). + private sourceAsWritten: string | null = null; constructor( filePath: string, @@ -762,6 +765,11 @@ export class TreeSitterExtractor { this.sourceIsPreParsed = options?.sourceIsPreParsed === true; } + /** The text `getSignature` reads: the source as written when the extractor asks for it. */ + private get signatureSource(): string { + return this.sourceAsWritten ?? this.source; + } + /** * Parse and extract from the source code */ @@ -811,7 +819,9 @@ export class TreeSitterExtractor { // saw (identical outside the blanked directive lines). Skipped when the // kernel route point already applied it (sourceIsPreParsed). if (this.extractor?.preParse && !this.sourceIsPreParsed) { - this.source = this.extractor.preParse(this.source, this.filePath); + const preParsed = this.extractor.preParse(this.source, this.filePath); + if (preParsed !== this.source && this.extractor.signatureAsWritten) this.sourceAsWritten = this.source; + this.source = preParsed; } this.tree = parser.parse(this.source) ?? null; if (!this.tree) { @@ -2160,7 +2170,7 @@ export class TreeSitterExtractor { } const docstring = this.docstringFor(node); - const signature = this.extractor.getSignature?.(node, this.source); + const signature = this.extractor.getSignature?.(node, this.signatureSource); const visibility = this.extractor.getVisibility?.(node); const isExported = commonJsExport || this.extractor.isExported?.(node, this.source); const isAsync = this.extractor.isAsync?.(node); @@ -2521,7 +2531,7 @@ export class TreeSitterExtractor { } const docstring = this.docstringFor(node); - const signature = this.extractor.getSignature?.(node, this.source); + const signature = this.extractor.getSignature?.(node, this.signatureSource); const visibility = this.extractor.getVisibility?.(node); const isAsync = this.extractor.isAsync?.(node); const isStatic = this.extractor.isStatic?.(node); @@ -4088,7 +4098,7 @@ export class TreeSitterExtractor { docstring?: string ): void { if (!this.extractor) return; - const signature = this.extractor.getSignature?.(node, this.source); + const signature = this.extractor.getSignature?.(node, this.signatureSource); const extra: Partial = { docstring, signature }; if (receiver) extra.qualifiedName = this.composeReceiverQualifiedName(receiver, name); else extra.isExported = this.extractor.isExported?.(node, this.source); @@ -4291,7 +4301,7 @@ export class TreeSitterExtractor { const mname = getNodeText(nameNode, this.source); if (mname) { const methodNode = this.createNode('method', mname, m, { - signature: this.extractor?.getSignature?.(m, this.source), + signature: this.extractor?.getSignature?.(m, this.signatureSource), }); if (methodNode) { for (const type of goInterfaceMethodTypeNames(m, decl, this.source)) { @@ -8525,7 +8535,7 @@ export class TreeSitterExtractor { // free routines, so there's no duplicate. if (!parentId && !fullName.includes('.')) { const fnNode = this.createNode('function', fullName, declProc, { - signature: this.extractor?.getSignature?.(declProc, this.source), + signature: this.extractor?.getSignature?.(declProc, this.signatureSource), visibility: this.extractor?.getVisibility?.(declProc), }); if (fnNode) { From 88ebefa1353806a48f914d6d66807b24dfa235e6 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 04:23:40 +0000 Subject: [PATCH 18/95] fix(pascal): calls inside a Delphi `for var x in` loop are indexed (#2781) The Pascal grammar's for-in rule takes a plain variable, so a loop that declares its variable inline (Delphi 10.3 and later), like `for var lItem in AList do lItem.DropIfEmpty;` or `for var lHandle: THandle in lHandles do`, became an ERROR node. A paren-less call in its body lost its statement shape and reached no graph, so the routine was missing from the callee's callers. The Pascal preParse added for issue #2545 now also blanks the inline `var` and the loop variable's `: Type`, offsets kept, so the loop parses as the grammar's `for x in` form. The type runs to the `in` at nesting depth 0, so generic and dotted types are whole; `for var i := 0 to`, which the grammar already parses, and `for var` inside comments or strings are left alone. The resolver still types the loop variable: it reads a receiver's declaration from the source as written, so `lHandle.Close` reaches THandle's Close, not a namesake. Validation: - New __tests__/pascal-for-in-var.test.ts (6 tests, the issue's unit included): 5 fail with main's pascal-preparse.ts, all pass with it. - Every textual `for var x in` in the corpus is blanked except 9 that sit in comments or strings (DelphiMVCFramework 47 of 49, DWScript 89 of 96), no other text is touched, and no file parses worse. - A/B against main 929c65c8, both arms built from their own sources: SynEdit, HeidiSQL and mORMot2 are byte-identical. DelphiMVCFramework: errors in the 20 changed files drop from 57 to 10, edges are identical (its loop bodies hold parenthesized calls, which main's error recovery already kept), +1 unresolved ref for the VCL's own GetControls in a loop's iterator. DWScript (2,497 files, mostly DWScript-language test scripts): edges -41 +30, every one triaged. The two real-library files gain correct calls; one paren-less right-hand-side read main kept only through error recovery (`var v := FIter.CollectInteger`) is gone until issue #2514's paren-less calls land. The rest are script files: 7 new calls inside the loops, one call moved from the file node to its routine, one stray top-level statement no longer credited to the routine error recovery had stretched over it, 21 file-level guesses error recovery made on one script's stray top-level statements, and 8 garbage nodes per enumerations script whose multi-line names now hold the blanked text. - Controls leveldb (C++, kernel) and SDWebImage (Objective-C, wasm) are byte-identical in dump-graph. - Cost: no measurable change; index time medians of 3 interleaved rounds: HeidiSQL 5.50 to 5.49 s, mORMot2 13.96 to 13.94 s, DWScript 13.98 to 13.94 s wall. Fixes #2546 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/pascal-for-in-var.test.ts | 237 ++++++++++++++++++++ src/extraction/languages/pascal-preparse.ts | 76 ++++++- 3 files changed, 312 insertions(+), 2 deletions(-) create mode 100644 __tests__/pascal-for-in-var.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c4a7c0f47..08981f81c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph sync` no longer fails with "FOREIGN KEY constraint failed" in a project whose code lazy-loads a workspace package, like Supabase Studio's `dynamic(() => import('icons').then((mod) => mod.EdgeFunctions))` or vben's `() => import('@vben/layouts').then((m) => m.IFrameView)`. Before, every sync after an edit in such a project rolled back, so the index stopped following your changes until a full re-index. Thanks @bompus for the report and @inth3shadows. (#2565) - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. - In Delphi, a routine with an attribute on one of its parameters, like `const [ref] AValue: Integer` or DelphiMVCFramework's `const [MVCFromBody] Person: TPerson`, is now indexed with its calls, and so is every class member declared after it. Before, the routine was lost or its calls were credited to the file, the parameter showed up as a field of the class, and the members after it were missing. Thanks @stalexteam for the report. (#2545) +- In Delphi, a `for … in` loop that declares its variable inline, like `for var lItem in AList do` or `for var lHandle: THandle in lHandles do`, now has its calls indexed, and a call on a typed loop variable reaches that type's method. Before, the whole loop was skipped, so a routine calling a method only from such a loop was missing from that method's callers. Thanks @stalexteam for the report. (#2546) - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. diff --git a/__tests__/pascal-for-in-var.test.ts b/__tests__/pascal-for-in-var.test.ts new file mode 100644 index 0000000000..0d782bdb2f --- /dev/null +++ b/__tests__/pascal-for-in-var.test.ts @@ -0,0 +1,237 @@ +/** + * A Delphi `for … in` loop that declares its variable inline (#2546): + * `for var lItem in AList do`, `for var lHandle: THandle in lHandles do` + * (Delphi 10.3 and later). + * + * The Pascal grammar's for-in rule takes a plain variable, so the whole loop + * was an ERROR node and no call in its body reached the graph. The extractor's + * preParse now blanks the inline `var` and its type, offsets kept, so the loop + * parses the way the grammar knows it. The resolver still finds the type: it + * reads a receiver's declaration from the source as written. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { getParser, initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { preParsePascal } from '../src/extraction/languages/pascal-preparse'; + +beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['pascal']); +}); + +// The issue's own unit. +const LOOP_PAS = `unit Loop; + +interface + +type + TItem = class + public + function DropIfEmpty: Boolean; + end; + +procedure WithVar(const AList: TArray); +procedure WithDecl(const AList: TArray); + +implementation + +function TItem.DropIfEmpty: Boolean; +begin + Result := True; +end; + +procedure WithVar(const AList: TArray); +begin + for var lItem in AList do + lItem.DropIfEmpty; +end; + +procedure WithDecl(const AList: TArray); +var + lItem: TItem; +begin + for lItem in AList do + lItem.DropIfEmpty; +end; + +end. +`; + +function callsFrom(code: string, qualifiedName: string, file = 'Loop.pas'): string[] { + const result = extractFromSource(file, code); + const from = result.nodes.find((n) => n.qualifiedName === qualifiedName); + return result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls' && r.fromNodeId === from?.id) + .map((r) => `${r.referenceName}:${r.line}`); +} + +async function withIndex(files: Record, check: (cg: CodeGraph) => void): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-for-var-')); + try { + for (const [name, body] of Object.entries(files)) fs.writeFileSync(path.join(dir, name), body); + const cg = await CodeGraph.init(dir, { index: true }); + try { + check(cg); + } finally { + cg.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +function callerNames(cg: CodeGraph, qualifiedName: string): string[] { + const target = cg.getNodesByKind('method').find((n) => n.qualifiedName === qualifiedName); + if (!target) return []; + return cg + .getIncomingEdges(target.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.source)?.name ?? '?') + .sort(); +} + +describe('Delphi `for var … in` loops (#2546)', () => { + it('reads the call in the body of a `for var x in` loop, as for a declared loop variable', () => { + expect(callsFrom(LOOP_PAS, 'WithVar')).toEqual(['lItem.DropIfEmpty:24']); + expect(callsFrom(LOOP_PAS, 'WithDecl')).toEqual(['lItem.DropIfEmpty:32']); + }); + + it('links both routines to the method the issue names', async () => { + await withIndex({ 'Loop.pas': LOOP_PAS }, (cg) => { + expect(callerNames(cg, 'TItem::DropIfEmpty')).toEqual(['WithDecl', 'WithVar']); + }); + }); + + it('resolves a typed loop variable on its declared type, not on a namesake', async () => { + const code = `unit Recv; + +interface + +type + THandleBox = class + public + procedure Close; + end; + + TDecoy = class + public + procedure Close; + end; + +procedure UseTyped(const AHandles: TArray); + +implementation + +procedure THandleBox.Close; +begin +end; + +procedure TDecoy.Close; +begin +end; + +procedure UseTyped(const AHandles: TArray); +begin + for var lHandle: THandleBox in AHandles do + lHandle.Close; +end; + +end. +`; + await withIndex({ 'Recv.pas': code }, (cg) => { + expect(callerNames(cg, 'THandleBox::Close')).toEqual(['UseTyped']); + expect(callerNames(cg, 'TDecoy::Close')).toEqual([]); + }); + }); + + it('parses generic and dotted loop types, loops inside anonymous methods and one-line loops', () => { + const code = `unit Forms; + +interface + +implementation + +procedure Run(const D: TDictionary>; const L: TArray); +begin + for var P: TPair> in D do + P.Value.Clear; + for var Q: System.Generics.Collections.TPair in Pairs do Q.Free; + TParallel.For(0, 9, procedure(I: Integer) + begin + for var X in L do X.Reset; + end); + if Assigned(L) then for var Y in L do Y.Done; +end; + +end. +`; + const tree = getParser('pascal')!.parse(preParsePascal(code))!; + try { + expect(tree.rootNode.hasError).toBe(false); + } finally { + tree.delete(); + } + // `P.Value.Clear` keeps the bare method name the extractor gives a chain + // on a variable; the anonymous method's call belongs to `Run`. + expect(callsFrom(code, 'Run', 'Forms.pas')).toEqual([ + 'Clear:10', + 'Q.Free:11', + 'TParallel.For:12', + 'X.Reset:14', + 'Assigned:16', + 'Y.Done:16', + ]); + }); + + it('blanks only the inline `var` and its type, keeping every offset', () => { + const blank = (line: string, ...parts: string[]) => + parts.reduce((s, part) => s.replace(part, ' '.repeat(part.length)), line); + const lines = [ + 'for var lItem in AList do', + 'for var lHandle: THandle in lHandles do', + 'for var P: TPair>', + ' in D do', + ]; + const out = preParsePascal(lines.join('\r\n')); + expect(out.split('\r\n')).toEqual([ + blank(lines[0]!, 'var'), + blank(lines[1]!, 'var', ': THandle'), + blank(lines[2]!, 'var', ': TPair>'), + lines[3], + ]); + }); + + it('leaves every other `for` alone', () => { + const code = `unit Keep; + +interface + +type + TStringHelper = record helper for string + function Upper: string; + end; + +implementation + +procedure Run(const L: TArray); +var + N: Integer; +begin + for var I := 0 to 9 do Use(I); + for var J: Integer := 0 to 9 do Use(J); + for N in L do Use(N); + for N := High(L) downto 0 do Use(N); + // for var Old in L do Use(Old); + { for var Older in L do } + WriteLn('for var Text in L do'); +end; + +end. +`; + expect(preParsePascal(code)).toBe(code); + }); +}); diff --git a/src/extraction/languages/pascal-preparse.ts b/src/extraction/languages/pascal-preparse.ts index 8b7c5fb176..5d907f3ef1 100644 --- a/src/extraction/languages/pascal-preparse.ts +++ b/src/extraction/languages/pascal-preparse.ts @@ -11,16 +11,29 @@ * at the `[` and every member declared after it in the class was lost. * The extractor reads nothing from an attribute, and a signature still * shows it: Pascal signatures read the source as written. + * - The inline variable of a `for … in` loop (Delphi 10.3): `for var lItem in + * AList do`, `for var lHandle: THandle in lHandles do`. The grammar's for-in + * rule takes a plain variable, so the whole loop became an error and no + * call in its body was read. Blanking `var` and the type leaves the loop + * the grammar knows. The resolver still finds the type: it reads a + * receiver's declaration from the source as written. */ const MODIFIERS: ReadonlySet = new Set(['const', 'var', 'out', 'constref']); const ROUTINE_KEYWORDS: ReadonlySet = new Set(['procedure', 'function', 'constructor', 'destructor', 'operator']); /** An attribute group longer than this is not one (and keeps the scan linear). */ const MAX_ATTRIBUTE_LENGTH = 2000; +/** A for-in loop variable's type longer than this is not one. */ +const MAX_LOOP_TYPE_LENGTH = 500; +/** Words that end a for-in variable's type without an `in`: not the for-in form. */ +const NOT_A_FOR_IN: ReadonlySet = new Set(['do', 'to', 'downto', 'begin', 'end']); +/** Whether a file can hold a `for var … in` loop at all (the scan below decides). */ +const FOR_VAR = /\bfor\s+var\b/i; const CH_TAB = 9, CH_LF = 10, CH_CR = 13, CH_SPACE = 32, CH_AMP = 38, CH_QUOTE = 39; const CH_LPAREN = 40, CH_RPAREN = 41, CH_STAR = 42, CH_DOT = 46, CH_SLASH = 47; -const CH_SEMI = 59, CH_LT = 60, CH_GT = 62, CH_LBRACKET = 91, CH_RBRACKET = 93, CH_LBRACE = 123; +const CH_COLON = 58, CH_SEMI = 59, CH_LT = 60, CH_EQ = 61, CH_GT = 62; +const CH_LBRACKET = 91, CH_RBRACKET = 93, CH_LBRACE = 123; /** Identifier characters, `&` (Delphi's keyword escape) and any non-ASCII letter. */ function isIdentStart(c: number): boolean { @@ -199,6 +212,62 @@ function scanParameterList(s: string, open: number, ranges: number[]): number { return i; } +/** The word starting at `i`, lowercased, or '' when no word starts there. */ +function lowerWordAt(s: string, i: number): string { + return isIdentStart(s.charCodeAt(i)) ? s.slice(i, wordEnd(s, i + 1)).toLowerCase() : ''; +} + +/** + * After a `for` keyword ending at `forEnd`, recognize `var Name in` or + * `var Name: Type in` and push the `var` range and the `: Type` range onto + * `ranges`. The type runs to the `in` at nesting depth 0, so generic types + * (`TPair>`) and dotted names are whole. Any other + * shape (`for var i := 0 to …`, which the grammar parses, or no `in`) adds + * nothing. Returns the index to resume scanning from. + */ +function scanForInVariable(s: string, forEnd: number, ranges: number[]): number { + const varStart = skipTrivia(s, forEnd); + if (lowerWordAt(s, varStart) !== 'var') return forEnd; + const varEnd = varStart + 3; + const nameStart = skipTrivia(s, varEnd); + if (!isIdentStart(s.charCodeAt(nameStart))) return forEnd; + let i = skipTrivia(s, wordEnd(s, nameStart + 1)); + let typeStart = -1; + if (s.charCodeAt(i) === CH_COLON && s.charCodeAt(i + 1) !== CH_EQ) { + typeStart = i; + const limit = Math.min(s.length, i + MAX_LOOP_TYPE_LENGTH); + let depth = 0; + i++; + for (;;) { + if (i >= limit) return forEnd; + const k = commentOrStringEnd(s, i); + if (k >= 0) { + i = k; + continue; + } + const c = s.charCodeAt(i); + if (c === CH_LT || c === CH_LPAREN || c === CH_LBRACKET) { + depth++; + } else if (c === CH_GT || c === CH_RPAREN || c === CH_RBRACKET) { + if (--depth < 0) return forEnd; + } else if (c === CH_SEMI || c === CH_EQ) { + return forEnd; // `for var i: Integer := 0 to …`, or no loop at all + } else if (isIdentStart(c)) { + const word = lowerWordAt(s, i); + if (depth === 0 && word === 'in') break; + if (NOT_A_FOR_IN.has(word)) return forEnd; + i = wordEnd(s, i + 1); + continue; + } + i++; + } + } + if (lowerWordAt(s, i) !== 'in') return forEnd; + ranges.push(varStart, varEnd); + if (typeStart >= 0) ranges.push(typeStart, i); + return i; +} + /** `s` with every `[from, to)` range in `ranges` blanked to spaces, newlines kept. */ function blankRanges(s: string, ranges: number[]): string { if (ranges.length === 0) return s; @@ -225,7 +294,7 @@ function followsClass(s: string, i: number): boolean { * the source unchanged when there is nothing to blank. */ export function preParsePascal(source: string): string { - if (source.indexOf('[') < 0) return source; + if (source.indexOf('[') < 0 && !FOR_VAR.test(source)) return source; const ranges: number[] = []; const n = source.length; let i = 0; @@ -248,6 +317,9 @@ export function preParsePascal(source: string): string { i = scanParameterList(source, open, ranges); continue; } + } else if (len === 3 && source.slice(i, e).toLowerCase() === 'for') { + i = scanForInVariable(source, e, ranges); + continue; } i = e; } else if (c >= 48 && c <= 57) { From 0c3e12ddbbcc467b595ca994941151a7dc60db23 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 04:53:59 +0000 Subject: [PATCH 19/95] fix(go): a call further along a chain links the method of the type the chain reaches (#2782) A Go method called on a receiver the extractor keeps no name for, such as a fluent chain's later link (model.WithContext(0).Where(1).First()), a call on what a function returns (time.Now().UTC().Format(...)) or a composite literal (GitOracle{}.Worktree()), reached the resolver as its bare name at the column where the chain starts. Name matching then took a project method of that name by proximity, or the only one: the issue's GORM-style Where and First went to another query type, the standard library's Format to the one project Format, and a literal's method to a namesake type. A new Go resolver step reads the chain at that column and types the receiver link by link: a call's result, or the one a multi-value assignment takes; a conversion's or a composite literal's type; a generic call given its type arguments; an embedded field by its type's name; a package-level variable from the call it is set to. A result written with type parameters keeps them as parameters, read from the declaration, since the indexed signature leaves them out, and the helper that reads them now includes a one-line function's only line. The method is the one Go selects on that type. A type from outside the project, or one that can't be read, links nothing instead of a guess. When a chain calls one name twice, the bare ref is the later call. Validation: 8 new tests; 7 fail on main, and the repeated-name case fails under a first-link rule; the 43 Go test files pass. A/B against main c326b6be, edges removed/added: gin 80/72, etcd 1726/1327, prometheus 2249/1265, sonic (gorm/gen) 910/877, kubernetes 88288/84331. Every removed edge was a name-match guess except 173 in kubernetes: 171 are the same edge now owned by another ref of its chain, and 2 wrong links of metav1.Now().Time.Unix(). Sampled retargets, drops and additions read correct; express and flask dumps are byte-identical. Indexing cost is within noise, median of 3 interleaved runs, wall/CPU seconds main -> fix: etcd 9.5/28.6 -> 8.6/25.4, prometheus 10.5/42.1 -> 10.5/42.6, kubernetes 121.9/413.7 -> 121.3/416.8. Fixes #2543 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/go-chained-receivers.test.ts | 370 +++++++++++++++++++++++++ src/resolution/index.ts | 7 +- src/resolution/name-matcher.ts | 254 +++++++++++++++-- 4 files changed, 611 insertions(+), 21 deletions(-) create mode 100644 __tests__/go-chained-receivers.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 08981f81c0..722fe254e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a call made through a type assertion, like `srv.(KVServer).Range(ctx, in)` in generated gRPC code, `c.Reader.(*pipe).Close()` or `v.(storage.Store).Fetch(key)`, now links to that method of the asserted type, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, it linked to whichever method shared the name, so every gRPC handler pointed at the `Unimplemented…Server` stub instead of the server interface. Re-index Go projects after upgrading. - In Go, a call through a variable that holds the result of a type assertion, like `st.Fetch(key)` after `st := v.(storage.Store)` or `rc.Read(p)` after `rc, ok := v.(ReadCloser)`, now links to that method of the asserted type, as a call made through the assertion itself does, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, such a call linked to whichever method shared the name or to nothing, and a variable named like a standard-library package, like grpc-go's `parser, ok := builder.(balancer.ConfigParser)`, was taken for that package. Re-index Go projects after upgrading. - In Go, a method called on an element of a slice, array or map, like kubectl's `schemas[ix].ValidateBytes(data)`, `pods[i].DeepCopy()`, `m.Items[len(m.Items)-1].Unmarshal(…)` in generated protobuf code or `(*in)[i].DeepCopyInto(&(*out)[i])` in every deepcopy file, now links to that method of the element's type. The type is read from what is indexed: a parameter, a variable, a struct field or a named slice type like `type ConjunctiveSchema []Schema`, and an element held by pointer, like `[]*Item`, reaches `Item`'s methods. Before, such a call linked to whichever method shared its name, often one in an unrelated package, or to nothing when many types declare it, as with Kubernetes' `DeepCopyInto`. An element whose type comes from outside your project, like `[]*net.IPNet`, or is a built-in one like `[]error`, now links to nothing rather than to a project method of that name. Re-index Go projects after upgrading. +- In Go, a method called further along a chain now links to that method of the type the chain has reached, like `Where` and `First` in a GORM-generated query such as `query.Card.WithContext(ctx).Where(…).First()`, a method called on what a function returns, like the client from `store.New(…)`, or one called on a composite literal, like `GitOracle{}.Worktree()`. The type is read from a function's or method's results, a conversion, a literal or an embedded field. Before, such a call was matched by its method's name alone, so a query's `Where` linked to another table's query type, a literal's method to a type that only shares its name, and `time.Now().UTC().Format(…)` to the one project method named `Format`. A chain whose type comes from outside your project, or can't be worked out, now links to nothing rather than to a guess. Re-index Go projects after upgrading. Thanks @BX-D for the report and @inth3shadows for the extra cases. (#2543) - In Go, a method a type declares in a file too large to index, like the `String`, `Marshal` and `Unmarshal` every Kubernetes API type has in `core/v1/generated.pb.go`, is no longer replaced by the method of that name a type it embeds has. Before, `pod.String()` on a `*v1.Pod` linked to `TypeMeta`'s `String`, a method the call never runs; it now links to nothing, since the pod's own method isn't in the graph. A method the type really gets from a type it embeds, like `GetName` from `ObjectMeta`, still links. Re-index Go projects after upgrading. - In Go, calls into a module whose `go.mod` is not at the project root now resolve, whether it is a `server/` backend next to a `web/` frontend or one of several modules side by side as in etcd, so calls like `store.New()` and `s.db.CreateItem()` find their targets instead of being treated as calls into a third-party package. A name written through a package, like a `job.OPCommand` result type or a field of type `artifact.Manager`, now links to that package's symbol rather than a same-named one elsewhere, which also corrects links in projects with a single `go.mod`. Re-index Go projects after upgrading. Thanks @GoDiao for the report and @danusha2345 for the fix. (#2322) - In Go, an exported type written without a package, like `Node` in `func Walk(v Visitor, node Node)`, now links to the type of that name in its own package, whether it is a struct, an interface or another kind of type. Before, it could link to a struct of the same name in an unrelated package, so prometheus's PromQL parser functions pointed at the Kubernetes discovery `Node` struct instead of the parser's own `Node` interface, or, for a result type like `Appender` in `func (f *fanout) Appender(…) Appender`, to the method itself. A type from a package outside your project, like `apiv1.Node` or `http.Handler`, no longer links to a struct of the same name in the file that uses it; a variadic parameter like `...storage.Filter` now links to that package's type; a route handler written as a method value, like `h.Follow` in an Echo or Gin app, links to that method rather than a model struct named `Follow`; and a method called on another call's result or on an indexed value, like `err[i].Error()` or a chained `.String()`, is no longer recorded as creating a struct of the same name. Re-index Go projects after upgrading. diff --git a/__tests__/go-chained-receivers.test.ts b/__tests__/go-chained-receivers.test.ts new file mode 100644 index 0000000000..9f097df0af --- /dev/null +++ b/__tests__/go-chained-receivers.test.ts @@ -0,0 +1,370 @@ +/** + * A Go method call whose receiver the extractor keeps no name for — a fluent + * chain's later link (`model.WithContext(0).Where(1).First()`), a call on what + * a function returns (`time.Now().UTC().Format(…)`), a composite literal + * (`GitOracle{}.Worktree()`) — reaches the resolver by its bare method name. + * Name matching then picked one of the project's methods of that name by + * proximity: a GORM-style chain's `Where` and `First` went to another query + * type's, a standard-library chain's `Format` to the only project `Format`, + * and a composite literal's method to a namesake type's (#2543). + * + * The receiver is now typed: a call gives its function's or method's first + * result (or the one a multi-value assignment takes), a conversion and a + * composite literal their type, an embedded field its type. The method is the + * one Go selects on that type, and nothing when the type comes from outside + * the project or can't be read. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +const GO_MOD = 'module example.com/app\n\ngo 1.22\n'; + +let tempDir: string; +let cg: CodeGraph | null = null; + +afterEach(() => { + cg?.close(); + cg = null; + if (tempDir) fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +async function index(files: Record): Promise { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2543-')); + for (const [rel, source] of Object.entries({ 'go.mod': GO_MOD, ...files })) { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), source); + } + cg = await CodeGraph.init(tempDir, { index: true }); + cg.resolveReferences(); + return cg; +} + +/** ` ` of every `calls` / `instantiates` edge out of the function `fromName`, sorted. */ +function callees(graph: CodeGraph, fromName: string): string[] { + const from = graph.getNodesByKind('function').find((n) => n.name === fromName); + expect(from, fromName).toBeDefined(); + return graph + .getOutgoingEdges(from!.id) + .filter((e) => e.kind === 'calls' || e.kind === 'instantiates') + .map((e) => { + const target = graph.getNode(e.target); + return `${e.kind} ${target?.kind} ${target?.qualifiedName}`; + }) + .sort(); +} + +describe('Go method calls on a receiver the extractor names no variable for (#2543)', () => { + it("follows the issue's factory → field → fluent chain to the session query type", async () => { + const graph = await index({ + 'query/query.go': 'package query\n\ntype Query struct { Session Session }\n', + 'query/card.go': + 'package query\n\ntype CardDo struct{}\nfunc (c CardDo) Where(value int) CardDo { return c }\nfunc (c CardDo) First() int { return 1 }\n', + 'query/session.go': + 'package query\n\ntype Session struct{}\ntype SessionDo struct{}\n' + + 'func (s Session) WithContext(ctx int) SessionDo { return SessionDo{} }\n' + + 'func (s SessionDo) Where(value int) SessionDo { return s }\nfunc (s SessionDo) First() int { return 2 }\n', + 'mysql/conn.go': 'package mysql\n\nimport "example.com/app/query"\n\nfunc Query() *query.Query { return &query.Query{} }\n', + 'mysql/session.go': + 'package mysql\n\nfunc Find() int {\n model := Query().Session\n return model.WithContext(0).\n Where(1).\n First()\n}\n', + }); + expect(callees(graph, 'Find')).toEqual([ + 'calls function Query', + 'calls method Session::WithContext', + 'calls method SessionDo::First', + 'calls method SessionDo::Where', + ]); + }); + + it("calls the composite literal's own type's method", async () => { + const graph = await index({ + 'main.go': `package main + +type GitOracle struct{} + +func (GitOracle) Worktree() string { return "" } + +type FakeOracle struct{} + +func (FakeOracle) Worktree() string { return "" } + +func use() string { return GitOracle{}.Worktree() } + +func useParen() string { return (&FakeOracle{}).Worktree() } + +func main() { use(); useParen() } +`, + }); + expect(callees(graph, 'use').filter((e) => e.startsWith('calls'))).toEqual(['calls method GitOracle::Worktree']); + expect(callees(graph, 'useParen').filter((e) => e.startsWith('calls'))).toEqual(['calls method FakeOracle::Worktree']); + }); + + it('links nothing for a chain on a standard-library value', async () => { + const graph = await index({ + 'main.go': `package main + +import ( + "net/http/httptest" + "time" +) + +type Report struct{} + +func (r Report) Format() string { return "" } +func (r Report) Get(key string) string { return key } + +func stamp() string { return time.Now().UTC().Format(time.RFC3339) } + +func header() string { + w := httptest.NewRecorder() + return w.Header().Get("Content-Type") +} + +func main() { stamp(); header() } +`, + }); + expect(callees(graph, 'stamp')).toEqual([]); + expect(callees(graph, 'header')).toEqual([]); + }); + + it('types a call result, a multi-value result, a conversion and an embedded field', async () => { + const graph = await index({ + 'store/store.go': `package store + +type KV interface { + Get(key string) string +} + +type Client struct { + KV + name string +} + +func (c *Client) Txn() *Txn { return &Txn{} } + +type Txn struct{} + +func (t *Txn) If(ok bool) *Txn { return t } +func (t *Txn) Commit() error { return nil } + +type Response struct{} + +func (r *Response) Header() string { return "" } + +func New(name string) (*Client, error) { return &Client{name: name}, nil } + +func Begin() (int, *Txn) { return 0, &Txn{} } +`, + 'store/other.go': `package store + +type Other struct{} + +func (o *Other) If(ok bool) *Other { return o } +func (o *Other) Commit() error { return nil } +func (o *Other) Get(key string) string { return key } +func (o *Other) Header() string { return "" } +`, + 'app/app.go': `package app + +import "example.com/app/store" + +type holder struct{ cli *store.Client } + +func run(h holder, raw any) error { + cli, err := store.New("x") + if err != nil { + return err + } + _ = h.cli.KV.Get("a") + _ = (*store.Response)(nil).Header() + _, txn := store.Begin() + _ = txn.If(true).Commit() + return cli.Txn().If(true).Commit() +} +`, + }); + expect(callees(graph, 'run').filter((e) => e.startsWith('calls'))).toEqual([ + 'calls function Begin', + 'calls function New', + 'calls method Client::Txn', + 'calls method KV::Get', + 'calls method Response::Header', + 'calls method Txn::Commit', + 'calls method Txn::Commit', + 'calls method Txn::If', + 'calls method Txn::If', + ]); + }); + + it('types a package-level variable from the call it is set to', async () => { + const graph = await index({ + 'scheme/scheme.go': `package scheme + +type Scheme struct{} + +func New() *Scheme { return &Scheme{} } + +func (s *Scheme) Test(name string) *Tester { return &Tester{} } + +type Tester struct{} + +func (t *Tester) Value(v any) *Tester { return t } +func (t *Tester) ExpectValid() {} +`, + 'scheme/decoy.go': `package scheme + +type Decoy struct{} + +func (d *Decoy) Value(v any) *Decoy { return d } +func (d *Decoy) ExpectValid() {} +`, + 'pkg/doc.go': 'package pkg\n\nimport "example.com/app/scheme"\n\nvar localScheme = scheme.New()\n', + 'pkg/doc_test.go': `package pkg + +func check() { + localScheme.Test("x").Value(1).ExpectValid() +} +`, + }); + expect(callees(graph, 'check')).toEqual([ + 'calls method Scheme::Test', + 'calls method Tester::ExpectValid', + 'calls method Tester::Value', + ]); + }); + + it('types a generic call and a generic literal given their type arguments', async () => { + const graph = await index({ + 'lib/lib.go': `package lib + +type ErrorList []string + +func (l ErrorList) MarkShortCircuit() ErrorList { return l } + +func PtrNoNils[T any](v *T) ErrorList { return nil } + +func Identity[T any](v T) T { return v } + +type Validator interface{ Validate() error } + +func Pick[V Validator](vs []V) V { return vs[0] } + +type List[T any] struct{ items []T } + +func (l List[T]) Len() int { return len(l.items) } +`, + 'lib/decoy.go': `package lib + +type Other struct{} + +func (Other) MarkShortCircuit() Other { return Other{} } +func (Other) Len() int { return 0 } +func (Other) Validate() error { return nil } + +// Named like Identity's type parameter. +type T struct{} + +func (T) Run() {} +`, + 'app/app.go': `package app + +import "example.com/app/lib" + +type A struct{} + +func (A) Run() {} + +type Report struct{} + +func (Report) Mark() {} + +type Box[T any] struct{ v T } + +func (b Box[T]) Size() int { return 0 } + +type Decoy struct{} + +func (Decoy) Run() {} +func (Decoy) Mark() {} +func (Decoy) Size() int { return 0 } + +type Probe struct{} + +func (Probe) Validate() error { return nil } + +func wrap[T any](v T) Report { return Report{} } + +func Check(v *int, probes []Probe) int { + lib.PtrNoNils[int](v).MarkShortCircuit() + lib.Identity[A](A{}).Run() + wrap[int](1).Mark() + _ = lib.Pick(probes).Validate() + return lib.List[int]{}.Len() + Box[int]{}.Size() +} +`, + }); + // `Identity`'s result is its type parameter, not the type `lib` has of + // that name; `Pick`'s has the methods of its constraint. + expect(callees(graph, 'Check').filter((e) => e.startsWith('calls method'))).toEqual([ + 'calls method Box::Size', + 'calls method ErrorList::MarkShortCircuit', + 'calls method List::Len', + 'calls method Report::Mark', + 'calls method Validator::Validate', + ]); + }); + + it('reads a later call of a name the chain already called on another type', async () => { + const graph = await index({ + 'main.go': `package main + +type A struct{} + +func (a *A) Next() *B { return &B{} } + +type B struct{} + +func (b *B) Next() *C { return &C{} } + +type C struct{} + +func walk(a *A) *C { return a.Next().Next() } + +func main() { walk(&A{}) } +`, + }); + expect(callees(graph, 'walk')).toEqual(['calls method A::Next', 'calls method B::Next']); + }); + + it("links nothing when the receiver's type can't be read, rather than guess by name", async () => { + const graph = await index({ + 'main.go': `package main + +type A struct{} + +func (A) Run() {} + +type B struct{} + +func (B) Run() {} + +type holder struct{ get func() A } + +func use(h holder) { + h.get().Run() +} + +func useLiteral() { + var cfg struct{ inner struct{ a A } } + cfg.inner.a.Run() +} + +func main() { use(holder{}); useLiteral() } +`, + }); + expect(callees(graph, 'use')).toEqual([]); + expect(callees(graph, 'useLiteral')).toEqual([]); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 8c5a5899b7..5913fab803 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -1422,6 +1422,11 @@ export class ReferenceResolver { // deepcopy's `(*in)[i].DeepCopyInto(…)`): to a method of the element type. const indexed = matchGoIndexedCall(ref, this.context); if (indexed !== undefined) return indexed; + // And one on any other receiver the extractor keeps no name for: a + // fluent chain's later link, a call's result, a composite literal + // (`model.WithContext(0).Where(1)`, `GitOracle{}.Worktree()`, #2543). + const chained = matchGoChainedCall(ref, this.context); + if (chained !== undefined) return chained; // A section or snippet a Shopify theme's file names is that theme's own, // or nothing: Shopify never looks in another theme (see ./shopify-themes). diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 96524c2b15..d4a0d05a13 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -20497,8 +20497,8 @@ function goAssertedType(ref: UnresolvedRef, context: ResolutionContext): string } interface GoChainLink { - kind: 'operand' | 'select' | 'assert' | 'call' | 'index'; - /** An operand's or selector's name, an assertion's type, what a call's or an index's brackets hold. */ + kind: 'operand' | 'select' | 'assert' | 'call' | 'index' | 'composite'; + /** An operand's or selector's name, an assertion's type, what a call's, an index's or a composite literal's brackets hold. */ text: string; } @@ -20507,8 +20507,10 @@ interface GoChainLink { * or a parenthesized expression), then each selector, type assertion, call * and index. A line break ends it, except right after a `.`, where Go * inserts no semicolon. `end.at` receives the offset reading stopped at. + * With `composite`, a composite literal's braces right after a type + * (`GitOracle{}`, `query.Card{…}`, `List[int]{}`) are a link too. */ -function goChainLinks(text: string, start: number, end?: { at: number }): GoChainLink[] { +function goChainLinks(text: string, start: number, end?: { at: number }, composite = false): GoChainLink[] { const links: GoChainLink[] = []; const NAME = /[A-Za-z_]\w*/y; let i = start; @@ -20548,6 +20550,13 @@ function goChainLinks(text: string, start: number, end?: { at: number }): GoChai const inner = group(); if (inner === null) break; link.text = inner; + } else if ( + composite && text[i] === '{' && text[start] !== '(' && links.length <= 3 && + links.every((l) => l.kind === 'operand' || l.kind === 'select' || l.kind === 'index') + ) { + const inner = group(); + if (inner === null) break; + links.push({ kind: 'composite', text: inner }); } else break; } if (end) end.at = i; @@ -20764,6 +20773,50 @@ export function matchGoIndexedCall(ref: UnresolvedRef, context: ResolutionContex return goMethodOnElement(receiver, ref, context); } +/** + * A Go method called on a receiver the extractor keeps no name for — a + * fluent chain's later link (`model.WithContext(0).Where(1).First()`), a + * call on what a function returns (`time.Now().UTC().Format(…)`, + * `query.Use(db).User.Where(…)`), a composite literal (`GitOracle{}.Worktree()`, + * `(MsgPack{data}).Render(w)`) or a field three links deep — arrives by its + * bare name, at the column where its chain starts (#2543). Name matching + * then picked one of the project's methods of that name by proximity: a + * GORM-style chain's `Where` and `First` went to another query type's, and + * a standard-library chain's `Format` to the only project `Format`. + * + * The receiver is typed as matchGoIndexedCall types one, and further: a + * call gives its function's or method's first result (goCalledType, + * goMethodResultType), a conversion `T(x)` and a composite literal their + * type. The method is then the one Go selects on that type, or nothing when + * the type comes from outside the project. A receiver that can't be typed + * — a func value's call, a generic result, a receiver the index doesn't + * hold — links nothing either: guessing by the name is what picked the + * wrong query type. Undefined only when the call is not such a chain link. + */ +export function matchGoChainedCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + if (ref.language !== 'go' || ref.referenceKind !== 'calls' || !/^[A-Za-z_]\w*$/.test(ref.referenceName)) return undefined; + const index = goScopeIndex(ref.filePath, context); + if (!index || ref.column < 0) return undefined; + const at = (index.lineStarts[ref.line - 1] ?? index.code.length) + ref.column; + const code = index.code; + // A column inside a name is no expression's start, and a call written at + // its own name calls a function. + if (at > 0 && /\w/.test(code[at - 1]!)) return undefined; + const name = ref.referenceName; + if (code.startsWith(name, at) && /^\s*\(/.test(code.slice(at + name.length, at + name.length + 16))) return undefined; + // Every call of a chain shares its column. A first link the extractor + // names (`x.M`, `a.b.M`, `F().M`) arrives under that name, so a bare one + // of the same name is a later link: the last `.name(…)` is read. + const links = goChainLinks(code, at, undefined, true); + let k = -1; + for (let i = links.length - 2; i > 0 && k < 0; i--) { + if (links[i]!.kind === 'select' && links[i]!.text === name && links[i + 1]!.kind === 'call') k = i; + } + if (k < 0) return undefined; + const receiver = goReceiverChainType(links.slice(0, k), code[at] === '(', at, ref.filePath, index, context, 0); + return receiver ? goMethodOnElement(receiver, ref, context) ?? null : null; +} + /** * The method `ref` names on the type of its receiver. A pointer's is the * type pointed to's. @@ -20904,35 +20957,73 @@ function goNamedTypeDecls(t: GoTypeText, context: ResolutionContext): { name: st * reads. `paren` says the first link is a parenthesized expression. */ function goReceiverChainType( - links: GoChainLink[], paren: boolean, at: number, filePath: string, index: GoScopeIndex, + links: GoChainLink[], paren: boolean, at: number, filePath: string, index: GoScopeIndex | null, context: ResolutionContext, depth: number, + /** Which result of the chain's last link, a call returning several values, is wanted. */ + result = 0, ): GoTyped { const head = links[0]; if (!head || head.kind !== 'operand' || depth > GO_VALUE_DEPTH) return undefined; - // A call's result (`x.Items()[0]`) isn't typed here. - if (links.some((l) => l.kind === 'call')) return undefined; + // The result a call link gives: the one asked for when it ends the chain. + const resultOf = (k: number): number => (k === links.length - 1 ? result : 0); let t: GoTyped; let i = 1; - if (paren) { + if (paren && links[1]?.kind === 'call' && /^\*?\s*[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)?$/.test(head.text.trim())) { + // `(*pb.T)(r)`: a conversion to the type the parentheses hold, when they hold one. + const type = goTypeTextAt(head.text, filePath, at, index, context); + const decls = goNamedTypeDecls({ ...type, text: type.text.replace(/^\*/, '') }, context); + if (decls === null) return null; + if (!decls) return undefined; + t = type; + i = 2; + } else if (paren) { t = goExprType(head.text, at, filePath, index, context, depth + 1); } else { - const decl = goLocalDeclAt(index, head.text, at); - if (decl) { + const decl = index ? goLocalDeclAt(index, head.text, at) : undefined; + if (decl && index) { + // A func value's call (`run(x).M()`) is not typed here. + if (links[1]?.kind === 'call') return undefined; t = goDeclaredValueType(decl, filePath, index, context, depth + 1); } else if (context.getImportMappings(filePath, 'go').some((m) => m.localName === head.text)) { - // `pkg.Var`: a package-level variable of the package imported. + // `pkg.Var`, `pkg.New(…)`, `pkg.T{…}`, `pkg.T(x)`: a package-level + // variable, function or type of the package imported. if (links[1]?.kind !== 'select') return undefined; const dir = goImportPackageDir(head.text, filePath, context); if (dir == null) return dir === null && isGoOutsidePackage(head.text, filePath, context) ? null : undefined; - t = goPackageVarTypeText(links[1].text, dir, filePath, true, context, depth + 1); - i = 2; + // Type arguments written at the call or the literal: `pkg.F[T](…)`, `pkg.List[int]{…}`. + const args = goTypeArgsLink(links, 2); + const written = `${head.text}.${links[1].text}${args}`; + const at2 = args ? 3 : 2; + const next = links[at2]?.kind; + if (next === 'composite') t = goTypeTextAt(written, filePath, at, index, context); + else if (next === 'call') t = goCalledType(links[1].text, dir, written, filePath, true, at, index, context, depth + 1, resultOf(at2)); + else t = goPackageVarTypeText(links[1].text, dir, filePath, true, context, depth + 1); + i = next === 'composite' || next === 'call' ? at2 + 1 : 2; + } else if (links[1]?.kind === 'composite' || (links[1]?.kind === 'index' && links[2]?.kind === 'composite')) { + // `T{…}`, `List[int]{…}`. + const args = goTypeArgsLink(links, 1); + t = goTypeTextAt(`${head.text}${args}`, filePath, at, index, context); + i = args ? 3 : 2; + } else if (links[1]?.kind === 'call' || (links[1]?.kind === 'index' && links[2]?.kind === 'call')) { + // `new(T)`, `make(T, …)`, `append(xs, …)`, a function of the package, + // one given its type arguments (`F[T](…)`), a conversion `T(x)`. + const args = goTypeArgsLink(links, 1); + t = !args && /^(?:new|make|append)$/.test(head.text) + ? goExprType(`${head.text}(${links[1].text})`, at, filePath, index, context, depth + 1) + : goCalledType(head.text, goPackageDir(filePath), `${head.text}${args}`, filePath, false, at, index, context, depth + 1, resultOf(args ? 2 : 1)); + i = args ? 3 : 2; } else { t = goPackageVarTypeText(head.text, goPackageDir(filePath), filePath, false, context, depth + 1); } } + if (result > 0 && links[links.length - 1]?.kind !== 'call') return undefined; for (; i < links.length && t; i++) { const link = links[i]!; - if (link.kind === 'select') t = goFieldTypeOf(t, link.text, context, depth + 1); + if (link.kind === 'select' && links[i + 1]?.kind === 'call') { + // `x.M(…)`: what the method returns. + t = goMethodResultType(t, link.text, filePath, context, depth + 1, resultOf(i + 1)); + i++; + } else if (link.kind === 'select') t = goFieldTypeOf(t, link.text, context, depth + 1); else if (link.kind === 'index') t = goIsSliceIndex(link.text) ? goSlicedType(t, context, depth + 1) : goElementOf(t, context, depth + 1); // `.(T)` is a T, written where the expression is. else if (link.kind === 'assert' && link.text !== 'type') t = goTypeTextAt(link.text, filePath, at, index, context); @@ -20941,6 +21032,104 @@ function goReceiverChainType( return t; } +/** + * The type arguments a Go chain's link `k` gives the function or type before + * it, `[T]` of `F[T](…)` or `[int]` of `List[int]{…}`: an index link a call + * or a composite literal follows. Empty for any other link. + */ +function goTypeArgsLink(links: GoChainLink[], k: number): string { + const link = links[k]; + const next = links[k + 1]?.kind; + return link?.kind === 'index' && (next === 'call' || next === 'composite') ? `[${link.text}]` : ''; +} + +/** + * The type a Go call `name(…)` gives, `name` read in the package in `dir` + * (`written` as the call spells it): its function's first result, or the + * type itself for a conversion `T(x)`. A package's declarations of the + * function must agree (build tags); null for a function from outside the + * project, undefined for one the index doesn't hold or a result written + * with the function's type parameters. + */ +function goCalledType( + name: string, dir: string, written: string, fromFile: string, imported: boolean, at: number, index: GoScopeIndex | null, + context: ResolutionContext, depth: number, result = 0, +): GoTyped { + if (depth > GO_VALUE_DEPTH) return undefined; + const clause = imported ? undefined : goPackageClause(fromFile, context); + const testFile = (f: string): boolean => /_test\.go$/.test(f); + const fns = context.getNodesByName(name).filter((n) => + n.language === 'go' && n.kind === 'function' && goPackageDir(n.filePath) === dir && + (imported ? !testFile(n.filePath) : goPackageClause(n.filePath, context) === clause && (testFile(fromFile) || !testFile(n.filePath)))); + if (fns.length === 0) { + // `T(x)`: a conversion, when the package declares the type. + return result === 0 && goPackageTypes(name, dir, context).length > 0 ? goTypeTextAt(written, fromFile, at, index, context) : undefined; + } + let found: GoTyped; + for (const fn of fns) { + const t = goSignatureResultType(fn, context, result); + if (t === undefined || (found !== undefined && found?.text !== t.text)) return undefined; + found = t; + } + return found; +} + +/** + * The type a Go method call `x.name(…)` gives on an `x` of type t: the first + * result of the method Go selects (goMethodOnElement). Null when the method + * is no project type's (t comes from outside the project, or a project type + * gets it from an embedded outside one): what it returns is unknown to the + * project too. + */ +function goMethodResultType(t: GoTypeText, name: string, filePath: string, context: ResolutionContext, depth: number, result = 0): GoTyped { + if (depth > GO_VALUE_DEPTH) return undefined; + const probe: UnresolvedRef = { + fromNodeId: '', referenceName: name, referenceKind: 'calls', line: 1, column: 0, filePath, language: 'go', + }; + const method = goMethodOnElement(t, probe, context); + if (!method) return method; + const node = context.getNodeById?.(method.targetNodeId); + return node ? goSignatureResultType(node, context, result) : undefined; +} + +/** + * The first result type of a Go function or method node, read in its own + * file: `CardDo` of `(value int) CardDo`, `*Response` of `(ctx context.Context) + * (*Response, error)`, `int` of `() (n int, err error)`. Undefined for none. + * The indexed signature leaves a generic function's `[T any]` out, so the + * type parameters in scope at its declaration are read there and kept with + * the result: `T` of `func Get[T any]() T` is no package's type, while + * `Set[T]` is a `Set`. + */ +function goSignatureResultType(fn: Node, context: ResolutionContext, result = 0): GoTypeText | undefined { + const sig = fn.signature?.trim(); + if (!sig || sig[0] !== '(') return undefined; + const close = goClosingBracket(sig, 0); + if (close < 0) return undefined; + let text = sig.slice(close + 1).trim(); + if (!text) return undefined; + if (text[0] === '(') { + const end = goClosingBracket(text, 0); + if (end < 0) return undefined; + const parts = goSplitTopLevel(text.slice(1, end), ',').map((p) => p.trim()).filter(Boolean); + // Named results (`n int, err error`) share a type with the names before + // it (`a, b T`): a name's type is the first one written from it on. + const named = (p: string): string | undefined => { + const m = /^([A-Za-z_]\w*)\s+(\S[\s\S]*)$/.exec(p); + return m && !/^(?:chan|func|map|struct|interface)$/.test(m[1]!) ? m[2] : undefined; + }; + let wanted: string | undefined; + if (parts.some((p) => named(p) !== undefined)) { + for (let r = result; r < parts.length && wanted === undefined; r++) wanted = named(parts[r]!); + } else wanted = parts[result]; + if (!wanted) return undefined; + text = wanted.trim(); + } else if (result > 0) return undefined; + const index = goScopeIndex(fn.filePath, context); + const params = index ? goTypeParamsAt(index, fn.filePath, index.lineStarts[fn.startLine - 1] ?? 0, context) : undefined; + return { text: goNormalizeType(text), file: fn.filePath, params }; +} + /** A type written at offset `at` of a file, or at package level for no `index`. */ function goTypeTextAt(text: string, filePath: string, at: number, index: GoScopeIndex | null, context: ResolutionContext): GoTypeText { return { text: goNormalizeType(text), file: filePath, params: index ? goTypeParamsAt(index, filePath, at, context) : undefined }; @@ -20953,9 +21142,19 @@ function goTypeTextAt(text: string, filePath: string, at: number, index: GoScope */ function goExprType( expr: string, at: number, filePath: string, index: GoScopeIndex | null, context: ResolutionContext, depth: number, + /** For a call that returns several values, the one wanted. */ + result = 0, ): GoTyped { const s = expr.trim(); if (!s || depth > GO_VALUE_DEPTH) return undefined; + if (result > 0) { + // Only a call's results are counted: `v, ok := m[k]` is no call's. + if (!/\)\s*$/.test(s)) return undefined; + const end = { at: 0 }; + const links = goChainLinks(s, 0, end, true); + if (end.at !== s.length || links[links.length - 1]?.kind !== 'call') return undefined; + return goReceiverChainType(links, s.startsWith('('), at, filePath, index, context, depth + 1, result); + } if (s.startsWith('&')) { const t = goExprType(s.slice(1), at, filePath, index, context, depth + 1); return t ? { ...t, text: `*${t.text}` } : t; @@ -20983,14 +21182,19 @@ function goExprType( while (s[j] === ' ' || s[j] === '\t') j++; if ((s[j] === '{' || s[j] === '(') && goClosingBracket(s, j) === s.length - 1) { const type = goTypeTextAt(s.slice(0, typeEnd), filePath, at, index, context); - // `T(x)` is a conversion only when T is a type, not a function. - return s[j] === '{' || GO_TYPE_LITERAL.test(type.text) || goNamedTypeDecls(type, context) ? type : undefined; + if (s[j] === '{' || GO_TYPE_LITERAL.test(type.text)) return type; + // `T(x)` is a conversion only when T is a type. A name from a package + // outside the project, type or function, gives a value of an outside + // type; any other call is read as a call below. + const decls = goNamedTypeDecls(type, context); + if (decls) return type; + if (decls === null && /^[A-Za-z_]\w*\.[A-Za-z_]\w*$/.test(type.text)) return null; } } - if (!index) return undefined; - // A name, a field, an element, an assertion. + // A name, a field, an element, an assertion, a call, a composite literal's: + // at package level (no `index`) a name is never a local. const end = { at: 0 }; - const links = goChainLinks(s, 0, end); + const links = goChainLinks(s, 0, end, true); if (end.at !== s.length) return undefined; return goReceiverChainType(links, s.startsWith('('), at, filePath, index, context, depth + 1); } @@ -21013,7 +21217,10 @@ function goDeclaredValueType(decl: GoDecl, filePath: string, index: GoScopeIndex const values = goSplitTopLevel(rhs, ','); // `v, ok := m[key]`, `v, ok := x.(T)`. const value = values.length === n ? values[k] : values.length === 1 && n === 2 && k === 0 && /[\])]\s*$/.test(rhs) ? rhs : undefined; - return value === undefined ? undefined : goExprType(value, decl.at, filePath, index, context, depth + 1); + if (value !== undefined) return goExprType(value, decl.at, filePath, index, context, depth + 1); + // `c, err := f()`, `_, cli := newClient(…)`: the call's k-th result. + if (values.length === 1 && n > 1 && /\)\s*$/.test(rhs)) return goExprType(rhs, decl.at, filePath, index, context, depth + 1, k); + return undefined; } if (decl.type === undefined) return undefined; // A variadic `xs ...T` is a `[]T`. @@ -21085,6 +21292,12 @@ function goFieldTypeOf(t: GoTypeText, name: string, context: ResolutionContext, if (!struct) continue; const own = struct.members.fields.get(name); if (own !== undefined) found.push({ text: goNormalizeType(own), file: struct.file, params: struct.members.params }); + // An embedded field is named after its type: `client.KV` for `struct { KV; … }`. + for (const e of struct.members.embedded) { + if (goNormalizeType(e).replace(/^\*/, '').replace(/\[[\s\S]*\]$/, '').replace(/^[A-Za-z_]\w*\./, '') === name) { + found.push({ text: goNormalizeType(e), file: struct.file, params: struct.members.params }); + } + } for (const e of struct.members.embedded) { if (seen.has(`${struct.file}\0${e}`)) continue; seen.add(`${struct.file}\0${e}`); @@ -21285,7 +21498,8 @@ function goTypeParamsAt(index: GoScopeIndex, filePath: string, at: number, conte if (!params) { params = new Map(); const lineStart = index.lineStarts[fn.startLine - 1] ?? 0; - const head = index.code.slice(lineStart, index.lineStarts[Math.min(fn.endLine, fn.startLine + 40) - 1] ?? index.code.length); + // Through the end line: a one-line function's head is its only line. + const head = index.code.slice(lineStart, index.lineStarts[Math.min(fn.endLine, fn.startLine + 40)] ?? index.code.length); // `func Name[T any](`, or `func (s *Set[T, U]) Name(`, whose type // arguments name its type's parameters, constrained where the type is. const generic = /^\s*func\s*(?:\(\s*(?:[A-Za-z_]\w*\s+)?\*?\s*([A-Za-z_]\w*)\s*(\[)|[A-Za-z_]\w*\s*(\[))/.exec(head); From 1890b10cc4929b5239876d9af7b3034208357111 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 06:00:50 +0000 Subject: [PATCH 20/95] fix(explore): a named function too long for its share renders instead of the file's tail (#2783) Root cause: an oversize cluster keeps its first-ranked member however big it is, and the ceiling windows it (CG-30). Ties on importance went to the shorter span, so in the issue's file, one 200-line function plus the line that calls it, the one-line `path = build_report()` took that guarantee. The search matches it through its signature, so it is an entry point too. The function the query named then no longer fit beside it and was dropped, and `codegraph_explore build_report` returned the file's last six lines under the verbatim-source header. A file asked for by path did the same, because every definition in it ranks below the bare edge lines. Where a body was windowed, nothing in the block said so. The footer's advice, ask for the symbol by name, returned the same window, and it suggested `codegraph_node`, which the default MCP surface does not list. Fix, all in the explore render path (query time only): - Rule 1: among cluster members of one importance, a definition the query named ranks ahead of one its search only matched, before the shorter span. - Rule 2: a fragment does not spend the oversize guarantee. When the kept members render under MIN_CHARS, the member they priced out is kept and windowed like a first-ranked one. That member must be one the query named, or a definition in a file pinned by path with nothing named in the cluster. - Rule 3: a body the ceiling window cuts ends with the per-symbol view's note, "... lines X-Y of `name` elided - codegraph_explore `file:X-Y` returns them" (one shared helper now). Each note is paid for by its own part's last lines, never a focus line, the member's first line or a member the part shows whole. So no section grows and no other file's budget moves. - Rule 4: the trim footer and the dynamic-boundary note name `codegraph_node` only when it is on the served tool surface. Validation: - New explore-queried-oversize-symbol.test.ts, 9 tests: 6 fail on main. Reverting any one rule fails that rule's own tests (2, 2, 3 and 1). - All 37 explore suites pass (414 tests). - Deterministic A/B: one index per repo, a fresh handler per query, five arms (main, then each rule added in turn). 820 queries per arm: the 50 longest functions by name, their files by path, caller/callee pairs, call chains, random names and flow questions. - The queried function's head (def plus 10 lines) renders, main -> fix: express 50 -> 50, flask 49 -> 50, excalidraw 33 -> 48, django 43 -> 47, airflow 43 -> 48, of 50 each. - Queried body lines rendered, main -> fix: excalidraw 6,615 -> 9,391, django 5,337 -> 5,662, airflow 6,320 -> 6,749. - Rule 1 changed 32 responses, each hand-triaged as the queried function now rendering. Only 4 of the 504 ordinary flow queries are among them. - Rule 2 changed none. - Rule 3 wrote 838 notes, all checked against source and index, at 2.8 to 3.9 source lines each. Bytes a note frees flow to lower files by the existing carry-forward. - Rule 4's 236 diffs are exactly the removed mention. - No response exceeds the 25,000-char inline cap. Fixes #2486 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../explore-queried-oversize-symbol.test.ts | 223 ++++++++++++++++++ src/mcp/tools.ts | 164 ++++++++++++- 3 files changed, 377 insertions(+), 11 deletions(-) create mode 100644 __tests__/explore-queried-oversize-symbol.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 722fe254e4..4251151a4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -218,6 +218,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph_explore`'s "Dynamic-dispatch links among your symbols" list now shows every link on its own line. Before, the links from an interface method to its implementations all read alike, like `TrySet → TrySet`, so only one was listed, chosen by the order of the index rather than by your query: asking about gin's `tryToSetValue setter.TrySet multipartRequest.TrySet setByForm` showed `headerSource`'s implementation and hid `formSource`'s. Both ends of such a link are now named with their type, like `setter.TrySet → formSource.TrySet`. When there are more links than the list shows, the implementations your query names come first and those in tests or examples last, and a hop the Flow already shows is not repeated. - `codegraph_explore`, `codegraph_node`, `codegraph_callers`, `codegraph_callees`, `codegraph_impact` and the `codegraph callers`, `callees` and `impact` commands now find a symbol written with its package, the way Go code names it: `refresh.Discovery.refresh`, `storage.Queryable.Querier`, or `gin.Context.Next` in gin's root package. Before, such a name found nothing, so a question written that way lost the symbol and explore's Flow had nothing to connect. A name the search ranked below other results, like gin's `binding.Default`, is found as well, and a Python or Rust module followed by a class and method works the same way. - When `codegraph_explore` finds more dynamic-dispatch links than its list shows, the symbols you named now take turns, so each one's links make the list: asking about prometheus's `refresh.Discovery.refresh hetzner.refresher.refresh hcloudDiscovery.refresh` now lists hetzner's `robotDiscovery` beside the links into other discovery packages that used to fill the list. +- `codegraph_explore` now shows a function you name even when it is too long for its share of the answer: its definition and the start of its body, with a note where the body is cut that says which lines were left out and which `codegraph_explore` query returns them. Before, a file made of one long function and a line that calls it could come back as just its last few lines, without the function, and asking for the file by path did the same. A trimmed answer also no longer suggests `codegraph_node`, which isn't offered by default. Thanks @Huy3ko for the report and @danusha2345 for the first fix. (#2486) - In C and C++, a struct, class, union or enum defined in a variable declaration, like `struct Foo { … } foo;`, is now indexed with its methods and enum values, next to the variable. Before, at most the variable was, so the type was missing from search and `codegraph_explore`, a class deriving from it had no base, and calls to its methods linked to nothing: in redis, the Sentinel's `sentinelState` and `redis-cli`'s `config` struct were missing. A struct, union or enum with no name of its own now takes the name of the variable it declares, so redis's `static struct { … } SPT;` is found as `SPT`, and one declared inside a function is named after its variable instead of ``. The comment above the declaration becomes the type's description. Re-index C and C++ projects after upgrading. - In Objective-C, a struct, union or enum defined in a variable declaration, like `static struct { Class isa; } placeholder;` or `enum { stateType, stateParamName } state;`, is now indexed with its enum values, next to the variable, as in C and C++. Before, only the variable was, so the type was missing from search and `codegraph_explore`. The same went for a `typedef enum { … } Name;` or `typedef struct … Name;` written after a line holding only a macro, like `OF_ASSUME_NONNULL_BEGIN` or `RCT_EXTERN_C_BEGIN`: ObjFW's `OFZIPArchiveEntryCompressionMethod` and every value in it were missing, and so was React Native's `RCTMethodInfo`. A struct, union or enum with no name of its own takes the name of the variable it declares, so ObjFW's class-cluster placeholders are found as `placeholder`, one declared inside a function or method is named after its variable instead of ``, and the comment above the declaration becomes the type's description. As in C, a type that a `.m` file defines links only from that file, so a call elsewhere that just shares its name, like ObjFW's calls of AmigaOS's `Lock(path, mode)`, doesn't link to the runtime's private `struct Lock`. Re-index Objective-C projects after upgrading. - In Objective-C, a macro written alone on its line, like `NS_ASSUME_NONNULL_BEGIN`, `OF_SUBCLASSING_RESTRICTED` or React Native's `RCT_EXTERN_C_BEGIN`, no longer hides the declaration below it. Before, the parser read such a macro as the start of a declaration, so an `@interface` under it could come out as a function and a `typedef` under it as two variables: many of ObjFW's classes, with their properties and methods and the classes they extend and protocols they adopt, were missing, and so were classes and protocols in React Native and KSCrash headers. A message to `super` from a class whose `@interface` follows an attribute macro like `NS_SWIFT_NAME(…)` still reaches the superclass. Re-index Objective-C projects after upgrading. diff --git a/__tests__/explore-queried-oversize-symbol.test.ts b/__tests__/explore-queried-oversize-symbol.test.ts new file mode 100644 index 0000000000..d9eb00fe5f --- /dev/null +++ b/__tests__/explore-queried-oversize-symbol.test.ts @@ -0,0 +1,223 @@ +/** + * A symbol the query asks for renders even when it is too long for its file's + * share of the answer, and the block says where it was cut (#2486). + * + * `shrinkCluster` keeps the first-ranked member of an oversize cluster however + * big it is and lets the ceiling window it (CG-30). Among members of one + * importance, the shorter span ranked first. So in a file that is one long + * function plus the line that calls it, + * + * def build_report(...): # 200 statements, 15 KB + * ... + * if __name__ == '__main__': + * path = build_report() # a search hit too: its signature names it + * + * the one-line `path` took that guarantee, the function then did not fit beside + * it, and `codegraph_explore build_report` returned the file's last six lines, + * under a header calling the block the verbatim source. Asking for the file by + * path did the same. A body the window does cut now carries the line range and + * the explore query that returns the rest, since asking for the symbol by name + * again returns the same window. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler } from '../src/mcp/tools'; + +const STATEMENTS = 200; +const DEF = "def build_report(output_path: str = 'reports/report.pdf'):"; + +/** The issue's own reproducer, line for line. */ +function reportSource(): string { + return [ + '"""Generate the phase-1 validation report as PDF."""', + 'from reportlab.platypus import SimpleDocTemplate', + '', + '', + DEF, + ' """Build the phase-1 validation report as a PDF."""', + ' doc = SimpleDocTemplate(output_path)', + ' story = []', + ...Array.from({ length: STATEMENTS }, (_, i) => + ` story.append(Paragraph(f'section ${i} of the report', styles['BodyText']))`), + ' doc.build(story)', + ' return output_path', + '', + '', + "if __name__ == '__main__':", + ' path = build_report()', + ' print(path)', + '', + ].join('\n'); +} + +/** + * The long function, then a 27-line function the search matches only through + * its docstring. Not a fragment, so nothing but rank decides which of the two + * the cluster keeps. + */ +function publishSource(): string { + return [ + '"""Report generation."""', + '', + '', + DEF, + ' """Build the phase-1 validation report as a PDF."""', + ' story = []', + ...Array.from({ length: 260 }, (_, i) => + ` story.append(Paragraph(f'section ${i} of the report body', styles['BodyText']))`), + ' return output_path', + '', + '', + 'def publish():', + ' """Run build_report and copy the result where readers find it."""', + ' target = build_report()', + ...Array.from({ length: 24 }, (_, i) => ` print('publishing step ${i} of the release checklist', target)`), + ' return target', + '', + ].join('\n'); +} + +/** A constant the query can name, right above a long function that has nothing to do with it. */ +function limitsSource(): string { + return [ + '"""Limits for the report."""', + '', + 'MAX_SECTIONS = 5', + '', + 'def render_everything(doc):', + ...Array.from({ length: STATEMENTS }, (_, i) => + ` doc.write(f'unrelated statement number ${i} of the long neighbour function')`), + ' return doc', + '', + ].join('\n'); +} + +type Project = { dir: string; cg: CodeGraph }; + +async function project(files: Record): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2486-')); + for (const [rel, text] of Object.entries(files)) fs.writeFileSync(path.join(dir, rel), text); + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + return { dir, cg }; +} + +function destroy(p: Project | undefined): void { + if (!p) return; + p.cg.destroy(); + fs.rmSync(p.dir, { recursive: true, force: true }); +} + +/** A handler per call: a session that already holds a file's lines elides them. */ +async function explore(p: Project, query: string): Promise { + const result = await new ToolHandler(p.cg).execute('codegraph_explore', { query }); + return result.content?.[0]?.text ?? ''; +} + +/** The `\t` form explore renders a source line in. */ +const rendered = (n: number, line: string): string => `${n}\t${line}`; + +describe('#2486 — the queried symbol renders when it alone overruns its file\'s budget', () => { + let report: Project | undefined; + let byName = ''; + // The default tool surface: a shell that exports an allowlist must not change what this asserts. + const allowlist = process.env.CODEGRAPH_MCP_TOOLS; + + beforeAll(async () => { + delete process.env.CODEGRAPH_MCP_TOOLS; + report = await project({ 'report_gen.py': reportSource() }); + byName = await explore(report, 'build_report'); + }, 120_000); + afterAll(() => { + destroy(report); + if (allowlist === undefined) delete process.env.CODEGRAPH_MCP_TOOLS; + else process.env.CODEGRAPH_MCP_TOOLS = allowlist; + }); + + it('fixture shape: one function far bigger than the whole response budget, beside a one-line search hit', () => { + const nodes = report!.cg.getNodesInFile('report_gen.py'); + const fn = nodes.find((n) => n.name === 'build_report' && n.kind === 'function')!; + expect(fn.startLine).toBe(5); + expect(fn.endLine).toBe(STATEMENTS + 10); + expect(reportSource().length).toBeGreaterThan(13_000); + // The search matches the variable through its signature, `= build_report()`. + const hit = nodes.find((n) => n.name === 'path')!; + expect(hit.startLine).toBe(hit.endLine); + expect(hit.startLine - fn.endLine).toBeLessThan(7); + }); + + it('renders the definition and the head of its body, not the file tail', () => { + expect(byName).toContain(rendered(5, DEF)); + expect(byName).toContain("section 0 of the report'"); + expect(byName.length).toBeLessThanOrEqual(19_500); + }); + + it('says inside the block where the body was cut, and that query returns the rest', async () => { + const note = byName.match(/… lines (\d+)-(\d+) of `build_report` elided — codegraph_explore `(report_gen\.py:\d+-\d+)` returns them/); + expect(note, 'no cut note in:\n' + byName.slice(-1500)).not.toBeNull(); + const [, from, to, followUp] = note!; + expect(Number(to)).toBe(STATEMENTS + 10); + // The note sits right after the last line shown. + expect(byName).toContain(`${Number(from) - 1}\t`); + expect(byName).not.toContain(`\n${from}\t`); + const rest = await explore(report!, followUp!); + expect(rest).toContain(rendered(Number(from), ` story.append(Paragraph(f'section ${Number(from) - 9} of the report', styles['BodyText']))`)); + expect(rest).toContain(rendered(STATEMENTS + 10, ' return output_path')); + }); + + it('names no follow-up tool the default surface does not list', () => { + expect(byName).toContain('Some file sections were trimmed for size'); + expect(byName).not.toContain('codegraph_node'); + }); + + it('a file asked for by path renders from its first definition, not its tail', async () => { + const byPath = await explore(report!, 'report_gen.py'); + expect(byPath).toContain(rendered(5, DEF)); + expect(byPath).toContain("section 0 of the report'"); + expect(byPath).toContain('of `build_report` elided — codegraph_explore `report_gen.py:'); + expect(byPath.length).toBeLessThanOrEqual(19_500); + }); + + it('a query naming the one-liner too still renders the function', async () => { + const both = await explore(report!, 'build_report path'); + expect(both).toContain(rendered(5, DEF)); + expect(both).toContain("section 0 of the report'"); + }); + + it('still names codegraph_node when the tool is served', async () => { + process.env.CODEGRAPH_MCP_TOOLS = 'explore,node'; + try { + expect(await explore(report!, 'build_report')).toContain('(or `codegraph_node`)'); + } finally { + delete process.env.CODEGRAPH_MCP_TOOLS; + } + }); +}); + +describe('#2486 — a name the query spelled outranks a search hit of the same importance', () => { + let publish: Project | undefined; + afterAll(() => destroy(publish)); + + it('renders the named function, not only the docstring match beside it', async () => { + publish = await project({ 'report_gen.py': publishSource() }); + const text = await explore(publish, 'build_report'); + expect(text).toContain(rendered(4, DEF)); + expect(text).toContain("section 0 of the report body'"); + expect(text).toContain('of `build_report` elided — codegraph_explore `report_gen.py:'); + }, 120_000); +}); + +describe('#2486 — what the query did not ask for is left as it was', () => { + let limits: Project | undefined; + afterAll(() => destroy(limits)); + + it('a one-line constant the query names renders without the long function beside it', async () => { + limits = await project({ 'limits.py': limitsSource() }); + const text = await explore(limits, 'MAX_SECTIONS'); + expect(text).toContain(rendered(3, 'MAX_SECTIONS = 5')); + expect(text).not.toContain('unrelated statement number 100'); + }, 120_000); +}); diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index e952492ada..ffba2116a6 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -1050,6 +1050,16 @@ export function formatGapMarker( return `\n\n... (gap: ${names}) ...\n\n`; } +/** + * The note a body cut short carries where it is cut: which of its lines were + * left out, and the explore query that returns them. A line range is a query + * explore renders as exactly that span, so the rest is one call away; asking + * for the symbol by name again would return the same window (#2486). + */ +export function elidedBodyNote(filePath: string, name: string, from: number, to: number): string { + return `\n… lines ${from}-${to} of \`${name}\` elided — codegraph_explore \`${filePath}:${from}-${to}\` returns them`; +} + /** * Join rendered parts with gap markers that name whatever the trim skipped. * @@ -2149,6 +2159,19 @@ export class ToolHandler { return !allow || allow.has(name.replace(/^codegraph_/, '')); } + /** + * Whether a tool is on the surface `tools/list` shows: the allowlist when one + * is set, the default surface otherwise. Not the same question as + * `isToolAllowed` (with no allowlist, every defined tool may still be + * called). Explore names a follow-up tool only when the agent can see it: + * `codegraph_node` is not on the default surface (#2486). + */ + private isToolServed(name: string): boolean { + const short = name.replace(/^codegraph_/, ''); + const allow = this.toolAllowlist(); + return allow ? allow.has(short) : DEFAULT_MCP_TOOLS.has(short); + } + /** * Get tool definitions with dynamic descriptions based on project size. * The codegraph_explore tool description includes a budget recommendation @@ -3674,7 +3697,7 @@ export class ToolHandler { '', ...notes, '', - '> These sites choose their call target at runtime (registry / bus / reflection) — the site shown IS where the flow continues. To follow it, run codegraph_explore or codegraph_node on a candidate; source for the sites above is included below.', + `> These sites choose their call target at runtime (registry / bus / reflection) — the site shown IS where the flow continues. To follow it, run codegraph_explore${this.isToolServed('node') ? ' or codegraph_node' : ''} on a candidate; source for the sites above is included below.`, '', ].join('\n'); } @@ -5819,7 +5842,17 @@ export class ToolHandler { // renders. Parts are SPAN-KEYED: every part carries the exact line range // its text was sliced from (see `buildSection`). const SPINE_WINDOW = 28; // lines each side of a focus line (the next-hop call site) - type SectionPart = { range: ExploreLineRange; text: string }; + type SectionPart = { + range: ExploreLineRange; + text: string; + /** + * Set on a part a ceiling window cut inside a member's body: the member's + * name and lines. Its `text` then ends with the note `noteCutBodies` + * writes, and `renoteCutBodies` rewrites that note when the part after it + * is dropped. + */ + cut?: { host: string; hostStart: number; hostEnd: number }; + }; /** * Bounded overshoot for one cluster's render (CG-30). * @@ -6162,9 +6195,7 @@ export class ToolHandler { const windows = bodyWindows.get(n.id)!; windows.forEach((w, k) => { const holeEnd = (windows[k + 1]?.start ?? n.endLine + 1) - 1; - const hole = holeEnd > w.end - ? `\n… lines ${w.end + 1}-${holeEnd} of \`${n.name}\` elided — codegraph_explore \`${filePath}:${w.end + 1}-${holeEnd}\` returns them` - : ''; + const hole = holeEnd > w.end ? elidedBodyNote(filePath, n.name, w.end + 1, holeEnd) : ''; skel.push({ range: w, text: renderSpan(w) + hole }); }); coveredUntil = n.endLine; @@ -6453,7 +6484,13 @@ export class ToolHandler { // qualified `SQLCompiler.as_sql` is no longer outranked by a denser cluster // around `SQLInsertCompiler.as_sql` that merely matched the bare name. const EXACT_IMPORTANCE = 11; - const ranges: Array<{ start: number; end: number; name: string; kind: string; importance: number; spine: boolean; spineCallLine?: number; qualifiedName?: string }> = [...rangeNodes.values()] + const ranges: Array<{ + start: number; end: number; name: string; kind: string; importance: number; spine: boolean; spineCallLine?: number; qualifiedName?: string; + /** A definition the query spelled by name, not one its search only matched. */ + named?: boolean; + /** A symbol's own range, as opposed to an edge line or an anchored span. */ + definition?: boolean; + }> = [...rangeNodes.values()] // Drop whole-file envelope nodes (containers covering >50% of the file). .filter(n => !(ENVELOPE_KINDS.has(n.kind) && (n.endLine - n.startLine + 1) > fileLines.length * 0.5)) .map(n => { @@ -6468,7 +6505,11 @@ export class ToolHandler { // processRunExecutionData, the named flow ENTRY at L1562, is a large // low-density method that lost the budget to denser blocks and got cut, so // the agent Read it back — the very thing explore exists to prevent). - return { start: n.startLine, end: n.endLine, name: n.name, kind: n.kind, importance, spine: flow.pathNodeIds.has(n.id), spineCallLine: flow.spineCallSites.get(n.id), qualifiedName: n.qualifiedName }; + return { + start: n.startLine, end: n.endLine, name: n.name, kind: n.kind, importance, + spine: flow.pathNodeIds.has(n.id), spineCallLine: flow.spineCallSites.get(n.id), qualifiedName: n.qualifiedName, + named: namedSeedIds.has(n.id) || flow.namedNodeIds.has(n.id), definition: true, + }; }); // Add edge source locations in this file — captures template references @@ -6764,11 +6805,22 @@ export class ToolHandler { // ahead of spine because the exact branch below keeps members in this // order until the cap: a long spine method first would price out the // body the agent singled out. + // + // Among members of one importance, a definition the query spelled by + // name goes ahead of one its search only matched, and only then does the + // shorter span win. The first member is the one kept however big it is, + // so this decides which symbol that guarantee covers. Shorter-first gave + // it to `path = build_report()` — a search hit on its signature, one + // line — and `build_report`, the 200-line function the query named, + // then no longer fit and the section was the file's last six lines + // (#2486). const byImportance = [...c.members].sort((a, b) => (guarded ? Number(isProtectedMember(b)) - Number(isProtectedMember(a)) : 0) || Number(b.importance >= EXACT_IMPORTANCE) - Number(a.importance >= EXACT_IMPORTANCE) || (b.spine ? 1 : 0) - (a.spine ? 1 : 0) - || b.importance - a.importance || (a.end - a.start) - (b.end - b.start) || a.start - b.start); + || b.importance - a.importance + || Number(!!b.named) - Number(!!a.named) + || (a.end - a.start) - (b.end - b.start) || a.start - b.start); const keep: ExploreRange[] = []; if (c.maxImportance >= EXACT_IMPORTANCE) { // An EXACT target prices EVERY member in rendered chars, protected @@ -6798,13 +6850,23 @@ export class ToolHandler { // exact cost of the protected set; each incidental member then adds its // own padded span, an over-estimate that errs toward the cluster below. let rendered: number | undefined; + // The best-ranked member dropped for size that the query asked for: + // one it named, or — in a file it pinned by path and named nothing in + // this cluster of — a definition. See below the loop. Not the second + // kind while a cluster below is owed room (`guarded`): nothing here + // outranks what that cluster's named members are waiting for. + const askedForInFile = !guarded && pinnedSet.has(filePath) && !c.members.some(isProtectedMember); + let droppedAskedFor: ExploreRange | undefined; for (const r of byImportance) { const sz = sizeOf(r) + GAP_MARKER.length; // Always keep the most important range, even if it alone is oversize — // an empty section sends the agent to Read, which costs far more. How // far it may overshoot is bounded by the caller's ceiling (CG-30), which // windows a runaway member instead of dropping it. - if (keep.length > 0 && kept + sz > cap) continue; + if (keep.length > 0 && kept + sz > cap) { + if (!droppedAskedFor && (r.named || (askedForInFile && r.definition))) droppedAskedFor = r; + continue; + } if (guarded && keep.length > 0 && !isProtectedMember(r)) { rendered ??= sectionText(dedupeSpans(partsOfMembers(keep)).parts).length; const cost = paddedCost(r) + GAP_MARKER.length; @@ -6814,6 +6876,18 @@ export class ToolHandler { keep.push(r); kept += sz; } + // A fragment does not spend the guarantee above. When what was kept + // renders to less than a readable slice, the member it priced out is + // kept the way a first-ranked one is, and the caller's ceiling windows + // it. Two shapes reach this: the query names a one-liner beside the + // long function it also names (`build_report path`), and a file asked + // for by path, where every definition ranks below the bare edge lines + // (importance 1 against 2) — `report_gen.py` kept `path = build_report()` + // and its call edge and dropped the function around them (#2486). + if (droppedAskedFor + && sectionText(dedupeSpans(partsOfMembers(keep)).parts).length < EXPLORE_ALLOCATION.MIN_CHARS) { + keep.push(droppedAskedFor); + } } if (keep.length === c.members.length) return null; return partsOfMembers(keep); @@ -6844,6 +6918,66 @@ export class ToolHandler { return [...exactFocus, ...(c.spineCallLine ? [c.spineCallLine] : []), ...named]; }; + /** + * The note a part's `cut` reads as, for the part's lines ending at `end`: + * the member's lines from there to the next part shown, or to its end. + */ + const cutNoteOf = (p: SectionPart, next: SectionPart | undefined, end: number): string => { + const cut = p.cut!; + const holeEnd = Math.min(cut.hostEnd, next ? next.range.start - 1 : Infinity); + return holeEnd > end ? elidedBodyNote(filePath, cut.host, end + 1, holeEnd) : ''; + }; + /** + * Each cut part's note written again from its `cut`, for after a part + * behind it was dropped: the hole it names now runs on to the next part + * that is left. + */ + const renoteCutBodies = (parts: ReadonlyArray): SectionPart[] => + parts.map((p, i) => (p.cut ? { ...p, text: renderSpan(p.range) + cutNoteOf(p, parts[i + 1], p.range.end) } : p)); + /** + * `windowed` (a ceiling window of `original`) with a note wherever the + * window cut a part short inside a member's body. The innermost member + * wins, so a method cut inside a class names the method. A part that ends + * where its source part ends was not cut, so a gap the member shrink left + * between whole members is never noted: its gap marker names it already. + * + * Each note is paid by the part it ends, which gives up lines from its end + * until it is no longer than it was — never its member's first line, a + * focus line, a line of a window's minimum, or a line of a member the part + * shows whole (a class's note must not cut a method inside it that was + * complete). So a note costs a line or two of the body it is cut from and + * nothing anywhere else: the window's spare room is what the files below + * draw on, and on a saturated response it is spoken for. A part that + * cannot pay goes out as it was, without a note. + */ + const noteCutBodies = ( + windowed: ReadonlyArray, original: ReadonlyArray, + c: ExploreCluster, focus: ReadonlyArray, + ): SectionPart[] => { + const marked = windowed.map((w): SectionPart => { + const source = original.find((p) => p.range.start <= w.range.start && w.range.end <= p.range.end); + if (!source || w.range.end >= source.range.end) return w; + const host = c.members + .filter((m) => m.definition && m.start <= w.range.end && m.end > w.range.end) + .sort((a, b) => (a.end - a.start) - (b.end - b.start))[0]; + return host ? { ...w, cut: { host: host.name, hostStart: host.start, hostEnd: host.end } } : w; + }); + return marked.map((p, i) => { + if (!p.cut) return p; + let lowest = Math.max(p.range.start + MIN_WINDOW_LINES - 1, p.cut.hostStart); + for (const l of focus) if (l >= p.range.start && l <= p.range.end) lowest = Math.max(lowest, l); + for (const m of c.members) { + if (m.definition && m.start >= p.range.start && m.end <= p.range.end) lowest = Math.max(lowest, m.end); + } + for (let end = p.range.end; end >= lowest; end--) { + const range = { start: p.range.start, end }; + const text = renderSpan(range) + cutNoteOf(p, marked[i + 1], end); + if (text.length <= p.text.length) return { range, text, cut: p.cut }; + } + return { range: p.range, text: p.text }; + }); + }; + /** * One cluster's final parts: built, shrunk if it overruns `cap`, then * passed through the session history (CG-18). @@ -6892,7 +7026,12 @@ export class ToolHandler { if (!Number.isFinite(ceiling) || sectionText(r.parts).length <= ceiling) return r; // Windows are subsets of spans dedupeSpans already cleared, so the record // still only ever claims source that was actually sent. - const parts = windowToCeiling(r.parts, ceiling, focusLinesOf(c)); + // A body the window cuts says so where it is cut, with the query for + // the rest (#2486): the agent otherwise learns of the cut only from a + // footer whose advice — ask for the symbol by name — returns this same + // window. + const focus = focusLinesOf(c); + const parts = noteCutBodies(windowToCeiling(r.parts, ceiling, focus), r.parts, c, focus); return { parts, covered: r.covered, shrunk: true }; }; if (sectionText(base.parts).length <= Math.min(cap, guard ?? Infinity)) { @@ -7252,6 +7391,9 @@ export class ToolHandler { if (!holdsFocus(reshrunk.parts[k]!)) { drop = k; break; } } reshrunk.parts.splice(drop, 1); + // The part before the dropped one may have noted a cut that now runs + // on through the dropped part's lines. + reshrunk.parts = renoteCutBodies(reshrunk.parts); reshrunk.shrunk = true; } renderedClusters.set(idx, reshrunk); @@ -7414,7 +7556,7 @@ export class ToolHandler { ? exploreCompletenessNotes(filesIncluded, trimmedShown, [...renderedFilePaths, ...fileGroups.keys()]) .map((note) => ['', '---', note]) : anyFileTrimmed || trimmedShown.length > 0 - ? [['', `> Some file sections were trimmed for size. Elided symbols are named inside gap markers as \`name (file:line)\` and preferred in the file header — run another \`codegraph_explore\` (or \`codegraph_node\`) with those exact names for their source.`]] + ? [['', `> Some file sections were trimmed for size. Elided symbols are named inside gap markers as \`name (file:line)\` and preferred in the file header — run another \`codegraph_explore\`${this.isToolServed('node') ? ' (or `codegraph_node`)' : ''} with those exact names for their source.`]] : []; /** Whether a trimmed section survives in `text` — picks a fallback note's wording after a cut. */ const trimmedIn = (text: string): boolean => From 800e582d5513ce0de6648f10717582f20da3ebc6 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 06:19:32 +0000 Subject: [PATCH 21/95] fix(go): conversions, external test packages and build-tag twins link as Go builds them (#2784) Three Go shapes recorded edges the source doesn't make. A conversion like Category(s), (*Pair)(p) or model.Kind(3) became a calls edge to the type, or an instantiates edge to a struct. A bare call in a package's external tests (package a_test) could bind to a same-named helper of package a in the same directory, which a_test can't see. And a name a package defines once per build target, like sep in sep_unix.go and sep_other.go, linked to whichever definition came first, so a Linux caller reached the !unix file. A Go call whose target is a type is now a references edge. Two files of one directory under different package clauses see each other's top-level names only through an import: qualified, or bare after a dot import and exported. A reference whose definition has build-constrained twins in its package reaches the one that builds for the most typical target its own file builds for: linux/amd64, then darwin/arm64, windows/amd64 and the rest. Constraints are read from the file name's GOOS/GOARCH suffix and the //go:build or // +build lines, with tags as go build sets them by default. Validation: 9 new tests in three files, all failing on main, and each part switched off fails its own file; the Go tests pass. A/B against main 0c3e12dd, edges changed as conversions / twin retargets: gin 56/0, etcd 556/89, prometheus 812/5270, sonic 49/0, x/sys 550/3072, gopsutil 31/179, fsnotify 3/7, kubernetes 16039/328. Nodes and refs are unchanged, and express and flask dumps are byte-identical. Every kind change sits on a conversion-shaped call, and the twin retargets read right by their files' build lines (prometheus labels to the default stringlabels file, x/sys syscalls to syscall_unix_gc.go). Indexing cost, wall/CPU seconds, median of 3: x/sys 6.7/20.5 -> 7.1/21.7, etcd 9.6/29.9 -> 9.8/30.0, prometheus 12.5/49.2 -> 12.9/50.2; kubernetes, 2 runs each, 108.1/385.8 -> 110.4/390.0. Fixes #2581 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/go-build-constraint-twins.test.ts | 110 ++++++++++ __tests__/go-conversion-references.test.ts | 114 ++++++++++ __tests__/go-external-test-package.test.ts | 145 +++++++++++++ __tests__/go-framework-own-package.test.ts | 3 +- src/resolution/index.ts | 16 +- src/resolution/name-matcher.ts | 220 +++++++++++++++++++- 7 files changed, 601 insertions(+), 8 deletions(-) create mode 100644 __tests__/go-build-constraint-twins.test.ts create mode 100644 __tests__/go-conversion-references.test.ts create mode 100644 __tests__/go-external-test-package.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 4251151a4f..3dedebbd8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,6 +56,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a call through a variable that holds the result of a type assertion, like `st.Fetch(key)` after `st := v.(storage.Store)` or `rc.Read(p)` after `rc, ok := v.(ReadCloser)`, now links to that method of the asserted type, as a call made through the assertion itself does, including one the type gets from a struct or interface it embeds, and to nothing when the type comes from outside your project. Before, such a call linked to whichever method shared the name or to nothing, and a variable named like a standard-library package, like grpc-go's `parser, ok := builder.(balancer.ConfigParser)`, was taken for that package. Re-index Go projects after upgrading. - In Go, a method called on an element of a slice, array or map, like kubectl's `schemas[ix].ValidateBytes(data)`, `pods[i].DeepCopy()`, `m.Items[len(m.Items)-1].Unmarshal(…)` in generated protobuf code or `(*in)[i].DeepCopyInto(&(*out)[i])` in every deepcopy file, now links to that method of the element's type. The type is read from what is indexed: a parameter, a variable, a struct field or a named slice type like `type ConjunctiveSchema []Schema`, and an element held by pointer, like `[]*Item`, reaches `Item`'s methods. Before, such a call linked to whichever method shared its name, often one in an unrelated package, or to nothing when many types declare it, as with Kubernetes' `DeepCopyInto`. An element whose type comes from outside your project, like `[]*net.IPNet`, or is a built-in one like `[]error`, now links to nothing rather than to a project method of that name. Re-index Go projects after upgrading. - In Go, a method called further along a chain now links to that method of the type the chain has reached, like `Where` and `First` in a GORM-generated query such as `query.Card.WithContext(ctx).Where(…).First()`, a method called on what a function returns, like the client from `store.New(…)`, or one called on a composite literal, like `GitOracle{}.Worktree()`. The type is read from a function's or method's results, a conversion, a literal or an embedded field. Before, such a call was matched by its method's name alone, so a query's `Where` linked to another table's query type, a literal's method to a type that only shares its name, and `time.Now().UTC().Format(…)` to the one project method named `Format`. A chain whose type comes from outside your project, or can't be worked out, now links to nothing rather than to a guess. Re-index Go projects after upgrading. Thanks @BX-D for the report and @inth3shadows for the extra cases. (#2543) +- In Go, three kinds of call now link where Go says they go. A conversion like `Category(s)`, `(*Pair)(p)` or `model.Kind(3)` is recorded as a use of the type rather than a call to it or a new instance of it. A call in a package's external tests (`package a_test`) reaches the helper those tests declare, not a same-named one in `package a` beside them, which they can't see. And a function, method or type a package defines once per platform or build tag, like `sep` in `sep_unix.go` and `sep_other.go`, now links to the definition a default build on Linux uses, or the one for the platform the calling file is written for, like Windows code reaching the Windows definition. Before, the second and third picked whichever definition came first, so a Linux caller could reach the Windows or `!unix` code. Re-index Go projects after upgrading. Thanks @inth3shadows for the report. (#2581) - In Go, a method a type declares in a file too large to index, like the `String`, `Marshal` and `Unmarshal` every Kubernetes API type has in `core/v1/generated.pb.go`, is no longer replaced by the method of that name a type it embeds has. Before, `pod.String()` on a `*v1.Pod` linked to `TypeMeta`'s `String`, a method the call never runs; it now links to nothing, since the pod's own method isn't in the graph. A method the type really gets from a type it embeds, like `GetName` from `ObjectMeta`, still links. Re-index Go projects after upgrading. - In Go, calls into a module whose `go.mod` is not at the project root now resolve, whether it is a `server/` backend next to a `web/` frontend or one of several modules side by side as in etcd, so calls like `store.New()` and `s.db.CreateItem()` find their targets instead of being treated as calls into a third-party package. A name written through a package, like a `job.OPCommand` result type or a field of type `artifact.Manager`, now links to that package's symbol rather than a same-named one elsewhere, which also corrects links in projects with a single `go.mod`. Re-index Go projects after upgrading. Thanks @GoDiao for the report and @danusha2345 for the fix. (#2322) - In Go, an exported type written without a package, like `Node` in `func Walk(v Visitor, node Node)`, now links to the type of that name in its own package, whether it is a struct, an interface or another kind of type. Before, it could link to a struct of the same name in an unrelated package, so prometheus's PromQL parser functions pointed at the Kubernetes discovery `Node` struct instead of the parser's own `Node` interface, or, for a result type like `Appender` in `func (f *fanout) Appender(…) Appender`, to the method itself. A type from a package outside your project, like `apiv1.Node` or `http.Handler`, no longer links to a struct of the same name in the file that uses it; a variadic parameter like `...storage.Filter` now links to that package's type; a route handler written as a method value, like `h.Follow` in an Echo or Gin app, links to that method rather than a model struct named `Follow`; and a method called on another call's result or on an indexed value, like `err[i].Error()` or a chained `.String()`, is no longer recorded as creating a struct of the same name. Re-index Go projects after upgrading. diff --git a/__tests__/go-build-constraint-twins.test.ts b/__tests__/go-build-constraint-twins.test.ts new file mode 100644 index 0000000000..bf75ed6eef --- /dev/null +++ b/__tests__/go-build-constraint-twins.test.ts @@ -0,0 +1,110 @@ +/** + * A Go package can define one name once per build target: `sep` in + * `sep_unix.go` (`//go:build unix`) and `sep_other.go` (`//go:build !unix`). + * Name matching took whichever came first, so a Linux caller reached the + * `!unix` definition (#2581). A reference now reaches the definition that + * builds for the most typical target its own file builds for: Linux on + * amd64, then a Mac, then Windows and the rest. The constraint is read from + * the `//go:build` line, the legacy `// +build` lines and the file name's + * `_GOOS` / `_GOARCH` suffix. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +const GO_MOD = 'module example.com/app\n\ngo 1.22\n'; + +let tempDir: string; +let cg: CodeGraph | null = null; + +afterEach(() => { + cg?.close(); + cg = null; + if (tempDir) fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +async function index(files: Record): Promise { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2581-build-')); + for (const [rel, source] of Object.entries({ 'go.mod': GO_MOD, ...files })) { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), source); + } + cg = await CodeGraph.init(tempDir, { index: true }); + cg.resolveReferences(); + return cg; +} + +/** ` ` of every call out of the function `fromName`, sorted. */ +function callsFrom(graph: CodeGraph, fromName: string): string[] { + const from = graph.getNodesByKind('function').find((n) => n.name === fromName); + expect(from, fromName).toBeDefined(); + return graph + .getOutgoingEdges(from!.id) + .filter((e) => e.kind === 'calls') + .map((e) => { + const target = graph.getNode(e.target); + return `${target?.qualifiedName} ${target?.filePath}`; + }) + .sort(); +} + +describe('Go definitions under complementary build constraints (#2581)', () => { + it("calls the issue's `unix` definition, not the `!unix` one", async () => { + const graph = await index({ + 'main.go': 'package main\n\nfunc main() { println(sep()) }\n', + 'sep_unix.go': '//go:build unix\n\npackage main\n\nfunc sep() string { return "/" }\n', + 'sep_other.go': '//go:build !unix\n\npackage main\n\nfunc sep() string { return "\\\\" }\n', + }); + expect(callsFrom(graph, 'main')).toEqual(['sep sep_unix.go']); + }); + + it('reads legacy `// +build` lines and `_GOOS` / `_GOARCH` file names', async () => { + const graph = await index({ + 'main.go': 'package main\n\nfunc main() { println(page(), dir(), cpu()) }\n', + // No system in these names: only the `+build` lines tell them apart. + 'page_small.go': '// +build plan9\n\npackage main\n\nfunc page() int { return 1 }\n', + 'page_std.go': '// +build !plan9\n\npackage main\n\nfunc page() int { return 4096 }\n', + 'dir_darwin.go': 'package main\n\nfunc dir() string { return "/Users" }\n', + 'dir_linux.go': 'package main\n\nfunc dir() string { return "/home" }\n', + 'cpu_386.go': 'package main\n\nfunc cpu() int { return 32 }\n', + 'cpu_amd64.go': 'package main\n\nfunc cpu() int { return 64 }\n', + }); + expect(callsFrom(graph, 'main')).toEqual(['cpu cpu_amd64.go', 'dir dir_linux.go', 'page page_std.go']); + }); + + it('follows the build target of the calling file, and falls back to a Mac', async () => { + const graph = await index({ + 'main.go': 'package main\n\nfunc main() { open(); notify() }\n', + 'run_windows.go': 'package main\n\nfunc run() { open() }\n', + 'open_other.go': '//go:build !windows\n\npackage main\n\nfunc open() {}\n', + 'open_windows.go': 'package main\n\nfunc open() {}\n', + 'notify_aix.go': 'package main\n\nfunc notify() {}\n', + 'notify_darwin.go': 'package main\n\nfunc notify() {}\n', + }); + expect(callsFrom(graph, 'main')).toEqual(['notify notify_darwin.go', 'open open_other.go']); + expect(callsFrom(graph, 'run')).toEqual(['open open_windows.go']); + }); + + it("applies to a method and to a call through another package's import", async () => { + const graph = await index({ + 'main.go': `package main + +import "example.com/app/osutil" + +func main() { + var f file + _ = f.close() + _ = osutil.Sep() +} +`, + 'file.go': 'package main\n\ntype file struct{}\n', + 'file_other.go': '//go:build !unix\n\npackage main\n\nfunc (f *file) close() error { return nil }\n', + 'file_unix.go': '//go:build unix\n\npackage main\n\nfunc (f *file) close() error { return nil }\n', + 'osutil/sep_other.go': '//go:build !unix\n\npackage osutil\n\nfunc Sep() string { return "\\\\" }\n', + 'osutil/sep_unix.go': '//go:build unix\n\npackage osutil\n\nfunc Sep() string { return "/" }\n', + }); + expect(callsFrom(graph, 'main')).toEqual(['Sep osutil/sep_unix.go', 'file::close file_unix.go']); + }); +}); diff --git a/__tests__/go-conversion-references.test.ts b/__tests__/go-conversion-references.test.ts new file mode 100644 index 0000000000..6d500fe422 --- /dev/null +++ b/__tests__/go-conversion-references.test.ts @@ -0,0 +1,114 @@ +/** + * A Go call whose callee is a type is a conversion: `Category(s)` with + * `type Category string`, `(*Pair)(p)`, `model.Kind(3)`. It uses the type, + * and calls or instantiates nothing, so it is recorded as a `references` + * edge to the type. Before, it was a `calls` edge to a named non-struct type + * and an `instantiates` edge to a struct (#2581). + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +const GO_MOD = 'module example.com/app\n\ngo 1.22\n'; + +let tempDir: string; +let cg: CodeGraph | null = null; + +afterEach(() => { + cg?.close(); + cg = null; + if (tempDir) fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +async function index(files: Record): Promise { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2581-conv-')); + for (const [rel, source] of Object.entries({ 'go.mod': GO_MOD, ...files })) { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), source); + } + cg = await CodeGraph.init(tempDir, { index: true }); + cg.resolveReferences(); + return cg; +} + +/** ` :` of every edge out of the function `fromName`, sorted. */ +function edgesFrom(graph: CodeGraph, fromName: string): string[] { + const from = graph.getNodesByKind('function').find((n) => n.name === fromName); + expect(from, fromName).toBeDefined(); + return graph + .getOutgoingEdges(from!.id) + .filter((e) => e.kind === 'calls' || e.kind === 'instantiates' || e.kind === 'references') + .map((e) => { + const target = graph.getNode(e.target); + return `${e.kind} ${target?.kind} ${target?.qualifiedName} :${e.line}`; + }) + .sort(); +} + +describe('Go type conversions (#2581)', () => { + it("records the issue's `Category(s)` as a use of the type, not a call", async () => { + const graph = await index({ + 'main.go': `package main + +type Category string + +func use(s string) Category { + return Category(s) // a conversion, not a call +} + +func main() { use("") } +`, + }); + expect(edgesFrom(graph, 'use')).toEqual([ + 'references type_alias Category :5', + 'references type_alias Category :6', + ]); + }); + + it('records conversions to struct, pointer, slice and imported types the same way, and keeps real calls', async () => { + const graph = await index({ + 'model/model.go': `package model + +type Kind int + +type Point struct{ X, Y int } + +func Make() Kind { return 0 } +`, + 'main.go': `package main + +import "example.com/app/model" + +type Celsius float64 + +type Pair struct{ X, Y int } + +type Names []string + +func convert(p *Pair, raw []string, v float64, pt model.Point) { + _ = Celsius(v) + _ = (*Pair)(p) + _ = Names(raw) + _ = model.Kind(3) + _ = Pair(pt) + _ = model.Make() + _ = Pair{X: 1} +} + +func main() { convert(nil, nil, 0, model.Point{}) } +`, + }); + const edges = edgesFrom(graph, 'convert'); + expect(edges.filter((e) => /:1[2-8]$/.test(e))).toEqual([ + 'calls function Make :17', + 'instantiates struct Pair :18', + 'references struct Pair :13', + 'references struct Pair :16', + 'references type_alias Celsius :12', + 'references type_alias Kind :15', + 'references type_alias Names :14', + ]); + }); +}); diff --git a/__tests__/go-external-test-package.test.ts b/__tests__/go-external-test-package.test.ts new file mode 100644 index 0000000000..dfcb1c8503 --- /dev/null +++ b/__tests__/go-external-test-package.test.ts @@ -0,0 +1,145 @@ +/** + * A package's external tests (`package a_test`) share its directory but are + * a package of their own. A name one of them declares at the top level + * reaches the other only through an import: written `a.Real`, or bare after + * `import . "…/a"`, and then only an exported one. Before, a bare call in an + * external test bound to package `a`'s test helper of the same name (#2581). + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +const GO_MOD = 'module example.com/a\n\ngo 1.22\n'; + +let tempDir: string; +let cg: CodeGraph | null = null; + +afterEach(() => { + cg?.close(); + cg = null; + if (tempDir) fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +async function index(files: Record): Promise { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2581-xtest-')); + for (const [rel, source] of Object.entries({ 'go.mod': GO_MOD, ...files })) { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), source); + } + cg = await CodeGraph.init(tempDir, { index: true }); + cg.resolveReferences(); + return cg; +} + +/** ` ` of every call or type use out of the function `fromName`, sorted. */ +function edgesFrom(graph: CodeGraph, fromName: string): string[] { + const from = graph.getNodesByKind('function').find((n) => n.name === fromName); + expect(from, fromName).toBeDefined(); + return graph + .getOutgoingEdges(from!.id) + .filter((e) => e.kind === 'calls' || e.kind === 'references') + .map((e) => { + const target = graph.getNode(e.target); + return `${e.kind} ${target?.qualifiedName} ${target?.filePath}`; + }) + .sort(); +} + +const PACKAGE_A = `package a + +func Real() {} + +func helper() {} + +type fixture struct{} +`; + +describe('Go external test packages (#2581)', () => { + it("binds the issue's external test to its own package's helper", async () => { + const graph = await index({ + 'a.go': 'package a\n\nfunc Real() {}\n', + 'a_internal_test.go': `package a + +import "testing" + +func corpus(t *testing.T) []string { return nil } + +func TestInternal(t *testing.T) { _ = corpus(t) } +`, + 'b_ext_helpers_test.go': `package a_test + +import "testing" + +func corpus(t *testing.T, root string) []string { return nil } +`, + 'a_ext_test.go': `package a_test + +import "testing" + +func TestExt(t *testing.T) { _ = corpus(t, "") } +`, + }); + expect(edgesFrom(graph, 'TestExt')).toEqual(['calls corpus b_ext_helpers_test.go']); + expect(edgesFrom(graph, 'TestInternal')).toEqual(['calls corpus a_internal_test.go']); + }); + + it('reaches the package under test only through its import', async () => { + const graph = await index({ + 'a.go': PACKAGE_A, + 'qualified_test.go': `package a_test + +import ( + "testing" + + "example.com/a" +) + +func TestQualified(t *testing.T) { a.Real() } +`, + 'dot_test.go': `package a_test + +import ( + "testing" + + . "example.com/a" +) + +func TestDot(t *testing.T) { Real() } + +func TestUnexported(t *testing.T) { helper() } +`, + 'plain_test.go': `package a_test + +type fixture struct{ name string } +`, + 'use_test.go': `package a_test + +func check(f fixture) {} +`, + }); + expect(edgesFrom(graph, 'TestQualified')).toEqual(['calls Real a.go']); + expect(edgesFrom(graph, 'TestDot')).toEqual(['calls Real a.go']); + // A dot import brings only the exported names. + expect(edgesFrom(graph, 'TestUnexported')).toEqual([]); + expect(edgesFrom(graph, 'check')).toEqual(['references fixture plain_test.go']); + }); + + it("does not bind package a's code to a name only its external tests declare", async () => { + const graph = await index({ + 'a.go': 'package a\n\nfunc Real() {}\n', + 'a_internal_test.go': `package a + +import "testing" + +func TestInternal(t *testing.T) { extOnly() } +`, + 'ext_test.go': `package a_test + +func extOnly() {} +`, + }); + expect(edgesFrom(graph, 'TestInternal')).toEqual([]); + }); +}); diff --git a/__tests__/go-framework-own-package.test.ts b/__tests__/go-framework-own-package.test.ts index 137fa58804..dacafd9c35 100644 --- a/__tests__/go-framework-own-package.test.ts +++ b/__tests__/go-framework-own-package.test.ts @@ -190,7 +190,8 @@ describe('Go framework heuristics pick only in the package Go reads the name fro }); it('a conversion and a composite literal still reach their own package’s struct', () => { - expect(linksFrom('errors.go', 'Error', { name: '_' })).toEqual(['instantiates errors.go:Error']); + // `(*Error)(nil)` is a conversion: a use of the type, not an instantiation (#2581). + expect(linksFrom('errors.go', 'Error', { name: '_' })).toEqual(['references errors.go:Error']); expect(linksFrom('errors.go', 'Error', { name: 'newError' })).toEqual([ 'instantiates errors.go:Error', 'references errors.go:Error', diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 5913fab803..1d07be21d5 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -1351,10 +1351,12 @@ export class ReferenceResolver { this.context, ); const scoped = this.gateRustScope(candidate, ref); - const resolved = this.gateSuperSelfCall( + // A Go name its package defines once per build target is the definition + // the reference's file builds with (see preferGoBuildTwin). + const resolved = preferGoBuildTwin(this.gateSuperSelfCall( scoped?.resolvedBy === 'framework' ? this.gateFrameworkLanguage(scoped, ref) : this.gateLanguage(scoped, ref), ref, - ); + ), ref, this.context); // A Dart class whose field holds its own type (`class Node { Node? next; }`) // names itself: a self-edge that says nothing (#2327). if (resolved && ref.language === 'dart' && ref.referenceKind === 'references' && @@ -1752,6 +1754,14 @@ export class ReferenceResolver { } } + // A Go call whose callee is a type is a conversion — `Category(s)`, + // `(*Pair)(p)`, `model.Kind(3)` — which uses the type, and calls or + // instantiates nothing (#2581). + if (kind === 'calls' && ref.original.language === 'go') { + const targetNode = this.nodeById(ref.targetNodeId); + if (targetNode && GO_TYPE_KINDS.has(targetNode.kind)) kind = 'references'; + } + // Promote "calls" to "instantiates" when the resolved target is a // class/struct/union. Languages without a `new` keyword (Python, Ruby) // express instantiation as `Foo()` — extraction can't tell that diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index d4a0d05a13..e3523c96af 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -3354,9 +3354,21 @@ function isSfcPrivate(n: Node, context: ResolutionContext): boolean { * exported name from anywhere, an unexported one only inside its own package, * which is a directory. By the name's first letter, not the extractor's flag: * the flag is unset for every Go method, exported or not. + * + * A package's external tests (`package a_test` beside `package a`) share its + * directory but are a package of their own: a name one of them declares at + * the top level reaches the other only through an import, written `a.Real`, + * or bare after `import . "…/a"`, and then only an exported one. A test + * helper `corpus` in `package a_test` is not package `a`'s `corpus` (#2581). */ -function isGoNameVisible(candidate: Node, ref: UnresolvedRef): boolean { - return /^[A-Z]/.test(candidate.name) || path.posix.dirname(candidate.filePath) === path.posix.dirname(ref.filePath); +function isGoNameVisible(candidate: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + const dir = goPackageDir(candidate.filePath); + if (dir !== goPackageDir(ref.filePath)) return /^[A-Z]/.test(candidate.name); + if (candidate.filePath === ref.filePath || TYPE_MEMBER_KINDS.has(candidate.kind) || ref.referenceName.includes('.')) return true; + const own = goPackageClause(ref.filePath, context); + const theirs = goPackageClause(candidate.filePath, context); + if (own === null || theirs === null || own === theirs) return true; + return /^[A-Z]/.test(candidate.name) && goDotImports(ref.filePath, context).dirs.includes(dir); } /** @@ -3457,7 +3469,7 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex // no `.mm` file's `vector`. (A class is the runtime's, wherever it is // written.) if (isObjcFileLocalElsewhere(candidate, ref)) return false; - if (lang === 'go') return isGoNameVisible(candidate, ref); + if (lang === 'go') return isGoNameVisible(candidate, ref, context); if (lang === 'rust') { if (candidate.visibility !== 'private') return true; if (isRustTraitImplMethod(candidate, context)) return true; @@ -11805,7 +11817,7 @@ export function matchByExactName( // And a Go name that isn't exported, only its own package's: prometheus // tsdb's `[]*memSeries` is head.go's, not `tsdb/agent`'s, which scored the // same (a child package's directory shares as much of the path) and won. - (n.language !== 'go' || isGoNameVisible(n, ref)) && + (n.language !== 'go' || isGoNameVisible(n, ref, context)) && dartExtensionDecl(n, context)?.named !== false && // A Scala package object's member, only where it is in scope — ahead of // the ranking, so cats.laws' `Eq` can be the `cats` package object's. @@ -20449,6 +20461,206 @@ function goPackageClause(filePath: string, context: ResolutionContext): string | return name; } +/** What `go build` is asked to build for. */ +interface GoBuildTarget { + goos: string; + goarch: string; +} + +/** The operating systems and architectures go/build reads from a file name (its syslist). */ +const GO_KNOWN_OS: ReadonlySet = new Set([ + 'aix', 'android', 'darwin', 'dragonfly', 'freebsd', 'hurd', 'illumos', 'ios', 'js', 'linux', 'nacl', 'netbsd', + 'openbsd', 'plan9', 'solaris', 'wasip1', 'windows', 'zos', +]); +const GO_KNOWN_ARCH: ReadonlySet = new Set([ + '386', 'amd64', 'amd64p32', 'arm', 'armbe', 'arm64', 'arm64be', 'loong64', 'mips', 'mipsle', 'mips64', 'mips64le', + 'mips64p32', 'mips64p32le', 'ppc', 'ppc64', 'ppc64le', 'riscv', 'riscv64', 's390', 's390x', 'sparc', 'sparc64', 'wasm', +]); +/** The systems the `unix` build tag holds for. */ +const GO_UNIX_OS: ReadonlySet = new Set([ + 'aix', 'android', 'darwin', 'dragonfly', 'freebsd', 'hurd', 'illumos', 'ios', 'linux', 'netbsd', 'openbsd', 'solaris', +]); + +/** + * The targets a Go name with build-constrained twins is read for, the most + * typical first: a Linux server, a Mac, Windows, then every other system on + * its usual architecture and Linux on every other architecture. + */ +const GO_BUILD_TARGETS: readonly GoBuildTarget[] = (() => { + const usual: Record = { js: 'wasm', wasip1: 'wasm', ios: 'arm64', android: 'arm64', aix: 'ppc64', zos: 's390x', hurd: '386' }; + const targets: GoBuildTarget[] = [ + { goos: 'linux', goarch: 'amd64' }, { goos: 'darwin', goarch: 'arm64' }, { goos: 'windows', goarch: 'amd64' }, + { goos: 'linux', goarch: 'arm64' }, { goos: 'darwin', goarch: 'amd64' }, { goos: 'freebsd', goarch: 'amd64' }, + ]; + const add = (goos: string, goarch: string): void => { + if (!targets.some((t) => t.goos === goos && t.goarch === goarch)) targets.push({ goos, goarch }); + }; + for (const goos of GO_KNOWN_OS) add(goos, usual[goos] ?? 'amd64'); + for (const goarch of GO_KNOWN_ARCH) add('linux', goarch); + return targets; +})(); + +/** + * Whether a build tag holds for a target, as the go command sets them by + * default: the system and architecture (`android` is also `linux`, `ios` + * `darwin`, `illumos` `solaris`), `unix` on a Unix system, the `gc` compiler, + * cgo, and every `go1.N` release. Any other tag (`ignore`, `purego`, a + * project's `integration`) is unset. + */ +function goBuildTagHolds(tag: string, t: GoBuildTarget): boolean { + if (tag === t.goos || tag === t.goarch) return true; + switch (tag) { + case 'unix': return GO_UNIX_OS.has(t.goos); + case 'linux': return t.goos === 'android'; + case 'darwin': return t.goos === 'ios'; + case 'solaris': return t.goos === 'illumos'; + case 'gc': return true; + case 'cgo': return t.goos !== 'js' && t.goos !== 'wasip1'; + default: return /^go1\.\d+$/.test(tag); + } +} + +type GoBuildConstraint = (t: GoBuildTarget) => boolean; + +/** A `//go:build` expression (`unix && !(linux || darwin)`) as a test of a target, or null when it can't be read. */ +function goParseBuildExpr(text: string): GoBuildConstraint | null { + const tokens = text.match(/&&|\|\||[()!]|[A-Za-z0-9_.]+|\S/g) ?? []; + let i = 0; + const or = (): GoBuildConstraint | null => { + let left = and(); + while (left && tokens[i] === '||') { + i++; + const a = left, b = and(); + left = b && ((t) => a(t) || b(t)); + } + return left; + }; + const and = (): GoBuildConstraint | null => { + let left = not(); + while (left && tokens[i] === '&&') { + i++; + const a = left, b = not(); + left = b && ((t) => a(t) && b(t)); + } + return left; + }; + const not = (): GoBuildConstraint | null => { + const tok = tokens[i++]; + if (tok === '!') { + const inner = not(); + return inner && ((t) => !inner(t)); + } + if (tok === '(') { + const inner = or(); + return tokens[i++] === ')' ? inner : null; + } + return tok !== undefined && /^[A-Za-z0-9_.]+$/.test(tok) ? (t) => goBuildTagHolds(tok, t) : null; + }; + const expr = or(); + return expr && i === tokens.length ? expr : null; +} + +/** A legacy `// +build` line's options (`linux,amd64 darwin`, `!plan9`) as a test of a target, or null when it can't be read. */ +function goParsePlusBuild(text: string): GoBuildConstraint | null { + const options = text.trim().split(/\s+/).filter(Boolean).map((option) => option.split(',').map((term) => { + const neg = term.startsWith('!'); + const tag = neg ? term.slice(1) : term; + return /^[A-Za-z0-9_.]+$/.test(tag) ? { neg, tag } : null; + })); + if (options.length === 0 || options.some((terms) => terms.some((term) => term === null))) return null; + return (t) => options.some((terms) => terms.every((term) => goBuildTagHolds(term!.tag, t) !== term!.neg)); +} + +const GO_BUILD_CONSTRAINTS = new WeakMap>(); + +/** + * The targets a Go file builds for, as go/build reads them: a `_linux`, + * `_amd64` or `_windows_arm64` suffix on its name (before `_test`), and the + * `//go:build` line above its package clause, or the legacy `// +build` lines + * when it has none. Null for a file that builds everywhere, or whose + * constraint can't be read. + */ +function goFileConstraint(filePath: string, context: ResolutionContext): GoBuildConstraint | null { + let memo = GO_BUILD_CONSTRAINTS.get(context); + if (!memo) GO_BUILD_CONSTRAINTS.set(context, (memo = new Map())); + if (memo.has(filePath)) return memo.get(filePath)!; + const parts: GoBuildConstraint[] = []; + // go/build ignores what comes before the name's first `_`, so `linux.go` names no system. + const base = path.posix.basename(filePath.replace(/\\/g, '/')).replace(/\..*$/, ''); + const suffix = base.includes('_') ? base.slice(base.indexOf('_') + 1).split('_') : []; + if (suffix[suffix.length - 1] === 'test') suffix.pop(); + const last = suffix[suffix.length - 1]; + const prev = suffix[suffix.length - 2]; + if (prev !== undefined && GO_KNOWN_OS.has(prev) && last !== undefined && GO_KNOWN_ARCH.has(last)) { + parts.push((t) => goBuildTagHolds(prev, t) && goBuildTagHolds(last, t)); + } else if (last !== undefined && (GO_KNOWN_OS.has(last) || GO_KNOWN_ARCH.has(last))) { + parts.push((t) => goBuildTagHolds(last, t)); + } + // The constraint comes before the package clause, after only blank lines and line comments. + let goBuild: string | undefined; + const plusBuild: string[] = []; + for (const line of (context.getFileLines?.(filePath) ?? context.readFile(filePath)?.split(/\r?\n/) ?? []).slice(0, 200)) { + // (Past a byte-order mark.) + const text = (line.charCodeAt(0) === 0xfeff ? line.slice(1) : line).trim(); + if (text === '') continue; + if (!text.startsWith('//')) break; + const go = /^\/\/go:build\s+(.*)$/.exec(text); + if (go && goBuild === undefined) goBuild = go[1]; + const plus = /^\/\/\s*\+build\s+(.*)$/.exec(text); + if (plus) plusBuild.push(plus[1]!); + } + const header = goBuild !== undefined ? [goParseBuildExpr(goBuild)] : plusBuild.map(goParsePlusBuild); + let readable = true; + for (const c of header) { + if (c) parts.push(c); + else readable = false; + } + const constraint: GoBuildConstraint | null = !readable || parts.length === 0 ? null : (t) => parts.every((c) => c(t)); + memo.set(filePath, constraint); + return constraint; +} + +const GO_BUILD_TWINS = new WeakMap>(); + +/** + * The definition Go builds, of a name its package defines more than once for + * different targets — `sep` in `sep_unix.go` (`//go:build unix`) and + * `sep_other.go` (`//go:build !unix`), a method in `file_linux.go` and + * `file_windows.go`. Name matching took whichever came first, so a Linux + * caller's `sep()` reached the `!unix` file and the Unix definition showed no + * caller (#2581). The definition kept is the one that builds for the most + * typical target the reference's own file builds for: Linux on amd64 first, + * then a Mac, then Windows and the rest (GO_BUILD_TARGETS), so a file that + * only builds for Windows reaches the Windows definition. The result stands + * when no twin builds for those targets, or two do. + */ +export function preferGoBuildTwin(result: ResolvedRef | null, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (!result || ref.language !== 'go') return result; + const target = context.getNodeById?.(result.targetNodeId); + // A file that builds everywhere can have no twin: the package would declare the name twice. + if (!target || target.language !== 'go' || !goFileConstraint(target.filePath, context)) return result; + let memo = GO_BUILD_TWINS.get(context); + if (!memo) GO_BUILD_TWINS.set(context, (memo = new Map())); + let twins = memo.get(target.id); + if (!twins) { + const dir = goPackageDir(target.filePath); + const clause = goPackageClause(target.filePath, context); + twins = context.getNodesByName(target.name).filter((n) => + n.language === 'go' && n.kind === target.kind && n.qualifiedName === target.qualifiedName && + goPackageDir(n.filePath) === dir && goPackageClause(n.filePath, context) === clause); + memo.set(target.id, twins); + } + if (twins.length < 2) return result; + const own = goFileConstraint(ref.filePath, context); + for (const t of GO_BUILD_TARGETS) { + if (own && !own(t)) continue; + const built = twins.filter((n) => goFileConstraint(n.filePath, context)?.(t) ?? true); + if (built.length === 1) return built[0]!.id === target.id ? result : { ...result, targetNodeId: built[0]!.id }; + if (built.length > 1) return result; + } + return result; +} + /** * The names a Go file may bind from a type assertion — the first name before * each `:=` or `=` on a line that asserts a type — so that only calls through From c46523079f6b00aae0fbbbaebecf519d3d768603 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 06:45:02 +0000 Subject: [PATCH 22/95] fix(context): the busiest file is found even when 20 test or generated files outrank it (#2785) Root cause: getDominantFile and getTopRouteFile took the top 20 files in SQL and only then dropped test and generated files in code. Twenty test files busier than the app's busiest file left nothing, so getDominantFile returned null and the context builder's core-directory boost never applied. Nineteen left the dominant file without its runner-up, and nextEdgeCount 0 passes the "3x the next file" test for a file that is not dominant at all. getTopRouteFile returned null the same way, and measured its 30% share against the routes of the 20 files in the window, not the project's. Fix: - DOMINANT_FILE_SQL ranks every file. The window saved no work: ORDER BY needs every group, so the aggregate runs to the end either way, and only one row per file comes back. - computeDominantFile filters the ranking 20 rows at a time until it has the dominant file and its runner-up. The generated-file lookup stays a bounded probe per batch. - getTopRouteFile counts every route file, with ties broken by path. Validation: - New dominant-file-window.test.ts, 6 tests, all failing on main: the issue's layout with 20 and with 19 test files, 25 header-declared generated files (past the first batch), and three route shapes. - The dominant-file cache, plan and allocation suites and the route-file detector test pass. - Seven of the eight real repos give the same dominant file before and after (express, flask, excalidraw, django, airflow, grpc-go, etcd). The closest to the window was airflow, with the runner-up at rank 16. - envoyproxy/go-control-plane crosses the window: 254 generated and test files outrank its first hand-written file. Main returns null; the fix returns xdsmatcher/pkg/matcher/matcher.go (167 edges, runner-up 165). - getTopRouteFile on main reported django's admin/sites.py at 8 of 16 routes and airflow's ui router at 65 of 65. Counting every route file gives 8 of 32 and 65 of 332, both under 30%, so null as documented. - Cost, median of 3 interleaved rounds of 5: getDominantFile on airflow (8,564 ranked files) 581 -> 587 ms, django 189 -> 181 ms, go-control-plane 277 -> 284 ms. getTopRouteFile on airflow 1.3 -> 2.5 ms. - Explore A/B, main vs fix: 996 queries on express, flask, excalidraw, django, airflow and go-control-plane, all byte-identical. Fixes #2578 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/dominant-file-window.test.ts | 154 +++++++++++++++++++++++++ src/db/queries.ts | 63 +++++++--- 3 files changed, 202 insertions(+), 16 deletions(-) create mode 100644 __tests__/dominant-file-window.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 3dedebbd8f..5498f3eeea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, a call through a module-level instance, like `_svc.delete(obj)` after `_svc = SceneService()`, now links to the method its class has or inherits, even when other classes have a method of that name. (#2582) - In Python, a call on a module imported from outside your project, like `subprocess.run(…)` after `import subprocess`, no longer links to a project method that happens to share its name. Thanks @Egrn for the report. (#2599) - `codegraph_explore`, the CLI `codegraph explore` and the prompt hook no longer stall for a minute or more on a large project after a full index; the step that finds the project's busiest file no longer slows to a crawl when a few files are huge, like a generated Android `R.java` (#2571). Thanks @yangfengfan2011. +- `codegraph_explore` and `codegraph context` now find your project's busiest file, and favor the code around it, even when twenty or more test or generated files are busier still, such as a package of protobuf stubs. Before, the search looked only at the 20 busiest files and then set the tests and generated ones aside, so it could find no file at all, or judge one far ahead of the next when it was not. `getTopRouteFile()` in the library counts every file's routes the same way. Thanks @inth3shadows for the report. (#2578) - In JavaScript and TypeScript, a call written as the body of an `if`, `while`, `for` or `do` without braces, like `if (e.key === "Escape") setOpen(false);`, no longer links to a same-named property or method in another file when the name is the file's own, such as a `setOpen` from `useState` (#2585). - `codegraph sync` no longer fails with "FOREIGN KEY constraint failed" in a project whose code lazy-loads a workspace package, like Supabase Studio's `dynamic(() => import('icons').then((mod) => mod.EdgeFunctions))` or vben's `() => import('@vben/layouts').then((m) => m.IFrameView)`. Before, every sync after an edit in such a project rolled back, so the index stopped following your changes until a full re-index. Thanks @bompus for the report and @inth3shadows. (#2565) - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. diff --git a/__tests__/dominant-file-window.test.ts b/__tests__/dominant-file-window.test.ts new file mode 100644 index 0000000000..dfbaac906b --- /dev/null +++ b/__tests__/dominant-file-window.test.ts @@ -0,0 +1,154 @@ +/** + * The dominant file and the top route file are chosen from every candidate, + * not from a 20-row window that test and generated files can fill (#2578). + * + * Both queries took the top 20 files in SQL and only then dropped test and + * generated files in code. Twenty test files with more in-file edges than the + * app's busiest file left nothing, so `getDominantFile` returned null; nineteen + * left the dominant file without its runner-up, and `nextEdgeCount: 0` passes + * the context builder's "3x the next file" test for a file that is not + * dominant at all. `getTopRouteFile` returned null the same way, and measured + * its 30% share against the routes of the 20 files in the window instead of + * the project's. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +let root: string | undefined; +let cg: CodeGraph | undefined; + +afterEach(() => { + cg?.close(); + cg = undefined; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = undefined; +}); + +/** `n + 1` functions, each calling the next: n call edges inside the file. */ +function chain(prefix: string, n: number, header = ''): string { + return header + + Array.from({ length: n }, (_, i) => `export function ${prefix}${i}() { return ${prefix}${i + 1}(); }`).join('\n') + + `\nexport function ${prefix}${n}() { return 0; }\n`; +} + +async function project(files: Record): Promise { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-2578-')); + for (const [rel, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), text); + } + cg = await CodeGraph.init(root, { index: true }); + return cg; +} + +const queries = (g: CodeGraph) => (g as any).queries; + +/** Edges between two symbols of `file`: what the dominant-file ranking counts. */ +function inFileEdges(g: CodeGraph, file: string): number { + return ((g as any).db.db.prepare(` + SELECT COUNT(*) AS n FROM edges e + JOIN nodes s ON e.source = s.id JOIN nodes t ON e.target = t.id + WHERE s.file_path = ? AND t.file_path = ? + `).get(file, file) as { n: number }).n; +} + +/** The issue's layout: the app's two busiest files, below `tests` test files that each outrank both. */ +function appWithTests(tests: number): Record { + const files: Record = { + 'src/core.ts': chain('core', 60), + 'src/second.ts': chain('second', 50), + }; + for (let k = 0; k < tests; k++) files[`tests/t${k}.test.ts`] = chain(`t${k}_`, 70); + return files; +} + +describe('getDominantFile looks past the test and generated files that outrank the app (#2578)', () => { + it('finds the dominant file when 20 test files outrank it', async () => { + const g = await project(appWithTests(20)); + const core = inFileEdges(g, 'src/core.ts'); + const second = inFileEdges(g, 'src/second.ts'); + // Fixture shape: every test file outranks the app's busiest file. + expect(inFileEdges(g, 'tests/t0.test.ts')).toBeGreaterThan(core); + expect(queries(g).getDominantFile()).toEqual({ filePath: 'src/core.ts', edgeCount: core, nextEdgeCount: second }); + }, 60_000); + + it('keeps the runner-up when 19 test files leave room for only one app file in the top 20', async () => { + const g = await project(appWithTests(19)); + const dominant = queries(g).getDominantFile(); + expect(dominant?.filePath).toBe('src/core.ts'); + // 0 here would pass the context builder's `edgeCount >= 3 * nextEdgeCount` + // directory boost for a file with a close runner-up. + expect(dominant?.nextEdgeCount).toBe(inFileEdges(g, 'src/second.ts')); + expect(dominant!.edgeCount).toBeLessThan(3 * dominant!.nextEdgeCount); + }, 60_000); + + it('looks past files whose header says they are generated, beyond the first batch', async () => { + const files: Record = { + 'src/core.ts': chain('core', 60), + 'src/second.ts': chain('second', 50), + }; + // Named like hand-written code; only the banner says otherwise, so the + // index's generated flag is what has to drop them. + for (let k = 0; k < 25; k++) files[`src/models/model${k}.ts`] = chain(`m${k}_`, 70, '// @generated by the model compiler. DO NOT EDIT.\n'); + const g = await project(files); + expect(queries(g).getGeneratedPathsAmong(['src/models/model0.ts', 'src/core.ts'])).toEqual(new Set(['src/models/model0.ts'])); + expect(queries(g).getDominantFile()).toEqual({ + filePath: 'src/core.ts', + edgeCount: inFileEdges(g, 'src/core.ts'), + nextEdgeCount: inFileEdges(g, 'src/second.ts'), + }); + }, 60_000); +}); + +/** An Express router file with one handler per route. */ +function routes(paths: string[]): string { + return `const express = require('express'); +const router = express.Router(); + +${paths.map((p, i) => `function handler${i}(req, res) { res.json({ at: '${p}' }); }`).join('\n')} + +${paths.map((p, i) => `router.get('${p}', handler${i});`).join('\n')} + +module.exports = router; +`; +} + +const EXPRESS_PACKAGE = JSON.stringify({ name: 'shop', dependencies: { express: '^4.19.0' } }); + +describe('getTopRouteFile counts every route file (#2578)', () => { + it('finds the app\'s routes file when 20 test servers hold more routes', async () => { + const files: Record = { + 'package.json': EXPRESS_PACKAGE, + 'src/routes.js': routes(['/orders', '/orders/:id', '/orders/:id/items', '/orders/:id/status']), + }; + for (let k = 0; k < 20; k++) files[`tests/server${k}.test.js`] = routes(['/a', '/b', '/c', '/d', '/e'].map((p) => `/t${k}${p}`)); + const g = await project(files); + expect(g.getTopRouteFile()).toEqual({ filePath: 'src/routes.js', routeCount: 4, totalRoutes: 4 }); + }, 60_000); + + it('measures the share against every route, not the 20 busiest files', async () => { + const files: Record = { + 'package.json': EXPRESS_PACKAGE, + 'src/routes.js': routes(Array.from({ length: 30 }, (_, i) => `/r${i}`)), + }; + for (let k = 0; k < 40; k++) files[`src/features/f${k}.js`] = routes([`/f${k}`]); + const g = await project(files); + // 30 of 70 routes: a 43% share, and the total is all 70. + expect(g.getTopRouteFile()).toEqual({ filePath: 'src/routes.js', routeCount: 30, totalRoutes: 70 }); + }, 60_000); + + it('answers null when routing is diffuse across more than 20 files', async () => { + const files: Record = { + 'package.json': EXPRESS_PACKAGE, + 'src/routes.js': routes(Array.from({ length: 10 }, (_, i) => `/r${i}`)), + }; + for (let k = 0; k < 40; k++) files[`src/features/f${k}.js`] = routes([`/f${k}`]); + const g = await project(files); + // 10 of 50 routes is 20%, under the 30% bar. Counting only the 20 busiest + // files made it 10 of 29. + expect(g.getTopRouteFile()).toBeNull(); + }, 60_000); +}); diff --git a/src/db/queries.ts b/src/db/queries.ts index 5b02ba255d..0e8ab69007 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -257,6 +257,12 @@ function rowToFileRecord(row: FileRow): FileRecord { * under-count nodes per file and pick that plan, and a few huge files (a * generated R.java) then cost ~100 s (#2571). Ties go to the path, so the * plan never decides between equal counts. + * + * Every file, not a top-N window: the test and generated files the caller + * drops can fill any window, and then the real dominant file, or its + * runner-up, was never among the rows (#2578). The window saved nothing + * either: ORDER BY needs every group, so the aggregate runs to the end + * whatever the LIMIT; only the rows handed back differ, one per file. */ export const DOMINANT_FILE_SQL = ` SELECT n.file_path AS file_path, COUNT(*) AS edge_count @@ -266,9 +272,15 @@ export const DOMINANT_FILE_SQL = ` WHERE n.file_path = m.file_path GROUP BY n.file_path ORDER BY edge_count DESC, n.file_path - LIMIT 20 `; +/** + * Ranked candidates the low-value filter is asked about at a time, in + * `computeDominantFile`: the generated flag is a database lookup, so it is + * made per batch rather than for every file with an in-file edge. + */ +const DOMINANT_FILE_BATCH = 20; + /** * Query builder for the knowledge graph database */ @@ -1189,8 +1201,8 @@ export class QueryBuilder { private computeDominantFile(): { filePath: string; edgeCount: number; nextEdgeCount: number } | null { if (!this.stmts.getDominantFile) { - // Pull top 20 candidates; we then filter out test/generated files - // in code (regex-grade matching that SQL LIKE can't express). The + // Every file ranked; test/generated files are then filtered out in + // code (regex-grade matching that SQL LIKE can't express). The // generated-file filter is critical — without it, etcd's // `api/etcdserverpb/rpc.pb.go` (1916 in-file edges, generated // protobuf stub) outranks the real `server/etcdserver/server.go` @@ -1200,13 +1212,26 @@ export class QueryBuilder { this.stmts.getDominantFile = this.db.prepare(DOMINANT_FILE_SQL); } const rows = this.stmts.getDominantFile.all() as Array<{ file_path: string; edge_count: number }>; - const generated = this.getGeneratedPathsAmong(rows.map(r => r.file_path)); - const filtered = rows.filter(r => !isLowValueFile(r.file_path, generated)); - if (filtered.length === 0 || filtered[0]!.edge_count < 20) return null; + // The first two files the filter keeps, in rank order: the dominant file + // and its runner-up. Filtering a 20-row window instead returned null once + // 20 test files outranked the real code, and an understated runner-up + // (0) once 19 did, which passes the caller's "3x the next file" test for + // a file that is not dominant at all (#2578). + const kept: Array<{ file_path: string; edge_count: number }> = []; + for (let i = 0; i < rows.length && kept.length < 2; i += DOMINANT_FILE_BATCH) { + const batch = rows.slice(i, i + DOMINANT_FILE_BATCH); + const generated = this.getGeneratedPathsAmong(batch.map(r => r.file_path)); + for (const r of batch) { + if (isLowValueFile(r.file_path, generated)) continue; + kept.push(r); + if (kept.length === 2) break; + } + } + if (kept.length === 0 || kept[0]!.edge_count < 20) return null; return { - filePath: filtered[0]!.file_path, - edgeCount: filtered[0]!.edge_count, - nextEdgeCount: filtered[1]?.edge_count ?? 0, + filePath: kept[0]!.file_path, + edgeCount: kept[0]!.edge_count, + nextEdgeCount: kept[1]?.edge_count ?? 0, }; } @@ -1221,6 +1246,12 @@ export class QueryBuilder { * Excludes test/generated files from candidacy. Returns null if there * are fewer than 3 non-test routes total, or if no file holds at least * 30% of them (diffuse routing → no single answer file). + * + * Every route file is counted. A 20-row window, filtered afterwards, + * returned null once 20 test files held more routes than the app's own, + * and measured the 30% share against the routes of those 20 files rather + * than of the project (#2578). One row per file that has a route, so the + * window saved next to nothing. */ getTopRouteFile(): { filePath: string; routeCount: number; totalRoutes: number } | null { if (!this.stmts.getTopRouteFile) { @@ -1229,8 +1260,7 @@ export class QueryBuilder { FROM nodes WHERE kind = 'route' GROUP BY file_path - ORDER BY cnt DESC - LIMIT 20 + ORDER BY cnt DESC, file_path `); } const rows = this.stmts.getTopRouteFile.all() as Array<{ file_path: string; cnt: number }>; @@ -3455,11 +3485,12 @@ export class QueryBuilder { /** * Which of `filePaths` the index flagged as tool-generated (schema v9+). * - * Bounded-lookup by design: every consumer already holds a short candidate - * list (a ranked file group, an FTS result page, a LIMIT-20 aggregate), so - * this stays a partial-index probe over a handful of paths — no whole-repo - * set to materialize, and no cache to invalidate, which means a ranking call - * can never serve a verdict the last sync already replaced. + * Bounded-lookup by design: every consumer already holds a candidate list + * (a ranked file group, an FTS result page, a batch of the dominant-file + * ranking, the files that hold a route), so this stays a partial-index + * probe over those paths — no whole-repo set to materialize, and no cache + * to invalidate, which means a ranking call can never serve a verdict the + * last sync already replaced. * * Returns ONLY the content/index signal; callers union it with * {@link isGeneratedFile} so pre-v9 databases (column present, all zeros From 7180fd066670e1b888d768d001a98fa94d4ff011 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 07:10:12 +0000 Subject: [PATCH 23/95] fix(js,ts): a dynamic import from Node or a package no longer links to a project namesake (#2788) Issue 2491 was reported against 1.6.1: `import { resolve } from 'node:path'`, bare `'fs'` and `require('node:fs')` bindings linked their calls to same-named project functions. Main no longer does that for any static form. PR 2120 (released in 1.6.2) declines a name match for a name the file imports from outside the repository. PR 2437 made that hold in projects with a catch-all tsconfig path alias, and PR 2671 extended it to function values such as `promisify(readFile)`. What still linked is the same binding written as a dynamic import. The out-of-repo check reads the file's static import mappings only. The name matcher deliberately treats `const { x } = await import('./m')` as an import, not a local shadow, so the call fell through to a same-named project function. vite's `const { createServer } = await import('node:http'); return createServer(app)` linked to vite's own `createServer`. Fix (name-matcher.ts): - isOutOfRepoBinding also asks dynamicImportSource for the module a dynamic import binds the name from, where the reference sits. - A per-file scan (FIFO of 64, cleared with the other name-matcher memos) finds `const { a, b: c, default: d } = await import('x')` declarations and `import('x').then(({ a }) => ...)` callbacks on the raw text. Only files with a match are comment- and string-blanked, to drop matches in comments and strings and to find where each binding's scope ends: the enclosing block (a top-level await binds the whole module) or the callback body. - A binding counts only when the reference sits inside that scope after it, and the innermost one wins, also over a static import of the name. nitro's `async function createWebSocketProxy()` imports crossws's `createWebSocketProxy` inside itself: only that inner call is dropped, and the file's other calls still reach the local function. - jsBlockEnd and jsBodyEnd are the forward forms of jsBlockOpenAtEnd and jsBodyOpenAtEnd, which now call them. Validation: - __tests__/dynamic-import-binding.test.ts: 5 of 6 tests fail with main's name-matcher.ts (the project-module control passes there); all 6 pass on the kernel and with CODEGRAPH_KERNEL=0. Four mutations each fail their test: no scope end, no callback form, no dynamic source, and no memo clearing (the in-process sync test). - Sync equals a fresh index (CLI) for adding and removing the dynamic import, for the namesake file arriving later, and for the nitro shape. - A/B on 9 TS/JS repos plus 2 controls, both arms with the kernel: 6 edges removed, 0 added, 0 retargeted, all 6 wrong. vite has 3 (node:http createServer, node:fs/promises readFile, rolldown watch), memfs 2 (node:util promisify), nitro 1 (the crossws self-loop). pathe, c12, unbuild, fs-extra, unenv and nuxt are byte-identical, as are the flask and gin controls. - Cost: fresh index median of 3, interleaved: vite 9.17 s main vs 9.50 s, nitro 7.43 s vs 7.12 s (noise on a shared box). The scan itself over every vite source file takes 40 ms (24 files reach the blanking pass). - Kernel parity is unchanged: 772/776 files byte-parity, 4 deferred, over nitro/src and vite's src (this is a resolver-only change). Fixes #2491 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/dynamic-import-binding.test.ts | 168 +++++++++++++++++++++++ src/resolution/name-matcher.ts | 128 +++++++++++++++-- 3 files changed, 288 insertions(+), 9 deletions(-) create mode 100644 __tests__/dynamic-import-binding.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 5498f3eeea..cd658d3c00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -214,6 +214,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In React, a tag a component only mentions in a comment or a string no longer links it to that component: a JSDoc example, markup commented out like `{/* */}`, or an error message like react-router's "You cannot render a inside another " or shadcn's "useCarousel must be used within a ". Nothing in a TypeScript `.ts` file, which can't hold JSX, is read as a tag either, like the XML in outline's OpenSearch template. Before, these counted as renders, so react-router's hooks showed up among the callers of `Router`, `Routes` and `Route`, and `codegraph_explore`, callers and impact followed renders that never happen. A tag after an apostrophe, a URL or a backtick in a component's text, like `

Don't have an account? `, still links as before. Re-index React projects after upgrading. - In JavaScript and TypeScript, a name imported from a package your `package.json` lists no longer links to a project symbol that only shares its name when `tsconfig.json` or `jsconfig.json` has a catch-all path alias, like `"*": ["./typings/*"]` or `"*": ["src/*", "node_modules/*"]`, or an alias that points the package at a file in `node_modules`, like `"lit/decorators": ["./node_modules/lit/decorators.js"]`. Such an alias made every package look like part of the project, so `import { Typography } from '@mui/material'` was linked to the project's own `Typography` and every lit `@property()` decorator to an unrelated class's `property` field, and `codegraph callers`, impact and `codegraph affected` listed code that never used them. An import the alias does map to a file of your project, like a `.d.ts` you keep for an untyped package or `components/Button` through `"*": ["src/*"]`, links as before. Re-index affected projects after upgrading. - In JavaScript and TypeScript, a function the file imports from a package and passes along as a value, like react-router's `Link` in `styled(Link)`, lodash's `noop` in `createContext({ onClose: noop })` or chart.js's `Legend` in `Chart.register(Legend)`, no longer links to a project function that only shares its name. Before, whenever the project had exactly one function of that name, every such value linked to it: outline's breadcrumbs, menus and document lists showed up among the callers of an email template's `Link`, and a test's own `noop` stub in cord-field among the code the app's contexts use, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A function the file declares itself, including one that hides the import inside a function, and one imported from a file of your project link as before. Re-index JavaScript and TypeScript projects after upgrading. +- In JavaScript and TypeScript, a function a file loads with a dynamic import from Node or a package, like `const { createServer } = await import('node:http')`, `const { default: chalk } = await import('chalk')` or `import('node:fs/promises').then(({ readFile }) => …)`, no longer links to a project function that only shares its name. Before, vite's call to Node's `createServer` showed up among the callers of vite's own `createServer`. The import counts only where it is in scope, so the rest of the file links as before, and a dynamic import of your own project's module still leads to that module. Thanks @Ancienttwo for the report. (#2491) - In JavaScript and TypeScript, a value the file imports from your own project and passes along now links only to what it imports: a constant like `isCloudHosted`, a class like a `Star` model, a whole module (`import * as Slack from '../slack'`), or a stylesheet, image or GraphQL document (`import TCO21 from './themes/TCO21.scss'`) no longer links to a function elsewhere that only shares its name. Before, when the project had exactly one function of that name, such a value linked to it: outline's settings screens showed up among the callers of the server's `isCloudHosted()` check, and its stores among those of the `` icon, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A component that wraps the function declared beside it, like `export default styled(Disclosure)`, and a page loaded lazily, like `lazy(() => import('../pages/Home'))`, still lead to that function. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript, an import of a TypeScript module written without an extension, like `import AudioModal from './audio_modal'` in a `.jsx` file next to `audio_modal.tsx`, now links to that module, as it already did from a TypeScript file. Before, a `.js` or `.jsx` file's imports of TypeScript modules were never followed, so in a codebase moving to TypeScript they were matched by name instead: an import under another name linked nothing, and a name that repeats could link to the wrong file, so mastodon's `` and `` tags rendered other components of those names than the ones each file imports. A JavaScript module beside a TypeScript one of the same name still comes first. Re-index JavaScript projects after upgrading. - `codegraph_explore`'s Flow now goes through the implementation your query names when an interface method has several that lead to the same place. Before, it took whichever one came first in the index: asking about prometheus's `Engine.execEvalStmt Queryable.Querier fanout.Querier NewMergeQuerier` traced the call through the TSDB's `DB.Querier` and left the `fanout.Querier` you named off the Flow. Any two routes of the same length are now settled this way, in favor of the one that passes through more of the symbols you named. diff --git a/__tests__/dynamic-import-binding.test.ts b/__tests__/dynamic-import-binding.test.ts new file mode 100644 index 0000000000..bed9ed8410 --- /dev/null +++ b/__tests__/dynamic-import-binding.test.ts @@ -0,0 +1,168 @@ +/** + * Issue #2491: a function imported from a Node built-in names nothing in the + * project. The static forms (`import { resolve } from 'node:path'`, bare + * `'fs'`, `require('node:fs')`) stopped linking to same-named project + * functions in #2120; a DYNAMIC import still did. vite's + * `const { createServer } = await import('node:http'); return createServer(app)` + * linked to vite's own `createServer`, and `readFile` from + * `await import('node:fs/promises')` to a playground helper. + * + * The binding counts only where it is in scope: nitro's + * `async function createWebSocketProxy()` imports crossws's + * `createWebSocketProxy` inside itself, so the call there is crossws's, while + * the file's other calls still mean the local function. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const FILES: Record = { + 'package.json': JSON.stringify({ name: 'app', type: 'module', dependencies: { crossws: '^0.3.0', chalk: '^5.0.0' } }), + 'src/server/index.ts': `export function createServer(app: unknown): unknown { return app; } +`, + 'src/io.ts': `export function readFile(p: string): string { return p; } +export function chalk(s: string): string { return s; } +export class Worker { constructor(readonly file: string) {} } +`, + 'src/helper.ts': `export function helper(): number { return 1; } +`, + 'src/http.ts': `export async function resolveHttpServer(app: unknown) { + const { createServer } = await import('node:http'); + return createServer(app as never); +} +`, + 'src/wasm.ts': `const instantiateFromFile = async (fileUrl: string) => { + const { readFile: read, readFile } = await import('node:fs/promises'); + await read(fileUrl); + return readFile(fileUrl); +}; +export async function paint(s: string) { + const { default: chalk } = await import('chalk'); + return chalk(s); +} +export function later(p: string) { + return import('node:fs/promises').then(({ readFile }) => readFile(p)); +} +export async function spawn() { + const { Worker } = await import('node:worker_threads'); + return new Worker('x'); +} +export async function wrap() { + const { readFile } = await import('node:fs/promises'); + return [readFile].map((f) => f); +} +export { instantiateFromFile }; +`, + 'src/proxy.ts': `async function createWebSocketProxy(opts: string) { + const { createWebSocketProxy } = await import('crossws'); + return createWebSocketProxy(opts as never); +} +export function outer(o: string) { return createWebSocketProxy(o); } +export async function inner(p: string) { + if (p) { + const { readFile } = await import('node:fs/promises'); + await readFile(p); + } + return readFile(p); +} +export async function project() { + const { helper } = await import('./helper'); + return helper(); +} +`, + 'src/top.mjs': `const { readFile } = await import('node:fs/promises'); +export async function fromTop(p) { return readFile(p); } +`, +}; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-dynamic-import-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}, 60_000); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `file:line name` of every edge of `kinds` leaving `file`. */ +function edgesFrom(file: string, kinds: string[]): string[] { + const out: string[] = []; + for (const node of cg.getNodesInFile(file)) { + for (const e of cg.getOutgoingEdges(node.id)) { + if (!kinds.includes(e.kind)) continue; + const target = cg.getNode(e.target)!; + out.push(`${e.line} ${target.filePath}:${target.name}`); + } + } + return out; +} + +describe('a name a dynamic import binds from outside the project (#2491)', () => { + it("never links to a same-named project function: vite's `await import('node:http')`", () => { + expect(edgesFrom('src/http.ts', ['calls'])).not.toContain('3 src/server/index.ts:createServer'); + }); + + it('holds for renamed and default bindings, `.then` destructuring, constructors and function values', () => { + const edges = edgesFrom('src/wasm.ts', ['calls', 'instantiates', 'references', 'function_ref']); + expect(edges.filter((e) => e.includes('src/io.ts'))).toEqual([]); + }); + + it('holds for a top-level `await import()` binding', () => { + expect(edgesFrom('src/top.mjs', ['calls']).filter((e) => e.includes('src/io.ts'))).toEqual([]); + }); + + it("binds only where it is in scope: nitro's function importing its own name", () => { + const edges = edgesFrom('src/proxy.ts', ['calls']); + // Inside the function, `createWebSocketProxy(opts)` is crossws's: no self-loop. + expect(edges).not.toContain('3 src/proxy.ts:createWebSocketProxy'); + // Outside it, the call still means the file's own function. + expect(edges).toContain('5 src/proxy.ts:createWebSocketProxy'); + // Inside the `if` block the name is node's; after the block it is bound by nothing, as before. + expect(edges).not.toContain('9 src/io.ts:readFile'); + expect(edges).toContain('11 src/io.ts:readFile'); + }); + + it("keeps a dynamic import of the project's own module", () => { + expect(edgesFrom('src/proxy.ts', ['calls'])).toContain('15 src/helper.ts:helper'); + }); +}); + +describe('sync reads the dynamic import an edit adds or removes (#2491)', () => { + it('drops the name match when the import appears and restores it when it goes', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-dynamic-import-sync-')); + const write = (rel: string, content: string): void => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + const plain = 'export async function a(p: string) {\n return readFile(p);\n}\n'; + const dynamic = "export async function a(p: string) {\n const { readFile } = await import('node:fs/promises');\n return readFile(p);\n}\n"; + write('src/io.ts', 'export function readFile(p: string): string { return p; }\n'); + write('src/use.ts', plain); + const graph = await CodeGraph.init(dir, { index: true }); + const calls = (): string[] => + graph.getNodesInFile('src/use.ts').flatMap((n) => + graph.getOutgoingEdges(n.id).filter((e) => e.kind === 'calls').map((e) => `${graph.getNode(e.target)!.filePath}:${graph.getNode(e.target)!.name}`)); + try { + expect(calls()).toContain('src/io.ts:readFile'); + write('src/use.ts', dynamic); + await graph.sync(); + expect(calls()).not.toContain('src/io.ts:readFile'); + write('src/use.ts', plain); + await graph.sync(); + expect(calls()).toContain('src/io.ts:readFile'); + } finally { + graph.close(); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e3523c96af..0862d98144 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -11324,12 +11324,17 @@ function jsBindingEnclosesReference(name: string, fn: Node, ref: UnresolvedRef, * declaration starts) is still open at the end of `code`. */ function jsBlockOpenAtEnd(code: string, at: number): boolean { + return jsBlockEnd(code, at) === code.length; +} + +/** Where the block that holds `at` closes: its `}`, or the end of `code`. */ +function jsBlockEnd(code: string, at: number): number { let depth = 0; for (let i = at; i < code.length; i++) { if (code[i] === '{') depth++; - else if (code[i] === '}' && --depth < 0) return false; + else if (code[i] === '}' && --depth < 0) return i; } - return true; + return code.length; } const JS_OPERAND_END = /[\w$)\]}'"`]/; @@ -11343,18 +11348,23 @@ const JS_OPERAND_START = /[\w$'"`]/; * statement where no semicolon ends the last. */ function jsBodyOpenAtEnd(code: string, at: number): boolean { + return jsBodyEnd(code, at) === code.length; +} + +/** Where the body that starts at `at` (see jsBodyOpenAtEnd) ends, or the end of `code`. */ +function jsBodyEnd(code: string, at: number): number { let i = at; while (i < code.length && WHITESPACE.test(code[i]!)) i++; - if (code[i] === '{') return jsBlockOpenAtEnd(code, i + 1); + if (code[i] === '{') return jsBlockEnd(code, i + 1); let depth = 0; for (; i < code.length; i++) { const c = code[i]!; if (c === '(' || c === '[' || c === '{') { depth++; } else if (c === ')' || c === ']' || c === '}') { - if (--depth < 0) return false; + if (--depth < 0) return i; } else if (depth === 0 && (c === ',' || c === ';')) { - return false; + return i; } else if (depth === 0 && c === '\n') { // `const f = (x) => x + 1` with no semicolon, above the reference: the // line before ends an operand (a name, a closing bracket, a quote) and @@ -11364,11 +11374,11 @@ function jsBodyOpenAtEnd(code: string, at: number): boolean { while (before >= 0 && WHITESPACE.test(code[before]!)) before--; let after = i + 1; while (after < code.length && WHITESPACE.test(code[after]!)) after++; - if (JS_OPERAND_END.test(code[before] ?? '') && (after === code.length || JS_OPERAND_START.test(code[after]!))) return false; + if (JS_OPERAND_END.test(code[before] ?? '') && (after === code.length || JS_OPERAND_START.test(code[after]!))) return i; i = after - 1; } } - return true; + return code.length; } /** @@ -17815,6 +17825,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { SEALED_MODULES.delete(context); LOCAL_BINDING_MEMO.delete(context); LOCAL_BINDING_SITES.delete(context); + DYNAMIC_IMPORT_SCANS.delete(context); SELECTOR_NAMES.delete(context); GET_STATE_FILES.delete(context); DESTRUCTURED_FILES.delete(context); @@ -19881,14 +19892,113 @@ export function reExportSource(name: string, ref: UnresolvedRef, context: Resolu /** * Does the file bind `name` by importing it from outside the repository? So * does a barrel's `export { name } from 'pkg'`: its `imports` reference names - * the package's export. + * the package's export. So does a dynamic import in scope at the reference + * (dynamicImportSource), which wins over the file's static one, as an inner + * binding does. */ function isOutOfRepoBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { const binding = context.getImportMappings?.(ref.filePath, ref.language)?.find((m) => m.localName === name); - const source = binding?.source ?? (ref.referenceKind === 'imports' ? reExportSource(name, ref, context) : undefined); + const source = dynamicImportSource(name, ref, context) ?? binding?.source ?? + (ref.referenceKind === 'imports' ? reExportSource(name, ref, context) : undefined); return source !== undefined && context.isOutOfRepoImport?.(source, ref.filePath, ref.language) === true; } +/** + * A file's dynamic `import()` bindings: what each one binds, from which + * module, and the stretch of the file it is in scope for (offsets in the + * file's text). Read once per file; kept for the last few files, since + * references arrive file by file. + */ +interface DynamicImportScan { + names: ReadonlySet; + lineStarts: number[]; + bindings: Array<{ names: string[]; source: string; at: number; scopeEnd: number }>; +} +const DYNAMIC_IMPORT_SCANS = new WeakMap>(); +const DYNAMIC_IMPORT_SCANS_KEEP = 64; +const NO_DYNAMIC_IMPORTS: DynamicImportScan = { names: new Set(), lineStarts: [], bindings: [] }; +/** `const { a, b: c, default: d } = await import('x')`. */ +const DYNAMIC_IMPORT_DECLARATION = /\b(?:const|let|var)\s*\{([^{}]*)\}\s*=\s*await\s+import\s*\(\s*(['"])([^'"\n]+)\2\s*\)/g; +/** `import('x').then(({ a }) => …)`, or a `function ({ a }) {…}` callback. */ +const DYNAMIC_IMPORT_CALLBACK = + /\bimport\s*\(\s*(['"])([^'"\n]+)\1\s*\)\s*\.\s*then\s*\(\s*(?:async\s+)?(?:function\s*[\w$]*\s*)?\(\s*\{([^{}]*)\}\s*\)\s*(=>|\{)/g; + +/** The local names a destructuring list binds: `a`, `b: c` (c), `default: d` (d), `e = 1` (e). */ +function destructuredLocals(list: string): string[] { + const out: string[] = []; + for (const part of list.split(',')) { + const binding = part.replace(/=[\s\S]*$/, '').trim(); + if (binding.startsWith('...')) continue; + const colon = binding.indexOf(':'); + const local = (colon >= 0 ? binding.slice(colon + 1) : binding).trim(); + if (/^[A-Za-z_$][\w$]*$/.test(local)) out.push(local); + } + return out; +} + +function dynamicImportScan(filePath: string, context: ResolutionContext): DynamicImportScan { + let cache = DYNAMIC_IMPORT_SCANS.get(context); + if (!cache) DYNAMIC_IMPORT_SCANS.set(context, (cache = new Map())); + let scan = cache.get(filePath); + if (scan) return scan; + scan = NO_DYNAMIC_IMPORTS; + const source = context.readFile(filePath) ?? ''; + const dynamic = /\bimport\s*\(/.test(source); + const declarations = dynamic ? [...source.matchAll(DYNAMIC_IMPORT_DECLARATION)] : []; + const callbacks = dynamic ? [...source.matchAll(DYNAMIC_IMPORT_CALLBACK)] : []; + if (declarations.length > 0 || callbacks.length > 0) { + // Offsets survive both passes. A match in a comment or a string binds + // nothing, and brackets in strings, template literals and regex literals + // open no scope. + const regexes: number[] = []; + const chars = blankStringContents(stripCommentsForRegex(source, 'javascript'), regexes).split(''); + for (let r = 0; r + 1 < regexes.length; r += 2) chars.fill(' ', regexes[r]! + 1, regexes[r + 1]!); + const code = chars.join(''); + const bindings: DynamicImportScan['bindings'] = []; + for (const m of declarations) { + if (code[m.index!] !== source[m.index!]) continue; + bindings.push({ names: destructuredLocals(m[1]!), source: m[3]!, at: m.index!, scopeEnd: jsBlockEnd(code, m.index!) }); + } + for (const m of callbacks) { + if (code[m.index!] !== source[m.index!]) continue; + const body = m.index! + m[0].length; + const scopeEnd = m[4] === '{' ? jsBlockEnd(code, body) : jsBodyEnd(code, body); + bindings.push({ names: destructuredLocals(m[3]!), source: m[2]!, at: m.index!, scopeEnd }); + } + if (bindings.length > 0) { + const lineStarts = [0]; + for (let i = source.indexOf('\n'); i !== -1; i = source.indexOf('\n', i + 1)) lineStarts.push(i + 1); + scan = { names: new Set(bindings.flatMap((b) => b.names)), lineStarts, bindings }; + } + } + if (cache.size >= DYNAMIC_IMPORT_SCANS_KEEP) cache.delete(cache.keys().next().value!); + cache.set(filePath, scan); + return scan; +} + +/** + * The module a dynamic import binds `name` from, where the reference sits: + * vite's `const { createServer } = await import('node:http'); return + * createServer(app)` means node's `createServer`, never the project's. Only a + * binding in scope there counts — declared earlier in a block that is still + * open (a top-level `await import()` binds the whole module), or a parameter + * of the `import('x').then(({ a }) => …)` callback the reference is in: + * nitro's `async function createWebSocketProxy()` imports crossws's + * `createWebSocketProxy` inside itself, while the file's other calls mean + * the function. Of several in scope, the innermost: the last to start. + */ +function dynamicImportSource(name: string, ref: UnresolvedRef, context: ResolutionContext): string | undefined { + if (!JS_FAMILY.has(ref.language)) return undefined; + const scan = dynamicImportScan(ref.filePath, context); + if (!scan.names.has(name)) return undefined; + const at = (scan.lineStarts[ref.line - 1] ?? Number.POSITIVE_INFINITY) + ref.column; + let found: DynamicImportScan['bindings'][number] | undefined; + for (const binding of scan.bindings) { + if (binding.at < at && at <= binding.scopeEnd && binding.names.includes(name) && (!found || binding.at > found.at)) found = binding; + } + return found?.source; +} + /** A module a bundler loads as data, not code: a stylesheet, an image, a GraphQL document, JSON, a font. */ const ASSET_MODULE = /\.(?:css|scss|sass|less|styl|stylus|pcss|graphql|gql|png|jpe?g|gif|svg|webp|avif|ico|bmp|tiff?|json5?|ya?ml|toml|md|mdx|txt|html?|xml|csv|woff2?|ttf|otf|eot|mp3|mp4|webm|ogg|wav|flac|pdf|wasm)$/i; From 0953084bc5b4db687c10417731c425873866f226 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 07:17:14 +0000 Subject: [PATCH 24/95] fix(pascal): the receiver calls of a chained Delphi call are indexed (#2787) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Pascal call whose receiver is itself a call recorded only its outer method: in `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)` the receiver call `MD5FileSized` and the `TMeta.ExtendedPath` in its arguments were dropped, as were paren'd calls deeper in a chain (`A(1).B.C`) or in an index (`List[GetIndex(x)].Free`). The extractor read a receiver call only for a `TFoo.GetInstance()` factory chain. The extractor now walks a call's receiver chain and records each paren'd call in it, with the calls in its arguments and indexes. A paren-less link stays a read. A link inside a chain is recorded only when it can be named precisely (`Obj.Method`, a bare routine, a `TFoo.Factory().X` chain), never as the bare method name a chain like `Client.New.BaseURL(…)` leaves, which matched namesakes. The calls main already read keep their names. Receivers are very often typecasts (`TButton(Sender).Caption`), which read like calls. Delphi builds objects only through a constructor (`TFoo.Create`), so a bare-name Pascal call that resolves to a type is a cast: resolution now gives it a `references` edge to the type instead of `instantiates` or `calls`. That also corrects the edges main already made for value casts (`F := TFoo(Obj)`). Validation: - New __tests__/pascal-chained-calls.test.ts (7 tests, the issue's unit included): all 7 fail with main's sources. Without only the resolver rule the typecast test fails; without only the chain-link gate the fluent-chain test fails. - A/B against main 88ebefa1, both arms built from their own sources, diffed by natural keys: - Every removed edge is a cast changing kind at the same site, with the same source and target (SynEdit 491, HeidiSQL 1,262, mORMot2 4,832, DWScript 3,115, DelphiMVCFramework 2,761), apart from 10 function references that moved to an earlier use in the same routine. - New edges: SynEdit 55, HeidiSQL 249, mORMot2 1,196, DWScript 690, DelphiMVCFramework 2,727. Mostly casts read as references to their type (14 of 14 sampled right) and receiver calls. - 178 new call edges sampled: 140 right. instance-method 30/30, qualified-name 10/10, bare functions 26/30, bare methods 70/92, calls landing on a property, field or variable 0/12. The wrong ones follow rules main applies to every call: the TypeInfo intrinsic (main already links 1,709 such calls to a project method), calls through local procedural variables, and calls now read inside a receiver's arguments, named as main names them. - Controls leveldb (C++, kernel) and SDWebImage (Objective-C, wasm) are byte-identical in dump-graph. - Cost within noise: index time medians of 3 interleaved rounds, HeidiSQL 7.10 to 7.30 s, mORMot2 18.80 to 18.90 s, DWScript 20.00 to 19.40 s wall. Fixes #2547 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/pascal-chained-calls.test.ts | 318 +++++++++++++++++++++++++ src/extraction/tree-sitter.ts | 74 +++++- src/resolution/index.ts | 16 ++ 4 files changed, 398 insertions(+), 11 deletions(-) create mode 100644 __tests__/pascal-chained-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index cd658d3c00..8611ba779c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. - In Delphi, a routine with an attribute on one of its parameters, like `const [ref] AValue: Integer` or DelphiMVCFramework's `const [MVCFromBody] Person: TPerson`, is now indexed with its calls, and so is every class member declared after it. Before, the routine was lost or its calls were credited to the file, the parameter showed up as a field of the class, and the members after it were missing. Thanks @stalexteam for the report. (#2545) - In Delphi, a `for … in` loop that declares its variable inline, like `for var lItem in AList do` or `for var lHandle: THandle in lHandles do`, now has its calls indexed, and a call on a typed loop variable reaches that type's method. Before, the whole loop was skipped, so a routine calling a method only from such a loop was missing from that method's callers. Thanks @stalexteam for the report. (#2546) +- In Delphi, a call made on the result of another call, like `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)` or `List[IndexOf(Name)].Free`, now has the inner calls indexed too, so `MD5FileSized` and `TMeta.ExtendedPath` list that routine among their callers. A typecast like `TButton(Sender).Caption` or `F := TFoo(Obj)` now counts as a use of the type instead of creating one. Thanks @stalexteam for the report. (#2547) - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. diff --git a/__tests__/pascal-chained-calls.test.ts b/__tests__/pascal-chained-calls.test.ts new file mode 100644 index 0000000000..6a581db249 --- /dev/null +++ b/__tests__/pascal-chained-calls.test.ts @@ -0,0 +1,318 @@ +/** + * The receiver of a chained Delphi call (#2547): in + * `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)` only `IsSame` was + * recorded. The receiver call `MD5FileSized` and the `TMeta.ExtendedPath` in + * its arguments were dropped, and so were the paren'd calls deeper in a chain + * (`A(1).B.C`) or in an index (`List[GetIndex(x)].Free`). The extractor now + * reads every paren'd call in a receiver chain. A paren-less link (`.B`) may + * read a field or property, so it is still not a call, and a typecast like + * `TButton(Sender)` creates nothing, so it no longer reads as a construction. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; + +beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['pascal']); +}); + +// The issue's own unit. +const CHAIN_PAS = `unit Chain; + +interface + +type + TKey = record + function IsSame(const AOther: TKey): Boolean; + procedure Reset; + end; + + TMeta = class + class function ExtendedPath(const APath: string): string; static; + end; + +function MD5FileSized(const APath: string): TKey; +function TargetIsTheSame(const APath: string; const AKey: TKey): Boolean; +procedure Stmt(const APath: string); + +implementation + +function TKey.IsSame(const AOther: TKey): Boolean; +begin + Result := True; +end; + +procedure TKey.Reset; +begin +end; + +class function TMeta.ExtendedPath(const APath: string): string; +begin + Result := APath; +end; + +function MD5FileSized(const APath: string): TKey; +begin +end; + +function TargetIsTheSame(const APath: string; const AKey: TKey): Boolean; +begin + Result := MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey); +end; + +procedure Stmt(const APath: string); +begin + MD5FileSized(TMeta.ExtendedPath(APath)).Reset; +end; + +end. +`; + +function refsFrom(code: string, qualifiedName: string, kind = 'calls'): string[] { + const result = extractFromSource('Unit1.pas', code); + const from = result.nodes.find((n) => n.qualifiedName === qualifiedName); + return result.unresolvedReferences + .filter((r) => r.referenceKind === kind && r.fromNodeId === from?.id) + .map((r) => `${r.referenceName}:${r.line}`) + .sort(); +} + +async function withIndex(code: string, check: (cg: CodeGraph) => void): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-chain-')); + try { + fs.writeFileSync(path.join(dir, 'Unit1.pas'), code); + const cg = await CodeGraph.init(dir, { index: true }); + try { + check(cg); + } finally { + cg.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +function edgesFrom(cg: CodeGraph, sourceName: string): string[] { + const source = cg.getNodesByKind('function').concat(cg.getNodesByKind('method')).find((n) => n.name === sourceName); + if (!source) return []; + return cg + .getOutgoingEdges(source.id) + .filter((e) => e.kind !== 'contains') + .map((e) => `${e.kind} ${cg.getNode(e.target)?.qualifiedName ?? '?'}`) + .sort(); +} + +describe('Delphi chained calls (#2547)', () => { + it('records the receiver call and the calls in its arguments', () => { + expect(refsFrom(CHAIN_PAS, 'TargetIsTheSame')).toEqual(['IsSame:41', 'MD5FileSized:41', 'TMeta.ExtendedPath:41']); + expect(refsFrom(CHAIN_PAS, 'Stmt')).toEqual(['MD5FileSized:46', 'Reset:46', 'TMeta.ExtendedPath:46']); + }); + + it('links each routine to all three callees the issue names', async () => { + await withIndex(CHAIN_PAS, (cg) => { + expect(edgesFrom(cg, 'TargetIsTheSame')).toEqual([ + 'calls MD5FileSized', + 'calls TKey::IsSame', + 'calls TMeta::ExtendedPath', + ]); + expect(edgesFrom(cg, 'Stmt')).toEqual(['calls MD5FileSized', 'calls TKey::Reset', 'calls TMeta::ExtendedPath']); + }); + }); + + it('reads paren\'d calls anywhere in a receiver chain, in every context', () => { + const code = `unit Unit1; + +interface + +implementation + +procedure Run; +begin + A(1).B.C(2); + X := A(3).B.C; + if F(4).Ready then Exit; + List[GetIndex(5)].Free; + Y := Items[Find(6)].Value; + GetHandler(7)(8); + Z := Obj.Prop.Method(9); +end; + +end. +`; + // `B`, `C` (line 10), `Ready`, `Value` and `Prop` are paren-less links: reads, not calls. + expect(refsFrom(code, 'Run')).toEqual([ + 'A:10', + 'A:9', + 'C:9', + 'F:11', + 'Find:13', + 'Free:12', + 'GetHandler:14', + 'GetIndex:12', + 'Method:15', + ]); + }); + + it('names a fluent chain\'s inner links only where they can be named precisely', () => { + const code = `unit Unit1; + +interface + +implementation + +procedure Run; +begin + Client.New.BaseURL('x').Accept('y').Get('/'); + RESTClient.HandleRedirects(False).Accept('z').Get('/'); + Builder.WithName(Name).Build; +end; + +end. +`; + // The outer `Get` keeps the name a call has always had. `BaseURL` and + // `Accept` would be bare names on a chain whose type is unknown, which + // match namesakes, so they are left out; `RESTClient.HandleRedirects` + // and `Builder.WithName` name their receiver. + expect(refsFrom(code, 'Run')).toEqual(['Build:11', 'Builder.WithName:11', 'Get:10', 'Get:9', 'RESTClient.HandleRedirects:10']); + }); + + it('gives a factory its caller and still resolves the chain on what it returns', async () => { + const code = `unit Unit1; + +interface + +type + TBar = class + procedure DoIt; + end; + TDecoy = class + procedure DoIt; + end; + TFoo = class + class function GetInstance: TBar; + end; + +procedure Run; + +implementation + +procedure TBar.DoIt; begin end; +procedure TDecoy.DoIt; begin end; +class function TFoo.GetInstance: TBar; begin Result := nil; end; + +procedure Run; +begin + TFoo.GetInstance().DoIt(); +end; + +end. +`; + await withIndex(code, (cg) => { + expect(edgesFrom(cg, 'Run')).toEqual(['calls TBar::DoIt', 'calls TFoo::GetInstance']); + }); + }); + + it('reads a typecast as a reference to its type, never a construction, wherever it is written', async () => { + const code = `unit Unit1; + +interface + +type + TFoo = class + public + procedure DoIt; + function Ready: Boolean; + end; + + TDecoy = class + public + procedure DoIt; + end; + + NullableInt = record + procedure SetNull; + end; + + TPair = record + Lo, Hi: Integer; + end; + PPair = ^TPair; + +function GetObj(N: Integer): TObject; +procedure Run(Sender: TObject; var V; P: Pointer); + +implementation + +procedure TFoo.DoIt; begin end; +function TFoo.Ready: Boolean; begin Result := True; end; +procedure TDecoy.DoIt; begin end; +procedure NullableInt.SetNull; begin end; +function GetObj(N: Integer): TObject; begin Result := nil; end; + +procedure Run(Sender: TObject; var V; P: Pointer); +var + F: TFoo; +begin + TFoo(Sender).DoIt; + TFoo(GetObj(1)).DoIt(); + if TFoo(Sender).Ready then Exit; + F := TFoo(Sender); + NullableInt(V).SetNull; + PPair(P)^.Lo := 1; +end; + +end. +`; + await withIndex(code, (cg) => { + expect(edgesFrom(cg, 'Run')).toEqual([ + 'calls GetObj', + 'calls NullableInt::SetNull', + 'calls TFoo::DoIt', + 'calls TFoo::DoIt', + 'references NullableInt', + 'references PPair', + 'references TFoo', + 'references TFoo', + 'references TFoo', + 'references TFoo', + ]); + }); + }); + + it('captures a routine passed by name to a receiver call, as for any call', async () => { + const code = `unit Unit1; + +interface + +type + TJob = class + procedure Start; + end; + +function Schedule(Proc: TProc): TJob; +procedure Worker; +procedure Run; + +implementation + +procedure TJob.Start; begin end; +function Schedule(Proc: TProc): TJob; begin Result := nil; end; +procedure Worker; begin end; + +procedure Run; +begin + Schedule(Worker).Start; +end; + +end. +`; + await withIndex(code, (cg) => { + expect(edgesFrom(cg, 'Run')).toEqual(['calls Schedule', 'calls TJob::Start', 'references Worker']); + }); + }); +}); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 0d10db3aa1..e6d627ac3e 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -8560,9 +8560,13 @@ export class TreeSitterExtractor { } /** - * Extract function calls from a Pascal expression + * Extract function calls from a Pascal expression. A `receiverLink` call is + * a link inside another call's receiver chain (#2547): it is recorded only + * when it can be named precisely — `Obj.Method`, a bare routine, or a + * `TFoo.Factory().Method` chain — never as the bare method name a longer + * chain like `Client.New.BaseURL(…)` would leave, which matches namesakes. */ - private extractPascalCall(node: SyntaxNode): void { + private extractPascalCall(node: SyntaxNode, receiverLink = false): void { if (this.nodeStack.length === 0) return; const callerId = this.nodeStack[this.nodeStack.length - 1]; if (!callerId) return; @@ -8613,7 +8617,8 @@ export class TreeSitterExtractor { calleeName = getNodeText(firstChild, this.source); } - if (calleeName) { + const vagueLink = receiverLink && firstChild.type === 'exprDot' && !calleeName.includes('.'); + if (calleeName && !vagueLink) { this.unresolvedReferences.push({ fromNodeId: callerId, referenceName: calleeName, @@ -8623,6 +8628,12 @@ export class TreeSitterExtractor { }); } + // The calls in the callee itself: a chained call's receiver, like + // `MD5FileSized(…)` in `MD5FileSized(…).IsSame(AKey)` (#2547), or the call + // whose result is called in `GetHandler()(X)`. + if (firstChild.type === 'exprDot') this.extractPascalReceiverCalls(firstChild.namedChild(0)); + else if (firstChild.type !== 'identifier') this.extractPascalReceiverCalls(firstChild); + // Also visit arguments for nested calls const args = node.namedChildren.find( (c: SyntaxNode) => c.type === 'exprArgs' @@ -8632,6 +8643,46 @@ export class TreeSitterExtractor { } } + /** + * The calls inside the receiver of a Pascal call or member access, which + * the outer reference doesn't cover (#2547): `MD5FileSized(…)` and the + * `TMeta.ExtendedPath(APath)` in its arguments for + * `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)`, `A(1)` in + * `A(1).B.C`, `GetIndex(x)` in `List[GetIndex(x)].Free`. A paren'd call + * there is a call like any other. A paren-less link (`.B`) may read a field + * or property, so it is not one. A typecast (`TButton(Sender)`) is written + * like a call; resolution tells it apart by what the name resolves to. + */ + private extractPascalReceiverCalls(node: SyntaxNode | null): void { + if (!node) return; + switch (node.type) { + case 'identifier': + return; + case 'exprCall': + this.extractPascalReceiverCall(node, true); + return; + case 'exprDot': + case 'exprTpl': + this.extractPascalReceiverCalls(node.namedChild(0)); + return; + case 'exprSubscript': { + this.extractPascalReceiverCalls(node.namedChild(0)); + const index = node.namedChildren.find((c: SyntaxNode) => c.type === 'exprArgs'); + if (index) this.visitPascalBlock(index); + return; + } + default: + this.visitPascalBlock(node); + } + } + + /** A receiver call, read as any call is, its arguments' routine values included (#756). */ + private extractPascalReceiverCall(call: SyntaxNode, receiverLink: boolean): void { + this.extractPascalCall(call, receiverLink); + const args = call.namedChildren.find((c: SyntaxNode) => c.type === 'exprArgs'); + if (args) this.maybeCaptureFnRefs(args, 'exprArgs'); + } + /** * Extract a PAREN-LESS Pascal method/procedure call (`Obj.Method;`, * `TFoo.GetInstance.DoIt;`). Pascal lets a no-arg method drop its parens, so it @@ -8671,10 +8722,11 @@ export class TreeSitterExtractor { if (innerCallee && /^[TI][A-Z]/.test(innerCallee)) { calleeName = `${innerCallee}().${method}`; // The T/I-prefixed inner is itself a real call — record it too. - if (receiver.type === 'exprCall') this.extractPascalCall(receiver); + if (receiver.type === 'exprCall') this.extractPascalReceiverCall(receiver, false); else this.extractPascalParenlessCall(receiver); } else { calleeName = method; // non-class receiver: a bare method ref (no field-access ref) + this.extractPascalReceiverCalls(receiver); // `MD5FileSized(…).Reset;` (#2547) } } else { // Simple: `Obj.Method` → the dotted name (resolves via the receiver / bare name). @@ -8682,6 +8734,7 @@ export class TreeSitterExtractor { .filter((c: SyntaxNode) => c.type === 'identifier') .map((id: SyntaxNode) => getNodeText(id, this.source)) .join('.'); + this.extractPascalReceiverCalls(receiver); // `List[GetIndex(x)].Free;` } if (calleeName) { @@ -8716,16 +8769,15 @@ export class TreeSitterExtractor { // A STATEMENT-level bare exprDot is a paren-less call (`Obj.Free;`, // `TFoo.GetInstance.DoIt;`). Anywhere else (assignment side, condition, // expression) a bare exprDot is ambiguous with a field/property access, - // so there we only descend for paren'd inner calls. + // so there we only read the paren'd calls in its receiver chain + // (`X := A(1).B.C` reads `A(1)`); a call right before the read is + // named as any call is. if (node.type === 'statement') { this.extractPascalParenlessCall(child); } else { - for (let j = 0; j < child.namedChildCount; j++) { - const grandchild = child.namedChild(j); - if (grandchild?.type === 'exprCall') { - this.extractPascalCall(grandchild); - } - } + const receiver = child.namedChild(0); + if (receiver?.type === 'exprCall') this.extractPascalReceiverCall(receiver, false); + else this.extractPascalReceiverCalls(receiver); } } else { this.visitPascalBlock(child); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 1d07be21d5..055e96ba68 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -153,6 +153,9 @@ const GO_BUILT_INS = new Set([ 'string', 'bool', 'byte', 'rune', 'any', ]); +/** What a Pascal typecast's name resolves to: `TFoo(x)` casts, it doesn't call (see createEdges). */ +const PASCAL_CAST_TARGET_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface', 'enum', 'type_alias']); + const PASCAL_UNIT_PREFIXES = [ 'System.', 'Winapi.', 'Vcl.', 'Fmx.', 'Data.', 'Datasnap.', 'Soap.', 'Xml.', 'Web.', 'REST.', 'FireDAC.', 'IBX.', @@ -1762,6 +1765,19 @@ export class ReferenceResolver { if (targetNode && GO_TYPE_KINDS.has(targetNode.kind)) kind = 'references'; } + // A Pascal call written with a bare name that resolves to a type is a + // typecast: `TButton(Sender).Caption`, `PInt64Rec(P)^.Lo`, `F := + // TFoo(Obj)`. Delphi builds an object only through a constructor + // (`TFoo.Create`), so the code refers to the type and creates nothing. + if ( + kind === 'calls' && + ref.original.language === 'pascal' && + !ref.original.referenceName.includes('.') && + PASCAL_CAST_TARGET_KINDS.has(this.nodeById(ref.targetNodeId)?.kind ?? '') + ) { + kind = 'references'; + } + // Promote "calls" to "instantiates" when the resolved target is a // class/struct/union. Languages without a `new` keyword (Python, Ruby) // express instantiation as `Foo()` — extraction can't tell that From 214ef50efe9fb53a64623ab8c6678b6ef99cf293 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 07:45:00 +0000 Subject: [PATCH 25/95] fix(explore): names the query spells that the index never saw are said, not glossed over (#2789) Root cause: explore answers whatever the query's words match. When a name the query spelled is not in the index at all, the answer is built from the other words and says nothing about the missing ones. In the report, the tool projects under a .NET repo's root build/ folder are skipped by default. Asking for ReleaseNotesBuilder, ChangelogParser, PersistentData and AddFeatures returned an unrelated Program.Main and a C++ Parser, and the agent read the wrong code. The miss diagnostics (issue #2026) ran only when nothing at all matched. Fix: - Query tokens written like code names (a camelCase hump, snake_case, or a qualified name) that resolve to no node at all are checked with a new bounded probe, getUnseenNames. It reports names the index has never seen: no node called that in any case, and no name, qualified name or path, signature or docstring word starting with it. - A qualified name is judged by its member. Express's app.handle is indexed as handle, so the receiver alone never makes a name missing. - Names that pass are listed in a note right under the summary line, success-shaped. The note points to codegraph status and never to Read. It names the .gitignore re-include for a default-skipped folder like build/ as the user's choice, since indexing scope is the user's call. - A single word (Program, Main, How) is never named: it may be English. - The total-miss answer gains the same note. - The seeding's query tokenizer moved into one shared function, unchanged. The skip itself is unchanged here. On main a root build/ folder is still skipped by default: PRs 2681 and 2687 re-opened build folders below src/ only. The documented opt-in, !/build/ in .gitignore, indexes it, and sync picks the opt-in up, verified on main and on 1.6.2. Validation: - New explore-unindexed-names.test.ts, 8 tests: 6 fail on main. - All 38 explore suites pass, plus the tool-allowlist and dynamic-boundary suites (466 tests). - The reporter's repo, newrelic/newrelic-dotnet-agent 418a0f2f, now commits the !/build/ workaround. With that line removed and the index synced, main answers the reporter's query with VersionFileGenerator/Program.cs Main and Sicily/Parser.cpp, saying nothing. The fix adds the note naming exactly the four missing names. - Ordinary queries: 996 on express, flask, excalidraw, django, airflow and go-control-plane, all byte-identical to main. - Miss probes: 120 queries mixing real names with invented ones, or only invented ones. Main never notes anything. The fix names every invented name in all 120 and no real name. The note costs about 5 source lines on a full response. Fixes #2462 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/explore-unindexed-names.test.ts | 196 ++++++++++++++++++++++ src/db/queries.ts | 37 ++++ src/index.ts | 8 + src/mcp/tools.ts | 78 ++++++++- 5 files changed, 314 insertions(+), 6 deletions(-) create mode 100644 __tests__/explore-unindexed-names.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 8611ba779c..879d07a534 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -223,6 +223,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph_explore`, `codegraph_node`, `codegraph_callers`, `codegraph_callees`, `codegraph_impact` and the `codegraph callers`, `callees` and `impact` commands now find a symbol written with its package, the way Go code names it: `refresh.Discovery.refresh`, `storage.Queryable.Querier`, or `gin.Context.Next` in gin's root package. Before, such a name found nothing, so a question written that way lost the symbol and explore's Flow had nothing to connect. A name the search ranked below other results, like gin's `binding.Default`, is found as well, and a Python or Rust module followed by a class and method works the same way. - When `codegraph_explore` finds more dynamic-dispatch links than its list shows, the symbols you named now take turns, so each one's links make the list: asking about prometheus's `refresh.Discovery.refresh hetzner.refresher.refresh hcloudDiscovery.refresh` now lists hetzner's `robotDiscovery` beside the links into other discovery packages that used to fill the list. - `codegraph_explore` now shows a function you name even when it is too long for its share of the answer: its definition and the start of its body, with a note where the body is cut that says which lines were left out and which `codegraph_explore` query returns them. Before, a file made of one long function and a line that calls it could come back as just its last few lines, without the function, and asking for the file by path did the same. A trimmed answer also no longer suggests `codegraph_node`, which isn't offered by default. Thanks @Huy3ko for the report and @danusha2345 for the first fix. (#2486) +- When a `codegraph_explore` query names symbols the index has never seen, the answer now says so first: it lists those names and points to `codegraph status`, and to how a folder skipped by default, like a root `build/` folder, is brought into the index. Before, the answer showed only what the query's other words matched, such as an unrelated `Main` and a C++ `Parser`, with nothing to say the names asked about were missing. Thanks @tippmar-nr for the report. (#2462) - In C and C++, a struct, class, union or enum defined in a variable declaration, like `struct Foo { … } foo;`, is now indexed with its methods and enum values, next to the variable. Before, at most the variable was, so the type was missing from search and `codegraph_explore`, a class deriving from it had no base, and calls to its methods linked to nothing: in redis, the Sentinel's `sentinelState` and `redis-cli`'s `config` struct were missing. A struct, union or enum with no name of its own now takes the name of the variable it declares, so redis's `static struct { … } SPT;` is found as `SPT`, and one declared inside a function is named after its variable instead of ``. The comment above the declaration becomes the type's description. Re-index C and C++ projects after upgrading. - In Objective-C, a struct, union or enum defined in a variable declaration, like `static struct { Class isa; } placeholder;` or `enum { stateType, stateParamName } state;`, is now indexed with its enum values, next to the variable, as in C and C++. Before, only the variable was, so the type was missing from search and `codegraph_explore`. The same went for a `typedef enum { … } Name;` or `typedef struct … Name;` written after a line holding only a macro, like `OF_ASSUME_NONNULL_BEGIN` or `RCT_EXTERN_C_BEGIN`: ObjFW's `OFZIPArchiveEntryCompressionMethod` and every value in it were missing, and so was React Native's `RCTMethodInfo`. A struct, union or enum with no name of its own takes the name of the variable it declares, so ObjFW's class-cluster placeholders are found as `placeholder`, one declared inside a function or method is named after its variable instead of ``, and the comment above the declaration becomes the type's description. As in C, a type that a `.m` file defines links only from that file, so a call elsewhere that just shares its name, like ObjFW's calls of AmigaOS's `Lock(path, mode)`, doesn't link to the runtime's private `struct Lock`. Re-index Objective-C projects after upgrading. - In Objective-C, a macro written alone on its line, like `NS_ASSUME_NONNULL_BEGIN`, `OF_SUBCLASSING_RESTRICTED` or React Native's `RCT_EXTERN_C_BEGIN`, no longer hides the declaration below it. Before, the parser read such a macro as the start of a declaration, so an `@interface` under it could come out as a function and a `typedef` under it as two variables: many of ObjFW's classes, with their properties and methods and the classes they extend and protocols they adopt, were missing, and so were classes and protocols in React Native and KSCrash headers. A message to `super` from a class whose `@interface` follows an attribute macro like `NS_SWIFT_NAME(…)` still reaches the superclass. Re-index Objective-C projects after upgrading. diff --git a/__tests__/explore-unindexed-names.test.ts b/__tests__/explore-unindexed-names.test.ts new file mode 100644 index 0000000000..ed9c8429b8 --- /dev/null +++ b/__tests__/explore-unindexed-names.test.ts @@ -0,0 +1,196 @@ +/** + * Names the query spells that the index has never seen are said plainly, + * instead of answering with code that only matched the query's other words + * (#2462). + * + * A .NET repo kept real tool projects in a root `build/` folder, which the + * default skip list leaves unindexed. `codegraph_explore "ReleaseNotesBuilder + * Program Main ChangelogParser PersistentData Entry AddFeatures"` came back as + * an unrelated `Program.Main` and a C++ `Parser`, with nothing to say that four + * of the names matched nothing. The agent read the wrong code. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler, isIdentifierShapedToken } from '../src/mcp/tools'; + +const QUERY = 'ReleaseNotesBuilder Program Main ChangelogParser PersistentData Entry AddFeatures'; +const MISSING = ['ReleaseNotesBuilder', 'ChangelogParser', 'PersistentData', 'AddFeatures']; + +/** The report's layout: the product under src/, its release tooling under a root build/. */ +const FILES: Record = { + '.gitignore': 'bin/\nobj/\n', + 'src/Agent/Program.cs': `namespace Agent +{ + public static class Program + { + public static int Main(string[] args) + { + var host = new AgentHost(); + return host.Run(args); + } + } + + public class AgentHost + { + public int Run(string[] args) { return args.Length; } + } +} +`, + 'src/native/parser.cpp': `#include +namespace native { +class Parser { +public: + explicit Parser(std::string text) : text_(std::move(text)) {} + int parse() { return static_cast(text_.size()); } +private: + std::string text_; +}; +} // namespace native +`, + // `ChangelogWriter` and `auditTrail` are seen only through a docstring and a + // folder name, never as a symbol. + 'src/auditTrail/trail.ts': `/** Hands the audit entries to the ChangelogWriter. */ +export function recordTrail(): number { return 1; } +`, + // Indexed as `handle`, with no `app.` on it, as express's own is. + 'src/server/app.js': `const app = {}; +app.handle = function handle(req, res) { return res; }; +module.exports = app; +`, + 'build/ReleaseNotesBuilder/Program.cs': `namespace ReleaseNotesBuilder +{ + public static class Program + { + public static int Main(string[] args) + { + var parser = new ChangelogParser(args[0]); + var data = PersistentData.Load(args[1]); + foreach (var entry in parser.Entries()) entry.AddFeatures(data); + return 0; + } + } +} +`, + 'build/ReleaseNotesBuilder/ChangelogParser.cs': `using System.Collections.Generic; +namespace ReleaseNotesBuilder +{ + public class ChangelogParser + { + private readonly string _path; + public ChangelogParser(string path) { _path = path; } + public IEnumerable Entries() { yield return new Entry(_path); } + } +} +`, + 'build/ReleaseNotesBuilder/Entry.cs': `using System.Collections.Generic; +namespace ReleaseNotesBuilder +{ + public class Entry + { + private readonly List _features = new List(); + public Entry(string title) { } + public void AddFeatures(PersistentData data) { _features.AddRange(data.Features); } + } +} +`, + 'build/ReleaseNotesBuilder/PersistentData.cs': `using System.Collections.Generic; +namespace ReleaseNotesBuilder +{ + public class PersistentData + { + public List Features { get; } = new List(); + public static PersistentData Load(string path) { return new PersistentData(); } + } +} +`, +}; + +describe('#2462 — explore says which named symbols are not in the index', () => { + let dir: string; + let cg: CodeGraph; + const explore = async (query: string) => { + const result = await new ToolHandler(cg).execute('codegraph_explore', { query }); + return { text: result.content?.[0]?.text ?? '', isError: result.isError === true }; + }; + + beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2462-')); + for (const [rel, text] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), text); + } + cg = CodeGraph.initSync(dir); + await cg.indexAll(); + }, 120_000); + + afterAll(() => { + cg?.destroy(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('fixture shape: the root build/ folder is skipped by default, so its names are not indexed', () => { + expect(cg.getFiles().map((f) => f.path).filter((p) => p.startsWith('build/'))).toEqual([]); + expect(cg.getNodesByName('Main').map((n) => n.filePath)).toEqual(['src/Agent/Program.cs']); + }); + + it('names the missing symbols above the unrelated matches, success-shaped', async () => { + const { text, isError } = await explore(QUERY); + expect(isError).toBe(false); + expect(text).toContain(`**Not in the index:** ${MISSING.map((n) => `\`${n}\``).join(', ')}.`); + expect(text).toContain("so the code below matches the query's other words, not them"); + // Above the source, where it is read first. + expect(text.indexOf('**Not in the index:**')).toBeLessThan(text.indexOf('**`src/Agent/Program.cs`**')); + // Where to look, never "Read"; including a skipped folder is the user's call. + expect(text).toContain('`codegraph status` shows what is.'); + expect(text).toContain('the user can include that folder by adding a re-include such as `!/build/` to `.gitignore`.'); + expect(text).not.toMatch(/\bRead (?:the|these|those|it)\b/); + }); + + it('never names a single word, which may be English, nor a name the index has', async () => { + const { text } = await explore(QUERY); + const note = text.split('\n').find((l) => l.startsWith('> **Not in the index:**'))!; + for (const word of ['Program', 'Main', 'Entry']) expect(note).not.toContain(`\`${word}\``); + expect((await explore('Program Main AgentHost Run')).text).not.toContain('Not in the index'); + expect((await explore('How does the agent host run its main program?')).text).not.toContain('Not in the index'); + }); + + it('a name seen only in a docstring or a folder name is in the index', () => { + expect(cg.getUnseenNames(['ChangelogWriter', 'auditTrail', 'agenthost', 'Agent.Program', ...MISSING])) + .toEqual(MISSING); + }); + + it('a qualified name is in the index when its member is, whatever the receiver', () => { + // `app.handle` is indexed as `handle`; only a member never seen makes it missing. + expect(cg.getUnseenNames(['app.handle', 'Zorblax.frobnicate', 'pkg::recordTrail'])).toEqual(['Zorblax.frobnicate']); + }); + + it('says so too when nothing matched at all', async () => { + const { text, isError } = await explore('ReleaseNotesBuilder PersistentData'); + expect(isError).toBe(false); + expect(text).toContain('No relevant code found'); + expect(text).toContain('**Not in the index:** `ReleaseNotesBuilder`, `PersistentData`.'); + expect(text).not.toContain('the code below'); + }); + + it('goes away once the folder is re-included and synced', async () => { + fs.writeFileSync(path.join(dir, '.gitignore'), 'bin/\nobj/\n!/build/\n'); + await cg.sync(); + const { text } = await explore(QUERY); + expect(text).not.toContain('Not in the index'); + expect(text).toContain('build/ReleaseNotesBuilder/ChangelogParser.cs'); + }, 120_000); +}); + +describe('isIdentifierShapedToken', () => { + it('takes names written like code and leaves single words alone', () => { + for (const t of ['ChangelogParser', 'addFeatures', 'build_report', 'MAX_SIZE', 'Session.request', 'pkg::run', '$store.items']) { + expect(isIdentifierShapedToken(t), t).toBe(true); + } + for (const t of ['Program', 'Main', 'How', 'parser', 'SQL', 'i18n', 'v1.2']) { + expect(isIdentifierShapedToken(t), t).toBe(false); + } + }); +}); diff --git a/src/db/queries.ts b/src/db/queries.ts index 0e8ab69007..be8a403c47 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -359,6 +359,7 @@ export class QueryBuilder { getDominantFile?: SqliteStatement; getChangeStamp?: SqliteStatement; getTopRouteFile?: SqliteStatement; + unseenName?: SqliteStatement; getRoutingManifest?: SqliteStatement; insertNameSegment?: SqliteStatement; } = {}; @@ -1881,6 +1882,42 @@ export class QueryBuilder { }; } + /** + * Which of `names` the index has never seen: no node is called that, in any + * case, and no indexed name, qualified name, signature or docstring has a + * word starting with it. A file's path is its qualified name, so a folder or + * file named that counts as seen too. Explore uses this to say plainly that + * a name the query spelled is not in the index (#2462). + * + * A qualified name (`app.handle`, `pkg::run`) is asked about by its last + * segment: the index does not always record the receiver it was written on + * — express's `app.handle = function handle(…)` is indexed as `handle` — so + * only a member it has never seen makes the whole name unseen. + * + * Bounded like `getExploreMissDiagnostics`: two index probes per name (the + * lower(name) index and FTS postings), never a source scan, at most 16 names. + */ + getUnseenNames(names: readonly string[]): string[] { + const checked = [...new Set(names)].filter((n) => n.length > 0 && n.length <= 128).slice(0, 16); + if (checked.length === 0) return []; + if (!this.stmts.unseenName) { + this.stmts.unseenName = this.db.prepare(` + SELECT EXISTS (SELECT 1 FROM nodes WHERE lower(name) = lower(?)) + OR EXISTS (SELECT 1 FROM nodes_fts WHERE nodes_fts MATCH ?) AS seen + `); + } + const stmt = this.stmts.unseenName; + return checked.filter((name) => { + const member = name.split(/::|\./).pop()!; + // FTS splits the phrase on everything but letters and digits, so a + // snake_case name is matched as the words it is made of. + const phrase = member.replace(/"/g, ''); + if (phrase.length === 0) return false; + const row = stmt.get(member, `{name qualified_name signature docstring} : "${phrase}"*`) as { seen: number }; + return !row.seen; + }); + } + /** * FTS5 search with prefix matching */ diff --git a/src/index.ts b/src/index.ts index 8708eff2d1..55dc2c434c 100644 --- a/src/index.ts +++ b/src/index.ts @@ -2119,6 +2119,14 @@ export class CodeGraph { return this.queries.getExploreMissDiagnostics(query); } + /** + * Which of `names` the index has never seen, by name, path, signature or + * docstring — the names explore reports as not in the index (#2462). + */ + getUnseenNames(names: readonly string[]): string[] { + return this.queries.getUnseenNames(names); + } + /** * The COBOL copybooks a query names (`CVACT01Y`, `MYCOPYBOOK`): each * member's indexed copybook file(s) and every `COPY` / `EXEC SQL INCLUDE` diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index ffba2116a6..42abfb0e98 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -1060,6 +1060,54 @@ export function elidedBodyNote(filePath: string, name: string, from: number, to: return `\n… lines ${from}-${to} of \`${name}\` elided — codegraph_explore \`${filePath}:${from}-${to}\` returns them`; } +/** + * The symbol-shaped tokens of an explore query, the way its named-symbol + * seeding reads them: split on spaces, commas and brackets, a source-file + * extension dropped (`Program.cs` names `Program`), at least 3 characters, + * plain or qualified identifiers only, deduped, at most 16. + */ +export function exploreQueryTokens(query: string): string[] { + const FILE_EXT = /\.(?:java|kt|kts|ts|tsx|js|jsx|mjs|cjs|cs|py|go|rb|php|swift|rs|cpp|cc|cxx|c|h|hpp|scala|lua|dart|vue|svelte|astro|erl|hrl)$/i; + return [...new Set( + query.split(/[\s,()[\]]+/) + .map((t) => t.replace(FILE_EXT, '').trim()) + .filter((t) => t.length >= 3 && /^[A-Za-z_$][\w$]*(?:(?:::|\.)[\w$]+)*$/.test(t)) + )].slice(0, 16); +} + +/** + * A query token written the way code names things — `ChangelogParser`, + * `addFeatures`, `build_report`, `Session.request`, `pkg::run` — rather than + * a single word, which may just as well be English (`How`, `Parser`, `main`). + * Only these are named back as missing from the index (#2462): a question's + * ordinary words must never be reported as code the index lacks. + */ +export function isIdentifierShapedToken(token: string): boolean { + return /[a-z][A-Z]/.test(token) + || /[A-Za-z0-9]_[A-Za-z0-9]/.test(token) + || /[A-Za-z_$][\w$]*(?:\.|::)[A-Za-z_$]/.test(token); +} + +/** + * The note explore gives for names the query spelled that the index has never + * seen (#2462). With results, they match the query's other words, and an agent + * reading them as the answer to the missing names reads the wrong code. It + * points at what decides the answer — whether those files are indexed — and + * never at Read. Including a skipped folder is worded as the user's choice: + * indexing scope is the user's call, never the agent's (AGENTS.md). + */ +export function notInIndexNote(names: readonly string[], withResults: boolean): string { + const one = names.length === 1; + const list = names.map((n) => `\`${n}\``).join(', '); + const effect = withResults + ? `, so the code below matches the query's other words, not ${one ? 'it' : 'them'}.` + : '.'; + return `> **Not in the index:** ${list}. No indexed symbol, file, signature or comment uses ${one ? 'this name' : 'these names'}${effect}` + + ` If ${one ? 'it is' : 'they are'} defined in this project, the file${one ? '' : 's'} defining ${one ? 'it' : 'them'} ${one ? 'is' : 'are'} not indexed:` + + ` \`codegraph status\` shows what is. If ${one ? 'it lives' : 'they live'} in a folder skipped by default, like \`build/\` or \`dist/\`,` + + ' the user can include that folder by adding a re-include such as `!/build/` to `.gitignore`.'; +} + /** * Join rendered parts with gap markers that name whatever the trim skipped. * @@ -4330,6 +4378,12 @@ export class ToolHandler { explanation += miss.candidates.length > 0 ? `\nCandidates to retry with codegraph_explore (shared words, not confirmed answers): ${cappedList(miss.candidates, 350)}` : '\nNo shared-word symbol candidates found; retry codegraph_explore with literal symbol/file names or code terms.'; + // Names written like code that the index has never seen may live in + // files it does not index (#2462): say which, and where to look. + const named = exploreQueryTokens(matchQuery).filter(isIdentifierShapedToken); + let unseen: string[] = []; + try { unseen = named.length > 0 ? cg.getUnseenNames(named) : []; } catch { /* never fail an explore call */ } + if (unseen.length > 0) explanation += `\n\n${notInIndexNote(unseen, false)}`; } const empty = `No relevant code found for "${query}"${missNote}${explanation}`; // Still an explore call, so it is still recorded: an empty answer spends a @@ -4420,8 +4474,10 @@ export class ToolHandler { // Files declaring a TYPE the query named by name — the counter-case guard // for the declaration-only penalty (CG-28). Populated in the token loop. const namedTypeFiles = new Set(); + // Tokens written like code names that resolved to no node at all, not even + // a camel-infix definer: candidates for the "not in the index" note (#2462). + const unresolvedNamedTokens: string[] = []; { - const FILE_EXT = /\.(?:java|kt|kts|ts|tsx|js|jsx|mjs|cjs|cs|py|go|rb|php|swift|rs|cpp|cc|cxx|c|h|hpp|scala|lua|dart|vue|svelte|astro|erl|hrl)$/i; const CALLABLE = new Set(['method', 'function', 'component', 'constructor']); // Variables/constants seed too: in Svelte/React a `$state` variable // (`chatAtBottom`, `feedAtBottom`) is exactly the kind of symbol an agent @@ -4433,11 +4489,7 @@ export class ToolHandler { const isTestPath = (p: string) => /(^|\/)(tests?|specs?|__tests__|testdata|mocks?|fixtures?)\//i.test(p) || /\.(test|spec)\.[a-z]+$/i.test(p); const bodyLines = (n: Node) => Math.max(0, (n.endLine ?? n.startLine) - n.startLine); const callerCount = (n: Node) => { try { return cg.getCallers(n.id).length; } catch { return 0; } }; - const tokens = [...new Set( - matchQuery.split(/[\s,()[\]]+/) - .map((t) => t.replace(FILE_EXT, '').trim()) - .filter((t) => t.length >= 3 && /^[A-Za-z_$][\w$]*(?:(?:::|\.)[\w$]+)*$/.test(t)) - )].slice(0, 16); + const tokens = exploreQueryTokens(matchQuery); // PascalCase tokens in the query are type/file disambiguators — when the // agent writes "DataRequest task validate", the `task`/`validate` it wants // are DataRequest's, NOT the same-named overloads in Validation.swift / @@ -4548,6 +4600,7 @@ export class ToolHandler { .sort((a, b) => a.name.length - b.name.length) .slice(0, 3); } + if (raw.length === 0 && cands.length === 0 && isIdentifierShapedToken(t)) unresolvedNamedTokens.push(t); // Bare lowercase words only seed defs their query-siblings corroborate // (see the NL-stopword guard above). Filtering CANDS (not picks) applies // the guard uniformly to both branches below, including the >3-def @@ -5120,6 +5173,19 @@ export class ToolHandler { ]; const summaryLineIdx = 2; + // Names the query spelled that the index has never seen (#2462): said + // first, under the summary, because the code below then answers the + // query's other words. A query for a root `build/` folder's tools, which + // the default skip list leaves unindexed, came back as an unrelated + // `Program.Main` and a C++ `Parser`, read as the answer. + let unseenNames: string[] = []; + if (unresolvedNamedTokens.length > 0) { + try { + unseenNames = cg.getUnseenNames(unresolvedNamedTokens); + } catch { /* the note must never fail an explore call */ } + } + if (unseenNames.length > 0) lines.push(notInIndexNote(unseenNames, true), ''); + // A named COBOL copybook's include sites, ALL of them (#2342) — the source // below has room for a few include windows, and "what includes X" is the // question a copybook name asks. From e69546e4a3f4be96646001a6f01ae5f4cf8a4fcb Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 08:10:10 +0000 Subject: [PATCH 26/95] fix(js,ts): code outside tests resolves as if Jest manual mocks were not there (#2790) Issue 2455: when production code imports a workspace package the resolver cannot follow (`import { formatLabel } from 'acme-shared'`), its names fall back to name matching, and a Jest manual mock beside the code (`__mocks__/acme-shared.ts`) is the nearest namesake. So the call, the import's names, type references, constructions and `extends` all linked to the mock, and the real function lost its callers. The visibility gate keeps test suites from production code, but `__mocks__/` is not one of them. jsx-render, Vue templates and the event-emitter pass pick by name with their own lookups, which no gate covers. A manual mock stands in for a module only while Jest or Vitest runs, and it re-declares that module's names by design. Rejecting it after the ranking picked it (what the gate does for `__tests__/`) leaves the call unlinked even where the real function is the only other candidate. It also leaves sync behind a fresh index: deleting the mock re-opens nothing, while a fresh index then links the real function (main has this today for `__tests__/`). Fix: - ReferenceResolver.resolveOne resolves a reference from code that is not a test (isTestPath) on a view of the graph without manual mocks, when the project has any. Every lookup by name, lower-case name, qualified name, kind and method answers as if no `__mocks__/` file were indexed. Only names a mock declares are filtered, and those answers are cached. The view is swapped in as this.context for the call, so every strategy (frameworks, imports, name matching, fuzzy, function refs) reads it, and clearCaches drops it with the memos keyed by it. Test code keeps the whole graph. - jsxChild, the Vue template resolver and the event-emitter handler lookup skip a mock's component or function for a non-test file (isManualMockHiddenFrom). - The synthesized-edge merge drops any edge from non-test code into a node a manual mock declares, whatever pass made it. - An import that names the mock's own path (`import x from './__mocks__/uuid'` in a vitest setup file) still links. Jest's `moduleNameMapper` and automock are not read: they act only while Jest runs, so they never change what production code calls. Validation: - __tests__/jest-manual-mocks.test.ts: 5 of 6 tests fail with main's sources. Each of five rules fails a test when removed: the view (three tests, the in-process sync case among them, which adds and deletes a mock), the JSX, Vue and event lookups, and the merge guard. - Sync equals a fresh index (CLI) when the mock or the real module arrives later, when the production file is edited, and when the mock is deleted. - A/B on 8 TS/JS repos plus 2 controls, both arms with the kernel. Edges from production code into mocks: webdriverio 23 to 0, outline 21 to 0, social-app 3 to 0, insomnia 19 to 13 (the 13 left are setup-vitest.ts importing mocks by path). webdriverio gains 23 right edges and 142 confidence raises that the mocks had blocked. jest, nuxt, flask and gin are byte-identical. On webdriverio, production code's edges equal main's on a copy without the mocks, 20,885 of 20,885 with metadata. insomnia matches the same way, apart from the 13 path imports. - Cost: fresh index, median of 3 interleaved: webdriverio 12.67 s main vs 12.71 s, insomnia 12.84 s vs 12.60 s. Fixes #2455 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/jest-manual-mocks.test.ts | 220 +++++++++++++++++++++++++ src/resolution/callback-synthesizer.ts | 31 +++- src/resolution/index.ts | 105 +++++++++++- src/resolution/name-matcher.ts | 16 ++ 5 files changed, 368 insertions(+), 5 deletions(-) create mode 100644 __tests__/jest-manual-mocks.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 879d07a534..25296bd419 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -216,6 +216,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In JavaScript and TypeScript, a name imported from a package your `package.json` lists no longer links to a project symbol that only shares its name when `tsconfig.json` or `jsconfig.json` has a catch-all path alias, like `"*": ["./typings/*"]` or `"*": ["src/*", "node_modules/*"]`, or an alias that points the package at a file in `node_modules`, like `"lit/decorators": ["./node_modules/lit/decorators.js"]`. Such an alias made every package look like part of the project, so `import { Typography } from '@mui/material'` was linked to the project's own `Typography` and every lit `@property()` decorator to an unrelated class's `property` field, and `codegraph callers`, impact and `codegraph affected` listed code that never used them. An import the alias does map to a file of your project, like a `.d.ts` you keep for an untyped package or `components/Button` through `"*": ["src/*"]`, links as before. Re-index affected projects after upgrading. - In JavaScript and TypeScript, a function the file imports from a package and passes along as a value, like react-router's `Link` in `styled(Link)`, lodash's `noop` in `createContext({ onClose: noop })` or chart.js's `Legend` in `Chart.register(Legend)`, no longer links to a project function that only shares its name. Before, whenever the project had exactly one function of that name, every such value linked to it: outline's breadcrumbs, menus and document lists showed up among the callers of an email template's `Link`, and a test's own `noop` stub in cord-field among the code the app's contexts use, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A function the file declares itself, including one that hides the import inside a function, and one imported from a file of your project link as before. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript and TypeScript, a function a file loads with a dynamic import from Node or a package, like `const { createServer } = await import('node:http')`, `const { default: chalk } = await import('chalk')` or `import('node:fs/promises').then(({ readFile }) => …)`, no longer links to a project function that only shares its name. Before, vite's call to Node's `createServer` showed up among the callers of vite's own `createServer`. The import counts only where it is in scope, so the rest of the file links as before, and a dynamic import of your own project's module still leads to that module. Thanks @Ancienttwo for the report. (#2491) +- In JavaScript and TypeScript, code outside your tests no longer links to a Jest or Vitest manual mock under `__mocks__/`. Before, when an import of a workspace package couldn't be followed, a call like `formatLabel(name)` linked by name to the mock beside it instead of the package's own function, so the real function lost its callers, and the same happened to rendered components and event handlers. Production code is now matched as if the mocks weren't there, so it reaches the real function, and adding or deleting a mock leaves it alone. Tests, other mocks, and a file that imports a mock by its path, like a `jest.setup.ts`, still reach the mock. Thanks @Y-Osawa2022 for the report and @danusha2345. (#2455) - In JavaScript and TypeScript, a value the file imports from your own project and passes along now links only to what it imports: a constant like `isCloudHosted`, a class like a `Star` model, a whole module (`import * as Slack from '../slack'`), or a stylesheet, image or GraphQL document (`import TCO21 from './themes/TCO21.scss'`) no longer links to a function elsewhere that only shares its name. Before, when the project had exactly one function of that name, such a value linked to it: outline's settings screens showed up among the callers of the server's `isCloudHosted()` check, and its stores among those of the `` icon, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A component that wraps the function declared beside it, like `export default styled(Disclosure)`, and a page loaded lazily, like `lazy(() => import('../pages/Home'))`, still lead to that function. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript, an import of a TypeScript module written without an extension, like `import AudioModal from './audio_modal'` in a `.jsx` file next to `audio_modal.tsx`, now links to that module, as it already did from a TypeScript file. Before, a `.js` or `.jsx` file's imports of TypeScript modules were never followed, so in a codebase moving to TypeScript they were matched by name instead: an import under another name linked nothing, and a name that repeats could link to the wrong file, so mastodon's `` and `` tags rendered other components of those names than the ones each file imports. A JavaScript module beside a TypeScript one of the same name still comes first. Re-index JavaScript projects after upgrading. - `codegraph_explore`'s Flow now goes through the implementation your query names when an interface method has several that lead to the same place. Before, it took whichever one came first in the index: asking about prometheus's `Engine.execEvalStmt Queryable.Querier fanout.Querier NewMergeQuerier` traced the call through the TSDB's `DB.Querier` and left the `fanout.Querier` you named off the Flow. Any two routes of the same length are now settled this way, in favor of the one that passes through more of the symbols you named. diff --git a/__tests__/jest-manual-mocks.test.ts b/__tests__/jest-manual-mocks.test.ts new file mode 100644 index 0000000000..ad3bdbc96b --- /dev/null +++ b/__tests__/jest-manual-mocks.test.ts @@ -0,0 +1,220 @@ +/** + * Issue #2455: a Jest (or Vitest) manual mock under `__mocks__/` stands in for + * a module only while a test runs. Production code never means it: when + * `import { formatLabel } from 'acme-shared'` could not be followed, the + * production call fell back to name matching and bound to + * `__mocks__/acme-shared.ts`, the nearer namesake, so the real function lost + * its caller. The same happened to the import's names, type references, + * constructions, `extends`, and a rendered ``. + * + * Test code may still reach a mock (that is when Jest substitutes it), and so + * may a file that imports the mock by its own path, like a `jest.setup.ts`. + * Jest's `moduleNameMapper` and automock only act while Jest runs, so a + * config that maps the package to the mock changes nothing for production + * code. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const MOCK = 'packages/api/__mocks__/acme-shared.tsx'; +const REAL = 'packages/common/utils.tsx'; + +const FILES: Record = { + 'package.json': JSON.stringify({ name: 'acme-monorepo', private: true, workspaces: ['packages/*'] }), + 'jest.config.js': `module.exports = { moduleNameMapper: { '^acme-shared$': '/packages/api/__mocks__/acme-shared.tsx' }, automock: true }; +`, + 'packages/common/package.json': JSON.stringify({ name: 'acme-shared', version: '1.0.0' }), + [REAL]: `export function formatLabel(name: string): string { return name.trim(); } +export function isAdmin(role: string): boolean { return role === 'admin'; } +export interface Label { text: string } +export class Client { send(): number { return 1; } } +export class BaseStore { load(): number { return 1; } } +export function Chart() { return null; } +`, + 'packages/api/package.json': JSON.stringify({ name: 'acme-api', dependencies: { 'acme-shared': 'workspace:*', react: '^18.0.0' } }), + [MOCK]: `export function formatLabel(name: string): string { return 'mock'; } +export function isAdmin(role: string): boolean { return true; } +export interface Label { text: string; mocked: true } +export class Client { send(): number { return 0; } } +export class BaseStore { load(): number { return 0; } } +export function Chart() { return null; } +export function resetMocks(): void {} +`, + 'packages/api/__mocks__/helpers.ts': `export function mockHelper() { return resetMocks(); } +`, + 'packages/api/src/handler.tsx': `import { formatLabel, isAdmin, Client, BaseStore, Chart } from 'acme-shared'; +import type { Label } from 'acme-shared'; + +export function handleRequest(name: string, role: string): string { + if (!isAdmin(role)) return 'forbidden'; + return formatLabel(name); +} +export function label(l: Label): string { return l.text; } +export function send() { return new Client().send(); } +export class Store extends BaseStore {} +export function View() { return ; } +`, + 'packages/api/src/handler.test.ts': `export function setUp(): void { resetMocks(); } +`, + 'packages/api/jest.setup.ts': `import { resetMocks } from './__mocks__/acme-shared'; +resetMocks(); +`, + // An event bus: production code emits; production code registers `handleReady`, which a mock + // declares too, and a test registers the mock's `onSavedMock`. + 'packages/api/src/bus.ts': `export function start(bus: { emit(e: string): void }) { + bus.emit('ready'); + bus.emit('saved'); +} +`, + 'packages/api/src/wire.ts': `export function wire(bus: { on(e: string, f: () => void): void }) { + bus.on('ready', handleReady); +} +`, + 'packages/api/src/handlers.ts': `export function handleReady(): void {} +`, + 'packages/api/__mocks__/handlers.ts': `export function handleReady(): void {} +export function onSavedMock(): void {} +`, + 'packages/api/src/bus.test.ts': `export function listen(bus: { on(e: string, f: () => void): void }) { + bus.on('saved', onSavedMock); +} +`, + // A mock beside one of the app's own modules: an unimported call never means the mock. + 'packages/web/package.json': JSON.stringify({ name: 'web', dependencies: { vue: '^3.0.0' } }), + 'packages/web/src/api.ts': `export function fetchUser(id: string) { return id; } +`, + 'packages/web/src/__mocks__/api.ts': `export function fetchUser(id: string) { return 'mock-' + id; } +export function onlyMock() { return 2; } +`, + 'packages/web/src/page.ts': `import { fetchUser } from './api'; +export function page() { return fetchUser('1'); } +export function guess() { return onlyMock(); } +`, + 'packages/web/src/components/Badge.vue': ` +`, + 'packages/web/src/components/__mocks__/Badge.vue': ` +`, + // Only a mock declares `Pill`: the template's `` is a component from elsewhere. + 'packages/web/src/components/__mocks__/Pill.vue': ` +`, + // `__mocks__/` sorts before `avatar/`: a pick by order alone would take the mock. + 'packages/web/src/components/__mocks__/Avatar.vue': ` +`, + 'packages/web/src/components/avatar/Avatar.vue': ` +`, + 'packages/web/src/components/Card.vue': ` +`, +}; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-jest-manual-mocks-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}, 60_000); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `kind name@file` of every edge but `contains` leaving the nodes of `file`. */ +function edgesFrom(file: string): string[] { + const out: string[] = []; + for (const node of cg.getNodesInFile(file)) { + for (const e of cg.getOutgoingEdges(node.id)) { + if (e.kind === 'contains') continue; + const target = cg.getNode(e.target)!; + out.push(`${e.kind} ${target.name}@${target.filePath}`); + } + } + return out; +} + +describe('a Jest manual mock under __mocks__/ (#2455)', () => { + it("is never what production code means by name: the issue's calls and imports reach the package", () => { + const edges = edgesFrom('packages/api/src/handler.tsx'); + expect(edges.filter((e) => e.endsWith(`@${MOCK}`))).toEqual([]); + // What the name means without the mock: the workspace package's own function. + expect(edges).toEqual(expect.arrayContaining([ + `calls formatLabel@${REAL}`, `calls isAdmin@${REAL}`, `imports formatLabel@${REAL}`, `imports isAdmin@${REAL}`, + ])); + }); + + it('nor a type it names, a class it constructs or extends, or a component it renders', () => { + const edges = edgesFrom('packages/api/src/handler.tsx'); + for (const kind of ['references Label', 'instantiates Client', 'extends BaseStore', 'calls Chart']) { + expect(edges).not.toContain(`${kind}@${MOCK}`); + expect(edges).toContain(`${kind}@${REAL}`); + } + }); + + it("nor a mock beside the app's own module, from a call or a Vue template", () => { + expect(edgesFrom('packages/web/src/page.ts')).not.toContain('calls onlyMock@packages/web/src/__mocks__/api.ts'); + expect(edgesFrom('packages/web/src/page.ts')).toContain('calls fetchUser@packages/web/src/api.ts'); + const card = edgesFrom('packages/web/src/components/Card.vue'); + expect(card).not.toContain('calls Badge@packages/web/src/components/__mocks__/Badge.vue'); + expect(card).not.toContain('calls Pill@packages/web/src/components/__mocks__/Pill.vue'); + expect(card).toContain('calls Badge@packages/web/src/components/Badge.vue'); + expect(card).toContain('calls Avatar@packages/web/src/components/avatar/Avatar.vue'); + }); + + it('nor a handler an event bus pairs with what production code emits', () => { + const edges = edgesFrom('packages/api/src/bus.ts'); + // Registered by production code: the handler it can mean. + expect(edges).toContain('calls handleReady@packages/api/src/handlers.ts'); + expect(edges).not.toContain('calls handleReady@packages/api/__mocks__/handlers.ts'); + // Registered by a test: what the test hands the bus, never what production code calls. + expect(edges).not.toContain('calls onSavedMock@packages/api/__mocks__/handlers.ts'); + }); + + it('is still reached from test code, from another mock, and through an import of its own path', () => { + expect(edgesFrom('packages/api/src/handler.test.ts')).toContain(`calls resetMocks@${MOCK}`); + expect(edgesFrom('packages/api/__mocks__/helpers.ts')).toContain(`calls resetMocks@${MOCK}`); + expect(edgesFrom('packages/api/jest.setup.ts')).toContain(`calls resetMocks@${MOCK}`); + }); +}); + +describe('sync: adding or deleting a manual mock leaves production code as it was (#2455)', () => { + it('matches the graph before the mock came and after it went', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-jest-manual-mocks-sync-')); + const write = (files: Record): void => { + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + } + }; + write({ + 'package.json': FILES['package.json']!, + 'packages/common/package.json': FILES['packages/common/package.json']!, + [REAL]: FILES[REAL]!, + 'packages/api/package.json': FILES['packages/api/package.json']!, + 'packages/api/src/handler.tsx': FILES['packages/api/src/handler.tsx']!, + }); + const graph = await CodeGraph.init(dir, { index: true }); + const production = (): string[] => + graph.getNodesInFile('packages/api/src/handler.tsx').flatMap((n) => + graph.getOutgoingEdges(n.id).filter((e) => e.kind !== 'contains').map((e) => `${e.kind} ${graph.getNode(e.target)!.name}@${graph.getNode(e.target)!.filePath}`)).sort(); + try { + const before = production(); + expect(before).toContain(`calls formatLabel@${REAL}`); + write({ [MOCK]: FILES[MOCK]! }); + await graph.sync(); + expect(production()).toEqual(before); + fs.rmSync(path.join(dir, MOCK)); + await graph.sync(); + expect(production()).toEqual(before); + } finally { + graph.close(); + fs.rmSync(dir, { recursive: true, force: true }); + } + }, 60_000); +}); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 0dac3e1e1d..4c2755688b 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -41,10 +41,11 @@ import { createYielder, type MaybeYield } from './cooperative-yield'; import { crossTierEdges, hasCrossTierPattern, hasTestRequestPattern, testRequestEdges } from './tier-synthesizer'; import { enclosingFn, makeLineAt } from './synth-utils'; import { resolveImportPath } from './import-resolver'; -import { cppSpecializationArguments, crossesCodeBoundary, goRefQualification, jsCodeBindsName } from './name-matcher'; +import { cppSpecializationArguments, crossesCodeBoundary, goRefQualification, isManualMockHiddenFrom, isManualMockPath, jsCodeBindsName } from './name-matcher'; import { goStdlibMethods } from './go-stdlib-methods'; import { declaredComponent, exportedComponent, lazyValueModules, wrappedComponentName } from './frameworks/react'; import { jsxCode } from './jsx-code'; +import { isTestPath } from '../search/query-utils'; const REGISTRAR_NAME = /^(on[A-Z]\w*|subscribe|addListener|addEventListener|register|watch|listen|addCallback)$/; const DISPATCHER_NAME = /(emit|trigger|notify|dispatch|fire|publish|flush)/i; @@ -350,7 +351,9 @@ async function eventEmitterEdges(ctx: ResolutionContext, onYield: MaybeYield): P while ((m = ON_RE.exec(content))) { const handlerName = m[2] || m[3]; if (!handlerName) continue; - const handler = ctx.getNodesByName(handlerName).find((n) => n.kind === 'function' || n.kind === 'method'); + // A manual mock's function is a handler only test code registers (#2455). + const handler = ctx.getNodesByName(handlerName).find((n) => + (n.kind === 'function' || n.kind === 'method') && !isManualMockHiddenFrom(n.filePath, file)); if (!handler) continue; const map = handlersByEvent.get(m[1]!) ?? new Map(); map.set(handler.id, `${file}:${lineOf(m.index)}`); handlersByEvent.set(m[1]!, map); @@ -1984,7 +1987,9 @@ function jsxChild( ): Node | undefined { const own = ownValueOf(name); if (own) return own.component ?? own.declaration; - const candidates = ctx.getNodesByName(name).filter((n) => JSX_CHILD_KINDS.has(n.kind)); + // A manual mock's component renders only in a test (#2455): production + // code's `` never means `__mocks__/…`'s `Chart`. + const candidates = ctx.getNodesByName(name).filter((n) => JSX_CHILD_KINDS.has(n.kind) && !isManualMockHiddenFrom(n.filePath, file)); const local = candidates.find((n) => n.filePath === file); if (local) return local; if (importedFromPackage(ctx, name, file)) return undefined; @@ -2668,7 +2673,7 @@ async function vueTemplateEdges(ctx: ResolutionContext, onYield: MaybeYield): Pr // the package's: no other project symbol, nor a Nuxt auto-imported // component (`fallback`), is what it renders or calls. const resolve = (name: string, kinds: Set, fallback?: Node): Node | undefined => { - const matches = ctx.getNodesByName(name).filter((n) => kinds.has(n.kind)); + const matches = ctx.getNodesByName(name).filter((n) => kinds.has(n.kind) && !isManualMockHiddenFrom(n.filePath, file)); const local = matches.find((n) => n.filePath === file); if (local || importedFromPackage(ctx, name, file)) return local; return matches[0] ?? fallback; @@ -5173,6 +5178,16 @@ function synthPassCostHint(name: string): number { /** Fixed non-registry steps: goMethodContains, goImplements, dedupe-merge, insertMergedEdges. */ const FIXED_SYNTH_STEPS = 4; export const SYNTH_PROGRESS_STEPS = SYNTH_PASSES.length + FIXED_SYNTH_STEPS; + +/** The ids of every node a Jest or Vitest manual mock (`__mocks__/`) declares. */ +function manualMockNodeIds(ctx: ResolutionContext): Set { + const ids = new Set(); + for (const file of ctx.getAllFiles()) { + if (!isManualMockPath(file)) continue; + for (const n of ctx.getNodesInFile(file)) ids.add(n.id); + } + return ids; +} export async function synthesizeCallbackEdges( queries: QueryBuilder, ctx: ResolutionContext, @@ -5390,12 +5405,20 @@ export async function synthesizeCallbackEdges( } } + // A Jest or Vitest manual mock answers only for test code (#2455): whatever + // pass paired them, production code never links into a `__mocks__/` file. + const mockNodes = manualMockNodeIds(ctx); + const fromTestCode = (id: string): boolean => { + const source = ctx.getNodeById?.(id); + return !source || isTestPath(source.filePath); + }; const merged: Edge[] = []; const seen = new Set(); for (const e of passEdges.flat()) { const key = `${e.source}>${e.target}`; if (seen.has(key)) continue; seen.add(key); + if (mockNodes.size > 0 && mockNodes.has(e.target) && !fromTestCode(e.source)) continue; merged.push(e); } __mark('dedupe-merge'); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 055e96ba68..5f2e3494ad 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding, isManualMockPath } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -51,6 +51,7 @@ import { lexicalPathWithinRoot } from '../utils'; import type { CrossFileNodes, ReExport } from './types'; import { resurrectRefFromDroppedEdge } from './resurrect-ref'; import { LRUCache } from './lru-cache'; +import { isTestPath } from '../search/query-utils'; import { JS_BUILT_INS } from './js-builtins'; import { builtinModules } from 'module'; import { parse as parseJsonc } from 'jsonc-parser'; @@ -551,6 +552,9 @@ export class ReferenceResolver { * Clear internal caches */ clearCaches(): void { + this.mocklessContext = null; + this.mocklessBase = null; + this.manualMocks = undefined; this.nodeCache.clear(); this.fileCache.clear(); this.importMappingCache.clear(); @@ -1335,6 +1339,22 @@ export class ReferenceResolver { } resolveOne(ref: UnresolvedRef): ResolvedRef | null { + // Production code is resolved as if the project's manual mocks were not + // indexed (see mocklessContextFor); every strategy below reads the graph + // through this.context, so it answers for that view. + const outer = this.context; + const base = outer === this.mocklessContext && this.mocklessBase ? this.mocklessBase : outer; + const context = this.mocklessContextFor(ref, base) ?? base; + if (context === outer) return this.resolveOneIn(ref); + this.context = context; + try { + return this.resolveOneIn(ref); + } finally { + this.context = outer; + } + } + + private resolveOneIn(ref: UnresolvedRef): ResolvedRef | null { // A C/C++ "call" whose name is a function-like macro visible in this // translation unit is a macro expansion, not a call — it must never bind // to a same-named function in another file (#1838). @@ -3730,6 +3750,89 @@ export class ReferenceResolver { return ref.referenceKind === 'calls' && !isRustForeignImplCallable(target, ref, this.context) ? null : result; } + /** + * The context production code is resolved in when the project has Jest or + * Vitest manual mocks (`__mocks__/`): the graph without them. A mock + * re-declares its module's names by design and stands in for the module + * only while a test runs, so production code means what it would mean were + * the mocks not there (#2455): `formatLabel(name)`, whose import of a + * workspace package was not followed, is the package's `formatLabel`, never + * the nearer mock's. Adding or deleting a mock then changes nothing for + * production code, as a fresh index would have it. Test code, a mock + * included, keeps the whole graph: a mock is what Jest hands it. Null when + * the whole graph applies. + */ + private mocklessContextFor(ref: UnresolvedRef, base: ResolutionContext): ResolutionContext | null { + if (isTestPath(ref.filePath)) return null; + const mocks = this.manualMockNames(); + if (!mocks) return null; + if (this.mocklessContext && this.mocklessBase === base) return this.mocklessContext; + const isMock = (n: Node): boolean => isManualMockPath(n.filePath); + const real = (nodes: Node[]): Node[] => (nodes.some(isMock) ? nodes.filter((n) => !isMock(n)) : nodes); + // Only a name some mock declares can lose a node: other lookups pass + // straight through, and the filtered answers are kept. + const byName = new Map(); + const byLowerName = new Map(); + const byKind = new Map(); + const getMethodMatches = base.getMethodMatches; + const iterateNodesByKind = base.iterateNodesByKind; + this.mocklessBase = base; + this.mocklessContext = { + ...base, + getNodesByName: (name) => { + if (!mocks.names.has(name)) return base.getNodesByName(name); + let nodes = byName.get(name); + if (!nodes) byName.set(name, (nodes = real(base.getNodesByName(name)))); + return nodes; + }, + getNodesByLowerName: (lowerName) => { + if (!mocks.lowerNames.has(lowerName)) return base.getNodesByLowerName(lowerName); + let nodes = byLowerName.get(lowerName); + if (!nodes) byLowerName.set(lowerName, (nodes = real(base.getNodesByLowerName(lowerName)))); + return nodes; + }, + getNodesByQualifiedName: (qualifiedName) => real(base.getNodesByQualifiedName(qualifiedName)), + getNodesByKind: (kind) => { + let nodes = byKind.get(kind); + if (!nodes) byKind.set(kind, (nodes = real(base.getNodesByKind(kind)))); + return nodes; + }, + getMethodMatches: getMethodMatches + ? (typeName, methodName, language) => { + const nodes = getMethodMatches(typeName, methodName, language); + return mocks.names.has(methodName) ? real(nodes) : nodes; + } + : undefined, + iterateNodesByKind: iterateNodesByKind + ? function* (kind) { + for (const n of iterateNodesByKind(kind)) if (!isMock(n)) yield n; + } + : undefined, + }; + return this.mocklessContext; + } + + /** Built on first use from `mocklessBase`; clearCaches drops it, and the memos keyed by it go with it. */ + private mocklessContext: ResolutionContext | null = null; + private mocklessBase: ResolutionContext | null = null; + /** The names the project's manual mocks declare, null when it has none; read once per cache window. */ + private manualMocks: { names: Set; lowerNames: Set } | null | undefined; + + private manualMockNames(): { names: Set; lowerNames: Set } | null { + if (this.manualMocks !== undefined) return this.manualMocks; + const files = [...(this.knownFiles ?? this.queries.getAllFilePaths())].filter(isManualMockPath); + const names = new Set(); + const lowerNames = new Set(); + if (files.length > 0) { + for (const n of this.queries.getNodesByFiles(files)) { + names.add(n.name); + lowerNames.add(n.name.toLowerCase()); + } + } + this.manualMocks = names.size > 0 ? { names, lowerNames } : null; + return this.manualMocks; + } + /** Is `filePath` (project-relative) one of the files the index holds? */ private isIndexedFile(filePath: string): boolean { const normalized = filePath.replace(/\\/g, '/'); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 0862d98144..a396c42087 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1736,6 +1736,22 @@ function isTestSuitePath(filePath: string): boolean { return /(?:^|\/)(?:tests?|__tests__|specs?|e2e|unittests|[\w.]+[-_]unittests?)\//.test(lower) || /(?:^|\/)[A-Za-z0-9]*(?:Test|Tests|Spec)\//.test(filePath); } +/** A Jest or Vitest manual mock: a file under a `__mocks__/` directory. */ +export function isManualMockPath(filePath: string): boolean { + return /(?:^|\/)__mocks__\//.test(filePath.replace(/\\/g, '/').toLowerCase()); +} + +/** + * Whether a definition in `candidatePath` is a manual mock that a reference + * from `fromPath`, which is no test code, cannot mean (#2455): the rule the + * resolver's mock-free view applies to name matching, for the passes that + * pick a component or a handler by name. A production `` never + * renders a mock's `Chart`. + */ +export function isManualMockHiddenFrom(candidatePath: string, fromPath: string): boolean { + return isManualMockPath(candidatePath) && !isTestPath(fromPath); +} + const fileStem = (filePath: string): string => { const name = filePath.slice(filePath.lastIndexOf('/') + 1); const dot = name.lastIndexOf('.'); From 8651990abdbd939f5b6c221aea0049ac3d90d2c4 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 08:48:56 +0000 Subject: [PATCH 27/95] fix(js,ts): an import spelled with a trailing slash or a ./ segment links its file (#2792) Issue 2577: `import('@/widgets/Chart/')`, `from 'src/widgets/./Chart'`, a doubled slash, a `..` after an alias, and a workspace package's `from '@acme/ui/'` lost the edges the plain spelling gets. The file import was dropped, the names were matched by name alone, and the lazily loaded component rendered the importing file's own `lazy(...)` constant. Root cause: the fallback-alias, workspace, direct-path and own-config branches of the import resolver join the specifier as written and probe the result through tryWithExtensions. `fileExists` falls back to the disk, which accepts any spelling of a file, so the resolver returned `src/widgets/Chart//index.tsx` or `src/widgets/./Chart/index.tsx`, and every lookup keyed by the indexed path missed it. The relative and tsconfig-path branches go through path.resolve and were already fine. Fix (import-resolver.ts): tryWithExtensions spells each candidate as the index keys files (indexedSpelling): for the ES module family, a path with a doubled slash or a `.`/`..` segment is posix-normalized before it is probed and returned. That is how Node, tsc and the bundlers read a specifier. A trailing slash still names only a folder, because `Chart/` plus `.tsx` is no file. Other languages are untouched: a C include's `a/../b.h` follows the folder on disk, as PR 2610 noted. Validation: - __tests__/import-path-spelling.test.ts: 4 of 5 fail on main (the plain spelling passes). They cover the issue's spellings, a doubled slash, `..`, a trailing slash beside a same-named file, and a workspace member. All pass on the kernel and with CODEGRAPH_KERNEL=0. Dropping the normalization, or its doubled-slash check, fails 4. - Sync equals a fresh index (CLI) when the folder arrives after its importer, when the importer is re-spelled, when the folder is deleted and when the target is edited. - A/B on 11 TS/JS repos plus 2 controls, both arms with the kernel: pulse-sms-web (53 imports like '@/utils/') gains 65 edges and corrects 7, including 2 imports that had name-matched to same-named API classes; react-one gains 11 file imports and corrects 2 bindings. All 85 are right. The other nine (jvue-admin, vue-invoice-app, vue3-antdv-admin, v-ui, vue-blog-admin, dreammart, vite, outline, social-app) and the flask and gin controls are byte-identical. - Cost: fresh index, median of 3 interleaved: vite 7.59 s main vs 7.49 s, outline 16.63 s vs 16.94 s. Fixes #2577 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/import-path-spelling.test.ts | 123 +++++++++++++++++++++++++ src/resolution/import-resolver.ts | 32 ++++++- 3 files changed, 152 insertions(+), 4 deletions(-) create mode 100644 __tests__/import-path-spelling.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 25296bd419..690677cdfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -299,6 +299,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In React, a component declared through a wrapper with no render function of its own, like styled-components' `styled(ActionButton)` (with `.attrs(…)`, `.withConfig(…)` or type arguments too), emotion's and MUI's `styled(Box)(…)`, `memo(Button)` or `forwardRef(Render)`, now links to the component it wraps, so a render flow in `codegraph_explore`, a component's callers and its impact go on through the wrapper instead of stopping at it. The calls and values in a wrapper's styles, like `${(props) => theme.color(props)}` or a shared `css` block, now belong to the wrapper, as they do to any other constant, so a theme helper used only in styles has callers. A component that wraps its render function in `forwardRef(…)` or `memo(…)` now shows in the impact of the top-level constants it reads, as a function component does. A wrapper around a component that comes from a package, like react-router's `Link` or antd's `Typography.Text`, links to nothing, and the file that declares a wrapper no longer shows as referencing the component it wraps. Re-index React projects after upgrading. - In TypeScript, a type written with a qualifier, like `React.ReactNode`, `Y.Doc`, `TChat.ChatMessage` or `EQueryType.CLICKHOUSE`, now links to the member of what the qualifier names: a type from a package, like React's, no longer links to a class or component of your own that shares its name, and a member of an enum or a namespace your project declares links to that member instead of to another enum's or a same-named component. Re-index TypeScript projects after upgrading. - In JavaScript and TypeScript, an import written from the app's root with a leading slash, like bigbluebutton's `import Icon from '/imports/ui/components/common/icon/component'`, now links to the file it names when the app's `tsconfig.json` or `jsconfig.json` maps it, as `"paths": { "/*": ["*"] }` does. Before, such an import was taken for a package, so the module looked unused and every name it imports linked by name instead, often to the wrong one of several components of the same name, or to nothing. Re-index projects that import this way after upgrading. +- In JavaScript and TypeScript, an import spelled with a trailing slash, a `./` or `../` segment or a doubled slash, like `import('@/widgets/Chart/')`, `from 'src/widgets/./Chart'` or a workspace package's `from '@acme/ui/'`, now links like the plain spelling: to the file it names, and a lazily loaded component to the component that file exports. Before, the import was dropped, its names were matched by name alone, and the lazily loaded component pointed at the importing file's own `lazy(…)` constant. A trailing slash still names a folder: `@/widgets/Chart/` is `Chart/index.tsx`, never a `Chart.tsx` beside it. Thanks @inth3shadows for the report and @JiaMingWang-CN. (#2577) - In JavaScript and TypeScript, a call whose callee is not a name, like an immediately invoked function `(() => { … })()`, a curried call such as `connect(mapState)(View)`, `handlers[type]()` or `(0, fn)()`, no longer stores the callee's whole source text as the name of a reference that nothing in the project can answer. A single immediately invoked function could leave its entire body in the index that way. The calls inside such a callee, and the inner call of a curried one, are still recorded. A call through parentheses, a cast or a non-null assertion, like `(fn)()`, `(fn as F)()` or `fn!()`, now links to `fn` exactly as `fn()` does, so a parameter named `fn` no longer links to a same-named property elsewhere in the project. In TypeScript, a call with type arguments behind an operator, like `!settings.get('X')` or `void apiRequest()`, now links to what it calls, as `await load()` already did. - In JavaScript, TypeScript, Python, Go, Rust, Java and Kotlin, a function named in the value of a top-level constant or variable, or of a class field, like `const routes = { home }`, `HANDLERS = {"home": home}`, a Go `map[string]func()` table, a Rust `static` list of functions, `static handlers = { click: onClick }`, a Java `List.of(Registry::home)` field or a Kotlin `listOf(::home)` property, is now referenced once, by the constant, variable, field or property that holds it. The file or class around it no longer also shows up as a second reference from the same place, so a function's callers and impact list each table once. - `codegraph sync` and the file watcher now follow an edit that changes which declaration a module exports as its default, like `export default A` becoming `export default B` while `A` stays, or the module dropping its default. References through `import Y from './x'`, `{ default as Y }`, `require('./x').default`, a barrel that passes the default on or a route that renders it as a layout, and calls on a value typed by it, now link to the new default, or stay unlinked, as a fresh index has them. Before, they kept pointing at the old declaration until the importing file changed or the project was indexed again. diff --git a/__tests__/import-path-spelling.test.ts b/__tests__/import-path-spelling.test.ts new file mode 100644 index 0000000000..8f81b95453 --- /dev/null +++ b/__tests__/import-path-spelling.test.ts @@ -0,0 +1,123 @@ +/** + * Issue #2577: an import spelled with a trailing slash (`@/widgets/Chart/`), a + * `.` segment (`src/widgets/./Chart`), a doubled slash or a `..` segment names + * the same file as the plain spelling, but the alias, workspace and direct-path + * branches of the import resolver joined the specifier as written. The disk + * check accepts any spelling of a file, so the import came back as + * `src/widgets/Chart//index.tsx`, which no file node is keyed by: the file + * import was dropped, the names fell back to name matching, and a lazily loaded + * component rendered the importing file's own `lazy(…)` constant. + * + * A trailing slash still names a folder: `@/widgets/Chart/` is + * `Chart/index.tsx`, never a `Chart.tsx` beside it. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const CHART = `export default function ChartView() { return null; } +export function Legend() { return 1; } +`; +/** An app that imports and lazily loads `Chart` through `spec`. */ +const app = (spec: string): string => `import { lazy } from 'react'; +import { Legend } from '${spec}'; +const Chart = lazy(() => import('${spec}')); +export function App() { Legend(); return ; } +`; + +const FILES: Record = { + 'package.json': JSON.stringify({ name: 'root', private: true, workspaces: ['packages/*'], dependencies: { react: '^18.0.0' } }), + 'src/widgets/Chart/index.tsx': CHART, + 'src/Plain.tsx': app('@/widgets/Chart'), + 'src/TrailingSlash.tsx': app('@/widgets/Chart/'), + 'src/DotSegment.tsx': app('src/widgets/./Chart'), + 'src/DoubleSlash.tsx': app('@/widgets//Chart'), + 'src/DotDot.tsx': app('@/other/../widgets/Chart'), + 'src/other/x.ts': `export const x = 1; +`, + // A trailing slash names the folder, not the file beside it. + 'src/cards/Card.tsx': `export function Legend() { return 2; } +`, + 'src/cards/Card/index.tsx': CHART, + 'src/Folder.tsx': app('@/cards/Card/'), + // A workspace member, imported as `@acme/ui/` and through a `.` segment. + 'packages/ui/package.json': JSON.stringify({ name: '@acme/ui' }), + 'packages/ui/index.tsx': CHART, + 'packages/ui/widgets/legend.ts': `export function Legend() { return 3; } +`, + 'packages/app/package.json': JSON.stringify({ name: 'app', dependencies: { '@acme/ui': 'workspace:*', react: '^18.0.0' } }), + 'packages/app/src/Member.tsx': app('@acme/ui/'), + 'packages/app/src/MemberPath.ts': `import { Legend } from '@acme/ui/./widgets/legend'; +export function show() { return Legend(); } +`, +}; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-import-path-spelling-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}, 60_000); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `kind name@file` of every edge but `contains` leaving the nodes of `file`. */ +function edgesFrom(file: string): string[] { + const out: string[] = []; + for (const node of cg.getNodesInFile(file)) { + for (const e of cg.getOutgoingEdges(node.id)) { + if (e.kind === 'contains') continue; + const target = cg.getNode(e.target)!; + out.push(`${e.kind} ${target.name}@${target.filePath}`); + } + } + return out; +} + +/** What the plain spelling gives: the file import, the lazily loaded component, and the named import by import. */ +function expectLinked(file: string, module: string): void { + const edges = edgesFrom(file); + expect(edges).toContain(`imports ${path.posix.basename(module)}@${module}`); + expect(edges).toContain(`calls ChartView@${module}`); + expect(edges).toContain(`calls Legend@${module}`); + expect(edges.filter((e) => e.startsWith('calls Chart@'))).toEqual([]); +} + +describe('an import spelled with a trailing slash or a `.`, `..` or empty segment (#2577)', () => { + it('links like the plain spelling', () => { + expectLinked('src/Plain.tsx', 'src/widgets/Chart/index.tsx'); + }); + + it('with a trailing slash', () => { + expectLinked('src/TrailingSlash.tsx', 'src/widgets/Chart/index.tsx'); + }); + + it('with a `.` segment, a doubled slash and a `..` segment', () => { + expectLinked('src/DotSegment.tsx', 'src/widgets/Chart/index.tsx'); + expectLinked('src/DoubleSlash.tsx', 'src/widgets/Chart/index.tsx'); + expectLinked('src/DotDot.tsx', 'src/widgets/Chart/index.tsx'); + }); + + it('a trailing slash names the folder, not a same-named file beside it', () => { + expectLinked('src/Folder.tsx', 'src/cards/Card/index.tsx'); + expect(edgesFrom('src/Folder.tsx')).not.toContain('calls Legend@src/cards/Card.tsx'); + }); + + it('through a workspace member', () => { + expectLinked('packages/app/src/Member.tsx', 'packages/ui/index.tsx'); + expect(edgesFrom('packages/app/src/MemberPath.ts')).toEqual(expect.arrayContaining([ + 'imports legend.ts@packages/ui/widgets/legend.ts', + 'calls Legend@packages/ui/widgets/legend.ts', + ])); + }); +}); diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index d7e8447717..6d11a974eb 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -1330,14 +1330,38 @@ function resolveAliasedImport( return tryWithExt(importPath); } -/** The file `basePath` names: with the language's extensions or index files, as is, or as an emitted `.js`. */ +/** + * The file `basePath` names: with the language's extensions or index files, as + * is, or as an emitted `.js` — each spelled as the index keys it + * (indexedSpelling). + */ function tryWithExtensions(basePath: string, language: Language, context: ResolutionContext): string | null { for (const ext of EXTENSION_RESOLUTION[language] || []) { - const candidate = basePath + ext; + const candidate = indexedSpelling(basePath + ext, language); if (context.fileExists(candidate)) return candidate; } - if (context.fileExists(basePath)) return basePath; - return findSourceForEmittedSpecifier(basePath, language, context); + const asIs = indexedSpelling(basePath, language); + if (context.fileExists(asIs)) return asIs; + return findSourceForEmittedSpecifier(asIs, language, context); +} + +/** + * A path as the index keys its files. The alias, workspace and direct-path + * branches join an import specifier as written, so `@/widgets/Chart/` made + * `src/widgets/Chart//index.tsx` and `src/widgets/./Chart` made + * `src/widgets/./Chart/index.tsx`. The disk check accepts either spelling, but + * no file node or `files` row is keyed by it, so the import linked nothing + * (#2577). An ES module specifier is read lexically — by Node, tsc and the + * bundlers alike — so `.` and empty segments drop out and `..` folds, here as + * there. A trailing slash still names a folder: `Chart/` gives + * `Chart/index.tsx`, never `Chart.tsx`. Other languages keep the path as + * found: a C include's `a/../b.h` follows the folder `a` on disk, symlinks + * included. + */ +function indexedSpelling(filePath: string, language: Language): string { + return ESM_IMPORT_LANGUAGES.has(language) && /\/\/|(?:^|\/)\.\.?(?:\/|$)/.test(filePath) + ? path.posix.normalize(filePath) + : filePath; } /** From b2a84873edc29b801368bd1f5a9cbd6527bda268 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 09:04:05 +0000 Subject: [PATCH 28/95] fix(daemon): a long sync of another project no longer gets the shared daemon killed (#2793) Root cause: the daemon arms its liveness watchdog with its own project's index files (codegraph.db + -wal), and since PR #1231 a silent timeout defers the kill while those files advance. But the daemon also writes every other project a session opens with projectPath: the project lifecycle takes that project's writer lock, starts its watcher and runs the catch-up sync, all storing on the main thread. A long synchronous stretch of such a sync left the daemon's own files still, so the watchdog read it as a wedge and SIGKILLed a healthy daemon mid-sync; the next session reattached the project and the cycle repeated. A proxy serving in-process, armed with no files at all, had the same blind spot for its own project. Fix: addWatchdogProgressPaths() lets a process add files to its watchdog at runtime. The project lifecycle adds a project's db and -wal the moment this process becomes the project's writer, and drops them after it closes the project, so only indexes this process writes count, never one another process owns. Each change re-arms the watchdog with a new child whose argv carries the whole list; the old child is stopped after the new one starts. Added files take effect at once (a sync may block the loop right after); dropped ones wait a turn, so a server releasing its projects on the way out re-arms once or not at all. A message over the heartbeat pipe was tried first and dropped: Node's Writable holds a second write while one is still in flight, so the -wal line of a db + -wal registration never reached the child before a block that followed in the same tick (4 of 4 runs). Validation: - New __tests__/liveness-watchdog-projects.test.ts (11 cases, real watchdog child, lowered timeout): main's source 9 fail / 2 pass (the two guards), fix 11 pass. The lifecycle case arms the watchdog for project A, attaches B through the project lifecycle and stores into B's index on the main thread for 3s: killed at ~1s on main, runs to completion with the fix. - 8 mutations, each failed at least one case: no lifecycle registration, registration before the writer-lock check, no drop on release, no re-arm on add, no re-arm on drop, old child not stopped, drops re-armed at once, adds deferred. - Real daemon on Windows (watchdog timeout 5s, daemon keyed on a 2-file project, second project excalidraw 1d4dc2c with all 687 TS files edited, attached with codegraph_status projectPath), 3 interleaved rounds: main killed 3/3 about 13s after attaching B ("Main thread unresponsive for ~5s"), leaving B with 27,698 edges and 72,534 pending refs; the fix survived 3/3 through 7.8-9.3s main-thread blocks and finished B (61,120 edges, 0 pending), and 3/3 again on the final build (9.7-9.9s blocks). Fixes #2404 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/liveness-watchdog-projects.test.ts | 346 +++++++++++++++++++ src/mcp/liveness-watchdog.ts | 174 ++++++++-- src/mcp/project-lifecycle.ts | 20 +- 4 files changed, 511 insertions(+), 30 deletions(-) create mode 100644 __tests__/liveness-watchdog-projects.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 690677cdfd..27e5019787 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - VB.NET designer code such as `New System.Drawing.Point(…)` or `Me.Size = New System.Drawing.Size(…)` is no longer linked to a project type or member that only shares the name, and a class nested inside another is matched by its bare name only from inside that class or one that inherits it. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) +- The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404) - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) - `codegraph install` and `upgrade` no longer break Codex's `config.toml` when CodeGraph's `[mcp_servers.codegraph]` entry is written differently from how CodeGraph writes it, such as indented, with spaces or quotes inside the brackets, or with Windows line endings: the entry is now updated in place, where before a second copy was added and Codex could load none of the file. A file already broken that way is repaired on the next install or upgrade, an example of the entry inside a multiline setting such as `developer_instructions` is no longer mistaken for it, uninstall removes the entry however it is written, and an entry you wrote with dotted keys or as an inline table is left as you wrote it. When your `mcp_servers` settings are one inline table, CodeGraph now says it can't add itself instead of breaking the file. Thanks @dencoseca for the report, and @danusha2345 and @rudycelekli for the first fixes. (#2562) diff --git a/__tests__/liveness-watchdog-projects.test.ts b/__tests__/liveness-watchdog-projects.test.ts new file mode 100644 index 0000000000..b97fde92a1 --- /dev/null +++ b/__tests__/liveness-watchdog-projects.test.ts @@ -0,0 +1,346 @@ +/** + * The liveness watchdog counts disk progress on every index the process + * writes, not only the one it was armed with (#2404). + * + * The shared daemon arms its watchdog with its own project's index files, yet + * it also syncs every other project a session names by `projectPath`, on the + * same main thread. A long sync of one of those read as a wedge (heartbeat + * silence, no change on the watched files), so the healthy daemon was + * SIGKILLed mid-sync, and again after every reattach. A proxy serving + * in-process, armed with no files at all, had the same blind spot. + * + * Each case spawns a real process with the real watchdog child and a short + * timeout (the technique of liveness-watchdog.test.ts), then blocks that + * process's main thread while a file or an index advances. + */ +import { describe, it, expect, beforeAll, afterEach } from 'vitest'; +import { spawn } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { getCodeGraphDir } from '../src/directory'; +import { getWriterPidPath } from '../src/mcp/writer-lock'; + +const WATCHDOG = path.resolve(__dirname, '../dist/mcp/liveness-watchdog.js'); +const LIFECYCLE = path.resolve(__dirname, '../dist/mcp/project-lifecycle.js'); + +beforeAll(() => { + for (const file of [WATCHDOG, LIFECYCLE]) { + if (!fs.existsSync(file)) throw new Error(`Build the project first: ${file} is missing (run npm run build).`); + } +}); + +const dirs: string[] = []; +afterEach(async () => { + // A killed process may hold its SQLite files for a moment on Windows. + for (const dir of dirs.splice(0)) { + const deadline = Date.now() + 5000; + for (;;) { + try { + fs.rmSync(dir, { recursive: true, force: true }); + break; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code ?? ''; + if (!['EBUSY', 'ENOTEMPTY', 'EPERM'].includes(code) || Date.now() > deadline) throw error; + await new Promise((resolve) => setTimeout(resolve, 50)); + } + } + } +}); + +function tempDir(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-wdp-')); + dirs.push(dir); + return dir; +} + +interface Outcome { + code: number | null; + signal: NodeJS.Signals | 'TIMEOUT' | null; + stdout: string; + stderr: string; +} + +/** + * Run `body` in a fresh Node process whose watchdog times out after + * `timeoutMs`. Resolves once the process and its watchdog child are gone (the + * child shares the process's stderr, where it writes its kill notice). + */ +function runChild(body: string, timeoutMs: number, hardTimeoutMs = 15_000): Promise { + const child = spawn(process.execPath, ['-e', body], { + env: { ...process.env, CODEGRAPH_NO_WATCHDOG: '0', CODEGRAPH_WATCHDOG_TIMEOUT_MS: String(timeoutMs) }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (d) => { stdout += d; }); + child.stderr.on('data', (d) => { stderr += d; }); + return new Promise((resolve) => { + const timer = setTimeout(() => { + child.kill('SIGKILL'); + resolve({ code: null, signal: 'TIMEOUT', stdout, stderr }); + }, hardTimeoutMs); + child.on('close', (code, signal) => { + clearTimeout(timer); + resolve({ code, signal, stdout, stderr }); + }); + }); +} + +/** The process blocked its main thread and ran to its own exit(5). */ +function expectSurvived(r: Outcome): void { + expect(r.stdout, r.stderr).toContain('BLOCKING'); + expect(r.stderr).not.toContain('Main thread unresponsive'); + expect(r.signal).toBeNull(); + expect(r.code, r.stderr).toBe(5); +} + +/** The process blocked its main thread and the watchdog killed it there. */ +function expectKilled(r: Outcome): void { + expect(r.stdout, r.stderr).toContain('BLOCKING'); + expect(r.stderr).toContain('Main thread unresponsive'); + expect(r.code).not.toBe(5); +} + +/** Grow `file` every 100ms for `forMs`. */ +function growFile(file: string, forMs: number): Promise { + return new Promise((resolve) => { + const iv = setInterval(() => { fs.appendFileSync(file, 'x'.repeat(64)); }, 100); + setTimeout(() => { clearInterval(iv); resolve(); }, forMs); + }); +} + +/** Spawned-process source: block the main thread for `ms`, then exit(5). */ +const block = (ms: number): string => ` + require('fs').writeSync(1, 'BLOCKING\\n'); + const end = Date.now() + ${ms}; + while (Date.now() < end) {} + process.exit(5); +`; + +describe('files added to the watchdog at runtime (#2404)', () => { + it('counts a file added after arming as progress, like an armed one', async () => { + const file = path.join(tempDir(), 'other.db-wal'); + fs.writeFileSync(file, 'seed'); + const [r] = await Promise.all([ + runChild(` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + installMainThreadWatchdog(); + addWatchdogProgressPaths([${JSON.stringify(file)}]); + setTimeout(() => { ${block(2500)} }, 200); + `, 500), + growFile(file, 3500), + ]); + expectSurvived(r); + }, 20_000); + + it('counts every file one call adds, right before a block', async () => { + // Two files in one call, then the main thread blocks at once and only the + // second advances, as a sync advances the -wal it registered after the db. + const dir = tempDir(); + const db = path.join(dir, 'other.db'); + const wal = path.join(dir, 'other.db-wal'); + fs.writeFileSync(db, 'seed'); + fs.writeFileSync(wal, 'seed'); + const [r] = await Promise.all([ + runChild(` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + installMainThreadWatchdog(); + setTimeout(() => { + addWatchdogProgressPaths([${JSON.stringify(db)}, ${JSON.stringify(wal)}]); + ${block(2500)} + }, 200); + `, 500), + growFile(wal, 3500), + ]); + expectSurvived(r); + }, 20_000); + + it('counts a file added before the watchdog was armed', async () => { + const file = path.join(tempDir(), 'other.db-wal'); + fs.writeFileSync(file, 'seed'); + const [r] = await Promise.all([ + runChild(` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + addWatchdogProgressPaths([${JSON.stringify(file)}]); + installMainThreadWatchdog(); + setTimeout(() => { ${block(2500)} }, 200); + `, 500), + growFile(file, 3500), + ]); + expectSurvived(r); + }, 20_000); + + it('keeps counting a file while another registration of it holds', async () => { + const file = path.join(tempDir(), 'other.db-wal'); + fs.writeFileSync(file, 'seed'); + const [r] = await Promise.all([ + runChild(` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + installMainThreadWatchdog(); + const first = addWatchdogProgressPaths([${JSON.stringify(file)}]); + addWatchdogProgressPaths([${JSON.stringify(file)}]); + first(); + first(); // idempotent: must not drop the second registration + setTimeout(() => { ${block(2500)} }, 200); + `, 500), + growFile(file, 3500), + ]); + expectSurvived(r); + }, 20_000); + + it('stops counting a file once dropped', async () => { + const file = path.join(tempDir(), 'other.db-wal'); + fs.writeFileSync(file, 'seed'); + const [r] = await Promise.all([ + runChild(` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + installMainThreadWatchdog(); + const drop = addWatchdogProgressPaths([${JSON.stringify(file)}]); + setTimeout(() => { drop(); setTimeout(() => { ${block(2500)} }, 100); }, 100); + `, 500), + growFile(file, 3500), + ]); + expectKilled(r); + }, 20_000); + + it('leaves no watchdog process behind as files come and go', async () => { + const file = path.join(tempDir(), 'other.db-wal'); + fs.writeFileSync(file, 'seed'); + const child = spawn(process.execPath, ['-e', ` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + const handle = installMainThreadWatchdog(); + const drop = addWatchdogProgressPaths([${JSON.stringify(file)}]); + setTimeout(() => { + drop(); + setTimeout(() => { + addWatchdogProgressPaths([${JSON.stringify(file)}]); + setTimeout(() => { handle.stop(); process.exit(0); }, 300); + }, 200); + }, 200); + `], { + env: { ...process.env, CODEGRAPH_NO_WATCHDOG: '0', CODEGRAPH_WATCHDOG_TIMEOUT_MS: '5000', CODEGRAPH_MCP_DEBUG: '1' }, + stdio: ['ignore', 'ignore', 'pipe'], + }); + let stderr = ''; + child.stderr.on('data', (d) => { stderr += d; }); + // 'exit', not 'close': a watchdog child left alive would hold the stderr pipe. + const code = await new Promise((resolve) => child.on('exit', (c) => resolve(c))); + expect(code, stderr).toBe(0); + const armed = (): number[] => [...stderr.matchAll(/armed \(child pid (\d+)\)/g)].map((m) => Number(m[1])); + for (const until = Date.now() + 2000; armed().length < 4 && Date.now() < until;) { + await new Promise((resolve) => setTimeout(resolve, 25)); + } + const pids = armed(); + // The first child, then one per change (added, dropped, added again). + expect(pids, stderr).toHaveLength(4); + const alive = (pid: number): boolean => { try { process.kill(pid, 0); return true; } catch { return false; } }; + const deadline = Date.now() + 5000; + while (pids.some(alive) && Date.now() < deadline) await new Promise((resolve) => setTimeout(resolve, 50)); + const survivors = pids.filter(alive); + for (const pid of survivors) { try { process.kill(pid, 'SIGKILL'); } catch { /* gone */ } } + expect(survivors).toEqual([]); + }, 20_000); + + it('starts no new watchdog for files dropped on the way out', async () => { + // A server letting go of its projects as it exits drops their files all at + // once; that must not spawn a watchdog per project just to exit with it. + const files = [0, 1, 2].map((i) => path.join(tempDir(), `other${i}.db-wal`)); + const child = spawn(process.execPath, ['-e', ` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + installMainThreadWatchdog(); + const drops = ${JSON.stringify(files)}.map((file) => addWatchdogProgressPaths([file])); + setTimeout(() => { for (const drop of drops) drop(); process.exit(0); }, 200); + `], { + env: { ...process.env, CODEGRAPH_NO_WATCHDOG: '0', CODEGRAPH_WATCHDOG_TIMEOUT_MS: '5000', CODEGRAPH_MCP_DEBUG: '1' }, + stdio: ['ignore', 'ignore', 'pipe'], + }); + let stderr = ''; + child.stderr.on('data', (d) => { stderr += d; }); + const code = await new Promise((resolve) => child.on('close', (c) => resolve(c))); + expect(code, stderr).toBe(0); + // The first child, then one per file added; none for the drops. + expect([...stderr.matchAll(/armed \(child pid (\d+)\)/g)], stderr).toHaveLength(4); + }, 20_000); + + it('does not mistake a file joining the watch list for the disk advancing', async () => { + // The file never changes. The first silent timeout must compare it with + // itself, not with the list from before it joined: kill at ~1.5s, well + // before the process's own exit at ~2.6s. + const file = path.join(tempDir(), 'other.db-wal'); + fs.writeFileSync(file, 'seed'); + const r = await runChild(` + const { installMainThreadWatchdog, addWatchdogProgressPaths } = require(${JSON.stringify(WATCHDOG)}); + installMainThreadWatchdog(); + addWatchdogProgressPaths([${JSON.stringify(file)}]); + setTimeout(() => { ${block(2400)} }, 200); + `, 1500); + expectKilled(r); + }, 20_000); +}); + +describe('another project this process writes (#2404)', () => { + /** A project with an empty index, as `codegraph init` leaves one. */ + function makeProject(): { root: string; db: string } { + const root = tempDir(); + CodeGraph.initSync(root).close(); + return { root, db: path.join(getCodeGraphDir(root), 'codegraph.db') }; + } + + /** + * Arm the watchdog for project A, as the daemon keyed on A arms it, attach + * project B through the project lifecycle the daemon opens every + * `projectPath` with, then store into B's index synchronously on the main + * thread for 3s, as a long sync of B does. The graph object stands in for + * CodeGraph: the lifecycle only watches, syncs and closes it. + */ + function attachAndStore(a: { db: string }, b: { root: string; db: string }, opts: { release?: boolean } = {}): Promise { + return runChild(` + const { installMainThreadWatchdog } = require(${JSON.stringify(WATCHDOG)}); + const { acquireProject } = require(${JSON.stringify(LIFECYCLE)}); + const { DatabaseSync } = require('node:sqlite'); + const graph = { watch: () => true, sync: async () => ({}), close() {}, isIndexing: () => false }; + installMainThreadWatchdog({ progressPaths: [${JSON.stringify(a.db)}, ${JSON.stringify(`${a.db}-wal`)}] }); + const lease = acquireProject(${JSON.stringify(b.root)}, () => graph, {}); + lease.ready().then(async () => { + if (${opts.release ? 'true' : 'false'}) { + await lease.release(); + await new Promise((resolve) => setTimeout(resolve, 50)); // dropped files re-arm on the next turn + } + require('fs').writeSync(1, 'BLOCKING\\n'); + const db = new DatabaseSync(${JSON.stringify(b.db)}); + db.exec('CREATE TABLE IF NOT EXISTS store_2404 (x BLOB)'); + const end = Date.now() + 3000; + while (Date.now() < end) db.exec('INSERT INTO store_2404 VALUES (randomblob(2048))'); + db.close(); + process.exit(5); + }); + `, 1000); + } + + it('lets a long store into another project it now writes run to completion', async () => { + const a = makeProject(); + const b = makeProject(); + const r = await attachAndStore(a, b); + expectSurvived(r); + }, 20_000); + + it('does not count a project whose writer is another process', async () => { + const a = makeProject(); + const b = makeProject(); + // B's live writer is someone else (this test process, as a direct server + // that has caught up): the spawned process only reads B, so B's files + // advancing says nothing about its own main thread. + fs.writeFileSync(getWriterPidPath(b.root), JSON.stringify({ pid: process.pid, mode: 'direct', startedAt: Date.now(), ready: true }) + '\n'); + const r = await attachAndStore(a, b); + expectKilled(r); + }, 20_000); + + it('stops counting a project once it is released', async () => { + const a = makeProject(); + const b = makeProject(); + const r = await attachAndStore(a, b, { release: true }); + expectKilled(r); + }, 20_000); +}); diff --git a/src/mcp/liveness-watchdog.ts b/src/mcp/liveness-watchdog.ts index e0f2e2e2df..4e459de628 100644 --- a/src/mcp/liveness-watchdog.ts +++ b/src/mcp/liveness-watchdog.ts @@ -30,10 +30,11 @@ * * **Won't fire on real work.** Heavy parsing runs in the parse worker * (off-thread) and the daemon's indexing shells out to a child process, so the - * daemon's main thread only ever does fast, bounded work. The default timeout - * is ~300× the 5h #850 wedge shorter, yet far longer than any legitimate - * main-thread block. Opt out with `CODEGRAPH_NO_WATCHDOG=1`; tune with - * `CODEGRAPH_WATCHDOG_TIMEOUT_MS`. + * daemon's main thread mostly does fast, bounded work; the exception, a sync + * storing into SQLite, is what the disk-progress deferral below is for. The + * default timeout is ~300× the 5h #850 wedge shorter, yet far longer than any + * legitimate main-thread block. Opt out with `CODEGRAPH_NO_WATCHDOG=1`; tune + * with `CODEGRAPH_WATCHDOG_TIMEOUT_MS`. * * **Disk-progress deferral (`progressPaths`).** The CLI `index`/`init` path is * different: it runs the SQLite store on this thread, and one long synchronous @@ -49,6 +50,18 @@ * coinciding with unrelated file activity — or I/O hung beyond all reason — * still dies. A true wedge with no disk progress dies at the base timeout, * exactly as before. + * + * **Every index the process writes counts (#2404).** An MCP server syncs on + * its main thread too, and not only the project it was armed for: the shared + * daemon writes every project a session names by `projectPath`, and a proxy's + * in-process fallback writes its own. {@link addWatchdogProgressPaths} adds a + * project's index files the moment this process becomes its writer and drops + * them when it lets go; each change re-arms the watchdog with a new child + * whose argv carries the whole list. A new process gets its list the instant + * it is spawned, whatever the parent does next: a message on the heartbeat + * pipe could not promise that, since a pipe write queued behind one still in + * flight waits for an event-loop turn, and a sync can block the loop right + * after the project is added. */ import * as fs from 'fs'; import * as os from 'os'; @@ -174,27 +187,71 @@ export interface WatchdogOptions { /** * Files whose size/mtime advancing counts as forward progress (the SQLite * DB + `-wal` for an in-process indexer). With paths supplied, a silent - * timeout only kills when the files did NOT advance — see the header. Omit - * for pure heartbeat behavior (the daemon, whose main thread never runs - * long synchronous work). + * timeout only kills when the files did NOT advance — see the header. The + * files of other indexes the process comes to write join at runtime through + * {@link addWatchdogProgressPaths}; with neither, a silent timeout kills. */ progressPaths?: string[]; } /** - * Install the main-thread liveness watchdog for a long-lived process. Returns a - * handle to stop it, or `null` when disabled or when the child can't be spawned - * (degraded, never throws — a missing watchdog must never keep a process from - * starting). + * Files added with {@link addWatchdogProgressPaths} and not yet dropped, with + * how many registrations hold each. */ -export function installMainThreadWatchdog(options: WatchdogOptions = {}): WatchdogHandle | null { - if (isEnvTruthy(process.env.CODEGRAPH_NO_WATCHDOG)) return null; +const addedProgressPaths = new Map(); - const timeoutMs = parseWatchdogTimeoutMs(process.env.CODEGRAPH_WATCHDOG_TIMEOUT_MS); - const checkMs = deriveCheckIntervalMs(timeoutMs); - const capMs = timeoutMs * PROGRESS_CAP_MULTIPLIER; - const progressPaths = options.progressPaths ?? []; +/** + * Tells each armed watchdog the list of files changed: `added` when a file + * joined it, false when files only left it. + */ +const armedWatchdogs = new Set<(added: boolean) => void>(); +/** + * Count `paths` advancing as forward progress for this process's liveness + * watchdog, armed now or later, until the returned function is called + * (#2404). For the index files of a project this process has just become the + * writer of: its syncs store on the main thread, and without this a long one + * reads as a wedge whenever the project is not the one the watchdog was armed + * for. Registrations of one file nest, and the returned function is + * idempotent. + */ +export function addWatchdogProgressPaths(paths: readonly string[]): () => void { + let added = false; + for (const file of paths) { + const held = addedProgressPaths.get(file) ?? 0; + addedProgressPaths.set(file, held + 1); + if (held === 0) added = true; + } + if (added) for (const changed of armedWatchdogs) changed(true); + let dropped = false; + return () => { + if (dropped) return; + dropped = true; + let removed = false; + for (const file of paths) { + const held = addedProgressPaths.get(file) ?? 0; + if (held > 1) { + addedProgressPaths.set(file, held - 1); + } else if (held === 1) { + addedProgressPaths.delete(file); + removed = true; + } + } + if (removed) for (const changed of armedWatchdogs) changed(false); + }; +} + +/** One watchdog child process, watching a fixed list of files. */ +interface WatchdogChild { + paths: readonly string[]; + /** Write one heartbeat byte. */ + beat(): void; + /** End the child: EOF, then a kill for good measure. */ + stop(): void; +} + +/** Spawn a watchdog child for `paths`, or null when it can't be spawned. */ +function spawnWatchdogChild(timeoutMs: number, checkMs: number, capMs: number, paths: readonly string[]): WatchdogChild | null { let child: ChildProcess; try { // No execArgv inheritance (unlike Worker), so the child carries none of our @@ -202,7 +259,7 @@ export function installMainThreadWatchdog(options: WatchdogOptions = {}): Watchd // fd 2 so the kill notice lands wherever the parent logs (daemon.log). child = spawn( process.execPath, - ['-e', CHILD_SOURCE, String(process.pid), String(timeoutMs), String(capMs), ...progressPaths], + ['-e', CHILD_SOURCE, String(process.pid), String(timeoutMs), String(capMs), ...paths], { stdio: ['pipe', 'ignore', 'inherit'], windowsHide: true, @@ -228,27 +285,88 @@ export function installMainThreadWatchdog(options: WatchdogOptions = {}): Watchd stdin.on('error', () => { /* child gone; heartbeat writes are best-effort */ }); child.on('error', (err) => debug(`child error: ${err.message}`)); - // Heartbeat: a byte per tick. When the main thread wedges, these stop and the - // child's timeout fires. unref'd so it never keeps the process alive itself. - const heartbeat = setInterval(() => { - try { stdin.write('\n'); } catch { /* child gone */ } - }, checkMs); - heartbeat.unref(); - // Neither the child nor its pipe should keep the parent alive past its work. child.unref(); try { (stdin as unknown as { unref?: () => void }).unref?.(); } catch { /* ignore */ } - debug(`armed (child pid ${child.pid ?? '?'}): timeoutMs=${timeoutMs} checkMs=${checkMs} progressPaths=${progressPaths.length}`); + debug(`armed (child pid ${child.pid ?? '?'}): timeoutMs=${timeoutMs} checkMs=${checkMs} progressPaths=${paths.length}`); + + return { + paths, + beat(): void { + try { stdin.write('\n'); } catch { /* child gone */ } + }, + stop(): void { + try { stdin.end(); } catch { /* ignore */ } // EOF -> child exits cleanly + try { child.kill(); } catch { /* ignore */ } // belt-and-suspenders + }, + }; +} + +/** + * Install the main-thread liveness watchdog for a long-lived process. Returns a + * handle to stop it, or `null` when disabled or when the child can't be spawned + * (degraded, never throws — a missing watchdog must never keep a process from + * starting). + */ +export function installMainThreadWatchdog(options: WatchdogOptions = {}): WatchdogHandle | null { + if (isEnvTruthy(process.env.CODEGRAPH_NO_WATCHDOG)) return null; + + const timeoutMs = parseWatchdogTimeoutMs(process.env.CODEGRAPH_WATCHDOG_TIMEOUT_MS); + const checkMs = deriveCheckIntervalMs(timeoutMs); + const capMs = timeoutMs * PROGRESS_CAP_MULTIPLIER; + const armedPaths = options.progressPaths ?? []; + const watchedPaths = (): string[] => [...new Set([...armedPaths, ...addedProgressPaths.keys()])]; + + let current = spawnWatchdogChild(timeoutMs, checkMs, capMs, watchedPaths()); + if (!current) return null; + + // Heartbeat: a byte per tick. When the main thread wedges, these stop and the + // child's timeout fires. unref'd so it never keeps the process alive itself. + const heartbeat = setInterval(() => current?.beat(), checkMs); + heartbeat.unref(); + + // The list of files changed (#2404): a child for the new list takes over, + // then the old one goes. Always from a turning event loop, so the old child, + // its last heartbeat recent, has no reason to kill meanwhile; a spawn that + // fails leaves it watching the old list. An added file takes effect at once, + // since a sync may block the loop right after. Dropped files wait for the + // next turn, so a process letting go of several projects, as one does on its + // way out, re-arms once or not at all. + let pendingRearm: NodeJS.Timeout | null = null; + const rearm = (): void => { + if (pendingRearm) clearTimeout(pendingRearm); + pendingRearm = null; + if (!current) return; + const paths = watchedPaths(); + if (paths.length === current.paths.length && paths.every((p) => current!.paths.includes(p))) return; + const next = spawnWatchdogChild(timeoutMs, checkMs, capMs, paths); + if (!next) return; + const previous = current; + current = next; + previous.stop(); + }; + const changed = (added: boolean): void => { + if (added) { + rearm(); + } else if (!pendingRearm) { + pendingRearm = setTimeout(rearm, 0); + pendingRearm.unref(); + } + }; + armedWatchdogs.add(changed); let stopped = false; return { stop(): void { if (stopped) return; stopped = true; + armedWatchdogs.delete(changed); + if (pendingRearm) clearTimeout(pendingRearm); + pendingRearm = null; clearInterval(heartbeat); - try { stdin.end(); } catch { /* ignore */ } // EOF -> child exits cleanly - try { child.kill(); } catch { /* ignore */ } // belt-and-suspenders + current?.stop(); + current = null; }, }; } diff --git a/src/mcp/project-lifecycle.ts b/src/mcp/project-lifecycle.ts index 1a634815eb..88ffc50ae9 100644 --- a/src/mcp/project-lifecycle.ts +++ b/src/mcp/project-lifecycle.ts @@ -1,10 +1,12 @@ /** Shared synchronization ownership for projects accessed by MCP engines (#1835). */ import { realpathSync } from 'fs'; +import * as path from 'path'; import type { Socket } from 'net'; import type CodeGraph from '../index'; -import { canonicalProjectRoot, isInitialized } from '../directory'; +import { canonicalProjectRoot, getCodeGraphDir, isInitialized } from '../directory'; import { LockUnavailableError, watchDisabledReason } from '../sync'; import { getDaemonSocketCandidates } from './daemon-paths'; +import { addWatchdogProgressPaths } from './liveness-watchdog'; import { connectWithHello } from './proxy'; import { markWriterReady, readWriterLock, releaseWriterLock, tryAcquireWriterLock } from './writer-lock'; @@ -20,6 +22,8 @@ interface Project { socket: Socket | null; timer: NodeJS.Timeout; options: Parameters[0]; + /** Drops the index files from the liveness watchdog's progress; set while this process writes them (#2404). */ + dropProgress: (() => void) | null; } export interface ProjectLease { @@ -42,7 +46,7 @@ export function acquireProject( if (!project) { const cg = open(); project = { key, cg, refs: 0, owner: false, caughtUp: false, retirement: null, gate: null, socket: null, options, - timer: setInterval(() => { void ready(root, project!); }, 1000) }; + dropProgress: null, timer: setInterval(() => { void ready(root, project!); }, 1000) }; project.timer.unref(); projects.set(key, project); } @@ -86,6 +90,9 @@ function retire(root: string, project: Project): Promise { projects.delete(project.key); project.cg.close(); if (project.owner) releaseWriterLock(root); + // After the close, which may still checkpoint the WAL on this thread. + project.dropProgress?.(); + project.dropProgress = null; resolve(); }; finish(); @@ -144,6 +151,15 @@ async function activate(root: string, project: Project): Promise { return; } project.owner = true; + // This process writes the index from here on, in long synchronous stretches + // on the main thread: the catch-up below and every watcher sync. The + // liveness watchdog must count its files advancing as progress, as it does + // for the project it was armed with; otherwise a long sync of any other + // project reads as a wedge and the server is killed mid-sync (#2404). + if (!project.dropProgress) { + const db = path.join(getCodeGraphDir(root), 'codegraph.db'); + project.dropProgress = addWatchdogProgressPaths([db, `${db}-wal`]); + } const disabled = watchDisabledReason(root); if (disabled) { process.stderr.write(`[CodeGraph MCP] File watcher disabled for ${root} — ${disabled}.\n`); From 8073637d667d660aa0654bf6ea021f98402d1dd4 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 09:44:58 +0000 Subject: [PATCH 29/95] fix(db): reopening an interrupted index no longer gets the server killed at every start (#2794) Root cause: a run killed inside a bulk-load window (a crash, an OOM kill, the liveness watchdog itself) leaves that window's secondary indexes dropped, and since PR #1562 every writable open rebuilds them in DatabaseConnection.open(), one CREATE INDEX after another on the caller's thread. Each one scans and sorts its whole table. On a large index the sort writes nothing to the database or its WAL, so on an MCP server the liveness watchdog saw a silent main thread with no disk progress, and killed the process before the index committed. Each restart died at the same index, and the proxy's in-process fallback died the same way on the request it re-served. Fix: an open on a watched event loop leaves the rebuild to a worker thread with its own connection, and the project is served once the rebuild is done. - DatabaseConnection.open({ deferIndexRecovery }) records the missing DDL instead of running it; recoverIndexesOffThread() runs it on a worker (single-flight, rejects on failure so the next call retries; SQLite's default temp store, so the sort spills to disk; the worker collects its garbage and exits by itself, per worker-teardown.ts). - CodeGraph.open() (the MCP default project, the CLI) defers and awaits it. - openSync({ deferIndexRecovery }) plus cg.recoverIndexes() for the MCP's synchronous opens: the ToolHandler's projectPath projects and the engine's retry path. ToolHandler.execute awaits the rebuild before serving (not time-boxed: reads without the indexes scan whole tables), the project lifecycle awaits it before its catch-up sync writes, and the retry path starts the query pool after it. - Synchronous opens that ask for nothing keep rebuilding in line, as before. Validation: - Reproduced on main b2a84873: a django 79502d0 init killed 1.5s into the resolution window left the field report's index set missing (5 ref + 5 edge indexes, index_state indexing); the in-line rebuild is about 750ms of main-thread work there. - Real daemon on Windows, a 3M-ref / 1M-edge index left the same way (781 MB), watchdog 1.5s: main was killed in 4 of 4 starts, stuck at 2 of 10 indexes, the launcher's in-process fallback killed too, no reply. The fix rebuilt all 10 in about 11s with no main-thread block over 506ms, answered the call, and the next start opened in about 120ms. - New __tests__/index-recovery-off-thread.test.ts (9 cases): main's source 8 fail / 1 pass (the in-line guard), fix 9 pass. Spies show no CREATE INDEX on the server thread for CodeGraph.open, a projectPath call (watched and not), the retry path and the startup path; a real watchdog (500ms) kills main mid-rebuild of a 300k-ref index and leaves the fix alone. 6 mutations each failed at least one case. - Contributor PR #1888 and the stale draft PR #1919 were read for guidance; this ports their worker recovery to current main's open paths. Fixes #1887 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/index-recovery-off-thread.test.ts | 261 ++++++++++++++++++++ src/db/index.ts | 114 ++++++++- src/index.ts | 41 ++- src/mcp/engine.ts | 15 +- src/mcp/project-lifecycle.ts | 4 + src/mcp/tools.ts | 24 +- 7 files changed, 445 insertions(+), 15 deletions(-) create mode 100644 __tests__/index-recovery-off-thread.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 27e5019787..0672e8da56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404) +- An index left incomplete by an indexing run that was stopped partway, by a crash, an out-of-memory kill or the server's own hang check, no longer gets the background server killed with "Main thread unresponsive" every time it starts: on a large project, restoring the database indexes the run had set aside took longer than the hang check allows, so each restart died at the same point and the index was never repaired. They are now restored in the background, and the project is served once they are back. Thanks @cbeaulieu-gt for the report and the first fix. (#1887) - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) - `codegraph install` and `upgrade` no longer break Codex's `config.toml` when CodeGraph's `[mcp_servers.codegraph]` entry is written differently from how CodeGraph writes it, such as indented, with spaces or quotes inside the brackets, or with Windows line endings: the entry is now updated in place, where before a second copy was added and Codex could load none of the file. A file already broken that way is repaired on the next install or upgrade, an example of the entry inside a multiline setting such as `developer_instructions` is no longer mistaken for it, uninstall removes the entry however it is written, and an entry you wrote with dotted keys or as an inline table is left as you wrote it. When your `mcp_servers` settings are one inline table, CodeGraph now says it can't add itself instead of breaking the file. Thanks @dencoseca for the report, and @danusha2345 and @rudycelekli for the first fixes. (#2562) diff --git a/__tests__/index-recovery-off-thread.test.ts b/__tests__/index-recovery-off-thread.test.ts new file mode 100644 index 0000000000..125639995a --- /dev/null +++ b/__tests__/index-recovery-off-thread.test.ts @@ -0,0 +1,261 @@ +/** + * Recovery of an interrupted bulk load runs off the event loop (#1887). + * + * A run killed inside a bulk-load window (a crash, an OOM kill, the liveness + * watchdog) leaves the window's secondary indexes dropped, and the next open + * rebuilds them (#1556). Rebuilding one scans and sorts its whole table, so on + * a large index the rebuild ran for minutes on the opener's main thread: an MCP + * server's liveness watchdog killed it before the index committed, and every + * restart was killed the same way. Opens on a watched event loop now leave the + * rebuild to a worker thread with its own connection, and wait for it before + * the project is served. + * + * The spies record the SQL every connection on THIS thread runs: a rebuild + * here shows up as a CREATE INDEX. Everything runs against the built dist, as + * the MCP engine tests do (the engine lazy-loads the CodeGraph chain). + */ +import { describe, it, expect, beforeAll, afterEach, vi } from 'vitest'; +import { spawn } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; + +const DIST = path.resolve(__dirname, '../dist'); +const CodeGraph = (require(path.join(DIST, 'index.js')) as typeof import('../src/index')).default; +const { DatabaseConnection, getDatabasePath } = require(path.join(DIST, 'db', 'index.js')) as typeof import('../src/db'); +const { createDatabase } = require(path.join(DIST, 'db', 'sqlite-adapter.js')) as typeof import('../src/db/sqlite-adapter'); +const { MCPEngine } = require(path.join(DIST, 'mcp', 'engine.js')) as typeof import('../src/mcp/engine'); + +beforeAll(() => { + if (!fs.existsSync(path.join(DIST, 'index.js'))) throw new Error('Build the project first (npm run build).'); +}); + +/** The indexes the resolution phase's ref and edge windows drop. */ +const WINDOW_INDEXES = [ + 'idx_unresolved_from_node', 'idx_unresolved_name', 'idx_unresolved_file_path', 'idx_unresolved_from_name', + 'idx_unresolved_failed_tail', 'idx_edges_kind', 'idx_edges_source_kind', 'idx_edges_target_kind', + 'idx_edges_provenance', 'idx_edges_synthesis_site', +]; + +const dirs: string[] = []; +const engines: Array<{ stop(): Promise }> = []; +afterEach(async () => { + vi.restoreAllMocks(); + for (const engine of engines.splice(0)) await engine.stop(); + for (const dir of dirs.splice(0)) { + const deadline = Date.now() + 5000; + for (;;) { + try { + fs.rmSync(dir, { recursive: true, force: true }); + break; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code ?? ''; + if (!['EBUSY', 'ENOTEMPTY', 'EPERM'].includes(code) || Date.now() > deadline) throw error; + await new Promise((resolve) => setTimeout(resolve, 50)); + } + } + } +}); + +/** + * A project whose index a killed run left inside the resolution phase's bulk + * window: rows in place, the window's ten indexes dropped. `failed` refs keep a + * catch-up sync from resolving them again. + */ +function interruptedProject(rows: { nodes: number; refs: number; edges: number }): { root: string; dbPath: string } { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-recover-')); + dirs.push(root); + CodeGraph.initSync(root).close(); + const dbPath = getDatabasePath(root); + const conn = DatabaseConnection.open(dbPath); + const db = conn.getDb(); + conn.beginBulkNodeLoad(); + db.exec(`WITH RECURSIVE n(i) AS (SELECT 0 UNION ALL SELECT i + 1 FROM n WHERE i < ${rows.nodes - 1}) + INSERT INTO nodes (id, kind, name, qualified_name, file_path, language, start_line, end_line, start_column, end_column, updated_at) + SELECT 'n' || i, 'function', 'fn_' || (i * 7919 % ${rows.nodes}), 'mod.fn_' || i, 'src/f' || (i % 500) || '.ts', 'typescript', i, i + 3, 0, 1, 0 FROM n`); + db.exec(`WITH RECURSIVE n(i) AS (SELECT 0 UNION ALL SELECT i + 1 FROM n WHERE i < ${rows.refs - 1}) + INSERT INTO unresolved_refs (from_node_id, reference_name, reference_kind, line, col, file_path, language, status, name_tail) + SELECT 'n' || (i % ${rows.nodes}), 'callee_' || (i * 104729 % 1000003), 'calls', i % 2000, i % 80, + 'src/f' || (i * 31 % 500) || '.ts', 'typescript', 'failed', 'tail_' || (i * 13 % 50021) FROM n`); + db.exec(`WITH RECURSIVE n(i) AS (SELECT 0 UNION ALL SELECT i + 1 FROM n WHERE i < ${rows.edges - 1}) + INSERT OR IGNORE INTO edges (source, target, kind, line, col, provenance) + SELECT 'n' || (i % ${rows.nodes}), 'n' || (i * 7907 % ${rows.nodes}), CASE i % 2 WHEN 0 THEN 'calls' ELSE 'references' END, + i, 0, CASE WHEN i % 10 = 0 THEN 'heuristic' ELSE NULL END FROM n`); + conn.endBulkNodeLoad(); + conn.beginBulkRefLoad(); + conn.beginBulkEdgeLoad(); + conn.close(); + return { root, dbPath }; +} + +/** Which of the window's indexes the database holds now. */ +function presentWindowIndexes(dbPath: string): string[] { + const { db } = createDatabase(dbPath, { readOnly: true }); + try { + const names = new Set((db.prepare("SELECT name FROM sqlite_master WHERE type = 'index'").all() as Array<{ name: string }>).map((r) => r.name)); + return WINDOW_INDEXES.filter((name) => names.has(name)); + } finally { + db.close(); + } +} + +/** Record the index DDL every connection on this thread runs from now on. */ +function spyOnIndexDdl(): string[] { + const { db: probe } = createDatabase(':memory:'); + const proto = Object.getPrototypeOf(probe) as { exec(sql: string): void; prepare(sql: string): unknown }; + probe.close(); + const seen: string[] = []; + const record = (sql: string): void => { if (/^\s*CREATE\s+INDEX/i.test(sql)) seen.push(sql); }; + const exec = proto.exec; + const prepare = proto.prepare; + vi.spyOn(proto, 'exec').mockImplementation(function (this: unknown, sql: string) { record(sql); return exec.call(this, sql); }); + vi.spyOn(proto, 'prepare').mockImplementation(function (this: unknown, sql: string) { record(sql); return prepare.call(this, sql); }); + return seen; +} + +const SMALL = { nodes: 2000, refs: 20000, edges: 5000 }; + +describe('recovering an interrupted bulk load off the event loop (#1887)', () => { + it('leaves a deferred rebuild to a worker that restores every index', async () => { + const { dbPath } = interruptedProject(SMALL); + expect(presentWindowIndexes(dbPath)).toEqual([]); + const seen = spyOnIndexDdl(); + const conn = DatabaseConnection.open(dbPath, { deferIndexRecovery: true }); + try { + expect(conn.hasDeferredIndexRecovery()).toBe(true); + expect(presentWindowIndexes(dbPath)).toEqual([]); + const first = conn.recoverIndexesOffThread(); + expect(conn.recoverIndexesOffThread()).toBe(first); // one rebuild at a time + await first; + expect(conn.hasDeferredIndexRecovery()).toBe(false); + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + await conn.recoverIndexesOffThread(); // nothing left: resolves at once + } finally { + conn.close(); + } + expect(seen).toEqual([]); + }, 30_000); + + it('defers nothing on a healthy index', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-recover-')); + dirs.push(root); + CodeGraph.initSync(root).close(); + const conn = DatabaseConnection.open(getDatabasePath(root), { deferIndexRecovery: true }); + try { + expect(conn.hasDeferredIndexRecovery()).toBe(false); + } finally { + conn.close(); + } + }); + + it('opens with CodeGraph.open() only once the indexes are back, rebuilt off this thread', async () => { + const { root, dbPath } = interruptedProject(SMALL); + const seen = spyOnIndexDdl(); + const cg = await CodeGraph.open(root); + try { + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + expect(cg.hasPendingIndexRecovery()).toBe(false); + } finally { + cg.close(); + } + expect(seen).toEqual([]); + }, 30_000); + + it('still rebuilds in line for a synchronous open that asks for no deferral', () => { + const { root, dbPath } = interruptedProject(SMALL); + const cg = CodeGraph.openSync(root); + cg.close(); + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + }, 30_000); + + it('serves a projectPath call once the indexes are back, without rebuilding on the server thread', async () => { + const { root, dbPath } = interruptedProject(SMALL); + const engine = new MCPEngine({ watch: false }); + engines.push(engine); + const seen = spyOnIndexDdl(); + const result = await engine.getToolHandler().execute('codegraph_search', { query: 'fn_1', projectPath: root }); + expect(result.isError).toBeFalsy(); + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + expect(seen).toEqual([]); + }, 30_000); + + it('rebuilds before a watched projectPath project is caught up and served', async () => { + const { root, dbPath } = interruptedProject(SMALL); + const engine = new MCPEngine({ watch: true }); + engines.push(engine); + const seen = spyOnIndexDdl(); + // The catch-up sync writes through the main connection; started during the + // rebuild it would wait out the rebuild's write lock with this thread blocked. + const events: string[] = []; + const recover = CodeGraph.prototype.recoverIndexes; + vi.spyOn(CodeGraph.prototype, 'recoverIndexes').mockImplementation(function (this: InstanceType) { + const pending = this.hasPendingIndexRecovery(); + return recover.call(this).then(() => { if (pending) events.push('rebuilt'); }); + }); + const sync = CodeGraph.prototype.sync; + vi.spyOn(CodeGraph.prototype, 'sync').mockImplementation(function (this: InstanceType, ...args) { + events.push('sync'); + return sync.apply(this, args); + }); + const result = await engine.getToolHandler().execute('codegraph_search', { query: 'fn_1', projectPath: root }); + expect(result.isError).toBeFalsy(); + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + expect(seen).toEqual([]); + expect(events[0]).toBe('rebuilt'); + expect(events).toContain('sync'); + }, 30_000); + + it('opens the default project at once on the retry path and serves it rebuilt', async () => { + const { root, dbPath } = interruptedProject(SMALL); + const engine = new MCPEngine({ watch: false }); + engines.push(engine); + const seen = spyOnIndexDdl(); + engine.retryInitializeSync(root); + expect(engine.hasDefaultCodeGraph()).toBe(true); + const result = await engine.getToolHandler().execute('codegraph_search', { query: 'fn_1' }); + expect(result.isError).toBeFalsy(); + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + expect(seen).toEqual([]); + }, 30_000); + + it('opens the default project with the indexes back on the startup path', async () => { + const { root, dbPath } = interruptedProject(SMALL); + const engine = new MCPEngine({ watch: false }); + engines.push(engine); + const seen = spyOnIndexDdl(); + await engine.ensureInitialized(root); + expect(engine.hasDefaultCodeGraph()).toBe(true); + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + expect(seen).toEqual([]); + }, 30_000); + + /** + * End to end with the real liveness watchdog, armed for heartbeats only and + * with a short timeout: a rebuild on the opener's thread is one silent block + * of about a second on this index and gets the process killed; off the + * thread, the heartbeat never stops. + */ + it('keeps a watched process alive while CodeGraph.open() rebuilds a large index', async () => { + const { root, dbPath } = interruptedProject({ nodes: 10000, refs: 300000, edges: 150000 }); + const watchdog = path.join(DIST, 'mcp', 'liveness-watchdog.js'); + const child = spawn(process.execPath, ['-e', ` + (async () => { + const { default: CodeGraph, initGrammars } = require(${JSON.stringify(path.join(DIST, 'index.js'))}); + await initGrammars(); // grammar loading is not what this measures + require(${JSON.stringify(watchdog)}).installMainThreadWatchdog(); + const cg = await CodeGraph.open(${JSON.stringify(root)}); + cg.close(); + process.exit(0); + })().catch((err) => { console.error(err); process.exit(23); }); + `], { + env: { ...process.env, CODEGRAPH_NO_WATCHDOG: '0', CODEGRAPH_WATCHDOG_TIMEOUT_MS: '500' }, + stdio: ['ignore', 'ignore', 'pipe'], + }); + let stderr = ''; + child.stderr.on('data', (d) => { stderr += d; }); + const code = await new Promise((resolve) => child.on('close', (c) => resolve(c))); + expect(stderr).not.toContain('Main thread unresponsive'); + expect(code, stderr).toBe(0); + expect(presentWindowIndexes(dbPath)).toEqual(WINDOW_INDEXES); + }, 60_000); +}); diff --git a/src/db/index.ts b/src/db/index.ts index cfe7c10a72..662e5301b2 100644 --- a/src/db/index.ts +++ b/src/db/index.ts @@ -160,9 +160,16 @@ export class DatabaseConnection { } /** - * Open an existing database - */ - static open(dbPath: string, options: { readOnly?: boolean } = {}): DatabaseConnection { + * Open an existing database. + * + * `deferIndexRecovery` leaves the secondary indexes an interrupted bulk load + * dropped for {@link recoverIndexesOffThread} instead of rebuilding them + * here. Rebuilding one scans and sorts its whole table: on a large index + * that is minutes of synchronous work, and on a watched event loop the + * liveness watchdog kills the process before the index commits, every time + * it starts (#1887). Reads stay correct meanwhile, only slower. + */ + static open(dbPath: string, options: { readOnly?: boolean; deferIndexRecovery?: boolean } = {}): DatabaseConnection { if (!fs.existsSync(dbPath)) { throw new Error(`Database not found: ${dbPath}`); } @@ -194,7 +201,8 @@ export class DatabaseConnection { // beginBulkNodeLoad and endBulkNodeLoad): the FTS triggers are missing and // nodes_fts is stale. Rebuild + recreate so search stays in sync. conn.healBulkNodeLoad(); - conn.healBulkSecondaryIndexes(); + if (options.deferIndexRecovery) conn.deferredIndexDdls = conn.missingBulkSecondaryIndexDdls(); + else conn.healBulkSecondaryIndexes(); // Self-heal a killed session's leftover oversized WAL (#1431) — one // statSync when healthy, off-thread checkpoint+truncate when not. @@ -436,6 +444,14 @@ export class DatabaseConnection { /** Recreate every secondary index a killed bulk parse/ref/edge window may leave dropped. */ private healBulkSecondaryIndexes(): void { + for (const ddl of this.missingBulkSecondaryIndexDdls()) this.db.exec(ddl); + } + + /** + * The DDL that recreates every secondary index a killed bulk parse/ref/edge + * window may leave dropped; empty, after one count, when none is missing. + */ + private missingBulkSecondaryIndexDdls(): string[] { const names = [...new Set([ ...DatabaseConnection.BULK_PARSE_INDEX_NAMES, ...DatabaseConnection.BULK_REF_INDEX_NAMES, @@ -445,15 +461,99 @@ export class DatabaseConnection { const row = this.db .prepare(`SELECT count(*) AS c FROM sqlite_master WHERE type = 'index' AND name IN (${placeholders})`) .get(...names) as { c: number } | undefined; - if ((row?.c ?? 0) >= names.length) return; + if ((row?.c ?? 0) >= names.length) return []; const schemaPath = path.join(__dirname, 'schema.sql'); const schema = fs.readFileSync(schemaPath, 'utf-8'); - for (const idx of names) { + return names.map((idx) => { const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`)); if (!m) throw new Error(`schema.sql: index ${idx} not found for crash recovery`); - this.db.exec(m[0]); + return m[0]; + }); + } + + /** Index DDL {@link open} left for {@link recoverIndexesOffThread}. */ + private deferredIndexDdls: string[] = []; + private indexRecovery: Promise | null = null; + + /** Whether {@link open} left dropped secondary indexes for {@link recoverIndexesOffThread}. */ + hasDeferredIndexRecovery(): boolean { + return this.deferredIndexDdls.length > 0; + } + + /** + * Recreate the secondary indexes {@link open} deferred, on a worker thread + * with its own connection, so this thread's event loop, and the #850 + * liveness watchdog's heartbeat, keep turning however long the rebuild takes + * (#1887). Each index commits on its own, so an interrupted recovery keeps + * what it finished. Not best-effort like {@link runMaintenance}: rejects when + * a statement fails, and the next call tries again. Single-flight; resolves + * at once when nothing was deferred. + */ + recoverIndexesOffThread(): Promise { + if (this.deferredIndexDdls.length === 0) return Promise.resolve(); + this.indexRecovery ??= this.runRecoveryOffThread(this.deferredIndexDdls) + .then(() => { this.deferredIndexDdls = []; }) + .finally(() => { this.indexRecovery = null; }); + return this.indexRecovery; + } + + private async runRecoveryOffThread(statements: string[]): Promise { + let WorkerCtor: typeof import('node:worker_threads').Worker; + try { + WorkerCtor = (await import('node:worker_threads')).Worker; + } catch { + // No worker threads in this runtime: recover here, as open() would have. + for (const ddl of statements) this.db.exec(ddl); + return; } + // SQLite's default temp storage, not this connection's MEMORY: the sort + // behind a big index spills to disk instead of growing the heap. The + // worker collects its garbage before it ends, so it exits with no marking + // in flight (worker-teardown.ts), and nothing terminates it. + const workerSource = ` + const { workerData, parentPort } = require('node:worker_threads'); + let error = null; + try { + const { DatabaseSync } = require('node:sqlite'); + const db = new DatabaseSync(workerData.dbPath); + try { + db.exec('PRAGMA busy_timeout = 30000'); + for (const sql of workerData.statements) db.exec(sql); + } finally { + db.close(); + } + } catch (e) { + error = String((e && e.message) || e); + } + try { + const v8 = require('v8'); + v8.setFlagsFromString('--expose-gc'); + const gc = require('vm').runInNewContext('gc'); + v8.setFlagsFromString('--no-expose-gc'); + gc(); + } catch {} + parentPort.postMessage({ error }); + `; + await new Promise((resolve, reject) => { + let settled = false; + const worker = new WorkerCtor(workerSource, { eval: true, workerData: { dbPath: this.dbPath, statements } }); + worker.once('message', (message: { error: string | null }) => { + settled = true; + if (message.error) reject(new Error(`Could not restore the indexes of ${this.dbPath}: ${message.error}`)); + else resolve(); + }); + worker.once('error', (err) => { + if (settled) return; + settled = true; + reject(err); + }); + worker.once('exit', (code) => { + if (settled) return; + settled = true; + reject(new Error(`Restoring the indexes of ${this.dbPath} stopped early (worker exit ${code}).`)); + }); + }); } /** diff --git a/src/index.ts b/src/index.ts index 55dc2c434c..b702890af6 100644 --- a/src/index.ts +++ b/src/index.ts @@ -393,9 +393,19 @@ export class CodeGraph { throw new Error(`Invalid CodeGraph directory: ${validation.errors.join(', ')}`); } - // Open database + // Open database. The secondary indexes an interrupted bulk load dropped are + // rebuilt off this thread, and finished before the instance is returned + // (#1887): rebuilt in line on a large index, they blocked the thread longer + // than the liveness watchdog allows, and every restart was killed the same + // way. const dbPath = getDatabasePath(resolvedRoot); - const db = DatabaseConnection.open(dbPath, { readOnly: options.readOnly }); + const db = DatabaseConnection.open(dbPath, { readOnly: options.readOnly, deferIndexRecovery: true }); + try { + await db.recoverIndexesOffThread(); + } catch (err) { + db.close(); + throw err; + } const queries = new QueryBuilder(db.getDb()); const instance = new CodeGraph(db, queries, resolvedRoot); @@ -462,9 +472,14 @@ export class CodeGraph { } /** - * Open synchronously (without sync) + * Open synchronously (without sync). + * + * `deferIndexRecovery` leaves the secondary indexes an interrupted bulk load + * dropped for {@link recoverIndexes} to rebuild off this thread, instead of + * rebuilding them here (#1887): for a caller on a watched event loop, which + * then awaits {@link recoverIndexes} before it relies on the project. */ - static openSync(projectRoot: string, options: Pick = {}): CodeGraph { + static openSync(projectRoot: string, options: Pick & { deferIndexRecovery?: boolean } = {}): CodeGraph { const resolvedRoot = path.resolve(projectRoot); // Check if initialized @@ -480,12 +495,28 @@ export class CodeGraph { // Open database const dbPath = getDatabasePath(resolvedRoot); - const db = DatabaseConnection.open(dbPath, { readOnly: options.readOnly }); + const db = DatabaseConnection.open(dbPath, { readOnly: options.readOnly, deferIndexRecovery: options.deferIndexRecovery }); const queries = new QueryBuilder(db.getDb()); return new CodeGraph(db, queries, resolvedRoot); } + /** + * Rebuild, on a worker thread, the secondary indexes an interrupted bulk + * load dropped and `openSync({ deferIndexRecovery: true })` left (#1887). + * Resolves at once when nothing is left, which is every open but the first + * after such an interruption; rejects when the rebuild fails, and the next + * call tries again. Reads stay correct meanwhile, only slower. + */ + recoverIndexes(): Promise { + return this.db.recoverIndexesOffThread(); + } + + /** Whether {@link recoverIndexes} still has indexes to rebuild. */ + hasPendingIndexRecovery(): boolean { + return this.db.hasDeferredIndexRecovery(); + } + /** * Check if a directory has been initialized as a CodeGraph project */ diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index cc0b43ba1d..1f8470e9aa 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -246,12 +246,23 @@ export class MCPEngine { this.cg = null; } assertNoRebuild(resolvedRoot); - this.cg = loadCodeGraph().openSync(resolvedRoot, { readOnly: this.opts.readOnly }); + // Indexes an interrupted bulk load dropped are rebuilt off this thread + // (#1887); the catch-up and tool calls wait for them. + this.cg = loadCodeGraph().openSync(resolvedRoot, { readOnly: this.opts.readOnly, deferIndexRecovery: true }); this.projectPath = resolvedRoot; this.toolHandler.setDefaultCodeGraph(this.cg); this.startWatching(); this.catchUpSync(); - this.maybeStartPool(resolvedRoot); + // The pool's workers open the project themselves: start them on a + // rebuilt one, not racing the rebuild for its write lock. + const opened = this.cg; + if (opened?.hasPendingIndexRecovery()) { + void opened.recoverIndexes().catch(() => { /* reported by the tool call waiting on it */ }).then(() => { + if (this.cg === opened) this.maybeStartPool(resolvedRoot); + }); + } else { + this.maybeStartPool(resolvedRoot); + } } catch (err) { // Still failing — caller will try again on the next tool call. this.toolHandler.setDefaultOpenFailure(err instanceof WslSharedIndexError ? err : null); diff --git a/src/mcp/project-lifecycle.ts b/src/mcp/project-lifecycle.ts index 88ffc50ae9..8158b5a209 100644 --- a/src/mcp/project-lifecycle.ts +++ b/src/mcp/project-lifecycle.ts @@ -118,6 +118,10 @@ function ready(root: string, project: Project): Promise { async function activate(root: string, project: Project): Promise { if (!isInitialized(root)) return; + // Rebuild what an interrupted bulk load dropped (off this thread, #1887) + // before writing: a sync's writes would wait out the rebuild's write lock + // with this thread blocked. + if (project.cg.hasPendingIndexRecovery?.()) await project.cg.recoverIndexes(); const writer = tryAcquireWriterLock(root, 'fallback'); if (writer.kind === 'taken') { // Keep a real daemon session, so its idle timeout cannot strand this reader. diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 42abfb0e98..2be63a7c05 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -2425,7 +2425,9 @@ export class ToolHandler { } } - const open = () => loadCodeGraph().openSync(canonicalRoot); + // Indexes an interrupted bulk load dropped are rebuilt off this thread, + // before the project is served (awaitIndexRecovery, #1887). + const open = () => loadCodeGraph().openSync(canonicalRoot, { deferIndexRecovery: true }); const cg = this.projectLifecycle?.open(canonicalRoot, open) ?? open(); this.projectCache.set(canonicalRoot, cg); this.projectUsedAt.set(canonicalRoot, Date.now()); @@ -2461,6 +2463,25 @@ export class ToolHandler { } } + /** + * Wait for a project to finish rebuilding, off this thread, the secondary + * indexes an interrupted bulk load dropped (#1887). Not time-boxed like the + * catch-up gate: without them a read scans whole tables, which on a large + * index stalls the thread the rebuild was moved off. Never throws: a failed + * rebuild leaves reads correct, and the next call tries again. + */ + private async awaitIndexRecovery(cg: CodeGraph): Promise { + if (!cg.hasPendingIndexRecovery?.()) return; + try { + await cg.recoverIndexes(); + } catch (err) { + process.stderr.write( + `[CodeGraph MCP] Could not restore the indexes an interrupted indexing run left out of ${cg.getProjectRoot()}: ` + + `${err instanceof Error ? err.message : String(err)}. Queries still work, more slowly; the next call tries again.\n` + ); + } + } + private async awaitProjectGate(projectPath: string): Promise { const cg = this.getCodeGraph(projectPath); if (!this.projectLifecycle || cg === this.cg) return; @@ -2906,6 +2927,7 @@ export class ToolHandler { } const project = await this.getCodeGraph(args.projectPath as string | undefined); + await this.awaitIndexRecovery(project); // Recover a watcher disabled by prolonged lock contention on the next call. // The stale banner remains until the watcher finishes its full scan; // frequent calls cannot bypass its cooldown (#1959). From 1040d09f26b7a258a213f24047815f2703b53f18 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 10:23:24 +0000 Subject: [PATCH 30/95] fix(mcp): a daemon that never answers no longer stalls a new session for minutes (#2795) Root cause: a launcher that found no answering daemon spawned one and then polled for it 240 times, 25ms apart. A probe that connects but gets no hello waits out the 3s hello timeout, so a process that holds the daemon's pipe and never says hello made the poll take about 240 x 3s, plus the first probe: about 12 minutes, with the session's tool calls held meanwhile. In practice that holder is a daemon whose main thread is blocked: the OS still accepts connections on its pipe, and the daemon the launcher spawns cannot take its lock, so it exits. Fix: the poll has a wall-clock budget instead of a probe count (the 6s it was meant to be), and each probe's hello wait is capped by what is left of it (pollForDaemon, connectAnyDaemonCandidate, a hello timeout on connectWithHello). Spent, the session is served in-process exactly as when a spawned daemon never binds: read-only while the stuck daemon still holds the writer slot. The proxy's existing retry with backoff then reaches the daemon again. The liveness watchdog kills a wedged daemon after its 60s window (longer while its disk advances), and the next retry spawns the replacement and proxies to it. The bound sits well inside that window, so a session neither waits on the wedge nor gives up on the daemon. Validation: - A real daemon wedged by a preload's busy loop on Windows, then a second session's codegraph_status: - no watchdog on the wedged daemon: main answered after 733.2s; the fix after 9.9s, and later calls in 0.5s while the daemon stayed wedged; - the default 60s watchdog (it killed the wedged daemon at about 121s, one deferral for its own off-thread writes): main answered after 125.2s; the fix after 9.9s in-process, then proxied to the replacement daemon at 121.1s. - New __tests__/daemon-silent-holder.test.ts (8 cases): main's source 0 pass, fix 8 pass. Fake-clock tests of the budget and the per-probe cap; a real silent socket honouring the hello timeout; the real launcher against a project whose pipe, lock and writer slot are held by a holder that never says hello: first call served in-process in about 10s (main: none in 45s); and after the holder dies, the session proxies to the daemon that replaces it. 3 mutations each failed at least one case. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/daemon-silent-holder.test.ts | 292 +++++++++++++++++++++++++ src/mcp/index.ts | 99 ++++++--- src/mcp/proxy.ts | 21 +- 4 files changed, 377 insertions(+), 36 deletions(-) create mode 100644 __tests__/daemon-silent-holder.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0672e8da56..ffda40983c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404) - An index left incomplete by an indexing run that was stopped partway, by a crash, an out-of-memory kill or the server's own hang check, no longer gets the background server killed with "Main thread unresponsive" every time it starts: on a large project, restoring the database indexes the run had set aside took longer than the hang check allows, so each restart died at the same point and the index was never repaired. They are now restored in the background, and the project is served once they are back. Thanks @cbeaulieu-gt for the report and the first fix. (#1887) +- A new agent session no longer waits up to about 12 minutes for its first answer when the background server is stuck: a server whose process still holds its connection open but never responds made the session wait out a long series of retries. The session now gives up within seconds and answers on its own, and switches back to the background server once that has been restarted. - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) - `codegraph install` and `upgrade` no longer break Codex's `config.toml` when CodeGraph's `[mcp_servers.codegraph]` entry is written differently from how CodeGraph writes it, such as indented, with spaces or quotes inside the brackets, or with Windows line endings: the entry is now updated in place, where before a second copy was added and Codex could load none of the file. A file already broken that way is repaired on the next install or upgrade, an example of the entry inside a multiline setting such as `developer_instructions` is no longer mistaken for it, uninstall removes the entry however it is written, and an entry you wrote with dotted keys or as an inline table is left as you wrote it. When your `mcp_servers` settings are one inline table, CodeGraph now says it can't add itself instead of breaking the file. Thanks @dencoseca for the report, and @danusha2345 and @rudycelekli for the first fixes. (#2562) diff --git a/__tests__/daemon-silent-holder.test.ts b/__tests__/daemon-silent-holder.test.ts new file mode 100644 index 0000000000..4a853a1eee --- /dev/null +++ b/__tests__/daemon-silent-holder.test.ts @@ -0,0 +1,292 @@ +/** + * A process that holds the daemon's pipe but never says hello no longer stalls + * a session's first tool call for minutes. + * + * In practice that holder is a shared daemon whose main thread is blocked: the + * OS still accepts connections on its pipe, but the hello never comes. After + * the first probe the launcher spawned a daemon (which can't take the lock + * from a live holder) and polled for one 240 times; each probe waited out the + * whole 3s hello timeout, so the first call waited about 12 minutes. The poll + * now has a wall-clock budget, after which the session is served in-process + * as when a spawned daemon never binds, and the proxy's retries (#2277) pick + * the daemon up again once its watchdog has replaced it. + */ +import { afterEach, describe, expect, it } from 'vitest'; +import { ChildProcessWithoutNullStreams, spawn, spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as net from 'net'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { connectAnyDaemonCandidate, pollForDaemon } from '../src/mcp/index'; +import { connectWithHello, DAEMON_HELLO_TIMEOUT_MS } from '../src/mcp/proxy'; +import { getDaemonPidPath, getDaemonSocketCandidates } from '../src/mcp/daemon-paths'; +import { getWriterPidPath } from '../src/mcp/writer-lock'; +import { CodeGraphPackageVersion } from '../src/mcp/version'; +import { WASM_RUNTIME_FLAGS } from '../src/extraction/wasm-runtime-flags'; +import { recordSpawns, removeSpawnLog, settleLosingCandidates } from './daemon-candidates'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +/** A clock the poll runs on, moved only by its sleeps and by the probes. */ +function fakeClock(): { now(): number; sleep(ms: number): Promise; advance(ms: number): void } { + let t = 1_000; + return { now: () => t, sleep: async (ms) => { t += ms; }, advance: (ms) => { t += ms; } }; +} + +describe('the poll for a spawned daemon', () => { + it('gives up within its budget when the pipe holder never says hello', async () => { + const clock = fakeClock(); + const started = clock.now(); + const waits: number[] = []; + // What the launcher's probe does: wait for a hello, at most to the deadline. + const silent = async (deadline: number): Promise => { + const wait = Math.min(DAEMON_HELLO_TIMEOUT_MS, deadline - clock.now()); + waits.push(wait); + clock.advance(wait); + return null; + }; + expect(await pollForDaemon(silent, 6_000, 25, clock)).toBeNull(); + // 240 probes of 3s each was the old wait: about 12 minutes. + expect(clock.now() - started).toBeLessThanOrEqual(6_000 + 25); + expect(waits.every((w) => w > 0 && w <= DAEMON_HELLO_TIMEOUT_MS)).toBe(true); + expect(waits.length).toBeLessThanOrEqual(3); + }); + + it('attaches the moment a daemon answers', async () => { + const clock = fakeClock(); + const started = clock.now(); + const socket = { daemon: true }; + const binds = async (): Promise => (clock.now() - started >= 400 ? socket : null); + expect(await pollForDaemon(binds, 6_000, 25, clock)).toBe(socket); + expect(clock.now() - started).toBeLessThan(500); + }); + + it('stops at once for a daemon of another version', async () => { + const clock = fakeClock(); + const started = clock.now(); + for (const answer of ['older-version', 'version-mismatch'] as const) { + expect(await pollForDaemon(async () => answer, 6_000, 25, clock)).toBeNull(); + } + expect(clock.now() - started).toBe(50); + }); +}); + +describe('the probe of the socket candidates', () => { + it('caps each hello wait at what is left before the deadline', async () => { + const clock = fakeClock(); + const calls: Array<[string, number]> = []; + const silent = async (where: string, _version?: string, timeout = DAEMON_HELLO_TIMEOUT_MS): Promise => { + calls.push([where, timeout]); + clock.advance(timeout); + return null; + }; + const deadline = clock.now() + DAEMON_HELLO_TIMEOUT_MS + 1_000; + expect(await connectAnyDaemonCandidate(['first', 'second', 'third'], deadline, silent, clock.now)).toBeNull(); + expect(calls).toEqual([['first', DAEMON_HELLO_TIMEOUT_MS], ['second', 1_000]]); + }); + + it('gives each candidate the full hello timeout without a deadline, and returns the first answer', async () => { + const clock = fakeClock(); + const calls: Array<[string, number]> = []; + const socket = new net.Socket(); + const second = async (where: string, _version?: string, timeout = DAEMON_HELLO_TIMEOUT_MS): Promise => { + calls.push([where, timeout]); + return where === 'second' ? socket : null; + }; + expect(await connectAnyDaemonCandidate(['first', 'second', 'third'], undefined, second, clock.now)).toBe(socket); + expect(calls).toEqual([['first', DAEMON_HELLO_TIMEOUT_MS], ['second', DAEMON_HELLO_TIMEOUT_MS]]); + }); +}); + +describe('a probe of a pipe holder that never says hello', () => { + const servers: net.Server[] = []; + const held: net.Socket[] = []; + const dirs: string[] = []; + afterEach(async () => { + for (const s of held.splice(0)) s.destroy(); + for (const server of servers.splice(0)) await new Promise((resolve) => server.close(() => resolve())); + for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('gives up after the hello timeout it is given', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-silent-')); + dirs.push(dir); + const where = process.platform === 'win32' + ? `\\\\.\\pipe\\cg-silent-${process.pid}-${Date.now()}` + : path.join(dir, 's.sock'); + const server = net.createServer((socket) => { held.push(socket); }); + servers.push(server); + await new Promise((resolve) => server.listen(where, resolve)); + const started = Date.now(); + expect(await connectWithHello(where, CodeGraphPackageVersion, 300)).toBeNull(); + const took = Date.now() - started; + expect(took).toBeGreaterThanOrEqual(250); + expect(took).toBeLessThan(DAEMON_HELLO_TIMEOUT_MS); + }); +}); + +/** + * End to end: the real launcher against a project whose daemon pipe, lock and + * writer slot are held by a "daemon" that never says hello (this test + * process), as a daemon whose main thread is blocked keeps them. + */ +describe('a session whose daemon never says hello', () => { + let root: string; + let launcher: ChildProcessWithoutNullStreams | null = null; + let holder: net.Server | null = null; + const held: net.Socket[] = []; + + const lockPid = (): number | null => { + try { return (JSON.parse(fs.readFileSync(getDaemonPidPath(root), 'utf8')) as { pid: number }).pid; } catch { return null; } + }; + const alive = (pid: number): boolean => { + try { process.kill(pid, 0); return true; } catch (err) { return (err as NodeJS.ErrnoException).code === 'EPERM'; } + }; + + async function project(): Promise { + root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'cg-silent-holder-'))); + fs.writeFileSync(path.join(root, 'a.ts'), 'export function alpha(): number { return 1; }\n'); + (await CodeGraph.init(root, { index: true })).close(); + } + + async function holdPipe(): Promise { + const pipe = getDaemonSocketCandidates(root)[0]!; + holder = net.createServer((socket) => { held.push(socket); }); + await new Promise((resolve) => holder!.listen(pipe, resolve)); + fs.writeFileSync(getDaemonPidPath(root), JSON.stringify({ + pid: process.pid, version: CodeGraphPackageVersion, socketPath: pipe, startedAt: Date.now() - 60_000, + })); + fs.writeFileSync(getWriterPidPath(root), JSON.stringify({ + pid: process.pid, mode: 'daemon', startedAt: Date.now() - 60_000, ready: true, + }) + '\n'); + } + + /** What the liveness watchdog's kill leaves: no pipe, and records naming a dead pid. */ + async function killHolder(): Promise { + for (const s of held.splice(0)) s.destroy(); + if (holder) await new Promise((resolve) => holder!.close(() => resolve())); + holder = null; + const dead = spawnSync(process.execPath, ['-e', '']).pid!; + const pipe = getDaemonSocketCandidates(root)[0]!; + fs.writeFileSync(getDaemonPidPath(root), JSON.stringify({ + pid: dead, version: CodeGraphPackageVersion, socketPath: pipe, startedAt: Date.now() - 120_000, + })); + fs.writeFileSync(getWriterPidPath(root), JSON.stringify({ + pid: dead, mode: 'daemon', startedAt: Date.now() - 120_000, ready: true, + }) + '\n'); + } + + function launch(env: NodeJS.ProcessEnv): { send(m: unknown): void; out(): string; err(): string; replyTo(id: number): Promise } { + const recorder = recordSpawns(root); + const child = spawn(process.execPath, [...WASM_RUNTIME_FLAGS, ...recorder.args, BIN, 'serve', '--mcp', '--path', root], { + cwd: os.tmpdir(), + env: { + ...process.env, ...recorder.env, ...env, + CODEGRAPH_TELEMETRY: '0', DO_NOT_TRACK: '1', CODEGRAPH_NO_UPDATE_CHECK: '1', + }, + stdio: ['pipe', 'pipe', 'pipe'], + }) as ChildProcessWithoutNullStreams; + launcher = child; + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (d) => { stdout += d; }); + child.stderr.on('data', (d) => { stderr += d; }); + child.stdin.on('error', () => { /* gone */ }); + const answered = (id: number): boolean => + stdout.split('\n').some((l) => { try { return JSON.parse(l).id === id; } catch { return false; } }); + return { + send: (m) => { child.stdin.write(JSON.stringify(m) + '\n'); }, + out: () => stdout, + err: () => stderr, + replyTo: (id) => { + const asked = Date.now(); + return new Promise((resolve) => { + const iv = setInterval(() => { + if (answered(id)) { clearInterval(iv); resolve(Date.now() - asked); } + else if (Date.now() - asked > 45_000) { clearInterval(iv); resolve(null); } + }, 50); + }); + }, + }; + } + + const initialize = { jsonrpc: '2.0', id: 1, method: 'initialize', params: { protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 't', version: '0' } } }; + const status = (id: number) => ({ jsonrpc: '2.0', id, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const reply = (out: string, id: number) => out.split('\n').map((l) => { try { return JSON.parse(l); } catch { return null; } }).find((m) => m?.id === id); + + afterEach(async () => { + if (launcher && launcher.exitCode === null && launcher.signalCode === null) { + const exited = new Promise((resolve) => launcher!.once('exit', () => resolve())); + launcher.stdin.end(); + await Promise.race([exited, new Promise((resolve) => setTimeout(resolve, 30_000))]); + if (launcher.exitCode === null && launcher.signalCode === null) launcher.kill('SIGKILL'); + } + launcher = null; + // Spawned daemons that lost the lock leave on their own, and must be gone + // before the pipe, lock and fixture go away. One that won idles out. + const winner = (): number => (holder ? process.pid : lockPid() ?? process.pid); + await settleLosingCandidates(root, winner, 60_000); + const daemon = winner(); + if (daemon !== process.pid) { + const until = Date.now() + 30_000; + while (alive(daemon) && Date.now() < until) await new Promise((resolve) => setTimeout(resolve, 100)); + if (alive(daemon)) process.kill(daemon, 'SIGKILL'); + } + removeSpawnLog(root); + for (const s of held.splice(0)) s.destroy(); + if (holder) await new Promise((resolve) => holder!.close(() => resolve())); + holder = null; + const deadline = Date.now() + 10_000; + for (;;) { + try { + fs.rmSync(root, { recursive: true, force: true }); + break; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code ?? ''; + if (!['EBUSY', 'ENOTEMPTY', 'EPERM'].includes(code) || Date.now() > deadline) throw error; + await new Promise((resolve) => setTimeout(resolve, 100)); + } + } + }, 150_000); + + it('serves the first tool call in-process within the connect budget', async () => { + await project(); + await holdPipe(); + // One connect attempt: this is about how long the first one may take. + const session = launch({ CODEGRAPH_DAEMON_RETRY_MS: '0' }); + session.send(initialize); + session.send(status(2)); + const took = await session.replyTo(2); + // The first probe's hello timeout (3s) plus the 6s budget, with room for a + // loaded machine; the old poll answered after about 12 minutes. + expect(took, session.err()).not.toBeNull(); + expect(took!).toBeLessThan(30_000); + expect(session.err()).toContain('Shared daemon unavailable; serving this session in-process'); + expect(reply(session.out(), 2).result.content[0].text).toContain('CodeGraph Status'); + }, 90_000); + + it('hands the session to the daemon that replaces the silent one', async () => { + await project(); + await holdPipe(); + const session = launch({ CODEGRAPH_DAEMON_RETRY_MS: '1000', CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '1000' }); + session.send(initialize); + session.send(status(2)); + expect(await session.replyTo(2), session.err()).not.toBeNull(); + expect(session.err()).toContain('serving this session in-process'); + // The silent daemon's liveness watchdog kills it; the session's next retry + // spawns a replacement and proxies to it. + await killHolder(); + const until = Date.now() + 60_000; + while (!session.err().includes('Shared daemon reachable; proxying this session to it') && Date.now() < until) { + await new Promise((resolve) => setTimeout(resolve, 100)); + } + expect(session.err()).toContain('Shared daemon reachable; proxying this session to it'); + session.send(status(3)); + expect(await session.replyTo(3), session.err()).not.toBeNull(); + const daemon = lockPid(); + expect(daemon).not.toBeNull(); + expect(daemon).not.toBe(process.pid); + expect(alive(daemon!)).toBe(true); + }, 120_000); +}); diff --git a/src/mcp/index.ts b/src/mcp/index.ts index e41bd43e05..6535649f71 100644 --- a/src/mcp/index.ts +++ b/src/mcp/index.ts @@ -49,7 +49,7 @@ import { tryAcquireDaemonLock, } from './daemon'; import { clearStaleDaemonArtifacts, stopOlderDaemon } from './daemon-registry'; -import { connectWithHello, runLocalHandshakeProxy } from './proxy'; +import { connectWithHello, DAEMON_HELLO_TIMEOUT_MS, runLocalHandshakeProxy } from './proxy'; import { CodeGraphPackageJsonPath, CodeGraphPackageVersion } from './version'; import { readWriterLock, @@ -163,21 +163,75 @@ function makeFallbackEngine(root: string): MCPEngine { } /** - * How long a launcher waits for a freshly-spawned daemon to bind its socket - * before giving up and running in-process. The daemon binds the socket *before* - * the (backgrounded) engine/grammar warm-up, so this only needs to cover node - * process startup. 60 × 100ms = 6s of headroom for a cold/slow box; on the - * common path the socket appears within a few rounds. + * How long a launcher waits for a freshly-spawned daemon to answer before + * giving up and running in-process. The daemon binds the socket *before* the + * (backgrounded) engine/grammar warm-up, so this only needs to cover node + * process startup: 6s of headroom for a cold/slow box; on the common path the + * socket appears within a few rounds. + * + * A budget of wall time, not a count of probes. A process that holds the pipe + * but never says hello — in practice a daemon whose main thread is blocked — + * makes every probe wait out its whole hello timeout, so the old 240 probes + * held a session's first tool call for about 12 minutes. Spent, the session is + * served in-process exactly as when the daemon never binds, and the proxy keeps + * retrying with backoff (#2277): a daemon its liveness watchdog kills (60s by + * default) is replaced by the next retry, never bypassed for good. */ +const DAEMON_CONNECT_BUDGET_MS = 6_000; // Poll finely (25ms) so the proxy attaches the instant the freshly-spawned // daemon binds, instead of waiting up to a coarse 100ms after — shaves the -// cold-start handshake (the window the headless agent races). Same ~6s total -// give-up budget (240 × 25ms), just finer granularity; socket-connect probes -// are cheap. Paired with deferring the CodeGraph load (engine.ts) off the bind -// path, this narrows the "No such tool available" race window. -const DAEMON_CONNECT_MAX_RETRIES = 240; +// cold-start handshake (the window the headless agent races); socket-connect +// probes are cheap. Paired with deferring the CodeGraph load (engine.ts) off the +// bind path, this narrows the "No such tool available" race window. const DAEMON_CONNECT_RETRY_DELAY_MS = 25; +/** + * Probe the daemon's socket candidates in order (the in-project path first); + * the first that answers wins. With a `deadline`, no hello wait runs past it. + * The connect and the clock are injectable for tests. + */ +export async function connectAnyDaemonCandidate( + candidates: readonly string[], + deadline = Infinity, + connect: typeof connectWithHello = connectWithHello, + now: () => number = Date.now, +): Promise>> { + for (const candidate of candidates) { + const left = deadline - now(); + if (left <= 0) return null; + const s = await connect(candidate, CodeGraphPackageVersion, Math.min(DAEMON_HELLO_TIMEOUT_MS, left)); + // A wrong-version daemon IS up — definitive; propagate so the caller + // replaces it (an older release) or serves in-process instead of + // spawning + polling for 6s. Don't keep probing fallbacks past it. + if (s) return s; + } + return null; +} + +/** + * Probe for a daemon every `retryDelayMs` until one answers or `budgetMs` of + * wall time is spent. `probe` gets the deadline, and caps its own hello wait to + * it. Resolves the socket, or null once the budget is spent or a daemon of + * another version answers. The clock is injectable for tests. + */ +export async function pollForDaemon( + probe: (deadline: number) => Promise, + budgetMs: number = DAEMON_CONNECT_BUDGET_MS, + retryDelayMs: number = DAEMON_CONNECT_RETRY_DELAY_MS, + clock: { now(): number; sleep(ms: number): Promise } = { now: Date.now, sleep }, +): Promise { + const deadline = clock.now() + budgetMs; + for (;;) { + await clock.sleep(retryDelayMs); + if (clock.now() >= deadline) return null; + const answer = await probe(deadline); + // Another version answering, older or not, waits for the next retry: one + // replacement per attempt, so this can never loop. + if (answer === 'older-version' || answer === 'version-mismatch') return null; + if (answer) return answer; + } +} + /** Whether `CODEGRAPH_NO_DAEMON` was set to a truthy value. */ function daemonOptOutSet(): boolean { const raw = process.env.CODEGRAPH_NO_DAEMON; @@ -684,16 +738,7 @@ export class MCPServer { // bound with zero coordination. The in-project candidate is tried first, so a // normal repo pays nothing extra (it connects on the very first probe). const candidates = getDaemonSocketCandidates(root); - const connectAnyCandidate = async (): Promise>> => { - for (const candidate of candidates) { - const s = await connectWithHello(candidate); - // A wrong-version daemon IS up — definitive; propagate so the caller - // replaces it (an older release) or serves in-process instead of - // spawning + polling for 6s. Don't keep probing fallbacks past it. - if (s) return s; - } - return null; - }; + const connectAnyCandidate = (deadline?: number) => connectAnyDaemonCandidate(candidates, deadline); const getDaemonSocket = async () => { // Fast path: a daemon may already be listening (on either candidate). const probe = await connectAnyCandidate(); @@ -710,15 +755,9 @@ export class MCPServer { // replacement this launcher holds the writer slot for it meanwhile. try { spawnDetachedDaemon(root, older === 'stopped'); - for (let attempt = 0; attempt < DAEMON_CONNECT_MAX_RETRIES; attempt++) { - await sleep(DAEMON_CONNECT_RETRY_DELAY_MS); - const s = await connectAnyCandidate(); - // Another version answering now, older or not, waits for the next - // retry: one replacement per attempt, so this can never loop. - if (s === 'older-version' || s === 'version-mismatch') return null; - if (s) return s; - } - return null; // never bound — the proxy serves this session in-process + // Null when nothing answered in time — never bound, or a holder that + // never says hello: the proxy serves this session in-process. + return await pollForDaemon(connectAnyCandidate); } finally { // The new daemon has taken the slot over, or never came; a no-op unless still ours. if (older === 'stopped') releaseWriterLock(root); diff --git a/src/mcp/proxy.ts b/src/mcp/proxy.ts index c0d9f6cafb..b9fa35d806 100644 --- a/src/mcp/proxy.ts +++ b/src/mcp/proxy.ts @@ -40,6 +40,13 @@ import { WORKER_START_SETTLE_MS } from '../worker-teardown'; /** Default poll cadence for the PPID watchdog (same as the direct server). */ const DEFAULT_PPID_POLL_MS = 5000; +/** + * How long a probe waits for a daemon's hello once connected. A healthy daemon + * sends it the moment it accepts; one whose main thread is blocked accepts at + * the OS level and never sends it. + */ +export const DAEMON_HELLO_TIMEOUT_MS = 3_000; + /** * How long a proxy serving in-process waits before trying the shared daemon * again, doubling after each miss up to the cap (#2277). Tunable through @@ -153,11 +160,13 @@ export async function runProxy( * release ({@link isOlderRelease}; the launcher replaces it, #2335) and * `'version-mismatch'` for any other version, which is left alone. Unlike * {@link runProxy} it does NOT pipe — the caller owns the socket. Used by the - * local-handshake proxy's background connect. + * local-handshake proxy's background connect. `helloTimeoutMs` bounds the wait + * for the hello, so a caller with a deadline can spend no more than it has. */ export async function connectWithHello( socketPath: string, expectedVersion: string = CodeGraphPackageVersion, + helloTimeoutMs: number = DAEMON_HELLO_TIMEOUT_MS, ): Promise { if (process.platform !== 'win32' && !fs.existsSync(socketPath)) return null; const socket = net.createConnection(socketPath); @@ -172,7 +181,7 @@ export async function connectWithHello( // AF_UNIX-over-DrvFs (WSL2 /mnt drives). A no-op guard makes the error // recoverable: the follow-up 'close' drives the caller's normal fallback. socket.on('error', () => { /* absorbed — see #974; 'close' drives the fallback */ }); - const hello = await readHelloLine(socket).catch(() => null); + const hello = await readHelloLine(socket, helloTimeoutMs).catch(() => null); if (!hello) { socket.destroy(); return null; // no daemon yet — caller should keep polling @@ -601,10 +610,10 @@ function startPpidWatchdogNoSocket(onDeath: () => void): void { /** * Read one CRLF/LF-terminated JSON line from the socket, parse it as the * daemon hello, and return it. Bounded to {@link MAX_HELLO_LINE_BYTES} so a - * malicious or broken peer can't OOM us. Times out at 3s — a healthy daemon - * sends hello immediately on accept. + * malicious or broken peer can't OOM us. Times out after `timeoutMs` (3s by + * default) — a healthy daemon sends hello immediately on accept. */ -function readHelloLine(socket: net.Socket): Promise { +function readHelloLine(socket: net.Socket, timeoutMs: number = DAEMON_HELLO_TIMEOUT_MS): Promise { return new Promise((resolve, reject) => { let buffer = ''; const cleanup = () => { @@ -647,7 +656,7 @@ function readHelloLine(socket: net.Socket): Promise { const timer = setTimeout(() => { cleanup(); reject(new Error('timed out waiting for daemon hello')); - }, 3000); + }, timeoutMs); timer.unref?.(); socket.on('data', onData); socket.on('error', onError); From 63ca5681db79f2c3d7230f240b257ebe3460df1a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 10:56:50 +0000 Subject: [PATCH 31/95] fix(db): a database SQLite can't open is reported as that, not as a Node version problem (#2796) Root cause: createDatabase() caught everything the node:sqlite connection threw and reported it as "CodeGraph requires node:sqlite (Node.js 22.5+)", though the module had loaded and SQLite itself had failed: a database another program held open (SQLITE_CANTOPEN on Windows), a folder the user can't write, an I/O error on a network share. Failures in the connection's first statements (locked, not a database, a WAL I/O error) came out raw, without the path. The MCP server then told the agent "No CodeGraph project is loaded" for a default project whose index exists, or "Tool execution failed: ... requires node:sqlite" for a projectPath one. Meanwhile a runtime that really lacks node:sqlite never got the hint: isInitialized's schema probe loaded the module outside its try, so the CLI and MCP server died with "Unhandled rejection: No such built-in module: node:sqlite". Fix: - src/db/node-sqlite.ts loads node:sqlite in one place; NodeSqliteUnavailableError carries the Node 22.5+ hint, now only for a module that cannot be loaded. The schema probe treats it as "unknown", so the command goes on to the open and reports it from inside its own catch. - src/db/open-error.ts: DatabaseOpenError names SQLite's code and the path, with a hint for BUSY, PERM, READONLY, IOERR (network share), CORRUPT, FULL, CANTOPEN (folder, permissions, another program holding the file) and NOTADB, keeping code/errcode/errstr and the cause. The Windows/WSL shared-index guidance still wins for its case. - DatabaseConnection.open/initialize wrap the connection's first statements as well as the constructor, and close the connection on failure. - MCP: the default project's open failure is reported to a tool call instead of "no project loaded", and a DatabaseOpenError or a missing node:sqlite is answered isError with the retry-once note (a malfunction, per AGENTS.md), worded as what failed rather than "an internal codegraph error". Not-indexed projects keep their success-shaped guidance. Validation: - Windows, an indexed project whose codegraph.db another process holds open with no sharing (a real SQLITE_CANTOPEN): CLI status, sync and query, an MCP default-project call and a projectPath call all said "requires node:sqlite" on main, or "No CodeGraph project is loaded" for the default project; with the fix all name SQLITE_CANTOPEN, the path and the hint. With --no-experimental-sqlite (node:sqlite unloadable), main's CLI and MCP server died with an unhandled rejection; the fix prints the Node 22.5+ explanation and the MCP call answers with it. - New __tests__/sqlite-open-error.test.ts (21 cases): main's source 18 fail / 3 pass, fix 21 pass. Real CANTOPEN (a directory, a file held exclusively), NOTADB and BUSY opens, the CLI and the MCP engine, plus each SQLite code's message. 5 mutations each failed at least one case. - The POSIX-only wsl-shared-index case for a real I/O error outside WSL now expects the named error (DatabaseOpenError, SQLITE_IOERR, the path) instead of the bare "disk I/O error"; it is skipped on Windows, so it was not run here. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/sqlite-open-error.test.ts | 286 ++++++++++++++++++++++++++++ __tests__/wsl-shared-index.test.ts | 10 +- src/db/index.ts | 33 +++- src/db/node-sqlite.ts | 36 ++++ src/db/open-error.ts | 93 +++++++++ src/db/sqlite-adapter.ts | 18 +- src/directory.ts | 6 +- src/mcp/engine.ts | 15 +- src/mcp/tools.ts | 32 +++- 10 files changed, 505 insertions(+), 25 deletions(-) create mode 100644 __tests__/sqlite-open-error.test.ts create mode 100644 src/db/node-sqlite.ts create mode 100644 src/db/open-error.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ffda40983c..726fef7e11 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404) - An index left incomplete by an indexing run that was stopped partway, by a crash, an out-of-memory kill or the server's own hang check, no longer gets the background server killed with "Main thread unresponsive" every time it starts: on a large project, restoring the database indexes the run had set aside took longer than the hang check allows, so each restart died at the same point and the index was never repaired. They are now restored in the background, and the project is served once they are back. Thanks @cbeaulieu-gt for the report and the first fix. (#1887) - A new agent session no longer waits up to about 12 minutes for its first answer when the background server is stuck: a server whose process still holds its connection open but never responds made the session wait out a long series of retries. The session now gives up within seconds and answers on its own, and switches back to the background server once that has been restarted. +- When CodeGraph can't open a project's database, it now says what went wrong: SQLite's error, the file's path, and what to check, such as another program holding the file open, a folder you can't write to, or a network share. Before, every such failure read "CodeGraph requires node:sqlite (Node.js 22.5+)", and an agent was told no project was loaded. That hint now appears only when Node really can't load its built-in SQLite, a case that used to end in an unhandled error instead. - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) - `codegraph install` and `upgrade` no longer break Codex's `config.toml` when CodeGraph's `[mcp_servers.codegraph]` entry is written differently from how CodeGraph writes it, such as indented, with spaces or quotes inside the brackets, or with Windows line endings: the entry is now updated in place, where before a second copy was added and Codex could load none of the file. A file already broken that way is repaired on the next install or upgrade, an example of the entry inside a multiline setting such as `developer_instructions` is no longer mistaken for it, uninstall removes the entry however it is written, and an entry you wrote with dotted keys or as an inline table is left as you wrote it. When your `mcp_servers` settings are one inline table, CodeGraph now says it can't add itself instead of breaking the file. Thanks @dencoseca for the report, and @danusha2345 and @rudycelekli for the first fixes. (#2562) diff --git a/__tests__/sqlite-open-error.test.ts b/__tests__/sqlite-open-error.test.ts new file mode 100644 index 0000000000..a9aca88e4f --- /dev/null +++ b/__tests__/sqlite-open-error.test.ts @@ -0,0 +1,286 @@ +/** + * A database SQLite cannot open is reported as what went wrong, with the path + * and a short hint, not as "CodeGraph requires node:sqlite (Node.js 22.5+)". + * + * Before, every failure inside the SQLite connection's constructor came out as + * that Node-version hint, so a database another program held open, one in a + * folder the user can't write, or one on a network share sent users after + * their Node version. A failure in the connection's first statements (locked, + * not a database, an I/O error) came out raw, with no path. The hint is now + * for a runtime that cannot load node:sqlite at all; the Windows/WSL shared + * index keeps its own guidance. + */ +import { afterEach, beforeAll, describe, expect, it } from 'vitest'; +import { spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { DatabaseConnection, DatabaseOpenError, NodeSqliteUnavailableError, getDatabasePath } from '../src/db'; +import { toDatabaseOpenError } from '../src/db/open-error'; +import { WslSharedIndexError } from '../src/db/wsl-shared-index'; +import { __setWslWindowsDriveForTests } from '../src/sync/watch-policy'; + +// Through require: vite does not resolve the node:sqlite builtin. +// eslint-disable-next-line @typescript-eslint/no-require-imports +const { DatabaseSync } = require('node:sqlite') as { DatabaseSync: new (path: string) => { exec(sql: string): void; close(): void } }; +const DIST = path.resolve(__dirname, '../dist'); +const BIN = path.join(DIST, 'bin', 'codegraph.js'); +const { MCPEngine } = require(path.join(DIST, 'mcp', 'engine.js')) as typeof import('../src/mcp/engine'); + +beforeAll(() => { + if (!fs.existsSync(BIN)) throw new Error('Build the project first (npm run build).'); +}); + +const dirs: string[] = []; +const cleanups: Array<() => void | Promise> = []; +afterEach(async () => { + for (const fn of cleanups.splice(0).reverse()) await fn(); + __setWslWindowsDriveForTests(null); + for (const dir of dirs.splice(0)) { + const deadline = Date.now() + 10_000; + for (;;) { + try { + fs.rmSync(dir, { recursive: true, force: true }); + break; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code ?? ''; + if (!['EBUSY', 'ENOTEMPTY', 'EPERM'].includes(code) || Date.now() > deadline) throw error; + await new Promise((resolve) => setTimeout(resolve, 100)); + } + } + } +}); + +function tempDir(): string { + const dir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'cg-open-error-'))); + dirs.push(dir); + return dir; +} + +/** An indexed project, closed again. */ +async function indexedProject(): Promise<{ root: string; dbPath: string }> { + const root = tempDir(); + fs.writeFileSync(path.join(root, 'a.ts'), 'export function alpha(): number { return 1; }\n'); + (await CodeGraph.init(root, { index: true })).close(); + return { root, dbPath: getDatabasePath(root) }; +} + +/** A node:sqlite error as SQLite reports it. */ +function sqliteError(errcode: number, message: string): Error { + return Object.assign(new Error(message), { code: 'ERR_SQLITE_ERROR', errcode, errstr: message }); +} + +/** + * Hold `file` open with no sharing, as a backup or antivirus scan can on + * Windows (libuv's UV_FS_O_EXLOCK): SQLite then can't open it at all. + */ +function holdExclusively(file: string): void { + const fd = fs.openSync(file, fs.constants.O_RDWR | 0x10000000); + cleanups.push(() => fs.closeSync(fd)); +} + +/** + * Lock the database from another connection so that no other can read it: + * exclusive locking mode, inside a write transaction. + */ +function lockDatabase(dbPath: string): void { + const holder = new DatabaseSync(dbPath); + holder.exec('PRAGMA locking_mode = EXCLUSIVE'); + holder.exec('BEGIN EXCLUSIVE'); + holder.exec('CREATE TABLE IF NOT EXISTS hold_2795 (x)'); + cleanups.push(() => { holder.exec('ROLLBACK'); holder.close(); }); +} + +function thrownBy(fn: () => unknown): unknown { + try { + fn(); + } catch (err) { + return err; + } + throw new Error('expected a throw'); +} + +describe('naming a failed open', () => { + const dbPath = path.join('/projects', 'app', '.codegraph', 'codegraph.db'); + + it.each([ + [14, 'unable to open database file', 'SQLITE_CANTOPEN', 'Check that the folder exists'], + [526, 'unable to open database file', 'SQLITE_CANTOPEN', 'holds the file open'], + [10, 'disk I/O error', 'SQLITE_IOERR', 'network share'], + [3850, 'disk I/O error', 'SQLITE_IOERR', 'local disk'], + [5, 'database is locked', 'SQLITE_BUSY', 'locked'], + [26, 'file is not a database', 'SQLITE_NOTADB', 'codegraph init'], + [11, 'database disk image is malformed', 'SQLITE_CORRUPT', 'codegraph index'], + [8, 'attempt to write a readonly database', 'SQLITE_READONLY', 'read-only'], + ])('reports SQLite code %i as %s, with the path and a hint', (errcode, message, name, hint) => { + const original = sqliteError(errcode, message); + const err = toDatabaseOpenError(original, dbPath) as DatabaseOpenError; + expect(err).toBeInstanceOf(DatabaseOpenError); + expect(err.message).toContain(dbPath); + expect(err.message).toContain(`${message} (${name}).`); + expect(err.message).toContain(hint); + expect(err.message).not.toContain('node:sqlite'); + expect(err.sqliteCode).toBe(name); + // Callers branching on SQLite's codes see what they saw before. + expect(err.code).toBe('ERR_SQLITE_ERROR'); + expect(err.errcode).toBe(errcode); + expect(err.cause).toBe(original); + }); + + it('names a code it has no hint for, without one', () => { + const err = toDatabaseOpenError(sqliteError(1, 'no such table: nodes'), dbPath) as DatabaseOpenError; + expect(err.message).toBe(`Could not open the CodeGraph database at ${dbPath}: no such table: nodes (SQLITE_ERROR).`); + }); + + it('leaves alone what SQLite did not report', () => { + const plain = new TypeError('The "path" argument must be of type string'); + expect(toDatabaseOpenError(plain, dbPath)).toBe(plain); + const missing = new NodeSqliteUnavailableError(new Error('No such built-in module: node:sqlite')); + expect(toDatabaseOpenError(missing, dbPath)).toBe(missing); + }); + + it('keeps the Windows/WSL guidance for an index the two share', () => { + __setWslWindowsDriveForTests(() => true); + const err = toDatabaseOpenError(sqliteError(10, 'disk I/O error'), '/mnt/c/app/.codegraph/codegraph.db'); + expect(err).toBeInstanceOf(WslSharedIndexError); + expect(toDatabaseOpenError(err, '/mnt/c/app/.codegraph/codegraph.db')).toBe(err); + }); + + it('keeps the node:sqlite explanation for a runtime that cannot load it', () => { + const err = new NodeSqliteUnavailableError(new Error('No such built-in module: node:sqlite')); + expect(err.message).toContain('CodeGraph requires node:sqlite (Node.js 22.5+)'); + expect(err.message).toContain('No such built-in module: node:sqlite'); + }); +}); + +describe('a database SQLite cannot open', () => { + it('reports a path SQLite cannot open as that, not as a Node version problem', () => { + // A directory where the file should be: SQLITE_CANTOPEN in the open itself. + const dbPath = path.join(tempDir(), 'codegraph.db'); + fs.mkdirSync(dbPath); + const err = thrownBy(() => DatabaseConnection.open(dbPath)) as Error; + expect(err).toBeInstanceOf(DatabaseOpenError); + expect(err.message).toContain(`Could not open the CodeGraph database at ${dbPath}: unable to open database file (SQLITE_CANTOPEN).`); + expect(err.message).not.toContain('node:sqlite'); + }); + + it('reports a file that is not a database, with its path', () => { + const dbPath = path.join(tempDir(), 'codegraph.db'); + fs.writeFileSync(dbPath, 'not a database '.repeat(300)); + const err = thrownBy(() => DatabaseConnection.open(dbPath)) as DatabaseOpenError; + expect(err).toBeInstanceOf(DatabaseOpenError); + expect(err.sqliteCode).toBe('SQLITE_NOTADB'); + expect(err.message).toContain(dbPath); + expect(err.message).toContain('codegraph init'); + }); + + it('reports a database another connection keeps locked, and opens it once released', async () => { + const { dbPath } = await indexedProject(); + lockDatabase(dbPath); + const err = thrownBy(() => DatabaseConnection.open(dbPath)) as DatabaseOpenError; + expect(err).toBeInstanceOf(DatabaseOpenError); + expect(err.sqliteCode).toBe('SQLITE_BUSY'); + expect(err.message).toContain(dbPath); + await cleanups.pop()!(); + DatabaseConnection.open(dbPath).close(); + }, 30_000); + + it.runIf(process.platform === 'win32')('reports a database another program holds open on Windows', async () => { + const { root, dbPath } = await indexedProject(); + holdExclusively(dbPath); + let err: unknown; + try { + (await CodeGraph.open(root)).close(); + } catch (e) { + err = e; + } + expect(err).toBeInstanceOf(DatabaseOpenError); + expect((err as Error).message).toContain(`Could not open the CodeGraph database at ${dbPath}: unable to open database file (SQLITE_CANTOPEN).`); + expect((err as Error).message).toContain('holds the file open'); + expect((err as Error).message).not.toContain('node:sqlite'); + }, 30_000); +}); + +/** Run the built CLI; output with colour codes removed. */ +function cli(args: string[], nodeFlags: string[] = []): { status: number | null; output: string } { + const r = spawnSync(process.execPath, [...nodeFlags, '--liftoff-only', '--disable-warning=ExperimentalWarning', BIN, ...args], { + encoding: 'utf8', + env: { ...process.env, CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_TELEMETRY: '0', DO_NOT_TRACK: '1', CODEGRAPH_NO_UPDATE_CHECK: '1', NO_COLOR: '1' }, + timeout: 120_000, + }); + return { status: r.status, output: `${r.stdout}${r.stderr}`.replace(/\x1b\[[0-9;]*m/g, '') }; +} + +describe('the CLI', () => { + it.runIf(process.platform === 'win32')('names the SQLite failure in status, sync and query', async () => { + const { root, dbPath } = await indexedProject(); + holdExclusively(dbPath); + for (const args of [['status', root], ['sync', root], ['query', 'alpha', '--path', root]]) { + const { status, output } = cli(args); + expect(status, output).not.toBe(0); + expect(output).toContain(`Could not open the CodeGraph database at ${dbPath}`); + expect(output).toContain('(SQLITE_CANTOPEN)'); + expect(output).not.toContain('requires node:sqlite'); + } + }, 120_000); + + it('explains a runtime without node:sqlite, without an unhandled rejection', async () => { + const { root } = await indexedProject(); + for (const args of [['status', root], ['sync', root]]) { + const { status, output } = cli(args, ['--no-experimental-sqlite']); + expect(status, output).not.toBe(0); + expect(output).toContain('CodeGraph requires node:sqlite (Node.js 22.5+)'); + expect(output).not.toContain('Unhandled rejection'); + } + }, 120_000); +}); + +describe('the MCP server', () => { + const engines: Array<{ stop(): Promise }> = []; + afterEach(async () => { + for (const engine of engines.splice(0)) await engine.stop(); + }); + + const text = (r: { content: Array<{ text: string }> }): string => r.content.map((c) => c.text).join('\n'); + + it('answers a call on the default project with why its index could not be opened', async () => { + const { root, dbPath } = await indexedProject(); + lockDatabase(dbPath); + const engine = new MCPEngine({ watch: false }); + engines.push(engine); + await engine.ensureInitialized(root); + expect(engine.hasDefaultCodeGraph()).toBe(false); + const result = await engine.getToolHandler().execute('codegraph_search', { query: 'alpha' }); + // A malfunction: isError, with the retry-once note, saying what failed. + expect(result.isError).toBe(true); + expect(text(result)).toContain(`Could not open the CodeGraph database at ${dbPath}`); + expect(text(result)).toContain('(SQLITE_BUSY)'); + expect(text(result)).toContain('Retry the call once'); + expect(text(result)).not.toContain('No CodeGraph project is loaded'); + expect(text(result)).not.toContain('internal codegraph error'); + }, 60_000); + + it('answers a projectPath call with why that index could not be opened', async () => { + const { root, dbPath } = await indexedProject(); + lockDatabase(dbPath); + const engine = new MCPEngine({ watch: false }); + engines.push(engine); + const result = await engine.getToolHandler().execute('codegraph_search', { query: 'alpha', projectPath: root }); + expect(result.isError).toBe(true); + expect(text(result)).toContain(`Could not open the CodeGraph database at ${dbPath}`); + expect(text(result)).toContain('(SQLITE_BUSY)'); + expect(text(result)).toContain('Retry the call once'); + expect(text(result)).not.toContain('node:sqlite'); + }, 60_000); + + it('still answers an unindexed default project with the not-indexed guidance', async () => { + const root = tempDir(); + const engine = new MCPEngine({ watch: false }); + engines.push(engine); + await engine.ensureInitialized(root); + const result = await engine.getToolHandler().execute('codegraph_search', { query: 'alpha' }); + expect(result.isError).toBeFalsy(); + expect(text(result)).toContain('No CodeGraph project is loaded'); + }, 30_000); +}); diff --git a/__tests__/wsl-shared-index.test.ts b/__tests__/wsl-shared-index.test.ts index a23a1a6859..fc57e8114a 100644 --- a/__tests__/wsl-shared-index.test.ts +++ b/__tests__/wsl-shared-index.test.ts @@ -25,7 +25,7 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import CodeGraph from '../src/index'; -import { DatabaseConnection } from '../src/db'; +import { DatabaseConnection, DatabaseOpenError } from '../src/db'; import { WslSharedIndexError, isSqliteIoError, toWslSharedIndexError } from '../src/db/wsl-shared-index'; import { codeGraphDirNameFor, findNearestCodeGraphRoot, getCodeGraphDir, isInitialized } from '../src/directory'; import { __setWslWindowsDriveForTests, isWslWindowsDrive } from '../src/sync/watch-policy'; @@ -252,7 +252,7 @@ describe.runIf(process.platform !== 'win32')('a real SQLite I/O error on open', expect((thrown as Error).message).toContain('CODEGRAPH_DIR=.codegraph-wsl'); }); - it('stays the raw SQLite error everywhere else', () => { + it('stays the SQLite error everywhere else, named and placed', () => { __setWslWindowsDriveForTests(() => false); let thrown: unknown; try { @@ -260,9 +260,11 @@ describe.runIf(process.platform !== 'win32')('a real SQLite I/O error on open', } catch (err) { thrown = err; } - expect(thrown).toBeInstanceOf(Error); + expect(thrown).toBeInstanceOf(DatabaseOpenError); expect(thrown).not.toBeInstanceOf(WslSharedIndexError); - expect((thrown as Error).message).toBe('disk I/O error'); + // SQLite's own failure with the path, not the WSL guidance. + expect((thrown as Error).message).toContain(`${dbPath}: disk I/O error (SQLITE_IOERR).`); + expect((thrown as Error).message).not.toContain('CODEGRAPH_DIR'); }); describe('through the MCP engine', () => { diff --git a/src/db/index.ts b/src/db/index.ts index 662e5301b2..c165d7e137 100644 --- a/src/db/index.ts +++ b/src/db/index.ts @@ -10,8 +10,11 @@ import * as path from 'path'; import { SchemaVersion } from '../types'; import { runMigrations, getCurrentVersion, CURRENT_SCHEMA_VERSION } from './migrations'; import { getCodeGraphDir, statInode } from '../directory'; +import { toDatabaseOpenError } from './open-error'; export { SqliteDatabase, SqliteBackend } from './sqlite-adapter'; +export { DatabaseOpenError } from './open-error'; +export { NodeSqliteUnavailableError } from './node-sqlite'; /** * Apply connection-level PRAGMAs. Shared by `initialize` and `open` so the two @@ -44,6 +47,22 @@ function configureConnection(db: SqliteDatabase, readOnly = false): void { db.pragma(`journal_size_limit = ${WAL_HEAL_THRESHOLD_BYTES}`); } +/** + * Run the rest of an open on the connection `db`. Its first statements read + * the file, so a database SQLite cannot use (locked, damaged, not a database, + * an I/O error) fails here as well as when the connection is made. Such a + * failure names SQLite's code and the path ({@link DatabaseOpenError}), and + * leaves no connection open. + */ +function withOpenFailures(db: SqliteDatabase, dbPath: string, finish: () => T): T { + try { + return finish(); + } catch (err) { + try { db.close(); } catch { /* the open's failure is the one to report */ } + throw toDatabaseOpenError(err, dbPath); + } +} + /** * WAL size past which `healOversizedWal` (run at every `open`) checkpoints and * truncates the file, and to which `journal_size_limit` clips the WAL after any @@ -107,9 +126,12 @@ export class DatabaseConnection { fs.mkdirSync(dir, { recursive: true }); } - // Create and configure database const { db, backend } = createDatabase(dbPath); + return withOpenFailures(db, dbPath, () => DatabaseConnection.initializeOn(db, backend, dbPath)); + } + /** The rest of {@link initialize}: configure the connection and create the schema. */ + private static initializeOn(db: SqliteDatabase, backend: SqliteBackend, dbPath: string): DatabaseConnection { configureConnection(db); // Run schema initialization, splitting FTS5 from the rest so @@ -175,7 +197,16 @@ export class DatabaseConnection { } const { db, backend } = createDatabase(dbPath, options); + return withOpenFailures(db, dbPath, () => DatabaseConnection.openOn(db, backend, dbPath, options)); + } + /** The rest of {@link open}: configure the connection, then migrate and repair a writable one. */ + private static openOn( + db: SqliteDatabase, + backend: SqliteBackend, + dbPath: string, + options: { readOnly?: boolean; deferIndexRecovery?: boolean }, + ): DatabaseConnection { configureConnection(db, options.readOnly); // Detect FTS5 availability for search fallback (#1532) diff --git a/src/db/node-sqlite.ts b/src/db/node-sqlite.ts new file mode 100644 index 0000000000..788e9f13a3 --- /dev/null +++ b/src/db/node-sqlite.ts @@ -0,0 +1,36 @@ +/** + * Node's built-in `node:sqlite`, loaded in one place, so that a runtime + * without it fails with the one explanation that fits it: CodeGraph needs + * Node.js 22.5+, or its own bundled runtime. Any other failure to open a + * database is SQLite's own, and is reported as such (see ./open-error.ts). + */ + +/** The parts of `node:sqlite` CodeGraph uses. */ +export interface NodeSqliteModule { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + DatabaseSync: new (path: string, options?: { readOnly?: boolean }) => any; +} + +/** `node:sqlite` is missing or could not be loaded in this Node.js runtime. */ +export class NodeSqliteUnavailableError extends Error { + constructor(cause: unknown) { + const reason = cause instanceof Error ? cause.message : String(cause); + super( + `Node.js ${process.version} could not load its built-in SQLite module (${reason}).\n` + + 'CodeGraph requires node:sqlite (Node.js 22.5+). Install the self-contained CodeGraph ' + + 'release (it bundles a compatible Node), or run on Node 22.5+.', + { cause } + ); + this.name = 'NodeSqliteUnavailableError'; + } +} + +/** Load `node:sqlite`, or throw {@link NodeSqliteUnavailableError}. */ +export function loadNodeSqlite(): NodeSqliteModule { + try { + // eslint-disable-next-line @typescript-eslint/no-require-imports + return require('node:sqlite') as NodeSqliteModule; + } catch (err) { + throw new NodeSqliteUnavailableError(err); + } +} diff --git a/src/db/open-error.ts b/src/db/open-error.ts new file mode 100644 index 0000000000..345d7745d5 --- /dev/null +++ b/src/db/open-error.ts @@ -0,0 +1,93 @@ +/** + * Opening a CodeGraph database failed in SQLite itself: report what SQLite + * said, with the database's path and a short hint at the usual cause. Before, + * every such failure came out as "CodeGraph requires node:sqlite (Node.js + * 22.5+)", which sent users after their Node version when the file was locked, + * unreadable, or on a network share. That hint is now only for a runtime that + * cannot load `node:sqlite` at all (./node-sqlite.ts); an index Windows and WSL + * share keeps its own guidance (./wsl-shared-index.ts). + */ + +import { NodeSqliteUnavailableError } from './node-sqlite'; +import { toWslSharedIndexError, WslSharedIndexError } from './wsl-shared-index'; + +/** SQLite's primary result codes, by number. node:sqlite reports the extended code; its low byte is the primary one. */ +const SQLITE_CODE_NAMES: readonly string[] = [ + 'SQLITE_OK', 'SQLITE_ERROR', 'SQLITE_INTERNAL', 'SQLITE_PERM', 'SQLITE_ABORT', 'SQLITE_BUSY', + 'SQLITE_LOCKED', 'SQLITE_NOMEM', 'SQLITE_READONLY', 'SQLITE_INTERRUPT', 'SQLITE_IOERR', + 'SQLITE_CORRUPT', 'SQLITE_NOTFOUND', 'SQLITE_FULL', 'SQLITE_CANTOPEN', 'SQLITE_PROTOCOL', + 'SQLITE_EMPTY', 'SQLITE_SCHEMA', 'SQLITE_TOOBIG', 'SQLITE_CONSTRAINT', 'SQLITE_MISMATCH', + 'SQLITE_MISUSE', 'SQLITE_NOLFS', 'SQLITE_AUTH', 'SQLITE_FORMAT', 'SQLITE_RANGE', 'SQLITE_NOTADB', +]; + +/** What to check, for the failures an open meets in practice. */ +const HINTS: Readonly> = { + SQLITE_BUSY: 'Another process has the database locked; retry when it is done.', + SQLITE_PERM: 'Check that you can read and write the file and its folder.', + SQLITE_READONLY: 'The database or its folder is read-only, and CodeGraph has to write its index there.', + SQLITE_IOERR: + 'SQLite could not read or write the file. On a network share, or another filesystem without ' + + 'reliable file locking, keep the project and its index on a local disk.', + SQLITE_CORRUPT: 'The database is damaged; run "codegraph index" to rebuild it.', + SQLITE_FULL: 'The disk is full.', + SQLITE_CANTOPEN: + 'Check that the folder exists and that you can read and write it, and that no other program ' + + '(a backup or antivirus scan, say) holds the file open.', + SQLITE_NOTADB: 'The file is not a SQLite database; move it aside and run "codegraph init" to build a new index.', +}; + +/** A failure SQLite reported while opening the database at `dbPath`. */ +export class DatabaseOpenError extends Error { + /** node:sqlite's error code ("ERR_SQLITE_ERROR"), kept from the original. */ + readonly code: unknown; + /** SQLite's extended result code, kept from the original. */ + readonly errcode: unknown; + readonly errstr: unknown; + /** SQLite's name for the primary result code, e.g. "SQLITE_CANTOPEN". */ + readonly sqliteCode: string; + readonly dbPath: string; + + constructor(original: Error, dbPath: string) { + const codes = original as Error & { code?: unknown; errcode?: unknown; errstr?: unknown }; + const primary = typeof codes.errcode === 'number' ? codes.errcode & 0xff : -1; + const sqliteCode = SQLITE_CODE_NAMES[primary] ?? `SQLite error ${String(codes.errcode)}`; + const hint = HINTS[sqliteCode]; + super( + `Could not open the CodeGraph database at ${dbPath}: ${original.message} (${sqliteCode}).${hint ? ` ${hint}` : ''}`, + { cause: original } + ); + this.name = 'DatabaseOpenError'; + this.code = codes.code; + this.errcode = codes.errcode; + this.errstr = codes.errstr; + this.sqliteCode = sqliteCode; + this.dbPath = dbPath; + } +} + +/** + * The error to throw for `err`, raised while opening the database at + * `dbPath`: the Windows/WSL guidance for an index the two share, a + * {@link DatabaseOpenError} for any other failure SQLite reported (it carries + * a result code), and anything else unchanged, the missing-`node:sqlite` + * explanation included. + */ +export function toDatabaseOpenError(err: unknown, dbPath: string): unknown { + if (err instanceof DatabaseOpenError || err instanceof WslSharedIndexError || err instanceof NodeSqliteUnavailableError) { + return err; + } + const shared = toWslSharedIndexError(err, dbPath); + if (shared) return shared; + if (err instanceof Error && typeof (err as { errcode?: unknown }).errcode === 'number') { + return new DatabaseOpenError(err, dbPath); + } + return err; +} + +/** + * A project's index exists but could not be opened: what a caller reports as + * that, not as "no index here". The Windows/WSL case has its own guidance. + */ +export function isDatabaseOpenFailure(err: unknown): err is DatabaseOpenError | NodeSqliteUnavailableError { + return err instanceof DatabaseOpenError || err instanceof NodeSqliteUnavailableError; +} diff --git a/src/db/sqlite-adapter.ts b/src/db/sqlite-adapter.ts index 39b2d8ba0f..8e7a104f64 100644 --- a/src/db/sqlite-adapter.ts +++ b/src/db/sqlite-adapter.ts @@ -10,6 +10,8 @@ * wasm fallback. When run from source instead, it requires Node >= 22.5. */ +import { loadNodeSqlite } from './node-sqlite'; +import { toDatabaseOpenError } from './open-error'; import { toWslSharedIndexError } from './wsl-shared-index'; export interface SqliteStatement { @@ -57,8 +59,7 @@ class NodeSqliteAdapter implements SqliteDatabase { private readonly _dbPath: string; constructor(dbPath: string, opts?: { readOnly?: boolean }) { - // eslint-disable-next-line @typescript-eslint/no-require-imports - const { DatabaseSync } = require('node:sqlite'); + const { DatabaseSync } = loadNodeSqlite(); this._dbPath = dbPath; this._db = opts?.readOnly ? new DatabaseSync(dbPath, { readOnly: true }) : new DatabaseSync(dbPath); } @@ -200,15 +201,8 @@ export function createDatabase(dbPath: string, opts?: { readOnly?: boolean }): { try { return { db: new NodeSqliteAdapter(dbPath, opts), backend: 'node-sqlite' }; } catch (error) { - // node:sqlite loaded and SQLite itself failed — not a missing-module case. - const shared = toWslSharedIndexError(error, dbPath); - if (shared) throw shared; - const msg = error instanceof Error ? error.message : String(error); - throw new Error( - 'Failed to open SQLite via the built-in node:sqlite module.\n' + - 'CodeGraph requires node:sqlite (Node.js 22.5+). Install the self-contained\n' + - 'CodeGraph release (it bundles a compatible Node), or run on Node 22.5+.\n' + - `Underlying error: ${msg}` - ); + // A runtime without node:sqlite says so (NodeSqliteUnavailableError); + // anything SQLite itself reported names its code and the path. + throw toDatabaseOpenError(error, dbPath); } } diff --git a/src/directory.ts b/src/directory.ts index c458457e4e..eb359a0c62 100644 --- a/src/directory.ts +++ b/src/directory.ts @@ -8,6 +8,7 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import { isWslWindowsDrive } from './sync/watch-policy'; +import { loadNodeSqlite } from './db/node-sqlite'; /** The default per-project data directory name. */ export const DEFAULT_CODEGRAPH_DIR = '.codegraph'; @@ -214,10 +215,11 @@ function hasCodeGraphSchema(dbPath: string, st: fs.Stats): boolean { * second connection opened and closed here leaves the first one's locks alone. */ function probeSchema(dbPath: string): 'schema' | 'no-schema' | 'not-sqlite' | 'unknown' { - // eslint-disable-next-line @typescript-eslint/no-require-imports - const { DatabaseSync } = require('node:sqlite'); let db: any = null; try { + // A runtime without node:sqlite can't inspect the file: `unknown`, so the + // command goes on to open it and reports why that is impossible. + const { DatabaseSync } = loadNodeSqlite(); db = new DatabaseSync(dbPath, { readOnly: true }); const row = db.prepare("SELECT 1 AS ok FROM sqlite_master WHERE type = 'table' AND name = 'nodes'").get(); return row !== undefined ? 'schema' : 'no-schema'; diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 1f8470e9aa..8764620465 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -16,6 +16,8 @@ import type CodeGraph from '../index'; import { resolveServerRoot } from '../directory'; import { ToolHandler } from './tools'; import { WslSharedIndexError } from '../db/wsl-shared-index'; +import { DatabaseOpenError, isDatabaseOpenFailure } from '../db/open-error'; +import type { NodeSqliteUnavailableError } from '../db/node-sqlite'; import { assertNoRebuild, releaseWriterLock, tryAcquireWriterLock, writerLockHeldMessage } from './writer-lock'; import { QueryPool, resolvePoolSize } from './query-pool'; import { endFreshnessMeasurements } from './index-freshness'; @@ -33,6 +35,15 @@ const loadCodeGraph = (): typeof import('../index').default => /** How often the per-tool-call retry may re-run the sub-project down-scan. */ const RETRY_SUBSCAN_TTL_MS = 5_000; +/** + * Why the default project failed to open, when a tool call should say so + * rather than "no project loaded": the Windows/WSL shared index (#995), or an + * index that exists but SQLite could not open, or no node:sqlite at all. + */ +function reportableOpenFailure(err: unknown): WslSharedIndexError | DatabaseOpenError | NodeSqliteUnavailableError | null { + return err instanceof WslSharedIndexError || isDatabaseOpenFailure(err) ? err : null; +} + export interface MCPEngineOptions { /** Serve existing index contents without syncing, watching, or claiming a writer slot. */ readOnly?: boolean; @@ -265,7 +276,7 @@ export class MCPEngine { } } catch (err) { // Still failing — caller will try again on the next tool call. - this.toolHandler.setDefaultOpenFailure(err instanceof WslSharedIndexError ? err : null); + this.toolHandler.setDefaultOpenFailure(reportableOpenFailure(err)); } } @@ -396,7 +407,7 @@ export class MCPEngine { const msg = err instanceof Error ? err.message : String(err); process.stderr.write(`[CodeGraph MCP] Failed to open project at ${resolvedRoot}: ${msg}\n`); // The agent otherwise hears only "no project loaded" (#995). - this.toolHandler.setDefaultOpenFailure(err instanceof WslSharedIndexError ? err : null); + this.toolHandler.setDefaultOpenFailure(reportableOpenFailure(err)); } } diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 2be63a7c05..d3fc528ac8 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -8,6 +8,8 @@ import type CodeGraph from '../index'; import type { QueryPool } from './query-pool'; import { findNearestCodeGraphRoot, isSameIndexRoot } from '../directory'; import { WslSharedIndexError } from '../db/wsl-shared-index'; +import { DatabaseOpenError, isDatabaseOpenFailure } from '../db/open-error'; +import type { NodeSqliteUnavailableError } from '../db/node-sqlite'; // Lazy-load the heavy CodeGraph chain off the MCP startup path — see the same // helper in engine.ts. ToolHandler must load to answer tools/list (static // schemas), but it must NOT drag in sqlite/query layers before the daemon binds; @@ -105,6 +107,20 @@ export class NotIndexedError extends Error {} * {@link NotIndexedError} it answers SUCCESS-shaped — never `isError`, which * would teach the agent to abandon codegraph for projects that work fine. */ +/** + * A project whose index exists but could not be opened: SQLite's own failure, + * with the path and what to check, or a runtime without node:sqlite. A real + * malfunction, so it stays `isError` with the retry-once note, but it says + * what failed instead of calling it an internal error. + */ +function databaseOpenFailureText(err: DatabaseOpenError | NodeSqliteUnavailableError): string { + return ( + `${err.message}\n` + + 'Retry the call once; if it persists, continue without codegraph for this task and pass ' + + 'the message above on to the user.' + ); +} + function wslSharedIndexGuidance(err: WslSharedIndexError): string { return ( `${err.message}\n\n` + @@ -2021,9 +2037,10 @@ export class ToolHandler { private knownSubprojects: string[] = []; private knownSubprojectsBase: string | null = null; // Why the default project failed to open, when that is worth telling the - // agent instead of "no project loaded" — today only the Windows/WSL - // shared-index error (#995). Engine-maintained; cleared by a successful open. - private defaultOpenFailure: WslSharedIndexError | null = null; + // agent instead of "no project loaded": the Windows/WSL shared-index error + // (#995), or an index that exists but could not be opened (SQLite's own + // failure, or no node:sqlite). Engine-maintained; cleared by a successful open. + private defaultOpenFailure: WslSharedIndexError | DatabaseOpenError | NodeSqliteUnavailableError | null = null; // Per-start-path cache of the git worktree/index mismatch (issue #155). The // mismatch is a fixed property of (where the request came from → which // .codegraph/ it resolves to), so the up-to-two `git rev-parse` spawns run @@ -2083,7 +2100,7 @@ export class ToolHandler { * call that needs it answers with that fix rather than "no project loaded". * `null` clears it. */ - setDefaultOpenFailure(err: WslSharedIndexError | null): void { + setDefaultOpenFailure(err: WslSharedIndexError | DatabaseOpenError | NodeSqliteUnavailableError | null): void { this.defaultOpenFailure = err; } @@ -3021,6 +3038,10 @@ export class ToolHandler { if (err instanceof PathRefusalError) { return this.errorResult(err.message); } + // An index that exists but could not be opened: a malfunction, said as what it is. + if (isDatabaseOpenFailure(err)) { + return this.errorResult(databaseOpenFailureText(err)); + } return this.errorResult( `Tool execution failed: ${err instanceof Error ? err.message : String(err)}. ` + 'This is an internal codegraph error — retry the call once; if it persists, ' + @@ -3109,6 +3130,9 @@ export class ToolHandler { if (err instanceof PathRefusalError) { return this.errorResult(err.message); } + if (isDatabaseOpenFailure(err)) { + return this.errorResult(databaseOpenFailureText(err)); + } return this.errorResult( `Tool execution failed: ${err instanceof Error ? err.message : String(err)}. ` + 'This is an internal codegraph error — retry the call once; if it persists, ' + From 407efb1c65885f4f89ac29a795c461de2a1c7784 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 11:10:10 +0000 Subject: [PATCH 32/95] fix(pascal): Delphi calls written without parentheses link to the routine (#2797) Delphi calls a routine that takes no arguments without parentheses (`Before;`, `X := FProvider.Value + 1`, `if Ready then`), and only a statement-level `Obj.Method;` was recorded, so most call sites were missing from callers, callees and impact. Bare names in arguments and assignments were captured as function values and matched by name alone, which linked locals and parameters to other classes' methods. The extractor now records each name written without parentheses with how it is used (a statement, a value, an event's method, a receiver), the routine's class, and the declared type of a receiver the routine declares. It leaves out what the routine declares, data the class and the unit declare, names the class's code assigns or passes to Inc/Dec, names Delphi's conventions give to data, and anything in a `with` body but an event's method. A new resolver links such a name only to a method or function Delphi's scope rules reach: the class and its ancestors, the unit, then the units it uses, last first, or a method of a receiver's declared type. Data of that name links nothing. A function runs (calls); a procedure read as a value, an event's method, and a routine that needs arguments written without them are method pointers (references). Where the tree has errors, the class and declarations come from the source text, and a routine's own header is no call. `class sealed (TBase)` and `class abstract (TBase)` heads now name their ancestors. Pascal function values capture only `@Handler`. A/B against main 8073637d (SynEdit, HeidiSQL, mORMot2, DWScript, DelphiMVCFramework): added edges 3011/4626/6501/5092/6776; main's name-matched function values now linked by scope 2387/2727/948/726/712, 114/281/554/342/586 of them as calls; removed 85/139/802/333/707, triaged as main's false positives (locals and parameters linked by name, other classes' methods, routine headers misparsed in broken trees) and own-class retargets, less deliberate losses (reads in `with` bodies, lowercase or type-like names read as values). Sampled precision of added edges: 196/196. leveldb and SDWebImage byte-identical. CPU, median of 3 interleaved: +15.2%, +16.2%, +8.7%, +12.9%, and +0.8% for DelphiMVCFramework (one run each). Fixes #2514 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/pascal-parenless-calls.test.ts | 749 +++++++++++++++++++++++ src/db/queries.ts | 9 + src/extraction/function-ref.ts | 6 +- src/extraction/tree-sitter.ts | 521 +++++++++++++++- src/resolution/index.ts | 12 + src/resolution/name-matcher.ts | 4 +- src/resolution/pascal-scope.ts | 483 +++++++++++++++ src/resolution/types.ts | 11 + 9 files changed, 1779 insertions(+), 17 deletions(-) create mode 100644 __tests__/pascal-parenless-calls.test.ts create mode 100644 src/resolution/pascal-scope.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 726fef7e11..74d35ba919 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Delphi, a routine with an attribute on one of its parameters, like `const [ref] AValue: Integer` or DelphiMVCFramework's `const [MVCFromBody] Person: TPerson`, is now indexed with its calls, and so is every class member declared after it. Before, the routine was lost or its calls were credited to the file, the parameter showed up as a field of the class, and the members after it were missing. Thanks @stalexteam for the report. (#2545) - In Delphi, a `for … in` loop that declares its variable inline, like `for var lItem in AList do` or `for var lHandle: THandle in lHandles do`, now has its calls indexed, and a call on a typed loop variable reaches that type's method. Before, the whole loop was skipped, so a routine calling a method only from such a loop was missing from that method's callers. Thanks @stalexteam for the report. (#2546) - In Delphi, a call made on the result of another call, like `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)` or `List[IndexOf(Name)].Free`, now has the inner calls indexed too, so `MD5FileSized` and `TMeta.ExtendedPath` list that routine among their callers. A typecast like `TButton(Sender).Caption` or `F := TFoo(Obj)` now counts as a use of the type instead of creating one. Thanks @stalexteam for the report. (#2547) +- In Delphi, a routine called without parentheses, the usual way to call one that takes no arguments, like `Before;`, `if Ready then`, `Result := FProvider.Value + 1` or `QueryTabs.ActiveTab.MemoFilename`, now counts as a call, so it shows up in callers, callees and impact, and a method given to an event, like `OnClick := ButtonClick`, is linked as a reference. A name is linked only to a method or function its code reaches, in its class and the classes it inherits from (`class sealed (TBase)` included), its unit, or the units it uses, never to a field, property or variable of that name. Thanks @OleksiyPenkov for the report, and @drakeo338 and @OleksiyPenkov for their pull requests. (#2514) - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. diff --git a/__tests__/pascal-parenless-calls.test.ts b/__tests__/pascal-parenless-calls.test.ts new file mode 100644 index 0000000000..04bf175634 --- /dev/null +++ b/__tests__/pascal-parenless-calls.test.ts @@ -0,0 +1,749 @@ +/** + * Delphi calls a parameterless routine without parentheses (#2514): + * `Before;`, `Result := FProvider.Value + 1`, `if Ready then`, and a method + * given to an event, `OnGet := FProvider.Value`. None of these was linked, so + * callers, callees and impact missed most call sites in real Delphi code. + * The same spelling reads a field, a property, a variable or a constant, so a + * name is linked only when Delphi's scope rules reach a method or function of + * that name: the routine's own declarations, its class and ancestors, the + * unit, then the units it uses. Data of that name links nothing. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; + +beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['pascal']); +}); + +async function withIndex(files: Record, check: (cg: CodeGraph) => void): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-parenless-')); + try { + for (const [name, code] of Object.entries(files)) fs.writeFileSync(path.join(dir, name), code); + const cg = await CodeGraph.init(dir, { index: true }); + try { + check(cg); + } finally { + cg.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +/** The calls and references a routine makes, as `kind Target::Name`. */ +function edgesFrom(cg: CodeGraph, qualifiedName: string): string[] { + const source = cg.getNodesByKind('function').concat(cg.getNodesByKind('method')).find((n) => n.qualifiedName === qualifiedName); + expect(source, qualifiedName).toBeDefined(); + return cg + .getOutgoingEdges(source!.id) + .filter((e) => e.kind === 'calls' || e.kind === 'references') + .map((e) => `${e.kind} ${cg.getNode(e.target)?.qualifiedName ?? '?'}`) + .sort(); +} + +// The issue's own unit. +const UNIT_A = `unit UnitA; + +interface + +type + TGetIntEvent = function: Integer of object; + + TProvider = class + public + function Value: Integer; + function ValueParens(): Integer; + end; + + TConsumer = class + private + FProvider: TProvider; + FOnGet: TGetIntEvent; + public + function UseNoParens: Integer; + function UseParens: Integer; + function UseEvent: Integer; + procedure Wire; + property OnGet: TGetIntEvent read FOnGet write FOnGet; + end; + +implementation + +function TProvider.Value: Integer; +begin + Result := 1; +end; + +function TProvider.ValueParens(): Integer; +begin + Result := 2; +end; + +function TConsumer.UseNoParens: Integer; +begin + Result := FProvider.Value + 1; +end; + +function TConsumer.UseParens: Integer; +begin + Result := FProvider.ValueParens() + 1; +end; + +procedure TConsumer.Wire; +begin + OnGet := FProvider.Value; +end; + +function TConsumer.UseEvent: Integer; +begin + Result := FOnGet; +end; + +end. +`; + +describe('Delphi calls without parentheses (#2514)', () => { + it('links the issue\'s paren-less call as a call and the event\'s method as a reference', async () => { + await withIndex({ 'UnitA.pas': UNIT_A }, (cg) => { + expect(edgesFrom(cg, 'TConsumer::UseNoParens')).toEqual(['calls TProvider::Value']); + expect(edgesFrom(cg, 'TConsumer::UseParens')).toEqual(['calls TProvider::ValueParens']); + expect(edgesFrom(cg, 'TConsumer::Wire')).toEqual(['references TProvider::Value']); + // `FOnGet` is a field: reading it calls whatever was assigned, which the code doesn't say. + expect(edgesFrom(cg, 'TConsumer::UseEvent')).toEqual([]); + }); + }); + + it('links a bare name to the class\'s method or the unit\'s routine, as a statement or for its value', async () => { + const code = `unit Unit1; + +interface + +type + TMin = class + public + procedure Before; + procedure After; + function Value: Integer; + procedure UseValue; + end; + +procedure FreeProc; + +implementation + +procedure FreeProc; +begin +end; + +procedure TMin.Before; +begin +end; + +procedure TMin.After; +begin + Before; + freeproc; +end; + +function TMin.Value: Integer; +begin + Result := 1; +end; + +procedure TMin.UseValue; +var + X: Integer; +begin + X := Value; + if Value > 0 then + Before; +end; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + // Pascal names are case-insensitive: `freeproc` is `FreeProc`. + expect(edgesFrom(cg, 'TMin::After')).toEqual(['calls FreeProc', 'calls TMin::Before']); + expect(edgesFrom(cg, 'TMin::UseValue')).toEqual(['calls TMin::Before', 'calls TMin::Value', 'calls TMin::Value']); + }); + }); + + it('never links a name that reads a field, property, variable, constant or parameter', async () => { + const base = `unit Base; + +interface + +type + TBase = class + protected + FReady: Boolean; + public + property Ready: Boolean read FReady; + end; + +implementation + +end. +`; + const decoys = `unit Decoys; + +interface + +function Ready: Boolean; +function Count: Integer; +function Items: Integer; +function Limit: Integer; +function Total: Integer; +function FItems: Integer; + +implementation + +function Ready: Boolean; begin Result := True; end; +function Count: Integer; begin Result := 0; end; +function Items: Integer; begin Result := 0; end; +function Limit: Integer; begin Result := 0; end; +function Total: Integer; begin Result := 0; end; +function FItems: Integer; begin Result := 0; end; + +end. +`; + const main = `unit Main; + +interface + +uses Decoys, Base; + +type + TList = class(TBase) + private + FItems: Integer; + function GetCount: Integer; + public + property Count: Integer read GetCount; + function Run(Total: Integer): Integer; + end; + +const + Limit = 10; + +var + Items: Integer; + +implementation + +function TList.GetCount: Integer; +begin + Result := 0; +end; + +function TList.Run(Total: Integer): Integer; +begin + Result := Count + Items + Limit + Total + FItems; + if Ready then + Result := 0; +end; + +end. +`; + await withIndex({ 'Base.pas': base, 'Decoys.pas': decoys, 'Main.pas': main }, (cg) => { + // `Ready` is the inherited property, which the unit's own + // `uses Decoys` doesn't reach past. + expect(edgesFrom(cg, 'TList::Run')).toEqual([]); + }); + }); + + it('follows the class\'s ancestors and the units it uses, the last one first', async () => { + const lib = `unit Lib; + +interface + +type + TBase = class + public + procedure Setup; + end; + +procedure Helper; + +implementation + +procedure TBase.Setup; begin end; +procedure Helper; begin end; + +end. +`; + const other = `unit Other; + +interface + +procedure Helper; + +implementation + +procedure Helper; begin end; + +end. +`; + const unused = `unit Unused; + +interface + +procedure Teardown; + +implementation + +procedure Teardown; begin end; + +end. +`; + const main = `unit Main; + +interface + +uses Other, Lib; + +type + TChild = class(TBase) + public + procedure Run; + end; + +implementation + +procedure TChild.Run; +begin + Setup; + Helper; + Teardown; +end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Other.pas': other, 'Unused.pas': unused, 'Main.pas': main }, (cg) => { + const helpers = cg.getNodesByKind('function').filter((n) => n.name === 'Helper'); + const libHelper = helpers.find((n) => n.filePath.endsWith('Lib.pas')); + const run = cg.getNodesByKind('method').find((n) => n.qualifiedName === 'TChild::Run')!; + const targets = cg.getOutgoingEdges(run.id).filter((e) => e.kind === 'calls').map((e) => e.target); + // `Lib` is used last, so its `Helper` hides `Other`'s; `Unused` isn't used at all. + expect(targets).toContain(libHelper!.id); + expect(edgesFrom(cg, 'TChild::Run')).toEqual(['calls Helper', 'calls TBase::Setup']); + }); + }); + + it('links a function used as a receiver, through fields, parameters and locals of a declared type', async () => { + // The issue's real project: `FProjectPanel.ActiveModelSeries.Visible`. + const code = `unit Unit1; + +interface + +type + TSeries = class + public + procedure BeginUpdate; + function Visible: Boolean; + end; + + TPanel = class + public + function ActiveSeries: TSeries; + end; + + TForm1 = class + private + FPanel: TPanel; + function CurrentPanel: TPanel; + public + procedure Run; + procedure RunWith(APanel: TPanel); + end; + +implementation + +procedure TSeries.BeginUpdate; begin end; +function TSeries.Visible: Boolean; begin Result := True; end; +function TPanel.ActiveSeries: TSeries; begin Result := nil; end; +function TForm1.CurrentPanel: TPanel; begin Result := FPanel; end; + +procedure TForm1.Run; +begin + FPanel.ActiveSeries.BeginUpdate; + if CurrentPanel.ActiveSeries.Visible then + Exit; +end; + +procedure TForm1.RunWith(APanel: TPanel); +var + S: TSeries; +begin + S := APanel.ActiveSeries; + if S.Visible then + Exit; +end; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + expect(edgesFrom(cg, 'TForm1::Run')).toEqual([ + 'calls TForm1::CurrentPanel', + 'calls TPanel::ActiveSeries', + 'calls TPanel::ActiveSeries', + 'calls TSeries::BeginUpdate', + 'calls TSeries::Visible', + ]); + expect(edgesFrom(cg, 'TForm1::RunWith')).toEqual(['calls TPanel::ActiveSeries', 'calls TSeries::Visible']); + }); + }); + + it('gives a method to an event or an argument as a reference, a method pointer', async () => { + const code = `unit Unit1; + +interface + +type + TNotify = procedure of object; + + TButton = class + public + OnClick: TNotify; + procedure Click; + end; + + TItem = class + public + constructor Create(AOwner: TObject); + end; + + TForm1 = class + private + FButton: TButton; + procedure ButtonClick; + procedure Register(Handler: TNotify); + function Title: string; + function Compare(A, B: Integer): Integer; + procedure Sort(Fn: TCompare); + public + procedure Setup; + procedure SetupWith; + end; + +implementation + +constructor TItem.Create(AOwner: TObject); begin end; +procedure TButton.Click; begin end; +procedure TForm1.ButtonClick; begin end; +procedure TForm1.Register(Handler: TNotify); begin end; +function TForm1.Title: string; begin Result := ''; end; +function TForm1.Compare(A, B: Integer): Integer; begin Result := A - B; end; +procedure TForm1.Sort(Fn: TCompare); begin end; + +procedure TForm1.Setup; +var + Item: TObject; +begin + FButton.OnClick := ButtonClick; + Register(ButtonClick); + // A function that needs arguments, written without them, is its pointer. + Sort(Compare); + // A constructor written without the arguments it needs is an inherited + // one outside the project, TObject.Create. + Item := TItem.Create; +end; + +procedure TForm1.SetupWith; +var + S: string; +begin + with FButton do + begin + Click; + OnClick := ButtonClick; + S := Title; + end; +end; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + expect(edgesFrom(cg, 'TForm1::Setup')).toEqual([ + 'calls TForm1::Register', + 'calls TForm1::Sort', + 'references TForm1::ButtonClick', + 'references TForm1::ButtonClick', + 'references TForm1::Compare', + ]); + // In a `with` body a name may be the `with` object's member: `Click` + // is the button's, and `Title` could be. Only the event's method is + // linked there. + expect(edgesFrom(cg, 'TForm1::SetupWith')).toEqual(['references TForm1::ButtonClick']); + }); + }); + + it('finds the class of a generic class\'s method, whose node spells its constraints', async () => { + const code = `unit Unit1; + +interface + +type + TBox = class + public + procedure Clear; + procedure Reset; + end; + + TOther = class + public + procedure Clear; + procedure Reset; + end; + +implementation + +procedure TOther.Clear; begin end; +procedure TOther.Reset; begin end; +procedure TBox.Clear; begin end; + +procedure TBox.Reset; +begin + Clear; +end; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + const reset = cg.getNodesByKind('method').filter((n) => n.name === 'Reset'); + const targets = reset.flatMap((r) => cg.getOutgoingEdges(r.id).filter((e) => e.kind === 'calls')) + .map((e) => cg.getNode(e.target)?.qualifiedName); + expect(targets).toEqual(['TBox::Clear']); + }); + }); + + it('links a form\'s event handler named after its component, and a C library\'s function', async () => { + const code = `unit Unit1; + +interface + +type + TNotify = procedure(Sender: TObject) of object; + + TButton = class + public + OnClick: TNotify; + end; + + TForm1 = class + private + btnOK: TButton; + public + procedure btnOKClick(Sender: TObject); + procedure Setup; + function Check: Integer; + end; + +function ERR_get_error: Cardinal; cdecl; +function mach_absolute_time: UInt64; cdecl; + +implementation + +function ERR_get_error: Cardinal; cdecl; +begin + Result := 0; +end; + +function mach_absolute_time: UInt64; cdecl; +begin + Result := 0; +end; + +procedure TForm1.btnOKClick(Sender: TObject); +begin +end; + +procedure TForm1.Setup; +begin + btnOK.OnClick := btnOKClick; +end; + +function TForm1.Check: Integer; +begin + Result := ERR_get_error + mach_absolute_time; +end; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + expect(edgesFrom(cg, 'TForm1::Setup')).toEqual(['references TForm1::btnOKClick']); + expect(edgesFrom(cg, 'TForm1::Check')).toEqual(['calls ERR_get_error', 'calls mach_absolute_time']); + }); + }); + + it('reads an ancestor through a class the unit declares ahead (`TBase = class;`)', async () => { + const code = `unit Unit1; + +interface + +type + TChild = class; + + TBase = class + public + function Ready: Boolean; + end; + + TChild = class(TBase) + public + procedure Run; + end; + +implementation + +function TBase.Ready: Boolean; +begin + Result := True; +end; + +procedure TChild.Run; +begin + if Ready then + Exit; +end; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + expect(edgesFrom(cg, 'TChild::Run')).toEqual(['calls TBase::Ready']); + }); + }); + + it('links the calls of a unit\'s initialization and finalization sections', async () => { + const code = `unit Unit1; + +interface + +procedure Setup; +procedure Teardown; + +implementation + +procedure Setup; +begin +end; + +procedure Teardown; +begin +end; + +initialization + Setup; + +finalization + Teardown; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + const file = cg.getNodesByKind('file').find((n) => n.filePath.endsWith('Unit1.pas'))!; + const targets = cg.getOutgoingEdges(file.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)?.name).sort(); + expect(targets).toEqual(['Setup', 'Teardown']); + }); + }); + + it('takes a name the class\'s code assigns or passes to Inc for a variable, as Delphi does', async () => { + // `Run` is a field of a base class the project doesn't have (a VCL + // highlighter's, say): the unit's `Run` function is not what it reads. + const helpers = `unit Helpers; + +interface + +function Run: Integer; +function Limit: Integer; + +implementation + +function Run: Integer; begin Result := 0; end; +function Limit: Integer; begin Result := 0; end; + +end. +`; + const main = `unit Main; + +interface + +uses Helpers; + +type + TMyHighlighter = class(TSynCustomHighlighter) + public + procedure Next; + procedure Reset; + function Count: Integer; + end; + +implementation + +procedure TMyHighlighter.Next; +var + C: Char; +begin + Inc(Run); + C := FLine[Run]; + if Limit > 0 then + Exit; +end; + +procedure TMyHighlighter.Reset; +begin + Limit := 0; +end; + +function TMyHighlighter.Count: Integer; +begin + Count := Run; +end; + +end. +`; + await withIndex({ 'Helpers.pas': helpers, 'Main.pas': main }, (cg) => { + expect(edgesFrom(cg, 'TMyHighlighter::Next')).toEqual([]); + expect(edgesFrom(cg, 'TMyHighlighter::Count')).toEqual([]); + }); + }); + + it('leaves out names a routine declares itself, even where a method shares the name', async () => { + const code = `unit Unit1; + +interface + +type + TMin = class + public + function Value: Integer; + function Items: Integer; + procedure Run(Value: Integer); + end; + +implementation + +function TMin.Value: Integer; begin Result := 1; end; +function TMin.Items: Integer; begin Result := 1; end; + +procedure TMin.Run(Value: Integer); +var + X: Integer; +begin + X := Value; + for var Items in [1, 2] do + X := X + Items; +end; + +end. +`; + await withIndex({ 'Unit1.pas': code }, (cg) => { + expect(edgesFrom(cg, 'TMin::Run')).toEqual([]); + }); + }); +}); diff --git a/src/db/queries.ts b/src/db/queries.ts index be8a403c47..58904634b4 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -4023,6 +4023,15 @@ export class QueryBuilder { return rows.map((r) => r.name); } + /** The names of a language's nodes of the given kinds (repeats included), decoding nothing else. */ + getNodeNamesOfKinds(language: Language, kinds: readonly NodeKind[]): string[] { + // `+language`: the kind index narrows further than the language one. + const rows = this.db + .prepare(`SELECT name FROM nodes WHERE kind IN (${kinds.map(() => '?').join(', ')}) AND +language = ?`) + .all(...kinds, language) as Array<{ name: string }>; + return rows.map((r) => r.name); + } + /** * The import node of every C / C++ `#include`, narrowed to the path it * spells and where it is written: the include graph reads nothing else, and diff --git a/src/extraction/function-ref.ts b/src/extraction/function-ref.ts index 798b158d53..6ef3befb50 100644 --- a/src/extraction/function-ref.ts +++ b/src/extraction/function-ref.ts @@ -355,9 +355,13 @@ const PASCAL_SPEC: FnRefSpec = { idTypes: new Set(['identifier']), dispatch: new Map([ ['exprArgs', { mode: 'args' }], - ['assignment', { mode: 'rhs', field: 'rhs' }], // OnClick := Handler + ['assignment', { mode: 'rhs', field: 'rhs' }], // OnClick := @Handler ]), unwrap: new Map([['exprUnary', 'operand']]), // @Handler + // A bare name (`OnClick := Handler`, `Register(Handler)`) is a paren-less + // name the extractor records itself, linked by scope (#2514); only the + // explicit `@Handler` is a function value here. + addressOfOnly: true, }; /** diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index e6d627ac3e..b440947d43 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -649,6 +649,32 @@ function keepsBareTsJsReceiver(node: SyntaxNode, source: string): boolean { */ const REACT_HANDLER_HOOKS = /^(?:React\.)?use(?:Callback|EffectEvent|Event)$/; +/** Bare Pascal names that are never a project routine's call (#2514), lowercased. */ +const PASCAL_NOT_ROUTINES: ReadonlySet = new Set(['self', 'result', 'true', 'false', 'nil', 'exit', 'break', 'continue']); + +/** A routine header of the Pascal source text, for a broken tree's regions (#2514). */ +interface PascalTextRoutine { + line: number; + classQn: string | null; + routine: string | null; + locals: Map; +} + +/** + * How a Pascal name written without parentheses may run a routine (#2514): + * as a statement (`call`), for its value (`read`), as an event's method + * (`ref`), or as a receiver or an indexed value (`recv`). See + * resolution/pascal-scope. + */ +type PascalNameMode = 'call' | 'read' | 'ref' | 'recv'; + +/** + * Names Delphi's conventions give to data, never to a routine: a field + * (`FCount`), a type (`TList`, `EAbort`), a constant (`MAX_PATH`, `CRLF`). + * A C library's functions keep their own case (`ERR_get_error`). + */ +const PASCAL_DATA_NAME = /^(?:F[A-Z][a-z]|[TE][A-Z][a-z]|[A-Z][A-Z0-9_]*$)/; + export class TreeSitterExtractor { /** * The node's prose, from either place it can live: a preceding comment @@ -728,6 +754,27 @@ export class TreeSitterExtractor { // the precision guard (the #932 table-dispatch philosophy scoped to locals). private cppLocalFnPtrs = new Map>>(); private methodIndex: Map | null = null; // lookup key → node ID for Pascal defProc lookup + // Paren-less Pascal names (#2514): the names each routine or anonymous + // method being walked declares itself (innermost last), the class the + // routine belongs to, how many `with` bodies the walk is inside (a bare name + // there may be the `with` object's), and per class the names its own data + // and the unit's data take (lowercased). + private pascalScopes: Array<{ + locals: Map; + classQn: string | null; + routine: string | null; + /** The line of the routine's header (0: none), for a broken tree's regions. */ + startLine: number; + }> = []; + /** The routine headers of the source text, by line, for a broken tree's regions (#2514). */ + private pascalTextRoutines: PascalTextRoutine[] | null = null; + private pascalSourceLines: string[] | null = null; + private pascalWithDepth = 0; + private pascalDataNameSets = new Map>(); + /** Bare names read in an expression, kept until the file shows which are variables (#2514). */ + private pascalReads: Array<{ ref: UnresolvedReference; scope: string; name: string }> = []; + /** Per routine class (`''`: none), the names the file assigns to or passes to `Inc`/`Dec`. */ + private pascalAssigned = new Map>(); // Function-as-value capture (#756): per-language spec + candidates collected // during the walk, gated & flushed into unresolvedReferences at end-of-file // (see flushFnRefCandidates). @@ -870,6 +917,7 @@ export class TreeSitterExtractor { // nodes and import refs are complete and the file node is still pushed. this.flushFnRefCandidates(); this.flushValueRefs(); + this.flushPascalReads(); if (packageNodeId) this.nodeStack.pop(); this.nodeStack.pop(); @@ -8331,9 +8379,19 @@ export class TreeSitterExtractor { return true; } - // block (begin..end) → visit for calls - if (nodeType === 'block') { - this.visitPascalBlock(node); + // block (begin..end) → visit for calls. Outside a routine — a program's + // main block, `initialization` and `finalization` — the names in it are + // the unit's (#2514). Where the tree has errors, a block outside a routine + // (a `blockTr` there) may be the body of one whose header didn't parse; + // the source text says whose it is. + if (nodeType === 'block' || nodeType === 'blockTr' || nodeType === 'initialization' || nodeType === 'finalization') { + const outside = this.pascalScopes.length === 0; + if (outside) { + this.pascalScopes.push({ locals: new Map([['result', null]]), classQn: null, routine: null, startLine: 0 }); + } + // A `blockTr` may also hold declarations, which are walked as they always were. + this.visitPascalBlock(node, nodeType === 'blockTr'); + if (outside) this.pascalScopes.pop(); return true; } @@ -8553,12 +8611,107 @@ export class TreeSitterExtractor { (c: SyntaxNode) => c.type === 'block' ); if (block) { + // `TList.Add` belongs to `TList`, whose node may spell its parameters + // with their constraints (`TList`). + const owner = fullName.includes('.') ? fullName.replace(/<[^>]*>/g, '').split('.').slice(0, -1).join('::') : null; + this.pascalScopes.push({ + locals: this.pascalRoutineLocals(node), + classQn: owner, + routine: shortNameKey.replace(/<.*$/, ''), + startLine: node.startPosition.row + 1, + }); this.nodeStack.push(parentId); this.visitPascalBlock(block); this.nodeStack.pop(); + this.pascalScopes.pop(); } } + /** + * The names a routine (a `defProc`) or an anonymous method (a `lambda`) + * declares before its body, lowercased, with the class or record type each + * is declared with (null when it has none, or none a name can stand for): + * parameters, `var`/`const` sections, nested routines and `Result`. The + * body's inline declarations join as the walk reaches them + * (`declarePascalInline`). A bare name among them is the routine's own, + * never a call. + */ + private pascalRoutineLocals(routine: SyntaxNode): Map { + const names = new Map(); + const header = routine.type === 'lambda' ? routine : routine.namedChildren.find((c: SyntaxNode) => c.type === 'declProc'); + names.set('result', this.pascalTypeName(header?.namedChildren.find((c: SyntaxNode) => c.type === 'typeref'))); + const declare = (decl: SyntaxNode): void => { + const type = this.pascalTypeName(decl.namedChildren.find((c: SyntaxNode) => c.type === 'type' || c.type === 'typeref')); + for (const c of decl.namedChildren) { + if (c.type === 'identifier') names.set(getNodeText(c, this.source).toLowerCase(), type); + } + }; + const args = header?.namedChildren.find((c: SyntaxNode) => c.type === 'declArgs'); + for (const arg of args?.namedChildren ?? []) { + if (arg.type === 'declArg') declare(arg); + } + for (const c of routine.namedChildren) { + if (c.type === 'declVars' || c.type === 'declConsts') { + for (const decl of c.namedChildren) declare(decl); + } else if (c.type === 'defProc') { + const name = getChildByField(c.namedChildren.find((x: SyntaxNode) => x.type === 'declProc') ?? c, 'name'); + if (name?.type === 'identifier') names.set(getNodeText(name, this.source).toLowerCase(), null); + } + } + return names; + } + + /** + * A name a routine's body declares where it stands — an inline `var`, a + * `for` or `for … in` loop's variable, an `on E: EFoo do` handler's — joins + * the routine's locals before the code that uses it is walked. + */ + private declarePascalInline(node: SyntaxNode): void { + const locals = this.pascalScopes[this.pascalScopes.length - 1]?.locals; + if (!locals) return; + const declare = (id: SyntaxNode | null | undefined, type: string | null): void => { + if (id?.type !== 'identifier') return; + const name = getNodeText(id, this.source).toLowerCase(); + // A loop over a variable the `var` section declares keeps its type. + if (type !== null || !locals.has(name)) locals.set(name, type); + }; + switch (node.type) { + case 'varAssignDef': + case 'varDef': { + let type = this.pascalTypeName(node.namedChildren.find((c: SyntaxNode) => c.type === 'typeref' || c.type === 'type')); + const value = node.parent?.type === 'assignment' ? node.parent.namedChild(node.parent.namedChildCount - 1) : null; + if (!type && value?.type === 'exprDot') { + // `var List := TStringList.Create;` + const [cls, ctor] = [value.namedChild(0), value.namedChild(value.namedChildCount - 1)]; + if (cls?.type === 'identifier' && ctor?.type === 'identifier' && /^create$/i.test(getNodeText(ctor, this.source))) { + type = getNodeText(cls, this.source); + } + } + declare(node.namedChildren.find((c: SyntaxNode) => c.type === 'identifier'), type); + return; + } + case 'foreach': + declare(getChildByField(node, 'iterator'), null); + return; + case 'for': + declare(node.namedChildren.find((c: SyntaxNode) => c.type === 'assignment')?.namedChild(0), null); + return; + case 'exceptionHandler': + declare(getChildByField(node, 'variable'), this.pascalTypeName(node.namedChildren.find((c: SyntaxNode) => c.type === 'typeref'))); + return; + } + } + + /** The class or record name a declared type (`type`/`typeref`) stands for: `TFoo`, `Sys.TFoo` → `TFoo`, `TList` → `TList`. */ + private pascalTypeName(type: SyntaxNode | null | undefined): string | null { + let ref = type?.type === 'type' ? type.namedChild(0) : type; + if (ref?.type !== 'typeref') return null; + ref = ref.namedChild(0); + if (ref?.type === 'typerefDot') ref = ref.namedChild(ref.namedChildCount - 1); + else if (ref?.type === 'typerefTpl') ref = ref.namedChild(0); + return ref?.type === 'identifier' ? getNodeText(ref, this.source) : null; + } + /** * Extract function calls from a Pascal expression. A `receiverLink` call is * a link inside another call's receiver chain (#2547): it is recorded only @@ -8649,24 +8802,31 @@ export class TreeSitterExtractor { * `TMeta.ExtendedPath(APath)` in its arguments for * `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)`, `A(1)` in * `A(1).B.C`, `GetIndex(x)` in `List[GetIndex(x)].Free`. A paren'd call - * there is a call like any other. A paren-less link (`.B`) may read a field - * or property, so it is not one. A typecast (`TButton(Sender)`) is written - * like a call; resolution tells it apart by what the name resolves to. + * there is a call like any other. A paren-less link (`B` in `A.B.C`, `GetList` + * in `GetList.Count`) is read for its value, which a function gives + * (#2514): it is recorded as a paren-less name. A typecast + * (`TButton(Sender)`) is written like a call; resolution tells it apart by + * what the name resolves to. */ - private extractPascalReceiverCalls(node: SyntaxNode | null): void { + private extractPascalReceiverCalls(node: SyntaxNode | null, names = true): void { if (!node) return; switch (node.type) { case 'identifier': + if (names) this.extractPascalParenlessName(node, null); return; case 'exprCall': this.extractPascalReceiverCall(node, true); return; case 'exprDot': + if (names) this.extractPascalParenlessName(node, null); + this.extractPascalReceiverCalls(node.namedChild(0), names); + return; case 'exprTpl': - this.extractPascalReceiverCalls(node.namedChild(0)); + // `TList.Create`: a generic type's name, not a value's. + this.extractPascalReceiverCalls(node.namedChild(0), false); return; case 'exprSubscript': { - this.extractPascalReceiverCalls(node.namedChild(0)); + this.extractPascalReceiverCalls(node.namedChild(0), names); const index = node.namedChildren.find((c: SyntaxNode) => c.type === 'exprArgs'); if (index) this.visitPascalBlock(index); return; @@ -8749,16 +8909,23 @@ export class TreeSitterExtractor { } /** - * Recursively visit a Pascal block/statement tree for call expressions + * Recursively visit a Pascal block/statement tree for call expressions. + * With `declarations`, a declaration in it is visited as a declaration. */ - private visitPascalBlock(node: SyntaxNode): void { + private visitPascalBlock(node: SyntaxNode, declarations = false): void { for (let i = 0; i < node.namedChildCount; i++) { const child = node.namedChild(i); if (!child) continue; + if (declarations && /^(?:decl|defProc$|interface$|implementation$)/.test(child.type)) { + this.visitNode(child); + continue; + } // Function-as-value capture (#756): Pascal bodies are walked here, not // in visitNode/visitForCallsAndStructure, so the capture hook fires here // — assignment RHS is the Delphi event-wiring idiom (`OnFire := Handler`). this.maybeCaptureFnRefs(child, child.type); + this.declarePascalInline(child); + this.learnPascalVariable(child); if (child.type === 'exprCall') { this.extractPascalCall(child); // The walker doesn't descend into a call's arguments — dispatch the @@ -8773,16 +8940,342 @@ export class TreeSitterExtractor { // (`X := A(1).B.C` reads `A(1)`); a call right before the read is // named as any call is. if (node.type === 'statement') { - this.extractPascalParenlessCall(child); + // Where the tree has errors, `procedure TFoo.Bar;` may parse as + // one: a routine's header calls nothing (#2514). + if (!this.isPascalHeaderName(child)) { + this.extractPascalParenlessCall(child); + } } else { const receiver = child.namedChild(0); if (receiver?.type === 'exprCall') this.extractPascalReceiverCall(receiver, false); else this.extractPascalReceiverCalls(receiver); - } + this.extractPascalParenlessName(child, node); + } + } else if (child.type === 'identifier') { + this.extractPascalParenlessName(child, node); + } else if (child.type === 'with') { + this.pascalWithDepth++; + this.visitPascalBlock(child, declarations); + this.pascalWithDepth--; + } else if (child.type === 'lambda') { + const outer = this.pascalScopes[this.pascalScopes.length - 1]; + this.pascalScopes.push({ + locals: this.pascalRoutineLocals(child), + classQn: outer?.classQn ?? null, + routine: null, + startLine: outer?.startLine ?? 0, + }); + this.visitPascalBlock(child, declarations); + this.pascalScopes.pop(); + } else { + // A `begin … end` block in it holds statements only, as it always has. + this.visitPascalBlock(child, declarations && child.type !== 'block'); + } + } + } + + /** + * A name written without parentheses where a parameterless routine may be + * called (#2514): `Before;`, `Result := FProvider.Value + 1`, `if Ready + * then`, `OnGet := Handler`, `GetList.Count`. Recorded as a `references` + * ref named `{mode}.Name` or `{mode}.Receiver.Name`, with the routine's + * class after an `@` (see resolution/pascal-scope), which links only a + * routine the name reaches, or nothing. A null `parent` marks a receiver's + * link. Names the routine declares itself and names the class's or unit's + * own data takes are left out here; a receiver the routine declares is + * written as its declared type (`{read}.:TProvider.Value`), or left out + * when it has none. + */ + private extractPascalParenlessName(name: SyntaxNode, parent: SyntaxNode | null): void { + const scope = this.pascalScopeAt(name.startPosition.row + 1); + if (!scope) return; + // Where the tree has errors, a routine's header may parse as statements + // (`procedure Foo;`): the name it declares is no call. + if (this.isPascalHeaderName(name)) return; + const callerId = this.nodeStack[this.nodeStack.length - 1]; + if (!callerId) return; + // Most bare names are the routine's own or the class's data: settled + // before what the name's place in the tree says. + if (name.type === 'identifier') { + const lower = getNodeText(name, this.source).toLowerCase(); + if (PASCAL_NOT_ROUTINES.has(lower) || this.pascalLocalType(lower, scope.locals) !== undefined) return; + if (this.pascalDataNames(scope.classQn).has(lower)) return; + } + const mode: PascalNameMode | null = parent ? this.pascalNameMode(name, parent) : 'recv'; + if (!mode) return; + // In a `with` body a name may be the `with` object's member, which the + // object's type would have to say. Only the method an event is given is + // linked there (`OnClick := ButtonClick`), the form's, as it is written. + if (this.pascalWithDepth > 0 && (mode !== 'ref' || name.type !== 'identifier')) return; + let path: string[]; + if (name.type === 'identifier') { + path = [getNodeText(name, this.source)]; + } else { + const links = this.pascalIdentifierPath(name); + if (!links) return; + path = links; + } + const head = path[0]!; + const lower = head.toLowerCase(); + const local = this.pascalLocalType(lower, scope.locals); + // Read for its value, a name Delphi's conventions give to data is never a + // routine's: lowercase words are locals, fields and enum values (`clRed`), + // where a C library's functions aren't (`mach_absolute_time`), and see + // PASCAL_DATA_NAME. A statement on its own runs a routine, and an event is + // given one, whatever its name (`btnOKClick`). + const read = mode !== 'call' && mode !== 'ref'; + if (read && PASCAL_DATA_NAME.test(path[path.length - 1]!)) return; + if (path.length === 1) { + if (PASCAL_NOT_ROUTINES.has(lower) || local !== undefined) return; + if (read && /^[a-z][A-Za-z]*$/.test(head)) return; + if (this.pascalDataNames(scope.classQn).has(lower)) return; + } else if (local !== undefined) { + // A local's declared class stands for it; a pointer type (`PNode`), + // which no class node declares, can't. A local's member as a receiver + // is left out: what it reaches is almost always data. + if (local === null || mode === 'recv' || /^P[A-Z]/.test(local)) return; + path[0] = `:${local}`; + } else if (lower !== 'self' && PASCAL_NOT_ROUTINES.has(lower)) { + return; + } + // The routine's class rides along: the reference's own node may be the + // file when the implementation matched no declaration. + const ref: UnresolvedReference = { + fromNodeId: callerId, + referenceName: `{${mode}${scope.classQn ? `@${scope.classQn}` : ''}}.${path.join('.')}`, + referenceKind: 'references', + line: name.startPosition.row + 1, + column: name.startPosition.column, + }; + if (path.length === 1 && mode !== 'call') this.pascalReads.push({ ref, scope: scope.classQn ?? '', name: lower }); + else this.unresolvedReferences.push(ref); + } + + /** + * A bare name the code assigns to (`Run := 0`) or passes to `Inc`/`Dec` is + * a variable, a field or a property wherever the same class's code reads it + * (#2514), unless the routine declares it, or it is the function's own name + * (its result, in the old style), or a `with` object may hold it. + */ + private learnPascalVariable(node: SyntaxNode): void { + if (this.pascalWithDepth > 0) return; + let target: SyntaxNode | null = null; + if (node.type === 'assignment') { + target = node.namedChild(0); + } else if (node.type === 'exprCall') { + const callee = node.namedChild(0); + if (callee?.type === 'identifier' && /^(?:inc|dec)$/i.test(getNodeText(callee, this.source))) { + target = node.namedChildren.find((c: SyntaxNode) => c.type === 'exprArgs')?.namedChild(0) ?? null; + } + } + if (target?.type !== 'identifier') return; + const scope = this.pascalScopeAt(target.startPosition.row + 1); + if (!scope) return; + const name = getNodeText(target, this.source).toLowerCase(); + if (name === scope.routine || this.pascalLocalType(name, scope.locals) !== undefined) return; + const key = scope.classQn ?? ''; + let names = this.pascalAssigned.get(key); + if (!names) this.pascalAssigned.set(key, (names = new Set())); + names.add(name); + } + + /** The bare names read in an expression, less those the file shows are variables of their class. */ + private flushPascalReads(): void { + for (const { ref, scope, name } of this.pascalReads) { + if (!this.pascalAssigned.get(scope)?.has(name)) this.unresolvedReferences.push(ref); + } + this.pascalReads = []; + } + + /** + * Whether a node is the name a routine's header declares (`TFoo.Bar` in + * `procedure TFoo.Bar;`), which a tree with errors may parse as a statement. + */ + private isPascalHeaderName(node: SyntaxNode): boolean { + if (!this.tree?.rootNode.hasError) return false; + const line = (this.pascalSourceLines ??= this.source.split(/\r?\n/))[node.startPosition.row] ?? ''; + return /^\s*(?:class\s+)?(?:procedure|function|constructor|destructor|operator)\s+$/i.test(line.slice(0, node.startPosition.column)); + } + + /** + * The declared type of a name the enclosing routines declare (null: none it + * can be named by), or undefined when none does. `textLocals` are the + * declarations the source text gives a broken tree's routine. + */ + private pascalLocalType(lower: string, textLocals?: Map | null): string | null | undefined { + for (let i = this.pascalScopes.length - 1; i >= 0; i--) { + const locals = this.pascalScopes[i]!.locals; + if (locals.has(lower)) return locals.get(lower); + } + return textLocals?.has(lower) ? textLocals.get(lower) : undefined; + } + + /** + * The class and routine a name at `line` belongs to; null outside any code. + * Where the tree has errors, a routine whose header didn't parse is a block + * outside any routine, or part of the routine above it, so the source text + * says: a header between the tree's routine and the name is the name's, + * with the declarations the text gives it. + */ + private pascalScopeAt(line: number): { classQn: string | null; routine: string | null; locals: Map | null } | null { + const scope = this.pascalScopes[this.pascalScopes.length - 1]; + if (!scope) return null; + const own = { classQn: scope.classQn, routine: scope.routine, locals: null }; + if (!this.tree?.rootNode.hasError) return own; + const routines = (this.pascalTextRoutines ??= this.readPascalTextRoutines()); + let lo = 0; + let hi = routines.length - 1; + let hit = -1; + while (lo <= hi) { + const mid = (lo + hi) >> 1; + if (routines[mid]!.line <= line) { + hit = mid; + lo = mid + 1; } else { - this.visitPascalBlock(child); + hi = mid - 1; + } + } + const text = hit >= 0 ? routines[hit]! : null; + // Inside a routine the tree parsed, only a later header overrides it. + if (!text || text.line <= scope.startLine || (scope.startLine > 0 && text.routine === null)) return own; + return text; + } + + /** + * Each routine header of the source text (`procedure TFoo.Bar(…)`), with + * its class, its name and what it declares before its body — parameters, + * `Result`'s type, `var` and `const` sections — and where the unit's own + * code resumes: after an `end;` that starts its line, which closes a + * routine, and at `initialization` and `finalization`. An indented header + * is a routine nested in the one above, whose class and names it shares. + */ + private readPascalTextRoutines(): PascalTextRoutine[] { + const lines = (this.pascalSourceLines ??= this.source.split(/\r?\n/)); + const header = /^(\s*)(?:class\s+)?(?:procedure|function|constructor|destructor|operator)\s+([A-Za-z_]\w*(?:<[^>]*>)?(?:\.[A-Za-z_]\w*(?:<[^>]*>)?)*)/i; + const routines: PascalTextRoutine[] = []; + let outer: PascalTextRoutine | null = null; + for (let i = 0; i < lines.length; i++) { + const text = lines[i]!; + if (/^(?:end;|\s*(?:initialization|finalization)\b)/i.test(text)) { + routines.push({ line: i + 1, classQn: null, routine: null, locals: new Map() }); + outer = null; + continue; + } + const m = header.exec(text); + if (!m) continue; + const nested: boolean = m[1] !== '' && outer !== null; + const parts = m[2]!.replace(/<[^>]*>/g, '').split('.'); + const routine = parts.pop()!.toLowerCase(); + let decl = text.slice(m.index + m[0].length); + for (let j = i + 1; j < lines.length && j <= i + 200 && !/^\s*(?:begin|asm)\b/i.test(lines[j]!) && !header.test(lines[j]!); j++) { + decl += '\n' + lines[j]; + } + decl = decl.replace(/\{[^}]*\}|\(\*[\s\S]*?\*\)|\/\/[^\n]*/g, ' '); + const locals: Map = new Map(nested ? outer!.locals : []); + const result = /^\s*(?:\([^)]*\))?\s*:\s*([A-Za-z_][\w.]*)/.exec(decl); + locals.set('result', result ? result[1]!.split('.').pop()! : null); + for (const d of decl.matchAll(/([A-Za-z_]\w*(?:\s*,\s*[A-Za-z_]\w*)*)\s*:(?!=)\s*([A-Za-z_][\w.]*)?/g)) { + const type = d[2] ? d[2].split('.').pop()! : null; + for (const name of d[1]!.split(',')) locals.set(name.trim().toLowerCase(), type); + } + if (nested) { + // The outer routine's body follows its nested ones: it keeps the + // outer routine's class and name, and sees the names of both. + routines.push({ line: i + 1, classQn: outer!.classQn, routine: outer!.routine, locals }); + } else { + outer = { line: i + 1, classQn: parts.length > 0 ? parts.join('::') : null, routine, locals }; + routines.push(outer); + } + } + return routines; + } + + /** + * How a name in `parent` may run a parameterless routine: `call` as a + * statement of its own, `read` for its value in an expression or an + * argument, `ref` as the method an event is given, `recv` as what is + * indexed; null where it can't (an assignment's target, a case label, a + * set literal, a type). + */ + private pascalNameMode(name: SyntaxNode, parent: SyntaxNode): PascalNameMode | null { + const is = (field: string): boolean => parent.childForFieldName(field)?.id === name.id; + switch (parent.type) { + case 'statement': + return parent.namedChildCount === 1 && name.type === 'identifier' ? 'call' : null; + case 'exprSubscript': + return parent.namedChild(0)?.id === name.id ? 'recv' : null; + case 'assignment': { + const target = parent.namedChild(0); + if (!target || target.id === name.id) return null; + const last = target.type === 'exprDot' + ? target.namedChildren.filter((c: SyntaxNode) => c.type === 'identifier').pop() + : target; + return last && /^F?On[A-Z]/.test(getNodeText(last, this.source)) ? 'ref' : 'read'; + } + case 'exprArgs': + return parent.parent?.type === 'exprCall' || parent.parent?.type === 'exprSubscript' || parent.parent?.type === 'inherited' ? 'read' : null; + case 'exprBinary': { + const op = parent.namedChildren.find((c: SyntaxNode) => c.id !== name.id && /^k[A-Z]/.test(c.type)); + return op && (op.type === 'kIs' || op.type === 'kAs') && parent.namedChild(0)?.id !== name.id ? null : 'read'; + } + case 'exprUnary': + return parent.child(0)?.type === 'kAt' ? null : 'read'; + case 'if': + case 'ifElse': + case 'while': + case 'repeat': + return is('condition') ? 'read' : null; + case 'case': + return 'read'; + case 'foreach': + return is('iterable') ? 'read' : null; + case 'for': + return is('end') ? 'read' : null; + case 'exprParens': + return 'read'; + default: + return null; + } + } + + /** The identifiers of a plain dotted name (`FProvider.Value`, `Self.A.B`), or null. */ + private pascalIdentifierPath(dot: SyntaxNode): string[] | null { + const links: string[] = []; + let n: SyntaxNode | null = dot; + while (n?.type === 'exprDot') { + const last = n.namedChild(n.namedChildCount - 1); + if (last?.type !== 'identifier') return null; + links.unshift(getNodeText(last, this.source)); + n = n.namedChild(0); + } + if (n?.type !== 'identifier') return null; + links.unshift(getNodeText(n, this.source)); + return links; + } + + /** + * The names data takes for a routine of class `classQn`, lowercased: the + * class's fields, properties and constants, and the unit's variables and + * constants. A bare name among them reads that data. + */ + private pascalDataNames(classQn: string | null): Set { + const key = classQn ?? ''; + let names = this.pascalDataNameSets.get(key); + if (names) return names; + names = new Set(); + const prefix = classQn ? `${classQn.toLowerCase()}::` : null; + for (const n of this.nodes) { + if (n.kind === 'enum_member') { + names.add(n.name.toLowerCase()); // enum values are visible unqualified + continue; } + if (n.kind !== 'field' && n.kind !== 'property' && n.kind !== 'constant' && n.kind !== 'variable') continue; + const qn = n.qualifiedName.replace(/<[^>]*>/g, '').toLowerCase(); + const own = prefix !== null && qn.startsWith(prefix) && qn.indexOf('::', prefix.length) < 0; + if (own || !qn.includes('::')) names.add(n.name.toLowerCase()); } + this.pascalDataNameSets.set(key, names); + return names; } } diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 5f2e3494ad..06d435e490 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -28,6 +28,7 @@ import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './ import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; import { clearDartLibraryMemos } from './dart-libraries'; import { clearVbnetReceiverMemos, isVbMemberRead, isVbPathCall, matchVbMemberRead, matchVbPathCall } from './vbnet-receivers'; +import { clearPascalScopeMemos, isPascalParenlessRef, matchPascalParenlessRef } from './pascal-scope'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { gateDartLocal, clearDartLocalScopeMemos } from './dart-local-scope'; import { clearCppTypeAliasMemos } from './cpp-type-aliases'; @@ -587,6 +588,7 @@ export class ReferenceResolver { clearSwiftTypeVisibility(this.context); clearDartLibraryMemos(this.context); clearVbnetReceiverMemos(this.context); + clearPascalScopeMemos(this.context); clearTypeParameterMemos(this.context); clearDartLocalScopeMemos(this.context); clearCppTypeAliasMemos(this.context); @@ -909,6 +911,12 @@ export class ReferenceResolver { return result; }, + getNodeNamesOfKinds: (language, kinds) => this.queries.getNodeNamesOfKinds(language, kinds), + + rememberNode: (node: Node) => { + if (!this.nodeByIdCache.has(node.id)) this.nodeByIdCache.set(node.id, node); + }, + getNodeById: (id: string) => { return this.nodeById(id); }, @@ -1428,6 +1436,10 @@ export class ReferenceResolver { // ./cpp-supertypes). if (isCppSupertypeRef(ref)) return matchCppSupertype(ref, this.context); + // A Pascal name written without parentheses (#2514) links only to a + // routine its scope reaches, or through a receiver to a method of the + // receiver's declared type; data of that name links nothing. + if (isPascalParenlessRef(ref)) return matchPascalParenlessRef(ref, this.context); // A Dart member read (`x.area`) links the getter the receiver's type // reaches, as a call, or nothing — never a guess by name (#2338). if (isDartMemberRead(ref)) return matchDartMemberRead(ref, this.context); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index a396c42087..d300409f2e 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -18233,7 +18233,7 @@ function headTypeArguments(cls: Node, sup: string, context: ResolutionContext): const DECLARED_SUPERS = new WeakMap>(); const INHERITED_METHODS = new WeakMap>(); -function declaredSupertypes(cls: Node, context: ResolutionContext): string[] { +export function declaredSupertypes(cls: Node, context: ResolutionContext): string[] { let memo = DECLARED_SUPERS.get(context); if (!memo) DECLARED_SUPERS.set(context, (memo = new Map())); const hit = memo.get(cls.id); @@ -18257,7 +18257,7 @@ function readDeclaredSupertypes(cls: Node, context: ResolutionContext): string[] const names = (text: string | undefined): string[] => text ? [...text.matchAll(/([A-Za-z_][\w.:\\]*)/g)].map((m) => m[1]!.split(/::|\.|\\/).pop()!).filter((w) => !/^(?:metaclass|object)$/.test(w)) : []; switch (cls.language) { - case 'pascal': return names(/=\s*class\s*\(([^)]*)\)/i.exec(head)?.[1]); + case 'pascal': return names(/=\s*class\s*(?:sealed|abstract)?\s*\(([^)]*)\)/i.exec(head)?.[1]); // A base built by a class's factory — Django's `class Manager( // BaseManager.from_queryset(QuerySet))` — subclasses that class: the // qualifier is the supertype, not the method called on it. diff --git a/src/resolution/pascal-scope.ts b/src/resolution/pascal-scope.ts new file mode 100644 index 0000000000..ec04b11b83 --- /dev/null +++ b/src/resolution/pascal-scope.ts @@ -0,0 +1,483 @@ +/** + * Pascal names written without parentheses (#2514): a parameterless routine + * is called that way in Delphi — `Before;`, `Result := FProvider.Value + 1`, + * `if Ready then` — and the same spelling reads a field, a property, a + * variable or a constant. The extractor records such a name as a + * `references` ref named `{mode}.Name` or `{mode}.Receiver.Name`: + * + * - `{call}` — a statement on its own, which only a routine can be; + * - `{read}` — a name in an expression or an argument, where a function + * (or a constructor) is called for its value and a procedure, which has + * none, can only be a method pointer; + * - `{ref}` — the value given to an event (`OnClick := Handler`), a method + * pointer; + * - `{recv}` — a receiver (`GetList` in `GetList.Count`, `B` in `A.B.C`) or + * an indexed value, which only a function can be. + * + * Written without its arguments, a routine that needs some is a method + * pointer too, in any of them. + * + * The routine's class follows an `@` (`{call@TConsumer}.Before`), because the + * reference's own node is the file when an implementation matched no + * declaration. A receiver the routine declares itself is written as its + * declared type after a `:` (`{read}.:TProvider.Value`). + * + * This module links such a ref only to a method or function the name reaches + * by Delphi's scope rules — the routine's class and its ancestors, then the + * unit, then the units it uses, last first — or, through a receiver, a method + * of the receiver's declared type. The first declaration of the name in that + * order decides: when it is data (a field, property, variable, constant, enum + * value or type), nothing is linked. A bare name the routine declares itself + * never reaches here (the extractor skips it). + */ + +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; +import { declaredSupertypes } from './name-matcher'; + +type Mode = 'call' | 'read' | 'ref' | 'recv'; +interface Found { + node: Node; + resolvedBy: ResolvedRef['resolvedBy']; + confidence: number; +} + +const PARENLESS = /^\{(call|read|ref|recv)(?:@([^}]+))?\}\.(.+)$/; +const CLASS_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface']); +/** What a unit declares by name at its top level. */ +const UNIT_KINDS: ReadonlySet = new Set([ + 'function', 'variable', 'constant', 'class', 'struct', 'interface', 'enum', 'type_alias', 'enum_member', +]); +/** What a class declares as its members. */ +const MEMBER_KINDS: ReadonlySet = new Set(['method', 'field', 'property', 'constant', 'variable']); +const MAX_HIERARCHY_DEPTH = 8; + +/** Whether `ref` is a paren-less Pascal name (see the file comment), which only this module resolves. */ +export function isPascalParenlessRef(ref: UnresolvedRef): boolean { + return ref.language === 'pascal' && ref.referenceKind === 'references' && PARENLESS.test(ref.referenceName); +} + +/** The routine a paren-less Pascal name reaches in its scope, or null (see the file comment). */ +export function matchPascalParenlessRef(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const m = PARENLESS.exec(ref.referenceName); + if (!m) return null; + const mode = m[1] as Mode; + const path = m[3]!.split('.'); + // Most such names read data, often the framework's: a name no Pascal + // routine has is settled here, before any lookup in scope. + if (routineNames(context)?.has(path[path.length - 1]!.toLowerCase()) === false) return null; + let cls: Node | null = null; + if (m[2]) { + cls = classByQualifiedName(m[2], ref.filePath!, context); + } else { + const routine = context.getNodeById?.(ref.fromNodeId); + if (routine) cls = classOfRoutine(routine, context); + } + const found = path.length === 1 ? nameInScope(path[0]!, cls, ref, context) : memberThroughPath(path, cls, ref, context); + if (!found) return null; + // Written without arguments, a routine that needs some can only be a + // method pointer — save a constructor, whose name then means an inherited + // one outside the project (`TObject.Create`) — and so is a procedure read + // for a value, and an event's method; any other runs. + const needsArgs = needsArguments(found.node); + if (needsArgs && routineKeyword(found.node, context) === 'constructor') return null; + const pointer = mode === 'ref' || needsArgs || (mode !== 'call' && !yieldsValue(found.node, context)); + // A statement runs its routine, and a receiver is a routine's value. + if (pointer && (mode === 'call' || mode === 'recv')) return null; + const edgeKind = pointer ? 'references' : 'calls'; + context.rememberNode?.(found.node); + return { + original: ref, + targetNodeId: found.node.id, + confidence: found.confidence, + resolvedBy: found.resolvedBy, + edgeKind, + // A method pointer is a function used as a value, as `@Handler` is (#756). + ...(edgeKind === 'references' ? { metadata: { fnRef: true } } : {}), + }; +} + +// --------------------------------------------------------------------------- +// Scope lookup +// --------------------------------------------------------------------------- + +function isRoutine(node: Node): boolean { + return node.kind === 'method' || node.kind === 'function'; +} + +/** A bare name, looked up the way Delphi looks it up from a routine of class `cls` (null: a free routine). */ +function nameInScope(name: string, cls: Node | null, ref: UnresolvedRef, context: ResolutionContext): Found | null { + const lower = name.toLowerCase(); + const memo = memos(context).inScope; + const key = `${cls?.id ?? ''}|${ref.filePath}|${lower}`; + if (memo.has(key)) return memo.get(key)!; + let found: Found | null = null; + const member = cls ? memberInHierarchy(cls, lower, context) : null; + if (member) { + if (isRoutine(member)) found = { node: member, resolvedBy: 'instance-method', confidence: 0.9 }; + } else { + const own = unitDeclaration(ref.filePath!, lower, context); + if (own) { + if (isRoutine(own)) found = { node: own, resolvedBy: 'exact-match', confidence: 0.9 }; + } else { + const used = usedUnitFiles(ref.filePath!, context); + for (let i = used.length - 1; i >= 0; i--) { + const decl = unitDeclaration(used[i]!, lower, context); + if (!decl) continue; + if (isRoutine(decl)) found = { node: decl, resolvedBy: 'import', confidence: 0.85 }; + break; + } + } + } + memo.set(key, found); + return found; +} + +/** A dotted name: the method its receiver's declared type has. */ +function memberThroughPath(path: string[], cls: Node | null, ref: UnresolvedRef, context: ResolutionContext): Found | null { + const head = headOf(path[0]!, cls, ref, context); + if (!head) return null; + let type: Node | null; + let rest = path.slice(1); + if (head.unitFile) { + // `UnitName.Routine`, `UnitName.GlobalVar.Method`. + const decl = unitDeclaration(head.unitFile, rest[0]!.toLowerCase(), context); + if (!decl) return null; + if (rest.length === 1) return isRoutine(decl) ? { node: decl, resolvedBy: 'import', confidence: 0.9 } : null; + type = typeOfMember(decl, context); + rest = rest.slice(1); + } else { + type = head.type; + } + for (const link of rest.slice(0, -1)) { + const member = type ? memberInHierarchy(type, link.toLowerCase(), context) : null; + if (!member) return null; + type = typeOfMember(member, context); + } + const member = type ? memberInHierarchy(type, rest[rest.length - 1]!.toLowerCase(), context) : null; + return member && isRoutine(member) ? { node: member, resolvedBy: 'instance-method', confidence: 0.9 } : null; +} + +/** + * What a dotted name's first link is: a value of a known class (a local's + * declared type, written `:TFoo`), a class itself, or a unit. + */ +function headOf(name: string, cls: Node | null, ref: UnresolvedRef, context: ResolutionContext): { type: Node | null; unitFile?: string } | null { + if (name.startsWith(':')) return { type: classNamed(name.slice(1), ref.filePath!, context) }; + const lower = name.toLowerCase(); + if (lower === 'self') return cls ? { type: cls } : null; + if (cls) { + const member = memberInHierarchy(cls, lower, context); + if (member) return { type: typeOfMember(member, context) }; + } + const decl = unitDeclaration(ref.filePath!, lower, context) ?? + usedUnitFiles(ref.filePath!, context).reduceRight((hit, file) => hit ?? unitDeclaration(file, lower, context), undefined); + if (decl) return { type: CLASS_KINDS.has(decl.kind) ? decl : typeOfMember(decl, context) }; + const unitFile = unitFileNamed(lower, ref.filePath!, context); + return unitFile ? { type: null, unitFile } : null; +} + +// --------------------------------------------------------------------------- +// Classes, units and declarations +// --------------------------------------------------------------------------- + +interface Memos { + routineNames?: Set | null; + named: Map; + inScope: Map; + classOfRoutine: Map; + classByQn: Map; + ancestors: Map; + usedFiles: Map; + classByName: Map; + unitsByName?: Map; + unitFiles: Map; + hierarchy: Map; + memberTypes: Map; + keywords: Map; +} +const MEMOS = new WeakMap(); + +function memos(context: ResolutionContext): Memos { + let m = MEMOS.get(context); + if (!m) { + m = { + named: new Map(), inScope: new Map(), classOfRoutine: new Map(), classByQn: new Map(), + ancestors: new Map(), usedFiles: new Map(), classByName: new Map(), unitFiles: new Map(), hierarchy: new Map(), + memberTypes: new Map(), keywords: new Map(), + }; + MEMOS.set(context, m); + } + return m; +} + +/** Every Pascal method's and function's name, lowercased, or null when the context can't list them. */ +function routineNames(context: ResolutionContext): Set | null { + const memo = memos(context); + if (memo.routineNames === undefined) { + const names = context.getNodeNamesOfKinds?.('pascal', ['method', 'function']); + memo.routineNames = names ? new Set(names.map((n) => n.toLowerCase())) : null; + } + return memo.routineNames; +} + +/** The Pascal nodes a name names, whatever their case. */ +function pascalNamed(lower: string, context: ResolutionContext): Node[] { + const memo = memos(context).named; + let nodes = memo.get(lower); + if (!nodes) { + nodes = context.getNodesByLowerName(lower).filter((n) => n.language === 'pascal'); + memo.set(lower, nodes); + } + return nodes; +} + +/** Drops this module's per-context memos (the resolver clears them between runs). */ +export function clearPascalScopeMemos(context: ResolutionContext): void { + MEMOS.delete(context); +} + +/** + * The class (or record) an implementation header names (`TOuter::TInner`, + * generic parameters dropped): the one `filePath` declares, whatever its case + * or parameters, else one the file sees. + */ +function classByQualifiedName(qualifiedName: string, filePath: string, context: ResolutionContext): Node | null { + const memo = memos(context).classByQn; + const key = `${filePath}|${qualifiedName}`; + if (memo.has(key)) return memo.get(key)!; + const exact = context.getNodesByQualifiedName(qualifiedName).filter((n) => n.language === 'pascal' && CLASS_KINDS.has(n.kind)); + const lower = qualifiedName.toLowerCase(); + const cls = declarationOf(exact.filter((n) => n.filePath === filePath)) ?? + declarationOf(context.getNodesInFile(filePath).filter((n) => + CLASS_KINDS.has(n.kind) && n.qualifiedName.replace(/<[^>]*>/g, '').toLowerCase() === lower)) ?? + declarationOf(exact) ?? + classNamed(qualifiedName.split('::').pop()!, filePath, context); + memo.set(key, cls); + return cls; +} + +/** + * The class declaration among same-named nodes, not a forward declaration + * (`TFoo = class;`, which names no members or ancestors): the longest one. + */ +function declarationOf(classes: Node[]): Node | null { + let best: Node | null = null; + for (const c of classes) if (!best || c.endLine - c.startLine > best.endLine - best.startLine) best = c; + return best; +} + +/** The class a method belongs to: its qualified name's owner, declared in the method's own file. */ +function classOfRoutine(routine: Node, context: ResolutionContext): Node | null { + if (routine.kind !== 'method') return null; + const memo = memos(context).classOfRoutine; + if (memo.has(routine.id)) return memo.get(routine.id)!; + const owner = routine.qualifiedName.slice(0, Math.max(0, routine.qualifiedName.lastIndexOf('::'))); + const candidates = owner ? context.getNodesByQualifiedName(owner).filter((n) => n.language === 'pascal' && CLASS_KINDS.has(n.kind)) : []; + const cls = declarationOf(candidates.filter((n) => n.filePath === routine.filePath)) ?? declarationOf(candidates); + memo.set(routine.id, cls); + return cls; +} + +/** A class and its ancestors the project declares, nearest first. */ +function ancestorsOf(cls: Node, context: ResolutionContext): Node[] { + const memo = memos(context).ancestors; + const hit = memo.get(cls.id); + if (hit) return hit; + const order: Node[] = []; + const seen = new Set(); + let frontier = [cls]; + for (let depth = 0; depth < MAX_HIERARCHY_DEPTH && frontier.length > 0; depth++) { + const next: Node[] = []; + for (const c of frontier) { + if (seen.has(c.id)) continue; + seen.add(c.id); + order.push(c); + for (const name of declaredSupertypes(c, context)) { + const sup = classNamed(name, c.filePath, context); + if (sup) next.push(sup); + } + } + frontier = next; + } + memo.set(cls.id, order); + return order; +} + +/** The qualified name of what declares a node (`TFoo::Bar` → `tfoo`), lowercased. */ +function ownerOf(node: Node): string { + return node.qualifiedName.slice(0, Math.max(0, node.qualifiedName.lastIndexOf('::'))).toLowerCase(); +} + +/** + * The member a name reaches in a class or, failing that, in its ancestors, + * nearest first; a parameterless overload before the others. Null when no + * class of the hierarchy declares it. + */ +function memberInHierarchy(cls: Node, lower: string, context: ResolutionContext): Node | null { + const memo = memos(context).hierarchy; + const key = `${cls.id}|${lower}`; + if (memo.has(key)) return memo.get(key)!; + let found: Node | null = null; + const candidates = pascalNamed(lower, context).filter((n) => MEMBER_KINDS.has(n.kind)); + if (candidates.length > 0) { + for (const c of ancestorsOf(cls, context)) { + const owner = c.qualifiedName.toLowerCase(); + const members = candidates.filter((n) => n.filePath === c.filePath && ownerOf(n) === owner); + if (members.length > 0) { + found = members.find((n) => !needsArguments(n)) ?? members[0]!; + break; + } + } + } + memo.set(key, found); + return found; +} + +/** A unit's top-level declaration of a name (its enum values included). */ +function unitDeclaration(filePath: string, lower: string, context: ResolutionContext): Node | undefined { + const decls = pascalNamed(lower, context).filter((n) => + n.filePath === filePath && UNIT_KINDS.has(n.kind) && (n.kind === 'enum_member' || !n.qualifiedName.includes('::'))); + // A routine declared in the interface and defined in the implementation is + // one name; of overloads, one called without arguments first. + return decls.find((n) => !isRoutine(n)) ?? decls.find((n) => !needsArguments(n)) ?? decls[0]; +} + +/** The files of the units a file uses, in the order its `uses` clauses name them. */ +function usedUnitFiles(filePath: string, context: ResolutionContext): string[] { + const memo = memos(context).usedFiles; + const hit = memo.get(filePath); + if (hit) return hit; + const files: string[] = []; + const imports = context.getNodesInFile(filePath) + .filter((n) => n.kind === 'import' && n.language === 'pascal') + .sort((a, b) => a.startLine - b.startLine || a.startColumn - b.startColumn); + for (const imp of imports) { + const file = unitFileNamed(imp.name.toLowerCase(), filePath, context); + if (file && file !== filePath && !files.includes(file)) files.push(file); + } + memo.set(filePath, files); + return files; +} + +/** The file of the unit a name names, nearest to `fromFile` when several units share it. */ +function unitFileNamed(lower: string, fromFile: string, context: ResolutionContext): string | null { + const m = memos(context); + const key = `${fromFile}|${lower}`; + if (m.unitFiles.has(key)) return m.unitFiles.get(key)!; + if (!m.unitsByName) { + m.unitsByName = new Map(); + for (const n of context.getNodesByKind('module')) { + if (n.language !== 'pascal') continue; + const name = n.name.toLowerCase(); + const files = m.unitsByName.get(name); + if (files) files.push(n.filePath); + else m.unitsByName.set(name, [n.filePath]); + } + } + const file = nearest(m.unitsByName.get(lower) ?? [], fromFile); + m.unitFiles.set(key, file); + return file; +} + +/** A class (or record or interface) a type name names, seen from `fromFile`. */ +function classNamed(typeName: string, fromFile: string, context: ResolutionContext): Node | null { + const simple = typeName.split('.').pop()!.toLowerCase(); + const memo = memos(context).classByName; + const key = `${fromFile}|${simple}`; + if (memo.has(key)) return memo.get(key)!; + const candidates = pascalNamed(simple, context).filter((n) => CLASS_KINDS.has(n.kind)); + let cls = declarationOf(candidates.filter((n) => n.filePath === fromFile)); + if (!cls && candidates.length > 0) { + // The unit used last declares the name Delphi sees; else the one unit that does. + const used = usedUnitFiles(fromFile, context); + for (let i = used.length - 1; i >= 0 && !cls; i--) cls = declarationOf(candidates.filter((n) => n.filePath === used[i])); + if (!cls && candidates.every((n) => n.filePath === candidates[0]!.filePath)) cls = declarationOf(candidates); + } + memo.set(key, cls); + return cls; +} + +/** The file whose path shares the longest directory prefix with `from`. */ +function nearest(files: string[], from: string): string | null { + if (files.length <= 1) return files[0] ?? null; + const dirs = from.split('/'); + let best: string | null = null; + let bestScore = -1; + for (const f of files) { + const parts = f.split('/'); + let score = 0; + while (score < parts.length && score < dirs.length && parts[score] === dirs[score]) score++; + if (score > bestScore) { + best = f; + bestScore = score; + } + } + return best; +} + +// --------------------------------------------------------------------------- +// Declared types +// --------------------------------------------------------------------------- + +function fileLines(filePath: string, context: ResolutionContext): string[] { + return context.getFileLines?.(filePath) ?? context.readFile(filePath)?.split(/\r?\n/) ?? []; +} + +function escapeRegExp(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +/** The class a member's value has: a field's or variable's declared type, a property's, a function's result. */ +function typeOfMember(member: Node, context: ResolutionContext): Node | null { + const memo = memos(context).memberTypes; + if (memo.has(member.id)) return memo.get(member.id)!; + let typeName: string | undefined; + if (member.kind === 'method' || member.kind === 'function') { + typeName = member.returnType ?? /:\s*([A-Za-z_][\w.]*)\s*;?\s*$/.exec(member.signature ?? '')?.[1]; + } else { + const line = fileLines(member.filePath, context)[member.startLine - 1] ?? ''; + const name = escapeRegExp(member.name); + const re = member.kind === 'property' + ? new RegExp(`\\bproperty\\s+${name}\\s*(?:\\[[^\\]]*\\])?\\s*:\\s*([A-Za-z_][\\w.]*)`, 'i') + : new RegExp(`\\b${name}\\b(?:\\s*,\\s*[A-Za-z_]\\w*)*\\s*:\\s*([A-Za-z_][\\w.]*)`, 'i'); + typeName = re.exec(line)?.[1]; + } + const type = typeName ? classNamed(typeName, member.filePath, context) : null; + memo.set(member.id, type); + return type; +} + +/** Whether a routine has a parameter without a default value, which a call must pass. */ +function needsArguments(routine: Node): boolean { + const signature = routine.signature?.trim(); + if (!signature?.startsWith('(')) return false; + let depth = 0; + let end = signature.length; + for (let i = 0; i < signature.length; i++) { + if (signature[i] === '(') depth++; + else if (signature[i] === ')' && --depth === 0) { + end = i; + break; + } + } + return signature.slice(1, end).split(';').some((param) => param.trim() !== '' && !param.includes('=')); +} + +/** Whether a routine gives a value — a function or a constructor — rather than a procedure. */ +function yieldsValue(routine: Node, context: ResolutionContext): boolean { + if (routine.returnType) return true; + const keyword = routineKeyword(routine, context); + return keyword === 'function' || keyword === 'constructor'; +} + +/** The keyword that declares a routine (`function`, `procedure`, `constructor`, …), lowercased, or null. */ +function routineKeyword(routine: Node, context: ResolutionContext): string | null { + const memo = memos(context).keywords; + const hit = memo.get(routine.id); + if (hit !== undefined) return hit; + const line = fileLines(routine.filePath, context)[routine.startLine - 1] ?? ''; + const keyword = /\b(function|procedure|constructor|destructor|operator)\s/i.exec(line)?.[1]!.toLowerCase() ?? null; + memo.set(routine.id, keyword); + return keyword; +} diff --git a/src/resolution/types.ts b/src/resolution/types.ts index f6e8610a6c..8caf285f20 100644 --- a/src/resolution/types.ts +++ b/src/resolution/types.ts @@ -146,6 +146,17 @@ export interface ResolutionContext { getAllFiles(): string[]; /** Get nodes by lowercase name (O(1) lookup for fuzzy matching) */ getNodesByLowerName(lowerName: string): Node[]; + /** + * The names of a language's nodes of the given kinds, nothing else decoded: + * a cheap pre-filter before lookups by name. Optional for external/test + * contexts. + */ + getNodeNamesOfKinds?(language: Language, kinds: readonly Node['kind'][]): string[]; + /** + * Hands the resolver a node a resolution found by other means, so the gates + * after it don't fetch it again by id. Optional for external/test contexts. + */ + rememberNode?(node: Node): void; /** * Direct supertypes of the type named `typeName` (same language): the classes * it extends and the interfaces / protocols / traits it implements/conforms to, From c1744e7e7630f2781486a722012cd5de08c9da76 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 11:44:29 +0000 Subject: [PATCH 33/95] fix(prompt-hook): desktop reminders and other sessions' messages no longer drive the hook (#2798) Claude Code's desktop app puts blocks (which local sessions are linked, and similar status) in front of the prompt it submits, and UserPromptSubmit hooks receive them inside `prompt`. The hook gated, picked the project and explored on the whole string, so the reminder's words drove it: a linked session titled after its work, like "trace-order-flow", carries structural keywords, so every turn, a bare "yes" included, got a full explore of the reminder. A reminder in front of a task notification or a subagent hand-back also defeated the start-anchored envelope checks, and a message from another session (a envelope) was never skipped at all. The hook now removes every complete block before the envelope checks, the gate, project selection and explore, returns early when nothing is left, and skips a prompt that is entirely a envelope, as it already did for task notifications and subagent hand-backs. The scan stops at the last closing tag and the attribute run stops at `<`, so a pasted prompt full of unclosed tags stays linear (520 KB in under 1 ms; the `[^>]*` form took 83 ms on a tenth of that and grows quadratically). Validation: - Prompt shapes come from the Claude Code desktop transcripts on this machine (314 transcripts): reminders always lead, one to four of them, separated by one or two newlines, and cross-session messages follow one to four reminders. - frontload-hook.test.ts: 15 new tests (8 spawn the built hook with Claude Code-shaped payloads) fail on main and pass with the fix; the file's 48 other tests pass on both. - Built CLI, private HOME, 13 payloads: on main every prompt behind a reminder got a full explore of the reminder (2.1 to 5.0 KB); with the fix the 8 that are not user input inject nothing and exit in about 200 ms instead of 680 ms, and a question behind reminders injects byte-for-byte what the same question alone does. Issue #2635. Contributor PR #2642 (yang0228) was read for guidance. Fixes #2635 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/frontload-hook.test.ts | 173 +++++++++++++++++++++++++++---- src/bin/codegraph.ts | 15 +-- src/directory.ts | 38 +++++++ 4 files changed, 201 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 74d35ba919..d8d9c4e2ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, a call through a module-level instance, like `_svc.delete(obj)` after `_svc = SceneService()`, now links to the method its class has or inherits, even when other classes have a method of that name. (#2582) - In Python, a call on a module imported from outside your project, like `subprocess.run(…)` after `import subprocess`, no longer links to a project method that happens to share its name. Thanks @Egrn for the report. (#2599) - `codegraph_explore`, the CLI `codegraph explore` and the prompt hook no longer stall for a minute or more on a large project after a full index; the step that finds the project's busiest file no longer slows to a crawl when a few files are huge, like a generated Android `R.java` (#2571). Thanks @yangfengfan2011. +- The Claude Code prompt hook now goes by what you typed. Claude Code's desktop app puts status notes, like which other sessions are linked, in front of your prompt, and the hook read them as part of it, so every turn, even a one-word "yes", could get code context about the note's words, and so could a task notification, a subagent's report or a message from another session that came after one. The notes are now skipped, and a message from another session gets no context, as task notifications and subagent reports already didn't. Thanks @WonderMr for the report and @yang0228 for the first fix. (#2635) - `codegraph_explore` and `codegraph context` now find your project's busiest file, and favor the code around it, even when twenty or more test or generated files are busier still, such as a package of protobuf stubs. Before, the search looked only at the 20 busiest files and then set the tests and generated ones aside, so it could find no file at all, or judge one far ahead of the next when it was not. `getTopRouteFile()` in the library counts every file's routes the same way. Thanks @inth3shadows for the report. (#2578) - In JavaScript and TypeScript, a call written as the body of an `if`, `while`, `for` or `do` without braces, like `if (e.key === "Escape") setOpen(false);`, no longer links to a same-named property or method in another file when the name is the file's own, such as a `setOpen` from `useState` (#2585). - `codegraph sync` no longer fails with "FOREIGN KEY constraint failed" in a project whose code lazy-loads a workspace package, like Supabase Studio's `dynamic(() => import('icons').then((mod) => mod.EdgeFunctions))` or vben's `() => import('@vben/layouts').then((m) => m.IFrameView)`. Before, every sync after an edit in such a project rolled back, so the index stopped following your changes until a full re-index. Thanks @bompus for the report and @inth3shadows. (#2565) diff --git a/__tests__/frontload-hook.test.ts b/__tests__/frontload-hook.test.ts index c85700b9c0..f4c1a9beae 100644 --- a/__tests__/frontload-hook.test.ts +++ b/__tests__/frontload-hook.test.ts @@ -7,13 +7,13 @@ * `planFrontload` / `findIndexedSubprojectRoots` directly (the hook's decision * logic), plus the built CLI's confidence tiers against a real index. */ -import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach, vi } from 'vitest'; import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import { spawnSync } from 'node:child_process'; import { CodeGraph } from '../src'; -import { planFrontload, isTaskNotification, isAgentMessage, findIndexedSubprojectRoots, unsafeIndexRootReason, isStructuralPrompt, hasStructuralKeyword, extractCodeTokens, PROMPT_HOOK_INJECTION_MAX, CLAUDE_CODE_INLINE_HOOK_OUTPUT_LIMIT, capPromptHookInjection } from '../src/directory'; +import { planFrontload, isTaskNotification, isAgentMessage, isCrossSessionMessage, stripSystemReminders, findIndexedSubprojectRoots, unsafeIndexRootReason, isStructuralPrompt, hasStructuralKeyword, extractCodeTokens, PROMPT_HOOK_INJECTION_MAX, CLAUDE_CODE_INLINE_HOOK_OUTPUT_LIMIT, capPromptHookInjection } from '../src/directory'; // Make the built-in exports configurable so HOME can point at a real temp // fixture without changing the process environment or the user's home files. @@ -35,6 +35,37 @@ function mkWorkspaceRoot(dir: string): string { return dir; } +/** + * Run the built CLI's `prompt-hook` in `dir` with a UserPromptSubmit payload + * shaped like the one Claude Code sends, and return what it injects. + */ +function runPromptHook(dir: string, prompt: string): string { + const payload = { + session_id: '0f1e2d3c-4b5a-6978-8796-a5b4c3d2e1f0', + transcript_path: path.join(dir, 'transcript.jsonl'), + cwd: dir, + permission_mode: 'default', + hook_event_name: 'UserPromptSubmit', + prompt, + }; + const result = spawnSync(process.execPath, [path.resolve(__dirname, '../dist/bin/codegraph.js'), 'prompt-hook'], { + cwd: dir, + input: JSON.stringify(payload), + encoding: 'utf8', + timeout: 15_000, + env: { + ...process.env, + CODEGRAPH_TELEMETRY: '0', DO_NOT_TRACK: '1', CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_NO_RELAUNCH: '1', CODEGRAPH_WASM_RELAUNCHED: '1', + // Enable only the hook subprocess under test. + CODEGRAPH_NO_PROMPT_HOOK: '0', CODEGRAPH_PROMPT_HOOK: '1', + }, + }); + expect(result.error).toBeUndefined(); + expect(result.status, result.stderr).toBe(0); + return result.stdout; +} + describe('planFrontload — front-load hook project resolution (#964)', () => { let tmp: string; beforeEach(() => { tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'cg-frontload-'))); }); @@ -321,24 +352,7 @@ export class OrderStateMachine { }); afterEach(() => { fs.rmSync(tmp, { recursive: true, force: true }); }); - function hook(prompt: string): string { - const result = spawnSync(process.execPath, [path.resolve(__dirname, '../dist/bin/codegraph.js'), 'prompt-hook'], { - cwd: tmp, - input: JSON.stringify({ cwd: tmp, prompt }), - encoding: 'utf8', - timeout: 15_000, - env: { - ...process.env, - CODEGRAPH_TELEMETRY: '0', DO_NOT_TRACK: '1', CODEGRAPH_NO_DAEMON: '1', - CODEGRAPH_NO_RELAUNCH: '1', CODEGRAPH_WASM_RELAUNCHED: '1', - // Enable only the hook subprocess under test. - CODEGRAPH_NO_PROMPT_HOOK: '0', CODEGRAPH_PROMPT_HOOK: '1', - }, - }); - expect(result.error).toBeUndefined(); - expect(result.status, result.stderr).toBe(0); - return result.stdout; - } + const hook = (prompt: string): string => runPromptHook(tmp, prompt); it('stays silent for everyday Portuguese, Spanish and German without indexed evidence', () => { for (const prompt of [ @@ -390,6 +404,70 @@ export class OrderStateMachine { }); }); +describe('prompt-hook reads past the desktop app\'s system reminders (#2635)', () => { + // Shaped like the prompts Claude Code's desktop app submits: one or more + // reminder blocks, separated by one or two newlines, then what the user + // typed or another session's message. A linked session's title names the + // work, so the reminder carries structural words ("trace", "flow") of its own. + const linked = '\nLinked sessions now (status, not an instruction): trace-order-flow = local_5f0c2a1e-9b7d-4c3a-8e21-3d4f5a6b7c8d (working). To message one, use SendMessage with its id.\n'; + const workspace = '\nThis session runs in a desktop workspace; write temporary files to its scratch folder.\n'; + const reminders = `${linked}\n${workspace}\n\n`; + let tmp: string; + + beforeAll(async () => { + tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'cg-hook-reminders-'))); + fs.writeFileSync(path.join(tmp, 'orders.ts'), ` +export class OrderStateMachine { + submitOrder() { return this.transition('submitted'); } + transition(state: string) { return state.length > 0; } +} +`); + const cg = await CodeGraph.init(tmp, { silent: true }); + try { await cg.indexAll(); } finally { cg.destroy(); } + }); + afterAll(() => { fs.rmSync(tmp, { recursive: true, force: true }); }); + + const hook = (prompt: string): string => runPromptHook(tmp, prompt); + + it('stays silent on a short reply that follows reminders', () => { + expect(hook('yes')).toBe(''); + expect(hook(`${reminders}yes`)).toBe(''); + }); + + it('stays silent when the prompt is only reminders', () => { + expect(hook(reminders)).toBe(''); + expect(hook(` \n${linked}\n\n${workspace}\n`)).toBe(''); + }); + + it('explores what the user typed, not the reminders', () => { + const output = hook(`${reminders}how does submitOrder reach transition?`); + expect(output).toContain('Structural context from CodeGraph'); + expect(output).toContain('**Exploration: how does submitOrder reach transition?**'); + expect(output).not.toContain('Linked sessions'); + expect(output).not.toContain('trace-order-flow'); + }); + + it('gates on what the user typed: prose that names indexed symbols gets the MEDIUM tier', () => { + const output = hook(`${linked}\n\nstate machine`); + expect(output).toContain('CodeGraph found indexed symbols matching this prompt'); + expect(output).toContain('OrderStateMachine'); + expect(output).not.toContain('Structural context from CodeGraph'); + }); + + it.each([ + ['a message from another session', '\nI traced how OrderStateMachine.submitOrder() reaches transition(). Ready for review.\n'], + ['a task notification', '\nb7d3k2\ncompleted\n

Background command "trace submitOrder flow" completed (exit code 0)\n'], + ['a subagent hand-back', '\n[Subagent hand-back] submitOrder() calls transition() on OrderStateMachine.\n'], + ])('stays silent on %s behind reminders', (_label, envelope) => { + expect(hook(`${linked}\n\n${envelope}`)).toBe(''); + expect(hook(`${reminders}${envelope}`)).toBe(''); + }); + + it('stays silent on a message from another session on its own', () => { + expect(hook('\nhow does OrderStateMachine work?\n')).toBe(''); + }); +}); + describe('extractCodeTokens — candidate symbols the hook verifies against the graph', () => { it('pulls camelCase / PascalCase / snake_case / call / member tokens', () => { expect(extractCodeTokens('prepareArticlePublish 的调用链')).toContain('prepareArticlePublish'); @@ -494,3 +572,58 @@ describe('subagent hand-backs (#2184)', () => { expect(isAgentMessage('trace AuthService login')).toBe(false); }); }); + +describe('messages from other sessions (#2635)', () => { + const message = '\nTraced how AuthService.login calls TokenStore.save.\n'; + + it('skips the complete envelope', () => { + expect(isCrossSessionMessage(message)).toBe(true); + expect(isCrossSessionMessage(` \n${message}\n`)).toBe(true); + expect(isCrossSessionMessage('trace AuthService login flow')).toBe(true); + }); + it('does not suppress a user question that mentions the marker', () => { + expect(isCrossSessionMessage(`Why does ${message} trigger the hook?`)).toBe(false); + expect(isCrossSessionMessage(`${message} Explain this.`)).toBe(false); + expect(isCrossSessionMessage('trace AuthService')).toBe(false); + expect(isCrossSessionMessage('trace AuthService')).toBe(false); + expect(isCrossSessionMessage('trace AuthService login')).toBe(false); + }); +}); + +describe('stripSystemReminders (#2635)', () => { + const reminder = '\nLinked sessions now (status, not an instruction): trace-auth = local_5f0c2a1e (working).\n'; + + it('drops the reminders in front of the prompt, however they are separated', () => { + expect(stripSystemReminders(`${reminder}\n\nhow does AuthService.login work?`)).toBe('how does AuthService.login work?'); + expect(stripSystemReminders(`${reminder}\n${reminder}\n\nyes`)).toBe('yes'); + expect(stripSystemReminders(`${reminder}yes`)).toBe('yes'); + expect(stripSystemReminders('status\nyes')).toBe('yes'); + }); + + it('leaves nothing when the prompt is only reminders', () => { + expect(stripSystemReminders(reminder)).toBe(''); + expect(stripSystemReminders(` \n${reminder}\n\n${reminder}\n`)).toBe(''); + }); + + it('keeps the user\'s text around a reminder elsewhere in the prompt, and keeps envelopes for the checks after it', () => { + expect(stripSystemReminders(`why does the hook react to this?\n\n${reminder}`)).toBe('why does the hook react to this?'); + expect(stripSystemReminders(`log line one\n${reminder}\nlog line two`)).toBe('log line one\n\n\nlog line two'); + const envelope = 'trace AuthService'; + expect(stripSystemReminders(`${reminder}\n\n${envelope}`)).toBe(envelope); + expect(isCrossSessionMessage(stripSystemReminders(`${reminder}\n\n${envelope}`))).toBe(true); + }); + + it('leaves a prompt that only mentions the tag alone', () => { + expect(stripSystemReminders('how does AuthService.login work?')).toBe('how does AuthService.login work?'); + expect(stripSystemReminders('why does show up in the hook input?')).toBe('why does show up in the hook input?'); + expect(stripSystemReminders('trace AuthService')).toBe('trace AuthService'); + expect(stripSystemReminders('a stray closing tag')).toBe('a stray closing tag'); + }); + + it('stays linear on a long pasted prompt full of unclosed tags', () => { + const pasted = `${'`; + const started = Date.now(); + stripSystemReminders(pasted); + expect(Date.now() - started).toBeLessThan(1_000); + }); +}); diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 296d950d16..119bd3c876 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -53,7 +53,7 @@ try { import { Command } from 'commander'; import * as path from 'path'; import * as fs from 'fs'; -import { getCodeGraphDir, isInitialized, hasSchemalessDb, hasForeignDbFile, unsafeIndexRootReason, findNearestCodeGraphRoot, planFrontload, isTaskNotification, isAgentMessage, hasStructuralKeyword, extractCodeTokens, capPromptHookInjection, codeGraphDirName, DEFAULT_CODEGRAPH_DIR } from '../directory'; +import { getCodeGraphDir, isInitialized, hasSchemalessDb, hasForeignDbFile, unsafeIndexRootReason, findNearestCodeGraphRoot, planFrontload, isTaskNotification, isAgentMessage, isCrossSessionMessage, stripSystemReminders, hasStructuralKeyword, extractCodeTokens, capPromptHookInjection, codeGraphDirName, DEFAULT_CODEGRAPH_DIR } from '../directory'; import { extractProseCandidates } from '../search/identifier-segments'; import { detectWorktreeIndexMismatch, worktreeMismatchWarning } from '../sync/worktree'; import { createShimmerProgress } from '../ui/shimmer-progress'; @@ -1512,11 +1512,14 @@ program let input: { prompt?: string; cwd?: string } = {}; try { input = JSON.parse(raw); } catch { return; } - const prompt = String(input.prompt || ''); - // System-injected task notifications and subagent hand-backs are not - // user prompts: exit before any project lookup or explore work (#1832, - // #2184). - if (isTaskNotification(prompt) || isAgentMessage(prompt)) return; + // The desktop app's blocks are host status, not what + // the user typed: gate, pick the project and explore on the rest + // (#2635). + const prompt = stripSystemReminders(String(input.prompt || '')); + // System-injected task notifications, subagent hand-backs and messages + // from other sessions are not user prompts, and neither is nothing: + // exit before any project lookup or explore work (#1832, #2184, #2635). + if (!prompt || isTaskNotification(prompt) || isAgentMessage(prompt) || isCrossSessionMessage(prompt)) return; // Gate telemetry: how often each tier fires vs. no-ops — counter names // only, NEVER prompt content (see TELEMETRY.md). This is the data that diff --git a/src/directory.ts b/src/directory.ts index eb359a0c62..2e7445d380 100644 --- a/src/directory.ts +++ b/src/directory.ts @@ -1125,3 +1125,41 @@ export function isTaskNotification(prompt: string): boolean { export function isAgentMessage(prompt: string): boolean { return /^\s*]*)?>[\s\S]*<\/agent-message>\s*$/.test(prompt); } + +/** + * A message another local Claude Code session sends to this one arrives as a + * `…` prompt, which + * UserPromptSubmit hooks also receive (#2635). Same rule as + * {@link isTaskNotification}: only a prompt that is entirely that envelope is + * skipped. + */ +export function isCrossSessionMessage(prompt: string): boolean { + return /^\s*]*)?>[\s\S]*<\/cross-session-message>\s*$/.test(prompt); +} + +const SYSTEM_REMINDER_CLOSE = ''; +/** One complete reminder block. The attribute run stops at `<`, so a tag left + * open in pasted text can't make the scan quadratic. */ +const SYSTEM_REMINDER_BLOCK = /]*)?>[\s\S]*?<\/system-reminder>/g; + +/** + * The prompt the user typed, without the `…` + * blocks Claude Code's desktop app puts in front of it, such as which other + * local sessions are linked. UserPromptSubmit hooks receive them as + * part of the prompt (#2635), but they are the host's status, not the user's + * words: gated and explored on, they made every turn, a bare "yes" included, + * inject context about the reminder, and they hid a task notification, a + * subagent hand-back or a cross-session message behind them from + * {@link isTaskNotification}, {@link isAgentMessage} and + * {@link isCrossSessionMessage}. Every complete block is removed and the rest + * trimmed; a `` the user only mentions, with no closing tag, + * stays. + */ +export function stripSystemReminders(prompt: string): string { + // Only the text up to the last closing tag can hold a complete block, and + // bounding the scan there keeps an unclosed tag from rescanning the tail. + const end = prompt.lastIndexOf(SYSTEM_REMINDER_CLOSE); + if (end < 0) return prompt.trim(); + const head = prompt.slice(0, end + SYSTEM_REMINDER_CLOSE.length); + return (head.replace(SYSTEM_REMINDER_BLOCK, '\n') + prompt.slice(head.length)).trim(); +} From e8ddc4e2be02578d735c350df55f1769b7d6978a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 11:54:48 +0000 Subject: [PATCH 34/95] fix(js,ts): imports of a repo's own packages follow their package.json entries (#2799) Issue 2502: a pnpm (or npm, Yarn, Bun) workspace member that declares its entry with `main`, `module`, `types` or `exports`, like `"main": "./src/index.ts"`, has no index file at its root, so `import { caseStatus } from '@acme/core'` failed and its calls fell back to name matching, where a same-named private function nearer the caller took them. Issue 2456: a sibling package another declares as a path dependency (`"acme-shared": "file:../common"`), with no root `workspaces`, was not known as a package at all. Root cause: workspace-packages.ts mapped a member's name to its folder only (its header said it did not yet honour `exports` or `main`), and the resolver probed `/index.*`. Path dependencies were read only from ohpm manifests. Fix: - workspace-packages.ts reads each package's declared entries: `exports` (subpaths, `*` patterns, fallback arrays, nested conditions, read with import/require/node/module/default first, other conditions next, types last) and the root's source/module/main/types/typings. It reads `file:`, `link:` and `portal:` dependencies from every package.json in the existing manifest walk: the root manifest's for every file, any other's for the files under it, as npm installs them. - The resolver takes the first declared target, `exports` before the main-style fields, that the index holds or that is build output the index lacks, read as the source it is built from (dist/, build/, lib/, out/, es/, esm/, cjs/ mirrored under src/, then without a format folder). Then the package's folder, as before, and a path into its build output (`@acme/utils/lib/path`) read as its source. A file a local build left on disk never wins over an indexed source, and no mapping links nothing. - Sync: a full reconcile reads the manifests again and, when their digest moved, re-opens every JavaScript-family reference, as a moved Python root set does. A file added inside a package re-checks the imports of that package that landed elsewhere in it. A full index drops the cached map. Validation: - __tests__/workspace-package-entries.test.ts (26 tests): 25 fail on main, the control passes; all pass on the kernel and with CODEGRAPH_KERNEL=0. Each of 17 rules removed in turn fails at least one test. - Sync equals a fresh index in 15 CLI scenarios (entries, subpaths, a member renamed, added or leaving the globs, file: deps added, removed or retargeted, an entry's file appearing). On real repos: two vben manifest edits (237 edge rows move) and a vite playground file: edit, synced graph identical to a fresh index. - A/B on 14 TS/JS repos and 2 controls: vben +2,955 edges, insomnia +2,222, webdriverio +2,472, jest +1,700, nuxt +706, memfs +470, vite +251, t3-turbo +47, social-app +40, vue3-antdv-admin +8, with name-matched namesakes corrected; nitro, outline, pathe, c12, flask and gin are byte-identical. - Cost: fresh index within 3% (median of 3, interleaved); a full sync reads the manifests, +30 to 160 ms on these monorepos. Fixes #2502 Fixes #2456 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/workspace-package-entries.test.ts | 370 ++++++++++++++++++ src/db/queries.ts | 31 ++ src/index.ts | 47 ++- src/resolution/import-resolver.ts | 120 +++++- src/resolution/index.ts | 36 +- src/resolution/module-importers.ts | 35 +- src/resolution/workspace-packages.ts | 393 +++++++++++++++++--- 8 files changed, 965 insertions(+), 68 deletions(-) create mode 100644 __tests__/workspace-package-entries.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d8d9c4e2ab..be4d0d65d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -223,6 +223,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In JavaScript and TypeScript, a function the file imports from a package and passes along as a value, like react-router's `Link` in `styled(Link)`, lodash's `noop` in `createContext({ onClose: noop })` or chart.js's `Legend` in `Chart.register(Legend)`, no longer links to a project function that only shares its name. Before, whenever the project had exactly one function of that name, every such value linked to it: outline's breadcrumbs, menus and document lists showed up among the callers of an email template's `Link`, and a test's own `noop` stub in cord-field among the code the app's contexts use, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A function the file declares itself, including one that hides the import inside a function, and one imported from a file of your project link as before. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript and TypeScript, a function a file loads with a dynamic import from Node or a package, like `const { createServer } = await import('node:http')`, `const { default: chalk } = await import('chalk')` or `import('node:fs/promises').then(({ readFile }) => …)`, no longer links to a project function that only shares its name. Before, vite's call to Node's `createServer` showed up among the callers of vite's own `createServer`. The import counts only where it is in scope, so the rest of the file links as before, and a dynamic import of your own project's module still leads to that module. Thanks @Ancienttwo for the report. (#2491) - In JavaScript and TypeScript, code outside your tests no longer links to a Jest or Vitest manual mock under `__mocks__/`. Before, when an import of a workspace package couldn't be followed, a call like `formatLabel(name)` linked by name to the mock beside it instead of the package's own function, so the real function lost its callers, and the same happened to rendered components and event handlers. Production code is now matched as if the mocks weren't there, so it reaches the real function, and adding or deleting a mock leaves it alone. Tests, other mocks, and a file that imports a mock by its path, like a `jest.setup.ts`, still reach the mock. Thanks @Y-Osawa2022 for the report and @danusha2345. (#2455) +- In JavaScript and TypeScript, an import of a package from your own repository now leads to the file its `package.json` names: the `exports` entry for the import, conditions and subpaths like `"./utils": "./src/utils.ts"` included, or else its `main`, `module`, `types` or `source`. A package another one depends on by path, like `"acme-shared": "file:../common"` or `link:`, now counts as part of your project too, for the files of the package that declares it, even with no root `workspaces`. Before, an import of a pnpm workspace member declared with `"main": "./src/index.ts"` and no `index` file at its root, or of a `file:` dependency, couldn't be followed, so its calls were matched by name alone and a same-named private function nearer the caller could take them. An entry that names build output, like `./dist/esm/index.mjs`, or an import of a path inside it, like `@jsonjoy.com/fs-node-utils/lib/types/misc`, leads to the matching source file, like `src/index.ts` or `src/types/misc.ts`, when your project has one, and to nothing otherwise. `codegraph sync` follows edits to these `package.json` fields. Re-index JavaScript and TypeScript monorepos after upgrading. Thanks @elderbernardi and @Y-Osawa2022 for the reports, and @maxmilian and @danusha2345. (#2502, #2456) - In JavaScript and TypeScript, a value the file imports from your own project and passes along now links only to what it imports: a constant like `isCloudHosted`, a class like a `Star` model, a whole module (`import * as Slack from '../slack'`), or a stylesheet, image or GraphQL document (`import TCO21 from './themes/TCO21.scss'`) no longer links to a function elsewhere that only shares its name. Before, when the project had exactly one function of that name, such a value linked to it: outline's settings screens showed up among the callers of the server's `isCloudHosted()` check, and its stores among those of the `` icon, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A component that wraps the function declared beside it, like `export default styled(Disclosure)`, and a page loaded lazily, like `lazy(() => import('../pages/Home'))`, still lead to that function. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript, an import of a TypeScript module written without an extension, like `import AudioModal from './audio_modal'` in a `.jsx` file next to `audio_modal.tsx`, now links to that module, as it already did from a TypeScript file. Before, a `.js` or `.jsx` file's imports of TypeScript modules were never followed, so in a codebase moving to TypeScript they were matched by name instead: an import under another name linked nothing, and a name that repeats could link to the wrong file, so mastodon's `` and `` tags rendered other components of those names than the ones each file imports. A JavaScript module beside a TypeScript one of the same name still comes first. Re-index JavaScript projects after upgrading. - `codegraph_explore`'s Flow now goes through the implementation your query names when an interface method has several that lead to the same place. Before, it took whichever one came first in the index: asking about prometheus's `Engine.execEvalStmt Queryable.Querier fanout.Querier NewMergeQuerier` traced the call through the TSDB's `DB.Querier` and left the `fanout.Querier` you named off the Flow. Any two routes of the same length are now settled this way, in favor of the one that passes through more of the symbols you named. diff --git a/__tests__/workspace-package-entries.test.ts b/__tests__/workspace-package-entries.test.ts new file mode 100644 index 0000000000..855f36e0b5 --- /dev/null +++ b/__tests__/workspace-package-entries.test.ts @@ -0,0 +1,370 @@ +/** + * Issues #2456 and #2502: a package of the repository is imported by its name, + * and that import has to land on the file the package's manifest names. + * + * - #2502: a pnpm (or npm/yarn/bun) workspace member that declares its entry + * with `main` / `module` / `types` or an `exports` map, like + * `"main": "./src/index.ts"`, had no `index.*` at its root, so + * `import { caseStatus } from '@acme/core'` failed and its calls fell back to + * name matching: a same-named private function nearer the caller took them. + * - #2456: a sibling package an npm `file:` dependency points at, with no root + * `workspaces`, was not known as a package at all. + * + * A declared entry that names build output the index does not hold (`dist/`) + * is read as the source it is built from when the mapping is clear + * (`dist/index.js` → `src/index.ts`), and otherwise links nothing new. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let roots: string[] = []; +let graphs: CodeGraph[] = []; + +afterEach(() => { + for (const graph of graphs) graph.close(); + graphs = []; + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); + roots = []; +}); + +function write(root: string, files: Record): void { + for (const [rel, content] of Object.entries(files)) { + const file = path.join(root, rel); + if (content === null) { + fs.rmSync(file, { force: true }); + continue; + } + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, content); + } +} + +async function index(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-ws-entries-')); + roots.push(root); + write(root, files); + const graph = await CodeGraph.init(root, { index: true }); + graphs.push(graph); + return graph; +} + +/** The files `file`'s import statements link it to. */ +function moduleImports(graph: CodeGraph, file: string): string[] { + const out = new Set(); + for (const node of graph.getNodesInFile(file)) { + for (const e of graph.getOutgoingEdges(node.id)) { + if (e.kind !== 'imports') continue; + const target = graph.getNode(e.target)!; + if (target.kind === 'file') out.add(target.filePath); + } + } + return [...out].sort(); +} + +/** `name@file` of every call `file` makes. */ +function callsFrom(graph: CodeGraph, file: string): string[] { + const out: string[] = []; + for (const node of graph.getNodesInFile(file)) { + for (const e of graph.getOutgoingEdges(node.id)) { + if (e.kind !== 'calls') continue; + const target = graph.getNode(e.target)!; + out.push(`${target.name}@${target.filePath}`); + } + } + return out.sort(); +} + +/** A pnpm workspace whose `@acme/core` member has the given manifest. */ +function pnpm(core: Record, extra: Record = {}): Record { + return { + 'package.json': JSON.stringify({ name: 'repo', private: true }), + 'pnpm-workspace.yaml': 'packages:\n - "packages/*"\n - "apps/*"\n', + 'packages/core/package.json': JSON.stringify({ name: '@acme/core', ...core }), + 'packages/core/src/index.ts': 'export function coreFn(): number { return 1; }\n', + 'apps/web/package.json': JSON.stringify({ name: '@acme/web', dependencies: { '@acme/core': 'workspace:*' } }), + 'apps/web/main.ts': "import { coreFn } from '@acme/core';\nexport const run = () => coreFn();\n", + ...extra, + }; +} + +describe('a workspace member imported by name lands on the entry its manifest declares (#2502)', () => { + it.each([ + ['main', { main: './src/index.ts' }], + ['module', { module: './src/index.ts' }], + ['types', { types: './src/index.ts' }], + ['a string `exports`', { exports: './src/index.ts' }], + ['`exports["."]`', { exports: { '.': './src/index.ts' } }], + ['conditions straight under `exports`', { exports: { types: './dist/index.d.ts', import: './src/index.ts' } }], + ])('through %s', async (_label, manifest) => { + const graph = await index(pnpm(manifest)); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toContain('coreFn@packages/core/src/index.ts'); + }, 120_000); + + it('through conditions, nested ones included, whichever the index holds', async () => { + // `types` and `require` name build output; the source sits under `import` → `default`. + const exports = { + '.': { + types: './dist/index.d.ts', + import: { types: './dist/index.d.mts', default: './src/index.ts' }, + require: './dist/index.cjs', + }, + }; + const graph = await index(pnpm({ exports })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it("through the condition its importers resolve by, read as its source, before a custom condition's source", async () => { + // @wdio/logger's layout: `import` names build output, `browserSource` a browser-only source file. + const exports = { + '.': { + browserSource: './src/browser.ts', + browser: './build/browser.js', + types: './build/index.d.ts', + import: './build/index.js', + require: './build/index.cjs', + }, + }; + const graph = await index(pnpm({ exports }, { + 'packages/core/src/browser.ts': 'export function coreFn(): number { return 5; }\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it('through the implementation a condition names before a declaration file', async () => { + const graph = await index(pnpm({ exports: { '.': { types: './src/index.d.ts', import: './src/index.ts' } } }, { + 'packages/core/src/index.d.ts': 'export declare function coreFn(): number;\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it('never through build output that is on disk but not indexed', async () => { + // A local build left `dist/` behind; `.gitignore` keeps it out of the index. + const graph = await index(pnpm({ main: './dist/index.js' }, { + '.gitignore': 'dist/\n', + 'packages/core/dist/index.js': 'exports.coreFn = function () { return 1; };\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it('through a subpath export and a `./*` pattern', async () => { + const graph = await index(pnpm( + { exports: { '.': './src/index.ts', './utils': './src/utils.ts', './schema/*': { import: './src/schema/*.ts' } } }, + { + 'packages/core/src/utils.ts': 'export function trim(s: string): string { return s.trim(); }\n', + 'packages/core/src/schema/case.ts': "export const caseTable = 'case';\n", + 'apps/web/text.ts': "import { trim } from '@acme/core/utils';\nexport const clean = (s: string) => trim(s);\n", + 'apps/web/table.ts': "import { caseTable } from '@acme/core/schema/case';\nexport const tableName = () => caseTable;\n", + }, + )); + expect(moduleImports(graph, 'apps/web/text.ts')).toEqual(['packages/core/src/utils.ts']); + expect(callsFrom(graph, 'apps/web/text.ts')).toContain('trim@packages/core/src/utils.ts'); + expect(moduleImports(graph, 'apps/web/table.ts')).toEqual(['packages/core/src/schema/case.ts']); + }, 120_000); + + it('through build output the index lacks, read as the source it is built from', async () => { + const graph = await index(pnpm({ + main: './dist/cjs/index.cjs', + types: './dist/types/index.d.ts', + exports: { '.': { import: './dist/esm/index.mjs', require: './dist/cjs/index.cjs' }, './utils': './dist/utils.js' }, + }, { + 'packages/core/src/utils.ts': 'export function trim(s: string): string { return s.trim(); }\n', + 'apps/web/text.ts': "import { trim } from '@acme/core/utils';\nexport const clean = (s: string) => trim(s);\n", + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + expect(moduleImports(graph, 'apps/web/text.ts')).toEqual(['packages/core/src/utils.ts']); + }, 120_000); + + it('through a path into its build output, read as the source it is built from', async () => { + // memfs's layout: `import { … } from '@jsonjoy.com/fs-node-utils/lib/types/misc'` is + // `lib/types/misc.js`, built from `src/types/misc.ts`. A local build on disk changes nothing. + const graph = await index(pnpm({ main: './lib/index.js' }, { + '.gitignore': 'lib/\n', + 'packages/core/lib/strings.js': 'exports.trim = function (s) { return s; };\n', + 'packages/core/src/strings.ts': 'export function trim(s: string): string { return s.trim(); }\n', + 'packages/core/src/types/misc.ts': 'export type Encoding = string;\n', + 'apps/web/text.ts': + "import { trim } from '@acme/core/lib/strings';\nimport type { Encoding } from '@acme/core/lib/types/misc';\nexport const clean = (s: string, e: Encoding) => trim(s) + e;\n", + })); + expect(moduleImports(graph, 'apps/web/text.ts')).toEqual(['packages/core/src/strings.ts', 'packages/core/src/types/misc.ts']); + expect(callsFrom(graph, 'apps/web/text.ts')).toEqual(['trim@packages/core/src/strings.ts']); + }, 120_000); + + it('through the source tree a build folder mirrors before the folder read as a build format', async () => { + // Vite's layout: `dist/node/index.js` is built from `src/node/index.ts`, not `src/index.ts`. + const graph = await index(pnpm({ main: './dist/node/index.js', types: './dist/node/index.d.ts' }, { + 'packages/core/src/node/index.ts': 'export function coreFn(): number { return 3; }\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/node/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toContain('coreFn@packages/core/src/node/index.ts'); + }, 120_000); + + it("through `exports` first, then the root's `main`-style fields when `exports` leads to no indexed file", async () => { + // `exports` read as its source beats a root `index.js` that `main` names… + const first = await index(pnpm({ exports: { '.': './dist/index.mjs' }, main: './index.js' }, { + 'packages/core/index.js': "module.exports = require('./dist/index.cjs');\n", + })); + expect(moduleImports(first, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + // …and `source` is read when `exports` names build output with no clear source. + const then = await index(pnpm({ exports: { '.': './dist/acme-core.mjs' }, source: './src/main.ts' }, { + 'packages/core/src/main.ts': 'export function coreFn(): number { return 4; }\n', + })); + expect(moduleImports(then, 'apps/web/main.ts')).toEqual(['packages/core/src/main.ts']); + }, 120_000); + + it('and links nothing new when build output has no clear source', async () => { + // `dist/bundle.js` has no `src/bundle.*`: no guess at another file. + const graph = await index(pnpm({ main: './dist/bundle.js' })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual([]); + }, 120_000); + + it("so a same-named private function no longer takes the package's callers (the issue's repro)", async () => { + const graph = await index({ + 'package.json': JSON.stringify({ name: 'repro', private: true }), + 'pnpm-workspace.yaml': 'packages:\n - "packages/*"\n - "apps/*"\n', + 'packages/core/package.json': JSON.stringify({ + name: '@acme/core', type: 'module', main: './src/index.ts', exports: { '.': './src/index.ts' }, + }), + 'packages/core/src/index.ts': "export * from './status.js';\n", + 'packages/core/src/status.ts': + "export function caseStatus(input: { open: boolean }): { value: string } {\n return { value: input.open ? 'open' : 'closed' };\n}\n", + 'packages/db/package.json': JSON.stringify({ name: '@acme/db', type: 'module' }), + 'packages/db/src/load.ts': + "import { caseStatus } from '@acme/core';\nexport const loadStatus = (open: boolean) => caseStatus({ open }).value;\n", + 'apps/web/package.json': JSON.stringify({ name: '@acme/web', type: 'module' }), + 'apps/web/lib/list.ts': + "import { caseStatus } from '@acme/core';\nexport const listCases = (cases: { open: boolean }[]) => cases.map((c) => caseStatus(c).value);\n", + 'apps/worker/package.json': JSON.stringify({ name: '@acme/worker', type: 'module' }), + 'apps/worker/src/run.ts': + "import { caseStatus } from '@acme/core';\nexport const run = () => caseStatus({ open: true }).value;\n", + 'apps/worker/src/interpret.ts': + 'function caseStatus(raw: string): string { return raw.trim(); }\nexport const interpret = (raw: string) => caseStatus(raw);\n', + }); + for (const file of ['packages/db/src/load.ts', 'apps/web/lib/list.ts', 'apps/worker/src/run.ts']) { + expect(callsFrom(graph, file)).toContain('caseStatus@packages/core/src/status.ts'); + expect(callsFrom(graph, file)).not.toContain('caseStatus@apps/worker/src/interpret.ts'); + } + }, 120_000); +}); + +describe('a package an npm `file:` dependency points at is imported by its name (#2456)', () => { + /** The issue's layout: no root manifest, `acme-api` depends on `file:../common`. */ + const fileDep = (spec: string, extra: Record = {}): Record => ({ + 'packages/common/package.json': JSON.stringify({ name: 'acme-shared', version: '1.0.0' }), + 'packages/common/utils.ts': + "export function formatLabel(name: string): string { return name.trim(); }\nexport function isAdmin(role: string): boolean { return role === 'admin'; }\n", + 'packages/api/package.json': JSON.stringify({ name: 'acme-api', dependencies: { 'acme-shared': spec } }), + 'packages/api/src/handler.ts': + "import { formatLabel, isAdmin } from 'acme-shared/utils';\nexport function handleRequest(name: string, role: string): string {\n return isAdmin(role) ? formatLabel(name) : 'forbidden';\n}\n", + ...extra, + }); + + it.each(['file:../common', 'link:../common'])('with %s and no root `workspaces`', async (spec) => { + const graph = await index(fileDep(spec, { + // A same-named function nearer the caller, which name matching would pick. + 'packages/api/src/format.ts': "export function formatLabel(name: string): string { return '!' + name; }\n", + })); + expect(moduleImports(graph, 'packages/api/src/handler.ts')).toEqual(['packages/common/utils.ts']); + expect(callsFrom(graph, 'packages/api/src/handler.ts')).toEqual([ + 'formatLabel@packages/common/utils.ts', 'isAdmin@packages/common/utils.ts', + ]); + }, 120_000); + + it('for every file when the root manifest declares it', async () => { + // An Expo app's local native module, as `link:` from the root package.json. + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app', dependencies: { '@acme/native': 'link:./modules/native' } }), + 'modules/native/package.json': JSON.stringify({ name: '@acme/native', main: './src/index.ts' }), + 'modules/native/src/index.ts': 'export function share(text: string): string { return text; }\n', + 'src/screens/Share.ts': "import { share } from '@acme/native';\nexport const onShare = () => share('hi');\n", + }); + expect(moduleImports(graph, 'src/screens/Share.ts')).toEqual(['modules/native/src/index.ts']); + expect(callsFrom(graph, 'src/screens/Share.ts')).toEqual(['share@modules/native/src/index.ts']); + }, 120_000); + + it("only for the files under the manifest that declares it", async () => { + // A test fixture's `left-pad` stub is the fixture's; the app's `left-pad` stays the npm package. + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app', dependencies: { 'left-pad': '^1.3.0' } }), + 'src/pad.ts': "import leftPad from 'left-pad';\nexport const pad = (s: string) => leftPad(s, 4);\n", + 'test/fixture/package.json': JSON.stringify({ name: 'fixture', dependencies: { 'left-pad': 'file:../stub-left-pad' } }), + 'test/fixture/use.ts': "import leftPad from 'left-pad';\nexport const use = () => leftPad('x', 2);\n", + 'test/stub-left-pad/package.json': JSON.stringify({ name: 'left-pad', main: 'index.ts' }), + 'test/stub-left-pad/index.ts': 'export default function leftPad(s: string, n: number): string { return s.padStart(n); }\n', + }); + expect(moduleImports(graph, 'test/fixture/use.ts')).toEqual(['test/stub-left-pad/index.ts']); + expect(moduleImports(graph, 'src/pad.ts')).toEqual([]); + expect(callsFrom(graph, 'src/pad.ts')).toEqual([]); + }, 120_000); + + it("through the target's own `exports`", async () => { + const graph = await index({ + 'packages/common/package.json': JSON.stringify({ name: 'acme-shared', exports: { './utils': './src/utils.ts' } }), + 'packages/common/src/utils.ts': 'export function formatLabel(name: string): string { return name; }\n', + 'packages/api/package.json': JSON.stringify({ name: 'acme-api', dependencies: { 'acme-shared': 'file:../common' } }), + 'packages/api/handler.ts': "import { formatLabel } from 'acme-shared/utils';\nexport const h = (n: string) => formatLabel(n);\n", + }); + expect(moduleImports(graph, 'packages/api/handler.ts')).toEqual(['packages/common/src/utils.ts']); + }, 120_000); +}); + +describe('sync follows a manifest edit (#2456, #2502)', () => { + it("re-links importers when a member's `main` or `exports` changes, as a fresh index would", async () => { + const files = pnpm({ main: './dist/bundle.js' }, { + 'packages/core/src/next.ts': 'export function coreFn(): number { return 2; }\n', + }); + const graph = await index(files); + const root = roots[roots.length - 1]!; + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual([]); + + write(root, { 'packages/core/package.json': JSON.stringify({ name: '@acme/core', main: './src/index.ts' }) }); + await graph.sync(); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + + write(root, { 'packages/core/package.json': JSON.stringify({ name: '@acme/core', exports: { '.': { import: './src/next.ts' } } }) }); + await graph.sync(); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/next.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toContain('coreFn@packages/core/src/next.ts'); + }, 120_000); + + it('re-links importers when a `file:` dependency appears', async () => { + const graph = await index({ + 'packages/common/package.json': JSON.stringify({ name: 'acme-shared' }), + 'packages/common/utils.ts': 'export function formatLabel(name: string): string { return name; }\n', + 'packages/api/package.json': JSON.stringify({ name: 'acme-api' }), + 'packages/api/handler.ts': "import { formatLabel } from 'acme-shared/utils';\nexport const h = (n: string) => formatLabel(n);\n", + }); + const root = roots[roots.length - 1]!; + expect(moduleImports(graph, 'packages/api/handler.ts')).toEqual([]); + write(root, { 'packages/api/package.json': JSON.stringify({ name: 'acme-api', dependencies: { 'acme-shared': 'file:../common' } }) }); + await graph.sync(); + expect(moduleImports(graph, 'packages/api/handler.ts')).toEqual(['packages/common/utils.ts']); + }, 120_000); +}); + +describe('sync follows the entry a manifest declares when its file appears (#2502)', () => { + it.each([ + ['an `exports` entry', { exports: { '.': './src/index.ts' } }], + ['the source of a `main` in `dist/`', { main: './dist/index.js' }], + ])("re-links importers that fell back to the package's root index, through %s", async (_label, manifest) => { + const files = pnpm(manifest, { + 'packages/core/index.ts': "export function caseStatus(open: boolean): string { return 'root'; }\n", + 'packages/core/src/status.ts': "export function caseStatus(open: boolean): string { return open ? 'open' : 'closed'; }\n", + 'apps/web/main.ts': "import { caseStatus } from '@acme/core';\nexport const run = () => caseStatus(true);\n", + }); + delete files['packages/core/src/index.ts']; + const graph = await index(files); + const root = roots[roots.length - 1]!; + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toEqual(['caseStatus@packages/core/index.ts']); + + // The barrel names no `caseStatus` of its own: only the import leads the call there. + write(root, { 'packages/core/src/index.ts': "export * from './status';\n" }); + await graph.sync(); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toEqual(['caseStatus@packages/core/src/status.ts']); + }, 120_000); +}); diff --git a/src/db/queries.ts b/src/db/queries.ts index 58904634b4..88004853d9 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -3403,6 +3403,37 @@ export class QueryBuilder { })); } + /** + * The module edges (`imports` edges into a file node) into any file under + * the folder `dir` (projectRoot-relative, no trailing slash), with the + * source file and language a resurrection needs: where an import of a + * package by its name can land, whichever file of the package its manifest + * or the folder's index makes the entry. + */ + getModuleImportsUnder( + dir: string + ): Array { + // `dir/` up to `dir0` ('0' is the byte after '/'). + const rows = this.db + .prepare( + `SELECT e.*, tgt.file_path AS target_file_path, src.file_path AS source_file_path, src.language AS source_language + FROM nodes tgt + JOIN edges e ON e.target = tgt.id AND e.kind = 'imports' + JOIN nodes src ON src.id = e.source + WHERE tgt.kind = 'file' AND tgt.file_path >= ? AND tgt.file_path < ?` + ) + .all(`${dir}/`, `${dir}0`) as Array< + EdgeRow & { target_file_path: string; source_file_path: string; source_language: Language } + >; + return rows.map((row) => ({ + ...rowToEdge(row), + edgeId: row.id, + targetFilePath: row.target_file_path, + sourceFilePath: row.source_file_path, + sourceLanguage: row.source_language, + })); + } + /** * The module edges (`imports` edges into a file node) into the files an * import of `dir/name` can find instead of a file added there: `dir/name.*` diff --git a/src/index.ts b/src/index.ts index b702890af6..0a62d02413 100644 --- a/src/index.ts +++ b/src/index.ts @@ -68,6 +68,7 @@ import { loadDeprioritizePatterns } from './project-config'; import { CodeGraphPackageVersion } from './mcp/version'; import { extractSegmentSearchWords, segmentLookupVariants, splitIdentifierSegments } from './search/identifier-segments'; import { createYielder } from './resolution/cooperative-yield'; +import { ESM_IMPORT_LANGUAGES } from './resolution/import-resolver'; import { minRefsForPool } from './resolution/resolver-pool'; // Re-export types for consumers @@ -150,6 +151,8 @@ export interface IndexOptions { */ /** project_metadata key for the Python package-root digest (see sync). */ const PYTHON_ROOTS_METADATA_KEY = 'python_root_fingerprint'; +/** Metadata key: the digest of the workspace packages and path dependencies the index was resolved by. */ +const WORKSPACE_METADATA_KEY = 'js_workspace_fingerprint'; export class CodeGraph { private db: DatabaseConnection; @@ -772,6 +775,7 @@ export class CodeGraph { // real full index — a sync touches a subset, so it must NOT advance the // extraction stamp (the bulk would still be stale). See extraction-version.ts. if (result.success) this.stampPythonRootFingerprint(); + if (result.success) this.stampWorkspaceFingerprint(); if (result.success && result.filesIndexed > 0) { try { @@ -882,6 +886,13 @@ export class CodeGraph { } catch { /* metadata is advisory */ } } + /** Record the workspace-package digest a later sync compares against (see sync). */ + private stampWorkspaceFingerprint(fingerprint = this.resolver.getWorkspaceFingerprint()): void { + try { + this.queries.setMetadata(WORKSPACE_METADATA_KEY, fingerprint); + } catch { /* metadata is advisory */ } + } + async sync(options: IndexOptions = {}): Promise { return this.indexMutex.withLock(async () => { const startedAt = Date.now(); @@ -1047,6 +1058,35 @@ export class CodeGraph { } } + // Workspace packages and path dependencies (#2456, #2502). A + // package.json edit — a member's name, its `main` or `exports`, a + // `file:` dependency — moves what an import of the package resolves to + // in files it never touched, and no indexed file changes with it. A + // full reconcile (`codegraph sync`, a git hook, the watcher's catch-up) + // reads the manifests again; when what they declare moved, every + // JavaScript-family reference is re-opened, as a moved Python root set + // re-opens Python's below, and the orphan sweep resolves them by the + // new map exactly as a fresh index would. Package.json edits that leave + // the map alone (a version bump, a new registry dependency) cost one + // read. The watcher's scoped syncs follow source files only and skip + // the read: a monorepo's manifests take a walk of the tree to find. + // An index stamped by an older engine has no digest: record one. + if (fullReconcile) { + const tWorkspace = Date.now(); + const before = this.queries.getMetadata(WORKSPACE_METADATA_KEY); + const now = this.resolver.refreshWorkspacePackages(); + if (before !== null && before !== now) { + let reopened = 0; + for (const language of ESM_IMPORT_LANGUAGES) reopened += this.orchestrator.resurrectResolutionEdgesForLanguage(language); + refreshSynthesis = true; + this.queries.setMetadata('synthesis_pending', '1'); + if (process.env.CODEGRAPH_SYNTH_TIMINGS) { + console.error(`[phase-timing] sync-workspace-packages: ${Date.now() - tWorkspace}ms (${reopened} refs re-opened)`); + } + } + if (before !== now) this.stampWorkspaceFingerprint(now); + } + // Resolve references if files were updated const filesChanged = result.filesAdded > 0 || result.filesModified > 0; // The last edge before the failed-ref retry below: the rebind's @@ -1170,7 +1210,9 @@ export class CodeGraph { // `index.ts` re-exporting what it imports: what the files importing // it bound by name in its absence, or failed, goes back to pending // for the sweep below, which binds it through the module as a fresh - // index does. (The removal half runs before removed files are deleted.) + // index does. So does what an import of a workspace package bound + // through another of its files while the entry its manifest names + // was missing. (The removal half runs before removed files are deleted.) const importReader = createImportReader(this.projectRoot); const added = (result.addedFilePaths ?? []).flatMap((filePath) => { const file = this.queries.getFileByPath(filePath); @@ -1180,7 +1222,8 @@ export class CodeGraph { const tAdded = Date.now(); const reopened = reopenReferencesThroughAddedFiles( this.queries, importReader, added, edgesBeforeResolution, - (specifier, fromFile, language) => this.resolver.resolveModuleFile(specifier, fromFile, language) + (specifier, fromFile, language) => this.resolver.resolveModuleFile(specifier, fromFile, language), + (file) => this.resolver.workspacePackageDirOf(file) ); if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-added-module-importers: ${Date.now() - tAdded}ms (${reopened} refs re-opened)`); } diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index 6d11a974eb..01712149b3 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -11,7 +11,7 @@ import { Language, Node } from '../types'; import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, ReExport } from './types'; import { applyAliases, aliasAndBaseUrlCandidates } from './path-aliases'; import { extractLocalExportAliases } from './alias-binding'; -import { resolveWorkspaceImport } from './workspace-packages'; +import { resolveWorkspaceImport, workspaceImportTargets, type WorkspaceImportTargets } from './workspace-packages'; import { stripCommentsForRegex } from './strip-comments'; import { makeLineAt } from './synth-utils'; import { dartDirectiveFile } from './dart-libraries'; @@ -389,7 +389,7 @@ function resolveImportPathUncached( // alias map first (custom prefixes like `@components/*` would // otherwise be misclassified as npm). What still looks like a package // may name a file through the importing app's own config. - if (isExternalImport(importPath, language, context)) { + if (isExternalImport(importPath, language, context, { fromFile })) { return resolveFromOwnConfig(importPath, fromFile, language, context); } @@ -1105,12 +1105,16 @@ const RUST_STDLIB_ROOTS = new Set(['std', 'core', 'alloc', 'proc_macro']); * asked `resolveImportPath` whether an alias maps the specifier to a file. * Matching a prefix proves nothing by itself: a catch-all `"*"` pattern * (`"*": ["./typings/*"]`) has an empty prefix and matches every package. + * + * `fromFile`, the importing file, lets a package another package depends on + * by path (`"acme-shared": "file:../common"`) count as the project's for the + * files under the manifest that declares it (#2456). */ export function isExternalImport( importPath: string, language: Language, context?: ResolutionContext, - options: { aliasPrefixes?: boolean } = {} + options: { aliasPrefixes?: boolean; fromFile?: string } = {} ): boolean { // Relative imports are not external if (importPath.startsWith('.')) { @@ -1122,7 +1126,7 @@ export function isExternalImport( // workspace map first so they aren't misclassified as external (#629). The // map is null for single-package repos, so this is a no-op there. const workspaces = context?.getWorkspacePackages?.(); - if (workspaces && resolveWorkspaceImport(importPath, workspaces)) { + if (workspaces && resolveWorkspaceImport(importPath, workspaces, options.fromFile)) { return false; } @@ -1244,7 +1248,8 @@ const EMITTED_TO_SOURCE_EXTENSIONS: ReadonlyArray boolean = (file) => context.fileExists(file) ): string | null { if (!EMITTED_SPECIFIER_LANGUAGES.has(language)) return null; for (const [emitted, sources] of EMITTED_TO_SOURCE_EXTENSIONS) { @@ -1252,7 +1257,7 @@ function findSourceForEmittedSpecifier( const stem = relativePath.replace(emitted, ''); for (const ext of sources) { const candidate = stem + ext; - if (context.fileExists(candidate)) return candidate; + if (exists(candidate)) return candidate; } return null; } @@ -1297,16 +1302,23 @@ function resolveAliasedImport( } } - // 1.5 Workspace packages (`@scope/ui/widgets` → `packages/ui/widgets`). - // Resolves a monorepo member import to the member's directory; the - // extension/index permutations below then find its barrel (#629). + // 1.5 Workspace packages and path dependencies: the entry the package's + // manifest declares for the import (#2502), else the member's + // directory (`@scope/ui/widgets` → `packages/ui/widgets`), where the + // extension/index permutations find its barrel (#629). A path into the + // package's build output the index lacks (`@acme/utils/lib/path`) is + // read as the source it is built from (`src/path.ts`), as a declared + // entry is, before a file only a local build left on disk. const workspaces = context.getWorkspacePackages?.(); - if (workspaces) { - const base = resolveWorkspaceImport(importPath, workspaces); - if (base) { - const hit = tryWithExt(base); - if (hit) return hit; - } + const targets = workspaces ? workspaceImportTargets(importPath, workspaces, fromFile) : null; + if (targets) { + const declared = declaredWorkspaceEntry(targets, language, context); + if (declared) return declared; + const direct = tryWithExt(targets.directory); + if (direct && (context.isIndexedFile?.(direct) ?? true)) return direct; + const built = builtSourceOf(targets.directory, targets.packageDir, language, context); + if (built) return built; + if (direct) return direct; } // 2. Hard-coded fallback list. Kept for projects that use these @@ -1364,6 +1376,80 @@ function indexedSpelling(filePath: string, language: Language): string { : filePath; } +/** + * The file a package's declared entry for an import lands on (#2502): the + * first declared target, in order (`exports` by condition, then the root's + * `main`-style fields), that the index holds, or that is build output the + * index lacks (`dist/index.js`), read as the source it is built from + * (`src/index.ts`). So `import`'s `./build/index.js` leads to `src/index.ts` + * before a custom condition's `./src/browser.ts` does. Null when none holds + * — the caller then tries the package's directory, as it did before entries + * were read. + */ +function declaredWorkspaceEntry(targets: WorkspaceImportTargets, language: Language, context: ResolutionContext): string | null { + for (const target of targets.declared) { + const hit = indexedTarget(target, language, context) ?? builtSourceOf(target, targets.packageDir, language, context); + if (hit) return hit; + } + return null; +} + +/** `target`, build output the index lacks, as the indexed source it is built from (see builtSourcesOf); null when there is none. */ +function builtSourceOf(target: string, packageDir: string, language: Language, context: ResolutionContext): string | null { + for (const source of builtSourcesOf(target, packageDir)) { + const hit = indexedTarget(source, language, context); + if (hit) return hit; + } + return null; +} + +/** + * `target` as a file the index holds: as written, with the language's + * extensions or index files, or as the source of an emitted `.js`. Only the + * index counts: build output a developer has built locally is on disk, but + * nothing links to it. + */ +function indexedTarget(target: string, language: Language, context: ResolutionContext): string | null { + const indexed = (file: string): boolean => context.isIndexedFile?.(file) ?? context.fileExists(file); + const asIs = indexedSpelling(target, language); + if (indexed(asIs)) return asIs; + for (const ext of EXTENSION_RESOLUTION[language] || []) { + const candidate = indexedSpelling(target + ext, language); + if (indexed(candidate)) return candidate; + } + return findSourceForEmittedSpecifier(asIs, language, context, indexed); +} + +/** A package's build folder: `dist/`, `build/`, `lib/`, `out/`, `es/`, `esm/`, `cjs/`. */ +const BUILD_ROOT = /^(?:dist|build|lib|out|output|es|esm|cjs|umd)\//; +/** The format subfolder a bundler adds below it (`dist/esm/`, `dist/types/`), or the source folder tsc mirrors (`dist/src/`). */ +const BUILD_FORMAT = /^(?:esm|cjs|es|mjs|module|modules|commonjs|types|typings|umd|node|browser|src)\//; +/** An emitted file's extension, declaration files included. */ +const EMITTED_EXTENSION = /(?:\.d)?\.(?:[cm]?[jt]s|[jt]sx)$/; + +/** + * The source files a package's build output may be built from, + * extension-less, most literal first: the build folder mirrored under + * `src/` (vite's `dist/node/index.js` → `src/node/index`), then without a + * format subfolder (`dist/esm/index.mjs`, `dist/types/index.d.ts` → + * `src/index`). The caller links one only if the index holds it, so a + * mapping that is not clear links nothing. Empty for a target outside a + * build folder. + */ +function builtSourcesOf(target: string, packageDir: string): string[] { + const prefix = packageDir === '' ? '' : packageDir + '/'; + if (!target.startsWith(prefix)) return []; + const inPackage = target.slice(prefix.length); + const root = BUILD_ROOT.exec(inPackage); + if (!root) return []; + const mirrored = inPackage.slice(root[0].length); + const format = BUILD_FORMAT.exec(mirrored); + const stems = [mirrored, ...(format ? [mirrored.slice(format[0].length)] : [])] + .map((rest) => rest.replace(EMITTED_EXTENSION, '')) + .filter((stem) => stem !== '' && !stem.startsWith('src/')); + return [...new Set(stems)].map((stem) => `${prefix}src/${stem}`); +} + /** * A bare specifier `isExternalImport` takes for a package, resolved through * the importing app's own tsconfig / jsconfig — the one nearest the file, @@ -4242,7 +4328,7 @@ export function isBoundToOutOfRepoImport( if (imp.localName !== name) continue; // A package the repository declares itself (`declare module 'x'`) has // its declarations in the repository. - return isExternalImport(imp.source, ref.language, context) && + return isExternalImport(imp.source, ref.language, context, { fromFile: ref.filePath }) && resolveImportPath(imp.source, ref.filePath, ref.language, context) === null && !ambientModuleDeclares(imp.source, imp.exportedName, imp.isDefault, ref, context); } @@ -4315,7 +4401,7 @@ export function importBindingTail(ref: UnresolvedRef, context: ResolutionContext const binding = mappings.find((m) => m.localName === name) ?? (receiver && ref.referenceKind === 'calls' ? localTypeBinding(receiver, ref, context, mappings) : undefined); if (!binding) return undefined; - if (isExternalImport(binding.source, ref.language, context) && + if (isExternalImport(binding.source, ref.language, context, { fromFile: ref.filePath }) && // A monorepo app's own tsconfig alias (`#/views/…`): isExternalImport reads the root's only. !context.getNearestAliases?.(ref.filePath)?.patterns.some((p) => p.prefix !== '' && binding.source.startsWith(p.prefix)) && // A module the app's `baseUrl` or catch-all `paths` reads (`layout/MainLayout`). diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 06d435e490..4093a92d96 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -45,7 +45,7 @@ import { createYielder, type MaybeYield } from './cooperative-yield'; import { MAX_SOURCE_FILE_SIZE_BYTES } from '../file-limits'; import { loadProjectAliases, type AliasMap } from './path-aliases'; import { findGoModuleForImport, goModulePackageDir, loadGoModule, type GoModule } from './go-module'; -import { loadWorkspacePackages, type WorkspacePackages } from './workspace-packages'; +import { loadWorkspacePackages, workspaceFingerprint, workspacePackageDirOf, type WorkspacePackages } from './workspace-packages'; import { logDebug } from '../errors'; import { stripCommentsForRegex } from './strip-comments'; import { lexicalPathWithinRoot } from '../utils'; @@ -365,6 +365,9 @@ export class ReferenceResolver { * Initialize the resolver (detect frameworks, etc.) */ initialize(): void { + // A full index resolves by the package.json files on disk now, as the + // fingerprint it stamps says, even in a process that read them earlier. + this.workspacePackages = undefined; this.frameworks = detectFrameworks(this.context); this.frameworksByLanguage.clear(); this.clearCaches(); @@ -706,7 +709,7 @@ export class ReferenceResolver { // `"lit/decorators": ["./node_modules/lit/decorators.js"]` lands in // `node_modules`, topcoder's `config` package on its `config/` folder. isOutOfRepoImport: (source, fromFile, language) => { - if (!isExternalImport(source, language, this.context, { aliasPrefixes: false })) return false; + if (!isExternalImport(source, language, this.context, { aliasPrefixes: false, fromFile })) return false; const resolved = resolveImportPath(source, fromFile, language, this.context); if (resolved !== null && this.isIndexedFile(resolved)) return false; return this.isDeclaredOutsidePackage(source, fromFile); @@ -2850,6 +2853,35 @@ export class ReferenceResolver { return pythonRootFingerprint(this.diskContext()); } + /** + * The workspace packages and path dependencies as package.json files + * declare them now, read again from disk, and their fingerprint (see + * `workspaceFingerprint`). The resolver keeps the map for its lifetime, so + * a sync calls this to see a manifest edit; when the map moved, the + * resolver takes the new one and drops every memo resolved through the old + * one, and the caller re-opens what was linked by it. + */ + refreshWorkspacePackages(): string { + const fresh = loadWorkspacePackages(this.projectRoot); + const fingerprint = workspaceFingerprint(fresh); + if (this.workspacePackages === undefined || workspaceFingerprint(this.workspacePackages) !== fingerprint) { + this.workspacePackages = fresh; + this.clearCaches(); + } + return fingerprint; + } + + /** The fingerprint of the workspace map this resolver resolves by (see `workspaceFingerprint`). */ + getWorkspaceFingerprint(): string { + return workspaceFingerprint(this.context.getWorkspacePackages?.() ?? null); + } + + /** The folder of the workspace package or path dependency `file` is in (see `workspacePackageDirOf`), or null. */ + workspacePackageDirOf(file: string): string | null { + const ws = this.context.getWorkspacePackages?.(); + return ws ? workspacePackageDirOf(ws, file) : null; + } + /** What a sync must re-open after these Python files were added or removed (see `pythonReopenScope`). */ getPythonReopenScope(changedFiles: string[]): ReturnType { return pythonReopenScope(this.diskContext(), changedFiles); diff --git a/src/resolution/module-importers.ts b/src/resolution/module-importers.ts index e1ff0a6e3c..f6a82da321 100644 --- a/src/resolution/module-importers.ts +++ b/src/resolution/module-importers.ts @@ -161,6 +161,9 @@ export function reopenReferencesThroughRemovedFiles( /** Resolves a module specifier written in `fromFile` as its import does: the project file it names, or null. */ export type ModuleResolver = (specifier: string, fromFile: string, language: Language) => string | null; +/** The folder of the project's own package (a workspace member, a path dependency) `file` is in, or null. */ +export type PackageDirOf = (file: string) => string | null; + /** * The other half: after a sync's retries, re-open the references the files * importing any of `added` make through those imports — the edges a name @@ -170,8 +173,10 @@ export type ModuleResolver = (specifier: string, fromFile: string, language: Lan * them. (The failed-import retry looks those up by the module's name, and * skips a name as common as `models` or `test`.) With `resolveModule`, also an * import that resolved to a file beside an added one that now outranks it - * (`./button` to `button.vue` while `button.ts` was missing), and what that - * statement binds. Returns the number re-opened. + * (`./button` to `button.vue` while `button.ts` was missing), or, with + * `packageDirOf`, to another file of the package an added file is in (see + * outrankedImports), and what that statement binds. Returns the number + * re-opened. */ export function reopenReferencesThroughAddedFiles( queries: QueryBuilder, @@ -179,8 +184,9 @@ export function reopenReferencesThroughAddedFiles( added: ReadonlyArray<{ path: string; language: Language }>, resolvedUpToEdgeId: number, resolveModule?: ModuleResolver, + packageDirOf?: PackageDirOf, ): number { - const outranked = resolveModule ? outrankedImports(queries, added, resolveModule) : []; + const outranked = resolveModule ? outrankedImports(queries, added, resolveModule, packageDirOf) : []; const bindings = bindingsOf(importersOf(queries, added, reader, outranked), reader); const failed = [...bindings].flatMap(([file, { modules, names }]) => queries.getFailedReferencesThrough(file, modules, names)); return reopenModuleEdges(queries, outranked) + @@ -430,17 +436,38 @@ interface OutrankedImport { * another extension of its name, or its folder's own entry — when the * import's specifier now names the added file: `./button` resolved to * `button.vue` while `button.ts` was missing, and `button.ts` comes first. + * With `packageDirOf`, also the imports of a package by its name into any + * file of the package an added file is in: the entry its manifest declares + * (`./src/index.ts`, or the source of `./dist/index.js`), or the source of a + * path into its build output, outranks the file an import fell back to while + * it was missing, such as the folder's own index (#2502). */ function outrankedImports( queries: QueryBuilder, added: ReadonlyArray<{ path: string; language: Language }>, resolveModule: ModuleResolver, + packageDirOf?: PackageDirOf, ): OutrankedImport[] { const out: OutrankedImport[] = []; + const packageImports = new Map>(); for (const file of added) { if (!BINDING_LANGUAGES.has(file.language)) continue; const { dir, name } = importedName(file.path); - for (const edge of queries.getModuleImportsNamed(dir, name)) { + const edges = queries.getModuleImportsNamed(dir, name); + const packageDir = packageDirOf?.(file.path); + if (packageDir) { + let imports = packageImports.get(packageDir); + if (!imports) { + imports = queries.getModuleImportsUnder(packageDir).filter((edge) => { + const specifier = edge.metadata?.refName; + return typeof specifier === 'string' && !specifier.startsWith('.') && !specifier.startsWith('/'); + }); + packageImports.set(packageDir, imports); + } + const seen = new Set(edges.map((edge) => edge.edgeId)); + for (const edge of imports) if (!seen.has(edge.edgeId)) edges.push(edge); + } + for (const edge of edges) { const specifier = edge.metadata?.refName; if (edge.targetFilePath === file.path || typeof specifier !== 'string') continue; if (resolveModule(specifier, edge.sourceFilePath, edge.sourceLanguage) === file.path) { diff --git a/src/resolution/workspace-packages.ts b/src/resolution/workspace-packages.ts index 0d1e2de8ea..5594de535f 100644 --- a/src/resolution/workspace-packages.ts +++ b/src/resolution/workspace-packages.ts @@ -11,27 +11,37 @@ * `0 callers` on a live component (issue #629). * * This module maps each member package's declared `name` to its - * directory so the resolver can rewrite `@scope/ui/widgets` → - * `packages/ui/widgets` and then run normal extension/index resolution. + * directory, and its package.json's declared entries (`exports`, `main`, + * `module`, `types`) to files, so the resolver can rewrite `@scope/ui` → + * `packages/ui/src/index.ts` and `@scope/ui/widgets` → the file its + * `exports` names, or `packages/ui/widgets` with normal extension/index + * resolution when the manifest names none (#2502). * - * Scope deliberately small for v1 (mirrors path-aliases.ts): * - reads `workspaces` (array OR `{ packages: [...] }`) from package.json, * plus a minimal `pnpm-workspace.yaml` `packages:` list * - expands `*`, `?` and `{a,b}` within a folder name and `**` across * folders (`packages/*`, pnpm's nested `packages/**\/*`), and leaves out * what a negated glob (`!**\/example`) matches - * - subpath resolution is directory-based (`@scope/ui/sub` → `/sub`); - * it does NOT yet honour a member's `exports` map or `main` field - * - returns null when the project declares no workspaces, so single- + * - reads the sibling packages a package.json depends on by path + * (`"acme-shared": "file:../common"`, `link:`, `portal:`), which need no + * workspace declaration at all: the root manifest's for every file, any + * other's for the files under it, as npm installs a package's + * dependencies for that package (#2456) + * - returns null when the project declares none of these, so single- * package repos pay nothing and see no behaviour change. */ +import { createHash } from 'crypto'; import * as fs from 'fs'; import * as path from 'path'; import { logDebug } from '../errors'; export interface WorkspacePackages { - /** Member package `name` → directory relative to projectRoot (posix). */ + /** + * Package `name` → directory relative to projectRoot (posix): the workspace + * members, ohpm's `file:` dependencies, and the root package.json's path + * dependencies. + */ byName: Map; /** * Member package `name` → its declared ENTRY FILE relative to projectRoot @@ -40,15 +50,64 @@ export interface WorkspacePackages { * the member's real barrel even when it doesn't follow an index-file * convention — and independent of the CONSUMER's language (a `.ts` file * importing an `.ets` barrel resolves without `.ets` in the TS candidate - * list). Absent for npm/pnpm members (their index conventions cover it). + * list). npm/pnpm packages use `entriesByDir`. */ entryByName?: Map; + /** + * The entries an npm/pnpm/yarn/bun package's package.json declares, by the + * package's directory (see PackageEntries). + */ + entriesByDir?: Map; + /** + * The path dependencies a package.json below the root declares: its + * directory → dependency name → target directory (all projectRoot-relative + * posix). Seen only from the files under that manifest — a test fixture's + * `"left-pad": "file:../stub"` is the fixture's own, and `left-pad` stays the + * npm package for everyone else. The root manifest's are in `byName`. + */ + scopedByName?: Map>; +} + +/** + * What a package.json declares a package's importers reach, as targets + * relative to projectRoot (posix), best candidate first. A target may name + * build output the index does not hold; the resolver then reads the source it + * is built from, where that is clear. + */ +export interface PackageEntries { + /** + * The `exports` map: `.`, `./utils`, `./schema/*` → targets. A string, an + * array or a conditions-only `exports` is the `.` entry. A key mapped to + * `null` (not exported) holds no target. + */ + exports?: Map; + /** + * The package root's `source`, `module`, `main`, `types` and `typings` + * fields, read after `exports` (as TypeScript's older `node` resolution + * reads them in its place). + */ + main?: string[]; +} + +/** Where an import of a workspace or path-dependency package lands, before any file is probed. */ +export interface WorkspaceImportTargets { + /** + * The entries the package declares for the import, best first: what + * `exports` names for it in condition order, then, for the package root, + * its `main`-style fields. + */ + declared: string[]; + /** The package's directory plus the import's subpath (`packages/ui/widgets`), extension-less. */ + directory: string; + /** The package's directory. */ + packageDir: string; } /** * Load workspace member packages for `projectRoot`. Returns `null` when - * the project declares no workspaces (the common single-package case) — - * callers then skip all workspace logic. + * the project declares no workspaces and no package depends on another by + * path (the common single-package case) — callers then skip all workspace + * logic. * * Cheap to call repeatedly only via the resolver's per-instance cache; * this function itself touches the filesystem, so the resolver memoises it @@ -75,19 +134,105 @@ export function loadWorkspacePackages(projectRoot: string): WorkspacePackages | // oh-package.json5 declares its local siblings as `"data": "file:../../ // core/data"` dependencies, and code then imports the bare name // (`import { CartRepository } from "data"`). Same monorepo problem as npm - // workspaces, different manifest. + // workspaces, different manifest. An npm package does the same with + // `"acme-shared": "file:../common"` and no root `workspaces` (#2456). + const { ohpm, npm } = collectPathDeps(projectRoot); const entryByName = new Map(); - for (const [name, dir] of collectOhpmFileDeps(projectRoot)) { + for (const [name, dir] of ohpm) { if (byName.has(name)) continue; byName.set(name, dir); const entry = readOhpmMain(projectRoot, dir); if (entry) entryByName.set(name, entry); } + for (const [name, dir] of npm.get('') ?? []) { + if (!byName.has(name)) byName.set(name, dir); + } + npm.delete(''); + + if (byName.size === 0 && npm.size === 0) return null; - if (byName.size === 0) return null; + // What each npm-style package's manifest declares for its importers. + const entriesByDir = new Map(); + const packageDirs = new Set(byName.values()); + for (const names of npm.values()) for (const dir of names.values()) packageDirs.add(dir); + for (const dir of packageDirs) { + const entries = readPackageEntries(projectRoot, dir); + if (entries) entriesByDir.set(dir, entries); + } - logDebug('workspace packages loaded', { count: byName.size }); - return { byName, entryByName: entryByName.size > 0 ? entryByName : undefined }; + logDebug('workspace packages loaded', { count: byName.size, scoped: npm.size, entries: entriesByDir.size }); + return { + byName, + entryByName: entryByName.size > 0 ? entryByName : undefined, + entriesByDir: entriesByDir.size > 0 ? entriesByDir : undefined, + scopedByName: npm.size > 0 ? npm : undefined, + }; +} + +/** + * Condition keys an `exports` target is read through first: the ones + * TypeScript and the bundlers resolve by. Any other condition (`browser`, + * `development`, a custom `@acme/source`) comes next, and `types`/`typings` + * last — a declaration file is the package as tsc reads it, the + * implementation is what its importers run. + */ +const RESOLVING_CONDITIONS = new Set(['import', 'require', 'node', 'module', 'module-sync', 'default']); +const TYPE_CONDITIONS = new Set(['types', 'typings']); + +/** An `exports` value's targets, best first: a string, a fallback array, or (nested) conditions, by key order within each tier. */ +function exportTargets(value: unknown): string[] { + if (typeof value === 'string') return [value]; + if (Array.isArray(value)) return value.flatMap(exportTargets); + if (!value || typeof value !== 'object') return []; + const conditions = Object.entries(value as Record); + const tier = (key: string): number => (RESOLVING_CONDITIONS.has(key) ? 0 : TYPE_CONDITIONS.has(key) ? 2 : 1); + return [0, 1, 2].flatMap((t) => conditions.filter(([key]) => tier(key) === t).flatMap(([, v]) => exportTargets(v))); +} + +/** The package.json fields an importer of the package root reaches without `exports`, best first. */ +const MAIN_FIELDS = ['source', 'module', 'main', 'types', 'typings']; + +/** Read a package's declared entries (see PackageEntries); null when its manifest declares none. */ +function readPackageEntries(projectRoot: string, dirRel: string): PackageEntries | null { + let pkg: Record | null; + try { + pkg = JSON.parse(fs.readFileSync(path.join(projectRoot, dirRel, 'package.json'), 'utf-8')) as Record | null; + } catch { + return null; + } + if (!pkg || typeof pkg !== 'object') return null; + const inPackage = (target: string): string | null => packageTarget(projectRoot, dirRel, target); + const out: PackageEntries = {}; + const exp = pkg.exports; + if (exp !== undefined && exp !== null) { + const exports = new Map(); + const subpaths = typeof exp === 'object' && !Array.isArray(exp) && Object.keys(exp).some((key) => key.startsWith('.')); + const entries: Array<[string, unknown]> = subpaths + ? Object.entries(exp as Record).filter(([key]) => key.startsWith('.')) + : [['.', exp]]; + for (const [key, value] of entries) { + exports.set(key, [...new Set(exportTargets(value).map(inPackage).filter((t): t is string => t !== null))]); + } + out.exports = exports; + } + const main = MAIN_FIELDS + .map((field) => pkg![field]) + .filter((value): value is string => typeof value === 'string') + .map(inPackage) + .filter((t): t is string => t !== null); + if (main.length > 0) out.main = [...new Set(main)]; + return out.exports || out.main ? out : null; +} + +/** A package-relative target (`./src/index.ts`) → projectRoot-relative posix, or null if it leaves the package. */ +function packageTarget(projectRoot: string, dirRel: string, target: string): string | null { + const trimmed = target.trim(); + if (!trimmed) return null; + const packageDir = path.resolve(projectRoot, dirRel); + const abs = path.resolve(packageDir, trimmed); + const inside = path.relative(packageDir, abs); + if (inside.startsWith('..') || path.isAbsolute(inside)) return null; + return path.relative(projectRoot, abs).replace(/\\/g, '/'); } /** @@ -124,11 +269,18 @@ function readOhpmMain(projectRoot: string, dirRel: string): string | null { * cross-module link. Registry dependencies (`@ohos/axios: "^2.0.0"`) don't * use `file:` and are ignored, staying external. * + * The same walk reads every package.json's `file:`, `link:` and `portal:` + * dependencies (#2456), by the directory of the manifest that declares them + * (`''` for the root): npm installs a package's dependencies for that package, + * so the files under it are the ones that see them. Only a directory of the + * project counts, other than the root and nothing under `node_modules` — a + * tarball or a path into installed packages maps nothing. + * * The walk is bounded (depth + directory budget) and prunes build/dependency - * dirs, so non-ArkTS projects pay one readdir at the root and nothing else - * (they have no oh-package.json5 anywhere shallow). + * dirs; a manifest is parsed only where the walk finds one. */ const OHPM_MANIFEST = 'oh-package.json5'; +const NPM_MANIFEST = 'package.json'; const OHPM_WALK_MAX_DEPTH = 6; const OHPM_WALK_DIR_BUDGET = 8000; const OHPM_SKIP_DIRS = new Set([ @@ -136,9 +288,10 @@ const OHPM_SKIP_DIRS = new Set([ 'build', 'dist', 'out', 'oh-package-lock.json5', ]); -function collectOhpmFileDeps(projectRoot: string): Map { - const byName = new Map(); +function collectPathDeps(projectRoot: string): { ohpm: Map; npm: Map> } { + const ohpm = new Map(); const ambiguous = new Set(); + const npm = new Map>(); const queue: Array<{ rel: string; depth: number }> = [{ rel: '', depth: 0 }]; let visited = 0; @@ -161,6 +314,18 @@ function collectOhpmFileDeps(projectRoot: string): Map { queue.push({ rel: rel ? `${rel}/${e.name}` : e.name, depth: depth + 1 }); continue; } + if (e.name === NPM_MANIFEST) { + const deps = new Map(); + for (const [name, target] of readNpmPathDeps(path.join(abs, e.name))) { + const targetAbs = path.resolve(abs, target); + const targetRel = path.relative(projectRoot, targetAbs).replace(/\\/g, '/'); + if (targetRel === '' || targetRel.startsWith('..') || path.isAbsolute(targetRel)) continue; + if (/(?:^|\/)node_modules(?:\/|$)/.test(targetRel) || !isDirectory(targetAbs)) continue; + deps.set(name, targetRel); + } + if (deps.size > 0) npm.set(rel, deps); + continue; + } if (e.name !== OHPM_MANIFEST) continue; const deps = readOhpmFileDeps(path.join(abs, e.name)); @@ -168,18 +333,47 @@ function collectOhpmFileDeps(projectRoot: string): Map { const targetAbs = path.resolve(abs, target); const targetRel = path.relative(projectRoot, targetAbs).replace(/\\/g, '/'); if (targetRel.startsWith('..')) continue; // escapes the project - const existing = byName.get(name); + const existing = ohpm.get(name); if (existing === undefined) { - if (!ambiguous.has(name)) byName.set(name, targetRel); + if (!ambiguous.has(name)) ohpm.set(name, targetRel); } else if (existing !== targetRel) { - byName.delete(name); + ohpm.delete(name); ambiguous.add(name); } } } } - return byName; + return { ohpm, npm }; +} + +function isDirectory(abs: string): boolean { + try { + return fs.statSync(abs).isDirectory(); + } catch { + return false; + } +} + +/** A package.json's dependencies on a path (`file:`, `link:`, `portal:`), from every dependency field → [name, target]. */ +function readNpmPathDeps(manifestAbs: string): Array<[string, string]> { + let pkg: Record | null; + try { + pkg = JSON.parse(fs.readFileSync(manifestAbs, 'utf-8')) as Record | null; + } catch { + return []; + } + if (!pkg || typeof pkg !== 'object') return []; + const out: Array<[string, string]> = []; + for (const field of ['dependencies', 'devDependencies', 'optionalDependencies', 'peerDependencies']) { + const deps = pkg[field]; + if (!deps || typeof deps !== 'object') continue; + for (const [name, value] of Object.entries(deps as Record)) { + const m = typeof value === 'string' ? /^(?:file|link|portal):(.+)$/.exec(value.trim()) : null; + if (m && m[1]!.trim()) out.push([name, m[1]!.trim()]); + } + } + return out; } /** Parse one oh-package.json5's dependencies → [name, file-target] pairs. */ @@ -208,32 +402,145 @@ function readOhpmFileDeps(manifestAbs: string): Array<[string, string]> { * Rewrite a bare workspace import to a path relative to projectRoot, * WITHOUT an extension — the caller applies the language's extension/index * resolution. `@scope/ui/widgets` → `packages/ui/widgets`; the bare package - * name `@scope/ui` → its directory. Returns `null` when no member package - * name matches. + * name `@scope/ui` → its directory, or the ohpm entry its manifest names. + * `fromFile`, the importing file, sees the path dependencies the manifests + * above it declare. Returns `null` when no package name matches. */ export function resolveWorkspaceImport( importPath: string, - ws: WorkspacePackages + ws: WorkspacePackages, + fromFile?: string ): string | null { - // Longest matching package name wins, so `@scope/ui/core` prefers a - // `@scope/ui/core` package over a `@scope/ui` one when both exist. - let bestName: string | null = null; - for (const name of ws.byName.keys()) { - if (importPath === name || importPath.startsWith(name + '/')) { - if (!bestName || name.length > bestName.length) bestName = name; - } - } - if (!bestName) return null; - const dir = ws.byName.get(bestName)!; - const subpath = importPath.slice(bestName.length); // '' or '/widgets' + const hit = matchPackage(importPath, ws, fromFile); + if (!hit) return null; + const subpath = importPath.slice(hit.name.length); // '' or '/widgets' // A bare member import resolves straight to the member's declared entry - // file when the manifest names one (ohpm `main`) — the caller's exact-path - // check hits it without extension/index guessing. - if (!subpath) { - const entry = ws.entryByName?.get(bestName); - if (entry) return entry; + // file when the ohpm manifest names one — the caller's exact-path check + // hits it without extension/index guessing. + const ohpm = subpath ? undefined : ws.entryByName?.get(hit.name); + return ohpm ?? (hit.dir + subpath).replace(/\/{2,}/g, '/'); +} + +/** + * Where an import of a package of the project lands (see + * WorkspaceImportTargets): what its manifest declares for the import, and + * the directory form. Null when no package name matches. + */ +export function workspaceImportTargets( + importPath: string, + ws: WorkspacePackages, + fromFile?: string +): WorkspaceImportTargets | null { + const hit = matchPackage(importPath, ws, fromFile); + if (!hit) return null; + const subpath = importPath.slice(hit.name.length); // '' or '/widgets' + const directory = (hit.dir + subpath).replace(/\/{2,}/g, '/'); + const ohpm = subpath ? undefined : ws.entryByName?.get(hit.name); + const declared = ohpm ? [ohpm] : declaredEntries(ws.entriesByDir?.get(hit.dir), subpath); + return { declared, directory, packageDir: hit.dir }; +} + +/** + * The package an import names: a path dependency of the nearest manifest + * above `fromFile` that declares a matching name, else a workspace member or + * a root path dependency. The longest name wins, so `@scope/ui/core` prefers + * a `@scope/ui/core` package over a `@scope/ui` one when both exist. + */ +function matchPackage(importPath: string, ws: WorkspacePackages, fromFile?: string): { name: string; dir: string } | null { + const scoped = scopedPackage(importPath, ws, fromFile); + if (scoped) return scoped; + const name = longestName(importPath, ws.byName.keys()); + return name === null ? null : { name, dir: ws.byName.get(name)! }; +} + +/** The path dependency the nearest declaring manifest above `fromFile` maps `importPath` to. */ +function scopedPackage(importPath: string, ws: WorkspacePackages, fromFile?: string): { name: string; dir: string } | null { + if (!fromFile || !ws.scopedByName) return null; + let dir = fromFile.replace(/\\/g, '/'); + while (dir !== '') { + const slash = dir.lastIndexOf('/'); + dir = slash < 0 ? '' : dir.slice(0, slash); + const names = ws.scopedByName.get(dir); + if (!names) continue; + const name = longestName(importPath, names.keys()); + if (name !== null) return { name, dir: names.get(name)! }; + } + return null; +} + +function longestName(importPath: string, names: Iterable): string | null { + let best: string | null = null; + for (const name of names) { + if ((importPath === name || importPath.startsWith(name + '/')) && (best === null || name.length > best.length)) best = name; + } + return best; +} + +/** + * The targets a package's manifest declares for `subpath` (`''` or + * `/widgets`), best first: its `exports` entry, then, for the package root, + * its `main`-style fields. + */ +function declaredEntries(entries: PackageEntries | undefined, subpath: string): string[] { + if (!entries) return []; + const exported = entries.exports ? exportedTargets(entries.exports, '.' + subpath) : []; + return subpath ? exported : [...new Set([...exported, ...(entries.main ?? [])])]; +} + +/** `exports`' targets for `key`: the exact key's, else the `*` pattern's with the longest prefix, its `*`s filled in. */ +function exportedTargets(exports: Map, key: string): string[] { + const exact = exports.get(key); + if (exact) return exact; + let best: { prefix: string; match: string; targets: string[] } | null = null; + for (const [pattern, targets] of exports) { + const star = pattern.indexOf('*'); + if (star < 0 || pattern.indexOf('*', star + 1) >= 0) continue; + const prefix = pattern.slice(0, star); + const suffix = pattern.slice(star + 1); + if (key.length < prefix.length + suffix.length || !key.startsWith(prefix) || !key.endsWith(suffix)) continue; + if (best && prefix.length <= best.prefix.length) continue; + best = { prefix, match: key.slice(prefix.length, key.length - suffix.length), targets }; + } + return best ? best.targets.map((t) => t.split('*').join(best!.match)) : []; +} + +/** + * The workspace map as a stable string: every name, scope, directory and + * declared entry an import can be resolved by. Two loads that agree on it + * resolve every package import the same way; sync compares it with the one + * the index was built with to see a package.json edit (#2456, #2502). + */ +export function workspaceFingerprint(ws: WorkspacePackages | null): string { + if (!ws) return 'none'; + const sorted = (map: Map | undefined): Array<[string, V]> => + map ? [...map.entries()].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) : []; + const canonical = JSON.stringify({ + byName: sorted(ws.byName), + entryByName: sorted(ws.entryByName), + scoped: sorted(ws.scopedByName).map(([dir, names]) => [dir, sorted(names)]), + entries: sorted(ws.entriesByDir).map(([dir, e]) => [dir, sorted(e.exports), e.main ?? []]), + }); + return createHash('sha256').update(canonical).digest('hex'); +} + +/** Each map's package folders, outermost first. */ +const PACKAGE_DIRS = new WeakMap(); + +/** + * The folder of the package `file` is in — a workspace member, or a path + * dependency's target — the outermost when packages nest, or null. Every + * import of a package by its name lands in its folder, inner packages' + * files included. + */ +export function workspacePackageDirOf(ws: WorkspacePackages, file: string): string | null { + let dirs = PACKAGE_DIRS.get(ws); + if (!dirs) { + const all = new Set(ws.byName.values()); + for (const names of ws.scopedByName?.values() ?? []) for (const dir of names.values()) all.add(dir); + dirs = [...all].filter((dir) => dir !== '').sort((a, b) => a.length - b.length); + PACKAGE_DIRS.set(ws, dirs); } - return (dir + subpath).replace(/\/{2,}/g, '/'); + return dirs.find((dir) => file.startsWith(dir + '/')) ?? null; } /** Read workspace glob patterns from package.json + pnpm-workspace.yaml. */ From 5d81d292fa60562d379f08e62514a2269c9a102a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 13:49:07 +0000 Subject: [PATCH 35/95] fix(js,ts): a call or new never lands on an interface or a type alias (#2801) A call or `new` that a strategy resolved by name or through an import could target a TypeScript interface or type alias, which nothing can call or make an instance of. memfs's fs-core `result.ts` declares `interface Ok` beside `function Ok` (and `Err` beside `Err`), and `Ok(...)` through an import reached the interface. A `new` of a global that a project only declares a type for (`new Response()`, `new Worker()`, `new Event()`) was linked by name to that type, and jest's `write()` parameter call to a `type write`. Root cause: no rule on what a call may target. Whichever strategy found the name picked among same-named nodes by order, and nothing stopped it on a type. Fix (gateTargetKind): for the JS family, a `calls` or `instantiates` result whose target is an `interface` or a `type_alias` moves to the value the target's file declares under the same name (a function, class, component, constant or variable, first by position): `function Ok` beside `interface Ok`, a class merged with an interface. With no such value the result is dropped, and the reference stays unresolved. Validation: - __tests__/call-value-targets.test.ts (3 tests): all 3 fail on main and pass on the kernel. Each of 5 rules removed in turn fails at least one. - Sync equals a fresh index (CLI) when the value beside a type is added or removed, and when a project type for a global or a parameter appears. - A/B on 14 TS/JS repos and 2 controls: memfs's 11 Ok/Err calls move from the interfaces to the functions; 165 `new` of globals dropped (vite 27, nuxt 57, webdriverio 26, insomnia 47, jest 7, memfs 1) and jest's `write()` call on `type write`. Every dropped `new` names a global (Worker, Event, Response, Request, Headers, ReadableStream, Node's Module, mocha's Context/Suite) or a local variable. vben, outline, social-app, t3-turbo, nitro, vue3-antdv-admin, pathe, c12, flask and gin are byte-identical. - 91 related test files pass (2,178 tests). - Cost: fresh index within noise (median of 3, interleaved): vite 9.07 s to 8.74 s, nuxt 11.71 s to 10.93 s, insomnia 15.26 s to 15.11 s. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/call-value-targets.test.ts | 98 ++++++++++++++++++++++++++++ src/resolution/index.ts | 31 +++++++++ 3 files changed, 130 insertions(+) create mode 100644 __tests__/call-value-targets.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index be4d0d65d1..87850b3b93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -224,6 +224,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In JavaScript and TypeScript, a function a file loads with a dynamic import from Node or a package, like `const { createServer } = await import('node:http')`, `const { default: chalk } = await import('chalk')` or `import('node:fs/promises').then(({ readFile }) => …)`, no longer links to a project function that only shares its name. Before, vite's call to Node's `createServer` showed up among the callers of vite's own `createServer`. The import counts only where it is in scope, so the rest of the file links as before, and a dynamic import of your own project's module still leads to that module. Thanks @Ancienttwo for the report. (#2491) - In JavaScript and TypeScript, code outside your tests no longer links to a Jest or Vitest manual mock under `__mocks__/`. Before, when an import of a workspace package couldn't be followed, a call like `formatLabel(name)` linked by name to the mock beside it instead of the package's own function, so the real function lost its callers, and the same happened to rendered components and event handlers. Production code is now matched as if the mocks weren't there, so it reaches the real function, and adding or deleting a mock leaves it alone. Tests, other mocks, and a file that imports a mock by its path, like a `jest.setup.ts`, still reach the mock. Thanks @Y-Osawa2022 for the report and @danusha2345. (#2455) - In JavaScript and TypeScript, an import of a package from your own repository now leads to the file its `package.json` names: the `exports` entry for the import, conditions and subpaths like `"./utils": "./src/utils.ts"` included, or else its `main`, `module`, `types` or `source`. A package another one depends on by path, like `"acme-shared": "file:../common"` or `link:`, now counts as part of your project too, for the files of the package that declares it, even with no root `workspaces`. Before, an import of a pnpm workspace member declared with `"main": "./src/index.ts"` and no `index` file at its root, or of a `file:` dependency, couldn't be followed, so its calls were matched by name alone and a same-named private function nearer the caller could take them. An entry that names build output, like `./dist/esm/index.mjs`, or an import of a path inside it, like `@jsonjoy.com/fs-node-utils/lib/types/misc`, leads to the matching source file, like `src/index.ts` or `src/types/misc.ts`, when your project has one, and to nothing otherwise. `codegraph sync` follows edits to these `package.json` fields. Re-index JavaScript and TypeScript monorepos after upgrading. Thanks @elderbernardi and @Y-Osawa2022 for the reports, and @maxmilian and @danusha2345. (#2502, #2456) +- In JavaScript and TypeScript, a call or `new` no longer lands on an interface or a type alias. When a file declares both a type and a value of the same name, like `interface Ok` and `function Ok`, or a class merged with an interface, the call reaches the value, and a global like `new Response()`, `new Worker()` or `new Event()` no longer links to a project type that only shares its name. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript and TypeScript, a value the file imports from your own project and passes along now links only to what it imports: a constant like `isCloudHosted`, a class like a `Star` model, a whole module (`import * as Slack from '../slack'`), or a stylesheet, image or GraphQL document (`import TCO21 from './themes/TCO21.scss'`) no longer links to a function elsewhere that only shares its name. Before, when the project had exactly one function of that name, such a value linked to it: outline's settings screens showed up among the callers of the server's `isCloudHosted()` check, and its stores among those of the `` icon, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A component that wraps the function declared beside it, like `export default styled(Disclosure)`, and a page loaded lazily, like `lazy(() => import('../pages/Home'))`, still lead to that function. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript, an import of a TypeScript module written without an extension, like `import AudioModal from './audio_modal'` in a `.jsx` file next to `audio_modal.tsx`, now links to that module, as it already did from a TypeScript file. Before, a `.js` or `.jsx` file's imports of TypeScript modules were never followed, so in a codebase moving to TypeScript they were matched by name instead: an import under another name linked nothing, and a name that repeats could link to the wrong file, so mastodon's `` and `` tags rendered other components of those names than the ones each file imports. A JavaScript module beside a TypeScript one of the same name still comes first. Re-index JavaScript projects after upgrading. - `codegraph_explore`'s Flow now goes through the implementation your query names when an interface method has several that lead to the same place. Before, it took whichever one came first in the index: asking about prometheus's `Engine.execEvalStmt Queryable.Querier fanout.Querier NewMergeQuerier` traced the call through the TSDB's `DB.Querier` and left the `fanout.Querier` you named off the Flow. Any two routes of the same length are now settled this way, in favor of the one that passes through more of the symbols you named. diff --git a/__tests__/call-value-targets.test.ts b/__tests__/call-value-targets.test.ts new file mode 100644 index 0000000000..587dece5be --- /dev/null +++ b/__tests__/call-value-targets.test.ts @@ -0,0 +1,98 @@ +/** + * A call or `new` written with a name that a TypeScript file declares both as + * a type and as a value (`interface Ok` beside `function Ok`, a class merged + * with an interface) reached the type, and a `new` of a value that only a + * project type shares its name with (`interface Response` for the global + * `Response`) was linked to that type. Nothing calls an interface or a type + * alias, or makes an instance of one: the call reaches the value its file + * declares under the name, or links nothing. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let roots: string[] = []; +let graphs: CodeGraph[] = []; + +afterEach(() => { + for (const graph of graphs) graph.close(); + graphs = []; + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); + roots = []; +}); + +async function index(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-call-value-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + const file = path.join(root, rel); + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, content); + } + const graph = await CodeGraph.init(root, { index: true }); + graphs.push(graph); + return graph; +} + +/** `kind name(kind)@file` of every edge of `kinds` out of `file`'s nodes. */ +function edgesFrom(graph: CodeGraph, file: string, kinds: string[]): string[] { + const out: string[] = []; + for (const node of graph.getNodesInFile(file)) { + for (const e of graph.getOutgoingEdges(node.id)) { + if (!kinds.includes(e.kind)) continue; + const target = graph.getNode(e.target)!; + out.push(`${e.kind} ${target.name}(${target.kind})@${target.filePath}`); + } + } + return out.sort(); +} + +describe('a call or `new` never lands on an interface or a type alias', () => { + /** memfs's fs-core/src/result.ts: each result type beside the function that makes one. */ + const result = + 'export interface Ok {\n ok: true;\n value: T;\n}\n\nexport interface Err {\n ok: false;\n err: E;\n}\n\n' + + 'export type Result = Ok | Err;\n\n' + + 'export function Ok(value: T): Ok {\n return { ok: true, value };\n}\n\n' + + 'export function Err(err: E): Err {\n return { ok: false, err };\n}\n'; + + it('reaches the value declared beside the type, directly and through a barrel', async () => { + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app' }), + 'lib/result.ts': result, + 'lib/index.ts': "export * from './result';\n", + 'direct.ts': + "import { Ok, Err } from './lib/result';\n\nexport const direct = (fail: boolean) => (fail ? Err('no') : Ok(1));\nexport const typed = (r: Ok) => r;\n", + 'barrel.ts': "import { Ok } from './lib';\n\nexport const barrel = () => Ok(2);\n", + }); + expect(edgesFrom(graph, 'direct.ts', ['calls'])).toEqual(['calls Err(function)@lib/result.ts', 'calls Ok(function)@lib/result.ts']); + expect(edgesFrom(graph, 'barrel.ts', ['calls'])).toEqual(['calls Ok(function)@lib/result.ts']); + // A type position still names the type. + expect(edgesFrom(graph, 'direct.ts', ['references'])).toContain('references Ok(interface)@lib/result.ts'); + }, 120_000); + + it('links nothing for a value only a project type answers', async () => { + // insomnia's Postman `interface Response`, vite's `interface Worker`, jest-circus's `type Event`: globals. + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app' }), + 'types/postman.ts': 'export interface Response {\n code: number;\n}\n\nexport type Event = { name: string };\n', + 'types/worker.d.ts': 'export interface Worker {\n postMessage(data: unknown): void;\n}\n', + 'fetch.ts': "export function respond(): unknown {\n return [new Response('ok'), new Event('ready'), new Worker('w.js')];\n}\n", + // jest's DefaultReporter: a `write` parameter beside a `type write` declared for it. + 'types/stream.ts': 'export type write = (chunk: string) => boolean;\n', + 'flush.ts': "export function flush(write: (chunk: string) => boolean): void {\n write('done');\n}\n", + }); + expect(edgesFrom(graph, 'fetch.ts', ['calls', 'instantiates'])).toEqual([]); + expect(edgesFrom(graph, 'flush.ts', ['calls'])).toEqual([]); + }, 120_000); + + it('reaches the class an interface merges with', async () => { + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app' }), + 'store.ts': 'export interface Store {\n size: number;\n}\n\nexport class Store {\n size = 0;\n}\n', + 'use.ts': "import { Store } from './store';\n\nexport const make = () => new Store();\n", + }); + expect(edgesFrom(graph, 'use.ts', ['instantiates'])).toEqual(['instantiates Store(class)@store.ts']); + }, 120_000); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 4093a92d96..33d9eb2cab 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -158,6 +158,9 @@ const GO_BUILT_INS = new Set([ /** What a Pascal typecast's name resolves to: `TFoo(x)` casts, it doesn't call (see createEdges). */ const PASCAL_CAST_TARGET_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface', 'enum', 'type_alias']); +/** The values a TypeScript type may share its name with in its file, which a call or `new` of the name means. */ +const TYPE_COMPANION_VALUE_KINDS: ReadonlySet = new Set(['function', 'class', 'component', 'constant', 'variable']); + const PASCAL_UNIT_PREFIXES = [ 'System.', 'Winapi.', 'Vcl.', 'Fmx.', 'Data.', 'Datasnap.', 'Soap.', 'Xml.', 'Web.', 'REST.', 'FireDAC.', 'IBX.', @@ -3707,6 +3710,21 @@ export class ReferenceResolver { !ref.referenceName.includes('.') && !isGoBareName(ref, this.context)) return null; } + // A TypeScript interface or type alias is no value: nothing calls one or + // makes an instance of it. A strategy that found one by name meant the + // value its file declares beside it (`interface Ok` and `function Ok` in a + // result module); with none there, the name is a value from elsewhere + // that the project only types (`new Response()`, `new Worker()`), and + // links nothing. + if ((ref.referenceKind === 'calls' || ref.referenceKind === 'instantiates') && ESM_IMPORT_LANGUAGES.has(ref.language)) { + const target = this.nodeById(result.targetNodeId); + if (target && (target.kind === 'interface' || target.kind === 'type_alias')) { + const value = this.sameNamedValueOfType(target); + if (!value) return null; + result = { ...result, targetNodeId: value.id }; + } + } + // An `imports` reference names something importable — never a member that // only exists inside a type, nor its own import statement. A module path // that names no file (`./missing`, `stdio.h`, `pkg.mod`, a C# `using` of @@ -3948,6 +3966,19 @@ export class ReferenceResolver { } /** The one supertype-kind node a TypeScript value shares its name and file with. */ + /** + * The value a file declares beside a type of the same name — `export + * function Ok()` beside `export interface Ok`, a class merged with an + * interface — which a call or `new` written with the name reaches. The + * first by position; null when the file declares none. + */ + private sameNamedValueOfType(type: Node): Node | null { + return this.context + .getNodesInFile(type.filePath) + .filter((n) => n.name === type.name && TYPE_COMPANION_VALUE_KINDS.has(n.kind)) + .sort((a, b) => a.startLine - b.startLine || a.startColumn - b.startColumn)[0] ?? null; + } + private sameNamedTypeOfValue(value: Node): Node | null { if (value.kind !== 'constant' && value.kind !== 'variable') return null; if (value.language !== 'typescript' && value.language !== 'tsx') return null; From d82a4adc9dea4bed3ed0fc3d8f93a3e7544c17ef Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 14:52:16 +0000 Subject: [PATCH 36/95] fix(js,ts): a call or new in a parameter's default value is the function's call (#2802) A default runs inside the function each time a call leaves the argument out, but both extractors pushed the function and walked only its body, so `function renderWithNote(note = buildNote())` recorded nothing for the call: no edge and no unresolved reference. Arrows, methods, constructor parameter properties, class-field arrows, object-literal methods and destructured parameters lost theirs the same way. Only an anonymous arrow inside a body was walked whole, params included. The TS extractor (visitParameterDefaults) and the kernel's TS/JS walker (visit_parameter_defaults) now hand each default value, and a computed key in a destructured parameter, to the body walker, with the scope the body gets on the stack: the function for a named function, method or component, the enclosing scope for an anonymous function or an object-literal method. It runs before the body walk at every site that walks only a body, including a class field's arrow, a forwardRef/memo render function and a styled .attrs() function. Patterns are recursed through; the names a parameter binds, its type annotation and a bodiless signature are not walked. TS/JS only: a Python default runs once, when the def does, and is left as it was. Validation: the new test fails 13 of 14 on main 5d81d292 and passes on both engines, LF and CRLF; the kernel parity fixtures gained a defaults section. A/B against main on 14 TS/JS repos: 118 edges added, none removed or retargeted, each a call or `new` written in a default (nuxt 45, mostly kit utilities' `nuxt: Nuxt = useNuxt()`, webdriverio 30, memfs 17, jest 7, vben 5, social-app 5, vite 4, outline 3, nitro 2; insomnia, t3-turbo, vue3-antdv-admin, pathe and c12 none). Seven ride resolution the body already gets: six Vue/Nuxt auto-import self-links and one member call on memfs's `export default createProcess()`. The flask and gin control dumps are byte-identical. kernel-parity.mjs over the 14 repos: 15,408 files byte-parity, 0 with diffs. Index time, median of 8 interleaved runs on vite, nuxt, webdriverio and insomnia: +1.6% total, within noise (the last 5 rounds alone: -1.7%). A CLI sync that adds defaults, adds a file and gives an unchanged file's default its callee matches a fresh index. Fixes #2588 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/fixtures/kernel-parity/torture.js | 11 + __tests__/fixtures/kernel-parity/torture.tsx | 15 ++ __tests__/param-default-calls.test.ts | 269 +++++++++++++++++++ codegraph-kernel/src/tsjs/extractors.rs | 83 +++++- src/extraction/tree-sitter.ts | 63 +++++ 6 files changed, 436 insertions(+), 6 deletions(-) create mode 100644 __tests__/param-default-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 87850b3b93..4e2fecdc17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes - In TypeScript, an awaited call with type arguments like `await fetchApi("/me")` or `await api.get(url)` now links to the function it calls. Before, the call was dropped, so typed fetch wrappers looked like they had no callers. (#2586) +- In JavaScript and TypeScript, a call or `new` in a parameter's default value, like `function renderWithNote(note = buildNote())`, a destructured `{ size = defaultSize() }` or `constructor(private svc = new Service())`, now counts as the function's own call. Before, it was dropped, so `buildNote` was missing that caller and impact on it came back incomplete. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and the pull request. (#2588) - In React, a component rendered at module scope, in a top-level constant or inside a test's `it(…)` callback, like `createRoot(el).render()` or `render()`, now counts as rendered there, so an app's entry reaches its root component and a component's tests show among its callers. (#2587) - In Python, `import a.b as X` written inside a function now links its calls like `X.f()`, as the same import at module level already did. Thanks @Egrn for the report. (#2600) - In Python, `from app import email` no longer links the importing file to a project variable named `app`, such as a FastAPI or Flask `app = FastAPI()`. (#2584) diff --git a/__tests__/fixtures/kernel-parity/torture.js b/__tests__/fixtures/kernel-parity/torture.js index 3b47ae5291..71b8b3fc11 100644 --- a/__tests__/fixtures/kernel-parity/torture.js +++ b/__tests__/fixtures/kernel-parity/torture.js @@ -125,3 +125,14 @@ function calleeShapes(handlers, type, fn, ns) { (fn)(); (ns.fn)(); } +// --- parameter defaults are the function's code (#2588) ----------------------- +function defaultsOf(note = legacyHelper(1), { a = arrow(2), b: { c = new Widget() } = {} } = {}, [x = legacyHelper(3), , y = arrow(4)] = [], { [legacyHelper(5)]: keyed } = {}) { + return [note, a, c, x, y, keyed]; +} +class DefaultsHolder { + constructor(o = new Widget()) { this.o = o; } + pick(item = legacyHelper(6)) { return item; } + onPick = (item = arrow(7)) => item; +} +const defaultsApi = { load(note = legacyHelper(8)) { return note; } }; +export default function (note = arrow(9)) { return note; } diff --git a/__tests__/fixtures/kernel-parity/torture.tsx b/__tests__/fixtures/kernel-parity/torture.tsx index 55346e7702..cc01414fa8 100644 --- a/__tests__/fixtures/kernel-parity/torture.tsx +++ b/__tests__/fixtures/kernel-parity/torture.tsx @@ -280,3 +280,18 @@ export function NestedHandlers({ items, onPick }: { items: string[]; onPick: (a: const [a, b] = [() => 1, () => 2]; return items.map((i) => ); } + +// --- parameter defaults are the function's code (#2588) ---------------------- +export function defaultsOf(note = formatLabel('x'), { a = parseLabel('a'), b: { c = new FieldDelegator({}, []) } = {} as any }: any = {}, [first = formatLabel('f')]: string[] = [], { [parseLabel('k')]: keyed }: any = {}) { return [note, a, c, first, keyed]; } +export const defaultArrow = (cb = (e: unknown) => formatLabel(String(e)), typed: ReturnType = parseLabel('t')) => cb(typed); +export class DefaultsHolder { + constructor(private readonly delegate = new FieldDelegator({ send: (m: string) => m }, [])) {} + pick(item = formatLabel('i')) { return item; } + onPick = (item = parseLabel('p')) => item; + wrappedPick = withBatchedUpdates((item = formatLabel('w')) => item); +} +export const DefaultsApi = { load(note = formatLabel('l')) { return note; }, save: (note = parseLabel('s')) => note }; +export const DefaultsRef = forwardRef(function DefaultsRef({ size = parseLabel('z') }: any, ref: any) { return {size}; }); +export const DefaultsBox = styled.div.attrs((p = formatLabel('b')) => ({ p }))`color: red;`; +export function overloadedDefault(x: string): string; +export function overloadedDefault(x: string = formatLabel('o')): string { return x; } diff --git a/__tests__/param-default-calls.test.ts b/__tests__/param-default-calls.test.ts new file mode 100644 index 0000000000..4fe9221006 --- /dev/null +++ b/__tests__/param-default-calls.test.ts @@ -0,0 +1,269 @@ +/** + * A parameter's default value is code its function runs, each time a call + * leaves that argument out: `function render(note = buildNote())` calls + * buildNote as surely as `const note = buildNote()` in its body would. The + * extractors walked only the body, so a call or `new` in a default was never + * seen (#2588). Each default, and a computed key in a destructured parameter, + * is now walked with the function on the stack — for functions, arrows, + * methods, constructors, class-field arrows, object-literal methods and the + * render functions of React wrappers — in the kernel and the wasm extractor + * alike. The names a parameter binds and its type are not code, and a Python + * default, which runs once when the `def` does, is left as it was. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import type { ExtractionResult, Language } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +const TS_SOURCE = `import { buildNote, fallback, make, pickKey, logError, Options, Service, throttle } from './lib'; +export function plain(note = buildNote()) { return note; } +export const arrow = (note = buildNote()) => note; +export function destr({ n } = buildNote() as any) { return n; } +export function nested({ a = fallback(), b: { c = make() } = {} as any }: any) { return a + c; } +export function computed({ [pickKey()]: v }: any) { return v; } +export function rest(...[first = fallback()]: number[]) { return first; } +export function optional(x?: number, y: number = fallback()) { return (x ?? 0) + y; } +export function typed(x: ReturnType = buildNote()) { return x; } +export function withNew(opts = new Options()) { return opts; } +export function handler(onError = (e: unknown) => logError(e)) { onError(1); } +export async function* stream(note = buildNote()) { yield note; } +export class K { + constructor(private readonly svc = new Service()) {} + m(note = buildNote()) { return note; } + static s(note = buildNote()) { return note; } + field = (note = buildNote()) => note; + wrapped = throttle((note = fallback()) => note); +} +export const api = { load(note = buildNote()) { return note; }, save: (note = make()) => note }; +export function outer() { + function inner(note = buildNote()) { return note; } + return inner(); +} +export function over(x: number): number; +export function over(x: number = fallback()): number { return x; } +export function inBody() { return buildNote(); } +`; + +const TS_EXPECTED = [ + 'api::load calls buildNote', + 'api::save calls make', + 'arrow calls buildNote', + 'computed calls pickKey', + 'K::constructor instantiates Service', + 'destr calls buildNote', + 'K::field calls buildNote', + 'handler calls logError', + 'handler calls onError', + 'inBody calls buildNote', + 'outer::inner calls buildNote', + 'K::m calls buildNote', + 'K::s calls buildNote', + 'nested calls fallback', + 'nested calls make', + 'optional calls fallback', + 'outer calls inner', + 'over calls fallback', + 'plain calls buildNote', + 'rest calls fallback', + 'stream calls buildNote', + // The type annotation's `typeof shape` is a type, not a call. + 'typed calls buildNote', + 'withNew instantiates Options', + 'K::wrapped calls fallback', +]; + +const JS_SOURCE = `import { buildNote, fallback, pickKey, Options } from './lib'; +export function plain(note = buildNote()) { return note; } +export function nested({ a = fallback(), b: { c = buildNote() } = {} }) { return a + c; } +export function arr([x = fallback(), , y = buildNote()] = []) { return x + y; } +export function computed({ [pickKey()]: v }) { return v; } +export const arrow = (note = buildNote()) => note; +export class J { + constructor(o = new Options()) { this.o = o; } + m(note = buildNote()) { return note; } + field = (note = fallback()) => note; +} +export default function (note = buildNote()) { return note; } +export function inBody() { return buildNote(); } +`; + +const JS_EXPECTED = [ + 'J::m calls buildNote', + 'arr calls buildNote', + 'arr calls fallback', + 'arrow calls buildNote', + 'computed calls pickKey', + 'J::constructor instantiates Options', + 'J::field calls fallback', + 'inBody calls buildNote', + 'nested calls buildNote', + 'nested calls fallback', + 'plain calls buildNote', + // An anonymous function's code is its enclosing scope's: here, the file's. + 'src/plain.js calls buildNote', +]; + +const TSX_SOURCE = `import { forwardRef, memo } from 'react'; +import styled from 'styled-components'; +import { buildNote, fallback } from './lib'; +export const Button = forwardRef(function Button({ size = fallback() }: any, ref: any) { return {size}; }); +export const Card = memo(({ note = buildNote() }: any) => {note}); +export const Box = styled.div.attrs((p = buildNote()) => ({ p }))\` + color: red; +\`; +export const Nested = memo(forwardRef(({ title = buildNote() }: any, ref: any) =>

{title}

)); +export function Page({ title = fallback() }: { title?: string }) { return

{title}

; } +setTimeout((n = fallback()) => n, 0); +`; + +const TSX_EXPECTED = [ + 'Box calls buildNote', + 'Button calls fallback', + 'Card calls buildNote', + 'Nested calls buildNote', + 'Page calls fallback', + // An anonymous function's code is its enclosing scope's: here, the file's. + 'src/ui.tsx calls fallback', + 'src/ui.tsx calls setTimeout', +]; + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +describe('a call in a parameter default is the function’s call (#2588)', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['typescript', 'tsx', 'javascript', 'jsx']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', file: string, source: string, language: Language): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource(file, source, language); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + const result = tryKernelExtract(file, source, language); + expect(result, `kernel extraction of ${file}`).not.toBeNull(); + return result!; + } + + /** `caller kind callee` for every call and `new` the file records, by its caller's qualified name. */ + function callsOf(result: ExtractionResult): string[] { + const names = new Map(result.nodes.map((n) => [n.id, n.kind === 'file' ? n.filePath : n.qualifiedName])); + return result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls' || r.referenceKind === 'instantiates') + .map((r) => `${names.get(r.fromNodeId)} ${r.referenceKind} ${r.referenceName}`) + .sort(); + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + + for (const crlf of [false, true]) { + const eol = (s: string) => (crlf ? s.replace(/\n/g, '\r\n') : s); + const label = crlf ? ' (CRLF)' : ''; + + it.each(backends)(`in TypeScript functions, arrows, methods, fields and object literals: %s${label}`, (backend) => { + expect(callsOf(extract(backend, 'src/use.ts', eol(TS_SOURCE), 'typescript'))).toEqual([...TS_EXPECTED].sort()); + }); + + it.each(backends)(`in JavaScript, through assignment and destructuring patterns: %s${label}`, (backend) => { + expect(callsOf(extract(backend, 'src/plain.js', eol(JS_SOURCE), 'javascript'))).toEqual([...JS_EXPECTED].sort()); + expect(callsOf(extract(backend, 'src/plain.jsx', eol(JS_SOURCE), 'jsx'))).toEqual( + JS_EXPECTED.map((s) => s.replace('src/plain.js ', 'src/plain.jsx ')).sort() + ); + }); + + it.each(backends)(`in a React wrapper's render function and a styled attrs function: %s${label}`, (backend) => { + expect(callsOf(extract(backend, 'src/ui.tsx', eol(TSX_SOURCE), 'tsx'))).toEqual([...TSX_EXPECTED].sort()); + }); + } + + it.runIf(kernelAvailable)('the kernel and the wasm extractor agree on every node, edge and reference', () => { + const canon = (r: ExtractionResult) => ({ + nodes: r.nodes.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort())).sort(), + edges: r.edges.map((e) => JSON.stringify(e, Object.keys(e).sort())).sort(), + refs: r.unresolvedReferences.map((x) => JSON.stringify(x, Object.keys(x).sort())).sort(), + }); + for (const [file, source, language] of [ + ['src/use.ts', TS_SOURCE, 'typescript'], + ['src/plain.js', JS_SOURCE, 'javascript'], + ['src/ui.tsx', TSX_SOURCE, 'tsx'], + ] as const) { + expect(canon(extract('kernel', file, source, language)), file).toEqual(canon(extract('wasm', file, source, language))); + } + }); +}); + +describe('the default value links to what it calls', () => { + let root = ''; + let graph: CodeGraph | null = null; + + afterEach(() => { + graph?.close(); + graph = null; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = ''; + }); + + it('gives buildNote its callers from defaults, and leaves a Python default alone', async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-param-default-')); + const files: Record = { + 'package.json': JSON.stringify({ name: 'app' }), + // The issue's reproduction, verbatim. + 'src/f.ts': + 'export function buildNote() { return { id: 1 }; }\n' + + 'export function renderWithNote(note = buildNote()) {\n return note.id;\n}\n' + + 'export const arrow = (note = buildNote()) => note.id;\n' + + 'export class K { m(note = buildNote()) { return note.id; } }\n' + + 'export function destr({ n } = buildNote() as any) { return n; }\n' + + 'export function inBody() { const note = buildNote(); return note.id; }\n', + // Python evaluates a default once, when the `def` runs: not the function's call. + 'tool/ctl.py': 'def build():\n return 1\n\n\ndef uses_default(x=build()):\n return x\n\n\ndef in_body():\n return build()\n', + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + graph = await CodeGraph.init(root, { index: true }); + + const callers = (file: string, name: string): string[] => { + const target = graph!.getNodesInFile(file).find((n) => n.name === name)!; + return graph!.getIncomingEdges(target.id) + .filter((e) => e.kind === 'calls') + .map((e) => graph!.getNode(e.source)!.name) + .sort(); + }; + expect(callers('src/f.ts', 'buildNote')).toEqual(['arrow', 'destr', 'inBody', 'm', 'renderWithNote']); + expect(callers('tool/ctl.py', 'build')).toEqual(['in_body']); + }, 120_000); +}); diff --git a/codegraph-kernel/src/tsjs/extractors.rs b/codegraph-kernel/src/tsjs/extractors.rs index abaaa24a1b..61b343825a 100644 --- a/codegraph-kernel/src/tsjs/extractors.rs +++ b/codegraph-kernel/src/tsjs/extractors.rs @@ -45,7 +45,9 @@ impl<'t> Walker<'t> { if name == "" { // Still walk the body: module wrappers hold named inner functions // and calls that would otherwise be lost (#528). - if let Some(body) = body_of(node) { + let body = body_of(node); + self.visit_parameter_defaults(node, body); + if let Some(body) = body { self.visit_function_body(body); } return; @@ -68,12 +70,73 @@ impl<'t> Walker<'t> { self.extract_decorators_for(node, row); self.stack.push(Scope { row, kind: "function", name }); - if let Some(body) = body_of(node) { + let body = body_of(node); + self.visit_parameter_defaults(node, body); + if let Some(body) = body { self.visit_function_body(body); } self.stack.pop(); } + /// A parameter's default value is code the function runs, each time a + /// call leaves that argument out (#2588). Each default, and a computed key + /// in a destructured parameter, is walked as the body is, with the same + /// scope on the stack; the names a parameter binds and its type annotation + /// are not. A function with no body never runs. Mirrors + /// TreeSitterExtractor.visitParameterDefaults. + fn visit_parameter_defaults(&mut self, func: Node<'t>, body: Option>) { + let Some(body) = body else { return }; + // A class field holds its function: `onPick = (item = first()) => {…}`. + let owner = if matches!(func.kind(), "public_field_definition" | "field_definition") { + body.parent() + } else { + Some(func) + }; + if let Some(params) = owner.and_then(|o| o.child_by_field_name("parameters")) { + self.visit_pattern_defaults(params); + } + } + + /// The defaults in a parameter list or a destructuring pattern, in source + /// order. Mirrors TreeSitterExtractor.visitPatternDefaults. + fn visit_pattern_defaults(&mut self, node: Node<'t>) { + stack_guard!(); + match node.kind() { + "formal_parameters" | "object_pattern" | "array_pattern" | "rest_pattern" => { + for i in 0..node.named_child_count() { + if let Some(child) = node.named_child(i) { + self.visit_pattern_defaults(child); + } + } + } + "required_parameter" | "optional_parameter" => { + if let Some(pattern) = node.child_by_field_name("pattern") { + self.visit_pattern_defaults(pattern); + } + if let Some(value) = node.child_by_field_name("value") { + self.visit_function_body(value); + } + } + "assignment_pattern" | "object_assignment_pattern" => { + if let Some(left) = node.child_by_field_name("left") { + self.visit_pattern_defaults(left); + } + if let Some(right) = node.child_by_field_name("right") { + self.visit_function_body(right); + } + } + "pair_pattern" => { + if let Some(key) = node.child_by_field_name("key").filter(|k| k.kind() == "computed_property_name") { + self.visit_function_body(key); + } + if let Some(value) = node.child_by_field_name("value") { + self.visit_pattern_defaults(value); + } + } + _ => {} + } + } + /// The property a CommonJS export assignment binds a function to — /// `exports.NAME = ` / `module.exports.NAME = ` — or None for /// any other assignment. The node must be the assignment's whole @@ -174,7 +237,9 @@ impl<'t> Walker<'t> { self.stack.push(Scope { row, kind: "component", name: name.to_string() }); match inner_fn { Some(inner) => { - if let Some(body) = body_of(inner) { + let body = body_of(inner); + self.visit_parameter_defaults(inner, body); + if let Some(body) = body { self.visit_function_body(body); } } @@ -246,7 +311,9 @@ impl<'t> Walker<'t> { } else if matches!(arg.kind(), "arrow_function" | "function_expression") { // A style or `.attrs()` function runs as the component's // code: its body is walked, the function never named. - if let Some(body) = body_of(arg) { + let body = body_of(arg); + self.visit_parameter_defaults(arg, body); + if let Some(body) = body { self.visit_function_body(body); } } else { @@ -331,7 +398,9 @@ impl<'t> Walker<'t> { // Object-literal methods are ephemeral: walk the body only. if let Some(parent) = node.parent() { if matches!(parent.kind(), "object" | "object_expression") { - if let Some(body) = body_of(node) { + let body = body_of(node); + self.visit_parameter_defaults(node, body); + if let Some(body) = body { self.visit_function_body(body); } return; @@ -358,7 +427,9 @@ impl<'t> Walker<'t> { self.extract_decorators_for(node, row); self.stack.push(Scope { row, kind: "method", name }); - if let Some(body) = body_of(node) { + let body = body_of(node); + self.visit_parameter_defaults(node, body); + if let Some(body) = body { self.visit_function_body(body); } self.stack.pop(); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index b440947d43..fb9ae9ef04 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -2200,6 +2200,7 @@ export class TreeSitterExtractor { // descends into this subtree. (#528) const body = this.extractor.resolveBody?.(node, this.extractor.bodyField) ?? getChildByField(node, this.extractor.bodyField); + this.visitParameterDefaults(node, body); if (body) { this.visitFunctionBody(body, ''); } @@ -2248,12 +2249,70 @@ export class TreeSitterExtractor { this.nodeStack.push(funcNode.id); const body = this.extractor.resolveBody?.(node, this.extractor.bodyField) ?? getChildByField(node, this.extractor.bodyField); + this.visitParameterDefaults(node, body); if (body) { this.visitFunctionBody(body, funcNode.id); } this.nodeStack.pop(); } + /** + * A parameter's default value is code the function runs, each time a call + * leaves that argument out (#2588): `function render(note = buildNote())` + * calls buildNote as surely as `const note = buildNote()` in its body would. + * Each default, and a computed key in a destructured parameter, is walked as + * the body is, with the same scope on the stack, so its calls and `new` + * attribute to the function. The names a parameter binds and its type + * annotation are not walked. TS/JS only: elsewhere a default may run once, + * when the definition does (Python). A function with no body never runs. + * Mirrored in the kernel (tsjs/extractors.rs visit_parameter_defaults). + */ + private visitParameterDefaults(fn: SyntaxNode, body: SyntaxNode | null | undefined): void { + if (!body || !TS_JS_CHAIN_LANGUAGES.has(this.language)) return; + // A class field holds its function: `onPick = (item = first()) => {…}`. + const owner = fn.type === 'public_field_definition' || fn.type === 'field_definition' ? body.parent : fn; + const params = owner ? getChildByField(owner, 'parameters') : null; + if (params) this.visitPatternDefaults(params); + } + + /** The defaults in a parameter list or a destructuring pattern, in source order. */ + private visitPatternDefaults(node: SyntaxNode): void { + switch (node.type) { + case 'formal_parameters': + case 'object_pattern': + case 'array_pattern': + case 'rest_pattern': + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (child) this.visitPatternDefaults(child); + } + return; + case 'required_parameter': + case 'optional_parameter': { + const pattern = getChildByField(node, 'pattern'); + if (pattern) this.visitPatternDefaults(pattern); + const value = getChildByField(node, 'value'); + if (value) this.visitFunctionBody(value, ''); + return; + } + case 'assignment_pattern': + case 'object_assignment_pattern': { + const left = getChildByField(node, 'left'); + if (left) this.visitPatternDefaults(left); + const right = getChildByField(node, 'right'); + if (right) this.visitFunctionBody(right, ''); + return; + } + case 'pair_pattern': { + const key = getChildByField(node, 'key'); + if (key?.type === 'computed_property_name') this.visitFunctionBody(key, ''); + const value = getChildByField(node, 'value'); + if (value) this.visitPatternDefaults(value); + return; + } + } + } + /** * Detect a React component declared via an HOC wrapper whose result is itself a * component: `forwardRef(...)`, `memo(...)`, `React.forwardRef/memo(...)`, and @@ -2355,6 +2414,7 @@ export class TreeSitterExtractor { if (innerFn) { const body = this.extractor.resolveBody?.(innerFn, this.extractor.bodyField) ?? getChildByField(innerFn, this.extractor.bodyField); + this.visitParameterDefaults(innerFn, body); if (body) this.visitFunctionBody(body, compNode.id); } else { const value = getChildByField(declarator, 'value'); @@ -2422,6 +2482,7 @@ export class TreeSitterExtractor { // `styled(Box)(({ theme }) => …)` for a function of its own). const body = this.extractor!.resolveBody?.(arg, this.extractor!.bodyField) ?? getChildByField(arg, this.extractor!.bodyField); + this.visitParameterDefaults(arg, body); if (body) this.visitFunctionBody(body, ''); } else { this.visitFunctionBody(arg, ''); @@ -2556,6 +2617,7 @@ export class TreeSitterExtractor { if (node.parent?.type === 'object' || node.parent?.type === 'object_expression') { const body = this.extractor.resolveBody?.(node, this.extractor.bodyField) ?? getChildByField(node, this.extractor.bodyField); + this.visitParameterDefaults(node, body); if (body) { this.visitFunctionBody(body, ''); } @@ -2631,6 +2693,7 @@ export class TreeSitterExtractor { this.nodeStack.push(methodNode.id); const body = this.extractor.resolveBody?.(node, this.extractor.bodyField) ?? getChildByField(node, this.extractor.bodyField); + this.visitParameterDefaults(node, body); if (body) { this.visitFunctionBody(body, methodNode.id); } From 7b50ca832152e9948f568ae641ea34952fac1334 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 15:20:06 +0000 Subject: [PATCH 37/95] fix(pascal): Delphi calls to compiler intrinsics and plain data no longer link namesakes (#2803) A Pascal call to a name the compiler or the System unit gives meaning to (SizeOf, TypeInfo, High, Low, Length, Inc, Dec, Assigned, Ord, Exit, Include, Move, Assert, typecasts like Pointer(P)) went to name matching, which linked any project member of that name: mORMot's TRttiProp.TypeInfo, a record's length or high field, a property named Pointer. The built-in name check compared case, so length(S) and dec(N) got through too. And a call that resolved to a field, property or variable linked whatever data had the name. Now a bare call to a System name links only to a declaration its scope reaches: the class and its ancestors, the unit, the units it uses last first, skipping a unit's stand-in declared under {$IFDEF} or {$IF}. A call resolved to data stands only when the data's declared type is a routine type (written out, a project alias to one, generic or on two lines, or an RTL procedural type by name), retargeted to what the scope declares. A name the calling routine declares itself links nothing; a name the scope doesn't declare at all (a with object's member, a declaration the parse misplaced) links to the one routine-holding field or variable of that name. Built-in names compare without case; the RTL namespaces keep their spelling, except system. Scope lookups now see a unit whose unit header the parser lost, by its file name (its indented, lost class members aren't taken for its routines), and an include file shares its unit's scope and uses clauses. A with body's bare call to a System name is not extracted, since it may be the with object's. Validation, edges removed/added against origin/main: SynEdit -243/+50, HeidiSQL -858/+296, mORMot2 -9039/+987, DelphiMVCFramework -2420/+242, DWScript -705/+222; leveldb and SDWebImage byte-identical. Sampled precision of calls to data, before and after: 20/50 and 50/50; of calls to System names: 2/50 and 47/47. Sampled removed edges: 40/40 wrong. Follow-up to #2514. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/pascal-system-calls.test.ts | 786 ++++++++++++++++++++++ src/extraction/languages/pascal-system.ts | 31 + src/extraction/tree-sitter.ts | 7 +- src/resolution/index.ts | 38 +- src/resolution/pascal-scope.ts | 604 +++++++++++++++-- 6 files changed, 1406 insertions(+), 61 deletions(-) create mode 100644 __tests__/pascal-system-calls.test.ts create mode 100644 src/extraction/languages/pascal-system.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e2fecdc17..5afe0259cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Delphi, a `for … in` loop that declares its variable inline, like `for var lItem in AList do` or `for var lHandle: THandle in lHandles do`, now has its calls indexed, and a call on a typed loop variable reaches that type's method. Before, the whole loop was skipped, so a routine calling a method only from such a loop was missing from that method's callers. Thanks @stalexteam for the report. (#2546) - In Delphi, a call made on the result of another call, like `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)` or `List[IndexOf(Name)].Free`, now has the inner calls indexed too, so `MD5FileSized` and `TMeta.ExtendedPath` list that routine among their callers. A typecast like `TButton(Sender).Caption` or `F := TFoo(Obj)` now counts as a use of the type instead of creating one. Thanks @stalexteam for the report. (#2547) - In Delphi, a routine called without parentheses, the usual way to call one that takes no arguments, like `Before;`, `if Ready then`, `Result := FProvider.Value + 1` or `QueryTabs.ActiveTab.MemoFilename`, now counts as a call, so it shows up in callers, callees and impact, and a method given to an event, like `OnClick := ButtonClick`, is linked as a reference. A name is linked only to a method or function its code reaches, in its class and the classes it inherits from (`class sealed (TBase)` included), its unit, or the units it uses, never to a field, property or variable of that name. Thanks @OleksiyPenkov for the report, and @drakeo338 and @OleksiyPenkov for their pull requests. (#2514) +- In Delphi, a call to something the compiler provides, like `SizeOf(X)`, `TypeInfo(TFoo)`, `High(A)`, `Inc(N)`, `Assert(…)` or a typecast like `Pointer(P)`, written in any letter case, no longer shows up as a call to a project method, field or variable that happens to share its name, unless the calling code's own class or units declare one, like a list class's own `Delete`. A call through a field, property or variable now counts only when it holds a routine, like an `OnChange` event or a `reference to function` variable, and calls in `{$I}` include files and into units the parser could only partly read now find the routines those units declare. Thanks @OleksiyPenkov for the report. (#2514) - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. diff --git a/__tests__/pascal-system-calls.test.ts b/__tests__/pascal-system-calls.test.ts new file mode 100644 index 0000000000..b26e43c24c --- /dev/null +++ b/__tests__/pascal-system-calls.test.ts @@ -0,0 +1,786 @@ +/** + * A Pascal call to a name the compiler or the System unit gives meaning to, + * like `SizeOf`, `TypeInfo`, `High`, `Inc`, `Move` or a typecast to a built-in + * type such as `Pointer(P)`, linked to any project member of that name, and + * the check that should have stopped it compared case: Pascal ignores it, so + * `length(S)` and `dec(N)` went to a record's `length` field and a variable + * named `dec`. A call that resolved to a field, a property or a variable + * linked too, though only one holding a routine can run. Now such a call + * links only to a declaration its scope reaches: a member of the class + * (`Delete(I)` in a list class), or a routine of the unit or the units it + * uses, and data only when its declared type is a routine's. That scope + * takes in the include files a unit pulls in and the units whose header the + * parser lost, and stops at the calling routine's own parameters and locals. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; + +beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['pascal']); +}); + +async function withIndex(files: Record, check: (cg: CodeGraph) => void): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-system-')); + try { + for (const [name, code] of Object.entries(files)) fs.writeFileSync(path.join(dir, name), code); + const cg = await CodeGraph.init(dir, { index: true }); + try { + check(cg); + } finally { + cg.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +/** The calls a routine makes, as `line: Target::Name`. */ +function callsFrom(cg: CodeGraph, qualifiedName: string): string[] { + const source = cg.getNodesByKind('method').concat(cg.getNodesByKind('function')).find((n) => n.qualifiedName === qualifiedName); + expect(source, qualifiedName).toBeDefined(); + return cg + .getOutgoingEdges(source!.id) + .filter((e) => e.kind === 'calls') + .map((e) => `${e.line}: ${cg.getNode(e.target)?.qualifiedName ?? '?'}`) + .sort(); +} + +// Namesakes of the System unit's names, elsewhere in the project. +const LIB = `unit Lib; + +interface + +type + TRec = record + length: Integer; + high: Integer; + end; + + TOther = class + private + FObj: TObject; + public + function TypeInfo: Pointer; + procedure Move(A, B: Integer); + function SizeOf: Integer; + property Pointer: TObject read FObj; + end; + +var + dec: Integer; + +implementation + +function TOther.TypeInfo: Pointer; begin Result := nil; end; +procedure TOther.Move(A, B: Integer); begin end; +function TOther.SizeOf: Integer; begin Result := 0; end; + +end. +`; + +describe('Pascal calls to the System unit and to data', () => { + it('links an intrinsic, a System routine or a built-in typecast to no project namesake, whatever its case', async () => { + const main = `unit Main; + +interface + +uses Lib; + +type + TList2 = class + public + procedure Run(const S: string; var A: array of Integer); + end; + +implementation + +procedure TList2.Run(const S: string; var A: array of Integer); +var + P: Pointer; + N: Integer; +begin + N := length(S) + High(A) + SIZEOF(N); + dec(N); + P := TypeInfo(TList2); + Move(P, P, 0); + P := Pointer(Self); +end; + +end. +`; + await withIndex({ 'Lib.pas': LIB, 'Main.pas': main }, (cg) => { + expect(callsFrom(cg, 'TList2::Run')).toEqual([]); + }); + }); + + it('takes `system.` for the System unit whatever its case, and a lowercase `data.` for a variable', async () => { + const main = `unit Main; + +interface + +uses Lib; + +type + TStore = class + public + function GetObjectFrom(const Name: string): TObject; + end; + +procedure Run(data: TStore); + +implementation + +function TStore.GetObjectFrom(const Name: string): TObject; begin Result := nil; end; + +procedure Run(data: TStore); +var + P: Pointer; +begin + P := system.TypeInfo(TStore); + data.GetObjectFrom('setting'); +end; + +end. +`; + await withIndex({ 'Lib.pas': LIB, 'Main.pas': main }, (cg) => { + expect(callsFrom(cg, 'Run')).toEqual(['24: TStore::GetObjectFrom']); + }); + }); + + it('links one to the declaration that shadows it in the call\'s scope', async () => { + const main = `unit Main; + +interface + +uses Lib; + +type + TList2 = class + public + procedure Delete(Index: Integer); + procedure Clear; + end; + +function Length(const S: string): Integer; +procedure Tidy; + +implementation + +function Length(const S: string): Integer; begin Result := 0; end; + +procedure TList2.Delete(Index: Integer); begin end; + +procedure TList2.Clear; +begin + Delete(0); +end; + +procedure Tidy; +begin + Length('x'); +end; + +end. +`; + await withIndex({ 'Lib.pas': LIB, 'Main.pas': main }, (cg) => { + // The list's own `Delete`, and the unit's own `Length`. + expect(callsFrom(cg, 'TList2::Clear')).toEqual(['25: TList2::Delete']); + expect(callsFrom(cg, 'Tidy')).toEqual(['30: Length']); + }); + }); + + it('takes a unit\'s stand-in declared under a condition for the System unit\'s own', async () => { + const lib = `unit Lib; + +interface + +type +{$IFNDEF UNICODE} + UnicodeString = WideString; +{$ENDIF} + RawUtf8 = type AnsiString; + +{$IFNDEF COMPILER_3_UP} +procedure Assert(Expr: Boolean); { stub for Delphi 2 } +{$ENDIF} +function UpperCase(const S: RawUtf8): RawUtf8; + +implementation + +{$IFNDEF COMPILER_3_UP} +procedure Assert(Expr: Boolean); begin end; +{$ENDIF} +function UpperCase(const S: RawUtf8): RawUtf8; begin Result := S; end; + +end. +`; + const main = `unit Main; + +interface + +uses SysUtils, Lib; + +procedure Run(P: Pointer; const S: RawUtf8); + +implementation + +procedure Run(P: Pointer; const S: RawUtf8); +begin + Assert(P <> nil); + if UnicodeString(P) <> '' then + UpperCase(S); +end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Main.pas': main }, (cg) => { + // The used unit's `UpperCase` hides SysUtils' one; `Assert` and + // `UnicodeString` are the compiler's. + expect(callsFrom(cg, 'Run')).toEqual(['15: UpperCase']); + const run = cg.getNodesByKind('function').find((n) => n.qualifiedName === 'Run')!; + expect(cg.getOutgoingEdges(run.id).filter((e) => e.kind === 'references').map((e) => cg.getNode(e.target)?.name)) + .not.toContain('UnicodeString'); + }); + }); + + it('reaches a class\'s ancestor through a type alias', async () => { + const base = `unit Base; + +interface + +type + TSearchEvent = procedure(Sender: TObject; const Text: string) of object; + + TBaseTree = class + private + FOnSearch: TSearchEvent; + public + function Ready: Boolean; + property OnSearch: TSearchEvent read FOnSearch write FOnSearch; + end; + +implementation + +function TBaseTree.Ready: Boolean; begin Result := True; end; + +end. +`; + const main = `unit Main; + +interface + +uses Base; + +type + TAncestor = TBaseTree; + + TStringTree = class(TAncestor) + public + procedure Search(const Text: string); + end; + +implementation + +procedure TStringTree.Search(const Text: string); +begin + if Ready then + OnSearch(Self, Text); +end; + +end. +`; + await withIndex({ 'Base.pas': base, 'Main.pas': main }, (cg) => { + const search = cg.getNodesByKind('method').find((n) => n.qualifiedName === 'TStringTree::Search')!; + const targets = cg.getOutgoingEdges(search.id).filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)?.qualifiedName).sort(); + // `TAncestor = TBaseTree;` names the class the tree descends from: its + // `Ready` runs, and so does what its `OnSearch` event holds. + expect(targets).toEqual(['TBaseTree::OnSearch', 'TBaseTree::Ready']); + }); + }); + + it('links a name its scope doesn\'t declare to the one routine-holding field of that name, unless the routine declares it', async () => { + const lib = `unit Lib; + +interface + +type + THandleAlloc = function(Parent: Pointer; var Handle: Pointer): Integer; + THasher = function(Crc: Cardinal; Buf: Pointer; Len: Cardinal): Cardinal; + + TOciLib = class + public + HandleAlloc: THandleAlloc; + end; + + TDynHasher = class + private + FHasher: THasher; + public + property Hasher: THasher read FHasher; + property Direct: THasher read FHasher; + end; + +implementation + +end. +`; + const main = `unit Main; + +interface + +uses Lib; + +type + TConn = class + private + FOci: TOciLib; + FEnv: Pointer; + public + procedure Connect; + end; + +function Hash(Item: Pointer; Hasher: THasher): Cardinal; +procedure Test; + +implementation + +procedure TConn.Connect; +begin + with FOci do + HandleAlloc(nil, FEnv); +end; + +function Hash(Item: Pointer; Hasher: THasher): Cardinal; +begin + Result := Hasher(0, Item, 4); +end; + +procedure Test; + + procedure Direct(const Url: string); + begin + end; + +begin + Direct('/root'); +end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Main.pas': main }, (cg) => { + // The `with` object's member, which the class's scope doesn't hold. + expect(callsFrom(cg, 'TConn::Connect')).toEqual(['24: TOciLib::HandleAlloc']); + // A parameter and a nested routine, not the properties named so. + expect(callsFrom(cg, 'Hash')).toEqual([]); + expect(callsFrom(cg, 'Test')).toEqual([]); + }); + }); + + it('takes a parameter or a local of the calling routine for what its name calls', async () => { + const lib = `unit Lib; + +interface + +type + TLoad = procedure(Data: Pointer); + +var + load: TLoad; + +implementation + +end. +`; + const main = `unit Main; + +interface + +uses Classes, Lib; + +type + TFoo = class + private + FOnChange: TNotifyEvent; + public + procedure Fire(OnChange: TNotifyEvent); + procedure Run(Data: Pointer); + property OnChange: TNotifyEvent read FOnChange; + end; + +implementation + +procedure TFoo.Fire(OnChange: TNotifyEvent); +begin + OnChange(Self); +end; + +procedure TFoo.Run(Data: Pointer); +var + load: TLoad; +begin + load := nil; + load(Data); +end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Main.pas': main }, (cg) => { + expect(callsFrom(cg, 'TFoo::Fire')).toEqual([]); + expect(callsFrom(cg, 'TFoo::Run')).toEqual([]); + }); + }); + + it('reads a routine type declared on two lines, generic, or as an alias of another', async () => { + const main = `unit Main; + +interface + +uses SysUtils; + +type + THelperEvent = + procedure(const Value: Variant) of object; + TSetParams = reference to procedure(const Obj: T); + TFormatter = TFunc; + TCounter = Integer; + TCount = TCounter; + + TWriter = class + private + FOnHelp: THelperEvent; + FSetParams: TSetParams; + FFormatter: TFormatter; + FCount: TCount; + public + procedure Write(Obj: T; E: Exception); + end; + +implementation + +procedure TWriter.Write(Obj: T; E: Exception); +begin + FOnHelp(1); + FSetParams(Obj); + FFormatter(E); + FCount(1); +end; + +end. +`; + await withIndex({ 'Main.pas': main }, (cg) => { + expect(callsFrom(cg, 'TWriter::Write')).toEqual([ + '29: TWriter::FOnHelp', + '30: TWriter::FSetParams', + '31: TWriter::FFormatter', + ]); + }); + }); + + it('sees the declarations of a used unit whose header the parser lost', async () => { + // The `{$IFDEF}` around a local `var` section costs the parse its `unit` + // node; the unit is still the one its file is named after. + const lib = `unit Lib; + +interface + +type + THasher = function(Crc: Cardinal; Buf: Pointer; Len: Cardinal): Cardinal; + +var + crc32c: THasher; + +function FileSize(const FileName: string): Int64; + +implementation + +function FileSize(const FileName: string): Int64; +{$IFDEF MSWINDOWS} +var + Handle: THandle; +{$ENDIF} +begin + Result := 0; +end; + +end. +`; + const main = `unit Main; + +interface + +uses Lib; + +procedure Run(const Name: string); + +implementation + +procedure Run(const Name: string); +begin + if FileSize(Name) > 0 then + crc32c(0, nil, 0); +end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Main.pas': main }, (cg) => { + expect(cg.getNodesByKind('module').map((n) => n.name)).toEqual(['Main']); + expect(callsFrom(cg, 'Run')).toEqual(['13: FileSize', '14: crc32c']); + }); + }); + + it('gives an include file\'s code the scope of the unit that includes it', async () => { + const hash = `unit Hash; + +interface + +type + THasher = function(Crc: Cardinal; Buf: Pointer; Len: Cardinal): Cardinal; + +var + DefaultHasher: THasher; + +implementation + +end. +`; + const os = `unit Os; + +interface + +uses Hash; + +procedure Entropy; +procedure Init; + +implementation + +{$I Os.windows.inc} + +procedure Init; +begin + OsLog('init'); +end; + +end. +`; + const extra = `unit Extra; + +interface + +var + ExtraLog: procedure(const S: string); + +implementation + +end. +`; + // An include file may open with a `uses` clause of its own. + const inc = `uses + Extra; + +var + OsLog: procedure(const S: string); + +procedure Entropy; +var + Seed: Cardinal; +begin + Seed := DefaultHasher(0, nil, 0); + ExtraLog('seeded'); +end; +`; + await withIndex({ 'Hash.pas': hash, 'Extra.pas': extra, 'Os.pas': os, 'Os.windows.inc': inc }, (cg) => { + const entropy = cg.getNodesByKind('function').find((n) => n.name === 'Entropy' && n.filePath.endsWith('.inc')); + expect(entropy).toBeDefined(); + expect(cg.getOutgoingEdges(entropy!.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)?.qualifiedName).sort()) + .toEqual(['DefaultHasher', 'ExtraLog']); + // And the unit's own code sees what its include file declares. + expect(callsFrom(cg, 'Init')).toEqual(['16: OsLog']); + }); + }); + + it('takes no member of a class such a unit\'s parse lost for a routine of the unit', async () => { + // The anonymous record ends the class early, so `Insert` comes out as a + // routine of the unit, indented as no routine of the unit is. + const lib = `unit Lib; + +interface + +type + TClient = class + Items: array of record + Name: string; + end; + procedure Insert(Index: Integer; const S: string); + end; + +procedure Log(const S: string); + +implementation + +procedure TClient.Insert(Index: Integer; const S: string); begin end; + +procedure Log(const S: string); +{$IFDEF MSWINDOWS} +var + Handle: THandle; +{$ENDIF} +begin +end; + +end. +`; + const main = `unit Main; + +interface + +uses Lib; + +procedure Run(var S: string); + +implementation + +procedure Run(var S: string); +begin + Insert('x', S, 1); + Log(S); +end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Main.pas': main }, (cg) => { + const insert = cg.getNodesByKind('function').find((n) => n.name === 'Insert'); + expect(insert?.startColumn).toBe(4); + expect(callsFrom(cg, 'Run')).toEqual(['14: Log']); + }); + }); + + it('leaves out such a call in a `with` body, which may be the `with` object\'s', async () => { + const main = `unit Main; + +interface + +type + TItem = class + public + procedure Delete(Index: Integer); + end; + + TItems = class + public + function Add: TItem; + procedure Delete(Index: Integer); + procedure Assign(Source: TObject); + end; + +implementation + +procedure TItem.Delete(Index: Integer); begin end; +function TItems.Add: TItem; begin Result := nil; end; +procedure TItems.Delete(Index: Integer); begin end; + +procedure TItems.Assign(Source: TObject); +begin + with Add do + Delete(0); + Delete(1); +end; + +end. +`; + await withIndex({ 'Main.pas': main }, (cg) => { + // In the `with` body, `Delete` is the item's, not the list's; after it, the list's. + expect(callsFrom(cg, 'TItems::Assign')).toEqual(['28: TItems::Delete']); + }); + }); + + it('links a call to data only when the data holds a routine and the call\'s scope reaches it', async () => { + const lib = `unit Lib; + +interface + +type + TNotifyEvent = procedure(Sender: TObject) of object; + + TStranger = class + private + FOnChange: TNotifyEvent; + end; + +implementation + +end. +`; + const main = `unit Main; + +interface + +uses Lib; + +type + TList2 = class + private + FOnChange: TNotifyEvent; + FCount: Integer; + FProc: procedure of object; + FGetter: reference to function: Integer; + FOnLoad, // a list's type comes at its end + FOnSave: TNotifyEvent; + public + procedure Run; + procedure Other; + property OnChange: TNotifyEvent read FOnChange write FOnChange; + end; + + TOtherList = class + public + procedure Fire; + end; + +var + GLog: procedure(const Msg: string); + GCount: Integer; + +implementation + +procedure TList2.Run; +begin + FOnChange(Self); + OnChange(Self); + FProc(); + FGetter(); + GLog('x'); + FCount(1); + GCount(1); + FOnLoad(Self); +end; + +procedure TList2.Other; begin end; + +procedure TOtherList.Fire; +begin + FOnChange(Self); +end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Main.pas': main }, (cg) => { + expect(callsFrom(cg, 'TList2::Run')).toEqual([ + '35: TList2::FOnChange', + '36: TList2::OnChange', + '37: TList2::FProc', + '38: TList2::FGetter', + '39: GLog', + '42: TList2::FOnLoad', + ]); + // No class this method's scope reaches declares `FOnChange`. + expect(callsFrom(cg, 'TOtherList::Fire')).toEqual([]); + }); + }); +}); diff --git a/src/extraction/languages/pascal-system.ts b/src/extraction/languages/pascal-system.ts new file mode 100644 index 0000000000..14daeabb03 --- /dev/null +++ b/src/extraction/languages/pascal-system.ts @@ -0,0 +1,31 @@ +/** + * The names Delphi's compiler and System unit give meaning to, lowercased: + * intrinsics (`SizeOf`, `TypeInfo`, `High`, `Inc`, `Assigned`, `Exit`), the + * System and SysUtils routines everyone calls (`Move`, `FillChar`, `Copy`, + * `Format`, `WriteLn`), and the built-in types a typecast names (`Pointer(P)`, + * `Integer(X)`). Pascal names ignore case: `length(S)` is `Length(S)`. A call + * to one is the System unit's unless a declaration in its scope shadows it + * (see resolution/pascal-scope). + */ +export const PASCAL_SYSTEM_NAMES: ReadonlySet = new Set([ + // Intrinsics and System routines + 'abs', 'addr', 'arctan', 'assert', 'assigned', 'assignfile', 'atomiccmpexchange', 'atomicdecrement', + 'atomicexchange', 'atomicincrement', 'blockread', 'blockwrite', 'break', 'chdir', 'chr', 'closefile', 'concat', + 'continue', 'copy', 'cos', 'dec', 'default', 'delete', 'dispose', 'eof', 'eoln', 'exclude', 'exit', 'exp', + 'fail', 'filepos', 'filesize', 'fillchar', 'finalize', 'frac', 'freemem', 'getdir', 'getmem', 'gettypekind', + 'halt', 'hasweakref', 'hi', 'high', 'inc', 'include', 'initialize', 'insert', 'int', 'ioresult', + 'isconstvalue', 'ismanagedtype', 'length', 'ln', 'lo', 'low', 'mkdir', 'move', 'new', 'odd', 'ord', + 'paramcount', 'paramstr', 'pi', 'pred', 'ptr', 'random', 'randomize', 'read', 'readln', 'reallocmem', + 'returnaddress', 'rmdir', 'round', 'runerror', 'setlength', 'setstring', 'sin', 'sizeof', 'slice', 'sqr', + 'sqrt', 'str', 'succ', 'swap', 'trunc', 'typehandle', 'typeinfo', 'typeof', 'uniquestring', 'upcase', 'val', + 'vararrayredim', 'write', 'writeln', + // SysUtils routines the resolver has always taken for the RTL's + 'abort', 'date', 'datetostr', 'floattostr', 'format', 'freeandnil', 'inttostr', 'lowercase', 'now', 'pos', + 'strtodate', 'strtofloat', 'strtoint', 'time', 'trim', 'uppercase', + // Built-in types, called as typecasts + 'ansichar', 'ansistring', 'boolean', 'byte', 'bytebool', 'cardinal', 'char', 'comp', 'currency', 'double', + 'extended', 'int16', 'int32', 'int64', 'int8', 'integer', 'longbool', 'longint', 'longword', 'nativeint', + 'nativeuint', 'olevariant', 'pansichar', 'pchar', 'pointer', 'pwidechar', 'rawbytestring', 'real', + 'shortint', 'shortstring', 'single', 'smallint', 'string', 'uint16', 'uint32', 'uint64', 'uint8', + 'unicodestring', 'utf8string', 'variant', 'widechar', 'widestring', 'word', 'wordbool', +]); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index fb9ae9ef04..95c39a0e73 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -32,6 +32,7 @@ import { RazorExtractor } from './razor-extractor'; import { SvelteExtractor } from './svelte-extractor'; import { AstroExtractor } from './astro-extractor'; import { DfmExtractor } from './dfm-extractor'; +import { PASCAL_SYSTEM_NAMES } from './languages/pascal-system'; import { VueExtractor } from './vue-extractor'; import { MyBatisExtractor } from './mybatis-extractor'; import { CfmlExtractor } from './cfml-extractor'; @@ -8834,7 +8835,11 @@ export class TreeSitterExtractor { } const vagueLink = receiverLink && firstChild.type === 'exprDot' && !calleeName.includes('.'); - if (calleeName && !vagueLink) { + // In a `with` body a bare `Delete(I)` or `Assign(X)` may be the `with` + // object's member rather than the System unit's or the class's: which one + // the object's type would have to say (see resolution/pascal-scope). + const withMember = this.pascalWithDepth > 0 && firstChild.type === 'identifier' && PASCAL_SYSTEM_NAMES.has(calleeName.toLowerCase()); + if (calleeName && !vagueLink && !withMember) { this.unresolvedReferences.push({ fromNodeId: callerId, referenceName: calleeName, diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 33d9eb2cab..b088bef407 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -28,7 +28,9 @@ import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './ import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; import { clearDartLibraryMemos } from './dart-libraries'; import { clearVbnetReceiverMemos, isVbMemberRead, isVbPathCall, matchVbMemberRead, matchVbPathCall } from './vbnet-receivers'; -import { clearPascalScopeMemos, isPascalParenlessRef, matchPascalParenlessRef } from './pascal-scope'; +import { + clearPascalScopeMemos, gatePascalDataCall, isPascalParenlessRef, isPascalSystemCall, matchPascalParenlessRef, matchPascalSystemCall, +} from './pascal-scope'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { gateDartLocal, clearDartLocalScopeMemos } from './dart-local-scope'; import { clearCppTypeAliasMemos } from './cpp-type-aliases'; @@ -161,12 +163,18 @@ const PASCAL_CAST_TARGET_KINDS: ReadonlySet = new Set(['class', 'struct' /** The values a TypeScript type may share its name with in its file, which a call or `new` of the name means. */ const TYPE_COMPANION_VALUE_KINDS: ReadonlySet = new Set(['function', 'class', 'component', 'constant', 'variable']); +/** + * The RTL's unit namespaces, as it spells them. Spelled otherwise, a + * receiver is most likely a variable that hides one (`data.GetObjectFrom`, + * `rest.EngineAdd`); only `system.` is the System unit's in any case. + */ const PASCAL_UNIT_PREFIXES = [ 'System.', 'Winapi.', 'Vcl.', 'Fmx.', 'Data.', 'Datasnap.', 'Soap.', 'Xml.', 'Web.', 'REST.', 'FireDAC.', 'IBX.', 'IdHTTP', 'IdTCP', 'IdSSL', ]; +/** Lowercased: Pascal names ignore case. */ const PASCAL_BUILT_INS = new Set([ 'System', 'SysUtils', 'Classes', 'Types', 'Variants', 'StrUtils', 'Math', 'DateUtils', 'IOUtils', 'Generics.Collections', 'Generics.Defaults', @@ -185,7 +193,7 @@ const PASCAL_BUILT_INS = new Set([ 'TList', 'TStringList', 'TStrings', 'TStream', 'TMemoryStream', 'TFileStream', 'Exception', 'EAbort', 'EConvertError', 'EAccessViolation', 'IInterface', 'IUnknown', -]); +].map((name) => name.toLowerCase())); const C_BUILT_INS = new Set([ // Standard C library functions @@ -1416,19 +1424,23 @@ export class ReferenceResolver { !reachesAmbientModule(ref, specifier!, declaringFile, this.context)) return null; } if (!resolved || ref.referenceKind !== 'calls') return resolved; + // A Pascal call to data runs only a routine the data holds, in scope (see + // gatePascalDataCall). + const call = ref.language === 'pascal' ? gatePascalDataCall(resolved, ref, this.context) : resolved; + if (!call) return null; - const target = this.nodeById(resolved.targetNodeId); - if (!target) return resolved; + const target = this.nodeById(call.targetNodeId); + if (!target) return call; const dot = ref.referenceName.lastIndexOf('.'); const memberName = dot >= 0 ? ref.referenceName.slice(dot + 1) : null; const forwarded = resolveAliasBinding(target, memberName, this.context); - if (!forwarded || forwarded.id === resolved.targetNodeId) return resolved; + if (!forwarded || forwarded.id === call.targetNodeId) return call; return this.gateLanguage({ - ...resolved, + ...call, targetNodeId: forwarded.id, - confidence: Math.min(resolved.confidence, 0.85), + confidence: Math.min(call.confidence, 0.85), }, ref); } @@ -1446,6 +1458,10 @@ export class ReferenceResolver { // routine its scope reaches, or through a receiver to a method of the // receiver's declared type; data of that name links nothing. if (isPascalParenlessRef(ref)) return matchPascalParenlessRef(ref, this.context); + // So does a Pascal call to a name the compiler or the System unit gives + // meaning to (`SizeOf`, `TypeInfo`, `length(S)`, `Pointer(P)`), whatever + // its case: only a declaration in the call's scope shadows it. + if (isPascalSystemCall(ref, this.context)) return matchPascalSystemCall(ref, this.context); // A Dart member read (`x.area`) links the getter the receiver's type // reaches, as a call, or nothing — never a guess by name (#2338). if (isDartMemberRead(ref)) return matchDartMemberRead(ref, this.context); @@ -3281,12 +3297,14 @@ export class ReferenceResolver { } } - // Pascal/Delphi built-ins and standard library units + // Pascal/Delphi built-ins, whatever their case: Pascal names ignore it + // (`length(S)`, `system.TypeInfo(T)`). And standard library units. if (ref.language === 'pascal') { - if (PASCAL_UNIT_PREFIXES.some((p) => name.startsWith(p))) { + const lower = name.toLowerCase(); + if (PASCAL_UNIT_PREFIXES.some((p) => name.startsWith(p)) || lower.startsWith('system.')) { return true; } - if (PASCAL_BUILT_INS.has(name)) { + if (PASCAL_BUILT_INS.has(lower)) { return true; } } diff --git a/src/resolution/pascal-scope.ts b/src/resolution/pascal-scope.ts index ec04b11b83..a128a0c946 100644 --- a/src/resolution/pascal-scope.ts +++ b/src/resolution/pascal-scope.ts @@ -34,6 +34,7 @@ import type { Node } from '../types'; import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; import { declaredSupertypes } from './name-matcher'; +import { PASCAL_SYSTEM_NAMES } from '../extraction/languages/pascal-system'; type Mode = 'call' | 'read' | 'ref' | 'recv'; interface Found { @@ -50,6 +51,15 @@ const UNIT_KINDS: ReadonlySet = new Set([ ]); /** What a class declares as its members. */ const MEMBER_KINDS: ReadonlySet = new Set(['method', 'field', 'property', 'constant', 'variable']); +/** Declarations that hold a value rather than run code. */ +const DATA_KINDS: ReadonlySet = new Set(['field', 'property', 'variable', 'constant', 'enum_member']); +/** + * The RTL's routine types whose names don't say so (`TProc`, `TFunc`), + * lowercased; the project's own are read from their declarations. + */ +const RTL_PROCEDURAL_TYPES: ReadonlySet = new Set([ + 'tproc', 'tfunc', 'tpredicate', 'tcomparison', 'tthreadprocedure', 'tlistsortcompare', 'tstringlistsortcompare', +]); const MAX_HIERARCHY_DEPTH = 8; /** Whether `ref` is a paren-less Pascal name (see the file comment), which only this module resolves. */ @@ -97,6 +107,194 @@ export function matchPascalParenlessRef(ref: UnresolvedRef, context: ResolutionC }; } +/** Whether `ref` is a bare Pascal call to one of the System unit's names (see PASCAL_SYSTEM_NAMES). */ +export function isPascalSystemCall(ref: UnresolvedRef, context: ResolutionContext): boolean { + return ref.language === 'pascal' && ref.referenceKind === 'calls' && PASCAL_SYSTEM_NAMES.has(ref.referenceName.toLowerCase()) && + isBareCall(ref, context); +} + +/** + * Whether a Pascal call named by a bare name is written bare: its expression + * starts with the name (`Reset(F)`). The last call of a chain is named bare + * too (`Reset` in `GetKey(P).Reset`), and its scope is its receiver's type. + */ +function isBareCall(ref: UnresolvedRef, context: ResolutionContext): boolean { + if (!/^[A-Za-z_]\w*$/.test(ref.referenceName)) return false; + const line = fileLines(ref.filePath!, context)[ref.line - 1] ?? ''; + return /^[A-Za-z_]\w*/.exec(line.slice(ref.column ?? 0))?.[0].toLowerCase() === ref.referenceName.toLowerCase(); +} + +/** + * A call to one of the System unit's names is the System unit's, which the + * project doesn't hold, unless a declaration in the call's scope shadows it: + * a member of the class (`Delete(I)` in a list class), a routine of the unit + * or the units it uses. A project type of that name is a typecast. Data of + * that name runs only when it holds a routine (see isProceduralData). A unit + * that declares one under a condition stands in for compilers without it + * (`{$IFNDEF UNICODE} UnicodeString = WideString; {$ENDIF}`): the code is + * most likely built with one that has it. + */ +export function matchPascalSystemCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const decl = declarationInScope(ref.referenceName.toLowerCase(), classOfCall(ref, context), ref, context); + if (!decl) return null; + if (DATA_KINDS.has(decl.node.kind) && !isProceduralData(decl.node, context)) return null; + if (!decl.node.qualifiedName.includes('::') && declaredUnderCondition(decl.node, context)) return null; + context.rememberNode?.(decl.node); + return { original: ref, targetNodeId: decl.node.id, confidence: decl.confidence, resolvedBy: decl.resolvedBy }; +} + +/** + * A Pascal call another strategy resolved to data — a field, a property, a + * variable, a constant — runs what the data holds, so it stands only for data + * whose declared type is a routine's. A bare name is what its scope declares + * (`FOnChange(Self)` in the class that declares its own `FOnChange`): the + * routine or the routine-holding data there, whichever namesake the strategy + * chose. Any other is a namesake: `length(S)` was a record's `length` field, + * `Pointer(P)` a property named `Pointer`. A name the calling routine + * declares itself, as a parameter, a local or a nested routine, is that + * declaration, which the graph doesn't hold. A name its scope doesn't + * declare at all is a `with` object's member (`with OCI do HandleAlloc(...)`) + * or a declaration the parse put in the wrong place: it stands for the one + * routine-holding field or variable the project declares by that name. + */ +export function gatePascalDataCall(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const target = context.getNodeById?.(result.targetNodeId); + if (!target || !DATA_KINDS.has(target.kind)) return result; + if (!isBareCall(ref, context)) return target.language === 'pascal' && isProceduralData(target, context) ? result : null; + const lower = ref.referenceName.toLowerCase(); + if (declaredInRoutine(lower, ref, context)) return null; + const decl = declarationInScope(lower, classOfCall(ref, context), ref, context); + if (!decl) { + const only = target.language === 'pascal' && pascalNamed(lower, context).every((n) => n.id === target.id || n.kind === 'import'); + return only && isProceduralData(target, context) ? result : null; + } + if (!isRoutine(decl.node) && !(DATA_KINDS.has(decl.node.kind) && isProceduralData(decl.node, context))) return null; + if (decl.node.id === target.id) return result; + context.rememberNode?.(decl.node); + return { ...result, targetNodeId: decl.node.id, confidence: decl.confidence, resolvedBy: decl.resolvedBy }; +} + +/** + * Whether the routine making a call declares the called name itself, as a + * parameter, a local or a nested routine (`function Hash(Item: Pointer; + * Hasher: THasher)` calling `Hasher(0, Item, 4)`): from the routine's header + * down to the call. + */ +function declaredInRoutine(lower: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + const lines = fileLines(ref.filePath!, context); + // The header in the text: a method's node may start at its declaration in + // the class, above the fields and the unit's variables. + const from = routineHeaderAt(ref.filePath!, ref.line, context); + const name = escapeRegExp(lower); + const mentions = new RegExp(`\\b${name}\\b`, 'i'); + const declares = new RegExp(`\\b(?:procedure|function)\\s+${name}\\b|\\b${name}\\s*(?:,\\s*[A-Za-z_]\\w*\\s*)*:(?!=)`, 'i'); + for (let i = Math.max(1, from); i <= ref.line && i <= lines.length; i++) { + const text = lines[i - 1]!; + if (mentions.test(text) && declares.test(text.replace(/\/\/.*$|\{[^}]*\}|'[^']*'/g, ' '))) return true; + } + return false; +} + +/** The line of the nearest routine header at column 0 at or above a line, else the line itself. */ +function routineHeaderAt(filePath: string, line: number, context: ResolutionContext): number { + headerClassAt(filePath, line, context); + const headers = memos(context).headerClasses.get(filePath) ?? []; + let found = line; + for (const h of headers) { + if (h.line > line) break; + if (h.routine) found = h.line; + } + return found; +} + +/** + * Whether a declaration is inside an `{$IFDEF}`, `{$IFNDEF}`, `{$IF}` or + * `{$IFOPT}` block, or its `{$ELSE}`. + */ +function declaredUnderCondition(node: Node, context: ResolutionContext): boolean { + const memo = memos(context).conditionDepths; + let depths = memo.get(node.filePath); + const lines = fileLines(node.filePath, context); + if (!depths) { + depths = []; + let depth = 0; + for (const text of lines) { + depths.push(depth); + if (text.includes('{$')) depth = conditionDepthAfter(text, depth); + } + memo.set(node.filePath, depths); + } + const line = lines[node.startLine - 1] ?? ''; + return conditionDepthAfter(line.slice(0, node.startColumn), depths[node.startLine - 1] ?? 0) > 0; +} + +/** The depth of conditional blocks after a line's directives (`{$IFDEF X}` opens one, `{$ENDIF}` closes it). */ +function conditionDepthAfter(text: string, depth: number): number { + for (const m of text.matchAll(/\{\$(ifdef|ifndef|ifopt|ifend|endif|if)\b/gi)) { + const directive = m[1]!.toLowerCase(); + depth = directive === 'endif' || directive === 'ifend' ? Math.max(0, depth - 1) : depth + 1; + } + return depth; +} + +/** + * The class whose method makes a call: the one the nearest method header + * above the call names in the source text (`procedure TFoo.Bar;`), which holds + * where the tree had errors and gave the code to the file, or to a routine + * that swallowed the ones after it; else the reference's method's class. + */ +function classOfCall(ref: UnresolvedRef, context: ResolutionContext): Node | null { + const qn = headerClassAt(ref.filePath!, ref.line, context); + const cls = qn ? classByQualifiedName(qn, ref.filePath!, context) : null; + if (cls) return cls; + const routine = context.getNodeById?.(ref.fromNodeId); + return routine?.kind === 'method' ? classOfRoutine(routine, context) : null; +} + +/** + * The class the nearest routine header at or above a line names + * (`procedure TFoo.Bar;` → `TFoo`), or null when it is a free routine's, or + * the unit's code resumes there: after an `end;` that starts its line, or at + * `initialization`. + */ +function headerClassAt(filePath: string, line: number, context: ResolutionContext): string | null { + const memo = memos(context).headerClasses; + let headers = memo.get(filePath); + if (!headers) { + headers = []; + const lines = fileLines(filePath, context); + const header = /^(?:class\s+)?(?:procedure|function|constructor|destructor|operator)\s+([A-Za-z_][\w.<>, ]*?)\s*[(:;]/i; + for (let i = 0; i < lines.length; i++) { + const text = lines[i]!; + // Each starts its line with one of c, d, e, f, i, o, p (either case). + const first = text.charCodeAt(0) | 0x20; + if (first !== 0x63 && first !== 0x64 && first !== 0x65 && first !== 0x66 && first !== 0x69 && first !== 0x6f && first !== 0x70) continue; + if (/^(?:end;|(?:initialization|finalization)\b)/i.test(text)) { + headers.push({ line: i + 1, classQn: null, routine: false }); + continue; + } + const m = header.exec(text); + if (!m) continue; + const parts = m[1]!.replace(/<[^>]*>/g, '').replace(/\s+/g, '').split('.'); + headers.push({ line: i + 1, classQn: parts.length > 1 ? parts.slice(0, -1).join('::') : null, routine: true }); + } + memo.set(filePath, headers); + } + let lo = 0; + let hi = headers.length - 1; + let hit: string | null = null; + while (lo <= hi) { + const mid = (lo + hi) >> 1; + if (headers[mid]!.line <= line) { + hit = headers[mid]!.classQn; + lo = mid + 1; + } else { + hi = mid - 1; + } + } + return hit; +} + // --------------------------------------------------------------------------- // Scope lookup // --------------------------------------------------------------------------- @@ -105,27 +303,36 @@ function isRoutine(node: Node): boolean { return node.kind === 'method' || node.kind === 'function'; } -/** A bare name, looked up the way Delphi looks it up from a routine of class `cls` (null: a free routine). */ +/** A bare name, looked up the way Delphi looks it up from a routine of class `cls` (null: a free routine): a routine, or null. */ function nameInScope(name: string, cls: Node | null, ref: UnresolvedRef, context: ResolutionContext): Found | null { - const lower = name.toLowerCase(); + const decl = declarationInScope(name.toLowerCase(), cls, ref, context); + return decl && isRoutine(decl.node) ? decl : null; +} + +/** + * The first declaration a bare name reaches from a routine of class `cls` + * (null: a free routine), whatever it declares: a member of the class or its + * ancestors, then a declaration of the unit, then of the units it uses, last + * first. Null when the project declares none of them, as for the System + * unit's names. + */ +function declarationInScope(lower: string, cls: Node | null, ref: UnresolvedRef, context: ResolutionContext): Found | null { const memo = memos(context).inScope; const key = `${cls?.id ?? ''}|${ref.filePath}|${lower}`; if (memo.has(key)) return memo.get(key)!; let found: Found | null = null; const member = cls ? memberInHierarchy(cls, lower, context) : null; if (member) { - if (isRoutine(member)) found = { node: member, resolvedBy: 'instance-method', confidence: 0.9 }; + found = { node: member, resolvedBy: 'instance-method', confidence: 0.9 }; } else { - const own = unitDeclaration(ref.filePath!, lower, context); + const own = ownUnitDeclaration(ref.filePath!, lower, context); if (own) { - if (isRoutine(own)) found = { node: own, resolvedBy: 'exact-match', confidence: 0.9 }; + found = { node: own, resolvedBy: 'exact-match', confidence: 0.9 }; } else { const used = usedUnitFiles(ref.filePath!, context); - for (let i = used.length - 1; i >= 0; i--) { - const decl = unitDeclaration(used[i]!, lower, context); - if (!decl) continue; - if (isRoutine(decl)) found = { node: decl, resolvedBy: 'import', confidence: 0.85 }; - break; + for (let i = used.length - 1; i >= 0 && !found; i--) { + const decl = unitDeclaration(used[i]!, lower, context, ref.filePath!); + if (decl) found = { node: decl, resolvedBy: 'import', confidence: 0.85 }; } } } @@ -141,7 +348,7 @@ function memberThroughPath(path: string[], cls: Node | null, ref: UnresolvedRef, let rest = path.slice(1); if (head.unitFile) { // `UnitName.Routine`, `UnitName.GlobalVar.Method`. - const decl = unitDeclaration(head.unitFile, rest[0]!.toLowerCase(), context); + const decl = unitDeclaration(head.unitFile, rest[0]!.toLowerCase(), context, ref.filePath!); if (!decl) return null; if (rest.length === 1) return isRoutine(decl) ? { node: decl, resolvedBy: 'import', confidence: 0.9 } : null; type = typeOfMember(decl, context); @@ -170,8 +377,9 @@ function headOf(name: string, cls: Node | null, ref: UnresolvedRef, context: Res const member = memberInHierarchy(cls, lower, context); if (member) return { type: typeOfMember(member, context) }; } - const decl = unitDeclaration(ref.filePath!, lower, context) ?? - usedUnitFiles(ref.filePath!, context).reduceRight((hit, file) => hit ?? unitDeclaration(file, lower, context), undefined); + const decl = ownUnitDeclaration(ref.filePath!, lower, context) ?? + usedUnitFiles(ref.filePath!, context).reduceRight( + (hit, file) => hit ?? unitDeclaration(file, lower, context, ref.filePath!), undefined); if (decl) return { type: CLASS_KINDS.has(decl.kind) ? decl : typeOfMember(decl, context) }; const unitFile = unitFileNamed(lower, ref.filePath!, context); return unitFile ? { type: null, unitFile } : null; @@ -191,10 +399,20 @@ interface Memos { usedFiles: Map; classByName: Map; unitsByName?: Map; + genericTypes?: Map; + projectFiles?: { byName: Map; pascal: string[] }; + includes: Map; + includers: Map; + ownFiles: Map; + unitFileSets: Map; unitFiles: Map; hierarchy: Map; memberTypes: Map; keywords: Map; + procedural: Map; + headerClasses: Map>; + outlines: Map; + conditionDepths: Map; } const MEMOS = new WeakMap(); @@ -204,7 +422,8 @@ function memos(context: ResolutionContext): Memos { m = { named: new Map(), inScope: new Map(), classOfRoutine: new Map(), classByQn: new Map(), ancestors: new Map(), usedFiles: new Map(), classByName: new Map(), unitFiles: new Map(), hierarchy: new Map(), - memberTypes: new Map(), keywords: new Map(), + memberTypes: new Map(), keywords: new Map(), procedural: new Map(), headerClasses: new Map(), outlines: new Map(), + conditionDepths: new Map(), unitFileSets: new Map(), includes: new Map(), includers: new Map(), ownFiles: new Map(), }; MEMOS.set(context, m); } @@ -248,11 +467,13 @@ function classByQualifiedName(qualifiedName: string, filePath: string, context: if (memo.has(key)) return memo.get(key)!; const exact = context.getNodesByQualifiedName(qualifiedName).filter((n) => n.language === 'pascal' && CLASS_KINDS.has(n.kind)); const lower = qualifiedName.toLowerCase(); - const cls = declarationOf(exact.filter((n) => n.filePath === filePath)) ?? - declarationOf(context.getNodesInFile(filePath).filter((n) => - CLASS_KINDS.has(n.kind) && n.qualifiedName.replace(/<[^>]*>/g, '').toLowerCase() === lower)) ?? - declarationOf(exact) ?? - classNamed(qualifiedName.split('::').pop()!, filePath, context); + let cls: Node | null = null; + for (const file of ownUnitFiles(filePath, context)) { + cls = declarationOf(exact.filter((n) => n.filePath === file)) ?? + declarationOf(outlineOf(file, context).classes.filter((n) => n.qualifiedName.replace(/<[^>]*>/g, '').toLowerCase() === lower)); + if (cls) break; + } + cls ??= declarationOf(exact) ?? classNamed(qualifiedName.split('::').pop()!, filePath, context); memo.set(key, cls); return cls; } @@ -334,46 +555,209 @@ function memberInHierarchy(cls: Node, lower: string, context: ResolutionContext) return found; } -/** A unit's top-level declaration of a name (its enum values included). */ -function unitDeclaration(filePath: string, lower: string, context: ResolutionContext): Node | undefined { +/** + * A unit's top-level declaration of a name (its enum values included), as + * code in `fromFile` sees it: another unit's member that its parse lost (see + * unitRoutineColumn) is not one, while in its own unit it is most likely the + * member the code calls. + */ +function unitDeclaration(filePath: string, lower: string, context: ResolutionContext, fromFile = filePath): Node | undefined { const decls = pascalNamed(lower, context).filter((n) => - n.filePath === filePath && UNIT_KINDS.has(n.kind) && (n.kind === 'enum_member' || !n.qualifiedName.includes('::'))); + n.filePath === filePath && UNIT_KINDS.has(n.kind) && (n.kind === 'enum_member' || !n.qualifiedName.includes('::')) && + !(fromFile !== filePath && n.kind === 'function' && n.startColumn > unitRoutineColumn(filePath, context))); // A routine declared in the interface and defined in the implementation is // one name; of overloads, one called without arguments first. return decls.find((n) => !isRoutine(n)) ?? decls.find((n) => !needsArguments(n)) ?? decls[0]; } +/** + * How far a unit's own routines are indented at most. In a unit whose + * `unit` header the parser lost, the members of a class it couldn't read + * come out as routines of the unit, indented deeper than the unit's own + * (`function Http: IHttpClient;` in a class body); elsewhere, any. + */ +function unitRoutineColumn(filePath: string, context: ResolutionContext): number { + return outlineOf(filePath, context).routineColumn; +} + +/** What scope lookups need of a file's nodes, read once: its classes, its `uses` in order, unitRoutineColumn. */ +function outlineOf(filePath: string, context: ResolutionContext): { classes: Node[]; imports: Node[]; routineColumn: number } { + const memo = memos(context).outlines; + let outline = memo.get(filePath); + if (!outline) { + const nodes = context.getNodesInFile(filePath); + let routineColumn = Infinity; + if (!nodes.some((n) => n.kind === 'module')) { + for (const n of nodes) { + if (n.kind === 'function' && !n.qualifiedName.includes('::')) routineColumn = Math.min(routineColumn, n.startColumn); + } + } + outline = { + classes: nodes.filter((n) => CLASS_KINDS.has(n.kind)), + imports: nodes.filter((n) => n.kind === 'import' && n.language === 'pascal') + .sort((a, b) => a.startLine - b.startLine || a.startColumn - b.startColumn), + routineColumn, + }; + memo.set(filePath, outline); + } + return outline; +} + /** The files of the units a file uses, in the order its `uses` clauses name them. */ -function usedUnitFiles(filePath: string, context: ResolutionContext): string[] { +function usedUnitFiles(fromFile: string, context: ResolutionContext): string[] { const memo = memos(context).usedFiles; - const hit = memo.get(filePath); + const hit = memo.get(fromFile); if (hit) return hit; + // The unit's `uses` clauses, then those of the files it includes: mORMot's + // `.inc` files open the unit's implementation with one. + const unit = unitFilesOf(fromFile, context); const files: string[] = []; - const imports = context.getNodesInFile(filePath) - .filter((n) => n.kind === 'import' && n.language === 'pascal') - .sort((a, b) => a.startLine - b.startLine || a.startColumn - b.startColumn); - for (const imp of imports) { - const file = unitFileNamed(imp.name.toLowerCase(), filePath, context); - if (file && file !== filePath && !files.includes(file)) files.push(file); - } - memo.set(filePath, files); + for (const own of unit) { + for (const imp of outlineOf(own, context).imports) { + const file = unitFileNamed(imp.name.toLowerCase(), own, context); + if (file && !unit.includes(file) && !files.includes(file)) files.push(file); + } + } + memo.set(fromFile, files); + return files; +} + +/** The unit's own declaration of a name, seen from one of its files (see ownUnitFiles). */ +function ownUnitDeclaration(fromFile: string, lower: string, context: ResolutionContext): Node | undefined { + for (const file of ownUnitFiles(fromFile, context)) { + const decl = unitDeclaration(file, lower, context); + if (decl) return decl; + } + return undefined; +} + +/** + * The files whose declarations are a file's unit's own, the file first: the + * unit is its own file and the include files it pulls in, so code in + * `mormot.core.os.pas` sees a variable `mormot.core.os.posix.inc` declares, + * and that file's code sees the unit's (see unitFilesOf). + */ +function ownUnitFiles(filePath: string, context: ResolutionContext): string[] { + const memo = memos(context).ownFiles; + let own = memo.get(filePath); + if (!own) { + const unit = unitFilesOf(filePath, context); + own = unit[0] === filePath ? unit : [filePath, ...unit.filter((f) => f !== filePath)]; + memo.set(filePath, own); + } + return own; +} + +/** + * The files one unit is made of: its own file, then the include files it + * pulls in (`{$I name.inc}`, `{$INCLUDE name}`), nested ones too. For an + * include file, the unit that includes it, nearest it when several do. + */ +function unitFilesOf(filePath: string, context: ResolutionContext): string[] { + const m = memos(context); + const hit = m.unitFileSets.get(filePath); + if (hit) return hit; + let root = filePath; + for (let i = 0; i < 3 && /\.inc$/i.test(root); i++) { + const up = includerOf(root, context); + if (!up || up === filePath) break; + root = up; + } + const files = [root]; + for (let i = 0; i < files.length && files.length < 32; i++) { + for (const inc of includesOf(files[i]!, context)) if (!files.includes(inc)) files.push(inc); + } + if (!files.includes(filePath)) files.push(filePath); + m.unitFileSets.set(filePath, files); return files; } +/** The include files a Pascal file pulls in (`{$I name.inc}`, `{$INCLUDE name}`), each the one of that name nearest it. */ +function includesOf(filePath: string, context: ResolutionContext): string[] { + const m = memos(context); + let includes = m.includes.get(filePath); + if (includes) return includes; + includes = []; + if (PASCAL_SOURCE.test(filePath)) { + for (const text of fileLines(filePath, context)) { + if (!text.includes('{$') || !/\{\$i(?:nclude)?\s/i.test(text)) continue; + for (const match of text.matchAll(/\{\$(?:I|INCLUDE)\s+['"]?([^'"}\s*]+)/gi)) { + let name = match[1]!.replace(/\\/g, '/'); + name = name.slice(name.lastIndexOf('/') + 1).toLowerCase(); + if (!name.includes('.')) name += '.inc'; + const inc = nearest(projectFiles(context).byName.get(name) ?? [], filePath); + if (inc && inc !== filePath && !includes.includes(inc)) includes.push(inc); + } + } + } + m.includes.set(filePath, includes); + return includes; +} + +/** The Pascal file that includes an include file: of the forty nearest it, the nearest whose text does. */ +function includerOf(incFile: string, context: ResolutionContext): string | null { + const m = memos(context); + if (m.includers.has(incFile)) return m.includers.get(incFile)!; + // The files of its directory first, those named most like it first of + // them (`mormot.core.os.pas` for `mormot.core.os.posix.inc`): how many + // characters of its path a file shares, the directory counting first. + const shared = (f: string): number => { + let n = 0; + while (n < f.length && n < incFile.length && f.charCodeAt(n) === incFile.charCodeAt(n)) n++; + return f.lastIndexOf('/', n - 1) * 1000 + n; + }; + const candidates = projectFiles(context).pascal.filter((f) => f !== incFile) + .map((f) => ({ f, n: shared(f) })).sort((a, b) => b.n - a.n).slice(0, 40); + const includer = candidates.find(({ f }) => includesOf(f, context).includes(incFile))?.f ?? null; + m.includers.set(incFile, includer); + return includer; +} + +const PASCAL_SOURCE = /\.(?:pas|dpr|dpk|lpr|inc)$/i; + +/** The project's files by lowercased name, and its Pascal sources. */ +function projectFiles(context: ResolutionContext): { byName: Map; pascal: string[] } { + const m = memos(context); + if (!m.projectFiles) { + const byName = new Map(); + const pascal: string[] = []; + for (const file of context.getAllFiles()) { + const base = file.slice(file.lastIndexOf('/') + 1).toLowerCase(); + const same = byName.get(base); + if (same) same.push(file); + else byName.set(base, [file]); + if (PASCAL_SOURCE.test(file)) pascal.push(file); + } + m.projectFiles = { byName, pascal }; + } + return m.projectFiles; +} + /** The file of the unit a name names, nearest to `fromFile` when several units share it. */ function unitFileNamed(lower: string, fromFile: string, context: ResolutionContext): string | null { const m = memos(context); const key = `${fromFile}|${lower}`; if (m.unitFiles.has(key)) return m.unitFiles.get(key)!; if (!m.unitsByName) { - m.unitsByName = new Map(); + const units = new Map(); + const add = (name: string, file: string): void => { + const files = units.get(name); + if (files) files.push(file); + else units.set(name, [file]); + }; + const headed = new Set(); for (const n of context.getNodesByKind('module')) { if (n.language !== 'pascal') continue; - const name = n.name.toLowerCase(); - const files = m.unitsByName.get(name); - if (files) files.push(n.filePath); - else m.unitsByName.set(name, [n.filePath]); + add(n.name.toLowerCase(), n.filePath); + headed.add(n.filePath); } + // A unit's file is named after it, which Delphi enforces: one whose + // `unit` header the parser lost to an error is the unit its file names. + for (const file of context.getAllFiles()) { + if (headed.has(file) || !/\.pas$/i.test(file)) continue; + add(file.slice(file.lastIndexOf('/') + 1, -'.pas'.length).toLowerCase(), file); + } + m.unitsByName = units; } const file = nearest(m.unitsByName.get(lower) ?? [], fromFile); m.unitFiles.set(key, file); @@ -381,23 +765,42 @@ function unitFileNamed(lower: string, fromFile: string, context: ResolutionConte } /** A class (or record or interface) a type name names, seen from `fromFile`. */ -function classNamed(typeName: string, fromFile: string, context: ResolutionContext): Node | null { +function classNamed(typeName: string, fromFile: string, context: ResolutionContext, depth = 0): Node | null { const simple = typeName.split('.').pop()!.toLowerCase(); const memo = memos(context).classByName; const key = `${fromFile}|${simple}`; if (memo.has(key)) return memo.get(key)!; - const candidates = pascalNamed(simple, context).filter((n) => CLASS_KINDS.has(n.kind)); - let cls = declarationOf(candidates.filter((n) => n.filePath === fromFile)); - if (!cls && candidates.length > 0) { - // The unit used last declares the name Delphi sees; else the one unit that does. + const named = pascalNamed(simple, context); + const visible = (kinds: ReadonlySet): Node[] => { + // In the unit's own files, then in the unit used last, which declares the + // name Delphi sees; else in the one unit that does. + const candidates = named.filter((n) => kinds.has(n.kind)); + for (const file of ownUnitFiles(fromFile, context)) { + const own = candidates.filter((n) => n.filePath === file); + if (own.length > 0) return own; + } const used = usedUnitFiles(fromFile, context); - for (let i = used.length - 1; i >= 0 && !cls; i--) cls = declarationOf(candidates.filter((n) => n.filePath === used[i])); - if (!cls && candidates.every((n) => n.filePath === candidates[0]!.filePath)) cls = declarationOf(candidates); + for (let i = used.length - 1; i >= 0; i--) { + const there = candidates.filter((n) => n.filePath === used[i]); + if (there.length > 0) return there; + } + return candidates.length > 0 && candidates.every((n) => n.filePath === candidates[0]!.filePath) ? candidates : []; + }; + let cls = declarationOf(visible(CLASS_KINDS)); + if (!cls && depth < 4) { + // A type alias names the class it stands for: `TVTAncestor = TVTAncestorVcl;`. + for (const alias of visible(ALIAS_KINDS)) { + const line = (fileLines(alias.filePath, context)[alias.startLine - 1] ?? '').replace(/\/\/.*$|\{[^}]*\}/g, ''); + const target = /=\s*([A-Za-z_][\w.]*)\s*;/.exec(line)?.[1]; + if (target && (cls = classNamed(target, alias.filePath, context, depth + 1))) break; + } } memo.set(key, cls); return cls; } +const ALIAS_KINDS: ReadonlySet = new Set(['type_alias']); + /** The file whose path shares the longest directory prefix with `from`. */ function nearest(files: string[], from: string): string | null { if (files.length <= 1) return files[0] ?? null; @@ -436,18 +839,119 @@ function typeOfMember(member: Node, context: ResolutionContext): Node | null { if (member.kind === 'method' || member.kind === 'function') { typeName = member.returnType ?? /:\s*([A-Za-z_][\w.]*)\s*;?\s*$/.exec(member.signature ?? '')?.[1]; } else { - const line = fileLines(member.filePath, context)[member.startLine - 1] ?? ''; - const name = escapeRegExp(member.name); - const re = member.kind === 'property' - ? new RegExp(`\\bproperty\\s+${name}\\s*(?:\\[[^\\]]*\\])?\\s*:\\s*([A-Za-z_][\\w.]*)`, 'i') - : new RegExp(`\\b${name}\\b(?:\\s*,\\s*[A-Za-z_]\\w*)*\\s*:\\s*([A-Za-z_][\\w.]*)`, 'i'); - typeName = re.exec(line)?.[1]; + typeName = /^([A-Za-z_][\w.]*)/.exec(declaredTypeText(member, context))?.[1]; } const type = typeName ? classNamed(typeName, member.filePath, context) : null; memo.set(member.id, type); return type; } +/** + * The text of the type a field, variable, constant or property declares, + * from its declaration on: `TFoo; // the foo` for `FFoo: TFoo; // the foo`. + * A name in a comma list (`FOnLoad,` above `FOnSave: TSaveEvent;`) takes the + * type the list ends with, a few lines down. Empty when none is written, as + * for a published `property OnClick;`. + */ +function declaredTypeText(node: Node, context: ResolutionContext): string { + const lines = fileLines(node.filePath, context); + const name = escapeRegExp(node.name); + const re = node.kind === 'property' + ? new RegExp(`\\bproperty\\s+${name}\\s*(?:\\[[^\\]]*\\])?\\s*:\\s*(.*)$`, 'i') + : new RegExp(`\\b${name}\\b(?:\\s*,\\s*[A-Za-z_]\\w*)*\\s*,?\\s*:(?!=)\\s*(.*)$`, 'i'); + let text = ''; + for (let i = node.startLine - 1; i < node.startLine + 5 && i < lines.length; i++) { + text += ' ' + lines[i]!.replace(/\/\/.*$|\{[^}]*\}|\(\*.*?\*\)/g, ' '); + const m = re.exec(text); + if (m) return m[1]!.trim(); + // A list goes on only after a comma. + if (node.kind === 'property' || !/,\s*$/.test(text)) break; + } + return ''; +} + +/** + * Whether data holds a routine, by its declared type: one written in place + * (`FProc: procedure of object`, `F: reference to function: Integer`), a + * project type declared as one (`TOnLog = procedure(...) of object;`), or a + * type the project doesn't declare that the RTL's names say is one + * (`TNotifyEvent`, `TWndMethod`, `TProc`). + */ +function isProceduralData(node: Node, context: ResolutionContext): boolean { + const memo = memos(context).procedural; + const hit = memo.get(node.id); + if (hit !== undefined) return hit; + const procedural = isProceduralTypeText(declaredTypeText(node, context), context, 0); + memo.set(node.id, procedural); + return procedural; +} + +/** + * Whether a written type is a routine's: written out + * (`reference to function: Integer`), or named, with the name's declaration + * saying so, through an alias to another such name + * (`TFormatter = TFunc;`). A name the project doesn't + * declare is a routine type when the RTL's names say so: `TProc`, `TFunc`, + * and the names Delphi gives them (`TNotifyEvent`, `TWndMethod`). + */ +function isProceduralTypeText(text: string, context: ResolutionContext, depth: number): boolean { + if (PROCEDURAL_TYPE.test(text)) return true; + const typeName = /^(?:type\s+)?([A-Za-z_][\w.]*)/i.exec(text)?.[1]?.split('.').pop()?.toLowerCase(); + if (!typeName || depth > 3) return false; + const types = typesNamed(typeName, context); + if (types.length === 0) return RTL_PROCEDURAL_TYPES.has(typeName) || /^t\w*(?:event|proc|method|callback)$/.test(typeName); + return types.some((t) => t.kind === 'type_alias' && isProceduralTypeText(aliasedTypeText(t, context), context, depth + 1)); +} + +/** + * The project's Pascal types a name names, generic ones by the name before + * their parameters: `TSetParams` for `TSetParams`. + */ +function typesNamed(lower: string, context: ResolutionContext): Node[] { + const m = memos(context); + if (!m.genericTypes) { + const generic = new Map(); + const add = (n: Node): void => { + const open = n.name.indexOf('<'); + if (n.language !== 'pascal' || open <= 0 || !(n.kind === 'type_alias' || CLASS_KINDS.has(n.kind))) return; + const base = n.name.slice(0, open).trim().toLowerCase(); + const list = generic.get(base); + if (list) list.push(n); + else generic.set(base, [n]); + }; + const kinds = ['type_alias', 'class', 'struct', 'interface'] as const; + // The names alone first, where the index can tell them, to fetch only + // the generic types. + const names = context.getNodeNamesOfKinds?.('pascal', kinds); + if (names) { + for (const name of new Set(names)) if (name.includes('<')) context.getNodesByName(name).forEach(add); + } else { + for (const kind of kinds) for (const n of context.iterateNodesByKind?.(kind) ?? context.getNodesByKind(kind)) add(n); + } + m.genericTypes = generic; + } + return pascalNamed(lower, context).filter((t) => t.kind === 'type_alias' || CLASS_KINDS.has(t.kind)) + .concat(m.genericTypes.get(lower) ?? []); +} + +/** + * What a type declaration says after its `=`, which may start on the next + * line: `procedure(...) of object` for `TOnHelper =` above it. + */ +function aliasedTypeText(alias: Node, context: ResolutionContext): string { + const lines = fileLines(alias.filePath, context); + let text = ''; + for (let i = alias.startLine - 1; i < alias.startLine + 2 && i < lines.length; i++) { + text += ' ' + lines[i]!.replace(/\/\/.*$|\{[^}]*\}|\(\*.*?\*\)/g, ' '); + const eq = text.indexOf('='); + if (eq >= 0 && text.slice(eq + 1).trim()) return text.slice(eq + 1).trim(); + } + return ''; +} + +/** A routine type written out: `procedure(...) of object`, `function: T`, `reference to procedure`. */ +const PROCEDURAL_TYPE = /^(?:reference\s+to\s+)?(?:procedure|function)\b/i; + /** Whether a routine has a parameter without a default value, which a call must pass. */ function needsArguments(routine: Node): boolean { const signature = routine.signature?.trim(); From 89cd08fab80fdf443d2851a7721fea286a9ea2d0 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 15:48:53 +0000 Subject: [PATCH 38/95] fix(explore): a long query, like a pasted log, answers in seconds on a large index (#2804) The prompt hook hands codegraph_explore the whole prompt, so a pasted log, document or source file arrives as a query of hundreds of words. findRelevantContext searched every word several times, and the reads behind those searches scanned the index each time: - findNodesByNameSubstring, four times per symbol word: `name LIKE '%word%'` with a leading wildcard reads every node of those kinds, about 80 ms each on vscode (562,636 nodes); - the LIKE fallback after an FTS miss (`name LIKE` or `qualified_name LIKE '%word%'`), about 120 ms per miss, and most words of a log miss; - the fuzzy fallback after that re-read `SELECT DISTINCT name` each time (about 150 ms), and getAllNodeNames, documented as cached, was not; - FTS joined nodes on the TEXT id instead of the rowid; - scorePathRelevance re-split every query word for every scored path, and the deprioritize predicate stat'ed codegraph.json per candidate. On vscode a 1.7k-character log took 80 s in explore, a 10k-character skill document 258 s, and the hook gave up on both. A per-statement profile put 65-80% of that in those scans. The fix: - A name-substring index answers the substring lookups from memory with exactly the rows SQLite returns, in its order: ASCII-only case folding and `_` as one character, like LIKE, and ties in length in the order the plan reads rows, the kind index's or rowid's. The plan is asked of SQLite once per statement shape, since a small index is scanned whole; a `%`, a language filter or any other plan goes to SQLite as before. - A word of letters and digits that no name and no letters-and-digits run of a qualified name holds can match nothing, so the LIKE fallback returns empty without scanning for it. - The fuzzy fallback reads the name list once and skips names outside its edit-distance length window. - These reads are memoized per database change stamp, like the dominant file (PR #1864): rebuilt after any write from any connection, never kept inside a transaction, dropped on rebind. - FTS joins on rowid, with CROSS JOIN keeping nodes_fts the outer loop as before. - Path relevance prepares each query once and each path is scored once per search; the deprioritize patterns are read once per search. - findRelevantContext searches at most 64 symbol names and 128 search terms of a query. Both lists put code-shaped words first, so a long paste is searched by its identifiers and its opening words. A 16-symbol bag (75 terms) stays whole. This is the one intended change in output, for long queries only. Validation (vscode ff0a86dc, median of 3 interleaved, fresh process): - explore, ordinary queries: 6 of 6 outputs byte-identical, 5.5-7.5 s fresh (the dominant-file aggregation is about 4 s of that) and 0.9-2.1 s warm vs 2.7-3.9 s on main; - explore, long queries: log 1.7k 80.1 to 10.7 s, markdown 3k 73.3 to 9.0 s, Java 3k 108.8 to 12.2 s, TS 3k 63.4 to 11.9 s, markdown 10k 258.3 to 10.8 s, Java 10k 144.7 to 11.1 s, TS 10k 146.7 to 9.6 s; - the real prompt hook on the issue's prompt shapes: short 7.2 to 7.0 s, log 84.2 to 12.9 s, markdown 3k 78.1 to 10.2 s, Java 3k 113.1 to 13.3 s, markdown 10k 261.7 to 11.3 s, Java 10k 146.3 to 11.1 s; - without the word caps, all 7 long queries gave byte-identical output 4.3-9.0x faster; - django 79502d08 and prometheus ed26d727: 11 of 11 queries byte-identical, tracebacks included, 1.0-2.9x faster; - the substring index against the LIKE scan on vscode: 4,830 of 4,830 lookups identical (805 words, snake_case included); - explore-long-query.test.ts: 13 tests; 8 fail on main's source. Issue #2620. Closed contributor PR #2621 (retsamekohC) was read for guidance; open PR #2260 (danusha2345) overlaps on the name-list memo and path relevance. Fixes #2620 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/explore-long-query.test.ts | 390 +++++++++++++++++++++++++++ src/context/index.ts | 57 ++-- src/db/name-substring-index.ts | 222 +++++++++++++++ src/db/queries.ts | 219 ++++++++++++++- src/index.ts | 51 +++- src/search/query-utils.ts | 93 +++++-- 7 files changed, 970 insertions(+), 63 deletions(-) create mode 100644 __tests__/explore-long-query.test.ts create mode 100644 src/db/name-substring-index.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 5afe0259cb..ae920b5276 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, a call through a module-level instance, like `_svc.delete(obj)` after `_svc = SceneService()`, now links to the method its class has or inherits, even when other classes have a method of that name. (#2582) - In Python, a call on a module imported from outside your project, like `subprocess.run(…)` after `import subprocess`, no longer links to a project method that happens to share its name. Thanks @Egrn for the report. (#2599) - `codegraph_explore`, the CLI `codegraph explore` and the prompt hook no longer stall for a minute or more on a large project after a full index; the step that finds the project's busiest file no longer slows to a crawl when a few files are huge, like a generated Android `R.java` (#2571). Thanks @yangfengfan2011. +- `codegraph_explore`, the CLI `codegraph explore` and the Claude Code prompt hook now answer a long query, like a pasted log, document or source file, in seconds on a large project instead of minutes. Every word of such a query used to search the whole index several times over, so the prompt hook ran out of time on long prompts and added nothing. Those searches now reuse what they read, and a very long query is searched by its code names and its opening words rather than by every word in it. Answers to ordinary questions and lists of symbol names are unchanged. Thanks @retsamekohC for the report and the first fix. (#2620) - The Claude Code prompt hook now goes by what you typed. Claude Code's desktop app puts status notes, like which other sessions are linked, in front of your prompt, and the hook read them as part of it, so every turn, even a one-word "yes", could get code context about the note's words, and so could a task notification, a subagent's report or a message from another session that came after one. The notes are now skipped, and a message from another session gets no context, as task notifications and subagent reports already didn't. Thanks @WonderMr for the report and @yang0228 for the first fix. (#2635) - `codegraph_explore` and `codegraph context` now find your project's busiest file, and favor the code around it, even when twenty or more test or generated files are busier still, such as a package of protobuf stubs. Before, the search looked only at the 20 busiest files and then set the tests and generated ones aside, so it could find no file at all, or judge one far ahead of the next when it was not. `getTopRouteFile()` in the library counts every file's routes the same way. Thanks @inth3shadows for the report. (#2578) - In JavaScript and TypeScript, a call written as the body of an `if`, `while`, `for` or `do` without braces, like `if (e.key === "Escape") setOpen(false);`, no longer links to a same-named property or method in another file when the name is the file's own, such as a `setOpen` from `useState` (#2585). diff --git a/__tests__/explore-long-query.test.ts b/__tests__/explore-long-query.test.ts new file mode 100644 index 0000000000..5f635564fe --- /dev/null +++ b/__tests__/explore-long-query.test.ts @@ -0,0 +1,390 @@ +/** + * Long `codegraph_explore` queries on a large index (#2620). + * + * The prompt hook hands explore the whole prompt, so a pasted log, document + * or source file reaches it as a query of hundreds of words, and every word + * cost several reads that each scanned the index: on a 560k-node index such + * a query took one to four minutes and the hook gave up. These tests pin the + * two halves of the fix: + * + * - the per-word reads give exactly the rows they gave before, now from + * memory (the name-substring lookups, the fuzzy fallback's name list, the + * LIKE fallback's "nothing to find" check), read once per database state; + * - a query searches at most MAX_QUERY_SYMBOLS names and MAX_QUERY_TERMS + * terms, code-shaped words first, so its cost no longer grows with its + * length. + */ +import { describe, it, expect, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src'; +import { QueryBuilder } from '../src/db/queries'; +import { MAX_QUERY_SYMBOLS, MAX_QUERY_TERMS } from '../src/context'; +import { scorePathRelevance, extractSearchTerms, normalizeNameToken, isTestFile } from '../src/search/query-utils'; +import type { SqliteDatabase } from '../src/db/sqlite-adapter'; + +function queriesOf(cg: CodeGraph): QueryBuilder { + return (cg as unknown as { queries: QueryBuilder }).queries; +} +function rawDb(cg: CodeGraph): SqliteDatabase { + return (cg as unknown as { db: { getDb(): SqliteDatabase } }).db.getDb(); +} + +/** Count the statements run whose SQL matches `pattern`, through the adapter. */ +function countStatements(db: SqliteDatabase, pattern: RegExp): { count: number } { + const counter = { count: 0 }; + const prepare = db.prepare.bind(db); + vi.spyOn(db, 'prepare').mockImplementation((sql: string) => { + const stmt = prepare(sql); + if (!pattern.test(sql)) return stmt; + const wrap = any>(fn: T): T => ((...args: any[]) => { counter.count++; return fn(...args); }) as T; + return { ...stmt, all: wrap(stmt.all.bind(stmt)), get: wrap(stmt.get.bind(stmt)), iterate: wrap(stmt.iterate.bind(stmt)), run: stmt.run.bind(stmt) }; + }); + return counter; +} + +const DEF = ['class', 'interface', 'struct', 'union', 'trait', 'protocol', 'enum', 'type_alias']; +const CALL = ['function', 'method', 'component']; +const SEED = ['function', 'method', 'component', 'variable', 'constant']; + +/** The lookup as it was before the index: a LIKE scan, ties left to the plan. */ +function likeScan(db: SqliteDatabase, substring: string, kinds: string[], excludePrefix: boolean, limit: number): string[] { + let sql = 'SELECT nodes.*, 1.0 as score FROM nodes WHERE name LIKE ?'; + const params: (string | number)[] = [`%${substring}%`]; + if (excludePrefix) { sql += ' AND name NOT LIKE ?'; params.push(`${substring}%`); } + sql += ` AND kind IN (${kinds.map(() => '?').join(',')}) ORDER BY length(name) ASC LIMIT ?`; + return (db.prepare(sql).all(...params, ...kinds, limit) as Array<{ id: string }>).map((row) => row.id); +} + +/** Many names of equal length across kinds and files, so ties decide the order. */ +function writeTieProject(dir: string): void { + const files: Record = {}; + for (let f = 0; f < 6; f++) { + const lines: string[] = []; + for (let i = 0; i < 12; i++) { + const n = (f * 7 + i * 5) % 12; + lines.push(`export function load${n}Item${f}(): void {}`); + lines.push(`export class Item${n}Loader${f} { loadItem${n}(): void {} }`); + lines.push(`export interface Item${n}Shape${f} { loadItem${n}Shape(): void; }`); + lines.push(`export const ITEM_${n}_LOADER_${f} = ${i};`); + } + lines.push('export function load_item_snake(): void {}'); + lines.push('export function loadXitemXsnake(): void {}'); + lines.push('export class LOADITEMLOUD {}'); + lines.push('export function größeLaden(): void {}'); + lines.push('export function GrößeLaden(): void {}'); + lines.push('export function ÉtatLoader(): void {}'); + lines.push('export function étatLoader(): void {}'); + lines.push('export function 𝑥ItemLoader(): void {}'); + files[`src/part${(f * 5) % 6}/items${f}.ts`] = lines.join('\n') + '\n'; + } + for (const [rel, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), text); + } +} + +const PATTERNS = [ + 'Item', 'item', 'ITEM', 'Loader', 'Load', 'load', 'Item1', 'Item1loader', 'Loaditem', 'Shape', 'Oader', 'O', + 'Load_item', 'Item_', '_', 'I_em', 'Load_i', 'Item_1', 'Größe', 'größe', 'GRÖSSE', 'État', 'état', 'Etat', + '𝑥', '𝑥item', 'Xitem', 'Loud', 'Missingword', 'Item.', 'Item(', +]; + +describe('name-substring lookups give the rows a LIKE scan gives (#2620)', () => { + let dir: string; + const open: CodeGraph[] = []; + afterEach(() => { + for (const cg of open.splice(0)) cg.close(); + vi.restoreAllMocks(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function setup(): Promise { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-substr-')); + writeTieProject(dir); + const cg = await CodeGraph.init(dir, { index: true }); + open.push(cg); + return cg; + } + + function expectSameRows(cg: CodeGraph, plan: 'rowid' | 'index'): void { + const q = queriesOf(cg); + const db = rawDb(cg); + let compared = 0; + for (const substring of PATTERNS) { + for (const kinds of [DEF, CALL, SEED, ['method'], ['constant', 'function']]) { + for (const excludePrefix of [false, true]) { + for (const limit of [200, 7, 1]) { + const expected = likeScan(db, substring, kinds, excludePrefix, limit); + const actual = q.findNodesByNameSubstring(substring, { kinds: kinds as never, excludePrefix, limit }).map((r) => r.node.id); + expect(actual, `${substring} ${kinds.join(',')} excludePrefix=${excludePrefix} limit=${limit}`).toEqual(expected); + compared++; + } + } + } + } + expect(compared).toBeGreaterThan(900); + // The answers came from the index, under the plan SQLite chose. + const memo = (q as unknown as { readMemos: Map } | null }> }) + .readMemos.get('nameSubstringIndex'); + expect(memo?.value).toBeTruthy(); + const plans = [...memo!.value!.plans.values()]; + expect(plans).toContain(plan); + expect(plans).not.toContain(null); + } + + it('matches when SQLite scans the table (a small index)', async () => { + const cg = await setup(); + const plan = rawDb(cg).prepare( + `EXPLAIN QUERY PLAN SELECT * FROM nodes WHERE name LIKE '%x%' AND kind IN (${DEF.map(() => '?').join(',')}) ORDER BY length(name) LIMIT 200`, + ).all(...DEF) as Array<{ detail: string }>; + expect(plan.map((r) => r.detail)).toContain('SCAN nodes'); + expectSameRows(cg, 'rowid'); + }); + + it('matches when SQLite reads through the kind index (a large index)', async () => { + const cg = await setup(); + const db = rawDb(cg); + // The statistics of a large index (vscode's), so the planner reads by kind. + db.exec('ANALYZE'); + db.exec("UPDATE sqlite_stat1 SET stat = '562636' || substr(stat, instr(stat, ' ')) WHERE tbl = 'nodes' AND instr(stat, ' ') > 0"); + db.exec("UPDATE sqlite_stat1 SET stat = '562636 1001 3 1 1' WHERE idx = 'idx_nodes_kind'"); + db.exec('ANALYZE sqlite_schema'); + const plan = db.prepare( + `EXPLAIN QUERY PLAN SELECT * FROM nodes WHERE name LIKE '%x%' AND kind IN (${DEF.map(() => '?').join(',')}) ORDER BY length(name) LIMIT 200`, + ).all(...DEF) as Array<{ detail: string }>; + expect(plan.map((r) => r.detail).some((d) => d.startsWith('SEARCH nodes USING INDEX idx_nodes_kind'))).toBe(true); + expectSameRows(cg, 'index'); + }); + + it('asks SQLite inside a transaction, and for a lookup without a kind filter', async () => { + const cg = await setup(); + const q = queriesOf(cg); + const db = rawDb(cg); + const scans = countStatements(db, /^\s*SELECT nodes\.\*, 1\.0 as score\s+FROM nodes\s+WHERE name LIKE \?/); + db.exec('BEGIN'); + try { + expect(q.findNodesByNameSubstring('Item', { kinds: DEF as never, limit: 50 }).map((r) => r.node.id)) + .toEqual(likeScan(db, 'Item', DEF, false, 50)); + } finally { + db.exec('ROLLBACK'); + } + expect(scans.count).toBe(2); // the lookup itself, and the comparison + q.findNodesByNameSubstring('Item', { limit: 50 }); + expect(scans.count).toBe(3); + q.findNodesByNameSubstring('Item', { kinds: DEF as never, limit: 50 }); + expect(scans.count).toBe(3); + }); +}); + +describe('per-word reads happen once per database state (#2620)', () => { + let dir: string; + const open: CodeGraph[] = []; + afterEach(() => { + for (const cg of open.splice(0)) cg.close(); + vi.restoreAllMocks(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function setup(): Promise { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-perword-')); + fs.mkdirSync(path.join(dir, 'src', 'zorbtown'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'src', 'zorbtown', 'orders.ts'), [ + 'export class OrderStateMachine {', + ' submitOrder(): boolean { return this.transition("submitted"); }', + ' transition(state: string): boolean { return state.length > 0; }', + '}', + 'export function linkSessions(ids: string[]): string[] { return ids; }', + ].join('\n') + '\n'); + const cg = await CodeGraph.init(dir, { index: true }); + open.push(cg); + return cg; + } + + it('reads the name list once for every word the fuzzy fallback tries', async () => { + const cg = await setup(); + const reads = vi.spyOn(queriesOf(cg), 'getAllNodeNames'); + for (const word of ['submitordr', 'transitoin', 'linksesions', 'qwertyuiop', 'zxcvbnmasd']) cg.searchNodes(word); + expect(reads).toHaveBeenCalledTimes(1); + // A typo still finds its symbol. + expect(cg.searchNodes('transitoin').map((r) => r.node.name)).toContain('transition'); + }); + + it('reads the names again after a sync or another connection\'s write', async () => { + const cg = await setup(); + expect(cg.searchNodes('ordermachne').map((r) => r.node.name)).not.toContain('OrderMachine'); + fs.writeFileSync(path.join(dir, 'src', 'zorbtown', 'more.ts'), 'export class OrderMachine {}\n'); + await cg.sync(); + expect(cg.searchNodes('ordermachne').map((r) => r.node.name)).toContain('OrderMachine'); + + const reader = await CodeGraph.open(dir); + open.push(reader); + expect(reader.searchNodes('invoicemachne').map((r) => r.node.name)).not.toContain('InvoiceMachine'); + fs.writeFileSync(path.join(dir, 'src', 'zorbtown', 'invoice.ts'), 'export class InvoiceMachine {}\n'); + await cg.sync(); + expect(reader.searchNodes('invoicemachne').map((r) => r.node.name)).toContain('InvoiceMachine'); + }); + + it('reads the names fresh inside a transaction, and does not keep them', async () => { + const cg = await setup(); + const db = rawDb(cg); + expect(cg.searchNodes('linksesions').map((r) => r.node.name)).toContain('linkSessions'); + db.exec('BEGIN'); + try { + db.exec("DELETE FROM nodes WHERE name = 'linkSessions'"); + expect(cg.searchNodes('linksesions').map((r) => r.node.name)).not.toContain('linkSessions'); + } finally { + db.exec('ROLLBACK'); + } + expect(cg.searchNodes('linksesions').map((r) => r.node.name)).toContain('linkSessions'); + }); + + it('skips the LIKE scan for a word no name or qualified name holds, and still finds one inside a qualified name', async () => { + const cg = await setup(); + const scans = countStatements(rawDb(cg), /qualified_name LIKE \?/); + // No token starts with these, so FTS misses and the LIKE fallback runs. + expect(cg.searchNodes('qwzxv')).toEqual([]); + expect(cg.searchNodes('plmokn')).toEqual([]); + expect(scans.count).toBe(0); + // "rbtow" is inside the folder name in a file's qualified name (its path), + // and inside no symbol name. + const inPath = cg.searchNodes('rbtow'); + expect(inPath.map((r) => r.node.filePath)).toContain('src/zorbtown/orders.ts'); + // "tateMach" is inside a name. + expect(cg.searchNodes('tatemach').map((r) => r.node.name)).toContain('OrderStateMachine'); + expect(scans.count).toBe(2); + }); +}); + +describe('a long query searches a bounded number of words (#2620)', () => { + let dir: string; + const open: CodeGraph[] = []; + afterEach(() => { + for (const cg of open.splice(0)) cg.close(); + vi.restoreAllMocks(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function setup(): Promise { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-longquery-')); + fs.writeFileSync(path.join(dir, 'orders.ts'), [ + 'export class OrderStateMachine {', + ' submitOrder(): boolean { return this.transition("submitted"); }', + ' transition(state: string): boolean { return state.length > 0; }', + '}', + 'export function reconcileLedger(): void {}', + 'export function zebrafinchquota(): void {}', + ].join('\n') + '\n'); + const cg = await CodeGraph.init(dir, { index: true }); + open.push(cg); + return cg; + } + + /** Distinct lowercase filler words that name nothing in the project. */ + function filler(n: number): string { + const words: string[] = []; + for (let i = 0; i < n; i++) words.push(`word${String.fromCharCode(97 + (i % 26))}${String.fromCharCode(97 + Math.floor(i / 26) % 26)}x`); + return words.join(' '); + } + + it('runs a bounded number of searches however long the query is', async () => { + const cg = await setup(); + const q = queriesOf(cg); + const search = vi.spyOn(q, 'searchNodes'); + const substring = vi.spyOn(q, 'findNodesByNameSubstring'); + await cg.findRelevantContext(`how does submitOrder work? ${filler(400)}`); + // A name is searched as a definition prefix (with its stems) and by four + // substring lookups; a term once. + expect(search.mock.calls.length).toBeLessThanOrEqual(MAX_QUERY_SYMBOLS * 4 + MAX_QUERY_TERMS); + expect(substring.mock.calls.length).toBeLessThanOrEqual(MAX_QUERY_SYMBOLS * 4); + }); + + it('searches a bag of symbol names whole', async () => { + const cg = await setup(); + const search = vi.spyOn(queriesOf(cg), 'searchNodes'); + const bag = 'AuthService AuthController TokenStore SessionManager UserRepository PasswordHasher JwtStrategy ' + + 'RefreshTokenService LoginHandler LogoutHandler AuthGuard RolesGuard PermissionService AuditLogger ' + + 'RateLimiter CsrfMiddleware reconcileLedger'; + expect(extractSearchTerms(bag).length).toBeGreaterThan(64); + expect(extractSearchTerms(bag).length).toBeLessThanOrEqual(MAX_QUERY_TERMS); + await cg.findRelevantContext(bag); + const searched = new Set(search.mock.calls.map((c) => String(c[0]).toLowerCase())); + for (const term of extractSearchTerms(bag)) expect(searched).toContain(term); + }); + + it('still searches every word of a query within the bound', async () => { + const cg = await setup(); + const search = vi.spyOn(queriesOf(cg), 'searchNodes'); + await cg.findRelevantContext(`${filler(20)} zebrafinchquota`); + const searched = new Set(search.mock.calls.map((c) => String(c[0]).toLowerCase())); + expect(searched).toContain('zebrafinchquota'); + expect(searched).toContain('wordaax'); + }); + + it('keeps the identifiers of a long paste, wherever they are in it', async () => { + const cg = await setup(); + const result = await cg.findRelevantContext(`${filler(300)} where does reconcileLedger run?`); + expect([...result.nodes.values()].map((n) => n.name)).toContain('reconcileLedger'); + }); + + it('drops the plain words past the bound', async () => { + const cg = await setup(); + const search = vi.spyOn(queriesOf(cg), 'searchNodes'); + await cg.findRelevantContext(`${filler(300)} zebrafinchquota`); + const searched = new Set(search.mock.calls.map((c) => String(c[0]).toLowerCase())); + expect(searched).not.toContain('zebrafinchquota'); + }); +}); + +describe('path relevance scores are unchanged by the per-query cache (#2620)', () => { + /** scorePathRelevance as it was, re-splitting the query for every path. */ + function reference(filePath: string, query: string, projectNameTokens?: Set, isDeprioritized?: boolean): number { + const pathLower = filePath.toLowerCase(); + const fileName = path.basename(filePath).toLowerCase(); + const dirName = path.dirname(filePath).toLowerCase(); + let score = 0; + const allWords = query.split(/\s+/).filter((w) => w.length > 0); + if (allWords.length === 0) return 0; + const words = projectNameTokens && projectNameTokens.size > 0 + ? allWords.filter((w) => !projectNameTokens.has(normalizeNameToken(w))) + : allWords; + const scored = words.length > 0 ? words : allWords; + for (const word of scored) { + const subtokens = extractSearchTerms(word, { stems: false }); + if (subtokens.length === 0) continue; + if (subtokens.some((t) => fileName.includes(t))) score += 10; + if (subtokens.some((t) => dirName.includes(t))) score += 5; + else if (subtokens.some((t) => pathLower.includes(t))) score += 3; + } + const queryLower = query.toLowerCase(); + const isTestQuery = queryLower.includes('test') || queryLower.includes('spec'); + if ((!isTestQuery && isTestFile(filePath)) || isDeprioritized === true) score -= 15; + return score; + } + + it('scores every path as before, repeated words and project names included', () => { + const queries = [ + 'how does the order service submit an order', + 'OrderService.submit order ORDER order_service submitOrder', + 'SuperBizAgent agent routes for the agent', + 'test the order service', ' ', '', 'spec', 'the of and', + Array.from({ length: 200 }, (_, i) => ['order', 'service', 'ledger', 'Submit', 'superbizagent'][i % 5]).join(' '), + ]; + const paths = [ + 'src/orders/order-service.ts', 'src/orders/__tests__/order-service.test.ts', 'SuperBizAgent/agent/routes.ts', + 'lib/ledger.rb', 'packages/submit/index.js', 'examples/order/main.go', 'README.md', 'a', + ]; + const tokens = [undefined, new Set(), new Set(['superbizagent'])]; + for (const query of queries) { + for (const filePath of paths) { + for (const projectNameTokens of tokens) { + for (const dep of [undefined, true, false]) { + expect(scorePathRelevance(filePath, query, projectNameTokens, dep), `${query} | ${filePath}`) + .toBe(reference(filePath, query, projectNameTokens, dep)); + } + } + } + } + }); +}); diff --git a/src/context/index.ts b/src/context/index.ts index 4f924eb40a..189329a9db 100644 --- a/src/context/index.ts +++ b/src/context/index.ts @@ -37,6 +37,21 @@ import { describeSynthesizedHop } from '../graph/synthesized-hop'; */ const MAX_COPYBOOK_DECLARATION_ENTRIES = 3; +/** + * The most symbol names, and search terms, of one query that + * `findRelevantContext` searches the index for. A question or a bag of symbol + * names stays well below both: a name like `RefreshTokenService` is one + * symbol but several terms (`refreshtokenservice`, `refresh`, `token`, + * `service`, stems). A pasted log, document or source file runs to hundreds + * of words, and every word costs several searches of the index, so such a + * query took minutes on a large project and the prompt hook gave up on it + * (#2620). Both lists put code-shaped words (camelCase, snake_case, dotted + * names) first and keep the order they appear in, so a long query is searched + * by its identifiers and its opening words. + */ +export const MAX_QUERY_SYMBOLS = 64; +export const MAX_QUERY_TERMS = 128; + /** * Extract likely symbol names from a natural language query * @@ -220,9 +235,14 @@ export class ContextBuilder { * and its non-production budget cap are deliberately NOT joined: those REMOVE * content, and `deprioritize` is a ranking lever by definition — `exclude` is * the lever for taking things out of reach. + * + * Returned as a predicate with the patterns read once, for the candidate + * loops of one search: asking the live matcher per candidate re-checked + * `codegraph.json` hundreds of thousands of times on a long query (#2620). */ - private isDeprioritized(filePath: string): boolean { - return this.queries.getDeprioritizedPathMatcher()?.(filePath) ?? false; + private deprioritizedPaths(): (filePath: string) => boolean { + const matcher = this.queries.getDeprioritizedPathSnapshot?.() ?? this.queries.getDeprioritizedPathMatcher?.(); + return (filePath) => matcher?.(filePath) ?? false; } /** @@ -473,7 +493,7 @@ export class ContextBuilder { // === HYBRID SEARCH === // Step 1: Extract potential symbol names from query - const symbolsFromQuery = extractSymbolsFromQuery(query); + const symbolsFromQuery = extractSymbolsFromQuery(query).slice(0, MAX_QUERY_SYMBOLS); logDebug('Extracted symbols from query', { query, symbols: symbolsFromQuery }); // Step 2: Look up exact matches for extracted symbols @@ -639,7 +659,8 @@ export class ContextBuilder { // fell through to the fuzzy fallback and returned whatever program name // sat within edit distance (`CVACT01Y` → `CBACT01C`). const searchTerms = extractSearchTerms(query) - .filter((term) => !copybooks.members.has(term.toUpperCase())); + .filter((term) => !copybooks.members.has(term.toUpperCase())) + .slice(0, MAX_QUERY_TERMS); if (searchTerms.length > 0) { // Search each term individually to get broader coverage, // then boost results that match multiple terms @@ -759,7 +780,7 @@ export class ContextBuilder { // than nodes matching just one generic term. Without this, "ExecutionUtils" // (matches only "execution") fills budget slots meant for "ShardSearchRequest" // (matches "shard" + "search" + "request"). - const queryTermsForBoost = extractSearchTerms(query); + const queryTermsForBoost = extractSearchTerms(query).slice(0, MAX_QUERY_TERMS); if (queryTermsForBoost.length >= 2) { // Group terms that are substrings of each other (stem variants of the same // root word). "indexed", "indexe", "index" should count as ONE concept match, @@ -853,6 +874,18 @@ export class ContextBuilder { // SEPARATE LIKE batch so one hot single-word term can't crowd classes // out of the length-ordered 200-row batch. const camelCallableKinds: NodeKind[] = ['function', 'method', 'component']; + // A path's relevance to the query depends on nothing else, and the + // candidates of every term share their files: score each path once. + const isDeprioritized = this.deprioritizedPaths(); + const pathScores = new Map(); + const pathScoreOf = (filePath: string): number => { + let score = pathScores.get(filePath); + if (score === undefined) { + score = scorePathRelevance(filePath, query, undefined, isDeprioritized(filePath)); + pathScores.set(filePath, score); + } + return score; + }; const camelSearchedTerms = new Set(); const searchIdSet = new Set(searchResults.map(r => r.node.id)); // Track per-node term hits for multi-term boosting @@ -902,12 +935,7 @@ export class ContextBuilder { if (searchIdSet.has(r.node.id)) continue; if (isTestFile(r.node.filePath) && !isTestQuery) continue; - const pathScore = scorePathRelevance( - r.node.filePath, - query, - undefined, - this.isDeprioritized(r.node.filePath), - ); + const pathScore = pathScoreOf(r.node.filePath); const brevityBonus = Math.max(0, 6 - (name.length - titleCased.length) / 4); termCandidates.push({ node: r.node, score: 8 + brevityBonus + pathScore }); } @@ -994,12 +1022,7 @@ export class ContextBuilder { const compoundResults: SearchResult[] = []; for (const [, entry] of compoundTermMap) { if (entry.terms.size >= 2) { - const pathScore = scorePathRelevance( - entry.node.filePath, - query, - undefined, - this.isDeprioritized(entry.node.filePath), - ); + const pathScore = pathScoreOf(entry.node.filePath); const brevityBonus = Math.max(0, 6 - entry.node.name.length / 8); compoundResults.push({ node: entry.node, diff --git a/src/db/name-substring-index.ts b/src/db/name-substring-index.ts new file mode 100644 index 0000000000..8ba3491e86 --- /dev/null +++ b/src/db/name-substring-index.ts @@ -0,0 +1,222 @@ +/** + * An in-memory answer to the substring lookup `codegraph_explore` runs for + * every word of its query (#2620): + * + * SELECT … FROM nodes WHERE name LIKE '%word%' [AND name NOT LIKE 'word%'] + * AND kind IN (…) ORDER BY length(name) LIMIT n + * + * A leading `%` can't use an index, so SQLite reads every node of those kinds + * for each call: about 80 ms on a 560k-node index, four calls a word, and a + * pasted log or document has hundreds of words. This index reads the names + * once and answers each call with one string search. + * + * It returns exactly what SQLite would. LIKE folds case for ASCII letters only, + * `_` matches any one character, and `length()` counts characters, not UTF-16 + * units, so all three are done that way here. SQLite leaves rows of equal + * length in the order its plan read them: through `idx_nodes_kind` (kind, + * file_path, start_line, id) on a large index, by rowid when it scans a small + * one. The index keeps both orders, and the caller says which plan SQLite + * chose. A pattern with a `%` is left to SQLite. + */ + +/** + * The kinds the index holds: the definitions and callables explore's + * camel-case and compound steps ask about, and the variables and constants its + * named-symbol seeding adds. + */ +export const NAME_SUBSTRING_KINDS: readonly string[] = [ + 'class', 'component', 'constant', 'enum', 'function', 'interface', 'method', + 'protocol', 'struct', 'trait', 'type_alias', 'union', 'variable', +]; + +/** Separates the names in {@link NameSubstringIndex.hay}; no name may contain it. */ +const SEPARATOR = '\u0000'; + +export interface NameSubstringIndex { + /** Kind codes: `kinds[entryKind[e]]` is entry `e`'s kind. */ + readonly kinds: readonly string[]; + /** Every distinct ASCII-lowercased name, each followed by {@link SEPARATOR}. */ + readonly hay: string; + /** Where each distinct name starts in `hay`, plus one past the end. */ + readonly starts: Int32Array; + /** The entries of distinct name `i` are `nameEntries[nameEntryStart[i] … nameEntryStart[i+1])`, ascending. */ + readonly nameEntryStart: Int32Array; + readonly nameEntries: Int32Array; + /** Per entry, in index order. */ + readonly entryRowid: Float64Array; + readonly entryKind: Uint8Array; + readonly entryLength: Int32Array; +} + +/** Lowercase ASCII letters only, as SQLite's LIKE compares them. */ +function asciiLower(text: string): string { + return /[A-Z]/.test(text) ? text.replace(/[A-Z]+/g, (run) => run.toLowerCase()) : text; +} + +/** `length(text)` as SQLite counts it: characters, so a surrogate pair is one. */ +function sqliteLength(text: string): number { + let length = text.length; + for (let i = 0; i < text.length; i++) { + const code = text.charCodeAt(i); + if (code >= 0xd800 && code <= 0xdbff && i + 1 < text.length) { + const next = text.charCodeAt(i + 1); + if (next >= 0xdc00 && next <= 0xdfff) { length--; i++; } + } + } + return length; +} + +/** + * Build the index from rows of {@link NAME_SUBSTRING_KINDS} read in + * `idx_nodes_kind` order, which becomes the entries' order. Returns null when + * a name holds the separator, so the caller keeps asking SQLite. + */ +export function buildNameSubstringIndex( + rows: Iterable<{ rowid: number; kind: string; name: string }>, +): NameSubstringIndex | null { + const kindCodes = new Map(NAME_SUBSTRING_KINDS.map((kind, i) => [kind, i])); + const nameIds = new Map(); + const names: string[] = []; + const rowids: number[] = []; + const kinds: number[] = []; + const lengths: number[] = []; + const nameOf: number[] = []; + for (const row of rows) { + const code = kindCodes.get(row.kind); + if (code === undefined) continue; + const lowered = asciiLower(row.name); + if (lowered.includes(SEPARATOR)) return null; + let id = nameIds.get(lowered); + if (id === undefined) { + id = names.length; + nameIds.set(lowered, id); + names.push(lowered); + } + rowids.push(row.rowid); + kinds.push(code); + lengths.push(sqliteLength(row.name)); + nameOf.push(id); + } + + const starts = new Int32Array(names.length + 1); + let offset = 0; + for (let i = 0; i < names.length; i++) { + starts[i] = offset; + offset += names[i]!.length + SEPARATOR.length; + } + starts[names.length] = offset; + + // Group entries by name, keeping index order inside each group. + const nameEntryStart = new Int32Array(names.length + 1); + for (const id of nameOf) nameEntryStart[id + 1]!++; + for (let i = 0; i < names.length; i++) nameEntryStart[i + 1]! += nameEntryStart[i]!; + const fill = nameEntryStart.slice(0, names.length); + const nameEntries = new Int32Array(nameOf.length); + for (let entry = 0; entry < nameOf.length; entry++) nameEntries[fill[nameOf[entry]!]!++] = entry; + + return { + kinds: NAME_SUBSTRING_KINDS, + hay: names.join(SEPARATOR) + SEPARATOR, + starts, + nameEntryStart, + nameEntries, + entryRowid: Float64Array.from(rowids), + entryKind: Uint8Array.from(kinds), + entryLength: Int32Array.from(lengths), + }; +} + +/** The distinct name whose span in `hay` holds `offset`. */ +function nameAt(starts: Int32Array, offset: number): number { + let lo = 0; + let hi = starts.length - 2; + while (lo < hi) { + const mid = (lo + hi + 1) >> 1; + if (starts[mid]! <= offset) lo = mid; else hi = mid - 1; + } + return lo; +} + +/** + * A matcher for a LIKE pattern without `%`: where the next match at or after + * `from` starts, and whether one starts exactly at `at`. Without `_` this is a + * plain string search; each `_` matches any one character but the separator. + */ +function patternMatcher(needle: string): { next(hay: string, from: number): number; startsAt(hay: string, at: number): boolean } { + if (!needle.includes('_')) { + return { + next: (hay, from) => hay.indexOf(needle, from), + startsAt: (hay, at) => hay.startsWith(needle, at), + }; + } + // `u`, so a wildcard takes a whole character, as SQLite's `_` does. + const source = needle + .split('_') + .map((part) => part.replace(/[\^$\\.*+?()[\]{}|/]/g, '\\$&')) + .join('[^\\u0000]'); + const anywhere = new RegExp(source, 'gu'); + const anchored = new RegExp(source, 'uy'); + return { + next(hay, from) { + anywhere.lastIndex = from; + const match = anywhere.exec(hay); + return match ? match.index : -1; + }, + startsAt(hay, at) { + anchored.lastIndex = at; + return anchored.test(hay); + }, + }; +} + +/** + * The order SQLite's plan reads candidate rows in, which its ORDER BY keeps + * for rows of equal length: the kind index's, or the table's by rowid. + */ +export type ScanOrder = 'index' | 'rowid'; + +/** + * The rowids SQLite's substring lookup returns, in its order, or null when the + * index can't answer exactly (a `%` in the pattern, an empty pattern, or a + * kind it doesn't hold). `order` is the order the plan reads rows in. + */ +export function searchNameSubstring( + index: NameSubstringIndex, + substring: string, + kinds: readonly string[], + excludePrefix: boolean, + limit: number, + order: ScanOrder, +): number[] | null { + const needle = asciiLower(substring); + if (needle.length === 0 || needle.includes('%') || needle.includes(SEPARATOR)) return null; + const wanted = new Uint8Array(index.kinds.length); + for (const kind of kinds) { + const code = index.kinds.indexOf(kind); + if (code < 0) return null; + wanted[code] = 1; + } + + const { hay, starts, nameEntryStart, nameEntries, entryKind, entryLength } = index; + const matcher = patternMatcher(needle); + const hits: number[] = []; + for (let at = matcher.next(hay, 0); at >= 0; ) { + const name = nameAt(starts, at); + if (!(excludePrefix && matcher.startsAt(hay, starts[name]!))) { + for (let k = nameEntryStart[name]!; k < nameEntryStart[name + 1]!; k++) { + const entry = nameEntries[k]!; + if (wanted[entryKind[entry]!]) hits.push(entry); + } + } + at = matcher.next(hay, starts[name + 1]!); + } + // Entry numbers follow the kind index, rowids the table: either breaks + // length ties the way SQLite's plan does. + const rowid = index.entryRowid; + hits.sort(order === 'index' + ? (a, b) => (entryLength[a]! - entryLength[b]!) || (a - b) + : (a, b) => (entryLength[a]! - entryLength[b]!) || (rowid[a]! - rowid[b]!)); + // A negative LIMIT is no limit in SQLite. + if (limit >= 0 && hits.length > limit) hits.length = limit; + return hits.map((entry) => index.entryRowid[entry]!); +} diff --git a/src/db/queries.ts b/src/db/queries.ts index 88004853d9..181d307c1c 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -23,6 +23,7 @@ import { parseQuery, boundedEditDistance } from '../search/query-parser'; import { isGeneratedFile } from '../extraction/generated-detection'; import { splitIdentifierSegments } from '../search/identifier-segments'; import { isModuleKey, referenceNameTail, MEMBER_TAIL_SQL, MODULE_PARKED_SQL } from './reference-tail'; +import { NAME_SUBSTRING_KINDS, buildNameSubstringIndex, searchNameSubstring, type NameSubstringIndex, type ScanOrder } from './name-substring-index'; /** * Files that should not be candidates for "dominant file" detection: tests, @@ -107,6 +108,15 @@ const THROUGH_NAMES = (column: string): string => `( */ export const DEPRIORITIZED_NAME_BONUS_SCALE = 0.75; +/** + * A `deprioritize` path predicate (#982) that can also hand out a copy with + * the project's patterns read once, for loops over many paths (#2620). + */ +export interface DeprioritizedPathMatcher { + (filePath: string): boolean; + snapshot?: () => (filePath: string) => boolean; +} + /** * The fields that tell one node of a file from another across a re-index of * that file, whose node ids change with every line shift (#2276). @@ -308,6 +318,12 @@ export class QueryBuilder { | { stamp: string; value: { filePath: string; edgeCount: number; nextEdgeCount: number } | null } | undefined; + // Query-independent reads that a search repeats for every word of its query + // — the name list the fuzzy fallback scans, the name-substring index — each + // tagged with the change stamp it was read under, like dominantFileMemo + // (#2620). See memoizedRead(). + private readMemos = new Map(); + // Prepared statements (lazily initialized) private stmts: { insertNode?: SqliteStatement; @@ -448,6 +464,26 @@ export class QueryBuilder { // different databases report the same one — the memo goes with the old // connection, or a worker following a rebuilt index keeps its answer (#1864). this.dominantFileMemo = undefined; + this.readMemos.clear(); + } + + /** + * `compute()`'s answer, reused until the database changes (see + * getChangeStamp), or null inside a transaction, where a rollback could + * leave a stale answer under an unchanged stamp: the caller then reads the + * database directly, as it did before the memo. + */ + private memoizedRead(key: string, compute: () => T): T | null { + if (this.db.inTransaction !== false) { + this.readMemos.delete(key); + return null; + } + const stamp = this.getChangeStamp(); + const memo = this.readMemos.get(key); + if (memo?.stamp === stamp) return memo.value as T; + const value = compute(); + this.readMemos.set(key, { stamp, value }); + return value; } private edgeKindStmt(sql: string): SqliteStatement { @@ -486,6 +522,17 @@ export class QueryBuilder { return this.isDeprioritizedPath; } + /** + * The `deprioritize` predicate with the project's patterns read once, for a + * loop that asks about many paths. The plain predicate re-checks + * `codegraph.json` on every call, which a long query's candidate loops made + * hundreds of thousands of times (#2620). + */ + getDeprioritizedPathSnapshot(): ((filePath: string) => boolean) | undefined { + const matcher = this.isDeprioritizedPath as DeprioritizedPathMatcher | undefined; + return matcher?.snapshot ? matcher.snapshot() : matcher; + } + // =========================================================================== // Node Operations // =========================================================================== @@ -1698,6 +1745,7 @@ export class QueryBuilder { // Apply multi-signal scoring if (results.length > 0 && (text || query)) { const scoringQuery = text || query; + const isDeprioritizedPath = this.getDeprioritizedPathSnapshot(); results = results.map(r => { // A path the project de-prioritized is saying its symbol NAMES are not // the answer, so the exact-name bonus has to be damped too. The -15 path @@ -1705,8 +1753,9 @@ export class QueryBuilder { // on #982's repro, a `usage()` helper sat at 74.8 vs 51.2 for the top // product symbol — -15 lands at 59.8, still ahead). Damped, not zeroed, // so the tree stays findable when it genuinely is what you asked for. - // Evaluated once and reused: the predicate stats the config file. - const deprioritized = this.isDeprioritizedPath?.(r.node.filePath) ?? false; + // Evaluated once per result and reused, from a snapshot that read the + // config once for the whole loop. + const deprioritized = isDeprioritizedPath?.(r.node.filePath) ?? false; const nameBonus = nameMatchBonus(r.node.name, scoringQuery); return { ...r, @@ -1789,15 +1838,20 @@ export class QueryBuilder { const lowered = text.toLowerCase(); const maxDist = lowered.length <= 4 ? 1 : 2; - // Pull the distinct name list once. The set is cached on QueryBuilder - // by getAllNodeNames(); even on a 200k-node project the distinct - // name set is typically O(10k) because most names repeat. The - // candidate-cap below bounds memory regardless. - const allNames = this.getAllNodeNames(); + // The distinct name list, lowercased, read once until the database + // changes: this fallback runs for every query word that FTS and LIKE both + // miss, which is most words of a pasted log, and the DISTINCT scan alone + // took ~150 ms a call on a 560k-node index (#2620). Inside a transaction + // it is read fresh, as before. + const { names, lowered: loweredNames, starts } = this.fuzzyNames(); const candidates: Array<{ name: string; dist: number }> = []; - for (const name of allNames) { - const dist = boundedEditDistance(name.toLowerCase(), lowered, maxDist); - if (dist <= maxDist) candidates.push({ name, dist }); + for (let i = 0; i < names.length; i++) { + // An edit distance is at least the length difference, so a name outside + // this window could never pass the bound: skipping it changes nothing. + const end = starts[i + 1]! - 1; + if (Math.abs(end - starts[i]! - lowered.length) > maxDist) continue; + const dist = boundedEditDistance(loweredNames.slice(starts[i]!, end), lowered, maxDist); + if (dist <= maxDist) candidates.push({ name: names[i]!, dist }); } candidates.sort((a, b) => a.dist - b.dist); @@ -1952,10 +2006,18 @@ export class QueryBuilder { // nameMatchBonus) can promote results that BM25 alone undervalues. const ftsLimit = Math.max(limit * 5, 100); + // Joined on rowid: nodes_fts is external-content over nodes with + // content_rowid='rowid', so its `id` column is read back from the very row + // the rowid names. The integer key gives the same rows without a lookup + // through the TEXT id index for each match, which roughly halves this + // query on a large index, where explore runs it once or twice for every + // query word (#2620). CROSS JOIN keeps nodes_fts the outer loop, as it was: + // on a small index the planner would otherwise scan nodes and probe FTS + // once per node. let sql = ` SELECT nodes.*, bm25(nodes_fts, 0, 20, 5, 1, 2) as score FROM nodes_fts - JOIN nodes ON nodes_fts.id = nodes.id + CROSS JOIN nodes ON nodes.rowid = nodes_fts.rowid WHERE nodes_fts MATCH ? `; @@ -1993,6 +2055,21 @@ export class QueryBuilder { private searchNodesLike(query: string, options: SearchOptions): SearchResult[] { const { kinds, languages, limit = 100, offset = 0 } = options; + // This runs when FTS found nothing, and a word FTS can't find is usually + // in no name at all: most words of a pasted log or document. Each scan + // read every node, about 120 ms on a 560k-node index (#2620). A word of + // ASCII letters and digits can only match inside a name, or inside one + // run of such characters in a qualified name, and lowercasing keeps those + // runs whole, so when neither lowercased list holds it there is nothing + // to find. Any other word is scanned as before. + if (/^[A-Za-z0-9]+$/.test(query)) { + const runs = this.memoizedRead('qualifiedNameRuns', () => this.readQualifiedNameRuns()); + if (runs !== null) { + const word = query.toLowerCase(); + if (!this.fuzzyNames().lowered.includes(word) && !runs.includes(word)) return []; + } + } + let sql = ` SELECT nodes.*, CASE @@ -2183,6 +2260,22 @@ export class QueryBuilder { sql += ' ORDER BY length(name) ASC LIMIT ?'; params.push(limit); + // Explore asks this four times for every word of its query, and the + // leading `%` makes each one a scan of every node of those kinds (#2620). + // The in-memory index answers with the same rows in the same order: ties + // in length come out in the order SQLite's plan for this very statement + // reads rows, the kind index's or the table's. Anything it can't answer + // exactly (no kind filter, a language filter, a `%` in the word, another + // plan, a transaction) is asked of SQLite. + if (kinds && kinds.length > 0 && !(languages && languages.length > 0)) { + const index = this.memoizedRead('nameSubstringIndex', () => this.readNameSubstringIndex()); + const order = index ? this.substringScanOrder(index, sql, params, excludePrefix ? 2 : 1) : null; + const rowids = order ? searchNameSubstring(index!.index, substring, kinds, excludePrefix === true, limit, order) : null; + if (rowids) { + return this.getNodeRowsByRowid(rowids).map((row) => ({ node: rowToNode(row), score: 1.0 })); + } + } + const rows = this.db.prepare(sql).all(...params) as (NodeRow & { score: number })[]; return rows.map((row) => ({ node: rowToNode(row), @@ -2190,6 +2283,76 @@ export class QueryBuilder { })); } + /** + * The order SQLite's plan for a substring lookup reads candidate rows in, + * which is the order its ORDER BY leaves ties of equal length in: the kind + * index's (kind, file_path, start_line, id), or the table's (rowid). Which + * one depends on the statistics — a small index is scanned whole — so it is + * asked of SQLite once per statement shape. Null for any other plan. + */ + private substringScanOrder( + memo: { index: NameSubstringIndex; plans: Map }, + sql: string, + params: readonly (string | number)[], + patternCount: number, + ): ScanOrder | null { + // The patterns (the first `patternCount` parameters) can't change the + // plan, since a leading `%` uses no index; the kinds and the limit can. + const key = `${sql}\u0000${params.slice(patternCount).join('\u0000')}`; + let order = memo.plans.get(key); + if (order === undefined) { + order = null; + try { + const details = (this.db.prepare(`EXPLAIN QUERY PLAN ${sql}`).all(...params) as Array<{ detail: string }>) + .map((row) => row.detail) + .filter((detail) => /^(SCAN|SEARCH) (TABLE )?nodes\b/.test(detail)); + if (details.length === 1) { + if (/^SEARCH (TABLE )?nodes USING (COVERING )?INDEX idx_nodes_kind \(/.test(details[0]!)) order = 'index'; + else if (/^SCAN (TABLE )?nodes$/.test(details[0]!)) order = 'rowid'; + } + } catch { + order = null; + } + memo.plans.set(key, order); + } + return order; + } + + /** + * Every node of the kinds the name-substring index holds, read in + * idx_nodes_kind order (kind, file_path, start_line, id): the order its + * results break length ties by. + */ + private readNameSubstringIndex(): { index: NameSubstringIndex; plans: Map } | null { + const rows = this.db + .prepare( + `SELECT rowid AS rowid, kind, name FROM nodes WHERE kind IN (${NAME_SUBSTRING_KINDS.map(() => '?').join(',')})` + + ' ORDER BY kind, file_path, start_line, id' + ) + .all(...NAME_SUBSTRING_KINDS) as Array<{ rowid: number; kind: string; name: string }>; + const index = buildNameSubstringIndex(rows); + return index ? { index, plans: new Map() } : null; + } + + /** Whole node rows for `rowids`, in that order; a rowid no longer present is skipped. */ + private getNodeRowsByRowid(rowids: readonly number[]): NodeRow[] { + if (rowids.length === 0) return []; + const byRowid = new Map(); + for (let i = 0; i < rowids.length; i += SQLITE_PARAM_CHUNK_SIZE) { + const chunk = rowids.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); + const rows = this.db + .prepare(`SELECT rowid AS rowid_, nodes.* FROM nodes WHERE rowid IN (${chunk.map(() => '?').join(',')})`) + .all(...chunk) as Array; + for (const row of rows) byRowid.set(row.rowid_, row); + } + const out: NodeRow[] = []; + for (const rowid of rowids) { + const row = byRowid.get(rowid); + if (row) out.push(row); + } + return out; + } + // =========================================================================== // Edge Operations // =========================================================================== @@ -4054,6 +4217,40 @@ export class QueryBuilder { return rows.map((r) => r.name); } + /** + * The distinct node names, in {@link getAllNodeNames} order, and the same + * names lowercased and newline-joined: name `i` lowercased is + * `lowered.slice(starts[i], starts[i + 1] - 1)`. Read once until the + * database changes (fresh inside a transaction). The fuzzy fallback compares + * against it, and the LIKE fallback asks it whether any name holds a word. + */ + private fuzzyNames(): { names: string[]; lowered: string; starts: Int32Array } { + const read = (): { names: string[]; lowered: string; starts: Int32Array } => { + const names = this.getAllNodeNames(); + const lowered = names.map((name) => name.toLowerCase()); + const starts = new Int32Array(names.length + 1); + for (let i = 0; i < lowered.length; i++) starts[i + 1] = starts[i]! + lowered[i]!.length + 1; + return { names, lowered: lowered.join('\n') + '\n', starts }; + }; + return this.memoizedRead('fuzzyNames', read) ?? read(); + } + + /** + * The runs of ASCII letters and digits in the distinct qualified names, + * lowercased and newline-joined: where a word LIKE could find in a + * qualified name has to be. + */ + private readQualifiedNameRuns(): string { + const runs = new Set(); + const rows = this.db.prepare('SELECT DISTINCT qualified_name AS name FROM nodes').all() as Array<{ name: string }>; + for (const row of rows) { + for (const run of row.name.toLowerCase().split(/[^a-z0-9]+/)) { + if (run) runs.add(run); + } + } + return [...runs].join('\n'); + } + /** The names of a language's nodes of the given kinds (repeats included), decoding nothing else. */ getNodeNamesOfKinds(language: Language, kinds: readonly NodeKind[]): string[] { // `+language`: the kind index narrows further than the language one. diff --git a/src/index.ts b/src/index.ts index 0a62d02413..0a16f6f79a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -30,7 +30,7 @@ import { } from './types'; import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db'; import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve'; -import { QueryBuilder } from './db/queries'; +import { QueryBuilder, type DeprioritizedPathMatcher } from './db/queries'; import { importPathKeys, moduleReferenceKeys } from './db/reference-tail'; import { isInitialized, @@ -218,24 +218,47 @@ export class CodeGraph { // on the pattern array's identity, which the cache keeps stable. let cachedPatterns: string[] | undefined; let cachedMatcher: ReturnType | undefined; - this.queries.setDeprioritizedPathMatcher((filePath: string): boolean => { + // The compiled matcher for the patterns `codegraph.json` holds right now, + // or null when there are none. + const currentMatcher = (): ReturnType | null => { + const patterns = loadDeprioritizePatterns(this.projectRoot); + if (patterns.length === 0) return null; + if (patterns !== cachedPatterns) { + cachedPatterns = patterns; + cachedMatcher = ignore().add(patterns); + } + return cachedMatcher!; + }; + const matches = (matcher: ReturnType | null, filePath: string): boolean => { + if (!matcher) return false; + const rel = path.isAbsolute(filePath) + ? path.relative(this.projectRoot, filePath) + : filePath; + if (!rel || rel.startsWith('..')) return false; + return matcher.ignores(rel.split(path.sep).join('/')); + }; + const deprioritized: DeprioritizedPathMatcher = (filePath: string): boolean => { try { - const patterns = loadDeprioritizePatterns(this.projectRoot); - if (patterns.length === 0) return false; - if (patterns !== cachedPatterns) { - cachedPatterns = patterns; - cachedMatcher = ignore().add(patterns); - } - const rel = path.isAbsolute(filePath) - ? path.relative(this.projectRoot, filePath) - : filePath; - if (!rel || rel.startsWith('..')) return false; - return cachedMatcher!.ignores(rel.split(path.sep).join('/')); + return matches(currentMatcher(), filePath); } catch { // Ranking must never take the search down with it. return false; } - }); + }; + // One `stat` for a whole loop of candidates instead of one per candidate + // (#2620): a search reads the config once, then asks the frozen matcher. + deprioritized.snapshot = () => { + let matcher: ReturnType | null = null; + try { matcher = currentMatcher(); } catch { matcher = null; } + return (filePath: string): boolean => { + try { + return matches(matcher, filePath); + } catch { + return false; + } + }; + }; + this.queries.setDeprioritizedPathMatcher(deprioritized); this.orchestrator = new ExtractionOrchestrator(this.projectRoot, this.queries); this.resolver = createResolver(this.projectRoot, this.queries); diff --git a/src/search/query-utils.ts b/src/search/query-utils.ts index 91bbdd09e1..cab9946b88 100644 --- a/src/search/query-utils.ts +++ b/src/search/query-utils.ts @@ -215,19 +215,30 @@ export function extractSearchTerms(query: string, options?: { stems?: boolean }) } /** - * Score path relevance to a query - * Higher score = more relevant path + * The query half of {@link scorePathRelevance}: the sub-tokens of each query + * word that is scored, with how many times the word occurs, and whether the + * query is about tests. */ -export function scorePathRelevance( - filePath: string, - query: string, - projectNameTokens?: Set, - isDeprioritized?: boolean, -): number { - const pathLower = filePath.toLowerCase(); - const fileName = path.basename(filePath).toLowerCase(); - const dirName = path.dirname(filePath).toLowerCase(); - let score = 0; +interface PathRelevanceQuery { + projectNameTokens: Set | undefined; + /** No words at all: every path scores 0. */ + empty: boolean; + words: Array<{ subtokens: string[]; count: number }>; + isTestQuery: boolean; +} + +/** + * The few most recent queries' halves. A search scores every candidate path + * against one query, and re-splitting every word of it for each path was + * most of the scoring time on a long query: a pasted page is hundreds of + * words, and explore scores thousands of paths (#2620). + */ +const pathRelevanceQueries = new Map(); +const PATH_RELEVANCE_QUERY_CACHE = 8; + +function pathRelevanceQuery(query: string, projectNameTokens: Set | undefined): PathRelevanceQuery { + const cached = pathRelevanceQueries.get(query); + if (cached && cached.projectNameTokens === projectNameTokens) return cached; // Score per original query WORD, not per sub-token. A single PascalCase word // splits into many sub-tokens (a project name "SuperBizAgent" → @@ -239,30 +250,72 @@ export function scorePathRelevance( // camelCase/snake split per word (so `getUserName` still matches a // `get_user_name` path) — we just attribute each word's matches once. const allWords = query.split(/\s+/).filter((w) => w.length > 0); - if (allWords.length === 0) return 0; // A query word that just names the PROJECT (its go.mod / package.json / repo // name) carries no discriminative path signal — drop it so the rest of the // query decides the ranking, instead of every file under a `…/` // tree winning on the project name alone (#720). Only when OTHER words remain, // so a bare project-name query still scores on its path. - const words = + const kept = projectNameTokens && projectNameTokens.size > 0 ? allWords.filter((w) => !projectNameTokens.has(normalizeNameToken(w))) : allWords; - const scored = words.length > 0 ? words : allWords; + const scored = kept.length > 0 ? kept : allWords; + // A word that occurs twice adds its score twice; words with the same + // sub-tokens score alike, so each such group is matched once and counted. + const groups = new Map(); for (const word of scored) { // Use base terms only — stem variants inflate path scores by generating // many near-duplicate terms that all match the same path segments. const subtokens = extractSearchTerms(word, { stems: false }); if (subtokens.length === 0) continue; + const key = subtokens.join('\u0000'); + const group = groups.get(key); + if (group) group.count++; + else groups.set(key, { subtokens, count: 1 }); + } + + const queryLower = query.toLowerCase(); + const prepared: PathRelevanceQuery = { + projectNameTokens, + empty: allWords.length === 0, + words: [...groups.values()], + isTestQuery: queryLower.includes('test') || queryLower.includes('spec'), + }; + pathRelevanceQueries.delete(query); + pathRelevanceQueries.set(query, prepared); + if (pathRelevanceQueries.size > PATH_RELEVANCE_QUERY_CACHE) { + pathRelevanceQueries.delete(pathRelevanceQueries.keys().next().value!); + } + return prepared; +} + +/** + * Score path relevance to a query + * Higher score = more relevant path + */ +export function scorePathRelevance( + filePath: string, + query: string, + projectNameTokens?: Set, + isDeprioritized?: boolean, +): number { + const prepared = pathRelevanceQuery(query, projectNameTokens); + if (prepared.empty) return 0; + + const pathLower = filePath.toLowerCase(); + const fileName = path.basename(filePath).toLowerCase(); + const dirName = path.dirname(filePath).toLowerCase(); + let score = 0; + + for (const { subtokens, count } of prepared.words) { // Exact filename match (strongest) - if (subtokens.some((t) => fileName.includes(t))) score += 10; + if (subtokens.some((t) => fileName.includes(t))) score += 10 * count; // Directory match - if (subtokens.some((t) => dirName.includes(t))) score += 5; + if (subtokens.some((t) => dirName.includes(t))) score += 5 * count; // General path match - else if (subtokens.some((t) => pathLower.includes(t))) score += 3; + else if (subtokens.some((t) => pathLower.includes(t))) score += 3 * count; } // Deprioritize test files unless the query is explicitly about tests, and @@ -274,9 +327,7 @@ export function scorePathRelevance( // standing statement by the project, so it is NOT waived. The name-bonus // damping at the call site is what keeps such a tree findable. // - a path that is both is docked ONCE, not twice. - const queryLower = query.toLowerCase(); - const isTestQuery = queryLower.includes('test') || queryLower.includes('spec'); - const offTarget = (!isTestQuery && isTestFile(filePath)) || isDeprioritized === true; + const offTarget = (!prepared.isTestQuery && isTestFile(filePath)) || isDeprioritized === true; if (offTarget) { score -= 15; } From c4dbc126b0575d676c5803a0ca15b0023bf2d848 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 16:26:56 +0000 Subject: [PATCH 39/95] fix(bundle): bundle Node 24.21.0 so Windows commands no longer abort as they exit (#2806) Root cause: every release bundles the Node that scripts/build-bundle.sh defaults to, since the release workflow passes only the target, and that default was v24.16.0. On Windows that Node can abort just as the process exits, after the command's work is done: a V8 worker posts a delayed task after the platform's delayed-task scheduler has started closing its uv_async handle, and libuv fails "Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c, line 94" with status 0xC0000409 (nodejs/node issue 56645). Node fixed it in 24.20.0 (nodejs/node PR 61999). Fix: the default moves to v24.21.0, the newest 24.x LTS release that is at least two weeks old (2026-09-07). It also carries the Windows 8.3 short-path file-watcher fix (nodejs/node PR 65118). V8 13.6.233.17, libuv 1.52.1 and the module ABI (137, N-API 10) are unchanged; SQLite goes from 3.53.0 to 3.53.4. A comment at the default says why it must stay on 24.x at 24.20.0 or later, and a test pins that, along with the workflow passing only the target. Validation, on Windows 11 x64: - Node's own regression test for the crash (test-process-exit-after-fetch-throw.js, from the v24.21.0 tag), 300 runs per arm, against the official win-x64 node.exe of each version (zips checked against nodejs.org's SHASUMS256.txt): v24.16.0 aborted 300/300 with 0xC0000409 and that async.c line 94 assertion, and the test as written failed 300/300; v24.21.0 passed 300/300. With the launchers' --liftoff-only both passed 300/300, since that flag only removes this test's trigger. - build-bundle.sh win32-x64 builds the bundle on this machine. Its node.exe is byte-identical to the official v24.21.0 one. --version through the Git Bash launcher, cmd and PowerShell, then init, status, query, callers and sync on a small TypeScript and Python project all exit 0, on the wasm path and with the native kernel bundled. - The real CLI (index, status -j, sync) run 300 times per runtime on the bundled app: 0 nonzero exits on either. - The runtime-facing test files (WASM flags, node:sqlite backend and open errors, launchers, worker teardown, version check) pass under Node 24.21.0: 9 files, 65 passed, 3 skipped (POSIX-only). - The new test fails on main's script (expected 16 to be greater than or equal to 20) and passes with the fix. Fixes #2617 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/bundle-launcher.test.ts | 20 ++++++++++++++++++++ scripts/build-bundle.sh | 10 ++++++++-- 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ae920b5276..0b3e62ce04 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404) - An index left incomplete by an indexing run that was stopped partway, by a crash, an out-of-memory kill or the server's own hang check, no longer gets the background server killed with "Main thread unresponsive" every time it starts: on a large project, restoring the database indexes the run had set aside took longer than the hang check allows, so each restart died at the same point and the index was never repaired. They are now restored in the background, and the project is served once they are back. Thanks @cbeaulieu-gt for the report and the first fix. (#1887) - A new agent session no longer waits up to about 12 minutes for its first answer when the background server is stuck: a server whose process still holds its connection open but never responds made the session wait out a long series of retries. The session now gives up within seconds and answers on its own, and switches back to the background server once that has been restarted. +- On Windows, a command like `codegraph sync` no longer sometimes crashes as it exits, after its work is done, printing `Assertion failed: !(handle->flags & UV_HANDLE_CLOSING)`, which made a script or CI run that had succeeded fail. The bundled Node runtime moves to 24.21.0, which fixes that crash in Node itself. Thanks @Ancienttwo for the report and the first fix. (#2617) - When CodeGraph can't open a project's database, it now says what went wrong: SQLite's error, the file's path, and what to check, such as another program holding the file open, a folder you can't write to, or a network share. Before, every such failure read "CodeGraph requires node:sqlite (Node.js 22.5+)", and an agent was told no project was loaded. That hint now appears only when Node really can't load its built-in SQLite, a case that used to end in an unhandled error instead. - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) diff --git a/__tests__/bundle-launcher.test.ts b/__tests__/bundle-launcher.test.ts index 04c0790240..c05fbc4a73 100644 --- a/__tests__/bundle-launcher.test.ts +++ b/__tests__/bundle-launcher.test.ts @@ -122,3 +122,23 @@ describe('bundled launchers (#1278)', () => { } }); }); + +describe('bundled Node runtime (#2617)', () => { + it('ships a Node 24 release with the fix for the Windows abort at exit', () => { + // The release workflow passes only the target, so the recipe's default + // version is the Node every release bundles. + const workflow = fs.readFileSync(path.resolve('.github/workflows/release.yml'), 'utf8'); + const calls = workflow.match(/build-bundle\.sh[^\r\n]*/g) ?? []; + expect(calls.length).toBeGreaterThan(0); + for (const call of calls) expect(call.trim()).toBe('build-bundle.sh "$t"'); + + const pin = /^NODE_VERSION="\$\{2:-v(\d+)\.(\d+)\.(\d+)\}"\r?$/m.exec(recipe); + expect(pin, 'default NODE_VERSION in scripts/build-bundle.sh').not.toBeNull(); + const [major, minor] = pin!.slice(1, 3).map(Number); + // The CLI refuses to run on Node 25 and later. + expect(major).toBe(24); + // Node 24 before 24.20.0 can abort on Windows as the process exits, after + // the command's work is done (0xC0000409, nodejs/node#56645). + expect(minor).toBeGreaterThanOrEqual(20); + }); +}); diff --git a/scripts/build-bundle.sh b/scripts/build-bundle.sh index 0fef1c98ed..4d571709ec 100755 --- a/scripts/build-bundle.sh +++ b/scripts/build-bundle.sh @@ -13,7 +13,7 @@ # scripts/build-bundle.sh [node-version] # target: darwin-arm64 | darwin-x64 | linux-x64 | linux-arm64 # | win32-x64 | win32-arm64 -# node-version: e.g. v24.16.0 (default below; pin for reproducible builds) +# node-version: e.g. v24.21.0 (default below; pin for reproducible builds) # # Output: # unix: release/codegraph-.tar.gz (launcher: bin/codegraph) @@ -21,7 +21,13 @@ set -euo pipefail TARGET="${1:?usage: build-bundle.sh [node-version]}" -NODE_VERSION="${2:-v24.16.0}" +# The release workflow passes no version, so this default is the Node every +# release bundles. Keep it on 24.x (the CLI refuses Node 25+) and at 24.20.0 or +# later: older 24.x releases can abort on Windows just as the process exits, +# after its work is done, with 0xC0000409 and "Assertion failed: +# !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c" (nodejs/node#56645, +# fixed by nodejs/node#61999 in 24.20.0; issue #2617). +NODE_VERSION="${2:-v24.21.0}" ROOT="$(cd "$(dirname "$0")/.." && pwd)" OUT="$ROOT/release" From 9ed79c6cf1a93e76ad4d1b9846f582701c7e8373 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 17:04:12 +0000 Subject: [PATCH 40/95] fix(add-lang): check-grammar fails a grammar that crashes or hangs its process (#2808) Root cause: scripts/add-lang/check-grammar.mjs ran the parse loop in its own process and printed "RESULT: PASS" as soon as the loop finished. A grammar can finish every parse and still take the process down after that. With the issue's vim grammar (tree-sitter-wasm 2.0.4) on Node 24, V8's optimizing tier aborts the process from a background compile job ("Fatal process out of memory: Zone") after PASS is printed, so the verdict and the exit code disagree. The check also ran without the V8 flag indexing always uses (--liftoff-only), so it judged grammars in a setup codegraph never runs them in, and an iteration count of 0 or a word passed without a single parse. Fix: the script runs the grammar in a child process, the way indexing does, and prints the verdict only after that child has exited. - The child gets indexing's V8 flags (WASM_RUNTIME_FLAGS) and loads every vendored grammar beside the one under test, like a polyglot index. - It parses the sample with one reused parser, walks every node of each tree and frees it, as extraction does. Every parse of the same sample must give the same tree. - It then parses the sample repeated to 1 MiB, the most indexing reads from one file, within the 10 s indexing gives one file. Only a crash or a hang counts there. - The parent fails the grammar when the child dies (any exit code or signal), runs past a timeout (CODEGRAPH_CHECK_GRAMMAR_TIMEOUT_MS, 120 s by default), or reports ERROR trees or differing trees. An iteration count that is not a whole number above 0 is refused (exit 2). The add-lang skill now says what counts as a FAIL. Validation: - New __tests__/add-lang-check-grammar.test.ts. A preload marks the process that runs WebAssembly, records its flags, and on request kills it or makes it hang at exit, after the parses. With main's script 4 of 5 tests fail: PASS printed for a process that then died, a hang until the test's 60 s limit, no --liftoff-only in the parsing process, and "0" passing. With the fix all 5 pass. The fifth (the tree-sitter-wasms Lua ABI-13 grammar fails with ERROR trees) passes on both. - The issue's vim grammar (sha256 df3906a0, the issue's file) on Node 24.21.0: main's script printed PASS and then died with the Zone OOM, exit 3, in 3 of 3 runs. The new check runs it with --liftoff-only, as indexing does, where it parses cleanly, 1 MiB included, and passes in 3 of 3. A copy of the new script with the flag withheld fails it in 3 of 3: "the parses finished, but the process running them then died (exit code 2147483651, 0x80000003)". - Old vs new on the 19 vendored grammars with real samples, plus the tree-sitter-wasms Lua and the vim grammar, on Node 22.22.0 and 24.21.0 (84 runs): the same verdict every time, except vim on Node 24, where main's PASS came with exit 3. No grammar's parses diverged. Fixes #2464 Co-Authored-By: Claude Opus 5.5 --- .claude/skills/add-lang/SKILL.md | 9 +- __tests__/add-lang-check-grammar.test.ts | 116 ++++++++++++ scripts/add-lang/check-grammar.mjs | 230 +++++++++++++++++++---- 3 files changed, 313 insertions(+), 42 deletions(-) create mode 100644 __tests__/add-lang-check-grammar.test.ts diff --git a/.claude/skills/add-lang/SKILL.md b/.claude/skills/add-lang/SKILL.md index 37cbdce556..24ac69ef42 100644 --- a/.claude/skills/add-lang/SKILL.md +++ b/.claude/skills/add-lang/SKILL.md @@ -60,11 +60,14 @@ still be unusable:** ```bash node scripts/add-lang/check-grammar.mjs path/to/valid-sample. ``` -It prints the grammar's ABI version and parses a valid sample many times in a -multi-grammar runtime. If it **FAILs** (ERROR trees on valid code — an old ABI +It prints the grammar's ABI version and runs it the way indexing does: in a +child process with indexing's V8 flags, beside the vendored grammars, parsing a +valid sample many times and once at the 1 MiB file-size limit. If it **FAILs** +(ERROR trees, or later parses giving a different tree, on valid code — an old ABI corrupting the shared WASM heap, which silently drops nested calls/imports on every file after the first; e.g. the tree-sitter-wasms **Lua** grammar is ABI 13 -and fails), do NOT use that wasm. **Vendor a newer (ABI 14/15) build instead:** +and fails — or the process running it crashing or hanging, even after every +parse finished), do NOT use that wasm. **Vendor a newer (ABI 14/15) build instead:** ```bash npm pack @tree-sitter-grammars/tree-sitter- # often ships a prebuilt *.wasm # or build one: npx tree-sitter build --wasm (needs Docker/emscripten) diff --git a/__tests__/add-lang-check-grammar.test.ts b/__tests__/add-lang-check-grammar.test.ts new file mode 100644 index 0000000000..93e50380d7 --- /dev/null +++ b/__tests__/add-lang-check-grammar.test.ts @@ -0,0 +1,116 @@ +/** + * scripts/add-lang/check-grammar.mjs decides whether a grammar wasm is safe to + * vendor. It must judge the grammar by how the process running it ends, not by + * a line printed before that: a grammar can finish every parse and then take + * the process down, as V8 does when a background compile job runs out of Zone + * memory, and the old check had already printed PASS by then (#2464). It must + * also run the grammar the way indexing does, with indexing's V8 flags. + * + * A preload stands in for such a grammar. It marks the process that runs + * WebAssembly, records how that process was launched, and on request makes it + * die or hang at exit, after the parses are done. + */ +import { afterEach, describe, expect, it } from 'vitest'; +import { spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { WASM_RUNTIME_FLAGS } from '../src/extraction/wasm-runtime-flags'; + +const SCRIPT = path.resolve('scripts/add-lang/check-grammar.mjs'); +const PYTHON = path.resolve('src/extraction/wasm/tree-sitter-python.wasm'); +const roots: string[] = []; + +const PRELOAD = ` +const fs = require('fs'); +const instantiate = WebAssembly.instantiate; +let ranWasm = false; +WebAssembly.instantiate = function (...args) { ranWasm = true; return instantiate.apply(this, args); }; +process.on('exit', () => { + if (!ranWasm) return; + fs.appendFileSync(process.env.CG_TEST_RECORD, JSON.stringify({ execArgv: process.execArgv }) + '\\n'); + // A hard kill rather than abort(): no crash report on macOS. + if (process.env.CG_TEST_AT_EXIT === 'die') process.kill(process.pid, 'SIGKILL'); + if (process.env.CG_TEST_AT_EXIT === 'hang') for (;;) {} +}); +`; + +function kit() { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-check-grammar-')); + roots.push(root); + const preload = path.join(root, 'grammar-process.cjs'); + fs.writeFileSync(preload, PRELOAD); + const python = path.join(root, 'sample.py'); + fs.writeFileSync(python, 'def add(a, b):\n return a + b\n\n\nclass Box:\n def size(self):\n return add(1, 2)\n'); + const lua = path.join(root, 'sample.lua'); + fs.writeFileSync(lua, 'local M = {}\n\nlocal function helper(x)\n return x * 2\nend\n\nfunction M.run(items)\n local out = {}\n for i, v in ipairs(items) do\n out[i] = helper(v)\n end\n return out\nend\n'); + return { root, preload, python, lua, record: path.join(root, 'record.jsonl') }; +} + +function check(k: ReturnType, args: string[], env: NodeJS.ProcessEnv = {}) { + const preload = `--require "${k.preload.replace(/\\/g, '/')}"`; + return spawnSync(process.execPath, [SCRIPT, ...args], { + encoding: 'utf8', + timeout: 60_000, + env: { + ...process.env, + NODE_OPTIONS: [process.env.NODE_OPTIONS, preload].filter(Boolean).join(' '), + CG_TEST_RECORD: k.record, + ...env, + }, + }); +} + +function records(k: ReturnType): Array<{ execArgv: string[] }> { + if (!fs.existsSync(k.record)) return []; + return fs.readFileSync(k.record, 'utf8').trim().split('\n').filter(Boolean).map((line) => JSON.parse(line)); +} + +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('add-lang check-grammar (#2464)', () => { + it('passes a healthy grammar, run in a child process with the V8 flags indexing uses', () => { + const k = kit(); + const run = check(k, [PYTHON, k.python]); + expect(run.status, run.stdout + run.stderr).toBe(0); + expect(run.stdout).toMatch(/RESULT: PASS/); + const ran = records(k); + expect(ran).toHaveLength(1); + for (const flag of WASM_RUNTIME_FLAGS) expect(ran[0]?.execArgv).toContain(flag); + }, 90_000); + + it('fails when the process running the grammar dies after the parses', () => { + const k = kit(); + const run = check(k, [PYTHON, k.python], { CG_TEST_AT_EXIT: 'die' }); + expect(run.status, run.stdout + run.stderr).toBe(1); + expect(run.stdout).not.toMatch(/RESULT: PASS/); + expect(run.stdout).toMatch(/RESULT: FAIL — the parses finished, but the process running them then died/); + }, 90_000); + + it('fails when the process running the grammar hangs instead of exiting', () => { + const k = kit(); + const run = check(k, [PYTHON, k.python], { CG_TEST_AT_EXIT: 'hang', CODEGRAPH_CHECK_GRAMMAR_TIMEOUT_MS: '5000' }); + expect(run.status, run.stdout + run.stderr).toBe(1); + expect(run.stdout).not.toMatch(/RESULT: PASS/); + expect(run.stdout).toMatch(/RESULT: FAIL — the parses finished, but the process running them had not exited/); + }, 90_000); + + it('still fails the tree-sitter-wasms Lua grammar, which corrupts the shared heap on reuse', () => { + const k = kit(); + const run = check(k, ['lua', k.lua]); + expect(run.status, run.stdout + run.stderr).toBe(1); + expect(run.stdout).toMatch(/ABI version: 13/); + expect(run.stdout).toMatch(/RESULT: FAIL — \d+\/20 parses produced ERROR trees/); + }, 90_000); + + it('refuses an iteration count that is not a whole number above 0', () => { + const k = kit(); + for (const iterations of ['0', 'many']) { + const run = check(k, [PYTHON, k.python, iterations]); + expect(run.status, `${iterations}: ${run.stdout}${run.stderr}`).toBe(2); + expect(run.stdout).not.toMatch(/RESULT:/); + } + }, 90_000); +}); diff --git a/scripts/add-lang/check-grammar.mjs b/scripts/add-lang/check-grammar.mjs index 461b129668..9aff935474 100755 --- a/scripts/add-lang/check-grammar.mjs +++ b/scripts/add-lang/check-grammar.mjs @@ -10,66 +10,218 @@ // upstream ABI-15 wasm. Run this on any new grammar first; if it FAILs, vendor a // newer build instead of using the tree-sitter-wasms one. // +// The grammar runs the way indexing runs it, in a child process: +// - with indexing's V8 flags (WASM_RUNTIME_FLAGS in +// src/extraction/wasm-runtime-flags.ts), so V8 compiles it as it would there; +// - next to every grammar codegraph vendors, like a polyglot index; +// - parsing the sample with one reused parser, walking every node of each tree +// and freeing it, as extraction does; every parse must give the same tree; +// - then once more at the largest file indexing reads (the sample repeated to +// 1 MiB, MAX_SOURCE_FILE_SIZE_BYTES in src/file-limits.ts), with the time +// indexing gives one file. +// This process prints the verdict only after that child has exited. A grammar +// can finish every parse and still take the process down, as when V8 aborts it +// from a background compile job ("Fatal process out of memory: Zone"), or hang; +// either one is a FAIL (#2464). +// // Usage: node scripts/add-lang/check-grammar.mjs [iterations] -// Exit: 0 healthy, 1 corruption / parse errors, 2 could not run. +// Exit: 0 healthy; 1 parse errors, corruption, or the check process crashed or +// hung; 2 could not run. +// CODEGRAPH_CHECK_GRAMMAR_TIMEOUT_MS: how long the child may run (default 120000). // NOTE: the sample must be SYNTACTICALLY VALID — a broken sample fails for the // wrong reason. -import { readFileSync, existsSync } from 'node:fs'; +import { readFileSync, existsSync, readdirSync } from 'node:fs'; +import { spawnSync } from 'node:child_process'; import { createRequire } from 'node:module'; -import { Parser, Language } from 'web-tree-sitter'; +import { fileURLToPath } from 'node:url'; +import path from 'node:path'; const require = createRequire(import.meta.url); -const fail = (code, msg) => { console.error(`[check-grammar] ${msg}`); process.exit(code); }; +const SCRIPT = fileURLToPath(import.meta.url); +const VENDORED_DIR = path.resolve(path.dirname(SCRIPT), '../../src/extraction/wasm'); +// Indexing's V8 flags. Keep in step with WASM_RUNTIME_FLAGS: +// __tests__/add-lang-check-grammar.test.ts fails when the child lacks one. +const INDEXING_WASM_FLAGS = ['--liftoff-only']; +// The most indexing reads from one file (MAX_SOURCE_FILE_SIZE_BYTES), and how +// long it gives one file to parse (DEFAULT_PARSE_TIMEOUT_MS in parse-pool.ts). +const MAX_FILE_BYTES = 1024 * 1024; +const FILE_BUDGET_MS = 10_000; +const CHILD_ENV = 'CODEGRAPH_CHECK_GRAMMAR_CHILD'; +const RESULT_TAG = 'CHECK_GRAMMAR_RESULT '; +const DEFAULT_TIMEOUT_MS = 120_000; +const SPECIAL = { csharp: 'c_sharp', 'c#': 'c_sharp' }; -const [token, sample, iterArg] = process.argv.slice(2); -if (!token || !sample) fail(2, 'usage: check-grammar.mjs [iterations]'); -if (!existsSync(sample)) fail(2, `sample not found: ${sample}`); -const iters = iterArg ? parseInt(iterArg, 10) : 20; +const fail = (code, msg) => { console.error(`[check-grammar] ${msg}`); process.exit(code); }; +const samePath = (a, b) => (process.platform === 'win32' ? a.toLowerCase() === b.toLowerCase() : a === b); -const SPECIAL = { csharp: 'c_sharp', 'c#': 'c_sharp' }; function resolveWasm(t) { - if (t.endsWith('.wasm')) return existsSync(t) ? t : fail(2, `wasm not found: ${t}`); + if (t.endsWith('.wasm')) return existsSync(t) ? path.resolve(t) : fail(2, `wasm not found: ${t}`); const base = SPECIAL[t.toLowerCase()] ?? t.toLowerCase(); try { return require.resolve(`tree-sitter-wasms/out/tree-sitter-${base}.wasm`); } catch { /* try vendored */ } - const vendored = `src/extraction/wasm/tree-sitter-${base}.wasm`; + const vendored = path.join(VENDORED_DIR, `tree-sitter-${base}.wasm`); if (existsSync(vendored)) return vendored; return fail(2, `no grammar for "${t}" — not in tree-sitter-wasms and not vendored`); } -const wasmPath = resolveWasm(token); -const source = readFileSync(sample, 'utf8'); +/** The parent: run the grammar in a child process, then judge how it ended. */ +function judge([token, sample, iterArg]) { + if (!token || !sample) fail(2, 'usage: check-grammar.mjs [iterations]'); + if (!existsSync(sample)) fail(2, `sample not found: ${sample}`); + const iters = iterArg === undefined ? 20 : Number(iterArg); + if (!Number.isInteger(iters) || iters < 1) fail(2, `iterations must be a whole number above 0, not "${iterArg}"`); + const wasmPath = resolveWasm(token); + const timeoutMs = Number(process.env.CODEGRAPH_CHECK_GRAMMAR_TIMEOUT_MS) || DEFAULT_TIMEOUT_MS; -try { await Parser.init(); } -catch { await Parser.init({ locateFile: () => require.resolve('web-tree-sitter/tree-sitter.wasm') }); } + const run = spawnSync(process.execPath, [...INDEXING_WASM_FLAGS, SCRIPT, wasmPath, path.resolve(sample), String(iters)], { + env: { ...process.env, [CHILD_ENV]: '1' }, + encoding: 'utf8', + timeout: timeoutMs, + maxBuffer: 64 * 1024 * 1024, + windowsHide: true, + }); + const timedOut = run.error?.code === 'ETIMEDOUT'; + if (run.error && !timedOut) fail(2, `could not run the check: ${run.error.message}`); -// Load a second, known-good grammar — the corruption surfaces under the -// multi-grammar runtime that real indexing uses, not a single grammar in isolation. -try { await Language.load(require.resolve('tree-sitter-wasms/out/tree-sitter-python.wasm')); } catch { /* ok */ } + const lines = (run.stdout ?? '').split(/\r?\n/); + const tagged = lines.find((line) => line.startsWith(RESULT_TAG)); + const result = tagged ? JSON.parse(tagged.slice(RESULT_TAG.length)) : null; + const passthrough = lines.filter((line) => !line.startsWith(RESULT_TAG)).join('\n').trim(); + if (passthrough) console.log(passthrough); + if (run.stderr) process.stderr.write(run.stderr.endsWith('\n') ? run.stderr : `${run.stderr}\n`); + if (run.status === 2 && !result) { process.exitCode = 2; return; } // the child said why it could not run -let language; -try { language = await Language.load(wasmPath); } -catch (e) { fail(2, `failed to load ${wasmPath}: ${e.message}`); } + console.log(`grammar: ${path.basename(wasmPath)}`); + if (result) { + console.log(` ABI version: ${result.abi}`); + console.log(` runtime: node ${[result.node, ...result.execArgv].join(' ')}, with ${result.companions} other grammars loaded`); + console.log(` parses: ${result.clean} clean / ${result.errors} with errors (of ${result.iters})`); + if (result.large) { + const { copies, bytes, nodes, ms } = result.large; + console.log(nodes === null + ? ` largest file: the sample ${copies}x (${bytes} bytes) was still parsing after ${FILE_BUDGET_MS / 1000} s, ` + + 'as long as indexing gives one file, and was stopped (not judged)' + : ` largest file: the sample ${copies}x (${bytes} bytes, ${nodes} nodes) in ${ms} ms`); + } + } -const parser = new Parser(); -parser.setLanguage(language); + const [code, verdict] = verdictFor(run, result, timedOut, timeoutMs); + console.log(verdict); + process.exitCode = code; +} -let ok = 0, err = 0; -for (let i = 0; i < iters; i++) { - const tree = parser.parse(source); - if (tree.rootNode.hasError) err++; else ok++; +function verdictFor(run, result, timedOut, timeoutMs) { + const seconds = Math.round(timeoutMs / 1000); + const finishedAnyway = 'Finishing the parses is not the same as being safe to use: indexing would ' + + 'go down the same way. Do not vendor this wasm.'; + if (timedOut) { + return [1, result + ? `RESULT: FAIL — the parses finished, but the process running them had not exited after ${seconds} s ` + + `and was stopped. ${finishedAnyway}` + : `RESULT: FAIL — the parses had not finished after ${seconds} s, so the process running them was stopped: ` + + 'the grammar hangs, or is far too slow, on this sample. Do not vendor this wasm.']; + } + if (run.status !== 0 || !result) { + return [1, result + ? `RESULT: FAIL — the parses finished, but the process running them then died (${describeEnd(run)}). ${finishedAnyway}` + : `RESULT: FAIL — the process running the parses died (${describeEnd(run)}) before they finished. ` + + 'Indexing would go down the same way. Do not vendor this wasm.']; + } + if (result.errors > 0) { + return [1, `RESULT: FAIL — ${result.errors}/${result.iters} parses produced ERROR trees on a valid sample. ` + + 'This grammar corrupts under web-tree-sitter; vendor a newer (ABI 14/15) wasm ' + + '(see SKILL.md "Find a grammar"). Confirm your sample is syntactically valid first.']; + } + if (result.diverged > 0) { + return [1, `RESULT: FAIL — ${result.diverged}/${result.iters - 1} later parses of the same sample gave a ` + + 'different tree than the first. This grammar corrupts under web-tree-sitter; vendor a newer ' + + '(ABI 14/15) wasm (see SKILL.md "Find a grammar").']; + } + return [0, 'RESULT: PASS — grammar parses cleanly and reuses safely, and the process running it exited cleanly.']; } -console.log(`grammar: ${wasmPath.split('/').pop()}`); -console.log(` ABI version: ${language.abiVersion}`); -console.log(` parses: ${ok} clean / ${err} with errors (of ${iters})`); -if (err > 0) { - console.log( - `RESULT: FAIL — ${err}/${iters} parses produced ERROR trees on a valid sample. ` + - `This grammar corrupts under web-tree-sitter; vendor a newer (ABI 14/15) wasm ` + - `(see SKILL.md "Find a grammar"). Confirm your sample is syntactically valid first.` - ); - process.exit(1); +function describeEnd(run) { + if (run.signal) return `signal ${run.signal}`; + const code = run.status; + return code > 255 ? `exit code ${code}, 0x${(code >>> 0).toString(16).toUpperCase()}` : `exit code ${code}`; } -console.log('RESULT: PASS — grammar parses cleanly and reuses safely.'); -process.exit(0); + +/** The child: load and exercise the grammar the way indexing does, then report. */ +async function runGrammar(wasmPath, samplePath, itersArg) { + const iters = Number(itersArg); + const { Parser, Language } = await import('web-tree-sitter'); + try { await Parser.init(); } + catch { await Parser.init({ locateFile: () => require.resolve('web-tree-sitter/tree-sitter.wasm') }); } + + // The other grammars an index loads beside this one: the corruption surfaces + // under the multi-grammar runtime that real indexing uses, not a single + // grammar in isolation. + let companions = 0; + for (const file of readdirSync(VENDORED_DIR).filter((f) => f.endsWith('.wasm')).sort()) { + const companion = path.join(VENDORED_DIR, file); + if (samePath(companion, wasmPath)) continue; + try { await Language.load(companion); companions++; } catch { /* not the grammar under test */ } + } + + let language; + try { language = await Language.load(wasmPath); } + catch (e) { + // No process.exit: on Windows a pipe write can still be pending. + console.error(`[check-grammar] failed to load ${wasmPath}: ${e.message}`); + process.exitCode = 2; + return; + } + + const parser = new Parser(); + parser.setLanguage(language); + const source = readFileSync(samplePath, 'utf8'); + + let clean = 0, errors = 0, diverged = 0, first = null; + for (let i = 0; i < iters; i++) { + const tree = parser.parse(source); + if (!tree) throw new Error('the parser returned no tree'); + const shape = walk(tree); + if (tree.rootNode.hasError) errors++; else clean++; + if (!first) first = shape; + else if (shape.nodes !== first.nodes || shape.hash !== first.hash) diverged++; + tree.delete(); + } + + // The largest file indexing reads, with the time indexing gives one file, + // unless the grammar has already failed. Repeating a sample can make code the + // language rejects, and recovering from that can be slow, so here only a + // crash or a hang counts. + let large = null; + if (errors === 0 && diverged === 0) { + const copies = Math.max(1, Math.floor(MAX_FILE_BYTES / (Buffer.byteLength(source) + 1))); + const big = Array(copies).fill(source).join('\n'); + const started = performance.now(); + const bigTree = parser.parse(big, null, { progressCallback: () => performance.now() - started > FILE_BUDGET_MS }); + large = { copies, bytes: Buffer.byteLength(big), nodes: bigTree ? walk(bigTree).nodes : null, ms: Math.round(performance.now() - started) }; + bigTree?.delete(); + } + parser.delete(); + + console.log(RESULT_TAG + JSON.stringify({ + abi: language.abiVersion, iters, clean, errors, diverged, companions, large, + node: process.version, execArgv: process.execArgv, + })); +} + +/** Visit every node, as extraction does; returns the tree's size and a hash of its shape. */ +function walk(tree) { + const cursor = tree.walk(); + let nodes = 0, hash = 2166136261; + const mix = (value) => { hash = Math.imul(hash ^ value, 16777619) >>> 0; }; + for (;;) { + nodes++; + mix(cursor.nodeTypeId); mix(cursor.currentFieldId); mix(cursor.startIndex); mix(cursor.endIndex); + if (cursor.gotoFirstChild()) continue; + while (!cursor.gotoNextSibling()) { + if (!cursor.gotoParent()) { cursor.delete(); return { nodes, hash }; } + } + } +} + +if (process.env[CHILD_ENV] === '1') await runGrammar(...process.argv.slice(2)); +else judge(process.argv.slice(2)); From 80c3680068d20b39c037afbcacf783c644b34876 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 17:23:05 +0000 Subject: [PATCH 41/95] fix(extraction): drop the extractVariables hook, which the core never called (#2809) Root cause: LanguageExtractor in src/extraction/tree-sitter-types.ts declared an optional extractVariables hook, documented together with the VariableInfo type it returned, but nothing ever called it. Variables are named by the core's own extractVariable, from its per-language branches and the variableTypes list. A language author who wrote the hook got nothing back, with no error and no warning; the reporter hit this while adding Nim. Fix: remove the hook and VariableInfo. No language implemented the hook, nothing else referred to either one, and the module is not part of the public API. The variableTypes doc now says how the core finds the declared names, and that a grammar which nests them deeper gets its own branch in extractVariable, as the add-lang skill already says. Wiring the hook in was the alternative, but nothing needs it, so it would be an API no language exercises; the open Nim PR uses a core branch instead. Validation: - New __tests__/language-extractor-hooks.test.ts reads LanguageExtractor's members with the TypeScript parser and checks that the extraction core reads each one through an extractor receiver. On main it fails, listing extractVariables; with the fix both tests pass, and all 59 remaining members are read. - The compiled JavaScript is byte-identical: of 1,164 dist files hashed, only tree-sitter-types.d.ts and its map changed. - expressjs/express (0dee2c52, 214 files) indexed before and after, on the native-kernel and the wasm paths: the graph dumps are byte-identical (12,450 lines; 1,135 nodes, 2,258 edges). Fixes #2465 Co-Authored-By: Claude Opus 5.5 --- __tests__/language-extractor-hooks.test.ts | 95 ++++++++++++++++++++++ src/extraction/tree-sitter-types.ts | 32 ++------ 2 files changed, 103 insertions(+), 24 deletions(-) create mode 100644 __tests__/language-extractor-hooks.test.ts diff --git a/__tests__/language-extractor-hooks.test.ts b/__tests__/language-extractor-hooks.test.ts new file mode 100644 index 0000000000..d7509ce573 --- /dev/null +++ b/__tests__/language-extractor-hooks.test.ts @@ -0,0 +1,95 @@ +/** + * Static source guard: every member of LanguageExtractor + * (src/extraction/tree-sitter-types.ts) is read by the extraction core. + * + * A hook the core never calls is a trap. A language author finds it, writes + * it, and gets nothing back: no error, no warning, no node. That happened with + * `extractVariables`, declared and documented but never called (#2465). + * + * "Read" means a property access through an `extractor` receiver + * (`this.extractor.x`, `this.extractor?.x`, `this.extractor!.x`, + * `extractor.x`, `extractor?.x`) or a destructuring of one, anywhere under + * src/extraction except the type file itself and the per-language extractors + * in languages/, which define these members rather than read them. The guard + * fails closed: a member read some other way shows up as unread. + */ +import { describe, it, expect } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import ts from 'typescript'; + +const EXTRACTION_DIR = path.join(__dirname, '..', 'src', 'extraction'); +const TYPES_FILE = path.join(EXTRACTION_DIR, 'tree-sitter-types.ts'); + +function parse(file: string): ts.SourceFile { + return ts.createSourceFile(file, fs.readFileSync(file, 'utf8'), ts.ScriptTarget.Latest, true); +} + +function declaredMembers(): string[] { + const source = parse(TYPES_FILE); + const iface = source.statements.find( + (s): s is ts.InterfaceDeclaration => ts.isInterfaceDeclaration(s) && s.name.text === 'LanguageExtractor' + ); + if (!iface) throw new Error('LanguageExtractor not found in tree-sitter-types.ts'); + return iface.members.flatMap((m) => (m.name && ts.isIdentifier(m.name) ? [m.name.text] : [])); +} + +function coreFiles(dir = EXTRACTION_DIR): string[] { + const out: string[] = []; + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const p = path.join(dir, entry.name); + if (entry.isDirectory()) { + if (entry.name !== 'languages') out.push(...coreFiles(p)); + } else if (entry.name.endsWith('.ts') && !entry.name.endsWith('.d.ts') && p !== TYPES_FILE) { + out.push(p); + } + } + return out; +} + +/** `extractor`, `this.extractor`, either one behind `!` or parentheses. */ +function isExtractorReceiver(node: ts.Expression): boolean { + while (ts.isNonNullExpression(node) || ts.isParenthesizedExpression(node)) node = node.expression; + if (ts.isIdentifier(node)) return node.text === 'extractor'; + return ts.isPropertyAccessExpression(node) && + node.expression.kind === ts.SyntaxKind.ThisKeyword && + node.name.text === 'extractor'; +} + +function membersRead(): Set { + const read = new Set(); + for (const file of coreFiles()) { + const visit = (node: ts.Node): void => { + if (ts.isPropertyAccessExpression(node) && isExtractorReceiver(node.expression)) { + read.add(node.name.text); + } else if (ts.isVariableDeclaration(node) && ts.isObjectBindingPattern(node.name) && + node.initializer && isExtractorReceiver(node.initializer)) { + for (const element of node.name.elements) { + const key = element.propertyName ?? element.name; + if (ts.isIdentifier(key)) read.add(key.text); + } + } + ts.forEachChild(node, visit); + }; + visit(parse(file)); + } + return read; +} + +describe('LanguageExtractor hooks (#2465)', () => { + it('declares only members the extraction core reads', () => { + const read = membersRead(); + const unread = declaredMembers().filter((name) => !read.has(name)); + expect(unread, 'LanguageExtractor members the extraction core never reads: call each one, or remove it').toEqual([]); + }); + + it('sees the members the core does read', () => { + // Guards the guard: if the receiver matching broke, every member would + // look unread, or none would be checked at all. + const read = membersRead(); + for (const name of ['functionTypes', 'variableTypes', 'extractImport', 'preParse', 'extractBareCall']) { + expect(read.has(name), name).toBe(true); + } + expect(declaredMembers().length).toBeGreaterThan(40); + }); +}); diff --git a/src/extraction/tree-sitter-types.ts b/src/extraction/tree-sitter-types.ts index ae629e5538..ade9fff8b9 100644 --- a/src/extraction/tree-sitter-types.ts +++ b/src/extraction/tree-sitter-types.ts @@ -25,23 +25,6 @@ export interface ImportInfo { handledRefs?: boolean; } -/** - * Information about a single variable within a declaration. - * Returned by a language's extractVariables hook. - */ -export interface VariableInfo { - /** Variable name */ - name: string; - /** Node kind: 'variable' or 'constant' */ - kind: NodeKind; - /** Optional signature string */ - signature?: string; - /** If set, this declarator is actually a function and should be extracted as such */ - delegateToFunction?: SyntaxNode; - /** The AST node to use for positioning (may differ from the declaration node) */ - positionNode?: SyntaxNode; -} - /** * Context object passed to language hooks that need to call back into the core extractor. * Provides a controlled API surface — hooks can create nodes, visit children, and add @@ -137,7 +120,14 @@ export interface LanguageExtractor { importTypes: string[]; /** Node types that represent function calls */ callTypes: string[]; - /** Node types that represent variable declarations (const, let, var, etc.) */ + /** + * Node types that represent variable declarations (const, let, var, etc.). + * By default the core names each declared variable from the node's direct + * `identifier` or `variable_declarator` children (`extractVariable` in + * tree-sitter.ts). A grammar that nests the names deeper, like Swift's + * `pattern` or Lua's `variable_list`, gets its own branch there; there is no + * per-language hook for it. + */ variableTypes: string[]; /** Node types that represent class fields (extracted as 'field' kind inside class bodies) */ fieldTypes?: string[]; @@ -331,12 +321,6 @@ export interface LanguageExtractor { */ extractImport?: (node: SyntaxNode, source: string) => ImportInfo | null; - /** - * Extract variable declarations from a variable declaration node. - * Returns info about each declared variable, allowing the core to create nodes. - */ - extractVariables?: (node: SyntaxNode, source: string) => VariableInfo[]; - /** * Extract receiver/owner type name from a method declaration. * Used by Go to get the struct receiver (e.g., "scrapeLoop" from "func (sl *scrapeLoop) run()"). From f3fdba2955fb2979c4c4f09e61d300f9f2fd277e Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 17:50:35 +0000 Subject: [PATCH 42/95] fix(pascal): calls inside nested Delphi routines are indexed, each under its routine (#2810) A Pascal routine may declare routines of its own between its header and its begin. The extractor walked only a routine's body block, so a nested routine's code was never read: what it called was missing, and a call to it from the routine that declares it went by name to any routine elsewhere with that name (in mORMot2's tests, Test, Check and One calls landed on other classes' methods). A nested routine is now a function node qualified under the routine it is nested in (TFoo::Bar::Inner), contained by it and walked with itself as the caller. It sees the parameters and locals of the routines around it that are declared above it. Its name means it in the routine that declares it, in the routines nested there after it and in its own body, ahead of the class and the unit. A call resolved to a nested routine from anywhere else is what the call's own scope declares, or nothing. A nested routine's bare calls reach the class of the method it is nested in first. Where a routine's tree has errors and its parse swallowed the unit's routines after it (column 0, or after the body), those are walked as the unit's own; of two bodies under {$IFDEF} the first is walked, as before. Validation against origin/main 7b50ca83, nodes and edges removed/added: SynEdit +239, -6/+1127; HeidiSQL +404, -178/+2352; mORMot2 +574, -956/+9565; DelphiMVCFramework +134, -279/+1049; DWScript +434, -346/+3351; leveldb and SDWebImage byte-identical. Sampled: calls to nested routines 38/38 right, calls made in them 44/59 (ordinary name matching's misses), removed edges 30/30 wrong links. Index time within about 5%, in proportion to the code now walked. Covers the nested-routine part of #2550. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/pascal-nested-routines.test.ts | 363 +++++++++++++++++++++++ __tests__/pascal-system-calls.test.ts | 2 +- src/extraction/tree-sitter.ts | 108 ++++--- src/resolution/index.ts | 15 +- src/resolution/pascal-scope.ts | 137 ++++++++- 6 files changed, 580 insertions(+), 46 deletions(-) create mode 100644 __tests__/pascal-nested-routines.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b3e62ce04..cf15b332bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Delphi, a call made on the result of another call, like `MD5FileSized(TMeta.ExtendedPath(APath)).IsSame(AKey)` or `List[IndexOf(Name)].Free`, now has the inner calls indexed too, so `MD5FileSized` and `TMeta.ExtendedPath` list that routine among their callers. A typecast like `TButton(Sender).Caption` or `F := TFoo(Obj)` now counts as a use of the type instead of creating one. Thanks @stalexteam for the report. (#2547) - In Delphi, a routine called without parentheses, the usual way to call one that takes no arguments, like `Before;`, `if Ready then`, `Result := FProvider.Value + 1` or `QueryTabs.ActiveTab.MemoFilename`, now counts as a call, so it shows up in callers, callees and impact, and a method given to an event, like `OnClick := ButtonClick`, is linked as a reference. A name is linked only to a method or function its code reaches, in its class and the classes it inherits from (`class sealed (TBase)` included), its unit, or the units it uses, never to a field, property or variable of that name. Thanks @OleksiyPenkov for the report, and @drakeo338 and @OleksiyPenkov for their pull requests. (#2514) - In Delphi, a call to something the compiler provides, like `SizeOf(X)`, `TypeInfo(TFoo)`, `High(A)`, `Inc(N)`, `Assert(…)` or a typecast like `Pointer(P)`, written in any letter case, no longer shows up as a call to a project method, field or variable that happens to share its name, unless the calling code's own class or units declare one, like a list class's own `Delete`. A call through a field, property or variable now counts only when it holds a routine, like an `OnChange` event or a `reference to function` variable, and calls in `{$I}` include files and into units the parser could only partly read now find the routines those units declare. Thanks @OleksiyPenkov for the report. (#2514) +- In Delphi, a routine declared inside another, between its header and its `begin`, is now indexed as a routine of its own under the one it belongs to (`TFoo.Bar`'s `Inner` is `TFoo::Bar::Inner`), with the calls its body makes. Before, its body was skipped, so what it called was missing from callers and impact, and a call to it from its own routine went to any routine of the same name elsewhere. Thanks @OleksiyPenkov for the pull request this follows. (#2550) - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. diff --git a/__tests__/pascal-nested-routines.test.ts b/__tests__/pascal-nested-routines.test.ts new file mode 100644 index 0000000000..7c594db952 --- /dev/null +++ b/__tests__/pascal-nested-routines.test.ts @@ -0,0 +1,363 @@ +/** + * A Pascal routine may declare routines of its own between its header and + * its `begin`. Their bodies were never walked, so whatever they called was + * missing from the graph, and a call to one from the routine that declares + * it went by name to any routine elsewhere that shares it. Now a nested + * routine is a function qualified under the routine it is nested in + * (`THost::Fire::Inner`), its calls are its own, and its name means it + * wherever it is in scope: in the routine that declares it, in the routines + * nested after it there, and in its own body. Its locals and parameters are + * in scope for its body and for the routines nested inside it. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; + +beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['pascal']); +}); + +async function withIndex(files: Record, check: (cg: CodeGraph) => void): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-nested-')); + try { + for (const [name, code] of Object.entries(files)) fs.writeFileSync(path.join(dir, name), code); + const cg = await CodeGraph.init(dir, { index: true }); + try { + check(cg); + } finally { + cg.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +function routine(cg: CodeGraph, qualifiedName: string) { + const node = cg.getNodesByKind('method').concat(cg.getNodesByKind('function')).find((n) => n.qualifiedName === qualifiedName); + expect(node, qualifiedName).toBeDefined(); + return node!; +} + +/** What a routine calls, as `file:Target`, sorted. */ +function callsFrom(cg: CodeGraph, qualifiedName: string): string[] { + return [...new Set(cg.getOutgoingEdges(routine(cg, qualifiedName).id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${path.basename(n.filePath)}:${n.qualifiedName}`))].sort(); +} + +// The routines elsewhere a nested routine's name could be taken for. +const LIB = `unit Lib; + +interface + +procedure Inner(N: Integer); +procedure Helper(N: Integer); +procedure Log(const S: string); + +implementation + +procedure Inner(N: Integer); begin end; +procedure Helper(N: Integer); begin end; +procedure Log(const S: string); begin end; + +end. +`; + +function host(implementation: string): string { + return `unit Host; + +interface + +uses Lib; + +type + THost = class + public + procedure Read; + procedure Fire; + function Count(O: TObject): Integer; + end; + + TOther = class + public + function Count(O: TObject): Integer; + end; + +implementation + +function THost.Count(O: TObject): Integer; begin Result := 0; end; +procedure THost.Read; begin end; +function TOther.Count(O: TObject): Integer; begin Result := 1; end; + +${implementation} + +end. +`; +} + +describe('Pascal nested routines', () => { + it('indexes a nested routine as a function of its own, called by the routine that declares it', async () => { + const main = host(`procedure THost.Fire; +var + Count: Integer; + + procedure Inner(N: Integer); + var + K: Integer; + begin + K := N + Count; + Helper(K); + end; + +begin + Inner(1); +end;`); + await withIndex({ 'Lib.pas': LIB, 'Host.pas': main }, (cg) => { + const inner = routine(cg, 'THost::Fire::Inner'); + expect(inner.kind).toBe('function'); + const contains = cg.getOutgoingEdges(routine(cg, 'THost::Fire').id).filter((e) => e.kind === 'contains'); + expect(contains.map((e) => e.target)).toContain(inner.id); + // Its call is its own, and the routine's call to `Inner` is to it, not to Lib's. + expect(callsFrom(cg, 'THost::Fire::Inner')).toEqual(['Lib.pas:Helper']); + expect(callsFrom(cg, 'THost::Fire')).toEqual(['Host.pas:THost::Fire::Inner']); + }); + }); + + it('follows calls through nesting levels, siblings above and the routine itself', async () => { + const main = host(`procedure THost.Fire; + + procedure First; + begin + Log('first'); + end; + + procedure Again(N: Integer); + begin + if N > 0 then + Again(N - 1); + First; + end; + + procedure Level1; + + procedure Level2; + begin + Again(2); + Level1; + end; + + begin + Level2; + end; + +begin + Level1; +end;`); + await withIndex({ 'Lib.pas': LIB, 'Host.pas': main }, (cg) => { + expect(callsFrom(cg, 'THost::Fire')).toEqual(['Host.pas:THost::Fire::Level1']); + expect(callsFrom(cg, 'THost::Fire::Level1')).toEqual(['Host.pas:THost::Fire::Level1::Level2']); + // A deeper routine reaches a routine nested further out, and the one it is in. + expect(callsFrom(cg, 'THost::Fire::Level1::Level2')).toEqual([ + 'Host.pas:THost::Fire::Again', + 'Host.pas:THost::Fire::Level1', + ]); + // A routine calls itself, and an earlier sibling, with or without parentheses. + expect(callsFrom(cg, 'THost::Fire::Again')).toEqual(['Host.pas:THost::Fire::Again', 'Host.pas:THost::Fire::First']); + expect(callsFrom(cg, 'THost::Fire::First')).toEqual(['Lib.pas:Log']); + }); + }); + + it('keeps a nested routine to the routine it is nested in', async () => { + const main = host(`procedure THost.Fire; + + procedure Outer; + + procedure Hidden(N: Integer); + begin + end; + + begin + Hidden(1); + end; + + procedure Peer; + begin + Hidden(2); + Helper(3); + end; + +begin + Outer; + Peer; +end; + +procedure Tidy; + + procedure Helper(N: Integer); + begin + end; + +begin + Helper(4); +end;`); + await withIndex({ 'Lib.pas': LIB, 'Host.pas': main }, (cg) => { + expect(callsFrom(cg, 'THost::Fire::Outer')).toEqual(['Host.pas:THost::Fire::Outer::Hidden']); + // Neither another routine's nested `Hidden` nor `Tidy`'s nested `Helper` is in scope here. + expect(callsFrom(cg, 'THost::Fire::Peer')).toEqual(['Lib.pas:Helper']); + expect(callsFrom(cg, 'Tidy')).toEqual(['Host.pas:Tidy::Helper']); + }); + }); + + it('keeps a later sibling out of scope, and a nested routine out of another unit', async () => { + const main = host(`procedure THost.Fire; + + procedure Early; + begin + Inner(1); + end; + + procedure Inner(N: Integer); + begin + end; + +begin + Early; + Inner(2); +end;`); + const other = `unit Other; + +interface + +uses Lib, Host; + +procedure Run; + +implementation + +procedure Run; +begin + Inner(3); +end; + +end. +`; + await withIndex({ 'Lib.pas': LIB, 'Host.pas': main, 'Other.pas': other }, (cg) => { + // Declared after it, the sibling isn't one `Early` can call: Lib's `Inner` is. + expect(callsFrom(cg, 'THost::Fire::Early')).toEqual(['Lib.pas:Inner']); + expect(callsFrom(cg, 'THost::Fire')).toEqual(['Host.pas:THost::Fire::Early', 'Host.pas:THost::Fire::Inner']); + expect(callsFrom(cg, 'Run')).toEqual(['Lib.pas:Inner']); + }); + }); + + it('gives a nested routine the class and the locals of the routines around it', async () => { + const main = host(`procedure THost.Fire; +var + Before: TObject; + + procedure Inner; + begin + Read; + Before.Free; + Helper(Count(Before)); + end; + +var + Log: Integer; + + procedure Second(Helper: Integer); + begin + Log := Helper + 1; + end; + +begin + Inner; + Second(1); +end;`); + await withIndex({ 'Lib.pas': LIB, 'Host.pas': main }, (cg) => { + // The class's `Read` and `Count` are in scope there, not `TOther`'s + // `Count`, and `Before` is the routine's local. + expect(callsFrom(cg, 'THost::Fire::Inner')).toEqual(['Host.pas:THost::Count', 'Host.pas:THost::Read', 'Lib.pas:Helper']); + // A parameter and a local of the routine around it hide Lib's routines. + expect(callsFrom(cg, 'THost::Fire::Second')).toEqual([]); + expect(callsFrom(cg, 'THost::Fire')).toEqual(['Host.pas:THost::Fire::Inner', 'Host.pas:THost::Fire::Second']); + }); + }); + + it('takes a routine a broken parse nests, at the start of its line, for one of the unit', async () => { + // The missing `;` after `procedure First` leaves `Second` inside it in the tree. + const script = `procedure First +begin +end; + +function Second : Integer +var + i : Integer; +begin + i := 1; + Result := Helper(i); +end; + +type + TThing = class + method Run : String; + method Stop; + end; + +method TThing.Run : String +begin + Result := ''; +end; + +method TThing.Stop +const + c = 2; +begin +end; +`; + const lib = `unit Lib; + +interface + +function Helper(N: Integer): Integer; + +implementation + +function Helper(N: Integer): Integer; begin Result := N; end; + +end. +`; + await withIndex({ 'Lib.pas': lib, 'Script.pas': script }, (cg) => { + const names = cg.getNodesByKind('function').filter((n) => n.filePath === 'Script.pas').map((n) => n.qualifiedName); + expect(names).toContain('Second'); + expect(names).not.toContain('First::Second'); + expect(callsFrom(cg, 'Second')).toEqual(['Lib.pas:Helper']); + }); + }); + + it('takes a nested function\'s own name for its result, and a nested name like a System routine\'s for the nested one', async () => { + const main = host(`procedure THost.Fire; + + function Calc: Integer; + begin + Calc := 5; + end; + + procedure Delete(Index: Integer); + begin + end; + +begin + if Calc > 0 then + Delete(1); +end;`); + await withIndex({ 'Lib.pas': LIB, 'Host.pas': main }, (cg) => { + expect(callsFrom(cg, 'THost::Fire::Calc')).toEqual([]); + expect(callsFrom(cg, 'THost::Fire')).toEqual(['Host.pas:THost::Fire::Calc', 'Host.pas:THost::Fire::Delete']); + }); + }); +}); diff --git a/__tests__/pascal-system-calls.test.ts b/__tests__/pascal-system-calls.test.ts index b26e43c24c..6714ee974f 100644 --- a/__tests__/pascal-system-calls.test.ts +++ b/__tests__/pascal-system-calls.test.ts @@ -378,7 +378,7 @@ end. expect(callsFrom(cg, 'TConn::Connect')).toEqual(['24: TOciLib::HandleAlloc']); // A parameter and a nested routine, not the properties named so. expect(callsFrom(cg, 'Hash')).toEqual([]); - expect(callsFrom(cg, 'Test')).toEqual([]); + expect(callsFrom(cg, 'Test')).toEqual(['39: Test::Direct']); }); }); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 95c39a0e73..94453c64ea 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -8670,61 +8670,103 @@ export class TreeSitterExtractor { if (!parentId) parentId = this.nodeStack[this.nodeStack.length - 1]; if (!parentId) return; - // Visit the block for calls - const block = node.namedChildren.find( - (c: SyntaxNode) => c.type === 'block' - ); - if (block) { - // `TList.Add` belongs to `TList`, whose node may spell its parameters - // with their constraints (`TList`). - const owner = fullName.includes('.') ? fullName.replace(/<[^>]*>/g, '').split('.').slice(0, -1).join('::') : null; - this.pascalScopes.push({ - locals: this.pascalRoutineLocals(node), - classQn: owner, - routine: shortNameKey.replace(/<.*$/, ''), - startLine: node.startPosition.row + 1, - }); - this.nodeStack.push(parentId); - this.visitPascalBlock(block); - this.nodeStack.pop(); - this.pascalScopes.pop(); + // `TList.Add` belongs to `TList`, whose node may spell its parameters + // with their constraints (`TList`). + const owner = fullName.includes('.') ? fullName.replace(/<[^>]*>/g, '').split('.').slice(0, -1).join('::') : null; + const routineId = parentId; + // What a routine nested in this one is qualified under: the routine's + // node, or its name as written when the implementation matched none. + const qualifiedName = (): string => { + const routine = this.nodes.find((n) => n.id === routineId); + return routine && routine.kind !== 'file' ? routine.qualifiedName : fullName.replace(/<[^>]*>/g, '').split('.').join('::'); + }; + this.walkPascalRoutine(node, routineId, owner, shortNameKey.replace(/<.*$/, ''), qualifiedName); + } + + /** + * A routine's declarations and body, walked for its calls with the routine + * as their caller. A routine nested in it (`procedure Inner;` between its + * header and `begin`) is a function of its own, qualified under it + * (`TFoo::Bar::Inner`), whose calls are its own; it sees the names the + * routine declares above it, as the routine's body sees them all. + */ + private walkPascalRoutine(node: SyntaxNode, routineId: string, classQn: string | null, routine: string, qualifiedName: () => string): void { + const locals = this.pascalRoutineLocals(node, true); + this.pascalScopes.push({ locals, classQn, routine, startLine: node.startPosition.row + 1 }); + this.nodeStack.push(routineId); + let block: SyntaxNode | null = null; + const misplaced: SyntaxNode[] = []; + const broken = node.hasError; + for (const c of node.namedChildren) { + if (c.type === 'declVars' || c.type === 'declConsts') { + for (const decl of c.namedChildren) this.declarePascalLocal(locals, decl); + } else if (c.type === 'defProc') { + // A nested routine comes before the body. Where the routine's tree + // has errors, its parse can swallow the unit's routines after it, + // which start their lines where it does: those are the unit's. + if (block || (broken && c.startPosition.column <= node.startPosition.column)) misplaced.push(c); + else this.extractPascalNestedRoutine(c, classQn, qualifiedName()); + } else if (c.type === 'block' && !block) { + // The first: `{$IFDEF}`s around two bodies leave a second. + block = c; + } } + if (block) this.visitPascalBlock(block); + this.nodeStack.pop(); + this.pascalScopes.pop(); + for (const c of misplaced) this.extractPascalDefProc(c); + } + + /** A routine nested in another (see walkPascalRoutine), under the qualified name of the one it is nested in. */ + private extractPascalNestedRoutine(node: SyntaxNode, classQn: string | null, outerQualifiedName: string): void { + const declProc = node.namedChildren.find((c: SyntaxNode) => c.type === 'declProc'); + const nameNode = declProc ? getChildByField(declProc, 'name') : null; + if (!declProc || nameNode?.type !== 'identifier') return; + const name = getNodeText(nameNode, this.source); + const qualifiedName = `${outerQualifiedName}::${name}`; + const fn = this.createNode('function', name, node, { + qualifiedName, + signature: this.extractor?.getSignature?.(declProc, this.signatureSource), + }); + if (!fn) return; + this.walkPascalRoutine(node, fn.id, classQn, name.toLowerCase(), () => qualifiedName); } /** * The names a routine (a `defProc`) or an anonymous method (a `lambda`) * declares before its body, lowercased, with the class or record type each * is declared with (null when it has none, or none a name can stand for): - * parameters, `var`/`const` sections, nested routines and `Result`. The - * body's inline declarations join as the walk reaches them + * parameters, `var`/`const` sections and `Result` (with `headerOnly`, the + * parameters and `Result`; walkPascalRoutine adds the sections in order). + * The body's inline declarations join as the walk reaches them * (`declarePascalInline`). A bare name among them is the routine's own, - * never a call. + * never a call. A nested routine's name is not: it is a routine to call. */ - private pascalRoutineLocals(routine: SyntaxNode): Map { + private pascalRoutineLocals(routine: SyntaxNode, headerOnly = false): Map { const names = new Map(); const header = routine.type === 'lambda' ? routine : routine.namedChildren.find((c: SyntaxNode) => c.type === 'declProc'); names.set('result', this.pascalTypeName(header?.namedChildren.find((c: SyntaxNode) => c.type === 'typeref'))); - const declare = (decl: SyntaxNode): void => { - const type = this.pascalTypeName(decl.namedChildren.find((c: SyntaxNode) => c.type === 'type' || c.type === 'typeref')); - for (const c of decl.namedChildren) { - if (c.type === 'identifier') names.set(getNodeText(c, this.source).toLowerCase(), type); - } - }; const args = header?.namedChildren.find((c: SyntaxNode) => c.type === 'declArgs'); for (const arg of args?.namedChildren ?? []) { - if (arg.type === 'declArg') declare(arg); + if (arg.type === 'declArg') this.declarePascalLocal(names, arg); } + if (headerOnly) return names; for (const c of routine.namedChildren) { if (c.type === 'declVars' || c.type === 'declConsts') { - for (const decl of c.namedChildren) declare(decl); - } else if (c.type === 'defProc') { - const name = getChildByField(c.namedChildren.find((x: SyntaxNode) => x.type === 'declProc') ?? c, 'name'); - if (name?.type === 'identifier') names.set(getNodeText(name, this.source).toLowerCase(), null); + for (const decl of c.namedChildren) this.declarePascalLocal(names, decl); } } return names; } + /** The names a parameter, `var` or `const` declaration declares, with the type it gives them (see pascalRoutineLocals). */ + private declarePascalLocal(names: Map, decl: SyntaxNode): void { + const type = this.pascalTypeName(decl.namedChildren.find((c: SyntaxNode) => c.type === 'type' || c.type === 'typeref')); + for (const c of decl.namedChildren) { + if (c.type === 'identifier') names.set(getNodeText(c, this.source).toLowerCase(), type); + } + } + /** * A name a routine's body declares where it stands — an inline `var`, a * `for` or `for … in` loop's variable, an `on E: EFoo do` handler's — joins diff --git a/src/resolution/index.ts b/src/resolution/index.ts index b088bef407..6a381191dc 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -29,7 +29,8 @@ import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformance import { clearDartLibraryMemos } from './dart-libraries'; import { clearVbnetReceiverMemos, isVbMemberRead, isVbPathCall, matchVbMemberRead, matchVbPathCall } from './vbnet-receivers'; import { - clearPascalScopeMemos, gatePascalDataCall, isPascalParenlessRef, isPascalSystemCall, matchPascalParenlessRef, matchPascalSystemCall, + clearPascalScopeMemos, gatePascalCall, isPascalParenlessRef, isPascalSystemCall, matchPascalNestedCall, + matchPascalParenlessRef, matchPascalSystemCall, } from './pascal-scope'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { gateDartLocal, clearDartLocalScopeMemos } from './dart-local-scope'; @@ -1424,9 +1425,10 @@ export class ReferenceResolver { !reachesAmbientModule(ref, specifier!, declaringFile, this.context)) return null; } if (!resolved || ref.referenceKind !== 'calls') return resolved; - // A Pascal call to data runs only a routine the data holds, in scope (see - // gatePascalDataCall). - const call = ref.language === 'pascal' ? gatePascalDataCall(resolved, ref, this.context) : resolved; + // A Pascal call to data runs only a routine the data holds, in scope, and + // one to a routine nested in another comes only from inside that one (see + // gatePascalCall). + const call = ref.language === 'pascal' ? gatePascalCall(resolved, ref, this.context) : resolved; if (!call) return null; const target = this.nodeById(call.targetNodeId); @@ -1462,6 +1464,11 @@ export class ReferenceResolver { // meaning to (`SizeOf`, `TypeInfo`, `length(S)`, `Pointer(P)`), whatever // its case: only a declaration in the call's scope shadows it. if (isPascalSystemCall(ref, this.context)) return matchPascalSystemCall(ref, this.context); + // A bare Pascal call to a routine nested where it is made (`Inner(1)` in + // the routine that declares `procedure Inner(N: Integer);`) is that one, + // and one made in a nested routine is what its scope declares first. + const nested = matchPascalNestedCall(ref, this.context); + if (nested) return nested; // A Dart member read (`x.area`) links the getter the receiver's type // reaches, as a call, or nothing — never a guess by name (#2338). if (isDartMemberRead(ref)) return matchDartMemberRead(ref, this.context); diff --git a/src/resolution/pascal-scope.ts b/src/resolution/pascal-scope.ts index a128a0c946..9f74e98129 100644 --- a/src/resolution/pascal-scope.ts +++ b/src/resolution/pascal-scope.ts @@ -144,8 +144,11 @@ export function matchPascalSystemCall(ref: UnresolvedRef, context: ResolutionCon } /** - * A Pascal call another strategy resolved to data — a field, a property, a - * variable, a constant — runs what the data holds, so it stands only for data + * What a Pascal call another strategy resolved stands for, when it landed on + * data or on a routine nested in another (see gatePascalNestedTarget). + * + * A call resolved to data — a field, a property, a variable, a constant — + * runs what the data holds, so it stands only for data * whose declared type is a routine's. A bare name is what its scope declares * (`FOnChange(Self)` in the class that declares its own `FOnChange`): the * routine or the routine-holding data there, whichever namesake the strategy @@ -157,9 +160,16 @@ export function matchPascalSystemCall(ref: UnresolvedRef, context: ResolutionCon * or a declaration the parse put in the wrong place: it stands for the one * routine-holding field or variable the project declares by that name. */ -export function gatePascalDataCall(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { +export function gatePascalCall(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { const target = context.getNodeById?.(result.targetNodeId); - if (!target || !DATA_KINDS.has(target.kind)) return result; + if (!target) return result; + if (DATA_KINDS.has(target.kind)) return gatePascalDataCall(target, result, ref, context); + if (isNestedRoutine(target, context)) return gatePascalNestedTarget(result, ref, context); + return result; +} + +/** See gatePascalCall: a call resolved to data. */ +function gatePascalDataCall(target: Node, result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { if (!isBareCall(ref, context)) return target.language === 'pascal' && isProceduralData(target, context) ? result : null; const lower = ref.referenceName.toLowerCase(); if (declaredInRoutine(lower, ref, context)) return null; @@ -309,14 +319,121 @@ function nameInScope(name: string, cls: Node | null, ref: UnresolvedRef, context return decl && isRoutine(decl.node) ? decl : null; } +/** + * The routine nested in the calling routine, or in one around it, that a + * bare name there names (`procedure Inner;` between `procedure TFoo.Bar;` + * and its `begin`, `TFoo::Bar::Inner`): the innermost one declared above the + * call, the calling routine itself among them. It comes before anything of + * the class or the unit. Null when none is. + */ +function nestedRoutineInScope(lower: string, ref: UnresolvedRef, context: ResolutionContext): Found | null { + // Most names no nested routine has: settled before any other lookup. + const named = pascalNamed(lower, context); + if (!named.some((n) => n.kind === 'function' && n.qualifiedName.includes('::'))) return null; + const memo = memos(context).nestedInScope; + const key = `${ref.fromNodeId}|${lower}`; + if (memo.has(key)) return memo.get(key)!; + let found: Found | null = null; + const caller = routineNode(ref.fromNodeId, context); + if (caller && caller.language === 'pascal') { + const chain = caller.qualifiedName.toLowerCase(); + let best: Node | null = null; + let depth = -1; + for (const n of named) { + if (n.filePath !== caller.filePath || n.startLine > ref.line || !isNestedRoutine(n, context)) continue; + const outer = ownerOf(n); + // Nested in the caller, or in a routine the caller is nested in. + if (chain !== outer && !chain.startsWith(`${outer}::`)) continue; + if (outer.length > depth || (outer.length === depth && n.startLine < best!.startLine)) { + best = n; + depth = outer.length; + } + } + if (best) found = { node: best, resolvedBy: 'exact-match', confidence: 0.95 }; + } + memo.set(key, found); + return found; +} + +/** + * Whether a node is a routine nested in another (see nestedRoutineInScope): + * a Pascal function qualified under a routine, which no code outside the one + * it is nested in can call. + */ +function isNestedRoutine(node: Node, context: ResolutionContext): boolean { + if (node.kind !== 'function' || node.language !== 'pascal') return false; + const cut = node.qualifiedName.lastIndexOf('::'); + if (cut < 0) return false; + const outer = node.qualifiedName.slice(0, cut); + const memo = memos(context).routineQns; + let routine = memo.get(outer); + if (routine === undefined) { + routine = context.getNodesByQualifiedName(outer).some((n) => n.language === 'pascal' && isRoutine(n)); + memo.set(outer, routine); + } + return routine; +} + +/** The routine a reference comes from, or null when it comes from a file or anything else. */ +function routineNode(id: string, context: ResolutionContext): Node | null { + const memo = memos(context).routineNodes; + if (memo.has(id)) return memo.get(id)!; + const node = context.getNodeById?.(id) ?? null; + const routine = node && isRoutine(node) ? node : null; + memo.set(id, routine); + return routine; +} + +/** + * A bare Pascal call (`Inner(1)`) to a routine nested where the call is (see + * nestedRoutineInScope). Made inside a nested routine, it reaches the rest of + * that routine's scope the same way: the class of the method it is nested in + * (`GetNext(Run)` in a tree method's nested routine is the tree's), its unit, + * the units it uses. Undefined when that scope holds no routine of the name, + * for the other strategies to resolve. + */ +export function matchPascalNestedCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | undefined { + if (ref.language !== 'pascal' || ref.referenceKind !== 'calls' || !isBareCall(ref, context)) return undefined; + const lower = ref.referenceName.toLowerCase(); + let found = nestedRoutineInScope(lower, ref, context); + if (!found) { + const caller = routineNode(ref.fromNodeId, context); + if (!caller || !isNestedRoutine(caller, context)) return undefined; + const decl = declarationInScope(lower, classOfCall(ref, context), ref, context); + if (!decl || !(isRoutine(decl.node) || (DATA_KINDS.has(decl.node.kind) && isProceduralData(decl.node, context)))) return undefined; + found = decl; + } + context.rememberNode?.(found.node); + return { original: ref, targetNodeId: found.node.id, confidence: found.confidence, resolvedBy: found.resolvedBy }; +} + +/** + * A Pascal call another strategy resolved to a routine nested in some other + * routine, which nothing outside that routine can call: a bare name is then + * what the call's scope declares, a routine or data holding one (mORMot's + * `StrLen` variable), and any other call links nothing. + */ +function gatePascalNestedTarget(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (!isBareCall(ref, context)) return null; + const lower = ref.referenceName.toLowerCase(); + const decl = declarationInScope(lower, classOfCall(ref, context), ref, context); + if (!decl || (!isRoutine(decl.node) && !(DATA_KINDS.has(decl.node.kind) && isProceduralData(decl.node, context)))) return null; + if (decl.node.id === result.targetNodeId) return result; + context.rememberNode?.(decl.node); + return { ...result, targetNodeId: decl.node.id, confidence: decl.confidence, resolvedBy: decl.resolvedBy }; +} + /** * The first declaration a bare name reaches from a routine of class `cls` - * (null: a free routine), whatever it declares: a member of the class or its - * ancestors, then a declaration of the unit, then of the units it uses, last - * first. Null when the project declares none of them, as for the System - * unit's names. + * (null: a free routine), whatever it declares: a routine nested in the + * calling routine or one around it, a member of the class or its ancestors, + * then a declaration of the unit, then of the units it uses, last first. + * Null when the project declares none of them, as for the System unit's + * names. */ function declarationInScope(lower: string, cls: Node | null, ref: UnresolvedRef, context: ResolutionContext): Found | null { + const nested = nestedRoutineInScope(lower, ref, context); + if (nested) return nested; const memo = memos(context).inScope; const key = `${cls?.id ?? ''}|${ref.filePath}|${lower}`; if (memo.has(key)) return memo.get(key)!; @@ -413,6 +530,9 @@ interface Memos { headerClasses: Map>; outlines: Map; conditionDepths: Map; + nestedInScope: Map; + routineNodes: Map; + routineQns: Map; } const MEMOS = new WeakMap(); @@ -424,6 +544,7 @@ function memos(context: ResolutionContext): Memos { ancestors: new Map(), usedFiles: new Map(), classByName: new Map(), unitFiles: new Map(), hierarchy: new Map(), memberTypes: new Map(), keywords: new Map(), procedural: new Map(), headerClasses: new Map(), outlines: new Map(), conditionDepths: new Map(), unitFileSets: new Map(), includes: new Map(), includers: new Map(), ownFiles: new Map(), + nestedInScope: new Map(), routineNodes: new Map(), routineQns: new Map(), }; MEMOS.set(context, m); } From 08478eea5323ce43ef7650e4968fa03a07fda6fa Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 18:11:37 +0000 Subject: [PATCH 43/95] fix(js,ts): a receiver declared outside the calling function goes by its declared type (#2811) react-dom's root.render() linked to the project's one class component render() when root was declared outside the calling function, a module's let root: Root | null or a let popupRoot: Root in the function around the one rendering, or held what createRoot(el) returned (#2589). The receiver reader scans only the calling function, so the type that rules a react-dom Root out was never read, and the call went to the one same-named method by its name. A TS/JS receiver the calling function doesn't type, in a function nested in another or one a module-scope variable names, is now read by its binding in scope at the call. A per-file brace-scope reader finds the innermost binding of the name: a const/let/var, a parameter, a destructuring, a for head, a function or class, a lone or destructured arrow parameter. A declaration or parameter written with a type, or a const/let/var whose whole value is a new of a class, types the call as the calling function's own declaration would; an annotated object literal stays its own members', and a new of a parameter names no class. A class component's render() and lifecycle methods are never a by-name guess for a receiver other than this or super, since React calls them, which also covers an untyped const r = createRoot(el). A first version that also took any value a package function returned to have no project method was dropped: it unlinked correct calls through useContext(...) values in social-app and outline. Validation: the new test fails 3 of 3 on main 89cd08fa and passes on the fix. A/B against main on 14 TS/JS repos: 35 edges removed, every one a wrong guess on a value whose type outside the calling function comes from outside the project (rolldown's watcher, node's Server and ChildProcess, playwright, unstorage, zod, NodeJS.WriteStream, HTMLElement) or nitro's ReactDOM.createRoot root; 18 added and 7 retargeted, all precise (memfs's onAbort watcher.close() now reaches FsaNodeFsWatcher::close, jest's currentSpec.cancel() Spec's); 92 same-target edges typed at 0.9 instead of guessed. The flask and gin control dumps are byte-identical. Index time, median of 8 interleaved runs on vite, nuxt, webdriverio and insomnia: +1.7% total, within noise (-3.6 to +5.2% per repo). A CLI sync that retypes a module binding and adds files matches a fresh index. Fixes #2589 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/js-receiver-binding-scope.test.ts | 142 +++++++ src/resolution/name-matcher.ts | 443 +++++++++++++++++++- 3 files changed, 576 insertions(+), 10 deletions(-) create mode 100644 __tests__/js-receiver-binding-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index cf15b332bd..94b262e8ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In TypeScript, an awaited call with type arguments like `await fetchApi("/me")` or `await api.get(url)` now links to the function it calls. Before, the call was dropped, so typed fetch wrappers looked like they had no callers. (#2586) - In JavaScript and TypeScript, a call or `new` in a parameter's default value, like `function renderWithNote(note = buildNote())`, a destructured `{ size = defaultSize() }` or `constructor(private svc = new Service())`, now counts as the function's own call. Before, it was dropped, so `buildNote` was missing that caller and impact on it came back incomplete. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and the pull request. (#2588) - In React, a component rendered at module scope, in a top-level constant or inside a test's `it(…)` callback, like `createRoot(el).render()` or `render()`, now counts as rendered there, so an app's entry reaches its root component and a component's tests show among its callers. (#2587) +- In JavaScript and TypeScript, react-dom's `root.render()` no longer links to your own class component's `render()`, such as an error boundary's, when `root` is declared outside the function making the call, like a module's `let root: Root | null`, or holds what `createRoot(el)` returned. A call on a variable declared outside the calling function now goes by the type that declaration gives it, so one declared with your own class, like `const store = new Store()` at the top of a file, links to that class's method, and a class component's `render()` and lifecycle methods, which only React calls, are no longer a guess for an untyped value. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report. (#2589) - In Python, `import a.b as X` written inside a function now links its calls like `X.f()`, as the same import at module level already did. Thanks @Egrn for the report. (#2600) - In Python, `from app import email` no longer links the importing file to a project variable named `app`, such as a FastAPI or Flask `app = FastAPI()`. (#2584) - In Python, a call through a module-level instance, like `_svc.delete(obj)` after `_svc = SceneService()`, now links to the method its class has or inherits, even when other classes have a method of that name. (#2582) diff --git a/__tests__/js-receiver-binding-scope.test.ts b/__tests__/js-receiver-binding-scope.test.ts new file mode 100644 index 0000000000..1db48f9554 --- /dev/null +++ b/__tests__/js-receiver-binding-scope.test.ts @@ -0,0 +1,142 @@ +/** + * react-dom's `root.render()` linked to the project's one class-component + * `render()` (#2589) when `root` was declared outside the calling function — a + * module's `let root: Root | null`, a `let popupRoot: Root` in the function + * around the one rendering — or held what `createRoot(el)` returned. A JS/TS + * receiver the calling function doesn't type is now read by its binding in + * scope at the call, as the function's own declaration is: the type `Root` + * says the method is react-dom's, and a project type the binding is written + * with or makes with `new` is the call's type. A nearer binding of the name — + * a parameter, an arrow's or a destructured one — still decides, and another + * function's binding holds in that function only. Nor is a class component's + * `render()` or lifecycle method a guess for a receiver nothing types: React + * calls those. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let roots: string[] = []; +let graphs: CodeGraph[] = []; + +afterEach(() => { + for (const graph of graphs) graph.close(); + graphs = []; + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); + roots = []; +}); + +async function index(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-receiver-scope-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + const file = path.join(root, rel); + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, content); + } + const graph = await CodeGraph.init(root, { index: true }); + graphs.push(graph); + return graph; +} + +/** `caller -> Owner::method` for every `calls` edge out of `file` that lands on a method. */ +function methodCalls(graph: CodeGraph, file: string): string[] { + const out: string[] = []; + for (const node of graph.getNodesInFile(file)) { + for (const e of graph.getOutgoingEdges(node.id)) { + if (e.kind !== 'calls') continue; + const target = graph.getNode(e.target)!; + if (target.kind === 'method') out.push(`${node.name} -> ${target.qualifiedName}`); + } + } + return out.sort(); +} + +const BOUNDARY = 'import { Component } from "react";\nexport class Boundary extends Component<{ children?: React.ReactNode }> {\n render() { return this.props.children; }\n}\n'; + +describe('a receiver is read by its binding in scope at the call', () => { + it("links react-dom's root to no project render, however it was declared (#2589)", async () => { + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app', dependencies: { react: '^18.0.0', 'react-dom': '^18.0.0' } }), + 'src/Boundary.tsx': BOUNDARY, + // The issue's reproduction. + 'src/popup.tsx': + 'import { createRoot, type Root } from "react-dom/client";\nlet root: Root | null = null;\n' + + 'export function showPopup(el: HTMLElement) {\n if (!root) root = createRoot(el);\n root.render(
);\n}\n', + 'src/popup2.tsx': + 'import { createRoot } from "react-dom/client";\nexport function mount(el: HTMLElement) {\n const r = createRoot(el);\n r.render(
);\n}\n', + 'src/popup3.tsx': + 'import { createRoot, type Root } from "react-dom/client";\nexport function mount3(el: HTMLElement) {\n const r: Root = createRoot(el);\n r.render(
);\n}\n' + + 'export function mount4(root: Root) {\n root.render(
);\n}\n', + 'src/more.tsx': + 'import { createRoot, type Root } from "react-dom/client";\nlet modRoot: Root;\n' + + 'export function a1(el: HTMLElement) { modRoot = createRoot(el); modRoot.render(
); }\n\n' + + 'export function outer(el: HTMLElement) {\n let popupRoot: Root | null = null;\n function renderPopup() { popupRoot!.render(
); }\n' + + ' popupRoot = createRoot(el);\n return renderPopup;\n}\n', + // Through the default export, at module scope and in a function, and in JavaScript. + 'src/main.tsx': + 'import ReactDOM from "react-dom/client";\nconst el = document.getElementById("root")!;\n' + + 'if (!el.innerHTML) {\n const root = ReactDOM.createRoot(el);\n root.render(
);\n}\n' + + 'const shared = ReactDOM.hydrateRoot(el,
);\nexport function rerender() {\n shared.render(
);\n}\n', + 'src/Card.jsx': 'import { Component } from "react";\nexport class Card extends Component {\n render() { return null; }\n}\n', + 'src/legacy.jsx': + 'import { createRoot } from "react-dom/client";\nexport function boot(el) {\n const root = createRoot(el);\n root.render(
);\n}\n', + }); + for (const file of ['src/popup.tsx', 'src/popup2.tsx', 'src/popup3.tsx', 'src/more.tsx', 'src/main.tsx', 'src/legacy.jsx']) { + expect(methodCalls(graph, file), file).toEqual([]); + } + }, 120_000); + + it("takes a project type from a binding outside the calling function", async () => { + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app' }), + 'src/queues.ts': + 'export class Store {\n dispatch(action: string) { return action; }\n}\n' + + 'export class Queue {\n dispatch(job: string) { return job; }\n}\n', + 'src/use.ts': + 'import { Store, Queue } from "./queues";\n' + + // A module's binding, named after neither class. + 'const s = new Store();\nlet q: Queue | undefined;\n' + + 'export function send() { s.dispatch("a"); }\nexport function drain() { q!.dispatch("b"); }\n' + + // The function around the one calling, and its parameter. + 'export function outer() {\n const pending = new Queue();\n function flush() { pending.dispatch("c"); }\n return flush;\n}\n' + + 'export function around(svc: Store) {\n function go() { svc.dispatch("d"); }\n return go;\n}\n', + }); + expect(methodCalls(graph, 'src/use.ts')).toEqual([ + 'drain -> Queue::dispatch', + 'flush -> Queue::dispatch', + 'go -> Store::dispatch', + 'send -> Store::dispatch', + ]); + }, 120_000); + + it('lets a nearer binding of the name decide', async () => { + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app', dependencies: { 'react-dom': '^18.0.0' } }), + // A method a call is guessed to is one of its own language's. + 'src/chart.tsx': 'export class Chart {\n render() { return 1; }\n}\n', + 'src/gauge.jsx': 'export class Gauge {\n close() { return 1; }\n}\n', + 'src/views.tsx': + 'import { createRoot, type Root } from "react-dom/client";\nimport { Chart } from "./chart";\n' + + 'let root: Root | null = null;\nexport function init(el: HTMLElement) { root = createRoot(el); }\n' + + // A typed parameter, an arrow's own parameter, a destructured one. + 'export function typed(root: Chart) { root.render(); }\n' + + 'export function each(items: Chart[]) { items.forEach((root) => root.render()); }\n' + + 'export function Panel({ root }: { root: any }) { return root.render(); }\n' + + // Another function's own declaration holds in that function only. + 'export function other() { const root = new Chart(); return root; }\n' + + 'export function show() { root!.render(
); }\n', + // A `new` of a parameter names no class to look the method up on. + 'src/later.jsx': + 'export function make(Backend) {\n const watcher = new Backend();\n function stop() { watcher.close(); }\n return stop;\n}\n', + }); + expect(methodCalls(graph, 'src/views.tsx')).toEqual([ + 'Panel -> Chart::render', + 'each -> Chart::render', + 'typed -> Chart::render', + ]); + expect(methodCalls(graph, 'src/later.jsx')).toEqual(['stop -> Gauge::close']); + }, 120_000); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index d300409f2e..c434635e1f 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -11632,15 +11632,7 @@ function enclosingOpenParen(source: string, at: number): number { * arrow. */ function bindsAtSites(source: string, name: string, sites: LocalBindingSites): boolean { - let patterns = LOCAL_BINDING_PATTERNS.get(name); - if (!patterns) { - patterns = localBindingPatterns(name, 'y'); - if (LOCAL_BINDING_PATTERNS.size >= LOCAL_BINDING_PATTERNS_CAP) { - LOCAL_BINDING_PATTERNS.delete(LOCAL_BINDING_PATTERNS.keys().next().value!); - } - LOCAL_BINDING_PATTERNS.set(name, patterns); - } - const { decl, fn, param } = patterns; + const { decl, fn, param } = stickyBindingPatterns(name); let from = 0; for (const at of sites.varDecls) { if (at < from) continue; @@ -17848,6 +17840,9 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { JS_FN_LOCAL_MEMO.delete(context); JS_CODE_BINDS.delete(context); JS_FN_CODE.delete(context); + JS_SCOPE_FILES.delete(context); + JS_MODULE_VALUES.delete(context); + REACT_LIFECYCLE_MEMO.delete(context); TS_FIELD_DECL_MEMO.delete(context); TS_CLASS_LINES.delete(context); TS_PASS_THROUGH_WRAPPERS.delete(context); @@ -17875,6 +17870,13 @@ function memoPatterns(key: string, build: () => RegExp[]): RegExp[] { return patterns; } +/** + * The type a TS annotation writes, captured: `Logger`, `Repository` (as + * `Repository`), and with brackets after it whole — `Item[]`, `Element["ids"]` + * — for matchMethodCall to read (arrayElementType, tsIndexedPropertyType). + */ +const TS_ANNOTATED_TYPE = '([A-Z][\\w.$]*(?:(?:\\s*<[^;=(){}<>]*(?:<[^;=(){}<>]*>[^;=(){}<>]*)*>)?\\s*\\[[^\\]]*\\])?)'; + export function localReceiverTypePatterns(language: Language, r: string): RegExp[] { return memoPatterns(`${language}|${r}`, () => buildLocalReceiverTypePatterns(language, r)); } @@ -17908,7 +17910,7 @@ function buildLocalReceiverTypePatterns(language: Language, r: string): RegExp[] // can tell an array of it (`items: Item[]`, `items: Item[]`), whose // own methods aren't an element's (arrayElementType), from one of its // properties (`groupIds: Element["groupIds"]`), which isn't it. - new RegExp(`\\b${r}\\b\\s*:\\s*([A-Z][\\w.$]*(?:(?:\\s*<[^;=(){}<>]*(?:<[^;=(){}<>]*>[^;=(){}<>]*)*>)?\\s*\\[[^\\]]*\\])?)`), // lg: Logger (annotation or typed param) + new RegExp(`\\b${r}\\b\\s*:\\s*${TS_ANNOTATED_TYPE}`), // lg: Logger (annotation or typed param) ]; case 'python': return [ @@ -18839,6 +18841,373 @@ function resolveAwaitedCallType( return { name: typeNode.name, filePath: typeNode.filePath }; } +/** + * A JS/TS file's code read for the bindings its names have (jsBindingAt): + * comments, strings, template literals and regex literals blanked; where each + * line starts; its braces as a tree of scopes, the first the file's own; and + * where a `const`/`let`/`var`, a `function`/`class` or an arrow can start. + * Per name, the bindings jsBindingSites finds. + */ +interface JsScopeFile { + code: string; + lineStarts: number[]; + scopes: { start: number; end: number; parent: number }[]; + varDecls: number[]; + fnDecls: number[]; + arrows: number[]; + bindings: Map; +} + +/** + * A binding of a name: where it holds (`from`, its block's `{` or its body's + * start, through `to`) and where it is written (`at`). A `const`/`let`/`var` + * of the name alone is a declaration, a parameter list's name a parameter, + * a `function` or `class` declaration a function. Any other binding — a + * destructuring, a `for` head, a lone or destructured arrow parameter — says + * nothing of the value. `reading` keeps what the binding says + * (jsBindingReading). + */ +interface JsBindingSite { + from: number; + to: number; + at: number; + kind: 'declaration' | 'parameter' | 'function' | 'other'; + reading?: JsBindingReading | null; +} + +/** + * What a binding says its value is: an instance of the type it is written + * with, or of the class its whole value makes with `new` (`raw` as written, + * on `line`). + */ +type JsBindingReading = { type: string; raw: string; line: number }; + +/** Per context, the files jsScopeFile read last, most recent last. */ +const JS_SCOPE_FILES = new WeakMap>(); +const JS_SCOPE_FILES_KEEP = 16; + +function jsScopeFile(filePath: string, context: ResolutionContext): JsScopeFile | null { + let files = JS_SCOPE_FILES.get(context); + if (!files) JS_SCOPE_FILES.set(context, (files = new Map())); + let file = files.get(filePath); + if (file !== undefined) { + files.delete(filePath); + } else { + file = null; + const source = context.readFile(filePath); + // A bundle's names are nothing a person declares a receiver with. + if (source !== null && source.length <= 1_000_000 && !isMinifiedContent(filePath, source)) { + const regexes: number[] = []; + const chars = blankStringContents(stripCommentsForRegex(source, 'javascript'), regexes).split(''); + for (let r = 0; r + 1 < regexes.length; r += 2) chars.fill(' ', regexes[r]! + 1, regexes[r + 1]!); + const code = chars.join(''); + const lineStarts = [0]; + const scopes = [{ start: -1, end: code.length, parent: -1 }]; + const open = [0]; + for (let i = 0; i < code.length; i++) { + const c = code[i]; + if (c === '\n') lineStarts.push(i + 1); + else if (c === '{') { + scopes.push({ start: i, end: code.length, parent: open[open.length - 1]! }); + open.push(scopes.length - 1); + } else if (c === '}' && open.length > 1) scopes[open.pop()!]!.end = i; + } + const offsets = (re: RegExp): number[] => Array.from(code.matchAll(re), (m) => m.index!); + const arrows: number[] = []; + for (let a = code.indexOf('=>'); a !== -1; a = code.indexOf('=>', a + 2)) arrows.push(a); + file = { code, lineStarts, scopes, varDecls: offsets(VAR_DECL_SITE), fnDecls: offsets(FN_DECL_SITE), arrows, bindings: new Map() }; + } + if (files.size >= JS_SCOPE_FILES_KEEP) files.delete(files.keys().next().value!); + } + files.set(filePath, file); + return file; +} + +/** The innermost scope of `file` that holds `offset`. */ +function jsScopeAt(file: JsScopeFile, offset: number): number { + const { scopes } = file; + let lo = 0, hi = scopes.length; + while (lo + 1 < hi) { + const mid = (lo + hi) >>> 1; + if (scopes[mid]!.start < offset) lo = mid; else hi = mid; + } + while (lo > 0 && scopes[lo]!.end < offset) lo = scopes[lo]!.parent; + return lo; +} + +/** localBindingPatterns, sticky, by name. */ +function stickyBindingPatterns(name: string): LocalBindingPatterns { + let patterns = LOCAL_BINDING_PATTERNS.get(name); + if (!patterns) { + patterns = localBindingPatterns(name, 'y'); + if (LOCAL_BINDING_PATTERNS.size >= LOCAL_BINDING_PATTERNS_CAP) { + LOCAL_BINDING_PATTERNS.delete(LOCAL_BINDING_PATTERNS.keys().next().value!); + } + LOCAL_BINDING_PATTERNS.set(name, patterns); + } + return patterns; +} + +/** Every binding of `name` in the file, read once per file and name. */ +function jsBindingSites(name: string, file: JsScopeFile): JsBindingSite[] { + const known = file.bindings.get(name); + if (known) return known; + const sites: JsBindingSite[] = []; + const { code } = file; + const { decl, fn, param } = stickyBindingPatterns(name); + const blockOf = (at: number) => { + const scope = file.scopes[jsScopeAt(file, at)]!; + return { from: scope.start, to: scope.end }; + }; + let resume = 0; + for (const at of file.varDecls) { + if (at < resume) continue; + decl.lastIndex = at; + const m = decl.exec(code); + if (!m) continue; + resume = at + m[0].length; + if (/\bfor\s*(?:await\s*)?\(\s*$/.test(code.slice(Math.max(0, at - 16), at))) { + // A `for` head's binding holds in the head and the statement after it. + sites.push({ from: code.lastIndexOf('(', at), to: jsForStatementEnd(code, at), at, kind: 'other' }); + continue; + } + // `const x = require('./x')` binds an import, which the file's imports answer for. + const plain = !/^(?:const|let|var)\s+[{[]/.test(m[0]) && !IMPORT_BINDING_VALUE.test(m[1] ?? ''); + sites.push({ ...blockOf(at), at, kind: plain ? 'declaration' : 'other' }); + } + for (const at of file.fnDecls) { + fn.lastIndex = at; + if (fn.test(code)) sites.push({ ...blockOf(at), at, kind: 'function' }); + } + const n = escapeRegExp(name); + const paramName = new RegExp(`(?:^\\(|,)\\s*${PARAM_MODIFIERS}(?:\\.\\.\\.)?(${n})(?![\\w$])`); + let tried = -1; + for (let at = code.indexOf(name); at !== -1; at = code.indexOf(name, at + 1)) { + // A whole name, and no member's (`el.root`), though a rest parameter's (`...root`) is one. + if ((at > 0 && /[\w$]/.test(code[at - 1]!)) || /[\w$]/.test(code[at + name.length] ?? '') || + (code[at - 1] === '.' && code.slice(Math.max(0, at - 3), at) !== '...')) continue; + const open = enclosingOpenParen(code, at); + // A control-flow head, `if (root) {`, binds nothing. + if (open >= 0 && /\b(?:if|while|for|switch|with)\s*$/.test(code.slice(Math.max(0, open - 12), open))) continue; + if (open < 0) { + // A destructured parameter: `({ root }) => …`, `function f([root]) {`. + const body = destructuredParameterBody(code, at); + if (body) sites.push({ ...body, at, kind: 'other' }); + continue; + } + if (open === tried) continue; + tried = open; + param.lastIndex = open; + const m = param.exec(code); + if (!m) continue; + const end = open + m[0].length; + const own = paramName.exec(m[0]); + sites.push({ + ...(code[end - 1] === '{' ? { from: end - 1, to: jsBlockEnd(code, end) } : { from: end, to: jsBodyEnd(code, end) }), + at: own ? open + own.index + own[0].length - name.length : at, + kind: 'parameter', + }); + } + // `name =>`: the name ends where the whitespace before the arrow starts. + for (const arrow of file.arrows) { + let end = arrow; + while (end > 0 && WHITESPACE.test(code[end - 1]!)) end--; + const start = end - name.length; + if (start >= 0 && code.startsWith(name, start) && (start === 0 || !ARROW_HEAD_CHAR.test(code[start - 1]!))) { + sites.push({ from: arrow + 2, to: jsBodyEnd(code, arrow + 2), at: start, kind: 'other' }); + } + } + file.bindings.set(name, sites); + return sites; +} + +/** Where a `for` statement whose head holds `at` ends: past the statement after its head. */ +function jsForStatementEnd(code: string, at: number): number { + let depth = 0; + for (let i = at; i < code.length; i++) { + const c = code[i]; + if (c === '(' || c === '[' || c === '{') depth++; + else if ((c === ')' || c === ']' || c === '}') && --depth < 0) return c === ')' ? jsBodyEnd(code, i + 1) : i; + } + return code.length; +} + +/** + * The body of the function whose parameter list destructures the name at + * `at` (`({ a, b: [root] }) => …`), or null when the name is in no + * parameter list: read back out of the braces and brackets around it to the + * `(` that opens the list, which a `=>` or a body's `{` follows (past a + * return type) and no `if`, `while`, `for`, `switch` or `with` heads. + */ +function destructuredParameterBody(code: string, at: number): { from: number; to: number } | null { + let depth = 0; + let nested = false; + let open = -1; + for (let i = at - 1; i >= 0 && at - i <= 2000; i--) { + const c = code[i]; + if (c === ')' || c === ']' || c === '}') depth++; + else if (c === '(' || c === '[' || c === '{') { + if (depth > 0) { + depth--; + } else if (c === '(') { + open = i; + break; + } else { + nested = true; + } + } else if (c === ';' && depth === 0) { + return null; + } + } + if (!nested || open < 0 || /\b(?:if|while|for|switch|with)\s*$/.test(code.slice(Math.max(0, open - 12), open))) return null; + let close = open + 1; + for (let d = 1; close < code.length && d > 0; close++) { + if (code[close] === '(') d++; + else if (code[close] === ')') d--; + } + const head = /^\s*(?::(?:[^=;{}()\n]|\{(?:[^{}]|\{[^{}]*\})*\})*)?(=>|\{)/.exec(code.slice(close, close + 2000)); + if (!head) return null; + const end = close + head[0].length; + return head[1] === '{' ? { from: end - 1, to: jsBlockEnd(code, end) } : { from: end, to: jsBodyEnd(code, end) }; +} + +/** + * The binding of `name` in scope at `offset` of a JS/TS file: the one whose + * scope holds it innermost, and of two in one scope the declaration nearest + * before it. Null when no binding of the name holds there. + */ +function jsBindingAt(name: string, file: JsScopeFile, offset: number): JsBindingSite | null { + let best: JsBindingSite | null = null; + for (const site of jsBindingSites(name, file)) { + if (site.from > offset || site.to < offset) continue; + if (!best || site.from > best.from || + (site.from === best.from && (site.at <= offset ? best.at > offset || site.at > best.at : best.at > offset && site.at < best.at))) { + best = site; + } + } + return best; +} + +/** + * What a binding of `name` says its value is (JsBindingReading), or null when + * it says nothing: a declaration or a parameter written with a type, or a + * declaration whose whole value is a `new` of a class. + */ +function jsBindingReading(name: string, site: JsBindingSite, file: JsScopeFile): JsBindingReading | null { + if (site.reading === undefined) { + site.reading = site.kind === 'parameter' ? jsParameterReading(name, site, file) + : site.kind === 'declaration' ? jsDeclarationReading(name, site, file) + : null; + } + return site.reading; +} + +/** The 1-based line of an offset in `file`. */ +function jsLineOf(file: JsScopeFile, offset: number): number { + let lo = 0, hi = file.lineStarts.length; + while (lo + 1 < hi) { + const mid = (lo + hi) >>> 1; + if (file.lineStarts[mid]! <= offset) lo = mid; else hi = mid; + } + return lo + 1; +} + +/** The type a parameter is written with (`root: Root`, `svc?: Store`). */ +function jsParameterReading(name: string, site: JsBindingSite, file: JsScopeFile): JsBindingReading | null { + const typed = new RegExp(`${escapeRegExp(name)}(?![\\w$])\\s*\\??\\s*:\\s*${TS_ANNOTATED_TYPE}`, 'y'); + typed.lastIndex = site.at; + const raw = typed.exec(file.code)?.[1]; + const type = raw ? normalizeInferredTypeName(raw) : null; + return raw && type ? { type, raw, line: jsLineOf(file, site.at) } : null; +} + +/** + * What a `const`/`let`/`var` says its value is: the type it is written with, + * or the class its whole value makes with `new` — not a `new` of a + * parameter or variable (`new Backend(root)`), which names no class. + */ +function jsDeclarationReading(name: string, site: JsBindingSite, file: JsScopeFile): JsBindingReading | null { + const { code } = file; + const head = new RegExp(`(?:const|let|var)\\s+${escapeRegExp(name)}(?![\\w$])\\s*(?:!?\\s*:\\s*${TS_ANNOTATED_TYPE})?`, 'y'); + head.lastIndex = site.at; + const m = head.exec(code); + if (!m) return null; + const valueAt = site.at + m[0].length; + const rest = code.slice(valueAt, valueAt + 400); + // An object literal is its own members' (resolveObjectPathMember), whatever + // type it is written with: vite's `const server: ViteDevServer = { … }` is + // the `close` it declares, not the interface's. + if (/^[^=;]*=(?![=>])\s*\{/.test(rest)) return null; + const line = jsLineOf(file, site.at); + if (m[1]) { + const type = normalizeInferredTypeName(m[1]); + return type ? { type, raw: m[1], line } : null; + } + const made = /^\s*=\s*new\s+([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)\s*(?:<[^;(){}]*>\s*)?\(/.exec(rest); + if (!made) return null; + const className = made[1]!.replace(/\s+/g, ''); + const bound = jsBindingAt(className.split('.')[0]!, file, valueAt); + if (bound && bound.kind !== 'function') return null; + // The `new` is the whole value: no member, index, call or operator follows it. + let close = valueAt + made[0].length; + let depth = 1; + for (const limit = close + 5000; close < code.length && close < limit && depth > 0; close++) { + if (code[close] === '(') depth++; + else if (code[close] === ')') depth--; + } + if (depth > 0 || !/^[ \t]*(?:[;,)}]|\r?\n(?![ \t]*(?:[.(\[?:]|\?\?|\|\||&&|as\b))|$)/.test(code.slice(close, close + 200))) return null; + const type = normalizeInferredTypeName(className); + return type ? { type, raw: className, line } : null; +} + +/** + * Per context and file: how many functions and methods hold each line, and + * the names the file's module scope gives a variable or constant node. + */ +const JS_MODULE_VALUES = new WeakMap }>>(); + +/** + * Whether the binding a call's receiver has may lie outside the function + * making the call, where the calling function's own declarations, which the + * receiver reader has read already, can't say what it is: in a function + * around that one, or at module scope, which declares the name. Code outside + * any function was read from the file's start. + */ +function jsMayBindOutside(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + let files = JS_MODULE_VALUES.get(context); + if (!files) JS_MODULE_VALUES.set(context, (files = new Map())); + let scope = files.get(ref.filePath); + if (!scope) { + const nodes = context.getNodesInFile(ref.filePath); + const functions = nodes.filter((n) => n.kind === 'function' || n.kind === 'method'); + const depth = new Int32Array(functions.reduce((last, n) => Math.max(last, n.endLine), 0) + 2); + for (const n of functions) { + depth[n.startLine]!++; + depth[n.endLine + 1]!--; + } + for (let line = 1; line < depth.length; line++) depth[line]! += depth[line - 1]!; + const names = new Set(nodes.filter((n) => (n.kind === 'variable' || n.kind === 'constant') && n.qualifiedName === n.name).map((n) => n.name)); + if (files.size >= 64) files.delete(files.keys().next().value!); + files.set(ref.filePath, (scope = { depth, names })); + } + const around = scope.depth[ref.line] ?? 0; + return around > 1 || (around === 1 && scope.names.has(name)); +} + +/** + * A JS/TS receiver the calling function doesn't type, read by its binding in + * scope at the call (#2589): a module's `let root: Root | null`, a `let + * popupRoot: Root` in the function around the one calling, or an outer + * function's typed parameter. Null when the binding says nothing of its value. + */ +function jsReceiverBindingReading(receiver: string, ref: UnresolvedRef, context: ResolutionContext): JsBindingReading | null { + if (!jsMayBindOutside(receiver, ref, context)) return null; + const file = jsScopeFile(ref.filePath, context); + if (!file) return null; + const site = jsBindingAt(receiver, file, (file.lineStarts[ref.line - 1] ?? file.code.length) + ref.column); + return site ? jsBindingReading(receiver, site, file) : null; +} + /** * Patterns that recover a PHP class property's declared type for a * `$this->prop` receiver. Deliberately NOT localReceiverTypePatterns: only @@ -19192,6 +19561,22 @@ export function matchMethodCall( if (!awaited.name || TS_PRIMITIVE_TYPES.has(awaited.name)) return null; inferredType = awaited.name; } + // A TS/JS receiver the calling function doesn't type is read by its + // binding in scope at the call (#2589): a module's `let root: Root | + // null`, a `let popupRoot: Root` in the function around the one calling, + // or an outer function's typed parameter types it, as the calling + // function's own declaration would. + if (!inferredType && !awaited && dotMatch && /^[A-Za-z_$][\w$]*$/.test(objectOrClass!) && + (ref.language === 'typescript' || ref.language === 'tsx' || ref.language === 'javascript' || ref.language === 'jsx') && + objectOrClass !== 'this' && objectOrClass !== 'super' && !isImportBinding(objectOrClass!, ref, context) && + isLocallyBoundJsName(objectOrClass!, ref.filePath, context)) { + const reading = nmTimedT('mc-jsbinding', ref, () => jsReceiverBindingReading(objectOrClass!, ref, context)); + if (reading) { + inferredType = reading.type; + decl.raw = reading.raw; + decl.line = reading.line; + } + } // A TS annotation with brackets after the type. An array of it (`items: // Item[]`): a method arrays have is the array's, which no project class // declares, where `distinctElements.find(…)` on vscode's `ExplorerItem[]` @@ -19731,6 +20116,14 @@ export function matchMethodCall( narrowed ||= kept.length !== targetMethods.length; targetMethods = kept; } + // A React class component's `render()` and lifecycle methods are React's + // to call (#2589): react-dom's `root.render()` is no error + // boundary's `render`, whatever holds the root. + if (JS_FAMILY.has(ref.language) && objectOrClass !== 'this' && objectOrClass !== 'super') { + const kept = targetMethods.filter((m) => !isReactLifecycleMethod(m, context)); + narrowed ||= kept.length !== targetMethods.length; + targetMethods = kept; + } // A python attribute chain nothing typed (matchPythonAttrCall) is guessed // only by the name of its last link, in the scoring below — Django's @@ -19871,6 +20264,36 @@ function isImportedModuleReceiver(receiver: string, ref: UnresolvedRef, context: return binding.isNamespace || context.isOutOfRepoImport?.(binding.source, ref.filePath, ref.language) === true; } +/** The methods React calls on a class component itself. */ +const REACT_LIFECYCLE_METHODS: ReadonlySet = new Set([ + 'render', 'componentDidMount', 'componentDidUpdate', 'componentWillUnmount', 'shouldComponentUpdate', + 'getSnapshotBeforeUpdate', 'componentDidCatch', 'componentWillMount', 'componentWillReceiveProps', + 'componentWillUpdate', 'UNSAFE_componentWillMount', 'UNSAFE_componentWillReceiveProps', 'UNSAFE_componentWillUpdate', +]); +const REACT_LIFECYCLE_MEMO = new WeakMap>(); + +/** + * A lifecycle method (`render`, `componentDidMount`, …) of a class that + * extends React's `Component` or `PureComponent` (or Preact's), read from the + * class's head. + */ +function isReactLifecycleMethod(n: Node, context: ResolutionContext): boolean { + if (n.kind !== 'method' || !REACT_LIFECYCLE_METHODS.has(n.name)) return false; + let memo = REACT_LIFECYCLE_MEMO.get(context); + if (!memo) REACT_LIFECYCLE_MEMO.set(context, (memo = new Map())); + let hit = memo.get(n.id); + if (hit === undefined) { + const ownerName = n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))).split('::').pop(); + const owner = ownerName ? context.getNodesInFile(n.filePath).find((c) => + c.kind === 'class' && c.name === ownerName && c.startLine <= n.startLine && c.endLine >= n.endLine) : undefined; + const lines = owner ? context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? [] : []; + const head = owner ? lines.slice(owner.startLine - 1, Math.min(owner.startLine + 3, n.startLine)).join(' ').split('{')[0]! : ''; + hit = /\bextends\s+(?:React\s*\.\s*)?(?:Pure)?Component\b/.test(head); + memo.set(n.id, hit); + } + return hit; +} + /** A method a Vue Options API component declares for itself (`index::handleLogin` in `index.vue`). */ function isVueComponentMethod(n: Node): boolean { return n.kind === 'method' && n.filePath.endsWith('.vue') && n.qualifiedName === `${path.posix.basename(n.filePath, '.vue')}::${n.name}`; From efb091ec2f1c3c82caf5d36c48c8673e0b284a88 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 19:43:54 +0000 Subject: [PATCH 44/95] fix(liquid): editor-saved Shopify JSON and {% sections %} groups link their sections (#2812) Shopify's theme editor writes a block comment ("IMPORTANT: The contents of this file are auto-generated...") at the top of every JSON template and section group it saves. The Liquid extractor read theme JSON with JSON.parse, which rejects comments and trailing commas, so such a file linked none of the sections it lists, and in an editor-saved theme most sections looked unused. All 14 of Shopify/horizon's templates and section groups carry the comment, so none of them linked. The links a JSON file did produce all sat at line 1, column 0. The same report found two more gaps. A layout's {% sections 'header-group' %} tag, which renders the section group sections/header-group.json, was never read. And a {% schema %} written inside {% comment %} or {% raw %} was indexed as the section's own schema, because only the tag scanner skipped those regions. Fix: - Read theme JSON with jsonc-parser's parseTree, which accepts comments, and allow trailing commas. Shopify's theme docs say theme JSON supports both, and Shopify's theme-check parses it with the same options. Any other parse error still links nothing. - Put each section reference at its "type" value's line and column. - Read the plural sections tag wherever the singular one is read: braced, whitespace-trimmed, or as a line of a {% liquid %} tag. It links to sections/.json, and its component is named ::sections:, apart from a section of the same name. The Shopify theme matcher already keeps any sections/ path inside the file's own theme, so a group resolves only there (its doc comment now says so). - Compute the comment and raw regions once per file and share them between the tag scanner and the schema extractor. A schema inside one is skipped, and the search resumes after the region. Validation: - 10 new regression tests in __tests__/extraction.test.ts fail with main's extractor and pass with the fix. Re-run on 08478eea: 30 passed and 10 failed with main's extractor, 40 passed with the fix. 3 strictness guards (a missing brace, an unclosed comment, a missing comma) pass on both. - A/B on base 80c36800, each arm built from its own sources with the kernel: 8 theme repos (Shopify/dawn, Shopify/horizon, Shopify/skeleton-theme, ChumsInc/chums-impulse-dev, agencecinq/cinquante-et-un, noclocks/shopify-devstore-theme, panoply/syncify, Shopify/theme-tools) and 2 controls (express, requests). The controls and skeleton-theme are byte-identical, and no node or edge is removed anywhere. All 299 changed reference edges were checked against the source text: 299 right, 0 wrong. 197 are new links (horizon 25, chums-impulse-dev 138, cinquante-et-un 11, shopify-devstore-theme 16, dawn 2, syncify 2, theme-tools 3), 14 of them from sections tags, and 102 moved from 1:0 to their "type" value. 22 sections tags add 44 import and component nodes. The only new failed refs are a "multirow" section that cinquante-et-un's theme does not have, and 8 sections tags in theme-tools' prettier-plugin test fixtures that name a group no file provides. - Cost: an extractor microbenchmark over 1,386 Liquid files (8 MB), median of 15 runs in 3 interleaved rounds, gave 436 ms on main and 439 ms with the fix. Index wall time, median of 3 interleaved runs, main vs fix: horizon 2891 vs 3064 ms, chums-impulse-dev 2429 vs 2368, syncify 3846 vs 3810, theme-tools 6698 vs 6620. The byte-identical controls moved as much (express 2312 vs 2445, requests 1946 vs 2080), so that is noise on a shared machine. - Rebased onto 08478eea, which adds only Pascal (PR #2810) and JS/TS resolution (PR #2811) changes. There the Liquid tests pass, and re-indexing the 8 theme repos gives graphs byte-identical to the pre-rebase fix arm, except for theme-tools' TypeScript calls, which PR #2811 changes. - Kernel parity: not applicable, the kernel does not extract Liquid. Fixes #2494 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/extraction.test.ts | 125 +++++++++++++++++++++++++++++ src/extraction/liquid-extractor.ts | 110 ++++++++++++++++++------- src/resolution/shopify-themes.ts | 18 +++-- 4 files changed, 215 insertions(+), 39 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94b262e8ed..942a9ae633 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -118,6 +118,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - JSON files in a `templates` or `sections` folder are now read as Shopify Liquid only inside a Shopify theme, where the folder that contains `templates` or `sections` also has `layout/theme.liquid` or `config/settings_schema.json`. Before, all such JSON was, so a .NET project template's `template.json`, a schematic or a CMS's content showed up in the index as an empty Liquid file, and one that lists `sections` could be linked to an unrelated Liquid file of the same name. Shopify themes, including one kept in a subfolder of a repository, are indexed as before, and the next sync removes JSON that an earlier version indexed this way. - In a repository that holds more than one Shopify theme, a theme's `{% render %}`, `{% include %}` and `{% section %}` tags and the sections its JSON templates list now link only to that theme's own snippets and sections, which is where Shopify looks for them. Before, a section or snippet the theme doesn't have was linked to another theme's file of the same name, and a theme kept in a subfolder could link to the root theme's copy instead of its own. A reference the theme can't satisfy now links to nothing. Projects with a single theme are indexed as before. Re-index projects with several Shopify themes after upgrading. - `codegraph sync` and the file watcher now link a Shopify section or snippet to the files that name it when it is added after them, or comes back after being deleted, whether they name it in a `{% render %}`, `{% include %}` or `{% section %}` tag or as a section `type` in a JSON template. Before, the link appeared only once a file naming it changed or the project was re-indexed, even in a repository with a single theme, so the new section or snippet looked unused and `codegraph affected` missed the files that use it. Re-index Shopify themes after upgrading to pick up links an earlier version missed. +- A Shopify JSON template or section group saved by the theme editor now links the sections it lists. The editor writes a comment at the top of every file it saves, and a file with that comment, or with a trailing comma, both of which Shopify accepts, linked none of its sections, so in an editor-saved theme most sections looked unused. A layout's `{% sections 'header-group' %}` tag now links to the section group it renders, each section a JSON file lists is linked from the line that names it rather than from the top of the file, and a `{% schema %}` written inside `{% comment %}` or `{% raw %}` is no longer indexed as the section's own. Re-index Shopify themes after upgrading. Thanks @Puma7 for the report and the pull request. (#2494) - `codegraph sync` and the file watcher now link a route to the page or layout it loads lazily when that file is added after the router was indexed, comes back after being deleted, or gets the component the route renders in a later edit, whether or not the import names the file's extension: React Router's `lazy: () => import('./pages/Team')`, Vue Router's `component: () => import('@/views/Login')` and Angular's `loadComponent: () => import('./home/home.component')`. Before, the route stayed unlinked until the router file changed or the project was re-indexed, so the page looked unused and `codegraph affected` missed the tests that reach it through the router. Re-index after upgrading to pick up links an earlier version missed. - A file named the way Google names its tests, like `wire_format_unittest.cc` in protocolbuffers/protobuf, `logging_unittest.cc` in google/glog or a Python `foo_unittest.py`, now counts as a test, as `wire_format_test.cc` already did. Before, it was taken for production code: search and `codegraph_explore` ranked it alongside the code it tests, `codegraph affected` never listed it, and calls in production code could link to a class or type a unittest declares for itself, as google/breakpad's `stack_frame_entries_.size()` did to a unittest's `StackHelper`. Re-index projects that have such files after upgrading. - A file in a `unittests/` folder, where llvm/llvm-project and google/breakpad keep their unit tests, or in a folder named like a unittest, such as `dcheck_unittest/` in google/glog, now counts as a test whatever its own name says, like LLVM's `CountCopyAndMove.h` helper. Before, such a file was taken for production code unless its name said test, and none of them were kept apart from the program the way a `tests/` folder is: search and `codegraph_explore` ranked them alongside production code, `codegraph affected` missed them, and production code linked to a unit test's own declarations, as LLVM's `Ctx` and `OpInfo` variables did to ones its unit tests declare. A folder named just `unittest/`, like CPython's `Lib/unittest/`, is still production code. Re-index projects that have such folders after upgrading. diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts index ed460dd4c8..eefa535c90 100644 --- a/__tests__/extraction.test.ts +++ b/__tests__/extraction.test.ts @@ -5417,6 +5417,40 @@ import 'package:flutter/material.dart'; .toEqual([['snippets/live.liquid', 18], ['snippets/after.liquid', 20]]); }); + /* A layout renders a section group, the JSON file `sections/.json`, + with the plural tag; only the singular one was read (#2494). */ + it.each([ + "{% sections 'header-group' %}", + '{%- sections "header-group" -%}', + "{% liquid\n sections 'header-group'\n%}", + ])('reads a section group from %j', (code) => { + const result = extractFromSource('layout/theme.liquid', code); + expect(result.nodes.filter((n) => n.kind === 'import').map((n) => n.name)).toEqual(['header-group']); + expect(result.unresolvedReferences.map((r) => r.referenceName)).toEqual(['sections/header-group.json']); + }); + + it('keeps a section and a section group of the same name apart', () => { + const result = extractFromSource('layout/theme.liquid', "{% section 'header' %}{% sections 'header' %}"); + expect(result.unresolvedReferences.map((r) => [r.referenceName, r.line, r.column])) + .toEqual([['sections/header.liquid', 1, 0], ['sections/header.json', 1, 22]]); + expect(result.nodes.filter((n) => n.kind === 'component').map((n) => n.qualifiedName)) + .toEqual(['layout/theme.liquid::section:header', 'layout/theme.liquid::sections:header']); + expect(new Set(result.nodes.map((n) => n.id)).size).toBe(result.nodes.length); + }); + + it.each(['comment', 'raw'])('indexes no schema written inside %s', (tag) => { + // An example schema in a comment is not the section's, and a lone + // opening tag in one must not swallow the real schema after it (#2494). + const code = [ + `{% ${tag} %}`, '{% schema %}', '{"name": "Ghost"}', '{% endschema %}', `{% end${tag} %}`, + `{%- ${tag} -%}{% schema %}{%- end${tag} -%}`, + '{% schema %}', '{"name": "Live"}', '{% endschema %}', + ].join('\n'); + const result = extractFromSource('sections/header.liquid', code); + expect(result.nodes.filter((n) => n.kind === 'constant').map((n) => [n.name, n.startLine])) + .toEqual([['Live', 7]]); + }); + it('should extract multiple imports', () => { const code = ` {% section 'header' %} @@ -7751,6 +7785,97 @@ describe('Liquid Shopify JSON template section resolution', () => { expect(cg.getFileDependencies('sections/header.liquid')).toEqual(['snippets/price.liquid']); }, 60_000); + // The notice Shopify's theme editor writes at the top of every JSON template + // and section group it saves. + const editorNotice = [ + '/*', + ' * ------------------------------------------------------------', + ' * IMPORTANT: The contents of this file are auto-generated.', + ' * ------------------------------------------------------------', + ' */', + ]; + const referencesFrom = (file: string): Array<[string | undefined, number | undefined, number | undefined]> => { + const fileNode = cg.getNodesInFile(file).find((n) => n.kind === 'file')!; + return cg.getOutgoingEdges(fileNode.id) + .filter((e) => e.kind === 'references') + .map((e): [string | undefined, number | undefined, number | undefined] => [cg.getNode(e.target)?.filePath, e.line, e.column]) + .sort((a, b) => a[1]! - b[1]!); + }; + + it.each(['\n', '\r\n'])('links the sections an editor-saved template and section group list, where each `type` is (%j)', async (newline) => { + // JSON.parse rejects the editor's comment, so such a file linked none of + // its sections, and a file without one linked them at line 1 (#2494). + writeFiles({ + 'layout/theme.liquid': `{% sections 'header-group' %}{{ content_for_layout }}\n`, + 'sections/header-group.json': [ + ...editorNotice, '{', ' "type": "header",', ' "name": "Header",', ' "sections": {', + ' "bar": { "type": "announcement-bar" },', ' "header": {', ' "type": "header"', ' }', + ' },', ' "order": ["bar", "header"]', '}', + ].join(newline), + 'sections/announcement-bar.liquid': `
\n`, + 'sections/header.liquid': `
\n`, + 'sections/main-product.liquid': `
{{ product.title }}
\n`, + 'templates/product.json': [ + ...editorNotice, '{', ' "sections": {', ' "main": {', ' "type": "main-product",', + ' "settings": { "note": "https://example.com/*not-a-comment*/ // nor this" }', ' },', + ' "again": { "type": "main-product" }', ' },', ' "order": ["main", "again"]', '}', + ].join(newline), + 'templates/index.json': `{\n "sections": {\n "main": { "type": "header" }\n },\n "order": ["main"]\n}\n`, + }); + cg = CodeGraph.initSync(tempDir); + await cg.indexAll(); + cg.resolveReferences(); + + expect(referencesFrom('templates/product.json')).toEqual([['sections/main-product.liquid', 9, 14]]); + expect(referencesFrom('sections/header-group.json')).toEqual([ + ['sections/announcement-bar.liquid', 10, 21], + ['sections/header.liquid', 12, 14], + ]); + expect(referencesFrom('templates/index.json')).toEqual([['sections/header.liquid', 3, 22]]); + expect(referencesFrom('layout/theme.liquid')).toEqual([['sections/header-group.json', 1, 0]]); + }); + + it('reads a JSON template with trailing commas, which Shopify accepts in theme JSON', () => { + const text = '// a note\n{"sections": {"main": {"type": "main-product",}, "hero": {"type": "hero"},}, "order": ["main", "hero",],}'; + expect(extractFromSource('templates/product.json', text, 'liquid').unresolvedReferences.map((r) => r.referenceName)) + .toEqual(['sections/main-product.liquid', 'sections/hero.liquid']); + }); + + it.each([ + ['a missing brace', '{"sections": {"main": {"type": "main-product"}}'], + ['an unclosed comment', '/* saved by the editor {"sections": {"main": {"type": "main-product"}}}'], + ['a missing comma', '{"sections": {"main": {"type": "main-product"} "hero": {"type": "hero"}}}'], + ])('links nothing from a JSON template with %s', (_case, text) => { + expect(extractFromSource('templates/product.json', text, 'liquid').unresolvedReferences).toEqual([]); + }); + + it("links a layout's section groups inside its own theme, and on to the sections they list", async () => { + // `{% sections 'header-group' %}` renders the theme's + // sections/header-group.json, whose sections render snippets (#2494). + writeFiles({ + ...severalThemes, + 'layout/theme.liquid': `{% sections 'header-group' %}{{ content_for_layout }}\n`, + 'sections/header-group.json': [...editorNotice, jsonTemplate('header')].join('\n'), + // This theme has no header group of its own, and the root theme's is not its. + 'examples/tailwind/layout/theme.liquid': `{% liquid\n sections 'header-group'\n%}{{ content_for_layout }}\n`, + }); + cg = CodeGraph.initSync(tempDir); + await cg.indexAll(); + cg.resolveReferences(); + + const linksOf = (file: string): string[] => cg.getFileDependencies(file).sort(); + expect(linksOf('layout/theme.liquid')).toEqual(['sections/header-group.json']); + expect(linksOf('sections/header-group.json')).toEqual(['sections/header.liquid']); + expect(linksOf('sections/header.liquid')).toEqual(['snippets/price.liquid']); + expect(linksOf('examples/tailwind/layout/theme.liquid')).toEqual([]); + + // Once the theme has a header group of its own, sync links the layout to it. + writeFiles({ 'examples/tailwind/sections/header-group.json': [...editorNotice, jsonTemplate('header')].join('\n') }); + await cg.sync(); + expect(linksOf('examples/tailwind/layout/theme.liquid')).toEqual(['examples/tailwind/sections/header-group.json']); + expect(linksOf('examples/tailwind/sections/header-group.json')).toEqual(['examples/tailwind/sections/header.liquid']); + }, 60_000); + it('finds the theme a file is in from the theme folder it sits in', () => { const markers = new Set(['layout/theme.liquid', 'examples/tailwind/config/settings_schema.json']); const exists = (relativePath: string): boolean => markers.has(relativePath); diff --git a/src/extraction/liquid-extractor.ts b/src/extraction/liquid-extractor.ts index f6d9aee60a..15850ff821 100644 --- a/src/extraction/liquid-extractor.ts +++ b/src/extraction/liquid-extractor.ts @@ -1,3 +1,4 @@ +import { parseTree, findNodeAtLocation, type ParseError } from 'jsonc-parser'; import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference } from '../types'; import { generateNodeId, NodeIdAllocator } from './tree-sitter-helpers'; @@ -6,9 +7,11 @@ import { generateNodeId, NodeIdAllocator } from './tree-sitter-helpers'; * * Liquid is a templating language (used by Shopify, Jekyll, etc.) that doesn't * have traditional functions or classes. Instead, we extract: - * - Section references ({% section 'name' %}) + * - Section references ({% section 'name' %}) and section group references + * ({% sections 'header-group' %} → `sections/header-group.json`) * - Snippet references ({% render 'name' %} and {% include 'name' %}) * - Schema blocks ({% schema %}...{% endschema %}) + * - The sections a Shopify JSON template or section group lists */ export class LiquidExtractor { private filePath: string; @@ -18,6 +21,8 @@ export class LiquidExtractor { private edges: Edge[] = []; private unresolvedReferences: UnresolvedReference[] = []; private errors: ExtractionError[] = []; + /** Where Liquid runs nothing — see `silentRegions`. Computed once per file. */ + private silent?: Array<[start: number, end: number]>; constructor(filePath: string, source: string) { this.filePath = filePath; @@ -100,31 +105,68 @@ export class LiquidExtractor { * mapping an id → `{ "type": "", ... }`; the `type` names a * `sections/.liquid` file. Emit a `references` edge to each, so a section * used only from a JSON template (the OS 2.0 norm) is no longer orphaned. + * + * The theme editor writes a block comment ("IMPORTANT: The contents of this + * file are auto-generated…") at the top of every template and section group + * it saves, and Shopify's theme JSON allows comments and trailing commas. + * Anything else that is not JSON, like a file cut off partway, links nothing. + * Each reference sits where its `type` value is written. */ private extractShopifyJsonSections(fromNodeId: string): void { - let parsed: unknown; - try { - parsed = JSON.parse(this.source); - } catch { - return; // not valid JSON (or a partial) — nothing to link - } - const sections = (parsed as { sections?: Record })?.sections; - if (!sections || typeof sections !== 'object') return; + const errors: ParseError[] = []; + const root = parseTree(this.source, errors, { allowTrailingComma: true }); + if (errors.length > 0 || root?.type !== 'object') return; + const sections = findNodeAtLocation(root, ['sections']); + if (sections?.type !== 'object') return; const seen = new Set(); - for (const key of Object.keys(sections)) { - const type = sections[key]?.type; - if (typeof type !== 'string' || seen.has(type)) continue; - seen.add(type); + for (const section of sections.children ?? []) { + const value = section.children?.[1]; + const type = value?.type === 'object' ? findNodeAtLocation(value, ['type']) : undefined; + if (type?.type !== 'string' || typeof type.value !== 'string' || seen.has(type.value)) continue; + seen.add(type.value); + const line = this.getLineNumber(type.offset); this.unresolvedReferences.push({ fromNodeId, - referenceName: `sections/${type}.liquid`, + referenceName: `sections/${type.value}.liquid`, referenceKind: 'references', - line: 1, - column: 0, + line, + column: type.offset - this.getLineStart(line), }); } } + /** + * The regions Liquid runs nothing in: `{% comment %}…{% endcomment %}` and + * `{% raw %}…{% endraw %}`, each from its opening tag to the end of its + * closing one (or to the end of the file, when it is never closed). The tags, + * schemas and declarations written inside one are examples, not code. + */ + private silentRegions(): Array<[start: number, end: number]> { + if (this.silent) return this.silent; + const regions: Array<[number, number]> = []; + const tags = /\{%[-]?\s*(\w+|#)([\s\S]*?)[-]?%\}/g; + let open: { name: string; start: number } | undefined; + let tag; + while ((tag = tags.exec(this.source)) !== null) { + const name = tag[1]!; + if (open) { + if (name === `end${open.name}`) { + regions.push([open.start, tags.lastIndex]); + open = undefined; + } + } else if (name === 'comment' || name === 'raw') { + open = { name, start: tag.index }; + } + } + if (open) regions.push([open.start, this.source.length]); + return (this.silent = regions); + } + + /** The silent region (see `silentRegions`) holding `index`, if any. */ + private silentRegionAt(index: number): [start: number, end: number] | undefined { + return this.silentRegions().find(([start, end]) => index >= start && index < end); + } + /** * Every occurrence of a Liquid tag, in BOTH spellings it can be written in. * @@ -152,18 +194,14 @@ export class LiquidExtractor { const blocks: Array<{ bodyStart: number; body: string }> = []; const tags = /\{%[-]?\s*(\w+|#)([\s\S]*?)[-]?%\}/g; const braced = new RegExp(`^\\{%[-]?\\s*(${tagPattern})\\s+${argPattern}`); - let suppressed: string | undefined; let tag; while ((tag = tags.exec(this.source)) !== null) { - const name = tag[1]!; - if (suppressed) { - if (name === `end${suppressed}`) suppressed = undefined; - continue; - } - if (name === 'comment' || name === 'raw') { - suppressed = name; + const silent = this.silentRegionAt(tag.index); + if (silent) { + tags.lastIndex = silent[1]; continue; } + const name = tag[1]!; if (name === 'liquid') { blocks.push({ bodyStart: tag.index + tag[0].indexOf(name) + name.length, @@ -280,13 +318,16 @@ export class LiquidExtractor { } /** - * Extract {% section 'name' %} references + * Extract {% section 'name' %} and {% sections 'group' %} references. A + * section is `sections/.liquid`; a section group, which a layout renders + * with the plural tag, is the JSON file `sections/.json` that lists its + * sections. */ private extractSectionReferences(fileNodeId: string): void { // Both spellings, as for render/include above. - for (const match of this.findTagOccurrences('section', `['"]([^'"]+)['"]`)) { + for (const match of this.findTagOccurrences('sections?', `['"]([^'"]+)['"]`)) { const fullMatch = match.fullMatch; - const sectionName = match.groups[1]; + const [tagType, sectionName] = match.groups; const line = this.getLineNumber(match.index); // Create an import node for searchability @@ -315,13 +356,13 @@ export class LiquidExtractor { }); // Create a component node for the section reference - const nodeId = this.nodeIds.generate(this.filePath, 'component', `section:${sectionName}`, line, match.index - this.getLineStart(line)); + const nodeId = this.nodeIds.generate(this.filePath, 'component', `${tagType}:${sectionName}`, line, match.index - this.getLineStart(line)); const node: Node = { id: nodeId, kind: 'component', name: sectionName!, - qualifiedName: `${this.filePath}::section:${sectionName}`, + qualifiedName: `${this.filePath}::${tagType}:${sectionName}`, filePath: this.filePath, language: 'liquid', startLine: line, @@ -340,10 +381,10 @@ export class LiquidExtractor { kind: 'contains', }); - // Add unresolved reference to the section file + // Add unresolved reference to the section file, or the group's JSON this.unresolvedReferences.push({ fromNodeId: fileNodeId, - referenceName: `sections/${sectionName}.liquid`, + referenceName: `sections/${sectionName}.${tagType === 'sections' ? 'json' : 'liquid'}`, referenceKind: 'references', line, column: match.index - this.getLineStart(line), @@ -360,6 +401,13 @@ export class LiquidExtractor { let match; while ((match = schemaRegex.exec(this.source)) !== null) { + // A schema inside {% comment %} or {% raw %} is an example, not the + // section's: look again after that region ends. + const silent = this.silentRegionAt(match.index); + if (silent) { + schemaRegex.lastIndex = silent[1]; + continue; + } const [fullMatch, schemaContent] = match; const startLine = this.getLineNumber(match.index); const endLine = this.getLineNumber(match.index + fullMatch.length); diff --git a/src/resolution/shopify-themes.ts b/src/resolution/shopify-themes.ts index 3c2f3c8b5c..99076599f8 100644 --- a/src/resolution/shopify-themes.ts +++ b/src/resolution/shopify-themes.ts @@ -2,13 +2,14 @@ * A Shopify theme's section and snippet references → that theme's own files. * * `{% render 'price' %}` renders the `snippets/price.liquid` of the theme the - * file belongs to; `{% section 'header' %}` and a JSON template's section - * `"type"` name a file in that theme's `sections/`. Shopify never looks in - * another theme. The Liquid extractor records these as `snippets/.liquid` - * and `sections/.liquid`, and the path matcher took any file whose path - * ends with that, so in a repository holding several themes (a theme tool's - * examples, a store's seasonal themes) a theme missing a section linked to - * another theme's copy of it. + * file belongs to; `{% section 'header' %}`, a JSON template's section + * `"type"` and `{% sections 'header-group' %}` name a file in that theme's + * `sections/`. Shopify never looks in another theme. The Liquid extractor + * records these as `snippets/.liquid`, `sections/.liquid` and, for + * a section group, `sections/.json`, and the path matcher took any file + * whose path ends with that, so in a repository holding several themes (a + * theme tool's examples, a store's seasonal themes) a theme missing a section + * linked to another theme's copy of it. * * From a file inside a theme (see `shopifyThemeRoot`), these references * resolve to the theme's own file or to nothing, and never fall through to the @@ -20,7 +21,8 @@ import type { ResolvedRef, ResolutionContext, UnresolvedRef } from './types'; /** * The file a section or snippet reference made inside a Shopify theme names: - * the theme's own `sections/.liquid` or `snippets/.liquid`. + * the theme's own `sections/.liquid`, `sections/.json` or + * `snippets/.liquid`. * undefined means the reference is not one, or is made outside any theme; * null means the theme has no such file, so no other strategy may guess one. */ From 68fcf2f1c7021d809ebba4d08286c16320cd9148 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 20:08:41 +0000 Subject: [PATCH 45/95] fix(js,ts): static and instance methods override only their own kind (#2813) The override-dispatch pass (interface-impl) linked a base class's method to every subclass method of the same name, so a static method counted as overriding an instance one, or the reverse, though a call to one never runs the other (issue #2761). TypeScript nodes recorded static already, so there the pass alone was wrong. JavaScript nodes never did: the extractor had no isStatic hook and the kernel answered nothing for the JS variants, so a JS class's static s() and s() were both stored as instance methods. JavaScript members now record their own static keyword with the hook TypeScript uses (hasStaticModifier, which also keeps a method named static() an instance one), and the kernel gives the JS variants the same answer, so both engines agree. The pass skips a JS/TS base and override whose static flags differ; a static method still overrides a static one, and an instance method an instance one. Validation: the new test fails 5 of 10 on main 08478eea (JS and JSX static flags in both engines, and the override pairs) and passes 10 of 10 on the fix. A/B against main on 20 repos: webpack's lib loses 15 edges, every one a cross-kind pair, such as Module's instance deserialize(context) to the static deserialize(context) factories of NormalModule and six other modules, or Module's static getSourceBasicTypes(module) to RuntimeModule's instance getSourceBasicTypes(); the same-kind pairs stay and no edge is added. No other repo's edges change. Nodes change only in is_static, 0 to 1, on 315 JavaScript static members, none in TypeScript or another language; the flask and gin control dumps are byte-identical. Kernel parity over the 20 TS/JS repos: 16,612 of 16,873 files byte-parity, none with diffs, the rest deferred to wasm as before. Index time, median of 3 interleaved runs on vite, nuxt, webdriverio, insomnia and webpack: -1.1% total, within noise. Fixes #2761 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/js-static-override-dispatch.test.ts | 179 ++++++++++++++++++ codegraph-kernel/src/tsjs/mod.rs | 11 +- src/extraction/languages/javascript.ts | 5 +- src/extraction/languages/typescript.ts | 20 +- src/resolution/callback-synthesizer.ts | 12 ++ 6 files changed, 213 insertions(+), 15 deletions(-) create mode 100644 __tests__/js-static-override-dispatch.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 942a9ae633..4ec3ae232c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -194,6 +194,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a type defined over a slice, map, function or basic type now counts as implementing the interfaces its methods satisfy, just as a struct does: prometheus's `type staticDiscoverer []*targetgroup.Group` is a `Discoverer`, gin's `type formSource map[string][]string` a `setter`, etcd's `type Int64Comparable int64` a `Comparable`, and an adapter like `type HandlerFunc func(…)` a `Handler`. Before, only structs were considered, so these types were missing from an interface's implementations in `codegraph_explore`, impact and the type hierarchy, and a call through the interface, like `Discoverer.Run` or `setter.TrySet`, stopped there instead of reaching their methods. Re-index Go projects after upgrading. - In Go, a type alias like `type Event = mvccpb.Event`, alone or inside a `type ( … )` group, is now indexed and links to the type it names, so impact on that type reaches the code that uses the alias. Before, aliases were missing: a use written through the alias's package, like `clientv3.Event` or prometheus's `storage.AOptions{…}`, linked to nothing, and one inside the package, like `&PrometheusVersion{}` in prometheus's `web` package, linked to whichever type of that name the project had, there a test helper's struct. A method called on a value of an alias's type now reaches the aliased type's method rather than a namesake's, and a generic alias like `type Items[T any] = List[T]` links to `List`, so `xs.Len()` on one reaches `List.Len` instead of nothing. Re-index Go projects after upgrading. - In Go, a struct that embeds another is no longer treated as overriding the methods it redeclares. gin's `Engine` embeds `RouterGroup` and has a `Use` of its own, but calling `Use` on a `RouterGroup` only ever runs `RouterGroup.Use`; before, `codegraph_explore`'s call path and impact went on from it into `Engine.Use` and the code only the engine runs. The same held for every struct or defined type embedded in another that redeclares one of its methods, like etcd's `Mutex` inside `lockerMutex`. A call through an interface still reaches each type that implements it, the embedding struct included. Re-index Go projects after upgrading. +- In JavaScript and TypeScript, a subclass's `static` method no longer counts as overriding an instance method of the same name in its base class, or the reverse, since calling one never runs the other. In webpack, `codegraph_explore`'s call path and impact went on from `Module`'s `deserialize(context)` into the `static deserialize(context)` of `NormalModule` and other modules. A static method still overrides its base's static method and an instance method its base's instance method, and JavaScript class members are now marked static, as TypeScript ones already were. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and @yang0228 for the pull request. (#2761) - In Go, a call through a struct field that holds a function, like prometheus's `d.refreshf(ctx)`, now reaches the functions put in that field, so `codegraph_explore` follows service discovery from `Discovery.Run` into each provider's `refresh` and names the line where each provider wires its method in, like `RefreshF: d.refresh`. The field can be set in a struct literal or by an assignment, copied from another struct's field (`refreshf: opts.RefreshF`), or given a method of an interface value, and two unrelated structs with a field of the same name, like `fn` or `handler`, stay apart. Before, such a flow stopped at the field. Re-index Go projects after upgrading. - In Go, a type defined from other types, like etcd's `type WatchChan <-chan WatchResponse` or gin's `type HandlerFunc func(*Context)` and `type HandlersChain []HandlerFunc`, now links to the types it is built from, so impact on `WatchResponse` or gin's `Context` reaches these types and the code that uses them. Before, such a type linked to nothing. One written through another package, like etcd's `type PutResponse pb.PutResponse`, links to that package's type, and a recursive one like `type stateFn func(*Lexer) stateFn` doesn't link to itself. Re-index Go projects after upgrading. - In Go, a struct now links to the types its fields are declared with, like etcd's `PutResponse` holding a `*ResponseHeader` or gin's `Context` holding a `HandlersChain`, so impact on `ResponseHeader` or `HandlersChain` reaches every struct that holds one and the code that uses those structs, and `codegraph_node` lists a struct's field types. Before, a struct linked only to the types it embeds. A field written through another package, like `*pb.ResponseHeader`, links to that package's type, one from outside your project, like `context.Context`, links to nothing, the types an embedded generic is filled in with, like the `Pod` in client-go's `listers.ResourceIndexer[*v1.Pod]`, link too, and a struct that holds itself, like a list node's `next *node`, doesn't link to itself. `codegraph_explore` still answers a question about a flow with the code that runs it, not with the structs that merely hold one of its types. Re-index Go projects after upgrading. diff --git a/__tests__/js-static-override-dispatch.test.ts b/__tests__/js-static-override-dispatch.test.ts new file mode 100644 index 0000000000..6c7d9e5c4b --- /dev/null +++ b/__tests__/js-static-override-dispatch.test.ts @@ -0,0 +1,179 @@ +/** + * The override-dispatch edges (`synthesizedBy: interface-impl`) linked a base + * class's method to a subclass's method of the same name even when one was + * `static` and the other not (#2761). In JavaScript and TypeScript a static + * method is inherited and overridden by the subclass's static method, an + * instance method by an instance method, and a call to one never runs the + * other. JavaScript members also never recorded `static` at all, in the + * kernel or the wasm extractor, so the edges couldn't tell them apart there. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import type { ExtractionResult, Language } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +const MEMBERS = `export class Flags { + static create() {} + run() {} + static handler = () => {}; + callback = () => {}; + static count = 0; + value = 0; + static async *stream() {} + static get size() { return 1; } + static() {} +} +`; + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +describe('a JS/TS class member records whether it is static (#2761)', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['typescript', 'tsx', 'javascript', 'jsx']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', file: string, source: string, language: Language): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource(file, source, language); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + const result = tryKernelExtract(file, source, language); + expect(result, `kernel extraction of ${file}`).not.toBeNull(); + return result!; + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + const variants = [['ts', 'typescript'], ['tsx', 'tsx'], ['js', 'javascript'], ['jsx', 'jsx']] as const; + + for (const backend of backends) { + it.each(variants)(`in %s, ${backend}`, (ext, language) => { + const members = extract(backend, `src/flags.${ext}`, MEMBERS, language).nodes + .filter((n) => n.kind === 'method' || n.kind === 'property') + .map((n) => `${n.name} ${n.kind} ${n.isStatic ? 'static' : 'instance'}`); + expect(members).toEqual([ + 'create method static', 'run method instance', 'handler method static', 'callback method instance', + 'count property static', 'value property instance', 'stream method static', 'size method static', + // A member named `static` is an instance method. + 'static method instance', + ]); + }); + } + + it.runIf(kernelAvailable)('the kernel and the wasm extractor agree on every node', () => { + for (const [ext, language] of variants) { + const canon = (r: ExtractionResult) => r.nodes.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort())).sort(); + expect(canon(extract('kernel', `src/flags.${ext}`, MEMBERS, language)), ext) + .toEqual(canon(extract('wasm', `src/flags.${ext}`, MEMBERS, language))); + } + }); +}); + +describe('an override pairs a static method with a static one only (#2761)', () => { + let root = ''; + let graph: CodeGraph | null = null; + + afterEach(() => { + graph?.close(); + graph = null; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = ''; + }); + + /** `base -> override` for every interface-impl edge, by qualified name and kind of method. */ + function overrides(g: CodeGraph, file: string): string[] { + const out: string[] = []; + const label = (id: string) => { + const n = g.getNode(id)!; + return `${n.isStatic ? 'static ' : ''}${n.qualifiedName}`; + }; + for (const node of g.getNodesInFile(file)) { + for (const e of g.getOutgoingEdges(node.id)) { + if (e.kind === 'calls' && e.metadata?.synthesizedBy === 'interface-impl') out.push(`${label(e.source)} -> ${label(e.target)}`); + } + } + return out.sort(); + } + + it('in TypeScript and JavaScript, for methods and callable fields', async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-static-override-')); + const classes = [ + // The issue's reproduction, and the other direction. + 'class B { s() {} }', + 'class A extends B { static s() {} }', + 'class StaticBase { static t() {} }', + 'class InstanceDerived extends StaticBase { t() {} }', + // A static overriding a static, an instance method an instance one. + 'class SB { static make() {} }', + 'class SD extends SB { static make() {} }', + 'class IB { go() {} }', + 'class ID extends IB { go() {} }', + // A base with both: each pairs with its own kind. + 'class Mixed { static m() {} m() {} }', + 'class StaticOnly extends Mixed { static m() {} }', + 'class InstanceOnly extends Mixed { m() {} }', + // A subclass with both, like webpack's modules with a `static + // deserialize()` beside the instance one: only that one overrides. + 'class Both extends IB { static go() {} go() {} }', + // Callable class fields. + 'class FieldBase { run = () => {}; static start = () => {}; }', + 'class FieldDerived extends FieldBase { static run = () => {}; start = () => {}; }', + '', + ].join('\n'); + fs.writeFileSync(path.join(root, 'package.json'), JSON.stringify({ name: 'app' })); + fs.mkdirSync(path.join(root, 'src')); + fs.writeFileSync(path.join(root, 'src/classes.ts'), classes); + fs.writeFileSync(path.join(root, 'src/classes.js'), classes); + fs.writeFileSync( + path.join(root, 'src/shapes.ts'), + 'export interface Shape { area(): number }\nexport class Square implements Shape { static area(): number { return 0; } }\nexport class Circle implements Shape { area(): number { return 1; } }\n', + ); + // Java, whose override edges this leaves as they were. + fs.writeFileSync(path.join(root, 'src/Base.java'), 'class Base { static void s() {} void r() {} }\nclass Derived extends Base { static void s() {} void r() {} }\n'); + graph = await CodeGraph.init(root, { index: true }); + + const expected = [ + 'IB::go -> Both::go', + 'IB::go -> ID::go', + 'Mixed::m -> InstanceOnly::m', + 'static Mixed::m -> static StaticOnly::m', + 'static SB::make -> static SD::make', + ]; + expect(overrides(graph, 'src/classes.ts')).toEqual(expected); + expect(overrides(graph, 'src/classes.js')).toEqual(expected); + expect(overrides(graph, 'src/shapes.ts')).toEqual(['Shape::area -> Circle::area']); + expect(overrides(graph, 'src/Base.java')).toEqual(['Base::r -> Derived::r', 'static Base::s -> static Derived::s']); + }, 120_000); +}); diff --git a/codegraph-kernel/src/tsjs/mod.rs b/codegraph-kernel/src/tsjs/mod.rs index 69860a0155..12eea4be1a 100644 --- a/codegraph-kernel/src/tsjs/mod.rs +++ b/codegraph-kernel/src/tsjs/mod.rs @@ -42,7 +42,7 @@ impl Variant { } } /// TS-family (typescript/tsx): type annotations, interfaces, enums, - /// aliases, visibility, isStatic. The JS family lacks all of those hooks. + /// aliases, visibility. The JS family lacks all of those hooks. fn is_ts(self) -> bool { matches!(self, Variant::Typescript | Variant::Tsx) } @@ -1072,13 +1072,10 @@ impl<'t> Walker<'t> { self.has_keyword_child(node, "async") } - /// TS has an isStatic hook; JS does not (None = field absent). + /// A member's own `static` keyword: the isStatic hook TS and JS share + /// (hasStaticModifier, #2761). fn is_static(&self, node: Node) -> Option { - if self.variant.is_ts() { - Some(self.has_keyword_child(node, "static")) - } else { - None - } + Some(self.has_keyword_child(node, "static")) } fn is_const_decl(&self, node: Node) -> bool { diff --git a/src/extraction/languages/javascript.ts b/src/extraction/languages/javascript.ts index 899e1530a3..b96956e466 100644 --- a/src/extraction/languages/javascript.ts +++ b/src/extraction/languages/javascript.ts @@ -1,6 +1,6 @@ import { getNodeText, getChildByField } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; -import { classifyTsClassMember } from './typescript'; +import { classifyTsClassMember, hasStaticModifier } from './typescript'; export const javascriptExtractor: LanguageExtractor = { functionTypes: ['function_declaration', 'generator_function_declaration', 'arrow_function', 'function_expression', 'generator_function'], @@ -77,6 +77,9 @@ export const javascriptExtractor: LanguageExtractor = { } return false; }, + // A `static` method or field, as in TypeScript (#2761): without it every JS + // class member read as an instance one. + isStatic: hasStaticModifier, isConst: (node) => { if (node.type === 'lexical_declaration') { for (let i = 0; i < node.childCount; i++) { diff --git a/src/extraction/languages/typescript.ts b/src/extraction/languages/typescript.ts index eae4cc8dff..40f5a64655 100644 --- a/src/extraction/languages/typescript.ts +++ b/src/extraction/languages/typescript.ts @@ -3,6 +3,18 @@ import type { LanguageExtractor } from '../tree-sitter-types'; import { hasFlowPragma } from '../grammars'; import type { Node as SyntaxNode } from 'web-tree-sitter'; +/** + * Whether a class member is written `static`: its own `static` keyword, never + * a member named `static()`. Shared by the TypeScript and JavaScript + * extractors, which parse class bodies alike. + */ +export function hasStaticModifier(node: SyntaxNode): boolean { + for (let i = 0; i < node.childCount; i++) { + if (node.child(i)?.type === 'static') return true; + } + return false; +} + /** * A TS/JS class field (`public_field_definition` / `field_definition`) is a * METHOD only when its value is callable — an arrow function, a function @@ -230,13 +242,7 @@ export const typescriptExtractor: LanguageExtractor = { } return false; }, - isStatic: (node) => { - for (let i = 0; i < node.childCount; i++) { - const child = node.child(i); - if (child?.type === 'static') return true; - } - return false; - }, + isStatic: hasStaticModifier, isConst: (node) => { // For lexical_declaration, check if it's 'const' or 'let' // For variable_declaration, it's always 'var' diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 4c2755688b..4727e6d001 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -822,6 +822,13 @@ const IFACE_OVERRIDE_LANGS = new Set([ 'java', 'kotlin', 'csharp', 'typescript', 'javascript', 'swift', 'scala', 'go', 'rust', 'arkts', ]); +/** + * Languages whose static and instance methods override only their own kind: + * a JS/TS class's statics are inherited and overridden by the subclass's + * statics, its instance methods by instance methods (#2761). tsx and jsx + * follow the same rule, though this pass doesn't walk their classes today. + */ +const STATIC_SPLIT_LANGS: ReadonlySet = new Set(['typescript', 'tsx', 'javascript', 'jsx']); /** * Whether a supertype edge out of a Go type is one of its embeddings (#2397): * declared, never synthesized (go-implements writes those), into a struct, an @@ -1671,6 +1678,11 @@ async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield for (const m of impls) { if (added >= MAX_CALLBACKS_PER_CHANNEL) break; if (bm.id === m.id) continue; + // A JS/TS static method overrides only a static one, and an + // instance method only an instance one (#2761): B's `s()` and the + // `static s()` of `class A extends B` are unrelated, and a call to + // one never runs the other. + if (STATIC_SPLIT_LANGS.has(m.language) && (bm.isStatic === true) !== (m.isStatic === true)) continue; const key = `${bm.id}>${m.id}`; if (seen.has(key)) continue; seen.add(key); From 53f72fec90e759607cdd3099ea85befdab0687f2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 21:49:29 +0000 Subject: [PATCH 46/95] fix(liquid): variables declared with {% capture %} are indexed, as assigned ones are (#2814) The Liquid extractor indexed a variable declared with {% assign %} as a variable node, in both spellings (braced, and as a line of a {% liquid %} tag) and skipping comment and raw regions, but it never read {% capture %}, Liquid's other way to declare a variable. A captured variable could not be found: search missed it, and codegraph_explore reported its name as not in the index. Fix: - extractAssignments reads capture tags through the same findTagOccurrences scanner as assign, so both spellings and the comment/raw rule apply. The name is word characters, as for assign, optionally in single or double quotes (Liquid reads 'title' as title), and must end at whitespace or the tag's end (-%} included). A name that goes on with "-" or ".", which the assign rule doesn't read either, gets no node rather than a cut-short one. - Captures are read after assigns. Node ids are keyed by file, kind, name and line, so an assign keeps its id when a capture of the same name shares its line. - One helper now builds the variable node and its contains edge for both. It computes the declaration's column once, where the assign code computed it three times, each a split of the whole file. Validation: - 13 new tests in __tests__/extraction.test.ts. 11 fail with main's extractor and pass with the fix: 8 declaration forms (braced, whitespace-trimmed, tight -%}, single- and double-quoted, {% liquid %} lines with LF and CRLF, a quoted {% liquid %} line), an assign and two captures of one name on one line (distinct ids, the assign keeps its), tags inside a capture body, and an index-level test that search finds captured variables in a non-theme folder. 2 guards pass on both: no capture from comment, raw, inline comments, prose, a nameless tag or a tag inside a string, and no cut-short hyphenated or dotted name. Red/green on the committed tree: 42 passed and 11 failed with main's extractor, 53 passed with the fix. - A/B on efb091ec, each arm built from its own checkout: 8 Shopify theme repos (Shopify/dawn, Shopify/horizon, Shopify/skeleton-theme, ChumsInc/chums-impulse-dev, agencecinq/cinquante-et-un, noclocks/shopify-devstore-theme, panoply/syncify, Shopify/theme-tools) and 2 controls (express, requests). The controls and skeleton-theme, which has no capture, are byte-identical. Every other theme gains only variable nodes, each with one contains edge: dawn 36, horizon 192, chums-impulse-dev 90, cinquante-et-un 25, shopify-devstore-theme 27, syncify 25, theme-tools 49, 444 in all (383 braced, 61 in {% liquid %} tags). That is exactly the count of capture tags a separate scan finds outside comment and raw regions, and a script checked each new node against the source text: it sits on a capture tag naming it, with no edge but its contains. 444 right, 0 wrong. Nothing is removed or moved, no unresolved ref changes, and no node present on both arms changed its id. Every capture and assign name in these themes is a plain word. - Cost: the extractor is faster with the fix. A microbenchmark over 1,386 Liquid files (8 MB), 15 runs in each of 3 interleaved rounds, gave per-round medians of 443-465 ms on main and 316-387 ms with the fix, because whole-file line-start scans fall from 29,807 to 20,671. Index wall time, median of 3 interleaved runs, main vs fix: horizon 2975 vs 2935 ms, chums-impulse-dev 2404 vs 2311, syncify 4329 vs 3867, theme-tools 8405 vs 8108, and the byte-identical controls express 2427 vs 2417 and requests 2006 vs 1942, which is within the noise of a shared machine. - Rebased onto 68fcf2f1, which adds only PR #2813 (JS/TS static and instance overrides, with a kernel change, so the kernel was rebuilt). There the Liquid tests pass, and re-indexing the 8 theme repos changes no Liquid or JSON line: 4 are byte-identical to the pre-rebase fix arm, and the others differ only in PR #2813's is_static flag on JavaScript methods (dawn 30, horizon 26, shopify-devstore-theme 14) and one synthesized TypeScript call in syncify. - Kernel parity: not applicable, the kernel does not extract Liquid. The issue also asks for README and docs-site wording about what Liquid support covers. That is PR #2815, a docs-only change held for the maintainer's review, so this change leaves the issue open. Refs #2498 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/extraction.test.ts | 76 ++++++++++++++++++++++++++++++ src/extraction/liquid-extractor.ts | 70 ++++++++++++++++----------- 3 files changed, 119 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ec3ae232c..644ee1d646 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,6 +119,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In a repository that holds more than one Shopify theme, a theme's `{% render %}`, `{% include %}` and `{% section %}` tags and the sections its JSON templates list now link only to that theme's own snippets and sections, which is where Shopify looks for them. Before, a section or snippet the theme doesn't have was linked to another theme's file of the same name, and a theme kept in a subfolder could link to the root theme's copy instead of its own. A reference the theme can't satisfy now links to nothing. Projects with a single theme are indexed as before. Re-index projects with several Shopify themes after upgrading. - `codegraph sync` and the file watcher now link a Shopify section or snippet to the files that name it when it is added after them, or comes back after being deleted, whether they name it in a `{% render %}`, `{% include %}` or `{% section %}` tag or as a section `type` in a JSON template. Before, the link appeared only once a file naming it changed or the project was re-indexed, even in a repository with a single theme, so the new section or snippet looked unused and `codegraph affected` missed the files that use it. Re-index Shopify themes after upgrading to pick up links an earlier version missed. - A Shopify JSON template or section group saved by the theme editor now links the sections it lists. The editor writes a comment at the top of every file it saves, and a file with that comment, or with a trailing comma, both of which Shopify accepts, linked none of its sections, so in an editor-saved theme most sections looked unused. A layout's `{% sections 'header-group' %}` tag now links to the section group it renders, each section a JSON file lists is linked from the line that names it rather than from the top of the file, and a `{% schema %}` written inside `{% comment %}` or `{% raw %}` is no longer indexed as the section's own. Re-index Shopify themes after upgrading. Thanks @Puma7 for the report and the pull request. (#2494) +- In Liquid, a variable declared with `{% capture %}`, like `{% capture title %}…{% endcapture %}`, a quoted `{% capture 'title' %}` or a `capture title` line inside `{% liquid %}`, is now indexed like one declared with `{% assign %}`, so search and `codegraph_explore` can find it. Before, only assigned variables were indexed, so a captured one could not be found, and `codegraph_explore` said its name was not in the index. Re-index Liquid projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2498) - `codegraph sync` and the file watcher now link a route to the page or layout it loads lazily when that file is added after the router was indexed, comes back after being deleted, or gets the component the route renders in a later edit, whether or not the import names the file's extension: React Router's `lazy: () => import('./pages/Team')`, Vue Router's `component: () => import('@/views/Login')` and Angular's `loadComponent: () => import('./home/home.component')`. Before, the route stayed unlinked until the router file changed or the project was re-indexed, so the page looked unused and `codegraph affected` missed the tests that reach it through the router. Re-index after upgrading to pick up links an earlier version missed. - A file named the way Google names its tests, like `wire_format_unittest.cc` in protocolbuffers/protobuf, `logging_unittest.cc` in google/glog or a Python `foo_unittest.py`, now counts as a test, as `wire_format_test.cc` already did. Before, it was taken for production code: search and `codegraph_explore` ranked it alongside the code it tests, `codegraph affected` never listed it, and calls in production code could link to a class or type a unittest declares for itself, as google/breakpad's `stack_frame_entries_.size()` did to a unittest's `StackHelper`. Re-index projects that have such files after upgrading. - A file in a `unittests/` folder, where llvm/llvm-project and google/breakpad keep their unit tests, or in a folder named like a unittest, such as `dcheck_unittest/` in google/glog, now counts as a test whatever its own name says, like LLVM's `CountCopyAndMove.h` helper. Before, such a file was taken for production code unless its name said test, and none of them were kept apart from the program the way a `tests/` folder is: search and `codegraph_explore` ranked them alongside production code, `codegraph affected` missed them, and production code linked to a unit test's own declarations, as LLVM's `Ctx` and `OpInfo` variables did to ones its unit tests declare. A folder named just `unittest/`, like CPython's `Lib/unittest/`, is still production code. Re-index projects that have such folders after upgrading. diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts index eefa535c90..8163436978 100644 --- a/__tests__/extraction.test.ts +++ b/__tests__/extraction.test.ts @@ -5325,6 +5325,64 @@ import 'package:flutter/material.dart'; expect(vars).toContain('heading'); }); + /* `{% capture name %}…{% endcapture %}` declares a variable as `assign` + does; only assigns were read (#2498). */ + it.each([ + ['{% capture captured %}Hello{% endcapture %}', 'captured', 1, 0], + ['{%- capture trimmed -%}Hi{%- endcapture -%}', 'trimmed', 1, 0], + ['{% capture tight-%}Hi{% endcapture %}', 'tight', 1, 0], + ["

{% capture 'single' %}Hi{% endcapture %}

", 'single', 1, 3], + ['{% capture "double" %}Hi{% endcapture %}', 'double', 1, 0], + ['{% liquid\n capture bare\n echo "Hi"\n endcapture\n%}', 'bare', 2, 2], + ['{% liquid\r\n capture crlf # a note\r\n endcapture\r\n%}', 'crlf', 2, 2], + ["{%- liquid\n capture 'quoted_line'\n endcapture\n-%}", 'quoted_line', 2, 2], + ] as const)('reads a capture declaration from %j', (code, name, line, column) => { + const result = extractFromSource('sections/card.liquid', code); + const variables = result.nodes.filter((n) => n.kind === 'variable'); + expect(variables.map((n) => [n.name, n.qualifiedName, n.startLine, n.startColumn])) + .toEqual([[name, `sections/card.liquid::${name}`, line, column]]); + const file = result.nodes.find((n) => n.kind === 'file')!; + expect(result.edges).toContainEqual({ source: file.id, target: variables[0]!.id, kind: 'contains' }); + }); + + it('reads no capture from comments, raw regions, prose or a tag without a name', () => { + const code = [ + '{% comment %}{% capture ghost %}x{% endcapture %}{% endcomment %}', + '{%- raw -%}{% capture ghost %}x{% endcapture %}{%- endraw -%}', + '{% # capture ghost %}', + '{% liquid', ' # capture ghost', ' comment', ' capture ghost', ' endcomment', '%}', + '{% capture %}x{% endcapture %}', + '

We capture ghost moments.

', + '{% assign text = "{% capture ghost %}" %}', + ].join('\n'); + const result = extractFromSource('sections/card.liquid', code); + expect(result.nodes.filter((n) => n.kind === 'variable').map((n) => n.name)).toEqual(['text']); + }); + + it('never cuts a capture name short', () => { + // Liquid allows `-` and `.` in a name, which the assign rule doesn't read + // either; such a name gets no node rather than a wrong one. + const code = ['{% capture ghost-name %}x{% endcapture %}', '{% capture ghost.name %}x{% endcapture %}'].join('\n'); + expect(extractFromSource('sections/card.liquid', code).nodes.filter((n) => n.kind === 'variable')).toEqual([]); + }); + + it('keeps an assign and captures of the same name on one line apart', () => { + const code = '{% assign title = 1 %}{% capture title %}one{% endcapture %}{% capture title %}two{% endcapture %}'; + const variables = extractFromSource('sections/card.liquid', code).nodes.filter((n) => n.kind === 'variable'); + expect(variables.map((n) => [n.name, n.startColumn])) + .toEqual([['title', 0], ['title', code.indexOf('{% capture')], ['title', code.lastIndexOf('{% capture')]]); + expect(new Set(variables.map((n) => n.id)).size).toBe(3); + // The assign keeps the id it had before captures were read. + expect(variables[0]!.id).toBe(generateNodeId('sections/card.liquid', 'variable', 'title', 1)); + }); + + it('still reads the tags inside a capture', () => { + const code = "{% capture card_html %}{% render 'card' %}{% assign inner = 1 %}{% endcapture %}"; + const result = extractFromSource('sections/product.liquid', code); + expect(result.nodes.filter((n) => n.kind === 'variable').map((n) => n.name).sort()).toEqual(['card_html', 'inner']); + expect(result.unresolvedReferences.map((r) => r.referenceName)).toEqual(['snippets/card.liquid']); + }); + it('should extract section inside a {% liquid %} block', () => { const code = ['{% liquid', " section 'header'", '%}'].join('\n'); const result = extractFromSource('layout/theme.liquid', code); @@ -7537,6 +7595,24 @@ describe('Liquid Shopify JSON template section resolution', () => { expect(cg.getNodesByKind('variable').some((n) => n.name === 'heading')).toBe(true); }); + it('indexes capture declarations as variables that search finds, in or out of a Shopify theme', async () => { + // A captured variable could not be found, while an assigned one could (#2498). + fs.mkdirSync(path.join(tempDir, '_includes'), { recursive: true }); + fs.writeFileSync(path.join(tempDir, '_includes/card.liquid'), [ + "{% assign existing = 'ok' %}", + '{% capture captured %}Hello{% endcapture %}', + '{% liquid', ' capture bare_capture', " echo 'x'", ' endcapture', '%}', + ].join('\n')); + cg = CodeGraph.initSync(tempDir); + await cg.indexAll(); + + for (const [name, line] of [['existing', 1], ['captured', 2], ['bare_capture', 4]] as const) { + const hit = cg.searchNodes(name, { kinds: ['variable'] }).find((r) => r.node.name === name); + expect(hit?.node, name).toMatchObject({ filePath: '_includes/card.liquid', language: 'liquid', startLine: line }); + expect(cg.getIncomingEdges(hit!.node.id)).toContainEqual(expect.objectContaining({ kind: 'contains' })); + } + }); + it('links a Shopify JSON template section `type` to its sections/.liquid', async () => { // Shopify OS 2.0 templates are JSON, referencing sections by `type` — not // a `{% section %}` Liquid tag — so a section used only from a JSON template diff --git a/src/extraction/liquid-extractor.ts b/src/extraction/liquid-extractor.ts index 15850ff821..4de1826c83 100644 --- a/src/extraction/liquid-extractor.ts +++ b/src/extraction/liquid-extractor.ts @@ -11,6 +11,7 @@ import { generateNodeId, NodeIdAllocator } from './tree-sitter-helpers'; * ({% sections 'header-group' %} → `sections/header-group.json`) * - Snippet references ({% render 'name' %} and {% include 'name' %}) * - Schema blocks ({% schema %}...{% endschema %}) + * - Variables declared with {% assign %} and {% capture %} * - The sections a Shopify JSON template or section group lists */ export class LiquidExtractor { @@ -55,7 +56,7 @@ export class LiquidExtractor { // Extract schema block this.extractSchema(fileNode.id); - // Extract assign statements as variables + // Extract assign and capture declarations as variables this.extractAssignments(fileNode.id); } } catch (error) { @@ -459,40 +460,53 @@ export class LiquidExtractor { } /** - * Extract {% assign var = value %} statements + * Extract {% assign var = value %} and {% capture var %}…{% endcapture %} + * declarations as variables */ private extractAssignments(fileNodeId: string): void { // Both spellings. Most of a modern theme's assigns are the bare kind. for (const match of this.findTagOccurrences('assign', `(\\w+)\\s*=`)) { - const variableName = match.groups[1]; - const line = this.getLineNumber(match.index); - - // Create a variable node - const nodeId = this.nodeIds.generate(this.filePath, 'variable', variableName!, line, match.index - this.getLineStart(line)); + this.addVariable(fileNodeId, match.groups[1]!, match); + } + // A capture declares its name as an assign does, and Liquid reads a quoted + // name (`{% capture 'title' %}`) as the name inside the quotes. A name that + // goes on with `-` or `.`, which the assign rule doesn't read either, gets + // no node rather than a cut-short one. Captures come after assigns, so an + // assign keeps its id when a capture of the same name shares its line. + for (const match of this.findTagOccurrences('capture', `(?:'(\\w+)'|"(\\w+)"|(\\w+))(?=\\s|-?%\\}|$)`)) { + const [, singleQuoted, doubleQuoted, bare] = match.groups; + this.addVariable(fileNodeId, (singleQuoted ?? doubleQuoted ?? bare)!, match); + } + } - const node: Node = { - id: nodeId, - kind: 'variable', - name: variableName!, - qualifiedName: `${this.filePath}::${variableName}`, - filePath: this.filePath, - language: 'liquid', - startLine: line, - endLine: line, - startColumn: match.index - this.getLineStart(line), - endColumn: match.index - this.getLineStart(line) + match.fullMatch.length, - updatedAt: Date.now(), - }; + /** A `variable` node, contained by the file, for a declaration that `findTagOccurrences` found. */ + private addVariable(fileNodeId: string, variableName: string, match: { fullMatch: string; index: number }): void { + const line = this.getLineNumber(match.index); + const column = match.index - this.getLineStart(line); + const nodeId = this.nodeIds.generate(this.filePath, 'variable', variableName, line, column); + + const node: Node = { + id: nodeId, + kind: 'variable', + name: variableName, + qualifiedName: `${this.filePath}::${variableName}`, + filePath: this.filePath, + language: 'liquid', + startLine: line, + endLine: line, + startColumn: column, + endColumn: column + match.fullMatch.length, + updatedAt: Date.now(), + }; - this.nodes.push(node); + this.nodes.push(node); - // Add containment edge from file - this.edges.push({ - source: fileNodeId, - target: nodeId, - kind: 'contains', - }); - } + // Add containment edge from file + this.edges.push({ + source: fileNodeId, + target: nodeId, + kind: 'contains', + }); } /** From ab92c90a540f419ab9d0eef841925aa1a85df43d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 22:11:29 +0000 Subject: [PATCH 47/95] fix(js,ts): a super.method() call links to the method of the class's parent (#2818) A TS/JS call written super.run() is recorded under its bare name, as issue #2497 found. Inside an override of run every strategy found the calling method itself, and the self-edge gate declined it, so the call was unlinked; elsewhere it went to whatever method had the name, often the class's own or a sibling class's in the same file, and, for a parent outside the project like HTMLElement, Set or Promise, to a project namesake. A call written on super where it starts is now resolved through the class's resolved extends edges, which the resolver persists before it resolves calls: the nearest ancestor's method of that name, a static one for a call in a static member, or nothing when no ancestor in the project has it. The call may sit in a method, a class field, or an arrow function inside either. One the index can't place in a class (a class expression's method, an object literal's) keeps the old path, as does one after a class keyword in the member, outside comments and strings. The self-edge gate now reads a TS/JS call at its own column, so a this.run() on the line of a super.run() keeps its edge. The issue's first part, a method call on the receiver of an aliased constructor, already resolves on main since PR #2661 and is unchanged. Validation: the new test fails 3 of 4 on main 68fcf2f1 and passes 4 of 4 on the fix; the fourth pins the class-expression guard, and a mutation of each guard fails a test. A/B against main on 22 repos (20 TS/JS, flask and gin): 660 call sites change, all super calls, checked against each class's extends chain and sampled by hand. 643 gain an edge to an ancestor's method (304 in three.js, like super.copy(source) reaching Object3D.copy; 210 in webpack, 85 in lit); 2 move off the class's own method to the parent's; 4 keep their target, now by the parent; 11 lose a guess, where the parent is a builtin or a library's class, like webpack's SortableSet extending Set, or a mixin's parameter. Nodes and all other edges are unchanged; 14 repos' dumps, the controls among them, are byte-identical. A sync that moves a parent, swaps a parent or adds an override to an intermediate class matches a fresh index. Index time, median of 3 interleaved runs on three.js, webpack, jest and insomnia: +1.7% total, noise; 7 interleaved rounds on three.js alone: -1.5%. Fixes #2497 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/js-super-calls.test.ts | 203 +++++++++++++++++++++++++++++++ src/resolution/index.ts | 26 ++++ src/resolution/js-super-calls.ts | 139 +++++++++++++++++++++ src/resolution/types.ts | 7 ++ 5 files changed, 376 insertions(+) create mode 100644 __tests__/js-super-calls.test.ts create mode 100644 src/resolution/js-super-calls.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 644ee1d646..82873bfcee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -196,6 +196,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a type alias like `type Event = mvccpb.Event`, alone or inside a `type ( … )` group, is now indexed and links to the type it names, so impact on that type reaches the code that uses the alias. Before, aliases were missing: a use written through the alias's package, like `clientv3.Event` or prometheus's `storage.AOptions{…}`, linked to nothing, and one inside the package, like `&PrometheusVersion{}` in prometheus's `web` package, linked to whichever type of that name the project had, there a test helper's struct. A method called on a value of an alias's type now reaches the aliased type's method rather than a namesake's, and a generic alias like `type Items[T any] = List[T]` links to `List`, so `xs.Len()` on one reaches `List.Len` instead of nothing. Re-index Go projects after upgrading. - In Go, a struct that embeds another is no longer treated as overriding the methods it redeclares. gin's `Engine` embeds `RouterGroup` and has a `Use` of its own, but calling `Use` on a `RouterGroup` only ever runs `RouterGroup.Use`; before, `codegraph_explore`'s call path and impact went on from it into `Engine.Use` and the code only the engine runs. The same held for every struct or defined type embedded in another that redeclares one of its methods, like etcd's `Mutex` inside `lockerMutex`. A call through an interface still reaches each type that implements it, the embedding struct included. Re-index Go projects after upgrading. - In JavaScript and TypeScript, a subclass's `static` method no longer counts as overriding an instance method of the same name in its base class, or the reverse, since calling one never runs the other. In webpack, `codegraph_explore`'s call path and impact went on from `Module`'s `deserialize(context)` into the `static deserialize(context)` of `NormalModule` and other modules. A static method still overrides its base's static method and an instance method its base's instance method, and JavaScript class members are now marked static, as TypeScript ones already were. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and @yang0228 for the pull request. (#2761) +- In JavaScript and TypeScript, a `super.run()` call now links to the method of the class's parent, or of the nearest ancestor that has it, like three.js's `super.copy(source)` reaching `Object3D.copy`, so `codegraph_explore`'s call path and impact follow an override into the method it builds on. Before, such a call inside a method of the same name was dropped, and elsewhere it went to whatever method had the name, often the class's own or a sibling class's in the same file. A call from a static method reaches the parent's static method, and a call to a parent from outside your project, like `HTMLElement`, `Set` or `Promise`, links nothing. Re-index JavaScript and TypeScript projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2497) - In Go, a call through a struct field that holds a function, like prometheus's `d.refreshf(ctx)`, now reaches the functions put in that field, so `codegraph_explore` follows service discovery from `Discovery.Run` into each provider's `refresh` and names the line where each provider wires its method in, like `RefreshF: d.refresh`. The field can be set in a struct literal or by an assignment, copied from another struct's field (`refreshf: opts.RefreshF`), or given a method of an interface value, and two unrelated structs with a field of the same name, like `fn` or `handler`, stay apart. Before, such a flow stopped at the field. Re-index Go projects after upgrading. - In Go, a type defined from other types, like etcd's `type WatchChan <-chan WatchResponse` or gin's `type HandlerFunc func(*Context)` and `type HandlersChain []HandlerFunc`, now links to the types it is built from, so impact on `WatchResponse` or gin's `Context` reaches these types and the code that uses them. Before, such a type linked to nothing. One written through another package, like etcd's `type PutResponse pb.PutResponse`, links to that package's type, and a recursive one like `type stateFn func(*Lexer) stateFn` doesn't link to itself. Re-index Go projects after upgrading. - In Go, a struct now links to the types its fields are declared with, like etcd's `PutResponse` holding a `*ResponseHeader` or gin's `Context` holding a `HandlersChain`, so impact on `ResponseHeader` or `HandlersChain` reaches every struct that holds one and the code that uses those structs, and `codegraph_node` lists a struct's field types. Before, a struct linked only to the types it embeds. A field written through another package, like `*pb.ResponseHeader`, links to that package's type, one from outside your project, like `context.Context`, links to nothing, the types an embedded generic is filled in with, like the `Pod` in client-go's `listers.ResourceIndexer[*v1.Pod]`, link too, and a struct that holds itself, like a list node's `next *node`, doesn't link to itself. `codegraph_explore` still answers a question about a flow with the code that runs it, not with the structs that merely hold one of its types. Re-index Go projects after upgrading. diff --git a/__tests__/js-super-calls.test.ts b/__tests__/js-super-calls.test.ts new file mode 100644 index 0000000000..357ce97d48 --- /dev/null +++ b/__tests__/js-super-calls.test.ts @@ -0,0 +1,203 @@ +/** + * A TS/JS call written `super.run(…)` runs the method of the class's parent, + * or of the nearest ancestor that has it (#2497). The extractor keeps such a + * call under its bare name, so inside an override of `run` every strategy + * found the calling method itself and the self-edge was declined, leaving the + * call unlinked; elsewhere the call went to whatever method had the name, like + * a sibling class's in the same file. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const BASE = `export class Base { + run() { return 1; } + init() {} + static make() { return new Base(); } + make() { return 2; } +} +`; + +const CHILDREN = `import { Base } from './base'; +export class Decoy { run() { return 'decoy'; } missing() {} } +export class Child extends Base { + run() { return super.run(); } +} +export class Worker extends Base { + work() { return super.run(); } +} +export class Lonely extends Base { + go() { return super.missing(); } +} +export class Mid extends Base {} +export class Leaf extends Mid { + run() { return super.run(); } +} +export class Mid2 extends Base { + run() { return 3; } +} +export class Leaf2 extends Mid2 { + run() { return super.run(); } +} +export class Maker extends Base { + static make() { return super.make(); } + make() { return super.make(); } + constructor() { + super(); + super.init(); + } + later = () => super.init(); + each() { return [1].map(() => super.run()); } + named() { + const go = () => super.run(); + return go(); + } + noted() { + // runs the parent class's run, as webpack writes "class-static-block" + const kind = 'class-static-block'; + return super.run(kind); + } + split() { + return super + .run(); + } +} +export class Counter extends Base { + run(n) { return n > 0 ? this.run(n - 1) : super.run(); } +} +`; + +const OUTSIDE = `export class Widget extends HTMLElement { + connectedCallback() { super.connectedCallback(); } + attach() { return super.connectedCallback(); } +} +export class Panel { + connectedCallback() {} +} +`; + +describe.each(['js', 'ts'])('a super call reaches the parent class (#2497): %s', (ext) => { + let root = ''; + let graph: CodeGraph | null = null; + + afterEach(() => { + graph?.close(); + graph = null; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = ''; + }); + + /** Every `calls` edge the index resolved (synthesized dispatch left out), by qualified names. */ + function calls(g: CodeGraph): string[] { + const out: string[] = []; + const label = (id: string) => { + const n = g.getNode(id)!; + return `${n.isStatic ? 'static ' : ''}${n.qualifiedName}`; + }; + for (const kind of ['method', 'function'] as const) { + for (const node of g.getNodesByKind(kind)) { + for (const e of g.getOutgoingEdges(node.id)) { + if (e.kind !== 'calls' || e.metadata?.synthesizedBy) continue; + out.push(`${label(e.source)} -> ${label(e.target)} [${e.metadata?.resolvedBy}]`); + } + } + } + return out.sort(); + } + + it('links the parent\'s or the nearest ancestor\'s method, of the caller\'s kind, and nothing else', async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-super-call-')); + fs.writeFileSync(path.join(root, 'package.json'), JSON.stringify({ name: 'app' })); + fs.mkdirSync(path.join(root, 'src')); + fs.writeFileSync(path.join(root, `src/base.${ext}`), BASE); + fs.writeFileSync(path.join(root, `src/children.${ext}`), CHILDREN); + fs.writeFileSync(path.join(root, `src/outside.${ext}`), OUTSIDE); + graph = await CodeGraph.init(root, { index: true }); + + expect(calls(graph)).toEqual([ + // The issue's override, and its renamed form: not the sibling Decoy's + // or Child's `run`, which share the file. + 'Child::run -> Base::run [instance-method]', + // Real recursion keeps its edge beside a super call on the same line. + 'Counter::run -> Base::run [instance-method]', + 'Counter::run -> Counter::run [exact-match]', + // A grandparent's method, and a nearer ancestor's over it. + 'Leaf2::run -> Mid2::run [instance-method]', + 'Leaf::run -> Base::run [instance-method]', + // A constructor, an arrow field, an arrow callback, a named arrow and + // a call split over two lines are the instance method's; a static + // one the static's. + 'Maker::constructor -> Base::init [instance-method]', + 'Maker::each -> Base::run [instance-method]', + 'Maker::later -> Base::init [instance-method]', + 'Maker::make -> Base::make [instance-method]', + 'Maker::named -> Maker::named::go [exact-match]', + 'Maker::named::go -> Base::run [instance-method]', + // A `class` in a comment or a string is no class expression. + 'Maker::noted -> Base::run [instance-method]', + 'Maker::split -> Base::run [instance-method]', + 'Worker::work -> Base::run [instance-method]', + 'static Maker::make -> static Base::make [instance-method]', + // A method no ancestor has (Decoy's `missing`), and a parent outside + // the project (HTMLElement, not Panel), link nothing. + ]); + }, 120_000); +}); + +it('a super call in a class expression is not read as the enclosing class\'s (#2497)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-super-call-expr-')); + let graph: CodeGraph | null = null; + try { + fs.writeFileSync(path.join(root, 'package.json'), JSON.stringify({ name: 'app' })); + fs.mkdirSync(path.join(root, 'src')); + fs.writeFileSync(path.join(root, 'src/base.js'), BASE); + fs.writeFileSync(path.join(root, 'src/holder.js'), [ + "import { Base } from './base.js';", + 'export class Other { run() { return 0; } }', + 'export class Holder extends Base {', + ' make() {', + ' return class extends Other { run() { return super.run(); } };', + ' }', + '}', + '', + ].join('\n')); + graph = await CodeGraph.init(root, { index: true }); + // The class expression's methods have no nodes, so its call is recorded + // from `make`; its `super` is Other, never Holder's parent. + const make = graph.getNodesByQualifiedName('Holder::make')[0]!; + const targets = graph.getOutgoingEdges(make.id).filter((e) => e.kind === 'calls').map((e) => graph!.getNode(e.target)!.qualifiedName); + expect(targets).not.toContain('Base::run'); + } finally { + graph?.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}, 120_000); + +it('a TypeScript class reaches the class it extends, not an interface it implements (#2497)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-super-call-ts-')); + let graph: CodeGraph | null = null; + try { + fs.writeFileSync(path.join(root, 'package.json'), JSON.stringify({ name: 'app' })); + fs.mkdirSync(path.join(root, 'src')); + fs.writeFileSync(path.join(root, 'src/base.ts'), 'export class Base { run(): number { return 1; } }\n'); + fs.writeFileSync(path.join(root, 'src/both.ts'), [ + "import { Base as Parent } from './base';", + 'export interface Runner { run(): number }', + 'export class Both extends Parent implements Runner {', + ' run(): number { return super.run(); }', + '}', + '', + ].join('\n')); + graph = await CodeGraph.init(root, { index: true }); + const both = graph.getNodesByQualifiedName('Both::run')[0]!; + const targets = graph.getOutgoingEdges(both.id) + .filter((e) => e.kind === 'calls') + .map((e) => `${graph!.getNode(e.target)!.qualifiedName} [${e.metadata?.resolvedBy}]`); + expect(targets).toEqual(['Base::run [instance-method]']); + } finally { + graph?.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}, 120_000); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 6a381191dc..2f7f9cb702 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -39,6 +39,7 @@ import { clearCppIncluderMemos } from './cpp-includers'; import { clearCFeatureVariantMemos } from './c-feature-variants'; import { clearAmbientModuleMemos, ambientModuleDeclarations } from './ambient-modules'; import { matchShopifyThemeFile } from './shopify-themes'; +import { isJsSuperCallLanguage, isJsSuperCallSite, matchJsSuperCall } from './js-super-calls'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveTsQualifiedTypeRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isDartImportRef, isLuaRequireRef, isJsPathImportRef, isBoundToOutOfRepoImport, reachesAmbientModule, importBindingTail, clearImportResolverMemos, resolveImportPath, isExternalImport, ESM_IMPORT_LANGUAGES, jsModuleFile, pythonRootFingerprint, pythonReopenScope, pythonPackageImporters, moduleDefaultExport, isJsFamilyFile, isModuleBindingName } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; import { resolveAliasBinding } from './alias-binding'; @@ -306,6 +307,8 @@ export class ReferenceResolver { private supertypeMemo = new Map(); // getSupertypeNodes' memo, by node id, under the same generation tag. private supertypeNodeMemo = new Map(); + // getSuperclassNodes' memo (`extends` edges alone), the same way. + private superclassNodeMemo = new Map(); /** Invalidate the getSupertypes memo — call when resolved edges may have advanced. */ private advanceSupertypeGeneration(): void { @@ -314,6 +317,7 @@ export class ReferenceResolver { // batches doesn't accrete dead entries. if (this.supertypeMemo.size > 50_000) this.supertypeMemo.clear(); if (this.supertypeNodeMemo.size > 50_000) this.supertypeNodeMemo.clear(); + if (this.superclassNodeMemo.size > 50_000) this.superclassNodeMemo.clear(); } // Node kinds are a small fixed set (~24), so this is a plain Map, not an LRU. // getNodesByKind returns the FULL node list for a kind; it was previously @@ -584,6 +588,7 @@ export class ReferenceResolver { this.methodOwnerIndexCache.clear(); this.supertypeMemo.clear(); this.supertypeNodeMemo.clear(); + this.superclassNodeMemo.clear(); this.supertypeGen++; this.nodesByKindCache.clear(); this.fileExistsMemo.clear(); @@ -990,6 +995,18 @@ export class ReferenceResolver { return nodes; }, + getSuperclassNodes: (id: string) => { + const hit = this.superclassNodeMemo.get(id); + if (hit && hit.gen === this.supertypeGen) return hit.nodes; + const nodes: Node[] = []; + for (const edge of this.queries.getOutgoingEdges(id, ['extends'])) { + const target = this.nodeById(edge.target); + if (target) nodes.push(target); + } + this.superclassNodeMemo.set(id, { gen: this.supertypeGen, nodes }); + return nodes; + }, + getImportMappings: (filePath: string, language) => { const cacheKey = filePath; const cached = this.importMappingCache.get(cacheKey); @@ -1499,6 +1516,12 @@ export class ReferenceResolver { const themeFile = matchShopifyThemeFile(ref, this.context); if (themeFile !== undefined) return themeFile; + // A TS/JS `super.run(…)` runs the method of the class's parent or a + // nearer ancestor, or one outside the project: never a method found by + // its name alone (see ./js-super-calls). + const superCall = matchJsSuperCall(ref, this.context); + if (superCall !== undefined) return superCall; + // Skip built-in/external references if (this.isBuiltInOrExternal(ref)) { return null; @@ -3810,6 +3833,9 @@ export class ReferenceResolver { */ private gateSuperSelfCall(result: ResolvedRef | null, ref: UnresolvedRef): ResolvedRef | null { if (!result || ref.referenceKind !== 'calls' || result.targetNodeId !== ref.fromNodeId) return result; + // A TS/JS call is read where it starts, so a `this.run()` that shares its + // line with a `super.run()` keeps its edge (#2497). + if (isJsSuperCallLanguage(ref.language)) return isJsSuperCallSite(ref, this.context) ? null : result; const name = ref.referenceName.slice(Math.max(ref.referenceName.lastIndexOf('.'), ref.referenceName.lastIndexOf(':')) + 1); if (!/^[A-Za-z_$][\w$]*$/.test(name)) return result; const line = (this.context.getFileLines?.(ref.filePath) ?? this.context.readFile(ref.filePath)?.split(/\r?\n/))?.[ref.line - 1]; diff --git a/src/resolution/js-super-calls.ts b/src/resolution/js-super-calls.ts new file mode 100644 index 0000000000..c8ca9ea32c --- /dev/null +++ b/src/resolution/js-super-calls.ts @@ -0,0 +1,139 @@ +/** + * A TS/JS call written `super.run(…)` runs the method of the class's parent + * (#2497): the one the class `extends`, or the nearest ancestor of that one + * that has the method, and a static method's `super` reaches the parent's + * static methods. The extractor keeps the call under its bare name, so the + * name strategies answered for it: inside an override of `run` they found the + * calling method itself, a self-edge gateSuperSelfCall then declined, and + * elsewhere they took any method of the name, like a sibling class's `run` in + * the same file, or a project `connectedCallback` for a web component whose + * parent is `HTMLElement`. + * + * The parent comes from the class's resolved `extends` edge, which the + * resolver persists before it resolves calls. A parent outside the project, + * or an ancestry without the method, links nothing: never a guess by name. + */ +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; +import { escapeRegExp } from '../utils'; +import { blankStringContents, stripCommentsForRegex } from './strip-comments'; + +const SUPER_CALL_LANGUAGES: ReadonlySet = new Set(['typescript', 'tsx', 'javascript', 'jsx']); +const IDENTIFIER = /^[A-Za-z_$][\w$]*$/; +/** How far up a class's ancestry a `super` call is looked for. */ +const MAX_ANCESTRY = 8; + +/** Whether `super` calls of `language` are read where each call starts (isJsSuperCallSite). */ +export function isJsSuperCallLanguage(language: string): boolean { + return SUPER_CALL_LANGUAGES.has(language); +} + +/** + * Whether the call `ref` records is written on `super` where it starts: + * `super.run(…)`, the member on the next line or two included. Only the + * call's own position counts, so a `this.run()` on the same line is not one. + */ +export function isJsSuperCallSite(ref: UnresolvedRef, context: ResolutionContext): boolean { + if (ref.referenceKind !== 'calls' || !SUPER_CALL_LANGUAGES.has(ref.language) || !IDENTIFIER.test(ref.referenceName)) return false; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); + const line = lines?.[ref.line - 1]; + if (!line || !line.startsWith('super', ref.column)) return false; + const written = new RegExp(String.raw`^super\s*\.\s*${escapeRegExp(ref.referenceName)}(?![\w$])`); + if (written.test(line.slice(ref.column))) return true; + return written.test(lines!.slice(ref.line - 1, ref.line + 2).join('\n').slice(ref.column)); +} + +/** + * The method a TS/JS `super.name(…)` call runs (see the file comment). + * Undefined when the call isn't written on `super`, or the index can't place + * it in a member of a class (an object literal's method, a class + * expression's), which leaves it to the other strategies; null when no + * ancestor the project declares has the method. + */ +export function matchJsSuperCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + if (!isJsSuperCallSite(ref, context) || !context.getSuperclassNodes) return undefined; + const caller = context.getNodeById?.(ref.fromNodeId); + const placed = caller ? classMemberAround(caller, context) : undefined; + if (!placed || writesClassBefore(placed.member, ref, context)) return undefined; + const target = inheritedMethod(placed.cls, ref.referenceName, placed.member.isStatic === true, context); + return target ? { original: ref, targetNodeId: target.id, confidence: 0.9, resolvedBy: 'instance-method' } : null; +} + +/** + * The class member a call made in `caller` belongs to, and its class: the + * caller itself when it is a method or a field of a class, or the member an + * arrow function nested in it is written in (`run() { const go = () => + * super.run(); }`), whose `super` is the member's. A `function` can't use + * `super`, so a function node named under a member is an arrow function. + */ +function classMemberAround(caller: Node, context: ResolutionContext): { member: Node; cls: Node } | undefined { + let member = caller; + for (let depth = 0; member.kind === 'function' && depth < 4; depth++) { + const outer = ownerOf(member, context, (n) => n.kind === 'function' || n.kind === 'method' || n.kind === 'property'); + if (!outer) return undefined; + member = outer; + } + if (member.kind !== 'method' && member.kind !== 'property') return undefined; + const cls = ownerOf(member, context, (n) => n.kind === 'class'); + return cls ? { member, cls } : undefined; +} + +/** The node `node` is declared in, by its qualified name, in its file and around it. */ +function ownerOf(node: Node, context: ResolutionContext, accept: (n: Node) => boolean): Node | undefined { + const cut = node.qualifiedName.lastIndexOf('::'); + if (cut <= 0) return undefined; + return context.getNodesByQualifiedName(node.qualifiedName.slice(0, cut)).find((n) => accept(n) && + n.filePath === node.filePath && n.startLine <= node.startLine && n.endLine >= node.endLine); +} + +/** + * Whether a `class` is written between where `member` starts and the call, + * outside comments and strings: a class expression's methods have no nodes, + * so a call in one is recorded from the member around it (`make() { return + * class extends Other { run() { super.run(); } }; }`), and its `super` is + * that class's. + */ +function writesClassBefore(member: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); + if (!lines || member.startLine > ref.line) return false; + const span = lines.slice(member.startLine - 1, ref.line); + span[span.length - 1] = (span[span.length - 1] ?? '').slice(0, ref.column); + span[0] = (span[0] ?? '').slice(member.startLine === ref.line ? Math.min(member.startColumn, ref.column) : member.startColumn); + const text = span.join('\n'); + if (!/(?([cls.id]); + let level = superclassesOf(cls, context); + for (let depth = 0; depth < MAX_ANCESTRY && level.length > 0; depth++) { + const next: Node[] = []; + for (const parent of level) { + if (seen.has(parent.id)) continue; + seen.add(parent.id); + const own = ownMethod(parent, name, isStatic, context); + if (own) return own; + next.push(...superclassesOf(parent, context)); + } + level = next; + } + return undefined; +} + +function superclassesOf(cls: Node, context: ResolutionContext): Node[] { + return (context.getSuperclassNodes?.(cls.id) ?? []).filter((n) => n.kind === 'class' && SUPER_CALL_LANGUAGES.has(n.language)); +} + +/** + * A method `cls` itself declares under `name`, static or instance as asked. + * Of a TypeScript method's overload signatures and its body, the body, which + * comes last. + */ +function ownMethod(cls: Node, name: string, isStatic: boolean, context: ResolutionContext): Node | undefined { + const own = context.getNodesByQualifiedName(`${cls.qualifiedName}::${name}`).filter((m) => + m.kind === 'method' && m.filePath === cls.filePath && m.startLine >= cls.startLine && m.endLine <= cls.endLine && + (m.isStatic === true) === isStatic); + return own.sort((a, b) => a.startLine - b.startLine || a.startColumn - b.startColumn).pop(); +} diff --git a/src/resolution/types.ts b/src/resolution/types.ts index 8caf285f20..b4e5dae4d5 100644 --- a/src/resolution/types.ts +++ b/src/resolution/types.ts @@ -175,6 +175,13 @@ export interface ResolutionContext { * Optional so external/test contexts compile without it. */ getSupertypeNodes?(id: string): Node[]; + /** + * The classes one class declaration extends, through its own resolved + * `extends` edges alone: the parent a `super.m()` call reaches, never an + * interface or a class it only implements (#2497). Optional so + * external/test contexts compile without it. + */ + getSuperclassNodes?(id: string): Node[]; /** * Look up a node by its id. Lets matchers derive the FROM-symbol's * enclosing-class scope (Swift implicit-self method scoping, `this.X` From df1cd1e18596e393bfd09f23d44fd5fe00a68724 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 22:42:08 +0000 Subject: [PATCH 48/95] fix(pascal): Delphi units with re-published properties and similar syntax parse whole (#2816) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SynEdit's main unit parsed into one ERROR node on main. A re-published property, `property OwnsObjects;` at line 303, is one of several Delphi constructs tree-sitter-pascal has no rule for. Each breaks the parse locally, and enough of them cost a whole unit: routines after a break lost their calls or had them credited to the file, their locals showed up as unit variables, and class members after an unnamed record went missing. The Pascal pre-parse now fixes each construct in place, offsets and newlines kept (signatures still read the source as written): - a re-published property (`property Cursor default crIBeam;`) is blanked; - a conditional right after a routine's header that holds no whole body (`{$IFDEF HASINLINE}inline;{$ENDIF}`) has its directives blanked, and so does one nested at the start of a conditional that holds no block; - branches that open the same `begin` keep only the first one, and an assembler version that opens a body beside a Pascal one is blanked; - a re-raise is blanked, or written as `;` right before an `end`; - an unnamed record type (`fIn: record … end`) is written as `string` with its fields blanked, so the declaration keeps its name; - a label right before `end` is blanked; - alternative statements after a case label, or after then/else/do without a `;`, read as one block: the directives become begin, `;` and end, so every branch keeps its calls. The last fix also undoes a regression the header fix caused alone: in HeidiSQL's SynRegExpr.pas a case statement full of `{$IFDEF}` alternatives broke the unit once main's accidental resync points were gone (659 references lost). With all fixes it has 1,712 against main's 1,681. Also fixed on the way: a routine nested in another credited the constants its parent's body reads to itself, when it was the last one before the body (the value-ref scan pulled in the next sibling, which is the outer routine's body). Validation: pascal-delphi-syntax (17 tests, 13 fail on main) and a new pascal-nested-routines case (fails on main); 14 mutations, one per rule and three guards, each fail a test. Two pascal-system-calls fixtures broke the parse on purpose with constructs this now reads; they use an old-style `begin … end.` initialization and an anonymous record holding a method instead, which still break it, and still fail when the guard they cover is removed. Parse census, main -> fix, files without a unit node: SynEdit 4 -> 1, HeidiSQL 6 -> 1, DelphiMVCFramework 15 -> 1, mORMot2 34 -> 6, DWScript 2100 -> 2086 (its test scripts have no unit header). Indexed A/B, edges removed/added: SynEdit 1889/2150, HeidiSQL 2571/3305, DelphiMVCFramework 4915/6227, mORMot2 15298/27195, DWScript 3758/4570; leveldb and SDWebImage identical. No file loses more call sites than it gains outside DelphiMVCFramework, where the six that do lose only wrong edges. Index time within noise (HeidiSQL +3% wall, +8% CPU for the code now read); the pre-parse itself takes 223 ms over mORMot2's 25 MB. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/pascal-delphi-syntax.test.ts | 521 +++++++++++++ __tests__/pascal-nested-routines.test.ts | 38 + __tests__/pascal-system-calls.test.ts | 21 +- src/extraction/languages/pascal-preparse.ts | 821 +++++++++++++++++++- src/extraction/tree-sitter.ts | 6 +- 6 files changed, 1360 insertions(+), 49 deletions(-) create mode 100644 __tests__/pascal-delphi-syntax.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 82873bfcee..0d43a8c3f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Delphi, a routine called without parentheses, the usual way to call one that takes no arguments, like `Before;`, `if Ready then`, `Result := FProvider.Value + 1` or `QueryTabs.ActiveTab.MemoFilename`, now counts as a call, so it shows up in callers, callees and impact, and a method given to an event, like `OnClick := ButtonClick`, is linked as a reference. A name is linked only to a method or function its code reaches, in its class and the classes it inherits from (`class sealed (TBase)` included), its unit, or the units it uses, never to a field, property or variable of that name. Thanks @OleksiyPenkov for the report, and @drakeo338 and @OleksiyPenkov for their pull requests. (#2514) - In Delphi, a call to something the compiler provides, like `SizeOf(X)`, `TypeInfo(TFoo)`, `High(A)`, `Inc(N)`, `Assert(…)` or a typecast like `Pointer(P)`, written in any letter case, no longer shows up as a call to a project method, field or variable that happens to share its name, unless the calling code's own class or units declare one, like a list class's own `Delete`. A call through a field, property or variable now counts only when it holds a routine, like an `OnChange` event or a `reference to function` variable, and calls in `{$I}` include files and into units the parser could only partly read now find the routines those units declare. Thanks @OleksiyPenkov for the report. (#2514) - In Delphi, a routine declared inside another, between its header and its `begin`, is now indexed as a routine of its own under the one it belongs to (`TFoo.Bar`'s `Inner` is `TFoo::Bar::Inner`), with the calls its body makes. Before, its body was skipped, so what it called was missing from callers and impact, and a call to it from its own routine went to any routine of the same name elsewhere. Thanks @OleksiyPenkov for the pull request this follows. (#2550) +- In Delphi, a unit that re-publishes an inherited property, like `property OwnsObjects;` or `property Cursor default crIBeam;`, is now indexed whole. So is one with a conditional right after a routine's header (`{$IFDEF HASINLINE}inline;{$ENDIF}`), an assembler and a Pascal version of one routine, a bare `raise`, a field or variable of an unnamed record type (`fIn: record … end`), a label just before `end`, or two versions of a statement in one `case` branch. Before, the parse broke at such a line, so the routines and class members after it went missing or their calls were credited to the file, which cost SynEdit's main unit most of its code. +- In Delphi, a constant read in a routine's body is no longer credited to the last routine nested inside it. - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. diff --git a/__tests__/pascal-delphi-syntax.test.ts b/__tests__/pascal-delphi-syntax.test.ts new file mode 100644 index 0000000000..639e112125 --- /dev/null +++ b/__tests__/pascal-delphi-syntax.test.ts @@ -0,0 +1,521 @@ +/** + * Delphi syntax the Pascal grammar has no rule for, which broke whole units + * of SynEdit, mORMot2, DWScript and HeidiSQL: a property re-published + * without its type (`property OwnsObjects;`, SynEdit.pas from line 303), a + * conditional right after a routine's header that holds no whole body, two + * branches that open the same `begin`, a re-raise, a field or variable of an + * anonymous record type, a label right before `end`, and alternative + * statements where the grammar takes one (`OP_X: {$IFDEF U} … {$ELSE} … + * {$ENDIF}`). The extractor's preParse blanks each, or writes a word the + * grammar knows in its place, offsets kept, so the unit parses whole and the + * routines after it keep their calls. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { getParser, initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { preParsePascal } from '../src/extraction/languages/pascal-preparse'; + +beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['pascal']); +}); + +/** Whether the unit parses whole: a `unit` node and no error anywhere. */ +function parsesWhole(code: string): boolean { + const tree = getParser('pascal')!.parse(preParsePascal(code))!; + try { + return !tree.rootNode.hasError && tree.rootNode.namedChildren.some((c) => c.type === 'unit'); + } finally { + tree.delete(); + } +} + +/** Every routine's calls, as `Caller->Callee`, and every node's qualified name, from the unit's extraction. */ +function extract(code: string, file: string): { calls: string[]; names: string[] } { + const result = extractFromSource(file, code); + const byId = new Map(result.nodes.map((n) => [n.id, n])); + return { + calls: result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => `${byId.get(r.fromNodeId)?.name}->${r.referenceName}`), + names: result.nodes.map((n) => n.qualifiedName), + }; +} + +const OVERRIDES = `unit Marks; + +interface + +type + TMarkList = class(TObjectList) + protected + function GetItem(Index: Integer): TObject; + property OwnsObjects; + public + constructor Create; + property Items[Index: Integer]: TObject read GetItem; default; + end; + + TEditor = class(TCustomControl) + published + property Align; + property Cursor default crIBeam; + class property Tag; + property Lines: TStrings read FLines; + procedure Paint; + end; + +implementation + +procedure TEditor.Paint; +begin + Refresh(1); +end; + +end. +`; + +const HEADERS = `unit Headers; + +interface + +procedure KeyUp; +function Size: Integer; +procedure Generate; + +implementation + +procedure KeyUp; +{$IFDEF LINUX} +var + Code: Byte; +{$ENDIF} +begin + Translate(1); +end; + +function Size: Integer; {$IFDEF HASINLINE}inline;{$ENDIF} +begin + Result := Measure(1); +end; + +procedure Generate; +{$IFDEF LLVM} +{$IFDEF LLVM_EXECUTE} +var + Engine: Pointer; +{$ENDIF} +{$ENDIF} +begin + Emit(1); +end; + +end. +`; + +const BEGINS = `unit Begins; + +interface + +procedure Click; +procedure After; + +implementation + +procedure Click; +var + I: Integer; +{$IFDEF FOLDING} + Index: Integer; +begin + Fold(1); + if I > 0 then + begin + Exit; + end; +{$ELSE} +begin +{$ENDIF} + Paint(1); +end; + +procedure After; +begin + Finish(1); +end; + +end. +`; + +const ASSEMBLER = `unit Assembler; + +interface + +function Log2Of(const V: Double): Double; +function DivMod10(var Dividend: Cardinal): Cardinal; +procedure Fill(P: Pointer); +procedure After; + +implementation + +procedure Fill(P: Pointer); +{$IFDEF FPC}nostackframe; assembler; asm {$ELSE} asm .noframe {$ENDIF} + mov eax, 1 +end; + +function Log2Of(const V: Double): Double; +{$IFDEF WIN32_ASM} +asm + fld1 + fld V + fyl2x +{$ELSE} +begin + Result := Log2(V); +{$ENDIF} +end; + +function DivMod10(var Dividend: Cardinal): Cardinal; +{$IFNDEF WIN32_ASM} +var + Divided: Cardinal; +begin + Divided := Dividend div 10; + Result := Remainder(Dividend, Divided); +{$ELSE} +const + C10: Cardinal = 10; +asm + mov ecx, eax + div C10 +{$ENDIF} +end; + +procedure After; +begin + Finish(1); +end; + +end. +`; + +const RERAISES = `unit Reraises; + +interface + +procedure Load; +procedure Parse; + +implementation + +procedure Load; +begin + try + Read(1); + except + Log(1); + raise; + end +end; + +procedure Parse; +begin + try + Scan(1); + except + on E: EConvertError do + Recover(1); + else raise + end; +end; + +end. +`; + +const RECORDS = `unit Records; + +interface + +type + TClient = class + private + fIn: record + Headers: Pointer; + URL, Method: string; + end; + fRows: array of packed record + Id: Integer; + end; + fCount: Integer; + public + procedure Send; + end; + + TCaches = array[1..4] of record + Size: Integer; + end; + + TState = record + Seed: Integer; + Data: record + case Integer of + 0: (Lc: Pointer) + end + end; + +var + PosixInject: record + GetSmbios: function(Index: Integer): string; + OnIdle: procedure of object; + end; + Ready: Boolean; + +const + Origin: record X, Y: Integer; end = (X: 0; Y: 0); + +implementation + +procedure TClient.Send; +begin + Post(fIn.URL); +end; + +end. +`; + +const LABELS = `unit Labels; + +interface + +function Find(P: PChar): Integer; +procedure After; + +implementation + +function Find(P: PChar): Integer; +label + fnd, quit; +begin + Result := 0; + if P = nil then + goto quit; + if Match(P) then + goto fnd; + goto quit; +fnd: + Result := 1; +quit: +end; + +procedure After; +begin + Finish(1); +end; + +end. +`; + +const ALTERNATIVES = `unit Alternatives; + +interface + +function Repeated(Op: Integer): Integer; +function Pick(Lookup: Pointer): Integer; + +implementation + +function Repeated(Op: Integer): Integer; +begin + Result := 0; + case Op of + 1: + {$IFDEF UNICODE} + begin + CountWide(Result); + end; + {$ELSE} + while Result < 10 do + CountAnsi(Result); + {$ENDIF} + 2: + Result := 2; + end; +end; + +function Pick(Lookup: Pointer): Integer; +begin + if Lookup = nil then + {$IFDEF ASMX86} + Result := PosAsm(0) + {$ELSE} + Result := PosPas(0) + {$ENDIF} + else + Result := PosLookup(Lookup); +end; + +end. +`; + +describe('Delphi syntax the Pascal grammar has no rule for', () => { + it.each([ + ['re-published properties', OVERRIDES], + ['conditionals after routine headers', HEADERS], + ['branches that open the same begin', BEGINS], + ['an assembler and a Pascal version of a body', ASSEMBLER], + ['re-raises', RERAISES], + ['anonymous records', RECORDS], + ['a label that ends a block', LABELS], + ['alternative statements where one goes', ALTERNATIVES], + ])('parses a unit whole past %s', (_name, code) => { + expect(parsesWhole(code)).toBe(true); + }); + + it('keeps a class whole past its re-published properties', () => { + const { calls, names } = extract(OVERRIDES, 'Marks.pas'); + expect(names).toEqual(expect.arrayContaining([ + 'TMarkList::GetItem', 'TMarkList::Create', 'TMarkList::Items', 'TEditor::Lines', 'TEditor::Paint', + ])); + // An override declares nothing new. + expect(names).not.toContain('TMarkList::OwnsObjects'); + expect(names).not.toContain('TEditor::Align'); + expect(calls).toEqual(['Paint->Refresh']); + }); + + it('gives each routine its calls past a conditional after its header', () => { + expect(extract(HEADERS, 'Headers.pas').calls.sort()).toEqual(['Generate->Emit', 'KeyUp->Translate', 'Size->Measure']); + expect(extract(BEGINS, 'Begins.pas').calls.sort()).toEqual(['After->Finish', 'Click->Fold', 'Click->Paint']); + expect(extract(ASSEMBLER, 'Assembler.pas').calls.sort()).toEqual(['After->Finish', 'DivMod10->Remainder', 'Log2Of->Log2']); + }); + + it('gives each routine its calls past a re-raise and a label that ends a block', () => { + expect(extract(RERAISES, 'Reraises.pas').calls.sort()).toEqual(['Load->Log', 'Load->Read', 'Parse->Recover', 'Parse->Scan']); + expect(extract(LABELS, 'Labels.pas').calls.sort()).toEqual(['After->Finish', 'Find->Match']); + }); + + it('keeps a declaration of an anonymous record type, and every declaration after it', () => { + const { calls, names } = extract(RECORDS, 'Records.pas'); + expect(names).toEqual(expect.arrayContaining([ + 'TClient::fIn', 'TClient::fRows', 'TClient::fCount', 'TClient::Send', 'TCaches', 'TState::Seed', 'TState::Data', + 'PosixInject', 'Ready', 'Origin', + ])); + // The record's own fields are no members of the class around it. + expect(names).not.toContain('TClient::URL'); + expect(names).not.toContain('GetSmbios'); + expect(calls).toEqual(['Send->Post']); + }); + + it('reads alternative statements where one goes as one block, keeping the calls of each', () => { + expect(extract(ALTERNATIVES, 'Alternatives.pas').calls.sort()).toEqual([ + 'Pick->PosAsm', 'Pick->PosLookup', 'Pick->PosPas', 'Repeated->CountAnsi', 'Repeated->CountWide', + ]); + }); + + it('keeps every offset and line, CRLF included', () => { + for (const code of [OVERRIDES, HEADERS, BEGINS, ASSEMBLER, RERAISES, RECORDS, LABELS, ALTERNATIVES]) { + const crlf = code.replace(/\n/g, '\r\n'); + const out = preParsePascal(crlf); + expect(out.length).toBe(crlf.length); + const breaks = (s: string) => [...s.matchAll(/\r\n/g)].map((m) => m.index); + expect(breaks(out)).toEqual(breaks(crlf)); + } + }); + + it('writes the words it needs where the grammar reads them', () => { + const blank = (s: string) => s.replace(/[^\r\n]/g, ' '); + expect(preParsePascal(' 1: {$IFDEF U} A(1); {$ELSE} B(2); {$ENDIF}\n')) + .toBe(' 1: begin A(1); ; B(2); end; \n'); + expect(preParsePascal(' Data: record X: Integer; end;\n')).toBe(` Data: string${blank(' X: Integer; end')};\n`); + expect(preParsePascal(' Data: packed record X: Integer; end end;\n')).toBe(` Data: string${blank(' record X: Integer; ')}; end;\n`); + expect(preParsePascal(' else raise\n end;\n')).toBe(' else ; \n end;\n'); + }); + + it('leaves alone what the grammar reads as it is', () => { + const code = `unit Keep; + +interface + +type + TList = class + property Items[Index: Integer]: T read GetItem; default; + property Count: Integer read FCount; + Value: {$IFDEF CPU64} Int64 {$ELSE} Integer {$ENDIF}; + end; + TRec = record + X: Integer; + end; + +function Load: Integer; + +implementation + +function Load: Integer; +{$IFDEF PUREPASCAL} +var + I: Integer; +begin + Result := Read(I); +end; +{$ELSE} +asm + mov eax, 1 +end; +{$ENDIF} + +procedure Run; +begin + with List do + {$IFDEF CPU64} + if A = B then + {$ELSE} + if C = D then + {$ENDIF} + Exit; + if Ready then {$IFDEF X} Go(1); {$ELSE} Stay(1); {$ENDIF} + raise EAbort.Create('stop'); + WriteLn('raise; property X; quit:'); +end; + +end. +`; + expect(preParsePascal(code)).toBe(code); + }); + + it('links a routine past a conditional after its header to what it calls', async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-delphi-syntax-')); + try { + fs.writeFileSync(path.join(dir, 'Lib.pas'), `unit Lib; + +interface + +procedure Translate(N: Integer); +procedure Emit(N: Integer); + +implementation + +procedure Translate(N: Integer); begin end; +procedure Emit(N: Integer); begin end; + +end. +`); + fs.writeFileSync(path.join(dir, 'Headers.pas'), HEADERS.replace('implementation\n', 'implementation\n\nuses Lib;\n')); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const callers = (name: string) => { + const target = cg.getNodesByKind('function').find((n) => n.qualifiedName === name && n.filePath === 'Lib.pas'); + return target ? cg.getIncomingEdges(target.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.source)?.name) : []; + }; + expect(callers('Translate')).toEqual(['KeyUp']); + expect(callers('Emit')).toEqual(['Generate']); + } finally { + cg.close(); + } + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/__tests__/pascal-nested-routines.test.ts b/__tests__/pascal-nested-routines.test.ts index 7c594db952..cc3cf43247 100644 --- a/__tests__/pascal-nested-routines.test.ts +++ b/__tests__/pascal-nested-routines.test.ts @@ -360,4 +360,42 @@ end;`); expect(callsFrom(cg, 'THost::Fire')).toEqual(['Host.pas:THost::Fire::Calc', 'Host.pas:THost::Fire::Delete']); }); }); + + it('gives a nested routine the constants it reads, not those its routine\'s body reads after it', async () => { + const code = `unit Values; + +interface + +const + MAX_DEPTH = 10; + OP_EXACTLY = 3; + OP_ANY = 4; + +function Parse(N: Integer): Integer; + +implementation + +function Parse(N: Integer): Integer; + + function Unquote(C: Integer): Integer; + begin + Result := C + OP_ANY; + end; + +begin + if N > MAX_DEPTH then + Result := OP_EXACTLY + else + Result := Unquote(N); +end; + +end. +`; + await withIndex({ 'Values.pas': code }, (cg) => { + const reads = cg.getOutgoingEdges(routine(cg, 'Parse::Unquote').id) + .filter((e) => e.kind === 'references') + .map((e) => cg.getNode(e.target)!.name); + expect(reads).toEqual(['OP_ANY']); + }); + }); }); diff --git a/__tests__/pascal-system-calls.test.ts b/__tests__/pascal-system-calls.test.ts index 6714ee974f..a3741e623d 100644 --- a/__tests__/pascal-system-calls.test.ts +++ b/__tests__/pascal-system-calls.test.ts @@ -483,7 +483,7 @@ end. }); it('sees the declarations of a used unit whose header the parser lost', async () => { - // The `{$IFDEF}` around a local `var` section costs the parse its `unit` + // An old-style `begin … end.` initialization costs the parse its `unit` // node; the unit is still the one its file is named after. const lib = `unit Lib; @@ -500,14 +500,12 @@ function FileSize(const FileName: string): Int64; implementation function FileSize(const FileName: string): Int64; -{$IFDEF MSWINDOWS} -var - Handle: THandle; -{$ENDIF} begin Result := 0; end; +begin + crc32c := nil; end. `; const main = `unit Main; @@ -606,16 +604,18 @@ end; }); it('takes no member of a class such a unit\'s parse lost for a routine of the unit', async () => { - // The anonymous record ends the class early, so `Insert` comes out as a - // routine of the unit, indented as no routine of the unit is. + // The anonymous record holding a method, which the parser can't read, + // ends the class early, so `Insert` comes out as a routine of the unit, + // indented as no routine of the unit is. const lib = `unit Lib; interface type TClient = class - Items: array of record + Items: record Name: string; + procedure Clear; end; procedure Insert(Index: Integer; const S: string); end; @@ -627,13 +627,10 @@ implementation procedure TClient.Insert(Index: Integer; const S: string); begin end; procedure Log(const S: string); -{$IFDEF MSWINDOWS} -var - Handle: THandle; -{$ENDIF} begin end; +begin end. `; const main = `unit Main; diff --git a/src/extraction/languages/pascal-preparse.ts b/src/extraction/languages/pascal-preparse.ts index 5d907f3ef1..8e3675b209 100644 --- a/src/extraction/languages/pascal-preparse.ts +++ b/src/extraction/languages/pascal-preparse.ts @@ -1,8 +1,8 @@ /** * Offset-preserving source fixes the Pascal grammar needs before it parses a * Delphi unit (the extractor's `preParse`). Each one blanks syntax the grammar - * has no rule for with spaces, keeping every newline, so node positions and - * text stay where they are. + * has no rule for with spaces, or writes a word it knows in its place, keeping + * every newline, so node positions and text stay where they are. * * - An attribute on a routine parameter, before or after its modifier: * `const [ref] AValue: Integer`, `[ref] const Source: TVarData`, @@ -17,6 +17,46 @@ * call in its body was read. Blanking `var` and the type leaves the loop * the grammar knows. The resolver still finds the type: it reads a * receiver's declaration from the source as written. + * - A property override, which names an inherited property without its type + * to publish it or change a specifier: `property OwnsObjects;`, `property + * Cursor default crIBeam;`. The grammar's property rule needs the type, so + * the class fell apart at the first override and every member declared + * after it was lost. An override declares nothing new to index. + * - A conditional right after a routine's header that doesn't hold whole + * bodies: `procedure KeyUp(…); {$IFDEF LINUX} var Code: Byte; {$ENDIF}`, + * `function Get: Integer; {$IFDEF HASINLINE}inline;{$ENDIF}`. The grammar + * reads a conditional there as alternative bodies only, so any other one + * broke the whole unit. Blanking its directives leaves its branches as + * plain code, which is how the grammar reads a conditional anywhere else. + * - A branch holding only a `begin` that another branch of its conditional + * opens too: `{$IFDEF FOLDING} Index: Integer; begin … {$ELSE} begin + * {$ENDIF}`. Read together, the branches open one block more than the code + * closes, so every routine after it was lost. The other branch's `begin` + * stays. An assembler version that opens the body beside a Pascal one, + * `{$IFDEF WIN32_ASM} asm … {$ELSE} begin … {$ENDIF} end;`, is blanked + * whole, so the Pascal version is the body. + * - A field, variable or typed constant of an anonymous record type: + * `AlgData: record case … end`, `Stack: array[1..32] of record First, Last: + * LongInt; end;`. The grammar knows a record only as a named type, so the + * record's `end` closed the class around it and the members declared after + * it were lost. The `record` (or the `packed` before it) is written over by + * `string` and the fields are blanked, so the declaration keeps its name: + * the fields belong to no type the graph could index them under. + * - A re-raise, `raise` with no exception (`except Log; raise; end`). The + * grammar's `raise` takes an exception, so it read the next word (often + * the `end`) as one and the unit broke. The `raise` is blanked, or written + * over by `;` right before an `end` (`else raise end`), where the grammar + * needs a statement to end the list. + * - A label that ends its block, right before `end`, `until`, `except` or + * `finally` (`quit: end;`). The grammar takes a label only before a + * statement, so it read the `end` as a name and the routine never closed, + * taking the routines after it along. The graph reads no labels. + * - Alternative statements where the grammar takes one: `OP_X: {$IFDEF U} + * begin … end; {$ELSE} while … do Inc(N); {$ENDIF}` after a case label, or + * `then {$IFDEF X} A := 1 {$ELSE} A := 2 {$ENDIF} else …`. Read together, + * the branches are two statements, so the case arm broke and took the + * routines after it along. The conditional's directives become `begin`, `;` + * and `end`, which makes its branches one block and keeps the code of each. */ const MODIFIERS: ReadonlySet = new Set(['const', 'var', 'out', 'constref']); @@ -27,13 +67,54 @@ const MAX_ATTRIBUTE_LENGTH = 2000; const MAX_LOOP_TYPE_LENGTH = 500; /** Words that end a for-in variable's type without an `in`: not the for-in form. */ const NOT_A_FOR_IN: ReadonlySet = new Set(['do', 'to', 'downto', 'begin', 'end']); -/** Whether a file can hold a `for var … in` loop at all (the scan below decides). */ -const FOR_VAR = /\bfor\s+var\b/i; +/** + * Whether a file can hold anything to fix at all (the scan below decides): a directive, an attribute, a `for var` + * loop, a property override, an anonymous record, a re-raise or a label (only a `goto` needs one). + */ +const MAY_NEED_FIXES = /\{\$|\[|\bfor\s+var\b|\bproperty\b|\brecord\b|\braise\b|\bgoto\b/i; +/** A property override longer than this, up to its `;`, is not one. */ +const MAX_OVERRIDE_LENGTH = 500; +/** A routine header's rest after its parameters, or a hint after it (`message WM_SIZE;`), longer than this is not one. */ +const MAX_HEADER_PART_LENGTH = 500; +/** An anonymous record longer than this, `record` to `end`, is left alone. */ +const MAX_RECORD_LENGTH = 20000; +/** The hints a routine's header can carry after its `;`: the grammar's procedure attributes. */ +const ROUTINE_HINTS: ReadonlySet = new Set([ + 'static', 'virtual', 'dynamic', 'abstract', 'override', 'overload', 'reintroduce', 'inline', + 'stdcall', 'cdecl', 'pascal', 'register', 'safecall', 'assembler', 'noreturn', 'local', 'far', + 'near', 'default', 'nodefault', 'deprecated', 'experimental', 'message', 'platform', + 'unimplemented', 'cppdecl', 'cvar', 'mwpascal', 'nostackframe', 'interrupt', 'iocheck', + 'hardfloat', 'softfloat', 'ms_abi_default', 'ms_abi_cdecl', 'saveregisters', 'sysv_abi_default', + 'sysv_abi_cdecl', 'vectorcall', 'varargs', 'winapi', 'public', 'export', 'alias', 'dispid', +]); +/** Words that end a block's statements, so a label right before one labels no statement, and a `raise` re-raises. */ +const LABEL_ENDS: ReadonlySet = new Set(['end', 'until', 'except', 'finally']); +/** Words that open a block an `end` closes, for telling whether a conditional branch leaves one open. */ +const BLOCK_OPENERS: ReadonlySet = new Set(['begin', 'asm', 'try', 'case', 'record']); +/** Words an anonymous record's fields can't hold: a `record` before them is something else. */ +const NOT_IN_RECORD_FIELDS: ReadonlySet = new Set([ + 'begin', 'class', 'object', 'interface', 'implementation', 'procedure', 'function', 'constructor', + 'destructor', 'property', 'type', 'var', 'const', 'private', 'protected', 'public', 'published', 'strict', +]); +/** Words that start a statement, for telling a conditional's branch holds statements rather than a type. */ +const STATEMENT_KEYWORDS: ReadonlySet = new Set([ + 'begin', 'if', 'while', 'for', 'repeat', 'case', 'try', 'with', 'raise', 'inherited', 'goto', 'asm', +]); +/** Words a type can start with, so a name before `(` that is one of them starts no call. */ +const TYPE_KEYWORDS: ReadonlySet = new Set([ + 'procedure', 'function', 'reference', 'class', 'record', 'object', 'interface', 'array', 'set', 'file', + 'packed', 'string', 'type', 'const', 'var', 'out', +]); +/** Words a statement can't end with: a branch ending in one holds a statement's head or a part of an expression. */ +const STATEMENT_CONTINUES: ReadonlySet = new Set([ + 'then', 'do', 'else', 'of', 'and', 'or', 'xor', 'not', 'div', 'mod', 'shl', 'shr', 'in', 'is', 'as', 'to', + 'downto', 'until', 'goto', 'on', +]); -const CH_TAB = 9, CH_LF = 10, CH_CR = 13, CH_SPACE = 32, CH_AMP = 38, CH_QUOTE = 39; -const CH_LPAREN = 40, CH_RPAREN = 41, CH_STAR = 42, CH_DOT = 46, CH_SLASH = 47; -const CH_COLON = 58, CH_SEMI = 59, CH_LT = 60, CH_EQ = 61, CH_GT = 62; -const CH_LBRACKET = 91, CH_RBRACKET = 93, CH_LBRACE = 123; +const CH_TAB = 9, CH_LF = 10, CH_CR = 13, CH_SPACE = 32, CH_DOLLAR = 36, CH_AMP = 38, CH_QUOTE = 39; +const CH_LPAREN = 40, CH_RPAREN = 41, CH_STAR = 42, CH_COMMA = 44, CH_DOT = 46, CH_SLASH = 47; +const CH_COLON = 58, CH_SEMI = 59, CH_LT = 60, CH_EQ = 61, CH_GT = 62, CH_AT = 64; +const CH_LBRACKET = 91, CH_RBRACKET = 93, CH_LBRACE = 123, CH_RBRACE = 125; /** Identifier characters, `&` (Delphi's keyword escape) and any non-ASCII letter. */ function isIdentStart(c: number): boolean { @@ -42,6 +123,9 @@ function isIdentStart(c: number): boolean { function isIdentPart(c: number): boolean { return (c >= 65 && c <= 90) || (c >= 97 && c <= 122) || c === 95 || (c >= 48 && c <= 57) || c >= 128; } +function isSpace(c: number): boolean { + return c === CH_SPACE || c === CH_TAB || c === CH_LF || c === CH_CR; +} function wordEnd(s: string, i: number): number { while (i < s.length && isIdentPart(s.charCodeAt(i))) i++; @@ -85,19 +169,70 @@ function commentOrStringEnd(s: string, i: number): number { return -1; } +/** Whether a comment starts at `i`. */ +function isCommentStart(s: string, i: number): boolean { + const c = s.charCodeAt(i); + return c === CH_LBRACE || (c === CH_LPAREN && s.charCodeAt(i + 1) === CH_STAR) || (c === CH_SLASH && s.charCodeAt(i + 1) === CH_SLASH); +} + /** The index of the first character from `i` that is neither whitespace nor in a comment. */ function skipTrivia(s: string, i: number): number { while (i < s.length) { - const c = s.charCodeAt(i); - if (c === CH_SPACE || c === CH_TAB || c === CH_LF || c === CH_CR) { - i++; - } else if (c === CH_LBRACE || (c === CH_LPAREN && s.charCodeAt(i + 1) === CH_STAR) || (c === CH_SLASH && s.charCodeAt(i + 1) === CH_SLASH)) { - i = commentOrStringEnd(s, i); + if (isSpace(s.charCodeAt(i))) i++; + else if (isCommentStart(s, i)) i = commentOrStringEnd(s, i); + else break; + } + return i; +} + +/** The index after the last character before `i` that is neither whitespace nor in a `{…}` or `(*…*)` comment. */ +function tokenEndBefore(s: string, i: number): number { + for (;;) { + while (i > 0 && isSpace(s.charCodeAt(i - 1))) i--; + if (i > 0 && s.charCodeAt(i - 1) === CH_RBRACE) { + const open = s.lastIndexOf('{', i - 2); + if (open < 0) return i; + i = open; + } else if (i > 1 && s.charCodeAt(i - 1) === CH_RPAREN && s.charCodeAt(i - 2) === CH_STAR) { + const open = s.lastIndexOf('(*', i - 3); + if (open < 0) return i; + i = open; } else { - break; + return i; } } - return i; +} + +/** The start of the identifier that ends at `i`, or -1 when none does. */ +function wordStartBefore(s: string, i: number): number { + let k = i; + while (k > 0 && isIdentPart(s.charCodeAt(k - 1))) k--; + if (k === i) return -1; + if (k > 0 && s.charCodeAt(k - 1) === CH_AMP) k--; + return isIdentStart(s.charCodeAt(k)) ? k : -1; +} + +/** The index after the first `;` from `i` outside parentheses and brackets, or -1 when none comes within `max` characters. */ +function semicolonEnd(s: string, i: number, max: number): number { + const limit = Math.min(s.length, i + max); + let depth = 0; + while (i < limit) { + const k = commentOrStringEnd(s, i); + if (k >= 0) { + i = k; + continue; + } + const c = s.charCodeAt(i); + if (c === CH_LPAREN || c === CH_LBRACKET) { + depth++; + } else if (c === CH_RPAREN || c === CH_RBRACKET) { + depth--; + } else if (c === CH_SEMI && depth <= 0) { + return i + 1; + } + i++; + } + return -1; } /** @@ -268,25 +403,587 @@ function scanForInVariable(s: string, forEnd: number, ranges: number[]): number return i; } -/** `s` with every `[from, to)` range in `ranges` blanked to spaces, newlines kept. */ -function blankRanges(s: string, ranges: number[]): string { - if (ranges.length === 0) return s; +/** + * After a `property` keyword running from `start` to `propertyEnd`, + * recognize a property override, which names an inherited property without + * its type to publish it or change a specifier — `property OwnsObjects;`, + * `property Cursor default crIBeam;`, `property Color stored IsColorStored;` + * — and push its range, keyword (or the `class` before it) to `;`, onto + * `ranges`. A name followed by `:` or by an index list (`property + * Items[Index: Integer]: T`) declares a property the grammar knows. Returns + * the index to resume scanning from. + */ +function scanPropertyOverride(s: string, start: number, propertyEnd: number, ranges: number[]): number { + const nameStart = skipTrivia(s, propertyEnd); + if (!isIdentStart(s.charCodeAt(nameStart))) return propertyEnd; + const i = skipTrivia(s, wordEnd(s, nameStart + 1)); + const next = s.charCodeAt(i); + if (next === CH_COLON || next === CH_LBRACKET) return propertyEnd; + const end = semicolonEnd(s, i, MAX_OVERRIDE_LENGTH); + if (end < 0) return propertyEnd; + const classStart = classBefore(s, start); + ranges.push(classStart >= 0 ? classStart : start, end); + return end; +} + +/** + * A conditional compiler directive group, from its `{$IF…}`, `{$IFDEF…}`, + * `{$IFNDEF…}` or `{$IFOPT…}` to its `{$ENDIF}` or `{$IFEND}`. `bounds` + * holds the `[start, end)` offsets of the group's own directives in order, + * its `{$ELSE}`s and `{$ELSEIF…}`s between, so branch `b` runs from + * `bounds[2b + 1]` to `bounds[2b + 2]`; `nested[b]` says whether another + * group sits in that branch. + */ +interface ConditionalGroup { + bounds: number[]; + nested: boolean[]; +} + +const COND_NONE = 0, COND_OPEN = 1, COND_ELSE = 2, COND_CLOSE = 3; + +/** What the `{$…}` directive at `i` does to a conditional: opens one, starts its next branch, closes it, or none of these. */ +function conditionalKind(s: string, i: number): number { + let j = i + 2; + while (j < s.length) { + const c = s.charCodeAt(j) | 32; + if (c < 97 || c > 122) break; + j++; + } + switch (s.slice(i + 2, j).toLowerCase()) { + case 'if': case 'ifdef': case 'ifndef': case 'ifopt': return COND_OPEN; + case 'else': case 'elseif': return COND_ELSE; + case 'endif': case 'ifend': return COND_CLOSE; + default: return COND_NONE; + } +} + +/** + * Every closed conditional group in `s`, innermost first. Directives in + * strings and comments don't count, nor do `(*$…*)` ones, which the grammar + * reads as comments. + */ +function conditionalGroups(s: string): ConditionalGroup[] { + const groups: ConditionalGroup[] = []; + const open: ConditionalGroup[] = []; + const n = s.length; + let i = 0; + while (i < n) { + const c = s.charCodeAt(i); + const k = c === CH_LBRACE || c === CH_QUOTE || c === CH_LPAREN || c === CH_SLASH ? commentOrStringEnd(s, i) : -1; + if (k < 0) { + i++; + continue; + } + if (c === CH_LBRACE && s.charCodeAt(i + 1) === CH_DOLLAR) { + const kind = conditionalKind(s, i); + const top = open[open.length - 1]; + if (kind === COND_OPEN) { + if (top) top.nested[top.nested.length - 1] = true; + open.push({ bounds: [i, k], nested: [false] }); + } else if (kind === COND_ELSE && top) { + top.bounds.push(i, k); + top.nested.push(false); + } else if (kind === COND_CLOSE && top) { + top.bounds.push(i, k); + groups.push(open.pop()!); + } + } + i = k; + } + return groups; +} + +/** + * Whether the code from `from` to `to` is a routine's locals and body, the + * one conditional the grammar reads right after a routine's header: it ends + * with the `end;` of a block that no routine declared in it owns. Code that + * closes a block it didn't open (a local class type) counts as a body too, + * so it stays the way the grammar reads it. + */ +function holdsBody(s: string, from: number, to: number): boolean { + const blocks: string[] = []; + let pending = 0; // routines declared at the top whose body hasn't come yet + let own = false; // whether the last top-level block is no declared routine's + let last = ''; + let beforeLast = ''; + let i = from; + while (i < to) { + const k = commentOrStringEnd(s, i); + if (k >= 0) { + if (s.charCodeAt(i) === CH_QUOTE) { + beforeLast = last; + last = "'"; + } + i = k; + continue; + } + const c = s.charCodeAt(i); + if (isIdentStart(c)) { + const e = wordEnd(s, i + 1); + const word = e - i <= 11 ? s.slice(i, e).toLowerCase() : 'word'; + if (blocks.length === 0) { + if (ROUTINE_KEYWORDS.has(word) && (last === '' || last === ';')) pending++; + else if ((word === 'forward' || word === 'external') && pending > 0) pending--; + } + if (word === 'end') { + if (blocks.pop() === undefined) return true; + } else if (word === 'begin' || word === 'asm') { + if (blocks.length === 0) { + own = pending === 0; + if (!own) pending--; + } + blocks.push(word); + } else if (word === 'try' || word === 'record' || (word === 'case' && blocks[blocks.length - 1] !== 'record')) { + blocks.push(word); + } + beforeLast = last; + last = word; + i = e; + continue; + } + if (!isSpace(c)) { + beforeLast = last; + last = s[i]!; + } + i++; + } + return blocks.length === 0 && own && beforeLast === 'end' && last === ';'; +} + +/** Whether the code from `from` to `to` holds a `begin` or an `asm`: a body, or a part of one. */ +function holdsBlock(s: string, from: number, to: number): boolean { + let i = from; + while (i < to) { + const k = commentOrStringEnd(s, i); + if (k >= 0) { + i = k; + continue; + } + if (isIdentStart(s.charCodeAt(i))) { + const e = wordEnd(s, i + 1); + if (e - i === 3 || e - i === 5) { + const word = s.slice(i, e).toLowerCase(); + if (word === 'begin' || word === 'asm') return true; + } + i = e; + continue; + } + i++; + } + return false; +} + +/** The index of the first character from `i` that is neither whitespace nor in a comment, stopping at a conditional group's opening directive. */ +function skipTriviaToGroup(s: string, i: number, groupAt: ReadonlyMap): number { + while (i < s.length) { + if (isSpace(s.charCodeAt(i))) i++; + else if (isCommentStart(s, i) && !groupAt.has(i)) i = commentOrStringEnd(s, i); + else break; + } + return i; +} + +/** + * From `i`, the index after a routine header's `;`, blank the directives of + * each conditional that follows the header, past the hints it carries + * (`overload;`, `inline;`), unless every branch of the conditional holds a + * whole body: the one conditional the grammar reads there. When the + * conditional holds no block at all (only locals or hints), the scan goes on + * in its first branch, now plain code, so a conditional nested at its start + * comes right after the header too. + */ +function blankConditionalsAfterHeader(s: string, i: number, groupAt: ReadonlyMap, ranges: number[]): void { + for (;;) { + i = skipTriviaToGroup(s, i, groupAt); + const group = groupAt.get(i); + if (group) { + const { bounds } = group; + let bodies = true; + for (let b = 1; b < bounds.length - 1 && bodies; b += 2) bodies = holdsBody(s, bounds[b]!, bounds[b + 1]!); + if (bodies) return; // the grammar reads these + ranges.push(...bounds); + i = holdsBlock(s, bounds[1]!, bounds[bounds.length - 2]!) ? bounds[bounds.length - 1]! : bounds[1]!; + continue; + } + if (!ROUTINE_HINTS.has(lowerWordAt(s, i))) return; + i = semicolonEnd(s, i, MAX_HEADER_PART_LENGTH); + if (i < 0) return; + } +} + +const OPEN_BEGIN = 0, OPEN_ASM = 1, OPEN_OTHER = 2; + +/** + * The blocks the code from `from` to `to` leaves open, in order, each as its + * opener's offset and kind (OPEN_BEGIN, OPEN_ASM, or OPEN_OTHER for `try`, + * `case` and `record`), or null when it closes a block it didn't open. + */ +function openBlocks(s: string, from: number, to: number): Array<[number, number]> | null { + const open: Array<[number, number]> = []; + let i = from; + while (i < to) { + const k = commentOrStringEnd(s, i); + if (k >= 0) { + i = k; + continue; + } + if (isIdentStart(s.charCodeAt(i))) { + const e = wordEnd(s, i + 1); + if (e - i >= 3 && e - i <= 6) { + const word = s.slice(i, e).toLowerCase(); + if (word === 'end') { + if (open.pop() === undefined) return null; + } else if (BLOCK_OPENERS.has(word)) { + open.push([i, word === 'begin' ? OPEN_BEGIN : word === 'asm' ? OPEN_ASM : OPEN_OTHER]); + } + } + i = e; + continue; + } + i++; + } + return open; +} + +/** + * In every conditional (holding no conditional of its own) whose branches + * each leave blocks open for the code after it to close. When they all leave + * the same number of `begin`s open, the `begin`s each branch after the first + * leaves open are blanked: read together, they would open that many blocks + * more than the code closes, and a branch's statements can sit in the first + * branch's block, as the graph doesn't read nesting. When each opens one + * body, some with `asm` and some with `begin` (an assembler and a Pascal + * version of a routine sharing one `end`), each assembler branch is blanked + * whole, so the Pascal one is the body; the graph reads nothing in assembler. + */ +function blankDuplicateBegins(s: string, groups: readonly ConditionalGroup[], ranges: number[]): void { + for (const { bounds, nested } of groups) { + if (nested.length < 2 || nested.includes(true)) continue; + const open: Array> = []; + for (let b = 0; b < nested.length; b++) { + const blocks = openBlocks(s, bounds[2 * b + 1]!, bounds[2 * b + 2]!); + if (!blocks || blocks.length === 0) break; + open.push(blocks); + } + if (open.length < nested.length) continue; + const kinds = (kind: number) => open.filter((o) => o.every(([, k]) => k === kind)).length; + if (kinds(OPEN_BEGIN) === open.length && open.every((o) => o.length === open[0]!.length)) { + for (let b = 1; b < open.length; b++) for (const [at] of open[b]!) ranges.push(at, at + 5); + } else if (open.every((o) => o.length === 1) && kinds(OPEN_ASM) > 0 && kinds(OPEN_BEGIN) > 0 && kinds(OPEN_BEGIN) + kinds(OPEN_ASM) === open.length) { + let kept = false; + for (let b = 0; b < open.length; b++) { + const [at, kind] = open[b]![0]!; + if (kind === OPEN_ASM) ranges.push(bounds[2 * b + 1]!, bounds[2 * b + 2]!); + else if (kept) ranges.push(at, at + 5); + else kept = true; + } + } + } +} + +const SHAPE_NONE = -1, SHAPE_STATEMENTS = 0, SHAPE_STATEMENTS_SEMI = 1, SHAPE_EMPTY = 2; +const START_NONE = 0, START_STATEMENT = 1, START_CALL = 2, START_NAME = 3; + +/** + * Whether the code from `from` to `to`, a conditional's branch, holds whole statements: its blocks and brackets + * close, every `if` has its `then` and every `while`, `for`, `with` and `on` its `do`, and it ends where a + * statement can. It must start with a statement's keyword, assign at its top level or call a routine (`Foo(…)`, + * `A.B(…)`); with `bare`, a lone name counts too (`Exit`, `List.Clear;`), which after a case label could just as + * well be a type. Returns SHAPE_STATEMENTS (SHAPE_STATEMENTS_SEMI when it ends with `;`), SHAPE_EMPTY for no code, + * or SHAPE_NONE. + */ +function statementsShape(s: string, from: number, to: number, bare: boolean): number { + const blocks: boolean[] = []; // an open block each: true for a `repeat`, which `until` closes + let depth = 0; + let heads = 0; // `if`, `while`, `for`, `with` and `on` still waiting for their `then` or `do` + let tokens = 0; + let start = START_NONE; + let name = 0; // reading the first dotted name: 1 after a part of it, 2 after its `.`, 0 when done + let last = ''; // the last token: a word lowercased ('word' when long), '0' for a number, else its character + let i = from; + while (i < to) { + const k = commentOrStringEnd(s, i); + if (k >= 0) { + if (s.charCodeAt(i) === CH_QUOTE) { + tokens++; + name = 0; + last = "'"; + } + i = k; + continue; + } + const c = s.charCodeAt(i); + if (isSpace(c)) { + i++; + continue; + } + tokens++; + if (isIdentStart(c)) { + const e = wordEnd(s, i + 1); + const word = e - i <= 9 ? s.slice(i, e).toLowerCase() : 'word'; + if (word === 'begin' || word === 'try' || word === 'case' || word === 'asm') blocks.push(false); + else if (word === 'repeat') blocks.push(true); + else if (word === 'end' && blocks.pop() !== false) return SHAPE_NONE; + else if (word === 'until' && blocks.pop() !== true) return SHAPE_NONE; + else if (word === 'if' || word === 'while' || word === 'for' || word === 'with' || word === 'on') heads++; + else if (word === 'then' || word === 'do') heads--; + if (tokens === 1) { + if (STATEMENT_KEYWORDS.has(word)) start = START_STATEMENT; + else if (!TYPE_KEYWORDS.has(word) && !STATEMENT_CONTINUES.has(word) && word !== 'end') name = 1; + } else { + name = name === 2 ? 1 : 0; + } + last = word; + i = e; + continue; + } + if (c >= 48 && c <= 57) { + name = 0; + last = '0'; + i = wordEnd(s, i + 1); + continue; + } + if (c === CH_COLON && s.charCodeAt(i + 1) === CH_EQ) { + if (depth === 0 && start !== START_CALL) start = START_STATEMENT; + name = 0; + last = ':='; + i += 2; + continue; + } + if (c === CH_LPAREN || c === CH_LBRACKET) { + if (c === CH_LPAREN && name === 1 && start === START_NONE) start = START_CALL; + depth++; + } else if ((c === CH_RPAREN || c === CH_RBRACKET) && --depth < 0) { + return SHAPE_NONE; + } else if (c === CH_SEMI && name === 1 && start === START_NONE) { + start = START_NAME; + } + name = c === CH_DOT && name === 1 ? 2 : 0; + last = s[i]!; + i++; + } + if (tokens === 0) return SHAPE_EMPTY; + if (depth !== 0 || blocks.length > 0 || heads !== 0) return SHAPE_NONE; + if (name === 1 && start === START_NONE) start = START_NAME; + if (start === START_NONE || (start === START_NAME && !bare)) return SHAPE_NONE; + const ends = last === ';' || last === ')' || last === ']' || last === '^' || last === "'" || last === '0' + || (isIdentStart(last.charCodeAt(0)) && !STATEMENT_CONTINUES.has(last)); + if (!ends) return SHAPE_NONE; + return last === ';' ? SHAPE_STATEMENTS_SEMI : SHAPE_STATEMENTS; +} + +/** Whether a blank range in `ranges` overlaps `[from, to)`. */ +function overlapsRange(ranges: readonly number[], from: number, to: number): boolean { + for (let r = 0; r < ranges.length; r += 2) if (ranges[r]! < to && ranges[r + 1]! > from) return true; + return false; +} + +/** Whether `[at, at + length)` holds no line break, so a word written there keeps every line where it is. */ +function onOneLine(s: string, at: number, length: number): boolean { + for (let i = at; i < at + length; i++) { + const c = s.charCodeAt(i); + if (c === CH_LF || c === CH_CR) return false; + } + return true; +} + +/** + * Read as one statement each conditional whose branches are alternative statements where the grammar takes just + * one: after a case label, `then`, `else` or `do` (`label` maps each such group's opening directive to whether a + * case label's `:` comes before it). Its opening directive becomes `begin`, each `{$ELSE}` a `;`, and its closing + * directive `end`, or `end;` when its branches end with `;`. Read together, the branches of one after a case label + * always broke the arm (`OP_X: begin … end; while … do Inc(N);`). After the others they break only when a branch + * before the last has no `;` (`then A {$ELSE} B {$ENDIF} else C`), since `then A; B;` parses as it is. A group + * holding another conditional is left alone, as is one whose branches end in a statement's head (`if … then`) for a + * body after them. + */ +function wrapAlternativeStatements( + s: string, + groups: readonly ConditionalGroup[], + label: ReadonlyMap, + ranges: readonly number[], + words: Array<[number, number, string]>, +): void { + for (const { bounds, nested } of groups) { + const afterLabel = label.get(bounds[0]!); + if (afterLabel === undefined || nested.length < 2 || nested.includes(true)) continue; + let filled = 0; + let semi = false; + let breaks = afterLabel; + let ok = true; + for (let b = 0; b < nested.length && ok; b++) { + const shape = statementsShape(s, bounds[2 * b + 1]!, bounds[2 * b + 2]!, !afterLabel); + if (shape === SHAPE_EMPTY) continue; + if (shape === SHAPE_NONE) ok = false; + if (filled > 0 && !semi) breaks = true; + filled++; + semi = shape === SHAPE_STATEMENTS_SEMI; + } + if (!ok || filled < 2 || !breaks || !onOneLine(s, bounds[0]!, 5)) continue; + let clear = true; + for (let d = 0; d < bounds.length && clear; d += 2) clear = !overlapsRange(ranges, bounds[d]!, bounds[d + 1]!); + if (!clear) continue; + words.push([bounds[0]!, bounds[1]!, 'begin']); + for (let d = 2; d < bounds.length - 2; d += 2) words.push([bounds[d]!, bounds[d + 1]!, ';']); + words.push([bounds[bounds.length - 2]!, bounds[bounds.length - 1]!, semi ? 'end;' : 'end']); + } +} + +/** + * The kind of statement position the token from `at` to `end` makes for what follows it: 1 for a case label's + * `:`, 0 for `then`, `else` or `do`, -1 for anything else. + */ +function statementPosition(s: string, at: number, end: number): number { + const length = end - at; + if (length === 1) return s.charCodeAt(at) === CH_COLON ? 1 : -1; + if (length < 2 || length > 4) return -1; + const word = s.slice(at, end).toLowerCase(); + return word === 'then' || word === 'else' || word === 'do' ? 0 : -1; +} + +/** + * The start of the declaration that the `record` keyword at `i` gives an + * anonymous type to: the first of the `Names` in `Names: [packed] record` or + * `Names: [packed] array[…] of [packed] record`, or the `TName` in `TName = + * array[…] of record`. -1 for a named record type (`TName = record`) and + * anything else. + */ +function anonymousRecordDeclStart(s: string, i: number): number { + let j = tokenEndBefore(s, i); + let w = wordStartBefore(s, j); + if (w >= 0 && lowerWordAt(s, w) === 'packed') { + j = tokenEndBefore(s, w); + w = wordStartBefore(s, j); + } + let array = false; + if (w >= 0 && lowerWordAt(s, w) === 'of') { + j = tokenEndBefore(s, w); + if (s.charCodeAt(j - 1) === CH_RBRACKET) { + const open = s.lastIndexOf('[', j - 1); + if (open < 0) return -1; + j = tokenEndBefore(s, open); + } + w = wordStartBefore(s, j); + if (w < 0 || lowerWordAt(s, w) !== 'array') return -1; + j = tokenEndBefore(s, w); + w = wordStartBefore(s, j); + if (w >= 0 && lowerWordAt(s, w) === 'packed') j = tokenEndBefore(s, w); + array = true; + } + const c = s.charCodeAt(j - 1); + if (c !== CH_COLON && !(c === CH_EQ && array)) return -1; + let start = wordStartBefore(s, tokenEndBefore(s, j - 1)); + if (start < 0 || c === CH_EQ) return start; + for (;;) { + const k = tokenEndBefore(s, start); + if (s.charCodeAt(k - 1) !== CH_COMMA) return start; + const name = wordStartBefore(s, tokenEndBefore(s, k - 1)); + if (name < 0) return -1; + start = name; + } +} + +/** + * The index after the `end` closing a record whose fields start at `i`, or -1 when they hold what a record's fields + * can't. A word like `procedure` or `object` is part of a field's type after `:`, `of` or `to` (`OnIdle: procedure + * of object`, `Get: reference to function: Integer`, `Args: array of const`), and words in brackets are a + * routine type's parameters. + */ +function recordEnd(s: string, i: number): number { + const limit = Math.min(s.length, i + MAX_RECORD_LENGTH); + let depth = 1; + let brackets = 0; + let prev = ''; // the token before: a word lowercased, or its character + while (i < limit) { + const k = commentOrStringEnd(s, i); + if (k >= 0) { + if (s.charCodeAt(i) === CH_QUOTE) prev = "'"; + i = k; + continue; + } + const c = s.charCodeAt(i); + if (isIdentStart(c)) { + const e = wordEnd(s, i + 1); + const word = s.slice(i, e).toLowerCase(); + if (brackets === 0) { + if (word === 'end') { + if (--depth === 0) return e; + } else if (word === 'record') { + depth++; + } else if (NOT_IN_RECORD_FIELDS.has(word) && prev !== ':' && prev !== 'of' && prev !== 'to') { + return -1; + } + } + prev = word; + i = e; + continue; + } + if (c === CH_LPAREN || c === CH_LBRACKET) brackets++; + else if (c === CH_RPAREN || c === CH_RBRACKET) brackets--; + else if (c === CH_EQ && brackets === 0) return -1; + if (!isSpace(c)) prev = s[i]!; + i++; + } + return -1; +} + +/** + * At a `record` keyword from `start` to `end`, recognize the declaration of a field, variable or typed constant of + * an anonymous record type, followed by its `;`, a typed constant's `=`, or the `)` or `end` after a last field. The + * `record` (or the `packed` before it) is written over by `string`, a type the grammar knows, and the fields after + * it are blanked through its `end`, so the declaration keeps its name and a typed constant its value. Returns the + * index to resume from. + */ +function scanAnonymousRecord(s: string, start: number, end: number, ranges: number[], words: Array<[number, number, string]>): number { + if (anonymousRecordDeclStart(s, start) < 0) return end; + const first = lowerWordAt(s, skipTrivia(s, end)); + if (first === '' || NOT_IN_RECORD_FIELDS.has(first)) return end; // ``, `R: record;` + const close = recordEnd(s, end); + if (close < 0) return end; + const next = skipTrivia(s, close); + const c = s.charCodeAt(next); + const last = lowerWordAt(s, next) === 'end'; // a last field, whose `;` Pascal lets go but the grammar needs + if (c !== CH_SEMI && c !== CH_EQ && c !== CH_RPAREN && !last) return end; + const packed = wordStartBefore(s, tokenEndBefore(s, start)); + const at = packed >= 0 && lowerWordAt(s, packed) === 'packed' ? packed : start; + words.push([at, at + 6, 'string']); + if (last) { + ranges.push(at + 6, close - 3); + words.push([close - 3, close, ';']); + } else { + ranges.push(at + 6, close); + } + return close; +} + +/** + * `s` with every `[from, to)` range in `ranges` blanked to spaces, newlines kept, and every `[from, to, word]` in + * `words` (which overlap neither each other nor a range) written over by its word, padded with spaces. Ranges may + * come in any order and overlap. + */ +function applyEdits(s: string, ranges: readonly number[], words: ReadonlyArray): string { + if (ranges.length === 0 && words.length === 0) return s; + const edits: Array = words.slice(); + for (let r = 0; r < ranges.length; r += 2) edits.push([ranges[r]!, ranges[r + 1]!, '']); + edits.sort((a, b) => a[0] - b[0]); let out = ''; let last = 0; - for (let r = 0; r < ranges.length; r += 2) { - const from = ranges[r]!; - const to = ranges[r + 1]!; - out += s.slice(last, from) + s.slice(from, to).replace(/[^\r\n]/g, ' '); + for (const [start, to, word] of edits) { + const from = Math.max(start, last); + if (to <= from) continue; + const written = from === start && word.length <= to - from ? word : ''; + out += s.slice(last, from) + written + s.slice(from + written.length, to).replace(/[^\r\n]/g, ' '); last = to; } return out + s.slice(last); } -/** Whether the word before index `i`, past whitespace, is `class`. */ -function followsClass(s: string, i: number): boolean { - let j = i - 1; - while (j >= 0 && (s.charCodeAt(j) === CH_SPACE || s.charCodeAt(j) === CH_TAB || s.charCodeAt(j) === CH_LF || s.charCodeAt(j) === CH_CR)) j--; - return j >= 4 && s.slice(j - 4, j + 1).toLowerCase() === 'class' && (j < 5 || !isIdentPart(s.charCodeAt(j - 5))); +/** The start of the `class` word before index `i`, past whitespace, or -1. */ +function classBefore(s: string, i: number): number { + let j = i; + while (j > 0 && isSpace(s.charCodeAt(j - 1))) j--; + const w = j - 5; + return w >= 0 && s.slice(w, j).toLowerCase() === 'class' && (w === 0 || !isIdentPart(s.charCodeAt(w - 1))) ? w : -1; } /** @@ -294,13 +991,30 @@ function followsClass(s: string, i: number): boolean { * the source unchanged when there is nothing to blank. */ export function preParsePascal(source: string): string { - if (source.indexOf('[') < 0 && !FOR_VAR.test(source)) return source; + if (!MAY_NEED_FIXES.test(source)) return source; + const directives = source.indexOf('{$') >= 0; const ranges: number[] = []; + const words: Array<[number, number, string]> = []; + const groups = directives ? conditionalGroups(source) : []; + const groupAt = new Map(); + for (const group of groups) groupAt.set(group.bounds[0]!, group); + // Each conditional's opening directive that comes where one statement goes: whether a case label is before it. + const label = new Map(); + // The last token read, start and end, for that; -1 right after a declaration the scan read past. + let lastAt = -1; + let lastEnd = -1; const n = source.length; let i = 0; while (i < n) { const k = commentOrStringEnd(source, i); if (k >= 0) { + if (source.charCodeAt(i) === CH_QUOTE) { + lastAt = i; + lastEnd = k; + } else if (lastAt >= 0 && source.charCodeAt(i + 1) === CH_DOLLAR && groupAt.has(i)) { + const position = statementPosition(source, lastAt, lastEnd); + if (position >= 0) label.set(i, position === 1); + } i = k; continue; } @@ -308,25 +1022,62 @@ export function preParsePascal(source: string): string { if (isIdentStart(c)) { const e = wordEnd(source, i + 1); const len = e - i; - const word = len >= 8 && len <= 11 ? source.slice(i, e).toLowerCase() : ''; + const word = len >= 6 && len <= 11 ? source.slice(i, e).toLowerCase() : ''; // `operator` is no reserved word in Delphi (a call `Operator([1, 2])` is // code), so it starts a header only after `class`. - if (ROUTINE_KEYWORDS.has(word) && (word !== 'operator' || followsClass(source, i))) { + if (ROUTINE_KEYWORDS.has(word) && (word !== 'operator' || classBefore(source, i) >= 0)) { const open = skipTrivia(source, routineNameEnd(source, e)); - if (source.charCodeAt(open) === CH_LPAREN) { - i = scanParameterList(source, open, ranges); - continue; + const next = source.charCodeAt(open) === CH_LPAREN ? scanParameterList(source, open, ranges) : e; + if (groupAt.size > 0) { + const headerEnd = semicolonEnd(source, next, MAX_HEADER_PART_LENGTH); + if (headerEnd >= 0) blankConditionalsAfterHeader(source, headerEnd, groupAt, ranges); } + i = next; + lastAt = -1; + continue; + } else if (word === 'property') { + i = scanPropertyOverride(source, i, e, ranges); + lastAt = -1; + continue; + } else if (word === 'record') { + i = scanAnonymousRecord(source, i, e, ranges, words); + lastAt = -1; + continue; } else if (len === 3 && source.slice(i, e).toLowerCase() === 'for') { i = scanForInVariable(source, e, ranges); + lastAt = -1; continue; + } else if (len === 5 && source.slice(i, e).toLowerCase() === 'raise') { + // A re-raise: blanked, or an empty statement where one must end a list (`else raise end`). + const next = skipTrivia(source, e); + const after = lowerWordAt(source, next); + if (source.charCodeAt(next) === CH_SEMI || after === 'else') ranges.push(i, e); + else if (LABEL_ENDS.has(after)) words.push([i, e, ';']); + } else { + let colon = e; + while (source.charCodeAt(colon) === CH_SPACE || source.charCodeAt(colon) === CH_TAB) colon++; + // A label ending its block (`quit: end;`), not an assembler one (`@z:`). + if (source.charCodeAt(colon) === CH_COLON && source.charCodeAt(colon + 1) !== CH_EQ && source.charCodeAt(i - 1) !== CH_AT + && LABEL_ENDS.has(lowerWordAt(source, skipTrivia(source, colon + 1)))) ranges.push(i, colon + 1); } + lastAt = i; + lastEnd = e; i = e; } else if (c >= 48 && c <= 57) { + lastAt = i; i = wordEnd(source, i + 1); // a number - } else { + lastEnd = i; + } else if (isSpace(c)) { i++; + } else { + lastAt = i; + i += c === CH_COLON && source.charCodeAt(i + 1) === CH_EQ ? 2 : 1; + lastEnd = i; } } - return blankRanges(source, ranges); + if (groups.length > 0) { + blankDuplicateBegins(source, groups, ranges); + if (label.size > 0) wrapAlternativeStatements(source, groups, label, ranges, words); + } + return applyEdits(source, ranges, words); } diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 94453c64ea..7310dc15b9 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -1309,8 +1309,10 @@ export class TreeSitterExtractor { // live in the sibling. Pull it in. (A body as a next sibling of the scope // node is unique to Dart/Pascal among the value-ref languages — every other // grammar nests the body inside the function node — so this is inert - // elsewhere.) - const sib = scope.node.nextNamedSibling; + // elsewhere.) A Pascal routine nested in another is stored as its whole + // `defProc`, body included, and the `block` after it is the outer + // routine's body: its reads are not the nested routine's. + const sib = scope.node.type === 'defProc' ? null : scope.node.nextNamedSibling; if (sib && (sib.type === 'function_body' || sib.type === 'block')) stack.push(sib); let visited = 0; while (stack.length > 0 && visited < TreeSitterExtractor.MAX_VALUE_REF_NODES) { From b8c3c032beb0f64016afa0a1241c6d433d6a3a3c Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 23:32:10 +0000 Subject: [PATCH 49/95] fix(bundle): bundle Node 24.22.1, the October security release, and verify the download (#2817) Release bundles were set to ship Node 24.21.0. The release workflow runs build-bundle.sh with no version, so the script's default decides the Node every bundle carries, and that default predates Node's October 2026 security release. v24.22.1 (2026-10-20) fixes 8 CVEs, 4 rated High (TLS, DNS, Brotli), and ships OpenSSL 3.5.9. It keeps V8 13.6.233.17, libuv 1.52.1 and module ABI 137, and it carries 24.20.0's fix for the Windows abort at exit (issue #2617). The script also never checked the Node download against the SHA-256 sums nodejs.org publishes. Fix: - The default and the usage example move to v24.22.1. The comment and a new test hold the default at 24.22.1 or later, on 24.x. - Step 1 downloads the release's SHASUMS256.txt and refuses the archive, before unpacking anything, unless its SHA-256 matches the listed one. The zip and tar.gz downloads now share that one code path (sha256sum, or shasum on macOS). - The unreleased CHANGELOG bullet for issue #2617 now names 24.22.1 and the check. BUNDLING.md describes the check. Validation (Windows 11 x64): - bundle-launcher.test.ts: 4 new tests (the version floor; a matching download is unpacked; a mismatched or unlisted one is refused before unpacking). With main's build-bundle.sh all 4 fail; with the fix the file has 7 passed and 3 POSIX-only skipped. - build-bundle.sh win32-x64 with the new default: the check passed (sha256 d2046d28...8b95fa5, as listed). The bundled node.exe is byte-identical to the official v24.22.1 one and prints v24.22.1 (OpenSSL 3.5.9). Its launchers from Git Bash, cmd and PowerShell, with no system Node on PATH, ran --version, init, status, query, callers and sync on a small TypeScript + Python fixture: all exit 0. - Node's own reproducer for the exit abort, 300 runs per mode, 4 at a time: the bundled v24.22.1 exited cleanly 300/300 in all three modes. The official v24.16.0 control aborted 299/300 with 0xC0000409 when the child ran directly. - 13 runtime-facing test files (sqlite, wasm flags, worker and stdin teardown, watcher, launcher, shim) run on the bundled node.exe: 136 passed, 22 skipped, 0 failed. Fixes #2780 Co-Authored-By: Claude Opus 5.5 --- BUNDLING.md | 4 ++ CHANGELOG.md | 2 +- __tests__/bundle-launcher.test.ts | 74 +++++++++++++++++++++++++++++++ scripts/build-bundle.sh | 54 ++++++++++++++-------- 4 files changed, 115 insertions(+), 19 deletions(-) diff --git a/BUNDLING.md b/BUNDLING.md index dc21ab5312..90d53ac9d1 100644 --- a/BUNDLING.md +++ b/BUNDLING.md @@ -33,6 +33,10 @@ scripts/build-bundle.sh linux-x64 # -> release/codegraph-linux-x64.ta scripts/build-bundle.sh win32-x64 # -> release/codegraph-win32-x64.zip ``` +The Node runtime is the official nodejs.org release named by the script's +default `NODE_VERSION` (or its second argument). The download is refused +unless it matches the SHA-256 listed in that release's `SHASUMS256.txt`. + Because dropping better-sqlite3 left **zero native addons**, building a bundle is pure file-packaging — **any** target builds on **any** OS (the whole matrix builds on one Linux runner). Cross-compilation isn't a concern; only *run-testing* a diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d43a8c3f9..0b4b6115d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,7 +51,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404) - An index left incomplete by an indexing run that was stopped partway, by a crash, an out-of-memory kill or the server's own hang check, no longer gets the background server killed with "Main thread unresponsive" every time it starts: on a large project, restoring the database indexes the run had set aside took longer than the hang check allows, so each restart died at the same point and the index was never repaired. They are now restored in the background, and the project is served once they are back. Thanks @cbeaulieu-gt for the report and the first fix. (#1887) - A new agent session no longer waits up to about 12 minutes for its first answer when the background server is stuck: a server whose process still holds its connection open but never responds made the session wait out a long series of retries. The session now gives up within seconds and answers on its own, and switches back to the background server once that has been restarted. -- On Windows, a command like `codegraph sync` no longer sometimes crashes as it exits, after its work is done, printing `Assertion failed: !(handle->flags & UV_HANDLE_CLOSING)`, which made a script or CI run that had succeeded fail. The bundled Node runtime moves to 24.21.0, which fixes that crash in Node itself. Thanks @Ancienttwo for the report and the first fix. (#2617) +- The bundled Node runtime moves to 24.22.1, Node's October security release, and every release now checks the Node it bundles against the checksums nodejs.org publishes. The new runtime also fixes a crash on Windows as a command like `codegraph sync` exits, after its work is done, printing `Assertion failed: !(handle->flags & UV_HANDLE_CLOSING)`, which made a script or CI run that had succeeded fail. Thanks @Ancienttwo for the report and the first fix, and @IdealBoy709 for the report on the security release. (#2617, #2780) - When CodeGraph can't open a project's database, it now says what went wrong: SQLite's error, the file's path, and what to check, such as another program holding the file open, a folder you can't write to, or a network share. Before, every such failure read "CodeGraph requires node:sqlite (Node.js 22.5+)", and an agent was told no project was loaded. That hint now appears only when Node really can't load its built-in SQLite, a case that used to end in an unhandled error instead. - `codegraph upgrade` no longer wipes your agent instructions file, such as a global `AGENTS.md`, `CLAUDE.md` or `GEMINI.md`, when it refreshes CodeGraph's section in it: a stray `` line above that section, like one left behind when part of an older section was deleted by hand, made the refresh replace everything in between, often the whole file. CodeGraph now rewrites only its own section, in `codegraph install` and `codegraph uninstall` too, and keeps everything around it exactly as it was, Windows line endings included. Thanks @pzaj2 for the report. (#2506) - `codegraph install`, `upgrade` and `uninstall` now write through an instructions or config file that is a symbolic link, such as a `~/.codex/AGENTS.md` or `~/.claude.json` linked from your dotfiles, instead of replacing the link with a copy: the shared file gets CodeGraph's settings, uninstall takes them out again, and the link stays in place. A link CodeGraph can't write through, like one to a read-only file or into a folder that doesn't exist, is now reported by name and left alone, and an upgrade still refreshes your other agents. Thanks @jguerra6 for the report and @danusha2345 for the first fix. (#2721) diff --git a/__tests__/bundle-launcher.test.ts b/__tests__/bundle-launcher.test.ts index c05fbc4a73..13b5b92c81 100644 --- a/__tests__/bundle-launcher.test.ts +++ b/__tests__/bundle-launcher.test.ts @@ -1,5 +1,6 @@ import { afterEach, describe, expect, it } from 'vitest'; import { spawnSync } from 'child_process'; +import * as crypto from 'crypto'; import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; @@ -142,3 +143,76 @@ describe('bundled Node runtime (#2617)', () => { expect(minor).toBeGreaterThanOrEqual(20); }); }); + +describe('bundled Node security release (#2780)', () => { + const archive = 'node-v24.22.1-linux-x64.tar.gz'; + + // Step 1 of the release recipe, run against a stand-in for nodejs.org: `curl` + // copies the requested file out of `site/` and logs the URL it was asked for. + function download(sums: (sha: string) => string) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-node-download-')); + roots.push(root); + const site = path.join(root, 'site'); + const work = path.join(root, 'work'); + fs.mkdirSync(path.join(site, 'node-v24.22.1-linux-x64', 'bin'), { recursive: true }); + fs.mkdirSync(work); + fs.writeFileSync(path.join(site, 'node-v24.22.1-linux-x64', 'bin', 'node'), '#!/bin/sh\n'); + const arg = (n: number) => windows ? `$(cygpath -u "$${n}")` : `$${n}`; + // Packed the way nodejs.org packs a release: one top-level folder. + const pack = shell(`set -eu\ncd "${arg(1)}"\ntar -czf ${archive} node-v24.22.1-linux-x64`, [site]); + expect(pack.status, pack.stderr).toBe(0); + const sha = crypto.createHash('sha256').update(fs.readFileSync(path.join(site, archive))).digest('hex'); + fs.writeFileSync(path.join(site, 'SHASUMS256.txt'), sums(sha)); + const step = recipe.slice(recipe.indexOf('# 1. Download'), recipe.indexOf('# 2. Build')); + const result = shell(`set -euo pipefail +WORK="${arg(1)}" +SITE="${arg(2)}" +TARGET=linux-x64 OSFAM=linux ARCH=x64 NODE_VERSION=v24.22.1 +curl() { + local url='' out='' + while [ $# -gt 0 ]; do + case "$1" in -o) out="$2"; shift 2 ;; -*) shift ;; *) url="$1"; shift ;; esac + done + printf '%s\\n' "$url" >> "$WORK/urls" + cp "$SITE/\${url##*/}" "$out" +} +${step} +printf 'NODE_BIN=%s\\n' "$NODE_BIN"`, [work, site]); + const urls = fs.existsSync(path.join(work, 'urls')) + ? fs.readFileSync(path.join(work, 'urls'), 'utf8').trim().split('\n') + : []; + return { result, work, urls }; + } + + it('ships Node 24.22.1, the October 2026 security release, or a later 24.x', () => { + const pin = /^NODE_VERSION="\$\{2:-v(\d+)\.(\d+)\.(\d+)\}"\r?$/m.exec(recipe); + expect(pin, 'default NODE_VERSION in scripts/build-bundle.sh').not.toBeNull(); + const [major, minor, patch] = pin!.slice(1, 4).map(Number); + expect(major).toBe(24); + // 24.22.1 fixes eight CVEs (four rated High) and ships OpenSSL 3.5.9. + expect(minor > 22 || (minor === 22 && patch >= 1), `v${major}.${minor}.${patch}`).toBe(true); + }); + + it('unpacks the Node download once it matches the SHA-256 nodejs.org publishes for it', () => { + const other = 'f'.repeat(64); + const { result, work, urls } = download((sha) => + `${other} node-v24.22.1-darwin-arm64.tar.gz\n${sha} ${archive}\n${other} node-v24.22.1-win-x64.zip\n`); + expect(result.status, result.stderr).toBe(0); + expect(urls).toEqual([ + `https://nodejs.org/dist/v24.22.1/${archive}`, + 'https://nodejs.org/dist/v24.22.1/SHASUMS256.txt', + ]); + expect(fs.existsSync(path.join(work, 'node-v24.22.1-linux-x64', 'bin', 'node'))).toBe(true); + expect(result.stdout).toMatch(/^NODE_BIN=.*node-v24\.22\.1-linux-x64\/bin\/node\r?$/m); + }); + + it.each([ + ['whose SHA-256 differs from the published one', () => `${'0'.repeat(64)} ${archive}\n`], + ['that the published list leaves out', (sha: string) => `${sha} node-v24.22.1-linux-arm64.tar.gz\n`], + ])('refuses a Node download %s, before unpacking it', (_, sums) => { + const { result, work } = download(sums); + expect(result.status).toBe(1); + expect(result.stderr).toContain(`${archive} does not match the SHA-256 that nodejs.org lists for v24.22.1`); + expect(fs.existsSync(path.join(work, 'node-v24.22.1-linux-x64'))).toBe(false); + }); +}); diff --git a/scripts/build-bundle.sh b/scripts/build-bundle.sh index 4d571709ec..4cfbf312f1 100755 --- a/scripts/build-bundle.sh +++ b/scripts/build-bundle.sh @@ -13,7 +13,7 @@ # scripts/build-bundle.sh [node-version] # target: darwin-arm64 | darwin-x64 | linux-x64 | linux-arm64 # | win32-x64 | win32-arm64 -# node-version: e.g. v24.21.0 (default below; pin for reproducible builds) +# node-version: e.g. v24.22.1 (default below; pin for reproducible builds) # # Output: # unix: release/codegraph-.tar.gz (launcher: bin/codegraph) @@ -22,12 +22,13 @@ set -euo pipefail TARGET="${1:?usage: build-bundle.sh [node-version]}" # The release workflow passes no version, so this default is the Node every -# release bundles. Keep it on 24.x (the CLI refuses Node 25+) and at 24.20.0 or -# later: older 24.x releases can abort on Windows just as the process exits, -# after its work is done, with 0xC0000409 and "Assertion failed: -# !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c" (nodejs/node#56645, -# fixed by nodejs/node#61999 in 24.20.0; issue #2617). -NODE_VERSION="${2:-v24.21.0}" +# release bundles. Keep it on 24.x (the CLI refuses Node 25+) and at 24.22.1 or +# later. 24.22.1 is Node's October 2026 security release (issue #2780). Before +# 24.20.0, Node 24 can abort on Windows just as the process exits, after its +# work is done, with 0xC0000409 and "Assertion failed: !(handle->flags & +# UV_HANDLE_CLOSING), file src\win\async.c" (nodejs/node#56645, fixed by +# nodejs/node#61999; issue #2617). +NODE_VERSION="${2:-v24.22.1}" ROOT="$(cd "$(dirname "$0")/.." && pwd)" OUT="$ROOT/release" @@ -39,24 +40,41 @@ OSFAM="${TARGET%-*}" # darwin | linux | win32 echo "[bundle] target=${TARGET} node=${NODE_VERSION}" -# 1. Download + extract the official Node runtime for the target platform. +# 1. Download + extract the official Node runtime for the target platform. The +# archive must match the SHA-256 that nodejs.org publishes for the release +# before anything is unpacked, so a bundle ships exactly that release's +# official build, never a truncated or substituted download (issue #2780). if [ "$OSFAM" = "win32" ]; then NODE_DIST="node-${NODE_VERSION}-win-${ARCH}" - NODE_URL="https://nodejs.org/dist/${NODE_VERSION}/${NODE_DIST}.zip" - echo "[bundle] downloading ${NODE_URL}" - curl -fsSL "$NODE_URL" -o "$WORK/node.zip" + NODE_ARCHIVE="${NODE_DIST}.zip" +else + NODE_DIST="node-${NODE_VERSION}-${TARGET}" + NODE_ARCHIVE="${NODE_DIST}.tar.gz" +fi +NODE_URL="https://nodejs.org/dist/${NODE_VERSION}/${NODE_ARCHIVE}" +echo "[bundle] downloading ${NODE_URL}" +curl -fsSL "$NODE_URL" -o "$WORK/$NODE_ARCHIVE" +curl -fsSL "https://nodejs.org/dist/${NODE_VERSION}/SHASUMS256.txt" -o "$WORK/SHASUMS256.txt" +EXPECTED_SHA="$(awk -v name="$NODE_ARCHIVE" '$2 == name { print $1 }' "$WORK/SHASUMS256.txt")" +if command -v sha256sum >/dev/null 2>&1; then + ACTUAL_SHA="$(sha256sum < "$WORK/$NODE_ARCHIVE" | cut -d' ' -f1)" +else + ACTUAL_SHA="$(shasum -a 256 < "$WORK/$NODE_ARCHIVE" | cut -d' ' -f1)" # macOS +fi +if [ -z "$EXPECTED_SHA" ] || [ "$ACTUAL_SHA" != "$EXPECTED_SHA" ]; then + echo "[bundle] error: ${NODE_ARCHIVE} does not match the SHA-256 that nodejs.org lists for ${NODE_VERSION} (listed: ${EXPECTED_SHA:-nothing}, downloaded: ${ACTUAL_SHA})" >&2 + exit 1 +fi +echo "[bundle] verified ${NODE_ARCHIVE} against nodejs.org's SHASUMS256.txt (sha256 ${ACTUAL_SHA})" +if [ "$OSFAM" = "win32" ]; then if command -v unzip >/dev/null 2>&1; then - unzip -q "$WORK/node.zip" -d "$WORK" + unzip -q "$WORK/$NODE_ARCHIVE" -d "$WORK" else - tar -xf "$WORK/node.zip" -C "$WORK" # bsdtar can read zip + tar -xf "$WORK/$NODE_ARCHIVE" -C "$WORK" # bsdtar can read zip fi NODE_BIN="$WORK/${NODE_DIST}/node.exe" else - NODE_DIST="node-${NODE_VERSION}-${TARGET}" - NODE_URL="https://nodejs.org/dist/${NODE_VERSION}/${NODE_DIST}.tar.gz" - echo "[bundle] downloading ${NODE_URL}" - curl -fsSL "$NODE_URL" -o "$WORK/node.tar.gz" - tar -xzf "$WORK/node.tar.gz" -C "$WORK" + tar -xzf "$WORK/$NODE_ARCHIVE" -C "$WORK" NODE_BIN="$WORK/${NODE_DIST}/bin/node" fi [ -f "$NODE_BIN" ] || { echo "[bundle] error: node binary not found ($NODE_BIN)" >&2; exit 1; } From 3f58e343832aacd7b941b82f05a65a77fbdffdb8 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 22 Oct 2026 23:32:31 +0000 Subject: [PATCH 50/95] fix(cli): index no longer blocks on a daemon lock naming a process that isn't CodeGraph (#2819) codegraph index told you to run `codegraph daemon stop`, which the CLI rejects ("too many arguments"), whenever it could not verify the process named in .codegraph/daemon.pid. In that state `codegraph daemon` lists nothing, because it lists only daemons that answer their socket. And `codegraph unlock` kept an old-format lock whose process was alive (the rule from issue #1834), so nothing the CLI offered got past it. The writer-lock message gave the same wrong advice. Root cause: a live pid in the lock was the only evidence, and a pid proves nothing once the OS gives it to another program. Every CodeGraph daemon since the first one (v0.9.5) runs ` ... codegraph.js serve --mcp --path `, so the holder's command line settles the question. Fix: - New src/mcp/process-identity.ts reads a process's command line and executable name from the OS: Win32_Process through PowerShell (tasklist as fallback) on Windows, /proc on Linux, ps elsewhere. Each lookup is capped at 5 s and never throws. A holder is a CodeGraph MCP server, provably something else, or unknown, and unknown is never evidence. - stopDaemonAt treats an unverified live holder that is provably not a CodeGraph server like a dead one. Its lock is cleared, under the same writer-slot claim, and the process is never signalled. clearStaleDaemonArtifacts (codegraph unlock) does the same for an old-format lock. A holder that may be a daemon keeps its lock, as before (issues #1553 and #1834). - daemonStopFailureMessage builds index's error from what remains: the pid, what it runs, and a command that clears it (taskkill /PID N /F or kill N), or which lock file to delete when only that will do. - writerLockHeldMessage names `codegraph daemon` instead of the missing `codegraph daemon stop`. Validation (Windows 11): - New daemon-lock-holder.test.ts (11 tests) and a daemon-registry test. With main's daemon-registry.ts, writer-lock.ts and codegraph.ts put back, 11 tests fail, including all 8 CLI cases and the writer-lock wording; with the fix all pass. Four existing tests used generic processes to stand in for a daemon that doesn't answer. Those stand-ins now carry a CodeGraph server's command line, and pass on main and here. - Real CLI with a private HOME, 4 cases: an old-format lock naming a live PING.EXE; an old-format lock naming a live Node process that isn't CodeGraph; a current-format lock naming that kind of Node process with a pipe nobody listens on; an old-format lock naming a Node process whose command line reads `codegraph.js serve --mcp`. On main all 4 index runs exit 1 with the old advice, `daemon stop` is rejected, `daemon` lists nothing, and unlock clears only the third. With the fix the first 3 rebuild (exit 0), each holder stays alive, and unlock clears the first too. The 4th exits 1, naming the pid and `taskkill /PID /F`; after that command, index exits 0. Contributor PR #2373 (@danusha2345) changes the two strings only; I read it for guidance. Fixes #2770 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cli-unlock.test.ts | 22 ++- __tests__/daemon-lock-holder.test.ts | 233 +++++++++++++++++++++++++++ __tests__/daemon-registry.test.ts | 33 +++- __tests__/index-command.test.ts | 10 +- __tests__/writer-lock.test.ts | 5 +- src/bin/codegraph.ts | 8 +- src/mcp/daemon-registry.ts | 104 +++++++++++- src/mcp/process-identity.ts | 135 ++++++++++++++++ src/mcp/writer-lock.ts | 2 +- 10 files changed, 532 insertions(+), 21 deletions(-) create mode 100644 __tests__/daemon-lock-holder.test.ts create mode 100644 src/mcp/process-identity.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b4b6115d9..e478835a5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - VB.NET designer code such as `New System.Drawing.Point(…)` or `Me.Size = New System.Drawing.Size(…)` is no longer linked to a project type or member that only shares the name, and a class nested inside another is matched by its bare name only from inside that class or one that inherits it. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) +- `codegraph index` no longer stops with advice to run `codegraph daemon stop`, a command that doesn't exist, when the project's background-server lock names a process that isn't CodeGraph, such as a lock left by a crash whose process number the system has since given to another program: it now clears that stale lock, leaving the process alone, and rebuilds, and `codegraph unlock` clears it too. When a CodeGraph process that doesn't answer holds the lock, such as an older background server, the message names that process and the command that stops it, and the message about another server already serving a project now points to `codegraph daemon`. Thanks @inth3shadows for the report and @danusha2345 for the first fix. (#2770) - The shared background server is no longer killed with "Main thread unresponsive" while it brings another project's index up to date, such as one a session names with `projectPath`: only its own project's index counted as progress, so a long update of any other looked like a hang, and the server was stopped and restarted again and again. Every index the server keeps up to date now counts, and so does the one a session updates on its own when the background server is unavailable. Thanks @mradwankhalil for the report and the first fix. (#2404) - An index left incomplete by an indexing run that was stopped partway, by a crash, an out-of-memory kill or the server's own hang check, no longer gets the background server killed with "Main thread unresponsive" every time it starts: on a large project, restoring the database indexes the run had set aside took longer than the hang check allows, so each restart died at the same point and the index was never repaired. They are now restored in the background, and the project is served once they are back. Thanks @cbeaulieu-gt for the report and the first fix. (#1887) - A new agent session no longer waits up to about 12 minutes for its first answer when the background server is stuck: a server whose process still holds its connection open but never responds made the session wait out a long series of retries. The session now gives up within seconds and answers on its own, and switches back to the background server once that has been restarted. diff --git a/__tests__/cli-unlock.test.ts b/__tests__/cli-unlock.test.ts index 56799f3856..9dc7244d78 100644 --- a/__tests__/cli-unlock.test.ts +++ b/__tests__/cli-unlock.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'; -import { execFile, execFileSync } from 'child_process'; +import { execFile, execFileSync, spawn } from 'child_process'; import * as fs from 'fs'; import * as net from 'net'; import * as os from 'os'; @@ -101,14 +101,22 @@ describe('codegraph unlock — daemon artifact recovery (#1553)', () => { }); it('preserves a live legacy lock whose daemon identity cannot be probed', () => { - const pidPath = getDaemonPidPath(tempDir); - fs.writeFileSync(pidPath, `${process.pid}\n`); + // The holder's command line reads like a CodeGraph MCP server's, so it may + // be the daemon. One that is plainly another program leaves a stale lock, + // which unlock clears (#2770). + const holder = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)', 'codegraph.js', 'serve', '--mcp'], { stdio: 'ignore' }); + try { + const pidPath = getDaemonPidPath(tempDir); + fs.writeFileSync(pidPath, `${holder.pid}\n`); - const output = runCodegraph(['unlock', tempDir], tempDir); + const output = runCodegraph(['unlock', tempDir], tempDir); - expect(output).toContain('No stale lock files found'); - expect(fs.readFileSync(pidPath, 'utf8')).toBe(`${process.pid}\n`); - expect(() => process.kill(process.pid, 0)).not.toThrow(); + expect(output).toContain('No stale lock files found'); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(`${holder.pid}\n`); + expect(() => process.kill(holder.pid!, 0)).not.toThrow(); + } finally { + holder.kill('SIGKILL'); + } }); it('removes a legacy lock whose PID is dead', () => { diff --git a/__tests__/daemon-lock-holder.test.ts b/__tests__/daemon-lock-holder.test.ts new file mode 100644 index 0000000000..5ab11539af --- /dev/null +++ b/__tests__/daemon-lock-holder.test.ts @@ -0,0 +1,233 @@ +/** + * Issue #2770: `codegraph index` could not stop a project's daemon whose lock + * named a live process it couldn't verify, and told the user to run + * `codegraph daemon stop`, a command the CLI rejects. `codegraph daemon` lists + * nothing in that state and `codegraph unlock` kept the lock, so nothing the + * CLI offered got past it. + * + * A lock whose live holder the OS shows is not a CodeGraph server (a pid the + * OS gave to another program, or a lock written by hand) is now treated as + * stale, the way a dead holder's is: `index` clears it and rebuilds, and + * `unlock` clears it, without signalling the process. A holder that does run a + * CodeGraph server keeps its lock, and the message names that process and a + * command that stops it. + */ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { spawn, spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { encodeLockInfo, getDaemonPidPath } from '../src/mcp/daemon-paths'; +import { isProcessAlive } from '../src/mcp/daemon-registry'; +import { + classifyLockHolder, + describeProcess, + isCodeGraphServerCommandLine, +} from '../src/mcp/process-identity'; +import { tryAcquireWriterLock, releaseWriterLock } from '../src/mcp/writer-lock'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); +const windows = process.platform === 'win32'; +const holders: number[] = []; +let root: string; + +/** + * Start a long-lived process to be named by the lock: `native` is no + * JavaScript runtime, `node` is Node running something else, and + * `codegraph-server` is Node with a CodeGraph MCP server's command line. + */ +function startHolder(kind: 'native' | 'node' | 'codegraph-server'): number { + const idle = ['-e', 'setInterval(() => {}, 1000)']; + const [command, args] = kind === 'native' + ? windows + ? [path.join(process.env.SystemRoot || 'C:\\Windows', 'System32', 'PING.EXE'), ['-n', '600', '127.0.0.1']] + : ['sleep', ['600']] + : kind === 'node' + ? [process.execPath, idle] + : [process.execPath, [...idle, path.join(root, 'install', 'dist', 'bin', 'codegraph.js'), 'serve', '--mcp', '--path', root]]; + const child = spawn(command, args as string[], { detached: true, stdio: 'ignore', windowsHide: true }); + child.unref(); + if (!child.pid) throw new Error(`could not start a ${kind} holder`); + holders.push(child.pid); + return child.pid; +} + +function runCli(args: string[]) { + return spawnSync(process.execPath, [BIN, ...args], { + cwd: root, + encoding: 'utf8', + timeout: 60_000, + env: { ...process.env, CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_TELEMETRY: '0', DO_NOT_TRACK: '1', CODEGRAPH_NO_PROMPT_HOOK: '1' }, + }); +} + +function writeLegacyLock(pid: number): string { + const lockPath = getDaemonPidPath(root); + fs.writeFileSync(lockPath, `${pid}\n`); + return lockPath; +} + +/** A current-format lock whose socket no daemon listens on. */ +function writeLock(pid: number): string { + const lockPath = getDaemonPidPath(root); + fs.writeFileSync(lockPath, encodeLockInfo({ + pid, + version: '1.6.2', + socketPath: windows ? '\\\\.\\pipe\\codegraph-2770-nobody-listens' : path.join(root, '.codegraph', 'nobody-listens.sock'), + startedAt: Date.now() - 60_000, + })); + return lockPath; +} + +function stopCommand(pid: number): string { + return windows ? `taskkill /PID ${pid} /F` : `kill ${pid}`; +} + +beforeEach(async () => { + root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'cg-lock-holder-'))); + fs.writeFileSync(path.join(root, 'app.ts'), 'export function originalSymbol() { return 1; }\n'); + const cg = await CodeGraph.init(root); + await cg.indexAll(); + cg.close(); +}); + +afterEach(() => { + for (const pid of holders.splice(0)) { + try { process.kill(pid, 'SIGKILL'); } catch { /* already gone */ } + } + releaseWriterLock(root); + fs.rmSync(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); +}); + +describe('what a lock holder is (#2770)', () => { + it('recognizes a CodeGraph MCP server by its command line', () => { + expect(isCodeGraphServerCommandLine( + '"C:\\Program Files\\nodejs\\node.exe" --liftoff-only --disable-warning=ExperimentalWarning ' + + '"C:\\Users\\me\\AppData\\Roaming\\npm\\node_modules\\@colbymchenry\\codegraph\\lib\\dist\\bin\\codegraph.js" serve --mcp --path D:\\proj', + )).toBe(true); + expect(isCodeGraphServerCommandLine('/opt/codegraph/node /opt/codegraph/lib/dist/bin/codegraph.js serve --mcp --path /src/app')).toBe(true); + expect(isCodeGraphServerCommandLine('node /usr/lib/node_modules/@colbymchenry/codegraph/dist/bin/codegraph.js index')).toBe(false); + expect(isCodeGraphServerCommandLine('node server.js serve --mcp')).toBe(false); + expect(isCodeGraphServerCommandLine('C:\\Windows\\Explorer.EXE')).toBe(false); + }); + + it('counts only what the OS shows as evidence', () => { + expect(classifyLockHolder({ name: 'explorer.exe', commandLine: 'C:\\Windows\\Explorer.EXE' })).toBe('other'); + expect(classifyLockHolder({ name: 'node.exe', commandLine: 'node.exe -e "setInterval(() => {}, 1000)"' })).toBe('other'); + expect(classifyLockHolder({ name: 'node', commandLine: 'node codegraph.js serve --mcp --path /p' })).toBe('codegraph-server'); + // No command line (another user's or an elevated process): only an + // executable that is no JavaScript runtime rules the holder out. + expect(classifyLockHolder({ name: 'svchost.exe', commandLine: null })).toBe('other'); + expect(classifyLockHolder({ name: 'node.exe', commandLine: null })).toBe('unknown'); + expect(classifyLockHolder({ name: '/usr/local/bin/node', commandLine: null })).toBe('unknown'); + expect(classifyLockHolder({ name: null, commandLine: null })).toBe('unknown'); + }); + + it('reads a running process from the OS', async () => { + const native = startHolder('native'); + const server = startHolder('codegraph-server'); + const nativeInfo = await describeProcess(native); + expect(nativeInfo.commandLine ?? '').toMatch(windows ? /ping/i : /sleep/); + expect(classifyLockHolder(nativeInfo)).toBe('other'); + expect(classifyLockHolder(await describeProcess(server))).toBe('codegraph-server'); + process.kill(native, 'SIGKILL'); + const end = Date.now() + 10_000; + while (isProcessAlive(native) && Date.now() < end) await new Promise((r) => setTimeout(r, 50)); + expect(classifyLockHolder(await describeProcess(native))).toBe('unknown'); + }, 30_000); +}); + +describe('codegraph index and unlock with a daemon lock naming a live process (#2770)', () => { + it.each([ + ['an old-format lock naming a program that is not a JavaScript runtime', 'native', writeLegacyLock], + ['an old-format lock naming a Node process that is not CodeGraph', 'node', writeLegacyLock], + ['a lock naming a Node process that is not CodeGraph', 'node', writeLock], + ] as const)('index clears %s and rebuilds, leaving the process alone', (_, kind, write) => { + const holder = startHolder(kind); + const lockPath = write(holder); + fs.writeFileSync(path.join(root, 'app.ts'), 'export function replacementSymbol() { return 2; }\n'); + + const index = runCli(['index', '--quiet', root]); + + expect(index.stderr + index.stdout).not.toContain('daemon stop'); + expect(index.status, index.stderr).toBe(0); + expect(fs.existsSync(lockPath)).toBe(false); + expect(isProcessAlive(holder)).toBe(true); + const cg = CodeGraph.openSync(root); + try { expect(cg.searchNodes('replacementSymbol').length).toBeGreaterThan(0); } + finally { cg.close(); } + }, 60_000); + + it('unlock clears an old-format lock naming a process that is not CodeGraph, leaving the process alone', () => { + const holder = startHolder('native'); + const lockPath = writeLegacyLock(holder); + + const unlock = runCli(['unlock', root]); + + expect(unlock.status, unlock.stderr).toBe(0); + expect(unlock.stdout).toContain('Removed stale lock artifacts'); + expect(fs.existsSync(lockPath)).toBe(false); + expect(isProcessAlive(holder)).toBe(true); + }, 60_000); + + it.each([ + ['an old-format lock', writeLegacyLock], + ['a lock', writeLock], + ] as const)('keeps %s held by a CodeGraph server that does not answer, and names a command that clears it', async (_, write) => { + const holder = startHolder('codegraph-server'); + const lockPath = write(holder); + + const index = runCli(['index', '--quiet', root]); + + expect(index.status).toBe(1); + const message = index.stderr + index.stdout; + expect(message).not.toContain('daemon stop'); + expect(message).toContain(`Process ${holder} runs a CodeGraph server`); + expect(message).toContain(`\`${stopCommand(holder)}\``); + expect(isProcessAlive(holder)).toBe(true); + expect(fs.existsSync(lockPath)).toBe(true); + + // The advice: stop that process, then retry `codegraph index`. + process.kill(holder, 'SIGKILL'); + const end = Date.now() + 10_000; + while (isProcessAlive(holder) && Date.now() < end) await new Promise((r) => setTimeout(r, 50)); + const retry = runCli(['index', '--quiet', root]); + expect(retry.status, retry.stderr).toBe(0); + }, 90_000); + + it('names the stale writer lock when the same process that is not CodeGraph holds it too', () => { + const holder = startHolder('native'); + writeLock(holder); + // Both of a crashed daemon's lock files, once the OS has reused its pid. + const writerPath = path.join(root, '.codegraph', 'writer.pid'); + fs.writeFileSync(writerPath, JSON.stringify({ pid: holder, mode: 'daemon', startedAt: Date.now() - 60_000, ready: true }) + '\n'); + + const index = runCli(['index', '--quiet', root]); + + expect(index.status).toBe(1); + const message = index.stderr + index.stdout; + expect(message).toContain(`names process ${holder}, which is not a CodeGraph process, so it is stale: delete ${writerPath}`); + fs.rmSync(writerPath); + const retry = runCli(['index', '--quiet', root]); + expect(retry.status, retry.stderr).toBe(0); + expect(isProcessAlive(holder)).toBe(true); + }, 90_000); + + it('the writer-lock message names `codegraph daemon`, a command the CLI accepts', () => { + expect(tryAcquireWriterLock(root, 'daemon').kind).toBe('acquired'); + try { + const index = runCli(['index', '--quiet', root]); + expect(index.status).toBe(1); + const message = index.stderr + index.stdout; + expect(message).toContain('writer lock held'); + expect(message).not.toContain('daemon stop'); + expect(message).toContain('pick it in `codegraph daemon`'); + } finally { + releaseWriterLock(root); + } + const daemon = runCli(['daemon']); + expect(daemon.stderr).not.toContain('too many arguments'); + expect(daemon.status, daemon.stderr).toBe(0); + }, 60_000); +}); diff --git a/__tests__/daemon-registry.test.ts b/__tests__/daemon-registry.test.ts index af8c46399d..49c2a4ed47 100644 --- a/__tests__/daemon-registry.test.ts +++ b/__tests__/daemon-registry.test.ts @@ -37,10 +37,15 @@ function waitForExit(child: ReturnType): Promise { : new Promise((resolve) => child.once('exit', () => resolve())); } +/** + * A live process standing in for a daemon. Its command line reads like a + * CodeGraph MCP server's: one that is plainly another program would make its + * lock stale (#2770). + */ function startDetachedProcess(): number { const source = [ "const { spawn } = require('child_process');", - "const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'ignore' });", + "const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)', 'codegraph.js', 'serve', '--mcp'], { detached: true, stdio: 'ignore' });", 'child.unref();', 'console.log(child.pid);', ].join(' '); @@ -101,6 +106,32 @@ describe('daemon-registry', () => { } }); + it('clears, without signalling it, the lock of a live process that is not a CodeGraph server (#2770)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-foreign-holder-')); + fs.mkdirSync(path.join(root, '.codegraph')); + const holder = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { stdio: 'ignore' }); + const pid = holder.pid!; + try { + fs.writeFileSync( + getDaemonPidPath(root), + encodeLockInfo({ pid, version: 'test', socketPath: path.join(root, '.codegraph', 'missing.sock'), startedAt: Date.now() }), + ); + expect((await stopDaemonAt(root, { preserveUnverified: true })).outcome).toBe('not-running'); + expect(fs.existsSync(getDaemonPidPath(root))).toBe(false); + expect(isProcessAlive(pid)).toBe(true); + + // An old-format lock has no socket to ask; the command line still tells. + fs.writeFileSync(getDaemonPidPath(root), `${pid}\n`); + expect(await clearStaleDaemonArtifacts(root)).toBe(true); + expect(fs.existsSync(getDaemonPidPath(root))).toBe(false); + expect(isProcessAlive(pid)).toBe(true); + } finally { + holder.kill('SIGKILL'); + await waitForExit(holder); + fs.rmSync(root, { recursive: true, force: true }); + } + }, 30_000); + it.each([false, true])('confirms daemon termination (refused: %s)', async (refused) => { const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-daemon-identity-')); fs.mkdirSync(path.join(root, '.codegraph')); diff --git a/__tests__/index-command.test.ts b/__tests__/index-command.test.ts index b4b405f8e6..92d57d6378 100644 --- a/__tests__/index-command.test.ts +++ b/__tests__/index-command.test.ts @@ -59,10 +59,16 @@ function isAlive(pid: number): boolean { } } +/** + * A live process standing in for a daemon that does not answer its socket. + * Its command line reads like a CodeGraph MCP server's: one that is plainly + * some other program would make the lock stale, and `index` would clear it + * (#2770). + */ function startDetachedProcess(): number { const source = [ "const { spawn } = require('child_process');", - "const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'ignore' });", + "const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)', 'codegraph.js', 'serve', '--mcp'], { detached: true, stdio: 'ignore' });", 'child.unref();', 'console.log(child.pid);', ].join(' '); @@ -150,7 +156,7 @@ describe('codegraph index — full re-index keeps the graph populated (#874)', ( ); try { - expect(() => runCodegraph(['index', '--quiet'], tempDir)).toThrow(/could not verify.*daemon/i); + expect(() => runCodegraph(['index', '--quiet'], tempDir)).toThrow(/did not answer as its daemon, so it was left running/); expect(isAlive(daemonPid)).toBe(true); expect(fs.existsSync(lockPath)).toBe(true); } finally { diff --git a/__tests__/writer-lock.test.ts b/__tests__/writer-lock.test.ts index a13b11350d..028f7f29ce 100644 --- a/__tests__/writer-lock.test.ts +++ b/__tests__/writer-lock.test.ts @@ -83,7 +83,10 @@ describe('writer lock (#1740)', () => { const msg = writerLockHeldMessage(r.existing, r.pidPath); expect(msg).toMatch(/writer lock held/i); expect(msg).toMatch(/CODEGRAPH_NO_DAEMON/); - expect(msg).toMatch(/daemon stop/); + // `codegraph daemon` is the picker that stops the shared daemon; the CLI + // rejects `codegraph daemon stop` (#2770). + expect(msg).toContain('pick it in `codegraph daemon`'); + expect(msg).not.toMatch(/daemon stop/); } }); diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 119bd3c876..aeae56d669 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -903,10 +903,12 @@ program try { // A live MCP daemon keeps SQLite handles open. Verify it by its socket // before stopping it so a stale pidfile can never signal another process. - const { stopDaemonAt } = await import('../mcp/daemon-registry'); - const daemonStop = await stopDaemonAt(fs.realpathSync(projectPath), { preserveUnverified: true }); + // A lock left naming a process that isn't CodeGraph is cleared (#2770). + const { stopDaemonAt, daemonStopFailureMessage } = await import('../mcp/daemon-registry'); + const realRoot = fs.realpathSync(projectPath); + const daemonStop = await stopDaemonAt(realRoot, { preserveUnverified: true }); if (daemonStop.outcome === 'unverified' || daemonStop.outcome === 'still-running') { - throw new Error('Could not verify that the active CodeGraph daemon has stopped. Run `codegraph daemon stop` to stop it, then retry `codegraph index`.'); + throw new Error(await daemonStopFailureMessage(realRoot, daemonStop)); } // Keep the writer slot through recreation AND indexing. A reconnecting diff --git a/src/mcp/daemon-registry.ts b/src/mcp/daemon-registry.ts index a9e5860116..707fbba221 100644 --- a/src/mcp/daemon-registry.ts +++ b/src/mcp/daemon-registry.ts @@ -32,12 +32,15 @@ import { type DaemonLockInfo, } from './daemon-paths'; import { + getWriterPidPath, readWriterLock, releaseWriterLock, swapWriterLock, tryAcquireWriterLock, + writerLockHeldMessage, type WriterLockInfo, } from './writer-lock'; +import { classifyLockHolder, describeProcess } from './process-identity'; import { isOlderRelease } from './version'; import { WORKER_START_SETTLE_MS } from '../worker-teardown'; @@ -217,13 +220,40 @@ export async function clearStaleDaemonArtifacts(root: string): Promise info = decodeLockInfo(lockContents); } catch { /* missing/corrupt */ } if (info && isProcessAlive(info.pid)) { - // A live legacy holder has no socket path to probe. That is inconclusive, - // not proof of PID reuse, so preserve its lock rather than risk two writers. - if (!canProbeDaemonIdentity(info) || await probeDaemonIdentity(info)) return false; + if (canProbeDaemonIdentity(info)) { + if (await probeDaemonIdentity(info)) return false; + } else if (!await holderIsForeign(info.pid)) { + // A live legacy holder has no socket path to probe. That is inconclusive, + // not proof of PID reuse, so preserve its lock rather than risk two + // writers, unless the OS shows the process isn't a CodeGraph server. + return false; + } } return cleanupDaemonArtifacts(root, lockContents); } +/** + * Whether the OS shows that the live process `pid` can't be a CodeGraph daemon + * (#2770): its command line is not a CodeGraph MCP server's. A lock naming it + * is stale, as one naming a dead process is. A process the OS won't describe + * is never counted, and neither is this one. + */ +async function holderIsForeign(pid: number): Promise { + if (pid === process.pid) return false; + return classifyLockHolder(await describeProcess(pid)) === 'other'; +} + +/** + * Clear the lock of a live holder that {@link holderIsForeign} rules out as a + * daemon, the way a dead holder's is cleared, never signalling it. Resolves + * null when the holder may be a daemon. + */ +async function clearForeignHolder(root: string, pid: number, lockContents: string | null): Promise { + if (!await holderIsForeign(pid)) return null; + const removed = cleanupDaemonArtifacts(root, lockContents); + return { root, pid, outcome: removed ? 'not-running' : 'unverified' }; +} + const sleep = (ms: number): Promise => new Promise((r) => setTimeout(r, ms)); /** How long `stopDaemonAt` gives a daemon to exit on SIGTERM before looking closer. */ @@ -297,18 +327,80 @@ export async function stopDaemonAt( return { root, pid, outcome: removed ? 'not-running' : 'unverified' }; } // Never signal a process merely because it reused a stale daemon PID. The - // daemon's immediate hello is the process-identity proof (#1553). + // daemon's immediate hello is the process-identity proof (#1553). A holder + // the OS shows is no CodeGraph server is no daemon either: its lock is + // stale, and is cleared without signalling it (#2770). if (!identity || !canProbeDaemonIdentity(identity)) { - return { root, pid, outcome: 'unverified' }; + return await clearForeignHolder(root, pid, lockContents) ?? { root, pid, outcome: 'unverified' }; } if (!await probeDaemonIdentity(identity)) { - if (options.preserveUnverified) return { root, pid, outcome: 'unverified' }; + if (options.preserveUnverified) { + return await clearForeignHolder(root, pid, lockContents) ?? { root, pid, outcome: 'unverified' }; + } const removed = cleanupDaemonArtifacts(root, lockContents); return { root, pid, outcome: removed ? 'not-running' : 'unverified' }; } return stopVerifiedDaemon(root, identity, lockContents, options.shutdownGraceMs); } +/** The OS command that stops process `pid`, for advice text. */ +function stopCommand(pid: number, force = false): string { + if (process.platform === 'win32') return `taskkill /PID ${pid} /F`; + return force ? `kill -9 ${pid}` : `kill ${pid}`; +} + +/** + * What `codegraph index` reports when {@link stopDaemonAt} left the project's + * daemon lock in place (outcome `unverified` or `still-running`): the process + * that holds it, what that process is, and a command that clears it (#2770). + */ +export async function daemonStopFailureMessage(root: string, result: StopResult): Promise { + const retry = 'then retry `codegraph index`'; + if (result.outcome === 'still-running' && result.pid !== null) { + return `The CodeGraph daemon serving this project (process ${result.pid}) did not exit when asked to stop. ` + + `Stop it with \`${stopCommand(result.pid, true)}\`, ${retry}.`; + } + const lockPath = getDaemonPidPath(root); + let holder: number | null = null; + try { + const current = decodeLockInfo(fs.readFileSync(lockPath, 'utf8')); + if (current && isProcessAlive(current.pid)) holder = current.pid; + } catch { /* no lock now */ } + if (holder !== null && result.pid !== null && holder !== result.pid) { + return `A new CodeGraph daemon (process ${holder}) started for this project while \`codegraph index\` was stopping the old one; ` + + 'retry `codegraph index`.'; + } + const kind = holder === null ? null : classifyLockHolder(await describeProcess(holder)); + if (holder !== null && kind === 'codegraph-server') { + return `Process ${holder} runs a CodeGraph server (\`codegraph serve --mcp\`) and holds this project's daemon lock, ` + + 'but did not answer as its daemon, so it was left running. It is probably an older or stuck CodeGraph daemon: ' + + `stop it with \`${stopCommand(holder)}\`, ${retry}.`; + } + if (holder !== null && kind === 'unknown') { + return `Process ${holder} holds this project's daemon lock (${lockPath}) but did not answer as a CodeGraph daemon, ` + + 'and CodeGraph could not tell what it is, so it was left running. ' + + `If it is a CodeGraph server, stop it with \`${stopCommand(holder)}\`; if it is not, delete ${lockPath}. ` + + 'Then retry `codegraph index`.'; + } + // The daemon lock names no daemon: no live process, or one that is not + // CodeGraph. Such a lock is only cleared while holding the project's writer + // lock, so that is what kept it. + const writer = readWriterLock(root); + const writerPath = getWriterPidPath(root); + if (writer && writer.pid > 0 && writer.pid !== process.pid && isProcessAlive(writer.pid)) { + if (writer.pid === holder) { + return `The project's writer lock (${writerPath}) names process ${holder}, which is not a CodeGraph process, ` + + `so it is stale: delete ${writerPath}, ${retry}.`; + } + return writerLockHeldMessage(writer, writerPath); + } + if (holder !== null) { + return `Process ${holder} is not a CodeGraph process, so this project's daemon lock is stale, but it could not be removed: ` + + `delete ${lockPath}, ${retry}.`; + } + return 'The project\'s CodeGraph daemon changed while `codegraph index` was stopping it; retry `codegraph index`.'; +} + /** * The signalling half of a stop, for a daemon whose socket hello just proved * `identity` against the lock read as `lockContents`: SIGTERM, wait, SIGKILL diff --git a/src/mcp/process-identity.ts b/src/mcp/process-identity.ts new file mode 100644 index 0000000000..db8113bb60 --- /dev/null +++ b/src/mcp/process-identity.ts @@ -0,0 +1,135 @@ +/** + * What the OS says a running process is, for judging a lock file that names it + * (#2770). + * + * A pid in `.codegraph/daemon.pid` proves nothing once the OS has given that + * number to another process, and a daemon that does not answer its socket hello + * cannot vouch for itself (#1553). The process's command line is the evidence + * left: every CodeGraph daemon runs ` … codegraph.js serve --mcp --path + * `. A lock whose live holder runs anything else is as stale as one whose + * holder has exited. When the OS withholds the command line (another user's or + * an elevated process), the executable's name can still rule out a holder that + * is no JavaScript runtime at all. + * + * Lookups are bounded and best-effort. What the OS won't say is `null`, and + * `null` is never evidence either way: such a holder stays `unknown`, and its + * lock is kept. + */ + +import { execFile } from 'child_process'; +import * as fs from 'fs'; +import * as path from 'path'; + +export interface ProcessDescription { + /** The executable's file name, such as `node.exe` or `node`; null when the OS won't say. */ + name: string | null; + /** The whole command line; null when the OS won't say. */ + commandLine: string | null; +} + +/** + * A live lock holder is a CodeGraph MCP server (`codegraph-server`), provably + * something else (`other`), or a process the OS won't describe (`unknown`). + */ +export type LockHolderKind = 'codegraph-server' | 'other' | 'unknown'; + +const LOOKUP_TIMEOUT_MS = 5_000; + +/** Runs a short OS query and resolves its stdout, or null when it fails or times out. */ +function query(file: string, args: string[]): Promise { + return new Promise((resolve) => { + try { + execFile(file, args, { encoding: 'utf8', timeout: LOOKUP_TIMEOUT_MS, windowsHide: true, maxBuffer: 1 << 20 }, + (error, stdout) => resolve(error ? null : stdout)); + } catch { + resolve(null); + } + }); +} + +function windowsTool(...parts: string[]): string { + return path.join(process.env.SystemRoot || process.env.windir || 'C:\\Windows', 'System32', ...parts); +} + +async function describeWindowsProcess(pid: number): Promise { + // Win32_Process has the command line; tasklist, the fallback when PowerShell + // can't run, has only the image name. + const script = + "$ErrorActionPreference = 'Stop'; [Console]::OutputEncoding = [Text.Encoding]::UTF8; " + + `$p = Get-CimInstance Win32_Process -Filter 'ProcessId = ${pid}'; ` + + 'if ($p) { [Console]::Out.Write((ConvertTo-Json -Compress @{ name = $p.Name; commandLine = $p.CommandLine })) }'; + const json = await query(windowsTool('WindowsPowerShell', 'v1.0', 'powershell.exe'), + ['-NoProfile', '-NonInteractive', '-NoLogo', '-Command', script]); + if (json?.trim()) { + try { + const found = JSON.parse(json) as { name?: unknown; commandLine?: unknown }; + return { + name: typeof found.name === 'string' && found.name ? found.name : null, + commandLine: typeof found.commandLine === 'string' && found.commandLine.trim() ? found.commandLine : null, + }; + } catch { /* fall back to tasklist */ } + } + const csv = await query(windowsTool('tasklist.exe'), ['/FI', `PID eq ${pid}`, '/FO', 'CSV', '/NH']); + for (const line of csv?.split(/\r?\n/) ?? []) { + const fields = /^"([^"]*)","(\d+)"/.exec(line); + if (fields && Number(fields[2]) === pid) return { name: fields[1] || null, commandLine: null }; + } + return { name: null, commandLine: null }; +} + +function describeLinuxProcess(pid: number): ProcessDescription { + let commandLine: string | null = null; + let name: string | null = null; + try { + const args = fs.readFileSync(`/proc/${pid}/cmdline`, 'utf8').split('\0').filter(Boolean); + if (args.length > 0) commandLine = args.join(' '); + } catch { /* gone, or hidden (hidepid) */ } + try { + name = fs.readFileSync(`/proc/${pid}/comm`, 'utf8').trim() || null; + } catch { /* gone, or hidden */ } + return { name, commandLine }; +} + +async function describePsProcess(pid: number): Promise { + const [args, comm] = await Promise.all([ + query('ps', ['-ww', '-o', 'args=', '-p', String(pid)]), + query('ps', ['-o', 'comm=', '-p', String(pid)]), + ]); + return { name: comm?.trim() || null, commandLine: args?.trim() || null }; +} + +/** What the OS says process `pid` is running. Never throws. */ +export async function describeProcess(pid: number): Promise { + if (!Number.isSafeInteger(pid) || pid <= 0) return { name: null, commandLine: null }; + if (process.platform === 'win32') return describeWindowsProcess(pid); + if (process.platform === 'linux') return describeLinuxProcess(pid); + return describePsProcess(pid); +} + +/** + * Whether a command line is a CodeGraph MCP server's, the only kind of process + * that writes `daemon.pid`: it names CodeGraph and has `serve` and `--mcp` as + * arguments. + */ +export function isCodeGraphServerCommandLine(commandLine: string): boolean { + const args = commandLine.replace(/["']/g, ' ').split(/\s+/); + return /codegraph/i.test(commandLine) && args.includes('serve') && args.some((a) => a === '--mcp' || a.startsWith('--mcp=')); +} + +/** Whether an executable could be a JavaScript runtime that runs CodeGraph. */ +function mayRunCodeGraph(name: string): boolean { + const base = path.basename(name.replace(/\\/g, '/')).toLowerCase().replace(/\.exe$/, ''); + return /node|codegraph|bun|deno/.test(base); +} + +/** + * What a live lock holder is. The command line decides when the OS gives it; + * otherwise only an executable that is no JavaScript runtime counts as `other`. + */ +export function classifyLockHolder(description: ProcessDescription): LockHolderKind { + if (description.commandLine !== null) { + return isCodeGraphServerCommandLine(description.commandLine) ? 'codegraph-server' : 'other'; + } + if (description.name !== null && !mayRunCodeGraph(description.name)) return 'other'; + return 'unknown'; +} diff --git a/src/mcp/writer-lock.ts b/src/mcp/writer-lock.ts index 7785cde674..20f3e94e2c 100644 --- a/src/mcp/writer-lock.ts +++ b/src/mcp/writer-lock.ts @@ -238,7 +238,7 @@ export function writerLockHeldMessage( return ( 'CodeGraph writer lock held by ' + who + '. ' + 'Only one live MCP writer may serve a project (auto-sync / index). ' + - 'Stop the other server (codegraph daemon stop if a shared daemon, or end the other MCP session), ' + + 'Stop the other server (if it is the shared daemon, pick it in `codegraph daemon` to stop it; otherwise end the other MCP session), ' + 'or unset CODEGRAPH_NO_DAEMON so additional clients proxy to the shared daemon. ' + 'If this is stale, delete ' + pidPath ); From e32021f6fbc498d6daea6d9b4d4f7f79ad5f3f8f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 00:35:03 +0000 Subject: [PATCH 51/95] fix(java,kotlin,csharp,scala): an overload links only the overrides with its parameters (#2820) The override pass (interface-impl) grouped a class's methods by name and linked every overload of an interface or base-class method to every same-named method of the class. Issue #2663's ExampleService got four edges for two overloads: putTairValue(String) also reached putTairValue(List, double), and the other way round, so flows, callers and impact walked into overloads a call never runs. Java, Kotlin, C# and Scala make an override repeat the parameter list of what it overrides, and the stored Java and Kotlin signatures already told the overloads apart; nothing compared them. A new override-parameters module picks, of a class's same-named methods, the ones that can override a base overload. The pass asks it only when a name is overloaded on either side, so a name declared once on each side links as before. A pair with another parameter count never links. Of those with the base's count, the ones whose parameter types are written alike win: qualifiers, annotations, final, default values and C#'s BCL names for its keywords aside, varargs as arrays. When none is, the ones whose types can't be told apart are kept: a type parameter of the base method or of a type around it matches any type in its place (save(T) implemented as save(String), List as List), a raw or nullable spelling of the same type may be it, and a method written in another language than the base is compared by count alone. An override's own type parameter matches no concrete type of the base, as these languages rule. Java, Kotlin and Scala lists come from the stored signatures; C# stores none, so its list is read from the declaration past attributes, comments and string literals, as the resolver's call-site overload choice already reads one. Extraction is unchanged, so both engines give the same input. Validation: - New override-overload-pairing test (12 cases, Java, Kotlin, C#, Scala): 11 fail on main, the single-method case passes on both; all pass with the fix on the kernel and on the wasm path. - A/B on main df1cd1e1, override edges main -> fix; nodes and every other edge identical in each: hadoop-common-project (hadoop 2c01c36f) 4651 -> 3762 (-1045 +156), guava 5765 -> 4360 (-1427 +22), mybatis-3 1126 -> 898 (-278 +50), Exposed 999 -> 884 (-115), okhttp 709 -> 665 (-44), Newtonsoft.Json 527 -> 404 (-125 +2), serilog 98 -> 91 (-7), twitter/util 838 -> 699 (-139). Controls gin (Go) and nest (TypeScript): graph dumps byte-identical. - Triage: of the 3180 removed edges, 1761 pair methods with different parameter counts and 1419 methods whose concrete parameter types differ; every one of the 230 added edges is a same-parameter override the per-class cap of 40 had cut off. An independent re-read finds no same-typed override missing next to any removed pair, and each group of base methods that lost every link into a class was checked against the source (the class overrides other overloads only, or the method is final or static). - Cost: the pass alone, median of 9 alternating runs, +3 to +96 ms per index (guava 715 -> 811 ms, hadoop-common 501 -> 555 ms); whole index, median of 3 interleaved: Newtonsoft.Json 12.3 s -> 11.9 s, okhttp 12.3 s -> 12.9 s, within noise. Fixes #2663 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/override-overload-pairing.test.ts | 385 ++++++++++++++++++ src/resolution/callback-synthesizer.ts | 34 +- src/resolution/override-parameters.ts | 410 ++++++++++++++++++++ src/resolution/type-parameters.ts | 19 +- 5 files changed, 840 insertions(+), 9 deletions(-) create mode 100644 __tests__/override-overload-pairing.test.ts create mode 100644 src/resolution/override-parameters.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index e478835a5e..15f9099acd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -199,6 +199,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a type alias like `type Event = mvccpb.Event`, alone or inside a `type ( … )` group, is now indexed and links to the type it names, so impact on that type reaches the code that uses the alias. Before, aliases were missing: a use written through the alias's package, like `clientv3.Event` or prometheus's `storage.AOptions{…}`, linked to nothing, and one inside the package, like `&PrometheusVersion{}` in prometheus's `web` package, linked to whichever type of that name the project had, there a test helper's struct. A method called on a value of an alias's type now reaches the aliased type's method rather than a namesake's, and a generic alias like `type Items[T any] = List[T]` links to `List`, so `xs.Len()` on one reaches `List.Len` instead of nothing. Re-index Go projects after upgrading. - In Go, a struct that embeds another is no longer treated as overriding the methods it redeclares. gin's `Engine` embeds `RouterGroup` and has a `Use` of its own, but calling `Use` on a `RouterGroup` only ever runs `RouterGroup.Use`; before, `codegraph_explore`'s call path and impact went on from it into `Engine.Use` and the code only the engine runs. The same held for every struct or defined type embedded in another that redeclares one of its methods, like etcd's `Mutex` inside `lockerMutex`. A call through an interface still reaches each type that implements it, the embedding struct included. Re-index Go projects after upgrading. - In JavaScript and TypeScript, a subclass's `static` method no longer counts as overriding an instance method of the same name in its base class, or the reverse, since calling one never runs the other. In webpack, `codegraph_explore`'s call path and impact went on from `Module`'s `deserialize(context)` into the `static deserialize(context)` of `NormalModule` and other modules. A static method still overrides its base's static method and an instance method its base's instance method, and JavaScript class members are now marked static, as TypeScript ones already were. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and @yang0228 for the pull request. (#2761) +- In Java, Kotlin, C# and Scala, each overload of an interface or base-class method now links only to the overrides with the same parameters. Before, every overload linked to every method of that name in the implementing class, so `ExampleService.putTairValue(String)` also led to `putTairValue(List, double)`, and `codegraph_explore`'s call path, callers and impact went into overloads that call never runs. Where a parameter is a type parameter, like `save(T entity)` implemented as `save(String entity)`, or the class is written in another language than its interface, the number of parameters decides; a method declared once on each side links as before. Re-index Java, Kotlin, C# and Scala projects after upgrading. Thanks @hxj1225 for the report and @danusha2345 for the pull request. (#2663) - In JavaScript and TypeScript, a `super.run()` call now links to the method of the class's parent, or of the nearest ancestor that has it, like three.js's `super.copy(source)` reaching `Object3D.copy`, so `codegraph_explore`'s call path and impact follow an override into the method it builds on. Before, such a call inside a method of the same name was dropped, and elsewhere it went to whatever method had the name, often the class's own or a sibling class's in the same file. A call from a static method reaches the parent's static method, and a call to a parent from outside your project, like `HTMLElement`, `Set` or `Promise`, links nothing. Re-index JavaScript and TypeScript projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2497) - In Go, a call through a struct field that holds a function, like prometheus's `d.refreshf(ctx)`, now reaches the functions put in that field, so `codegraph_explore` follows service discovery from `Discovery.Run` into each provider's `refresh` and names the line where each provider wires its method in, like `RefreshF: d.refresh`. The field can be set in a struct literal or by an assignment, copied from another struct's field (`refreshf: opts.RefreshF`), or given a method of an interface value, and two unrelated structs with a field of the same name, like `fn` or `handler`, stay apart. Before, such a flow stopped at the field. Re-index Go projects after upgrading. - In Go, a type defined from other types, like etcd's `type WatchChan <-chan WatchResponse` or gin's `type HandlerFunc func(*Context)` and `type HandlersChain []HandlerFunc`, now links to the types it is built from, so impact on `WatchResponse` or gin's `Context` reaches these types and the code that uses them. Before, such a type linked to nothing. One written through another package, like etcd's `type PutResponse pb.PutResponse`, links to that package's type, and a recursive one like `type stateFn func(*Lexer) stateFn` doesn't link to itself. Re-index Go projects after upgrading. diff --git a/__tests__/override-overload-pairing.test.ts b/__tests__/override-overload-pairing.test.ts new file mode 100644 index 0000000000..168c90782f --- /dev/null +++ b/__tests__/override-overload-pairing.test.ts @@ -0,0 +1,385 @@ +/** + * An overload is overridden by the overload with its parameters (#2663). The + * override edges (`synthesizedBy: interface-impl`) linked every overload of an + * interface or base-class method to every same-named method of the class, so + * `ExampleService.putTairValue(String)` reached + * `ExampleServiceImpl.putTairValue(List, double)` and the other way + * round. Java, Kotlin, C# and Scala make an override repeat the parameter list + * of what it overrides, so an overload now pairs with the methods of its + * parameter count, and of those with the ones whose parameter types are its + * own; where a type can't be compared (a type parameter, a type written in + * another language), the count alone decides. A name declared once on each + * side links as before. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const FILES: Record = { + // The issue's example. + 'src/main/java/com/example/ExampleService.java': `package com.example; + +import java.util.List; + +interface ExampleService { + Object putTairValue(String tairKey); + + Object putTairValue(List tairKey, double count); +} +`, + 'src/main/java/com/example/ExampleServiceImpl.java': `package com.example; + +import java.util.List; + +class ExampleServiceImpl implements ExampleService { + @Override + public Object putTairValue(String tairKey) { + return null; + } + + @Override + public Object putTairValue(List tairKey, double count) { + return null; + } +} +`, + 'src/main/java/com/example/Shapes.java': `package com.example; + +import java.util.List; + +// One parameter each, told apart by type; \`final\` and annotations aside. +interface Slots { + void put(String key); + void put(int index); +} + +class ArraySlots implements Slots { + public void put(@Deprecated final String key) {} + public void put(int index) {} +} + +// A subclass overriding one overload of its base. +class Base { + void draw(int x) {} + void draw(String label) {} +} + +class Sub extends Base { + @Override + void draw(int x) {} +} + +// An implementation with an overload of its own. +interface Logger { + void log(String msg); +} + +class FileLogger implements Logger { + public void log(String msg) {} + public void log(String msg, int level) {} +} + +// Written another way but the same types: qualified, varargs as an array, +// a C-style array, a raw type. +interface Codec { + void write(java.util.List lines); + void write(String... parts); + void write(int[] data, int length); + void write(List codes, String sep); +} + +class CodecImpl implements Codec { + public void write(List lines) {} + public void write(String[] parts) {} + public void write(int data[], int length) {} + public void write(List codes, String sep) {} +} + +// Generic methods: a type parameter can be any type. +interface Mapper { + T map(Class type); + Object map(String name); +} + +class ReflectMapper implements Mapper { + public U map(Class type) { return null; } + public Object map(String name) { return null; } +} + +// One method on each side: linked as before, whatever its parameters. +interface Single { + void only(int x); +} + +class SingleImpl implements Single { + public void only(long x) {} +} + +// A Java interface a Kotlin class implements: the count decides. +interface Sink { + void accept(int x); + void accept(String s, int n); +} + +// A method's own type parameter stands in for no concrete type of its base: +// the generic appendTo overrides only the generic one. +class Appender { + A appendTo(A appendable, java.util.Iterator parts) { return appendable; } + StringBuilder appendTo(StringBuilder builder, java.util.Iterator parts) { return builder; } +} + +class SkippingAppender extends Appender { + @Override + A appendTo(A appendable, java.util.Iterator parts) { return appendable; } +} +`, + 'src/main/kotlin/com/example/Repos.kt': `package com.example + +interface Repository { + fun save(entity: T) + fun save(entities: List, flush: Boolean) +} + +class UserRepository : Repository { + override fun save(entity: String) {} + fun save(entities: Map) {} + override fun save(entities: List, flush: Boolean) {} +} + +interface Cache { + fun put(key: String): Any? + fun put(keys: List, count: Double = 1.0): Any? + fun put(key: String, value: Int): Any? +} + +class MemoryCache : Cache { + override fun put(key: String): Any? = null + override fun put(keys: List, count: Double): Any? = null + override fun put(key: kotlin.String, value: Int): Any? = null +} + +open class Shape { + open fun draw(x: Int) {} + open fun draw(label: String) {} +} + +class Circle : Shape() { + override fun draw(x: Int) {} +} + +class KotlinSink : Sink { + override fun accept(x: Int) {} + override fun accept(s: String, n: Int) {} +} +`, + 'src/Store.cs': `using System; +using System.Collections.Generic; + +namespace Example +{ + public interface IStore + { + object Put(string key); + object Put(List keys, double count); + object Put(int index); + } + + public class Store : IStore + { + public object Put(string key) { return null; } + public object Put(List keys, double count) { return null; } + public object Put(int index) { return null; } + } + + public class ExplicitStore : IStore + { + object IStore.Put(string key) => null; + object IStore.Put(List keys, double count) => null; + object IStore.Put(int index) => null; + } + + // BCL names for the keywords, a namespace, a default value, an + // attribute and a comment that mention the method. + public class SpelledStore : IStore + { + [Obsolete("Put(int) is the one to use")] + public object Put(String key) => null; + public object Put(System.Collections.Generic.List keys, Double count = 1.0) => null; + // Put(string) and Put(List, double) above. + public object Put( + Int32 index) => null; + } + + public interface IRepo + { + void Save(T entity); + void Save(IList entities, bool flush); + } + + public class UserRepo : IRepo + { + public void Save(string entity) { } + public void Save(IList entities, bool flush) { } + } + + public abstract class Widget + { + public virtual void Draw(int x) { } + public virtual void Draw(string label) { } + public virtual void Move(ref int x) { } + public virtual void Move(int x) { } + } + + public class Button : Widget + { + public override void Draw(int x) { } + public override void Move(ref int x) { } + } +} +`, + 'src/Repo.scala': `package example + +trait Repo { + def find(id: Long): Any + def find(ids: List[Long], limit: Int = 10): Any +} + +class SqlRepo extends Repo { + def find(id: Long): Any = null + def find(ids: List[Long], limit: Int): Any = null +} +`, +}; + +let root = ''; +let graph: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-override-overloads-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + graph = await CodeGraph.init(root, { index: true }); +}, 120_000); + +afterAll(() => { + graph?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::name:line -> Owner::name:line` for every override edge out of `owner`'s methods. */ +function overrides(owner: string): string[] { + const label = (id: string): string => { + const n = graph.getNode(id)!; + return `${n.qualifiedName.split('::').slice(-2).join('::')}:${n.startLine}`; + }; + return graph + .getNodesByKind('method') + .filter((m) => m.qualifiedName.split('::').slice(-2)[0] === owner) + .flatMap((m) => graph.getOutgoingEdges(m.id) + .filter((e) => e.kind === 'calls' && e.metadata?.synthesizedBy === 'interface-impl') + .map((e) => `${label(e.source)} -> ${label(e.target)}`)) + .sort(); +} + +describe('an overload is overridden by the overload with its parameters (#2663)', () => { + it("Java: the issue's interface and implementation", () => { + expect(overrides('ExampleService')).toEqual([ + 'ExampleService::putTairValue:6 -> ExampleServiceImpl::putTairValue:6', + 'ExampleService::putTairValue:8 -> ExampleServiceImpl::putTairValue:11', + ]); + }); + + it('Java: by parameter types at the same count, past `final` and annotations', () => { + expect(overrides('Slots')).toEqual([ + 'Slots::put:7 -> ArraySlots::put:12', + 'Slots::put:8 -> ArraySlots::put:13', + ]); + }); + + it('Java: a subclass overrides only the overload it repeats', () => { + expect(overrides('Base')).toEqual(['Base::draw:18 -> Sub::draw:23']); + }); + + it("Java: an implementation's own overload is no override", () => { + expect(overrides('Logger')).toEqual(['Logger::log:29 -> FileLogger::log:33']); + }); + + it('Java: the same types written another way', () => { + expect(overrides('Codec')).toEqual([ + 'Codec::write:40 -> CodecImpl::write:47', + 'Codec::write:41 -> CodecImpl::write:48', + 'Codec::write:42 -> CodecImpl::write:49', + 'Codec::write:43 -> CodecImpl::write:50', + ]); + }); + + it('Java: a type parameter matches any type in its place', () => { + expect(overrides('Mapper')).toEqual([ + 'Mapper::map:55 -> ReflectMapper::map:60', + 'Mapper::map:56 -> ReflectMapper::map:61', + ]); + }); + + it("Java: an override's own type parameter stands in for no concrete type", () => { + expect(overrides('Appender')).toEqual(['Appender::appendTo:82 -> SkippingAppender::appendTo:87']); + }); + + it('a name declared once on each side links as before', () => { + expect(overrides('Single')).toEqual(['Single::only:66 -> SingleImpl::only:70']); + }); + + it('Kotlin: by count where a type parameter stands in, by type elsewhere', () => { + expect(overrides('Repository')).toEqual([ + 'Repository::save:4 -> UserRepository::save:10', + 'Repository::save:4 -> UserRepository::save:9', + 'Repository::save:5 -> UserRepository::save:11', + ]); + expect(overrides('Cache')).toEqual([ + 'Cache::put:15 -> MemoryCache::put:21', + 'Cache::put:16 -> MemoryCache::put:22', + 'Cache::put:17 -> MemoryCache::put:23', + ]); + expect(overrides('Shape')).toEqual(['Shape::draw:27 -> Circle::draw:32']); + }); + + it('Kotlin implementing a Java interface: by parameter count', () => { + expect(overrides('Sink')).toEqual([ + 'Sink::accept:75 -> KotlinSink::accept:36', + 'Sink::accept:76 -> KotlinSink::accept:37', + ]); + }); + + it('C#: by parameter types, read from the declarations', () => { + expect(overrides('IStore')).toEqual([ + 'IStore::Put:10 -> ExplicitStore::Put:24', + 'IStore::Put:10 -> SpelledStore::Put:35', + 'IStore::Put:10 -> Store::Put:17', + 'IStore::Put:8 -> ExplicitStore::Put:22', + 'IStore::Put:8 -> SpelledStore::Put:31', + 'IStore::Put:8 -> Store::Put:15', + 'IStore::Put:9 -> ExplicitStore::Put:23', + 'IStore::Put:9 -> SpelledStore::Put:33', + 'IStore::Put:9 -> Store::Put:16', + ]); + expect(overrides('IRepo')).toEqual([ + 'IRepo::Save:41 -> UserRepo::Save:47', + 'IRepo::Save:42 -> UserRepo::Save:48', + ]); + // `Move(ref int)` and `Move(int)` are two overloads. + expect(overrides('Widget')).toEqual([ + 'Widget::Draw:53 -> Button::Draw:61', + 'Widget::Move:55 -> Button::Move:62', + ]); + }); + + it('Scala: by parameter types, past a default value', () => { + expect(overrides('Repo')).toEqual([ + 'Repo::find:4 -> SqlRepo::find:9', + 'Repo::find:5 -> SqlRepo::find:10', + ]); + }); +}); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 4727e6d001..51a56a91f9 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -43,6 +43,7 @@ import { enclosingFn, makeLineAt } from './synth-utils'; import { resolveImportPath } from './import-resolver'; import { cppSpecializationArguments, crossesCodeBoundary, goRefQualification, isManualMockHiddenFrom, isManualMockPath, jsCodeBindsName } from './name-matcher'; import { goStdlibMethods } from './go-stdlib-methods'; +import { overrideFilter } from './override-parameters'; import { declaredComponent, exportedComponent, lazyValueModules, wrappedComponentName } from './frameworks/react'; import { jsxCode } from './jsx-code'; import { isTestPath } from '../search/query-utils'; @@ -808,7 +809,9 @@ async function cppOverrideEdges(queries: QueryBuilder, ctx: ResolutionContext, o * interface method. Bridge it like cpp-override: for each class that * `implements` an interface (or `extends` an abstract base), link each * base/interface method → the class's same-name method (the override) so - * trace/callees reach the implementation. Over-approximation accepted + * trace/callees reach the implementation. Of an overloaded name, an overload + * links the methods with its parameters where the language makes an override + * repeat them (./override-parameters, #2663). Over-approximation accepted * (reachability-correct); capped per class, gated to JVM languages. */ // Languages whose static `implements`/`extends` edges should bridge an @@ -1573,7 +1576,7 @@ function goPromotions( }; } -async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield): Promise { +async function interfaceOverrideEdges(queries: QueryBuilder, ctx: ResolutionContext, onYield: MaybeYield): Promise { let scanned255 = 0; const edges: Edge[] = []; const seen = new Set(); @@ -1610,6 +1613,21 @@ async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield protocolMemo.set(base.id, methods); return methods; }; + // Of a class's same-named methods, the ones that can override one of a + // base's overloads (#2663): Java, Kotlin, C# and Scala make an override + // repeat the parameters of what it overrides. Asked only for a name that + // is overloaded on either side. + const overridesOf = overrideFilter(ctx); + const nameCounts = new Map>(); + const declaredNamed = (base: Node, name: string): number => { + let counts = nameCounts.get(base.id); + if (!counts) { + counts = new Map(); + for (const m of baseMethodsOf(base)) counts.set(m.name, (counts.get(m.name) ?? 0) + 1); + nameCounts.set(base.id, counts); + } + return counts.get(name) ?? 0; + }; // A Go struct also satisfies an interface with the methods its embedded // types promote into it (goImplementsEdges counts them), and a call through // the interface then runs the embedded type's method. Those links are made @@ -1654,8 +1672,11 @@ async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield const promotes = goStruct && sup.provenance === 'heuristic' && base.kind === 'interface'; // Group impl methods by name to handle OVERLOADS: an interface `list()` and // `list(params)` are distinct nodes and a call may resolve to either, so - // link every base overload → every same-name impl overload (keying by name - // alone would drop all but one and miss the resolved overload). A Go + // each base overload links the same-name impl methods that can override + // it — by parameters where the language makes an override repeat them + // (overrideFilter), else every one (keying by name alone would drop all + // but one and miss the resolved overload). A name declared once on each + // side pairs as it is. A Go // interface's unexported method is matched in its own package only // (goMethodKey): a namesake the type declares elsewhere is another // method, and the one the interface asks for is promoted. @@ -1675,7 +1696,8 @@ async function interfaceOverrideEdges(queries: QueryBuilder, onYield: MaybeYield if (promotes) unmatched.push(bm); continue; } - for (const m of impls) { + const overloaded = impls.length > 1 || declaredNamed(base, bm.name) > 1; + for (const m of overloaded ? overridesOf(bm, impls) : impls) { if (added >= MAX_CALLBACKS_PER_CHANNEL) break; if (bm.id === m.id) continue; // A JS/TS static method overrides only a static one, and an @@ -5113,7 +5135,7 @@ export const SYNTH_PASSES: SynthPassDef[] = [ { name: 'ifaceEdges', gate: (has) => has('java', 'kotlin', 'csharp', 'swift', 'scala', 'go', 'rust', 'arkts', ...JS_FAMILY), - run: (q, _c, y) => interfaceOverrideEdges(q, y), + run: (q, c, y) => interfaceOverrideEdges(q, c, y), }, { name: 'kotlinExpectActual', gate: (has) => has('kotlin'), run: (q, _c, y) => kotlinExpectActualEdges(q, y) }, { name: 'goGrpcEdges', gate: (has) => has('go'), run: (q, _c, y) => goGrpcStubImplEdges(q, y) }, diff --git a/src/resolution/override-parameters.ts b/src/resolution/override-parameters.ts new file mode 100644 index 0000000000..617db39382 --- /dev/null +++ b/src/resolution/override-parameters.ts @@ -0,0 +1,410 @@ +/** + * Which of a class's same-named methods overrides an overload of a type it + * implements or extends (#2663). Java, Kotlin, C# and Scala make an override + * repeat the parameter list of the method it overrides, so of + * `ExampleServiceImpl`'s two `putTairValue`s, `putTairValue(String)` is the + * one a call to `ExampleService.putTairValue(String)` runs, never + * `putTairValue(List, double)`. The override pass + * (interfaceOverrideEdges) asks this module once a name is overloaded on + * either side; a name declared once on each side pairs as it always has. + * + * Parameter lists come from the stored signatures (`Object (String tairKey)` + * in Java, `(tairKey: String): Any?` in Kotlin, `(id: Long): Any` in Scala). + * C# methods store none, so theirs is read from the declaration, as the + * resolver's own overload choice reads it (name-matcher's overloadSetOf). + */ +import type { Node } from '../types'; +import type { ResolutionContext } from './types'; +import { typeParametersInScope } from './type-parameters'; +import { escapeRegExp } from '../utils'; + +/** The languages whose overrides repeat the parameter list of the method they override. */ +export const PARAMETER_OVERRIDE_LANGUAGES: ReadonlySet = new Set(['java', 'kotlin', 'csharp', 'scala']); + +/** How far a C# declaration is read for its parameter list: attributes, then a parameter a line. */ +const CSHARP_HEAD_LINES = 40; + +/** C#'s keyword for each BCL type it aliases: `String key` and `string key` are one parameter type. */ +const CSHARP_KEYWORD_TYPES: Readonly> = { + String: 'string', Object: 'object', Boolean: 'bool', Byte: 'byte', SByte: 'sbyte', Char: 'char', + Decimal: 'decimal', Double: 'double', Single: 'float', Int16: 'short', UInt16: 'ushort', Int32: 'int', + UInt32: 'uint', Int64: 'long', UInt64: 'ulong', IntPtr: 'nint', UIntPtr: 'nuint', +}; + +/** How two parameter types compare: alike, possibly alike, or told apart. */ +type Fit = 'same' | 'maybe' | 'differs'; + +/** + * A filter for one run of the override pass: `(base, methods)` keeps the + * methods that can override `base`, from same-named methods of a class + * implementing or extending its type, in their order: + * + * - those with as many parameters as `base`, never one with another count; + * - of those, the ones whose parameter types are `base`'s, when there are any; + * - else the ones whose types can't be told apart from `base`'s: a type + * parameter of `base`, its class or a class around it stands for any type + * in its place (`save(T entity)` implemented as `save(String entity)`), a + * type with its type arguments dropped or nullable may be the same type (a + * raw `List` for `List`), and the types of a method in another + * language (a Kotlin class implementing a Java interface) aren't compared + * at all. + * + * Every method, as before, when `base`'s parameters can't be read or its + * language isn't one of {@link PARAMETER_OVERRIDE_LANGUAGES}; a method whose + * own parameters can't be read is kept with those that might match. + */ +export function overrideFilter(context: ResolutionContext): (base: Node, methods: readonly Node[]) => Node[] { + const types = new Map(); + const typesOf = (m: Node): readonly string[] | null => { + let found = types.get(m.id); + if (found === undefined) { + found = parameterTypes(m, context); + types.set(m.id, found); + } + return found; + }; + // Asked of a base method only: an override's own type parameter facing a + // concrete type of the base is no override in these languages (` A + // appendTo(A, …)` does not override `appendTo(StringBuilder, …)`), and the + // overrides of one base sit in many files that would each be read. + const scopes = new Map>(); + const scopeOf = (m: Node): ReadonlySet => { + let found = scopes.get(m.id); + if (!found) { + found = typeParametersInScope(m, context); + scopes.set(m.id, found); + } + return found; + }; + return (base, methods) => { + const want = PARAMETER_OVERRIDE_LANGUAGES.has(base.language) ? typesOf(base) : null; + if (!want) return [...methods]; + const same: Node[] = []; + const maybe: Node[] = []; + for (const m of methods) { + const have = PARAMETER_OVERRIDE_LANGUAGES.has(m.language) ? typesOf(m) : null; + if (!have) { + maybe.push(m); + continue; + } + if (have.length !== want.length) continue; + if (m.language !== base.language) { + maybe.push(m); + continue; + } + let fit: Fit = 'same'; + for (let i = 0; i < want.length && fit !== 'differs'; i++) { + const f = compareTypes(want[i]!, () => scopeOf(base), have[i]!, base.language); + if (f !== 'same') fit = f; + } + if (fit === 'same') same.push(m); + else if (fit === 'maybe') maybe.push(m); + } + return same.length > 0 ? same : maybe; + }; +} + +/** + * How parameter type `a` of a base method compares with `b`, the override's + * type in the same place. `aScope` gives the type parameters the base can + * name, asked for only when nothing cheaper decides: reading them reads the + * base's source. + */ +function compareTypes(a: string, aScope: () => ReadonlySet, b: string, language: string): Fit { + if (a === b) return 'same'; + if (looseType(a, language) === looseType(b, language)) return 'maybe'; + return fitsWithTypeParameters(a, aScope(), b) ? 'maybe' : 'differs'; +} + +/** Past this length a type is not matched against a pattern: each type parameter in it is a `.+`. */ +const MAX_PATTERN_TYPE_LENGTH = 200; + +/** + * Whether `type` is `pattern` with each type parameter it names filled in: + * `List` is `List`, and `String` is `T`. A type too long to match + * cheaply might be, and so fits. + */ +function fitsWithTypeParameters(pattern: string, typeParameters: ReadonlySet, type: string): boolean { + if (typeParameters.size === 0) return false; + if (pattern.length > MAX_PATTERN_TYPE_LENGTH || type.length > MAX_PATTERN_TYPE_LENGTH) { + return [...typeParameters].some((name) => new RegExp(`(? { + if (i % 2 === 1 && typeParameters.has(part)) { + named = true; + return '.+'; + } + return escapeRegExp(part); + }) + .join(''); + return named && new RegExp(`^${source}$`).test(type); +} + +/** A type without its type arguments and nullable marks: `List?` is `List`. */ +function looseType(type: string, language: string): string { + const args = language === 'scala' ? /\[[^[\]]*\]/g : /<[^<>]*>/g; + let t = type; + for (let previous = ''; previous !== t; ) { + previous = t; + t = t.replace(args, ''); + } + return t.replace(/\?/g, ''); +} + +/** The parameter types `method` declares, each normalized (normalizeType); null when they can't be read. */ +function parameterTypes(method: Node, context: ResolutionContext): readonly string[] | null { + const list = parameterList(method, context); + if (list === null) return null; + const types: string[] = []; + for (const param of splitParameters(list)) { + const type = parameterType(param, method.language); + if (type === RECEIVER) continue; + if (type === null) return null; + types.push(normalizeType(type, method.language)); + } + return types; +} + +/** The text of `method`'s parameter list, without its parentheses; null when it can't be found. */ +function parameterList(method: Node, context: ResolutionContext): string | null { + if (method.language === 'csharp') return csharpParameterList(method, context); + const sig = method.signature?.trim(); + if (!sig) return null; + if (method.language === 'java') { + // The return type, then the list: `Object (List tairKey, double count)`. + const groups = parenGroups(sig); + const last = groups[groups.length - 1]; + return last && last[1] === sig.length - 1 ? sig.slice(last[0] + 1, last[1]) : null; + } + // Kotlin and Scala: the list, then the return type: `(keys: List): Any?`. + // A Scala `def size: Int` takes no parameter list at all. + if (method.language === 'scala' && sig.startsWith(':')) return ''; + const first = parenGroups(sig)[0]; + return first && first[0] === 0 ? sig.slice(1, first[1]) : null; +} + +/** + * The parameter list of a C# method, read from its declaration: past its + * attribute sections (`[Obsolete("Put(int) instead")]`), the `( … )` after + * its name — and type parameters, `Save(…)` — with comments and string + * literals blanked. An explicit interface implementation, `object + * IStore.Put(string key)`, is named `Put` like any other. + */ +function csharpParameterList(method: Node, context: ResolutionContext): string | null { + const lines = context.getFileLines?.(method.filePath) ?? context.readFile(method.filePath)?.split(/\r?\n/) ?? null; + if (!lines || method.startLine < 1 || method.startLine > lines.length) return null; + const end = Math.min(lines.length, Math.max(method.endLine, method.startLine), method.startLine - 1 + CSHARP_HEAD_LINES); + const first = lines[method.startLine - 1]!; + const head = blankCsharpTrivia( + [first.slice(Math.min(method.startColumn, first.length)), ...lines.slice(method.startLine, end)].join('\n') + ); + let at = 0; + for (;;) { + while (at < head.length && /\s/.test(head[at]!)) at++; + if (head[at] !== '[') break; + const close = closingBracket(head, at); + if (close < 0) return null; + at = close + 1; + } + const name = new RegExp(`(?()]*>)?\\s*\\(`, 'g'); + name.lastIndex = at; + const found = name.exec(head); + if (!found) return null; + const open = found.index + found[0].length - 1; + const close = closingBracket(head, open); + return close < 0 ? null : head.slice(open + 1, close); +} + +/** C# source with its comments and the contents of its string and character literals blanked, newlines kept. */ +function blankCsharpTrivia(text: string): string { + let out = ''; + for (let i = 0; i < text.length; i++) { + const c = text[i]!; + if (c === '/' && (text[i + 1] === '/' || text[i + 1] === '*')) { + const close = text[i + 1] === '/' ? text.indexOf('\n', i) : text.indexOf('*/', i + 2); + const stop = close < 0 ? text.length : text[i + 1] === '/' ? close : close + 2; + out += text.slice(i, stop).replace(/[^\n]/g, ' '); + i = stop - 1; + continue; + } + if (c === '"' || c === "'") { + // `@"…"` and `$@"…"` escape a quote by doubling it; the rest by a backslash. + const verbatim = c === '"' && (text[i - 1] === '@' || (text[i - 2] === '@' && text[i - 1] === '$')); + let j = i + 1; + while (j < text.length) { + if (verbatim ? text[j] === '"' && text[j + 1] === '"' : text[j] === '\\') { + j += 2; + continue; + } + if (text[j] === c || (!verbatim && text[j] === '\n')) break; + j++; + } + out += c + text.slice(i + 1, j).replace(/[^\n]/g, ' ') + (text[j] ?? ''); + i = j; + continue; + } + out += c; + } + return out; +} + +/** A parameter that is no parameter: Java's receiver, `void m(Foo this, int x)`. */ +const RECEIVER = Symbol('receiver'); + +/** The type text a parameter declares, as written; null when it can't be read. */ +function parameterType(param: string, language: string): string | null | typeof RECEIVER { + if (language === 'csharp') return csharpParameterType(param); + const bare = stripAnnotations(param).trim(); + if (language === 'java') { + // `final List lines`, `String... parts`, a C-style `int data[]`. + const m = /^([\s\S]*?)([A-Za-z_$][\w$]*)((?:\s*\[\s*\])*)$/.exec(bare.replace(/\bfinal\b/g, ' ').trim()); + if (!m || !m[1]!.trim()) return null; + if (m[2] === 'this') return RECEIVER; + return m[1]! + m[3]!.replace(/\s+/g, ''); + } + // Kotlin and Scala: `vararg keys: String`, `count: Double = 1.0`, `f: => Unit`. + const colon = topLevelIndex(bare, (i) => bare[i] === ':'); + if (colon < 0) return null; + let type = bare.slice(colon + 1); + const assign = topLevelIndex(type, (i) => type[i] === '=' && type[i + 1] !== '>'); + if (assign >= 0) type = type.slice(0, assign); + if (!type.trim()) return null; + return /\bvararg\b/.test(bare.slice(0, colon)) ? `vararg ${type}` : type; +} + +/** + * A C# parameter's type: past its attributes and the modifiers that are no + * part of it (`this`, `params`, `scoped`), keeping `ref`, `out` and `in`, + * which are, and before its name and default value. + */ +function csharpParameterType(param: string): string | null { + let p = param.trim(); + while (p.startsWith('[')) { + const close = closingBracket(p, 0); + if (close < 0) return null; + p = p.slice(close + 1).trim(); + } + const assign = topLevelIndex(p, (i) => p[i] === '=' && !/[=!<>]/.test(p[i - 1] ?? '') && !/[=>]/.test(p[i + 1] ?? '')); + if (assign >= 0) p = p.slice(0, assign).trim(); + const m = /^([\s\S]*?\S)\s+@?[A-Za-z_]\w*$/.exec(p); + if (!m) return null; + let type = m[1]!.trim(); + const kept: string[] = []; + for (let mod = /^(this|params|scoped|ref|out|in|readonly)\s+/.exec(type); mod; mod = /^(this|params|scoped|ref|out|in|readonly)\s+/.exec(type)) { + if (mod[1] !== 'this' && mod[1] !== 'params' && mod[1] !== 'scoped') kept.push(mod[1]!); + type = type.slice(mod[0].length); + } + return type ? [...kept, type].join(' ') : null; +} + +/** + * A parameter type as the languages compare it: qualifiers dropped + * (`java.util.List` is `List`), Java's varargs an + * array, C#'s BCL names its keywords (`String` is `string`), and whitespace + * kept only between two words (`out T`, `vararg String`). + */ +function normalizeType(type: string, language: string): string { + let t = type.replace(/\s+/g, ' ').trim(); + if (language === 'java') t = t.replace(/\.\.\./g, '[]'); + t = t.replace(/\b[A-Za-z_]\w*\s*::\s*/g, ''); + t = t.replace(/\b(?:[A-Za-z_$][\w$]*\s*\.\s*)+(?=[A-Za-z_$])/g, ''); + if (language === 'csharp') t = t.replace(/\b[A-Z][A-Za-z0-9]*\b/g, (w) => CSHARP_KEYWORD_TYPES[w] ?? w); + return t.replace(/ (?![\w$])|(? text[i] === ','); + const param = (comma < 0 ? list.slice(start) : list.slice(start, start + comma)).trim(); + if (param) params.push(param); + if (comma < 0) return params; + start += comma + 1; + } +} + +/** + * The first index in `text` outside brackets and string literals where `at` + * holds, or -1. A `<` opens type arguments when written right after a name + * (`List<`), and the `>` of `->` and `=>` closes none. + */ +function topLevelIndex(text: string, at: (i: number, text: string) => boolean): number { + let depth = 0; + let angles = 0; + for (let i = 0; i < text.length; i++) { + const c = text[i]!; + if (c === '"' || c === "'") { + i = closingQuote(text, i); + continue; + } + if (depth === 0 && angles === 0 && at(i, text)) return i; + if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') depth = Math.max(0, depth - 1); + else if (c === '<' && /[\w$?\]>]/.test(text[i - 1] ?? '') && text[i + 1] !== '<' && text[i + 1] !== '=') angles++; + else if (c === '>' && angles > 0 && text[i - 1] !== '-' && text[i - 1] !== '=') angles--; + } + return -1; +} + +/** The top-level `( … )` groups of `text`, as [open, close] index pairs; string literals skipped. */ +function parenGroups(text: string): Array<[number, number]> { + const groups: Array<[number, number]> = []; + for (let i = 0; i < text.length; i++) { + const c = text[i]!; + if (c === '"' || c === "'") i = closingQuote(text, i); + else if (c === '(') { + const close = closingBracket(text, i); + if (close < 0) return groups; + groups.push([i, close]); + i = close; + } + } + return groups; +} + +/** The index of the bracket closing the one at `open` — `(`, `[` or `{` — or -1; string literals skipped. */ +function closingBracket(text: string, open: number): number { + let depth = 0; + for (let i = open; i < text.length; i++) { + const c = text[i]!; + if (c === '"' || c === "'") i = closingQuote(text, i); + else if (c === '(' || c === '[' || c === '{') depth++; + else if ((c === ')' || c === ']' || c === '}') && --depth === 0) return i; + } + return -1; +} + +/** The index of the quote closing the string or character literal opening at `open`, or the text's end. */ +function closingQuote(text: string, open: number): number { + const quote = text[open]!; + let j = open + 1; + while (j < text.length && text[j] !== quote) j += text[j] === '\\' ? 2 : 1; + return Math.min(j, text.length - 1); +} diff --git a/src/resolution/type-parameters.ts b/src/resolution/type-parameters.ts index 42a2a63edb..91f1d8ba49 100644 --- a/src/resolution/type-parameters.ts +++ b/src/resolution/type-parameters.ts @@ -149,10 +149,23 @@ function scalaMemberInScope(result: ResolvedRef, ref: UnresolvedRef, context: Re return /^[A-Za-z_]\w*$/.test(owner.name) && /^[A-Za-z_]\w*$/.test(target.name) ? imports.test(text) : true; } -function enclosingDeclarations(ref: UnresolvedRef, context: ResolutionContext): Node[] { +function enclosingDeclarations(at: { filePath: string; line: number }, context: ResolutionContext): Node[] { return context - .getNodesInFile(ref.filePath) - .filter((n) => DECLARATION_KINDS.has(n.kind) && n.startLine <= ref.line && n.endLine >= ref.line); + .getNodesInFile(at.filePath) + .filter((n) => DECLARATION_KINDS.has(n.kind) && n.startLine <= at.line && n.endLine >= at.line); +} + +/** + * The type parameters in scope at a declaration: its own and those of the + * declarations around it — a method's (` T map(Class type)`), its + * class's (`interface Repository`), an enclosing class's. + */ +export function typeParametersInScope(node: Node, context: ResolutionContext): ReadonlySet { + const names = new Set(); + for (const declaration of enclosingDeclarations({ filePath: node.filePath, line: node.startLine }, context)) { + for (const name of declaredTypeParameters(declaration, context).keys()) names.add(name); + } + return names; } /** The type parameters a declaration declares, each with the default a C++ head gives it (`typename Parser = TcParser`). */ From d5fc1fbb21689b78b89e1ffb1144b6e44931b806 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 01:50:54 +0000 Subject: [PATCH 52/95] fix(python): calls like reg.get() on a typed local or parameter reach the class's method (#2821) In Python, a call of a method named like a builtin dict, list, string or file method (get, update, items, pop, close and the rest of the builtin-method list) was dropped by the builtin filter before any strategy could read its receiver. Only a receiver whose capitalized name is a project class, or an imported project module, got past it. So `reg = SystemPackRegistry(); reg.get(k)`, `registry = load_registry()` with `def load_registry() -> SystemPackRegistry`, and a parameter `reg: SystemPackRegistry` gave no edge, while `self.reg.get()` linked through the attribute reader of PR #1926. A call the filter drops now gets one way back, by evidence only: matchPythonTypedLocalCall. It reads the def that binds the receiver where the call reads it (the calling def, or one it is nested in). It types the name only when every binding there gives it the same project class: - a parameter gives what its annotation names ("T", Optional[T], T | None); - an assignment gives its own annotation, or what its whole value makes: a constructor, or a factory typed by its return annotation or by its returns (the attribute reader's pythonValueEvidence and pythonFactoryType); - `x = None` gives nothing either way; - each class is the one the file's imports name (pythonClassNamed), so a class imported from outside the project is none of the project's; - the method must be the class's own or a declared base's. Everything else stays dropped: any other binding of the name (a loop, with, except or comprehension target, a tuple or augmented assignment, a walrus, an import, del, a nested def or class of the name, a nonlocal write from a nested def), a value that names no class ({}, json.load(p), Foo().data(), a union), a rebound callee, a module-level name. These are the shapes the reporter's follow-up found making wrong edges with a looser reader. Like an attribute call, the edge records the other files its answer read (typeFrom), so a sync that edits the factory or a base re-opens it. Validation: - New __tests__/python-typed-builtin-method-calls.test.ts, 20 tests: - the issue's three calls; annotated, quoted and Optional parameters; items, pop and close; a module-qualified constructor; typed, untyped and awaited factories; a None placeholder; a local annotation; a closure; typeFrom; two same-named classes; an inherited method; - the reporter's trap shapes, which stay unlinked; - three sync cases, each matching a fresh index. With main's source the 13 positive tests fail and the 7 trap tests pass. - A/B against main 3f58e343. Both arms were built from their own sources with the kernel. Rebased onto e32021f6 (PR #2820, outside Python), the fix arm's graphs are byte-identical to before the rebase: - requests: +15 edges; - sqlalchemy: +52 edges; - django's 2,938 Python files: +73 edges; - 0 removed and no other change in any of them; - express and gin (controls): byte-identical. All 140 added edges were read at their source: each is a constructed local, a factory-typed local or an annotated parameter calling its own class's method, or one a declared base has. The issue's fixtures link all three calls, and the verification kit's nine trap call sites stay unresolved, kernel and wasm alike. - Cost: a CPU profile puts the new reader at about 1.0 s on django and 1.2 s on sqlalchemy, around 1% of index CPU. Index CPU medians of 3 interleaved runs, main vs fix: sqlalchemy 98.6 s vs 98.5 s, django 97.9 s vs 99.1 s (single runs vary by several seconds on this shared machine). - kernel-parity.mjs: 295/295 Python files at parity. Fixes #2583 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../python-typed-builtin-method-calls.test.ts | 491 ++++++++++++++++++ src/resolution/index.ts | 19 +- src/resolution/name-matcher.ts | 252 +++++++++ 4 files changed, 760 insertions(+), 3 deletions(-) create mode 100644 __tests__/python-typed-builtin-method-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 15f9099acd..f98f1a3d90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, `import a.b as X` written inside a function now links its calls like `X.f()`, as the same import at module level already did. Thanks @Egrn for the report. (#2600) - In Python, `from app import email` no longer links the importing file to a project variable named `app`, such as a FastAPI or Flask `app = FastAPI()`. (#2584) - In Python, a call through a module-level instance, like `_svc.delete(obj)` after `_svc = SceneService()`, now links to the method its class has or inherits, even when other classes have a method of that name. (#2582) +- In Python, a call of your own class's method named like a dictionary, list or file method, such as `get`, `update`, `items`, `pop` or `close`, now links to that method when the function says what the variable holds: `reg = SystemPackRegistry()` then `reg.get(…)`, `registry = load_registry()` with `def load_registry() -> SystemPackRegistry`, or a parameter written `reg: SystemPackRegistry`. Before, these calls were dropped as built-in methods unless the variable was named after the class. A variable the function also sets to something else, like `{}` or what `json.load` returns, a class imported from another package, like `queue.Queue`, and a union type still link nothing. Re-index Python projects after upgrading. Thanks @inth3shadows for the report. (#2583) - In Python, a call on a module imported from outside your project, like `subprocess.run(…)` after `import subprocess`, no longer links to a project method that happens to share its name. Thanks @Egrn for the report. (#2599) - `codegraph_explore`, the CLI `codegraph explore` and the prompt hook no longer stall for a minute or more on a large project after a full index; the step that finds the project's busiest file no longer slows to a crawl when a few files are huge, like a generated Android `R.java` (#2571). Thanks @yangfengfan2011. - `codegraph_explore`, the CLI `codegraph explore` and the Claude Code prompt hook now answer a long query, like a pasted log, document or source file, in seconds on a large project instead of minutes. Every word of such a query used to search the whole index several times over, so the prompt hook ran out of time on long prompts and added nothing. Those searches now reuse what they read, and a very long query is searched by its code names and its opening words rather than by every word in it. Answers to ordinary questions and lists of symbol names are unchanged. Thanks @retsamekohC for the report and the first fix. (#2620) diff --git a/__tests__/python-typed-builtin-method-calls.test.ts b/__tests__/python-typed-builtin-method-calls.test.ts new file mode 100644 index 0000000000..8f826a33a0 --- /dev/null +++ b/__tests__/python-typed-builtin-method-calls.test.ts @@ -0,0 +1,491 @@ +/** + * Python calls of builtin-named methods through a typed local or parameter + * (#2583). + * + * `get`, `update`, `items`, `pop`, `close` and the other dict/list/file method + * names are dropped as builtins before any strategy can type the receiver. + * When the function that binds the receiver says what it holds — a + * constructor or a typed factory as the whole value, or a parameter's + * annotation — the call is that project class's method. The negative cases + * are the shapes where such typing produced a wrong edge in the reporter's + * follow-up: the type's import, a later rebinding, a chained constructor, + * module-level code, a union. Those stay unlinked. + */ +import { describe, it, expect, beforeAll, afterAll, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let dir: string; +let cg: CodeGraph; + +const files: Record = { + // The issue's own files. + 'loader.py': `class SystemPackRegistry: + def get(self, pack_id): + return pack_id + + def update(self): + return None + + def items(self): + return [] + + def pop(self, key): + return key + + def close(self): + return None + + +def load_registry() -> SystemPackRegistry: + return SystemPackRegistry() + + +def make_registry(): + return SystemPackRegistry() + + +async def open_registry() -> "SystemPackRegistry": + return SystemPackRegistry() +`, + 'test_packs.py': `from loader import SystemPackRegistry, load_registry + + +def test_pack(): + registry = load_registry() + return registry.get("x") + + +def test_ctor(): + reg = SystemPackRegistry() + reg.update() + return reg.get("y") +`, + 'more.py': `from typing import Optional +import loader +from loader import SystemPackRegistry, make_registry, open_registry + + +def annotated(reg: SystemPackRegistry): + reg.get("a") + reg.items() + + +def quoted(reg: "SystemPackRegistry"): + reg.pop("b") + + +def optional(reg: Optional[SystemPackRegistry] = None): + reg.close() + + +def ctor_more(): + reg = SystemPackRegistry() + reg.items() + reg.pop("d") + reg.close() + + +def through_module(): + reg = loader.SystemPackRegistry() + reg.items() + + +def untyped_factory(): + reg = make_registry() + reg.update() + + +async def awaited_factory(): + reg = await open_registry() + reg.close() + + +def none_first(flag): + reg = None + if flag: + reg = SystemPackRegistry() + reg.update() + + +def annotated_local(): + reg: SystemPackRegistry = make_registry() + reg.pop("e") + + +def closure(): + reg = SystemPackRegistry() + + def inner(): + return reg.get("f") + return inner +`, + // Two project classes named Handler: the import says which one. + 'a.py': `class Handler: + def get(self): + return "a" +`, + 'b.py': `class Handler: + def get(self): + return "b" +`, + 'handlers.py': `from b import Handler + + +def handle(): + h = Handler() + return h.get() +`, + // A method a declared base in another file has. + 'base.py': `class BaseStore: + def update(self, data): + return data +`, + 'derived.py': `from base import BaseStore + + +class Derived(BaseStore): + pass + + +def refresh(): + d = Derived() + d.update({}) +`, + // The reporter's follow-up: shapes the receiver typing must not link. + 'models.py': `class Queue: + def get(self): + return None + + +class Session: + def get(self, url): + return url + + def close(self): + return None + + +class Store: + def get(self, key): + return key + + def update(self, data): + return data + + def items(self): + return [] + + +class Foo: + def build(self): + return self + + def data(self): + return {} + + +class Counter: + def update(self, other): + return other + + +class StringIO: + def read(self): + return "" +`, + 'traps.py': `import json +from queue import Queue +from requests import Session +from collections import Counter +from io import StringIO +from models import Store, Foo + + +def outside_import(): + q = Queue() + q.get() + s = Session() + s.get("http://x") + c = Counter() + c.update("abc") + buf = StringIO() + buf.read() + + +def rebinding(p): + reg = Store() + reg = {} + reg.get("k") + data = Store() + data = json.load(p) + data.update({}) + + +def branches(flag, rows, path): + a = Store() if flag else {} + a.get("k") + if flag: + b = Store() + else: + b = {} + b.get("k") + c = Store() + for c in rows: + c.items() + d = Store() + with open(path) as d: + d.get("k") + e = Store() + return [e.get("k") for e in rows] + + +def chained(): + x = Foo().data() + x.get("k") + y = Foo().build() + y.get("k") + + +def unions(reg: "Store | dict"): + if isinstance(reg, dict): + reg.get("k") + + +def unannotated(reg): + reg.get("k") + + +def rebound_callee(load): + Store = load() + reg = Store() + reg.get("k") + + +_store = Store() +_store = {} + + +def rebound_global(): + _store.get("k") +`, + 'modlevel.py': `from models import Store + + +def make(): + reg = Store() + return reg + + +reg = {} +reg.get("k") +`, + 'outside_base.py': `from somepkg import BaseStore + + +class Child(BaseStore): + pass + + +def use_child(): + ch = Child() + ch.update({}) +`, +}; + +beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-2583-')); + for (const [rel, text] of Object.entries(files)) { + fs.writeFileSync(path.join(dir, rel), text); + } + cg = await CodeGraph.init(dir, { index: true }); +}); + +afterAll(() => { + cg?.destroy(); + fs.rmSync(dir, { recursive: true, force: true }); +}); + +/** The `calls` targets of the named function or method in `file`, as `qualifiedName@file`, sorted. */ +function callsFrom(file: string, qualifiedName: string): string[] { + const src = cg.getNodesInFile(file).find((n) => n.qualifiedName === qualifiedName && + (n.kind === 'function' || n.kind === 'method')); + expect(src, `${file}:${qualifiedName}`).toBeDefined(); + return cg.getOutgoingEdges(src!.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!) + .filter((n) => n.kind === 'method') + .map((n) => `${n.qualifiedName}@${n.filePath}`) + .sort(); +} + +/** The `calls` edges into the file-level code of `file`, to methods. */ +function moduleLevelCalls(file: string): string[] { + const fileNode = cg.getNodesInFile(file).find((n) => n.kind === 'file'); + expect(fileNode).toBeDefined(); + return cg.getOutgoingEdges(fileNode!.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!) + .filter((n) => n.kind === 'method') + .map((n) => `${n.qualifiedName}@${n.filePath}`); +} + +describe('Python builtin-named methods through typed locals and parameters (#2583)', () => { + it('links the issue\'s constructor-typed local', () => { + expect(callsFrom('test_packs.py', 'test_ctor')).toEqual([ + 'SystemPackRegistry::get@loader.py', + 'SystemPackRegistry::update@loader.py', + ]); + }); + + it('links a local typed by its factory\'s return annotation', () => { + expect(callsFrom('test_packs.py', 'test_pack')).toEqual(['SystemPackRegistry::get@loader.py']); + }); + + it('links calls through an annotated parameter, quoted or Optional too', () => { + expect(callsFrom('more.py', 'annotated')).toEqual([ + 'SystemPackRegistry::get@loader.py', + 'SystemPackRegistry::items@loader.py', + ]); + expect(callsFrom('more.py', 'quoted')).toEqual(['SystemPackRegistry::pop@loader.py']); + expect(callsFrom('more.py', 'optional')).toEqual(['SystemPackRegistry::close@loader.py']); + }); + + it('links items, pop and close on a constructed local, and one made through the module', () => { + expect(callsFrom('more.py', 'ctor_more')).toEqual([ + 'SystemPackRegistry::close@loader.py', + 'SystemPackRegistry::items@loader.py', + 'SystemPackRegistry::pop@loader.py', + ]); + expect(callsFrom('more.py', 'through_module')).toEqual(['SystemPackRegistry::items@loader.py']); + }); + + it('links a local typed by what a factory returns, awaited or not', () => { + expect(callsFrom('more.py', 'untyped_factory')).toEqual(['SystemPackRegistry::update@loader.py']); + expect(callsFrom('more.py', 'awaited_factory')).toEqual(['SystemPackRegistry::close@loader.py']); + }); + + it('reads past a None placeholder and trusts a local annotation', () => { + expect(callsFrom('more.py', 'none_first')).toEqual(['SystemPackRegistry::update@loader.py']); + expect(callsFrom('more.py', 'annotated_local')).toEqual(['SystemPackRegistry::pop@loader.py']); + }); + + it('links a call in a nested function through the enclosing function\'s local', () => { + expect(callsFrom('more.py', 'closure::inner')).toEqual(['SystemPackRegistry::get@loader.py']); + }); + + it('records the factory\'s file for a sync to re-open the call', () => { + const src = cg.getNodesInFile('test_packs.py').find((n) => n.qualifiedName === 'test_pack')!; + const edge = cg.getOutgoingEdges(src.id).find((e) => e.kind === 'calls' && cg.getNode(e.target)?.name === 'get'); + expect(edge?.metadata?.typeFrom).toEqual(['loader.py']); + }); + + it('takes the class the import names when two project classes share its name', () => { + expect(callsFrom('handlers.py', 'handle')).toEqual(['Handler::get@b.py']); + }); + + it('links a method a declared base in another file has', () => { + expect(callsFrom('derived.py', 'refresh')).toEqual(['BaseStore::update@base.py']); + }); + + it('links nothing for a class imported from outside the project that a project class is named like', () => { + expect(callsFrom('traps.py', 'outside_import')).toEqual([]); + }); + + it('links nothing for a local rebound to a dict or to what json.load returns', () => { + expect(callsFrom('traps.py', 'rebinding')).toEqual([]); + }); + + it('links nothing for a local bound in a branch, a loop, a with or a comprehension', () => { + expect(callsFrom('traps.py', 'branches')).toEqual([]); + }); + + it('links nothing more for a local holding what a method of a constructed object returns', () => { + expect(callsFrom('traps.py', 'chained')).toEqual(['Foo::build@models.py', 'Foo::data@models.py']); + }); + + it('links nothing for a union, an unannotated parameter, a rebound class name or a rebound global', () => { + expect(callsFrom('traps.py', 'unions')).toEqual([]); + expect(callsFrom('traps.py', 'unannotated')).toEqual([]); + expect(callsFrom('traps.py', 'rebound_callee')).toEqual([]); + expect(callsFrom('traps.py', 'rebound_global')).toEqual([]); + }); + + it('links nothing for module-level code, whatever a function above binds', () => { + expect(moduleLevelCalls('modlevel.py')).toEqual([]); + }); + + it('links nothing for a method a base from outside the project could hold', () => { + expect(callsFrom('outside_base.py', 'use_child')).toEqual([]); + }); +}); + +describe('sync after an edit to the file a typed local reads (#2583)', () => { + let tmp: string; + let graph: CodeGraph | undefined; + + afterEach(() => { + graph?.close(); + graph = undefined; + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + const write = (map: Record) => { + for (const [rel, text] of Object.entries(map)) fs.writeFileSync(path.join(tmp, rel), text); + }; + + const getCalls = (g: CodeGraph, qualifiedName: string): string[] => { + const src = g.getNodesInFile('use.py').find((n) => n.qualifiedName === qualifiedName)!; + return g.getOutgoingEdges(src.id).filter((e) => e.kind === 'calls') + .map((e) => g.getNode(e.target)!).filter((n) => n.kind === 'method') + .map((n) => `${n.qualifiedName}@${n.filePath}`).sort(); + }; + + /** Syncs, then checks the synced graph against a fresh index of the same files. */ + const syncedMatchesFresh = async (qualifiedName: string): Promise => { + await graph!.sync(); + const synced = getCalls(graph!, qualifiedName); + graph!.close(); + fs.rmSync(path.join(tmp, '.codegraph'), { recursive: true, force: true }); + graph = await CodeGraph.init(tmp, { index: true }); + expect(synced).toEqual(getCalls(graph, qualifiedName)); + return synced; + }; + + const USE = 'from loader import Registry, make\n\n\ndef ctor():\n reg = Registry()\n return reg.get("k")\n\n\ndef factory():\n reg = make()\n return reg.get("k")\n'; + + it('re-types the local when the factory changes what it returns', async () => { + tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-2583-sync-')); + const classes = 'class Registry:\n def get(self, k):\n return k\n\n\nclass Other:\n def get(self, k):\n return k\n\n\n'; + write({ 'loader.py': `${classes}def make() -> Registry:\n return Registry()\n`, 'use.py': USE }); + graph = await CodeGraph.init(tmp, { index: true }); + expect(getCalls(graph, 'factory')).toEqual(['Registry::get@loader.py']); + write({ 'loader.py': `${classes}def make() -> Other:\n return Other()\n` }); + expect(await syncedMatchesFresh('factory')).toEqual(['Other::get@loader.py']); + }); + + it('types the local once the factory gains a return annotation', async () => { + tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-2583-sync-')); + const cls = 'class Registry:\n def get(self, k):\n return k\n\n\n'; + write({ 'loader.py': `${cls}def make(kind):\n return kind()\n`, 'use.py': USE }); + graph = await CodeGraph.init(tmp, { index: true }); + expect(getCalls(graph, 'factory')).toEqual([]); + write({ 'loader.py': `${cls}def make(kind) -> Registry:\n return kind()\n` }); + expect(await syncedMatchesFresh('factory')).toEqual(['Registry::get@loader.py']); + }); + + it('links the call once the class gains the method, and drops it when the class loses it', async () => { + tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-2583-sync-')); + const without = 'class Registry:\n def other(self):\n pass\n\n\ndef make():\n return Registry()\n'; + const withGet = 'class Registry:\n def other(self):\n pass\n\n def get(self, k):\n return k\n\n\ndef make():\n return Registry()\n'; + write({ 'loader.py': without, 'use.py': USE }); + graph = await CodeGraph.init(tmp, { index: true }); + expect(getCalls(graph, 'ctor')).toEqual([]); + write({ 'loader.py': withGet }); + expect(await syncedMatchesFresh('ctor')).toEqual(['Registry::get@loader.py']); + write({ 'loader.py': without }); + expect(await syncedMatchesFresh('ctor')).toEqual([]); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 2f7f9cb702..5889a4fa98 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, isImportShadowedByJsBinding, isManualMockPath } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, matchPythonTypedLocalCall, isImportShadowedByJsBinding, isManualMockPath } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -1522,9 +1522,14 @@ export class ReferenceResolver { const superCall = matchJsSuperCall(ref, this.context); if (superCall !== undefined) return superCall; - // Skip built-in/external references + // Skip built-in/external references — except a python call of a + // builtin-named method through a local or a parameter its def types as a + // project class (`reg = Registry(); reg.get(k)`, #2583): that reading, + // by evidence only, is the one way back, and nothing guesses past it. if (this.isBuiltInOrExternal(ref)) { - return null; + return this.isPythonBuiltinNamedMethodCall(ref) + ? this.gateLanguage(matchPythonTypedLocalCall(ref, this.context), ref) + : null; } // CFML component paths in inheritance (#1152): `extends="coldbox.system.web. @@ -3216,6 +3221,14 @@ export class ReferenceResolver { return false; } + /** A python call `x.get(…)` of a method named like a builtin collection, string or file method. */ + private isPythonBuiltinNamedMethodCall(ref: UnresolvedRef): boolean { + if (ref.language !== 'python' || ref.referenceKind !== 'calls') return false; + const dot = ref.referenceName.indexOf('.'); + return dot > 0 && PYTHON_BUILT_IN_METHODS.has(ref.referenceName.slice(dot + 1)) && + !PYTHON_BUILT_IN_TYPES.has(ref.referenceName.slice(0, dot)); + } + /** * Check if reference is to a built-in or external symbol */ diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index c434635e1f..7ab8c30803 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -17856,6 +17856,9 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { PY_BASES_MEMO.delete(context); PY_FILE_CLASSES.delete(context); PY_ATTR_CALL_MEMO.delete(context); + PY_FILE_DEFS.delete(context); + PY_SCOPE_BINDINGS.delete(context); + PY_LOCAL_CLASS_MEMO.delete(context); } function memoPatterns(key: string, build: () => RegExp[]): RegExp[] { @@ -23407,6 +23410,255 @@ function pythonAttrCallOn( return hit.answer ? { ...hit.answer, original: ref } : hit.answer; } +/** + * A call of a builtin-named method — `get`, `update`, `items`, `close` — + * through a local or a parameter (#2583): `reg = SystemPackRegistry(); + * reg.get(k)`, `registry = load_registry()` with `def load_registry() -> + * SystemPackRegistry`, `def f(reg: SystemPackRegistry)`. The builtin filter + * drops such a call by its method's name before any strategy reads the + * receiver, and this is its one way back, by evidence only: the def that binds + * the name where the call reads it (the calling one, or one it is nested in) + * gives it one project class everywhere it binds it (pythonLocalClass), and + * the method is that class's own or a declared base's. A module-level name, + * and one its def leaves unknown or contested, stay dropped. The edge records + * the other files its answer was read from (`typeFrom`), as an attribute + * call's does, so a sync that edits the factory or a base re-opens it. + */ +export function matchPythonTypedLocalCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (ref.language !== 'python' || ref.referenceKind !== 'calls') return null; + const m = /^([A-Za-z_]\w*)\.(\w+)$/.exec(ref.referenceName); + if (!m || m[1] === 'self' || m[1] === 'cls') return null; + const receiver = m[1]!; + const scope = pythonBindingScope(receiver, ref, context); + if (!scope) return null; + const read = new Set(); + const cls = pythonLocalClass(scope, receiver, ref, context, read); + if (!cls || cls === 'runtime') return null; + const hit = pythonMethodOnClass(cls, m[2]!, ref, context, read); + if (!hit) return null; + read.delete(ref.filePath); + return read.size > 0 ? { ...hit, metadata: { typeFrom: [...read].sort() } } : hit; +} + +/** Per context: a python file's functions and methods, the latest-starting first, for the defs around a call. */ +const PY_FILE_DEFS = new WeakMap>(); + +function pythonFileDefs(filePath: string, context: ResolutionContext): Node[] { + const memo = pyMemo(PY_FILE_DEFS, context); + let hit = memo.get(filePath); + if (!hit) { + hit = context.getNodesInFile(filePath) + .filter((n) => n.kind === 'function' || n.kind === 'method') + .sort((a, b) => b.startLine - a.startLine); + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(filePath, hit); + } + return hit; +} + +/** + * Per context: a def and a name → what the def's parameter of that name is + * annotated with (pythonParamAnnotation), and every binding of the name in the + * def's own body. + */ +const PY_SCOPE_BINDINGS = new WeakMap>(); + +function pythonScopeBindings(scope: Node, name: string, context: ResolutionContext): { param: string | null | undefined; bindings: PythonBinding[] } { + const memo = pyMemo(PY_SCOPE_BINDINGS, context); + const key = `${scope.id}\0${name}`; + let hit = memo.get(key); + if (!hit) { + const lines = pythonMemberLines(scope.filePath, context); + const statements = pythonStatementStarts(scope.filePath, context); + // pythonLineBindings binds nothing on a statement that names the name + // only as a receiver or a container (`reg.get(…)`, `reg[k]`), its + // continuation lines included: its own word test rules those out first. + // An import binds the head of a dotted name (`import os.path`), and a star + // import, which binds any name, can't be written in a def. So the lines + // that can't bind skip its per-line work. + const n = escapeRegExp(name); + const word = new RegExp(`(? { + const test = /^\s*(?:from|import)\b/.test(lines[i]!) ? mention : word; + if (test.test(lines[i]!)) return true; + if (!statements[i]) return false; + for (let j = i + 1; j < lines.length && !statements[j]; j++) if (test.test(lines[j]!)) return true; + return false; + }; + const bindings: PythonBinding[] = []; + for (const i of pythonOwnLines(scope, scope.filePath, context)) { + if (names(i)) bindings.push(...pythonLineBindings(lines, i, statements, name)); + } + hit = { param: pythonParamAnnotation(scope.signature ?? '', name), bindings }; + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(key, hit); + } + return hit; +} + +/** + * The def whose local `name` is the one `ref` reads: the innermost def around + * the reference that binds the name, by a parameter or anywhere in its own + * body. Null when no def around it does (a module global), or when one + * declares it `global`. + */ +function pythonBindingScope(name: string, ref: UnresolvedRef, context: ResolutionContext): Node | null { + for (const scope of pythonFileDefs(ref.filePath, context)) { + if (scope.startLine > ref.line || scope.endLine < ref.line) continue; + const { param, bindings } = pythonScopeBindings(scope, name, context); + if (param !== undefined) return scope; + if (bindings.some((b) => b.kind === 'global')) return null; + if (bindings.length > 0) return scope; + } + return null; +} + +/** + * What the parameter `name` of a def with this signature (`(a, b: T = d)`) + * says of its type: the annotation as written, null for a parameter without + * one and for `*name` / `**name` (a tuple, a dict), undefined when the def has + * no parameter of that name. + */ +function pythonParamAnnotation(signature: string, name: string): string | null | undefined { + const open = signature.indexOf('('); + if (open < 0 || !signature.includes(name)) return undefined; + // Parameters split at the commas directly inside the parentheses; strings + // are blanked (same length) so a quoted comma or bracket doesn't count. + const blank = pythonBlankStrings(signature); + const params: Array<[number, number]> = []; + let depth = 0; + let from = open + 1; + for (let i = open; i < blank.length; i++) { + const c = blank[i]!; + if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') { + if (--depth === 0) { params.push([from, i]); break; } + } else if (c === ',' && depth === 1) { + params.push([from, i]); + from = i + 1; + } + } + for (const [start, end] of params) { + const head = /^\s*(\*{0,2})([A-Za-z_]\w*)\s*(:?)/.exec(signature.slice(start, end)); + if (!head || head[2] !== name) continue; + if (head[1] || !head[3]) return null; + // The annotation runs up to the default's `=`, outside brackets. + const at = start + head[0].length; + let d = 0; + let to = end; + for (let i = at; i < end; i++) { + const c = blank[i]!; + if (c === '(' || c === '[' || c === '{') d++; + else if (c === ')' || c === ']' || c === '}') d--; + else if (c === '=' && d === 0) { to = i; break; } + } + return signature.slice(at, to).trim() || null; + } + return undefined; +} + +/** Per context: a def and a name → the class pythonLocalClass reads, and the files read for it. */ +const PY_LOCAL_CLASS_MEMO = new WeakMap>(); + +/** + * The project class the local or parameter `name` of `scope` holds, the same + * for every binding of it there. A parameter holds what its annotation names + * (`"T"`, `Optional[T]` and `T | None` too); an assignment, what its own + * annotation names, or else what its whole value makes — a class's + * constructor, or a factory, typed by its `-> T` or by what it returns + * (pythonValueEvidence, pythonFactoryType) — or what an annotated parameter it + * copies holds; `name = None` says nothing either way. Each class is the one + * the file's imports name (pythonClassNamed), so one from outside the project, + * as `from queue import Queue` gives, is none of the project's ('runtime'). + * + * Anything else that binds the name leaves its type unknown (null): a + * parameter without an annotation, a `for` / `with` / `except` / comprehension + * target, a tuple or augmented assignment, a walrus, an import, `del`, a + * nested `def` or `class` of the name, a `nonlocal` write from a def nested in + * the scope. So does a value that names no class — `{}` is a dict, and + * `json.load(p)`, `Foo().data()` or a union are unknown — and a callee the + * scope or a def around it rebinds (`Store = load(); reg = Store()`). + */ +function pythonLocalClass( + scope: Node, + name: string, + ref: UnresolvedRef, + context: ResolutionContext, + read: Set, +): PythonTypeAnswer { + const memo = pyMemo(PY_LOCAL_CLASS_MEMO, context); + const key = `${scope.id}\0${name}`; + let hit = memo.get(key); + if (!hit) { + const own = new Set(); + hit = { answer: pythonLocalClassUncached(scope, name, ref, context, own), read: [...own] }; + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(key, hit); + } + for (const f of hit.read) read.add(f); + return hit.answer; +} + +function pythonLocalClassUncached( + scope: Node, + name: string, + ref: UnresolvedRef, + context: ResolutionContext, + read: Set, +): PythonTypeAnswer { + const filePath = scope.filePath; + const { param, bindings } = pythonScopeBindings(scope, name, context); + const evidence: PythonEvidence[] = []; + if (param !== undefined) { + const ev = param === null ? null : pythonAnnotationEvidence(param); + if (!ev) return null; + evidence.push(ev); + } + for (const b of bindings) { + if (b.kind !== 'assign') return null; + let ev: PythonAttrEvidence; + if (b.type) { + // `reg: Base = make()` trusts the annotation; `reg: A = B()` contradicts it. + const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; + if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) return null; + ev = pythonAnnotationEvidence(b.type); + } else if (b.value === 'None') { + continue; + } else if (/^[A-Za-z_]\w*$/.test(b.value)) { + // `reg = registry`: what the parameter's annotation names, unless the def rebinds it. + const copied = pythonScopeBindings(scope, b.value, context); + if (!copied.param || copied.bindings.length > 0) return null; + ev = pythonAnnotationEvidence(copied.param); + } else { + const at: UnresolvedRef = { ...ref, line: b.line + 1 }; + if (pythonValueCallees(b.value).some((c) => pythonBindsLocally(c.split('.')[0]!, at, context, false))) return null; + ev = pythonValueEvidence(b.value, null); + } + if (!ev) return null; + evidence.push(ev); + } + // Nothing here can be a project class: a builtin, or no evidence. + if (!evidence.some((ev) => ev !== 'runtime')) return evidence.length > 0 ? 'runtime' : null; + + // A def or class of the name nested in the scope binds it there, a def + // nested in it can rebind the scope's own through `nonlocal`, and `with … as + // (a, name)` binds it in a way pythonLineBindings doesn't read. + if (pythonFileDefs(filePath, context).some((n) => n.id !== scope.id && n.name === name && + n.startLine > scope.startLine && n.endLine <= scope.endLine) || + pythonFileClasses(filePath, context).some((n) => n.name === name && + n.startLine > scope.startLine && n.endLine <= scope.endLine)) return null; + const lines = pythonMemberLines(filePath, context); + const n = escapeRegExp(name); + const nonlocal = new RegExp(`^\\s*nonlocal\\b.*(? pythonEvidenceClass(ev, filePath, context, read, 0, null))); +} + /** * Python chained factory call — `Client.from_env().send()`, * `make_client().send()`, `Client().send()` — the python instance of the #750 From 628f69e275e0131babc2297d32ba9c78cbc76f3c Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 02:12:42 +0000 Subject: [PATCH 53/95] fix(kotlin): repository().pending() links the method of the type repository() declares (#2822) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue #2805: in a Kotlin test class with `private fun repository(): RealRepo`, `repository().pending(1)` went to a test double's `FakeRepo.pending` in another package, and `val repo = repository(); repo.pending(1)` to the interface `Repo.pending`. The extractor records a call on an unqualified call's result by the outer method's bare name (a capitalized inner, `Foo().bar()`, is kept as a chain), so the name matcher took whichever `pending` came first; and the local's type was read only from a constructor or an annotation, so the receiver's name guessed it. The name matcher now reads such a call where it is written (`repository().pending(1)`, `this.repository()…`, `nullable()?.pending()`) and finds the inner function the way Kotlin does: a method of the calling class, of a class it inherits from or one around it, read from their declarations, else a top-level function the file can name. The call is resolved on the type that function's signature declares, including what the type inherits and extensions on it, or links nothing when the type lacks the method: the #750 rule, as for `Foo.factory().bar()`. A local set from such a call alone (`val repo = repository()`) takes that type for the existing typed-local path. Edge cases the A/B surfaced: - a nested type is the one the function names, `Request.Builder` or a plain `Builder` written in `Request`, matched by whole names (`MockResponse.Builder` is no `Response.Builder`); - an extension is the one the calling file can name (Exposed declares `Expression.year()` in three modules); - a type from outside the project, a call inside an extension on one, and an extension the index keeps unattached (on a nullable receiver, or `fun Dns.Call.x()` filed under `Dns`) keep the name match as before. Validation: - New kotlin-call-result-receiver test (9 cases): 7 fail on main, the two that guard unchanged behavior pass on both; all pass with the fix on the kernel and on the wasm path. Mutation checks confirm the owner-path, scoped-nested-type, whole-segment and extension-visibility rules are each exercised. - A/B on main e32021f6, calls by call site, nodes identical: okhttp c317930: 82 retargeted, 176 gained, 3 dropped, 516 confidence only; JetBrains/Exposed 09f814c: 152 retargeted, 138 gained, 1 dropped, 1353 confidence only; nowinandroid 2982e63: 1 confidence only. Controls gin (Go), nest (TypeScript) and mybatis-3 (Java): byte-identical. - Triage: retargets go from name guesses to the declared type (`request().post()` → `Request.Builder.post`, `varchar(…).index()` → `Column.index`, `rowNumber().over()` → `RowNumber.over`, `newReader().readInt()` → `Hpack.Reader.readInt`); gains are locals now typed (`val connection = connect(peer)`); the 4 drops are Kotlin's `apply` scope function, which went to a test helper's or a Gradle plugin's `apply`. Every retarget group was checked against the source. - Cost: whole index, median of 3 interleaved: okhttp 10.5 s -> 10.9 s, Exposed 17.7 s -> 16.8 s, within noise. Fixes #2805 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/kotlin-call-result-receiver.test.ts | 365 ++++++++++++++++++ src/resolution/name-matcher.ts | 229 +++++++++++ 3 files changed, 595 insertions(+) create mode 100644 __tests__/kotlin-call-result-receiver.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f98f1a3d90..07c21e8db6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -201,6 +201,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a struct that embeds another is no longer treated as overriding the methods it redeclares. gin's `Engine` embeds `RouterGroup` and has a `Use` of its own, but calling `Use` on a `RouterGroup` only ever runs `RouterGroup.Use`; before, `codegraph_explore`'s call path and impact went on from it into `Engine.Use` and the code only the engine runs. The same held for every struct or defined type embedded in another that redeclares one of its methods, like etcd's `Mutex` inside `lockerMutex`. A call through an interface still reaches each type that implements it, the embedding struct included. Re-index Go projects after upgrading. - In JavaScript and TypeScript, a subclass's `static` method no longer counts as overriding an instance method of the same name in its base class, or the reverse, since calling one never runs the other. In webpack, `codegraph_explore`'s call path and impact went on from `Module`'s `deserialize(context)` into the `static deserialize(context)` of `NormalModule` and other modules. A static method still overrides its base's static method and an instance method its base's instance method, and JavaScript class members are now marked static, as TypeScript ones already were. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and @yang0228 for the pull request. (#2761) - In Java, Kotlin, C# and Scala, each overload of an interface or base-class method now links only to the overrides with the same parameters. Before, every overload linked to every method of that name in the implementing class, so `ExampleService.putTairValue(String)` also led to `putTairValue(List, double)`, and `codegraph_explore`'s call path, callers and impact went into overloads that call never runs. Where a parameter is a type parameter, like `save(T entity)` implemented as `save(String entity)`, or the class is written in another language than its interface, the number of parameters decides; a method declared once on each side links as before. Re-index Java, Kotlin, C# and Scala projects after upgrading. Thanks @hxj1225 for the report and @danusha2345 for the pull request. (#2663) +- In Kotlin, a call on what one of your functions returns, like `repository().pending(1)` or `val repo = repository(); repo.pending(1)` with a `private fun repository(): RealRepo` in a test class, now links to the method of the type that function declares, including one the type inherits or an extension on it. Before, the chained call went to whichever `pending` the name matched first, like a test double's in another package, and the local was guessed from its name, which reached the interface. The function is found in the calling class, a class it inherits from or one around it, or at the top level, and a nested type like `Request.Builder` is that type, not another class's `Builder`. When the declared type has no such method, the call now links to nothing rather than a method of the same name elsewhere; a type from outside your project is matched by name as before. Re-index Kotlin projects after upgrading. Thanks @mindcipherrr for the report. (#2805) - In JavaScript and TypeScript, a `super.run()` call now links to the method of the class's parent, or of the nearest ancestor that has it, like three.js's `super.copy(source)` reaching `Object3D.copy`, so `codegraph_explore`'s call path and impact follow an override into the method it builds on. Before, such a call inside a method of the same name was dropped, and elsewhere it went to whatever method had the name, often the class's own or a sibling class's in the same file. A call from a static method reaches the parent's static method, and a call to a parent from outside your project, like `HTMLElement`, `Set` or `Promise`, links nothing. Re-index JavaScript and TypeScript projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2497) - In Go, a call through a struct field that holds a function, like prometheus's `d.refreshf(ctx)`, now reaches the functions put in that field, so `codegraph_explore` follows service discovery from `Discovery.Run` into each provider's `refresh` and names the line where each provider wires its method in, like `RefreshF: d.refresh`. The field can be set in a struct literal or by an assignment, copied from another struct's field (`refreshf: opts.RefreshF`), or given a method of an interface value, and two unrelated structs with a field of the same name, like `fn` or `handler`, stay apart. Before, such a flow stopped at the field. Re-index Go projects after upgrading. - In Go, a type defined from other types, like etcd's `type WatchChan <-chan WatchResponse` or gin's `type HandlerFunc func(*Context)` and `type HandlersChain []HandlerFunc`, now links to the types it is built from, so impact on `WatchResponse` or gin's `Context` reaches these types and the code that uses them. Before, such a type linked to nothing. One written through another package, like etcd's `type PutResponse pb.PutResponse`, links to that package's type, and a recursive one like `type stateFn func(*Lexer) stateFn` doesn't link to itself. Re-index Go projects after upgrading. diff --git a/__tests__/kotlin-call-result-receiver.test.ts b/__tests__/kotlin-call-result-receiver.test.ts new file mode 100644 index 0000000000..d74d12712c --- /dev/null +++ b/__tests__/kotlin-call-result-receiver.test.ts @@ -0,0 +1,365 @@ +/** + * A Kotlin call on what an unqualified call returns is a call on the type + * that function declares (#2805). `repository().pending(1)`, with a private + * `fun repository(): RealRepo` in the calling class, went to whichever + * `pending` the name matched first — a test double's `FakeRepo.pending` in + * another package — and `val repo = repository(); repo.pending(1)` went to + * the interface `Repo.pending`, guessed from the local's name. Both now link + * `RealRepo.pending`, through the helper the class declares, inherits or the + * package declares at the top level; a type that has no such method links + * nothing rather than a namesake elsewhere. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const FILES: Record = { + // The issue's four files, verbatim. + 'src/main/kotlin/demo/domain/Repo.kt': `package demo.domain + +interface Repo { + fun pending(id: Long): Int +} +`, + 'src/main/kotlin/demo/data/RealRepo.kt': `package demo.data + +import demo.domain.Repo + +class RealRepo : Repo { + override fun pending(id: Long): Int = 1 +} +`, + 'src/test/kotlin/demo/testutil/FakeRepo.kt': `package demo.testutil + +import demo.domain.Repo + +class FakeRepo : Repo { + override fun pending(id: Long): Int = 2 +} +`, + 'src/test/kotlin/demo/data/RealRepoTest.kt': `package demo.data + +class RealRepoTest { + private fun repository(): RealRepo = RealRepo() + + fun viaConstructor() { + val repo = RealRepo() + repo.pending(1) + } + + fun viaHelperLocal() { + val repo = repository() + repo.pending(1) + } + + fun viaHelperChain() { + repository().pending(1) + } +} +`, + // More helpers and call shapes. + 'src/test/kotlin/demo/data/MoreHelperTest.kt': `package demo.data + +import demo.domain.Repo + +abstract class BaseRepoTest { + protected fun baseRepository(): RealRepo = RealRepo() +} + +class MoreHelperTest : BaseRepoTest() { + private fun repository(): RealRepo = RealRepo() + private fun asInterface(): Repo = RealRepo() + private fun nullable(): RealRepo? = null + + private fun runTest(block: () -> Unit) = block() + + fun viaBaseHelper() { + baseRepository().pending(1) + } + + fun viaInterfaceHelper() { + asInterface().pending(1) + } + + fun viaThis() { + this.repository().pending(1) + } + + fun viaSafeCall() { + nullable()?.pending(1) + } + + fun viaTopLevel() { + freshRepo().pending(1) + } + + fun viaLambdaLocal() { + runTest { + var repo = repository() + repo.pending(1) + } + } + + fun viaMissing() { + repository().missing() + } +} +`, + 'src/test/kotlin/demo/data/Fixtures.kt': `package demo.data + +fun freshRepo(): RealRepo = RealRepo() + +fun RealRepo.describe(): String = "real" + +class Codec { + class Reader { + fun readInt(): Int = 0 + } +} + +// A nested type sharing its name with the one the helper names (in Request.kt); an extension on a nested type. +class MockRequest { + class Builder { + fun post(): Builder = this + } +} + +class Mall { + class Builder { + fun open(): Mall = Mall() + } +} + +// Shop.newBuilder() declares a plain Builder, which is Shop.Builder. +fun Shop.reopen(): Shop = newBuilder().open() + +class Dns { + class Call +} + +fun Dns.Call.toEvents(): Int = 1 + +class Client { + private fun request(): Request.Builder = Request.Builder() + private fun newCall(): Dns.Call = Dns.Call() + + fun send() { + request().post() + } + + fun events() { + newCall().toEvents() + } +} + +class InheritedHelperTest { + private fun audited(): AuditedRepo = AuditedRepo() + private fun repository(): RealRepo = RealRepo() + private fun newReader(): Codec.Reader = Codec.Reader() + private fun text(): StringBuilder = StringBuilder() + + // A nested type the helper names through its outer class. + fun viaNested() { + newReader().readInt() + } + + // A type from outside the project: linked by the method's name, as before. + fun viaOutside() { + text().reverse() + } + + // A method the helper's type inherits; an extension function on it. + fun viaInherited() { + audited().audit() + } + + fun viaExtension() { + repository().describe() + } + + fun viaVisibleExtension() { + repository().label() + } +} +`, + 'src/main/kotlin/demo/data/Shop.kt': `package demo.data + +class Shop { + class Builder { + fun open(): Shop = Shop() + } + + fun newBuilder(): Builder = Builder() +} +`, + 'src/main/kotlin/demo/data/Request.kt': `package demo.data + +class Request { + class Builder { + fun post(): Builder = this + } +} +`, + // The helper's type and its base, in a file indexed after the caller's. + 'src/test/kotlin/demo/data/zz/Audit.kt': `package demo.data + +open class BaseAudit { + fun audit() {} +} + +class AuditedRepo : BaseAudit() +`, + // Two extensions of one name: the caller's package declares one, another package the other. + 'src/test/kotlin/demo/aa/OtherLabels.kt': `package demo.aa + +import demo.data.RealRepo + +fun RealRepo.label(): String = "other" +`, + 'src/test/kotlin/demo/data/zz/Labels.kt': `package demo.data + +fun RealRepo.label(): String = "own" +`, + // A namesake of both methods in an unrelated package. + 'src/test/kotlin/demo/other/Decoy.kt': `package demo.other + +class Decoy { + fun missing() {} + fun pending(id: Long): Int = 3 + fun audit() {} + fun describe(): String = "decoy" + fun readInt(): Int = 3 + fun reverse() {} +} +`, +}; + +let root = ''; +let graph: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-kotlin-call-result-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + graph = await CodeGraph.init(root, { index: true }); +}, 120_000); + +afterAll(() => { + graph?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** The qualified names `owner::fn` calls, sorted. */ +function callees(owner: string, fn: string): string[] { + const caller = graph.getNodesByKind('method').find((m) => + m.qualifiedName.endsWith(`::${owner}::${fn}`) || m.qualifiedName === `${owner}::${fn}`); + expect(caller, `${owner}::${fn}`).toBeDefined(); + return graph + .getOutgoingEdges(caller!.id) + .filter((e) => e.kind === 'calls') + .map((e) => graph.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('a Kotlin call on what an unqualified call returns (#2805)', () => { + it("the issue's local and chained forms reach the helper's declared type", () => { + expect(callees('RealRepoTest', 'viaConstructor')).toEqual(['demo.data::RealRepo::pending']); + expect(callees('RealRepoTest', 'viaHelperLocal')).toEqual([ + 'demo.data::RealRepo::pending', + 'demo.data::RealRepoTest::repository', + ]); + expect(callees('RealRepoTest', 'viaHelperChain')).toEqual([ + 'demo.data::RealRepo::pending', + 'demo.data::RealRepoTest::repository', + ]); + }); + + it('a helper the class inherits, one through `this.`, a safe call and a top-level function', () => { + expect(callees('MoreHelperTest', 'viaBaseHelper')).toEqual([ + 'demo.data::BaseRepoTest::baseRepository', + 'demo.data::RealRepo::pending', + ]); + expect(callees('MoreHelperTest', 'viaThis')).toEqual([ + 'demo.data::MoreHelperTest::repository', + 'demo.data::RealRepo::pending', + ]); + expect(callees('MoreHelperTest', 'viaSafeCall')).toEqual([ + 'demo.data::MoreHelperTest::nullable', + 'demo.data::RealRepo::pending', + ]); + expect(callees('MoreHelperTest', 'viaTopLevel')).toEqual([ + 'demo.data::RealRepo::pending', + 'demo.data::freshRepo', + ]); + }); + + it('a helper declared to return the interface reaches the interface method', () => { + expect(callees('MoreHelperTest', 'viaInterfaceHelper')).toEqual([ + 'demo.data::MoreHelperTest::asInterface', + 'demo.domain::Repo::pending', + ]); + }); + + it('a local set from a helper inside a lambda', () => { + expect(callees('MoreHelperTest', 'viaLambdaLocal')).toEqual([ + 'demo.data::MoreHelperTest::repository', + 'demo.data::MoreHelperTest::runTest', + 'demo.data::RealRepo::pending', + ]); + }); + + it("a method the helper's type inherits, and an extension function on it", () => { + expect(callees('InheritedHelperTest', 'viaInherited')).toEqual([ + 'demo.data::BaseAudit::audit', + 'demo.data::InheritedHelperTest::audited', + ]); + expect(callees('InheritedHelperTest', 'viaExtension')).toEqual([ + 'RealRepo::describe', + 'demo.data::InheritedHelperTest::repository', + ]); + }); + + it('a nested type named through its outer class; a type from outside the project as before', () => { + expect(callees('InheritedHelperTest', 'viaNested')).toEqual([ + 'demo.data::Codec::Reader::readInt', + 'demo.data::InheritedHelperTest::newReader', + ]); + // `StringBuilder.reverse()` is the JDK's: the project's namesake is the + // guess it was on main, left to the strategies this change doesn't touch. + expect(callees('InheritedHelperTest', 'viaOutside')).toEqual([ + 'demo.data::InheritedHelperTest::text', + 'demo.other::Decoy::reverse', + ]); + }); + + it('a nested type of a shared name is the one the helper names', () => { + expect(callees('Client', 'send')).toEqual([ + 'demo.data::Client::request', + 'demo.data::Request::Builder::post', + ]); + expect(callees('Shop', 'reopen')).toEqual([ + 'demo.data::Shop::Builder::open', + 'demo.data::Shop::newBuilder', + ]); + // An extension on a nested type is kept under the outer type's name: found by name, as before. + expect(callees('Client', 'events')).toEqual([ + 'Dns::toEvents', + 'demo.data::Client::newCall', + ]); + }); + + it("an extension the calling file can name, of two alike", () => { + const caller = graph.getNodesByKind('method').find((m) => m.qualifiedName.endsWith('::InheritedHelperTest::viaVisibleExtension'))!; + const targets = graph.getOutgoingEdges(caller.id).filter((e) => e.kind === 'calls').map((e) => graph.getNode(e.target)!) + .filter((n) => n.name === 'label').map((n) => n.filePath); + expect(targets).toEqual(['src/test/kotlin/demo/data/zz/Labels.kt']); + }); + + it('a method the declared type lacks links nothing, not a namesake elsewhere', () => { + expect(callees('MoreHelperTest', 'viaMissing')).toEqual(['demo.data::MoreHelperTest::repository']); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 7ab8c30803..aa3c74d84f 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -17491,6 +17491,219 @@ function importedFqnOf( return imports.find((i) => i.localName === typeName)?.source; } +/** How many lines past its first a Kotlin call is read for its arguments. */ +const KOTLIN_CALL_LINES = 8; + +/** An unqualified Kotlin call's head, up to its `(`: `repository(`, `this.repository(`, `make(`. */ +const KOTLIN_UNQUALIFIED_CALL = /^(?:this\s*\.\s*)?([a-z_]\w*)\s*(?:<[^<>()]*>)?\s*\(/; + +/** The index of the `)` closing the `(` at `open` in Kotlin source, string literals skipped; -1 when it doesn't close. */ +function kotlinClosingParen(text: string, open: number): number { + let depth = 0; + for (let i = open; i < text.length; i++) { + const c = text[i]!; + if (c === '"' || c === "'") { + for (i++; i < text.length && text[i] !== c; i++) if (text[i] === '\\') i++; + } else if (c === '(') depth++; + else if (c === ')' && --depth === 0) return i; + } + return -1; +} + +/** + * A Kotlin call written on what an unqualified call returns — + * `repository().pending(1)`, `this.repository().pending(1)`, + * `nullable()?.pending(1)` — which the extractor records by the outer + * method's bare name at the inner call's column (a capitalized inner is a + * constructor, kept as `Foo().pending`). The inner function's name, or null + * for any other shape: anything but `!!` between the inner call and the + * method makes another one (`repository().other().pending()`). + */ +function kotlinCallResultCallee(ref: UnresolvedRef, context: ResolutionContext): string | null { + if (ref.language !== 'kotlin' || ref.referenceKind !== 'calls' || !/^[A-Za-z_]\w*$/.test(ref.referenceName)) return null; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); + const first = lines?.[ref.line - 1]; + if (!lines || first === undefined) return null; + const text = [first.slice(Math.max(0, ref.column)), ...lines.slice(ref.line, ref.line + KOTLIN_CALL_LINES)].join('\n'); + const head = KOTLIN_UNQUALIFIED_CALL.exec(text); + if (!head) return null; + const close = kotlinClosingParen(text, head[0].length - 1); + if (close < 0) return null; + const outer = new RegExp(`^\\s*(?:!!\\s*)?\\??\\.\\s*${escapeRegExp(ref.referenceName)}\\s*[({<]`); + return outer.test(text.slice(close + 1)) ? head[1]! : null; +} + +/** + * The type an unqualified Kotlin call to `name` returns, as the function it + * calls declares it (#2805): a method of the calling class, else of a type + * around the call or one those inherit (kotlinHierarchyAt, read from their + * declarations), else a top-level function the calling file can name. + * Undefined when no such function is found, or those found don't declare one + * type: `fun repository() = RealRepo()` declares none. + */ +function kotlinCallResultType(name: string, ref: UnresolvedRef, context: ResolutionContext): string | undefined { + const fns = context.getNodesByName(name).filter((n) => n.language === 'kotlin' && (n.kind === 'method' || n.kind === 'function')); + if (fns.length === 0) return undefined; + const ownerOf = (qualifiedName: string): string => qualifiedName.slice(0, Math.max(0, qualifiedName.lastIndexOf('::'))); + const caller = context.getNodeById?.(ref.fromNodeId); + // Inside an extension function on a type from outside the project (`fun + // URI.toHttpUrlOrNull() = toString()…`), that type's own members are the + // first a bare call reaches, and they aren't known here. + if (caller?.kind === 'method') { + const owner = ownerOf(caller.qualifiedName); + const member = context.getNodesByQualifiedName(owner).some((n) => MEMBER_CLASS_KINDS.has(n.kind) || n.kind === 'module'); + if (!member && !isKotlinProjectType(owner.split(/::|\./).pop() ?? '', context)) return undefined; + } + let pool = caller ? fns.filter((n) => n.kind === 'method' && ownerOf(n.qualifiedName) === ownerOf(caller.qualifiedName)) : []; + if (pool.length === 0) { + const hierarchy = kotlinHierarchyAt(ref, context); + pool = fns.filter((n) => { + if (n.kind !== 'method') return false; + const owner = ownerOf(n.qualifiedName).split(/::|\./); + const type = owner.pop(); + // A companion object's members are called bare inside its class. + return hierarchy.has(type === 'Companion' ? owner.pop() ?? '' : type ?? ''); + }); + } + if (pool.length === 0) { + pool = fns.filter((n) => { + if (n.kind !== 'function') return false; + const pkg = kotlinFileScope(n.filePath, context).pkg; + return n.qualifiedName === (pkg ? `${pkg}::${n.name}` : n.name) && isKotlinTopLevelVisible(n, ref, context); + }); + } + const types = new Set(pool.map((n) => kotlinScopedReturnType(n, context) ?? '')); + return pool.length > 0 && types.size === 1 && !types.has('') ? [...types][0] : undefined; +} + +/** + * The type a Kotlin function declares it returns (kotlinDeclaredReturnType), + * a name written alone taken as the nested type of the class it is declared + * in, or of a class around that, that has the name: `fun newBuilder(): + * Builder` in `Request` returns `Request.Builder`, not `Headers.Builder`. + */ +function kotlinScopedReturnType(fn: Node, context: ResolutionContext): string | undefined { + const written = kotlinDeclaredReturnType(fn); + if (!written || written.includes('.')) return written; + const isType = (qualifiedName: string): boolean => + context.getNodesByQualifiedName(qualifiedName).some((n) => MEMBER_CLASS_KINDS.has(n.kind) || n.kind === 'module'); + const scopes = fn.qualifiedName.split('::').slice(0, -1); + for (let i = scopes.length; i > 0; i--) { + const scope = scopes.slice(0, i).join('::'); + if (isType(scope) && isType(`${scope}::${written}`)) return `${scopes[i - 1]!}.${written}`; + } + return written; +} + +/** + * The type a Kotlin function's signature declares it returns, as its type + * names from the outermost type on, without nullability, type arguments or a + * package: `(source: Buffer): Hpack.Reader` → `Hpack.Reader`, `(): RealRepo?` + * → `RealRepo`, `(): List` → `List`. Undefined for none, or a function type. + */ +function kotlinDeclaredReturnType(fn: Node): string | undefined { + const sig = fn.signature?.trim(); + if (!sig?.startsWith('(')) return undefined; + const close = kotlinClosingParen(sig, 0); + const written = close < 0 ? undefined : /^\s*:\s*([\s\S]+)$/.exec(sig.slice(close + 1))?.[1]; + const path = written?.trim().replace(/\?+$/, '').replace(/<[\s\S]*>$/, '').trim().split('.').map((p) => p.trim()); + if (!path || !path.every((p) => /^[A-Za-z_]\w*$/.test(p))) return undefined; + const types = path.slice(path.findIndex((p) => /^[A-Z]/.test(p))); + return types.length > 0 && types.every((p) => /^[A-Z]/.test(p)) ? types.join('.') : undefined; +} + +/** Whether a Kotlin or Java type of the project — a class, interface, enum or object — has the name `name`. */ +function isKotlinProjectType(name: string, context: ResolutionContext): boolean { + return kotlinProjectTypes(name, context).length > 0; +} + +/** The project's Kotlin and Java types — classes, interfaces, enums, objects — named `name`. */ +function kotlinProjectTypes(name: string, context: ResolutionContext): Node[] { + return context.getNodesByName(name).filter((n) => + (n.language === 'kotlin' || n.language === 'java') && (MEMBER_CLASS_KINDS.has(n.kind) || n.kind === 'module')); +} + +/** + * A Kotlin call on what an unqualified call returns (kotlinCallResultCallee): + * the method of the type that function declares (#2805), as a + * `Foo.factory().bar()` chain takes it (matchDottedCallChain), or nothing — + * never a namesake found by the method's name alone. A nested type the + * function names through its outer one, `Request.Builder`, is that one, not + * another type's `Builder`. Undefined when the shape or the declared type + * can't be read, for the strategies after it. + */ +function matchKotlinCallResultCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + const callee = kotlinCallResultCallee(ref, context); + if (!callee) return undefined; + const written = kotlinCallResultType(callee, ref, context); + const path = written?.split('.') ?? []; + const type = path[path.length - 1]; + // A type from outside the project is no help: its methods, and which of the + // project's extensions its supertypes take, aren't known here. + if (!type || !isKotlinProjectType(type, context)) return undefined; + const name = ref.referenceName; + const found = resolveMethodOnType(type, name, ref, context, 0.85, 'instance-method', importedFqnOf(type, ref, context)); + // Whole names: `MockResponse::Builder::body` is no `Response::Builder::body`. + const member = `${path.join('::')}::${name}`; + const isPathMember = (n: Node): boolean => n.qualifiedName === member || n.qualifiedName.endsWith(`::${member}`); + if (found) { + const target = context.getNodeById?.(found.targetNodeId); + // An extension function, kept under its receiver type's bare name, is + // the one the calling file can name: Exposed declares `Expression.year()` + // in three modules, and a test of one calls its own. + if (target && target.kind === 'method' && !target.qualifiedName.slice(0, target.qualifiedName.lastIndexOf('::')).includes('::')) { + const visible = context.getNodesByName(name).filter((n) => + n.kind === 'method' && n.qualifiedName === target.qualifiedName && isKotlinTopLevelVisible(n, ref, context)); + if (visible.some((n) => n.id === target.id)) return found; + return visible.length === 1 ? { ...found, targetNodeId: visible[0]!.id } : undefined; + } + if (path.length === 1 || (target && isPathMember(target))) return found; + // Another type's `Builder`: the method of the one the path names, or the strategies after this one. + const own = context.getNodesByName(name).filter((n) => n.kind === 'method' && isPathMember(n)); + return own.length === 1 ? { original: ref, targetNodeId: own[0]!.id, confidence: 0.85, resolvedBy: 'instance-method' } : undefined; + } + // An extension the index doesn't keep under the type may be it: one on a + // nullable receiver (`fun MediaType?.charsetOrUtf8()`) is a top-level + // function, one on a nested type (`fun Dns.Call.toEventsQueue()`) is kept + // under its outer type's bare name (`Dns::toEventsQueue`), where a member's + // owner carries its package. + const extension = context.getNodesByName(name).some((n) => n.language === 'kotlin' && + (n.kind === 'function' || (n.kind === 'method' && !n.qualifiedName.slice(0, n.qualifiedName.lastIndexOf('::')).includes('::')))); + return extension ? undefined : null; +} + +/** + * The type a Kotlin local holds when its nearest declaration in the calling + * function sets it from an unqualified call alone — `val repo = + * repository()`, `var repo = this.repository()!!` — as that function declares + * (kotlinCallResultType, #2805). Undefined for any other declaration, or none. + */ +function kotlinLocalCallResultType(receiver: string, ref: UnresolvedRef, context: ResolutionContext): string | undefined { + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); + if (!lines) return undefined; + const r = escapeRegExp(receiver); + const declares = new RegExp(`\\b(?:val|var)\\s+${r}\\b`); + const initializer = new RegExp(`^(?:val|var)\\s+${r}\\s*=\\s*(?:this\\s*\\.\\s*)?([a-z_]\\w*)\\s*(?:<[^<>()]*>)?\\s*\\(`); + const start = Math.max(0, enclosingScopeStartLine(ref, context) - 1); + for (let i = Math.min(lines.length, ref.line) - 1; i >= start; i--) { + const line = lines[i]!; + const at = declares.exec(i === ref.line - 1 ? line.slice(0, Math.max(0, ref.column)) : line); + if (!at) continue; + const text = [line.slice(at.index), ...lines.slice(i + 1, i + 1 + KOTLIN_CALL_LINES)].join('\n'); + const call = initializer.exec(text); + if (!call) return undefined; + const close = kotlinClosingParen(text, call[0].length - 1); + if (close < 0 || !/^[ \t]*(?:!![ \t]*)?(?:;|\/\/|\r?\n|$)/.test(text.slice(close + 1))) return undefined; + // Only a type of the project's (see matchKotlinCallResultCall), and a + // nested one only when no other type shares its name: the type is looked + // up by its last name from here on. + const path = kotlinCallResultType(call[1]!, ref, context)?.split('.') ?? []; + const types = path.length > 0 ? kotlinProjectTypes(path[path.length - 1]!, context) : []; + return types.length === 1 || (types.length > 1 && path.length === 1) ? path[path.length - 1] : undefined; + } + return undefined; +} + /** Per-context memo: `\0` → tsImportedTypeFile's answer. */ const TS_IMPORTED_TYPE_FILES = new WeakMap>(); @@ -19517,6 +19730,12 @@ export function matchMethodCall( // A pytest test's parameter is what its fixture returns: flaskbb's // `cli_runner.invoke(…)` is click's `CliRunner`, not the project's one `invoke`. if (!inferredType && ref.language === 'python' && dotMatch) inferredType = pythonFixtureReturnType(objectOrClass!, ref, context); + // A Kotlin local set from an unqualified call holds what that function + // declares it returns: `val repo = repository()` (#2805), where the + // receiver's name led to the interface `Repo`. + if (!inferredType && ref.language === 'kotlin' && dotMatch && /^[a-z_]\w*$/.test(objectOrClass!)) { + inferredType = nmTimedT('mc-kotlincall', ref, () => kotlinLocalCallResultType(objectOrClass!, ref, context)) ?? null; + } // `subprocess.run(…)` after `import subprocess`: a function of a module // from outside the project, not a same-named project method (#2599). if (!inferredType && ref.language === 'python' && dotMatch && isPythonOutsideModuleReceiver(objectOrClass!, ref, context)) return null; @@ -26517,6 +26736,16 @@ function matchReferenceInner( if (lent !== undefined) return lent; } + // A Kotlin call on what an unqualified call returns, which the extractor + // records by its bare name — `repository().pending(1)` — is a call on the + // type that function declares, or on nothing (#2805): the strategies below + // took any `pending`, a test double's in another package. What the type + // inherits counts, as inheritance is resolved before any call. + if (ref.language === 'kotlin' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName)) { + const onResult = nmTimedT('kotlinCallResult', ref, () => matchKotlinCallResultCall(ref, context)); + if (onResult !== undefined) return onResult; + } + // Try strategies in order of confidence let result: ResolvedRef | null; From 23a80a847ae2a90bf195af2ee030c00cc094bb00 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 02:38:35 +0000 Subject: [PATCH 54/95] fix(python): a from-import of a module with no dot in its name links the module's file (#2823) In Python, `from loader import X` with loader.py at a package root left the module reference `loader` unresolved, so the importing file had no imports edge to loader.py. `import loader` and `from pkg.loader import X` both link their module's file. The from-import's names (X) linked, but `codegraph affected` and impact could miss the file's dependency on the module. v1.6.2 behaves the same. Root cause: resolvePythonAbsoluteModule resolves only dotted module refs. It leaves bare names to resolveModuleImportToFile, because a bare imported name like `authentication` in `from app.api.routes import authentication` is ambiguous on its own. But resolveModuleImportToFile matches an import by its local name, which for `from loader import X` is X, never `loader`. So the module of an undotted from-import was never looked up by its path. When the file also bound the module's name some other way, the generic import loop resolved it through that binding: next to `from django.core import mail`, the module of `from mail import ...` became django/core/mail/__init__.py. Django's tests/mail/__init__.py is the module meant, and the statement's own names already resolve there. Fix: resolvePythonAbsoluteModule also takes the module a from-import names (isPythonFromImportModule), dotted or not. It resolves it from the package roots, as `import loader` is resolved. Imported names keep their path, and a module from outside the project still links nothing. Validation: - New __tests__/python-from-import-module-file.test.ts, 7 tests: a root-level module, a top-level package, a src layout, a module named on a continuation line, a submodule imported by name, sync after the module file is added (matching a fresh index), and an outside module with a same-named nested file (links nothing). With main's import-resolver.ts the 6 positive tests fail and the guard passes. Seven assertions in python-module-resolution.test.ts listed a file's import edges exactly while the module edge was missing; they now include the module's file. What they test is unchanged: where the imported name binds, and no cycle. - A/B against main d5fc1fbb. Both arms were built from their own sources with the kernel. Rebased onto 628f69e2 (PR #2822, Kotlin only), the fix arm's graphs are byte-identical to before the rebase: - requests: +2 edges; - sqlalchemy: +3,189; - django's 2,938 Python files: +103 edges and 2 retargeted; - every change is a from-import's module linking its file, and nothing else changed; - express and gin (controls): byte-identical. Triage of all 3,296 added edges: - 3,284 point at the file, or the package folder, where the same statement's imported names already resolve; - the other 12 were read by hand: multi-line statements, re-exported constants the name resolver doesn't follow, and a deliberately broken name in a Django test fixture, each on the right module file; - the 2 django retargets are the mail case above. - Cost: index CPU medians of 3 interleaved runs, main vs fix: sqlalchemy 100.7 s vs 100.3 s, django 98.4 s vs 97.8 s. Found while verifying another Python issue in the sweep; no issue was filed for it. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../python-from-import-module-file.test.ts | 129 ++++++++++++++++++ __tests__/python-module-resolution.test.ts | 10 +- src/resolution/import-resolver.ts | 14 +- 4 files changed, 145 insertions(+), 9 deletions(-) create mode 100644 __tests__/python-from-import-module-file.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 07c21e8db6..dd8a4d1d43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In JavaScript and TypeScript, react-dom's `root.render()` no longer links to your own class component's `render()`, such as an error boundary's, when `root` is declared outside the function making the call, like a module's `let root: Root | null`, or holds what `createRoot(el)` returned. A call on a variable declared outside the calling function now goes by the type that declaration gives it, so one declared with your own class, like `const store = new Store()` at the top of a file, links to that class's method, and a class component's `render()` and lifecycle methods, which only React calls, are no longer a guess for an untyped value. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report. (#2589) - In Python, `import a.b as X` written inside a function now links its calls like `X.f()`, as the same import at module level already did. Thanks @Egrn for the report. (#2600) - In Python, `from app import email` no longer links the importing file to a project variable named `app`, such as a FastAPI or Flask `app = FastAPI()`. (#2584) +- In Python, a from-import of a module whose name has no dot, like `from loader import Registry` with `loader.py` at the repository root, or `from mypackage import helper` in a `src/` layout, now links the importing file to that module's file, as `import loader` and `from pkg.loader import Registry` already did. Before, the file's dependency on the module itself was left out, so `codegraph affected` and impact could miss the files that import it. When the file also imported something else under the module's name, like `from django.core import mail` beside `from mail import …`, the module was linked to that instead. Re-index Python projects after upgrading. - In Python, a call through a module-level instance, like `_svc.delete(obj)` after `_svc = SceneService()`, now links to the method its class has or inherits, even when other classes have a method of that name. (#2582) - In Python, a call of your own class's method named like a dictionary, list or file method, such as `get`, `update`, `items`, `pop` or `close`, now links to that method when the function says what the variable holds: `reg = SystemPackRegistry()` then `reg.get(…)`, `registry = load_registry()` with `def load_registry() -> SystemPackRegistry`, or a parameter written `reg: SystemPackRegistry`. Before, these calls were dropped as built-in methods unless the variable was named after the class. A variable the function also sets to something else, like `{}` or what `json.load` returns, a class imported from another package, like `queue.Queue`, and a union type still link nothing. Re-index Python projects after upgrading. Thanks @inth3shadows for the report. (#2583) - In Python, a call on a module imported from outside your project, like `subprocess.run(…)` after `import subprocess`, no longer links to a project method that happens to share its name. Thanks @Egrn for the report. (#2599) diff --git a/__tests__/python-from-import-module-file.test.ts b/__tests__/python-from-import-module-file.test.ts new file mode 100644 index 0000000000..a258f7a4cb --- /dev/null +++ b/__tests__/python-from-import-module-file.test.ts @@ -0,0 +1,129 @@ +/** + * The module a Python `from X import y` names is a file of the project when X + * is one, dotted or not: `from loader import X` with `loader.py` at a package + * root links the importing file to it, as `import loader` and `from + * pkg.loader import X` already did. Before, an undotted module of a + * from-import was left unresolved, so the file lost its dependency on the + * module and `codegraph affected` missed it. A module from outside the + * project still links nothing. + */ + +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +let tempDir: string; +let cg: CodeGraph | null = null; + +function write(files: Record): void { + for (const [rel, source] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), source); + } +} + +/** `kind:name@file` of every target of an `imports` edge from `fromFile`'s file node, sorted. */ +function importEdges(fromFile: string): string[] { + const file = cg!.getNodesByKind('file').find((n) => n.filePath === fromFile)!; + expect(file).toBeDefined(); + return cg! + .getOutgoingEdges(file.id) + .filter((e) => e.kind === 'imports') + .map((e) => cg!.getNode(e.target)) + .filter((n): n is NonNullable => !!n) + .map((n) => `${n.kind}:${n.name}@${n.filePath}`) + .sort(); +} + +async function indexed(files: Record): Promise { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-from-module-')); + write(files); + cg = await CodeGraph.init(tempDir, { index: true }); +} + +afterEach(() => { + cg?.close(); + cg = null; + fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +const LOADER = 'class Registry:\n def get(self, k):\n return k\n\n\ndef load_registry():\n return Registry()\n'; + +describe('the module of a Python from-import links its file', () => { + it('links a top-level module at the project root, as `import loader` does', async () => { + await indexed({ + 'loader.py': LOADER, + 'packs.py': 'from loader import Registry, load_registry\n\n\ndef use():\n return load_registry()\n', + 'plain.py': 'import loader\n\n\ndef use():\n return loader.load_registry()\n', + }); + expect(importEdges('packs.py')).toEqual([ + 'class:Registry@loader.py', + 'file:loader.py@loader.py', + 'function:load_registry@loader.py', + ]); + expect(importEdges('plain.py')).toEqual(['file:loader.py@loader.py']); + }); + + it('links a top-level package to its __init__.py, as a dotted from-import links its package', async () => { + await indexed({ + 'pkg/__init__.py': 'def helper():\n return 1\n', + 'pkg/sub/__init__.py': 'def deep():\n return 2\n', + 'app.py': 'from pkg import helper\nfrom pkg.sub import deep\n', + }); + expect(importEdges('app.py')).toEqual([ + 'file:__init__.py@pkg/__init__.py', + 'file:__init__.py@pkg/sub/__init__.py', + 'function:deep@pkg/sub/__init__.py', + 'function:helper@pkg/__init__.py', + ]); + }); + + it('finds the module from the package roots, as in a src layout', async () => { + await indexed({ + 'pyproject.toml': '[project]\nname = "app"\n', + 'src/app/__init__.py': 'def helper():\n return 1\n', + 'tests/test_app.py': 'from app import helper\n\n\ndef test_helper():\n assert helper() == 1\n', + }); + expect(importEdges('tests/test_app.py')).toEqual([ + 'file:__init__.py@src/app/__init__.py', + 'function:helper@src/app/__init__.py', + ]); + }); + + it('links a module named on a continuation line of the statement', async () => { + await indexed({ 'loader.py': LOADER, 'packs.py': 'from \\\n loader import Registry\n' }); + expect(importEdges('packs.py')).toEqual(['class:Registry@loader.py', 'file:loader.py@loader.py']); + }); + + it('links nothing for a module from outside the project, even one a nested file is named like', async () => { + await indexed({ + 'tests/fixtures/requests.py': 'class Session:\n pass\n', + 'client.py': 'from requests import Session\nimport requests\n', + }); + expect(importEdges('client.py')).toEqual([]); + }); + + it('keeps a submodule the statement imports by name', async () => { + await indexed({ + 'pkg/__init__.py': '', + 'pkg/loader.py': LOADER, + 'app.py': 'from pkg import loader\n', + }); + expect(importEdges('app.py')).toEqual(['file:__init__.py@pkg/__init__.py', 'file:loader.py@pkg/loader.py']); + }); + + it('links the module once it is added, on sync as on a fresh index', async () => { + await indexed({ 'packs.py': 'from loader import Registry\n' }); + expect(importEdges('packs.py')).toEqual([]); + write({ 'loader.py': LOADER }); + await cg!.sync(); + const synced = importEdges('packs.py'); + cg!.close(); + fs.rmSync(path.join(tempDir, '.codegraph'), { recursive: true, force: true }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(synced).toEqual(importEdges('packs.py')); + expect(synced).toEqual(['class:Registry@loader.py', 'file:loader.py@loader.py']); + }); +}); diff --git a/__tests__/python-module-resolution.test.ts b/__tests__/python-module-resolution.test.ts index 05350950b8..99a0a1ec8d 100644 --- a/__tests__/python-module-resolution.test.ts +++ b/__tests__/python-module-resolution.test.ts @@ -338,7 +338,8 @@ def fetch(): cg = await CodeGraph.init(tempDir, { index: true }); expect(targets('analysis.py', 'analysis.py', 'imports')).toEqual([]); - expect(targets('report.py', 'report.py', 'imports')).toEqual(['summary@analysis.py']); + // The from-import's module is analysis.py itself; the name is its `summary`. + expect(targets('report.py', 'report.py', 'imports').sort()).toEqual(['analysis.py@analysis.py', 'summary@analysis.py']); expect(cg.findCircularDependencies()).toEqual([]); }); @@ -651,7 +652,8 @@ def main(): 'main.py': 'from pkg import f\n\n\ndef main():\n f()\n', }); cg = await CodeGraph.init(tempDir, { index: true }); - expect(targets('main.py', 'main.py', 'imports')).toEqual(['f@pkg/__init__.py']); + // The package itself is the from-import's module; `f` is its attribute. + expect(targets('main.py', 'main.py', 'imports').sort()).toEqual(['__init__.py@pkg/__init__.py', 'f@pkg/__init__.py']); }); it('binds it through a relative import too', async () => { @@ -675,7 +677,7 @@ def main(): 'main.py': 'from pkg import thing\n\n\ndef main():\n thing()\n', }); cg = await CodeGraph.init(tempDir, { index: true }); - expect(targets('main.py', 'main.py', 'imports')).toEqual(['thing@pkg/impl.py']); + expect(targets('main.py', 'main.py', 'imports').sort()).toEqual(['__init__.py@pkg/__init__.py', 'thing@pkg/impl.py']); }); it('still binds the submodule when the package does not bind the name', async () => { @@ -685,7 +687,7 @@ def main(): 'main.py': 'from pkg import f\n\n\ndef main():\n f.g()\n', }); cg = await CodeGraph.init(tempDir, { index: true }); - expect(targets('main.py', 'main.py', 'imports')).toEqual(['f.py@pkg/f.py']); + expect(targets('main.py', 'main.py', 'imports').sort()).toEqual(['__init__.py@pkg/__init__.py', 'f.py@pkg/f.py']); expect(targets('main', 'main.py', 'calls')).toEqual(['g@pkg/f.py']); }); }); diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index 01712149b3..c05f268215 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -30,6 +30,7 @@ import { tsImportedTypeFile, reExportSource, pythonFunctionModuleAlias, + isPythonFromImportModule, isRustAssociatedItem, tsGlobalScope, } from './name-matcher'; @@ -3661,11 +3662,14 @@ function resolvePythonAbsoluteModule( context: ResolutionContext ): ResolvedRef | null { if (ref.referenceKind !== 'imports') return null; - // Only a DOTTED `import a.b.c` ref carries its full module path. A bare leaf - // (`from app.api.routes import authentication`) is ambiguous on its own — three - // `authentication.py` files may exist — so leave it to resolveModuleImportToFile, - // which uses the import's source (`app.api.routes`) to build the full path. - if (!ref.referenceName.includes('.')) return null; + // Only a DOTTED `import a.b.c` ref carries its full module path, and so does + // the module a from-import names, dotted or not: `from loader import X` is + // the top-level `loader`, found from the package roots as `import loader` + // finds it. A bare leaf it imports (`from app.api.routes import + // authentication`) is ambiguous on its own — three `authentication.py` files + // may exist — so leave it to resolveModuleImportToFile, which uses the + // import's source (`app.api.routes`) to build the full path. + if (!ref.referenceName.includes('.') && !isPythonFromImportModule(ref, context)) return null; const hit = pythonModuleFileNode(ref.referenceName, ref.filePath, context); return hit ? { original: ref, targetNodeId: hit.id, confidence: 0.9, resolvedBy: 'import' } : null; } From 013b91c1be37ca6a6ee2c19bb89be56dd0faa37a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 03:16:07 +0000 Subject: [PATCH 55/95] fix(js,ts): a named class expression is a class with its own members (#2825) A named class expression, like register(class Widget extends Base {}) or module.exports = class Router {}, was not indexed as a class, as issue #2499 found. Its methods became free functions, its fields and supertypes were dropped, and calls in it had no class to start from, so its this-calls, super calls and own static calls went unresolved or to a by-name guess, and calls elsewhere guessed into its methods. Both engines now index a named class expression as a class, with its methods, properties, extends and implements. An anonymous one is still passed over. Its head (class Widget extends Base) is its signature, which marks it as an expression, and it is never exported by itself. JavaScript binds the name only inside the expression, and the resolver now scopes it the same way. A reference written outside every class expression resolves in a view of the graph that hides class expressions and their members, so new Widget() elsewhere links nothing, and a same-named class, function or binding keeps its links. A name only class expressions carry is unknown there. A reference inside one sees the whole graph, unless it names another class expression. A class a CommonJS module exports, module.exports = class Router {} or exports.Store = class Store {}, is the module's value, which require hands to every importer, so it is not scoped: it reads as a declaration does, and a require of it reaches its static methods and its instances' methods. The reader of CommonJS named exports takes exports.X = class Y {} too. Super calls in a class expression now resolve through the super-call matcher. Validation: the new test fails 11 of 13 on main d5fc1fbb and passes 13 of 13 on the fix, and a mutation of each of 7 parts (the view, the inside check, the name rule, the expression marker, the class node, the CommonJS export rule, the named-export reader) fails a test. A/B against main d5fc1fbb on 22 repos (20 TS/JS, flask and gin): 193 classes are added in 10 repos with their members, and 801 edge sites change in 7 repos, each bucketed and sampled by hand. 563 are jest's calls to its global test(), 0.4 guesses that move from one wrong namesake to another: the old one, a class expression's test() in an e2e fixture, is now a method. Of the rest, 85 are new extends, implements and type references of class expressions; 74 calls, news and references move to the member they are written in; 65 calls and news gain or change a target, inside class expressions (this-calls, own static calls) and through require (static calls, typed instance calls, new); and 14 guesses are removed, among them 3 super calls whose parent the index can't name. 12 repos' dumps, the controls among them, are byte-identical. Kernel parity holds on 22 repos. Three sync scenarios match a fresh index. Index time, median of 5 interleaved runs: lit +0.8%, webpack +2.7%, jest +0.6%; insomnia +12.8%, though it indexes byte-identically, holds no class expression, and its resolver time is the same on both arms, so that gap is other jobs' load. Fixes #2499 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/js-class-expressions.test.ts | 287 ++++++++++++++++++++++++ codegraph-kernel/src/tsjs/extractors.rs | 16 +- codegraph-kernel/src/tsjs/mod.rs | 15 +- src/extraction/languages/javascript.ts | 3 +- src/extraction/languages/typescript.ts | 5 +- src/extraction/tree-sitter.ts | 30 ++- src/resolution/alias-binding.ts | 7 +- src/resolution/index.ts | 187 +++++++++++++-- src/resolution/js-class-expressions.ts | 66 ++++++ 10 files changed, 592 insertions(+), 25 deletions(-) create mode 100644 __tests__/js-class-expressions.test.ts create mode 100644 src/resolution/js-class-expressions.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index dd8a4d1d43..204eb7343f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -204,6 +204,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Java, Kotlin, C# and Scala, each overload of an interface or base-class method now links only to the overrides with the same parameters. Before, every overload linked to every method of that name in the implementing class, so `ExampleService.putTairValue(String)` also led to `putTairValue(List, double)`, and `codegraph_explore`'s call path, callers and impact went into overloads that call never runs. Where a parameter is a type parameter, like `save(T entity)` implemented as `save(String entity)`, or the class is written in another language than its interface, the number of parameters decides; a method declared once on each side links as before. Re-index Java, Kotlin, C# and Scala projects after upgrading. Thanks @hxj1225 for the report and @danusha2345 for the pull request. (#2663) - In Kotlin, a call on what one of your functions returns, like `repository().pending(1)` or `val repo = repository(); repo.pending(1)` with a `private fun repository(): RealRepo` in a test class, now links to the method of the type that function declares, including one the type inherits or an extension on it. Before, the chained call went to whichever `pending` the name matched first, like a test double's in another package, and the local was guessed from its name, which reached the interface. The function is found in the calling class, a class it inherits from or one around it, or at the top level, and a nested type like `Request.Builder` is that type, not another class's `Builder`. When the declared type has no such method, the call now links to nothing rather than a method of the same name elsewhere; a type from outside your project is matched by name as before. Re-index Kotlin projects after upgrading. Thanks @mindcipherrr for the report. (#2805) - In JavaScript and TypeScript, a `super.run()` call now links to the method of the class's parent, or of the nearest ancestor that has it, like three.js's `super.copy(source)` reaching `Object3D.copy`, so `codegraph_explore`'s call path and impact follow an override into the method it builds on. Before, such a call inside a method of the same name was dropped, and elsewhere it went to whatever method had the name, often the class's own or a sibling class's in the same file. A call from a static method reaches the parent's static method, and a call to a parent from outside your project, like `HTMLElement`, `Set` or `Promise`, links nothing. Re-index JavaScript and TypeScript projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2497) +- In JavaScript and TypeScript, a named class expression, like `register(class Widget extends Base {…})`, `customElements.define('x-card', class XCard extends HTMLElement {…})` or `const Panel = class PanelImpl extends Base {…}`, is now indexed as a class with its own methods and the class it extends, so calls between its methods and its `super` calls link as a class declaration's do. Before, its methods were free functions or weren't indexed at all, and it extended nothing. Its name stays bound inside the expression alone, as JavaScript has it: `new Widget()` elsewhere links nothing, a constant, class or interface of the same name around it keeps its links, and code outside never reaches its methods by a guess. A class a CommonJS module exports, like webpack's `module.exports = class MultiCompiler {…}` or `exports.Store = class Store {…}`, is reached through `require` as a class declaration is, along with its static methods and the methods called on its instances. Re-index JavaScript and TypeScript projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2499) - In Go, a call through a struct field that holds a function, like prometheus's `d.refreshf(ctx)`, now reaches the functions put in that field, so `codegraph_explore` follows service discovery from `Discovery.Run` into each provider's `refresh` and names the line where each provider wires its method in, like `RefreshF: d.refresh`. The field can be set in a struct literal or by an assignment, copied from another struct's field (`refreshf: opts.RefreshF`), or given a method of an interface value, and two unrelated structs with a field of the same name, like `fn` or `handler`, stay apart. Before, such a flow stopped at the field. Re-index Go projects after upgrading. - In Go, a type defined from other types, like etcd's `type WatchChan <-chan WatchResponse` or gin's `type HandlerFunc func(*Context)` and `type HandlersChain []HandlerFunc`, now links to the types it is built from, so impact on `WatchResponse` or gin's `Context` reaches these types and the code that uses them. Before, such a type linked to nothing. One written through another package, like etcd's `type PutResponse pb.PutResponse`, links to that package's type, and a recursive one like `type stateFn func(*Lexer) stateFn` doesn't link to itself. Re-index Go projects after upgrading. - In Go, a struct now links to the types its fields are declared with, like etcd's `PutResponse` holding a `*ResponseHeader` or gin's `Context` holding a `HandlersChain`, so impact on `ResponseHeader` or `HandlersChain` reaches every struct that holds one and the code that uses those structs, and `codegraph_node` lists a struct's field types. Before, a struct linked only to the types it embeds. A field written through another package, like `*pb.ResponseHeader`, links to that package's type, one from outside your project, like `context.Context`, links to nothing, the types an embedded generic is filled in with, like the `Pod` in client-go's `listers.ResourceIndexer[*v1.Pod]`, link too, and a struct that holds itself, like a list node's `next *node`, doesn't link to itself. `codegraph_explore` still answers a question about a flow with the code that runs it, not with the structs that merely hold one of its types. Re-index Go projects after upgrading. diff --git a/__tests__/js-class-expressions.test.ts b/__tests__/js-class-expressions.test.ts new file mode 100644 index 0000000000..ec4761d3bb --- /dev/null +++ b/__tests__/js-class-expressions.test.ts @@ -0,0 +1,287 @@ +/** + * A named class expression, like `register(class Widget extends Base {…})` or + * `const Panel = class PanelImpl extends Base {…}`, is a class: it has a node, + * members of its own and the supertypes it names (#2499). Before, its methods + * were free functions or not indexed at all, and it extended nothing. Its + * name is bound inside the expression alone, so outside it the name reaches + * neither the class nor its members, and a binding of the same name around + * it keeps meaning what it meant. A class a CommonJS module exports is the + * module's value, which `require` hands over as it does a declaration. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import type { ExtractionResult, Language } from '../src/types'; + +const KERNEL_PATH = path.join(__dirname, '..', 'codegraph-kernel', 'prebuilds', `${process.platform}-${process.arch}`, 'codegraph-kernel.node'); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +const EXPRESSIONS = `class Base { run() { return 1; } } +register(class Widget extends Base { + run() { return super.run(); } + extra() { return this.run(); } +}); +export const Panel = class PanelImpl extends Base { + go() { return 3; } +}; +register(class extends Base { anonymous() { return 4; } }); +`; + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +describe('a named class expression is a class (#2499)', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['typescript', 'tsx', 'javascript', 'jsx']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', file: string, source: string, language: Language): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource(file, source, language); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + const result = tryKernelExtract(file, source, language); + expect(result, `kernel extraction of ${file}`).not.toBeNull(); + return result!; + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + const variants = [['ts', 'typescript'], ['tsx', 'tsx'], ['js', 'javascript'], ['jsx', 'jsx']] as const; + + for (const backend of backends) { + it.each(variants)(`in %s, ${backend}`, (ext, language) => { + const result = extract(backend, `src/widgets.${ext}`, EXPRESSIONS, language); + const symbols = result.nodes + .filter((n) => n.kind === 'class' || n.kind === 'method' || n.kind === 'function') + .map((n) => `${n.kind} ${n.qualifiedName}${n.signature && n.kind === 'class' ? ` (${n.signature})` : ''}${n.isExported ? ' exported' : ''}`); + expect(symbols).toEqual([ + 'class Base', + 'method Base::run', + 'class Widget (class Widget extends Base)', + 'method Widget::run', + 'method Widget::extra', + 'class Panel::PanelImpl (class PanelImpl extends Base)', + 'method Panel::PanelImpl::go', + // An anonymous class expression is passed over, as before. + 'function anonymous', + ]); + const supertypes = result.unresolvedReferences + .filter((r) => r.referenceKind === 'extends') + .map((r) => `${result.nodes.find((n) => n.id === r.fromNodeId)?.qualifiedName} extends ${r.referenceName}`); + expect(supertypes).toEqual(['Widget extends Base', 'Panel::PanelImpl extends Base']); + }); + } + + it.runIf(kernelAvailable)('the kernel and the wasm extractor agree on every node and reference', () => { + for (const [ext, language] of variants) { + const canon = (r: ExtractionResult) => ({ + nodes: r.nodes.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort())).sort(), + refs: r.unresolvedReferences.map((x) => JSON.stringify(x, Object.keys(x).sort())).sort(), + }); + expect(canon(extract('kernel', `src/widgets.${ext}`, EXPRESSIONS, language)), ext) + .toEqual(canon(extract('wasm', `src/widgets.${ext}`, EXPRESSIONS, language))); + } + }); +}); + +describe.each(['js', 'ts'])('a class expression\'s name reaches it from inside alone (#2499): %s', (ext) => { + let root = ''; + let graph: CodeGraph | null = null; + + afterEach(() => { + graph?.close(); + graph = null; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = ''; + }); + + /** `kind source -> target [resolvedBy confidence]` for the calls, supertypes and instantiations out of a file's nodes, synthesized ones left out. */ + function edgesFrom(g: CodeGraph, file: string): string[] { + const out: string[] = []; + for (const node of g.getNodesInFile(file)) { + for (const e of g.getOutgoingEdges(node.id)) { + if (!['calls', 'extends', 'implements', 'instantiates'].includes(e.kind) || e.metadata?.synthesizedBy) continue; + out.push(`${e.kind} ${node.qualifiedName} -> ${g.getNode(e.target)!.qualifiedName} [${e.metadata?.resolvedBy} ${e.metadata?.confidence}]`); + } + } + return out.sort(); + } + + it('links the class\'s supertypes, super and this calls, and nothing outside by its name', async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-class-expression-')); + const write = (rel: string, lines: string[]) => { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), lines.join('\n') + '\n'); + }; + write('package.json', [JSON.stringify({ name: 'app' })]); + write(`src/base.${ext}`, ['export class Base { run() { return 1; } }', 'export function register(T) { return T; }']); + write(`src/widgets.${ext}`, [ + "import { Base, register } from './base';", + 'register(class Widget extends Base {', + ' run() { return super.run(); }', + ' extra() { return this.run(); }', + // Another class expression's name is no binding in here either. + ' other() { return new PanelImpl(); }', + '});', + 'export const Panel = class PanelImpl extends Base {', + ' go() { return super.run(); }', + '};', + // The inner name is no binding out here: this throws at runtime. + 'export function make() { return new Widget(); }', + 'export function panel() { return new Panel(); }', + ]); + write(`src/shadow.${ext}`, [ + "import { Base, register } from './base';", + // A declaration and a constant share names with class expressions in their file. + 'export class Gadget { run() { return 0; } }', + 'export const list = [class Gadget extends Base { run() { return super.run(); } }];', + 'export const Tool = class Tool extends Base { static make() { return new Tool(); } };', + 'export function build() { return new Gadget(); }', + 'export function tool() { return new Tool(); }', + 'export function helper() { return Tool.make(); }', + ]); + // nitro's shape: an interface, and a class expression of its name that a + // factory returns. A value typed by the name is the interface's. + if (ext === 'ts') { + write('src/env.ts', [ + 'export interface Env { fetch(url: string): string }', + 'export function define() { return class Env { fetch(url: string) { return url; } }; }', + 'export function use(env: Env) { return env.fetch("/"); }', + ]); + } + graph = await CodeGraph.init(root, { index: true }); + + expect(edgesFrom(graph, `src/widgets.${ext}`)).toEqual([ + 'calls Panel::PanelImpl::go -> Base::run [instance-method 0.9]', + 'calls Widget::extra -> Widget::run [exact-match 0.4]', + 'calls Widget::run -> Base::run [instance-method 0.9]', + `calls src/widgets.${ext} -> register [import 0.9]`, + 'extends Panel::PanelImpl -> Base [import 0.9]', + 'extends Widget -> Base [import 0.9]', + // `new Panel()` is the constant's, as before; `new Widget()` outside + // the expression links nothing. + 'instantiates panel -> Panel [exact-match 0.9]', + ]); + expect(edgesFrom(graph, `src/shadow.${ext}`)).toEqual([ + 'calls list::Gadget::run -> Base::run [instance-method 0.9]', + 'extends Tool::Tool -> Base [import 0.9]', + 'extends list::Gadget -> Base [import 0.9]', + // Outside the expressions, the declaration and the constant keep the + // names, at the confidence they had; inside its own body, `Tool` is + // the class. + 'instantiates Tool::Tool::make -> Tool::Tool [exact-match 0.4]', + 'instantiates build -> Gadget [exact-match 0.9]', + 'instantiates tool -> Tool [exact-match 0.9]', + ]); + if (ext === 'ts') { + expect(edgesFrom(graph, 'src/env.ts').filter((e) => e.startsWith('calls use'))).toEqual(['calls use -> Env::fetch [instance-method 0.9]']); + } + }, 120_000); +}); + +it('a guess by a receiver\'s name never reaches a class expression\'s member from code its value can\'t reach (#2499)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-class-expression-guess-')); + let graph: CodeGraph | null = null; + try { + fs.writeFileSync(path.join(root, 'package.json'), JSON.stringify({ name: 'app' })); + fs.mkdirSync(path.join(root, 'src')); + // lit's SSR DOM shim declares its document and elements this way. + fs.writeFileSync(path.join(root, 'src/dom-shim.ts'), [ + 'export const DocumentShim = class Document {', + ' createElement(tag: string) { return tag; }', + '};', + 'export const HTMLElementShim = class HTMLElement {', + ' appendChild(child: unknown) { return child; }', + '};', + '', + ].join('\n')); + fs.writeFileSync(path.join(root, 'src/format.ts'), 'export class Formatter { appendChild(node: unknown) { return node; } }\n'); + fs.writeFileSync(path.join(root, 'src/view.ts'), [ + 'let container: HTMLElement;', + "export function view() { container = document.createElement('div'); return container; }", + // A class expression of its own, which it reaches. + 'export const Local = class LocalView { show() { return 1; } };', + 'export function mount(el: unknown) { container.appendChild(el); }', + '', + ].join('\n')); + graph = await CodeGraph.init(root, { index: true }); + for (const name of ['view', 'mount']) { + const fn = graph.getNodesByQualifiedName(name)[0]!; + expect(graph.getOutgoingEdges(fn.id).filter((e) => e.kind === 'calls').map((e) => graph!.getNode(e.target)!.qualifiedName), name).toEqual([]); + } + } finally { + graph?.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}, 120_000); + +it('a class a CommonJS module exports is reached through `require` as a declaration is (#2499)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-class-expression-cjs-')); + let graph: CodeGraph | null = null; + try { + fs.writeFileSync(path.join(root, 'package.json'), JSON.stringify({ name: 'app' })); + fs.mkdirSync(path.join(root, 'src')); + fs.writeFileSync(path.join(root, 'src/router.js'), [ + 'module.exports = class Router {', + ' static make() { return 1; }', + ' route() { return 2; }', + '};', + // A class expression the module doesn't export keeps its scope. + 'register(class Hidden { go() { return 3; } });', + '', + ].join('\n')); + fs.writeFileSync(path.join(root, 'src/store.js'), 'exports.Store = class StoreImpl {\n static open() { return 4; }\n};\n'); + fs.writeFileSync(path.join(root, 'src/app.js'), [ + "const Router = require('./router.js');", + "const { Store } = require('./store.js');", + 'exports.start = function start() {', + ' const r = new Router();', + ' Router.make();', + ' Store.open();', + ' new Hidden().go();', + ' return r.route();', + '};', + '', + ].join('\n')); + graph = await CodeGraph.init(root, { index: true }); + const start = graph.getNodesByQualifiedName('start')[0]!; + const targets = graph.getOutgoingEdges(start.id) + .filter((e) => e.kind === 'calls' || e.kind === 'instantiates') + .map((e) => `${e.kind} ${graph!.getNode(e.target)!.kind} ${graph!.getNode(e.target)!.qualifiedName} [${e.metadata?.resolvedBy}]`) + .sort(); + // The class, its static method and its instance's method, by what each + // module exports, `module.exports` or `exports.Store`; the class + // expression neither exports stays out of reach. + expect(targets).toEqual([ + 'calls method Router::make [import]', + 'calls method Router::route [instance-method]', + 'calls method StoreImpl::open [import]', + 'instantiates class Router [import]', + ]); + } finally { + graph?.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}, 120_000); diff --git a/codegraph-kernel/src/tsjs/extractors.rs b/codegraph-kernel/src/tsjs/extractors.rs index 61b343825a..e717f5a8de 100644 --- a/codegraph-kernel/src/tsjs/extractors.rs +++ b/codegraph-kernel/src/tsjs/extractors.rs @@ -368,10 +368,15 @@ impl<'t> Walker<'t> { pub(super) fn extract_class(&mut self, node: Node<'t>) { let resolved_body = body_of(node); // skipBodilessClass unset for TS/JS let name = self.extract_name(node); + // A class expression's name is bound inside the expression alone: + // never exported by it, and its head is its signature, which a + // declaration lacks (#2499). Mirrors TreeSitterExtractor.extractClass. + let expression = node.kind() == "class"; let extra = Extra { docstring: crate::docstring::preceding_docstring(node, self.src), visibility: self.visibility_of(node), - is_exported: Some(self.is_exported(node)), + is_exported: Some(!expression && self.is_exported(node)), + signature: if expression { resolved_body.map(|body| class_expression_head(self.src, node, body)) } else { None }, ..Extra::default() }; let Some(row) = self.create_node("class", &name, node, extra) else { @@ -2014,6 +2019,15 @@ impl<'t> Walker<'t> { } } +/// A class expression's head, `class Widget extends Base`: its text up to the +/// body, trailing spaces and line breaks dropped. Mirrors classExpressionHead +/// (tree-sitter.ts). +fn class_expression_head(src: &str, node: Node<'_>, body: Node<'_>) -> String { + src[node.start_byte()..body.start_byte()] + .trim_end_matches(|c: char| matches!(c, ' ' | '\t' | '\r' | '\n')) + .to_string() +} + /// `.replace(/\s+/g, ' ')` for the tuple-contract signature. fn collapse_ws(s: &str) -> String { let mut out = String::with_capacity(s.len()); diff --git a/codegraph-kernel/src/tsjs/mod.rs b/codegraph-kernel/src/tsjs/mod.rs index 12eea4be1a..cd53f3255b 100644 --- a/codegraph-kernel/src/tsjs/mod.rs +++ b/codegraph-kernel/src/tsjs/mod.rs @@ -82,7 +82,16 @@ fn is_function_type(kind: &str) -> bool { } fn is_class_type(v: Variant, kind: &str) -> bool { - kind == "class_declaration" || (v.is_ts() && kind == "abstract_class_declaration") + kind == "class_declaration" || kind == "class" || (v.is_ts() && kind == "abstract_class_declaration") +} + +/// Whether `node` is a class to extract: a class declaration, or a class +/// expression (`class`) with a name, `register(class Widget extends Base +/// {…})` (#2499); an anonymous one is passed over. Mirrors +/// TreeSitterExtractor.isClassTypeNode. +fn is_class_node(v: Variant, node: Node) -> bool { + let kind = node.kind(); + is_class_type(v, kind) && (kind != "class" || node.child_by_field_name("name").is_some()) } fn is_variable_type(kind: &str) -> bool { @@ -692,7 +701,7 @@ impl<'t> Walker<'t> { // `declare function`, a function in a `declare module` body or a `.d.ts`. self.extract_function(node, None); skip_children = true; - } else if is_class_type(self.variant, kind) { + } else if is_class_node(self.variant, node) { self.extract_class(node); skip_children = true; } else if is_method_type(self.variant, kind) @@ -839,7 +848,7 @@ impl<'t> Walker<'t> { } } - if is_class_type(self.variant, kind) { + if is_class_node(self.variant, node) { self.extract_class(node); return; } diff --git a/src/extraction/languages/javascript.ts b/src/extraction/languages/javascript.ts index b96956e466..17920e7fde 100644 --- a/src/extraction/languages/javascript.ts +++ b/src/extraction/languages/javascript.ts @@ -4,7 +4,8 @@ import { classifyTsClassMember, hasStaticModifier } from './typescript'; export const javascriptExtractor: LanguageExtractor = { functionTypes: ['function_declaration', 'generator_function_declaration', 'arrow_function', 'function_expression', 'generator_function'], - classTypes: ['class_declaration'], + // A named class expression too, as in TypeScript (#2499). + classTypes: ['class_declaration', 'class'], methodTypes: ['method_definition', 'field_definition'], // JS `field_definition` ≙ TS `public_field_definition`: plain fields are // properties, function-valued fields are methods (#808). diff --git a/src/extraction/languages/typescript.ts b/src/extraction/languages/typescript.ts index 40f5a64655..ec28e1e829 100644 --- a/src/extraction/languages/typescript.ts +++ b/src/extraction/languages/typescript.ts @@ -150,7 +150,10 @@ export const typescriptExtractor: LanguageExtractor = { preParse: blankFlowSyntax, functionTypes: ['function_declaration', 'generator_function_declaration', 'arrow_function', 'function_expression', 'generator_function'], isAmbientFunction: isAmbientFunctionSignature, - classTypes: ['class_declaration', 'abstract_class_declaration'], + // `class` is a class expression, `register(class Widget extends Base {…})`: + // a named one is a class with its members and supertypes (#2499), and an + // anonymous one is passed over (TreeSitterExtractor.isClassTypeNode). + classTypes: ['class_declaration', 'abstract_class_declaration', 'class'], // `method_signature` is the interface/type-literal form of a method; without it // an interface's members never enter the graph, so a `.d.ts` platform API has // no declaration node for call sites to attach to (#1638). Java/C# don't need diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 7310dc15b9..162714c979 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -430,6 +430,14 @@ const LITERAL_RECEIVER_TYPES = new Set([ */ const TS_JS_CHAIN_LANGUAGES = new Set(['typescript', 'tsx', 'javascript', 'jsx']); +/** Languages whose `class` node is a class expression (#2499). */ +const CLASS_EXPRESSION_LANGUAGES = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'arkts']); + +/** A class expression's head, `class Widget extends Base`: its text up to the body. Mirrored in the kernel (tsjs extract_class). */ +function classExpressionHead(source: string, node: SyntaxNode, body: SyntaxNode): string { + return source.substring(node.startIndex, body.startIndex).replace(/[ \t\r\n]+$/, ''); +} + /** Receiver node types (TS/JS grammars) that continue a member chain downward. */ const TS_JS_CHAIN_RECEIVER_TYPES = new Set(['member_expression', 'subscript_expression']); @@ -1543,7 +1551,7 @@ export class TreeSitterExtractor { skipChildren = true; } // Check for class declarations - else if (this.extractor.classTypes.includes(nodeType)) { + else if (this.isClassTypeNode(node, nodeType)) { // Some languages reuse class_declaration for structs/enums (e.g. Swift) const classification = this.extractor.classifyClassNode?.(node) ?? 'class'; if (classification === 'struct') { @@ -2540,6 +2548,16 @@ export class TreeSitterExtractor { /** * Extract a class */ + /** + * Whether `node` is a class to extract: one of the language's class types, + * except a TS/JS class expression without a name (`register(class extends + * Base {…})`), which is passed over as before (#2499). + */ + private isClassTypeNode(node: SyntaxNode, nodeType: string): boolean { + if (!this.extractor!.classTypes.includes(nodeType)) return false; + return nodeType !== 'class' || !CLASS_EXPRESSION_LANGUAGES.has(this.language) || !!getChildByField(node, 'name'); + } + private extractClass(node: SyntaxNode, kind: NodeKind = 'class'): void { if (!this.extractor) return; @@ -2555,13 +2573,19 @@ export class TreeSitterExtractor { const name = extractName(node, this.source, this.extractor); const docstring = this.docstringFor(node); const visibility = this.extractor.getVisibility?.(node); - const isExported = this.extractor.isExported?.(node, this.source); + // A class expression's name is bound inside the expression alone, so the + // class is never what a module exports by it, and it carries its head + // (`class Widget extends Base`) as its signature: a class declaration + // has none, which is how resolution tells the two apart (#2499). + const expression = node.type === 'class' && CLASS_EXPRESSION_LANGUAGES.has(this.language); + const isExported = expression ? false : this.extractor.isExported?.(node, this.source); const scoped = this.writtenScopeQualifiedName(node, name); const classNode = this.createNode(kind, name, node, { docstring, visibility, isExported, + ...(expression && resolvedBody ? { signature: classExpressionHead(this.source, node, resolvedBody) } : {}), ...this.cppBodyEnd(resolvedBody), ...(scoped !== undefined ? { qualifiedName: scoped } : {}), }); @@ -7420,7 +7444,7 @@ export class TreeSitterExtractor { // Extract structural nodes found inside function bodies. // Each extract method visits its own children, so we return after extracting. - if (this.extractor!.classTypes.includes(nodeType)) { + if (this.isClassTypeNode(node, nodeType)) { const classification = this.extractor!.classifyClassNode?.(node) ?? 'class'; if (classification === 'struct') this.extractStruct(node); else if (classification === 'enum') this.extractEnum(node); diff --git a/src/resolution/alias-binding.ts b/src/resolution/alias-binding.ts index 1af7958e8e..0f3ff6c601 100644 --- a/src/resolution/alias-binding.ts +++ b/src/resolution/alias-binding.ts @@ -131,11 +131,12 @@ export function extractLocalExportAliases(content: string): Array<{ exportedName } } } - // CommonJS: `exports.setCharset = function setCharset(…)`, `module.exports.x = impl`. - const memberRe = /^[ \t]*(?:module\.)?exports\.([A-Za-z_$][\w$]*)\s*=\s*(?:(?:async\s+)?function\s*\*?\s*([A-Za-z_$][\w$]*)\s*\(|([A-Za-z_$][\w$]*)\s*;?[ \t]*$)/gm; + // CommonJS: `exports.setCharset = function setCharset(…)`, `module.exports.x = impl`, + // and a named class expression, `exports.Router = class Router {…}` (#2499). + const memberRe = /^[ \t]*(?:module\.)?exports\.([A-Za-z_$][\w$]*)\s*=\s*(?:(?:async\s+)?function\s*\*?\s*([A-Za-z_$][\w$]*)\s*\(|class\s+(?!extends\b)([A-Za-z_$][\w$]*)|([A-Za-z_$][\w$]*)\s*;?[ \t]*$)/gm; let member: RegExpExecArray | null; while ((member = memberRe.exec(content)) !== null) { - out.push({ exportedName: member[1]!, localName: member[2] ?? member[3]! }); + out.push({ exportedName: member[1]!, localName: member[2] ?? member[3] ?? member[4]! }); } // `module.exports = { parse, format: formatDate }`. const objectRe = /^[ \t]*module\.exports\s*=\s*\{([^{}]*)\}/gm; diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 5889a4fa98..e2e717afc1 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -40,6 +40,7 @@ import { clearCFeatureVariantMemos } from './c-feature-variants'; import { clearAmbientModuleMemos, ambientModuleDeclarations } from './ambient-modules'; import { matchShopifyThemeFile } from './shopify-themes'; import { isJsSuperCallLanguage, isJsSuperCallSite, matchJsSuperCall } from './js-super-calls'; +import { CLASS_EXPRESSION_REF_LANGUAGES, isCommonJsExportedClass, isJsClassExpression, writtenIn, type ClassExpressionIndex } from './js-class-expressions'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveTsQualifiedTypeRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isDartImportRef, isLuaRequireRef, isJsPathImportRef, isBoundToOutOfRepoImport, reachesAmbientModule, importBindingTail, clearImportResolverMemos, resolveImportPath, isExternalImport, ESM_IMPORT_LANGUAGES, jsModuleFile, pythonRootFingerprint, pythonReopenScope, pythonPackageImporters, moduleDefaultExport, isJsFamilyFile, isModuleBindingName } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; import { resolveAliasBinding } from './alias-binding'; @@ -575,6 +576,10 @@ export class ReferenceResolver { this.mocklessContext = null; this.mocklessBase = null; this.manualMocks = undefined; + this.classExpressionFreeViews = new WeakMap(); + this.classExpressionFreeBases = new WeakMap(); + this.classExpressionIndex = undefined; + this.classExpressionOnlyNameSet = null; this.nodeCache.clear(); this.fileCache.clear(); this.importMappingCache.clear(); @@ -1193,33 +1198,37 @@ export class ReferenceResolver { */ private hasAnyPossibleMatch(name: string): boolean { if (!this.knownNames) return true; // no pre-filter available + // In the view outside the class expressions, names only their classes + // and members carry name nothing (classExpressionFreeContextFor). + const only = this.classExpressionFreeBases.has(this.context) ? this.classExpressionOnlyNames() : null; + const known = (n: string): boolean => this.knownNames!.has(n) && !(only?.has(n) ?? false); // Direct name match - if (this.knownNames.has(name)) return true; + if (known(name)) return true; // For qualified names like "obj.method" or "Class::method", check the parts const dotIdx = name.indexOf('.'); if (dotIdx > 0) { const receiver = name.substring(0, dotIdx); const member = name.substring(dotIdx + 1); - if (this.knownNames.has(receiver) || this.knownNames.has(member)) return true; + if (known(receiver) || known(member)) return true; // Also check capitalized receiver (instance-method resolution) const capitalized = receiver.charAt(0).toUpperCase() + receiver.slice(1); - if (this.knownNames.has(capitalized)) return true; + if (known(capitalized)) return true; // JVM FQN: `com.example.foo.Bar` — the only useful segment is the // last one (`Bar`); the earlier check finds `example.foo.Bar` which // never matches a node name. const lastDot = name.lastIndexOf('.'); if (lastDot > dotIdx) { const tail = name.substring(lastDot + 1); - if (tail && this.knownNames.has(tail)) return true; + if (tail && known(tail)) return true; } } const colonIdx = name.indexOf('::'); if (colonIdx > 0) { const receiver = name.substring(0, colonIdx); const member = name.substring(colonIdx + 2); - if (this.knownNames.has(receiver) || this.knownNames.has(member)) return true; + if (known(receiver) || known(member)) return true; // Multi-segment path `a::b::c` (a Rust/C++ module call like // `database::profiles::find`) — the only segment that names a symbol is // the last (`c`); `member` above is `b::c`, which never matches a node @@ -1228,7 +1237,7 @@ export class ReferenceResolver { const lastColon = name.lastIndexOf('::'); if (lastColon > colonIdx) { const tail = name.substring(lastColon + 2); - if (tail && this.knownNames.has(tail)) return true; + if (tail && known(tail)) return true; } } @@ -1242,9 +1251,9 @@ export class ReferenceResolver { if (sepIdx > 0) { const receiver = name.substring(0, sepIdx); const member = name.substring(sepIdx + 1); - if (this.knownNames.has(member) || this.knownNames.has(receiver)) return true; + if (known(member) || known(receiver)) return true; const capitalized = receiver.charAt(0).toUpperCase() + receiver.slice(1); - if (this.knownNames.has(capitalized)) return true; + if (known(capitalized)) return true; } } @@ -1252,7 +1261,7 @@ export class ReferenceResolver { const slashIdx = name.lastIndexOf('/'); if (slashIdx > 0) { const fileName = name.substring(slashIdx + 1); - if (this.knownNames.has(fileName)) return true; + if (known(fileName)) return true; } return false; @@ -1380,11 +1389,20 @@ export class ReferenceResolver { resolveOne(ref: UnresolvedRef): ResolvedRef | null { // Production code is resolved as if the project's manual mocks were not - // indexed (see mocklessContextFor); every strategy below reads the graph - // through this.context, so it answers for that view. + // indexed (see mocklessContextFor), and code outside the named class + // expressions as if their classes were not (classExpressionFreeContextFor); + // every strategy below reads the graph through this.context, so it + // answers for that view. + const outer = this.context; + const unviewed = this.classExpressionFreeBases.get(outer) ?? outer; + const base = unviewed === this.mocklessContext && this.mocklessBase ? this.mocklessBase : unviewed; + const mockless = this.mocklessContextFor(ref, base) ?? base; + return this.resolveOneWith(this.classExpressionFreeContextFor(ref, mockless) ?? mockless, ref); + } + + /** resolveOneIn with every strategy reading the graph through `context`. */ + private resolveOneWith(context: ResolutionContext, ref: UnresolvedRef): ResolvedRef | null { const outer = this.context; - const base = outer === this.mocklessContext && this.mocklessBase ? this.mocklessBase : outer; - const context = this.mocklessContextFor(ref, base) ?? base; if (context === outer) return this.resolveOneIn(ref); this.context = context; try { @@ -3959,6 +3977,149 @@ export class ReferenceResolver { return this.manualMocks; } + /** + * The context a TS/JS reference is resolved in outside the named class + * expressions (#2499): one without the classes they declare or their + * members. An expression's name is bound inside it alone, and its class's + * value travels in ways resolution doesn't follow. So outside, a reference + * means what it would were those classes not indexed, with the confidence + * it had: `const Panel = class Panel {…}` keeps `new Panel()` on the + * constant, an interface beside a same-named class expression keeps its + * typed calls, and an `HTMLElement` type stays clear of lit's + * `HTMLElementShim = class HTMLElement {…}`. A class a CommonJS module + * exports isn't among them (classExpressions). Null inside a class + * expression, where the whole graph applies. + */ + private classExpressionFreeContextFor(ref: UnresolvedRef, base: ResolutionContext): ResolutionContext | null { + if (!CLASS_EXPRESSION_REF_LANGUAGES.has(ref.language)) return null; + const expressions = this.classExpressions(); + if (!expressions) return null; + const dot = ref.referenceName.indexOf('.'); + const named = expressions.byName.get(dot > 0 ? ref.referenceName.slice(0, dot) : ref.referenceName); + // Written in a class expression, a reference sees its class and the + // classes around it, as JavaScript scopes them, unless it names another + // one, which is bound only inside that one. + const inside = expressions.byFile.get(ref.filePath)?.some((n) => writtenIn(n, ref)) ?? false; + if (inside && !(named && !named.some((n) => writtenIn(n, ref)))) return null; + let view = this.classExpressionFreeViews.get(base); + if (!view) { + view = this.classExpressionFreeView(base); + this.classExpressionFreeViews.set(base, view); + this.classExpressionFreeBases.set(view, base); + } + return view; + } + + /** + * `base` without the classes class expressions declare and their members. + * The lookups an import reads a module's exports by (getNodesInFileNamed, + * getExportedNodesInFile) are left whole. + */ + private classExpressionFreeView(base: ResolutionContext): ResolutionContext { + const { hiddenIds } = this.classExpressions()!; + const hidden = (n: Node): boolean => hiddenIds.has(n.id); + const keep = (nodes: Node[]): Node[] => (nodes.some(hidden) ? nodes.filter((n) => !hidden(n)) : nodes); + const byKind = new Map(); + const getMethodMatches = base.getMethodMatches; + const iterateNodesByKind = base.iterateNodesByKind; + const { files } = this.classExpressions()!; + return { + ...base, + getNodesByName: (name) => keep(base.getNodesByName(name)), + getNodesByLowerName: (lowerName) => keep(base.getNodesByLowerName(lowerName)), + getNodesByQualifiedName: (qualifiedName) => keep(base.getNodesByQualifiedName(qualifiedName)), + getNodesInFile: (filePath) => (files.has(filePath) ? keep(base.getNodesInFile(filePath)) : base.getNodesInFile(filePath)), + getNodesByKind: (kind) => { + let nodes = byKind.get(kind); + if (!nodes) byKind.set(kind, (nodes = keep(base.getNodesByKind(kind)))); + return nodes; + }, + getMethodMatches: getMethodMatches + ? (typeName, methodName, language) => keep(getMethodMatches(typeName, methodName, language)) + : undefined, + iterateNodesByKind: iterateNodesByKind + ? function* (kind) { + for (const n of iterateNodesByKind(kind)) if (!hidden(n)) yield n; + } + : undefined, + }; + } + + /** classExpressionFreeContextFor's views, by the context each hides from and what it hides, and each view's context; clearCaches drops both. */ + private classExpressionFreeViews = new WeakMap(); + private classExpressionFreeBases = new WeakMap(); + /** + * The names that only class expressions' classes and members carry, which + * name nothing where classExpressionFreeView hides those: lit's DOM shims + * (`class HTMLElement`, `class Event`) would otherwise send every + * `HTMLElement` and `new Event()` across the project through the whole + * resolution only to find nothing, where before the name pre-filter + * dropped them. Read once per cache window. + */ + private classExpressionOnlyNames(): Set { + if (this.classExpressionOnlyNameSet) return this.classExpressionOnlyNameSet; + const only = new Set(); + const expressions = this.classExpressions(); + if (expressions) { + const names = new Set(); + for (const id of expressions.hiddenIds) { + const n = this.nodeById(id); + if (n) names.add(n.name); + } + for (const name of names) { + if (!this.queries.getNodesByName(name).some((n) => !expressions.hiddenIds.has(n.id))) only.add(name); + } + } + this.classExpressionOnlyNameSet = only; + return only; + } + + /** classExpressionOnlyNames' answer; clearCaches drops it. */ + private classExpressionOnlyNameSet: Set | null = null; + + /** + * The classes named class expressions declare: by name, the files holding + * them, the ids of them and their members, and the one a member belongs + * to. Null when there are none. Read once per cache window. A class a + * CommonJS module exports is left out: the module binds it, so it is read + * as a declaration is. + */ + private classExpressionIndex: ClassExpressionIndex | null | undefined; + + private classExpressions(): ClassExpressionIndex | null { + if (this.classExpressionIndex !== undefined) return this.classExpressionIndex; + const byName = new Map(); + const byFile = new Map(); + const byKey = new Map(); + const files = new Set(); + const hiddenIds = new Set(); + const linesOf = (file: string): readonly string[] => + this.context.getFileLines?.(file) ?? this.context.readFile(file)?.split(/\r?\n/) ?? []; + for (const n of this.queries.getNodesByKind('class')) { + if (!isJsClassExpression(n) || isCommonJsExportedClass(n, linesOf(n.filePath))) continue; + const named = byName.get(n.name); + if (named) named.push(n); else byName.set(n.name, [n]); + const inFile = byFile.get(n.filePath); + if (inFile) inFile.push(n); else byFile.set(n.filePath, [n]); + byKey.set(`${n.filePath}\0${n.qualifiedName}`, n); + files.add(n.filePath); + hiddenIds.add(n.id); + } + // A member is its class's when it is declared in its file under its + // qualified name (`Widget::run` of `Widget`). + const ownerOf = (member: Node): Node | undefined => { + const cut = member.qualifiedName.lastIndexOf('::'); + return cut > 0 ? byKey.get(`${member.filePath}\0${member.qualifiedName.slice(0, cut)}`) : undefined; + }; + if (files.size > 0) { + for (const n of this.queries.getNodesByFiles([...files])) { + if ((n.kind === 'method' || n.kind === 'property' || n.kind === 'field') && ownerOf(n)) hiddenIds.add(n.id); + } + } + this.classExpressionIndex = byName.size > 0 ? { byName, byFile, files, hiddenIds, ownerOf } : null; + return this.classExpressionIndex; + } + /** Is `filePath` (project-relative) one of the files the index holds? */ private isIndexedFile(filePath: string): boolean { const normalized = filePath.replace(/\\/g, '/'); diff --git a/src/resolution/js-class-expressions.ts b/src/resolution/js-class-expressions.ts new file mode 100644 index 0000000000..1f7ad26909 --- /dev/null +++ b/src/resolution/js-class-expressions.ts @@ -0,0 +1,66 @@ +/** + * A named class expression's name is bound inside the expression alone + * (#2499): `register(class Gadget {…})` declares no `Gadget` for the code + * around it, where `new Gadget()` throws a ReferenceError. The extractor gives + * such a class a node, so its members are its own and its `extends` is + * known, and marks it with its head as its signature (`class Gadget`), which + * a class declaration doesn't carry. + * + * The class's name is bound nowhere outside the expression, and its value + * travels in ways resolution doesn't follow, except an import of what a + * module exports. So the resolver reads a reference outside the class + * expressions without their classes and members (ReferenceResolver's + * classExpressionFreeContextFor): there the reference means what it meant + * before those classes were indexed. A binding of the same name around an + * expression, like the `Panel` of `const Panel = class Panel {…}`, keeps the + * name, at the confidence it had. A class a CommonJS module exports, + * `module.exports = class Router {…}`, is the module's value, which every + * importer binds, so it isn't scoped: it is read as a declaration is. + */ +import type { Node } from '../types'; +import type { UnresolvedRef } from './types'; + +/** Languages a class expression's class and the references to its name can be written in: TS/JS, and the scripts of Vue, Svelte and Astro files. */ +export const CLASS_EXPRESSION_REF_LANGUAGES: ReadonlySet = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'arkts', 'vue', 'svelte', 'astro']); + +/** What the resolver knows of the classes named class expressions declare (ReferenceResolver.classExpressions). */ +export interface ClassExpressionIndex { + /** The classes, by name. */ + byName: Map; + /** The classes, by the file that holds them. */ + byFile: Map; + /** The files that hold them. */ + files: Set; + /** The ids of the classes and of their members. */ + hiddenIds: Set; + /** The class a member belongs to, when it is one of them. */ + ownerOf: (member: Node) => Node | undefined; +} + +/** Whether `node` is a class a named class expression declares. */ +export function isJsClassExpression(node: Node): boolean { + return node.kind === 'class' && !!node.signature && CLASS_EXPRESSION_REF_LANGUAGES.has(node.language); +} + +/** The end of the text before a class a CommonJS module exports: `module.exports = `, `exports.Router = `, `module.exports.Router = `. */ +const COMMONJS_EXPORT_BEFORE = /(?:^|[^\w$.])(?:module\.)?exports(?:\.[A-Za-z_$][\w$]*)?\s*=\s*$/; + +/** + * Whether a class expression is what its CommonJS module exports, as in + * `module.exports = class Router {…}` or `exports.Router = class Router {…}` + * (`lines`: its file's lines). An importer reaches that class, its statics + * and, through an instance, its methods, as it reaches a declaration's. + */ +export function isCommonJsExportedClass(node: Node, lines: readonly string[]): boolean { + const line = lines[node.startLine - 1]; + if (line === undefined) return false; + return COMMONJS_EXPORT_BEFORE.test(`${lines[node.startLine - 2] ?? ''}\n${line.slice(0, node.startColumn)}`); +} + +/** Whether the reference is written inside `node`'s source range. */ +export function writtenIn(node: Node, ref: UnresolvedRef): boolean { + if (node.filePath !== ref.filePath) return false; + const afterStart = ref.line > node.startLine || (ref.line === node.startLine && ref.column >= node.startColumn); + const beforeEnd = ref.line < node.endLine || (ref.line === node.endLine && ref.column <= node.endColumn); + return afterStart && beforeEnd; +} From 9ee6718ad195c4be422614032a869196de705c5f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 03:37:25 +0000 Subject: [PATCH 56/95] fix(vue): a +`, + // Types and JSX together, which only the TSX grammar reads. + 'src/Card.vue': ` +`, + // Controls: TypeScript stays TypeScript, no lang stays JavaScript. + 'src/Plain.vue': ` + +`, + 'src/Legacy.vue': ` + +`, +}; + +let roots: string[] = []; +let graphs: CodeGraph[] = []; + +afterEach(() => { + for (const graph of graphs) graph.close(); + graphs = []; + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); + roots = []; +}); + +async function indexProject(): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vue-tsx-')); + roots.push(root); + for (const [file, text] of Object.entries(PROJECT)) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + } + const graph = await CodeGraph.init(root, { index: true }); + graphs.push(graph); + return graph; +} + +const named = (nodes: Node[], kind: string, name: string) => nodes.find((n) => n.kind === kind && n.name === name); + +/** + * The file's extraction errors. The file must be stored: one whose grammar + * was not loaded is never parsed, and nothing at all is stored for it. + */ +function errorsOf(graph: CodeGraph, file: string) { + const record = graph.getFile(file); + expect(record, `${file} is stored`).not.toBeNull(); + return record!.errors ?? []; +} + +describe('Vue \n', 'svelte')).toEqual(['javascript']); + }); +}); diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index e405adcce2..66c69687b7 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -463,7 +463,9 @@ function resolveWasmPath(lang: GrammarLanguage): string { * SFC languages (svelte/vue/astro) have no grammar of their own — their * extractors delegate \n\n", + 'pages/push.vue': + '\n' + + '\n', + 'pages/target.vue': '\n', + 'pages/login.vue': '\n', + 'middleware/auth.global.ts': + "export default defineNuxtRouteMiddleware((to) => {\n if (to.path !== '/login') {\n return navigateTo('/login');\n }\n});\n", + 'pages/account.vue': + '\n' + + '\n', + 'composables/useNav.ts': 'export function useNav() {\n return { go: (to: string) => navigateTo(to) };\n}\n', + // Two project functions named navigateTo: a plugin's own, and a helper's. + 'plugins/keys.client.ts': + 'export default defineNuxtPlugin(() => {\n' + + ' const navigateTo = (to: string) => {\n console.log(to);\n };\n' + + " window.addEventListener('keydown', () => navigateTo('/target'));\n" + + '});\n', + 'lib/open.ts': 'export function navigateTo(url: string) {\n return window.open(url);\n}\n', + 'pages/docs.vue': + "\n" + + '\n', +}; + +const roots: string[] = []; +function indexProject(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-nuxt-nav-')); + roots.push(root); + for (const [file, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + } + return CodeGraph.init(root, { index: true }); +} + +afterAll(() => { + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('Nuxt navigateTo', () => { + let cg: CodeGraph; + beforeAll(async () => { + cg = await indexProject(NUXT_APP); + }); + afterAll(() => cg?.close()); + + const route = (name: string): Node => cg.getNodesByKind('route').find((r) => r.name === name)!; + const sym = (name: string, file: string): Node => cg.getNodesInFile(file).find((n) => n.name === name && n.kind !== 'import')!; + /** The edges a node makes from its `navigateTo` calls. */ + const navigateToEdges = (from: Node): Edge[] => + cg.getOutgoingEdges(from.id).filter((e) => (e.metadata as Record | undefined)?.refName === 'navigateTo'); + const destinations = (from: Node) => + navigateToEdges(from).map((e) => `${e.kind} ${(e.metadata as Record).href} -> ${cg.getNode(e.target)?.name}`); + + it('navigates from a page to the page it names, with no edge from the page to itself', () => { + expect(destinations(sym('index', 'pages/index.vue'))).toEqual(['navigates /target -> /target']); + expect(navigateToEdges(sym('index', 'pages/index.vue'))[0]!.target).toBe(route('/target').id); + // router.push in the same app, unchanged. + expect(cg.getOutgoingEdges(sym('go', 'pages/push.vue').id).filter((e) => e.kind === 'navigates').map((e) => e.target)).toEqual([ + route('/target').id, + ]); + }); + + it('navigates from route middleware', () => { + const file = cg.getNodesInFile('middleware/auth.global.ts').find((n) => n.kind === 'file')!; + expect(destinations(file)).toEqual(['navigates /login -> /login']); + }); + + it('reads an object destination and a call written in the template', () => { + expect(destinations(sym('goLogin', 'pages/account.vue'))).toEqual(['navigates /login -> /login']); + expect(destinations(sym('account', 'pages/account.vue'))).toEqual(['navigates /target -> /target']); + }); + + it('leaves a computed destination, a path no page serves and an external URL unresolved', () => { + for (const [name, file] of [ + ['goNext', 'pages/account.vue'], + ['goMissing', 'pages/account.vue'], + ['goDocs', 'pages/account.vue'], + ['useNav', 'composables/useNav.ts'], + ] as const) { + expect(navigateToEdges(sym(name, file)), name).toEqual([]); + } + // Neither project function named navigateTo is called by any of them, nor + // does any navigateTo call point at its own caller. + const namesakes = cg.getNodesByName('navigateTo').map((n) => n.id); + expect(namesakes).toHaveLength(2); + const intoNamesakes = cg.getIncomingEdgesTo(namesakes, ['calls']); + expect(intoNamesakes.map((e) => `${cg.getNode(e.source)?.name} -> ${cg.getNode(e.target)?.filePath}`)).toEqual([ + 'openDocs -> lib/open.ts', + ]); + }); + + it('calls the navigateTo a file imports, and navigates through the one a file declares', () => { + const docs = cg.getOutgoingEdges(sym('openDocs', 'pages/docs.vue').id).filter((e) => e.kind === 'calls'); + expect(docs.map((e) => cg.getNode(e.target)).map((n) => `${n?.name} ${n?.filePath}`)).toEqual(['navigateTo lib/open.ts']); + // The plugin's wrapper is called with a page's path: where it goes, as before. + const plugin = cg.getNodesInFile('plugins/keys.client.ts').find((n) => n.kind === 'file')!; + expect(destinations(plugin)).toEqual(['navigates /target -> /target']); + }); +}); + +describe('navigateTo outside Nuxt', () => { + it('is the project function a plain Vue app imports', async () => { + const cg = await indexProject({ + 'package.json': JSON.stringify({ name: 'app', dependencies: { vue: '^3.5.0', 'vue-router': '^4.4.0' } }), + 'tsconfig.json': JSON.stringify({ compilerOptions: { baseUrl: '.', paths: { '@/*': ['src/*'] } } }), + 'src/router/index.ts': + "import { createRouter, createWebHistory } from 'vue-router';\n" + + "export default createRouter({ history: createWebHistory(), routes: [{ path: '/login', component: () => import('@/views/Login.vue') }] });\n", + 'src/views/Login.vue': '\n', + 'src/utils/nav.ts': 'export function navigateTo(path: string) {\n window.location.assign(path);\n}\n', + 'src/views/Home.vue': + "\n" + + '\n', + }); + try { + const go = cg.getNodesInFile('src/views/Home.vue').find((n) => n.name === 'go')!; + const calls = cg.getOutgoingEdges(go.id).filter((e) => e.kind === 'calls'); + expect(calls.map((e) => cg.getNode(e.target)?.filePath)).toEqual(['src/utils/nav.ts']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/sync-navigation-retry.test.ts b/__tests__/sync-navigation-retry.test.ts index 79070b1e27..1d3a05ca1f 100644 --- a/__tests__/sync-navigation-retry.test.ts +++ b/__tests__/sync-navigation-retry.test.ts @@ -315,6 +315,24 @@ const CASES: Case[] = [ from: 'src/views/Home.vue', expected: '/login → /login', }, + { + // Nuxt's auto-imported navigateTo, waiting for its page (#2634). + name: 'Nuxt (navigateTo)', + files: { + 'package.json': JSON.stringify({ name: 'site', dependencies: { nuxt: '4', vue: '3' } }), + 'pages/index.vue': + '\n' + + '\n', + 'pages/login.vue': '\n', + }, + route: { file: 'pages/login.vue', before: null }, + from: 'pages/index.vue', + expected: '/login → /login', + }, { name: 'Angular', files: { diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index e94571780e..8421fb46b7 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -1,6 +1,6 @@ # Framework & language coverage — what is done, what is left -**Last verified: 2026-08-29** (Angular row: 2026-10-06; React Router row: 2026-10-19) against the build at that date. Re-verify with the +**Last verified: 2026-08-29** (Angular row: 2026-10-06; React Router row: 2026-10-19; Vue Router / Nuxt row: 2026-10-22) against the build at that date. Re-verify with the queries in [Checking this file is still true](#checking-this-file-is-still-true) before trusting a row; this is a snapshot, not a live view. @@ -40,7 +40,7 @@ guessed. | Next.js | `frameworks/nextjs.ts` | `next-router-synthesizer.ts` | `nextjs.test.ts` | next-saas-starter | | React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges), proshop-v2 (28), react-redux-realworld (22), react-boilerplate (`styled(Link)`), takenote (v5 ``), bulletproof-react (nested `children`, `lazy` routes, `paths.x.path` constants through an app's own tsconfig alias: 0 → 9 named, linked routes), and its `paths.x.getHref(id)` links and `navigate(paths.x.getHref())` calls (0 → 11 navigates). Route tables another file hands the router (`tableRoutes` in `frameworks/react.ts`, trap 13): the ASP.NET Core React template's `AppRoutes.map(… )` on jasontaylordev/CleanArchitecture's `ClientApp-React` (0 → 6 routes, 11 navigates), replaysMike/Binner (0 → 46, 87), GavinLonDigital/RankingApp, GarrettHays/PaydirtPickem, charlessolar/eShopOnContainersDDD; `useRoutes(routes)` from another file and `[...sessionRoutes]` on uilibrary/matx-react (0 → 7, 16), and its `[...materialRoutes]` once the jsconfig `"baseUrl": "src"` import naming it resolved (7 → 21 routes, each bound through `lazy(() => import(…))` to its page, trap 20); codedthemes' `createBrowserRouter([MainRoutes, LoginRoutes])` on berry and mantis (9 and 8 routes, no navigation they can resolve; each bound to its lazily loaded page and its layout since their `baseUrl` / `"*"` imports resolve, where before they named the local `Loadable(lazy(…))` value); a table `.map`ped inside `` on RADeveloping/chickadeeinvest; `useRoutes([...])` in place on minimal-ui-kit/material-kit-react (0 → 7, bound through `lazy(() => import(…))` to each page). A table a function returns, handed over through a variable (`const routes = getRouteMap({ … })` in a class's `render()`, then `createBrowserRouter(routes)`; trap 13): Medusa's admin dashboard (0 → 221 routes, each bound to its page, 613 layout edges, 92 navigations newly linked: 60 `navigate()` calls and 32 ``s; all 821 edges an AST oracle on the TypeScript compiler expects, and no others), Mercur's admin panel (0 → 216; 775 of 775), serenedb's serene-ui (`buildRoutes = () => [...appRoutes, ...loaderRoutes]`: 0 → 4), 22 control repos byte-identical, and a late-file sync equal to a fresh index on both dashboards. Route objects that render `` (`element: `, `Component: Outlet`, a `lazy` loader handing back `{ Component: Outlet }`; trap 14): both dashboards and mercurjs/vendor-panel drop every `layout:Outlet` reference they made (93, 106 and 87 rows, none of which linked anything) and the 13, 5 and 5 layout edges that drew a product's detail page around its variants' pages; React Router's error-boundaries example's `/` becomes its `Layout` page instead of React Router's own `Outlet`; 21 other repos byte-identical, and 2,677 surveyed router files change only by dropping 40 such references. A table behind `useMemo` (trap 13): anotherIlnaz/tg-tt-bot-stage (`useMemo(() => getRoutes(personalAcc), [personalAcc])`: 0 → 12 routes, 16 navigations), dingtalkwukong/bewild-clone (`useMemo(() => buildRoutes(), [])`: 0 → 13, 4) and the-ctrlv/SpaceR (`useMemo(getRoutes, [])` through a `baseUrl` import: 0 → 10, 21), each route bound to the page its table names; EyeSeeTea/training-app, whose builder maps a config into routes, and BANKFORM1, whose route file imports `createBrowserRouter` and so was read already, unchanged, with 25 other repos byte-identical. JSX index routes and layouts (trap 14): proshop-v2's `createRoutesFromElements` (`/` is HomeScreen inside App, not App; 30 layout edges, the `path=''` guards included; Screens 30 → 74 transitions, the guards' redirects drawn from the screens they guard), chickadeeinvest's table inside `}>` (its 4 routes get the layout), React Router 6.29's own examples and tests, refine's 281 examples (one `/` per app, 191 layout edges, 21 new navigations), crwn-clothing, the vanlife course app, volun-mern (105 layout edges, six levels deep), spotify/vispana, react_persist_login, chushi (`path={"agents"}`: 2 → 6 routes), hathor-explorer (`path=""` at the root), cboard, react-router-auth-v6: 578 of 581 new layout edges name a component of an enclosing `` (vanlife's 3 inherit a default-import resolution bug that already misbinds its pages), 27 of 27 new navigations precise, no navigation lost, 16 control repos byte-identical. Lazy route modules that are barrels (`exportedComponent` in `frameworks/react.ts`): luci-go's milo/ui (`export * from './x'` index files and `export { X as Component }` clauses: 0 → 51 lazy routes linked; the other 40 load a file under `src/build/`, which the default ignore skips, export only a `loader` or a `handle`, or pick a named export in an async `lazy`), Medusa's admin dashboard (all 193 route-map modules resolve through its `export { X as Component } from` barrels), four more apps (9 routes), 16 control repos byte-identical. Lazy loaders that pick their page (`lazyRouteReference` in `frameworks/react.ts`, trap 18): milo/ui's three tabs (`const { TestTab } = await import('…/tabs')`), 20 apps from a GitHub survey of `createBrowserRouter` + `lazy` (636 picked-page edges, each checked against the target file's own export: 467 by name, 169 as its `export default`; 42 routes written with `async lazy()` now indexed, among them React Router's own `lazy-loading-router-provider` example), an AST oracle on the TypeScript compiler agreeing with the text reader on all 2,538 loaders of 876 surveyed router files, a late-file sync equal to a fresh index on 8 of those apps, and 26 control repos byte-identical, among them ghostfolio, whose 30 Angular lazy routes stay Angular's with React detected in the same project. Barrels that pass on a page they import (`import { UserLayout } from './UserLayout'; export { UserLayout }`, `import Login from './Login'; export default Login`; `exportedComponent` follows the import): Arshia76/DigiShop.Frontend (0 → 27 route and layout edges), 49 other repos byte-identical; `export default withAuth(Page)` is followed nowhere. A page in a folder below the barrel's (`pages/Home/index.tsx` behind `@/pages`) whose route failed waits under the barrel's module key (`module:pages`), which the page's own keys never give: a sync that adds the page, or gives it a definition, walks up the import edges to the barrels that pass it on (`reopenReferencesThroughBarrels` in `resolution/module-importers.ts`) and re-opens what waits under theirs. A late-file sync on DigiShop then equals a fresh index (13 of its routes stayed unlinked before), and 27 other repos sync exactly as before. A name the route file declares itself (trap 19), such as a lazy page or layout, a `Loadable(lazy(…))` or an `observer(…)` value, binds to what that declaration renders wherever the file uses it, in routes, layouts and JSX alike: outline (+31 edges, 7 moved off another file's namesake or onto the screen a lazy declaration loads), mastodon (+20, 9 moved: `lazy(() => import(…).then(({ Emoji }) => …))` values and `let Emoji` locals had gone to other files' components), SigNoz (+7, 4 moved), excalidraw (+3, 1 moved: the Next.js example's `` was the docs site's), bulletproof-react (+6), material-kit-react and create-t3-turbo (+1 each), with matx, mantis, proshop, next-saas-starter and berry byte-identical | | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | -| Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | +| Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts`, `nuxt-navigate-to.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies. Nuxt's `navigateTo` (trap 21): mealie 13 of its 14 calls, nuxt.com 17 of 24, elk 9 of 10 outside the plugin that declares its own, atidone 2 of 2, pages and route middleware alike; the rest are computed, an API endpoint (`{ external: true }`), or a page of a Nuxt layer (`extends: ['./layers/nuxi']`), whose pages sit in a route table of their own. nuxt/ui's docs: 0 of 4, two computed and two whose only page is a catch-all (`[...slug].vue`, named `*slug`) or an optional parameter (`[[tag]].vue`, `:tag?`), which `matchRoute` doesn't read yet, for markup links either | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts`, `sveltekit-route-names.test.ts` | sveltekit-realworld (31 edges); shadcn-svelte and skeleton (`(group)` layouts: 13 and 23 edges), svelte.dev (74), kit's test apps (47) | | Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders, 32 bindings), Ghostfolio (189 edges, 170 renders, 215 bindings), ngx-admin (routes and renders; its menus are config; 13 bindings), CleanArchitecture's `src/Web/ClientApp` (signals; 12 bindings), angular-spotify (Nx libs behind barrels: 14 routes), jira-clone (class-constant paths, mount-only redirects), jhipster (60), ionic-conference (18), Angular-JumpStart (18) | @@ -435,6 +435,20 @@ Each of these cost real debugging time; they are not hypothetical. project keeps for it (`"*": ["./typings/*"]`, as tsc reads it). A route resolves its lazy module as TypeScript whatever the file is written in, which is why the TypeScript extension list probes `/index.jsx` last. +21. **A navigation call named like a function must not fall back to one.** + Nuxt auto-imports `navigateTo`, so a call to it is written bare, with no + import to say where it comes from. When Vue Router can't read where it goes + — a computed destination, a path no page serves, an external URL — the + reference would reach the name matcher, which links any project function + of that name: elk declares a `navigateTo` of its own inside a plugin. The + Vue resolver used to sink every `navigateTo` into an edge to its caller, + which hid the navigation too, and which sync never revisits (trap 16). + `FrameworkResolver.ownsReference` names the references a framework answers + alone: `nuxtResolver` owns a bare `navigateTo` that no declaration or + import in the file binds, and one no router links stays unresolved, so a + sync binds it when its page appears. An absolute URL is no path of the app: + Vue Router's call reader keeps only destinations that start with `/`, as + its markup reader does. --- diff --git a/src/resolution/frameworks/vue-router.ts b/src/resolution/frameworks/vue-router.ts index f84dad897b..a62e0629e2 100644 --- a/src/resolution/frameworks/vue-router.ts +++ b/src/resolution/frameworks/vue-router.ts @@ -535,7 +535,10 @@ export const vueRouterResolver: FrameworkResolver = { if (!href) return null; // Every arm of a conditional destination is somewhere this call goes; the // first is this reference's resolution and the rest ride as `alsoTargets`. - const targets = destinationsForHref(href, routes); + // An arm that is no path of the app — an absolute URL, which Nuxt's + // `navigateTo(url, { external: true })` leaves the app for — opens none of + // its routes, as `` is read (vue-router-synthesizer). + const targets = destinationsForHref(href, routes).filter((t) => t.href.path.startsWith('/')); const target = targets[0]; if (!target) return null; return { diff --git a/src/resolution/frameworks/vue.ts b/src/resolution/frameworks/vue.ts index b8a5a3c2ee..257c536fdf 100644 --- a/src/resolution/frameworks/vue.ts +++ b/src/resolution/frameworks/vue.ts @@ -28,13 +28,15 @@ const VUE_COMPILER_MACROS = new Set([ ]); /** - * Nuxt auto-imported composables and utilities + * Nuxt auto-imported composables and utilities. `navigateTo` is not one of + * them here: `navigateTo('/x')` is a navigation, which Vue Router's resolver + * links to the page it opens, and Nuxt owns the call when it can't (see + * `nuxtResolver.ownsReference`, #2634). */ const NUXT_AUTO_IMPORTS = new Set([ // Routing 'useRoute', 'useRouter', - 'navigateTo', 'abortNavigation', // Data fetching 'useFetch', @@ -213,6 +215,25 @@ export const nuxtResolver: FrameworkResolver = { return resolvePageComponent(ref, context); }, + /** + * Nuxt's auto-imported `navigateTo` (#2634). Vue Router's resolver links a + * call to the page it opens; one it can't link — a computed destination, a + * path no page serves, an external URL — still calls Nuxt's, never a + * project function that shares the name, such as the `navigateTo` elk + * declares inside a plugin. A file that declares or imports a `navigateTo` + * calls that one, which the other strategies find (an import from `#app` + * links nothing, as any package's does). + */ + ownsReference(ref: UnresolvedRef, context: ResolutionContext): boolean { + return ( + ref.referenceKind === 'calls' && + ref.referenceName === 'navigateTo' && + VUE_SCRIPT_LANGUAGES.has(ref.language) && + !declaresOwnFunction(ref, context) && + !context.getImportMappings(ref.filePath, ref.language).some((m) => m.localName === ref.referenceName) + ); + }, + extract(filePath: string, _content: string) { const nodes: Node[] = []; const references: UnresolvedRef[] = []; diff --git a/src/resolution/index.ts b/src/resolution/index.ts index e2e717afc1..5e3e46ea0f 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -1713,6 +1713,14 @@ export class ReferenceResolver { } if (this.profileStages) this.stageAdd('frameworks', ref, fwEarly !== null, tFw); if (fwEarly) return fwEarly; + // A reference a framework answers alone ends with the frameworks' answer + // (see FrameworkResolver.ownsReference): Nuxt's `navigateTo(x)` with a + // destination no route serves is no namesake's call. + if (this.frameworksFor(ref.language).some((f) => f.ownsReference?.(ref, this.context))) { + return candidates.length > 0 + ? candidates.reduce((best, curr) => (curr.confidence > best.confidence ? curr : best)) + : null; + } // A retained untyped chain supplies effect/call-site evidence only. In // particular, importing its root does not make the root its call target. // A path through module namespaces is not untyped: `z.coerce.number()` diff --git a/src/resolution/types.ts b/src/resolution/types.ts index b4e5dae4d5..d5d5dcb634 100644 --- a/src/resolution/types.ts +++ b/src/resolution/types.ts @@ -373,6 +373,17 @@ export interface FrameworkResolver { * Asked only about references written in the languages `resolve()` sees. */ claimsReference?(name: string): boolean; + /** + * A reference this framework answers alone. When no framework's `resolve()` + * links it, it stays unresolved: no import or name strategy guesses past it, + * as they would for a project function that only shares its name. Nuxt's + * auto-imported `navigateTo('/x')` names a page, which Vue Router reads; + * one whose destination is computed, or a path no page serves, calls no + * function of the project — elk declares a `navigateTo` of its own inside a + * plugin. Unresolved, a sync can still revisit it when a route appears. + * Asked only about references written in the languages `resolve()` sees. + */ + ownsReference?(ref: UnresolvedRef, context: ResolutionContext): boolean; /** * Extract framework-specific nodes and references from a file. * From 5b47bc4963433507957b1112dc58ea26f0978c56 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 04:41:00 +0000 Subject: [PATCH 58/95] fix(js,ts): .tsx and .jsx classes link their overrides as .ts and .js classes do (#2828) The interface-impl pass links a base class's or an interface's method to each subclass's override, so a call through the base reaches the method that runs. It took a class's methods, and its supertype, by language: typescript and javascript were in its language set, and tsx and jsx were not. A class in a .tsx or .jsx file got no override edges, and neither did a .ts class whose base or interface was in a .tsx file. The pass's own gate, the static/instance split of #2761 and the other JS-family lists already count tsx and jsx, and the set's history (#494, #708) gives no reason for leaving them out. tsx and jsx are now in the set, so these classes get the edges that .ts and .js classes get, statics paired with statics. Validation: the new test fails 4 of 4 on main 013b91c1 and passes 4 of 4 on the fix, and dropping tsx or jsx alone from the set fails the tests for it. A/B against main on 11 repos: outline gains 262 override edges in 42 base-subclass pairs (26 emails overriding BaseEmail's subject, render and preview; editor nodes, marks and extensions overriding Node, Mark and Extension) and lexical 27 in 12, grouped by pair and sampled against the source, none pairing a static with an instance method. Nodes and all other edges are unchanged. excalidraw, social-app, insomnia, two .jsx apps and the controls (three, webpack, flask, gin) are byte-identical: their .tsx classes extend React's or other packages' classes. explore on outline now leads with the flow BaseEmail.send -> render -> WelcomeEmail.render, where main finds none. A sync that adds an interface method, adds an override and drops a subclass's base matches a fresh index. The pass takes 65-108 ms on both arms, and index time, median of 3 interleaved runs on lexical, outline, excalidraw and social-app, is +1.7% in total, noise. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/jsx-override-dispatch.test.ts | 97 +++++++++++++++++++++++++ src/resolution/callback-synthesizer.ts | 9 ++- 3 files changed, 103 insertions(+), 4 deletions(-) create mode 100644 __tests__/jsx-override-dispatch.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e069b7045..88de4b516a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -203,6 +203,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a type alias like `type Event = mvccpb.Event`, alone or inside a `type ( … )` group, is now indexed and links to the type it names, so impact on that type reaches the code that uses the alias. Before, aliases were missing: a use written through the alias's package, like `clientv3.Event` or prometheus's `storage.AOptions{…}`, linked to nothing, and one inside the package, like `&PrometheusVersion{}` in prometheus's `web` package, linked to whichever type of that name the project had, there a test helper's struct. A method called on a value of an alias's type now reaches the aliased type's method rather than a namesake's, and a generic alias like `type Items[T any] = List[T]` links to `List`, so `xs.Len()` on one reaches `List.Len` instead of nothing. Re-index Go projects after upgrading. - In Go, a struct that embeds another is no longer treated as overriding the methods it redeclares. gin's `Engine` embeds `RouterGroup` and has a `Use` of its own, but calling `Use` on a `RouterGroup` only ever runs `RouterGroup.Use`; before, `codegraph_explore`'s call path and impact went on from it into `Engine.Use` and the code only the engine runs. The same held for every struct or defined type embedded in another that redeclares one of its methods, like etcd's `Mutex` inside `lockerMutex`. A call through an interface still reaches each type that implements it, the embedding struct included. Re-index Go projects after upgrading. - In JavaScript and TypeScript, a subclass's `static` method no longer counts as overriding an instance method of the same name in its base class, or the reverse, since calling one never runs the other. In webpack, `codegraph_explore`'s call path and impact went on from `Module`'s `deserialize(context)` into the `static deserialize(context)` of `NormalModule` and other modules. A static method still overrides its base's static method and an instance method its base's instance method, and JavaScript class members are now marked static, as TypeScript ones already were. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and @yang0228 for the pull request. (#2761) +- In TypeScript and JavaScript, a class in a `.tsx` or `.jsx` file now links its base class's and interfaces' methods to its overrides, as a class in a `.ts` or `.js` file already did. `codegraph_explore`'s call path, callers and impact now follow a call through the base into the override, like outline's `BaseEmail.send()` reaching each email's `render()`. This also works across file types, such as an interface in a `.ts` file implemented in a `.tsx` one. Before, these classes had no override links at all. Re-index projects with `.tsx` or `.jsx` files after upgrading. - In Java, Kotlin, C# and Scala, each overload of an interface or base-class method now links only to the overrides with the same parameters. Before, every overload linked to every method of that name in the implementing class, so `ExampleService.putTairValue(String)` also led to `putTairValue(List, double)`, and `codegraph_explore`'s call path, callers and impact went into overloads that call never runs. Where a parameter is a type parameter, like `save(T entity)` implemented as `save(String entity)`, or the class is written in another language than its interface, the number of parameters decides; a method declared once on each side links as before. Re-index Java, Kotlin, C# and Scala projects after upgrading. Thanks @hxj1225 for the report and @danusha2345 for the pull request. (#2663) - In Kotlin, a call on what one of your functions returns, like `repository().pending(1)` or `val repo = repository(); repo.pending(1)` with a `private fun repository(): RealRepo` in a test class, now links to the method of the type that function declares, including one the type inherits or an extension on it. Before, the chained call went to whichever `pending` the name matched first, like a test double's in another package, and the local was guessed from its name, which reached the interface. The function is found in the calling class, a class it inherits from or one around it, or at the top level, and a nested type like `Request.Builder` is that type, not another class's `Builder`. When the declared type has no such method, the call now links to nothing rather than a method of the same name elsewhere; a type from outside your project is matched by name as before. Re-index Kotlin projects after upgrading. Thanks @mindcipherrr for the report. (#2805) - In JavaScript and TypeScript, a `super.run()` call now links to the method of the class's parent, or of the nearest ancestor that has it, like three.js's `super.copy(source)` reaching `Object3D.copy`, so `codegraph_explore`'s call path and impact follow an override into the method it builds on. Before, such a call inside a method of the same name was dropped, and elsewhere it went to whatever method had the name, often the class's own or a sibling class's in the same file. A call from a static method reaches the parent's static method, and a call to a parent from outside your project, like `HTMLElement`, `Set` or `Promise`, links nothing. Re-index JavaScript and TypeScript projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2497) diff --git a/__tests__/jsx-override-dispatch.test.ts b/__tests__/jsx-override-dispatch.test.ts new file mode 100644 index 0000000000..ec7d3b70bf --- /dev/null +++ b/__tests__/jsx-override-dispatch.test.ts @@ -0,0 +1,97 @@ +/** + * The override-dispatch edges (`synthesizedBy: interface-impl`) link an + * interface's or a base class's method to the method of the same name that a + * class implementing or extending it declares, so a call through the base + * reaches the override. They were made for `.ts` and `.js` classes only: a + * `.tsx` or `.jsx` class got none, though it is a TypeScript or JavaScript + * class, and neither did a `.ts` class whose base or interface lives in a + * `.tsx` file, like outline's emails extending `BaseEmail`. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +describe('override dispatch in .tsx and .jsx classes', () => { + let root = ''; + let graph: CodeGraph | null = null; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-jsx-override-')); + const write = (rel: string, lines: string[]) => fs.writeFileSync(path.join(root, rel), lines.join('\n') + '\n'); + fs.writeFileSync(path.join(root, 'package.json'), JSON.stringify({ name: 'app' })); + fs.mkdirSync(path.join(root, 'src')); + const classes = [ + 'export class Widget { render() { return null; } static create() { return 1; } }', + 'export class Button extends Widget { render() { return null; } static create() { return 2; } }', + // A static method overrides only a static one (#2761). + 'export class Toggle extends Widget { static render() { return null; } }', + ]; + write('src/widgets.tsx', classes); + write('src/widgets.jsx', classes); + // An interface and an abstract base in a .ts file, implemented in .tsx. + write('src/shape.ts', [ + 'export interface Shape { area(): number }', + 'export abstract class Figure { abstract draw(): string; describe(): string { return this.draw(); } }', + ]); + write('src/Circle.tsx', [ + "import { Shape, Figure } from './shape';", + 'export class Circle extends Figure implements Shape {', + ' area(): number { return 3; }', + ' draw(): string { return "o"; }', + ' view() { return
; }', + '}', + ]); + // outline's shape: a base in a .tsx file, a subclass in a .ts one. + write('src/BaseEmail.tsx', ['export class BaseEmail { subject() { return ""; } render() { return

; } }']); + write('src/WelcomeEmail.ts', [ + "import { BaseEmail } from './BaseEmail';", + 'export class WelcomeEmail extends BaseEmail { subject() { return "Welcome"; } }', + ]); + graph = await CodeGraph.init(root, { index: true }); + }, 120_000); + + afterAll(() => { + graph?.close(); + graph = null; + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + /** `base -> override` for every interface-impl edge out of a file's nodes, by qualified name (and file, when another). */ + function overrides(file: string): string[] { + const g = graph!; + const out: string[] = []; + const label = (id: string) => { + const n = g.getNode(id)!; + return `${n.isStatic ? 'static ' : ''}${n.qualifiedName}${n.filePath === file ? '' : ` (${path.basename(n.filePath)})`}`; + }; + for (const node of g.getNodesInFile(file)) { + for (const e of g.getOutgoingEdges(node.id)) { + if (e.kind === 'calls' && e.metadata?.synthesizedBy === 'interface-impl') out.push(`${label(e.source)} -> ${label(e.target)}`); + } + } + return out.sort(); + } + + const sameFile = ['Widget::render -> Button::render', 'static Widget::create -> static Button::create']; + + it('in a .tsx file, statics to statics', () => { + expect(overrides('src/widgets.tsx')).toEqual(sameFile); + }); + + it('in a .jsx file, statics to statics', () => { + expect(overrides('src/widgets.jsx')).toEqual(sameFile); + }); + + it('from a .ts interface and abstract base to the .tsx class implementing them', () => { + expect(overrides('src/shape.ts')).toEqual([ + 'Figure::draw -> Circle::draw (Circle.tsx)', + 'Shape::area -> Circle::area (Circle.tsx)', + ]); + }); + + it('from a .tsx base to the .ts class extending it', () => { + expect(overrides('src/BaseEmail.tsx')).toEqual(['BaseEmail::subject -> WelcomeEmail::subject (WelcomeEmail.ts)']); + }); +}); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 51a56a91f9..d747898abf 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -820,16 +820,17 @@ async function cppOverrideEdges(queries: QueryBuilder, ctx: ResolutionContext, o // kind that holds methods" — i.e. the shape this loop expects. Swift and // Scala fit shape-wise (Swift `protocol`/`class`, Scala `trait`/`class`) // and are added below; their concrete-side nodes can be a `struct` (Swift) -// or an `object` (Scala) so the loop also iterates those kinds. +// or an `object` (Scala) so the loop also iterates those kinds. A `.tsx` or +// `.jsx` class is a TypeScript or JavaScript class, and its base or interface +// is often in a `.ts` file, or the other way round. const IFACE_OVERRIDE_LANGS = new Set([ - 'java', 'kotlin', 'csharp', 'typescript', 'javascript', 'swift', 'scala', 'go', 'rust', + 'java', 'kotlin', 'csharp', 'typescript', 'tsx', 'javascript', 'jsx', 'swift', 'scala', 'go', 'rust', 'arkts', ]); /** * Languages whose static and instance methods override only their own kind: * a JS/TS class's statics are inherited and overridden by the subclass's - * statics, its instance methods by instance methods (#2761). tsx and jsx - * follow the same rule, though this pass doesn't walk their classes today. + * statics, its instance methods by instance methods (#2761). */ const STATIC_SPLIT_LANGS: ReadonlySet = new Set(['typescript', 'tsx', 'javascript', 'jsx']); /** From dcb985f1d202ca084527d0abf60b3ba28c063a97 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 05:20:21 +0000 Subject: [PATCH 59/95] fix(ruby): a call on a namespaced class links its caller to the class and its method (#2827) A Ruby call on a constant written with its namespace, like Billing::ChargeCard.call(...) or ::Foo.call, recorded no edge. Both extractors (the kernel's ruby.rs and the wasm one) emitted the receiver's references ref only for a bare `constant` receiver, so a scope_resolution receiver got none. The call ref itself, Billing::ChargeCard.call, matched no resolver strategy either: the method-call patterns reject a `::` in the receiver. A class used only through such calls showed no callers and no dependents. Fix: - Both engines treat a constant path (a constant, or a scope_resolution of constants, a leading `::` included) as a constant receiver and emit the references ref under the receiver as written. A path through a value (obj::Thing, self.class::LIMIT) still emits none, and .new stays an instantiation. - A new resolver step, src/resolution/ruby-constant-paths.ts, resolves a Ruby references ref written as such a path, and a calls ref on one, the way Ruby looks the constant up: the class or module around the reference, then each namespace outside it, then the top level; a `::`-rooted path at the top level only. A call reaches the class's or module's own method, else the first one its ancestry declares (rubyAncestry, now exported). The step runs before the pre-filter, which dropped `::`-rooted names, and it is exclusive: a path the project doesn't declare (ActiveRecord::Base) or a class without the method links nothing, never a namesake by its last name, and production code never reaches a class only a test declares. Validation: - New __tests__/ruby-namespaced-receiver.test.ts (8 cases, each on the kernel and on wasm, plus a sync case) and a parity case in kernel-ruby-parity.test.ts. On main 16 of the 22 tests in the two files fail (the rest are guards and the old parity cases); all pass with the fix. Reverting only the resolver step, or only the wasm extractor, turns 13 and 10 of them red. - A/B against main (kernel path), every edge triaged: sinatra +23 edges, rubocop +168, mastodon's Ruby tree (4,363 files) +676; 0 removed, nodes identical. 866 of 867 are right: references by the path as written or under an enclosing namespace (each such case checked against the call site's own nesting), calls to a class method, a module_function method, or an inherited class method. 1 is wrong: Paperclip::Transcoder.make reaches the instance method `make`, since a singleton method and an instance method share one qualified name today; a bare Transcoder.make does the same on main. Controls express and flask are byte-identical. - kernel-parity.mjs --lang ruby: sinatra 147/147, rubocop 1799/1801 (the same 2 deferred on main), mastodon 3333/3333, 0 diffs. The fix arm's kernel and wasm graphs are byte-identical on sinatra and rubocop. - Index time, median of 3 interleaved runs: +0.4% to +1.7%, the same as the controls (+0.8%, +1.3%). Fixes #2503 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/kernel-ruby-parity.test.ts | 41 +++ __tests__/ruby-namespaced-receiver.test.ts | 312 +++++++++++++++++++++ codegraph-kernel/src/ruby.rs | 23 +- docs/design/ruby-kernel-port-checklist.md | 18 +- src/extraction/tree-sitter.ts | 17 +- src/resolution/index.ts | 10 + src/resolution/name-matcher.ts | 2 +- src/resolution/ruby-constant-paths.ts | 118 ++++++++ 9 files changed, 529 insertions(+), 13 deletions(-) create mode 100644 __tests__/ruby-namespaced-receiver.test.ts create mode 100644 src/resolution/ruby-constant-paths.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 88de4b516a..0fbd831ffb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -151,6 +151,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, a call on a variable, parameter or member declared with template arguments nested inside its template arguments, like `autovector>` or `Striped>`, now links to the method of the class it is declared as. Before, the nested arguments hid that class: in RocksDB, `files_marked_for_compaction_.clear()` on such an `autovector` linked to an unrelated `CompactionInputFiles::clear` that only shares words with the variable's name, and `mutex_.Get(key)` on a `Striped` linked to nothing. Re-index C++ projects after upgrading. - In C++, a base class like `Message` in `class DynamicMessage final : public Message` now links to the class C++ itself would pick: the one declared in the namespaces around the class, or brought in by the file's `using` declarations and namespace aliases, and the class a `typedef` or `using` alias names. Before, a base class was matched by its name alone, so it could land on a class of the same name in another namespace or another language: in protocolbuffers/protobuf, `DynamicMessage` derived from the JSON parser's internal `Message` class, and in google/leveldb and RocksDB, iterators derived from an unrelated nested `Iterator` class such as the skip list's. The wrong base showed up in `codegraph_explore`'s type hierarchy and in impact, and tied the base's methods to the wrong overrides. A template parameter used as a base no longer links to a class that shares its name, and a base written from the global scope, like `public ::testing::Test`, now links to its class. Re-index C++ projects after upgrading. - In C++ and Rust, a call written with a qualifier, like `Status::OK()`, now links to a method of the class it names, never to one of a class whose name only ends the same way. Before, facebook/rocksdb's calls to `Status::OK()`, `Status::InvalidArgument(…)` and `Status::Corruption(…)` linked to `IOStatus`'s methods of those names, a subclass calling its base, like `return FileSystemWrapper::NewWritableFile(…)` or google/leveldb's `return DB::Put(…)`, linked to another subclass or to the calling method itself, and BurntSushi/ripgrep's `Path::new(…)` on the standard library's `Path` linked to its own `PrinterPath::new`. When classes in several languages share the name, the one in the call's own language is taken, so protocolbuffers/protobuf's C++ `DescriptorPool::generated_pool()` calls, which met its Ruby namesake first and linked to nothing, now link to the C++ method. A C++ call through a template parameter, like `CacheShard::ComputeHash(key)` in rocksdb's `template class ShardedCache`, no longer links to a class that merely shares words with the parameter's name, unless that class is the parameter's default, like `TcParser` in `template `. A template's value parameter, like `const PrefetchOpts& kOpts`, is no longer mistaken for a type parameter, so references to `PrefetchOpts` inside it now link. Re-index C++ and Rust projects after upgrading. +- In Ruby, a call on a class written with its namespace, like `Billing::ChargeCard.call(…)` or `::Billing::ChargeCard.call(…)`, now links the caller to that class and to the class method it calls, as `ChargeCard.call(…)` already linked to the class. Before, a class used only through such calls looked like it had no callers. The name is looked up the way Ruby does it, from the namespaces around the call outward, so `Users::Create.call` reaches `Users::Create` and never a `Billing::Create`, a call inside `module Api; module V2` reaches `Api::V2`'s class before another version's, and a library's class like `ActiveRecord::Base` is never taken for a project class of the same last name. Re-index Ruby projects after upgrading. Thanks @james-jang for the report and @drakeo338 for the pull request. (#2503) - In C++, a call through a class template's specialization written with its arguments, like fmtlib/fmt's `formatter::format(42, ctx)`, no longer links to a member defined for another specialization, like a test's `formatter::format`, and when the template declares no such member, as for `formatter::parse(ctx)`, the call is no longer matched to another class's method of that name, like `native_formatter::parse`. A call to the specialization a member is defined for, and to a template's own members, links as before. Re-index C++ projects after upgrading. - In C++, a function is now named by its declarator, never by something written in the clauses after its parameters. nlohmann/json's constructor `basic_json(CompatibleType&& val) noexcept(noexcept(JSONSerializer::to_json(…)))` was missing from the index, replaced by a method `JSONSerializer::to_json` that doesn't exist and that other `JSONSerializer::to_json(…)` calls linked to; a method declared with `throw(std::bad_alloc)` was indexed as `std::bad_alloc`, and a function with a trailing `requires std::integral` as `std::integral`. Re-index C++ projects after upgrading. - In C and C++, a header or source file named like a test, such as protocolbuffers/protobuf's `conformance_test.h` and `test_runner.h`, now counts as part of the code that `#include`s it, directly or through other headers. Before, calls from that code into it were dropped, so protobuf's conformance suites' `suite_.ReportFailure(…)` and `RunValidInputTest(…)` calls, and its unit tests' calls to the shared `TestUtil` helpers, linked to nothing. A test file that only tests include is still kept out of reach of the rest of your code. Re-index C and C++ projects after upgrading. diff --git a/__tests__/kernel-ruby-parity.test.ts b/__tests__/kernel-ruby-parity.test.ts index 70390e9925..9dcf00c517 100644 --- a/__tests__/kernel-ruby-parity.test.ts +++ b/__tests__/kernel-ruby-parity.test.ts @@ -107,6 +107,47 @@ describe.skipIf(!kernelBuilt)('kernel Ruby extraction parity', () => { assertParity('fixtures/torture.rb (crlf)', crlf, 30); }); + // A receiver written with its namespace (`Billing::ChargeCard.call`, `::Foo.call`) + // is a constant read, as a bare `ChargeCard.call`'s is (#2503): both backends + // emit the same `references` ref for it, under the receiver as written, at + // the receiver. A path through a value (`obj::Thing`, `self.class::LIMIT`) + // names no constant the index can know, and `.new` stays an instantiation. + it('namespaced constant receivers', () => { + const src = [ + 'class Callers', + ' def a', + ' Billing::ChargeCard.call(amount: 1)', + ' ::Billing::ChargeCard.call', + ' ::Foo.call', + ' Users::Create&.call', + ' A::B::C.run!', + ' Config::DEFAULTS.fetch(:x)', + ' Foo::Bar::baz', + ' obj::Thing.go', + ' self.class::LIMIT.times { }', + ' Billing::ChargeCard.new(amount: 3)', + ' end', + 'end', + '', + ].join('\n'); + assertParity('src/callers.rb', src); + process.env.CODEGRAPH_KERNEL = '0'; + const refs = extractFromSource('src/callers.rb', src, 'ruby').unresolvedReferences + .filter((r) => r.referenceKind === 'references' || r.referenceKind === 'instantiates') + .map((r) => `${r.referenceKind} ${r.referenceName} ${r.line}:${r.column}`); + delete process.env.CODEGRAPH_KERNEL; + expect(refs.sort()).toEqual([ + 'instantiates ChargeCard 12:4', + 'references ::Billing::ChargeCard 4:4', + 'references ::Foo 5:4', + 'references A::B::C 7:4', + 'references Billing::ChargeCard 3:4', + 'references Config::DEFAULTS 8:4', + 'references Foo::Bar 9:4', + 'references Users::Create 6:4', + ]); + }); + it('mixin implements refs carry filePath through the v2 ref-flag wire path', () => { const src = 'class Widget\n include Comparable\nend\n'; process.env.CODEGRAPH_KERNEL_LANGS = 'all'; diff --git a/__tests__/ruby-namespaced-receiver.test.ts b/__tests__/ruby-namespaced-receiver.test.ts new file mode 100644 index 0000000000..e62c0ca03b --- /dev/null +++ b/__tests__/ruby-namespaced-receiver.test.ts @@ -0,0 +1,312 @@ +/** + * A Ruby call on a constant written with its namespace — + * `Billing::ChargeCard.call(…)`, `::Foo.call` — depends on the class it names + * and calls that class's method, as `ChargeCard.call(…)` already did (#2503). + * The constant is the one Ruby's lookup finds from where it is written: the + * nearest namespace around the call that declares it, then the top level; a + * name written from the top level (`::Foo`) is looked up there only. A + * constant the project doesn't declare is a library's (`ActiveRecord::Base`) + * and links nothing, never a project class or method sharing its last name. + * + * Runs on the native kernel (skipped when none is built) and the wasm + * extractor. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { getKernel } from '../src/extraction/kernel/loader'; + +const FILES: Record = { + // The issue's three files. + 'app/services/billing/charge_card.rb': `module Billing + class ChargeCard + def self.call(**kw) = new(**kw).call + def initialize(amount:) = @amount = amount + def call = @amount + end +end +`, + 'app/services/checkout/complete.rb': `module Checkout + class Complete + def call + Billing::ChargeCard.call(amount: 1) + end + end +end +`, + 'app/services/billing/refund.rb': `module Billing + class Refund + def call + ChargeCard.call(amount: 2) + end + end +end +`, + // One class name in two namespaces: the caller names which one it means. + 'app/billing/create.rb': `module Billing + class Create + def self.call = 1 + end +end +`, + 'app/users/create.rb': `module Users + class Create + def self.call = 2 + end +end +`, + 'app/foo.rb': `class Foo + def self.call = 3 +end +`, + 'app/svc/foo.rb': `module Svc + class Foo + def self.call = 4 + end +end +`, + 'app/svc/runner.rb': `module Svc + class Runner + def billing = Billing::Create.call + def users = Users::Create.call + def users_safe = Users::Create&.call + def rooted = ::Billing::Create.call + def rooted_top = ::Foo.call + end +end +`, + // The same relative name under two API versions: the one around the call. + 'app/api/v1/users/create.rb': `module Api + module V1 + module Users + class Create + def self.call = 5 + end + end + end +end +`, + 'app/api/v2/users/create.rb': `module Api + module V2 + module Users + class Create + def self.call = 6 + end + end + end +end +`, + 'app/api/v2/users_controller.rb': `module Api + module V2 + class UsersController + def create + Users::Create.call + end + end + end +end +`, + // A library constant whose last name a project class shares. + 'app/base.rb': `class Base + def self.transaction = 7 +end +`, + 'app/models/order.rb': `class Order + def save! + ActiveRecord::Base.transaction { } + Stripe::Charge.create(amount: 1) + end +end +`, + // A test's stand-in for a library class, which production code never reaches. + 'spec/support/stripe.rb': `module Stripe + class Charge + def self.create(**) = 9 + end +end +`, + 'spec/models/order_spec.rb': `class OrderSpec + def test_charge + Stripe::Charge.create(amount: 1) + end +end +`, + // A module's method, a class method a superclass declares, a class declared + // with a compact name, a method the class doesn't declare, a constant value. + 'app/billing/helpers.rb': `module Billing + module Helpers + def self.fmt(x) = x + end + class Document + def self.build = new + end + class Invoice < Document + end + RATES = { usd: 1 } +end +`, + 'app/billing/statement.rb': `class Billing::Statement + def self.render = 8 +end +`, + 'app/billing/query.rb': `class Query + def where(x) = x +end +`, + 'app/billing/report.rb': `class Report + def run + Billing::Helpers.fmt(1) + Billing::Invoice.build + Billing::Statement.render + Billing::ChargeCard.where(amount: 1) + Billing::RATES.fetch(:usd) + end +end +`, +}; + +let root = ''; +let cg: CodeGraph | undefined; +let savedKernel: string | undefined; + +afterEach(() => { + cg?.close(); + cg = undefined; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = ''; + if (savedKernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = savedKernel; +}); + +function write(files: Record): void { + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } +} + +async function index(backend: string, files: Record = FILES): Promise { + savedKernel = process.env.CODEGRAPH_KERNEL; + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-ruby-nsrecv-')); + write(files); + cg = await CodeGraph.init(root, { index: true }); + return cg; +} + +/** ` :` of every `calls`/`references` edge written on `line` of `file`. */ +function edgesAt(graph: CodeGraph, file: string, line: number): string[] { + const ids = graph.getNodesInFile(file).map((n) => n.id); + return graph + .getOutgoingEdgesFrom(ids) + .filter((e) => (e.kind === 'calls' || e.kind === 'references') && e.line === line) + .map((e) => { + const t = graph.getNode(e.target)!; + return `${e.kind} ${t.qualifiedName}:${t.startLine}`; + }) + .sort(); +} + +/** `:` of every `references` edge into the class or module `qualifiedName`. */ +function referencedFrom(graph: CodeGraph, qualifiedName: string): string[] { + const target = graph.getNodesByQualifiedName(qualifiedName).find((n) => n.kind === 'class' || n.kind === 'module')!; + return graph + .getIncomingEdgesTo([target.id]) + .filter((e) => e.kind === 'references') + .map((e) => `${graph.getNode(e.source)!.filePath}:${e.line}`) + .sort(); +} + +const backends = [ + ['kernel', getKernel() === null], + ['wasm', false], +] as const; + +describe.each(backends)('Ruby call on a namespaced constant (%s)', (backend, skip) => { + it.skipIf(skip)('links the caller to the class, as a bare constant receiver does', async () => { + const graph = await index(backend); + expect(referencedFrom(graph, 'Billing::ChargeCard')).toEqual([ + 'app/billing/report.rb:6', + 'app/services/billing/refund.rb:4', + 'app/services/checkout/complete.rb:4', + ]); + }); + + it.skipIf(skip)("calls the class's method", async () => { + const graph = await index(backend); + expect(edgesAt(graph, 'app/services/checkout/complete.rb', 4)).toEqual([ + 'calls Billing::ChargeCard::call:3', + 'references Billing::ChargeCard:2', + ]); + }); + + it.skipIf(skip)('reaches the class of the namespace the caller names', async () => { + const graph = await index(backend); + const runner = 'app/svc/runner.rb'; + expect(edgesAt(graph, runner, 3)).toEqual(['calls Billing::Create::call:3', 'references Billing::Create:2']); + expect(edgesAt(graph, runner, 4)).toEqual(['calls Users::Create::call:3', 'references Users::Create:2']); + expect(edgesAt(graph, runner, 5)).toEqual(['calls Users::Create::call:3', 'references Users::Create:2']); + }); + + it.skipIf(skip)('looks a ::-rooted name up at the top level only', async () => { + const graph = await index(backend); + const runner = 'app/svc/runner.rb'; + expect(edgesAt(graph, runner, 6)).toEqual(['calls Billing::Create::call:3', 'references Billing::Create:2']); + // `::Foo` inside `module Svc` is the top-level Foo, not Svc::Foo. + expect(edgesAt(graph, runner, 7)).toEqual(['calls Foo::call:2', 'references Foo:1']); + }); + + it.skipIf(skip)('looks a relative name up in the namespaces around the call first', async () => { + const graph = await index(backend); + expect(edgesAt(graph, 'app/api/v2/users_controller.rb', 5)).toEqual([ + 'calls Api::V2::Users::Create::call:5', + 'references Api::V2::Users::Create:4', + ]); + }); + + it.skipIf(skip)('links nothing for a constant the project does not declare', async () => { + const graph = await index(backend); + expect(edgesAt(graph, 'app/models/order.rb', 3)).toEqual([]); + expect(referencedFrom(graph, 'Base')).toEqual([]); + }); + + it.skipIf(skip)('never links production code to a class only a test declares', async () => { + const graph = await index(backend); + expect(edgesAt(graph, 'app/models/order.rb', 4)).toEqual([]); + expect(edgesAt(graph, 'spec/models/order_spec.rb', 3)).toEqual([ + 'calls Stripe::Charge::create:3', + 'references Stripe::Charge:2', + ]); + }); + + it.skipIf(skip)('calls module methods, inherited class methods and compact-named classes, and guesses no method', async () => { + const graph = await index(backend); + const report = 'app/billing/report.rb'; + expect(edgesAt(graph, report, 3)).toEqual(['calls Billing::Helpers::fmt:3', 'references Billing::Helpers:2']); + expect(edgesAt(graph, report, 4)).toEqual(['calls Billing::Document::build:6', 'references Billing::Invoice:8']); + expect(edgesAt(graph, report, 5)).toEqual(['calls Billing::Statement::render:2', 'references Billing::Statement:1']); + // ChargeCard declares no `where`; Query's is not it. + expect(edgesAt(graph, report, 6)).toEqual(['references Billing::ChargeCard:2']); + // A constant holding a value: the call is the value's. + expect(edgesAt(graph, report, 7)).toEqual(['references Billing::RATES:10']); + }); +}); + +describe('Ruby call on a namespaced constant, after a sync', () => { + it('links the calls once a sync adds the class they name', async () => { + const { 'app/services/billing/charge_card.rb': chargeCard, ...rest } = FILES; + const graph = await index('default', rest); + expect(edgesAt(graph, 'app/services/checkout/complete.rb', 4)).toEqual([]); + + write({ 'app/services/billing/charge_card.rb': chargeCard! }); + expect((await graph.sync()).filesAdded).toBe(1); + // complete.rb did not change: only the retry of its parked refs links it. + expect(edgesAt(graph, 'app/services/checkout/complete.rb', 4)).toEqual([ + 'calls Billing::ChargeCard::call:3', + 'references Billing::ChargeCard:2', + ]); + }, 60_000); +}); diff --git a/codegraph-kernel/src/ruby.rs b/codegraph-kernel/src/ruby.rs index 3e4e33cb08..101a3271ff 100644 --- a/codegraph-kernel/src/ruby.rs +++ b/codegraph-kernel/src/ruby.rs @@ -820,10 +820,12 @@ impl<'t> Walker<'t> { }; self.push_ref(caller, &callee, calls_kind, line, col); - // Capitalized constant receiver (`Klass.static_call`, `RETRY_MAX.times`) - // → an ADDITIONAL references ref at the RECEIVER's position. - // (scope_resolution receivers get none — type ≠ constant.) - if !skip && receiver.kind() == "constant" { + // Capitalized constant receiver (`Klass.static_call`, `RETRY_MAX.times`), + // or one written with its namespace (`Billing::ChargeCard.call`, + // `::Foo.call` — #2503) → an ADDITIONAL references ref at the + // RECEIVER's position, under the receiver as written. A path through + // a value (`obj::Thing`, `self.class::LIMIT`) names no constant. + if !skip && is_constant_path(receiver) { self.push_ref_at( caller, &receiver_name.to_string(), @@ -1121,6 +1123,19 @@ impl<'t> Walker<'t> { } } +/// A constant, or a `scope_resolution` path of constants (`A::B::C`, a +/// top-level `::A` / `::A::B`): what Ruby reads as a constant (isRubyConstantPath). +fn is_constant_path(node: Node) -> bool { + match node.kind() { + "constant" => true, + "scope_resolution" => { + node.child_by_field_name("name").map_or(false, |n| n.kind() == "constant") + && node.child_by_field_name("scope").map_or(true, is_constant_path) + } + _ => false, + } +} + /// The Ruby `call` node whose argument_list (or keyword pair) contains `node` /// — nearest `call` ancestor within 4 parent hops (function-ref.ts:837). fn ruby_enclosing_call(node: Node) -> Option { diff --git a/docs/design/ruby-kernel-port-checklist.md b/docs/design/ruby-kernel-port-checklist.md index 3a02bac069..120e40ec65 100644 --- a/docs/design/ruby-kernel-port-checklist.md +++ b/docs/design/ruby-kernel-port-checklist.md @@ -429,13 +429,17 @@ field_expression machinery, which ruby therefore NEVER runs. Steps: normalized to `()`**, unlike java/php chain encodings); literal receivers are NOT filtered (`"literal".upcase` → `"literal".upcase`, `5.times` → `5.times` — unresolvable noise, PRESERVE). -6. Receiver node TYPE exactly `constant` (and not skipped) → ADDITIONAL - `references` ref {name: receiverName, line/col: **the RECEIVER's** - position} — `Klass.static_call` emits calls `Klass.static_call` + - references `Klass`; fires for VALUE constants too (`RETRY_MAX.times {…}` - → calls `RETRY_MAX.times` + references `RETRY_MAX` — pinned). A - `scope_resolution` receiver (`Foo::Bar.baz`) gets NO references ref - (type ≠ constant). PRESERVE. +6. Receiver a CONSTANT PATH (and not skipped) → ADDITIONAL `references` + ref {name: receiverName, line/col: **the RECEIVER's** position} — + `Klass.static_call` emits calls `Klass.static_call` + references `Klass`; + fires for VALUE constants too (`RETRY_MAX.times {…}` → calls + `RETRY_MAX.times` + references `RETRY_MAX` — pinned). A constant path is + a `constant`, or a `scope_resolution` whose `name` is a `constant` and + whose `scope` is absent or itself a constant path (#2503): `Foo::Bar.baz` + → references `Foo::Bar`, `::Foo.baz` → references `::Foo` (the leading + `::` kept — the resolver reads it as "from the top level"). A path + through a value (`obj::Thing.go`, `self.class::LIMIT.times`) gets NONE. + Both engines: `is_constant_path` / `isRubyConstantPath`. 7. return — inner receiver calls are ALSO visited afterwards (the body walker recurses children at 5277 — extractCall does not consume its subtree), so `chained.first_call.second_call` emits BOTH diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 162714c979..caf11234a4 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -305,6 +305,18 @@ function swiftPropertyInfo( return { nameNode: firstSimpleIdentifier(pattern), isLet, isComputed }; } +/** + * A Ruby constant, or a `scope_resolution` path of constants (`A::B::C`, a + * top-level `::A` / `::A::B`): what Ruby reads as a constant. A path through + * a value (`obj::Thing`, `self.class::LIMIT`) is not one. + */ +function isRubyConstantPath(node: SyntaxNode): boolean { + if (node.type === 'constant') return true; + if (node.type !== 'scope_resolution') return false; + const scope = getChildByField(node, 'scope'); + return getChildByField(node, 'name')?.type === 'constant' && (!scope || isRubyConstantPath(scope)); +} + /** True when `node` is (transitively) inside a C function body — i.e. a local, * not a file/namespace-scope declaration. Walks the parent chain to the root. */ function hasFunctionAncestor(node: SyntaxNode): boolean { @@ -5444,7 +5456,10 @@ export class TreeSitterExtractor { // — is itself a dependency on that constant; emit a `references` ref so a // class used only via its class methods still records a dependent (the edge // the old receiver-only callee happened to provide, now made explicit). - if (!skip && receiverNode.type === 'constant') { + // So is one written with its namespace, `Billing::ChargeCard.call` or + // `::Foo.call` (#2503), under the receiver as written; a path through a + // value (`obj::Thing`, `self.class::LIMIT`) names no constant. + if (!skip && isRubyConstantPath(receiverNode)) { this.unresolvedReferences.push({ fromNodeId: callerId, referenceName: receiverName, diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 5e3e46ea0f..c568b3deb0 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -41,6 +41,7 @@ import { clearAmbientModuleMemos, ambientModuleDeclarations } from './ambient-mo import { matchShopifyThemeFile } from './shopify-themes'; import { isJsSuperCallLanguage, isJsSuperCallSite, matchJsSuperCall } from './js-super-calls'; import { CLASS_EXPRESSION_REF_LANGUAGES, isCommonJsExportedClass, isJsClassExpression, writtenIn, type ClassExpressionIndex } from './js-class-expressions'; +import { matchRubyConstantPathRef } from './ruby-constant-paths'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveTsQualifiedTypeRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isDartImportRef, isLuaRequireRef, isJsPathImportRef, isBoundToOutOfRepoImport, reachesAmbientModule, importBindingTail, clearImportResolverMemos, resolveImportPath, isExternalImport, ESM_IMPORT_LANGUAGES, jsModuleFile, pythonRootFingerprint, pythonReopenScope, pythonPackageImporters, moduleDefaultExport, isJsFamilyFile, isModuleBindingName } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; import { resolveAliasBinding } from './alias-binding'; @@ -1574,6 +1575,15 @@ export class ReferenceResolver { const phpQualified = resolvePhpQualifiedClassRef(ref, this.context); if (phpQualified !== undefined) return this.gateLanguage(phpQualified, ref); + // A Ruby constant written with its namespace (`Billing::ChargeCard`, + // `::Foo`) and a call on one (`Billing::ChargeCard.call`) are what Ruby's + // constant lookup finds from where they are written, or nothing the + // project declares (see ./ruby-constant-paths). The pre-filter below + // drops a `::`-rooted name, and the strategies past it would take a + // namesake by its last name. + const rubyPath = matchRubyConstantPathRef(ref, this.context); + if (rubyPath !== undefined) return this.gateLanguage(rubyPath, ref); + // Fast pre-filter: skip if no symbol with this name exists anywhere // AND the name doesn't match a local import. The import escape is // necessary because re-export rename chains (`import { login } diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index aa3c74d84f..e4098cd842 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -7805,7 +7805,7 @@ function isRubyMethodInScope(method: Node, ref: UnresolvedRef, context: Resoluti } /** A Ruby class's qualified name, those of its superclasses, and of every module mixed into any of them. */ -function rubyAncestry(qn: string, context: ResolutionContext): Set { +export function rubyAncestry(qn: string, context: ResolutionContext): Set { let memo = RUBY_ANCESTRY.get(context); if (!memo) RUBY_ANCESTRY.set(context, (memo = new Map())); const hit = memo.get(qn); diff --git a/src/resolution/ruby-constant-paths.ts b/src/resolution/ruby-constant-paths.ts new file mode 100644 index 0000000000..b77e34fa24 --- /dev/null +++ b/src/resolution/ruby-constant-paths.ts @@ -0,0 +1,118 @@ +/** + * A Ruby constant written with its namespace — `Billing::ChargeCard`, or from + * the top level, `::Foo` and `::Billing::ChargeCard` — is the class, module + * or constant Ruby's lookup finds from where it is written (#2503): the + * nearest namespace around the reference that declares it, then the top + * level; one written from the top level is looked up there only. The + * extractor records such a call receiver as a `references` ref beside the + * call, as it does a bare one (`ChargeCard.call`), and the call itself as + * `Billing::ChargeCard.call`: a method of that class or module, its own or + * one its ancestry declares. + * + * Both resolve here alone. A path the project doesn't declare is a library's + * (`ActiveRecord::Base.transaction`), where the strategies past this one + * would take a project class or method sharing its last name; and a class + * the path names without the method (`Billing::ChargeCard.where`) has it from + * somewhere the index can't see, never from another class that declares one. + * Production code never names a class a test declares. A bare constant + * (`ChargeCard`) is left to the name strategies, as before. + */ +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; +import { isTestPath } from '../search/query-utils'; +import { rubyAncestry } from './name-matcher'; + +/** `A::B`, `::A`, `::A::B`: a constant path. Group 1 is a leading `::`. */ +const CONSTANT_PATH = /^(::)?([A-Z]\w*(?:::[A-Z]\w*)*)$/; +/** `A::B.m`, `::A.m`: a call on one. Group 3 is the method. */ +const CONSTANT_PATH_CALL = /^(::)?([A-Z]\w*(?:::[A-Z]\w*)*)\.([A-Za-z_]\w*[?!=]?)$/; + +/** What a Ruby constant is in the index: a class, a module, or a value (`RATES = {…}` is a `variable`). */ +const NAMESPACE_KINDS: ReadonlySet = new Set(['class', 'module']); +const VALUE_KINDS: ReadonlySet = new Set(['variable', 'constant']); +const CONSTANT_KINDS: ReadonlySet = new Set([...NAMESPACE_KINDS, ...VALUE_KINDS]); + +/** + * What a Ruby `references` ref written as a constant path names, or the + * method a call on one reaches (see the file comment). Undefined for any + * other ref, which the other strategies resolve; null when the project + * declares no such constant, or the class or module it names has no method + * of the name. + */ +export function matchRubyConstantPathRef(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + if (ref.language !== 'ruby') return undefined; + if (ref.referenceKind === 'references') { + const m = CONSTANT_PATH.exec(ref.referenceName); + if (!m || !isNamespaced(m)) return undefined; + const target = constantsNamed(m[2]!, m[1] !== undefined, ref, context)[0]; + return target ? { original: ref, targetNodeId: target.id, confidence: 0.9, resolvedBy: 'qualified-name' } : null; + } + if (ref.referenceKind === 'calls') { + const m = CONSTANT_PATH_CALL.exec(ref.referenceName); + if (!m || !isNamespaced(m)) return undefined; + const owner = constantsNamed(m[2]!, m[1] !== undefined, ref, context)[0]; + const method = owner && NAMESPACE_KINDS.has(owner.kind) ? methodOf(owner.qualifiedName, m[3]!, ref, context) : null; + return method ? { original: ref, targetNodeId: method.node.id, confidence: method.own ? 0.85 : 0.8, resolvedBy: 'qualified-name' } : null; + } + return undefined; +} + +/** Whether a matched path is written with a `::`, which a bare `Foo` is not. */ +function isNamespaced(m: RegExpExecArray): boolean { + return m[1] !== undefined || m[2]!.includes('::'); +} + +/** + * The declarations of the constant `path` names where `ref` is written: those + * of the first of `::`, `::`, … `` the + * project declares, the scope being the class or module around the + * reference — only `` itself when it is written from the top level. + * A class or module comes before a value of the name, then the reference's + * own file, then production code. + */ +function constantsNamed(path: string, absolute: boolean, ref: UnresolvedRef, context: ResolutionContext): Node[] { + const production = !isTestPath(ref.filePath); + let prefix = absolute ? '' : lexicalScope(ref, context); + for (;;) { + const qn = prefix ? `${prefix}::${path}` : path; + const found = context.getNodesByQualifiedName(qn).filter((n) => + n.language === 'ruby' && CONSTANT_KINDS.has(n.kind) && !(production && isTestPath(n.filePath))); + if (found.length > 0) return ordered(found, ref); + if (!prefix) return []; + const cut = prefix.lastIndexOf('::'); + prefix = cut >= 0 ? prefix.slice(0, cut) : ''; + } +} + +/** The qualified name of the innermost class or module around `ref`, or '' at the top level of its file. */ +function lexicalScope(ref: UnresolvedRef, context: ResolutionContext): string { + let inner: Node | undefined; + for (const n of context.getNodesInFile(ref.filePath)) { + if (!NAMESPACE_KINDS.has(n.kind) || n.language !== 'ruby' || n.startLine > ref.line || n.endLine < ref.line) continue; + // Of the ranges around a point, the innermost starts last. + if (!inner || n.startLine > inner.startLine || (n.startLine === inner.startLine && n.startColumn > inner.startColumn)) inner = n; + } + return inner?.qualifiedName ?? ''; +} + +/** + * The method `name` of the class or module `owner`: its own, else the first + * of its superclasses and mixed-in modules that declares one (rubyAncestry). + * Production code never reaches one a test declares. + */ +function methodOf(owner: string, name: string, ref: UnresolvedRef, context: ResolutionContext): { node: Node; own: boolean } | null { + const production = !isTestPath(ref.filePath); + for (const qn of rubyAncestry(owner, context)) { + const methods = context.getNodesByQualifiedName(`${qn}::${name}`).filter((n) => + n.kind === 'method' && n.language === 'ruby' && !(production && isTestPath(n.filePath))); + if (methods.length > 0) return { node: ordered(methods, ref)[0]!, own: qn === owner }; + } + return null; +} + +/** Anything before a value, then the reference's own file, then production code; the index's order within each. */ +function ordered(nodes: Node[], ref: UnresolvedRef): Node[] { + const rank = (n: Node): number => + (VALUE_KINDS.has(n.kind) ? 4 : 0) + (n.filePath === ref.filePath ? 0 : 2) + (isTestPath(n.filePath) ? 1 : 0); + return nodes.length < 2 ? nodes : [...nodes].sort((a, b) => rank(a) - rank(b)); +} From 8fdc8a1efb378f68c10a48d4b002d3cf6bb3776a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 06:25:23 +0000 Subject: [PATCH 60/95] fix(rust): calls on field chains, struct literals and outside-type paths link by type (#2831) Issue #2614 reported four Rust calls that linked to a same-named project method. Case 1 (a slice's contains) was fixed on main by the typed-receiver work. The other three still reproduced on main in both engines: vis.map.get("a") on a HashMap field went to Ext::get, J::from(i) under `use serde_json::Value as J` to a local `impl From for Fail`, and Xyz { x: 1.0 }.to_val() to Stat::to_val. Root cause. The extractor keeps one receiver level on a call, so a deeper field chain (vis.map.get, self.a.b.m, self.0.m) or a struct literal reaches the resolver as the bare method name, and the typed-receiver reading only ran for x.m and self.field.m refs. The name strategies then took the one project get, or one of the two to_val impls. J::from matched no qualified name, J names no project type, and a single capital letter is read as a possible generic parameter, so Strategy 3 took the crate's one from. The same family: under an unrenamed `use serde_json::Value`, Value::default() linked by qualified name to a project Value's impl. Fix. - matchRustWrittenReceiverCall, an exclusive hook beside the as_mut one: for a bare-named call, read the receiver from the text at the ref's column. Only a field chain rooted at a local, a parameter or self, or a struct literal (alone or in parentheses) is read; it is typed with rustValueType and the call resolved with rustTypedMethodCall. Any other receiver, an unreadable one, or a type no node names (one a macro declares) is left to the name strategies. A provided method of a project trait the type derives counts (axum-extra's #[derive(TypedPath)]). - rustFieldValue reads an element of a tuple-typed field (self.pair.0). - isRustPathCallReachable in gateRustScope: a call through a type path whose type the file's use, or the written path, names from std or a dependency reaches only the project's impl of a trait for that type, or for a trait's path the project's impls of that trait. A path dependency the project holds is the project's. Validation. - New __tests__/rust-written-receivers.test.ts (12 tests: the issue's crate verbatim, chain/literal/derive/path variants, guards): 9 fail on main's sources, all pass with the fix. The 27 Rust test files plus resolution and cross-language resolution pass. - A/B against main 5b47bc49 (kernel on), changed call sites triaged as removed wrong / added right / wrong to right / same target now by type: ripgrep 30 / 18 / 10 / 37 and tokio 281 / 58 / 54 / 141 (every site read), axum 18 / 7 / 0 / 7 (every site read), rust-analyzer without its parser test data (951 files) 737 / 186 / 434 / 843 (every path removal checked by its use; every group sampled). No right link lost. Two wrong links added on rust-analyzer: a block-scoped glob (`const _: () = { use syntax::ast::*; }`) is read file-wide by the existing glob lookup, so a `String` field there reads as ast::String; typed locals already misread it the same way on main. Nodes identical on every repo; controls gin and express byte-identical. - Kernel parity (Rust): 1226/1226 files on ripgrep, tokio and axum. Resolver-only change, so the kernel needs no mirror. - Cost: ripgrep median 6.7 s to 6.5 s, tokio 20.2 s to 20.8 s (interleaved, median of 3, noisy box); a CPU profile puts about 0.3 s of tokio's 57 s busy CPU in the new code and the typed reading it calls. Fixes #2614 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/rust-written-receivers.test.ts | 474 +++++++++++++++++++++++ src/resolution/index.ts | 12 +- src/resolution/name-matcher.ts | 292 ++++++++++++++ 4 files changed, 775 insertions(+), 4 deletions(-) create mode 100644 __tests__/rust-written-receivers.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0fbd831ffb..9ec68ad34a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -313,6 +313,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Rust, a variable that holds what a method call returns, like `let server = builder.port(80).build();`, `let settings = config.load()?;` or `let conn = client.connect().await;`, now has the type that method is declared to return, so calls on the variable link to that type's methods. A generic method's type comes from the call's turbofish, like `arg.get::()`, and a call on a type that implements one trait twice, like `Encoder` and `Encoder<&[u8]>`, links to the implementation that takes the argument given. Before, such calls were linked by name and could reach another type's method of the same name. Re-index Rust projects after upgrading. - In Rust, a `for` loop's variable now has the type of an item of what the loop goes over, like `for item in &self.items` over a `Vec`, `for (key, value) in &map` over a `HashMap`, or `for sc in cmd.get_subcommands()` over the `impl Iterator` a method returns, and each name in a tuple pattern like `let (tx, rx) = mpsc::channel(8);` has the type of its part of the tuple, so calls on them link to that type's methods. Before, such calls were linked by name: tokio's `tx.send(…)` calls on channel senders went to an internal type's `send`. Re-index Rust projects after upgrading. - In Rust, a call on a field of `self`, like `self.store.save()` or `self.inner.lock()`, now links to the method of the type the struct declares for that field, also when the field's type is a parameter the `impl` bounds by a trait (`impl Worker`), a trait object, or held in an `Arc` or a `Box`. A field whose type comes from the standard library or a dependency, like `jar: cookie::CookieJar` or `inner: std::sync::Mutex<…>`, no longer links to the project's own type of the same name, which often made a wrapper's method look like it called itself. Re-index Rust projects after upgrading. +- In Rust, a method called through a chain of fields or on a struct literal, like `vis.map.get("a")`, `self.inner.config.port()` or `Xyz { x: 1.0 }.to_val()`, now links to the method of the type the last field is declared with, or of the literal's own type. A call through a type from the standard library or a dependency, like `J::from(i)` after `use serde_json::Value as J` or `Value::default()` after `use serde_json::Value`, no longer links to a project function of the same name. Before, such calls were linked by the method's name alone, so a `HashMap` field's `get` reached an unrelated project `get` and a struct literal's `to_val()` reached another type's. Re-index Rust projects after upgrading. Thanks @Egrn for the report and @danusha2345. (#2614) - In Rust, a call written with a turbofish, like `matches.get_one::("name")`, `parse_config::()` or `Pool::::new()`, now links to the function it calls. Before, the type arguments stayed in the call's name, so no such call was linked. Re-index Rust projects after upgrading. - In Rust, a call through a value's declared type, like `reader.as_mut().poll_read(cx)` on `reader: Pin<&mut R>`, now reads the declaration in scope at the call. Before, a `let` of the same name inside an earlier block, or a struct literal's field of that name written above the call, could be taken for it. Re-index Rust projects after upgrading. - In C++, every name a declaration lists now has the declaration's type, like `strings_found` in google/breakpad's `ByteBuffer symbols_found, strings_found;` or `data_` in a class's `Table *index_, *data_;`, so calls on it link to that class's methods. Before, only the first name had its type read, and a call on another name linked to nothing or to the class of an earlier variable with the same name. diff --git a/__tests__/rust-written-receivers.test.ts b/__tests__/rust-written-receivers.test.ts new file mode 100644 index 0000000000..679f7ff400 --- /dev/null +++ b/__tests__/rust-written-receivers.test.ts @@ -0,0 +1,474 @@ +/** + * A Rust call resolves by the type its receiver is written with, where the + * source writes it (#2614): + * + * - a field chain, which the extractor records by the bare method name — + * `vis.map.get("a")` on a `HashMap` field, `self.inner.cfg.port()` — is a + * call on the type the struct declares for the last field; + * - a struct literal, `Xyz { x: 1.0 }.to_val()`, is a call on its own type; + * - a path through a type from std or a dependency — `J::from(i)` under + * `use serde_json::Value as J`, `serde_json::Value::from(i)` — reaches only + * the project's own impls of a trait for that type. + * + * By the name alone, the reporter's `vis.map.get("a")` went to `Ext::get` (an + * associated function, at that), `Xyz { … }.to_val()` to `Stat::to_val` (the + * same trait's other impl) and `J::from(i)` to `impl From for Fail`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +const graphs: Record<'issue' | 'variants' | 'keeps', CodeGraph> = {} as never; + +async function index(name: keyof typeof graphs, files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), `cg-rust-written-${name}-`)); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + graphs[name] = await CodeGraph.init(root, { index: true }); +} + +const manifest = (name: string): string => + `[package]\nname = "${name}"\nversion = "0.1.0"\nedition = "2021"\n\n[dependencies]\nserde_json = "1"\n`; + +beforeAll(async () => { + // The issue's crate, verbatim. + await index('issue', { + 'Cargo.toml': manifest('repro'), + 'src/main.rs': [ + 'use std::collections::HashMap;', + 'use serde_json::Value as J;', + '', + 'pub enum Val {', + ' Null,', + ' Int(i64),', + ' List(Vec),', + '}', + '', + 'pub struct Obj {', + ' keys: Vec,', + '}', + '', + 'impl Obj {', + ' pub fn contains(&self, k: &str) -> bool {', + ' self.keys.iter().any(|x| x == k)', + ' }', + '}', + '', + 'pub struct Ext;', + '', + 'impl Ext {', + ' pub fn get(name: &str) -> Ext {', + ' let _ = name;', + ' Ext', + ' }', + '}', + '', + 'pub struct Fail;', + '', + 'impl From for Fail {', + ' fn from(_: std::io::Error) -> Fail {', + ' Fail', + ' }', + '}', + '', + 'pub struct Xyz {', + ' x: f64,', + '}', + '', + 'pub struct Stat {', + ' n: f64,', + '}', + '', + 'pub trait ToVal {', + ' fn to_val(&self) -> Val;', + '}', + '', + 'impl ToVal for Xyz {', + ' fn to_val(&self) -> Val {', + ' Val::Int(self.x as i64)', + ' }', + '}', + '', + 'impl ToVal for Stat {', + ' fn to_val(&self) -> Val {', + ' Val::Int(self.n as i64)', + ' }', + '}', + '', + '// 1. slice::contains, not Obj::contains', + 'fn slice_has(names: &[&str], k: &str) -> bool {', + ' names.contains(&k)', + '}', + '', + '// 2. HashMap::get on a struct field, not Ext::get', + 'pub struct Vis {', + ' map: HashMap,', + '}', + '', + 'fn map_get(vis: &Vis) -> Option<&i32> {', + ' vis.map.get("a")', + '}', + '', + '// 3. serde_json::Value::from via alias J, not Fail::from', + 'fn to_json(i: i64) -> J {', + ' J::from(i)', + '}', + '', + '// 4. method on struct literal: Xyz::to_val, not Stat::to_val', + 'fn centroid() -> Val {', + ' Xyz { x: 1.0 }.to_val()', + '}', + '', + 'fn main() {', + ' let o = Obj { keys: vec![] };', + ' let _ = o.contains("a");', + ' let _ = Ext::get("x");', + ' let _ = Stat { n: 0.0 }.to_val();', + ' let _ = slice_has(&["a"], "a");', + ' let _ = map_get(&Vis { map: HashMap::new() });', + ' let _ = to_json(1);', + ' let _ = centroid();', + ' let _ = Val::List(vec![Val::Null]);', + '}', + '', + ].join('\n'), + }); + + // Field chains and struct literals of other shapes, beside namesakes of the + // methods they call; and paths through types from outside the project, with + // one `from` in the crate for a guess by the name to take. + await index('variants', { + 'Cargo.toml': manifest('variants'), + 'src/main.rs': 'mod chains;\nmod config;\nmod paths;\n\nfn main() {}\n', + 'src/config.rs': [ + 'pub struct Config {', + ' pub port: u16,', + '}', + '', + 'impl Config {', + ' pub fn load() -> Config {', + ' Config { port: 1 }', + ' }', + '', + ' pub fn port(&self) -> u16 {', + ' self.port', + ' }', + '}', + '', + '/// A project type named like `serde_json::Value`.', + 'pub struct Value;', + '', + 'impl Default for Value {', + ' fn default() -> Value {', + ' Value', + ' }', + '}', + '', + 'pub struct Fail;', + '', + 'impl From for Fail {', + ' fn from(_: std::io::Error) -> Fail {', + ' Fail', + ' }', + '}', + '', + ].join('\n'), + 'src/chains.rs': [ + 'use std::collections::HashMap;', + '', + 'use crate::config::Config;', + '', + 'pub struct Inner {', + ' pub map: HashMap,', + ' pub cfg: Config,', + '}', + '', + 'pub struct Outer {', + ' pub inner: Inner,', + ' pub pair: (Config, u8),', + '}', + '', + 'pub struct Wrap(pub Config);', + '', + '/// Namesakes of the methods the calls below make, beside them.', + 'pub struct Dial;', + '', + 'impl Dial {', + ' pub fn port(&self) -> u16 {', + ' 0', + ' }', + '', + ' pub fn get(&self, _k: &str) -> Option<&i32> {', + ' None', + ' }', + '}', + '', + 'pub enum Shape {', + ' Circle { r: f64 },', + '}', + '', + 'impl Shape {', + ' pub fn area(&self) -> f64 {', + ' 0.0', + ' }', + '}', + '', + 'pub struct Room {', + ' pub w: f64,', + '}', + '', + 'impl Room {', + ' pub fn area(&self) -> f64 {', + ' self.w', + ' }', + '', + ' pub fn fresh(&self) -> f64 {', + ' Self { w: 2.0 }.area()', + ' }', + '}', + '', + 'impl Outer {', + ' pub fn deep_std(&self) -> Option<&i32> {', + ' self.inner.map.get("a")', + ' }', + '', + ' pub fn deep_project(&self) -> u16 {', + ' self.inner.cfg.port()', + ' }', + '', + ' pub fn tuple_field(&self) -> u16 {', + ' self.pair.0.port()', + ' }', + '}', + '', + 'impl Wrap {', + ' pub fn tuple_self(&self) -> u16 {', + ' self.0.port()', + ' }', + '}', + '', + 'pub fn param_chain(o: &Outer) -> u16 {', + ' o.inner.cfg.port()', + '}', + '', + 'pub fn param_chain_std(o: &Outer) -> i32 {', + ' o.inner', + ' .map', + ' .get("b")', + ' .copied()', + ' .unwrap_or(0)', + '}', + '', + 'pub fn local_chain() -> u16 {', + ' let w = Wrap(Config::load());', + ' w.0.port()', + '}', + '', + 'pub fn literal() -> u16 {', + ' Config { port: 2 }.port()', + '}', + '', + 'pub fn literal_in_parens() -> u16 {', + ' (&Config { port: 3 }).port()', + '}', + '', + 'pub fn variant_literal() -> f64 {', + ' Shape::Circle { r: 1.0 }.area()', + '}', + '', + '/// A trait a derive implements (axum-extra\'s `#[derive(TypedPath)]`): the impl is the macro\'s.', + 'pub trait TypedPath {', + ' fn with_query_params(self, q: u8) -> u8', + ' where', + ' Self: Sized,', + ' {', + ' q', + ' }', + '}', + '', + '#[derive(TypedPath, Debug)]', + 'pub struct UsersShow {', + ' pub id: u32,', + '}', + '', + 'pub fn derived() -> u8 {', + ' UsersShow { id: 1 }.with_query_params(2)', + '}', + '', + ].join('\n'), + 'src/paths.rs': [ + 'use serde_json as json;', + 'use serde_json::Value;', + '', + 'pub fn written_outside(i: i64) -> serde_json::Value {', + ' serde_json::Value::from(i)', + '}', + '', + 'pub fn crate_alias(i: i64) -> json::Value {', + ' json::Value::from(i)', + '}', + '', + 'pub fn unrenamed_outside() -> Value {', + ' Value::default()', + '}', + '', + 'pub fn written_default() -> serde_json::Value {', + ' serde_json::Value::default()', + '}', + '', + ].join('\n'), + }); + + // The project's own impls on types from outside it, which those paths do reach. + await index('keeps', { + 'Cargo.toml': manifest('keeps'), + 'src/main.rs': [ + 'use std::io::Write as W;', + 'use std::path::PathBuf;', + '', + 'use serde_json::Value;', + '', + 'pub struct Fail;', + '', + 'impl From for PathBuf {', + ' fn from(_: Fail) -> PathBuf {', + ' PathBuf::new()', + ' }', + '}', + '', + 'pub trait Blank {', + ' fn blank() -> Self;', + '}', + '', + 'impl Blank for serde_json::Value {', + ' fn blank() -> Self {', + ' serde_json::Value::Null', + ' }', + '}', + '', + 'pub struct Sink;', + '', + 'impl std::io::Write for Sink {', + ' fn write(&mut self, buf: &[u8]) -> std::io::Result {', + ' Ok(buf.len())', + ' }', + '', + ' fn flush(&mut self) -> std::io::Result<()> {', + ' Ok(())', + ' }', + '}', + '', + 'fn to_path(f: Fail) -> PathBuf {', + ' PathBuf::from(f)', + '}', + '', + 'fn blank() -> Value {', + ' Value::blank()', + '}', + '', + 'fn drain(s: &mut Sink) {', + ' let _ = W::flush(s);', + '}', + '', + 'fn main() {', + ' let _ = to_path(Fail);', + ' let _ = blank();', + '}', + '', + ].join('\n'), + }); +}, 120_000); + +afterAll(() => { + for (const cg of Object.values(graphs)) cg?.close(); + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** The calls on `line` of `file`, as `qualifiedName resolvedBy`, sorted. */ +const callsAt = (name: keyof typeof graphs, file: string, line: number): string[] => { + const cg = graphs[name]; + const out: string[] = []; + for (const node of cg.getNodesInFile(file)) { + for (const edge of cg.getOutgoingEdges(node.id)) { + if (edge.kind !== 'calls' || edge.line !== line || edge.provenance === 'heuristic') continue; + out.push(`${cg.getNode(edge.target)!.qualifiedName} ${String(edge.metadata?.resolvedBy)}`); + } + } + return [...new Set(out)].sort(); +}; + +describe('the issue\'s crate (#2614)', () => { + it('a `HashMap` field\'s `get` is no project method', () => { + expect(callsAt('issue', 'src/main.rs', 72)).toEqual([]); + }); + + it('`J::from(i)` under `use serde_json::Value as J` is no project method', () => { + expect(callsAt('issue', 'src/main.rs', 77)).toEqual([]); + }); + + it('a struct literal calls its own type\'s method', () => { + expect(callsAt('issue', 'src/main.rs', 82)).toEqual(['Xyz::to_val instance-method']); + expect(callsAt('issue', 'src/main.rs', 89)).toEqual(['Stat::to_val instance-method']); + }); + + it('keeps what was already right', () => { + expect(callsAt('issue', 'src/main.rs', 63)).toEqual([]); + expect(callsAt('issue', 'src/main.rs', 87)).toEqual(['Obj::contains instance-method']); + expect(callsAt('issue', 'src/main.rs', 88)).toEqual(['Ext::get qualified-name']); + }); +}); + +describe('a Rust call on a field chain', () => { + it('of a std type is no project method', () => { + expect(callsAt('variants', 'src/chains.rs', 56)).toEqual([]); + expect(callsAt('variants', 'src/chains.rs', 79)).toEqual([]); + }); + + it('of a project type is that type\'s method, not a namesake beside the call', () => { + expect(callsAt('variants', 'src/chains.rs', 60)).toEqual(['Config::port instance-method']); + expect(callsAt('variants', 'src/chains.rs', 64)).toEqual(['Config::port instance-method']); + expect(callsAt('variants', 'src/chains.rs', 70)).toEqual(['Config::port instance-method']); + expect(callsAt('variants', 'src/chains.rs', 75)).toEqual(['Config::port instance-method']); + expect(callsAt('variants', 'src/chains.rs', 88)).toEqual(['Config::port instance-method']); + }); +}); + +describe('a Rust call on a struct literal', () => { + it('is its own type\'s method, in parentheses too', () => { + expect(callsAt('variants', 'src/chains.rs', 92)).toEqual(['Config::port instance-method']); + expect(callsAt('variants', 'src/chains.rs', 96)).toEqual(['Config::port instance-method']); + }); + + it('of an enum\'s variant is the enum\'s method, and `Self { … }` the impl type\'s', () => { + expect(callsAt('variants', 'src/chains.rs', 100)).toEqual(['Shape::area instance-method']); + expect(callsAt('variants', 'src/chains.rs', 50)).toEqual(['Room::area instance-method']); + }); + + it('reaches the method a trait it derives provides', () => { + expect(callsAt('variants', 'src/chains.rs', 119)).toEqual(['TypedPath::with_query_params instance-method']); + }); +}); + +describe('a Rust path through a type from outside the project', () => { + it('written out or through a crate\'s alias is no project type\'s function', () => { + expect(callsAt('variants', 'src/paths.rs', 5)).toEqual([]); + expect(callsAt('variants', 'src/paths.rs', 9)).toEqual([]); + }); + + it('a `use` names, or the path writes, is not a project namesake\'s', () => { + expect(callsAt('variants', 'src/paths.rs', 13)).toEqual([]); + expect(callsAt('variants', 'src/paths.rs', 17)).toEqual([]); + }); + + it('still reaches the project\'s own impls for that type, or of that trait', () => { + expect(callsAt('keeps', 'src/main.rs', 37)).toEqual(['PathBuf::from qualified-name']); + expect(callsAt('keeps', 'src/main.rs', 41)).toEqual(['Value::blank qualified-name']); + expect(callsAt('keeps', 'src/main.rs', 45)).toEqual(['Sink::flush instance-method']); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index c568b3deb0..816b020de8 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, matchPythonTypedLocalCall, isImportShadowedByJsBinding, isManualMockPath } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, isRustPathCallReachable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, matchPythonTypedLocalCall, isImportShadowedByJsBinding, isManualMockPath } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -3899,8 +3899,10 @@ export class ReferenceResolver { /** * A bare Rust name reaches only what is in scope — every strategy's result, * a framework resolver's `Ok(x)` → `struct Ok` construction included (see - * isRustNameInScope). And a call, however written, reaches a project impl - * on a std type only as Rust would pick it (isRustForeignImplCallable). + * isRustNameInScope). A call through a path whose type is from outside the + * project reaches only the project's impls for that type (or of that trait, + * isRustPathCallReachable). And a call, however written, reaches a project + * impl on a std type only as Rust would pick it (isRustForeignImplCallable). */ private gateRustScope(result: ResolvedRef | null, ref: UnresolvedRef): ResolvedRef | null { if (!result || ref.language !== 'rust') return result; @@ -3909,7 +3911,9 @@ export class ReferenceResolver { const target = this.nodeById(result.targetNodeId); if (!target) return result; if (bare && !isRustNameInScope(target, ref, this.context)) return null; - return ref.referenceKind === 'calls' && !isRustForeignImplCallable(target, ref, this.context) ? null : result; + if (ref.referenceKind !== 'calls') return result; + if (!bare && !isRustPathCallReachable(target, ref, this.context)) return null; + return isRustForeignImplCallable(target, ref, this.context) ? result : null; } /** diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e4098cd842..25a697a198 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -9285,6 +9285,72 @@ export function isRustForeignImplCallable(target: Node, ref: UnresolvedRef, cont return local === undefined || rustFirstArgIs(local, ref, context); } +/** + * Whether a Rust call through a type's path can mean `target` (#2614). A + * type the path names from std or a dependency — through the file's `use` + * (`J::from(i)` under `use serde_json::Value as J`, `Value::default()` under + * `use serde_json::Value`) or as written (`serde_json::Value::default()`) — + * is that crate's own: the project holds a function of it only as its own + * impl of a trait for it (`impl Blank for serde_json::Value`, `impl From + * for PathBuf`), and a call through a trait's path (`W::flush(s)` under `use + * std::io::Write as W`) reaches the project's impls of that trait. A + * same-named function of a project type is another type's: the reporter's + * `J::from(i)` went to a local `impl From for Fail`, a guess by + * the name alone, and `Value::default()` to a project `Value` of its own. + */ +export function isRustPathCallReachable(target: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (target.language !== 'rust') return true; + const segments = ref.referenceName.split('::'); + if (segments.length < 2 || !segments.every((s) => /^[A-Za-z_]\w*$/.test(s))) return true; + const type = segments[segments.length - 2]!; + if (!/^[A-Z]/.test(type) || type === 'Self') return true; + const outside = rustOutsidePathType(type, segments.slice(0, -2), ref, context); + if (outside === null) return true; + const impl = target.kind === 'method' ? rustTraitImplHeader(target, context) : null; + if (!impl) return false; + if (impl.trait === outside) return true; + if (impl.self !== outside) return false; + // The type the impl is for, as its own file names it: the outside one, not a project namesake. + const head = rustPathHead(impl.selfText.replace(/^(?:&\s*(?:'\w+\s+)?(?:mut\s+)?)+/, '')); + const lookup = rustTypeLookup(head?.name ?? impl.self, head?.segments ?? [], target.filePath, context, rustNodeOffset(target, context)); + return lookup.outside || !lookup.nodes.some((n) => RUST_OWNER_KINDS.has(n.kind)); +} + +/** + * The name of the type a Rust path `path::type` names from std or a + * dependency, where the file's `use`s or the path itself say it is from + * there — `J` under `use serde_json::Value as J` is `Value` — or null: one of + * the project's, or one nothing written says is from outside it. + */ +function rustOutsidePathType(type: string, path: string[], ref: UnresolvedRef, context: ResolutionContext): string | null { + const scopes = rustScopesOf(ref.filePath, context); + const at = scopes ? (scopes.lineStarts[ref.line - 1] ?? 0) + Math.max(0, ref.column) : undefined; + let root: string | undefined; + let real = type; + if (path.length === 0) { + const bound = rustUsePath(type, ref.filePath, at, context); + if (!bound || bound.length < 2) return null; + root = bound[0]; + real = bound[bound.length - 1]!; + } else { + root = rustUsePath(path[0]!, ref.filePath, at, context)?.[0] ?? path[0]; + } + return root !== undefined && isRustOutsideCrate(root, ref, context) ? real : null; +} + +/** Whether a Rust path's first segment names std, or a crate the manifests depend on that the project doesn't hold. */ +function isRustOutsideCrate(root: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (RUST_STD_ROOTS.has(root)) return true; + if (root === 'crate' || root === 'self' || root === 'super' || rustProjectCrates(context).has(root)) return false; + if (!rustDependencyCrates(context).has(root)) return false; + // A path dependency the project holds, a workspace member or not, is the project's. + const graph = rustCrateGraph(context); + const own = rustCrateDirOf(ref.filePath, graph); + if (own !== null && graph.crates.get(own)?.keys.has(root)) return false; + for (const crate of graph.crates.values()) if (crate.name === root) return false; + return true; +} + /** Whether `type` is a Rust enum that declares `variant` (#2328). */ function declaresRustVariant(type: Node, variant: string, context: ResolutionContext): boolean { if (type.kind !== 'enum') return false; @@ -10385,6 +10451,8 @@ function rustFieldValue( else written = rustValueType(recv, at, scopes, ref, context, depth + 1); if (!written) return null; const read = rustReadType(written, ref, context); + // `self.pair.0` on a field declared `(Config, u8)`: the tuple's field at that place. + if (read.kind === 'outside' && read.name === '(' && /^\d+$/.test(field)) return rustTupleField(written, Number(field)); if (read.kind !== 'named') return null; const lookup = rustTypeLookup(read.name, read.path, read.written.file, context, read.written.at); const structs = lookup.outside ? [] : lookup.nodes.filter((n) => n.kind === 'struct' || n.kind === 'union'); @@ -24330,6 +24398,225 @@ function matchRustAsMutChainCall(ref: UnresolvedRef, context: ResolutionContext) return rustLentCall(projected && decl.pinned ? `Pin<&mut ${decl.type}>` : decl.type, name, ref, context, tried); } +/** + * A Rust call on a field chain or a struct literal, which the extractor + * records by its bare name — `vis.map.get("a")`, `self.inner.cfg.port()`, + * `self.0.len()`, `Xyz { x: 1.0 }.to_val()` (#2614) — is a call on the type + * that receiver is written with: the last field's as its struct declares it, + * the literal's own (rustValueType, rustTypedMethodCall). By the name alone, + * the reporter's `vis.map.get("a")` on a `HashMap` field went to + * the project's one `get`, `Ext::get`, an associated function at that, and + * `Xyz { … }.to_val()` to `Stat::to_val`, the same trait's other impl. + * Undefined, for the name strategies to decide as before, where the receiver + * is anything else (a call's value, an index, a macro) or its type can't be + * read. + */ +function matchRustWrittenReceiverCall(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null | undefined { + const scopes = rustScopesOf(ref.filePath, context); + const lineStart = scopes?.lineStarts[ref.line - 1]; + if (!scopes || lineStart === undefined || ref.column < 0) return undefined; + const start = lineStart + ref.column; + if (start >= (scopes.lineStarts[ref.line] ?? scopes.code.length)) return undefined; + const receiver = rustWrittenCallReceiver(scopes, start, ref.referenceName); + if (receiver === null) return undefined; + const written = rustValueType(receiver, start, scopes, ref, context, 0); + if (!written) return undefined; + const read = rustReadType(written, ref, context); + // A type no declaration the index holds names — one a macro declares, like + // tokio's `type AtomicUnsignedShort = …` inside `cfg_has_atomic_u64! { … }` — + // says nothing of what the call reaches, unless a wrapper around it has the + // method itself (`Arc::clone`): the name strategies decide, as before. + if (read.kind === 'named' && !read.wrappers.some((w) => RUST_WRAPPER_OWN_METHODS.get(w)?.has(ref.referenceName))) { + const lookup = rustTypeLookup(read.name, read.path, read.written.file, context, read.written.at); + if (lookup.undeclared && rustMethodsNamed(lookup.name, ref.referenceName, context).length === 0) return undefined; + } + let result = rustTypedMethodCall(read, ref.referenceName, ref, context); + if (result === 'unread') return undefined; + if (result === null && read.kind === 'named') result = rustDerivedTraitMethod(read, ref.referenceName, ref, context); + if (result) { + let typed = RUST_TYPED_REFS.get(context); + if (!typed) RUST_TYPED_REFS.set(context, (typed = new WeakSet())); + typed.add(ref); + } + return result; +} + +/** + * The method a Rust project trait that a type derives provides, for a call + * on a value of the type: `#[derive(TypedPath)] struct UsersShow { … }` + * reaches axum-extra's `TypedPath::with_query_params` through the impl the + * derive writes, which the index doesn't hold. Null unless exactly one + * derived trait in scope at the call declares the method with a body. + */ +function rustDerivedTraitMethod(read: Extract, method: string, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const lookup = rustTypeLookup(read.name, read.path, read.written.file, context, read.written.at); + const types = lookup.outside ? [] : lookup.nodes.filter((n) => RUST_OWNER_KINDS.has(n.kind)); + if (types.length !== 1) return null; + const found: Node[] = []; + for (const name of rustDerivedNames(types[0]!, context)) { + const traits = context.getNodesByName(name).filter((n) => n.kind === 'trait' && n.language === 'rust' && n.name === name); + for (const m of rustTraitMethods(traits, method, context)) { + const trait = rustTraitOf(m, context); + if (trait && rustHasBody(m, context) && isRustTraitInScope(trait.name, ref, m, context)) found.push(m); + } + } + const owners = new Set(found.map((m) => rustTraitOf(m, context)?.id)); + return owners.size === 1 ? { original: ref, targetNodeId: found[0]!.id, confidence: 0.85, resolvedBy: 'instance-method' } : null; +} + +/** The names a Rust type's `#[derive(…)]` attributes write, each by its path's last segment. */ +function rustDerivedNames(decl: Node, context: ResolutionContext): string[] { + const scopes = rustScopesOf(decl.filePath, context); + if (!scopes) return []; + const start = rustNodeOffset(decl, context); + const keyword = new RegExp(`\\b(?:struct|enum|union)\\s+${escapeRegExp(decl.name)}\\b`).exec(scopes.code.slice(start, start + 2000)); + if (!keyword) return []; + const at = start + keyword.index; + // The attributes right before the keyword: `#[derive(TypedPath, Deserialize)] #[typed_path("…")] pub struct …`. + const attrs = /((?:#\s*\[(?:[^[\]]|\[[^[\]]*\])*\]\s*)+)(?:pub(?:\s*\([^()]*\))?\s+)?$/.exec(scopes.code.slice(Math.max(0, at - 2000), at))?.[1] ?? ''; + const names: string[] = []; + for (const m of attrs.matchAll(/\bderive\s*\(([^()]*)\)/g)) { + for (const part of m[1]!.split(',')) { + const name = part.trim().split('::').pop()?.trim(); + if (name && /^[A-Za-z_]\w*$/.test(name)) names.push(name); + } + } + return names; +} + +/** Past the whitespace at `i` in a Rust file's blanked text. */ +function rustSpaceEnd(code: string, i: number): number { + while (i < code.length && /\s/.test(code[i]!)) i++; + return i; +} + +/** Past the arguments of a call at `p` — `(…)`, after a turbofish `::<…>` — or -1 where no call's arguments start there. */ +function rustCallArgsEnd(scopes: RustScopes, p: number): number { + const { code, close } = scopes; + let q = rustSpaceEnd(code, p); + if (code.startsWith('::', q)) { + q = rustSpaceEnd(code, q + 2); + if (code[q] !== '<') return -1; + const end = rustAngleEnd(code.slice(q, q + 512), 0); + if (end < 0) return -1; + q = rustSpaceEnd(code, q + end + 1); + } + if (code[q] !== '(') return -1; + const end = close.get(q); + return end === undefined ? -1 : end + 1; +} + +/** + * A field chain or a struct literal at `from` in a Rust file's blanked text, + * up to `limit`: where it ends, how many fields it reads (`.inner`, `.0`), + * and whether it starts with a value that writes its type — a struct literal + * (`Xyz { … }`, `Shape::Circle { … }`, `Self { … }`) or one of these in + * parentheses, `(&Config { … })`. Null for anything else: a path's value, a + * call, a macro, a `match`. + */ +function rustFieldChainAt(scopes: RustScopes, from: number, limit: number): { end: number; hops: number; typed: boolean } | null { + const { code, close } = scopes; + let p = from; + let typed = false; + if (code[p] === '(') { + const end = close.get(p); + if (end === undefined || end >= limit) return null; + let q = rustSpaceEnd(code, p + 1); + if (code[q] === '&') { + q = rustSpaceEnd(code, q + 1); + if (/^mut\b/.test(code.slice(q, q + 4))) q = rustSpaceEnd(code, q + 3); + } + const inner = rustFieldChainAt(scopes, q, end); + if (!inner || rustSpaceEnd(code, inner.end) !== end || (inner.hops === 0 && !inner.typed)) return null; + p = end + 1; + typed = true; + } else { + const path = /^(?:[A-Za-z_]\w*\s*::\s*)*[A-Za-z_]\w*/.exec(code.slice(p, Math.min(limit, p + 512)))?.[0]; + if (!path) return null; + const segments = path.split('::').map((s) => s.trim()); + const last = segments[segments.length - 1]!; + let q = rustSpaceEnd(code, p + path.length); + // `Wrapper:: { … }`: the literal's type arguments. + if (code.startsWith('::', q)) { + const lt = rustSpaceEnd(code, q + 2); + const end = code[lt] === '<' ? rustAngleEnd(code.slice(lt, lt + 512), 0) : -1; + if (end < 0) return null; + q = rustSpaceEnd(code, lt + end + 1); + } + if (/^[A-Z]/.test(last) && code[q] === '{') { + const end = close.get(q); + if (end === undefined || end >= limit) return null; + p = end + 1; + typed = true; + } else if (segments.length === 1 && /^[a-z_]/.test(last)) { + // A local, a parameter or `self`; a keyword (`match x { … }`) reads no field after it. + p += path.length; + } else { + return null; + } + } + let hops = 0; + for (;;) { + let q = rustSpaceEnd(code, p); + if (q >= limit || code[q] !== '.' || code[q + 1] === '.') break; + q = rustSpaceEnd(code, q + 1); + const id = /^(?:[A-Za-z_]\w*|\d+)/.exec(code.slice(q, Math.min(limit, q + 128)))?.[0]; + // `.await` and a method call end the chain. + if (!id || id === 'await' || rustCallArgsEnd(scopes, q + id.length) >= 0) break; + p = q + id.length; + hops++; + } + return { end: p, hops, typed }; +} + +/** + * The receiver of the Rust call `name(…)` whose expression starts at `start` + * (where the extractor records a call it names bare), when it is a field + * chain — `vis.map`, `self.inner.cfg`, `self.0` — or a struct literal, alone + * or in parentheses (rustFieldChainAt), as the file's blanked text writes it. + * Null for any other receiver, and when a later link of the chain calls + * `name` again (`m.inner.get(k).get(j)`): that call is recorded at the same + * place, and can't be told from this one. + */ +function rustWrittenCallReceiver(scopes: RustScopes, start: number, name: string): string | null { + const { code, close } = scopes; + if (/\w/.test(code[start - 1] ?? '')) return null; + const chain = rustFieldChainAt(scopes, start, code.length); + // `x.name(…)` and `self.name(…)` are recorded with their receiver. + if (!chain || (chain.hops === 0 && !chain.typed)) return null; + const dot = rustSpaceEnd(code, chain.end); + if (code[dot] !== '.' || code[dot + 1] === '.') return null; + const at = rustSpaceEnd(code, dot + 1); + if (!code.startsWith(name, at) || /\w/.test(code[at + name.length] ?? '')) return null; + let p = rustCallArgsEnd(scopes, at + name.length); + if (p < 0) return null; + // The links after the call: `?`, `[…]`, `.field`, `.method(…)`. + for (;;) { + p = rustSpaceEnd(code, p); + const c = code[p]; + if (c === '?') { + p++; + continue; + } + if (c === '[') { + const end = close.get(p); + if (end === undefined) return null; + p = end + 1; + continue; + } + if (c !== '.' || code[p + 1] === '.') break; + p = rustSpaceEnd(code, p + 1); + const id = /^(?:[A-Za-z_]\w*|\d+)/.exec(code.slice(p, p + 128))?.[0]; + if (!id) break; + p += id.length; + const args = rustCallArgsEnd(scopes, p); + if (args < 0) continue; + if (id === name) return null; + p = args; + } + return code.slice(start, dot).trim(); +} + /** Std pointers whose `as_ref` / `as_mut` lend what they hold: `Box`, `NonNull` → `&mut T`. */ const RUST_LENDING_POINTERS: ReadonlySet = new Set(['Box', 'NonNull', 'Rc', 'Arc', 'Cow']); @@ -26731,9 +27018,14 @@ function matchReferenceInner( // extractor records by its bare name — `self.delay.as_mut().reset(…)` — is // a call on the type the receiver is declared with, or on nothing of the // project's (matchRustAsMutChainCall). + // So is one on a field chain or a struct literal — `vis.map.get("a")`, + // `Xyz { x: 1.0 }.to_val()` — a call on the type that receiver is written + // with (matchRustWrittenReceiverCall). if (ref.language === 'rust' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName)) { const lent = nmTimedT('rustAsMut', ref, () => matchRustAsMutChainCall(ref, context)); if (lent !== undefined) return lent; + const written = nmTimedT('rustWritten', ref, () => matchRustWrittenReceiverCall(ref, context)); + if (written !== undefined) return written; } // A Kotlin call on what an unqualified call returns, which the extractor From f7820b584fc901bbce31763ec0ae1dbcff3cac5f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 07:18:05 +0000 Subject: [PATCH 61/95] fix(dart): calls to a private name in another part of the same library link again (#2832) Dart keeps a `_name` to its library: the library file and every file its `part` directives name. isVisibleAcrossFiles still listed Dart in PRIVATE_IS_FILE_LOCAL (added with PR #1745 for Kotlin, Java, C#, Swift, Scala and PHP), and both Dart extractors mark a `_` function, method or class `private`. So once the name-matching pipeline had picked a private candidate in another file, the pick was declined and the reference stayed unresolved. A library split into parts, generated `.g.dart` and `.freezed.dart` parts included, lost the calls and type uses between its files. Fix: - dartPrivateVisible (dart-libraries.ts) says whether a file belongs to the reference's library, read from `part` / `part of` alone. Unlike inSameDartLibrary it needs no pubspec.yaml, since a part names its library by a path or a library name. When the reference's library can't be found, the name stays file-local, as before. - isVisibleAcrossFiles takes Dart out of PRIVATE_IS_FILE_LOCAL and lets a private Dart candidate through when it is in the reference's library. - matchByExactName drops another library's private Dart candidates before the ranking, as it does for C#, PHP, Java and Go names the file can't see. Otherwise another library's namesake can rank first, be declined, and take the library's own declaration down with it. Without a pubspec nothing else filters a top-level `_name`, and a member's class hierarchy is matched by type name, which can reach another library's class of the same name (the SDK's wasm and VM `_ImmutableLinkedHashMapMixin`). Validation: - New __tests__/dart-part-privacy.test.ts, run with and without a pubspec: the issue's library plus a receiver call, a generated part's `_$UserFromJson` and `_$Counter`, another library that imports the one declaring the names, a namesake beside a part, a part named by library name, and a part whose library is missing. 6 of 12 fail on main's resolver, 12 pass. With the pre-ranking filter disabled, 1 fails (the namesake without a pubspec). - A/B vs main 58eb57a3, each arm with its own dist and kernel; the kernel and wasm paths give the same changed-site sets. Re-run after the rebases, on 5b47bc49 (kernel) and 8fdc8a1e (wasm), with the same changed sites. Edges added: getx (217 Dart files) 0, bloc (616) +49, riverpod (1,300) +542, localsend (393) +1, protobuf.dart (225) +143, apidash (954) +342, dart-lang/sdk sdk/lib (480, no pubspec) +207. Nothing removed or retargeted, and 1,064 edges changed only their confidence. Every newly resolved ref has its edge at the same site; nodes and files are unchanged. - Triage of all 1,284 added edges: 1,274 right. An independent part/part-of oracle confirms 1,195 (a top-level `_name` unique in its library, a member of the call site's own hierarchy, or a member of the receiver's declared type); 79 were checked by hand. 10 are wrong, all receiver-name guesses (instance-method 0.65-0.8) that pick one implementation of an abstract member, or the first of a tie: the guess public names already get, which the file-local rule had hidden. The 1,064 confidence changes are same-file targets that had another library's namesake beside them (exact-match 0.7 to 0.9). - gin and moshi controls byte-identical, kernel and wasm. Kernel and wasm full-index dumps byte-identical on every Dart repo. kernel-parity --lang dart: 0 diffs on all seven repos. - Index time, median of 3 interleaved runs on the kernel path, CPU with workers, main vs fix: riverpod 55.6 / 50.3 s, sdk/lib 59.4 / 59.6 s, apidash 29.5 / 30.8 s, bloc 18.6 / 18.3 s, protobuf.dart 14.7 / 14.3 s. Within noise. Fixes #2466 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/dart-part-privacy.test.ts | 201 ++++++++++++++++++++++++++++ src/resolution/dart-libraries.ts | 15 +++ src/resolution/name-matcher.ts | 35 ++++- 4 files changed, 248 insertions(+), 4 deletions(-) create mode 100644 __tests__/dart-part-privacy.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ec68ad34a..d4189422f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -114,6 +114,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Dart, a call, a constructor call or a type written without a receiver now links only to a declaration the file can see: one in its own library (the file and its `part` files) or one exported by a library it imports, with `export` chains, `show`, `hide` and `as` prefixes followed the way Dart follows them. So a generated `.g.dart` file reaches its own library's function or type instead of a same-named one in a neighbouring file, a call to a package function such as `fakeAsync`, `runApp` or `unawaited` no longer links to an unrelated project function of that name, and a bare call is never taken for another type's enum value or static constant. Re-index Dart projects after upgrading. - In Dart, an `import` or `export` now links to the file it names: `package:my_app/src/cart.dart` to that package's `lib/src/cart.dart`, and a relative path to the file it points to from the importing file. Before, only the file name at the end was matched, so `import 'package:flutter/widgets.dart'` linked to whichever project file was called `widgets.dart`, and most imports of your own files didn't link to the file at all, which threw off `codegraph affected` and the files listed as using a file. An import of a Dart SDK library or of a package from outside your project now links nothing, and when packages in one repository share a name, the one the importing app depends on by `path:` is used. Re-index Dart projects after upgrading. - In Dart, a `part` directive, like `part 'user.g.dart';` or `part 'src/widgets.dart';`, now links a library to the part's file the way an import does. A library and its parts are one library, so a change to a part, whether a generated `.g.dart` or `.freezed.dart` file or one you wrote, now reaches the library and the code that imports it in `codegraph affected` and in the files listed as using the part. Before, the two files were connected only when a call or a type happened to cross between them, which a library made only of `part` lines never has. A part your project doesn't have, like a generated file that isn't committed, links nothing. Re-index Dart projects after upgrading. +- In Dart, a private name declared in another file of the same library, the library file or one of its `part` files, now links again, as Dart shares such names across the whole library: a call to a `_format()` helper, a `_Foo()` constructor call, or a class extending Riverpod's generated `_$Counter`. Before, a library split into parts, including the code generated into `.g.dart` and `.freezed.dart` files like json_serializable's `_$UserFromJson(json)`, lost the calls and type uses between its files from `callers` and impact. A private name of another library is still never linked. Re-index Dart projects after upgrading. Thanks @libin1528 for the report and @maxmilian. (#2466) - In Dart, an annotation now links to the constant or constructor it names, read the way it is written: `@riverpod` reaches Riverpod's `const riverpod = Riverpod();`, `@Riverpod(keepAlive: true)` the `Riverpod` class, an annotation written through a class, like `@SomeClass.value` or `@SomeClass.named(…)`, that class's own constant, enum value or constructor, and `@meta.immutable` only what the `meta` import brings in. Before, an annotation was matched like a function decorator, so a method, getter or function that only shared its name won: every `@riverpod` provider in Riverpod's own repository was linked to an unrelated getter called `riverpod`, and `callers` and impact for the real annotation missed them. Re-index Dart projects after upgrading. - In Dart, an annotation is no longer lost when a comment sits between it and the declaration, like `@override` followed by `// ignore: must_call_super`, a `///` comment written below the annotations, or `@riverpod` followed by an `// expect_lint: …` line. Before, the comment hid the annotation, so a provider written that way was missing from the annotation's `callers` and impact. Re-index Dart projects after upgrading. - In Dart, a call written through an import prefix, like `http.get(…)` after `import 'package:http/http.dart' as http;`, or `utils.Listener()` and `kit.Box.create()` through a prefix of your own, now links to what the library imported with that prefix exports, following its `export` chains, `show` and `hide`, and to nothing when that library is a package outside your project. Before, the prefix was taken for a variable, so `http.get(…)` could link to an unrelated `Http` class's `get` method, and calls made through a prefix into your own libraries were missing from `callers` and impact. Re-index Dart projects after upgrading. diff --git a/__tests__/dart-part-privacy.test.ts b/__tests__/dart-part-privacy.test.ts new file mode 100644 index 0000000000..4da99b6d2b --- /dev/null +++ b/__tests__/dart-part-privacy.test.ts @@ -0,0 +1,201 @@ +/** + * A Dart `_name` is private to its library — the library file and every file + * its `part` directives name — not to the one file that declares it (#2466). + * A library split into parts calls its own private helpers across those + * files, and those calls were dropped as if each file were a library of its + * own. A file of another library still never reaches them, importing the + * library or not. Whether a pubspec.yaml sits above the files makes no + * difference: a part names its library by a path or by a library name. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +function writeTree(root: string, files: Record): void { + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } +} + +/** ` -> :` for every `calls` edge out of `files`. */ +function callEdges(cg: CodeGraph, files: string[]): string[] { + return edgesOf(cg, files, ['calls']).map((e) => e.slice(e.indexOf(' ') + 1)); +} + +/** ` -> :` for every edge of `kinds` out of `files`. */ +function edgesOf(cg: CodeGraph, files: string[], kinds: string[]): string[] { + const ids = files.flatMap((f) => cg.getNodesInFile(f).map((n) => n.id)); + return cg.getOutgoingEdgesFrom(ids) + .filter((e) => kinds.includes(e.kind)) + .map((e) => { + const t = cg.getNode(e.target)!; + return `${e.kind} ${cg.getNode(e.source)!.qualifiedName} -> ${t.filePath}:${t.qualifiedName}`; + }) + .sort(); +} + +/** + * The issue's library (a library file and three parts, one of them in a + * subdirectory), a library with a generated part, and a library that uses both. + */ +const GREETER: Record = { + 'lib/greeter.dart': `part 'greeter_format.dart'; +part 'greeter_shout.dart'; +part 'src/_greeter_whisper.dart'; + +class Greeter with _Formatting, _Shouting, _Whispering { + String greet(String name) => _format(name); // library file -> private member in a part + String hello() => _local(); // same-file control + String _local() => 'hi'; +} + +void _trace(String s) => print(s); +`, + 'lib/greeter_format.dart': `part of 'greeter.dart'; + +mixin _Formatting { + String _format(String name) { + _trace(name); // part -> private top-level function in the library file + return 'Hello, $name'; + } +} +`, + 'lib/greeter_shout.dart': `part of 'greeter.dart'; + +mixin _Shouting on _Formatting { + String shout(String name) => _format(name).toUpperCase(); // part -> sibling part + String echo(Greeter g) => g._local(); // part -> private member, through a receiver +} +`, + 'lib/src/_greeter_whisper.dart': `part of '../greeter.dart'; + +mixin _Whispering on _Formatting { + String whisper(String name) => _format(name).toLowerCase(); // part in a subdirectory -> sibling part + void log(String s) => _trace(s); // -> the library file's, not the namesake beside this part +} +`, + // What code generators write into a part: json_serializable's `_$UserFromJson`, + // and a base class like riverpod_generator's `_$Counter`. + 'lib/models/user.dart': `part 'user.g.dart'; + +class User { + final String name; + User(this.name); + factory User.fromJson(Map json) => _$UserFromJson(json); +} + +class Counter extends _$Counter {} +`, + 'lib/models/user.g.dart': `part of 'user.dart'; + +User _$UserFromJson(Map json) => User(json['name'] as String); + +abstract class _$Counter {} +`, + // Another library: the greeter library's privates are not its to name, even through an import. + 'lib/other.dart': `import 'greeter.dart'; +import 'models/user.dart'; + +void callsOther(Greeter g, Object o) { + _trace('x'); + g._local(); + if (o is _$Counter) _$UserFromJson({}); +} +`, +}; + +const GREETER_FILES = ['lib/greeter.dart', 'lib/greeter_format.dart', 'lib/greeter_shout.dart', 'lib/src/_greeter_whisper.dart']; + +const GREETER_EDGES = [ + 'Greeter::greet -> lib/greeter_format.dart:_Formatting::_format', + 'Greeter::hello -> lib/greeter.dart:Greeter::_local', + '_Formatting::_format -> lib/greeter.dart:_trace', + '_Shouting::echo -> lib/greeter.dart:Greeter::_local', + '_Shouting::shout -> lib/greeter_format.dart:_Formatting::_format', + '_Whispering::log -> lib/greeter.dart:_trace', + '_Whispering::whisper -> lib/greeter_format.dart:_Formatting::_format', +]; + +for (const pubspec of [true, false]) { + describe(`Dart private names across a library's parts (${pubspec ? 'with' : 'without'} a pubspec.yaml)`, () => { + let root = ''; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-dart-part-private-')); + writeTree(root, { + ...(pubspec ? { 'pubspec.yaml': 'name: greeter\n' } : {}), + ...GREETER, + // A library that keeps its own `_trace` and `_local`, beside the + // greeter's part in lib/src/ and so nearer it than the greeter's own. + 'lib/src/farewell.dart': `void _trace(String s) => print(s); + +class Farewell { + String _local() => 'bye'; + String bye() => _local(); +} + +void wave() => _trace('wave'); +`, + // A library whose part is named by the library's name. + 'lib/named.dart': `library app.named; + +part 'named_part.dart'; + +int _seed() => 6; +`, + 'lib/named_part.dart': `part of app.named; + +int usesSeed() => _seed(); +`, + // A part whose library the project doesn't have: its own file is all it can see. + 'lib/stray_part.dart': `part of 'missing.dart'; + +int usesStray() => _seed(); +`, + }); + cg = await CodeGraph.init(root, { index: true }); + }); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + it('links a call to a private member or function declared in another file of the library', () => { + expect(callEdges(cg, GREETER_FILES)).toEqual(GREETER_EDGES); + }); + + it('links the private function and class a generated part declares', () => { + expect(edgesOf(cg, ['lib/models/user.dart'], ['calls', 'extends'])).toEqual([ + 'calls User::fromJson -> lib/models/user.g.dart:_$UserFromJson', + 'extends Counter -> lib/models/user.g.dart:_$Counter', + ]); + }); + + it("never links another library's private names, even when the file imports that library", () => { + // Only the public `Greeter` the parameter names. + expect(edgesOf(cg, ['lib/other.dart'], ['calls', 'references', 'instantiates'])).toEqual([ + 'references callsOther -> lib/greeter.dart:Greeter', + ]); + }); + + it("keeps another library's private namesakes to that library", () => { + expect(callEdges(cg, ['lib/src/farewell.dart'])).toEqual([ + 'Farewell::bye -> lib/src/farewell.dart:Farewell::_local', + 'wave -> lib/src/farewell.dart:_trace', + ]); + }); + + it('finds the library a part names by library name', () => { + expect(callEdges(cg, ['lib/named_part.dart'])).toEqual(['usesSeed -> lib/named.dart:_seed']); + }); + + it("links nothing from a part whose library isn't in the project", () => { + expect(callEdges(cg, ['lib/stray_part.dart'])).toEqual([]); + }); + }); +} diff --git a/src/resolution/dart-libraries.ts b/src/resolution/dart-libraries.ts index 2301899864..edc6691025 100644 --- a/src/resolution/dart-libraries.ts +++ b/src/resolution/dart-libraries.ts @@ -794,3 +794,18 @@ export function inSameDartLibrary(fromFile: string, declFile: string, context: R const scope = scopeOf(fromFile, context, memoFor(context)); return scope !== null && scope.own.has(declFile); } + +/** + * Whether a private `_name` declared in `declFile` can be named in `fromFile`: + * both are files of one library, the library file and its parts. Dart keeps a + * `_name` to its library, not to its file, so a library split into `part`s + * calls its own private helpers across them. Unlike inSameDartLibrary this + * needs no pubspec.yaml above the files: a part names its library by a path + * or a library name. False when the library of `fromFile` is unknown. + */ +export function dartPrivateVisible(fromFile: string, declFile: string, context: ResolutionContext): boolean { + if (fromFile === declFile) return true; + const memo = memoFor(context); + const library = libraryOf(fromFile, context, memo); + return library !== null && membersOf(library, context, memo).has(declFile); +} diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 25a697a198..55fbe56008 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -12,7 +12,7 @@ import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, isSuperty import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_ARRAY_METHODS, JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; -import { dartImportPrefixes, dartLibrarySees, dartPrefixSees, inSameDartLibrary } from './dart-libraries'; +import { dartImportPrefixes, dartLibrarySees, dartPrefixSees, dartPrivateVisible, inSameDartLibrary } from './dart-libraries'; import { isDartLocallyBound } from './dart-local-scope'; import { breakVbTie, isVbMemberInScope, isVbNestedTypeInScope, isVbTypeQualifiedBy, matchVbTypedCall, preferVbProject, sameVbProject } from './vbnet-receivers'; import { cppAliasedClass, cppAliasedTypeName, cppAliasedTypeWritten, cppDeclaresLocalAlias, cppParentScope, cppScopesWithin, cppTemplateArguments, cppTemplateParameters, cppTypeNamesDeclared, cppTypeSegments, isCppPointerType, resolveCppAliasedType, stripCppTemplateArguments, type CppAliasedType } from './cpp-type-aliases'; @@ -2150,9 +2150,10 @@ function luaModuleFile(name: string, decl: Node, context: ResolutionContext): st /** * Languages in which `visibility: 'private'` on a definition means no other * FILE can name it: a Kotlin `private fun` is file- or class-local, and the - * same holds for Java, C#, Swift, Scala, Dart and PHP members. + * same holds for Java, C#, Swift, Scala and PHP members. (A Dart `_name` is + * its library's, which can span several files: see isVisibleAcrossFiles.) */ -const PRIVATE_IS_FILE_LOCAL = new Set(['kotlin', 'java', 'csharp', 'swift', 'scala', 'dart', 'php']); +const PRIVATE_IS_FILE_LOCAL = new Set(['kotlin', 'java', 'csharp', 'swift', 'scala', 'php']); /** Per-context memo: node id → "this C/C++ function is declared `static`". */ const C_STATIC_MEMO = new WeakMap>(); @@ -3404,9 +3405,12 @@ function isGoNameVisible(candidate: Node, ref: UnresolvedRef, context: Resolutio * - **Objective-C**: a struct, union, enum, typedef or global a `.m` / `.mm` * source file defines is local to that file, as in C; other files reach a * global through its header's `extern` declaration. A class is not. - * - **Kotlin, Java, C#, Swift, Scala, Dart, PHP**: `private` is class- or + * - **Kotlin, Java, C#, Swift, Scala, PHP**: `private` is class- or * file-local. An Android `editor.apply()` resolved onto an unrelated class's * `private fun apply`. + * - **Dart**: a `_name` is private to its library — the library file and the + * files its `part` directives name — so another file of the library names + * it, and a file of another library never does (#2466). * - **Go**: an unexported (lowercase) identifier is package-local, and a * package is a directory. Judged by the name's case: the extractor's * `isExported` is unset for every Go method. @@ -3496,6 +3500,13 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex const owner = rustModuleDir(candidate.filePath); return ref.filePath.startsWith(owner + '/'); } + // A Dart `_name` is its library's, which spans the library file and its + // parts: riverpod_generator's `class Counter extends _$Counter` names the + // class it wrote into `main.g.dart`, and protobuf.dart's GeneratedMessage + // calls `_fieldSet._clear()` on the FieldSet another part declares. + if (lang === 'dart') { + return candidate.visibility !== 'private' || (ref.language === 'dart' && dartPrivateVisible(ref.filePath, candidate.filePath, context)); + } if (PRIVATE_IS_FILE_LOCAL.has(lang)) return candidate.visibility !== 'private'; // An R test file runs in an environment of its own (testthat): its top-level // `c <- ggplot(…)` is not what the package's 2,843 `c(…)` calls mean. The @@ -4364,6 +4375,21 @@ function isDartTopLevelVisible(candidate: Node, ref: UnresolvedRef, context: Res : dartPrefixSees(ref.filePath, prefix, candidate.filePath, candidate.name, context); } +/** + * Whether `n` is a Dart `_name` that another library declares, which nothing + * written in the reference's library can mean (#2466). It leaves the + * candidates before one is picked, so the library's own declaration wins + * where another library's namesake would have ranked first and then been + * declined: the Dart SDK's wasm `_ImmutableLinkedHashMapMixin` calls the + * `_createIndex` of the `_MapCreateIndexMixin` it is `on`, not the one the + * VM's mixin of the same name declares. isVisibleAcrossFiles applies the + * same rule to the pipeline's pick. + */ +function isDartPrivateOfAnotherLibrary(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + return n.language === 'dart' && n.visibility === 'private' && n.filePath !== ref.filePath && + !dartPrivateVisible(ref.filePath, n.filePath, context); +} + /** * The import prefix a Dart type name is written through — the `p` of * `p.Report`, `p.Color.red` — or null. A type reference's column is the @@ -11844,6 +11870,7 @@ export function matchByExactName( !(ref.language === 'dart' && isDartConstructor(n, context)) && !(dartBare && isDartTypeValue(n) && !isDartMethodInScope(n, ref, context)) && !((dartBare || dartTypeRef) && !isDartTopLevelVisible(n, ref, context, dartPrefix)) && + !(ref.language === 'dart' && isDartPrivateOfAnotherLibrary(n, ref, context)) && !(phpSelf && (n.kind !== 'method' || !isPhpMethodInScope(n, ref, phpSelf, context))) && !(pythonShape && !fitsPythonCallShape(n, pythonShape, ref, context)) && !(rustBare && !isRustNameInScope(n, ref, context)) && From 16a8e77c469a4b24552cbb631730212264bbf53f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 08:22:15 +0000 Subject: [PATCH 62/95] fix(ruby): a call reaches the class method or the instance method its receiver calls (#2834) A Ruby class method (`def self.build`, `def Widget.build`, or a def inside `class << self`) was indexed exactly like an instance method: is_static 0, under the same qualified name as an instance method of that name (Widget::build twice). So every call took whichever of the two the class declared first. Widget.build could reach `def build`, and widget.build on an instance could reach `def self.build`. In the repro for issue 2503, `card = Billing::ChargeCard.new(...); card.call` linked to `def self.call` rather than `def call`. Fix: - Extraction, both engines: the Ruby isStatic hook (isRubyClassMethod in languages/ruby.ts, is_class_method in codegraph-kernel/src/ruby.rs) marks a singleton_method, or a method whose parent past one body_statement is a singleton_class, as static. Every other def is marked not static, the same way on both, so the graph only changes for class methods. - Resolution, Ruby only, where the call says which kind it is: rubyMethodsOfKindFirst orders a call's candidates with the wanted kind first. A call on a constant (Strategy 1, its inherited walk, and the namespaced path in ruby-constant-paths.ts) wants the class method. A receiver typed by `x = Widget.new` (resolveMethodOnType) wants the instance method. A bare call splits only a real pair, one qualified name among its candidates as both kinds, keeping the caller's kind: a block may run on another object, so nothing else moves. A receiver known only by its name (Strategies 2 and 3) says nothing of its kind and is left as it was: rubocop's `cop.cop_name` runs over cop classes. With only one kind present, every path picks what it did before. Validation: - New __tests__/ruby-singleton-methods.test.ts (6 cases, each on the kernel and on wasm) and a parity case in kernel-ruby-parity.test.ts. On main 9 of the 18 tests in the two files fail, every one that checks the new behaviour; the guards and the old parity cases pass. All pass with the fix. - A/B against main f7820b58 (kernel path): nodes are unchanged except is_static going 0 to 1 on the class methods, 167 on sinatra, 358 on rubocop and 266 on mastodon's Ruby tree. Edges: none added or removed. 20 changed their target or confidence: sinatra 10, rubocop 0, mastodon 10. 19 are right: 13 moved from the wrong twin to the right one, such as InlineRenderer.render to def self.render and sinatra's instance methods' force_encoding to def force_encoding, and 6 kept their target with higher confidence. One kept a target wrong in both arms: a call inside sinatra's define_method block. Controls express and flask are byte-identical. - The A/B narrowed two rules. Preferring instance methods for a receiver known only by its name moved rubocop's 19 right cop.cop_name edges, and dropping other-kind methods from any bare call moved block-DSL calls on sinatra. Both are gone: only a real pair is split. - kernel-parity.mjs --lang ruby is the same on both arms: sinatra 147/147, rubocop 1799/1801 (2 deferred), mastodon 3333/3333, 0 diffs. The fix arm's kernel and wasm graphs are byte-identical on sinatra and rubocop. - Index time, median of 3 interleaved runs: sinatra -3.7%, rubocop +0.5%, mastodon +0.7%, controls express -2.7% and flask +1.0%. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/kernel-ruby-parity.test.ts | 45 +++++ __tests__/ruby-singleton-methods.test.ts | 212 ++++++++++++++++++++++ codegraph-kernel/src/ruby.rs | 33 +++- docs/design/ruby-kernel-port-checklist.md | 16 +- src/extraction/languages/ruby.ts | 15 ++ src/resolution/name-matcher.ts | 93 ++++++++-- src/resolution/ruby-constant-paths.ts | 8 +- 8 files changed, 396 insertions(+), 27 deletions(-) create mode 100644 __tests__/ruby-singleton-methods.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d4189422f0..9aaca65a56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -206,6 +206,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Go, a struct that embeds another is no longer treated as overriding the methods it redeclares. gin's `Engine` embeds `RouterGroup` and has a `Use` of its own, but calling `Use` on a `RouterGroup` only ever runs `RouterGroup.Use`; before, `codegraph_explore`'s call path and impact went on from it into `Engine.Use` and the code only the engine runs. The same held for every struct or defined type embedded in another that redeclares one of its methods, like etcd's `Mutex` inside `lockerMutex`. A call through an interface still reaches each type that implements it, the embedding struct included. Re-index Go projects after upgrading. - In JavaScript and TypeScript, a subclass's `static` method no longer counts as overriding an instance method of the same name in its base class, or the reverse, since calling one never runs the other. In webpack, `codegraph_explore`'s call path and impact went on from `Module`'s `deserialize(context)` into the `static deserialize(context)` of `NormalModule` and other modules. A static method still overrides its base's static method and an instance method its base's instance method, and JavaScript class members are now marked static, as TypeScript ones already were. Re-index JavaScript and TypeScript projects after upgrading. Thanks @inth3shadows for the report and @yang0228 for the pull request. (#2761) - In TypeScript and JavaScript, a class in a `.tsx` or `.jsx` file now links its base class's and interfaces' methods to its overrides, as a class in a `.ts` or `.js` file already did. `codegraph_explore`'s call path, callers and impact now follow a call through the base into the override, like outline's `BaseEmail.send()` reaching each email's `render()`. This also works across file types, such as an interface in a `.ts` file implemented in a `.tsx` one. Before, these classes had no override links at all. Re-index projects with `.tsx` or `.jsx` files after upgrading. +- In Ruby, a class method, `def self.build` or a `def` inside `class << self`, is now told apart from an instance method of the same name. A call on the class, like `Widget.build`, links to the class method; a call on an instance, like `widget.build` after `widget = Widget.new`, links to the instance method; and a call without a receiver links to the one that matches the method it is written in. Before, both were stored as instance methods under one name, so a call linked to whichever the class declared first: in a service object with `def self.call` and `def call`, `card.call` on an instance linked to the class method. Re-index Ruby projects after upgrading. - In Java, Kotlin, C# and Scala, each overload of an interface or base-class method now links only to the overrides with the same parameters. Before, every overload linked to every method of that name in the implementing class, so `ExampleService.putTairValue(String)` also led to `putTairValue(List, double)`, and `codegraph_explore`'s call path, callers and impact went into overloads that call never runs. Where a parameter is a type parameter, like `save(T entity)` implemented as `save(String entity)`, or the class is written in another language than its interface, the number of parameters decides; a method declared once on each side links as before. Re-index Java, Kotlin, C# and Scala projects after upgrading. Thanks @hxj1225 for the report and @danusha2345 for the pull request. (#2663) - In Kotlin, a call on what one of your functions returns, like `repository().pending(1)` or `val repo = repository(); repo.pending(1)` with a `private fun repository(): RealRepo` in a test class, now links to the method of the type that function declares, including one the type inherits or an extension on it. Before, the chained call went to whichever `pending` the name matched first, like a test double's in another package, and the local was guessed from its name, which reached the interface. The function is found in the calling class, a class it inherits from or one around it, or at the top level, and a nested type like `Request.Builder` is that type, not another class's `Builder`. When the declared type has no such method, the call now links to nothing rather than a method of the same name elsewhere; a type from outside your project is matched by name as before. Re-index Kotlin projects after upgrading. Thanks @mindcipherrr for the report. (#2805) - In JavaScript and TypeScript, a `super.run()` call now links to the method of the class's parent, or of the nearest ancestor that has it, like three.js's `super.copy(source)` reaching `Object3D.copy`, so `codegraph_explore`'s call path and impact follow an override into the method it builds on. Before, such a call inside a method of the same name was dropped, and elsewhere it went to whatever method had the name, often the class's own or a sibling class's in the same file. A call from a static method reaches the parent's static method, and a call to a parent from outside your project, like `HTMLElement`, `Set` or `Promise`, links nothing. Re-index JavaScript and TypeScript projects after upgrading. Thanks @Puma7 for the report and the pull request. (#2497) diff --git a/__tests__/kernel-ruby-parity.test.ts b/__tests__/kernel-ruby-parity.test.ts index 9dcf00c517..72a25297e8 100644 --- a/__tests__/kernel-ruby-parity.test.ts +++ b/__tests__/kernel-ruby-parity.test.ts @@ -148,6 +148,51 @@ describe.skipIf(!kernelBuilt)('kernel Ruby extraction parity', () => { ]); }); + // A class method (`def self.x`, `def Widget.x`, a `def` in `class << self`) + // is static on both backends, every other def is not: they share a + // qualified name with any instance method of the name. + it('class methods are static', () => { + const src = [ + 'class Widget', + ' def build = self', + ' def self.build(opts) = new', + ' def Widget.named = 1', + ' class << self', + ' def make = new', + ' def other', + ' def nested = 2', + ' end', + ' end', + ' def run', + ' def inner = 3', + ' end', + 'end', + 'def self.top = 4', + 'def plain = 5', + '', + ].join('\n'); + assertParity('src/widget.rb', src); + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + delete process.env.CODEGRAPH_KERNEL; + const viaKernel = tryKernelExtract('src/widget.rb', src, 'ruby'); + const statics = viaKernel!.nodes + .filter((n) => n.kind === 'method' || n.kind === 'function') + .map((n) => `${n.kind} ${n.name}:${n.startLine} ${n.isStatic === true ? 'static' : n.isStatic === false ? 'instance' : 'unset'}`) + .sort(); + expect(statics).toEqual([ + 'function inner:12 instance', + 'function nested:8 instance', + 'function plain:16 instance', + 'function top:15 static', + 'method build:2 instance', + 'method build:3 static', + 'method make:6 static', + 'method named:4 static', + 'method other:7 static', + 'method run:11 instance', + ]); + }); + it('mixin implements refs carry filePath through the v2 ref-flag wire path', () => { const src = 'class Widget\n include Comparable\nend\n'; process.env.CODEGRAPH_KERNEL_LANGS = 'all'; diff --git a/__tests__/ruby-singleton-methods.test.ts b/__tests__/ruby-singleton-methods.test.ts new file mode 100644 index 0000000000..5307b35a2a --- /dev/null +++ b/__tests__/ruby-singleton-methods.test.ts @@ -0,0 +1,212 @@ +/** + * A Ruby class method — `def self.build`, `def Widget.build`, or a `def` + * inside `class << self` — is the class's own, not one its instances answer + * to. They were indexed as instance methods under the same qualified name as + * an instance method of that name (`Widget::build` twice), so a call picked + * whichever the class declared first: `Widget.build` could land on + * `def build`, and `widget.build` on `def self.build`. Class methods are now + * marked static, and a call reaches the one of the right kind: a call on the + * class (`Widget.build`, `Billing::Payout.call`) its class method, a call on + * an instance (`w = Widget.new; w.build`) its instance method, and a bare + * call the one of the calling method's kind. Only such pairs are split: a + * block may run on another object, so a bare call with no same-named method + * of the other kind keeps the target it had. A receiver known by its name + * alone (`widget.build`) may hold the class itself, so it splits nothing. + * + * Runs on the native kernel (skipped when none is built) and the wasm + * extractor. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { getKernel } from '../src/extraction/kernel/loader'; + +const FILES: Record = { + // The instance method first, so declaration order alone picks it. + 'app/widget.rb': `class Widget + def build + self + end + + def self.build(opts) + new + end + + class << self + def make + new + end + + def build_all + build(1) + end + end + + def run + build + Widget.build(1) + end + + def self.go + build(2) + end +end +`, + 'app/user.rb': `class User + def x + w = Widget.new + w.build + Widget.build(3) + Widget.make + end +end +`, + // #2503's control: an instance of a class whose class method comes first. + 'app/services/billing/charge_card.rb': `module Billing + class ChargeCard + def self.call(**kw) = new(**kw).call + def initialize(amount:) = @amount = amount + def call = @amount + end +end +`, + // A class method's block (a test DSL's) calls what an instance method and a + // top-level function are both named: no pair of one class, so no split. + 'app/spec_like.rb': `class SpecLike + def self.parses + run_case do + mock_app + end + end + + def mock_app = 1 +end +`, + 'app/helpers.rb': `def mock_app = 2 +`, + // A receiver named after a class that holds classes, as rubocop's + // `cop.cop_name` over its registry's cop classes does. + 'app/cop.rb': `class Cop + def self.cop_name = 'Style/Foo' + def cop_name = self.class.cop_name +end +`, + 'app/registry.rb': `class Registry + def names(cops) + cops.map { |cop| cop.cop_name } + end +end +`, + // A namespaced class whose instance method comes first. + 'app/services/billing/payout.rb': `module Billing + class Payout + def call = 1 + def self.call(**kw) = new.call + end +end +`, + 'app/services/checkout/start.rb': `module Checkout + class Start + def call + card = Billing::ChargeCard.new(amount: 3) + card.call + Billing::Payout.call(amount: 4) + payout = Billing::Payout.new + payout.call + end + end +end +`, +}; + +let root = ''; +let cg: CodeGraph | undefined; +let savedKernel: string | undefined; + +afterEach(() => { + cg?.close(); + cg = undefined; + if (root) fs.rmSync(root, { recursive: true, force: true }); + root = ''; + if (savedKernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = savedKernel; +}); + +async function index(backend: string): Promise { + savedKernel = process.env.CODEGRAPH_KERNEL; + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-ruby-singleton-')); + for (const [rel, content] of Object.entries(FILES)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); + return cg; +} + +/** `:` of the method each `calls` edge written on `line` of `file` reaches. */ +function callsAt(graph: CodeGraph, file: string, line: number): string[] { + const ids = graph.getNodesInFile(file).map((n) => n.id); + return graph.getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === 'calls' && e.line === line) + .map((e) => { const t = graph.getNode(e.target)!; return `${t.qualifiedName}:${t.startLine}`; }) + .sort(); +} + +const backends = [ + ['kernel', getKernel() === null], + ['wasm', false], +] as const; + +describe.each(backends)('Ruby class methods (%s)', (backend, skip) => { + it.skipIf(skip)('are static, and instance methods are not', async () => { + const graph = await index(backend); + const statics = graph.getNodesInFile('app/widget.rb') + .filter((n) => n.kind === 'method') + .map((n) => `${n.name}:${n.startLine}:${n.isStatic === true ? 'static' : 'instance'}`) + .sort(); + expect(statics).toEqual([ + 'build:2:instance', + 'build:6:static', + 'build_all:15:static', + 'go:25:static', + 'make:11:static', + 'run:20:instance', + ]); + }); + + it.skipIf(skip)('a call on the class reaches its class method', async () => { + const graph = await index(backend); + expect(callsAt(graph, 'app/user.rb', 5)).toEqual(['Widget::build:6']); + expect(callsAt(graph, 'app/user.rb', 6)).toEqual(['Widget::make:11']); + expect(callsAt(graph, 'app/widget.rb', 22)).toEqual(['Widget::build:6']); + expect(callsAt(graph, 'app/services/checkout/start.rb', 6)).toEqual(['Billing::Payout::call:4']); + }); + + it.skipIf(skip)('a call on an instance reaches its instance method', async () => { + const graph = await index(backend); + expect(callsAt(graph, 'app/user.rb', 4)).toEqual(['Widget::build:2']); + expect(callsAt(graph, 'app/services/checkout/start.rb', 5)).toEqual(['Billing::ChargeCard::call:5']); + expect(callsAt(graph, 'app/services/checkout/start.rb', 8)).toEqual(['Billing::Payout::call:3']); + }); + + it.skipIf(skip)('a bare call reaches the method of the calling method kind', async () => { + const graph = await index(backend); + expect(callsAt(graph, 'app/widget.rb', 21)).toEqual(['Widget::build:2']); + expect(callsAt(graph, 'app/widget.rb', 26)).toEqual(['Widget::build:6']); + expect(callsAt(graph, 'app/widget.rb', 16)).toEqual(['Widget::build:6']); + }); + + it.skipIf(skip)('a receiver known by its name alone splits nothing: it may hold the class', async () => { + const graph = await index(backend); + expect(callsAt(graph, 'app/registry.rb', 3)).toEqual(['Cop::cop_name:2']); + }); + + it.skipIf(skip)('a bare call with no method of the other kind by its name keeps its target', async () => { + const graph = await index(backend); + expect(callsAt(graph, 'app/spec_like.rb', 4)).toEqual(['SpecLike::mock_app:8']); + }); +}); diff --git a/codegraph-kernel/src/ruby.rs b/codegraph-kernel/src/ruby.rs index 101a3271ff..96e004e59a 100644 --- a/codegraph-kernel/src/ruby.rs +++ b/codegraph-kernel/src/ruby.rs @@ -17,7 +17,7 @@ use crate::buffers::{ build_meta, edge_kind_index, node_kind_index, Arena, BoolFlags, EdgeRow, EmitOut, NodeRow, - RefRow, StrRef, Tables, FLAG_IS_EXPORTED, FUNCTION_REF_CODE, NONE, NONE_STR, + RefRow, StrRef, Tables, FLAG_IS_EXPORTED, FLAG_IS_STATIC, FUNCTION_REF_CODE, NONE, NONE_STR, REF_FLAG_FILE_PATH, }; use crate::docstring::preceding_docstring; @@ -97,6 +97,9 @@ struct Extra { docstring: Option, signature: Option, visibility: Option, + /// The isStatic hook (languages/ruby.ts): set on every def, true for a + /// class method (is_class_method). + is_static: Option, } struct ValueScope<'t> { @@ -283,10 +286,15 @@ impl<'t> Walker<'t> { let id_ref = self.arena.put(&id); let doc_ref = opt_str(&mut self.arena, extra.docstring.as_deref()); let sig_ref = opt_str(&mut self.arena, extra.signature.as_deref()); + // No isExported/isAsync hooks; isStatic only for defs. + let mut flags = BoolFlags::default(); + if let Some(v) = extra.is_static { + flags.set(FLAG_IS_STATIC, v); + } let row = self.tables.push_node(&NodeRow { kind: node_kind_index(kind).unwrap(), visibility: extra.visibility.unwrap_or(0), - flags: BoolFlags::default(), // no isExported/isAsync/isStatic hooks + flags, start_line, end_line, start_column: self.col_of(node), @@ -615,6 +623,7 @@ impl<'t> Walker<'t> { docstring: preceding_docstring(node, self.src), signature: None, // no getSignature hook visibility: Some(self.visibility_of(node)), + is_static: Some(is_class_method(node)), }; let Some(row) = self.create_node("function", &name, node, extra) else { return }; // (ruby ∉ TYPE_ANNOTATION_LANGUAGES; decorators are a structural no-op) @@ -633,6 +642,7 @@ impl<'t> Walker<'t> { docstring: preceding_docstring(node, self.src), signature: None, visibility: Some(self.visibility_of(node)), + is_static: Some(is_class_method(node)), }; let Some(row) = self.create_node("method", &name, node, extra) else { return }; self.stack.push(Scope { row, kind: "method", name }); @@ -649,6 +659,7 @@ impl<'t> Walker<'t> { docstring: preceding_docstring(node, self.src), signature: None, visibility: Some(self.visibility_of(node)), + ..Extra::default() }; let Some(row) = self.create_node("class", &name, node, extra) else { return }; @@ -692,7 +703,7 @@ impl<'t> Walker<'t> { } let name = self.text(left).to_string(); let signature = right.map(|r| util::init_signature(self.text(r))); - self.create_node("variable", &name, node, Extra { docstring, signature, visibility: None }); + self.create_node("variable", &name, node, Extra { docstring, signature, ..Extra::default() }); } /// extractImport — every non-body `call` lands here (importTypes:['call']). @@ -1136,6 +1147,22 @@ fn is_constant_path(node: Node) -> bool { } } +/// isRubyClassMethod (languages/ruby.ts): `def self.x` / `def Widget.x` (a +/// singleton_method), or a `def` inside `class << self`. +fn is_class_method(node: Node) -> bool { + match node.kind() { + "singleton_method" => true, + "method" => { + let mut parent = node.parent(); + if parent.map_or(false, |p| p.kind() == "body_statement") { + parent = parent.and_then(|p| p.parent()); + } + parent.map_or(false, |p| p.kind() == "singleton_class") + } + _ => false, + } +} + /// The Ruby `call` node whose argument_list (or keyword pair) contains `node` /// — nearest `call` ancestor within 4 parent hops (function-ref.ts:837). fn ruby_enclosing_call(node: Node) -> Option { diff --git a/docs/design/ruby-kernel-port-checklist.md b/docs/design/ruby-kernel-port-checklist.md index 120e40ec65..26f1993e6c 100644 --- a/docs/design/ruby-kernel-port-checklist.md +++ b/docs/design/ruby-kernel-port-checklist.md @@ -253,14 +253,18 @@ Hooks PRESENT (port each exactly): (see extractImport below). Hooks ABSENT (the walker must NOT do these): `preParse`, `resolveName`, -`recoverMangledName`, `isMisparsedFunction`, `isConst`, `isStatic`, +`recoverMangledName`, `isMisparsedFunction`, `isConst`, `isExported`, `isAsync`, `getSignature`, `getReturnType`, `getReceiverType`, `resolveBody`, `classifyClassNode`, `classifyMethodNode`, `extractPropertyName`, `interfaceKind`, `extraClassNodeTypes`, `packageTypes`/`extractPackage`, `extractModifiers`, `synthesizeMembers`, `skipBodilessClass`, `methodsAreTopLevel`. Consequences: every ruby -function/method node has signature/isAsync/isStatic/returnType/isExported -**undefined** (file node isExported:false); **no isConst means every +function/method node has signature/isAsync/returnType/isExported +**undefined** (file node isExported:false). `isStatic` IS a hook +(isRubyClassMethod / `is_class_method`): every function/method node carries +it — true for a `singleton_method` (`def self.x`, `def Foo.x`) and for a +`method` whose parent, past one `body_statement`, is a `singleton_class` +(`class << self`), false for every other def (a nested def included). **no isConst means every `assignment` extracts kind `'variable'`, never `'constant'` — including `MAX = 3`** (value-ref targeting still works: kind variable is a target); `attr_accessor`/`attr_reader`/`attr_writer` synthesize NOTHING (no @@ -279,9 +283,9 @@ non-decorator sibling — always a no-op). | every node | visitNode hook first (943) | mixin calls + modules handled there (above); handled → scanFnRefSubtree + STOP | | every node | maybeCaptureFnRefs (990) | fires for `argument_list` / `pair` (the RUBY_SPEC dispatch keys) in visitNode context too — this is how class-body hook-DSL symbols are captured | | `method` | functionTypes:994 | inside class-like (class/module/…, isInsideClassLikeNode:1486 — **`module` counts**, 1498) AND in methodTypes → extractMethod:1737; else extractFunction:1517. skipChildren | -| `singleton_method` (`def self.x` / `def Foo.x` / `def obj.x`) | NOT functionTypes → methodTypes:1027 | no classifyMethodNode → extractMethod. At top level: not class-like, no methodsAreTopLevel, no receiver hook → gate 1747 sends it to extractFunction → a top-level `def self.x` is a plain **function** node named x. `object` field ignored everywhere — `def self.x` vs `def x` are indistinguishable in the graph (isStatic undefined) | +| `singleton_method` (`def self.x` / `def Foo.x` / `def obj.x`) | NOT functionTypes → methodTypes:1027 | no classifyMethodNode → extractMethod. At top level: not class-like, no methodsAreTopLevel, no receiver hook → gate 1747 sends it to extractFunction → a top-level `def self.x` is a plain **function** node named x. `object` field ignored everywhere; `def self.x` and `def x` share a qualified name, and only isStatic (true for the singleton_method) tells them apart | | `class` | classTypes:1005 | no classifyClassNode → extractClass:1679 (kind 'class') | -| `singleton_class` (`class << self`) | NO branch | recursed → its `body` body_statement's defs extract as methods of the OUTER class, indistinguishable from instance methods (probed) | +| `singleton_class` (`class << self`) | NO branch | recursed → its `body` body_statement's defs extract as methods of the OUTER class, under the same qualified names as instance methods, with isStatic true (probed) | | `assignment` (top level / class / module) | variableTypes:1098 | gate: `!isInsideClassLikeNode() \|\| isClassScopeConstantAssignment` (1508: type==='assignment' AND (`left` field ?? namedChild(0)).type === `'constant'`). File scope: identifier AND constant LHS both extract. Class/module scope: ONLY constant-LHS. Then extractVariable:2538 + scanFnRefSubtree (1110) + skipChildren — **the RHS is never walked: top-level/class-level `X = Foo.new` emits NO instantiates/calls** | | `operator_assignment` (`+=`, `\|\|=`) | no branch | not `assignment` → nothing extracted at any scope; recursed (children emit nothing either at non-body scope) | | `call` (top level / class / module body) | **importTypes:1209 — NEVER callTypes** | extractImport:3170 (below). require/require_relative → import node + refs; **every other call → NOTHING** (hook null → 3350 `if (this.extractor.extractImport) return;`). skipChildren stays false → children ARE visited: nested calls also land here (nothing), argument_lists get fn-ref capture. This kills `attr_accessor`, `has_many`, `define_method` (+ its do_block body!), `get '/x' do…end` route blocks, bare DSL calls — **all invisible to extraction at non-body scope** except fn-ref candidates + the rails regex extractor | @@ -330,7 +334,7 @@ extractFilePackage:1397 (no packageTypes → no namespace node); `value=` — keep the suffix in the name). `` path (1549) can't trigger (grammar requires a name). No misparse hook. Node extras: docstring (getPrecedingDocstring), signature undefined, visibility (sibling scan!), - isExported/isAsync/isStatic/returnType undefined. extractTypeAnnotations → + isExported/isAsync/returnType undefined, isStatic from the hook (above). extractTypeAnnotations → no-op (ruby ∉ TYPE_ANNOTATION_LANGUAGES:5752). extractDecoratorsFor → no-op. Push, walk `body` field (`body_statement`) via visitFunctionBody, pop. **`parameters` (method_parameters) are NEVER walked — a call in a diff --git a/src/extraction/languages/ruby.ts b/src/extraction/languages/ruby.ts index 8679a15c1a..f053fa8162 100644 --- a/src/extraction/languages/ruby.ts +++ b/src/extraction/languages/ruby.ts @@ -2,6 +2,20 @@ import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText, getChildByField } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; +/** + * A class method: `def self.x` or `def Widget.x` (a singleton_method), or a + * `def` inside `class << self`. It shares its qualified name with an instance + * method of the same name (`Widget::build` for both), so only this tells the + * two apart. + */ +function isRubyClassMethod(node: SyntaxNode): boolean { + if (node.type === 'singleton_method') return true; + if (node.type !== 'method') return false; + let parent = node.parent; + if (parent?.type === 'body_statement') parent = parent.parent; + return parent?.type === 'singleton_class'; +} + export const rubyExtractor: LanguageExtractor = { functionTypes: ['method'], classTypes: ['class'], @@ -103,6 +117,7 @@ export const rubyExtractor: LanguageExtractor = { return name; }, + isStatic: isRubyClassMethod, getVisibility: (node) => { // Ruby visibility is based on preceding visibility modifiers let sibling = node.previousNamedSibling; diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 55fbe56008..122dece68b 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -7830,6 +7830,33 @@ function isRubyMethodInScope(method: Node, ref: UnresolvedRef, context: Resoluti return rubyAncestry(here.qualifiedName, context).has(method.qualifiedName.slice(0, cut)); } +/** + * Of a Ruby class's two methods of one name, `def build` and `def self.build` + * (both `Widget::build`), the one a bare call means: a bare call is a call on + * `self`, so in an instance method it is the instance method, and in a class + * method (`def self.x`, `class << self`) the class method. Widget's `run` + * calling `build` reaches `def build`, and its `def self.go` calling `build` + * reaches `def self.build`. Only such pairs are split: inside a block that + * runs on another object (`Module.new do … end`, a test's `it do … end`) the + * caller's kind says nothing of `self`, and any other candidate stays as it + * was. A call written anywhere but in a method splits nothing. + */ +function rubyMethodsOfCallerKind(candidates: Node[], ref: UnresolvedRef, context: ResolutionContext): Node[] { + if (candidates.length < 2) return candidates; + const caller = context.getNodeById?.(ref.fromNodeId); + if (!caller || caller.kind !== 'method' || caller.language !== 'ruby') return candidates; + const wantClass = caller.isStatic === true; + const isRubyMethod = (n: Node): boolean => n.kind === 'method' && n.language === 'ruby'; + // Per qualified name, which kinds the candidates declare: 1 class, 2 instance. + const kinds = new Map(); + for (const n of candidates) { + if (isRubyMethod(n)) kinds.set(n.qualifiedName, (kinds.get(n.qualifiedName) ?? 0) | (n.isStatic === true ? 1 : 2)); + } + const twinOfOtherKind = (n: Node): boolean => + isRubyMethod(n) && kinds.get(n.qualifiedName) === 3 && (n.isStatic === true) !== wantClass; + return candidates.some(twinOfOtherKind) ? candidates.filter((n) => !twinOfOtherKind(n)) : candidates; +} + /** A Ruby class's qualified name, those of its superclasses, and of every module mixed into any of them. */ export function rubyAncestry(qn: string, context: ResolutionContext): Set { let memo = RUBY_ANCESTRY.get(context); @@ -11950,7 +11977,8 @@ export function matchByExactName( const ranked = dartBare ? nearestDartMembers(preferOwnDartLibrary(filtered, ref, context), ref, context) : dartTypeRef && dartPrefix === null ? preferOwnDartLibrary(filtered, ref, context) : swiftShape && swiftShape.shape !== 'chained' ? nearestSwiftMembers(filtered, ref, context) - : kotlinBare ? lexicalKotlinMembers(filtered, ref, context) : cppSpecializationPool(filtered, ref, context); + : kotlinBare ? lexicalKotlinMembers(filtered, ref, context) + : rubyBare ? rubyMethodsOfCallerKind(filtered, ref, context) : cppSpecializationPool(filtered, ref, context); // (Above the ceiling below nothing is picked, so no translation unit is read.) const candidates = ranked.length <= AMBIGUOUS_NAME_CEILING ? withinCppTranslationUnit(ranked, ref, context, true) : ranked; @@ -13125,7 +13153,11 @@ export function resolveMethodOnType( // intentionally in ANOTHER file (#314) — is unaffected: those blocks return // early whenever an import pins the class. if (ref.referenceKind === 'function_ref' && matches.length !== 1) return null; - const ordered = preferCallSiteFile(withinCppTranslationUnit(cppVisibleFirst(matches, ref, context), ref, context), ref.filePath); + // A Ruby receiver is typed only by `x = Widget.new`, so it is an instance: + // `x.build` reaches `def build`, not `def self.build`, wherever each is. + const ordered = rubyMethodsOfKindFirst( + preferCallSiteFile(withinCppTranslationUnit(cppVisibleFirst(matches, ref, context), ref, context), ref.filePath), + ref.language === 'ruby' ? 'instance' : undefined); return { original: ref, targetNodeId: ordered[0]!.id, @@ -18609,18 +18641,18 @@ function tsClassHead(lines: readonly string[], cls: Node): string { } /** A method named `name` on a supertype of the given classes, nearest first. */ -function inheritedClassMethod(classes: Node[], name: string, context: ResolutionContext): Node | null { +function inheritedClassMethod(classes: Node[], name: string, context: ResolutionContext, kind?: RubyMethodKind): Node | null { if (classes.length === 0) return null; let memo = INHERITED_METHODS.get(context); if (!memo) INHERITED_METHODS.set(context, (memo = new Map())); - const key = `${classes.map((c) => c.id).join(',')}|${name}`; + const key = `${classes.map((c) => c.id).join(',')}|${name}|${kind ?? ''}`; if (memo.has(key)) return memo.get(key)!; - const found = findInheritedClassMethod(classes, name, context); + const found = findInheritedClassMethod(classes, name, context, kind); memo.set(key, found); return found; } -function findInheritedClassMethod(classes: Node[], name: string, context: ResolutionContext): Node | null { +function findInheritedClassMethod(classes: Node[], name: string, context: ResolutionContext, kind?: RubyMethodKind): Node | null { const seen = new Set(classes.map((c) => c.id)); let frontier = classes; for (let depth = 0; depth < 5 && frontier.length > 0; depth++) { @@ -18630,8 +18662,8 @@ function findInheritedClassMethod(classes: Node[], name: string, context: Resolu for (const decl of context.getNodesByName(sup)) { if (decl.language !== cls.language || !isMethodOwnerKind(decl) || seen.has(decl.id)) continue; seen.add(decl.id); - const method = context.getNodesInFile(decl.filePath).find((n) => n.kind === 'method' && n.name === name && - n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))).split(/::|\./).pop() === decl.name); + const method = rubyMethodsOfKindFirst(context.getNodesInFile(decl.filePath).filter((n) => n.kind === 'method' && n.name === name && + n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))).split(/::|\./).pop() === decl.name), kind)[0]; if (method) return method; next.push(decl); } @@ -20242,7 +20274,9 @@ export function matchMethodCall( return { original: ref, targetNodeId: method.id, confidence: own ? 0.85 : 0.8, resolvedBy: 'qualified-name' }; } - const methodNode = classStrategyMethod(classNode, methodName!, context); + // A Ruby call on a class is a call of its class method: `Widget.build` + // reaches `def self.build`, not `def build`, whichever comes first. + const methodNode = classStrategyMethod(classNode, methodName!, context, ref.language === 'ruby' ? 'class' : undefined); if (methodNode) { return { @@ -20255,7 +20289,8 @@ export function matchMethodCall( } // A class method the named class inherits — Horse's `THorse.Get(…)` is // THorseCore's, three `class(…)` heads up — before any guess by name. - const inherited = inheritedClassMethod(classCandidates.filter((c) => c.language === ref.language), methodName!, context); + const inherited = inheritedClassMethod(classCandidates.filter((c) => c.language === ref.language), methodName!, context, + ref.language === 'ruby' ? 'class' : undefined); if (inherited) return { original: ref, targetNodeId: inherited.id, confidence: 0.8, resolvedBy: 'qualified-name' }; return null; }); @@ -20344,6 +20379,10 @@ export function matchMethodCall( if (!isLexicallyReachable(classNode, ref, context)) continue; if (productionCaller && isTestPath(classNode.filePath)) continue; + // A Ruby receiver named after a class may hold the class as well as an + // instance of it (rubocop's `cop.cop_name` runs over its registry's + // cop classes, so it is `def self.cop_name`), so neither of a class's + // two methods of the name is preferred: the first declared is taken. const methodNode = classStrategyMethod(classNode, methodName!, context); if (methodNode) { @@ -25688,9 +25727,12 @@ function tsFieldCall( * the caller's loop, so a nested `Outer::Logger` can take a top-level * `Logger.log()` (declared first, or the only one with the method). Choosing * among them needs each language's scope rules. + * + * `kind` picks between a Ruby class's two methods of the name, `def self.m` + * and `def m`, which share one qualified name (rubyMethodsOfKindFirst). */ -function findOwnMethod(nodesInFile: Node[], classNode: Node, methodName: string): Node | undefined { - const methods = nodesInFile.filter((n) => n.kind === 'method' && n.name === methodName); +function findOwnMethod(nodesInFile: Node[], classNode: Node, methodName: string, kind?: RubyMethodKind): Node | undefined { + const methods = rubyMethodsOfKindFirst(nodesInFile.filter((n) => n.kind === 'method' && n.name === methodName), kind); const exact = `${classNode.qualifiedName}::${methodName}`; const byName = `${classNode.name}::${methodName}`; return ( @@ -25706,9 +25748,30 @@ function findOwnMethod(nodesInFile: Node[], classNode: Node, methodName: string) * `Form` reaches `BaseForm::is_valid`, which the substring test this replaced * only found because `BaseForm` contains `Form`. */ -function classStrategyMethod(classNode: Node, methodName: string, context: ResolutionContext): Node | undefined { - return findOwnMethod(context.getNodesInFile(classNode.filePath), classNode, methodName) ?? - inheritedClassMethod([classNode], methodName, context) ?? undefined; +function classStrategyMethod(classNode: Node, methodName: string, context: ResolutionContext, kind?: RubyMethodKind): Node | undefined { + return findOwnMethod(context.getNodesInFile(classNode.filePath), classNode, methodName, kind) ?? + inheritedClassMethod([classNode], methodName, context, kind) ?? undefined; +} + +/** + * Which of a Ruby class's two methods of one name a call reaches: a call on + * the class (`Widget.build`) its class method, `def self.build` or one in + * `class << self`; a call on an instance (`widget.build`) its instance + * method, `def build`. Both share one qualified name, `Widget::build`, and + * only `isStatic` tells them apart. + */ +export type RubyMethodKind = 'class' | 'instance'; + +/** + * `methods` with the ones of `kind` first, in their own order: a pick of the + * first then takes one of the kind wherever there is one, and still the + * first otherwise. Methods of other languages are never static-split. + */ +export function rubyMethodsOfKindFirst(methods: T[], kind: RubyMethodKind | undefined): T[] { + if (!kind || methods.length < 2) return methods; + const fits = (m: T): boolean => m.language === 'ruby' && (m.isStatic === true) === (kind === 'class'); + const first = methods.filter(fits); + return first.length === 0 || first.length === methods.length ? methods : [...first, ...methods.filter((m) => !fits(m))]; } /** diff --git a/src/resolution/ruby-constant-paths.ts b/src/resolution/ruby-constant-paths.ts index b77e34fa24..8d6925912c 100644 --- a/src/resolution/ruby-constant-paths.ts +++ b/src/resolution/ruby-constant-paths.ts @@ -20,7 +20,7 @@ import type { Node } from '../types'; import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; import { isTestPath } from '../search/query-utils'; -import { rubyAncestry } from './name-matcher'; +import { rubyAncestry, rubyMethodsOfKindFirst } from './name-matcher'; /** `A::B`, `::A`, `::A::B`: a constant path. Group 1 is a leading `::`. */ const CONSTANT_PATH = /^(::)?([A-Z]\w*(?:::[A-Z]\w*)*)$/; @@ -98,14 +98,16 @@ function lexicalScope(ref: UnresolvedRef, context: ResolutionContext): string { /** * The method `name` of the class or module `owner`: its own, else the first * of its superclasses and mixed-in modules that declares one (rubyAncestry). - * Production code never reaches one a test declares. + * Of a class's `def self.m` and `def m`, which share one qualified name, the + * call on the class reaches the class method. Production code never reaches + * one a test declares. */ function methodOf(owner: string, name: string, ref: UnresolvedRef, context: ResolutionContext): { node: Node; own: boolean } | null { const production = !isTestPath(ref.filePath); for (const qn of rubyAncestry(owner, context)) { const methods = context.getNodesByQualifiedName(`${qn}::${name}`).filter((n) => n.kind === 'method' && n.language === 'ruby' && !(production && isTestPath(n.filePath))); - if (methods.length > 0) return { node: ordered(methods, ref)[0]!, own: qn === owner }; + if (methods.length > 0) return { node: rubyMethodsOfKindFirst(ordered(methods, ref), 'class')[0]!, own: qn === owner }; } return null; } From 51c4124daddd788f622f140abf69ea5abad3f956 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 08:47:14 +0000 Subject: [PATCH 63/95] fix(cfml): a dotted extends finds its component when a test suite mirrors its path (#2835) A CFML component that extends another by its dotted path, like extends="contenscms.core.basecore", got no extends edge when a test suite held a file of the same name in a folder of the same name, such as a TestBox spec at tests/testbox/tests/specs/unit/core/basecore.cfc. resolveCfmlComponentPath counts the path's folders that match a candidate's, from the right. contenscms is a server mapping onto contens/, so only core matched, for the component and the spec alike, and a tie gives no edge. The rule the resolver applies elsewhere since PR #2209, that production code never names a symbol in a test suite, never ran on this path: it returns before the name matcher's visibility gate. Fix: - A component in a test suite is no candidate for a reference from production code (isTestSuitePath, now exported from the name matcher). This also keeps application code from extending a test file that happens to match more folders. - A test's tie between a component and a suite file that mirrors its path goes to the component outside the test suites. Two production components still tie. Validation: - cfml-inheritance-resolution.test.ts: 2 new cases (the issue's tree in script and tag form, a test extending the component, a mirror that matches more folders, a path only a test suite holds, a test naming its own suite) fail on main's resolver; all 12 pass with the fix. - The issue's repro: page and tagpage now extend contens/core/basecore.cfc (qualified-name 0.9); on main both stayed unresolved. - A/B vs main 8fdc8a1e (CFML has no kernel path): FW/1 (305 CFML files), BlogCFC (167), ColdBox (668), ContentBox (726), Preside (1,791) and CFWheels (the repo, and its vendor/wheels core of 1,528 CFML files as a project of its own): no node, edge, reference or file error changed. None has a component and a test-suite file that tie on a dotted path. express and flask controls: full dumps byte-identical (kernel path). - Index time, median of 3 interleaved runs, CPU with workers, main vs fix: Preside 63.6 / 60.4 s, Wheels core 35.5 / 32.7 s. Within noise (main alone ranged 59-67 s and 35-38 s). Fixes #2650 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cfml-inheritance-resolution.test.ts | 34 +++++++++++++++++++ src/resolution/index.ts | 27 ++++++++++----- src/resolution/name-matcher.ts | 2 +- 4 files changed, 55 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9aaca65a56..b879cfdd58 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -131,6 +131,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph sync` and the file watcher now link a route to the page or layout it loads lazily when that file is added after the router was indexed, comes back after being deleted, or gets the component the route renders in a later edit, whether or not the import names the file's extension: React Router's `lazy: () => import('./pages/Team')`, Vue Router's `component: () => import('@/views/Login')` and Angular's `loadComponent: () => import('./home/home.component')`. Before, the route stayed unlinked until the router file changed or the project was re-indexed, so the page looked unused and `codegraph affected` missed the tests that reach it through the router. Re-index after upgrading to pick up links an earlier version missed. - A file named the way Google names its tests, like `wire_format_unittest.cc` in protocolbuffers/protobuf, `logging_unittest.cc` in google/glog or a Python `foo_unittest.py`, now counts as a test, as `wire_format_test.cc` already did. Before, it was taken for production code: search and `codegraph_explore` ranked it alongside the code it tests, `codegraph affected` never listed it, and calls in production code could link to a class or type a unittest declares for itself, as google/breakpad's `stack_frame_entries_.size()` did to a unittest's `StackHelper`. Re-index projects that have such files after upgrading. - A file in a `unittests/` folder, where llvm/llvm-project and google/breakpad keep their unit tests, or in a folder named like a unittest, such as `dcheck_unittest/` in google/glog, now counts as a test whatever its own name says, like LLVM's `CountCopyAndMove.h` helper. Before, such a file was taken for production code unless its name said test, and none of them were kept apart from the program the way a `tests/` folder is: search and `codegraph_explore` ranked them alongside production code, `codegraph affected` missed them, and production code linked to a unit test's own declarations, as LLVM's `Ctx` and `OpInfo` variables did to ones its unit tests declare. A folder named just `unittest/`, like CPython's `Lib/unittest/`, is still production code. Re-index projects that have such folders after upgrading. +- In CFML, a dotted `extends` like `extends="contenscms.core.basecore"` now links to the component it names when a test suite mirrors that component's folder and file name, such as a TestBox spec at `tests/specs/unit/core/basecore.cfc`, and application code is never linked to a component inside a test suite. Before, the two files tied, so the base class was missing from the graph, and so was everything the component inherits from it. Thanks @HarryMuc for the report. (#2650) - `codegraph_explore` now leaves out of its answers every file that search already treats as a test or an example: a C++ `FooTest.cpp` or `foo_test.cc`, a Google `foo_unittest.cc`, anything in an LLVM-style `unittests/` folder or an `e2e/` or `testing/` folder, and a project's `examples/`, `samples/` and `benchmarks/`. Before, explore went by a shorter list of its own, so a unit test or an example that called the code you asked about could take a place in the answer meant for that code: in protocolbuffers/protobuf, a question about `DescriptorPool` came back with three of its unit tests. The same files are no longer taken for a project's main routes file either, such as an end-to-end suite's mock server. No re-index is needed. - `codegraph sync` and the file watcher now link a file to a module it imports under a name of its own when the module is added after the file was indexed, comes back after being deleted, or gets the export in a later edit: a default import like `import tagsController from './tag/tag.controller'`, a namespace import like `import * as api from './api'`, or an aliased one like `import { Component as GroupsPageWrapper } from './groups_page'`, in JavaScript, TypeScript, Vue, Svelte and Astro. Before, the import and every use of the name, such as a call, `new`, a base class, a JSX or template tag or an Express `app.use` mount, stayed unlinked until the importing file changed or the project was re-indexed, so the module looked unused and `codegraph affected` missed the files that use it. Re-index after upgrading to pick up links an earlier version missed. - `codegraph sync` and the file watcher now also link a call or read on a name a file imports, like `NODE_PATH_EXCLUDELIST.includes(path)` after `import { NODE_PATH_EXCLUDELIST } from './constants'`, `store.get(key)` through a default import or `api.get()` through a namespace import, when the module gets the export in a later edit, is added after the importing file was indexed, or comes back after being deleted, in JavaScript, TypeScript, Vue, Svelte and Astro. Before, such a call stayed unlinked until the importing file changed or the project was re-indexed, because sync looked it up only by its method's name, which the module seldom declares, and skipped a name as common as `includes`, `get` or `map` altogether. Re-index after upgrading to pick up links an earlier version missed. diff --git a/__tests__/cfml-inheritance-resolution.test.ts b/__tests__/cfml-inheritance-resolution.test.ts index e1a815ebf4..879da7f900 100644 --- a/__tests__/cfml-inheritance-resolution.test.ts +++ b/__tests__/cfml-inheritance-resolution.test.ts @@ -90,6 +90,40 @@ describe('CFML component-path inheritance resolution (#1152)', () => { expect(edges.filter((e) => e.src === 'Main')).toHaveLength(0); }); + it('passes over a test suite that mirrors the component path (#2650)', async () => { + // `contenscms` is a server mapping onto `contens/`, so only `core` + // corroborates either file: the component and the TestBox spec named + // after it tied, and a tie yields no edge. + write('contens/core/basecore.cfc', `component {\n function load() { return 1; }\n}\n`); + write('tests/testbox/tests/specs/unit/core/basecore.cfc', `component extends="testbox.system.BaseSpec" {}\n`); + write('contens/model/page.cfc', `component extends="contenscms.core.basecore" {}\n`); + write('contens/model/tagpage.cfc', `\n\n\n`); + // A test that names the component means the component too. + write('tests/testbox/tests/specs/unit/model/fakecore.cfc', `component extends="contenscms.core.basecore" {}\n`); + const edges = await load(); + expect(has(edges, 'page', 'basecore', 'contens/core/basecore.cfc')).toBe(true); + expect(has(edges, 'tagpage', 'basecore', 'contens/core/basecore.cfc')).toBe(true); + expect(has(edges, 'fakecore', 'basecore', 'contens/core/basecore.cfc')).toBe(true); + expect(edges.filter((e) => e.tgtFile.startsWith('tests/'))).toHaveLength(0); + }); + + it('never resolves production code into a test suite, even on a closer path (#2650)', async () => { + write('contens/core/basecore.cfc', `component {}\n`); + // The mirror repeats `contenscms/core`, so it corroborates more segments. + write('tests/unit/contenscms/core/basecore.cfc', `component {}\n`); + write('contens/model/page.cfc', `component extends="contenscms.core.basecore" {}\n`); + // Only a test suite holds `contenscms.core.queue`: no edge from production code. + write('tests/specs/core/queue/worker.cfc', `component {}\n`); + write('contens/model/job.cfc', `component extends="contenscms.core.queue.worker" {}\n`); + // A test naming the suite's own path still reaches it. + write('tests/specs/core/queue/workerTest.cfc', `component extends="tests.specs.core.queue.worker" {}\n`); + const edges = await load(); + expect(has(edges, 'page', 'basecore', 'contens/core/basecore.cfc')).toBe(true); + expect(edges.filter((e) => e.src === 'page')).toHaveLength(1); + expect(edges.filter((e) => e.src === 'job')).toHaveLength(0); + expect(has(edges, 'workerTest', 'worker', 'tests/specs/core/queue/worker.cfc')).toBe(true); + }); + it('resolves a relative path against the referencing file (the FW/1 shape)', async () => { write('examples/base.cfc', `component {\n function shared() { return 1; }\n}\n`); write('examples/sub/app.cfc', `component extends="../base" {}\n`); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 816b020de8..01e363cace 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, isRustPathCallReachable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, matchPythonTypedLocalCall, isImportShadowedByJsBinding, isManualMockPath } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, isRustPathCallReachable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, goDotImportedSupertype, GO_TYPE_KINDS, matchGoAssertedCall, matchGoIndexedCall, matchGoChainedCall, preferGoBuildTwin, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves, matchPythonTypedLocalCall, isImportShadowedByJsBinding, isManualMockPath, isTestSuitePath } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -3498,6 +3498,12 @@ export class ReferenceResolver { * edge. Directory comparison is case-insensitive (CFML path resolution is); * the class segment itself is matched exactly, which real code satisfies — * dotted paths are written to match the on-disk file name. + * + * A test suite is no candidate for production code (#2209's rule), and on a + * tie it yields to the one component outside the test suites (#2650): a + * TestBox spec mirroring the component's path (`tests/specs/unit/core/ + * basecore.cfc` beside `core/basecore.cfc`) corroborates the same segments, + * and a mapping root like `contenscms` (onto `contens/`) matches neither. */ private resolveCfmlComponentPath(ref: UnresolvedRef): ResolvedRef | null { const cfmlCandidates = (name: string): Node[] => @@ -3541,10 +3547,13 @@ export class ReferenceResolver { const className = segments[segments.length - 1]!; const dirSegments = segments.slice(0, -1); - let best: Node | null = null; + // Production code never names a component in a test suite; a test can + // name one, by the path to it. + const fromTest = isTestPath(ref.filePath); let bestScore = 0; - let tie = false; + let tied: Node[] = []; for (const cand of cfmlCandidates(className)) { + if (!fromTest && isTestSuitePath(cand.filePath)) continue; const dirs = cand.filePath.replace(/\\/g, '/').split('/').slice(0, -1); // Count matching directory segments right-to-left: for // `coldbox.system.web.Controller` vs `system/web/Controller.cfc`, @@ -3559,15 +3568,17 @@ export class ReferenceResolver { score++; } if (score > bestScore) { - best = cand; bestScore = score; - tie = false; + tied = [cand]; } else if (score === bestScore && score > 0) { - tie = true; + tied.push(cand); } } - if (!best || bestScore === 0 || tie) return null; - return { original: ref, targetNodeId: best.id, confidence: 0.9, resolvedBy: 'qualified-name' }; + // A test's tie between a component and a suite mirroring its path: the + // path names the component, as it does from production code. + if (tied.length > 1) tied = tied.filter((c) => !isTestSuitePath(c.filePath)); + if (bestScore === 0 || tied.length !== 1) return null; + return { original: ref, targetNodeId: tied[0]!.id, confidence: 0.9, resolvedBy: 'qualified-name' }; } /** diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 122dece68b..1eb0a6a69e 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1724,7 +1724,7 @@ const JVM_TYPE_KINDS: ReadonlySet = new Set(['class', 'interface', 'enum * its own code may use. A `foo_unittest/` directory is a suite, as a * `foo_unittest.cc` file is: glog builds each one as a test program of its own. */ -function isTestSuitePath(filePath: string): boolean { +export function isTestSuitePath(filePath: string): boolean { if (!isTestPath(filePath)) return false; const lower = filePath.toLowerCase(); const name = lower.slice(lower.lastIndexOf('/') + 1); From cf3e6bc525743de7a5ee771ae26448b796a6d9fa Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 09:09:05 +0000 Subject: [PATCH 64/95] fix(ts): a type-only re-export like export type * from './types' now parses (#2837) TypeScript 5.0's type-only re-export of a whole module, `export type * from './x'` and `export type * as ns from './x'`, has no rule in the vendored tree-sitter-typescript grammar (v0.23.2, the revision the kernel's crate compiles) or in upstream master: `type` parses as an ERROR inside the export statement. The graph was already right, since the import resolver reads the statement from the file (PRs #1955 and #2526), but the error tree meant a barrel of re-exports got the "parse produced no symbols" warning and a parse_error row, `codegraph status` listed it among the files it could not parse, and the native kernel deferred every file with such a line to wasm. The TypeScript extractor's preParse (.ts and .tsx) now blanks that `type` to four spaces where a statement starts (a line start, or after `;`, `{`, `}` or a block comment) when `*` is followed by `from` or `as`, so the line reads as the `export *` it is to the graph. Offsets are kept. The kernel route applies the preParse before calling the kernel, so both engines parse the same bytes and no Rust change is needed. A grammar patch would mean vendoring and regenerating the grammar for both dialects, two wasm rebuilds and a kernel off the crates.io crate, for a statement `export *` already models. A literal pre-check skips other files: 16 ms over strapi's 4,659 TS files. The only node change is an import node's stored statement text, which now reads `export * from './x'`. Import nodes are filtered from search, explore and the context builder, and nothing in resolution reads it. status-index-health.test.ts used `export type * from` as its unparseable file (issue #2336); it now uses a file with merge-conflict markers. Validation: - New ts-export-type-star.test.ts, 15 tests: 13 fail with main's typescript.ts, 11 with the blank unwired, all pass with the fix (kernel and wasm, .ts and .tsx, LF and CRLF). - A/B, kernel engine, main vs fix, dumps by natural keys: masto.js, kysely, TanStack/query, graffle and strapi keep every edge and ref (162,421 and 192,248 on strapi). The only changes are 883 import signatures, one per `export type *` statement, and 76 cleared warnings (masto.js 11, kysely 1, query 4, graffle 28, strapi 32). The Go and Python controls gin and requests are byte-identical. - Kernel parity, typescript and tsx: 0 diffs on every sweep. Deferrals drop masto.js 11 to 0, graffle 203 to 13, strapi 105 to 69, query 32 to 27, kysely 14 to 12; the rest are other grammar gaps. - Index time, median of 3 interleaved: graffle 12.7 s vs 13.1 s, query 11.2 s vs 11.4 s, within noise. Fixes #2474 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/status-index-health.test.ts | 13 +- __tests__/ts-export-type-star.test.ts | 240 +++++++++++++++++++++++++ src/extraction/languages/typescript.ts | 28 ++- 4 files changed, 277 insertions(+), 5 deletions(-) create mode 100644 __tests__/ts-export-type-star.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b879cfdd58..298b336b83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -282,6 +282,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704) - An Express route keeps its named handler when a middleware before it contains an arrow function. `rateLimit({ keyGenerator: (req) => req.ip })` or `(req, res, next) => next()` no longer hides the handler at the end of the route (re-index to update an existing project). - A TypeScript type re-exported through a barrel (`export type { Foo } from`, `export { type Foo } from`, `export type * from`) now links to its real declaration instead of a same-named type elsewhere in the project, and import lists with comments, string names, or a JSDoc `@import` are read correctly. Re-index to update an existing project. +- In TypeScript, a file that re-exports a whole module's types, like `export type * from './types'` or `export type * as Types from './types'`, now parses cleanly. Before, the parser couldn't read that line, so a barrel made of re-exports was reported as producing no symbols and `codegraph status` listed it among the files it could not parse. Re-index a project that uses it after upgrading. Thanks @gavinwilliams for the report and @danusha2345 for the pull request. (#2474) - Git sync hooks now refresh the project they were installed for, so a project inside a larger repository stays current after commit, pull and checkout; installing them no longer breaks an existing hook written in another language, and runs even when your own hook ends with `exit`. - In TypeScript and JavaScript, a call through a field the calling class inherits, like `this._store.add(…)` in a class whose base class declares `_store`, now links to the method of the field's class, with the base class found the way the subclass's file imports it, under its own name, an alias or a re-export. Before, such a call linked only when some other class with the calling class's name happened to declare the field, which could be another app's class: that last resort now needs a type the calling file imports or declares. A call on a field or variable whose type comes from a package outside your project, like `TelemetryReporter` from `@vscode/extension-telemetry`, no longer links to a project class that only shares the type's name, unless a file in your project declares that package (`declare module '…'`). Re-index TypeScript and JavaScript projects after upgrading. - In TypeScript and JavaScript, a call on a variable, parameter or field whose class the file imports, like `store.add(…)` with `store: DisposableStore`, now links to the method of the class that import names when other classes in the project share its name, such as a vendored copy, a test mock or another app's class. Before, it linked to whichever of them was indexed first, or to the one in the nearest folder: in vscode, calls on classes imported from vscode's own `vs/base` and `vs/platform` reached the copies the Copilot extension vendors or its test fixtures, in bitwarden the desktop app's `initService.init()` reached the browser extension's `InitService`, and in react-native `Keyboard` and `WebSocket` reached the Jest mock of `NativeEventEmitter`. A call whose first match was a test file's class, which then linked to nothing, now reaches the imported class too. The import is followed through re-exports, `import type`, namespace imports like `new async.Delayer()` and `require(…)`. Re-index TypeScript and JavaScript projects after upgrading. diff --git a/__tests__/status-index-health.test.ts b/__tests__/status-index-health.test.ts index b94562c8eb..7a5571e103 100644 --- a/__tests__/status-index-health.test.ts +++ b/__tests__/status-index-health.test.ts @@ -5,9 +5,9 @@ * "Up to date" used to mean only "every file's content hash matches the * index". Two kinds of file pass that check with nothing usable in the graph: * - * - a file the parser could not read — `export type * from` is valid - * TypeScript 5.0 the bundled grammar has no rule for, so the file is stored - * with a parse error and no symbols (#2336); + * - a file the parser could not read, like one left with merge-conflict + * markers, which is stored with a parse error and no symbols (#2336; the + * report's file, TypeScript 5.0's `export type * from`, parses since #2474); * - a row stored without its symbols — by an engine whose grammar failed to * load (#2335), or the #1541 wipe — which `codegraph sync` re-indexes. * @@ -96,7 +96,12 @@ describe('status reports files indexed without their symbols (#2336, #2335)', () it('a file the parser could not read keeps status from reporting "up to date"', async () => { fs.writeFileSync( path.join(dir, 'src', 'index.ts'), - 'export * from "./constants/index.js";\nexport type * from "./types/database.js";\n' + 'export * from "./constants/index.js";\n' + + '<<<<<<< HEAD\n' + + 'export type { Row } from "./types/database.js";\n' + + '=======\n' + + 'export type { RowId } from "./types/database.js";\n' + + '>>>>>>> feature\n' ); await index(); diff --git a/__tests__/ts-export-type-star.test.ts b/__tests__/ts-export-type-star.test.ts new file mode 100644 index 0000000000..eec3300f24 --- /dev/null +++ b/__tests__/ts-export-type-star.test.ts @@ -0,0 +1,240 @@ +/** + * TypeScript 5.0's type-only re-export of a whole module parses (#2474): + * + * export { FooClient, type FooClientOptions } from "./client.js"; + * export type * from "./types.js"; + * export type * as Shapes from "./shapes.js"; + * + * The grammar has no rule for `export type *` (upstream's neither), so `type` + * parsed as an error. A barrel of re-exports was then reported as producing no + * symbols, `codegraph status` listed it among the files it could not parse, + * and the native kernel handed every file holding such a line to the wasm + * parser. The TypeScript extractor's preParse now blanks that `type` to spaces + * where a statement starts, offsets kept, so the line reads as the `export *` + * it is to the graph, and both engines parse the same bytes. The import + * resolver reads the statement as written from the file, so a name imported + * through the barrel still reaches the file that declares it. + */ +import { describe, it, expect, beforeAll, beforeEach, afterEach, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel'; +import { blankExportTypeStar } from '../src/extraction/languages/typescript'; +import type { ExtractionResult, Language } from '../src/types'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1'; + +/** The issue's barrel, plus the namespace form. */ +const BARREL = [ + 'export { FooClient, FooApiError, type FooClientOptions } from "./client.js";', + 'export { DAYS_OF_WEEK } from "./types.js";', + 'export type * from "./types.js";', + 'export type * as Shapes from "./shapes.js";', + '', +].join('\n'); + +describe('blankExportTypeStar', () => { + it('blanks the `type` of a type-only star re-export, keeping every offset', () => { + const cases: Array<[string, string]> = [ + ['export type * from "./types";', 'export * from "./types";'], + ['export type * as Types from "./types";', 'export * as Types from "./types";'], + ["export type * as 'my types' from './types';", "export * as 'my types' from './types';"], + ['export type *from"./types";', 'export *from"./types";'], + ['export\ttype\t*\tfrom "./types";', 'export\t \t*\tfrom "./types";'], + // `from` on the next line. + ['export type *\n from "./types";', 'export *\n from "./types";'], + // Indented in an ambient module, and after `{`, `;` or a block comment on the same line. + ['declare module "m" {\n export type * from "./y";\n}', 'declare module "m" {\n export * from "./y";\n}'], + ['declare module "m" { export type * from "./y"; }', 'declare module "m" { export * from "./y"; }'], + ['export * from "./a"; export type * from "./b";', 'export * from "./a"; export * from "./b";'], + ['/* types */ export type * from "./b";', '/* types */ export * from "./b";'], + ['export * from "./a";\r\nexport type * from "./b";\r\n', 'export * from "./a";\r\nexport * from "./b";\r\n'], + ]; + for (const [source, expected] of cases) { + const blanked = blankExportTypeStar(source); + expect(blanked).toBe(expected); + expect(blanked).toHaveLength(source.length); + } + }); + + it('leaves everything else as written', () => { + for (const source of [ + 'export type Star = "*";', + 'export type { A } from "./a";', + 'export { type A } from "./a";', + 'export * from "./a";', + 'export * as ns from "./a";', + // Not where a statement starts: a mention in a comment, a doc or a string. + '// Re-export them with export type * from "./types".', + '/**\n * export type * from "./types";\n */', + 'const hint = "export type * from \'./types\'";', + ]) { + expect(blankExportTypeStar(source)).toBe(source); + } + }); +}); + +const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const; + +describe('a type-only star re-export parses in both engines', () => { + let savedEnv: Record = {}; + + beforeAll(async () => { + await initGrammars(); + await loadGrammarsForLanguages(['typescript', 'tsx']); + }); + + beforeEach(() => { + savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]])); + resetKernelForTests(); + }); + + afterEach(() => { + for (const k of ENV_KEYS) { + if (savedEnv[k] === undefined) delete process.env[k]; + else process.env[k] = savedEnv[k]; + } + resetKernelForTests(); + }); + + function extract(backend: 'kernel' | 'wasm', file: string, source: string, language: Language): ExtractionResult { + if (backend === 'wasm') { + process.env.CODEGRAPH_KERNEL = '0'; + return extractFromSource(file, source, language); + } + delete process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL_LANGS = 'all'; + // Null when the kernel defers the file to the wasm parser, as it does every file with a parse error. + const result = tryKernelExtract(file, source, language); + expect(result, `kernel extraction of ${file}`).not.toBeNull(); + return result!; + } + + const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const); + + for (const crlf of [false, true]) { + const eol = (s: string) => (crlf ? s.replace(/\n/g, '\r\n') : s); + for (const [file, language] of [ + ['src/index.ts', 'typescript'], + ['src/index.tsx', 'tsx'], + ] as const) { + it.each(backends)(`a barrel in ${file}${crlf ? ' (CRLF)' : ''}: %s`, (backend) => { + const result = extract(backend, file, eol(BARREL), language); + expect(result.errors).toEqual([]); + // Each statement imports its module, as `export * from` does. + expect(result.nodes.filter((n) => n.kind === 'import').map((n) => `${n.name}@${n.startLine}`)).toEqual([ + './client.js@1', + './types.js@2', + './types.js@3', + './shapes.js@4', + ]); + expect(result.unresolvedReferences.map((r) => `${r.referenceName}@${r.line}`).sort()).toEqual([ + './client.js@1', + './shapes.js@4', + './types.js@2', + './types.js@3', + 'DAYS_OF_WEEK@2', + 'FooApiError@1', + 'FooClient@1', + 'FooClientOptions@1', + ]); + }); + } + } + + it.each(backends)('in a declaration file\'s declare module body: %s', (backend) => { + const source = 'declare module "sdk" {\n export type * from "./types";\n export function connect(): void;\n}\n'; + const result = extract(backend, 'typings/sdk.d.ts', source, 'typescript'); + expect(result.errors).toEqual([]); + expect(result.nodes.filter((n) => n.kind !== 'file').map((n) => `${n.kind} ${n.name}`)).toEqual([ + 'import ./types', + 'function connect', + ]); + }); + + it.runIf(kernelAvailable)('the kernel and the wasm extractor agree', () => { + const canon = (r: ExtractionResult) => [ + ...r.nodes.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort())).sort(), + ...r.unresolvedReferences.map((ref) => JSON.stringify(ref, Object.keys(ref).sort())).sort(), + ]; + for (const [file, language] of [ + ['src/index.ts', 'typescript'], + ['src/index.tsx', 'tsx'], + ] as const) { + expect(canon(extract('kernel', file, BARREL, language))).toEqual(canon(extract('wasm', file, BARREL, language))); + } + }); +}); + +describe('indexing a barrel with `export type *` (#2474)', () => { + let root = ''; + let cg: CodeGraph; + + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-export-type-star-')); + const files: Record = { + 'packages/sdk/src/client.ts': + 'export interface FooClientOptions { baseUrl: string }\n' + + 'export class FooApiError extends Error {}\n' + + 'export class FooClient {\n' + + ' constructor(private opts: FooClientOptions) {}\n' + + ' get(path: string) { return this.opts.baseUrl + path; }\n' + + '}\n', + 'packages/sdk/src/types.ts': "export const DAYS_OF_WEEK = ['mon', 'tue'];\nexport type Schedule = { day: string };\n", + 'packages/sdk/src/shapes.ts': 'export interface Circle { radius: number }\n', + 'packages/sdk/src/index.ts': BARREL, + // Same names, declared where nothing re-exports them: what a name match could pick. + 'app/shapes.ts': 'export interface FooClientOptions { decoy: true }\nexport type Schedule = { decoy: true };\n', + 'app/main.ts': + 'import { FooClient, type FooClientOptions } from "../packages/sdk/src/index.js";\n' + + 'import type { Schedule } from "../packages/sdk/src/index.js";\n' + + 'export function run(o: FooClientOptions, s: Schedule) {\n' + + ' return new FooClient(o).get(s.day);\n' + + '}\n', + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); + }); + + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + it('stores the barrel without a parse error', () => { + const barrel = cg.getFile('packages/sdk/src/index.ts'); + expect(barrel).not.toBeNull(); + expect(barrel!.errors ?? []).toEqual([]); + }); + + it('links each name imported through the barrel to the file that declares it', () => { + const ids = cg.getNodesInFile('app/main.ts').map((n) => n.id); + const imported = cg + .getOutgoingEdgesFrom(ids, ['imports']) + .map((e) => cg.getNode(e.target)!) + .filter((t) => t.kind !== 'file') + .map((t) => `${t.name} -> ${t.filePath}`) + .sort(); + expect(imported).toEqual([ + 'FooClient -> packages/sdk/src/client.ts', + 'FooClientOptions -> packages/sdk/src/client.ts', + // Reached only through `export type * from "./types.js"`. + 'Schedule -> packages/sdk/src/types.ts', + ]); + }); +}); diff --git a/src/extraction/languages/typescript.ts b/src/extraction/languages/typescript.ts index ec28e1e829..5b435d9141 100644 --- a/src/extraction/languages/typescript.ts +++ b/src/extraction/languages/typescript.ts @@ -69,6 +69,32 @@ export function blankFlowSyntax(source: string, filePath?: string): string { .replace(/\.\.\.(?=\s*[},])/g, ' '); } +/** Whether the file writes `export type *` anywhere: most files don't, and skip the rest. */ +const EXPORT_TYPE_STAR_HINT = /export\s+type\s*\*/; +/** + * `export type *` where a statement starts (a line's first word, or after `;`, + * `{`, `}` or the end of a block comment), followed by `from` or `as`. + */ +const EXPORT_TYPE_STAR = /(^|[;{}]|\*\/)([ \t]*export\s+)type(?=\s*\*\s*(?:from|as)\b)/gm; + +/** + * TypeScript 5.0's type-only re-export of a whole module, `export type * from + * './types'` and `export type * as ns from './types'`, with its `type` blanked + * to spaces (offsets survive), so it reads as the `export *` it is to the + * graph. The grammar has no rule for it (nor does upstream's): `type` parsed + * as an error, so a barrel of re-exports was reported as producing no symbols + * and counted by `codegraph status` as a file it could not parse, and the + * kernel handed every such file to the wasm parser (#2474). Only a statement + * start is blanked, so a mention in a comment or a doc keeps its text. The + * import resolver reads the statement as written, `type` included, from the + * file itself (`extractReExports`). Applied before both engines parse (the + * kernel's preParse hoist), so they agree. + */ +export function blankExportTypeStar(source: string): string { + if (!EXPORT_TYPE_STAR_HINT.test(source)) return source; + return source.replace(EXPORT_TYPE_STAR, '$1$2 '); +} + /** * Whether `node` is declared in a `declare module 'x' { … }` or `declare * global { … }` body, which exports it without an `export` keyword. @@ -147,7 +173,7 @@ export function isAmbientFunctionSignature(node: SyntaxNode, filePath: string): } export const typescriptExtractor: LanguageExtractor = { - preParse: blankFlowSyntax, + preParse: (source, filePath) => blankExportTypeStar(blankFlowSyntax(source, filePath)), functionTypes: ['function_declaration', 'generator_function_declaration', 'arrow_function', 'function_expression', 'generator_function'], isAmbientFunction: isAmbientFunctionSignature, // `class` is a class expression, `register(class Widget extends Base {…})`: From d90f6abec79b4e2a90691c94b4cb8f27a187e555 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 09:59:26 +0000 Subject: [PATCH 65/95] fix(cfml): and cfinclude() link the code to the file they include (#2838) Only cfscript's `include "x.cfm";`, written outside any function, linked the including code to the template. CfmlExtractor never read ``, which the grammar parses as a cf_selfclose_tag whose name is only in the text after `` is extracted like the script form: an import node in the scope it sits in (function, component, or the template's file) and an imports reference from that scope. `cfinclude(...)` and `include template="...";` make the same import and no call, and an `include` in a function body counts, from the function. - A path built at runtime (`#dir#/x.cfm`) is no import, in every form. - A CFML include of a template path resolves as CFML does: relative to the including file's directory, or from the project root for a leading `/` (less sure, 0.85), to the file at that path, ignoring case when one file matches. A path that names no file there goes to the file-name matcher, as before, but only ever to a file, and never to the including file. Validation: - New cfml-include.test.ts: the tag form at component level (same folder, ../, upper case, unquoted), in a function body inside , in a template and from the web root; the script forms; runtime-built paths, cfinclude(template=variables.x) and (no import); a relative path with a namesake elsewhere; an unmatched path that another template also includes, and a path whose only namesake is the includer. 5 of 7 fail on main, all 7 pass. - A/B vs main 51c4124d (CFML has no kernel path) on FW/1, BlogCFC, ColdBox, ContentBox, Preside, CFWheels and its vendor/wheels core as a project of its own: 127 new imports edges and 152 import nodes (36 more resolved again to the same file, more surely); 9 import nodes of runtime-built paths gone; 1 edge gone, CFWheels' starter-app views/wheels/wheels.cfm including itself. No edge moved to another target. Every CFML imports edge in the fix graphs (225) targets a file named like the include path, and every relative path that stays inside the project reaches exactly that file but one: Wheels' Public.cfc including views/routetesterprocess.cfm, no file beside it, matched by name to public/views/routetesterprocess.cfm as before. express and flask controls byte-identical. - Index time, median of 4 interleaved runs, CPU with workers, main vs fix: Preside 52.6 / 50.8 s, Wheels core 28.2 / 29.5 s. Within noise. Refs #2651 (part 1; part 2 is a feature request and stays open) Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cfml-include.test.ts | 162 +++++++++++++++++++++++++++ src/extraction/cfml-extractor.ts | 65 ++++++++++- src/extraction/languages/cfscript.ts | 52 ++++++++- src/extraction/tree-sitter.ts | 12 ++ src/resolution/index.ts | 59 ++++++++++ 6 files changed, 346 insertions(+), 5 deletions(-) create mode 100644 __tests__/cfml-include.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 298b336b83..0a3a62537a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -145,6 +145,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In JavaScript and TypeScript, a file that re-exports a module, like an `index.ts` with `export * from './button'` or `export { Button } from './button'`, now counts as importing that module, the way an `import` statement does. `codegraph affected` and a module's dependents now include the barrels that re-export it; before, a barrel made only of `export *` lines showed no dependency at all. Re-index JavaScript and TypeScript projects after upgrading. - `codegraph sync` and the file watcher now also update the files that import through an `index.ts` re-exporting another barrel, like `export * from './dropdown-menu'`, when that inner barrel is deleted or added. Before, their imports stayed linked through the deleted barrel, or stayed linked the way they were while it was missing, until each importing file changed or the project was re-indexed. - An import in a Vue, Svelte or Astro component now links the component to the file it imports, as an import in a script does, and so does an import of a scoped package that a `tsconfig` path or a workspace package maps to a project file, like `import { debounce } from '@excalidraw/common'`. Before, only the names such an import binds were linked, so a barrel or a module imported for its side effects showed no dependency from those files in `codegraph affected`, and `codegraph sync` could not update them when a barrel they import through was deleted or added. `codegraph sync` and the file watcher also link such an import again, and one written as `.` or `..`, when the file it names comes back. Re-index JavaScript, TypeScript, Vue, Svelte and Astro projects after upgrading. +- In CFML, `` now links the component, function or template it is written in to the file it includes, as `include "formlistshared.cfm";` in a script already did, and so do `cfinclude(template="…")`, `include template="…";` and an `include` inside a function. A relative path like `../shared/helpers.cfm` now finds that file even when another file has the same name. Before, these includes left no trace in the graph, and `cfinclude(…)` showed up as a call to a function that doesn't exist. Thanks @HarryMuc for the report. (#2651) - In a pnpm, npm, Yarn or Bun workspace whose globs reach packages more than one folder down, like pnpm's `packages/**/*` or npm's `libs/**/*`, those packages now count as part of the project, so an import of one is no longer taken for an outside package's, and `codegraph sync` and the file watcher relink the calls through it when its files change. A workspace glob can now also name packages by part of a folder name (`packages/plugin-*`) or by a folder below the wildcard (`apps/*/web`), and a negated glob like `!**/example` leaves packages out, as the package managers do. Before, only the folders directly under a glob's first wildcard were read, and a negated glob was ignored. Re-index after upgrading. - In a project that uses Vue, an import written through an `@/` or `~/` alias, like `import { Button } from '@/components'`, now links the file it imports, as it does in any other project, so `codegraph sync` and the file watcher update the files that import a module when that module, or a barrel they import through, is added or deleted. Before, Vue support linked such an import to the first declaration or the component in that file, which sync does not follow, and an import through a `tsconfig` alias that starts like one of Nuxt's built-in modules, like `#app/where`, linked the importing file to itself. Re-index Vue projects after upgrading. - `codegraph sync` and the file watcher now move an import written without an extension, like `import { sizeOf } from './button'`, to a file added beside the one it found before when the new file comes first, such as `button.ts` next to `button.vue`. Before, the import and the names it binds stayed linked to `button.vue` until the importing file changed or the project was re-indexed. JavaScript, TypeScript, Vue, Svelte and Astro. diff --git a/__tests__/cfml-include.test.ts b/__tests__/cfml-include.test.ts new file mode 100644 index 0000000000..33924221e1 --- /dev/null +++ b/__tests__/cfml-include.test.ts @@ -0,0 +1,162 @@ +/** + * CFML includes (#2651). Only cfscript's `include "x.cfm";`, written outside + * any function, linked the including code to the template: the tag form + * `` — the common one in tag-based code — made + * nothing anywhere, the function form `cfinclude(template="x.cfm")` became a + * call to a function named `cfinclude`, and an `include` inside a function + * was dropped. A template path is relative to the including file's directory + * (or to the web root with a leading `/`), so `../shared/helpers.cfm` is that + * file even when another `helpers.cfm` exists. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +let dir = ''; +let edges: string[] = []; +let imports: string[] = []; +let unresolved: string[] = []; +let importTargetKinds: string[] = []; + +const files: Record = { + // Tag form at component scope: same directory, relative, upper case, unquoted. + 'app/core/baseform.cfc': ` +\t +\t +\t +\t +\t + +`, + // Tag form in a function body, nested in control flow; runtime-built paths and are no includes. + 'app/core/report.cfc': ` +\t +\t\t +\t\t\t +\t\t +\t\t +\t\t +\t\t +\t + +`, + // Tag form in a template, and a path from the web root. + 'app/page.cfm': ` + +`, + // A template and a function including a path no file has (a server + // mapping's): the function may not link to the template's import of it. + 'app/one.cfm': `\n`, + 'app/core/runtests.cfc': `component {\n\tfunction run() {\n\t\tinclude "/mapped/runner.cfm";\n\t}\n}\n`, + // The only file of the name an include names is the includer itself. + 'app/views/wheels/wheels.cfm': `\n`, + // Script forms: in a function body, the attribute form, the function form. + 'app/core/scriptform.cfc': `component { +\tinclude "formlistshared.cfm"; +\tfunction render() { +\t\tinclude "../shared/helpers.cfm"; +\t\tcfinclude(template="upper.cfm"); +\t\tcfinclude("bare.cfm"); +\t\tCFINCLUDE( TEMPLATE = 'formlistshared.cfm' ); +\t} +\tfunction other() { +\t\tinclude template="formlistshared.cfm"; +\t\tcfinclude(template=variables.view); +\t\tinclude "#variables.view#"; +\t\tinclude "#variables.dir#/helpers.cfm"; +\t} +} +`, + 'app/core/formlistshared.cfm': `\n`, + 'app/core/Upper.cfm': `\n`, + 'app/core/bare.cfm': `\n`, + 'app/shared/helpers.cfm': `\n`, + // A namesake the relative path does not name. + 'other/shared/helpers.cfm': `\n`, +}; + +beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cfml-include-')); + for (const [rel, body] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), body); + } + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + edges = db + .prepare( + `SELECT s.qualified_name src, t.file_path tgt FROM edges e + JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE e.kind = 'imports'` + ) + .all() + .map((r: any) => `${r.src} -> ${r.tgt}`) + .sort(); + imports = db.prepare(`SELECT qualified_name q FROM nodes WHERE kind = 'import'`).all().map((r: any) => r.q).sort(); + importTargetKinds = db + .prepare(`SELECT DISTINCT t.kind k FROM edges e JOIN nodes t ON t.id = e.target WHERE e.kind = 'imports'`) + .all() + .map((r: any) => r.k); + unresolved = db + .prepare(`SELECT reference_kind k, reference_name n FROM unresolved_refs`) + .all() + .map((r: any) => `${r.k} ${r.n}`); + cg.close?.(); +}); + +afterAll(() => { + if (dir) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('CFML includes (#2651)', () => { + it('links at component scope to the template, relative to the including file', () => { + expect(edges).toEqual(expect.arrayContaining([ + 'app/core/baseform.cfc::baseform -> app/core/formlistshared.cfm', + 'app/core/baseform.cfc::baseform -> app/shared/helpers.cfm', + 'app/core/baseform.cfc::baseform -> app/core/Upper.cfm', + 'app/core/baseform.cfc::baseform -> app/core/bare.cfm', + ])); + expect(imports).toContain('baseform::formlistshared.cfm'); + }); + + it('links a in a function body from that function', () => { + expect(edges).toContain('report::render -> app/core/formlistshared.cfm'); + expect(imports).toContain('report::render::formlistshared.cfm'); + }); + + it('links a in a template, including one from the web root', () => { + expect(edges).toContain('app/page.cfm -> app/core/formlistshared.cfm'); + expect(edges).toContain('app/page.cfm -> app/shared/helpers.cfm'); + }); + + it('makes nothing of a template built at runtime or of ', () => { + expect(edges.filter((e) => e.startsWith('report::render ->'))).toEqual(['report::render -> app/core/formlistshared.cfm']); + expect(imports.filter((q) => q.startsWith('report::'))).toEqual(['report::render::formlistshared.cfm']); + expect(edges.filter((e) => e.startsWith('scriptform::other ->'))).toEqual(['scriptform::other -> app/core/formlistshared.cfm']); + expect(imports.filter((q) => q.includes('#'))).toEqual([]); + }); + + it('links the script forms: in a function body, `include template=…` and cfinclude(…)', () => { + expect(edges).toEqual(expect.arrayContaining([ + 'scriptform::render -> app/shared/helpers.cfm', + 'scriptform::render -> app/core/Upper.cfm', + 'scriptform::render -> app/core/bare.cfm', + 'scriptform::render -> app/core/formlistshared.cfm', + 'scriptform::other -> app/core/formlistshared.cfm', + ])); + expect(unresolved.filter((r) => /cfinclude/i.test(r))).toEqual([]); + }); + + it('never links a relative include to a namesake the path does not name', () => { + expect(edges.filter((e) => e.endsWith('other/shared/helpers.cfm'))).toEqual([]); + }); + + it('links an include to a file or to nothing: never to an import statement, nor to the including file', () => { + expect(importTargetKinds).toEqual(['file']); + expect(edges.filter((e) => e.startsWith('app/one.cfm') || e.startsWith('runtests::'))).toEqual([]); + expect(edges.filter((e) => e.startsWith('app/views/wheels/wheels.cfm'))).toEqual([]); + }); +}); diff --git a/src/extraction/cfml-extractor.ts b/src/extraction/cfml-extractor.ts index a1a668706a..fb6ef8655e 100644 --- a/src/extraction/cfml-extractor.ts +++ b/src/extraction/cfml-extractor.ts @@ -417,6 +417,11 @@ export class CfmlExtractor { case 'cf_function_tag': if (declScope) this.extractFunctionTag(node, parentClassName ? containerId : undefined, containerId, parentClassName); return; + case 'cf_selfclose_tag': + // ``, wherever it is written (#2651). Its + // attributes are still walked below for `#hash#` expressions. + if (this.selfCloseTagName(node) === 'include') this.extractIncludeTag(node, containerId); + break; case 'cf_tag': { const name = this.cfTagName(node); if (name === 'interface' && declScope && !parentClassName) { @@ -559,6 +564,60 @@ export class CfmlExtractor { return nameNode ? this.source.substring(nameNode.startIndex, nameNode.endIndex).toLowerCase() : undefined; } + /** + * A void tag's name, lowercased, without the `cf` prefix (`` → + * `include`). The grammar keeps no node for it: it is the text after `` (#2651) includes the template + * as cfscript's `include "formlistshared.cfm";` does, so it is extracted the + * same way: an import node in the scope it sits in (the function, the + * component, or the file of a template) and an `imports` reference from + * that scope, which the resolver links to the file. A template built at + * runtime (`template="#dir#/x.cfm"`) names no file the index can link. + */ + private extractIncludeTag(tag: SyntaxNode, scopeId: string): void { + const valueNode = this.tagAttrValueNode(tag, 'template', true); + const template = valueNode ? this.source.substring(valueNode.startIndex, valueNode.endIndex).trim() : ''; + if (!template) return; + // Qualified like the generic extractor's import nodes: under the scope's + // own name, without the file path (`baseform::formlistshared.cfm`). + const scope = this.nodes.find((n) => n.id === scopeId); + const prefix = !scope || scope.kind === 'file' ? '' : scope.kind === 'method' ? scope.qualifiedName : scope.name; + const line = tag.startPosition.row + 1; + const column = tag.startPosition.column; + const id = this.nodeIds.generate(this.filePath, 'import', template, line, column); + this.nodes.push({ + id, + kind: 'import', + name: template, + qualifiedName: prefix ? `${prefix}::${template}` : template, + filePath: this.filePath, + language: this.language, + startLine: line, + endLine: tag.endPosition.row + 1, + startColumn: column, + endColumn: tag.endPosition.column, + signature: this.source.substring(tag.startIndex, tag.endIndex).trim(), + updatedAt: Date.now(), + }); + this.edges.push({ source: scopeId, target: id, kind: 'contains' }); + this.unresolvedReferences.push({ + fromNodeId: scopeId, + referenceName: template, + referenceKind: 'imports', + filePath: this.filePath, + line, + column, + language: this.language, + }); + } + /** * Delegate a `...` tag body to the cfscript grammar. * With `parentClassName` set (the block sits at component scope), functions @@ -691,9 +750,10 @@ export class CfmlExtractor { /** * The `attribute_value` node of a tag attribute — `undefined` when the tag * has no such attribute, `null` when its value isn't plain text (empty, or - * a `#hash#` expression). + * a `#hash#` expression). With `wholeValue`, also `null` when only part of + * the value is plain text (`template="#dir#/x.cfm"`). */ - private tagAttrValueNode(tag: SyntaxNode, attrName: string): SyntaxNode | null | undefined { + private tagAttrValueNode(tag: SyntaxNode, attrName: string, wholeValue = false): SyntaxNode | null | undefined { const attrs: SyntaxNode[] = []; for (let i = 0; i < tag.namedChildCount; i++) { const child = tag.namedChild(i); @@ -716,6 +776,7 @@ export class CfmlExtractor { const valueWrapper = attr.namedChildren.find( (c: SyntaxNode) => c.type === 'quoted_cf_attribute_value' || c.type === 'cf_attribute_value' ); + if (wholeValue && valueWrapper?.namedChildCount !== 1) return null; return valueWrapper?.namedChildren.find((c: SyntaxNode) => c.type === 'attribute_value') ?? null; } return undefined; diff --git a/src/extraction/languages/cfscript.ts b/src/extraction/languages/cfscript.ts index 361806f4bb..68cd21aeb5 100644 --- a/src/extraction/languages/cfscript.ts +++ b/src/extraction/languages/cfscript.ts @@ -17,6 +17,44 @@ function cfmlVisibility(node: SyntaxNode): 'public' | 'private' | 'protected' | return undefined; } +/** + * A string literal's text when it is written out in full, or null when a + * `#…#` expression builds it at runtime (`"#dir#/x.cfm"`): such a path names + * no file the index can link. + */ +function literalString(node: SyntaxNode | null | undefined, source: string): string | null { + if (node?.type !== 'string' || node.namedChildren.some((c: SyntaxNode) => c.type !== 'string_fragment')) return null; + return getNodeText(node, source).replace(/^["']|["']$/g, '').trim() || null; +} + +/** The value of a `template=…` argument or attribute, when it is a literal string. */ +function templateArgument(node: SyntaxNode, source: string): string | null { + if (node.type !== 'assignment_expression') return null; + const left = getChildByField(node, 'left'); + if (left?.type !== 'identifier' || getNodeText(left, source).toLowerCase() !== 'template') return null; + return literalString(getChildByField(node, 'right'), source); +} + +/** + * Whether a call is `cfinclude(…)` — the function form of `include` and of + * `` (ACF 11+, Lucee): an include of a template, not a call (#2651). + */ +export function isCfIncludeCall(node: SyntaxNode, source: string): boolean { + const fn = getChildByField(node, 'function'); + return fn?.type === 'identifier' && getNodeText(fn, source).toLowerCase() === 'cfinclude'; +} + +/** The literal template `cfinclude(template="x.cfm")` or `cfinclude("x.cfm")` names. */ +function cfIncludeCallTemplate(node: SyntaxNode, source: string): string | null { + const args = getChildByField(node, 'arguments'); + if (!args) return null; + for (const arg of args.namedChildren) { + const named = templateArgument(arg, source); + if (named) return named; + } + return literalString(args.namedChildren[0], source); +} + export const cfscriptExtractor: LanguageExtractor = { functionTypes: ['function_declaration', 'function_expression', 'arrow_function'], classTypes: ['component'], @@ -45,10 +83,18 @@ export const cfscriptExtractor: LanguageExtractor = { const importText = source.substring(node.startIndex, node.endIndex).trim(); if (node.type === 'include_statement') { - // `include "path/to/file.cfm";` — the included template path. + // `include "path/to/file.cfm";` — the included template path — or the + // attribute form, `include template="path/to/file.cfm";` (#2651). A + // path built at runtime (`include "#view#";`) is no import. const expr = node.namedChildren.find((c: SyntaxNode) => c.type === 'string'); - if (!expr) return null; - const moduleName = getNodeText(expr, source).replace(/^["']|["']$/g, ''); + const attr = expr ? undefined : node.namedChildren.find((c: SyntaxNode) => c.type === 'assignment_expression'); + const moduleName = expr ? literalString(expr, source) : attr ? templateArgument(attr, source) : null; + return moduleName ? { moduleName, signature: importText } : null; + } + + if (node.type === 'call_expression') { + // `cfinclude(template="path/to/file.cfm");` (#2651). + const moduleName = isCfIncludeCall(node, source) ? cfIncludeCallTemplate(node, source) : null; return moduleName ? { moduleName, signature: importText } : null; } diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index caf11234a4..c0449a40d7 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -27,6 +27,7 @@ import { appendCppMacroMethods } from './languages/cpp-macro-methods'; import { rustImplItemOwner, rustImplTypeName } from './languages/rust'; import { goAliasTypeNames, goEmbeddedTypeName, goInterfaceMethodTypeNames, goStructFieldTypeNames } from './languages/go'; import { dartMisparsedGenericCall, dartReceiverOf, isDartTypeName, pushDartTypeAliasRefs, pushDartTypeRefs } from './languages/dart'; +import { isCfIncludeCall } from './languages/cfscript'; import { LiquidExtractor } from './liquid-extractor'; import { RazorExtractor } from './razor-extractor'; import { SvelteExtractor } from './svelte-extractor'; @@ -5166,6 +5167,13 @@ export class TreeSitterExtractor { const callerId = this.nodeStack[this.nodeStack.length - 1]; if (!callerId) return; + // CFML `cfinclude(template="x.cfm")` includes a template, as `include + // "x.cfm";` does (#2651): an import of the file, never a call. + if (this.language === 'cfscript' && isCfIncludeCall(node, this.source)) { + this.extractImport(node); + return; + } + // VB.NET: `foo(args)` is syntactically ambiguous between a call and an // index read, so the grammar parses non-empty parens as // array_access_expression (field `array`, not `function`) — even Roslyn @@ -7249,6 +7257,10 @@ export class TreeSitterExtractor { // handler paths live in a raw token tree the call walker can't see. if (nodeType === 'macro_invocation') this.extractRustRouteMacro(node); + // A CFML `include "x.cfm";` in a function body includes the template + // there, as it does at component scope (#2651). + if (nodeType === 'include_statement' && this.language === 'cfscript') this.extractImport(node); + if (this.extractor!.callTypes.includes(nodeType)) { this.extractCall(node); } else if (INSTANTIATION_KINDS.has(nodeType) || this.isVbnetConstructorShapedArrayCreation(node)) { diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 01e363cace..536f86894f 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -1568,6 +1568,21 @@ export class ReferenceResolver { return this.resolveCfmlComponentPath(ref); } + // A CFML include names a template by its path from the including file's + // directory, or from the web root (#2651): `` is that file, not whichever `helpers.cfm` file-name + // matching would settle on. A path that names no file here (one through a + // server mapping, `/contenscms/…`) is matched by its file name, as before, + // but only to a file: the symbol strategies below took a path no file + // matched for another template's import statement of the same path. Nor + // is it the including file, which a file-name match can land on (Wheels' + // `views/wheels/wheels.cfm` including `../../wheels/public/wheels.cfm`). + if ((ref.language === 'cfml' || ref.language === 'cfscript') && ref.referenceKind === 'imports' && isCfmlTemplatePath(ref.referenceName)) { + const included = this.resolveCfmlIncludePath(ref) ?? matchByFilePath(ref, this.context); + if (included && this.nodeById(included.targetNodeId)?.filePath === ref.filePath) return null; + return this.gateLanguage(included, ref); + } + // A PHP class written with a namespace in it — `new Alias\X()` through a // `use Ns as Alias;` namespace alias, `extends Sub\Base`, `\Ns\X::make()` // (#2256). The pre-filter below would drop most of these before any import @@ -3581,6 +3596,41 @@ export class ReferenceResolver { return { original: ref, targetNodeId: tied[0]!.id, confidence: 0.9, resolvedBy: 'qualified-name' }; } + /** + * The file a CFML include names (#2651) — `include "x.cfm";`, + * ``, `cfinclude(template="x.cfm")`. A template + * path is relative to the directory of the file that includes it + * (`formlistshared.cfm`, `../shared/helpers.cfm`), or to the web root when + * it starts with `/`: the project root, when the project is the web root — + * an assumption, so such a match is less sure. Null when no indexed file is + * there. CFML code written on a case-blind file system often spells a path + * in another case, so a single file that matches ignoring case counts too. + */ + private resolveCfmlIncludePath(ref: UnresolvedRef): ResolvedRef | null { + const written = ref.referenceName.trim().replace(/\\/g, '/'); + const fromRoot = written.startsWith('/'); + const parts = fromRoot ? [] : ref.filePath.replace(/\\/g, '/').split('/').slice(0, -1); + for (const seg of written.split('/')) { + if (seg === '' || seg === '.') continue; + if (seg === '..') { + if (parts.length === 0) return null; // above the project root + parts.pop(); + } else { + parts.push(seg); + } + } + const fileName = parts[parts.length - 1]; + if (!fileName) return null; + const wanted = parts.join('/'); + const files = this.context.getNodesByLowerName(fileName.toLowerCase()).filter((n) => n.kind === 'file'); + const exact = files.find((n) => n.filePath === wanted); + if (exact) return { original: ref, targetNodeId: exact.id, confidence: fromRoot ? 0.85 : 0.95, resolvedBy: 'file-path' }; + const folded = files.filter((n) => n.filePath.toLowerCase() === wanted.toLowerCase()); + return folded.length === 1 + ? { original: ref, targetNodeId: folded[0]!.id, confidence: fromRoot ? 0.8 : 0.9, resolvedBy: 'file-path' } + : null; + } + /** * Resolve a `this.` function-as-value reference (#756/#808) to the * ENCLOSING CLASS's own member — never a same-named symbol elsewhere. The @@ -4267,6 +4317,15 @@ export class ReferenceResolver { } } +/** + * Whether a CFML `imports` name is the path of a template an include names + * (`formlistshared.cfm`, `../shared/helpers.cfm`, `/app/x.cfm`), not a + * cfscript `import com.foo.Bar;` or `import "java:…";` (#2651). + */ +function isCfmlTemplatePath(name: string): boolean { + return !name.includes(':') && (name.includes('/') || name.includes('\\') || /\.(?:cfml?|cfc|cfs)$/i.test(name.trim())); +} + /** The project-relative directory holding `p` ('' for the project root). */ function goDirOf(p: string): string { const dir = path.posix.dirname(p.replace(/\\/g, '/')); From f406fe00644520aec481f9557ae2779ebd039515 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 11:49:44 +0000 Subject: [PATCH 66/95] fix(cfml): createObject("component", path) links its component, and .init() on it (#2841) `new a.b.Batch()` made an instantiates edge, but `createObject("component", "a.b.Batch")`, the form older CFML uses most, was extracted as a call to the engine function createObject, which never resolves. A call chained onto either construction, `createObject(...).init()` or `new a.b.Batch().init()`, was extracted as a bare `init`, which the name matcher bound to whichever component's init came first (0.4), or to none when there are many. Fix: - cfscript extraction (tag expressions included): a component createObject, with two arguments, one, or named ones, makes an instantiates reference under the dotted path as written, not a call. Java and other object types, and names built at runtime, are unchanged. - A call on a construction is extracted as `().`. - The resolver reads a dotted instantiates path with resolveCfmlComponentPath, the rules extends uses, and resolves a chained call to the method of that component or one it inherits (matched ignoring case; inheritance is wired before calls). A bare component name is the one in the calling file's folder, else the only one so named. Without both, the call gets no edge rather than a guess by name. - resolveCfmlComponentPath also accepts a component at the project root for a path with one folder segment, its file named like it: `wheels.JobWorker` in Wheels' own source, where `wheels` maps onto the root. Without it Wheels' core lost calls like `new wheels.JobWorker().processNext()` that name matching had found. It ranks below any folder match; extends gains the same. Validation: - New cfml-create-object.test.ts (5 cases) and a root-component case in cfml-inheritance-resolution.test.ts: 5 of the 6 fail on main (the sixth pins the variable typing that already worked), all pass. - A/B vs main d90f6abe on FW/1, BlogCFC, ColdBox, ContentBox, Preside, CFWheels and its vendor/wheels core indexed as its own project: 1,022 edges removed: 563 resolved again to the same target by type, 444 moved from a method guessed by name to the constructed component's own or inherited method, and 15 dropped, all name guesses for components outside the index, without the method, or tied. 1,342 added: 641 instantiates, 38 calls, and in Wheels' core 653 extends and 10 implements to root components. Oracles over the fix graphs: of the 983 chained-construction call edges, 965 land in the file named like the component and 18 on a method its supertypes declare (each checked); all 608 dotted instantiates and, in Wheels' core, all 786 dotted extends/implements edges target a file named like the path's last segment. One site is doubtful: Wheels' core public/docs/core.cfm names the application's app.models.Model, which that subtree lacks; one `models` folder corroborates a test asset's Model.cfc, as it would for extends. express and flask controls byte-identical. - Index time, median of 4 interleaved runs, CPU with workers, main vs fix: Preside 58.7 / 54.5 s, Wheels core 32.2 / 33.0 s. Within noise. Fixes #2652 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cfml-create-object.test.ts | 131 ++++++++++++++++++ __tests__/cfml-inheritance-resolution.test.ts | 16 +++ src/extraction/languages/cfscript.ts | 55 +++++++- src/extraction/tree-sitter.ts | 29 +++- src/resolution/index.ts | 76 +++++++++- 6 files changed, 304 insertions(+), 4 deletions(-) create mode 100644 __tests__/cfml-create-object.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a3a62537a..044d6c6cb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,6 +43,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. - In C# and VB.NET, a constant no longer stands in for a type with the same name: `new Station { … }` links to class `Station` even when another class declares `private const string Station`, `new Version(…)` no longer links to a `const string Version`, and a field like `static readonly Meter Meter` no longer points at itself. Thanks @EvanYu1980 for the report and @drakeo338. (#2337) +- In CFML, `createObject("component", "contenscms.core.queue.batchcontext")`, and the one-argument `createObject("contenscms.core.queue.batchcontext")`, now link to the component they create, found by its path the way `extends` finds one, as `new` already did. A call chained onto the new object, like `createObject(…).init()` or `new contenscms.core.queue.batchcontext().init()`, now links to that component's `init`, or to the one it inherits. In a framework's own source, a path like `wheels.JobWorker`, whose first part is the framework itself, now also finds the component at the project root, for `extends` as well. Before, `createObject` showed up as a call to a function that doesn't exist, and the chained `init` was linked to whichever component's `init` came first, or to none. Thanks @HarryMuc for the report. (#2652) - In VB.NET, a call on a variable, parameter, field or property now reaches a method of the type it is declared with — one that type inherits, or an extension method written for it — and nothing when that type comes from outside your project, instead of any project method that merely shares the name. - In VB.NET, a call to one of .NET's own methods such as `Add`, `Contains`, `Clear` or `Dispose` on a value whose type isn't known is no longer linked to a project method of that name, unless the value is named after that method's class. - In VB.NET, a type name is looked up the way VB.NET does it — through the namespaces around it, the file's and project's `Imports` (aliases included), and the caller's own project — so a class declared in several namespaces or projects no longer draws every call to whichever copy was indexed first, and two candidates nothing tells apart get no link at all. diff --git a/__tests__/cfml-create-object.test.ts b/__tests__/cfml-create-object.test.ts new file mode 100644 index 0000000000..3fb1e8e253 --- /dev/null +++ b/__tests__/cfml-create-object.test.ts @@ -0,0 +1,131 @@ +/** + * CFML `createObject("component", "a.b.Batch")` (#2652). `new a.b.Batch()` + * made an `instantiates` edge; createObject, the form older code uses most, + * made none — it was a call to an engine function that never resolves. And a + * call chained onto either construction, `createObject(…).init()` or `new + * a.b.Batch().init()`, was emitted as a bare `init`, which matched whichever + * component's `init` came first. A component path resolves as `extends` + * resolves one: by its folders, from the right. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +let dir = ''; +let edges: string[] = []; +let unresolved: string[] = []; + +const files: Record = { + // Other components with an `init`, which a guess by name would reach first. + 'contens/aaa/alpha.cfc': `component {\n\tfunction init() { return this; }\n\tfunction $init() { return this; }\n}\n`, + 'contens/core/mixin.cfc': `component {\n\tfunction $init() { return this; }\n}\n`, + // A framework's own source, where its mapping (`wheels`) is the project root. + 'JobWorker.cfc': `component {\n\tfunction processNext() {}\n}\n`, + 'tests/specs/JobSpec.cfc': `component { +\tfunction run() { +\t\tnew wheels.JobWorker().processNext(); +\t\tvar w = createObject("component", "wheels.JobWorker"); +\t} +} +`, + // An app's own Controller is not the framework's `wheels.Controller`. + 'app/controllers/Controller.cfc': `component {\n\tfunction init() { return this; }\n}\n`, + 'app/handlers/Main.cfc': `component {\n\tfunction index() {\n\t\tcreateObject("component", "wheels.Controller").init();\n\t}\n}\n`, + 'contens/aaa/beta.cfc': `component {\n\tfunction init() { return this; }\n}\n`, + 'contens/core/base.cfc': `component {\n\tfunction init() { return this; }\n}\n`, + 'contens/core/queue/batchcontext.cfc': ` +\t +\t + +`, + // Inherits its init. + 'contens/core/queue/jobrunner.cfc': `component extends="contenscms.core.base" {\n\tfunction start() {}\n}\n`, + 'contens/api/batch/collection.cfc': ` +\t +\t\t +\t\t +\t\t +\t + +`, + 'contens/api/batch/scriptcollection.cfc': `component { +\tfunction process() { +\t\tvar oRunner = createObject("contenscms.core.queue.jobrunner").init(); +\t\tvar oNew = new contenscms.core.queue.batchcontext().init(); +\t\tvar oMap = createObject("java", "java.util.HashMap"); +\t\tvar oBox = createObject("component", "testbox.system.TestBox").init(); +\t\tvar oDyn = createObject("component", "#variables.root#.batchcontext"); +\t\tvar oMix = new contenscms.core.mixin().$init(); +\t} +} +`, +}; + +beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cfml-create-object-')); + for (const [rel, body] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), body); + } + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + edges = db + .prepare( + `SELECT e.kind, e.line, s.qualified_name src, t.qualified_name tgt FROM edges e + JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE e.kind IN ('instantiates', 'calls')` + ) + .all() + .map((r: any) => `${r.src} -${r.kind}-> ${r.tgt} @${r.line}`) + .sort(); + unresolved = db + .prepare(`SELECT reference_kind k, reference_name n, line FROM unresolved_refs`) + .all() + .map((r: any) => `${r.k} ${r.n} @${r.line}`); + cg.close?.(); +}); + +afterAll(() => { + if (dir) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('CFML createObject (#2652)', () => { + it('links createObject("component", "a.b.c") to the component it makes, in tags and in script', () => { + expect(edges).toEqual(expect.arrayContaining([ + 'collection::process -instantiates-> contens/core/queue/batchcontext.cfc::batchcontext @3', + 'collection::process -instantiates-> contens/core/queue/jobrunner.cfc::jobrunner @5', + 'scriptcollection::process -instantiates-> contens/core/queue/jobrunner.cfc::jobrunner @3', + ])); + // Only the Java object and the component named at runtime are left as calls. + expect(unresolved.filter((r) => r.startsWith('calls createObject')).sort()).toEqual(['calls createObject @5', 'calls createObject @7']); + }); + + it('links a call chained onto the construction to that component\'s method, or one it inherits', () => { + expect(edges).toEqual(expect.arrayContaining([ + 'collection::process -calls-> batchcontext::init @3', + 'scriptcollection::process -calls-> base::init @3', + 'scriptcollection::process -calls-> batchcontext::init @4', + 'scriptcollection::process -calls-> mixin::$init @8', + ])); + expect(edges.filter((e) => /-> (alpha|beta)::\$?init/.test(e))).toEqual([]); + }); + + it('makes nothing of a component outside the project, another type of object or a name built at runtime', () => { + expect(edges.filter((e) => / @[567]$/.test(e) && e.startsWith('scriptcollection::'))).toEqual([]); + }); + + it('still types a variable assigned from createObject', () => { + expect(edges).toContain('collection::process -calls-> batchcontext::run @4'); + }); + + it('finds a component at the project root by a one-folder path, and only there', () => { + expect(edges).toEqual(expect.arrayContaining([ + 'JobSpec::run -calls-> JobWorker::processNext @3', + 'JobSpec::run -instantiates-> JobWorker.cfc::JobWorker @4', + ])); + expect(edges.filter((e) => e.startsWith('Main::index'))).toEqual([]); + }); +}); diff --git a/__tests__/cfml-inheritance-resolution.test.ts b/__tests__/cfml-inheritance-resolution.test.ts index 879da7f900..94d6b09a29 100644 --- a/__tests__/cfml-inheritance-resolution.test.ts +++ b/__tests__/cfml-inheritance-resolution.test.ts @@ -163,4 +163,20 @@ describe('CFML component-path inheritance resolution (#1152)', () => { const edges = await load(); expect(edges.filter((e) => e.src === 'Caller' || e.src === 'go')).toHaveLength(0); }); + + it('resolves a one-folder path to a component at the project root (a framework\'s own repo, #2652)', async () => { + // Wheels' own source: `wheels` is the mapping onto the root, so + // `wheels.Model` is Model.cfc there, with no folder to corroborate. + write('Model.cfc', `component {}\n`); + write('models/User.cfc', `component extends="wheels.Model" {}\n`); + // A longer path names folders the root component isn't in. + write('Util.cfc', `component {}\n`); + write('models/Helper.cfc', `component extends="coldbox.system.Util" {}\n`); + // An app's own Controller sits in folders the path doesn't name. + write('app/controllers/Controller.cfc', `component {}\n`); + write('app/controllers/Main.cfc', `component extends="wheels.Controller" {}\n`); + const edges = await load(); + expect(has(edges, 'User', 'Model', 'Model.cfc')).toBe(true); + expect(edges.filter((e) => e.src === 'Helper' || e.src === 'Main')).toHaveLength(0); + }); }); diff --git a/src/extraction/languages/cfscript.ts b/src/extraction/languages/cfscript.ts index 68cd21aeb5..fc951ad7af 100644 --- a/src/extraction/languages/cfscript.ts +++ b/src/extraction/languages/cfscript.ts @@ -19,8 +19,9 @@ function cfmlVisibility(node: SyntaxNode): 'public' | 'private' | 'protected' | /** * A string literal's text when it is written out in full, or null when a - * `#…#` expression builds it at runtime (`"#dir#/x.cfm"`): such a path names - * no file the index can link. + * `#…#` expression builds it at runtime (`"#dir#/x.cfm"`, `"#dir#.Batch"`): + * such a path names no file, and such a name no component, the index can + * link. */ function literalString(node: SyntaxNode | null | undefined, source: string): string | null { if (node?.type !== 'string' || node.namedChildren.some((c: SyntaxNode) => c.type !== 'string_fragment')) return null; @@ -55,6 +56,56 @@ function cfIncludeCallTemplate(node: SyntaxNode, source: string): string | null return literalString(args.namedChildren[0], source); } +/** A component name or dotted component path (`Batch`, `contenscms.core.queue.Batch`). */ +const COMPONENT_PATH = /^[A-Za-z_][\w$-]*(?:\.[A-Za-z_][\w$-]*)*$/; + +/** + * The component `createObject("component", "a.b.Batch")` makes — or the + * one-argument `createObject("a.b.Batch")` (Lucee, ACF), or with named + * arguments, `createObject(type="component", component="a.b.Batch")` — as + * written (#2652). Null for any other call, for another type of object + * (`createObject("java", …)`), and for a name built at runtime. + */ +export function cfmlCreatedComponent(call: SyntaxNode, source: string): string | null { + if (call.type !== 'call_expression') return null; + const fn = getChildByField(call, 'function'); + if (fn?.type !== 'identifier' || getNodeText(fn, source).toLowerCase() !== 'createobject') return null; + const args = getChildByField(call, 'arguments')?.namedChildren ?? []; + let type: string | null | undefined; + let name: string | null | undefined; + if (args.length > 0 && args.every((a: SyntaxNode) => a.type === 'assignment_expression')) { + for (const arg of args) { + const key = getNodeText(getChildByField(arg, 'left') ?? arg, source).toLowerCase(); + if (key === 'type') type = literalString(getChildByField(arg, 'right'), source); + else if (key === 'component') name = literalString(getChildByField(arg, 'right'), source); + } + } else if (args.length === 1) { + name = literalString(args[0], source); + } else if (args.length >= 2) { + type = literalString(args[0], source); + name = literalString(args[1], source); + } + if (type !== undefined && type?.toLowerCase() !== 'component') return null; + return name && COMPONENT_PATH.test(name) ? name : null; +} + +/** + * The component an expression constructs, when the expression is a + * construction a call is chained onto: `new a.b.Batch()` or a component + * `createObject(…)`, parenthesized or not (#2652). + */ +export function cfmlConstructedComponent(node: SyntaxNode, source: string): string | null { + let expr: SyntaxNode | null = node; + while (expr?.type === 'parenthesized_expression') expr = expr.namedChild(0); + if (!expr) return null; + if (expr.type === 'new_expression') { + const ctor = getChildByField(expr, 'constructor'); + const name = ctor ? getNodeText(ctor, source).replace(/\s+/g, '') : ''; + return COMPONENT_PATH.test(name) ? name : null; + } + return cfmlCreatedComponent(expr, source); +} + export const cfscriptExtractor: LanguageExtractor = { functionTypes: ['function_declaration', 'function_expression', 'arrow_function'], classTypes: ['component'], diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index c0449a40d7..dd6a81869e 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -27,7 +27,7 @@ import { appendCppMacroMethods } from './languages/cpp-macro-methods'; import { rustImplItemOwner, rustImplTypeName } from './languages/rust'; import { goAliasTypeNames, goEmbeddedTypeName, goInterfaceMethodTypeNames, goStructFieldTypeNames } from './languages/go'; import { dartMisparsedGenericCall, dartReceiverOf, isDartTypeName, pushDartTypeAliasRefs, pushDartTypeRefs } from './languages/dart'; -import { isCfIncludeCall } from './languages/cfscript'; +import { cfmlConstructedComponent, cfmlCreatedComponent, isCfIncludeCall } from './languages/cfscript'; import { LiquidExtractor } from './liquid-extractor'; import { RazorExtractor } from './razor-extractor'; import { SvelteExtractor } from './svelte-extractor'; @@ -5174,6 +5174,24 @@ export class TreeSitterExtractor { return; } + // CFML `createObject("component", "a.b.Batch")` makes an instance of the + // component, as `new a.b.Batch()` does (#2652): an instantiation of it, + // under the path as written, which the resolver reads as `extends`' + // component paths. Not a call: createObject is the engine's own function. + if (this.language === 'cfscript') { + const created = cfmlCreatedComponent(node, this.source); + if (created) { + this.unresolvedReferences.push({ + fromNodeId: callerId, + referenceName: created, + referenceKind: 'instantiates', + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + return; + } + } + // VB.NET: `foo(args)` is syntactically ambiguous between a call and an // index read, so the grammar parses non-empty parens as // array_access_expression (field `array`, not `function`) — even Roslyn @@ -5956,6 +5974,7 @@ export class TreeSitterExtractor { return; } const SKIP_RECEIVERS = new Set(['self', 'this', 'cls', 'super']); + let constructed: string | null = null; if (receiver && (receiver.type === 'identifier' || receiver.type === 'simple_identifier' || receiver.type === 'field_identifier')) { const receiverName = getNodeText(receiver, this.source); if (!SKIP_RECEIVERS.has(receiverName)) { @@ -6079,6 +6098,14 @@ export class TreeSitterExtractor { // (`!Settings.Defaults.isOn()`). Mirrored in the kernel's // extract_call (swift.rs). calleeName = `${swiftUnwrappedText(receiver, this.source).replace(/\s+/g, '')}.${methodName}`; + } else if (this.language === 'cfscript' && receiver && (constructed = cfmlConstructedComponent(receiver, this.source))) { + // CFML call on the component the receiver constructs — + // `createObject("component", "a.b.Batch").init()`, `new + // a.b.Batch().init()` (#2652). Encode `().`: + // the resolver finds the component as `new`/`extends` do and the + // method on it, or nothing. The bare `init` this used to emit + // matched whichever component's `init` came first. + calleeName = `${constructed}().${methodName}`; } else if ( this.language === 'cfscript' && receiver && diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 536f86894f..b95b2660d7 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -97,6 +97,9 @@ const CHAIN_SHAPE = /^(.+)\(\)\.(\w+)$/; /** PHP `$this->prop->method()` encoded as `this->prop.method` — no `()`, so CHAIN_SHAPE misses it. */ const PHP_PROP_SHAPE = /^this->\w+\.\w+$/; +/** A CFML call on a construction, `a.b.Batch().init` (#2652); CFML names may hold `$` (`$init`). */ +const CFML_CONSTRUCTED_CALL = /^([A-Za-z_$][\w$.-]*)\(\)\.([A-Za-z_$][\w$]*)$/; + /** * How many files that import a changed module by default a sync reads the * edges of, when the module forwards its default from another file, to see @@ -1567,6 +1570,15 @@ export class ReferenceResolver { ) { return this.resolveCfmlComponentPath(ref); } + // The component a CFML `createObject("component", "a.b.Batch")` makes, + // named by its component path as `extends` names one (#2652), and a call + // chained onto a construction — `createObject(…).init()`, `new + // a.b.Batch().init()`, encoded `a.b.Batch().init` — which is a method of + // that component or of nothing: no name match may guess past it. + if (ref.language === 'cfml' || ref.language === 'cfscript') { + if (ref.referenceKind === 'instantiates' && ref.referenceName.includes('.')) return this.resolveCfmlComponentPath(ref); + if (ref.referenceKind === 'calls' && CFML_CONSTRUCTED_CALL.test(ref.referenceName)) return this.resolveCfmlConstructedCall(ref); + } // A CFML include names a template by its path from the including file's // directory, or from the web root (#2651): ` (n.kind === 'class' || n.kind === 'interface') && (n.language === 'cfml' || n.language === 'cfscript')); + const dirOf = (p: string): string => p.slice(0, p.lastIndexOf('/') + 1); + const beside = components.filter((n) => dirOf(n.filePath) === dirOf(fromFile)); + if (beside.length === 1) return beside[0]; + return components.length === 1 ? components[0] : undefined; + } + + /** A CFML component's method by its lowercased name (CFML names ignore case): its own, else one a supertype has. */ + private cfmlMethodOf(cls: Node, lowerName: string, seen: Set): Node | undefined { + if (seen.has(cls.id) || seen.size >= 8) return undefined; + seen.add(cls.id); + const own = this.context.getNodesInFile(cls.filePath).find((n) => + n.kind === 'method' && + n.name.toLowerCase() === lowerName && + n.qualifiedName.toLowerCase() === `${cls.name}::${n.name}`.toLowerCase() && + n.startLine >= cls.startLine && n.endLine <= cls.endLine + ); + if (own) return own; + for (const supertype of this.context.getSupertypeNodes?.(cls.id) ?? []) { + const inherited = this.cfmlMethodOf(supertype, lowerName, seen); + if (inherited) return inherited; + } + return undefined; } /** From ef05970bb4e51c5ee43fa3ec365a525408498d53 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 12:17:25 +0000 Subject: [PATCH 67/95] test(daemon): the OS process lookup tests pass on a loaded Windows machine (#2842) daemon-lock-holder.test.ts > "reads a running process from the OS" failed under load on Windows, and so did the seven CLI tests in the file that look a process up. describeProcess gives each OS query 5 s: on a busy machine PowerShell takes longer than that to start and run its CIM query, so the lookup falls back to tasklist (the image name only, or nothing once tasklist runs out too) and the command line is null. The in-process test's toMatch(/ping/) then fails, and the CLI the other tests spawn sees a Node holder it can't classify, keeps the lock, and says so. In production that fallback is the intended fail-safe: an undescribed holder stays "unknown" and its lock is kept. The per-query budget is still 5 s unless CODEGRAPH_PROCESS_LOOKUP_TIMEOUT_MS says otherwise, and nothing outside the tests sets it. The test file sets it to 60 s for itself and for every CLI it spawns (an env var reaches both), gives the lookup tests time to match (240 s per CLI run, 30 s for a killed holder to go), and adds a Windows-only test that a lookup that runs out says nothing, so the holder stays "unknown". Validation (Windows, 16 logical CPUs, a 48-thread CPU burn running): - describeProcess on a live PING.EXE: the 5 s budget answered nothing in 8 of 8 lookups (about 11 s each); a 60 s budget read the command line in 6 of 6 (median 12.6 s, max 13.6 s). - The reported test alone: main 0 of 4, each "expected '' to match /ping/i"; this branch 2 of 2, and 6 of 6 with an earlier revision that passed the same 60 s budget as an argument. - The whole file: main 3 of 11, the 8 failures being exactly the tests that look a process up; this branch 12 of 12, twice. - No load: 12 of 12 in 20 s. Co-Authored-By: Claude Opus 5.5 --- __tests__/daemon-lock-holder.test.ts | 46 +++++++++++++++++++++++----- src/mcp/process-identity.ts | 13 +++++++- 2 files changed, 50 insertions(+), 9 deletions(-) diff --git a/__tests__/daemon-lock-holder.test.ts b/__tests__/daemon-lock-holder.test.ts index 5ab11539af..b9022d9774 100644 --- a/__tests__/daemon-lock-holder.test.ts +++ b/__tests__/daemon-lock-holder.test.ts @@ -12,7 +12,7 @@ * CodeGraph server keeps its lock, and the message names that process and a * command that stops it. */ -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'vitest'; import { spawn, spawnSync } from 'child_process'; import * as fs from 'fs'; import * as os from 'os'; @@ -32,6 +32,23 @@ const windows = process.platform === 'win32'; const holders: number[] = []; let root: string; +/** + * Each OS lookup gets a minute here, in this process and in the CLIs it runs, + * not the 5 s CodeGraph gives one: on a loaded machine PowerShell can take + * longer than that to start, and a Windows lookup that runs out falls back to + * tasklist, which has no command line to show. The tests that look a process + * up allow for it. + */ +const LOOKUP_ENV = 'CODEGRAPH_PROCESS_LOOKUP_TIMEOUT_MS'; +const savedLookupTimeout = process.env[LOOKUP_ENV]; +beforeAll(() => { + process.env[LOOKUP_ENV] = '60000'; +}); +afterAll(() => { + if (savedLookupTimeout === undefined) delete process.env[LOOKUP_ENV]; + else process.env[LOOKUP_ENV] = savedLookupTimeout; +}); + /** * Start a long-lived process to be named by the lock: `native` is no * JavaScript runtime, `node` is Node running something else, and @@ -57,7 +74,7 @@ function runCli(args: string[]) { return spawnSync(process.execPath, [BIN, ...args], { cwd: root, encoding: 'utf8', - timeout: 60_000, + timeout: 240_000, env: { ...process.env, CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_TELEMETRY: '0', DO_NOT_TRACK: '1', CODEGRAPH_NO_PROMPT_HOOK: '1' }, }); } @@ -132,9 +149,22 @@ describe('what a lock holder is (#2770)', () => { expect(classifyLockHolder(nativeInfo)).toBe('other'); expect(classifyLockHolder(await describeProcess(server))).toBe('codegraph-server'); process.kill(native, 'SIGKILL'); - const end = Date.now() + 10_000; + const end = Date.now() + 30_000; while (isProcessAlive(native) && Date.now() < end) await new Promise((r) => setTimeout(r, 50)); expect(classifyLockHolder(await describeProcess(native))).toBe('unknown'); + }, 240_000); + + // Windows starts no process within a millisecond, so both of its queries run out. + it.runIf(windows)('a lookup that runs out of time says nothing, so the holder stays unknown', async () => { + const native = startHolder('native'); + process.env[LOOKUP_ENV] = '1'; + try { + const info = await describeProcess(native); + expect(info).toEqual({ name: null, commandLine: null }); + expect(classifyLockHolder(info)).toBe('unknown'); + } finally { + process.env[LOOKUP_ENV] = '60000'; + } }, 30_000); }); @@ -157,7 +187,7 @@ describe('codegraph index and unlock with a daemon lock naming a live process (# const cg = CodeGraph.openSync(root); try { expect(cg.searchNodes('replacementSymbol').length).toBeGreaterThan(0); } finally { cg.close(); } - }, 60_000); + }, 300_000); it('unlock clears an old-format lock naming a process that is not CodeGraph, leaving the process alone', () => { const holder = startHolder('native'); @@ -169,7 +199,7 @@ describe('codegraph index and unlock with a daemon lock naming a live process (# expect(unlock.stdout).toContain('Removed stale lock artifacts'); expect(fs.existsSync(lockPath)).toBe(false); expect(isProcessAlive(holder)).toBe(true); - }, 60_000); + }, 300_000); it.each([ ['an old-format lock', writeLegacyLock], @@ -190,11 +220,11 @@ describe('codegraph index and unlock with a daemon lock naming a live process (# // The advice: stop that process, then retry `codegraph index`. process.kill(holder, 'SIGKILL'); - const end = Date.now() + 10_000; + const end = Date.now() + 30_000; while (isProcessAlive(holder) && Date.now() < end) await new Promise((r) => setTimeout(r, 50)); const retry = runCli(['index', '--quiet', root]); expect(retry.status, retry.stderr).toBe(0); - }, 90_000); + }, 600_000); it('names the stale writer lock when the same process that is not CodeGraph holds it too', () => { const holder = startHolder('native'); @@ -212,7 +242,7 @@ describe('codegraph index and unlock with a daemon lock naming a live process (# const retry = runCli(['index', '--quiet', root]); expect(retry.status, retry.stderr).toBe(0); expect(isProcessAlive(holder)).toBe(true); - }, 90_000); + }, 600_000); it('the writer-lock message names `codegraph daemon`, a command the CLI accepts', () => { expect(tryAcquireWriterLock(root, 'daemon').kind).toBe('acquired'); diff --git a/src/mcp/process-identity.ts b/src/mcp/process-identity.ts index db8113bb60..834cabd39f 100644 --- a/src/mcp/process-identity.ts +++ b/src/mcp/process-identity.ts @@ -35,11 +35,22 @@ export type LockHolderKind = 'codegraph-server' | 'other' | 'unknown'; const LOOKUP_TIMEOUT_MS = 5_000; +/** + * How long one OS query may run: 5 s, unless `CODEGRAPH_PROCESS_LOOKUP_TIMEOUT_MS` + * says otherwise. Tests set it: on a loaded machine PowerShell can take longer + * than 5 s to start, and a Windows lookup that runs out falls back to tasklist, + * which has no command line to show. + */ +function lookupTimeoutMs(): number { + const fromEnv = Number(process.env.CODEGRAPH_PROCESS_LOOKUP_TIMEOUT_MS); + return Number.isFinite(fromEnv) && fromEnv > 0 ? fromEnv : LOOKUP_TIMEOUT_MS; +} + /** Runs a short OS query and resolves its stdout, or null when it fails or times out. */ function query(file: string, args: string[]): Promise { return new Promise((resolve) => { try { - execFile(file, args, { encoding: 'utf8', timeout: LOOKUP_TIMEOUT_MS, windowsHide: true, maxBuffer: 1 << 20 }, + execFile(file, args, { encoding: 'utf8', timeout: lookupTimeoutMs(), windowsHide: true, maxBuffer: 1 << 20 }, (error, stdout) => resolve(error ? null : stdout)); } catch { resolve(null); From 6500a3782b19522e52be93aba7f55f93141c1932 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 23 Oct 2026 12:32:36 +0000 Subject: [PATCH 68/95] fix(cfml): a or block that fails to parse no longer flags the file (#2843) Both delegates in cfml-extractor.ts read a block with its own grammar through a fresh TreeSitterExtractor and copied its errors onto the file. That extractor warns "parse produced no symbols (tree has errors) - the file is indexed but contributes nothing to the graph" when its tree has errors and it found no symbols: true of the block, not of the file, and a `` body never declares a symbol. So `codegraph init` printed the warning and `codegraph status` counted the file among those whose symbols are missing, though its component, methods and calls were indexed. extractTagExpressions already drops the warning for the same reason. Fix: the delegates record that warning as one naming the block and its line, with code partial_parse, so `codegraph files --json` shows which block failed while init and status, which key on parse_error, no longer report the file. Other errors pass through unchanged. The issue's own example SQL parses cleanly with the bundled cfquery grammar, so the reporter's trigger was not reproduced; any query body the grammar can't read (here `EQV` in a condition) and any statement-only block with a syntax error showed the same warning. Validation: - New cfml-embedded-parse-warning.test.ts: the repro DAO keeps its class, both methods and the call inside the failing query, and records one partial_parse warning naming at line 6; a failing block likewise; getIndexHealth lists neither file. All 3 fail on main's extractor and pass. status-index-health.test.ts passes. - On the repro, main prints the whole-file warning during init and status says 1 file could not be parsed so its symbols are missing; with the fix neither does. - A/B vs main f406fe00: FW/1, BlogCFC, ColdBox, ContentBox, Preside and CFWheels unchanged; in Wheels' vendor/wheels core (indexed as its own project) one file changed, tests/html.cfm, whose block at line 8 fails: its warning now names that block. Nodes, edges and references are identical in every repo. express and flask controls byte-identical. - Index time, median of 4 interleaved runs, CPU with workers, main vs fix: Preside 57.1 / 54.5 s, Wheels core 33.2 / 32.8 s. Within noise. Fixes #2654 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cfml-embedded-parse-warning.test.ts | 93 +++++++++++++++++++ src/extraction/cfml-extractor.ts | 33 +++++-- 3 files changed, 121 insertions(+), 6 deletions(-) create mode 100644 __tests__/cfml-embedded-parse-warning.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 044d6c6cb2..37bfa09456 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -63,6 +63,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - For Hermes Agent, `codegraph install` and `upgrade` no longer add `mcp-codegraph` to `platform_toolsets.cli`. Hermes up to 0.22.2 reported that name as an unknown toolset on every start, and Hermes 0.22.4 reads it as the only MCP server to load, hiding your others from CLI sessions; an entry an earlier install added is taken out again. A refresh also no longer resets your own `mcp_servers.codegraph` settings, such as an absolute command path, timeouts or `enabled: false`, and a config indented or written differently from Hermes's default no longer gets a duplicate or unreadable entry. Thanks @Huy3ko for the report and @conifercone for confirming it. (#2485) - `codegraph upgrade` now upgrades a global npm install when run from your home folder, which holds npm's global folder on Windows and with tools like nvm: before, it ran `npm install` without `-g` there, which left the global copy on the old version and put a second copy in your home folder. A copy installed in a project is now upgraded in that project wherever you run the command, with a warning if npm left it on the old version, and a copy installed with pnpm, yarn or bun gets the command that upgrades it instead of npm. Thanks @timlincool for the report and @maxmilian for the first fix. (#2477) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) +- In CFML, a `` or `` block the parser can't read no longer marks its whole file as contributing nothing to the graph: `codegraph init` no longer warns about the file, and `codegraph status` no longer lists it among files whose symbols are missing, since its component, functions and calls are all indexed. The file's recorded errors now name the block and its line instead. Thanks @HarryMuc for the report. (#2654) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) - In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) - In Vue and Nuxt components, calls written in the template (`{{ formatDate(post) }}`, `:to="getAccountRoute(account)"`, `@click="save(item)"`) and calls in a destructuring declaration at the top of `