Skip to content

fix(python): a base list wrapped over several lines is read whole - #3001

Closed
JosefAschauer wants to merge 3 commits into
colbymchenry:mainfrom
JosefAschauer:fix/python-multiline-bases
Closed

JosefAschauer wants to merge 3 commits into
colbymchenry:mainfrom
JosefAschauer:fix/python-multiline-bases

Conversation

@JosefAschauer

Copy link
Copy Markdown
Contributor

Stacked on #2998. This branch contains #2998's commit (89624ed). The change here is the second commit, 3b6b3f8. I'll rebase it once #2998 lands.

Problem

The resolver's own reader of a class's bases (pythonBases, behind module-global and self. dispatch) reads only the class line. black and ruff wrap a long base list one base per line:

class Wrapped(
    Mixin,
    Base,
):
    ...

current = Wrapped()
current.run()   # no edge; the one-line form reaches Base.run

Such a class looked like it had no bases. Extraction already reads the whole list, so the class has its extends edges; only this reader missed them.

Fix

A new pythonBaseList reads from the class line to the list's closing parenthesis, counting nested parentheses, within the class's extent. It leaves out # comments, so a ) in a comment doesn't end the list. pythonBases uses it in place of the single-line regex.

Validation

  • New __tests__/python-multiline-bases.test.ts: module globals over a one-line list (control), a wrapped list, a wrapped list with comments (one ends in :)), and a parameterized base wrapped inside the list. Each resolves run to Base::run, with a decoy Other.run in the file.

    Mutation-checked twice: reading only the first line fails the three wrapped cases, and leaving comments in fails the comment case.

  • npx tsc --noEmit: clean.

  • Full suite on Linux: 8772 passed, 4 failed. These are the same environmental failures as fix(python): a parameterized base links its class, like class Leaf(Base[str]) #2998 (bundle-launcher ×2, cli-sync ×2).

  • Independent review (Grok): SHIP.

🤖 Generated with Claude Code

https://claude.ai/code/session_018hJjBUPFiLSFBssshFwHzq

@colbymchenry

Copy link
Copy Markdown
Owner

Thanks! The module-global case is real on main, your reader fixes it, and the tests pass on the newest main. One more reader has the same gap: the Python branch of declaredSupertypes, behind class-name calls like Wrapped.run(...), reads only three lines and needs the closing parenthesis on them, so a wrapped base list still has no bases there. Reusing pythonBaseList there would close it. Happy to look again once it's ready.

JosefAschauer added a commit to JosefAschauer/codegraph that referenced this pull request Oct 30, 2026
declaredSupertypes, behind a call through the class name like
Wrapped.run(x), read a Python base list from the first three lines and
stopped at the first closing parenthesis, so a list wrapped past them (or
holding a comment with a parenthesis) still had no bases there. It now
reads the list through pythonBaseList, as pythonBases does. Review note
from the maintainer on colbymchenry#3001.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hJjBUPFiLSFBssshFwHzq
@JosefAschauer

Copy link
Copy Markdown
Contributor Author

Thanks, good catch. declaredSupertypes' Python branch now reads the list through pythonBaseList (5d50558). The keyword-argument and factory rewrites (metaclass=…, BaseManager.from_queryset(QuerySet)) are kept as they were.

The test adds class-name calls (Wrapped.run(x)) next to the module-global ones:

  • a one-line control;
  • a wrapped list;
  • a list whose comment holds a );
  • decorated classes, both one-line and wrapped, to confirm a class node starts on its class line, which pythonBaseList relies on.

Without the change, the wrapped, commented and decorated-wrapped cases fail; with it, all 8 pass. On the branch:

  • tsc --noEmit is clean.
  • The full suite gives 8776 passed and 4 failed. The 4 are bundle-launcher (Windows archives) and cli-sync (quiet=true), and they fail the same way without this change.

This branch is still stacked on #2998, so it carries that commit too. Marking both ready.

@JosefAschauer
JosefAschauer marked this pull request as ready for review October 30, 2026 13:59
@JosefAschauer

Copy link
Copy Markdown
Contributor Author

Heads-up on ordering: this follow-up now changes the same case 'python' line in readDeclaredSupertypes as #3000 (type arguments are no supertypes), so whichever of the two lands second will conflict there. The two compose: read the list whole, then drop the brackets. This is the resolution I run locally on main plus both branches:

case 'python': {
  let bases = pythonBaseList(cls, context);
  while (bases && /\[[^[\]]*\]/.test(bases)) bases = bases.replace(/\[[^[\]]*\]/g, '');
  return names(bases?.replace(/\w+\s*=\s*[\w.]+/g, '').replace(/([A-Za-z_][\w.]*)\.\w+\s*\(/g, '$1 ('));
}

With both merged, the full suite passes apart from the known failures: bundle-launcher x2, cli-sync x2, and the explore-exact-target x2 that come from #2076. I'm happy to rebase whichever one is left once the other lands.

colbymchenry pushed a commit that referenced this pull request Oct 30, 2026
…se[str])

A Python class with a parameterized base had no extends edge. The grammar
parses Base[str] in a base list as a subscript, and the base-list reader took
only bare identifiers and attributes, so the base was skipped, on the wasm path
and in the kernel alike. Everything that follows the class hierarchy missed
such a class. The resolver's own text reader of a base list, pythonBases, also
looked up Base[str] verbatim and split Pair[str, int] at its inner comma, so
current = Leaf() then current.run() did not reach Base.run.

A subscript base is now unwrapped to its value before the identifier or
attribute check, in src/extraction/tree-sitter.ts and in the kernel's
python.rs. pythonBases drops bracketed type arguments before splitting the list.
Generic[T] and Protocol[T] stay outside the project like their plain forms; a
computed base, a subscript of a subscript and keyword arguments link nothing.

Validation:
- The test file runs on both arms: red on main's source and kernel (12 of 16
  fail, 6 per arm), green here (16 passed).
- Kernel parity with this kernel on 3,155 Python files of flask, requests,
  mypy, litestar, sqlalchemy, polar and django: 0 diffs, and the same 5 files
  defer to wasm as on main. The kernel and wasm edge dumps of the repos run
  both ways are identical.
- A/B against main on 7 Python repos, 457,948 edges: 1,492 added (901 extends,
  555 calls, 36 references), 0 removed, 607 call sites moved, 223 kept their
  target with higher confidence.
  polar: 491 self.get_one_or_none() style calls move from the protocol stub
  RepositoryProtocol at 0.4 to RepositoryBase and its mixin at 0.9.
  litestar: 12 calls in GenericSyncMockRepository move from
  AbstractAsyncRepository to AbstractSyncRepository.
  sqlalchemy: 435 extends and 554 calls added, like t.insert() reaching
  TableClause.insert.
  We read 44 sampled changes and every small bucket. They read right, except
  12 self.op() sites that move to the typing-only overload SQLCoreOperations.op
  and one Enum.copy that moves to TypeEngine.copy.
- 3 of the 901 extends are wrong: sqlalchemy's Sequence[_T], Tuple[int, ...]
  and MutableSet[_T] from typing link to its own classes of those names. The
  plain form does this on main (28 class X(Enum) there), and none of the 3
  brought a call edge.
- Controls express and gin: full dumps byte-identical.
- Index cost with PR #3001 on top, sqlalchemy, median of 3 interleaved: 23.7 s
  on main, 25.2 s here, with 1,143 more edges.

Original PR #2998 by @JosefAschauer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JosefAschauer and others added 3 commits October 30, 2026 20:24
pythonBases (behind module-global and self. dispatch) read only the
`class` line, so a base list black or ruff wraps one base per line looked
empty and `current = Wrapped(); current.run()` reached no base's method.
A new pythonBaseList reads from the `class` line to the list's closing
parenthesis, within the class, leaving out comments. Extraction already
read the whole list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hJjBUPFiLSFBssshFwHzq
declaredSupertypes, behind a call through the class name like
Wrapped.run(x), read a Python base list from the first three lines and
stopped at the first closing parenthesis, so a list wrapped past them (or
holding a comment with a parenthesis) still had no bases there. It now
reads the list through pythonBaseList, as pythonBases does. Review note
from the maintainer on colbymchenry#3001.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hJjBUPFiLSFBssshFwHzq
@JosefAschauer
JosefAschauer force-pushed the fix/python-multiline-bases branch from 17e2f5b to 00819ff Compare October 30, 2026 20:31
@JosefAschauer

Copy link
Copy Markdown
Contributor Author

Thanks for landing #2998, #3000 and #2999! I've rebased this one onto current main (08b187a). It now carries only its own three commits, and the readDeclaredSupertypes conflict with #3083 is resolved as described above: the list is read whole, then type arguments are dropped. tsc is clean and the Python base tests pass (59 passed). The full suite fails only bundle-launcher x2, cli-sync x2 and gitignore-pattern-parity x4, and those four fail the same way on plain main 08b187a here.

colbymchenry pushed a commit that referenced this pull request Oct 30, 2026
Both of the resolver's readers of a Python base list stopped early. pythonBases,
behind module-global and self. dispatch, read only the class line.
declaredSupertypes, behind class-name calls, read three lines and needed the
closing parenthesis on them. black and ruff wrap a long list one base per line,
so class Wrapped(Mixin, Base) written that way seemed to have no bases:
current = Wrapped() then current.run(), and Wrapped.run(x), reached nothing a
base declares. Extraction already read the whole list.

A new pythonBaseList reads from the class line to the list's closing
parenthesis, counting nested parentheses, within the class's extent, and leaves
out comments, so a ) in one doesn't end the list. Both readers use it.
declaredSupertypes still drops type arguments first, the composition the author
posted on PR #3001, and keeps its metaclass= and factory rewrites.

Validation:
- The test file: 7 of 9 fail without this change (the two one-line controls
  pass), 9 pass here. Putting the type-argument drop out of the merged
  case 'python' fails the new composition case and the two of PR #3083.
- Wrapped.run(None) on a wrapped base list reaches Base::run with the kernel
  and with wasm, as do a list over five lines, a decorated class, and a wrapped
  Base[Optional[User]].
- A/B on top of PR #2998, 7 Python repos, 459,440 edges: 118 added (102 calls,
  16 references), 0 removed, 959 call sites moved, 113 kept their target with
  higher confidence. requests, flask, litestar, mypy and django do not change.
  polar: 942 XRepository.from_session(session) calls move from a sibling
  repository's from_session at 0.65 to RepositoryBase.from_session at 0.85.
  sqlalchemy: wrapped heads like class Table( and class ColumnElement( now
  reach what they inherit: t.select() reaches FromClause.select. Its 17 moved
  sites read right; 15 of them land on the typing-only SQLCoreOperations
  overload of the same ColumnOperators method.
- Controls express and gin: full dumps byte-identical.
- No kernel change. Index cost with PR #2998, sqlalchemy, median of 3
  interleaved: 23.7 s on main, 25.2 s here.

Original PR #3001 by @JosefAschauer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry

Copy link
Copy Markdown
Owner

Thanks @JosefAschauer! This landed as #3103 with you as the commit author, using the combined resolution you posted for case 'python'. It made a big difference on real code: on polar, 942 from_session calls now reach RepositoryBase instead of a sibling repository. We added a test for wrapped bases with type arguments, made one line-split handle CRLF, and reworded the CHANGELOG bullet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants