Skip to content

gh-92041: Avoid registry scans for uncached registered frames - #160254

Open
mdeck wants to merge 4 commits into
python:mainfrom
mdeck:inspect-registered-frame-fastpath-20261022
Open

mdeck wants to merge 4 commits into
python:mainfrom
mdeck:inspect-registered-frame-fastpath-20261022

Conversation

@mdeck

@mdeck mdeck commented Oct 22, 2026 •

Copy link
Copy Markdown

When inspect.getmodule() encounters an uncached registered frame or traceback, it currently scans sys.modules to populate its filename map. Resolve that case directly from the frame's globals owner when both the registered-module identity and canonical filename match, then reuse the existing filename cache.

This is the frame-path split requested in #92042. It is independent of and complementary to #156685, which removes a redundant scan during key computation. Both changes compose cleanly in either order. This PR changes only the Python frame lookup path; it does not include registry watchers, C changes or negative caching.

The existing getabsfile(module) canonicalizer preserves source/bytecode normalization. Private execution namespaces, mismatched origins, module subclasses and dynamically supplied filenames, fileless modules and unrelated filename overrides retain the filename fallback. Existing filename-cache hits retain precedence. On an uncached filename collision, the verified globals owner wins over an unrelated module using the same filename. A later unrelated miss can repopulate the filename map from the registry and change that result; code-object lookups also share this map. This lookup-history dependence is documented and explicitly tested.

Performance

The original, source-pinned measurement matrix used one current-main CPython 3.16 -O3 Linux build, 15 interleaved fresh-process pairs per registry size, with 10,000 synthetic modules added to the ordinary registry:

Workload #156685 with its narrow bytecode fix Both changes Paired speedup, 95% CI
Cold registered stack, source context enabled, depth 2 96.12 ms 202.7 µs 455.7× (433.3–482.1)
Newly registered filename, other filename entries warm 2.051 ms 5.185 µs 393.9× (377.8–414.5)

The bytecode fix is a separate measurement variant, not part of this PR. The exact uncorrected #156685 also shows the independent cold registered-frame gain: 87.01 ms → 4.660 µs, paired speedup 19,427.8× (18,548.4–20,458.2).

These are workload-specific synthetic measurements. Cold tests reset inspect's filename dictionaries, not the interpreter or filesystem caches. The import-like test retains other warmed entries and includes inserting/removing the target registry entry and evicting its own filename metadata. Mixed stacks containing an unregistered generated frame still require a full scan and receive no comparable benefit. We claim no steady-state stack or moduleless-miss speedup.

Eight further fresh-process crossover runs with counterbalanced within-process ABBA/BAAB batches found no meaningful regression in the tested warm paths. For the exact current replacement versus both changes: warmed frame ratio 1.002× (0.991–1.010), warmed source-context stack ratio 1.006× (0.997–1.016). Identical-code A/A controls are included in the evidence; small cross-process changes are not treated as gains.

A separate before/after recheck of the review revisions uses eight fresh processes for cold workloads and 16 fresh processes for refined warm-path measurements, with counterbalanced within-process batches and identical-code controls. It finds no statistically resolved change: cold-stack ratio 1.006× (0.997–1.014), newly registered frame 0.998× (0.986–1.010), warmed frame 0.992× (0.981–1.004), warmed stack 0.999× (0.986–1.011). These intervals leave smaller effects unresolved; no additional gain is claimed. The existing cache-hit instruction prefix is identical. Scripts, snapshots, raw samples and validation logs.

Validation

  • Current revision: 1,138 tests reported, 4 skipped; all six suites pass (inspect, zipimport support, import, linecache, traceback and pydoc). All 16 frame-owner regression methods also pass composed with the bytecode-corrected replacement.
  • 648 differential cases: zero differences versus main for this candidate; the raw composition has exactly the replacement's 36 known bytecode differences, and the corrected composition has zero.
  • Fresh-namespace zipimport and getsource regression check passes for all six variants.
  • The new stored-filename/module-__getattr__ no-scan test fails on the parent for the expected scan and passes on this revision. 300 repeated dynamic-filename, dynamic-loader and module-subclass tests pass across parent and revision.
  • Documentation lint (sphinx-lint --enable=default-role) and git diff --check pass.

Measurement scripts, complete raw process samples, intervals, controls, exact sources/hashes and logs: https://github.com/mdeck/cpython/tree/inspect-frame-fastpath-evidence-20261022/Tools/benchmarks/inspect_getmodule_92041.

Developed and tested with Codex (AI assistance).

mdeck added 2 commits October 22, 2026 09:44
Resolve a frame or traceback from its registered globals owner only when
its canonical filename matches. Reuse the existing filename cache and
retain the fallback for other objects and namespaces.

Developed and tested with Codex (AI assistance).
The fixture left a stale __file__ value beneath a module subclass property.
LOAD_ATTR_MODULE specialization can read that value, making the assertion
depend on warmup state in the existing filename fallback, including main.

Remove the shadow value and exercise warmed getfile access, two changing
origins, and independent cold frame/traceback lookups. Keep production code
unchanged.

Verified the corrected test 50 times on each of main and the candidate;
all six focused suites pass (1,135 tests reported, four skipped).

mdeck commented Oct 22, 2026

Copy link
Copy Markdown
Author

@eendebakpt, would you be willing to review this follow-up to #92042? It implements the frame/traceback split you suggested, without the registry watchers or C changes, and is independent of and complementary to #156685.

CI is green; regression coverage and reproducible benchmarks are linked in the description. I’d particularly appreciate your view on verified frame globals taking precedence for an uncached filename collision. Thanks for your earlier feedback.

Comment thread Lib/inspect.py Outdated
Comment on lines +947 to +948
except (TypeError, FileNotFoundError):
pass

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
except (TypeError, FileNotFoundError):
pass
except FileNotFoundError, TypeError:
pass
return None

Comment thread Lib/inspect.py Outdated
Comment on lines +940 to +942
if (type(module) is not types.ModuleType or
module.__dict__ is not namespace or '__getattr__' in namespace or
type(namespace.get('__file__')) is not str):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (type(module) is not types.ModuleType or
module.__dict__ is not namespace or '__getattr__' in namespace or
type(namespace.get('__file__')) is not str):
if (type(module) is not types.ModuleType
or module.__dict__ is not namespace
or '__getattr__' in namespace
or type(namespace.get('__file__')) is not str):

Comment thread Lib/inspect.py Outdated
Comment on lines +968 to +969
frame = object if isframe(object) else (
object.tb_frame if istraceback(object) else None)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
frame = object if isframe(object) else (
object.tb_frame if istraceback(object) else None)
if isframe(object):
frame = object
elif istraceback(object):
frame = object.tb_frame
else:
frame = None

@picnixz
picnixz requested a review from eendebakpt October 22, 2026 18:23
Comment thread Lib/inspect.py Outdated
module = sys.modules.get(name)
# Leave dynamic module metadata to the existing filename fallback.
if (type(module) is not types.ModuleType or
module.__dict__ is not namespace or '__getattr__' in namespace or

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is '__getattr__' in namespace ever needed?

@property
def __file__(self):
return self.source
# Do not leave a shadow value that LOAD_ATTR_MODULE can read instead

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be #156474

@@ -0,0 +1,3 @@
Speed up :func:`inspect.getmodule` for frames and tracebacks whose globals
belong to a registered module with a matching filename. The globals owner

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This promises a bit more than the code does. For example:

import inspect, os, sys, tempfile, types

fn = os.path.join(tempfile.mkdtemp(), 'owner.py')
open(fn, 'w').close()
owner = types.ModuleType('owner_mod'); owner.__file__ = fn
other = types.ModuleType('other_mod'); other.__file__ = fn
sys.modules['owner_mod'] = owner
sys.modules['other_mod'] = other        # inserted later -> wins a registry scan

exec(compile("import sys\nframe = sys._getframe()\n", fn, 'exec'), owner.__dict__)
frame = owner.__dict__['frame']

print("1 frame, cold          :", inspect.getmodule(frame).__name__)
inspect.getmodule(compile("pass", "<nowhere>", "exec"))   # unrelated miss
print("2 frame, after a miss  :", inspect.getmodule(frame).__name__)
inspect.modulesbyfile.clear(); inspect._filesbymodname.clear()
print("3 frame, cold again    :", inspect.getmodule(frame).__name__)
inspect.modulesbyfile.clear(); inspect._filesbymodname.clear()
print("4 code object, cold    :", inspect.getmodule(frame.f_code).__name__)
inspect.modulesbyfile.clear(); inspect._filesbymodname.clear()
inspect.getmodule(frame)
print("5 code obj after frame :", inspect.getmodule(frame.f_code).__name__)

This is a pathological case (intentionally giving two modules the same filename), so maybe fine to keep the code as is. But rewrite the news entry.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also please indicate this change in the docs of inspect directly. This may be breaking for some people.

@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #35337785 | 📁 Comparing ef27738 against main (5b55b50)

  🔍 Preview build  

2 files changed
± library/inspect.html
± whatsnew/changelog.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants