Repository navigation
Slow inspect.getmodule #92041
Description
Bug report
inspect.getmodule is slow. It becomes slower as len(sys.modules) grows. It is slower with modules that don't resolve to valid file paths. This causes inspect.stack and others to be slow as well.
def run_test(module):
# cols have stack depths
# rows have len(sys.modules)
def print_line(h, *vals):
if not h:
print("%5s" % h, *["%6i" % v for v in vals])
else:
print("%5s" % h, *["%6.1f" % v for v in vals])
def add_modules(n):
import random
import sys
for _ in range(n):
sys.modules[f"foo_{random.randint(0,2**64)}"] = module
def measure(depth):
import inspect
import timeit
def nest(level):
if level > 0:
return nest(level-1)
else:
dur = timeit.timeit(lambda: inspect.stack(), number=1)
return dur * 1000
return nest(depth)
import sys
print_line("", *[2**e for e in range(7)])
for i in range(1, 5):
add_modules(n=10**i)
times = [measure(depth=2**e) for e in range(7)]
print_line(len(sys.modules), *times)
python 3.11.0a7+ optimized build ====
# sys has no __file__ and doesn't enter getmodule cache
>>> import sys
>>> run_test(sys)
1 2 4 8 16 32 64
87 2.1 0.8 1.0 1.3 2.0 3.3 6.1
187 2.0 2.5 2.7 3.6 5.5 9.5 16.8
1187 13.6 15.7 19.1 26.3 39.6 67.4 123.3
11187 134.1 150.3 183.4 252.0 383.5 750.3 1189.8
>>> exit()
>>> import random
>>> run_test(random)
1 2 4 8 16 32 64
87 2.1 0.7 0.9 1.2 1.7 2.9 5.6
187 2.3 1.1 1.5 1.8 2.5 4.3 7.8
1187 17.0 4.3 5.6 7.1 11.1 18.6 33.9
11187 164.6 37.4 45.7 61.6 94.7 161.0 297.7
>>> exit()Tables show time in milliseconds for a single inspect.stack() call.
Rows show len(sys.modules)
Columns indicate stack depth
Your environment
- CPython versions tested on: 3.9, 3.11.0a7+
- Operating system and architecture: MacOS, Linux, x64
Activity
mdeck commentedon May 8, 2022
Looking at the test data, we see
inspect.stack(alsoinspect.getouterframes) can take over 1 second (!!) in worst case with many loaded fileless modules and a deep call stack.Even the typical case performs poorly - 187 modules and 8 frame call stack takes almost 2 milliseconds. Note this is with code pasted into REPL, so all the frames are within uncached moduleless frames (the worst case)
#92042 reduces duration using simple caching to avoid recursion and repeated
sys.modulesiteration.Here is before & after, running test from a file instead of REPL (a more typical use case)
Times are in microseconds.
Built with
./configure --enable-optimizations --with-lto && make(updated to show test with
randominserted intosys.modulesrather thansys, more representative of typical case)mdeck commentedon May 8, 2022
Simply having the moduleless REPL frames somewhere in the stack has a profound effect on times:
(updated to show test with
randominserted intosys.modulesrather thansys, more representative of typical case)asottile-sentry commentedon Jan 26, 2024
haven't tracked down the exact tracebacks that are being processed but this led to a huge slowdown for us at sentry moving from python 3.10 to python 3.11
in one worker type this went from
inspect.stacktaking 4% of execution time to 54%Avoid expensive inspect calls in config validators
Avoid expensive inspect calls in config validators (#112085)
Avoid expensive inspect calls in config validators (#112085)
Speed up the frame helper
Fix scene integration doing blocking I/O in the event loop to import …
pythongh-92041: Resolve frame modules from globals
pythongh-92041: Address getmodule review findings
pythongh-92041: Preserve frame filename semantics
pythongh-92041: Preserve getmodule filename fallback
pythongh-92041: Safely cache inspect.getmodule filename lookups
pythongh-92041: Avoid a second sys.modules scan in inspect.getmodule
pythongh-92041: Avoid a second sys.modules scan in inspect.getmodule
pythongh-92041: Avoid a second sys.modules scan in inspect.getmodule
pythongh-92041: Avoid a second sys.modules scan in inspect.getmodule
pythongh-92041: Avoid a second sys.modules scan in inspect.getmodule
pythongh-92041: Test getmodule of a frame exec'd in a fresh namespace
pythongh-92041: Avoid registry scans for uncached registered frames
pythongh-92041: Address frame lookup review and document cache preced…