Repository navigation
Commit ff4959b for pdb colorization GH-113081 broke some pdb backtraces #142964
Description
Bug report
Bug description:
If you put a breakpoint here:
diff --git a/Lib/tokenize.py b/Lib/tokenize.py
index 117b485b934..9d215ea4ea1 100644
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -574,6 +574,7 @@ def _transform_msg(msg):
def _generate_tokens_from_c_tokenizer(source, encoding=None, extra_tokens=False):
"""Tokenize a source reading Python code as unicode strings using the internal C tokenizer"""
+ breakpoint()
if encoding is None:
it = _tokenize.TokenizerIter(source, extra_tokens=extra_tokens)
else:and run this:
./python.exe -m test test_tokenize -m test_random_files -uall -v
❯ ./python.exe -m test test_tokenize -m test_random_files -uall -v
== CPython 3.14.0a7+ (bisect/good-2bbcaedb75942389dacb51866948f40de5951c9c-4-g08d7687094c-dirty:08d768) [Clang 17.0.0 (clang-1700.6.3.2)]
== macOS-26.2-arm64-arm-64bit-Mach-O little-endian
== Python build: release
== cwd: /Users/pablogsal/github/python/3.14/build/test_python_worker_11047æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all
Using random seed: 1175774464
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 4.01 Run 1 test sequentially in a single process
0:00:00 load avg: 4.01 [1/1] test_tokenize
> /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb) c
test_random_files (test.test_tokenize.TestRoundtrip.test_random_files) ... > /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb)
> /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb) bt
(Pdb)
pdb cannot show the stack. Bisecting points to
ff4959b is the first bad commit (for #113081 / #133355)
commit ff4959b (HEAD)
Author: Tian Gao gaogaotiantian@hotmail.com
Date: Mon May 5 00:49:52 2025 -0700
gh-113081: Highlight source code in pdb (#133355)
Doc/library/pdb.rst | 8 +++-
Doc/whatsnew/3.14.rst | 5 ++
Lib/pdb.py | 31 +++++++++---
Lib/test/test_pdb.py | 37 +++++++++++++++
...-18-48-54.gh-issue-113081.JsLJ1X.rst | 1 +
5 files changed, 74 insertions(+), 8 deletions(-)
create mode 100644 Misc/NEWS.d/next/Library/2025-05-03-18-48-54.gh-issue-113081.JsLJ1X.rst
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
gaogaotiantian commentedon Dec 19, 2025
Okay I'm not 100% sure if this is considered a bug.
pdbuses some stdlib,tokenizeis one of them. If you put abreakpoint()in the source code, there will be some issues because of nested debugging. To be more specific, to colorize the source code,_pyreplis used and it usestokenize. If you do a soft breakpoint there (byb tokenize.py:lineno), it should work because tracefunc is not triggered. This is not an end-user issue because users can't edit python source code directly. Do you still consider this a bug and think we should fix it?[-]Commit ff4959b6b0cf6da5d2f59260a9103b31c812b0ba broke some pdb backtraces[/-][+]Commit ff4959b for pdb colorization GH-113081 broke some pdb backtraces[/+]gpshead commentedon Dec 22, 2025
could it make sense for
breakpoint()to understand when it has been invoked from stdlib code that the debugger it's about to drop into relies on and disable the fancier pyrepl features? (I don't know what the right way to do this would be... simple heuristic about being a breakpoint in stdlib code or within in a specific set of stdlib modules - maintaining that list wouldn't be too hard, we'd be the only ones using that and could add new things if they ever come up)gaogaotiantian commentedon Oct 20, 2026
This is fixed in #159918 .