Skip to content

gh-158195: Do not hold the type lock during stop-the-world - #159972

Draft
nascheme wants to merge 8 commits into
python:mainfrom
nascheme:gh-158195-unpin-type-lock
Draft

nascheme wants to merge 8 commits into
python:mainfrom
nascheme:gh-158195-unpin-type-lock

Conversation

@nascheme

@nascheme nascheme commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Fixes the free-threaded deadlock described in gh-158195 by removing type-lock pinning, rather than changing how stop-the-world restores critical sections. With this PR, the type lock and class dict lock can now be released normally while a thread waits to stop the world.

Design

Type slot, MRO and __bases__ updates are prepared under TYPE_LOCK and published with the world stopped. Without pinning, another thread can change the state between preparation and publication. Preparation itself can also release the lock or run reentrant Python code, so pinning did not fully protect against stale updates even before this change.

Publication now uses an optimistic retry:

  1. Snapshot a per-interpreter generation counter before preparing an update.
  2. Prepare the slot queue, compute the MRO or validate the new bases.
  3. Stop the world and compare the generation. If it changed, publish nothing and retry preparation; otherwise, publish the update.

The counter advances on slot-relevant class dict mutations, MRO and bases changes, and slot publication. Ordinary class attribute assignments do not advance it. A single counter keeps validation simple, at the cost of possible retries for unrelated type mutations. Existing type version tags cannot serve this purpose: they may already be invalid (zero) throughout preparation.

A stale update must be recomputed, not simply discarded: an unrelated mutation can invalidate the snapshot without doing the work needed to bring this type's slots or MRO up to date. Reentrancy and rollback checks still avoid overwriting an MRO or bases value that has already been replaced.

Supporting changes

  • Slot update queues own strong references to their types, keeping subclasses alive while the locks are released. References are dropped only after the version tags are invalidated, since dropping the last reference to a queued subclass runs type_dealloc(), which takes the type lock again.
  • Version tags are invalidated after publication, while the type lock is still held, so tags assigned during the wait cannot preserve stale cache entries.
  • Hierarchy-dependent __bases__ validation is repeated on retry; argument checks and the audit event run only once per assignment.
  • __bases__ assignment stores a private copy of the assigned tuple, so the rollback on error can tell by identity whether tp_bases was replaced again, even if another assignment stored the same tuple object.
  • Deferred refcounting is enabled only on the MRO tuple that is actually stored, so a rejected candidate is freed normally.

The generation counter and retry checks are free-threaded-only; the GIL build has no generation-tracking overhead.

Backporting

This fix should be backported to 3.14 and 3.15, since the deadlock exists in both. For ABI compatibility, the generation counter could become a process global, rather than inside the interpreter struct.

The change is structured so that the default (GIL) build is essentially unaffected:

  • publish_gen and the types_publish_* helpers exist only under Py_GIL_DISABLED. On the default build the macros expand to constants (gen is 0, stale is 0), so no retry happens.
  • try_set_tp_mro() / try_set_tp_bases() reduce on the default build to the same tp_mro == old_mro and tp_bases == new_bases compares the old code made at the same points (reent in mro_internal(), the undo loop, the bail path).
  • The slot queue code (slot_update_*, apply_type_slot_updates(), the loops in update_slot_after_setattr() and update_all_slots()) was already under Py_GIL_DISABLED.
  • Moving _PyType_Modified_Unlocked() after the flag update in type_set_abstractmethods() and _PyType_SetFlagsRecursive() has no observable effect on the default build: nothing runs in between.
  • Moving the audit hook and the tuple checks before BEGIN_TYPE_LOCK() has no effect there either; the lock is a no-op.
  • __bases__ assignment now stores a copy of the assigned tuple on both builds. The only observable difference is that C.__bases__ is no longer the same object that was assigned.
  • The rollback in mro_hierarchy_for_complete_type() is now conditional on tp_mro still being the MRO just set. This check is needed on the free-threaded build because another thread may replace the MRO while the type lock is released. On the default build, there is no intervening opportunity to run Python code. The conditional rollback therefore has no expected behavioral effect there.

In the free-threaded build, type slot, MRO and __bases__ updates are
prepared under TYPE_LOCK and published with the world stopped.  The lock
was "pinned" so that it stayed held while the thread waited for the
stop-the-world mutex.  That deadlocks against a thread that owns the
stop-the-world mutex and needs TYPE_LOCK to re-attach, which happens
when Python code (a custom mro(), a str subclass __eq__ of a class dict
key, or an automatic GC) runs under the type lock and stops the world.

Remove the pinning and publish optimistically instead.  A type publish
generation counter is incremented by every mutation that a prepared
update might depend on.  The preparing thread snapshots it, prepares,
stops the world and compares.  If the counter changed, nothing is
published and the update is prepared again.
This is more clear than passing the "gen" as a pointer and using
NULL to indicate that we don't want to check the generation.
Use explicit stop-the-world pauses and finite worker workloads. Replace
slot-update stress with a deterministic reentrancy test, and consolidate
queue reference-release coverage.
The rollback on error restores the old bases only if tp_bases is still
the tuple this assignment stored, identified by object identity.  That
breaks if another assignment stores the same tuple object in between.
Store a private copy so that the identity check is reliable.
Required since a dropped reference could run type_dealloc(), which takes
TYPE_LOCK.  Move the final slot_update_clear() after the version tags
are invalidated.
Both helpers now return 1 when stored, 0 when the expected value was
replaced and -1 when the publish generation changed.  The check_gen flag
is replaced by the TYPES_PUBLISH_GEN_NONE sentinel.
The checks are plain C and do not run Python code.
@nascheme
nascheme force-pushed the gh-158195-unpin-type-lock branch from 6a77bee to 7917d6a Compare October 21, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic-free-threading type-bug An unexpected behavior, bug, or error

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant