Skip to content

Leak the Core runtime when it is dropped during interpreter finalization - #2047

Closed
aleakos wants to merge 1 commit into
temporalio:mainfrom
aleakos:fix-runtime-drop-at-finalization
Closed

aleakos wants to merge 1 commit into
temporalio:mainfrom
aleakos:fix-runtime-drop-at-finalization

Conversation

@aleakos

@aleakos aleakos commented Oct 29, 2026

Copy link
Copy Markdown

What was changed

CoreRuntimeHandle::drop (temporalio/bridge/src/runtime.rs) now checks sys.is_finalizing() while attached. If the interpreter is finalizing, or the check itself fails, it leaks the Core runtime with std::mem::forget instead of dropping it. Otherwise the drop is unchanged: it releases the GIL and shuts the runtime down, as in #1998.

Why?

Fixes the exit hang in #2046: a process sometimes never exits after asyncio.run() returns.

At interpreter finalization the final GC frees the last RuntimeRef. Dropping the Core runtime then waits for every Tokio blocking-pool thread. One of those threads can be in pyo3_async_runtimes::generic::set_result, completing a Python future through call_soon_threadsafe, and asking for the GIL. During finalization CPython parks that thread forever in PyThread_hang_thread, whether the GIL is free or not. The drop never returns.

#1998 releases the GIL around the drop, but that does not help here. Python::try_attach still returns Some on the finalizing thread itself, so the drop takes the normal path and blocks. We first tried "leak when try_attach returns None". It still hung 4 times in 76 runs, and gdb showed the drop running inside the try_attach closure. So the finalization state has to be checked explicitly.

Py_IsFinalizing() is in the limited API only from 3.13, and the bridge is built abi3-py310, so the check calls sys.is_finalizing(). It exists in every supported version and needs only the stable ABI. Leaking is safe because the process is exiting: the operating system reclaims the threads and memory.

Checklist

  1. Closes [Bug] Process hangs forever at interpreter exit: Runtime drop waits for a Tokio thread that CPython parks in take_gil #2046

  2. How was this tested:

    All runs on Linux aarch64 (Docker, ghcr.io/astral-sh/uv:python3.13-bookworm), Python 3.13.11. Wheels built with maturin build --release, using CARGO_PROFILE_RELEASE_LTO=off and 16 codegen units because of build-host memory. The reproduction is repro.py from the issue: start a time-skipping environment from an existing binary, shut it down, exit. A run counts as hung only if it printed its last line and was still alive 60 s later.

    Wheel Runs Hangs
    1.35.0 from PyPI 89 3
    1.35.0 from source, unchanged 1 1 (the setup process deadlocked; gdb signature confirmed)
    1.35.0 from source, first attempt (leak only when try_attach returns None) 76 4
    1.35.0 from source, this change 200 0

    At the first attempt's rate (about 5 %), 200 clean runs by chance has a probability of about 3e-5.

    • A script that also runs a Worker with a failing activity and workflow before it exits: 60 runs with this change, 0 hangs, 0 failures.
    • tests/test_runtime.py: 13 passed with this change and 13 passed without it. The same one test fails on both wheels: test_opentelemetry_histogram_bucket_overrides, because opentelemetry-proto was not installed in the test environment.
    • Not run: cargo fmt --check, cargo clippy, the full test suite, other platforms (x86_64, macOS, Windows), Python 3.10 to 3.12.
  3. Any docs updates needed? No.

A deterministic regression test needs a way to force a Tokio thread to complete a Python future during finalization. The reproduction is probabilistic (about 1 hang in 15 to 60 runs), so it is not suitable as a CI test as it stands.

At interpreter finalization a Tokio thread that completes a Python
future asks for the GIL, and CPython parks it for good. Dropping the
Core runtime then waits for that thread forever, so the process never
exits after asyncio.run() returns.

Python::try_attach still attaches the finalizing thread itself, so its
result does not show this state. Check sys.is_finalizing() while
attached and leak the runtime while it is true. The process is exiting,
so the operating system reclaims it.

Fixes temporalio#2046
@aleakos
aleakos requested a review from a team as a code owner October 29, 2026 19:29
@aleakos aleakos closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Process hangs forever at interpreter exit: Runtime drop waits for a Tokio thread that CPython parks in take_gil

1 participant