Repository navigation
Conversation
At interpreter finalization a Tokio thread that completes a Python future asks for the GIL, and CPython parks it for good. Dropping the Core runtime then waits for that thread forever, so the process never exits after asyncio.run() returns. Python::try_attach still attaches the finalizing thread itself, so its result does not show this state. Check sys.is_finalizing() while attached and leak the runtime while it is true. The process is exiting, so the operating system reclaims it. Fixes temporalio#2046
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
CoreRuntimeHandle::drop(temporalio/bridge/src/runtime.rs) now checkssys.is_finalizing()while attached. If the interpreter is finalizing, or the check itself fails, it leaks the Core runtime withstd::mem::forgetinstead of dropping it. Otherwise the drop is unchanged: it releases the GIL and shuts the runtime down, as in #1998.Why?
Fixes the exit hang in #2046: a process sometimes never exits after
asyncio.run()returns.At interpreter finalization the final GC frees the last
RuntimeRef. Dropping the Core runtime then waits for every Tokio blocking-pool thread. One of those threads can be inpyo3_async_runtimes::generic::set_result, completing a Python future throughcall_soon_threadsafe, and asking for the GIL. During finalization CPython parks that thread forever inPyThread_hang_thread, whether the GIL is free or not. The drop never returns.#1998 releases the GIL around the drop, but that does not help here.
Python::try_attachstill returnsSomeon the finalizing thread itself, so the drop takes the normal path and blocks. We first tried "leak whentry_attachreturnsNone". It still hung 4 times in 76 runs, and gdb showed the drop running inside thetry_attachclosure. So the finalization state has to be checked explicitly.Py_IsFinalizing()is in the limited API only from 3.13, and the bridge is builtabi3-py310, so the check callssys.is_finalizing(). It exists in every supported version and needs only the stable ABI. Leaking is safe because the process is exiting: the operating system reclaims the threads and memory.Checklist
Closes [Bug] Process hangs forever at interpreter exit: Runtime drop waits for a Tokio thread that CPython parks in take_gil #2046
How was this tested:
All runs on Linux aarch64 (Docker,
ghcr.io/astral-sh/uv:python3.13-bookworm), Python 3.13.11. Wheels built withmaturin build --release, usingCARGO_PROFILE_RELEASE_LTO=offand 16 codegen units because of build-host memory. The reproduction isrepro.pyfrom the issue: start a time-skipping environment from an existing binary, shut it down, exit. A run counts as hung only if it printed its last line and was still alive 60 s later.try_attachreturnsNone)At the first attempt's rate (about 5 %), 200 clean runs by chance has a probability of about 3e-5.
Workerwith a failing activity and workflow before it exits: 60 runs with this change, 0 hangs, 0 failures.tests/test_runtime.py: 13 passed with this change and 13 passed without it. The same one test fails on both wheels:test_opentelemetry_histogram_bucket_overrides, becauseopentelemetry-protowas not installed in the test environment.cargo fmt --check,cargo clippy, the full test suite, other platforms (x86_64, macOS, Windows), Python 3.10 to 3.12.Any docs updates needed? No.
A deterministic regression test needs a way to force a Tokio thread to complete a Python future during finalization. The reproduction is probabilistic (about 1 hang in 15 to 60 runs), so it is not suitable as a CI test as it stands.