Bug report

Bug description:

If a ProcessPoolExecutor worker is killed (SIGKILL, the OOM killer) while it is writing its result to the result pipe, the future never completes, BrokenProcessPool is never raised, and the interpreter cannot exit. The docs say BrokenProcessPool is raised when a worker "has terminated in a non-clean fashion (for example, if it was killed from the outside)", and a worker killed before or after the send does raise it. The window is large for large results: memory peaks while the result is pickled and sent, which is when the OOM killer is likely to strike.

"""A ProcessPoolExecutor worker that is SIGKILLed while its result is in the pipe
leaves the future pending forever, and the interpreter cannot exit.

    python repro.py          # stops at the first delay whose future never completes
    python repro.py close    # same, but the parent first closes its copy of the
                             # result pipe's write end

Each attempt uses a fresh one-worker spawn executor. The worker builds a 1.2 GB
bytes result, writes its pid to a file and returns the result; the parent SIGKILLs
the worker DELAY seconds after the pid appears.
"""
import faulthandler
import os
import signal
import sys
import tempfile
import time
from concurrent.futures import ProcessPoolExecutor
from multiprocessing import get_context

NBYTES = 1_200_000_000


def big_result(marker):
    result = b"x" * NBYTES
    with open(marker, "w") as f:
        f.write(str(os.getpid()))
    return result


def attempt(delay, close):
    marker = os.path.join(tempfile.mkdtemp(), "pid")
    executor = ProcessPoolExecutor(max_workers=1, mp_context=get_context("spawn"))
    future = executor.submit(big_result, marker)
    if close:
        executor._result_queue._writer.close()
    while not (os.path.exists(marker) and os.path.getsize(marker)):
        time.sleep(0.005)
    with open(marker) as f:
        pid = int(f.read())
    time.sleep(delay)
    os.kill(pid, signal.SIGKILL)
    killed = time.monotonic()
    try:
        outcome = f"returned {len(future.result(timeout=5))} bytes"
    except TimeoutError:
        outcome = "PENDING 5 s after the worker was killed"
    except Exception as e:
        outcome = f"raised {type(e).__name__} after {time.monotonic() - killed:.2f} s"
        if e.__cause__ is not None:
            outcome += f" ({str(e.__cause__).strip(chr(10) + chr(39)).splitlines()[-1]})"
    executor.shutdown(wait=False, cancel_futures=True)
    return outcome


if __name__ == "__main__":
    close = sys.argv[1:] == ["close"]
    for step in range(21):
        delay = step * 0.05
        outcome = attempt(delay, close)
        print(f"{sys.version.split()[0]}{' close' if close else ''} delay={delay:.2f}: {outcome}", flush=True)
        if outcome.startswith("PENDING"):
            break
    faulthandler.dump_traceback_later(10, exit=True)  # bounds the exit: dumps every thread and exits

python repro.py on 3.14.7 (paths shortened):

3.14.7 delay=0.00: raised BrokenProcessPool after 0.02 s
3.14.7 delay=0.05: raised BrokenProcessPool after 0.03 s
3.14.7 delay=0.10: PENDING 5 s after the worker was killed
Timeout (0:00:10)!
Thread 0x0000000171c4b000 [QueueFeederThread] (most recent call first):
  File ".../lib/python3.14/threading.py", line 369 in wait
  File ".../lib/python3.14/multiprocessing/queues.py", line 249 in _feed
  File ".../lib/python3.14/threading.py", line 1024 in run
  File ".../lib/python3.14/threading.py", line 1082 in _bootstrap_inner
  File ".../lib/python3.14/threading.py", line 1044 in _bootstrap

Thread 0x0000000170c3f000 [Thread-3] (most recent call first):
  File ".../lib/python3.14/multiprocessing/connection.py", line 416 in _recv
  File ".../lib/python3.14/multiprocessing/connection.py", line 458 in _recv_bytes
  File ".../lib/python3.14/multiprocessing/connection.py", line 260 in recv
  File ".../lib/python3.14/concurrent/futures/process.py", line 513 in wait_result_broken_or_wakeup
  File ".../lib/python3.14/concurrent/futures/process.py", line 366 in run
  File ".../lib/python3.14/threading.py", line 1082 in _bootstrap_inner
  File ".../lib/python3.14/threading.py", line 1044 in _bootstrap

Thread 0x00000001f219e180 (most recent call first):
  File ".../lib/python3.14/threading.py", line 1133 in join
  File ".../lib/python3.14/concurrent/futures/process.py", line 106 in _python_exit
  File ".../lib/python3.14/threading.py", line 1548 in <lambda>
  File ".../lib/python3.14/threading.py", line 1577 in _shutdown

3.12.13 and 3.13.12 behave the same way, with the first PENDING at 0.50 s and 0.20 s; the window moves with result size and machine load, hence the sweep.

Cause. _ExecutorManagerThread.wait_result_broken_or_wakeup waits on the result pipe and the worker sentinels together (process.py:506), but as soon as the first bytes of a result are readable it calls result_reader.recv() (process.py:513), which reads the rest of the message with blocking reads that watch neither the sentinels nor a timeout (connection.py:410-422). EOF would end that read with OSError("got end of file during message"), and the pool would be marked broken, but EOF never arrives: the result queue is a SimpleQueue created in the parent (process.py:842, queues.py:360), so the parent holds a copy of the pipe's write end for the executor's lifetime. At exit, _python_exit() then joins the stuck manager thread with no timeout (process.py:98-106).

python repro.py close confirms this. It closes the parent's copy (executor._result_queue._writer.close()) right after submitting. Across all 21 delays (0 to 1 s) on each of 3.12.13, 3.13.12 and 3.14.7, every attempt raised BrokenProcessPool within 0.1 s (with cause OSError: got end of file during message when the kill landed mid-message) or returned the result. Every run exited normally:

3.14.7 close delay=0.45: raised BrokenProcessPool after 0.05 s (EOFError)
3.14.7 close delay=0.50: raised BrokenProcessPool after 0.09 s (OSError: got end of file during message)
3.14.7 close delay=0.55: raised BrokenProcessPool after 0.07 s (OSError: got end of file during message)

Neither shutdown(wait=False, cancel_futures=True) (the repro calls it) nor 3.14's kill_workers() helps. With executor.kill_workers() called on a pending attempt, the future is still pending 5 s later, and exit hangs in _python_exit as above. The worker is already dead; the write end that keeps the pipe open is the parent's.

History. This exact race was reported and fixed once. In bpo-9205 (gh-53451), Charles-François Natali described it: _poll() returns as soon as the length header has been sent, "The first _recv() will succeed, since the length is in the Pipe. The second one, however, will remain stuck on the read from the pipe/unix socket, because there's no more data. This can be reproduced easily by adding a short sleep right after the sending of the length of the buffer inside send_bytes(), and then sending a SIGKILL to a worker process." Antoine Pitrou agreed ("We would need to insert a select() before each raw read()") and did that. The fix shipped in dd69649 ("ProcessPoolExecutor now detects killed children and raises BrokenProcessPool in such a situation. Previously it would reliably freeze/deadlock."): from 3.3.0a1, Connection._recv(size, sentinels=...) selected on [handle] + sentinels before every read().

bdb1cf1 (bpo-12328, gh-56537, 2012, a fix for overlapped I/O on Windows) "Removes use of the SentinelReady exception and the sentinels argument to certain methods. concurrent.futures.process has been changed to use wait() instead of SentinelReady." That replaced the per-read check with one wait([reader] + sentinels) followed by a plain blocking recv(). The race came back, and the bpo-12328 thread does not discuss it. 3.3.0 already has that shape (wait([reader] + sentinels), then reader.recv()), and so do 3.12, 3.13, 3.14 and main.

Possible fixes.

• Read the result body in chunks under wait([reader] + sentinels), and treat a sentinel that fires mid-message as a broken pool. This restores the 3.3.0a1 behaviour.
• Bound _python_exit()'s join, so that a stuck manager thread cannot block interpreter exit.

Closing the parent's write end, as the repro does, works for one worker. In general it does not: the parent needs the write end to start replacement workers, and with several workers the live ones still hold write ends.

Related.

• ProcessPoolExecutor deadlock when a child process crashes while data is being sent in call queue #94777: the mirror image on the call queue; fixed in 2023 in terminate_broken(), which this case never reaches.
• ProcessPoolExecutor: pending futures never complete if executor is gc'd while a max_tasks_per_child= worker exits #152967 (fixed on main and 3.15): the same symptom, a pending future and an _python_exit hang, through a different path.
• ProcessPoolExecutor: after a worker dies, shutdown(wait=False) and submit() block until every other worker has exited #158413 and ProcessPoolExecutor.shutdown(wait=True) deadlocks when worker exit messages fill the result pipe #158570: other teardown hangs in the same manager thread.
• multiprocessing.Connection does not communicate pipe closure between parent and child #56697: EOF semantics of multiprocessing.Connection when another process holds a write end.

Found while debugging a hang in an RL training pipeline. The analysis and the reproducer were prepared with an AI assistant (Claude); every output above is from a real run.

CPython versions tested on:

3.12, 3.13, 3.14

Operating systems tested on:

macOS

Linked PRs

• Fix ProcessPoolExecutor hang when a worker is killed mid-result #159696
• gh-159689: Fix ProcessPoolExecutor hang when a worker is killed mid-result #159697