Repository navigation
Make os.scandir retry on system calls failing with EINTR #130209
Description
Bug report
Bug description:
In a long-running script, I hit:
[…]
File "/Users/gsnedders/ct.py", line 104, in worker
with os.scandir(path) as it:
~~~~~~~~~~^^^^^^
InterruptedError: [Errno 4] Interrupted system call: '/Users/gsnedders/Library/Containers/com.apple.ScreenTimeWidgetApplication.ScreenTimeWidgetExtension/Data/Library/Application Scripts'
No repro case is provided, because that would require making syscalls fail with EINTR reliably somehow.
PEP 475 (Retry system calls failing with EINTR) implies to me that os.scandir should be retrying, rather than just failing here.
It does appear that os.scandir doesn't always retry, looking at the source:
Line 16489 in ae30646
The iterator itself also seems to not retry:
Line 16290 in ae30646
CPython versions tested on:
3.13
Operating systems tested on:
macOS
Activity
picnixz commentedon Feb 16, 2025
I'm not sure but could
os.scandiralso have the same weirdness asos.close?Anyway, I think we should treat it as a feature request.
[-]scandir does not retry on system calls failing with EINTR[/-][+]Make `os.scandir` retry on system calls failing with EINTR[/+]larryhastings commentedon Oct 20, 2026
I'd like to propose broadening this issue. I hit the same class of failure in a real program, and after reading the kernel side I think the problem affects a bunch of functions.
Disclosure: this whole message was written for me by Claude Fable 5.1, after having read this existing issue, and examining both Python source and Linux kernel source. I read it over and confirmed it's basically accurate, but I'm not a signal expert by any stretch.
What happened
I have a custom-written podcast downloader. The main process installs a
SIGCHLDhandler withsignal.signal()(which uses a a loopback pipe to wake up aselectorsloop), then runs download/transcode jobs assubprocesschildren, and maintains / updates a large tree of files--the cached RSS feeds, the downloaded MP3 files--using conventional filesystem APIS, e.g.os.stat(),os.rename(),os.utime(),os.unlink()and so on. During development, I mounted the downloader's data via NFS (version 4).Every time a child exited, Linux sent my downloader's main process a
SIGCHLDsignal, and eventually one of those path operations in the main thread died withOSError: [Errno 4] Interrupted system call. (I didn't keep the traceback, so I can't say which call it was that time; the analysis below suggests many candidates.)Why this happened
signal.signal()installs handlers throughPyOS_setsig(), which usessigaction()withsa_flags = SA_ONSTACKonly (Python/pylifecycle.c). NoSA_RESTART. So the moment a program installs any Python-level handler, every syscall that the kernel would otherwise restart transparently starts failing withEINTRinstead.open,read,write,fstat,fsync,ftruncate,fchmod,fchown, thewait*family,select, sockets, ...). These detect theEINTRerror and automatically retry. But this was does not done for path-basedosfunctions:stat,lstat,rename,replace,utime,unlink,mkdir,rmdir,listdir,scandir,access,chmod,chown,link,symlink,readlink,truncate,statvfs,chdir. I checkedModules/posixmodule.con main: none of these have thewhile (res != 0 && errno == EINTR && !(async_err = PyErr_CheckSignals()))loop thatfchmod/fchown/mkfifohave. Soos.fchmod()retries butos.chmod()doesn't.The PEP's implicit assumption was that these calls don't block long enough to be interrupted. But for network and userspace filesystems that assumption doesn't hold.
Behavior on Linux 7.2.11
NFSv4 (
fs/nfs/nfs4proc.c). When the server answersNFS4ERR_DELAY,NFS4ERR_GRACE,NFS4ERR_FILE_OPEN, or a few layout/recall errors, the client backs off withnfs4_delay(), sleeping between 0.1s and 15s per round. Operations that set.interruptible = truedo that sleep inTASK_INTERRUPTIBLE, andnfs4_delay_interruptible()returns-ERESTARTSYSwhen any signal is pending. Those operations includelookup,getattr,access,readdir,remove,rmdir,mkdir,link,symlink,readlink,open,statfs. Since every path walk goes through lookup, permission (access) and attribute revalidation (getattr), any path-based syscall can come back-ERESTARTSYSwhen a signal lands mid-backoff:stat,rename,utime,unlink,listdir, all of them. (The RENAME and SETATTR operations themselves are not interruptible; they're hit through the path walk.) Everything else in the NFS client and sunrpc (rpc_wait_bit_killable,nfs_wait_bit_killable, the v3 jukebox retry) isTASK_KILLABLE, so only fatal signals break it.For NFS the error is always
-ERESTARTSYS, so a handler installed withSA_RESTARTmakes the kernel restart the call and nothing ever reaches userspace.signal.siginterrupt(sig, False)is therefore a working workaround for NFS, and it's what I did.FUSE (
fs/fuse/dev.c,request_wait_answer()). This is why I think the workaround isn't the answer. The kernel waits for the daemon withwait_event_interruptible(), so any signal interrupts. If the request hasn't been read by the daemon yet, the kernel pulls it off the queue and sets the error to a hard-EINTR(fuse_remove_pending_req()). If it has been read, the kernel sendsFUSE_INTERRUPTand the daemon decides whether to fail the operation withEINTR. A hard-EINTRis not restartable:SA_RESTARTdoes nothing for it. So on sshfs, rclone, or any FUSE filesystem, a Python program with a signal handler can getEINTRfromos.stat()and friends, and the only fix is to retry in userspace, which is exactly what PEP 475 was created to do.I don't know how other POSIX OSes behave here, but I'd be surprised if adding an automatic retry after
EINTRwas a bad move on those other OSes.Proposal
Extend PEP 475 retry-on-
EINTRto the remainingosfunctions that take a path or an fd and currently don't retry. The idiom already exists inposixmodule.c; this is mostly mechanical. The PEP's own rationale applies unchanged: the alternative is for every caller ofos.stat()to wrap it in a retry loop, which nobody does, so programs fail intermittently on network filesystems whenever they also handle a signal.I'm happy to work on a PR if there's agreement on the direction. (I'm at the core dev sprint right now! Lookin' for something to do!)
vstinner commentedon Oct 20, 2026
SA_RESTARTflag is not used on signal handlers, since Python has to callPyErr_CheckSignals()which can set an exception.Well, it's a bug :-) The should implement PEP 475. For example, use a loop such as:
Same here: it should implement PEP 475, so retry on EINTR and call PyErr_CheckSignals() in the meanwhile.