Repository navigation
os.dup2(fd, fd, inheritable=False) behaves inconsistently #77043
Description
| BPO | 32862 |
|---|---|
| Nosy | @benjaminp, @eryksun, @izbyshev |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-02-17.03:24:02.305>
labels = ['type-bug', '3.8', '3.9', '3.10', 'extension-modules', 'library']
title = 'os.dup2(fd, fd, inheritable=False) behaves inconsistently'
updated_at = <Date 2021-03-20.14:32:55.054>
user = 'https://github.com/izbyshev'bugs.python.org fields:
activity = <Date 2021-03-20.14:32:55.054>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2018-02-17.03:24:02.305>
creator = 'izbyshev'
dependencies = []
files = []
hgrepos = []
issue_num = 32862
keywords = ['patch']
message_count = 3.0
messages = ['312266', '312295', '312298']
nosy_count = 3.0
nosy_names = ['benjamin.peterson', 'eryksun', 'izbyshev']
pr_nums = ['5713']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32862'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
izbyshev commentedon Feb 17, 2018
os.dup2(fd, fd, inheritable=False) may fail or change fd inheritability in following ways:
POSIX without F_DUP2FD_CLOEXEC
1.1) dup3() is available (a common case for Linux): OSError (EINVAL, dup3() doesn't allow equal descriptors)
1.2) dup3() is not available: fd made non-inheritable
POSIX with F_DUP2FD_CLOEXEC (FreeBSD): inheritability is not changed
Windows: fd made non-inheritable
In contrast, os.dup2(fd, fd, inheritable=True) never changes fd inheritability (same as before PEP-446 landed). I suggest to make os.dup2(fd, fd, inheritable=False) behave the same.
eryksun commentedon Feb 18, 2018
In Windows the CRT file descriptor is actually still inheritable. This only makes the underlying OS handle non-inheritable. I don't think there's a way to make an existing FD non-inheritable using public CRT functions. See bpo-32865.
izbyshev commentedon Feb 18, 2018
@eryksun: Thank you for the note! I've commented on bpo-32865. This adds even more inconsistency to this corner case.
pythongh-77043: Make os.dup2(fd, fd, False) always remove inheritability
izbyshev commentedon Feb 22, 2023
I've revisited this issue, and here is corrected and expanded list of how
os.dup2(fd, fd, inheritable=False)behaves across platforms:On Windows, Linux before 2.6.27, and macOS
dup2()is attempted(validating the fd), and on success the fd is then separately made
non-inheritable.
On platforms where F_DUP2FD_CLOEXEC fcntl command is used:
dup3()on Linux). The source code confirms this.On Linux 2.6.27+
dup3()is used, which always fails with EINVAL (evenif the fd is invalid).
So overall we have the full range of possible behavior across platforms: inheritability is changed, not changed, or OSError is raised.
One way to fix this inconsistency would be to make
os.dup2(fd, fd, False)behave asos.dup2(fd, fd): just validate the fd and don't change its inheritability. This is what I originally attempted in PR #5713. But now, given that on all platforms except Solarisos.dup2(fd, fd, False)either fails or makes the fd non-inheritable, I think this change could be viewed as a regression: we probably don't want to introduce new cases where inheritable fds can unexpectedly appear, and the caller explicitly asks forinheritable=Falsehere.Likewise, changing
os.dup2(fd, fd, False)to always fail (likedup3()) would add a new error case on several platforms and can be viewed as a regression.If we were designing
os.dup2()from scratch, the right thing to do would probably be to makeinheritablealways affect the inheritability (and make itFalseby default). But now we can't change howos.dup2()behaves by default (even PEP-446 didn't touch it).I'm not sure whether it's worth to separate the default (
inheritableis not passed explicitly) and the explicitinheritable=Truecase and make the latter always make the returned fd inheritable. But I think that something needs to be done to at least fix theinheritable=Falsemess. So I've implemented PR #102148 to always make the fd non-inheritable in this case.Cc: @vstinner
pythongh-77043: Make os.dup2(fd, fd, False) always remove inheritability
vstinner commentedon Apr 5, 2023
As an user, I expect that
os.dup2(fd, fd, inheritable=False)raises an exception if fd is invalid, or makes the file descriptor non-inheritable.In Python in general, we attempt to have a consistent behavior on all platforms, adding platform-specific code for that if needed. For example, there is a lot of (platform-specific) code to have a portable behavior with Not-a-Number (NaN) in math functions.
For the special case
fd2 == fd, we should avoiddup3(). Maybe just have the same portable code on all platforms for this special case.izbyshev commentedon Apr 5, 2023
PR #102148 is in line with this.
I'm not aware of a portable way of validating an fd that is better than simply delegating to
dup2()(like in PR #102148). The obvious way to validate an fd (fcntl) doesn't exist on Windows. Moreover, fd validation is not as easy as it might sound (see is_valid_fd), and an fd might be valid for some operations, but not for others. I don't think CPython should invent its own semantics of the fd validity foros.dup2(), risking it be different from what the actualdup2()does for the sake of providing a consistent behavior. Theosmodule is low-level and has always been tied to the underlying libc, and users came to expect that. This situation is different from math functions.