Repository navigation
Skip directory entries that raise PermissionError in scandir - #15243
Danish-kh-IT wants to merge 3 commits into
Conversation
ege-arhan
left a comment
There was a problem hiding this comment.
Verified against the branch (c76c5bfa6). The new test passes, but the reported scenario still crashes — the PermissionError in #9089 does not come from entry.is_file().
scandir() only guards the per-entry probe (src/_pytest/pathlib.py:1004-1011); os.scandir(path) at src/_pytest/pathlib.py:996-1000 is still wrapped in except FileNotFoundError only, so recursing into a directory the user cannot read raises before _IGNORED_ERRORS is ever consulted:
$ mkdir -p p/locked p/ok && echo "def test_a(): pass" > p/ok/test_ok.py && chmod 000 p/locked
$ pytest p
E PermissionError: [Errno 13] Permission denied: '.../p/locked'
ERROR locked - PermissionError: ...
Same result for the symlink case from the issue (ln -s /root p/sym_to_root), and both tracebacks bottom out at pathlib.py:998, i.e. the os.scandir(path) call itself, not at entry.is_file(). visit() (pathlib.py:1026) reaches it as soon as entry.is_dir() is true, so norecursedirs/--ignore still cannot help — which is exactly what the report describes.
The unit test hides this because it patches os.scandir wholesale, so the failing call site is mocked out:
with unittest.mock.patch("os.scandir") as mock_scandir:
mock_scandir.return_value.__enter__.return_value = [denied_entry, ok_entry]
assert scandir("/fake/path") == [ok_entry]A pytester/tmpdir test that really chmods a subdirectory would have caught it.
Suggested handling for the unreadable directory itself:
try:
scandir_iter = os.scandir(path)
except OSError as err:
if _ignore_error(err):
return []
raiseFileNotFoundError remains covered since ENOENT is in _IGNORED_ERRORS, and a directory that cannot be listed then behaves like a missing one instead of aborting the session. Worth keeping the per-entry guard as well — a file that vanishes between readdir and stat is a different, still-real case.
Minor: EACCES is appended in the middle of the tuple ((ENOENT, ENOTDIR, EACCES, EBADF, ELOOP)), which breaks the alphabetical order the original comment/first three entries follow; also the changelog says "consistent with other transient filesystem errors", but EACCES is a permission fact rather than a race, so the wording could name the user-facing outcome (unreadable directories are skipped) instead.
Test collection crashed with PermissionError when a directory contained an entry whose is_file() raised it (for example a symlinked folder owned by root), because EACCES was not among the errors ignored while scanning a directory. Add EACCES to _IGNORED_ERRORS so such entries are skipped, consistent with ENOENT and the other transient errors; other OSErrors still propagate. Fixes pytest-dev#9089. Co-authored-by: Muse <noreply@muse.dev>
- scandir(): guard the os.scandir(path) call itself with _ignore_error, so recursing into an unreadable directory returns [] instead of crashing collection with PermissionError (the reported pytest-dev#9089 scenario); the per-entry guard is kept - _IGNORED_ERRORS reordered alphabetically - changelog reworded to name the outcome (unreadable directories are skipped) - tests: mock-based tests for the directory-level call (PermissionError returns [], a non-ignorable OSError still propagates) plus a pytester test that really chmods a subdirectory Co-authored-by: Muse <noreply@muse.dev>
c76c5bf to
0c52f40
Compare
The new chmod-based regression test exposed a second wall behind scandir(): while collecting an unreadable directory, _loadconftestmodules probes for <dir>/conftest.py with Path.is_file(), which stats inside the directory and raises PermissionError, interrupting the whole session. Treat a PermissionError from the probe as 'no conftest' - a directory that cannot be searched cannot provide one. Co-authored-by: Muse <noreply@muse.dev>
|
Update — the real chmod test you asked for earned its keep. With the scandir() guard in place it still failed on Linux CI, and the traceback showed a second wall behind the first: while collecting the unreadable directory, _loadconftestmodules probes for /conftest.py with Path.is_file(), which stats inside the directory and raises PermissionError, interrupting the whole session ("Interrupted: 1 error during collection"). That probe was unreachable before, because scandir() crashed first — including in your repro.Fixed in 839aaf5: a PermissionError from the conftest probe is now treated as "no conftest" — a directory that cannot be searched cannot provide one. The same change also covers the earlier round: scandir() guards the os.scandir(path) call itself, _IGNORED_ERRORS is alphabetical again, the changelog names the outcome (unreadable directories are skipped), and three tests were added (two mock-based ones for the directory-level call, plus the pytester chmod test). Local verification: testing/test_conftest.py + testing/test_config.py 365 passed, 5 skipped, 1 xfailed; testing/test_pathlib.py + test_collection.py + test_main.py 246 passed, 12 skipped, 1 xfailed; pre-commit all passed. The chmod test still skips on my Windows machine, so the Linux CI jobs are the real check for it. |
Fixes #9089.
scandir()insrc/_pytest/pathlib.pyprobes each directory entry withentry.is_file()and re-raises anyOSErrorwhose errno is not in_IGNORED_ERRORS.EACCESwas missing from that set, so a single entry that cannot be inspected because of insufficient permissions (for example a symlinked folder owned by root) crashed the whole collection withPermissionError, and neithernorecursedirsnor--ignorecould prevent it.This adds
EACCESto_IGNORED_ERRORS, so such entries are skipped, consistent with how a vanishing file (ENOENT) and the other transient errors are already handled. OtherOSErrors still propagate — the existingtest_scandir_handles_os_errorguarantee is unchanged.Tests: added
test_scandir_skips_permission_denied_entrynext totest_scandir_handles_os_error, using the same mock-based approach. Verified locally:testing/test_pathlib.py113 passed, 1 skipped;testing/test_collection.py+testing/test_main.py131 passed, 10 skipped, 1 xfailed.As noted in my comment on #9089, if maintainers would prefer a warning instead of silently skipping, I'm happy to adjust.