Repository navigation
Warn when pytest.mark.skip reason looks like a condition - #15252
LuanaPBueno wants to merge 1 commit into
Conversation
Closes pytest-dev#12582 Co-authored-by: Claude <noreply@anthropic.com>
Zac-HD
left a comment
There was a problem hiding this comment.
Thanks for taking this on Luana!
I'm excited to merge this soon; what do you think of my suggestion below?
| for node in ast.walk(tree): | ||
| # Attribute access or calls, for example ``sys.platform.startswith('win')``. | ||
| if isinstance(node, ast.Attribute | ast.Call): | ||
| return True | ||
| # Comparisons against values, for example ``os.name == 'nt'``; comparisons | ||
| # of plain names are ignored as they are also valid English, for example | ||
| # ``"nothing in particular"``. | ||
| if isinstance(node, ast.Compare) and not all( | ||
| isinstance(operand, ast.Name) for operand in [node.left, *node.comparators] | ||
| ): | ||
| return True | ||
| return False |
There was a problem hiding this comment.
I think I favor a simpler approach: rather than checking the contents of the expression, just reject those which contain only letters/space/dot:
| for node in ast.walk(tree): | |
| # Attribute access or calls, for example ``sys.platform.startswith('win')``. | |
| if isinstance(node, ast.Attribute | ast.Call): | |
| return True | |
| # Comparisons against values, for example ``os.name == 'nt'``; comparisons | |
| # of plain names are ignored as they are also valid English, for example | |
| # ``"nothing in particular"``. | |
| if isinstance(node, ast.Compare) and not all( | |
| isinstance(operand, ast.Name) for operand in [node.left, *node.comparators] | |
| ): | |
| return True | |
| return False | |
| return re.fullmatch(r"[A-z .]+", reason) is None |
it seems to me that this better captures the core "looks like valid english" issue, and is very simple.
ege-arhan
left a comment
There was a problem hiding this comment.
Tested the branch head locally. A module- or class-level pytest.mark.skip now warns once per collected item, not once per mark.
src/_pytest/skipping.py:213-221: the warning is emitted inside the per-item loop over item.iter_markers(name="skip"), and iter_markers yields inherited marks for every item, so a mark that decorates 3 tests produces 3 identical warnings.
Repro:
import pytest
pytestmark = pytest.mark.skip("sys.platform == 'win32'")
def test_a(): pass
def test_b(): pass
def test_c(): passpython -m pytest test_mod.py -W "default::pytest.PytestWarning" prints the same "looks like a condition" warning 3 times; same with a @pytest.mark.skip(...) class decorator (3 warnings for 3 methods). The added test only has a single test function, so it cannot catch this.
Suggestion: warn once per unique reason, e.g. keep a set() of already-warned reason strings on the session/config in evaluate_skip_marks, or move the check to collection time where the mark owner is known. Otherwise every module-level skip turns into noise proportional to the number of tests.
This adds a warning when the reason passed to
pytest.mark.skiplooks like a condition, likepytest.mark.skip("sys.platform == 'win32'"). That test is always skipped, and it's easy to mix up withskipifwithout noticing.While testing this, the rule suggested in the issue turned out to also catch normal reasons like "nothing in particular" or "Test is not isolated" (since
inandis notare valid comparisons in Python), and two existing tests broke because of it. So I made it a bit stricter: it only warns when there's an attribute access, a call, or a comparison with a value, likeos.name == 'nt'.I used Claude to help, but I reviewed all the changes, ran the tests locally and checked that the new tests fail without the fix. Happy to change anything!
Closes #12582
closes #XYZWto the PR description and/or commits.Co-authored-bycommit trailers.changelogdirectory.AUTHORSin alphabetical order (I did this in pytest-dev/pytest#15202 (already merged), so my name is already in the list).