python runtime_error ai_generated true

FAILED tests/test_state.py::test_second_run - AssertionError: assert 2 == 1 # passes when run alone, fails when run after tests/test_state.py::test_first_run

ID: python/pytest-flaky-test-ordering

Also available as: JSON · Markdown · 中文
80%Fix Rate
86%Confidence
0Evidence
2024-07-30First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.x active — — —
8.x active — — —

Root Cause

Test isolation is broken: a module-level mutable (cache, global, class attribute, or DB row) is mutated by an earlier test and not reset. Pytest's default collection order exposes the leak.

generic

中文

测试隔离被破坏：模块级可变对象（缓存、全局变量、类属性或数据库行）被较早的测试修改且未重置。Pytest 的默认收集顺序暴露了这种泄漏。

Workarounds

  1. 95% success 
    @pytest.fixture(autouse=True)
def reset_cache():
    from myapp import cache
    cache.clear()
    yield
    cache.clear()
  2. 85% success 
    pip install pytest-forked
pytest --forked tests/test_state.py
  3. 90% success 
    pytest --stepwise tests/test_state.py
# or:
pytest tests/test_state.py::test_first_run tests/test_state.py::test_second_run -v

Dead Ends

Common approaches that don't work:

  1. 80% fail

    Shuffling changes which test fails, not whether tests are isolated; flakiness moves around instead of disappearing.

  2. 85% fail

    Reruns the same broken state; if the first run mutated global state, retries still see the mutated value.