python runtime_error ai_generated true

FAILED tests/test_x.py::TestX::test_y - subtest failed # but exit code is 0 and CI passes

ID: python/pytest-subtests-failure-not-counted

Also available as: JSON · Markdown · 中文
80%Fix Rate
82%Confidence
0Evidence
2025-05-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.x active — — —
8.x active — — —

Root Cause

unittest subTest failures are reported but pytest's exit code may remain 0 if the test method itself doesn't propagate the failure, especially in older pytest or with plugins that swallow subtest outcomes.

generic

中文

unittest subTest 失败会被报告，但如果测试方法本身未传播失败，pytest 的退出码可能仍为 0，尤其在较旧 pytest 或吞掉子测试结果的插件中。

Workarounds

  1. 95% success 
    pip install -U pytest
# pytest >= 7.2 correctly reports subtest failures in exit code
  2. 85% success 
    def test_y(self):
    errors = []
    for case in cases:
        with self.subTest(case=case):
            try:
                check(case)
            except AssertionError as e:
                errors.append(e)
    if errors:
        raise AssertionError(errors)

Dead Ends

Common approaches that don't work:

  1. 85% fail

    Stops on first failure but doesn't change exit code semantics for subtest failures.

  2. 95% fail

    Changes traceback style, not exit code behavior.

  3. 70% fail

    Loses the per-subtest granularity that subTest was meant to provide.