python system_error ai_generated partial

pytest_benchmark.fixture.BenchmarkFixture: Benchmark 'test_sort' failed: 5.21ms > 5.00ms (threshold)

ID: python/pytest-benchmark-flaky-threshold

Also available as: JSON · Markdown · 中文
80%Fix Rate
84%Confidence
0Evidence
2025-10-11First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
4.0.x active — — —

Root Cause

pytest-benchmark's --benchmark-max-time or comparison threshold is too tight for noisy CI hardware; a few microseconds of variance trips the gate.

generic

中文

pytest-benchmark 的 --benchmark-max-time 或比较阈值对嘈杂的 CI 硬件过紧；几微秒的波动就会触发失败。

Workarounds

  1. 90% success 
    pytest --benchmark-only \
  --benchmark-compare=baseline \
  --benchmark-compare-fail=median:15%
  2. 87% success 
    pytest --benchmark-only --benchmark-disable-gc \
  --benchmark-save=ci-$CI_RUN_ID
# Compare offline; only fail on >25% median regression

Dead Ends

Common approaches that don't work:

  1. 80% fail

    Disables the benchmark entirely; performance regressions no longer surface.

  2. 70% fail

    Test ordering isn't the cause; hardware variance and CPU steal still produce flaky results.