ECC cuda hardware_error ai_generated partial

CUDA error: an illegal memory access was encountered due to ECC uncorrectable error

ID: cuda/ecc-uncorrectable

Also available as: JSON · Markdown
60%Fix Rate
80%Confidence
50Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —

Root Cause

GPU memory has an uncorrectable ECC (Error Correcting Code) error indicating failing DRAM cells. This is a hardware defect. The GPU retires bad memory pages but if too many fail, the GPU becomes unreliable. Common in data centers after extended use.

generic

Workarounds

  1. 50% success Reset the GPU and check if errors persist: nvidia-smi -r 
    nvidia-smi -r -i <gpu_id> to reset. Then check: nvidia-smi -q -d ECC for pending/retired pages.
  2. 95% success Replace the GPU if ECC error count is high and growing 
    Check nvidia-smi -q -d ECC. If retired_pages > 10 or volatile_uncorrectable > 0 persistently, GPU needs replacement.
  3. 85% success Move workload to a different GPU on the same machine 
    CUDA_VISIBLE_DEVICES=<other_gpu_id> python train.py

Dead Ends

Common approaches that don't work:

  1. Disable ECC to hide the errors: nvidia-smi -e 0 85% fail

    Disabling ECC does not fix the bad memory; it just stops detecting errors, leading to silent data corruption in computations

  2. Reduce GPU clock speed to work around memory errors 80% fail

    ECC errors are from physical memory defects, not timing issues; underclocking does not fix bad DRAM cells

Error Chain

Leads to:
Preceded by:
Frequently confused with: