CUDA_ERROR_ILLEGAL_ADDRESS pytorch runtime_error ai_generated true

RuntimeError: CUDA error: device-side assert triggered. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

ID: pytorch/cuda-error-devices-synchronize-abort

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2023-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
PyTorch 2.0.0 active — — —
CUDA 11.7 active — — —
CUDA 11.8 active — — —
CUDA 12.1 active — — —
Ubuntu 22.04 active — — —

Root Cause

A CUDA kernel encountered an assertion failure on the device (e.g., invalid index in embedding, negative dimension, or NaN in loss), which often causes subsequent operations to fail silently before this error surfaces.

generic

中文

CUDA 内核在设备上遇到了断言失败（例如，嵌入层中的无效索引、负维度或损失中的 NaN），这通常会导致后续操作静默失败，然后此错误才会显现。

Official Documentation

https://pytorch.org/docs/stable/notes/cuda.html#device-side-assertions

Workarounds

  1. 90% success Enable device-side assertions by setting environment variable TORCH_USE_CUDA_DSA=1 before running the script, then re-run to get a detailed stack trace pointing to the failing operation (e.g., embedding lookup with out-of-range index). Example: TORCH_USE_CUDA_DSA=1 python train.py 
    Enable device-side assertions by setting environment variable TORCH_USE_CUDA_DSA=1 before running the script, then re-run to get a detailed stack trace pointing to the failing operation (e.g., embedding lookup with out-of-range index). Example: TORCH_USE_CUDA_DSA=1 python train.py
  2. 80% success Add gradient clipping and NaN checks in the training loop: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0); if torch.isnan(loss): print('NaN loss'); return 
    Add gradient clipping and NaN checks in the training loop: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0); if torch.isnan(loss): print('NaN loss'); return
  3. 85% success Wrap the problematic operation in a try-except block and use torch.cuda.synchronize() to catch the exact error location. For example: try: output = model(input); torch.cuda.synchronize(); except RuntimeError as e: print(f'Error at iteration {i}: {e}') 
    Wrap the problematic operation in a try-except block and use torch.cuda.synchronize() to catch the exact error location. For example: try: output = model(input); torch.cuda.synchronize(); except RuntimeError as e: print(f'Error at iteration {i}: {e}')

中文步骤

  1. Enable device-side assertions by setting environment variable TORCH_USE_CUDA_DSA=1 before running the script, then re-run to get a detailed stack trace pointing to the failing operation (e.g., embedding lookup with out-of-range index). Example: TORCH_USE_CUDA_DSA=1 python train.py
  2. Add gradient clipping and NaN checks in the training loop: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0); if torch.isnan(loss): print('NaN loss'); return
  3. Wrap the problematic operation in a try-except block and use torch.cuda.synchronize() to catch the exact error location. For example: try: output = model(input); torch.cuda.synchronize(); except RuntimeError as e: print(f'Error at iteration {i}: {e}')

Dead Ends

Common approaches that don't work:

  1. Set torch.backends.cudnn.deterministic = True 95% fail

    Deterministic mode does not fix invalid tensor values or index errors; it only ensures reproducibility of operations.

  2. Increase batch size to trigger error less often 90% fail

    Larger batch sizes may hide the issue temporarily but do not address the root cause (e.g., out-of-range indices in embedding). The error will reappear on different data.

  3. Set CUDA_LAUNCH_BLOCKING=1 environment variable 85% fail

    While this helps identify the exact operation causing the error, it does not fix the underlying problem such as index errors or NaN values.