CUDA_ERROR_ILLEGAL_ADDRESS pytorch runtime_error ai_generated true

RuntimeError: CUDA 错误：触发了设备端断言。请使用 TORCH_USE_CUDA_DSA 编译以启用设备端断言。

RuntimeError: CUDA error: device-side assert triggered. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

ID: pytorch/cuda-error-devices-synchronize-abort

其他格式: JSON · Markdown 中文 · English
85%修复率
88%置信度
1证据数
2023-03-15首次发现

版本兼容性

版本状态引入弃用备注
PyTorch 2.0.0 active — — —
CUDA 11.7 active — — —
CUDA 11.8 active — — —
CUDA 12.1 active — — —
Ubuntu 22.04 active — — —

根因分析

CUDA 内核在设备上遇到了断言失败（例如，嵌入层中的无效索引、负维度或损失中的 NaN），这通常会导致后续操作静默失败，然后此错误才会显现。

English

A CUDA kernel encountered an assertion failure on the device (e.g., invalid index in embedding, negative dimension, or NaN in loss), which often causes subsequent operations to fail silently before this error surfaces.

generic

官方文档

https://pytorch.org/docs/stable/notes/cuda.html#device-side-assertions

解决方案

  1. Enable device-side assertions by setting environment variable TORCH_USE_CUDA_DSA=1 before running the script, then re-run to get a detailed stack trace pointing to the failing operation (e.g., embedding lookup with out-of-range index). Example: TORCH_USE_CUDA_DSA=1 python train.py
  2. Add gradient clipping and NaN checks in the training loop: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0); if torch.isnan(loss): print('NaN loss'); return
  3. Wrap the problematic operation in a try-except block and use torch.cuda.synchronize() to catch the exact error location. For example: try: output = model(input); torch.cuda.synchronize(); except RuntimeError as e: print(f'Error at iteration {i}: {e}')

无效尝试

常见但无效的做法:

  1. Set torch.backends.cudnn.deterministic = True 95% 失败

    Deterministic mode does not fix invalid tensor values or index errors; it only ensures reproducibility of operations.

  2. Increase batch size to trigger error less often 90% 失败

    Larger batch sizes may hide the issue temporarily but do not address the root cause (e.g., out-of-range indices in embedding). The error will reappear on different data.

  3. Set CUDA_LAUNCH_BLOCKING=1 environment variable 85% 失败

    While this helps identify the exact operation causing the error, it does not fix the underlying problem such as index errors or NaN values.