# RuntimeError: CUDA error: device-side assert triggered. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

- **ID:** `pytorch/cuda-error-devices-synchronize-abort`
- **Domain:** pytorch
- **Category:** runtime_error
- **Error Code:** `CUDA_ERROR_ILLEGAL_ADDRESS`
- **Verification:** ai_generated
- **Fix Rate:** 85%

## Root Cause

A CUDA kernel encountered an assertion failure on the device (e.g., invalid index in embedding, negative dimension, or NaN in loss), which often causes subsequent operations to fail silently before this error surfaces.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| PyTorch 2.0.0 | active | — | — |
| CUDA 11.7 | active | — | — |
| CUDA 11.8 | active | — | — |
| CUDA 12.1 | active | — | — |
| Ubuntu 22.04 | active | — | — |

## Workarounds

1. **Enable device-side assertions by setting environment variable TORCH_USE_CUDA_DSA=1 before running the script, then re-run to get a detailed stack trace pointing to the failing operation (e.g., embedding lookup with out-of-range index). Example: TORCH_USE_CUDA_DSA=1 python train.py** (90% success)
   ```
   Enable device-side assertions by setting environment variable TORCH_USE_CUDA_DSA=1 before running the script, then re-run to get a detailed stack trace pointing to the failing operation (e.g., embedding lookup with out-of-range index). Example: TORCH_USE_CUDA_DSA=1 python train.py
   ```
2. **Add gradient clipping and NaN checks in the training loop: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0); if torch.isnan(loss): print('NaN loss'); return** (80% success)
   ```
   Add gradient clipping and NaN checks in the training loop: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0); if torch.isnan(loss): print('NaN loss'); return
   ```
3. **Wrap the problematic operation in a try-except block and use torch.cuda.synchronize() to catch the exact error location. For example: try: output = model(input); torch.cuda.synchronize(); except RuntimeError as e: print(f'Error at iteration {i}: {e}')** (85% success)
   ```
   Wrap the problematic operation in a try-except block and use torch.cuda.synchronize() to catch the exact error location. For example: try: output = model(input); torch.cuda.synchronize(); except RuntimeError as e: print(f'Error at iteration {i}: {e}')
   ```

## Dead Ends

- **Set torch.backends.cudnn.deterministic = True** — Deterministic mode does not fix invalid tensor values or index errors; it only ensures reproducibility of operations. (95% fail)
- **Increase batch size to trigger error less often** — Larger batch sizes may hide the issue temporarily but do not address the root cause (e.g., out-of-range indices in embedding). The error will reappear on different data. (90% fail)
- **Set CUDA_LAUNCH_BLOCKING=1 environment variable** — While this helps identify the exact operation causing the error, it does not fix the underlying problem such as index errors or NaN values. (85% fail)