cuda runtime ai_generated true

RuntimeError: CUDA error: device-side assert triggered

ID: cuda/device-side-assert

Also available as: JSON · Markdown
88%Fix Rate
90%Confidence
50Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —
12 active — — —

Root Cause

A CUDA kernel assertion failed on the GPU, typically due to out-of-bounds index or invalid label in loss function.

generic

Workarounds

  1. 93% success Set CUDA_LAUNCH_BLOCKING=1 to get the exact line that triggered the assert 
    CUDA_LAUNCH_BLOCKING=1 python train.py
# This makes CUDA operations synchronous so the Python traceback
# shows exactly which operation triggered the assert.
# Look for the line number in the traceback — that's the offending op.

    Sources: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#assertion

  2. 90% success Check loss function inputs: labels must be in [0, num_classes) range 
    # Check label bounds before loss computation:
assert labels.min() >= 0, f"Negative label: {labels.min()}"
assert labels.max() < num_classes, f"Label {labels.max()} >= num_classes {num_classes}"

# Check embedding indices:
assert indices.max() < vocab_size, f"Index {indices.max()} >= vocab_size {vocab_size}"

Dead Ends

Common approaches that don't work:

  1. Restarting the Python process to clear the error 60% fail

    Device-side asserts are sticky; error persists until GPU context is reset

  2. Catching RuntimeError and continuing training 85% fail

    GPU is in an error state; all subsequent CUDA operations will fail

Error Chain

Frequently confused with: