cudaErrorNoKernelImageAvailable cuda build_error ai_generated true

CUDA error: no kernel image is available for execution on the device (cudaErrorNoKernelImageAvailable)

ID: cuda/device-incompatible-cubin

Also available as: JSON · Markdown · 中文
90%Fix Rate
88%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
CUDA 12.0 active — — —
PyTorch 2.0.0 active — — —
NVIDIA A100 (sm_80) active — — —

Root Cause

The compiled CUDA binary (cubin) was generated for a different GPU architecture than the one currently in use, e.g., sm_80 vs sm_86.

generic

中文

编译的CUDA二进制文件（cubin）是为与当前使用的GPU架构不同的架构生成的，例如sm_80与sm_86。

Official Documentation

https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#gpu-compilation

Workarounds

  1. 90% success Recompile the kernel with the correct architecture. For PyTorch, set environment variable `TORCH_CUDA_ARCH_LIST='8.6'` for Ampere GPUs. For custom CUDA code, compile with `-arch=sm_86`. 
    Recompile the kernel with the correct architecture. For PyTorch, set environment variable `TORCH_CUDA_ARCH_LIST='8.6'` for Ampere GPUs. For custom CUDA code, compile with `-arch=sm_86`.

中文步骤

  1. Recompile the kernel with the correct architecture. For PyTorch, set environment variable `TORCH_CUDA_ARCH_LIST='8.6'` for Ampere GPUs. For custom CUDA code, compile with `-arch=sm_86`.

Dead Ends

Common approaches that don't work:

  1. 80% fail

    The issue is architecture-specific, not CUDA version.

  2. 95% fail

    Caching only affects runtime memory, not compiled code.