cuda library_initialization ai_generated true

RuntimeError: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

ID: cuda/cublas-init-error

Also available as: JSON · Markdown
83%Fix Rate
79%Confidence
22Evidence
2023-04-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
12 active — — —
11 active — — —

Root Cause

CUBLAS_STATUS_NOT_INITIALIZED on CUDA 12.1 with A100 GPUs is most commonly caused by a mismatch between the cuBLAS library version and the CUDA toolkit version. This occurs when CUDA toolkit components are updated piecemeal rather than as a complete unit. Ensuring all CUDA 12.1 libraries are at consistent versions resolves most cases.

generic

Workarounds

  1. 87% success Install a consistent CUDA 12.1 toolkit with matching cuBLAS version 
    sudo apt install cuda-toolkit-12-1 (this installs all components at consistent versions) or download the full toolkit from https://developer.nvidia.com/cuda-12-1-0-download-archive

    Sources: https://developer.nvidia.com/cuda-12-1-0-download-archive

  2. 71% success Set CUBLAS_WORKSPACE_CONFIG environment variable 
    export CUBLAS_WORKSPACE_CONFIG=:4096:8

    Sources: https://docs.nvidia.com/cuda/cublas/index.html#results-reproducibility

  3. 80% success Use the PyTorch-bundled CUDA libraries instead of system CUDA 
    pip install torch --index-url https://download.pytorch.org/whl/cu121 (PyTorch ships its own cuBLAS) && export LD_LIBRARY_PATH=$(python -c 'import torch; print(torch.__path__[0])')/lib:$LD_LIBRARY_PATH

    Sources: https://pytorch.org/get-started/locally/

Dead Ends

Common approaches that don't work:

  1. Reinstalling cuBLAS alone (apt install --reinstall libcublas-12-1) 82% fail

    Reinstalling the same mismatched cuBLAS version does not fix the underlying version incompatibility. The issue is not a corrupted installation but a version mismatch between cuBLAS and the rest of the CUDA toolkit. If cuBLAS 12.1.3 is installed but the CUDA runtime expects 12.1.0, reinstalling 12.1.3 will not help.

  2. Calling torch.cuda.init() manually before operations 91% fail

    torch.cuda.init() initializes the CUDA context and driver API, but cuBLAS initialization is a separate step that happens when the first cuBLAS operation is invoked. If cuBLAS libraries are mismatched, explicit CUDA context initialization does not prevent the cuBLAS init failure.

  3. Setting CUDA_VISIBLE_DEVICES to a single GPU 88% fail

    Restricting visible devices to a single GPU does not resolve library version mismatches. This workaround is sometimes suggested because multi-GPU setups can mask the root cause, but the cuBLAS init error is per-device and occurs regardless of how many GPUs are visible.

Error Chain

Leads to:
Preceded by:
Frequently confused with: