| RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable |
— |
cuda |
device |
88% |
90% |
ai_generated |
| RuntimeError: Trying to backward through the graph a second time |
— |
cuda |
autograd_error |
92% |
90% |
ai_generated |
| RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False |
— |
cuda |
serialization_error |
96% |
95% |
ai_generated |
| CUDA error: context is destroyed (cudaErrorContextIsDestroyed) |
— |
cuda |
runtime_error |
82% |
85% |
ai_generated |
| CUDA error: cooperative launch of kernel exceeds device occupancy (cudaErrorCooperativeLaunchTooLarge) |
— |
cuda |
kernel_launch_error |
85% |
88% |
ai_generated |
| RuntimeError: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle) |
— |
cuda |
library_initialization |
83% |
79% |
ai_generated |
| RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED |
— |
cuda |
runtime_error |
82% |
85% |
ai_generated |
| RuntimeError: CUDA error: no kernel image is available for execution on the device |
— |
cuda |
compatibility_error |
88% |
90% |
ai_generated |
| RuntimeError: The NVIDIA driver on your system is too old |
— |
cuda |
driver_error |
90% |
90% |
ai_generated |
| RuntimeError: CUDA error: operation not permitted when stream is capturing |
— |
cuda |
runtime_error |
78% |
82% |
ai_generated |
| RuntimeError: CUDA error: initialization error |
— |
cuda |
init_error |
78% |
85% |
ai_generated |
| RuntimeError: CUDA lazy loading is not enabled |
— |
cuda |
config_error |
90% |
90% |
ai_generated |
| CUDA error: system not ready (cudaErrorSystemNotReady) |
— |
cuda |
initialization_error |
82% |
85% |
ai_generated |
| RuntimeError: CUDA error: unknown error |
— |
cuda |
runtime_error |
78% |
75% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED |
EXECUTION |
cuda |
runtime_error |
80% |
80% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED |
— |
cuda |
compatibility |
88% |
90% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED |
— |
cuda |
library_error |
85% |
87% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED on RNN backward |
EXECUTION |
cuda |
runtime_error |
82% |
85% |
ai_generated |
| RuntimeError: cuDNN version incompatibility |
— |
cuda |
compatibility_error |
88% |
90% |
ai_generated |
| RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR |
ERNAL |
cuda |
library_error |
78% |
85% |
ai_generated |
| RuntimeError: CUDA error: device-side assert triggered |
— |
cuda |
runtime |
88% |
90% |
ai_generated |
| CUDA error: an illegal memory access was encountered due to ECC uncorrectable error |
ECC |
cuda |
hardware_error |
60% |
80% |
ai_generated |
| CUDA error: an error occurred in the external semaphore (cudaErrorExternalDevice) |
— |
cuda |
interop_error |
78% |
82% |
ai_generated |
| RuntimeError: FlashAttention only supports Ampere GPUs or newer (sm >= 80) |
— |
cuda |
runtime_error |
85% |
88% |
ai_generated |
| CUDA error: the graph update was not performed because it included changes which violated constraints specific to instantiated graph update (cudaErrorGraphExecUpdateFailure) |
— |
cuda |
graph_error |
83% |
86% |
ai_generated |
| RuntimeError: CUDA error: an illegal memory access was encountered |
— |
cuda |
runtime |
85% |
88% |
ai_generated |
| CUDA error: invalid device ordinal (cudaErrorInvalidDevice) |
— |
cuda |
configuration_error |
92% |
95% |
ai_generated |
| CUDA error: invalid pitch argument (cudaErrorInvalidPitchValue) |
— |
cuda |
memory_error |
85% |
88% |
ai_generated |
| CUDA error: map buffer object failed (cudaErrorMapBufferObjectFailed) |
— |
cuda |
memory_error |
78% |
82% |
ai_generated |
| RuntimeError: CUDA error: misaligned address |
— |
cuda |
runtime_error |
55% |
78% |
ai_generated |
| RuntimeError: expected scalar type Half but found Float |
— |
cuda |
runtime_error |
88% |
88% |
ai_generated |
| RuntimeError: CUDA error: peer access is not supported between these two devices |
— |
cuda |
runtime_error |
65% |
80% |
ai_generated |
| RuntimeError: NCCL error: invalid argument - ncclInvalidArgument |
— |
cuda |
distributed_computing |
82% |
85% |
ai_generated |
| RuntimeError: NCCL error: Timeout (NCCL_ERROR_TIMEOUT) |
ERROR |
cuda |
runtime_error |
65% |
82% |
ai_generated |
| RuntimeError: NCCL communicator was aborted on rank 0. Original reason for aborting was: Timeout at NCCL sequence 42 |
— |
cuda |
distributed_communication |
74% |
72% |
ai_generated |
| RuntimeError: NCCL error: unhandled system error, NCCL version |
— |
cuda |
distributed_error |
75% |
82% |
ai_generated |
| CUDA error: no kernel image is available for execution on the device |
— |
cuda |
compatibility |
90% |
92% |
ai_generated |
| CUDA error: initialization error (cudaErrorInitializationError) |
— |
cuda |
initialization_error |
85% |
88% |
ai_generated |
| FileNotFoundError: nvcc not found. Please ensure CUDA is installed |
— |
cuda |
build |
94% |
96% |
ai_generated |
| NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver |
— |
cuda |
driver_error |
60% |
82% |
ai_generated |
| CUDA error: an NVLink error was detected on the link (cudaErrorNvlinkUncorrectable) |
— |
cuda |
hardware_error |
72% |
78% |
ai_generated |
| RuntimeError: NVRTC compilation failed |
— |
cuda |
runtime_error |
78% |
82% |
ai_generated |
| CUDA error: profiler not initialized (cudaErrorProfilerNotInitialized) |
— |
cuda |
profiling_error |
87% |
90% |
ai_generated |
| CUDA error: shared object initialization failed (cudaErrorSharedObjectInitFailed) |
— |
cuda |
initialization_error |
85% |
88% |
ai_generated |
| CUDA error: operation failed due to a previous error during capture (cudaErrorStreamCaptureMerge) |
— |
cuda |
graph_error |
80% |
84% |
ai_generated |
| RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu |
— |
cuda |
device_error |
95% |
93% |
ai_generated |
| CUDA error: invalid configuration argument (cudaErrorInvalidConfiguration) - too many threads per block |
— |
cuda |
kernel_launch_error |
92% |
95% |
ai_generated |
| torch.cuda.OutOfMemoryError: CUDA out of memory |
— |
cuda |
oom_error |
85% |
90% |
ai_generated |
| AssertionError: Torch not compiled with CUDA enabled |
— |
cuda |
install_error |
90% |
88% |
ai_generated |
| RuntimeError: Triton compilation failed: LLVM ERROR |
ERROR |
cuda |
runtime_error |
80% |
82% |
ai_generated |
| RuntimeError: The current CUDA version is not compatible with the installed PyTorch build. CUDA version: 12.1, PyTorch CUDA: 11.8 |
— |
cuda |
version_incompatibility |
92% |
88% |
ai_generated |
| CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate_v2 |
CUBLAS_STATUS_ALLOC_FAILED |
cuda |
runtime_error |
75% |
85% |
ai_generated |
| CUDA error: stream-order violation during graph launch (cudaErrorStreamOrderViolation) |
cudaErrorStreamOrderViolation |
cuda |
runtime_error |
85% |
88% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM when setting batch normalization epsilon to 0 |
CUDNN_STATUS_BAD_PARAM |
cuda |
type_error |
95% |
90% |
ai_generated |
| RuntimeError: NVRTC compilation failed: error: cannot open source file 'cuda_runtime.h' |
— |
cuda |
build_error |
90% |
87% |
ai_generated |
| RuntimeError: CUBLAS_STATUS_INVALID_VALUE when calling cublasGemmStridedBatchedEx with batch_count > 0 but A/B/C matrices have incompatible dimensions |
CUBLAS_STATUS_INVALID_VALUE |
cuda |
runtime_error |
80% |
82% |
ai_generated |
| RuntimeError: NCCL error: unhandled system error, NCCL version 2.18.5, error: IB device not found |
— |
cuda |
network_error |
80% |
86% |
ai_generated |
| CUDA error: driver is in a state that is invalid for the requested operation (cudaErrorDriverNotReady) |
cudaErrorDriverNotReady (804) |
cuda |
runtime_error |
78% |
85% |
ai_generated |
| CUDA error: MPS client failed to connect (cudaErrorMpsClientFailed) |
cudaErrorMpsClientFailed (803) |
cuda |
runtime_error |
82% |
88% |
ai_generated |
| CUDA error: invalid resource type (cudaErrorInvalidResourceType) |
cudaErrorInvalidResourceType (801) |
cuda |
type_error |
80% |
83% |
ai_generated |
| CUDA error: unsupported exec affinity (cudaErrorUnsupportedExecAffinity) |
cudaErrorUnsupportedExecAffinity (802) |
cuda |
runtime_error |
80% |
84% |
ai_generated |
| CUDA error: invalid program counter (cudaErrorInvalidPc) |
cudaErrorInvalidPc (805) |
cuda |
runtime_error |
75% |
82% |
ai_generated |
| CUDA error: invalid sync object (cudaErrorInvalidSyncObject) |
cudaErrorInvalidSyncObject (806) |
cuda |
runtime_error |
80% |
86% |
ai_generated |
| RuntimeError: CUDA error: CUBLAS_STATUS_ARCH_MISMATCH when calling cublasSgemm |
CUBLAS_STATUS_ARCH_MISMATCH |
cuda |
runtime_error |
82% |
85% |
ai_generated |
| CUDA error: driver version is insufficient for CUDA runtime version (cudaErrorInsufficientDriver) |
cudaErrorInsufficientDriver |
cuda |
config_error |
90% |
88% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM when calling cudnnBatchNormalizationForwardTraining |
CUDNN_STATUS_BAD_PARAM |
cuda |
runtime_error |
78% |
83% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM when calling cudnnRNNForwardTraining with input size mismatch |
CUDNN_STATUS_BAD_PARAM |
cuda |
runtime_error |
80% |
84% |
ai_generated |
| CUDA error: MPS server is not running (cudaErrorMpsServerNotReady) |
cudaErrorMpsServerNotReady |
cuda |
system_error |
88% |
82% |
ai_generated |
| CUDA error: MPS server: maximum partition size exceeded (cudaErrorMpsMaxPartitionSizeExceeded) |
cudaErrorMpsMaxPartitionSizeExceeded |
cuda |
resource_error |
75% |
85% |
ai_generated |
| RuntimeError: NVRTC compilation failed: error: cannot open source file 'cuda.h' |
NVRTC_ERROR_COMPILATION |
cuda |
build_error |
85% |
88% |
ai_generated |
| RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate_v2 |
CUBLAS_STATUS_ALLOC_FAILED |
cuda |
resource_error |
78% |
82% |
ai_generated |
| CUDA error: driver library is not loaded (cudaErrorDriverLibraryNotLoaded) |
cudaErrorDriverLibraryNotLoaded |
cuda |
system_error |
82% |
87% |
ai_generated |
| RuntimeError: CUDA error: peer access is not supported between these two devices (cudaErrorPeerAccessUnsupported) |
cudaErrorPeerAccessUnsupported |
cuda |
runtime_error |
70% |
84% |
ai_generated |
| RuntimeError: Tensor Cores are not supported on the current device architecture (compute capability < 7.0) |
CUDNN_STATUS_ARCH_MISMATCH |
cuda |
type_error |
90% |
86% |
ai_generated |
| RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatchedEx with invalid broadcast dimensions |
CUBLAS_STATUS_INVALID_VALUE |
cuda |
runtime_error |
82% |
85% |
ai_generated |
| RuntimeError: CUDA error: an illegal memory access was encountered after a cudaFree call on a tensor still in use |
cudaErrorIllegalAddress |
cuda |
runtime_error |
79% |
82% |
ai_generated |
| RuntimeError: CUDA error: operation not permitted when stream is capturing (streamCaptureInvalidated) |
cudaErrorStreamCaptureInvalidated |
cuda |
runtime_error |
81% |
87% |
ai_generated |
| RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR when computing SVD of a singular matrix |
CUSOLVER_STATUS_INTERNAL_ERROR |
cuda |
runtime_error |
76% |
84% |
ai_generated |
| RuntimeError: Triton compilation failed: LLVM ERROR: out of memory when compiling kernel with large shared memory |
— |
cuda |
build_error |
72% |
83% |
ai_generated |
| RuntimeError: CUDA error: an illegal memory access was encountered in an asynchronous operation |
cudaErrorIllegalAddress |
cuda |
runtime_error |
85% |
85% |
ai_generated |
| CUDA error: no kernel image is available for execution on the device (cudaErrorNoKernelImageAvailable) |
cudaErrorNoKernelImageAvailable |
cuda |
build_error |
90% |
88% |
ai_generated |
| RuntimeError: MPS backend out of memory (MPS allocated: 2.00 GB, limit: 2.00 GB) |
— |
cuda |
resource_error |
85% |
82% |
ai_generated |
| RuntimeError: NCCL error: version mismatch, expected 2.18.5 but got 2.19.1 |
ncclSystemError |
cuda |
config_error |
90% |
87% |
ai_generated |
| RuntimeError: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm_v2 with handle=0x0 |
CUBLAS_STATUS_INVALID_VALUE |
cuda |
runtime_error |
85% |
83% |
ai_generated |
| RuntimeError: Triton compilation failed: unsupported instruction 'mma.sync.aligned.m16n8k16.row.col.f16.f16.f16' |
— |
cuda |
build_error |
80% |
86% |
ai_generated |
| RuntimeError: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, batchCount ) |
CUBLAS_STATUS_INVALID_VALUE |
cuda |
runtime_error |
80% |
85% |
ai_generated |
| nvcc fatal : Cannot find compiler 'cl.exe' in PATH |
— |
cuda |
build_error |
85% |
90% |
ai_generated |
| CUDA error: MPS heap memory limit exceeded (cudaErrorMpsHeapMemoryLimitExceeded) |
cudaErrorMpsHeapMemoryLimitExceeded |
cuda |
resource_error |
80% |
87% |
ai_generated |
| RuntimeError: Triton Error [CUDA]: PTX assembly failed: ptxas fatal : Ptx assembly aborted due to errors |
— |
cuda |
build_error |
75% |
85% |
ai_generated |
| RuntimeError: CUDA error: the graph update was not performed because it included changes which violated constraints specific to instantiated CUDA graphs (cudaErrorGraphUpdateViolation) - memory pool mismatch |
cudaErrorGraphUpdateViolation |
cuda |
runtime_error |
80% |
86% |
ai_generated |
| RuntimeError: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched with lda=0 |
CUBLAS_STATUS_INVALID_VALUE (1) |
cuda |
runtime_error |
90% |
85% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED when calling cudnnRNNBackwardData_v8 with training mode enabled and double backward |
CUDNN_STATUS_NOT_SUPPORTED (5) |
cuda |
runtime_error |
78% |
82% |
ai_generated |
| RuntimeError: NVRTC compilation failed: error: Ptx assembly requires .target sm_52 or higher. Current target: sm_50 |
NVRTC_ERROR_COMPILATION (6) |
cuda |
build_error |
92% |
88% |
ai_generated |
| terminate called after throwing an instance of 'thrust::system::system_error' what(): after cudaGetLastError: an illegal memory access was encountered |
cudaErrorIllegalAddress (77) |
cuda |
runtime_error |
80% |
84% |
ai_generated |
| RuntimeError: CUBLAS_STATUS_INVALID_VALUE when calling cublasGemmStridedBatchedEx with batch_count=0 |
CUBLAS_STATUS_INVALID_VALUE (1) |
cuda |
runtime_error |
95% |
87% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM when calling cudnnBatchNormalizationForwardTraining with epsilon < 0 |
CUDNN_STATUS_BAD_PARAM (4) |
cuda |
runtime_error |
93% |
86% |
ai_generated |
| RuntimeError: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmEx( handle, opa, opb, m, n, k, &alpha, a, atype, lda, b, btype, ldb, &beta, c, ctype, ldc, compute_type, algo) |
CUBLAS_STATUS_NOT_SUPPORTED |
cuda |
runtime_error |
82% |
88% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM when calling cudnnSetRNNDescriptor_v8 with hiddenSize=4096, numLayers=4, and dropout=0.0 |
CUDNN_STATUS_BAD_PARAM |
cuda |
runtime_error |
90% |
85% |
ai_generated |
| RuntimeError: NCCL error: socket error: connect to 192.168.1.100:29500 failed: Connection timed out |
— |
cuda |
network_error |
80% |
87% |
ai_generated |
| RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasLtMatmulAlgoGetHeuristic |
CUBLAS_STATUS_ALLOC_FAILED |
cuda |
resource_error |
78% |
86% |
ai_generated |
| RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM when calling cudnnBatchNormalizationForwardTraining with epsilon=1e-06 |
CUDNN_STATUS_BAD_PARAM |
cuda |
runtime_error |
92% |
84% |
ai_generated |
| RuntimeError: Triton compilation failed: error: Kernel launch timed out after 300 seconds |
— |
cuda |
runtime_error |
75% |
83% |
ai_generated |