cudaErrorInvalidPc (805) cuda runtime_error ai_generated partial

CUDA error: invalid program counter (cudaErrorInvalidPc)

ID: cuda/cuda-error-invalid-pc

Also available as: JSON · Markdown · 中文
75%Fix Rate
82%Confidence
1Evidence
2025-04-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
CUDA 12.4 active — — —
CUDA 12.6 active — — —
NVIDIA Driver 550.54.10 active — — —
NVIDIA Driver 560.35.03 active — — —
PyTorch 2.5.0 active — — —

Root Cause

The GPU attempted to execute a kernel with an invalid program counter, typically due to a corrupted device function pointer, a miscompiled kernel, or an out-of-bounds jump in device code (e.g., from a misused function pointer or indirect call).

generic

中文

GPU 尝试执行具有无效程序计数器的内核，通常是由于损坏的设备函数指针、编译错误的内核或设备代码中的越界跳转（例如，误用函数指针或间接调用）。

Official Documentation

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html

Workarounds

  1. 85% success Compile the kernel with `-lineinfo` and run with `cuda-memcheck` or `compute-sanitizer` to identify the exact source line causing the invalid jump: `compute-sanitizer --tool memcheck ./my_app` 
    Compile the kernel with `-lineinfo` and run with `cuda-memcheck` or `compute-sanitizer` to identify the exact source line causing the invalid jump: `compute-sanitizer --tool memcheck ./my_app`
  2. 80% success Avoid using function pointers in device code if possible; replace them with switch statements or templates to eliminate indirect jumps. 
    Avoid using function pointers in device code if possible; replace them with switch statements or templates to eliminate indirect jumps.
  3. 75% success Ensure all device function pointers are initialized correctly and not left as null or garbage. For example, in CUDA C++: `typedef void (*func_t)(); func_t f = &my_device_func;` 
    Ensure all device function pointers are initialized correctly and not left as null or garbage. For example, in CUDA C++: `typedef void (*func_t)(); func_t f = &my_device_func;`

中文步骤

  1. Compile the kernel with `-lineinfo` and run with `cuda-memcheck` or `compute-sanitizer` to identify the exact source line causing the invalid jump: `compute-sanitizer --tool memcheck ./my_app`
  2. Avoid using function pointers in device code if possible; replace them with switch statements or templates to eliminate indirect jumps.
  3. Ensure all device function pointers are initialized correctly and not left as null or garbage. For example, in CUDA C++: `typedef void (*func_t)(); func_t f = &my_device_func;`

Dead Ends

Common approaches that don't work:

  1. 80% fail

    The error occurs during kernel execution, not during API calls; post-hoc checks do not prevent the invalid program counter from being reached.

  2. 90% fail

    Scheduling policy does not affect kernel correctness; the invalid PC is a code bug, not a synchronization issue.