cudaErrorMpsHeapMemoryLimitExceeded cuda resource_error ai_generated true

CUDA 错误：MPS 堆内存限制超出 (cudaErrorMpsHeapMemoryLimitExceeded)

CUDA error: MPS heap memory limit exceeded (cudaErrorMpsHeapMemoryLimitExceeded)

ID: cuda/mps-heap-limit-exceeded

其他格式: JSON · Markdown 中文 · English
80%修复率
87%置信度
1证据数
2023-09-05首次发现

版本兼容性

版本状态引入弃用备注
CUDA 11.8 active — — —
CUDA 12.2 active — — —
MPS 1.0 active — — —
NVIDIA Driver 535.54 active — — —

根因分析

在 NVIDIA 多进程服务 (MPS) 下，MPS 服务器设置的每客户端堆内存限制（通过 CUDA_MPS_HEAP_SIZE）已被当前进程耗尽，通常是由于分配了太多小张量或在长时间训练循环中未释放内存。

English

Under NVIDIA Multi-Process Service (MPS), the per-client heap memory limit set by the MPS server (via CUDA_MPS_HEAP_SIZE) has been exhausted by the current process, typically due to allocating too many small tensors or not freeing memory in a long-running training loop.

generic

官方文档

https://docs.nvidia.com/deploy/mps/index.html#topic_3_4_2

解决方案

  1. 在启动 MPS 守护进程前设置环境变量以增加 MPS 堆大小：`export CUDA_MPS_HEAP_SIZE=4G`（或更大的值如 `8G`），然后使用 `nvidia-cuda-mps-control -d` 重启 MPS。这会为每个客户端分配更多堆内存。
  2. 通过在训练循环中定期使用 `torch.cuda.empty_cache()` 减少内存碎片，或使用 `torch.zeros` 或 `torch.empty` 重用张量，而不是每次迭代创建新张量。
  3. 通过停止 MPS 守护进程从 MPS 切换到每 GPU 单进程（禁用 MPS）：`echo quit | nvidia-cuda-mps-control`。这完全移除堆限制，但会失去 MPS 的进程间通信优势。

无效尝试

常见但无效的做法:

  1. Increasing `torch.cuda.max_memory_allocated` via `torch.cuda.set_per_process_memory_fraction` 90% 失败

    The MPS heap limit is independent of the per-process memory fraction; changing the PyTorch memory limit does not affect the MPS server's heap allocation.

  2. Restarting only the CUDA process without restarting the MPS server 80% 失败

    The MPS server's heap limit is persistent across client restarts; the limit is still in effect unless the server is restarted.

  3. Setting `CUDA_MPS_HEAP_SIZE=0` to disable the limit 75% 失败

    Setting heap size to 0 may cause undefined behavior or default to a very small limit; the environment variable must be set to a positive value or unset to use the default (which is usually larger).