pytorch resource_error ai_generated true

RuntimeError: DataLoader worker (pid 12345) received signal 11 (Segmentation fault). Possible causes: shared memory exhaustion, insufficient shared memory, or too many workers.

ID: pytorch/data-loader-worker-segfault-shared-memory

Also available as: JSON · Markdown · 中文
85%Fix Rate
87%Confidence
1Evidence
2023-02-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
pytorch>=1.8.0 active — — —
linux>=4.18 active — — —
cuda>=11.0 active — — —

Root Cause

DataLoader worker processes exhaust the system's shared memory (/dev/shm) limit, causing a segmentation fault when trying to copy tensors from shared memory to the GPU.

generic

中文

DataLoader工作进程耗尽系统共享内存（/dev/shm）限制，在尝试将张量从共享内存复制到GPU时导致段错误。

Official Documentation

https://pytorch.org/docs/stable/data.html#multi-process-data-loading

Workarounds

  1. 90% success Increase the system shared memory limit by running the container or process with `--shm-size=8g` (Docker) or `mount -o remount,size=8G /dev/shm` (host). Alternatively, set `DataLoader(pin_memory=False, num_workers=0)` to bypass shared memory entirely. 
    Increase the system shared memory limit by running the container or process with `--shm-size=8g` (Docker) or `mount -o remount,size=8G /dev/shm` (host). Alternatively, set `DataLoader(pin_memory=False, num_workers=0)` to bypass shared memory entirely.
  2. 75% success Use `torch.utils.data.DataLoader(..., multiprocessing_context='spawn')` to force the spawn start method, which may reduce shared memory fragmentation on some systems. 
    Use `torch.utils.data.DataLoader(..., multiprocessing_context='spawn')` to force the spawn start method, which may reduce shared memory fragmentation on some systems.

中文步骤

  1. Increase the system shared memory limit by running the container or process with `--shm-size=8g` (Docker) or `mount -o remount,size=8G /dev/shm` (host). Alternatively, set `DataLoader(pin_memory=False, num_workers=0)` to bypass shared memory entirely.
  2. Use `torch.utils.data.DataLoader(..., multiprocessing_context='spawn')` to force the spawn start method, which may reduce shared memory fragmentation on some systems.

Dead Ends

Common approaches that don't work:

  1. Reduce the batch size to lower memory pressure. 70% fail

    Reducing batch size does not directly affect shared memory usage per worker; each worker still allocates its own copy of tensors in shared memory for pin_memory.

  2. Increase the number of DataLoader workers to speed up data loading. 90% fail

    Increasing the number of workers exacerbates shared memory exhaustion by spawning more processes that each allocate their own shared memory buffers.