pytorch system_error ai_generated partial

RuntimeError: DataLoader worker (pid 12345) received signal 11 (Segmentation fault). Possible causes: shared memory exhausted or dataset memory corruption

ID: pytorch/dataloader-worker-segfault

Also available as: JSON · Markdown · 中文
85%Fix Rate
82%Confidence
1Evidence
2023-06-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
pytorch>=1.13 active — — —
torchvision>=0.14 active — — —
cuda>=11.7 active — — —
python>=3.8 active — — —

Root Cause

A DataLoader worker process crashed with a segmentation fault, typically due to insufficient shared memory (on Linux, /dev/shm is too small) or a buggy dataset that corrupts memory (e.g., unsafe multiprocessing with non-picklable objects).

generic

中文

DataLoader工作进程因段错误崩溃，通常由于共享内存不足（Linux上/dev/shm太小）或数据集存在内存损坏错误（例如使用不可pickle对象的不安全多进程）。

Official Documentation

https://pytorch.org/docs/stable/data.html#multi-process-data-loading

Workarounds

  1. 90% success Increase shared memory size: In Docker, add --shm-size=8g; on bare metal, remount /dev/shm with larger size: sudo mount -o remount,size=8G /dev/shm 
    Increase shared memory size: In Docker, add --shm-size=8g; on bare metal, remount /dev/shm with larger size: sudo mount -o remount,size=8G /dev/shm
  2. 80% success Reduce num_workers to 0 or 1 to avoid multiprocessing issues: DataLoader(..., num_workers=0) 
    Reduce num_workers to 0 or 1 to avoid multiprocessing issues: DataLoader(..., num_workers=0)
  3. 75% success Switch multiprocessing context to 'spawn' (instead of 'fork'): torch.multiprocessing.set_start_method('spawn', force=True) 
    Switch multiprocessing context to 'spawn' (instead of 'fork'): torch.multiprocessing.set_start_method('spawn', force=True)

中文步骤

  1. Increase shared memory size: In Docker, add --shm-size=8g; on bare metal, remount /dev/shm with larger size: sudo mount -o remount,size=8G /dev/shm
  2. Reduce num_workers to 0 or 1 to avoid multiprocessing issues: DataLoader(..., num_workers=0)
  3. Switch multiprocessing context to 'spawn' (instead of 'fork'): torch.multiprocessing.set_start_method('spawn', force=True)

Dead Ends

Common approaches that don't work:

  1. Increasing num_workers to speed up data loading 90% fail

    More workers consume more shared memory, making the segfault more likely.

  2. Using pin_memory=True without increasing shared memory 85% fail

    Pin memory also uses shared memory, exacerbating the exhaustion.