pytorch resource_error ai_generated true

RuntimeError: DataLoader worker (pid 12345) received signal 11 (Segmentation fault). Possible causes: shared memory exhausted or corrupted shared memory files.

ID: pytorch/dataloader-shared-memory-segfault

Also available as: JSON · Markdown · 中文
78%Fix Rate
82%Confidence
1Evidence
2023-06-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.10 active — — —
torch>=2.0 active — — —
Linux kernel 5.x+ active — — —
Docker containers active — — —

Root Cause

DataLoader workers using multiprocessing with shared memory (shm) run out of /dev/shm space or encounter corrupted shared memory files, causing segfault.

generic

中文

DataLoader 工作进程使用多进程和共享内存（shm）时，/dev/shm 空间耗尽或共享内存文件损坏，导致段错误。

Official Documentation

https://pytorch.org/docs/stable/data.html#multi-process-data-loading

Workarounds

  1. 85% success Increase /dev/shm size: docker run --shm-size=8g ... or sudo mount -o remount,size=8G /dev/shm 
    Increase /dev/shm size: docker run --shm-size=8g ... or sudo mount -o remount,size=8G /dev/shm
  2. 75% success Set DataLoader with multiprocessing_context='spawn' and reduce num_workers: DataLoader(dataset, num_workers=2, multiprocessing_context='spawn') 
    Set DataLoader with multiprocessing_context='spawn' and reduce num_workers: DataLoader(dataset, num_workers=2, multiprocessing_context='spawn')
  3. 70% success Disable shared memory by setting DataLoader with persistent_workers=False and pin_memory=False 
    Disable shared memory by setting DataLoader with persistent_workers=False and pin_memory=False

中文步骤

  1. Increase /dev/shm size: docker run --shm-size=8g ... or sudo mount -o remount,size=8G /dev/shm
  2. Set DataLoader with multiprocessing_context='spawn' and reduce num_workers: DataLoader(dataset, num_workers=2, multiprocessing_context='spawn')
  3. Disable shared memory by setting DataLoader with persistent_workers=False and pin_memory=False

Dead Ends

Common approaches that don't work:

  1. Increasing num_workers to speed up loading 70% fail

    More workers consume more shared memory, exacerbating the exhaustion and making segfaults more frequent.

  2. Setting multiprocessing_context to 'fork' on Linux 60% fail

    Fork inherits parent's memory space, but shared memory issues remain; 'spawn' is recommended for PyTorch.