pytorch system_error ai_generated partial

RuntimeError: DataLoader worker (pid 12345) exited unexpectedly: Bus error (core dumped)

ID: pytorch/dataloader-worker-exit-with-bus-error

Also available as: JSON · Markdown · 中文
78%Fix Rate
83%Confidence
1Evidence
2024-02-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.12.0 active — — —
cuda>=11.0 active — — —
python>=3.8 active — — —

Root Cause

DataLoader worker process encountered a bus error due to shared memory corruption or misalignment, often caused by large tensors in shared memory with incompatible alignment.

generic

中文

数据加载器工作进程因共享内存损坏或对齐问题遇到总线错误，通常由共享内存中不兼容对齐的大张量引起。

Official Documentation

https://pytorch.org/docs/stable/data.html#multi-process-data-loading

Workarounds

  1. 85% success Set multiprocessing start method to 'spawn' instead of 'fork' to avoid shared memory corruption. 
    Set multiprocessing start method to 'spawn' instead of 'fork' to avoid shared memory corruption.
  2. 75% success Reduce DataLoader batch size or use prefetch_factor=2 to limit shared memory usage. 
    Reduce DataLoader batch size or use prefetch_factor=2 to limit shared memory usage.

中文步骤

  1. Set multiprocessing start method to 'spawn' instead of 'fork' to avoid shared memory corruption.
  2. Reduce DataLoader batch size or use prefetch_factor=2 to limit shared memory usage.

Dead Ends

Common approaches that don't work:

  1. Increasing the number of workers in DataLoader 90% fail

    More workers increase contention for shared memory and can exacerbate the bus error.

  2. Setting pin_memory=False in DataLoader 80% fail

    Bus errors are not primarily caused by pin_memory; disabling it may not address the underlying memory alignment issue.