pytorch runtime_error ai_generated partial

RuntimeError: DataLoader worker (pid 12345) timed out after 300 seconds waiting for data from queue. Consider increasing timeout or reducing batch size.

ID: pytorch/dataloader-queue-timeout

Also available as: JSON · Markdown · 中文
80%Fix Rate
86%Confidence
1Evidence
2024-06-18First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
torch>=1.7 active — — —
torch<=2.5.1 active — — —

Root Cause

DataLoader worker processes are stuck waiting for data from the prefetch queue, typically due to slow data loading, deadlock in custom dataset, or insufficient shared memory.

generic

中文

DataLoader 工作进程在等待预取队列数据时卡住，通常是由于数据加载慢、自定义数据集中的死锁或共享内存不足。

Official Documentation

https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

Workarounds

  1. 85% success Increase the timeout parameter in DataLoader. For example: DataLoader(dataset, batch_size=32, num_workers=4, timeout=600) to allow up to 600 seconds. 
    Increase the timeout parameter in DataLoader. For example: DataLoader(dataset, batch_size=32, num_workers=4, timeout=600) to allow up to 600 seconds.
  2. 90% success Reduce batch size or number of workers to lower memory and I/O pressure. For example: DataLoader(dataset, batch_size=16, num_workers=2) 
    Reduce batch size or number of workers to lower memory and I/O pressure. For example: DataLoader(dataset, batch_size=16, num_workers=2)
  3. 75% success Use prefetch_factor=1 in DataLoader to reduce prefetch queue size: DataLoader(dataset, batch_size=32, num_workers=4, prefetch_factor=1) 
    Use prefetch_factor=1 in DataLoader to reduce prefetch queue size: DataLoader(dataset, batch_size=32, num_workers=4, prefetch_factor=1)

中文步骤

  1. Increase the timeout parameter in DataLoader. For example: DataLoader(dataset, batch_size=32, num_workers=4, timeout=600) to allow up to 600 seconds.
  2. Reduce batch size or number of workers to lower memory and I/O pressure. For example: DataLoader(dataset, batch_size=16, num_workers=2)
  3. Use prefetch_factor=1 in DataLoader to reduce prefetch queue size: DataLoader(dataset, batch_size=32, num_workers=4, prefetch_factor=1)

Dead Ends

Common approaches that don't work:

  1. Increasing the number of workers to speed up loading 70% fail

    More workers can exacerbate shared memory contention and lead to more timeouts, especially if the bottleneck is I/O or CPU.

  2. Setting timeout=0 to disable timeout 80% fail

    Setting timeout=0 means no timeout, but the worker may hang indefinitely, causing the training to stall.