pytorch runtime_error ai_generated partial

运行时错误：DataLoader 工作进程（pid 12345）在等待队列数据 300 秒后超时。考虑增加超时时间或减小批次大小。

RuntimeError: DataLoader worker (pid 12345) timed out after 300 seconds waiting for data from queue. Consider increasing timeout or reducing batch size.

ID: pytorch/dataloader-queue-timeout

其他格式: JSON · Markdown 中文 · English
80%修复率
86%置信度
1证据数
2024-06-18首次发现

版本兼容性

版本状态引入弃用备注
torch>=1.7 active — — —
torch<=2.5.1 active — — —

根因分析

DataLoader 工作进程在等待预取队列数据时卡住，通常是由于数据加载慢、自定义数据集中的死锁或共享内存不足。

English

DataLoader worker processes are stuck waiting for data from the prefetch queue, typically due to slow data loading, deadlock in custom dataset, or insufficient shared memory.

generic

官方文档

https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

解决方案

  1. Increase the timeout parameter in DataLoader. For example: DataLoader(dataset, batch_size=32, num_workers=4, timeout=600) to allow up to 600 seconds.
  2. Reduce batch size or number of workers to lower memory and I/O pressure. For example: DataLoader(dataset, batch_size=16, num_workers=2)
  3. Use prefetch_factor=1 in DataLoader to reduce prefetch queue size: DataLoader(dataset, batch_size=32, num_workers=4, prefetch_factor=1)

无效尝试

常见但无效的做法:

  1. Increasing the number of workers to speed up loading 70% 失败

    More workers can exacerbate shared memory contention and lead to more timeouts, especially if the bottleneck is I/O or CPU.

  2. Setting timeout=0 to disable timeout 80% 失败

    Setting timeout=0 means no timeout, but the worker may hang indefinitely, causing the training to stall.