pytorch resource_error ai_generated true

RuntimeError: DataLoader worker (pid 12345) received signal 9 (SIGKILL). Possible causes: out of memory or system limit reached.

ID: pytorch/dataloader-num-workers-memory-leak

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2023-09-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
1.10 active — — —
1.11 active — — —
1.12 active — — —
1.13 active — — —
2.0 active — — —
2.1 active — — —
2.2 active — — —
2.3 active — — —

Root Cause

Too many DataLoader workers with high memory usage per sample cause the system OOM killer to terminate workers, especially when using large datasets or high-resolution images.

generic

中文

DataLoader 工作进程数量过多且每个样本内存使用量高，导致系统 OOM killer 终止进程，尤其在使用大数据集或高分辨率图像时。

Official Documentation

https://pytorch.org/docs/stable/data.html#multi-process-data-loading

Workarounds

  1. 90% success Reduce num_workers to a lower value, e.g., 2 or 4, or set to 0 for single-process loading to eliminate worker memory overhead. 
    Reduce num_workers to a lower value, e.g., 2 or 4, or set to 0 for single-process loading to eliminate worker memory overhead.
  2. 75% success Use prefetch_factor to limit the number of batches prefetched per worker, reducing peak memory usage. 
    Use prefetch_factor to limit the number of batches prefetched per worker, reducing peak memory usage.

中文步骤

  1. Reduce num_workers to a lower value, e.g., 2 or 4, or set to 0 for single-process loading to eliminate worker memory overhead.
  2. Use prefetch_factor to limit the number of batches prefetched per worker, reducing peak memory usage.

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Increasing workers exacerbates memory pressure, causing more frequent OOM kills.

  2. 50% fail

    This doesn't reduce per-worker memory; it just changes how data is split.