pytorch multiprocessing_error ai_generated true

RuntimeError: DataLoader worker (pid(s) 12345) exited unexpectedly: OSError: [Errno 9] Bad file descriptor

ID: pytorch/dataloader-bad-file-descriptor

Also available as: JSON · Markdown
80%Fix Rate
86%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

DataLoader worker process crashes with bad file descriptor. Usually caused by opening files in __init__ instead of __getitem__, or by fork-related resource sharing issues.

generic

Workarounds

  1. 92% success Open files lazily in __getitem__ instead of __init__ 
    Store file paths in __init__; open files in __getitem__: with open(self.paths[idx]) as f: data = f.read()

    Sources: https://pytorch.org/tutorials/

  2. 85% success Use multiprocessing start method 'spawn' instead of 'fork' 
    torch.multiprocessing.set_start_method('spawn')  # avoids fork-related fd inheritance issues
  3. 82% success Use persistent_workers=True to keep workers alive across epochs 
    DataLoader(dataset, num_workers=4, persistent_workers=True)  # workers keep their file handles open

Dead Ends

Common approaches that don't work:

  1. Set num_workers=0 as a permanent fix 60% fail

    Eliminates multiprocessing but makes data loading the bottleneck; training throughput drops significantly

  2. Increase ulimit for open files 70% fail

    The issue is file handles not being picklable across process fork, not hitting the limit