ENOMEM huggingface system_error ai_generated partial

OSError: Unable to mmap dataset file: [Errno 12] Cannot allocate memory

ID: huggingface/dataset-memory-mapping-failure

Also available as: JSON · Markdown · 中文
75%Fix Rate
85%Confidence
1Evidence
2023-08-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
datasets>=2.10.0 active — — —
Linux kernel >=5.4 active — — —
Python>=3.8 active — — —

Root Cause

The system does not have enough virtual memory address space to memory-map the dataset file, often due to large file size or system ulimit restrictions.

generic

中文

系统没有足够的虚拟内存地址空间来对数据集文件进行内存映射，通常是由于文件过大或系统 ulimit 限制所致。

Official Documentation

https://huggingface.co/docs/datasets/v2.14.0/en/loading#memory-mapping

Workarounds

  1. 70% success Reduce the dataset file size by splitting it into smaller shards before loading. Use `datasets.load_dataset('path/to/dataset', split='train[:50%]')` to load half the data. 
    Reduce the dataset file size by splitting it into smaller shards before loading. Use `datasets.load_dataset('path/to/dataset', split='train[:50%]')` to load half the data.
  2. 85% success Increase the system's vm.max_map_count using `sysctl -w vm.max_map_count=2000000` on Linux. This increases the number of memory-mapped regions allowed. 
    Increase the system's vm.max_map_count using `sysctl -w vm.max_map_count=2000000` on Linux. This increases the number of memory-mapped regions allowed.
  3. 90% success Use `load_dataset(..., streaming=True)` to avoid memory-mapping the entire file by streaming the data in chunks. 
    Use `load_dataset(..., streaming=True)` to avoid memory-mapping the entire file by streaming the data in chunks.

中文步骤

  1. 通过将数据集文件分割成更小的分片来减小文件大小，然后再加载。使用 `datasets.load_dataset('path/to/dataset', split='train[:50%]')` 加载一半数据。
  2. 在Linux上使用 `sysctl -w vm.max_map_count=2000000` 增加系统的 vm.max_map_count。这增加了允许的内存映射区域数量。
  3. 使用 `load_dataset(..., streaming=True)` 通过流式传输数据块来避免对整个文件进行内存映射。

Dead Ends

Common approaches that don't work:

  1. 90% fail

    The error is about virtual memory address space, not physical RAM. Adding RAM does not fix address space exhaustion.

  2. 80% fail

    The error occurs during mmap of the original dataset file, not cache. Disabling caching may cause other issues and does not fix the mmap failure.

  3. 95% fail

    This error is about mmap, not mlock. The ulimit for locked memory is unrelated to virtual memory mapping limits.