REI tensorflow resource_error ai_generated true

资源耗尽错误：从迭代器获取下一个元素失败：读取数据时内存不足

ResourceExhaustedError: Failed to get next element from iterator: Out of memory while reading data

ID: tensorflow/resource-exhausted-iterator-get-next

其他格式: JSON · Markdown 中文 · English
82%修复率
86%置信度
1证据数
2024-06-18首次发现

版本兼容性

版本状态引入弃用备注
tensorflow 2.13.0 active — — —
tensorflow 2.14.0 active — — —

根因分析

数据管道预取消耗过多内存，通常由于批大小过大、数据集元素过大或无限预取导致。

English

Data pipeline prefetching consumes too much memory, often due to large batch size, large dataset elements, or infinite prefetch.

generic

官方文档

https://www.tensorflow.org/api_docs/python/tf/data/Dataset#prefetch

解决方案

  1. Limit the prefetch buffer size using tf.data.Dataset.prefetch with a specific buffer size:
dataset = dataset.batch(32).prefetch(buffer_size=tf.data.AUTOTUNE)
# Or manually set buffer_size to a small number like 1 or 2
# Alternatively, use .prefetch(1) to limit to one batch
  2. Reduce batch size to lower memory usage per element:
dataset = dataset.batch(16).prefetch(tf.data.AUTOTUNE)
# Or use smaller batch size like 8
  3. Use tf.data.Dataset.cache to avoid re-reading large files, combined with controlled prefetch:
dataset = dataset.cache().batch(32).prefetch(1)

无效尝试

常见但无效的做法:

  1. 70% 失败

    Memory is consumed by prefetch buffer, not worker count.

  2. 90% 失败

    The error is about TensorFlow's internal buffer limits, not system memory capacity.