tensorflow memory_error ai_generated true

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor

ID: tensorflow/oom-allocating-tensor

Also available as: JSON · Markdown
85%Fix Rate
90%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

TensorFlow GPU memory exhausted. Batch too large or TF is allocating all GPU memory upfront.

generic

Workarounds

  1. 92% success Enable GPU memory growth to allocate on demand 
    gpus = tf.config.list_physical_devices('GPU'); tf.config.experimental.set_memory_growth(gpus[0], True)

    Sources: https://www.tensorflow.org/api_docs/python/tf

  2. 90% success Reduce batch size
  3. 85% success Set memory limit per GPU 
    tf.config.set_logical_device_configuration(gpu, [tf.config.LogicalDeviceConfiguration(memory_limit=4096)])

Dead Ends

Common approaches that don't work:

  1. Set TF_FORCE_GPU_ALLOW_GROWTH=true as environment variable only 55% fail

    Some TF versions ignore the env var; set it programmatically as well

  2. Install tensorflow-cpu to avoid GPU issues 82% fail

    Completely gives up GPU acceleration. Fix memory management instead.