data memory_error ai_generated true

ExecutorLostFailure: Out of memory error in executor

ID: data/spark-executor-oom

Also available as: JSON · Markdown
78%Fix Rate
83%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

Spark executor ran out of memory.

generic

Workarounds

  1. 88% success Increase spark.executor.memory and spark.executor.memoryOverhead
  2. 90% success Repartition data to reduce per-partition size

Dead Ends

Common approaches that don't work:

  1. Set executor memory to cluster maximum 72% fail

    Reduces parallelism drastically

  2. Disable memory overhead checking 80% fail

    Container killed by YARN