elasticsearch runtime_error ai_generated partial

OutOfMemoryError: Java heap space (field cache eviction) while loading fielddata for field [user.name]

ID: elasticsearch/field-cache-eviction-oom

Also available as: JSON · Markdown · 中文
74%Fix Rate
83%Confidence
1Evidence
2024-04-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Elasticsearch 7.10.0 active — — —
Elasticsearch 8.9.0 active — — —
Elasticsearch 8.14.0 active — — —

Root Cause

The fielddata cache for aggregations on text fields consumes excessive heap memory, leading to an OutOfMemoryError during cache eviction or loading.

generic

中文

用于文本字段聚合的字段数据缓存消耗过多堆内存，导致在缓存驱逐或加载期间发生 OutOfMemoryError。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html#fielddata-circuit-breaker

Workarounds

  1. 80% success Enable `eager_fielddata` on the field mapping to preload fielddata at index time, reducing peak memory usage during queries. Update mapping: `PUT my_index/_mapping {"properties": {"user.name": {"type": "text", "fielddata": true, "eager_fielddata": true}}}` 
    Enable `eager_fielddata` on the field mapping to preload fielddata at index time, reducing peak memory usage during queries. Update mapping: `PUT my_index/_mapping {"properties": {"user.name": {"type": "text", "fielddata": true, "eager_fielddata": true}}}`
  2. 76% success Limit the fielddata cache size and set a circuit breaker. Configure in `elasticsearch.yml`: `indices.fielddata.cache.size: 20%` and `indices.breaker.fielddata.limit: 40%`. Then restart the node. 
    Limit the fielddata cache size and set a circuit breaker. Configure in `elasticsearch.yml`: `indices.fielddata.cache.size: 20%` and `indices.breaker.fielddata.limit: 40%`. Then restart the node.

中文步骤

  1. Enable `eager_fielddata` on the field mapping to preload fielddata at index time, reducing peak memory usage during queries. Update mapping: `PUT my_index/_mapping {"properties": {"user.name": {"type": "text", "fielddata": true, "eager_fielddata": true}}}`
  2. Limit the fielddata cache size and set a circuit breaker. Configure in `elasticsearch.yml`: `indices.fielddata.cache.size: 20%` and `indices.breaker.fielddata.limit: 40%`. Then restart the node.

Dead Ends

Common approaches that don't work:

  1. 65% fail

    This only postpones the OOM error; large aggregations on high-cardinality text fields can still exhaust memory, and large heaps can cause longer GC pauses.

  2. 75% fail

    This can cause severe performance degradation because every aggregation will reload fielddata from disk, leading to high latency and potential OOM from concurrent loads.