elasticsearch data_error ai_generated true

TranslogCorruptedException: translog corruption detected at position 12345 while recovering shard [my_index][0]

ID: elasticsearch/translog-corruption-on-recovery

Also available as: JSON · Markdown · 中文
82%Fix Rate
87%Confidence
1Evidence
2023-09-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.10.0 active — — —
7.17.0 active — — —
8.0.0 active — — —

Root Cause

The translog file for a shard is corrupted due to a disk error, abrupt shutdown, or filesystem inconsistency, preventing shard recovery.

generic

中文

由于磁盘错误、突然关闭或文件系统不一致，分片的事务日志文件损坏，导致分片恢复失败。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/translog.html

Workarounds

  1. 85% success Truncate the corrupted translog by removing the translog directory for the affected shard (e.g., `rm -rf /var/lib/elasticsearch/nodes/0/indices/<index-uuid>/0/translog`) and then force a recovery from the primary or replica using `POST _cluster/reroute?retry_failed=true`. 
    Truncate the corrupted translog by removing the translog directory for the affected shard (e.g., `rm -rf /var/lib/elasticsearch/nodes/0/indices/<index-uuid>/0/translog`) and then force a recovery from the primary or replica using `POST _cluster/reroute?retry_failed=true`.
  2. 80% success If the shard is a replica, delete it and reallocate: `POST _cluster/reroute { "commands": [ { "cancel": { "index": "my_index", "shard": 0, "node": "node-1" } } ] }` then let Elasticsearch recreate it from the primary. 
    If the shard is a replica, delete it and reallocate: `POST _cluster/reroute { "commands": [ { "cancel": { "index": "my_index", "shard": 0, "node": "node-1" } } ] }` then let Elasticsearch recreate it from the primary.

中文步骤

  1. Truncate the corrupted translog by removing the translog directory for the affected shard (e.g., `rm -rf /var/lib/elasticsearch/nodes/0/indices/<index-uuid>/0/translog`) and then force a recovery from the primary or replica using `POST _cluster/reroute?retry_failed=true`.
  2. If the shard is a replica, delete it and reallocate: `POST _cluster/reroute { "commands": [ { "cancel": { "index": "my_index", "shard": 0, "node": "node-1" } } ] }` then let Elasticsearch recreate it from the primary.

Dead Ends

Common approaches that don't work:

  1. 90% fail

    Filesystem repair may not fix translog corruption at the application level; Elasticsearch may still see invalid data.

  2. 95% fail

    The corruption persists in the translog file; restarting will trigger the same error during recovery.