elasticsearch resource_error ai_generated true

ClusterBlockException: index [my_index] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; disk usage exceeded flood-stage watermark

ID: elasticsearch/index-read-only-due-to-disk-threshold

Also available as: JSON · Markdown · 中文
85%Fix Rate
90%Confidence
1Evidence
2023-09-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.17.10 active — — —
8.6.2 active — — —
8.11.0 active — — —

Root Cause

The disk usage on a node has exceeded the flood-stage watermark (default 95%), causing Elasticsearch to automatically set the index to read-only to prevent data loss.

generic

中文

节点上的磁盘使用量超过了洪水水位线（默认 95%），导致 Elasticsearch 自动将索引设置为只读以防止数据丢失。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

Workarounds

  1. 90% success Free disk space by deleting old indices or moving data to another node: DELETE /old_logs_index-2024-01 or use ILM to rollover. Then remove the block: PUT /my_index/_settings { "index.blocks.read_only_allow_delete": null } 
    Free disk space by deleting old indices or moving data to another node: DELETE /old_logs_index-2024-01 or use ILM to rollover. Then remove the block: PUT /my_index/_settings { "index.blocks.read_only_allow_delete": null }
  2. 85% success Add more disk space to the node or move shards to nodes with more space using the Cluster Reroute API: POST /_cluster/reroute { "commands": [{ "move": { "index": "my_index", "shard": 0, "from_node": "full_node", "to_node": "free_node" } }] } 
    Add more disk space to the node or move shards to nodes with more space using the Cluster Reroute API: POST /_cluster/reroute { "commands": [{ "move": { "index": "my_index", "shard": 0, "from_node": "full_node", "to_node": "free_node" } }] }

中文步骤

  1. 通过删除旧索引或将数据移动到其他节点释放磁盘空间：DELETE /old_logs_index-2024-01 或使用 ILM 滚动。然后移除块：PUT /my_index/_settings { "index.blocks.read_only_allow_delete": null }
  2. 为节点增加更多磁盘空间，或使用 Cluster Reroute API 将分片移动到有更多空间的节点：POST /_cluster/reroute { "commands": [{ "move": { "index": "my_index", "shard": 0, "from_node": "full_node", "to_node": "free_node" } }] }

Dead Ends

Common approaches that don't work:

  1. 95% fail

    Without freeing disk space, the block will be re-applied by the disk threshold monitor within minutes.

  2. 80% fail

    This only postpones the issue and risks running out of disk completely, causing data corruption.