DISK_WATERMARK_BLOCK elasticsearch resource_error ai_generated true

ClusterBlockException: index [my_index] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; disk usage exceeded threshold [95%] on node [node-1]

ID: elasticsearch/index-blocked-by-disk-threshold

Also available as: JSON · Markdown · 中文
85%Fix Rate
90%Confidence
1Evidence
2023-08-25First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Elasticsearch 7.10 active — — —
Elasticsearch 8.0 active — — —
Elasticsearch 8.9 active — — —

Root Cause

Disk usage on a node exceeded the high watermark (default 95%), causing Elasticsearch to block write operations to protect against data loss.

generic

中文

节点上的磁盘使用率超过了高水位线（默认95%），导致Elasticsearch阻止写入操作以防止数据丢失。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-usage-exceeded.html

Workarounds

  1. 90% success Free disk space by deleting old indices or moving data: `POST _reindex {"source": {"index": "old_index"}, "dest": {"index": "new_index"}}` then delete old index 
    Free disk space by deleting old indices or moving data: `POST _reindex {"source": {"index": "old_index"}, "dest": {"index": "new_index"}}` then delete old index
  2. 85% success Increase disk capacity by adding a new node or expanding storage, then rebalance shards: `PUT _cluster/settings {"transient": {"cluster.routing.rebalance.enable": "all"}}` 
    Increase disk capacity by adding a new node or expanding storage, then rebalance shards: `PUT _cluster/settings {"transient": {"cluster.routing.rebalance.enable": "all"}}`
  3. 70% success Temporarily disable the disk allocation decider to allow writes: `PUT _cluster/settings {"transient": {"cluster.routing.allocation.disk.threshold_enabled": false}}` (use with caution) 
    Temporarily disable the disk allocation decider to allow writes: `PUT _cluster/settings {"transient": {"cluster.routing.allocation.disk.threshold_enabled": false}}` (use with caution)

中文步骤

  1. 通过删除旧索引或移动数据释放磁盘空间：`POST _reindex {"source": {"index": "old_index"}, "dest": {"index": "new_index"}}` 然后删除旧索引
  2. 通过添加新节点或扩展存储增加磁盘容量，然后重新平衡分片：`PUT _cluster/settings {"transient": {"cluster.routing.rebalance.enable": "all"}}`
  3. 临时禁用磁盘分配决策器以允许写入：`PUT _cluster/settings {"transient": {"cluster.routing.allocation.disk.threshold_enabled": false}}`（谨慎使用）

Dead Ends

Common approaches that don't work:

  1. Manually removing the index block with `PUT my_index/_settings {"index.blocks.read_only_allow_delete": null}` without freeing disk space 90% fail

    The block will be reapplied by the cluster when the next disk check runs, as disk usage remains high.

  2. Increasing the disk watermark thresholds in elasticsearch.yml (e.g., `cluster.routing.allocation.disk.watermark.high: 98%`) 70% fail

    This only delays the problem and risks data loss if disk fills completely.

  3. Deleting the index entirely to free space 50% fail

    This is a destructive action that may cause data loss; it should only be a last resort.