elasticsearch resource_error ai_generated partial

ClusterHealthResponse: status=RED, unassigned_shards=5, reason=ALLOCATION_FAILED due to [1] shards not allocated because of rack awareness constraints

ID: elasticsearch/primary-shard-not-allocated-due-to-rack-awareness

Also available as: JSON · Markdown · 中文
78%Fix Rate
85%Confidence
1Evidence
2024-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.17.10 active — — —
8.11.0 active — — —
8.12.2 active — — —

Root Cause

Rack awareness allocation filtering prevents primary shard allocation because no node in the required rack has enough disk space or is available.

generic

中文

机架感知分配过滤阻止主分片分配，因为所需机架中没有节点具有足够的磁盘空间或可用。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-allocation-awareness

Workarounds

  1. 90% success Temporarily disable rack awareness allocation filtering: PUT _cluster/settings { "transient": { "cluster.routing.allocation.awareness.attributes": "" } } then re-enable after shards allocate 
    Temporarily disable rack awareness allocation filtering: PUT _cluster/settings { "transient": { "cluster.routing.allocation.awareness.attributes": "" } } then re-enable after shards allocate
  2. 95% success Add a node in the required rack and ensure it has sufficient disk space, then use the cluster rebalance API: POST _cluster/reroute?retry_failed=true 
    Add a node in the required rack and ensure it has sufficient disk space, then use the cluster rebalance API: POST _cluster/reroute?retry_failed=true

中文步骤

  1. 临时禁用机架感知分配过滤：PUT _cluster/settings { "transient": { "cluster.routing.allocation.awareness.attributes": "" } }，分片分配后再重新启用
  2. 在所需机架中添加一个节点并确保其有足够的磁盘空间，然后使用集群重路由API：POST _cluster/reroute?retry_failed=true

Dead Ends

Common approaches that don't work:

  1. 65% fail

    The issue is not overall disk space but the specific rack's node availability; increasing space on nodes in other racks does not satisfy the allocation filter

  2. 80% fail

    Re-creating the index with fewer shards does not address the rack awareness constraint; the new primary shards will still fail to allocate if the same rack is unavailable