shard_limit_exceeded elasticsearch resource_error ai_generated partial

ElasticsearchException: failed to create index [logs-2025.03] - this action would add [2] shards, but this cluster currently has [1000]/[1000] maximum shards open

ID: elasticsearch/shard-limit-exceeded-for-node

Also available as: JSON · Markdown · 中文
78%Fix Rate
85%Confidence
1Evidence
2024-06-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Elasticsearch 7.17.0 active — — —
Elasticsearch 8.11.0 active — — —
Elasticsearch 8.15.0 active — — —

Root Cause

The cluster has reached the maximum number of shards allowed, preventing the creation of new indices.

generic

中文

集群已达到允许的最大分片数，无法创建新索引。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/current/misc-cluster-settings.html#cluster-max-shards-per-node

Workarounds

  1. 85% success Reduce the number of shards per index by reindexing with fewer primary shards. For example, use `POST /_reindex` with a target index that has 1 primary shard instead of 5: `{"source": {"index": "logs-2025.03"}, "dest": {"index": "logs-2025.03-compact", "number_of_shards": 1}}` 
    Reduce the number of shards per index by reindexing with fewer primary shards. For example, use `POST /_reindex` with a target index that has 1 primary shard instead of 5: `{"source": {"index": "logs-2025.03"}, "dest": {"index": "logs-2025.03-compact", "number_of_shards": 1}}`
  2. 72% success Increase `cluster.max_shards_per_node` cautiously after auditing shard distribution. Run `GET _cluster/settings` to check current limit, then set via `PUT _cluster/settings` with `{"persistent": {"cluster.max_shards_per_node": 1500}}`. Ensure enough heap per node. 
    Increase `cluster.max_shards_per_node` cautiously after auditing shard distribution. Run `GET _cluster/settings` to check current limit, then set via `PUT _cluster/settings` with `{"persistent": {"cluster.max_shards_per_node": 1500}}`. Ensure enough heap per node.

中文步骤

  1. Reduce the number of shards per index by reindexing with fewer primary shards. For example, use `POST /_reindex` with a target index that has 1 primary shard instead of 5: `{"source": {"index": "logs-2025.03"}, "dest": {"index": "logs-2025.03-compact", "number_of_shards": 1}}`
  2. Increase `cluster.max_shards_per_node` cautiously after auditing shard distribution. Run `GET _cluster/settings` to check current limit, then set via `PUT _cluster/settings` with `{"persistent": {"cluster.max_shards_per_node": 1500}}`. Ensure enough heap per node.

Dead Ends

Common approaches that don't work:

  1. 70% fail

    This only delays the problem and can cause severe performance degradation because each node handles too many shards, leading to heap pressure and slow recovery.

  2. 45% fail

    Deleting indices may not free up shard slots if the cluster-level limit is still hit due to node distribution; also risks data loss.