elasticsearch resource_error ai_generated true

ElasticsearchException: failed to create index [my_index] - limit of total shards [1000] for cluster [my_cluster] exceeded

ID: elasticsearch/shards-limit-exceeded

Also available as: JSON · Markdown · 中文
82%Fix Rate
85%Confidence
1Evidence
2023-05-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.10.0 active — — —
7.17.0 active — — —
8.0.0 active — — —
8.5.0 active — — —

Root Cause

The cluster has reached its maximum allowed number of shards (default 1000 per node or cluster-level limit), preventing index creation or opening.

generic

中文

集群已达到允许的最大分片数量（默认每节点或集群级别限制为1000），阻止索引创建或打开。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/current/misc-cluster-settings.html#cluster-shard-limit

Workarounds

  1. 85% success Increase the cluster-level shard limit dynamically: PUT _cluster/settings { "persistent": { "cluster.max_shards_per_node": 2000 } } 
    Increase the cluster-level shard limit dynamically: PUT _cluster/settings { "persistent": { "cluster.max_shards_per_node": 2000 } }
  2. 80% success Reduce shard count by merging small indices or deleting unused indices: POST _cat/indices?v | awk '{print $3}' | xargs -I {} curl -XDELETE 'http://localhost:9200/{}' 
    Reduce shard count by merging small indices or deleting unused indices: POST _cat/indices?v | awk '{print $3}' | xargs -I {} curl -XDELETE 'http://localhost:9200/{}'
  3. 75% success Set a per-index shard limit to prevent future overflow: PUT _cluster/settings { "persistent": { "cluster.max_shards_per_node.frozen": 500 } } 
    Set a per-index shard limit to prevent future overflow: PUT _cluster/settings { "persistent": { "cluster.max_shards_per_node.frozen": 500 } }

中文步骤

  1. Increase the cluster-level shard limit dynamically: PUT _cluster/settings { "persistent": { "cluster.max_shards_per_node": 2000 } }
  2. Reduce shard count by merging small indices or deleting unused indices: POST _cat/indices?v | awk '{print $3}' | xargs -I {} curl -XDELETE 'http://localhost:9200/{}'
  3. Set a per-index shard limit to prevent future overflow: PUT _cluster/settings { "persistent": { "cluster.max_shards_per_node.frozen": 500 } }

Dead Ends

Common approaches that don't work:

  1. 95% fail

    Shard counts are persisted in cluster state and metadata, not reset on restart; this only causes temporary downtime without fixing the limit.

  2. 70% fail

    The limit applies to the entire cluster, not just one index. Deleting a single index does not free enough shards if the cluster is at capacity.

  3. 90% fail

    Adding replicas increases total shard count, making the problem worse. The limit caps total shards including replicas.