elasticsearch runtime_error ai_generated true

ElasticsearchTimeoutException: task [id:12345] cancelled with reason [timeout] after [30000ms] while waiting for completion

ID: elasticsearch/task-cancellation-timeout

Also available as: JSON · Markdown · 中文
79%Fix Rate
83%Confidence
1Evidence
2024-01-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
7.15.0 active — — —
7.17.15 active — — —
8.6.0 active — — —
8.10.0 active — — —

Root Cause

A long-running task (e.g., reindex, force merge, snapshot) exceeded the configured timeout or cancellation threshold, leading to premature termination.

generic

中文

长时间运行的任务（如重新索引、强制合并、快照）超过了配置的超时或取消阈值，导致提前终止。

Official Documentation

https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#task-cancellation

Workarounds

  1. 78% success Increase the task timeout for the specific operation: POST _reindex?wait_for_completion=false&timeout=10m { "source": { "index": "old" }, "dest": { "index": "new" } } 
    Increase the task timeout for the specific operation: POST _reindex?wait_for_completion=false&timeout=10m { "source": { "index": "old" }, "dest": { "index": "new" } }
  2. 82% success Check and update cluster-level task cancellation settings: PUT _cluster/settings { "persistent": { "task.max_cancellation_timeout": "120s" } } 
    Check and update cluster-level task cancellation settings: PUT _cluster/settings { "persistent": { "task.max_cancellation_timeout": "120s" } }
  3. 75% success Retry the task with a smaller batch size or fewer shards to reduce execution time: POST _reindex { "source": { "index": "old", "size": 500 }, "dest": { "index": "new" } } 
    Retry the task with a smaller batch size or fewer shards to reduce execution time: POST _reindex { "source": { "index": "old", "size": 500 }, "dest": { "index": "new" } }

中文步骤

  1. Increase the task timeout for the specific operation: POST _reindex?wait_for_completion=false&timeout=10m { "source": { "index": "old" }, "dest": { "index": "new" } }
  2. Check and update cluster-level task cancellation settings: PUT _cluster/settings { "persistent": { "task.max_cancellation_timeout": "120s" } }
  3. Retry the task with a smaller batch size or fewer shards to reduce execution time: POST _reindex { "source": { "index": "old", "size": 500 }, "dest": { "index": "new" } }

Dead Ends

Common approaches that don't work:

  1. 60% fail

    The error may be due to cluster-wide task cancellation settings (e.g., `task.max_cancellation_timeout`), not just the request timeout. Overriding locally may be ignored.

  2. 85% fail

    Task cancellation is recorded in cluster state, and restarting a single node does not reset the task manager on other nodes. The task will still be cancelled.

  3. 40% fail

    This is a security risk and may cause resource leaks. Also, it requires a node restart, which may not be feasible in production.