kafka runtime_error ai_generated true

org.apache.kafka.connect.errors.ConnectException: Exceeded max number of retries. Topic 'my_topic' partition 0, offset 12345: Elasticsearch cluster unavailable

ID: kafka/elasticsearch-sink-connector-backoff

Also available as: JSON · Markdown · 中文
80%Fix Rate
87%Confidence
1Evidence
2023-11-05First Seen

Root Cause

Kafka Connect Elasticsearch sink connector exhausted retries due to Elasticsearch cluster being unreachable or overloaded, causing task failure.

generic

中文

Kafka Connect Elasticsearch sink连接器因Elasticsearch集群不可达或过载而耗尽重试次数，导致任务失败。

Official Documentation

https://docs.confluent.io/kafka-connect-elasticsearch/current/index.html

Workarounds

  1. 80% success Check Elasticsearch cluster health and restart it if necessary. Use `curl -X GET 'localhost:9200/_cluster/health?pretty'` to verify status. 
    Check Elasticsearch cluster health and restart it if necessary. Use `curl -X GET 'localhost:9200/_cluster/health?pretty'` to verify status.
  2. 75% success Increase 'retry.backoff.ms' in connector config to a higher value (e.g., 10000 ms) and ensure 'max.retries' is reasonable (e.g., 10). This gives Elasticsearch more time to recover. 
    Increase 'retry.backoff.ms' in connector config to a higher value (e.g., 10000 ms) and ensure 'max.retries' is reasonable (e.g., 10). This gives Elasticsearch more time to recover.
  3. 90% success If Elasticsearch is temporarily down, pause the connector using `curl -X PUT 'http://localhost:8083/connectors/my-connector/pause'` and resume after restoring Elasticsearch. 
    If Elasticsearch is temporarily down, pause the connector using `curl -X PUT 'http://localhost:8083/connectors/my-connector/pause'` and resume after restoring Elasticsearch.

中文步骤

  1. Check Elasticsearch cluster health and restart it if necessary. Use `curl -X GET 'localhost:9200/_cluster/health?pretty'` to verify status.
  2. Increase 'retry.backoff.ms' in connector config to a higher value (e.g., 10000 ms) and ensure 'max.retries' is reasonable (e.g., 10). This gives Elasticsearch more time to recover.
  3. If Elasticsearch is temporarily down, pause the connector using `curl -X PUT 'http://localhost:8083/connectors/my-connector/pause'` and resume after restoring Elasticsearch.

Dead Ends

Common approaches that don't work:

  1. 60% fail

    This only delays failure; if Elasticsearch is truly down, retries will eventually exhaust and the connector will still fail, potentially filling logs.

  2. 90% fail

    The underlying Elasticsearch connectivity issue persists, so the new connector will encounter the same error.

  3. 70% fail

    The issue is not load but connectivity; reducing tasks doesn't fix the Elasticsearch unavailability.