kafka runtime_error ai_generated true

org.apache.kafka.connect.errors.ConnectException：超过最大重试次数。主题'my_topic'分区0，偏移量12345：Elasticsearch集群不可用

org.apache.kafka.connect.errors.ConnectException: Exceeded max number of retries. Topic 'my_topic' partition 0, offset 12345: Elasticsearch cluster unavailable

ID: kafka/elasticsearch-sink-connector-backoff

其他格式: JSON · Markdown 中文 · English
80%修复率
87%置信度
1证据数
2023-11-05首次发现

根因分析

Kafka Connect Elasticsearch sink连接器因Elasticsearch集群不可达或过载而耗尽重试次数，导致任务失败。

English

Kafka Connect Elasticsearch sink connector exhausted retries due to Elasticsearch cluster being unreachable or overloaded, causing task failure.

generic

官方文档

https://docs.confluent.io/kafka-connect-elasticsearch/current/index.html

解决方案

  1. Check Elasticsearch cluster health and restart it if necessary. Use `curl -X GET 'localhost:9200/_cluster/health?pretty'` to verify status.
  2. Increase 'retry.backoff.ms' in connector config to a higher value (e.g., 10000 ms) and ensure 'max.retries' is reasonable (e.g., 10). This gives Elasticsearch more time to recover.
  3. If Elasticsearch is temporarily down, pause the connector using `curl -X PUT 'http://localhost:8083/connectors/my-connector/pause'` and resume after restoring Elasticsearch.

无效尝试

常见但无效的做法:

  1. 60% 失败

    This only delays failure; if Elasticsearch is truly down, retries will eventually exhaust and the connector will still fail, potentially filling logs.

  2. 90% 失败

    The underlying Elasticsearch connectivity issue persists, so the new connector will encounter the same error.

  3. 70% 失败

    The issue is not load but connectivity; reducing tasks doesn't fix the Elasticsearch unavailability.