503 kafka network_error ai_generated partial

HTTP 503 Service Unavailable: Connect REST API is temporarily unavailable due to rebalance

ID: kafka/connect-rest-api-503-service-unavailable

Also available as: JSON · Markdown · 中文
79%Fix Rate
86%Confidence
1Evidence
2024-04-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kafka 3.4.0 active — — —
Kafka 3.5.1 active — — —
Kafka 3.7.0 active — — —

Root Cause

Kafka Connect's REST API returns 503 when the cluster is undergoing a rebalance, as worker leadership may be reassigned and endpoints are temporarily disabled.

generic

中文

当集群正在进行重平衡时，Kafka Connect的REST API返回503，因为工作节点领导权可能正在重新分配，端点被暂时禁用。

Official Documentation

https://kafka.apache.org/documentation/#connect_rest

Workarounds

  1. 93% success Implement retry logic with exponential backoff in your client. Example in Python: import time; for i in range(5): try: response = requests.get('http://connect-host:8083/connectors'); break; except requests.exceptions.HTTPError as e: if e.response.status_code == 503: time.sleep(2**i); else: raise 
    Implement retry logic with exponential backoff in your client. Example in Python: import time; for i in range(5): try: response = requests.get('http://connect-host:8083/connectors'); break; except requests.exceptions.HTTPError as e: if e.response.status_code == 503: time.sleep(2**i); else: raise
  2. 85% success Monitor the rebalance status using Kafka's metrics or logs, and only query the REST API after the rebalance completes. Check for 'REBALANCE_COMPLETED' log entries. 
    Monitor the rebalance status using Kafka's metrics or logs, and only query the REST API after the rebalance completes. Check for 'REBALANCE_COMPLETED' log entries.

中文步骤

  1. Implement retry logic with exponential backoff in your client. Example in Python: import time; for i in range(5): try: response = requests.get('http://connect-host:8083/connectors'); break; except requests.exceptions.HTTPError as e: if e.response.status_code == 503: time.sleep(2**i); else: raise
  2. Monitor the rebalance status using Kafka's metrics or logs, and only query the REST API after the rebalance completes. Check for 'REBALANCE_COMPLETED' log entries.

Dead Ends

Common approaches that don't work:

  1. Setting 'rest.advertised.host.name' to a different value 90% fail

    Increasing worker timeout does not prevent the 503 during rebalance; the API is intentionally disabled.

  2. Restarting a single Connect worker 85% fail

    Restarting one worker may trigger another rebalance, making the 503 persist longer.