kafka network_error ai_generated true

org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. Timeout: 30000 ms

ID: kafka/network-exception-read-timeout

Also available as: JSON · Markdown · 中文
82%Fix Rate
88%Confidence
1Evidence
2023-06-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kafka_2.13-3.3.1 active — — —
kafka_2.13-3.4.0 active — — —
kafka_2.13-3.5.0 active — — —

Root Cause

Kafka broker closed the TCP connection before sending a response, often due to request timeout or network congestion between client and broker.

generic

中文

Kafka 代理在发送响应前关闭了 TCP 连接，通常是由于请求超时或客户端与代理之间的网络拥塞。

Official Documentation

https://kafka.apache.org/documentation/#networking

Workarounds

  1. 80% success Increase `broker.request.timeout.ms` in `server.properties` (e.g., from 30000 to 60000) and restart the broker to allow longer processing time for slow requests. 
    Increase `broker.request.timeout.ms` in `server.properties` (e.g., from 30000 to 60000) and restart the broker to allow longer processing time for slow requests.
  2. 85% success Enable TCP keepalive on both client and broker: set `socket.connection.setup.timeout.ms=10000` and `socket.connection.setup.timeout.max.ms=30000` in `server.properties`, and on the client set `connections.max.idle.ms=300000`. 
    Enable TCP keepalive on both client and broker: set `socket.connection.setup.timeout.ms=10000` and `socket.connection.setup.timeout.max.ms=30000` in `server.properties`, and on the client set `connections.max.idle.ms=300000`.
  3. 90% success If using a load balancer, check idle timeout settings (e.g., AWS NLB idle timeout) and increase them to match Kafka's `connections.max.idle.ms`. 
    If using a load balancer, check idle timeout settings (e.g., AWS NLB idle timeout) and increase them to match Kafka's `connections.max.idle.ms`.

中文步骤

  1. Increase `broker.request.timeout.ms` in `server.properties` (e.g., from 30000 to 60000) and restart the broker to allow longer processing time for slow requests.
  2. Enable TCP keepalive on both client and broker: set `socket.connection.setup.timeout.ms=10000` and `socket.connection.setup.timeout.max.ms=30000` in `server.properties`, and on the client set `connections.max.idle.ms=300000`.
  3. If using a load balancer, check idle timeout settings (e.g., AWS NLB idle timeout) and increase them to match Kafka's `connections.max.idle.ms`.

Dead Ends

Common approaches that don't work:

  1. 85% fail

    The timeout is on the broker side; increasing client timeout does not prevent the broker from disconnecting.

  2. 90% fail

    Network issues (e.g., firewall, TCP keepalive) persist after reboot; the error recurs on the next request.

  3. 70% fail

    Disabling socket timeout can cause resource exhaustion and does not address the root cause of disconnection.