kafka network_error ai_generated true

java.io.IOException: Connection reset by peer (Socket write error)

ID: kafka/network-exception-connection-reset

Also available as: JSON · Markdown · 中文
78%Fix Rate
83%Confidence
1Evidence
2023-06-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2.8.0 active — — —
3.2.0 active — — —
3.5.0 active — — —

Root Cause

Kafka broker forcefully closed TCP connection due to request timeout, network instability, or broker overload causing socket write failure.

generic

中文

Kafka代理因请求超时、网络不稳定或代理过载而强制关闭TCP连接，导致套接字写入失败。

Official Documentation

https://kafka.apache.org/documentation/#network

Workarounds

  1. 80% success Increase broker's 'queued.max.requests' to 500 to handle burst traffic and reduce connection resets. 
    Increase broker's 'queued.max.requests' to 500 to handle burst traffic and reduce connection resets.
  2. 75% success Enable 'connections.max.idle.ms' with a lower value (e.g., 60000) to clean idle connections and reduce resource contention. 
    Enable 'connections.max.idle.ms' with a lower value (e.g., 60000) to clean idle connections and reduce resource contention.
  3. 85% success Add retry logic in producer with exponential backoff to handle transient connection resets. 
    Add retry logic in producer with exponential backoff to handle transient connection resets.

中文步骤

  1. 增加代理的 'queued.max.requests' 到500以处理突发流量并减少连接重置。
  2. 启用 'connections.max.idle.ms' 并设置为较低值（例如60000），以清理空闲连接并减少资源争用。
  3. 在生产者的重试逻辑中添加指数退避，以处理瞬态连接重置。

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Socket timeout affects idle connections, not active writes; connection reset still occurs on overload.

  2. 85% fail

    Firewall may cause resets, but disabling it is not a production fix; root cause is broker-side overload.

  3. 60% fail

    Backoff delays reconnection but does not prevent reset during active write; error occurs before backoff.