GRPC_HTTP2_PING_TIMEOUT grpc network_error ai_generated true

UNAVAILABLE: grpc: HTTP2 ping timeout after 10s, closing connection

ID: grpc/http2-ping-timeout

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC v1.48.x active — — —
gRPC v1.56.x active — — —
gRPC v1.63.x active — — —

Root Cause

gRPC client sent HTTP/2 pings but did not receive responses from the server within the timeout period, indicating network issues or server overload.

generic

中文

gRPC 客户端发送 HTTP/2 ping 但未在超时时间内收到服务器响应，表明网络问题或服务器过载。

Official Documentation

https://grpc.io/docs/guides/keepalive/#keepalive-ping-timeout

Workarounds

  1. 85% success Increase both client and server keepalive timeout values: set `GRPC_ARG_KEEPALIVE_TIMEOUT_MS` to 20000 (20 seconds) on the client, and ensure server `GRPC_ARG_KEEPALIVE_TIMEOUT_MS` is also increased to match. Example in C++: `channel_args->SetInt(GRPC_ARG_KEEPALIVE_TIMEOUT_MS, 20000);` 
    Increase both client and server keepalive timeout values: set `GRPC_ARG_KEEPALIVE_TIMEOUT_MS` to 20000 (20 seconds) on the client, and ensure server `GRPC_ARG_KEEPALIVE_TIMEOUT_MS` is also increased to match. Example in C++: `channel_args->SetInt(GRPC_ARG_KEEPALIVE_TIMEOUT_MS, 20000);`
  2. 80% success Check network path for packet loss or high latency using tools like `mtr` or `ping`. If behind a load balancer, ensure it supports HTTP/2 ping responses and has an appropriate timeout. 
    Check network path for packet loss or high latency using tools like `mtr` or `ping`. If behind a load balancer, ensure it supports HTTP/2 ping responses and has an appropriate timeout.
  3. 75% success Reduce the keepalive ping interval (`GRPC_ARG_KEEPALIVE_TIME_MS`) to 10000 (10 seconds) to detect failures faster and retry connections sooner. 
    Reduce the keepalive ping interval (`GRPC_ARG_KEEPALIVE_TIME_MS`) to 10000 (10 seconds) to detect failures faster and retry connections sooner.

中文步骤

  1. 增加客户端和服务器的 keepalive 超时时间：设置 `GRPC_ARG_KEEPALIVE_TIMEOUT_MS` 为 20000 毫秒（20 秒），确保服务器端也同步增加。
  2. 检查网络路径是否存在丢包或高延迟（使用 mtr 或 ping）。如果在负载均衡器后，确保支持 HTTP/2 ping 响应并设置适当的超时。
  3. 减少 keepalive ping 间隔（`GRPC_ARG_KEEPALIVE_TIME_MS`）到 10000 毫秒（10 秒），更快检测故障并重新连接。

Dead Ends

Common approaches that don't work:

  1. 65% fail

    This only extends the wait for ping responses; if the network is unreliable, pings still fail, and the connection closes after a longer delay.

  2. 55% fail

    Without pings, idle connections may be dropped by middleboxes (e.g., NATs, load balancers), leading to different errors like connection reset.

  3. 60% fail

    This pings only when streams are active; if the issue is network latency, pings still time out.