grpc protocol_error ai_generated true

UNAVAILABLE: grpc: received goaway from server due to too_many_pings

ID: grpc/too-many-pings

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2023-06-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC C++ 1.50+ active — — —
gRPC Go 1.50+ active — — —
Envoy 1.25+ active — — —
Istio 1.18+ active — — —

Root Cause

Client sends pings too frequently, exceeding server's max_pings_without_data or min_time_between_pings settings, causing server to send GOAWAY and close the connection.

generic

中文

客户端发送ping的频率过高，超过了服务器的max_pings_without_data或min_time_between_pings设置，导致服务器发送GOAWAY并关闭连接。

Official Documentation

https://grpc.io/docs/guides/keepalive/

Workarounds

  1. 90% success Set client-side keepalive parameters: keepalive_time=30s, keepalive_timeout=10s, and min_time_between_pings=5s. In Go: grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 30 * time.Second, Timeout: 10 * time.Second, MinTimeBetweenPings: 5 * time.Second}) 
    Set client-side keepalive parameters: keepalive_time=30s, keepalive_timeout=10s, and min_time_between_pings=5s. In Go: grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 30 * time.Second, Timeout: 10 * time.Second, MinTimeBetweenPings: 5 * time.Second})
  2. 85% success Configure server-side max_pings_without_data to a higher value (e.g., 5) or increase min_time_between_pings (e.g., 10s). In C++ server builder: builder.AddChannelArgument(GRPC_ARG_HTTP2_MIN_RECV_PING_INTERVAL_WITHOUT_DATA_MS, 10000); 
    Configure server-side max_pings_without_data to a higher value (e.g., 5) or increase min_time_between_pings (e.g., 10s). In C++ server builder: builder.AddChannelArgument(GRPC_ARG_HTTP2_MIN_RECV_PING_INTERVAL_WITHOUT_DATA_MS, 10000);

中文步骤

  1. Set client-side keepalive parameters: keepalive_time=30s, keepalive_timeout=10s, and min_time_between_pings=5s. In Go: grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 30 * time.Second, Timeout: 10 * time.Second, MinTimeBetweenPings: 5 * time.Second})
  2. Configure server-side max_pings_without_data to a higher value (e.g., 5) or increase min_time_between_pings (e.g., 10s). In C++ server builder: builder.AddChannelArgument(GRPC_ARG_HTTP2_MIN_RECV_PING_INTERVAL_WITHOUT_DATA_MS, 10000);

Dead Ends

Common approaches that don't work:

  1. Increasing keepalive time without adjusting ping interval 40% fail

    The issue is not the keepalive time but the ping frequency; increasing keepalive time alone may not reduce ping rate if min_time_between_pings is violated.

  2. Disabling keepalive entirely 30% fail

    Disabling keepalive can cause long-lived connections to be dropped by middleboxes (e.g., NAT, firewalls) that expect periodic traffic.