grpc protocol_error ai_generated true

UNAVAILABLE: grpc: received goaway from server due to max_pings_without_data

ID: grpc/grpc-max-pings-without-data

Also available as: JSON · Markdown · 中文
78%Fix Rate
84%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC v1.38.0 active — — —
gRPC v1.49.0 active — — —
gRPC v1.58.0 active — — —

Root Cause

The server sent a GOAWAY frame because the client sent too many keepalive pings without any data flow.

generic

中文

服务器发送了 GOAWAY 帧，因为客户端在没有数据流的情况下发送了过多保活 ping。

Official Documentation

https://grpc.io/docs/guides/keepalive/#keepalive-configuration

Workarounds

  1. 80% success Adjust client keepalive settings: reduce grpc.keepalive_time_ms to a higher value (e.g., 300000 ms) and increase grpc.keepalive_timeout_ms. Example in C++: channel_args.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, 300000); 
    Adjust client keepalive settings: reduce grpc.keepalive_time_ms to a higher value (e.g., 300000 ms) and increase grpc.keepalive_timeout_ms. Example in C++: channel_args.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, 300000);
  2. 75% success On the server, set the maximum allowed pings without data using environment variable GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA (e.g., 0 to disable the limit) or via channel args. 
    On the server, set the maximum allowed pings without data using environment variable GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA (e.g., 0 to disable the limit) or via channel args.
  3. 70% success Ensure client sends data frames periodically (e.g., by sending health check RPCs) to avoid triggering the ping-only limit. 
    Ensure client sends data frames periodically (e.g., by sending health check RPCs) to avoid triggering the ping-only limit.

中文步骤

  1. Adjust client keepalive settings: reduce grpc.keepalive_time_ms to a higher value (e.g., 300000 ms) and increase grpc.keepalive_timeout_ms. Example in C++: channel_args.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, 300000);
  2. On the server, set the maximum allowed pings without data using environment variable GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA (e.g., 0 to disable the limit) or via channel args.
  3. Ensure client sends data frames periodically (e.g., by sending health check RPCs) to avoid triggering the ping-only limit.

Dead Ends

Common approaches that don't work:

  1. 60% fail

    This may lead to undetected dead connections, causing other errors like 'stream removed'.

  2. 50% fail

    This can lead to resource exhaustion on the server if many clients ping aggressively.

  3. 85% fail

    The client will keep sending pings and get GOAWAY repeatedly, causing constant disconnections.