INTERNAL api protocol_error ai_generated true

gRPC error: INTERNAL: HTTP/2 ping timeout

ID: api/grpc-error-internal-http2-ping-timeout

Also available as: JSON · Markdown · 中文
85%Fix Rate
86%Confidence
1Evidence
2024-05-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC Go 1.62 active — — —
gRPC Python 1.60 active — — —
Envoy 1.28 active — — —
gRPC-Web 1.50 active — — —
Istio 1.20 active — — —

Root Cause

The gRPC client or server failed to receive a PING acknowledgment within the keepalive timeout period, causing the connection to be closed.

generic

中文

gRPC 客户端或服务器未能在 keepalive 超时时间内收到 PING 确认，导致连接关闭。

Official Documentation

https://grpc.io/docs/guides/keepalive/

Workarounds

  1. 90% success Set consistent keepalive parameters on both client and server. In gRPC Go, use grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 10 * time.Second, Timeout: 5 * time.Second, PermitWithoutStream: true}). On the server, set keepalive.EnforcementPolicy{MinTime: 10 * time.Second, PermitWithoutStream: true}. 
    Set consistent keepalive parameters on both client and server. In gRPC Go, use grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 10 * time.Second, Timeout: 5 * time.Second, PermitWithoutStream: true}). On the server, set keepalive.EnforcementPolicy{MinTime: 10 * time.Second, PermitWithoutStream: true}.
  2. 85% success If using Envoy, add 'http2_protocol_options: { connection_keepalive: { interval: 10s, timeout: 5s } }' to the cluster configuration. 
    If using Envoy, add 'http2_protocol_options: { connection_keepalive: { interval: 10s, timeout: 5s } }' to the cluster configuration.
  3. 80% success Check for network intermediaries (e.g., firewalls, load balancers) that may drop HTTP/2 PING frames. Configure them to allow HTTP/2 keepalive packets. 
    Check for network intermediaries (e.g., firewalls, load balancers) that may drop HTTP/2 PING frames. Configure them to allow HTTP/2 keepalive packets.

中文步骤

  1. Set consistent keepalive parameters on both client and server. In gRPC Go, use grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 10 * time.Second, Timeout: 5 * time.Second, PermitWithoutStream: true}). On the server, set keepalive.EnforcementPolicy{MinTime: 10 * time.Second, PermitWithoutStream: true}.
  2. If using Envoy, add 'http2_protocol_options: { connection_keepalive: { interval: 10s, timeout: 5s } }' to the cluster configuration.
  3. Check for network intermediaries (e.g., firewalls, load balancers) that may drop HTTP/2 PING frames. Configure them to allow HTTP/2 keepalive packets.

Dead Ends

Common approaches that don't work:

  1. 50% fail

    Keepalive is essential for detecting dead connections; disabling it can lead to stale connections.

  2. 70% fail

    The server may have a lower keepalive threshold, causing the server to close the connection first.

  3. 90% fail

    The issue is a configuration mismatch, not a code bug.