EGOAWAY grpc network_error ai_generated partial

UNAVAILABLE: grpc: connection closed during stream: GOAWAY received from peer

ID: grpc/connection-close-during-stream

Also available as: JSON · Markdown · 中文
70%Fix Rate
84%Confidence
1Evidence
2024-04-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC v1.59.0 active — — —
gRPC v1.62.0 active — — —
gRPC v1.65.0 active — — —

Root Cause

The peer sent an HTTP/2 GOAWAY frame, indicating it is shutting down or reloading, causing all active streams to be terminated with an unavailable status.

generic

中文

对端发送了 HTTP/2 GOAWAY 帧，表示正在关闭或重新加载，导致所有活跃流以不可用状态终止。

Official Documentation

https://httpwg.org/specs/rfc7540.html#GOAWAY

Workarounds

  1. 85% success Implement client-side retry with exponential backoff and jitter: `from grpc import exponential_backoff; for attempt in range(3): try: response = stub.Call(request); break except grpc.RpcError as e: if e.code() == grpc.StatusCode.UNAVAILABLE: time.sleep(backoff(attempt))` 
    Implement client-side retry with exponential backoff and jitter: `from grpc import exponential_backoff; for attempt in range(3): try: response = stub.Call(request); break except grpc.RpcError as e: if e.code() == grpc.StatusCode.UNAVAILABLE: time.sleep(backoff(attempt))`
  2. 80% success Configure the server to drain connections gracefully before shutdown by setting a GOAWAY grace period: `server.graceful_shutdown(timeout=30)` 
    Configure the server to drain connections gracefully before shutdown by setting a GOAWAY grace period: `server.graceful_shutdown(timeout=30)`
  3. 90% success Use a load balancer or proxy that can redirect traffic to healthy backends during server restarts. 
    Use a load balancer or proxy that can redirect traffic to healthy backends during server restarts.

中文步骤

  1. Implement client-side retry with exponential backoff and jitter: `from grpc import exponential_backoff; for attempt in range(3): try: response = stub.Call(request); break except grpc.RpcError as e: if e.code() == grpc.StatusCode.UNAVAILABLE: time.sleep(backoff(attempt))`
  2. Configure the server to drain connections gracefully before shutdown by setting a GOAWAY grace period: `server.graceful_shutdown(timeout=30)`
  3. Use a load balancer or proxy that can redirect traffic to healthy backends during server restarts.

Dead Ends

Common approaches that don't work:

  1. Increasing the client's retry count to 10 or more. 60% fail

    If the server is permanently down, retries will all fail; retries only help with transient conditions.

  2. Setting the environment variable GRPC_GOAWAY_MIN_PING_INTERVAL to a very low value. 90% fail

    This controls ping frequency, not GOAWAY handling; it may trigger additional GOAWAYs due to excessive pings.

  3. Switching to a different port on the same server. 95% fail

    GOAWAY is sent for the entire server, not a single port; all ports are affected during shutdown.