grpc network_error ai_generated true

CANCELLED: grpc: stream closed by server shutdown

ID: grpc/stream-interrupted-by-shutdown

Also available as: JSON · Markdown · 中文
80%Fix Rate
82%Confidence
1Evidence
2023-09-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC Python 1.46.0 active — — —
gRPC Go 1.50.0 active — — —

Root Cause

The server is gracefully shutting down, closing all active streams.

generic

中文

服务器正在正常关闭，导致所有活跃流被关闭。

Official Documentation

https://grpc.io/docs/guides/error-handling/

Workarounds

  1. 80% success Implement client-side reconnection logic with exponential backoff. Example in Python: `channel = grpc.insecure_channel('target', options=[('grpc.service_config', '{"methodConfig": [{"name": [{}], "retryPolicy": {"maxAttempts": 5, "initialBackoff": "0.1s", "maxBackoff": "1s", "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"]}}]}')])` 
    Implement client-side reconnection logic with exponential backoff. Example in Python: `channel = grpc.insecure_channel('target', options=[('grpc.service_config', '{"methodConfig": [{"name": [{}], "retryPolicy": {"maxAttempts": 5, "initialBackoff": "0.1s", "maxBackoff": "1s", "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"]}}]}')])`
  2. 90% success Use a load balancer to route requests to healthy server instances. 
    Use a load balancer to route requests to healthy server instances.

中文步骤

  1. Implement client-side reconnection logic with exponential backoff. Example in Python: `channel = grpc.insecure_channel('target', options=[('grpc.service_config', '{"methodConfig": [{"name": [{}], "retryPolicy": {"maxAttempts": 5, "initialBackoff": "0.1s", "maxBackoff": "1s", "backoffMultiplier": 2, "retryableStatusCodes": ["UNAVAILABLE"]}}]}')])`
  2. Use a load balancer to route requests to healthy server instances.

Dead Ends

Common approaches that don't work:

  1. 80% fail

    The server is down; retries will fail until the server is back.

  2. 60% fail

    Keepalive settings do not prevent server shutdown from closing streams.

  3. 95% fail

    The stream is permanently closed; a new stream must be created.