DEADLINE_EXCEEDED api runtime_error ai_generated true

gRPC error: DEADLINE_EXCEEDED: deadline exceeded waiting for stream response

ID: api/grpc-deadline-exceeded-streaming

Also available as: JSON · Markdown · 中文
82%Fix Rate
87%Confidence
1Evidence
2024-06-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC 1.58 active — — —
gRPC-Go 1.60 active — — —
gRPC-Java 1.61 active — — —
Envoy 1.28 active — — —

Root Cause

Client-set deadline (timeout) expired before the server finished sending all stream messages, often due to slow processing or network latency on long-lived streams.

generic

中文

客户端设置的截止时间（超时）在服务器完成发送所有流消息之前到期，通常由于长时间流上的慢处理或网络延迟。

Official Documentation

https://grpc.io/docs/guides/deadlines/

Workarounds

  1. 85% success Set a per-RPC deadline that matches expected stream duration. For server-streaming RPCs, use a longer deadline (e.g., 60s) and implement keepalive pings to detect dead connections. 
    Set a per-RPC deadline that matches expected stream duration. For server-streaming RPCs, use a longer deadline (e.g., 60s) and implement keepalive pings to detect dead connections.
  2. 80% success Implement server-side streaming with backpressure: use flow control to prevent the server from sending faster than the client can consume, reducing the risk of deadline exceeded due to buffering. 
    Implement server-side streaming with backpressure: use flow control to prevent the server from sending faster than the client can consume, reducing the risk of deadline exceeded due to buffering.

中文步骤

  1. 为每个 RPC 设置与预期流持续时间匹配的截止时间。对于服务器流式 RPC，使用较长的截止时间（例如 60 秒）并实现 keepalive ping 以检测死连接。
  2. 在服务器端实现带有背压的流式传输：使用流量控制防止服务器发送速度超过客户端消费速度，减少因缓冲导致截止时间超时的风险。

Dead Ends

Common approaches that don't work:

  1. 75% fail

    Unbounded timeouts mask underlying performance issues and may cause resource leaks; the correct fix is per-stream or per-RPC tuning, not a global increase.

  2. 90% fail

    This removes timeout protection, leading to hung connections and resource exhaustion; gRPC requires deadlines for reliability.