DEADLINE_EXCEEDED api runtime_error ai_generated partial

gRPC error: DEADLINE_EXCEEDED — server too busy to respond within deadline

ID: api/grpc-deadline-exceeded-server-busy

Also available as: JSON · Markdown · 中文
85%Fix Rate
86%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC 1.60+ active — — —
gRPC-Go 1.60+ active — — —
gRPC-Python 1.60+ active — — —

Root Cause

The gRPC server failed to process the request within the client-specified deadline due to high load, resource contention, or a slow downstream dependency.

generic

中文

gRPC 服务器由于高负载、资源争用或下游依赖缓慢，未能在客户端指定的截止时间内处理请求。

Official Documentation

https://grpc.io/docs/guides/error/

Workarounds

  1. 85% success Implement client-side retries with exponential backoff and jitter. Example gRPC-Go: `grpc.WithDefaultCallOptions(grpc.MaxRetryAttempts(3), grpc.WithBackoff(grpc.DefaultBackoffConfig))`. 
    Implement client-side retries with exponential backoff and jitter. Example gRPC-Go: `grpc.WithDefaultCallOptions(grpc.MaxRetryAttempts(3), grpc.WithBackoff(grpc.DefaultBackoffConfig))`.
  2. 90% success Scale up the gRPC server by adding more instances or increasing resources (CPU/memory). Monitor server metrics (e.g., `grpc_server_requests_in_flight`). 
    Scale up the gRPC server by adding more instances or increasing resources (CPU/memory). Monitor server metrics (e.g., `grpc_server_requests_in_flight`).
  3. 80% success Reduce the request complexity or implement server-side rate limiting to prevent overload. Example: use `grpc.MaxConcurrentStreams` in server config. 
    Reduce the request complexity or implement server-side rate limiting to prevent overload. Example: use `grpc.MaxConcurrentStreams` in server config.

中文步骤

  1. 实现客户端重试，使用指数退避和抖动。gRPC-Go 示例：`grpc.WithDefaultCallOptions(grpc.MaxRetryAttempts(3), grpc.WithBackoff(grpc.DefaultBackoffConfig))`。
  2. 通过添加更多实例或增加资源（CPU/内存）来扩展 gRPC 服务器。监控服务器指标（例如 `grpc_server_requests_in_flight`）。
  3. 降低请求复杂度或实现服务器端速率限制以防止过载。示例：在服务器配置中使用 `grpc.MaxConcurrentStreams`。

Dead Ends

Common approaches that don't work:

  1. 80% fail

    If the server is overloaded, a longer deadline just delays the timeout; the request may still fail or degrade other requests.

  2. 90% fail

    The server is still busy; immediate retries worsen load and likely fail again.

  3. 60% fail

    The error is a server-side timeout, not a connection failure; network is likely fine.