grpc runtime_error ai_generated true

DEADLINE_EXCEEDED: grpc: deadline exceeded after 5000ms while waiting for server to respond

ID: grpc/grpc-deadline-exceeded-server-busy

Also available as: JSON · Markdown · 中文
82%Fix Rate
86%Confidence
1Evidence
2024-03-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC v1.44.0 active — — —
gRPC v1.51.0 active — — —
gRPC v1.59.0 active — — —

Root Cause

The server took longer than the client's deadline to respond, often due to server overload or slow processing.

generic

中文

服务器响应时间超过客户端截止时间，通常是由于服务器过载或处理缓慢。

Official Documentation

https://grpc.io/docs/guides/deadlines/

Workarounds

  1. 85% success Profile the server to identify slow operations (e.g., database queries, external API calls). Use tools like pprof or tracing. Example in Go: go tool pprof http://localhost:6060/debug/pprof/profile 
    Profile the server to identify slow operations (e.g., database queries, external API calls). Use tools like pprof or tracing. Example in Go: go tool pprof http://localhost:6060/debug/pprof/profile
  2. 75% success Implement server-side rate limiting or request queuing to handle load spikes. Use a circuit breaker pattern to reject requests when the server is overloaded. 
    Implement server-side rate limiting or request queuing to handle load spikes. Use a circuit breaker pattern to reject requests when the server is overloaded.
  3. 80% success Increase the client deadline to a reasonable value (e.g., 10 seconds) and add retry with exponential backoff. Example in Python with grpc.aio: await asyncio.wait_for(stub.MyRpc(request), timeout=10) 
    Increase the client deadline to a reasonable value (e.g., 10 seconds) and add retry with exponential backoff. Example in Python with grpc.aio: await asyncio.wait_for(stub.MyRpc(request), timeout=10)

中文步骤

  1. Profile the server to identify slow operations (e.g., database queries, external API calls). Use tools like pprof or tracing. Example in Go: go tool pprof http://localhost:6060/debug/pprof/profile
  2. Implement server-side rate limiting or request queuing to handle load spikes. Use a circuit breaker pattern to reject requests when the server is overloaded.
  3. Increase the client deadline to a reasonable value (e.g., 10 seconds) and add retry with exponential backoff. Example in Python with grpc.aio: await asyncio.wait_for(stub.MyRpc(request), timeout=10)

Dead Ends

Common approaches that don't work:

  1. 60% fail

    This masks the problem but doesn't fix server performance; long deadlines can cause resource leaks.

  2. 75% fail

    The underlying performance issue (e.g., slow database queries) will reappear after restart.

  3. 80% fail

    Repeated retries can overwhelm the server further, worsening the problem.