networking grpc ai_generated true

rpc error: code = DeadlineExceeded desc = context deadline exceeded

ID: networking/grpc-deadline-exceeded

Also available as: JSON · Markdown
85%Fix Rate
88%Confidence
60Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

A gRPC call did not complete within its configured deadline. The server may be slow, overloaded, or the network path may have high latency. The client cancels the call and returns DeadlineExceeded (gRPC status code 4).

generic

Workarounds

  1. 88% success Profile the server-side handler to identify and optimize slow operations 
    1. Add server-side interceptors to log handler duration
2. Use distributed tracing (OpenTelemetry, Jaeger) to identify slow spans
3. Profile with pprof or equivalent: go tool pprof http://localhost:6060/debug/pprof/profile
4. Optimize slow database queries, external API calls, or computation
5. Consider caching frequently requested data
6. Set appropriate deadlines that account for the 95th percentile latency
  2. 85% success Implement proper deadline propagation and retry with exponential backoff 
    1. Propagate deadlines through the call chain: ctx, cancel := context.WithTimeout(ctx, remaining)
2. Use gRPC retry interceptors with exponential backoff
3. Configure max retry attempts (3-5 typically sufficient)
4. Add jitter to avoid thundering herd: backoff + random(0, backoff*0.1)
5. Use circuit breakers to stop calling consistently failing services
6. Monitor deadline propagation with tracing to ensure deadlines shrink correctly through the call chain

Dead Ends

Common approaches that don't work:

  1. Set very large deadlines or remove deadlines entirely 80% fail

    Removing deadlines means requests can hang indefinitely, consuming server resources and client goroutines/threads. In cascading microservice calls, missing deadlines cause resource exhaustion and cascading failures.

  2. Add aggressive retry logic without backoff or deadline propagation 75% fail

    Retrying immediately without backoff amplifies load on an already overloaded server. Without propagating the remaining deadline, each retry uses the full original deadline, potentially exceeding the caller's deadline.

Error Chain

Leads to:
Preceded by:
Frequently confused with: