8 communication resource_error ai_generated true

grpc::RESOURCE_EXHAUSTED: gRPC call failed with status code 8

ID: communication/grpc-resource-exhausted

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2023-11-05First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC 1.62 active — — —
gRPC-go v1.65 active — — —
gRPC-java 1.64 active — — —
envoy 1.30 active — — —

Root Cause

gRPC server or client exhausts memory, file descriptors, or concurrent stream limits, often due to stream leakage or insufficient resource quotas in the transport layer.

generic

中文

gRPC 服务器或客户端耗尽了内存、文件描述符或并发流限制，通常是由于流泄漏或传输层资源配额不足导致。

Official Documentation

https://grpc.github.io/grpc/core/md_doc_statuscodes.html

Workarounds

  1. 85% success Increase the gRPC server's maximum concurrent streams: set `GRPC_ARG_MAX_CONCURRENT_STREAMS` to a higher value (e.g., 1000) and restart the server. Also monitor `grpc.server.streams` metric. 
    Increase the gRPC server's maximum concurrent streams: set `GRPC_ARG_MAX_CONCURRENT_STREAMS` to a higher value (e.g., 1000) and restart the server. Also monitor `grpc.server.streams` metric.
  2. 80% success Add client-side connection pooling: reuse gRPC channels and limit the number of concurrent calls per channel using a semaphore or rate limiter. 
    Add client-side connection pooling: reuse gRPC channels and limit the number of concurrent calls per channel using a semaphore or rate limiter.
  3. 88% success Configure Envoy proxy to set `max_requests_per_connection` and `circuit_breakers` limits to prevent client-side resource exhaustion from overwhelming the gRPC server. 
    Configure Envoy proxy to set `max_requests_per_connection` and `circuit_breakers` limits to prevent client-side resource exhaustion from overwhelming the gRPC server.

中文步骤

  1. Increase the gRPC server's maximum concurrent streams: set `GRPC_ARG_MAX_CONCURRENT_STREAMS` to a higher value (e.g., 1000) and restart the server. Also monitor `grpc.server.streams` metric.
  2. Add client-side connection pooling: reuse gRPC channels and limit the number of concurrent calls per channel using a semaphore or rate limiter.
  3. Configure Envoy proxy to set `max_requests_per_connection` and `circuit_breakers` limits to prevent client-side resource exhaustion from overwhelming the gRPC server.

Dead Ends

Common approaches that don't work:

  1. Increase server memory and restart 70% fail

    The issue is often not raw memory but stream limit exhaustion (e.g., MAX_CONCURRENT_STREAMS). More memory doesn't increase stream quotas.

  2. Set grpc.max_receive_message_size to a very high value (e.g., 500MB) 90% fail

    This error is about connection-level resources, not message size. Changing message size doesn't address stream or FD exhaustion.

  3. Disable keepalive pings in gRPC client 80% fail

    Keepalive pings help detect dead connections and actually reduce resource leaks. Disabling them makes the problem worse.