RESOURCE_EXHAUSTED communication runtime_error ai_generated true

gRPC client stream failed with RESOURCE_EXHAUSTED: Too many outstanding requests

ID: communication/grpc-client-stream-failed-with-resource-exhausted

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-02-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC 1.54.0 active — — —
gRPC 1.60.0 active — — —
gRPC 1.62.0 active — — —

Root Cause

gRPC client-side flow control or concurrency limit exceeded, causing the channel to reject new streams.

generic

中文

gRPC 客户端侧流控制或并发限制超限，导致通道拒绝新流。

Official Documentation

https://grpc.io/docs/guides/error-codes/

Workarounds

  1. 85% success Configure gRPC client channel with increased max_concurrent_streams and flow control window: In C++, set `ChannelArguments` with `GRPC_ARG_MAX_CONCURRENT_STREAMS` to 1000 and `GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE` to 64KB. In Python, use `grpc.insecure_channel(target, options=[('grpc.max_concurrent_streams', 1000)])`. 
    Configure gRPC client channel with increased max_concurrent_streams and flow control window: In C++, set `ChannelArguments` with `GRPC_ARG_MAX_CONCURRENT_STREAMS` to 1000 and `GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE` to 64KB. In Python, use `grpc.insecure_channel(target, options=[('grpc.max_concurrent_streams', 1000)])`.
  2. 90% success Implement client-side rate limiting with a semaphore to limit in-flight requests: Use `asyncio.Semaphore(100)` in Python or `std::counting_semaphore` in C++ to throttle before sending. 
    Implement client-side rate limiting with a semaphore to limit in-flight requests: Use `asyncio.Semaphore(100)` in Python or `std::counting_semaphore` in C++ to throttle before sending.
  3. 75% success Enable gRPC client-side keepalive with aggressive timeout to detect dead streams: Set `grpc.keepalive_time_ms` to 10000 and `grpc.keepalive_timeout_ms` to 5000. 
    Enable gRPC client-side keepalive with aggressive timeout to detect dead streams: Set `grpc.keepalive_time_ms` to 10000 and `grpc.keepalive_timeout_ms` to 5000.

中文步骤

  1. Configure gRPC client channel with increased max_concurrent_streams and flow control window: In C++, set `ChannelArguments` with `GRPC_ARG_MAX_CONCURRENT_STREAMS` to 1000 and `GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE` to 64KB. In Python, use `grpc.insecure_channel(target, options=[('grpc.max_concurrent_streams', 1000)])`.
  2. Implement client-side rate limiting with a semaphore to limit in-flight requests: Use `asyncio.Semaphore(100)` in Python or `std::counting_semaphore` in C++ to throttle before sending.
  3. Enable gRPC client-side keepalive with aggressive timeout to detect dead streams: Set `grpc.keepalive_time_ms` to 10000 and `grpc.keepalive_timeout_ms` to 5000.

Dead Ends

Common approaches that don't work:

  1. Increase the gRPC max concurrent streams on the server side only 80% fail

    The issue is client-side concurrency limits; server-side changes don't help.

  2. Restart the client process to reset channel state 90% fail

    Restarting only temporarily clears the queue; the same condition recurs under load.

  3. Ignore the error and retry indefinitely with backoff 70% fail

    Without addressing the concurrency limit, retries will keep hitting the same error.