grpc runtime_error ai_generated true

DEADLINE_EXCEEDED: client idle timeout exceeded

ID: grpc/deadline-exceeded-client-idle-timeout

Also available as: JSON · Markdown · 中文
88%Fix Rate
88%Confidence
1Evidence
2024-03-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC v1.60.0 active — — —
gRPC v1.65.0 active — — —
gRPC-Java v1.62.0 active — — —

Root Cause

gRPC client channel idle timeout (default 30 minutes) expired before any RPC was made, causing the channel to close and pending calls to fail.

generic

中文

gRPC客户端通道空闲超时（默认30分钟）在发起任何RPC之前到期，导致通道关闭，挂起的调用失败。

Official Documentation

https://grpc.io/docs/guides/keepalive/

Workarounds

  1. 95% success Disable idle timeout: set channel argument 'grpc.idle_timeout=0' (Python: 'grpc.insecure_channel(target, options=[("grpc.idle_timeout", 0)])') to keep the channel open indefinitely. 
    Disable idle timeout: set channel argument 'grpc.idle_timeout=0' (Python: 'grpc.insecure_channel(target, options=[("grpc.idle_timeout", 0)])') to keep the channel open indefinitely.
  2. 85% success Enable keepalive pings to prevent idle timeout: configure 'grpc.keepalive_time_ms=10000', 'grpc.keepalive_timeout_ms=5000', and 'grpc.keepalive_permit_without_calls=1' to maintain channel activity. 
    Enable keepalive pings to prevent idle timeout: configure 'grpc.keepalive_time_ms=10000', 'grpc.keepalive_timeout_ms=5000', and 'grpc.keepalive_permit_without_calls=1' to maintain channel activity.
  3. 80% success Schedule a periodic health check RPC (e.g., every 20 minutes) to reset the idle timer; use a lightweight 'HealthCheck' service call to keep the channel alive. 
    Schedule a periodic health check RPC (e.g., every 20 minutes) to reset the idle timer; use a lightweight 'HealthCheck' service call to keep the channel alive.

中文步骤

  1. 禁用空闲超时：设置通道参数'grpc.idle_timeout=0'（Python: 'grpc.insecure_channel(target, options=[("grpc.idle_timeout", 0)])'）以保持通道无限期打开。
  2. 启用保活ping以防止空闲超时：配置'grpc.keepalive_time_ms=10000'、'grpc.keepalive_timeout_ms=5000'和'grpc.keepalive_permit_without_calls=1'以维持通道活动。
  3. 安排定期健康检查RPC（如每20分钟一次）以重置空闲计时器；使用轻量级的'HealthCheck'服务调用保持通道活跃。

Dead Ends

Common approaches that don't work:

  1. 95% fail

    Reducing the RPC deadline (e.g., from 10s to 5s) doesn't fix the idle timeout; the channel still closes after the idle period.

  2. 80% fail

    Increasing client keepalive time (e.g., 'grpc.keepalive_time=10s') without enabling keepalive ping prevents timeout because keepalive is disabled by default for idle channels.

  3. 70% fail

    Recreating the channel on each RPC avoids idle timeout but introduces high latency and resource overhead, potentially causing other errors like 'RESOURCE_EXHAUSTED'.