grpc network_error ai_generated partial

UNAVAILABLE: stream removed by load balancer

ID: grpc/stream-removed-by-load-balancer

Also available as: JSON · Markdown · 中文
78%Fix Rate
85%Confidence
1Evidence
2024-06-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
gRPC v1.62.0 active — — —
Envoy v1.29.0 active — — —
gRPC-LB v1.8.0 active — — —

Root Cause

gRPC load balancer (e.g., gRPC-LB or Envoy) removed an active stream due to backend health check failure or connection draining.

generic

中文

gRPC负载均衡器（如gRPC-LB或Envoy）因后端健康检查失败或连接排空而移除了活跃流。

Official Documentation

https://grpc.io/docs/guides/load-balancing/

Workarounds

  1. 80% success Configure gRPC client with retry policy and fallback: set 'grpc.enable_retries=1' and 'GRPC_RETRY_BUFFER_SIZE=32MB' in environment; implement a custom retry interceptor with exponential backoff (e.g., 100ms, 500ms, 2s). 
    Configure gRPC client with retry policy and fallback: set 'grpc.enable_retries=1' and 'GRPC_RETRY_BUFFER_SIZE=32MB' in environment; implement a custom retry interceptor with exponential backoff (e.g., 100ms, 500ms, 2s).
  2. 75% success Adjust load balancer health check interval and timeout: in Envoy, set 'health_check.interval: 5s' and 'health_check.timeout: 2s' to reduce false removals; ensure 'connection_drain_on_health_failure: false' for critical streams. 
    Adjust load balancer health check interval and timeout: in Envoy, set 'health_check.interval: 5s' and 'health_check.timeout: 2s' to reduce false removals; ensure 'connection_drain_on_health_failure: false' for critical streams.
  3. 85% success Implement client-side reconnection logic: use gRPC's 'Channelz' to monitor subchannel state and reconnect on 'TRANSIENT_FAILURE' (e.g., 'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)'). 
    Implement client-side reconnection logic: use gRPC's 'Channelz' to monitor subchannel state and reconnect on 'TRANSIENT_FAILURE' (e.g., 'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)').

中文步骤

  1. 配置gRPC客户端重试策略和回退：设置环境变量'grpc.enable_retries=1'和'GRPC_RETRY_BUFFER_SIZE=32MB'；实现自定义重试拦截器，使用指数退避（如100ms、500ms、2s）。
  2. 调整负载均衡器健康检查间隔和超时：在Envoy中设置'health_check.interval: 5s'和'health_check.timeout: 2s'以减少误移除；对关键流设置'connection_drain_on_health_failure: false'。
  3. 实现客户端重连逻辑：使用gRPC的'Channelz'监控子通道状态，在'TRANSIENT_FAILURE'时重连（如'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)'）。

Dead Ends

Common approaches that don't work:

  1. 85% fail

    Increasing client retry count without addressing load balancer health check fails because retries hit the same unhealthy backend.

  2. 70% fail

    Disabling TLS entirely reduces security and may mask underlying connection issues; the error persists if the load balancer still sees backend as unhealthy.

  3. 90% fail

    Restarting the client without adjusting load balancer configuration only temporarily reconnects; the error recurs on next health check failure.