# UNAVAILABLE: stream removed by load balancer

- **ID:** `grpc/stream-removed-by-load-balancer`
- **Domain:** grpc
- **Category:** network_error
- **Verification:** ai_generated
- **Fix Rate:** 78%

## Root Cause

gRPC load balancer (e.g., gRPC-LB or Envoy) removed an active stream due to backend health check failure or connection draining.

## Version Compatibility

| Version | Status | Introduced | Deprecated |
|---------|--------|------------|------------|
| gRPC v1.62.0 | active | — | — |
| Envoy v1.29.0 | active | — | — |
| gRPC-LB v1.8.0 | active | — | — |

## Workarounds

1. **Configure gRPC client with retry policy and fallback: set 'grpc.enable_retries=1' and 'GRPC_RETRY_BUFFER_SIZE=32MB' in environment; implement a custom retry interceptor with exponential backoff (e.g., 100ms, 500ms, 2s).** (80% success)
   ```
   Configure gRPC client with retry policy and fallback: set 'grpc.enable_retries=1' and 'GRPC_RETRY_BUFFER_SIZE=32MB' in environment; implement a custom retry interceptor with exponential backoff (e.g., 100ms, 500ms, 2s).
   ```
2. **Adjust load balancer health check interval and timeout: in Envoy, set 'health_check.interval: 5s' and 'health_check.timeout: 2s' to reduce false removals; ensure 'connection_drain_on_health_failure: false' for critical streams.** (75% success)
   ```
   Adjust load balancer health check interval and timeout: in Envoy, set 'health_check.interval: 5s' and 'health_check.timeout: 2s' to reduce false removals; ensure 'connection_drain_on_health_failure: false' for critical streams.
   ```
3. **Implement client-side reconnection logic: use gRPC's 'Channelz' to monitor subchannel state and reconnect on 'TRANSIENT_FAILURE' (e.g., 'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)').** (85% success)
   ```
   Implement client-side reconnection logic: use gRPC's 'Channelz' to monitor subchannel state and reconnect on 'TRANSIENT_FAILURE' (e.g., 'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)').
   ```

## Dead Ends

- **** — Increasing client retry count without addressing load balancer health check fails because retries hit the same unhealthy backend. (85% fail)
- **** — Disabling TLS entirely reduces security and may mask underlying connection issues; the error persists if the load balancer still sees backend as unhealthy. (70% fail)
- **** — Restarting the client without adjusting load balancer configuration only temporarily reconnects; the error recurs on next health check failure. (90% fail)