{
  "id": "grpc/stream-removed-by-load-balancer",
  "signature": "UNAVAILABLE: stream removed by load balancer",
  "signature_zh": "UNAVAILABLE: 负载均衡器移除了流",
  "regex": "UNAVAILABLE: stream removed by load balancer",
  "domain": "grpc",
  "category": "network_error",
  "subcategory": null,
  "root_cause": "gRPC load balancer (e.g., gRPC-LB or Envoy) removed an active stream due to backend health check failure or connection draining.",
  "root_cause_type": "generic",
  "root_cause_zh": "gRPC负载均衡器（如gRPC-LB或Envoy）因后端健康检查失败或连接排空而移除了活跃流。",
  "versions": [
    {
      "version": "gRPC v1.62.0",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    },
    {
      "version": "Envoy v1.29.0",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    },
    {
      "version": "gRPC-LB v1.8.0",
      "introduced": null,
      "deprecated": null,
      "removed": null,
      "behavior_change": null,
      "status": "active"
    }
  ],
  "os_specific": {},
  "dead_ends": [
    {
      "action": "",
      "why_fails": "Increasing client retry count without addressing load balancer health check fails because retries hit the same unhealthy backend.",
      "fail_rate": 0.85,
      "condition": "",
      "sources": []
    },
    {
      "action": "",
      "why_fails": "Disabling TLS entirely reduces security and may mask underlying connection issues; the error persists if the load balancer still sees backend as unhealthy.",
      "fail_rate": 0.7,
      "condition": "",
      "sources": []
    },
    {
      "action": "",
      "why_fails": "Restarting the client without adjusting load balancer configuration only temporarily reconnects; the error recurs on next health check failure.",
      "fail_rate": 0.9,
      "condition": "",
      "sources": []
    }
  ],
  "workarounds": [
    {
      "action": "Configure gRPC client with retry policy and fallback: set 'grpc.enable_retries=1' and 'GRPC_RETRY_BUFFER_SIZE=32MB' in environment; implement a custom retry interceptor with exponential backoff (e.g., 100ms, 500ms, 2s).",
      "success_rate": 0.8,
      "how": "Configure gRPC client with retry policy and fallback: set 'grpc.enable_retries=1' and 'GRPC_RETRY_BUFFER_SIZE=32MB' in environment; implement a custom retry interceptor with exponential backoff (e.g., 100ms, 500ms, 2s).",
      "condition": "",
      "sources": []
    },
    {
      "action": "Adjust load balancer health check interval and timeout: in Envoy, set 'health_check.interval: 5s' and 'health_check.timeout: 2s' to reduce false removals; ensure 'connection_drain_on_health_failure: false' for critical streams.",
      "success_rate": 0.75,
      "how": "Adjust load balancer health check interval and timeout: in Envoy, set 'health_check.interval: 5s' and 'health_check.timeout: 2s' to reduce false removals; ensure 'connection_drain_on_health_failure: false' for critical streams.",
      "condition": "",
      "sources": []
    },
    {
      "action": "Implement client-side reconnection logic: use gRPC's 'Channelz' to monitor subchannel state and reconnect on 'TRANSIENT_FAILURE' (e.g., 'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)').",
      "success_rate": 0.85,
      "how": "Implement client-side reconnection logic: use gRPC's 'Channelz' to monitor subchannel state and reconnect on 'TRANSIENT_FAILURE' (e.g., 'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)').",
      "condition": "",
      "sources": []
    }
  ],
  "workarounds_zh": [
    "配置gRPC客户端重试策略和回退：设置环境变量'grpc.enable_retries=1'和'GRPC_RETRY_BUFFER_SIZE=32MB'；实现自定义重试拦截器，使用指数退避（如100ms、500ms、2s）。",
    "调整负载均衡器健康检查间隔和超时：在Envoy中设置'health_check.interval: 5s'和'health_check.timeout: 2s'以减少误移除；对关键流设置'connection_drain_on_health_failure: false'。",
    "实现客户端重连逻辑：使用gRPC的'Channelz'监控子通道状态，在'TRANSIENT_FAILURE'时重连（如'channel.watch_connectivity_state(grpc.ChannelConnectivity.TRANSIENT_FAILURE)'）。"
  ],
  "transition_graph": {
    "leads_to": [],
    "preceded_by": [],
    "frequently_confused_with": []
  },
  "official_doc_url": "https://grpc.io/docs/guides/load-balancing/",
  "official_doc_section": null,
  "error_code": null,
  "verification_tier": "ai_generated",
  "confidence": 0.85,
  "fix_success_rate": 0.78,
  "resolvable": "partial",
  "first_seen": "2024-06-15",
  "last_confirmed": "2024-06-01",
  "last_updated": "2024-06-01",
  "evidence_count": 1,
  "tags": [],
  "locale": "en",
  "aliases": []
}