502 kubernetes network_error ai_generated true

502 Bad Gateway (Ingress/Nginx) — upstream prematurely closed connection

ID: kubernetes/ingress-nginx-502-bad-gateway-upstream

Also available as: JSON · Markdown · 中文
80%Fix Rate
88%Confidence
1Evidence
2023-06-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
ingress-nginx 1.7.0 active — — —
Kubernetes 1.27 active — — —
nginx 1.23 active — — —

Root Cause

The backend service (upstream) is not responding or closes connections prematurely, often due to high load or misconfigured timeouts.

generic

中文

后端服务（上游）无响应或过早关闭连接，通常由于负载过高或超时配置错误。

Official Documentation

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-timeouts

Workarounds

  1. 85% success Check upstream service health: `kubectl get pods -l app=<service>` and `kubectl logs <pod>`. Increase resources or add horizontal pod autoscaling if under load. 
    Check upstream service health: `kubectl get pods -l app=<service>` and `kubectl logs <pod>`. Increase resources or add horizontal pod autoscaling if under load.
  2. 80% success Add ingress annotation to adjust timeouts: `nginx.ingress.kubernetes.io/proxy-read-timeout: '600'` and `nginx.ingress.kubernetes.io/proxy-send-timeout: '600'`. 
    Add ingress annotation to adjust timeouts: `nginx.ingress.kubernetes.io/proxy-read-timeout: '600'` and `nginx.ingress.kubernetes.io/proxy-send-timeout: '600'`.

中文步骤

  1. 检查上游服务健康状态：`kubectl get pods -l app=<service>` 和 `kubectl logs <pod>`。如果负载过高，增加资源或启用水平 Pod 自动伸缩。
  2. 添加 Ingress 注释以调整超时：`nginx.ingress.kubernetes.io/proxy-read-timeout: '600'` 和 `nginx.ingress.kubernetes.io/proxy-send-timeout: '600'`。

Dead Ends

Common approaches that don't work:

  1. 60% fail

    Only temporarily clears the issue; the upstream problem remains.

  2. 70% fail

    Does not address the upstream service's inability to handle connections.

  3. 50% fail

    May mask the issue but can cause resource exhaustion if upstream is truly broken.