api network_error ai_generated true

502 Bad Gateway: upstream server unreachable

ID: api/http-502-bad-gateway-upstream-unreachable

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2023-06-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Nginx 1.24+ active — — —
HAProxy 2.8+ active — — —
AWS ALB (2024) active — — —

Root Cause

The API gateway or reverse proxy received an invalid response from the upstream server, often due to the upstream being down, overloaded, or misconfigured in the load balancer.

generic

中文

API 网关或反向代理从上游服务器收到无效响应，通常是由于上游服务器宕机、过载或负载均衡器配置错误。

Official Documentation

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

Workarounds

  1. 90% success Check upstream server health via a health check endpoint (e.g., `/health`). Example: `curl -I http://upstream-service:8080/health`. If it fails, restart the upstream service or check its logs. 
    Check upstream server health via a health check endpoint (e.g., `/health`). Example: `curl -I http://upstream-service:8080/health`. If it fails, restart the upstream service or check its logs.
  2. 85% success Verify load balancer configuration, ensuring the upstream server is correctly listed and the port matches. Example: `nginx -t` to test Nginx config, then `systemctl reload nginx`. 
    Verify load balancer configuration, ensuring the upstream server is correctly listed and the port matches. Example: `nginx -t` to test Nginx config, then `systemctl reload nginx`.
  3. 80% success Temporarily bypass the gateway by hitting the upstream directly (if network permits) to isolate the issue. Example: `curl http://10.0.1.5:8080/api/v1/resource`. 
    Temporarily bypass the gateway by hitting the upstream directly (if network permits) to isolate the issue. Example: `curl http://10.0.1.5:8080/api/v1/resource`.

中文步骤

  1. 通过健康检查端点（例如 `/health`）检查上游服务器健康状态。示例：`curl -I http://upstream-service:8080/health`。如果失败，重启上游服务或检查其日志。
  2. 验证负载均衡器配置，确保上游服务器正确列出且端口匹配。示例：`nginx -t` 测试 Nginx 配置，然后 `systemctl reload nginx`。
  3. 临时绕过网关直接访问上游（如果网络允许）以隔离问题。示例：`curl http://10.0.1.5:8080/api/v1/resource`。

Dead Ends

Common approaches that don't work:

  1. 90% fail

    The upstream server itself is unhealthy; restarting the gateway does not fix the upstream issue.

  2. 75% fail

    If the upstream is down, timeouts only delay failure detection; the root cause (e.g., crashed service) remains.

  3. 60% fail

    While DNS can cause this, the error often stems from upstream service failure (e.g., port 8080 not listening), not DNS resolution.