api network_error ai_generated partial

502 Bad Gateway: Upstream timeout

ID: api/http-502-bad-gateway-upstream-timeout

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2024-01-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Nginx 1.24 active — — —
AWS ALB active — — —
HAProxy 2.6 active — — —
Kong 3.4 active — — —
Istio 1.18 active — — —

Root Cause

The proxy or gateway server (e.g., Nginx, AWS ALB) did not receive a timely response from the upstream server, causing a timeout.

generic

中文

代理或网关服务器（例如 Nginx、AWS ALB）未及时收到上游服务器的响应，导致超时。

Official Documentation

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

Workarounds

  1. 85% success Increase the proxy timeout value. For Nginx, add 'proxy_read_timeout 120s;' in the location block. For AWS ALB, increase the idle timeout setting in the target group to 120 seconds. 
    Increase the proxy timeout value. For Nginx, add 'proxy_read_timeout 120s;' in the location block. For AWS ALB, increase the idle timeout setting in the target group to 120 seconds.
  2. 80% success Optimize the upstream server's response time by adding database indexes, caching, or scaling horizontally. Use a CDN or caching layer for static responses. 
    Optimize the upstream server's response time by adding database indexes, caching, or scaling horizontally. Use a CDN or caching layer for static responses.
  3. 75% success Implement a retry mechanism with exponential backoff. Example in Python: for i in range(3): try: response = requests.get(url, timeout=30); break except requests.exceptions.Timeout: time.sleep(2**i) 
    Implement a retry mechanism with exponential backoff. Example in Python: for i in range(3): try: response = requests.get(url, timeout=30); break except requests.exceptions.Timeout: time.sleep(2**i)

中文步骤

  1. Increase the proxy timeout value. For Nginx, add 'proxy_read_timeout 120s;' in the location block. For AWS ALB, increase the idle timeout setting in the target group to 120 seconds.
  2. Optimize the upstream server's response time by adding database indexes, caching, or scaling horizontally. Use a CDN or caching layer for static responses.
  3. Implement a retry mechanism with exponential backoff. Example in Python: for i in range(3): try: response = requests.get(url, timeout=30); break except requests.exceptions.Timeout: time.sleep(2**i)

Dead Ends

Common approaches that don't work:

  1. 70% fail

    The timeout is on the proxy side, not the client. The proxy's timeout configuration must be adjusted.

  2. 60% fail

    The upstream server may be slow due to load or inefficient queries, not a crash.

  3. 90% fail

    The upstream server is still under load; immediate retries will likely timeout again.