504 api network_error ai_generated partial

504 Gateway Timeout: upstream response timeout

ID: api/http-504-gateway-timeout-upstream

Also available as: JSON · Markdown · 中文
80%Fix Rate
89%Confidence
1Evidence
2023-08-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Nginx 1.25.0 active — — —
AWS API Gateway (2024) active — — —
Kong 3.6.0 active — — —
HAProxy 2.9.0 active — — —

Root Cause

An API gateway or reverse proxy (e.g., Nginx, AWS API Gateway) timed out while waiting for the upstream service to respond, typically due to slow processing or network issues.

generic

中文

API 网关或反向代理（如 Nginx、AWS API Gateway）在等待上游服务响应时超时，通常由于处理缓慢或网络问题。

Official Documentation

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504

Workarounds

  1. 85% success Increase the upstream timeout in the gateway configuration. For Nginx, add to the location block: `proxy_read_timeout 60s; proxy_connect_timeout 30s;`. Then reload: `nginx -s reload`. 
    Increase the upstream timeout in the gateway configuration. For Nginx, add to the location block: `proxy_read_timeout 60s; proxy_connect_timeout 30s;`. Then reload: `nginx -s reload`.
  2. 80% success Optimize the upstream service's response time by adding caching, database query optimization, or asynchronous processing. For example, use Redis caching for frequent queries. 
    Optimize the upstream service's response time by adding caching, database query optimization, or asynchronous processing. For example, use Redis caching for frequent queries.
  3. 75% success Implement a retry mechanism with exponential backoff in the client, but ensure idempotency. Example in Python: `requests.get(url, timeout=10); time.sleep(2**attempt)`. 
    Implement a retry mechanism with exponential backoff in the client, but ensure idempotency. Example in Python: `requests.get(url, timeout=10); time.sleep(2**attempt)`.

中文步骤

  1. Increase the upstream timeout in the gateway configuration. For Nginx, add to the location block: `proxy_read_timeout 60s; proxy_connect_timeout 30s;`. Then reload: `nginx -s reload`.
  2. Optimize the upstream service's response time by adding caching, database query optimization, or asynchronous processing. For example, use Redis caching for frequent queries.
  3. Implement a retry mechanism with exponential backoff in the client, but ensure idempotency. Example in Python: `requests.get(url, timeout=10); time.sleep(2**attempt)`.

Dead Ends

Common approaches that don't work:

  1. 90% fail

    The timeout occurs at the gateway, not the client; client-side changes are irrelevant.

  2. 70% fail

    Restarting resets the service but does not address underlying performance bottlenecks like slow queries or resource leaks.

  3. 80% fail

    Removing timeouts allows slow requests to accumulate, potentially crashing the service.