nginx network_error ai_generated true

no live upstreams while connecting to upstream

ID: nginx/no-live-upstreams-while-connecting-to-upstream

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2024-01-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
nginx 1.24.0 active — — —
nginx 1.22.1 active — — —
nginx 1.26.0 active — — —

Root Cause

All upstream servers in the upstream block are marked as down or have failed health checks, leaving no available targets.

generic

中文

upstream块中的所有上游服务器均被标记为故障或健康检查失败，导致没有可用目标。

Official Documentation

https://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream

Workarounds

  1. 90% success Check upstream server health and restart them: systemctl restart myapp. Also verify health_check settings in nginx.conf and ensure backup servers are defined: server backup1.example.com backup; 
    Check upstream server health and restart them: systemctl restart myapp. Also verify health_check settings in nginx.conf and ensure backup servers are defined: server backup1.example.com backup;
  2. 85% success Temporarily mark a server as up with max_fails=0 fail_timeout=0 in the upstream block: server 10.0.0.1:8080 max_fails=0 fail_timeout=0; 
    Temporarily mark a server as up with max_fails=0 fail_timeout=0 in the upstream block: server 10.0.0.1:8080 max_fails=0 fail_timeout=0;

中文步骤

  1. Check upstream server health and restart them: systemctl restart myapp. Also verify health_check settings in nginx.conf and ensure backup servers are defined: server backup1.example.com backup;
  2. Temporarily mark a server as up with max_fails=0 fail_timeout=0 in the upstream block: server 10.0.0.1:8080 max_fails=0 fail_timeout=0;

Dead Ends

Common approaches that don't work:

  1. Increase worker_connections to 10240 95% fail

    This error is about upstream availability, not connection limits. Increasing worker_connections cannot revive dead upstream servers.

  2. Set proxy_next_upstream to always retry 85% fail

    proxy_next_upstream only works if there is at least one live upstream. If all are dead, retry logic is useless.