nginx resource_error ai_generated true

nginx: [alert] 1024 worker_connections are not enough while connecting to upstream

ID: nginx/worker-connections-are-not-enough-while-connecting-to-upstream

Also available as: JSON · Markdown · 中文

80%Fix Rate

85%Confidence

1Evidence

2023-09-01First Seen

Version Compatibility

Version	Status	Introduced	Deprecated	Notes
nginx 1.24.0	active	—	—	—
nginx 1.22.1	active	—	—	—
nginx 1.20.2	active	—	—	—
nginx 1.18.0	active	—	—	—

Root Cause

The number of simultaneous connections exceeds the worker_connections limit, causing connection failures to upstream servers.

generic

中文

同时连接数超过了 worker_connections 限制，导致连接上游服务器失败。

Official Documentation

https://nginx.org/en/docs/ngx_core_module.html#worker_connections

Workarounds

85% success Increase worker_connections and adjust system limits: events { worker_connections 4096; multi_accept on; } Also increase system max open files: ulimit -n 65536 and set in /etc/security/limits.conf: * soft nofile 65536 * hard nofile 65536 Then reload nginx.
```
Increase worker_connections and adjust system limits:
events {
    worker_connections 4096;
    multi_accept on;
}
Also increase system max open files: ulimit -n 65536 and set in /etc/security/limits.conf:
* soft nofile 65536
* hard nofile 65536
Then reload nginx.
```
80% success Increase the number of worker_processes to distribute connections: worker_processes auto; This uses all CPU cores; each worker has its own connection pool.
```
Increase the number of worker_processes to distribute connections:
worker_processes auto;
This uses all CPU cores; each worker has its own connection pool.
```
70% success Implement rate limiting or connection limiting per IP to prevent abuse: limit_conn_zone $binary_remote_addr zone=addr:10m; limit_conn addr 100; This caps connections per client to stay within worker_connections.
```
Implement rate limiting or connection limiting per IP to prevent abuse:
limit_conn_zone $binary_remote_addr zone=addr:10m;
limit_conn addr 100;
This caps connections per client to stay within worker_connections.
```

中文步骤

增加 worker_connections 并调整系统限制：
events {
    worker_connections 4096;
    multi_accept on;
}
同时增加系统最大打开文件数：ulimit -n 65536 并在 /etc/security/limits.conf 中设置：
* soft nofile 65536
* hard nofile 65536
然后重新加载 nginx。

增加 worker_processes 数量以分配连接：
worker_processes auto;
这将使用所有 CPU 核心；每个 worker 有自己的连接池。

实施速率限制或按 IP 限制连接以防止滥用：
limit_conn_zone $binary_remote_addr zone=addr:10m;
limit_conn addr 100;
这将限制每个客户端的连接数，以保持在 worker_connections 范围内。

Dead Ends

Common approaches that don't work:

50% fail
System limits (ulimit -n, fs.file-max) may still cap the total connections; also the OS may have per-process limits.
40% fail
While it reduces connection duration, it increases connection churn and may still hit the limit under high concurrency.
30% fail
Multi_accept affects connection acceptance behavior but does not increase the maximum number of connections.