networking resource_error ai_generated partial

UDP: socket receive buffer overflow on port 12345, 5000 packets dropped

ID: networking/udp-socket-buffer-overflow

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2023-08-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Linux kernel 5.15+ active — — —
Linux kernel 6.5+ active — — —
FreeBSD 13.1+ active — — —

Root Cause

UDP socket buffer overflow occurs when the application does not read data fast enough from the receive buffer, causing the kernel to drop incoming packets to prevent memory exhaustion.

generic

中文

UDP 套接字缓冲区溢出发生在应用程序未能足够快地从接收缓冲区读取数据时，导致内核丢弃传入数据包以防止内存耗尽。

Official Documentation

https://www.kernel.org/doc/html/latest/networking/udp.html

Workarounds

  1. 80% success Increase the UDP receive buffer size system-wide: sysctl -w net.core.rmem_max=26214400 && sysctl -w net.core.rmem_default=26214400, then restart the application. 
    Increase the UDP receive buffer size system-wide: sysctl -w net.core.rmem_max=26214400 && sysctl -w net.core.rmem_default=26214400, then restart the application.
  2. 85% success Optimize the application to read UDP packets in larger batches or use non-blocking I/O: In Python, use socket.setblocking(0) and a loop to drain the buffer. 
    Optimize the application to read UDP packets in larger batches or use non-blocking I/O: In Python, use socket.setblocking(0) and a loop to drain the buffer.
  3. 80% success Use the SO_RCVBUF socket option to set a larger buffer per socket: In C, setsockopt(sock, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size)); 
    Use the SO_RCVBUF socket option to set a larger buffer per socket: In C, setsockopt(sock, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size));

中文步骤

  1. Increase the UDP receive buffer size system-wide: sysctl -w net.core.rmem_max=26214400 && sysctl -w net.core.rmem_default=26214400, then restart the application.
  2. Optimize the application to read UDP packets in larger batches or use non-blocking I/O: In Python, use socket.setblocking(0) and a loop to drain the buffer.
  3. Use the SO_RCVBUF socket option to set a larger buffer per socket: In C, setsockopt(sock, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size));

Dead Ends

Common approaches that don't work:

  1. 75% fail

    任意增加套接字接收缓冲区大小（例如 net.core.rmem_max = 10000000）可能延迟溢出，但无法解决应用程序处理速度慢的问题。

  2. 90% fail

    重启应用程序或服务器会暂时清空缓冲区，但如果应用程序的读取速率与传入流量速率不匹配，溢出会再次发生。

  3. 80% fail

    禁用 UDP 校验和卸载可能减少 CPU 负载，但无法解决由应用程序级背压引起的缓冲区溢出。