networking network_error ai_generated partial

TCP: Out-of-order packet received on connection 10.0.0.1:443 -> 192.168.1.50:54321, expected seq 12345, got 12390

ID: networking/tcp-out-of-order

Also available as: JSON · Markdown · 中文
72%Fix Rate
82%Confidence
1Evidence
2023-12-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Linux kernel 6.5.0-14-generic active — — —
FreeBSD 13.2 active — — —
Windows 11 Pro 23H2 active — — —
nginx 1.24.0 active — — —

Root Cause

A TCP packet arrived with a sequence number that does not match the expected next sequence number, indicating packet reordering in the network, often due to multipath routing, load balancers, or router bufferbloat.

generic

中文

TCP数据包的序列号与预期的下一个序列号不匹配，表明网络中发生了数据包重排序，通常是由于多路径路由、负载均衡器或路由器缓冲区膨胀。

Official Documentation

https://datatracker.ietf.org/doc/html/rfc9293

Workarounds

  1. 85% success Enable TCP reordering detection on the receiver: `sysctl -w net.ipv4.tcp_reordering=3` to allow up to 3 reordered packets before treating as loss 
    Enable TCP reordering detection on the receiver: `sysctl -w net.ipv4.tcp_reordering=3` to allow up to 3 reordered packets before treating as loss
  2. 78% success Use `tcpdump` to identify the path causing reordering and pin the connection to a single path via routing policy: `ip route add 192.168.1.50/32 via 10.0.0.1 dev eth0` 
    Use `tcpdump` to identify the path causing reordering and pin the connection to a single path via routing policy: `ip route add 192.168.1.50/32 via 10.0.0.1 dev eth0`

中文步骤

  1. Enable TCP reordering detection on the receiver: `sysctl -w net.ipv4.tcp_reordering=3` to allow up to 3 reordered packets before treating as loss
  2. Use `tcpdump` to identify the path causing reordering and pin the connection to a single path via routing policy: `ip route add 192.168.1.50/32 via 10.0.0.1 dev eth0`

Dead Ends

Common approaches that don't work:

  1. 65% fail

    Large buffers can hide the symptom but exacerbate bufferbloat, leading to increased latency and packet drops.

  2. 75% fail

    SACK is designed to handle out-of-order packets efficiently; disabling it makes retransmission less efficient and can degrade performance.

  3. 80% fail

    The reordering is a network-level issue; restarting the app does not fix the underlying path and the problem will recur.