K8S-EXEC-005 kubernetes network_error ai_generated true

error: unable to upgrade connection: dial tcp: i/o timeout

ID: kubernetes/kubectl-exec-tcp-timeout

Also available as: JSON · Markdown · 中文
82%Fix Rate
86%Confidence
1Evidence
2023-11-02First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
kubernetes 1.22 active — — —
kubernetes 1.23 active — — —
kubernetes 1.24 active — — —
kubernetes 1.27 active — — —

Root Cause

The kube-apiserver cannot establish a connection to the kubelet on the node for exec/logs/attach operations, usually due to network policies, firewall rules, or node unavailability.

generic

中文

kube-apiserver 无法与节点上的 kubelet 建立用于 exec/logs/attach 操作的连接，通常是由于网络策略、防火墙规则或节点不可用。

Official Documentation

https://kubernetes.io/docs/reference/kubectl/kubectl-exec/

Workarounds

  1. 90% success Verify network connectivity from the control plane node to the target node on port 10250 (kubelet port): `nc -zv <node-ip> 10250`. If blocked, update firewall rules or network policies to allow traffic. 
    Verify network connectivity from the control plane node to the target node on port 10250 (kubelet port): `nc -zv <node-ip> 10250`. If blocked, update firewall rules or network policies to allow traffic.
  2. 80% success Check if the node is in a 'NotReady' state. If so, investigate kubelet logs on the node (`journalctl -u kubelet`) and restart the kubelet service. 
    Check if the node is in a 'NotReady' state. If so, investigate kubelet logs on the node (`journalctl -u kubelet`) and restart the kubelet service.

中文步骤

  1. 验证从控制平面节点到目标节点在端口 10250（kubelet 端口）上的网络连通性：`nc -zv <node-ip> 10250`。如果被阻止，更新防火墙规则或网络策略以允许流量。
  2. 检查节点是否处于 'NotReady' 状态。如果是，在节点上调查 kubelet 日志（`journalctl -u kubelet`）并重启 kubelet 服务。

Dead Ends

Common approaches that don't work:

  1. Restart the kube-apiserver pod. 95% fail

    The issue is on the node side (kubelet or network), not the API server; restarting the API server won't help.

  2. Increase the kubelet's timeout settings. 70% fail

    The timeout is due to connectivity, not latency; increasing timeouts only delays the failure.