kubernetes network_error ai_generated true

connection refused: dial tcp 10.96.0.10:53: connect: connection refused (due to NetworkPolicy)

ID: kubernetes/networkpolicy-blocking-dns

Also available as: JSON · Markdown · 中文
88%Fix Rate
86%Confidence
1Evidence
2024-04-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Calico v3.27.2 active — — —
Cilium v1.15.0 active — — —
Kubernetes v1.28.5 active — — —
Kubernetes v1.29.1 active — — —

Root Cause

A NetworkPolicy is blocking DNS traffic (port 53) from the pod to the CoreDNS service, either by ingress or egress rules, preventing DNS resolution.

generic

中文

NetworkPolicy阻止了Pod到CoreDNS服务的DNS流量（端口53），可能是入站或出站规则，导致DNS解析失败。

Official Documentation

https://kubernetes.io/docs/concepts/services-networking/network-policies/

Workarounds

  1. 90% success Add an egress rule to allow DNS traffic in the pod's namespace: `kubectl edit networkpolicy <policy-name>` and add under egress: `- ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]` and ensure the destination includes the kube-system namespace or the CoreDNS pod selector. 
    Add an egress rule to allow DNS traffic in the pod's namespace: `kubectl edit networkpolicy <policy-name>` and add under egress: `- ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]` and ensure the destination includes the kube-system namespace or the CoreDNS pod selector.
  2. 85% success Test connectivity to CoreDNS: `kubectl run -it --rm debug --image=busybox -- sh` then run `nslookup kubernetes.default.svc.cluster.local`. If it fails, inspect NetworkPolicies in the namespace: `kubectl describe networkpolicy -n <namespace>` 
    Test connectivity to CoreDNS: `kubectl run -it --rm debug --image=busybox -- sh` then run `nslookup kubernetes.default.svc.cluster.local`. If it fails, inspect NetworkPolicies in the namespace: `kubectl describe networkpolicy -n <namespace>`

中文步骤

  1. Add an egress rule to allow DNS traffic in the pod's namespace: `kubectl edit networkpolicy <policy-name>` and add under egress: `- ports: [{port: 53, protocol: UDP}, {port: 53, protocol: TCP}]` and ensure the destination includes the kube-system namespace or the CoreDNS pod selector.
  2. Test connectivity to CoreDNS: `kubectl run -it --rm debug --image=busybox -- sh` then run `nslookup kubernetes.default.svc.cluster.local`. If it fails, inspect NetworkPolicies in the namespace: `kubectl describe networkpolicy -n <namespace>`

Dead Ends

Common approaches that don't work:

  1. Restart the pod or delete and recreate it 95% fail

    The NetworkPolicy is still in place, so restarting the pod will result in the same DNS failure. The policy must be modified.

  2. Disable all NetworkPolicies by deleting them 60% fail

    This is a security risk and defeats the purpose of network segmentation. A targeted fix is better, e.g., adding a policy that allows DNS.