kubernetes resource_error ai_generated true

failed to allocate for range 0: no IP addresses available in range: 10.244.0.0/16

ID: kubernetes/pod-network-ip-exhaustion

Also available as: JSON · Markdown · 中文
78%Fix Rate
85%Confidence
1Evidence
2024-02-20First Seen

Root Cause

The pod CIDR range is exhausted because too many pods are scheduled or there's a leak in IP address release, often due to a CNI plugin bug.

generic

中文

Pod CIDR 范围已耗尽，因为调度了太多 pod 或 IP 地址释放存在泄漏，通常是由于 CNI 插件错误。

Official Documentation

https://kubernetes.io/docs/concepts/cluster-administration/networking/#pod-cidr

Workarounds

  1. 75% success Flush the CNI IPAM database: For Calico, run 'calicoctl ipam release --ip=<specific-ip>' or 'calicoctl ipam release --all' to release unused IPs. 
    Flush the CNI IPAM database: For Calico, run 'calicoctl ipam release --ip=<specific-ip>' or 'calicoctl ipam release --all' to release unused IPs.
  2. 80% success Scale down unnecessary deployments to free pod IPs: 'kubectl scale deployment <name> --replicas=0' then scale back up. 
    Scale down unnecessary deployments to free pod IPs: 'kubectl scale deployment <name> --replicas=0' then scale back up.

中文步骤

  1. Flush the CNI IPAM database: For Calico, run 'calicoctl ipam release --ip=<specific-ip>' or 'calicoctl ipam release --all' to release unused IPs.
  2. Scale down unnecessary deployments to free pod IPs: 'kubectl scale deployment <name> --replicas=0' then scale back up.

Dead Ends

Common approaches that don't work:

  1. Restart all pods to release IPs 80% fail

    Restarting pods doesn't force the CNI to release stale IPs; the allocation table remains full.

  2. Increase the pod CIDR size in kube-controller-manager 70% fail

    Changing CIDR requires cluster reinitialization and doesn't free existing IPs; it's a complex operation with downtime.