kubernetes scheduling_error ai_generated true

0/3 nodes are available: 3 Insufficient cpu.

ID: kubernetes/pod-pending-resource-quota

Also available as: JSON · Markdown
88%Fix Rate
87%Confidence
62Evidence
2023-09-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
8 active — — —

Root Cause

Pod cannot be scheduled because the cluster has insufficient CPU or memory to satisfy the pod's resource requests. This can be caused by actual resource exhaustion, overly generous resource requests, ResourceQuota limits on the namespace, or a combination of these factors.

generic

Workarounds

  1. 90% success Inspect ResourceQuota and actual resource usage with kubectl describe quota and kubectl top nodes 
    Run 'kubectl describe quota -n <namespace>' to see quota limits vs current usage. Run 'kubectl top nodes' to see actual node resource consumption. Compare the pod's resource requests (from the deployment spec) against available capacity. If quota is the bottleneck, request a quota increase or reduce the pod's requests to realistic values based on actual usage.

    Sources: https://kubernetes.io/docs/concepts/policy/resource-quotas/

  2. 88% success Right-size resource requests based on actual usage metrics 
    Use 'kubectl top pods -n <namespace>' or a monitoring tool (Prometheus, Datadog) to measure actual CPU and memory usage over time. Set requests to the P95 usage value rather than arbitrary large numbers. Many teams over-request by 3-10x, wasting schedulable capacity. Use Vertical Pod Autoscaler (VPA) in recommendation mode to get suggested values.

    Sources: https://kubernetes.io/docs/tasks/run-application/vertical-pod-autoscale/

  3. 85% success Scale down or evict lower-priority deployments to free cluster capacity 
    Identify non-critical workloads with 'kubectl get deployments -n <namespace> --sort-by=.spec.replicas'. Scale down staging or batch workloads: 'kubectl scale deployment <name> --replicas=0'. For managed clusters, enable cluster autoscaler to add nodes automatically when demand exceeds capacity.

    Sources: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

Dead Ends

Common approaches that don't work:

  1. Increasing the number of replicas in the deployment hoping the scheduler will find room 95% fail

    Adding more replicas makes the problem worse, not better. Each additional replica requires its own resource allocation. If the cluster cannot schedule one pod due to insufficient CPU, requesting more pods with the same resource requirements only increases the total resource demand and creates more Pending pods.

  2. Deleting random pods from the namespace to free up resources 85% fail

    Pods managed by controllers (Deployments, ReplicaSets, StatefulSets, DaemonSets) are immediately recreated after deletion. The freed resources are reclaimed within seconds by the replacement pods. Deleting unmanaged pods may free resources temporarily but destroys workloads without solving the underlying capacity problem.

  3. Setting resource requests to 0 to bypass the scheduling constraint 75% fail

    While setting requests to 0 allows the pod to be scheduled on any node regardless of available resources, the pod runs as BestEffort QoS class. It will be the first to be evicted under memory pressure, and it can starve other workloads or be OOMKilled unpredictably. This trades a clear scheduling error for intermittent runtime failures that are harder to diagnose.

Error Chain

Leads to:
Preceded by: