kubernetes scheduling_error ai_generated partial

Warning FailedScheduling: 0/N nodes are available: insufficient cpu/memory

ID: kubernetes/failedscheduling

Also available as: JSON · Markdown
80%Fix Rate
85%Confidence
50Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
8 active — — —

Root Cause

No nodes have enough resources for the pod. Cluster is at capacity.

generic

Workarounds

  1. 90% success Reduce resource requests to match actual usage (check metrics first) 
    resources:
  requests:
    cpu: 100m
    memory: 128Mi

    Sources: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

  2. 85% success Scale up the cluster — add more nodes 
    # Check current node capacity:
kubectl describe nodes | grep -A 5 'Allocated resources'

# Scale up node pool (EKS example):
aws eks update-nodegroup-config --cluster-name my-cluster --nodegroup-name my-nodes --scaling-config minSize=2,maxSize=10,desiredSize=5

# Or enable cluster autoscaler

    Sources: https://kubernetes.io/docs/concepts/cluster-administration/

  3. 80% success Check for node affinity/taints preventing scheduling on available nodes 
    kubectl describe node <node> | grep -A5 Taints

    Sources: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

Dead Ends

Common approaches that don't work:

  1. Delete other pods to free resources 75% fail

    Those pods are running for a reason — may cause outages

  2. Remove resource requests from the pod spec 70% fail

    Pod may get OOMKilled without resource limits

Error Chain

Frequently confused with: