kubernetes runtime_error ai_generated true

0/4 nodes are available: 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.

ID: kubernetes/insufficient-node-selector

Also available as: JSON · Markdown · 中文
85%Fix Rate
84%Confidence
1Evidence
2023-11-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kubernetes 1.24 active — — —
Kubernetes 1.27 active — — —

Root Cause

The pod has node affinity rules or node selectors that do not match any available node's labels, and preemption cannot find victims because the pod's priority is too low or no lower-priority pods exist.

generic

中文

Pod 的节点亲和性规则或节点选择器与任何可用节点的标签都不匹配，并且抢占无法找到受害者，因为 Pod 的优先级太低或不存在更低优先级的 Pod。

Official Documentation

https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity

Workarounds

  1. 90% success Check the pod's node affinity/selector: `kubectl get pod <pod-name> -o yaml | grep -A 10 nodeAffinity`. Then verify node labels: `kubectl get nodes --show-labels`. Adjust the pod spec or node labels to match. 
    Check the pod's node affinity/selector: `kubectl get pod <pod-name> -o yaml | grep -A 10 nodeAffinity`. Then verify node labels: `kubectl get nodes --show-labels`. Adjust the pod spec or node labels to match.
  2. 85% success If the pod uses nodeSelector, add the required label to a node: `kubectl label node <node-name> <key>=<value>`. For example, if nodeSelector has `disktype: ssd`, run `kubectl label node worker1 disktype=ssd`. 
    If the pod uses nodeSelector, add the required label to a node: `kubectl label node <node-name> <key>=<value>`. For example, if nodeSelector has `disktype: ssd`, run `kubectl label node worker1 disktype=ssd`.
  3. 80% success Remove or relax the node affinity rules by editing the pod/deployment spec: `kubectl edit deployment <deployment-name>` and remove the `nodeSelector` or `nodeAffinity` fields. 
    Remove or relax the node affinity rules by editing the pod/deployment spec: `kubectl edit deployment <deployment-name>` and remove the `nodeSelector` or `nodeAffinity` fields.

中文步骤

  1. Check the pod's node affinity/selector: `kubectl get pod <pod-name> -o yaml | grep -A 10 nodeAffinity`. Then verify node labels: `kubectl get nodes --show-labels`. Adjust the pod spec or node labels to match.
  2. If the pod uses nodeSelector, add the required label to a node: `kubectl label node <node-name> <key>=<value>`. For example, if nodeSelector has `disktype: ssd`, run `kubectl label node worker1 disktype=ssd`.
  3. Remove or relax the node affinity rules by editing the pod/deployment spec: `kubectl edit deployment <deployment-name>` and remove the `nodeSelector` or `nodeAffinity` fields.

Dead Ends

Common approaches that don't work:

  1. 70% fail

    Increasing the pod's priority class does not help if no nodes match the affinity rules; it only affects preemption.

  2. 80% fail

    Adding more nodes without correct labels will not match the pod's selectors.