policy resource_error ai_generated true

Pod "my-pod" evicted due to resource quota: the pod has been evicted because its resource usage exceeds the quota for the namespace

ID: policy/kubernetes-pod-eviction-resource-quota

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2024-01-20First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Kubernetes v1.28 active — — —
Kubernetes v1.29 active — — —
kubelet v1.28.3 active — — —

Root Cause

Kubernetes ResourceQuota enforces limits on total CPU/memory per namespace; a pod exceeding its request/limit triggers eviction when cumulative usage hits the quota.

generic

中文

Kubernetes ResourceQuota 对每个命名空间的总 CPU/内存实施限制；当累计使用量达到配额时，超过其请求/限制的 Pod 会触发驱逐。

Official Documentation

https://kubernetes.io/docs/concepts/policy/resource-quotas/

Workarounds

  1. 85% success Check current namespace quota usage: kubectl describe quota -n <namespace>. Then reduce pod resource requests/limits to fit within available quota, e.g., in deployment YAML set resources.requests.cpu: "250m" and resources.requests.memory: "256Mi". 
    Check current namespace quota usage: kubectl describe quota -n <namespace>. Then reduce pod resource requests/limits to fit within available quota, e.g., in deployment YAML set resources.requests.cpu: "250m" and resources.requests.memory: "256Mi".
  2. 80% success Increase the namespace ResourceQuota by editing it: kubectl edit quota <quota-name> -n <namespace> and raising spec.hard.cpu or spec.hard.memory. 
    Increase the namespace ResourceQuota by editing it: kubectl edit quota <quota-name> -n <namespace> and raising spec.hard.cpu or spec.hard.memory.
  3. 70% success Use cluster autoscaling or node scaling to add more resources, then adjust quota accordingly. 
    Use cluster autoscaling or node scaling to add more resources, then adjust quota accordingly.

中文步骤

  1. 检查当前命名空间配额使用情况：kubectl describe quota -n <namespace>。然后减少 Pod 资源请求/限制以适配可用配额，例如在部署 YAML 中设置 resources.requests.cpu: "250m" 和 resources.requests.memory: "256Mi"。
  2. 通过编辑增加命名空间 ResourceQuota：kubectl edit quota <quota-name> -n <namespace> 并提高 spec.hard.cpu 或 spec.hard.memory。
  3. 使用集群自动缩放或节点缩放添加更多资源，然后相应调整配额。

Dead Ends

Common approaches that don't work:

  1. 90% fail

    Simply restarting the pod without adjusting resource limits causes immediate re-eviction.

  2. 70% fail

    Increasing the pod's resource requests without checking namespace quota may still fail if the quota is already maxed out.

  3. 80% fail

    Deleting the ResourceQuota object to bypass limits can lead to resource starvation for other pods.