DiskPressure cloud resource_error ai_generated true

PodEviction: The node had condition: [DiskPressure].

ID: cloud/azure-aks-pod-eviction-disk-pressure

Also available as: JSON · Markdown · 中文
85%Fix Rate
88%Confidence
1Evidence
2024-01-10First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
AKS 1.28 active — — —
AKS 1.29 active — — —
Kubernetes 1.27 active — — —
Azure Linux Node Image 2024 active — — —

Root Cause

AKS node's local disk usage exceeds 85% threshold, triggering kubelet to evict pods to free space, often due to container logs, images, or emptyDir volumes filling up the OS disk.

generic

中文

AKS节点的本地磁盘使用率超过85%阈值，触发kubelet驱逐Pod以释放空间，通常是由于容器日志、镜像或emptyDir卷填满了OS磁盘。

Official Documentation

https://learn.microsoft.com/en-us/azure/aks/manage-disk-usage

Workarounds

  1. 85% success Evict pods gracefully and clean up unused container images on the node: `kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data`, then SSH into node and run `docker system prune -a --force` or `nerdctl system prune -a --force` (depending on container runtime). Uncordon node: `kubectl uncordon <node-name>`. 
    Evict pods gracefully and clean up unused container images on the node: `kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data`, then SSH into node and run `docker system prune -a --force` or `nerdctl system prune -a --force` (depending on container runtime). Uncordon node: `kubectl uncordon <node-name>`.
  2. 90% success Configure log rotation for container logs by setting up a DaemonSet with fluent-bit or filebeat to ship logs and limit local storage, or adjust kubelet config: `--container-log-max-size=10Mi --container-log-max-files=3`. 
    Configure log rotation for container logs by setting up a DaemonSet with fluent-bit or filebeat to ship logs and limit local storage, or adjust kubelet config: `--container-log-max-size=10Mi --container-log-max-files=3`.

中文步骤

  1. Evict pods gracefully and clean up unused container images on the node: `kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data`, then SSH into node and run `docker system prune -a --force` or `nerdctl system prune -a --force` (depending on container runtime). Uncordon node: `kubectl uncordon <node-name>`.
  2. Configure log rotation for container logs by setting up a DaemonSet with fluent-bit or filebeat to ship logs and limit local storage, or adjust kubelet config: `--container-log-max-size=10Mi --container-log-max-files=3`.

Dead Ends

Common approaches that don't work:

  1. 95% fail

    Pods are recreated by controllers and the node remains under DiskPressure, causing immediate re-eviction.

  2. 80% fail

    New nodes may also fill up quickly if the root cause (e.g., log rotation misconfiguration) is not fixed; existing node remains under pressure.

  3. 90% fail

    Restarting kubelet does not free disk space; it only temporarily resets the pressure condition until usage rechecked.