docker resource_error ai_generated partial

Error response from daemon: mkdir /var/lib/docker/overlay2/...: no space left on device

ID: docker/overlay2-quota-exceeded

Also available as: JSON · Markdown · 中文
78%Fix Rate
85%Confidence
1Evidence
2023-03-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
Docker 20.10.0 active — — —
Docker 24.0.0 active — — —
Docker 25.0.0 active — — —
Docker CE 26.0.0 active — — —

Root Cause

The overlay2 filesystem has exhausted its inode or block quota, typically due to excessive container layers or dangling images consuming all available space.

generic

中文

overlay2 文件系统的 inode 或块配额已耗尽，通常是由于过多的容器层或悬空镜像占用了所有可用空间。

Official Documentation

https://docs.docker.com/storage/storagedriver/overlayfs-driver/

Workarounds

  1. 75% success Run 'docker system prune --all --volumes' to remove unused containers, networks, images, and volumes. Then verify with 'df -h /var/lib/docker' and 'df -i /var/lib/docker'. 
    Run 'docker system prune --all --volumes' to remove unused containers, networks, images, and volumes. Then verify with 'df -h /var/lib/docker' and 'df -i /var/lib/docker'.
  2. 60% success Increase the overlay2 storage quota by editing /etc/docker/daemon.json: add '{"storage-opts": ["overlay2.size=100G"]}' and restart Docker. Note: this works only with xfs filesystem and project quotas enabled. 
    Increase the overlay2 storage quota by editing /etc/docker/daemon.json: add '{"storage-opts": ["overlay2.size=100G"]}' and restart Docker. Note: this works only with xfs filesystem and project quotas enabled.

中文步骤

  1. Run 'docker system prune --all --volumes' to remove unused containers, networks, images, and volumes. Then verify with 'df -h /var/lib/docker' and 'df -i /var/lib/docker'.
  2. Increase the overlay2 storage quota by editing /etc/docker/daemon.json: add '{"storage-opts": ["overlay2.size=100G"]}' and restart Docker. Note: this works only with xfs filesystem and project quotas enabled.

Dead Ends

Common approaches that don't work:

  1. 40% fail

    Running 'docker system prune -a' without checking for running containers can remove essential images and containers, causing data loss.

  2. 60% fail

    Increasing overlay2 size via daemon.json 'storage-opts' often fails because the underlying filesystem (e.g., ext4) does not support per-directory quotas.

  3. 70% fail

    Rebooting the host only temporarily frees space if the root cause (e.g., log rotation) is not addressed.