CannotStartContainerError aws resource_error ai_generated true

STOPPED (CannotStartContainerError: API error (500): devmapper: Thin pool has X free data blocks which is less than the minimum required X total data blocks)

ID: aws/ecs-task-stopped-cannotstartcontainererror

Also available as: JSON · Markdown · 中文
75%Fix Rate
85%Confidence
1Evidence
2023-08-15First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
ECS 1.62.0 active — — —
Docker 20.10.17 active — — —
Amazon Linux 2 active — — —

Root Cause

ECS task fails because the Docker storage driver (devicemapper) on the container instance has insufficient free data blocks in its thin pool, often due to disk space exhaustion or misconfigured Docker storage.

generic

中文

ECS 任务失败，因为容器实例上的 Docker 存储驱动程序（devicemapper）的精简池中的空闲数据块不足，通常是由于磁盘空间耗尽或 Docker 存储配置错误。

Official Documentation

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-stopped-reason-cannotstartcontainererror.html

Workarounds

  1. 85% success Run `docker system prune -a --volumes` on the container instance to remove unused Docker objects and free thin pool space. Then restart the ECS task. 
    Run `docker system prune -a --volumes` on the container instance to remove unused Docker objects and free thin pool space. Then restart the ECS task.
  2. 70% success Increase the Docker base device size by editing `/etc/docker/daemon.json` to set `{"storage-opts": ["dm.basesize=50G"]}` and restarting Docker service. 
    Increase the Docker base device size by editing `/etc/docker/daemon.json` to set `{"storage-opts": ["dm.basesize=50G"]}` and restarting Docker service.

中文步骤

  1. Run `docker system prune -a --volumes` on the container instance to remove unused Docker objects and free thin pool space. Then restart the ECS task.
  2. Increase the Docker base device size by editing `/etc/docker/daemon.json` to set `{"storage-opts": ["dm.basesize=50G"]}` and restarting Docker service.

Dead Ends

Common approaches that don't work:

  1. Manually restarting the ECS agent or the container instance without cleaning up unused images/containers. 90% fail

    The underlying thin pool is still full; restarting does not free data blocks.

  2. Increasing the ECS task memory limit without addressing Docker storage. 95% fail

    The error is about disk storage blocks, not memory allocation.