aws resource_error ai_generated true

service unable to place tasks: reason: resource CPU was insufficient

ID: aws/ecs-task-stopped-resource-cpu

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2023-04-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
ECS 1.62.0 active — — —
AWS CLI 2.15.0 active — — —

Root Cause

ECS cluster or capacity provider does not have enough available CPU units to run the task definition's CPU reservation.

generic

中文

ECS集群或容量提供者没有足够的可用CPU单元来运行任务定义中预留的CPU资源。

Official Documentation

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-event-messages.html

Workarounds

  1. 75% success Scale out the ECS cluster by adding more EC2 instances or increasing Auto Scaling group size: aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --desired-capacity 10 --min-size 5 --max-size 20
    Scale out the ECS cluster by adding more EC2 instances or increasing Auto Scaling group size:
    aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --desired-capacity 10 --min-size 5 --max-size 20
  2. 85% success Use Fargate launch type or Fargate capacity provider with automatic scaling: aws ecs create-service --cluster my-cluster --service-name my-service --task-definition my-task --launch-type FARGATE --network-configuration ...
    Use Fargate launch type or Fargate capacity provider with automatic scaling:
    aws ecs create-service --cluster my-cluster --service-name my-service --task-definition my-task --launch-type FARGATE --network-configuration ...
  3. 70% success Reduce CPU reservation in the task definition to fit within available cluster resources: { "family": "my-task", "containerDefinitions": [{ "name": "my-container", "cpu": 256, "memory": 512 }] }
    Reduce CPU reservation in the task definition to fit within available cluster resources:
    {
      "family": "my-task",
      "containerDefinitions": [{
        "name": "my-container",
        "cpu": 256,
        "memory": 512
      }]
    }

中文步骤

  1. 通过增加更多EC2实例或扩展Auto Scaling组来扩展ECS集群:
    aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --desired-capacity 10 --min-size 5 --max-size 20
  2. 使用Fargate启动类型或Fargate容量提供者并启用自动扩展:
    aws ecs create-service --cluster my-cluster --service-name my-service --task-definition my-task --launch-type FARGATE --network-configuration ...
  3. 在任务定义中减少CPU预留以适应集群可用资源:
    {
      "family": "my-task",
      "containerDefinitions": [{
        "name": "my-container",
        "cpu": 256,
        "memory": 512
      }]
    }

Dead Ends

Common approaches that don't work:

  1. Increase task CPU reservation in task definition 95% fail

    Increasing CPU reservation makes the problem worse, as it requires even more CPU from the cluster.

  2. Set service desired count to 0 and back to original 90% fail

    Recycling the service does not create new capacity; the underlying resource shortage remains.

  3. Add more memory without adding CPU capacity 85% fail

    Memory is separate from CPU; the error is explicitly about CPU, so memory adjustments don't help.