aws resource_error ai_generated true

service unable to place tasks: reason: resource CPU was insufficient

ID: aws/ecs-task-stopped-resource-cpu

Also available as: JSON · Markdown · 中文
80%Fix Rate
85%Confidence
1Evidence
2023-04-12First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
ECS 1.62.0 active
AWS CLI 2.15.0 active

Root Cause

ECS cluster or capacity provider does not have enough available CPU units to run the task definition's CPU reservation.

generic

中文

ECS集群或容量提供者没有足够的可用CPU单元来运行任务定义中预留的CPU资源。

Official Documentation

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-event-messages.html

Workarounds

  1. 75% success Scale out the ECS cluster by adding more EC2 instances or increasing Auto Scaling group size: aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --desired-capacity 10 --min-size 5 --max-size 20
    Scale out the ECS cluster by adding more EC2 instances or increasing Auto Scaling group size:
    aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --desired-capacity 10 --min-size 5 --max-size 20
  2. 85% success Use Fargate launch type or Fargate capacity provider with automatic scaling: aws ecs create-service --cluster my-cluster --service-name my-service --task-definition my-task --launch-type FARGATE --network-configuration ...
    Use Fargate launch type or Fargate capacity provider with automatic scaling:
    aws ecs create-service --cluster my-cluster --service-name my-service --task-definition my-task --launch-type FARGATE --network-configuration ...
  3. 70% success Reduce CPU reservation in the task definition to fit within available cluster resources: { "family": "my-task", "containerDefinitions": [{ "name": "my-container", "cpu": 256, "memory": 512 }] }
    Reduce CPU reservation in the task definition to fit within available cluster resources:
    {
      "family": "my-task",
      "containerDefinitions": [{
        "name": "my-container",
        "cpu": 256,
        "memory": 512
      }]
    }

中文步骤

  1. 通过增加更多EC2实例或扩展Auto Scaling组来扩展ECS集群:
    aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --desired-capacity 10 --min-size 5 --max-size 20
  2. 使用Fargate启动类型或Fargate容量提供者并启用自动扩展:
    aws ecs create-service --cluster my-cluster --service-name my-service --task-definition my-task --launch-type FARGATE --network-configuration ...
  3. 在任务定义中减少CPU预留以适应集群可用资源:
    {
      "family": "my-task",
      "containerDefinitions": [{
        "name": "my-container",
        "cpu": 256,
        "memory": 512
      }]
    }

Dead Ends

Common approaches that don't work:

  1. Increase task CPU reservation in task definition 95% fail

    Increasing CPU reservation makes the problem worse, as it requires even more CPU from the cluster.

  2. Set service desired count to 0 and back to original 90% fail

    Recycling the service does not create new capacity; the underlying resource shortage remains.

  3. Add more memory without adding CPU capacity 85% fail

    Memory is separate from CPU; the error is explicitly about CPU, so memory adjustments don't help.