aws deployment_error ai_generated true

Stack is in UPDATE_ROLLBACK_FAILED state and can not be updated

ID: aws/cloudformation-update-rollback-failed

Also available as: JSON · Markdown
80%Fix Rate
85%Confidence
75Evidence
2018-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
2 active — — —

Root Cause

CloudFormation stack update failed and the rollback also failed, leaving the stack in a broken state. Usually caused by resources that were manually modified outside CloudFormation, or by resources that cannot be rolled back (e.g., RDS with deletion protection).

generic

Workarounds

  1. 85% success Use ContinueUpdateRollback to skip the failing resources 
    aws cloudformation continue-update-rollback \
  --stack-name my-stack \
  --resources-to-skip ResourceLogicalId1 ResourceLogicalId2
# Identify failing resources from stack events:
aws cloudformation describe-stack-events --stack-name my-stack

    Sources: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-continueupdaterollback.html

  2. 82% success Fix the external state that caused rollback failure, then continue rollback 
    1. Check stack events: aws cloudformation describe-stack-events --stack-name my-stack
2. Identify the resource that failed to rollback
3. Manually fix it (re-create deleted resources, remove deletion protection, etc.)
4. Run: aws cloudformation continue-update-rollback --stack-name my-stack

    Sources: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html

Dead Ends

Common approaches that don't work:

  1. Deleting the stack to start fresh 75% fail

    DELETE will fail if resources have deletion protection, dependencies from other stacks, or non-empty S3 buckets. Even if deletion succeeds, you lose all managed resources and state.

  2. Retrying the same update that caused the failure 95% fail

    The stack is in a failed state and won't accept updates until the rollback is completed. The same update will fail for the same reason.

Error Chain

Preceded by:
Frequently confused with: