SpotInstanceTermination cloud runtime_error ai_generated partial

EC2 Spot Instance termination notice: Your Spot Instance 'i-1234567890abcdef0' is terminated because the Spot price exceeded the maximum price you are willing to pay.

ID: cloud/aws-ec2-spot-instance-termination

Also available as: JSON · Markdown · 中文
75%Fix Rate
84%Confidence
1Evidence
2023-04-18First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
EC2 Spot Instances (all regions) active — — —
AWS CLI 2.13.0 active — — —

Root Cause

The Spot Instance's bid price is lower than the current Spot market price, causing AWS to reclaim the instance.

generic

中文

竞价实例的出价低于当前竞价市场价格，导致AWS回收该实例。

Official Documentation

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instances-termination-notices.html

Workarounds

  1. 80% success Use a Spot Fleet or EC2 Fleet with multiple instance types and bid prices. Example: `aws ec2 request-spot-fleet --spot-fleet-request-config file://config.json` where config.json includes diverse instance types (e.g., m5.large, c5.large) and a max bid equal to the On-Demand price. This increases chances of capacity availability. 
    Use a Spot Fleet or EC2 Fleet with multiple instance types and bid prices. Example: `aws ec2 request-spot-fleet --spot-fleet-request-config file://config.json` where config.json includes diverse instance types (e.g., m5.large, c5.large) and a max bid equal to the On-Demand price. This increases chances of capacity availability.
  2. 75% success Implement graceful shutdown handling. Use the EC2 metadata endpoint to check for termination notices (http://169.254.169.254/latest/meta-data/spot/termination-time) and save state before termination. Example script: `curl http://169.254.169.254/latest/meta-data/spot/termination-time 2>/dev/null && echo "Termination imminent"` 
    Implement graceful shutdown handling. Use the EC2 metadata endpoint to check for termination notices (http://169.254.169.254/latest/meta-data/spot/termination-time) and save state before termination. Example script: `curl http://169.254.169.254/latest/meta-data/spot/termination-time 2>/dev/null && echo "Termination imminent"`

中文步骤

  1. Use a Spot Fleet or EC2 Fleet with multiple instance types and bid prices. Example: `aws ec2 request-spot-fleet --spot-fleet-request-config file://config.json` where config.json includes diverse instance types (e.g., m5.large, c5.large) and a max bid equal to the On-Demand price. This increases chances of capacity availability.
  2. Implement graceful shutdown handling. Use the EC2 metadata endpoint to check for termination notices (http://169.254.169.254/latest/meta-data/spot/termination-time) and save state before termination. Example script: `curl http://169.254.169.254/latest/meta-data/spot/termination-time 2>/dev/null && echo "Termination imminent"`

Dead Ends

Common approaches that don't work:

  1. Increase the bid price to the On-Demand price 60% fail

    Setting bid to On-Demand price defeats the purpose of Spot Instances and may still not guarantee persistence during high demand. AWS may still terminate if capacity is reclaimed.

  2. Use a different instance type without changing bid 70% fail

    Different instance types have different Spot markets; the same bid may still be too low for the new type.