Hands-on AWS knowledge base for developers Managed Ops
Talk to Us

How to Reduce EC2 Costs With Spot Instances: Ops Playbook

Export PDF Print-friendly layout with a copyright watermark; choose Save as PDF in the print dialog
How to Reduce EC2 Costs With Spot Instances: Ops Playbook

TL;DR

  • Spot gives you the same vCPU and RAM for roughly 60–90% less than On-Demand, but only for workloads that survive a 2-minute termination notice. Batch jobs, CI runners, and stateless API replicas qualify. A single-instance database does not.
  • Your allocation strategy matters more than your price cap. price-capacity-optimized cut interruptions by an order of magnitude in my fleets compared with lowest-price — chasing the cheapest pool is how you get a 30% churn morning.
  • Compute is only one meter. EBS volumes, public IPv4 addresses, and cross-AZ data transfer keep billing after the instance is gone.

The EC2 bill has four meters, and Spot touches exactly one

On-Demand pricing is per-second with a 60-second minimum on Linux, billed against the instance type's hourly rate in that region. Stop the instance and compute billing stops — the EBS volumes underneath do not. Reserved Instances and Savings Plans are commitments you buy in advance against that same On-Demand rate, which is why they can't be applied to Spot. Spot is spare capacity sold at a market price that floats with supply and demand, capped at the On-Demand rate.

The price you pay for Spot is the market price at launch time for that specific instance type and Availability Zone. In practice it drifts slowly — I've watched an m6i.xlarge pool sit flat for days, then move 20% in an afternoon when a large customer showed up. AWS now changes Spot prices less aggressively than the old bidding model, but the capacity can still be reclaimed at any moment with a 2-minute warning.

The remaining three meters are storage, network, and addresses. A public IPv4 address costs roughly $3–4 per month per address, which adds up fast on a fleet of 40 instances where half the boxes are internal-only and still have one attached. EBS gp3 bills per GB-month plus IOPS and throughput above baseline. Data transfer out and cross-AZ traffic land on separate line items that nobody looks at until the invoice arrives.

Where the money actually leaks before Spot ever helps

The most common leak isn't the instance price — it's idle capacity. A t3.large running at 3% average CPU for a month costs the same as one at 80%. Pull the metric before you touch pricing:

aws cloudwatch get-metric-statistics 
  --namespace AWS/EC2 --metric-name CPUUtilization 
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 
  --period 86400 --statistics Average Maximum 
  --start-time 2025-01-01T00:00:00Z --end-time 2025-01-15T00:00:00Z 
  --query 'sort_by(Datapoints,&Timestamp)[*].[Timestamp,Average,Maximum]' 
  --output table


Second leak: orphaned storage. Detached volumes are pure loss and take ten seconds to find.

```bash
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[*].[VolumeId,Size,VolumeType,CreateTime]' --output table


Third leak: NAT Gateway processing. It charges an hourly fee plus a per-GB charge on every byte, and on a container fleet pulling images all day the per-GB meter often dwarfs the hourly one. If you're running Spot workers that constantly replace themselves and re-pull a 1.5 GB image, that's real money.

## Sizing the saving before you move a single instance

Use Spot Placement Scores first — it tells you, per region and AZ, how likely your exact instance-type list is to be fulfilled at your target capacity. Scores run 1–10; I don't move a fleet on anything below 7.

```bash
aws ec2 get-spot-placement-scores 
  --instance-types m6i.xlarge m6a.xlarge m5.xlarge c6i.xlarge c6a.xlarge 
  --target-capacity 10 --region us-east-1 
  --single-availability-zone --output table


Then confirm the current market rate for that pool:

```bash
aws ec2 describe-spot-price-history 
  --instance-types m6i.xlarge --product-descriptions "Linux/UNIX" 
  --start-time "$(date -u +%Y-%m-%dT%H:%M:%S)" 
  --query 'SpotPriceHistory[0].[InstanceType,AvailabilityZone,SpotPrice]' 
  --output text
# example shape (illustrative only — re-run in your region):
# m6i.xlarge  us-east-1a  0.042100


Here's how the numbers usually shake out on a 10-instance, 4-vCPU web tier. Treat the index column as a multiplier against your region's On-Demand rate, not a dollar figure:

| Configuration | Compute coverage | Relative monthly cost | Interruption exposure |
|---|---|---|---|
| All On-Demand | 0% Spot | 100 | None |
| 20% On-Demand base + 80% Spot | 80% Spot | ~30 | Low with diversified types |
| 100% Spot, `lowest-price` | 100% Spot | ~15 | High — single-pool churn |
| 100% Spot, `price-capacity-optimized` | 100% Spot | ~18 | Low |
| On-Demand + 1-yr Compute Savings Plan | 0% Spot | ~70 | None, 1-year lock-in |

The gap between the last two rows is the part people get wrong. `lowest-price` looks cheaper on paper and then eats the difference in failed requests and retries.

## The rollout: a mixed instances policy, not a script in a loop

Don't launch Spot instances one at a time and hand-manage replacement. Put them behind an Auto Scaling group with a mixed instances policy, so AWS handles rebalancing and the 2-minute replacement for you.

```bash
aws autoscaling create-auto-scaling-group 
  --auto-scaling-group-name web-spot 
  --min-size 2 --max-size 12 --desired-capacity 4 
  --vpc-zone-identifier "subnet-0aaa1111,subnet-0bbb2222,subnet-0ccc3333" 
  --mixed-instances-policy '{
    "LaunchTemplate": {
      "LaunchTemplateSpecification": {
        "LaunchTemplateId": "lt-0abc123", "Version": "$Latest"
      }
    },
    "InstancesDistribution": {
      "OnDemandBaseCapacity": 2,
      "OnDemandPercentageAboveBaseCapacity": 0,
      "SpotAllocationStrategy": "price-capacity-optimized"
    }
  }'


Two settings carry the whole design. `OnDemandBaseCapacity: 2` keeps a floor of always-on capacity so a regional capacity crunch can't take your service to zero. The diversified instance-type list in the launch template (aim for 10–15 families with matching vCPU/RAM) is what gives `price-capacity-optimized` room to place you in a deep pool instead of a cheap, shallow one.

Then wire the drain. The instance metadata endpoint gives you the termination time, but only if you ask with a v2 token:

```bash
TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" 
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

curl -s -o /dev/null -w "%{http_code}n" 
  -H "X-aws-ec2-metadata-token: $TOKEN" 
  http://169.254.169.254/latest/meta-data/spot/instance-action
# 404 = no notice yet.  200 = you have ~120 seconds.


**The lesson I paid for:** our drain script ran for three weeks without ever firing, and we only found out when a node died mid-batch and took a 40-minute job with it. The script was doing a plain `curl` against the metadata endpoint and getting a silent `401 Unauthorized` back, because the AMI had IMDSv2 set to required and the handler swallowed the error into `/dev/null`. I verified it by SSHing in and running the curl by hand with no token — 401, exactly as expected. Fix was the token call above plus alerting on any non-404 response code. Test your handler by actually terminating an instance and watching the logs; don't trust that "the script is deployed" means it works.

Also add an EventBridge rule on `EC2 Spot Instance Interruption Warning` for anything outside the ASG, and a lifecycle hook if you need graceful connection draining beyond what the 2-minute window gives you.

## Billing traps that show up on the second invoice

A few things that reliably surprise people:

- **Savings Plans don't apply to Spot.** If you prepaid for a 1-year Compute Savings Plan and then moved 80% of your fleet to Spot, you're still paying for the commitment and now under-using it. Right-size the commitment to your On-Demand base only.
- **Spot vCPU quotas are per-region and start low.** Newer accounts hit `An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation` well before they expect to. Request the increase a few days before launch.
- **`InsufficientInstanceCapacity` on launch usually means too-narrow a request.** One instance type, one AZ, one size — that's a request with no fallbacks.
- **Cross-AZ traffic.** Spot placement spreads you across AZs whether you like it or not, and chatty services between AZs pay per GB in each direction. Same-AZ placement isn't guaranteed on Spot.
- **Instance store is not EBS.** If the instance type has local NVMe, that data is gone at interruption, not just unattached.

If you're running this across multiple client accounts and don't want to babysit quotas and allocation strategies yourself, that's the sort of thing we handle at awsgood.com — but the CLI above will get you most of the way on a single account.

Start with the placement score and one non-critical workload. Move the batch tier first, watch the interruption rate for a week, then touch the API tier.

Sources

FAQ

How often do Spot instances actually get interrupted?

It depends almost entirely on your allocation strategy and instance-type diversity. With `price-capacity-optimized` and 10+ instance types across 3 AZs, I typically see interruption rates in the low single digits per week. With `lowest-price` on two instance types, I've seen 30% of a fleet replaced inside 40 minutes during a capacity crunch. Check the `SpotInterruption` and ASG activity history metrics before you assume a number.

Can I run production workloads on Spot instances?

Yes, if the workload is stateless or checkpointed and you keep an On-Demand base capacity. The pattern that works is a mixed instances policy with `OnDemandBaseCapacity` set to your minimum viable footprint, a diversified instance-type list, and a working 2-minute drain handler. What doesn't work is a single Spot instance running a database with no replica.

Do Spot instances work with Savings Plans or Reserved Instances?

No. Savings Plans and RIs apply to On-Demand usage only, and Spot usage is billed separately at the market rate. If you shift a large share of your fleet to Spot, revisit your commitment — otherwise you're paying for reserved capacity you no longer consume. Size the Savings Plan to the On-Demand base, not the total fleet.

Leave a Reply

Your email address will not be published. Required fields are marked *