www.awsgood.com
Let's start with a spot that's easy to get wrong: if you ask "how much does it cost per month to scale out with EC2 Auto Scaling" and only look at peak instance count, you'll definitely get it wrong — the bill only counts instance-hours. Below, we cover two threads: the cost formula and recovery drills, with drills as the main thread — being able to scale out but not being able to pull the data back is worse than not scaling out at all.
The formula is short; the hard part is estimating the "average instance count" accurately:
Monthly instance-hours ≈ Σ(instances in each period × hours in that period) × 30
Monthly cost ≈ monthly instance-hours × on-demand unit price + EBS + outbound data transfer
Here's an example with a realistic shape. The business keeps two instances running during the day, scales to 8 at the morning and evening peaks, and falls back to 3 in the afternoon:
| Period | Instances | Duration | Instance-hours/day |
|---|---|---|---|
| 00:00–09:00 | 2 | 9h | 18 |
| 09:00–12:00 | 8 | 3h | 24 |
| 12:00–14:00 | 3 | 2h | 6 |
| 14:00–18:00 | 8 | 4h | 32 |
| 18:00–24:00 | 3 | 6h | 18 |
| Total | — | 24h | 98 |
That's 98 instance-hours a day, roughly 2,940 a month, averaging 4.1 instances. At an on-demand rate on the order of $0.10/hour for m5.large (varies by region and instance type), the instance portion comes to around $290, a bit over ¥2,000; add the EBS root volume and traffic and you usually land on a single-digit to low-double-digit dollar remainder.
With the same configuration, if scale-in gets stuck and 8 instances run around the clock, that's 5,840 instance-hours, close to $580. The extra $290 isn't from scaling out — it's piled up because scale-in didn't take effect.
How to mix pricing models:
| Pricing model | vs. On-Demand | Which layer it covers | Trade-off |
|---|---|---|---|
| On-Demand | 100% | Peak scale-out layer | Most expensive |
| Savings Plans (1-year) | ~70% | 24/7 baseline | Unused committed hours are still billed |
| Standard RI, 1 year | ~60–70% | Long-term fixed specs | Locked to instance family and region; changing specs is a hassle |
| Spot | 30–40% or even lower | Stateless, interruptible scale-out layer | 2-minute interruption notice, reclaimed at any time |
In practice, cover the baseline with Savings Plans or RIs, leave the elastic layer to On-Demand and Spot, and don't try to save machine by machine.
## Can You Actually Recover When Something Goes Wrong? Four Scenarios You Must Drill
Auto Scaling failures basically fall into four categories, and each one can be drilled without waiting for a real incident.
The first is instances being killed by mistake or a health-check loop. The ELB health check fails, the ASG repeatedly replaces instances, and the service keeps flapping. The drill is simply to terminate one instance manually and watch how long the ALB target group takes to deregister it and how long the new instance takes to reach InService.
The second is insufficient capacity in a single AZ. This is the most common category for teams in China working with ap-east-1 and ap-southeast-1, and the scaling activity writes a failure directly:
```text
Launching a new EC2 instance: i-0a1b2c3d4e5f6a7b8. Status: Failed
Cause: InsufficientInstanceCapacity - We currently do not have sufficient m5.large capacity
in the Availability Zone you requested (ap-east-1a).
When you see this line, your first reaction should be to check whether the subnet is attached to only one AZ, not to raise max size.
The third is the AMI referenced by the launch template being deregistered. One slip while ops cleans up old images and scaling goes dead on the spot:
```text
Launching a new EC2 instance. Status: Failed
Cause: The image id '[ami-0123456789abcdef0]' does not exist
This error hides in "Activity history"; if you don't go digging, you'll never find it, and the instance count stays stuck below desired.
The fourth is new instances failing to come up after a release: a package download in user-data times out, or the startup script can't connect to the config center. The drill: copy the launch template, change it to the new user-data, and spin up a small ASG that takes no traffic to observe.
## Validating Backup Effectiveness: How to Prove an AMI and Snapshots Actually Work
`describe-images` showing available only means the image registered successfully; whether it can boot is a different matter. Here's how I usually verify:
```bash
aws ec2 describe-images --image-ids ami-0123456789abcdef0
--query 'Images[0].{State:State,Name:Name,Created:CreationDate,Snapshots:BlockDeviceMappings[*].Ebs.SnapshotId}'
If the output shows `State` is `available` and `Snapshots` is non-empty, you've only passed the first gate. The second gate is to actually launch a machine from it, attach it to a temporary target group, and run a command via SSM to confirm the application process is alive:
```bash
aws ssm send-command --instance-ids i-0test0123456789
--document-name "AWS-RunShellScript"
--parameters 'commands=["systemctl is-active nginx","curl -s -o /dev/null -w %{http_code} localhost:8080/health"]'
There's a gotcha in EBS snapshot recovery: new volumes are lazily loaded — they can be attached within minutes of creation, but the first block read has to be pulled from object storage, and IO latency can reach hundreds of milliseconds. For volumes on the order of 100 GB, run `fio` once in the drill to warm them up, rather than discovering the first-screen lag only when you actually switch over in production.
## Steps for a Complete Recovery Drill and RTO Records
The worst thing a drill can become is a "click and done." Follow this order and you can wrap up in under an hour.
1. Freeze the change window: don't touch the ASG configuration or ship releases during it.
2. Record the baseline: desired, InService instance count, ALB healthy host count.
3. Create the failure: set desired straight to 0, simulating the loss of the whole group.
4. Restore the configuration and start the clock, watching the scaling activities.
5. Record the RTO, roll back to the pre-drill configuration.
```bash
# Simulate the loss of all instances in the group
aws autoscaling set-desired-capacity --auto-scaling-group-name web-asg --desired-capacity 0
# Restore and watch how long each step takes
aws autoscaling set-desired-capacity --auto-scaling-group-name web-asg --desired-capacity 4
watch -n 15 "aws autoscaling describe-scaling-activities
--auto-scaling-group-name web-asg --max-items 5
--query 'Activities[*].[StartTime,StatusCode,Description]' --output table"
The output looks like this; the timestamps are your raw RTO data:
```text
2024-06-12T02:14:33Z Successful Launching a new EC2 instance: i-0a1b...
2024-06-12T02:17:52Z Successful Launching a new EC2 instance: i-0c9d...
Three and a half minutes from trigger to the first instance InService, about eight minutes until all four are ready — that's with a t3/m5 mix and dependencies pre-installed in the AMI. If the AMI still has to run yum to install packages, that number can stretch to over ten minutes.
The drill record table looks like this; fill in your own measured values:
| Scenario | RTO target | Measured | RPO | Verdict |
|---|---|---|---|---|
| Single instance terminated | 5 minutes | 3.5 minutes | 0 | Met |
| Whole group zeroed and rebuilt | 10 minutes | 8 minutes | 0 | Met |
| Single AZ unavailable | 15 minutes | Not drilled | 0 | To be added |
| Data volume accidentally deleted | 2 hours | Not met | 15 minutes | Snapshot frequency needs changing |
## Health Checks and Alarms: Don't Copy Thresholds from the Docs
The default thresholds in the docs probably don't apply to you. This is the set that produced the fewest false positives in our own operations:
| Metric | Threshold | Window |
|---|---|---|
| GroupInServiceInstances < GroupDesiredCapacity | sustained for 5 minutes | High |
| Failed appears in scaling activities | alert on a single occurrence | High |
| UnHealthyHostCount ≥ 1 | 2 consecutive periods | High |
| 5xx rate > 1% | sustained for 5 minutes | High |
| CPUUtilization target tracking at 60% | above 80% for 15 minutes | Medium |
The second one is the easiest to overlook, because when an instance dies the ASG replaces it automatically — nothing looks wrong on the surface, but you're actually losing capacity. A single command during health checks will catch it:
```bash
aws autoscaling describe-scaling-activities --auto-scaling-group-name web-asg
--query 'Activities[?StatusCode==`Failed`].[StartTime,Description]' --output table
We hit this ourselves once: a former colleague changed an ASG's max size from 10 to 4. When daytime traffic rose, it scaled only to 4 instances and stopped there; CPU was maxed out but no instance-level alarm fired, and the client only reported that "it's a bit slow." Only after comparing the time of the last successful scaling in `describe-scaling-activities` — stuck at two weeks earlier — did we locate max size. The verification method is crude but effective: put max size and the measured peak instance count on the health-check checklist; if the two are equal, you're already running against the ceiling.
## Common Questions
**Will my bill suddenly jump a lot during scaling out?** No. On-demand instances are billed by the second; a scaled-out machine costs only for as long as it runs — what jumps is capacity, not unit price. An outrageous jump is usually a badly configured scale-in policy, or instances being repeatedly rebuilt because they never pass health checks.
**Why doesn't scale-in take effect for so long?** First check the cooldown (300 seconds by default) and the target-tracking cooldown logic, then whether any instance has scale-in protection. Also, ALB request-count metrics are inherently insensitive during low-traffic periods.
**Do I have to buy RIs?** Only worthwhile if your baseline is stable at 3 or more instances and runs a full year. For highly variable workloads, start with Savings Plans for better flexibility. Actual discounts vary by region and instance type; don't blindly apply the numbers from the website.
**Can Spot fully replace On-Demand?** For stateless scale-out layers that can tolerate a 2-minute interruption, yes; don't touch databases or long-lived connection services.
Working out scaling costs isn't hard; the hard part is actually being able to bring things back up on the day something breaks. Drills, records, and threshold tuning — these three take two hours a month. If no one on the team is watching alarms and drills, awsgood.com takes over RTO records and alarm thresholds along with managed operations, but you still have to sign off on the drill results.