EC2 Image Builder Daily Operations: Backup, Scaling, and Recovery Drill in Practice

EC2 Image Builder Daily Operations: Backup, Scaling, and Recovery Drill in Practice
When it comes to operating EC2 Image Builder, it ultimately comes down to one thing: when something really goes wrong, can you launch a machine that can run business workloads within an hour? Build success is only a process score; recovery success is the result score.
Start with the conclusions
- Set the operations metric as "recovery time," not "build success rate." Run a drill every week that launches an instance from the latest AMI, with a target on the order of 30–60 minutes. If it exceeds that, you need to go back and check the recipe, IAM instance profile, and SSM registration steps.
- Monitor three numbers: consecutive failure count, build duration drift, and AMI accumulation. I generally trigger alarms on "2 consecutive build failures," which is much more reliable than alarming on a single failure and filters out occasional network jitter.
- For scaling, first add concurrent pipelines, then consider switching to larger instance types. The fault response order is fixed as "first switch back to the previous available AMI, then locate the root cause." Doing it in reverse will unnecessarily prolong the outage.
Below, we expand along the main thread of "can it be recovered."
Recovery drills should first cover these four scenarios
Many people only drill one scenario: delete the AMI and rebuild. That only covers the most ideal case. The failure surface in real environments is much broader.
| Scenario | How to trigger the drill | Main verification points | Target RTO |
|---|---|---|---|
| AMI accidentally deregistered or cleaned up by retention policy | Manually deregister a non-production version | Can it be found from the historical version list and made usable again | 15 minutes |
| Recipe or component changes cause new version build failures | Deliberately write an exit 1 in a component |
How long alarms take to arrive, whether recipe versions can be rolled back quickly | 20 minutes |
| Upstream base AMI updates cause compatibility issues | Point the base AMI to the latest AL2023 | Whether kernel modules, drivers, and agent versions still work | 60 minutes |
| Region-level failure requires cross-region launch | Distribute from us-east-1 to ap-northeast-1 | Whether cross-region AMIs have been distributed in advance, whether keys are available | More than 60 minutes |
The third category is the most easily overlooked. We encountered it once in production: after the Amazon Linux 2023 base AMI was updated, a self-compiled DKMS module on a certain machine did not keep up with the new kernel version. The instance came up, but the business process would not start. The build logs were all normal, and Succeeded was glaringly bright.
- 代购 SSL 证书 · DigiCert / GeoTrust / RapidSSL 单域名与通配符
- SSL/HTTPS 证书配置 · 证书安装 · 证书申请
- Wildcard 通配符 SSL 证书安装部署续费申请
The fourth category tests whether you have done cross-region distribution in advance. If you wait until a region really has a problem before copying the AMI, a 12 GB image will take at least a dozen minutes to copy across regions, not counting queuing.
Backup effectiveness verification: pipeline showing Succeeded does not mean recovery is possible
The Image Builder pipeline status only indicates that the build process has completed. An AMI being in AVAILABLE only indicates that registration succeeded. There are still several steps between these two things and "being able to boot and run business workloads."
I generally check in this order, starting with the pipeline output:
aws imagebuilder list-image-pipeline-images
--image-pipeline-arn arn:aws:imagebuilder:us-east-1:123456789012:image-pipeline/web-base
--query 'imageSummaryList[].{Ver:imageVersion,State:state.status,Reason:state.reason,AMI:outputResources.amis[0].image}'
--output table
--region us-east-1
Only continue if `State` in the output is `AVAILABLE`. If it is `FAILED`, check `Reason`.
Next, confirm that the AMI itself still exists and has not been deleted by someone else, and check whether its creation time is the same as what you thought:
```bash
aws ec2 describe-images
--image-ids ami-0abc
Leave a Reply