www.awsgood.com
When it comes to operating EC2 Image Builder, it ultimately comes down to one thing: when something really goes wrong, can you launch a machine that can run business workloads within an hour? Build success is only a process score; recovery success is the result score.
Below, we expand along the main thread of "can it be recovered."
Many people only drill one scenario: delete the AMI and rebuild. That only covers the most ideal case. The failure surface in real environments is much broader.
| Scenario | How to trigger the drill | Main verification points | Target RTO |
|---|---|---|---|
| AMI accidentally deregistered or cleaned up by retention policy | Manually deregister a non-production version | Can it be found from the historical version list and made usable again | 15 minutes |
| Recipe or component changes cause new version build failures | Deliberately write an exit 1 in a component |
How long alarms take to arrive, whether recipe versions can be rolled back quickly | 20 minutes |
| Upstream base AMI updates cause compatibility issues | Point the base AMI to the latest AL2023 | Whether kernel modules, drivers, and agent versions still work | 60 minutes |
| Region-level failure requires cross-region launch | Distribute from us-east-1 to ap-northeast-1 | Whether cross-region AMIs have been distributed in advance, whether keys are available | More than 60 minutes |
The third category is the most easily overlooked. We encountered it once in production: after the Amazon Linux 2023 base AMI was updated, a self-compiled DKMS module on a certain machine did not keep up with the new kernel version. The instance came up, but the business process would not start. The build logs were all normal, and Succeeded was glaringly bright.
The fourth category tests whether you have done cross-region distribution in advance. If you wait until a region really has a problem before copying the AMI, a 12 GB image will take at least a dozen minutes to copy across regions, not counting queuing.
The Image Builder pipeline status only indicates that the build process has completed. An AMI being in AVAILABLE only indicates that registration succeeded. There are still several steps between these two things and "being able to boot and run business workloads."
I generally check in this order, starting with the pipeline output:
aws imagebuilder list-image-pipeline-images
--image-pipeline-arn arn:aws:imagebuilder:us-east-1:123456789012:image-pipeline/web-base
--query 'imageSummaryList[].{Ver:imageVersion,State:state.status,Reason:state.reason,AMI:outputResources.amis[0].image}'
--output table
--region us-east-1
Only continue if `State` in the output is `AVAILABLE`. If it is `FAILED`, check `Reason`.
Next, confirm that the AMI itself still exists and has not been deleted by someone else, and check whether its creation time is the same as what you thought:
```bash
aws ec2 describe-images
--image-ids ami-0abc