在打印窗口把「目标」选为「另存为 PDF」即可保存,页面已带 awsgood.com 版权水印。 返回原文

www.awsgood.com

EC2 Image Builder Daily Operations: Backup, Scaling, and Recovery Drill in Practice

2026-11-11 AWS Managed Operations 约 3 分钟

EC2 Image Builder Daily Operations: Backup, Scaling, and Recovery Drill in Practice

When it comes to operating EC2 Image Builder, it ultimately comes down to one thing: when something really goes wrong, can you launch a machine that can run business workloads within an hour? Build success is only a process score; recovery success is the result score.

Start with the conclusions

Below, we expand along the main thread of "can it be recovered."

Recovery drills should first cover these four scenarios

Many people only drill one scenario: delete the AMI and rebuild. That only covers the most ideal case. The failure surface in real environments is much broader.

Scenario How to trigger the drill Main verification points Target RTO
AMI accidentally deregistered or cleaned up by retention policy Manually deregister a non-production version Can it be found from the historical version list and made usable again 15 minutes
Recipe or component changes cause new version build failures Deliberately write an exit 1 in a component How long alarms take to arrive, whether recipe versions can be rolled back quickly 20 minutes
Upstream base AMI updates cause compatibility issues Point the base AMI to the latest AL2023 Whether kernel modules, drivers, and agent versions still work 60 minutes
Region-level failure requires cross-region launch Distribute from us-east-1 to ap-northeast-1 Whether cross-region AMIs have been distributed in advance, whether keys are available More than 60 minutes

The third category is the most easily overlooked. We encountered it once in production: after the Amazon Linux 2023 base AMI was updated, a self-compiled DKMS module on a certain machine did not keep up with the new kernel version. The instance came up, but the business process would not start. The build logs were all normal, and Succeeded was glaringly bright.

The fourth category tests whether you have done cross-region distribution in advance. If you wait until a region really has a problem before copying the AMI, a 12 GB image will take at least a dozen minutes to copy across regions, not counting queuing.

Backup effectiveness verification: pipeline showing Succeeded does not mean recovery is possible

The Image Builder pipeline status only indicates that the build process has completed. An AMI being in AVAILABLE only indicates that registration succeeded. There are still several steps between these two things and "being able to boot and run business workloads."

I generally check in this order, starting with the pipeline output:

aws imagebuilder list-image-pipeline-images 
  --image-pipeline-arn arn:aws:imagebuilder:us-east-1:123456789012:image-pipeline/web-base 
  --query 'imageSummaryList[].{Ver:imageVersion,State:state.status,Reason:state.reason,AMI:outputResources.amis[0].image}' 
  --output table 
  --region us-east-1


Only continue if `State` in the output is `AVAILABLE`. If it is `FAILED`, check `Reason`.

Next, confirm that the AMI itself still exists and has not been deleted by someone else, and check whether its creation time is the same as what you thought:

```bash
aws ec2 describe-images 
  --image-ids ami-0abc

Sources