www.awsgood.com
When given an unfamiliar environment, I don't log into the console to look at charts first. I run a single command to flatten out the details of all instances—instance type, availability zone, launch time, and whether IMDSv2 is enforced.
aws ec2 describe-instances --region us-east-1
--query 'Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name,AZ:Placement.AvailabilityZone,Launch:LaunchTime,IMDSv2:MetadataOptions.HttpTokens,Tags:Tags}'
--output table
In the output, I focus on three things: instances with `LaunchTime` more than two or three years ago are most likely unmaintained; `HttpTokens` still being `optional` means IMDSv2 is not enforced, posing a risk of SSRF capturing temporary credentials; those without `Owner` or `Env` in `Tags` go straight onto the to-do list.
The asset table should have at least five columns: instance ID, one-sentence description of purpose, owner, dependent domain names/certificates, and backup strategy. Domains and certificates are the easiest to miss—when a client handed over an instance list but not the ACM certificates and Route 53 hosted zones, three months later the certificate expired and the site threw `NET::ERR_CERT_DATE_INVALID` directly on iOS. After that, I put certificate expiry dates directly into the asset table.
Permission handover has two tracks. One is IAM: confirm whether there is a dedicated operations role, who has `AdministratorAccess` attached, and whether it can be changed to on-demand temporary credentials with privilege escalation (paired with SSM Session Manager for keyless login, which also saves the trouble of opening port 22). The other is billing and root: the root email, MFA device, and payment method are not handed over—only a read-only `Billing` role is given. That's the bottom line.
For documentation handover, I generally only need three things: a simple architecture diagram (even a hand-drawn photo), incident records from the past six months, and a change log. If none of the three can be produced, it means this environment was previously "manual operations," and the baseline will need to be rebuilt as a new environment afterward—both the quote and the timeline must go up.
## How to Configure Monitoring Alerts: Which Metrics Must Have Thresholds, Which Are Pure Noise
The metrics EC2 sends to CloudWatch by default are only CPU, network, disk I/O, and status checks—**memory and disk usage are not included by default**. This is where beginners most easily fail—the dashboard looks perfectly normal, but the root volume is actually at 98%.
You must install the CloudWatch Agent to have `mem_used_percent` and `disk_used_percent`. After installation, verify first:
```bash
aws cloudwatch list-metrics --namespace CWAgent --region us-east-1
--query 'Metrics[].MetricName' --output text | tr 't' 'n' | sort -u
If the return is empty, the Agent isn't running or the configuration hasn't taken effect—don't rush to configure alerts. I've seen cases where memory alerts were configured and stayed at `INSUFFICIENT_DATA` for three months with nobody noticing; details in the pitfalls section below.
For thresholds, my habitual configuration:
| Metric | Collection Method | Suggested Threshold | Consecutive Data Points | Notes |
|---|---|---|---|---|
| `CPUUtilization` | Default, 5 minutes | 80% | 3 (15 minutes) | Single-point triggering is maddening; business peaks will inevitably cause false alarms |
| `StatusCheckFailed_System` | Default, 1 minute | ≥1 | 2 | Host-level issue; can only migrate via stop/start |
| `StatusCheckFailed_Instance` | Default, 1 minute | ≥1 | 3 | Usually kernel panic or full disk |
| `mem_used_percent` | Agent, 60 seconds | 85% | 3 | Memory leaks are more fatal than CPU spikes |
| `disk_used_percent` | Agent, 60 seconds | 85% | 2 | Small instance types with 20GB root volumes grow fast |
| `CPUCreditBalance` (t series) | Default | < 30 | 1 | Burstable instances slow down abruptly when credits are exhausted |
Don't just attach SNS email to alert actions. SNS emails received domestically are occasionally delayed by a few minutes, and during a P1 incident those few minutes are critical—I generally also push a webhook to an IM group. Also, the `treatMissingData` parameter must be explicitly set to `breaching` or `notBreaching`; the default `missing` will keep the alert in its original state when there's no data, which is equivalent to being blind.
## Inspection Checklist: What to Look at Daily, Weekly, and Monthly
Inspection is not clicking through every dashboard panel; it's looking at specific places with suspicion. My checklist looks roughly like this:
**Daily (within 10 minutes)**
- Whether any alert status is `INSUFFICIENT_DATA`—no data is more dangerous than having alerts
- Whether last night's DLM (Data Lifecycle Manager) snapshot task status is `completed`
- How much disk usage has grown compared to yesterday; growth rate says more about the problem than absolute value
**Weekly**
- Orphaned instances without an `Env` tag or not associated with an Auto Scaling group
- Non-80/443 ports open to `0.0.0.0/0` in security groups
- SSM managed instance online rate; for offline ones, first check whether the Agent has died
**Monthly**
- Pick one machine for a real recovery drill (not checking snapshot status—actually launching one)
- Patch baseline compliance rate
- Idle resources: instances with CPU below 2% for 30 consecutive days, unattached EBS volumes, unassociated Elastic IPs
Here's an order-of-magnitude judgment: a single `m5.xlarge` running on-demand in us-east-1 for a full month is in the low four figures in RMB (varies by region and instance type; refer to the AWS official pricing page), so three idle machines for a year is over thirty thousand. That's why I persuaded clients to add idle resource checks to the monthly inspection—more practical than any optimization recommendation.
For the 30-day handover pace, I generally schedule it like this: Week 1, read-only without changes, complete the asset table; Week 2, fill in the monitoring and alert baseline, downgrade or delete all noisy alerts; Week 3, do the first recovery drill and patch assessment; Week 4, produce the baseline report and confirm with the client which changes go into the change window. Don't touch production configuration in the first two weeks—this is a hard rule. Acting without understanding dependencies is equivalent to taking the blame for the previous handler.
## Real Testing and Pitfalls: An Alert Storm and That Alert Stuck at INSUFFICIENT_DATA
In March 2026, I took over a client with 12 EC2 instances. The previous operations person had set the `CPUUtilization` alert at 60%, single-point triggering, with a 5-minute evaluation period. Business peaks were at 10 a.m. and 8 p.m. daily, so SNS pushed over two hundred emails a day. Within less than two weeks, nobody in the group was looking at alerts anymore.
The real incident came at exactly this time. One `m5.large` running log processing had a 20GB root volume that went from 78% to 100% in two days. After it filled up, the Java process threw `No space left on device`, and the service was down for six hours before anyone noticed. And the disk alert had never been configured at all—because that metric isn't included by default.
My first fix attempt was wrong: I added the CloudWatch Agent and created a `disk_used_percent` alert with an 85% threshold. A week later, looking back, the status was still `INSUFFICIENT_DATA`. After investigating for a long time, there were two reasons: in the Agent configuration file, the `disk` section only had `mount_points` written but not `resources`, so collection never took effect; and the alert's `treatMissingData` used the default value `missing`, so it always displayed gray instead of red.
The verification method is simple—just check whether the metric has data points:
```bash
aws cloudwatch get-metric-statistics --namespace CWAgent
--metric-name disk_used_percent --dimensions Name=InstanceId,Value=i-0abc1234567890def
--start-time 2026-03-15T00:00:00Z --end-time 2026-03-15T06:00:00Z
--period 300 --statistics Maximum
If it returns `Datapoints: []`, it isn't being collected—don't suspect the alert configuration. After that, I changed the process to: first run this command to confirm there are data points, then create the alert, and finally manually trigger it once (temporarily set the threshold to 1%) to confirm the notification chain works. Doing it in reverse order means two days of back-and-forth.
Another type of pitfall is in snapshots. The client required hourly backups of critical volumes. After I configured a high-frequency policy, I saw API errors in CloudWatch: `SnapshotCreationPerVolumeRateExceeded`—there's a cap on the concurrent snapshot creation rate per volume, and configuring too densely gets you throttled directly. Changing to hourly for critical volumes and daily for normal volumes, combined with cross-region replication, solved the problem. This error can't be found without looking at logs; the snapshot task just shows "partially completed" in the console.
## Backup Drills, Scaling, and Incident Response: Which Tasks Can Be Handed Over
For backups, the configuration itself isn't difficult—what's difficult is **drills**. I require that every drill must launch a new instance from a snapshot, attach the volumes, get the service running, and verify data integrity, with the entire process completed within 60 minutes to pass. Just checking that the DLM task status is `completed` and feeling reassured is a very common form of self-comfort—a successful snapshot doesn't mean it can be restored.
Scaling comes in two types. For stateless web tiers, put them in an Auto Scaling group with a target tracking policy on `ASG Average CPU 60%` and an ALB in front—this can be handed over, and changing parameters doesn't affect business logic. Stateful legacy monoliths can only be scaled vertically, and changing instance types inevitably involves a downtime window; such changes must be approved by the client's business side. The managed operations team can only propose plans and window suggestions—they cannot make the call themselves.
I generally tier incident response into three levels: P1 is site or core transaction unavailability—respond within 15 minutes and give a preliminary root cause assessment within 2 hours; P2 is noticeable performance degradation or non-core feature abnormality—respond within 1 hour; P3 is alert investigation and configuration optimization—handled via ticket the next day. The tier table must be written into the contract appendix; otherwise, being woken up at 3 a.m. to handle a "disk at 82%" alert will drive people crazy.
### When Not to Use This Approach
There are several scenarios where I don't recommend managed operations. First, when compliance requirements prohibit logs and data from leaving the country, or require operations personnel to have specific qualifications—an external team's access itself won't pass the review. Second, when the business has only one machine and annual cloud spend is a few thousand yuan—managed operations fees would cost more than the server, so it's better to have the developer manage it on the side. Third, when the client just wants "someone to answer the phone" and is unwilling to change any configuration, grant read-only permissions, or cooperate with change windows—in this case even the monitoring baseline can't be established, and taking it on means taking the blame. Fourth, if the root account and billing permissions are unwilling to be separated—just walk away.
If the team has no dedicated operations staff but genuinely needs to establish this monitoring and inspection system, awsgood.com can do an environment baseline review and managed configuration based on the handover checklist, getting the asset table, alert thresholds, and inspection checklist in place first, so there's a reference when you take over yourself later.
## FAQ
**How much does EC2 managed operations generally cost?** There's no standard price; it's usually quoted by instance count and whether 7×24 response is included. For within 10 instances, with only business-hours inspection and alert triage, it's in the range of several hundred to one or two thousand RMB per month; with 7×24 on-call and recovery drills, the price goes up by multiples. The specifics depend on response time and change frequency.
**Why aren't memory and disk monitoring included by default?** These two types of metrics are inside the operating system and not visible at the hypervisor level, so they must be collected from within the instance via the CloudWatch Agent. Therefore, the first thing to do when a new machine goes online is install the Agent and verify the metrics have data points—don't wait three months until you discover the alert has been gray.
**Where's the dividing line between self-managed and handing over to a managed team?** The criterion is "who has the authority to change things when something goes wrong." Anything requiring changes to architecture, budget, or account permissions stays in-house; anything highly repetitive with clear operating manuals (inspection, patching, snapshots, alert triage) is most cost-effective to hand over.
**Won't too few alerts cause incidents to be missed?** No. The value of alerts lies in accuracy, not quantity. My experience is to compress alerts to no more than 5 effective notifications per week—that's when the team will actually look at them. If you can't compress them, it's mostly because thresholds are set too low or evaluation periods are too short.
**After taking over, I find the previous handler left no documentation at all—how do I start?** Work backward from `describe-instances` and CloudTrail event history. CloudTrail can show who changed what recently, which is more reliable than asking people. Fill in the asset table as you go, and make no changes in the first week.
## References
- [Amazon CloudWatch User Guide: Using Alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html)
- [Amazon EC2 User Guide: Status Checks](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring-system-instance-status-check.html)
- [Amazon CloudWatch User Guide: Install and Configure the CloudWatch Agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html)
- [Amazon EC2 User Guide: Automate Snapshot Creation with Data Lifecycle Manager](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html)