Hands-on AWS knowledge base for developers Managed Ops
Talk to Us

What Should an Application Load Balancer (ALB) Managed Operations Service Include? A Drill Checklist

Export PDF Print-friendly layout with a copyright watermark; choose Save as PDF in the print dialog
What Should an Application Load Balancer (ALB) Managed Operations Service Include? A Drill Checklist

Conclusion First

  1. The core of ALB managed operations is four things: metrics and alarm thresholds, a regular inspection checklist, failure drill and response processes, and configuration verification before and after changes. "Watching the console for you" does not count as managed operations.
  2. Set thresholds based on the business; don't copy default values. These three must be configured: HTTPCode_ELB_5XX_Count, UnHealthyHostCount, and p99 of TargetResponseTime; choose the rest based on traffic volume.
  3. What is worth outsourcing is repetitive judgment (alarm classification, certificate and rule inspections, drill execution); architecture decisions and production change approvals must stay in your own hands.

Why proactively create failures: most ALB failures are "half-down"

ALB instances themselves in managed regions such as us-east-1 and ap-northeast-1 rarely go completely down. The problems that really get people out of bed at midnight almost all occur in configuration and backend interactions: incorrect health check judgments, mismatched timeouts, instances mixed into target groups that never came up, weight switches not taking effect.

This type of failure is characterized as "half-down"—one of three backends is down, and what you see is "occasionally a bit slow" or "occasionally an error," not a full-site outage. By the time the alarm goes off on its own, it has often been running for two weeks. I usually tell customers: this type of problem cannot be detected by testing; you can only inject it proactively.

The idea of chaos engineering is not that mysterious: first define a steady state (e.g., p99 250ms, 5XX below 0.1%), then inject a controllable failure and see whether the steady state is broken. If it breaks, it is an architecture problem; if it does not break, consider it insurance purchased.

Real-world testing and pitfalls: long-tail 502s from keep-alive mismatch

In March 2026, a customer's API was behind an ALB in ap-northeast-1, with three m5.xlarge backends running Node.js 18. I was woken by an alarm at 2 a.m.; on CloudWatch, HTTPCode_ELB_5XX_Count was 2–5 every 5 minutes. Strangely, HTTPCode_Target_5XX_Count remained 0 the whole time.

This contrast is key: there are 5XX on the ELB side and none on the target side, meaning the request was never properly processed by the backend—the problem is at the connection layer. Grepping the access logs produced this: elb_status_code=502 target_status_code=- target_processing_time=-1.

Our first configuration change went in the wrong direction. At the time, we judged that health checks were too strict and causing false removal, so we changed interval from 30s to 10s and unhealthy_threshold from 3 to 2. As a result, there was not one fewer 502; instead, there was one more outage—a machine was directly removed after a single 3-second Full GC, and that machine's cache had not warmed up yet that day, so after traffic returned, latency fluctuated for ten minutes.

The real root cause was found using the time field in the access logs. Sorting the 502 requests by time, we found that the interval between two consecutive requests to the same target was concentrated at 55–62 seconds. SSHing in to check the Node configuration: server.keepAliveTimeout defaults to 5000ms, while the ALB idle timeout is 60 seconds. The ALB side thought the connection was still alive; Node had already closed it unilaterally, and reuse directly got an RST.

The fix is to make the backend live a little longer than the ALB:

// server.js, observe for one week after restart; ELB-side 502s drop to zero
server.keepAliveTimeout = 65000;   // > ALB idle timeout 60s
server.headersTimeout = 66000;


At the same time, I changed the health check parameters back to `interval 15s / timeout 5s / unhealthy_threshold 3`. This pitfall is basically impossible to reproduce in local load testing—during load testing, connections do not stay idle for 60 seconds, so that path is never hit.

One command used during troubleshooting; I suggest saving it as an alias on the bastion host:

```bash
aws elbv2 describe-target-health 
  --target-group-arn arn:aws:elasticloadbalancing:ap-northeast-1:123456789012:targetgroup/api-tg/1a2b3c4d5e6f7890 
  --region ap-northeast-1 
  --query 'TargetHealthDescriptions[].{Target:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason}' 
  --output table


## What the managed team should watch, and which work should be outsourced

Metrics are not about quantity; they are about having an action attached to every threshold. The following table is the default set we configure for customers, with scale referenced to a small business with around 500,000 daily requests:

| Metric | Suggested threshold | First action after trigger |
|---|---|---|
| `UnHealthyHostCount` | ≥ 1 for 2 consecutive 1-minute periods | Pull target group health reasons; distinguish between application not ready / incorrect check path |
| `HTTPCode_ELB_5XX_Count` | Proportion > 1% within 5 minutes, and `Target_5XX` is 0 | Check access logs, focusing on `target_status_code=-` |
| `HTTPCode_Target_5XX_Count` | > 20 within 5 minutes | Pull application logs; the ALB side is basically not responsible |
| `TargetResponseTime` p99 | > 2s for 5 consecutive minutes | Check whether a single instance is dragging things down; compare with other instances in the same target group |
| `RejectedConnectionCount` | > 0 | Target group capacity or backend accept queue is full |
| ACM certificate remaining days | < 30 days | Check whether the DNS validation record is still there |

Inspections are separated into daily and weekly. Daily: check whether there are "zombie" targets in the target group (instances already deleted but status still present), certificate remaining days, and the 5XX distribution over the past 24 hours. Weekly: check whether the number of listener rules is approaching the quota (100 rules per ALB by default; an increase can be requested), LCU usage trends, whether WAF block volume has changed abruptly, and the relationship between `deregistration delay` and release duration—the default is 300 seconds; if a rolling release stalls for five minutes, this is the cause, not the release system.

Which work is suitable to hand over; this table is fairly straightforward:

| Work item | Suggested ownership | Reason |
|---|---|---|
| Alarm classification and first-line response | Managed team | Highly repetitive; solved with process and on-call schedule |
| Certificate, rule, and quota inspections | Managed team | Clear check items; high cost of missed inspections |
| Drill execution and observation records | Managed team | Requires patience and fixed scripts |
| Listener rule and target group changes | Co-managed, executed by managed team | Change ticket + two-person review |
| Architecture selection, cross-region strategy | Yourself | Involves cost and business expectations |
| Production change window approval | Yourself | Responsibility cannot be outsourced |

On cost: the fixed hourly fee for one ALB in us-east-1 is on the order of a dozen-plus USD per month, plus LCU, which varies with region and usage; refer to the official AWS pricing page. Managed service fees are generally priced by number of ALBs and number of alarms, with little relation to traffic.

## Low-risk chaos drill starter checklist

Don't start by unplugging the network cable. Follow this order; the first drill does not require stopping anything:

1. Run a 7-day baseline first. Enable ALB access logs (landing in S3 is enough), and record p50/p99, 5XX ratio, and the normal fluctuation range of HealthyHostCount. Without a baseline, there is no judgment standard.
2. First experiment: remove 1/3 of the targets from the target group; do not stop the instances, just `deregister-targets`. Observe for 10 minutes.
3. Second experiment: in a weighted target group, switch the weight of a rule from 100 to 0, and verify how long it takes for the rollback path to take effect.
4. Third experiment: simulate all backends being down, confirm whether ALB returns 503 or 502, how long it takes for the alarm to be pushed to the group, and how long it takes for on-call to respond.

For containerized environments, you can use FIS directly; it is cleaner than manually removing machines:

```json
{
  "description": "stop-34pct-api-tasks",
  "targets": {
    "apiTasks": {
      "resourceType": "aws:ecs:task",
      "selectionMode": "PERCENT(34)",
      "parameters": {
        "cluster": "prod-api-cluster",
        "service": "api"
      }
    }
  },
  "actions": {
    "stopTasks": {
      "actionId": "aws:ecs:stop-task",
      "targets": { "Tasks": "apiTasks" }
    }
  },
  "stopConditions": [{
    "source": "aws:cloudwatch:alarm",
    "value": "arn:aws:cloudwatch:ap-northeast-1:123456789012:alarm:alb-5xx-burn"
  }],
  "roleArn": "arn:aws:iam::123456789012:role/fis-experiment-role"
}


`stopConditions` should not be omitted; this is the only gate between a drill and an incident.

Observe only three metrics: whether p99 exceeds 1.5 times the baseline, whether the 5XX ratio exceeds 1%, and the time from alarm trigger to a person receiving the message. Our internal requirement is to enter the IM group within 2 minutes; exceeding 2 minutes is considered a process defect, not a people problem.

### When not to use this approach

If there is only one backend and no redundancy, do not run machine-removal drills;

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *