Hands-on AWS knowledge base for developers Managed Ops
Talk to Us

What Should EC2 Instance Connect Managed Operations Include? A Checklist You Can Use at 3 A.M.

Export PDF Print-friendly layout with a copyright watermark; choose Save as PDF in the print dialog
What Should EC2 Instance Connect Managed Operations Include? A Checklist You Can Use at 3 A.M.

First, the conclusion

  • The core object of managed operations is a path, not a machine. From clicking “Connect” to getting a shell, EC2 Instance Connect spans five segments: IAM permissions, EIC Endpoint (or the instance public IP), security groups, the in-instance ec2-instance-connect package and sshd's AuthorizedKeysCommand, and CloudTrail auditing. If any segment breaks, the user sees the same Permission denied (publickey), so the managed service contract must specify inspection items segment by segment, rather than vaguely saying “ensure SSH is available.”
  • The checklist must be granular enough to be accepted. For example, “SendSSHPublicKey fails ≥3 times within 5 minutes triggers P2,” “EIC Endpoint status not available triggers P1, 15-minute response,” “once a quarter, run a recovery drill that rebuilds the Endpoint from scratch and completes within 30 minutes.” A managed service commitment without numbers cannot be reconciled when an incident occurs.
  • Only changing the login method without touching the AMI and security groups is equivalent to postponing the failure until the next scale-out. We have seen Auto Scaling launch 6 t3.medium instances at once and 5 of them could not connect; the root cause was that the launch template mixed in an old AMI without ec2-instance-connect preinstalled.

Alerted at 2:07 A.M.: do these in this order in the first 10 minutes

When the on-call phone rings, the alert text usually has only one sentence: “EC2 Instance Connect login failure rate increased.” Don't rush to open the console and click Connect; triage first.

0–2 minutes: confirm the blast radius. Clarify three things: browser console or CLI? Which region? What is the exact error text? The browser console and aws ec2-instance-connect ssh are two different failure surfaces—the former is affected by the instance's public IP, while the latter may just be expired local credentials. At the same time, pull the recent CloudTrail call records:

aws cloudtrail lookup-events 
  --region us-east-1 
  --lookup-attributes AttributeKey=EventName,AttributeValue=SendSSHPublicKey 
  --start-time "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" 
  --query 'Events[].CloudTrailEvent' --output text 
  | jq -r 'fromjson | [.eventTime, .errorCode // "OK", .requestParameters.instanceId] | @tsv'


If there are large numbers of `AccessDenied` in the output, the problem is in IAM; if it is `InvalidParameter` or `EC2InstanceNotFoundException`, investigate instance status and the Endpoint.

**2–5 minutes: confirm the instance itself is alive.** Many “can't connect” cases are actually because the instance is already down:

```bash
aws ec2 describe-instance-status 
  --instance-ids i-0a1b2c3d4e5f67890 
  --region us-east-1 --include-all-instances 
  --query 'InstanceStatuses[0].{State:InstanceState.Name,Sys:SystemStatus.Status,Inst:InstanceStatus.Status}'


`Sys` is impaired, meaning there is a problem with the underlying host; go directly to the recovery process and don't waste time on SSH.

**5–10 minutes: switch to the backup channel to stop the bleeding.** In production, I generally require SSM Session Manager to be kept as a second channel. It uses a different path from Instance Connect (the instance-side agent initiates outbound connections), so when one is down, the other is often still alive:

```bash
aws ssm start-session --target i-0a1b2c3d4e5f67890 --region us-east-1


Once inside, the first thing is not to fix things but to confirm that business processes are still running and the disk is not full. Login channel failures and business failures must be handled separately; don't mix them in the same troubleshooting effort.

## Real-world test and pitfalls: an instance migrated from CentOS 7 kept spinning when clicking Connect

At 2:07 a.m. on November 18, 2026, the customer's on-call group reported “production instances cannot connect.” These machines were CentOS 7 migrated from the IDC using MGN that week; the AMI was custom-built by the customer and ran in a private subnet in us-east-1.

Clicking Connect in the console, the page spun for about 25 seconds and popped up:


There was a problem connecting to your instance. Common causes include
the instance not having a public IP address, a security group not allowing
inbound SSH traffic, or the instance not having the EC2 Instance Connect
package installed.


**Where the first judgment was wrong:** The error text put “security group” first, so our first reaction was to allow port 22 and even attached a `0.0.0.0/0` rule. It still couldn't connect after the change. This made two mistakes—changing the security group before locating the issue, and conveniently opening a world-open rule, which cost time later during the postmortem to close.

**How we located it:** The key was to check whether sshd received the connection. On the instance (accessed via a jump host), run:

```bash
sudo tail -f /var/log/secure   # CentOS/RHEL; Ubuntu is /var/log/auth.log


The criterion is straightforward: when the client clicks Connect, if there is not a single new record in the log, the TCP connection never reached sshd, and the problem is at the network layer or the Endpoint side; if there is a record followed by `Permission denied (publickey)`, the problem is at the authentication layer, most likely the package is not installed or `AuthorizedKeysCommand` is not configured.

The connection we clicked at 02:23:41 that night: the last entry in the `secure` log was still the cron record at 02:19. Combined with `describe-instances`, `PublicIpAddress` was empty—the truth was that these machines were in a private subnet and had no public IP at all. The console's Instance Connect connects back over the public network, so it was bound to fail.

**How it was finally solved:** Create an EC2 Instance Connect Endpoint for the subnet, allow only port 22 from the endpoint security group in the security group, and install the package on the instance:

```bash
sudo yum install -y ec2-instance-connect
sudo systemctl restart sshd
grep -E 'AuthorizedKeysCommand' /etc/ssh/sshd_config


You should normally see two lines:


AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect


If not, add them manually and restart sshd. Verifying with the CLI is more reliable:

```bash
aws ec2-instance-connect ssh 
  --instance-id i-0a1b2c3d4e5f67890 
  --os-user ec2-user --connection-type eice --region us-east-1


After this path was working, we wrote “instances in private subnets must always use EIC Endpoint” into the delivery standard. Later, doing the math: that night took about 47 minutes from alert to recovery, of which 20 minutes were wasted changing security groups.

## Daily inspection and alert thresholds: 6 points worth watching on this path

Inspection is not “log in and take a look every day”; that's just ceremonial. What really needs watching is the things that fail silently

Sources

Leave a Reply

Your email address will not be published. Required fields are marked *