www.awsgood.com
ec2-instance-connect package and sshd's AuthorizedKeysCommand, and CloudTrail auditing. If any segment breaks, the user sees the same Permission denied (publickey), so the managed service contract must specify inspection items segment by segment, rather than vaguely saying “ensure SSH is available.”SendSSHPublicKey fails ≥3 times within 5 minutes triggers P2,” “EIC Endpoint status not available triggers P1, 15-minute response,” “once a quarter, run a recovery drill that rebuilds the Endpoint from scratch and completes within 30 minutes.” A managed service commitment without numbers cannot be reconciled when an incident occurs.t3.medium instances at once and 5 of them could not connect; the root cause was that the launch template mixed in an old AMI without ec2-instance-connect preinstalled.When the on-call phone rings, the alert text usually has only one sentence: “EC2 Instance Connect login failure rate increased.” Don't rush to open the console and click Connect; triage first.
0–2 minutes: confirm the blast radius. Clarify three things: browser console or CLI? Which region? What is the exact error text? The browser console and aws ec2-instance-connect ssh are two different failure surfaces—the former is affected by the instance's public IP, while the latter may just be expired local credentials. At the same time, pull the recent CloudTrail call records:
aws cloudtrail lookup-events
--region us-east-1
--lookup-attributes AttributeKey=EventName,AttributeValue=SendSSHPublicKey
--start-time "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)"
--query 'Events[].CloudTrailEvent' --output text
| jq -r 'fromjson | [.eventTime, .errorCode // "OK", .requestParameters.instanceId] | @tsv'
If there are large numbers of `AccessDenied` in the output, the problem is in IAM; if it is `InvalidParameter` or `EC2InstanceNotFoundException`, investigate instance status and the Endpoint.
**2–5 minutes: confirm the instance itself is alive.** Many “can't connect” cases are actually because the instance is already down:
```bash
aws ec2 describe-instance-status
--instance-ids i-0a1b2c3d4e5f67890
--region us-east-1 --include-all-instances
--query 'InstanceStatuses[0].{State:InstanceState.Name,Sys:SystemStatus.Status,Inst:InstanceStatus.Status}'
`Sys` is impaired, meaning there is a problem with the underlying host; go directly to the recovery process and don't waste time on SSH.
**5–10 minutes: switch to the backup channel to stop the bleeding.** In production, I generally require SSM Session Manager to be kept as a second channel. It uses a different path from Instance Connect (the instance-side agent initiates outbound connections), so when one is down, the other is often still alive:
```bash
aws ssm start-session --target i-0a1b2c3d4e5f67890 --region us-east-1
Once inside, the first thing is not to fix things but to confirm that business processes are still running and the disk is not full. Login channel failures and business failures must be handled separately; don't mix them in the same troubleshooting effort.
## Real-world test and pitfalls: an instance migrated from CentOS 7 kept spinning when clicking Connect
At 2:07 a.m. on November 18, 2026, the customer's on-call group reported “production instances cannot connect.” These machines were CentOS 7 migrated from the IDC using MGN that week; the AMI was custom-built by the customer and ran in a private subnet in us-east-1.
Clicking Connect in the console, the page spun for about 25 seconds and popped up:
There was a problem connecting to your instance. Common causes include
the instance not having a public IP address, a security group not allowing
inbound SSH traffic, or the instance not having the EC2 Instance Connect
package installed.
**Where the first judgment was wrong:** The error text put “security group” first, so our first reaction was to allow port 22 and even attached a `0.0.0.0/0` rule. It still couldn't connect after the change. This made two mistakes—changing the security group before locating the issue, and conveniently opening a world-open rule, which cost time later during the postmortem to close.
**How we located it:** The key was to check whether sshd received the connection. On the instance (accessed via a jump host), run:
```bash
sudo tail -f /var/log/secure # CentOS/RHEL; Ubuntu is /var/log/auth.log
The criterion is straightforward: when the client clicks Connect, if there is not a single new record in the log, the TCP connection never reached sshd, and the problem is at the network layer or the Endpoint side; if there is a record followed by `Permission denied (publickey)`, the problem is at the authentication layer, most likely the package is not installed or `AuthorizedKeysCommand` is not configured.
The connection we clicked at 02:23:41 that night: the last entry in the `secure` log was still the cron record at 02:19. Combined with `describe-instances`, `PublicIpAddress` was empty—the truth was that these machines were in a private subnet and had no public IP at all. The console's Instance Connect connects back over the public network, so it was bound to fail.
**How it was finally solved:** Create an EC2 Instance Connect Endpoint for the subnet, allow only port 22 from the endpoint security group in the security group, and install the package on the instance:
```bash
sudo yum install -y ec2-instance-connect
sudo systemctl restart sshd
grep -E 'AuthorizedKeysCommand' /etc/ssh/sshd_config
You should normally see two lines:
AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect
If not, add them manually and restart sshd. Verifying with the CLI is more reliable:
```bash
aws ec2-instance-connect ssh
--instance-id i-0a1b2c3d4e5f67890
--os-user ec2-user --connection-type eice --region us-east-1
After this path was working, we wrote “instances in private subnets must always use EIC Endpoint” into the delivery standard. Later, doing the math: that night took about 47 minutes from alert to recovery, of which 20 minutes were wasted changing security groups.
## Daily inspection and alert thresholds: 6 points worth watching on this path
Inspection is not “log in and take a look every day”; that's just ceremonial. What really needs watching is the things that fail silently