What Should EC2 Instance Connect Managed Operations Include? A Checklist You Can Use at 3 A.M.
Export PDF
Print-friendly layout with a copyright watermark; choose Save as PDF in the print dialog

First, the conclusion
- The core object of managed operations is a path, not a machine. From clicking “Connect” to getting a shell, EC2 Instance Connect spans five segments: IAM permissions, EIC Endpoint (or the instance public IP), security groups, the in-instance
ec2-instance-connectpackage and sshd'sAuthorizedKeysCommand, and CloudTrail auditing. If any segment breaks, the user sees the samePermission denied (publickey), so the managed service contract must specify inspection items segment by segment, rather than vaguely saying “ensure SSH is available.” - The checklist must be granular enough to be accepted. For example, “
SendSSHPublicKeyfails ≥3 times within 5 minutes triggers P2,” “EIC Endpoint status not available triggers P1, 15-minute response,” “once a quarter, run a recovery drill that rebuilds the Endpoint from scratch and completes within 30 minutes.” A managed service commitment without numbers cannot be reconciled when an incident occurs. - Only changing the login method without touching the AMI and security groups is equivalent to postponing the failure until the next scale-out. We have seen Auto Scaling launch 6
t3.mediuminstances at once and 5 of them could not connect; the root cause was that the launch template mixed in an old AMI withoutec2-instance-connectpreinstalled.
Alerted at 2:07 A.M.: do these in this order in the first 10 minutes
When the on-call phone rings, the alert text usually has only one sentence: “EC2 Instance Connect login failure rate increased.” Don't rush to open the console and click Connect; triage first.
0–2 minutes: confirm the blast radius. Clarify three things: browser console or CLI? Which region? What is the exact error text? The browser console and aws ec2-instance-connect ssh are two different failure surfaces—the former is affected by the instance's public IP, while the latter may just be expired local credentials. At the same time, pull the recent CloudTrail call records:
aws cloudtrail lookup-events
--region us-east-1
--lookup-attributes AttributeKey=EventName,AttributeValue=SendSSHPublicKey
--start-time "$(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ)"
--query 'Events[].CloudTrailEvent' --output text
| jq -r 'fromjson | [.eventTime, .errorCode // "OK", .requestParameters.instanceId] | @tsv'
If there are large numbers of `AccessDenied` in the output, the problem is in IAM; if it is `InvalidParameter` or `EC2InstanceNotFoundException`, investigate instance status and the Endpoint.
**2–5 minutes: confirm the instance itself is alive.** Many “can't connect” cases are actually because the instance is already down:
```bash
aws ec2 describe-instance-status
--instance-ids i-0a1b2c3d4e5f67890
--region us-east-1 --include-all-instances
--query 'InstanceStatuses[0].{State:InstanceState.Name,Sys:SystemStatus.Status,Inst:InstanceStatus.Status}'
`Sys` is impaired, meaning there is a problem with the underlying host; go directly to the recovery process and don't waste time on SSH.
**5–10 minutes: switch to the backup channel to stop the bleeding.** In production, I generally require SSM Session Manager to be kept as a second channel. It uses a different path from Instance Connect (the instance-side agent initiates outbound connections), so when one is down, the other is often still alive:
```bash
aws ssm start-session --target i-0a1b2c3d4e5f67890 --region us-east-1
Once inside, the first thing is not to fix things but to confirm that business processes are still running and the disk is not full. Login channel failures and business failures must be handled separately; don't mix them in the same troubleshooting effort.
## Real-world test and pitfalls: an instance migrated from CentOS 7 kept spinning when clicking Connect
At 2:07 a.m. on November 18, 2026, the customer's on-call group reported “production instances cannot connect.” These machines were CentOS 7 migrated from the IDC using MGN that week; the AMI was custom-built by the customer and ran in a private subnet in us-east-1.
Clicking Connect in the console, the page spun for about 25 seconds and popped up:
There was a problem connecting to your instance. Common causes include
the instance not having a public IP address, a security group not allowing
inbound SSH traffic, or the instance not having the EC2 Instance Connect
package installed.
**Where the first judgment was wrong:** The error text put “security group” first, so our first reaction was to allow port 22 and even attached a `0.0.0.0/0` rule. It still couldn't connect after the change. This made two mistakes—changing the security group before locating the issue, and conveniently opening a world-open rule, which cost time later during the postmortem to close.
**How we located it:** The key was to check whether sshd received the connection. On the instance (accessed via a jump host), run:
```bash
sudo tail -f /var/log/secure # CentOS/RHEL; Ubuntu is /var/log/auth.log
The criterion is straightforward: when the client clicks Connect, if there is not a single new record in the log, the TCP connection never reached sshd, and the problem is at the network layer or the Endpoint side; if there is a record followed by `Permission denied (publickey)`, the problem is at the authentication layer, most likely the package is not installed or `AuthorizedKeysCommand` is not configured.
The connection we clicked at 02:23:41 that night: the last entry in the `secure` log was still the cron record at 02:19. Combined with `describe-instances`, `PublicIpAddress` was empty—the truth was that these machines were in a private subnet and had no public IP at all. The console's Instance Connect connects back over the public network, so it was bound to fail.
**How it was finally solved:** Create an EC2 Instance Connect Endpoint for the subnet, allow only port 22 from the endpoint security group in the security group, and install the package on the instance:
```bash
sudo yum install -y ec2-instance-connect
sudo systemctl restart sshd
grep -E 'AuthorizedKeysCommand' /etc/ssh/sshd_config
You should normally see two lines:
AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect
If not, add them manually and restart sshd. Verifying with the CLI is more reliable:
```bash
aws ec2-instance-connect ssh
--instance-id i-0a1b2c3d4e5f67890
--os-user ec2-user --connection-type eice --region us-east-1
After this path was working, we wrote “instances in private subnets must always use EIC Endpoint” into the delivery standard. Later, doing the math: that night took about 47 minutes from alert to recovery, of which 20 minutes were wasted changing security groups.
## Daily inspection and alert thresholds: 6 points worth watching on this path
Inspection is not “log in and take a look every day”; that's just ceremonial. What really needs watching is the things that fail silently
Sources
- EC2 Instance Connect
- Set up EC2 Instance Connect
- Troubleshoot EC2 Instance Connect
- Connect to your instances using EC2 Instance Connect Endpoint
- Logging data events with CloudTrail
推荐
推荐服务 · 一站式办理
由云市场认证服务商承接,下单后按需沟通交付。
腾讯云市场
NGINX / APACHE / TOMCAT / IIS 证书配置 · SSL证书安装
各服务器环境 SSL 证书安装配置,教程看不懂?直接交给服务商办。
查看服务 →
阿里云市场
代购 SSL 证书 · DigiCert / GeoTrust / RapidSSL 单域名 DV
单域名 DV 证书代购,主流品牌可选,比官网渠道更省。
查看服务 →
阿里云市场
Wildcard 通配符 SSL 证书安装部署续费申请
一张通配符保护全部一级子域名,含安装部署与续费服务。
查看服务 →
腾讯云市场
SSL证书申请+配置 · 宝塔/WDCP 面板全站加密(OV 企业单域名等规格)
付费证书云上签发、一键部署到宝塔/WDCP 站点,OV 企业单域名可选。
查看服务 →
阿里云市场
企业级 EV / OV 证书与代码、驱动签名服务
SSL EV/OV 证书 + 软件代码签名、驱动签名,企业级一站式办理。
查看服务 →
阿里云市场
GlobalSign 代码签名证书服务(OV/EV 企业版)
对 exe/dll/驱动等软件做数字签名,消除「未知发布者」警告,企业版含申请代办。
查看服务 →
阿里云市场
HTTPS证书申请 · SSL证书安装部署 · 证书续费
证书过期、部署报错、需要续期?申请+部署+续费一站式办理。
查看服务 →
Leave a Reply