Remote Access to EC2 instances, the easy (and secure) way

Getting a shell on an EC2 instance can be a pain. It’s a classic security vs convenience tradeoff: AWS best practices would have you put important infrastructure in private subnets with restricted routing and no Internet-facing IP addresses. But every step to secure an instance makes it less convenient to use. In an ideal world, nobody should ever need to get a shell on such an instance. But the real world is imperfect. If an important production issue needs debugging, sometimes getting a shell on an EC2 instance is the best way to get to the bottom of an issue. When that happens, you want authorised users to be able to get shell access quickly and conveniently.

So what would you think if I told you that I can safely, securely and quickly get access to almost any of my EC2 instances by opening a terminal and typing in: 

ssh ec2-user@i-07e9bd6d349754cd,eu-west-2

Flashback. Let’s see how we get there from where we are now.

The sysadmin’s toolkit: SSH

Early in EC2 life, the only way to get shell onto an EC2 instance was through SSH. You would register an SSH public key with AWS, tell AWS to install that key when you create the instance, and open port 22 on the security group. Then you can SSH in using the public IP address. There’s a lot to like about this scenario:

✓ Sysadmins have been using SSH for decades - it’s a well-known and well-understood tool

✓ We can also use SSH’s partner SCP, so it’s easy to copy files to and from the instance

But there are downsides:

✗ An SSH port on a public IP address will get scanned and credential-stuffed on a near-continuous basis. Your security logs will be polluted with constant noise.

✗ Somebody needs to manage the SSH keys. Do you manually update authorized_keys with every engineer’s public key? Or do you share a private key with your engineers, or manage an SSH certification authority? In short, how do you deal with onboarding new engineers, and making sure that those who have left no longer have access?

✗ If you are following security best practices, your instances won’t have public IP addresses. This makes SSH hard to use - you end up using a bastion and have to make two or more “hops” to reach the instance that you want, which takes away a lot of convenience.

Introducing Session Manager

Later on, Systems Manager arrived, and then it grew a service called Session Manager. If your instance has the Systems Manager agent installed, and suitable permissions in its instance profile, then you can go to Systems Manager, click on an instance, select Actions → Start Session, and you instantly have a terminal in your web browser with a shell on your chosen instance. Instances don’t need a public IP address, they just need to be able to reach the Systems Manager API endpoints. In practice, this means VPC endpoints, NAT gateways, or a public IP address.

This fixes a lot of problems we had before:

✓ We no longer need to manage users and/or SSH keys on the instance. Instead we use IAM to grant users access to the Session Manager API calls. If an employee leaves, their IAM account is removed and they lose access.

✓ Session Manager can reach instances deep inside private subnets that would normally be inaccessible or require a bastion.

But, again, there are downsides:

✗ We cannot use the ubiquitous SSH tools that we are used to.

✗ Nor can we use SCP, so transferring files has to be done in a different way (such as via an S3 bucket).

✗ The terminal in the web browser is a bit clunky. In my experience, selecting text and copy-and-paste was not very smooth.

Let’s see if we can fix some of these problems. First of all, let’s go back to the command line. awscli can interact with the Systems Manager APIs, and with a bit of help from the Session Manager Plugin for AWS CLI, we can start a session from our own terminal:

aws ssm start-session --target i-07e9bd6d3497545cd --region eu-west-2

OK, so we’ve knocked one item off our list of downsides. But the other problems are still all there. We want to use SSH and SCP, just like we’re used to.

Bringing them both together

Fortunately we have another Systems Manager feature that can help us. It can run a plugin which, instead of connecting the session to a shell, ties it to an SSH daemon. We can combine this with OpenSSH’ ProxyCommand feature to run an SSH session tunnelled through Session Manager. We get back our ability to run SSH and SCP, without having to open up an SSH port anywhere.

Here’s how to set it up. Open the file ~/.ssh/config in a text editor (create it if it doesn’t exist) and then add this text:

# SSH over Session Manager
host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Now you can run a command like this:

ssh ec2-user@i-07e9bd6d3497545cd

And you have a terminal onto the EC2 instance, using the tools you are familiar with! (Note that you may have to set the AWS_DEFAULT_REGION environment variable before running this command.)

At this point we’ve solved many of the disadvantages we’ve come across, but there’s still one left. SSH keys need to be managed even when tunnelled through Session Manager. This means that we have to run parallel authorisation systems - IAM permissions for allowing use of Session Manager, and SSH keys to then actually log on to the instance. If only there was a way to integrate SSH key management into something that could be managed by IAM.

Luckily for us, there is, and it’s in a different AWS service called EC2 Instance Connect. It grants access to SSH in response to an API call, which of course can be controlled by IAM permissions. It grants access by temporarily installing an SSH public key - you provide your SSH public key, and Instance Connect opens up access using the key for 60 seconds. You can then initiate the SSH key tunnelled through Session Manager. No messing around with authorized_keys.

Instance Connect requires a daemon to be installed on each instance. You get this by default on recent Amazon Linux 2 AMIs. For other Linux distributions you may need to install this daemon yourself. See the EC2 Instance Connect document for details.

If we have the necessary agents on our instance, and even if we assigned no SSH key to it at all, we can run a command like this:

aws ec2-instance-connect send-ssh-public-key --region eu-west-2 --instance-id i-07e9bd6d3497545cd --availability-zone eu-west-2b --instance-os-user ec2-user --ssh-public-key file://$HOME/.ssh/id_rsa.pub

And then, within 60 seconds, follow it with our SSH command:

ssh ec2-user@i-07e9bd6d3497545cd

Making it easy

So now we’ve addressed all of our complaints. We can use our favourite tools, SSH and SCP, in the terminal window we are comfortable with. We don’t need to manage SSH keys. We don’t need our instances to have public IP addresses or open ports in security groups. We don’t even need the instances to have access to the Internet. We use IAM permissions exclusively - we can grant access to individuals easily, and revoke them in as little as 60 seconds.

Finally, let's tidy things up a bit. Instead of having to run two commands to start a session, let’s combine them in a script. This script also contains some parsing so we can specify a region at the same time.

Here’s my script to make it easy to use. Save this as ssm-ssh-shim.sh, and then follow the instructions at the top. Happy remote-instance-management!

#!/usr/bin/env bash

# Put into .ssh/config like this:
# host i-* mi-*
# ProxyCommand sh -c "/path/to/ssm-ssh-thunk.sh '%r' '%h' '%p'"
#
# Then you can SSH-via-SessionManager using a command like this:
# ssh ec2-user@i-07e9bd6d3497545cd,eu-west-2
# SCP works too:
# scp ec2-user@i-07e9bd6d3497545cd,eu-west-2:.bash_history history
#
# Works on instances that have both amazon-ssm-agent and ec2-instance-connect installed (e.g. any recent Amazon Linux).
# Credentials for AWS supplied in the normal fashion (environment variables, ~/.aws/credentials, etc.)

set -e

user="$1"
target="$2"
port="$3"

if [[ "${target}" =~ (m?i-[0-9a-f]+)(,([a-z0-9-]+))? ]]; then
instance_id="${BASH_REMATCH[1]}"
[[ "${BASH_REMATCH[3]}" != "" ]] && export AWS_DEFAULT_REGION="${BASH_REMATCH[3]}"
else
echo >&2 "Could not parse: ${target}"
exit 1
fi

echo >&2 "$(tput setaf 5)get instance details$(tput setaf 7)"
# Look up the AZ of the instance
az="$( aws ec2 describe-instances --instance-id "${instance_id}" | jq -r '.Reservations[0].Instances[0].Placement.AvailabilityZone' )"

# Locate an SSH public key
for file in ~/.ssh/id_rsa.pub ~/.ssh/id_dsa.pub; do
if [ -f "${file}" ]; then
echo >&2 "$(tput setaf 5)add ephemeral key$(tput setaf 7)"
aws ec2-instance-connect send-ssh-public-key \
--instance-id "${instance_id}" \
--availability-zone "${az}" \
--instance-os-user "${user}" \
--ssh-public-key "file://${file}"
break
fi
done

echo >&2 "$(tput setaf 5)start session$(tput setaf 7)"
exec aws ssm start-session --target "${instance_id}" --document-name AWS-StartSSHSession --parameters portNumber="${port}"

Limitations

Throughout this post I’ve glossed over some of the areas where Session Manager has clear wins over an SSH-based solution. In particular, Session Manager’s ability to save a transcript of the terminal session is highly useful if you need to keep a close eye on your servers for audit and compliance purposes. In a future blog post I’ll go into details on a pure-Session Manager approach and its unique advantages.

And all of this is only valid if you are working on systems which make heavy use of SSH - which in 2020 basically means Linux machines. For Windows servers there’s a unique set of challenges, and again in a future blog post I’ll demonstrate how Session Manager can be integrated with RDP.