Why SSH certificates are awesome

Big Tech companies such as Facebook, Google, Netflix, Uber with thousands of users and servers use SSH Certificates to manage access. This post will go over why that’s the case

What is SSH?

SSH is a protocol used by system administrators and developers to communicate securely with a server. Authentication can be done with SSH keys or passwords.

Passwords and why they’re not recommended

We all know passwords and have used them at some point. Those secret string of characters you have to remember to prove your identity.

Passwords are good and better than nothing, but securing a server with just a password is not recommended especially on critical systems.

1. Weak passwords can be brute-forced
2. Sometimes passwords have to be shared for group accounts.
3. Only one compromised server is enough to phish all user credentials.
4. Revocation of compromised user passwords is not easy or convenient
5. Without a second factor, they are a single point of failure

You can improve the security by using tools like fail2ban to block IP addresses that have multiple failed login attempts.

Why use keys?

Keys are the default authentication method on most cloud providers eg: Digital Ocean. There is a good reason for this. Keys are more secure than passwords

SSH keys are a matching set of cryptographic keys which can be used for authentication.

Each set contains a public and a private key. The public key can be shared freely without concern, while the private key must be vigilantly guarded and never exposed to anyone. source

Why SSH Keys are more secure

• One user account can have multiple keys. This makes auditing easier
• Keys can be password protected. Note the password is just for unlocking the key and never sent to the server
• Keys can be combined with passwords
• Keys can be revoked

Keys are created with the ssh-keygen command.

When creating keys, it is a good idea to encrypt them with a password. So that even if the key is leaked, the password will add another layer of security.

Managing SSH keys

To connect to a server using keys, a user’s public key has to be in the authorized_keys file on every server they want to connect to.

This method of managing keys is the most common and works okay for small teams but it can be a burden for large organizations with many users.

But growth creeps upon you, and before you know it you are securing significant assets with sub-standard methods

Key management issues

• The authorized_keys file on all servers has to be maintained regularly
• Issuing a new key is a long process
• Losing a key locks you out

The task of updating the authorized_keys file can and should be automated for large companies. This method “works” but there is a much better alternative

SSH Certificates

This is where public keys are signed by a Certificate Authority(CA). This method scales very well across thousands of machines. It can be used for both user and host verification. This method eliminates the Trust on first use requirement.

Advantages of certificates

SSH certificates are underrated and awesome in my opinion. They have some good features:

• Server configuration is done only once
• Updating the authorized_keys file on every server is not needed.
• Audits are easier since every key is unique
• Works perfectly for giving users access to offline/air-gapped servers
• Keys can expire

Available Metadata

The signed key can contain the following useful information

1. Key Expiry date & time

This is how long a key is valid. The value can range from minutes to years. Netflix set their certificates to expire after two minutes. That’s enough time for a developer to login into a server and start an ssh session. Note that the session does not terminate when a certificate expires, only when a user logs out

2. Principals

Principals control which user accounts and hosts a user has access to. For example, there can be a root_staging principal which allows users to login as root on a staging server

You can Parse this data offline with ssh-keygen -Lf /Path/To/Signed_Cert

You can also use this site

Open Source Implementations

There are many solutions to giving users SSH access. All these are available on GitHub. Note that documentation is not that good for some tools though

1. Keybase Certificate Authority bot

This is a keybase chatbot that signs public keys. it is cli based. The bot doesn’t need any ports open.

• Used internaly @ Keybase
• Only supports keybase Authentication
• Good Documentation

Blog post on keybase SSH

2. Cashier

• Controlled via a web interface
• Integrates well with LDAP or Active Directory
• Google, Github or Gitlab login available
• Key revocation page

Video on The Tool

3. BLESS - By Netflix

• Used internally @ Netflix,
• Runs on AWS Lambda
• Video on the tool

4. SSHrimp - by Stoggi

• Runs on AWS Lambda
• Authenticate with OpenID connect
• Still in Beta

5. Keymaster - Symantec

• Supportts Oauth2 login
• Integrates well with LDAP/Active Directory
• Made by Symantec
• Documentation

More Tools

• Smallstep CA
• Uber’s PAM module

Video Resources

For those that prefer videos to blog posts

• Public key Cryptography Explained
• The Key Exchange Problem
• Conference Talks on SSH Certificates

Further Reading

I’m not the first one to write about this topic, so here’s related content covering it

Blog Posts

• How SSH is secured At Facebook
• SSH at scale - CAs and Principals
• How to SSH Properly
• If you’re not using SSH certificates you’re doing SSH wrong

Whitepapers and Manuals

• Redhat Deployment Guide
• NIST whitepaper on SSH automation
• IETF Memo on recomended practice
• ssh-keygen manual

Image Credits

Illustrations from Stories by Freepik