For the majority of my personal computing history, I have only had one machine to work or play from. This changed a few years ago when I was able to cobble together a decent Gen 6 Intel i7 based on some used and new parts. I now had a machine for the office, and one for the basement. This meant I now need to manage 2 ssh keys. Fast forward a few more years, and now I have 3 MacOS laptops, the same trusty desktop in my office, and an AMD nuc style system for the basement (latter two running Archlinux).

Managing that many SSH keys means I am replicating effort, and increasing the probability I might make a mistake. Additionally creating new keys for each new machine doesn't scale well when I treat my laptops as interchangeable compute terminals. I currently try to use a YubiKey as my primary MFA flow for any service that supports it. Additionally I find a YubiKey is a great way to store TOTP codes that don't live in my password manager. I remembered when reading over some of the YubiKey docs a while back that there was a way to support SSH auth flows. This seemed like a potentially great route for me, hardware backed, pin-code secured portable keys!

Value Add

The more machines I need to manage, the less customization I wish to do to each one. This is because I feel I should be able to move from machine to machine and be productive in a short amount of time. With the way laptops increasing must be shipped off for repair, this has come in handy more than once. Traditionally I might need to generate an SSH key on each system. Authenticate to various servers, services, etc and then upload my new public key to each. This would cause friction where there need not be.

Using SSH keys is just another from of MFA where the private key is the something you have, and the password protecting it is the something you know. It's common however for folks to use their systems keychain to help manage the SSH agent service. Fundamentally this requires you to keep the key unlocked and in memory. For a single user system where you are the only one with root/admin, this isn't the worst. However I don't particularly like this auth flow if I am using a machine managed by someone else (e.g. an employer).

via GIPHY

Using a YubiKey to store your SSH private key would solve both the machine admin access and memory dump attacks. Fortunately I found the process easy enough to get working well, to the point where it has become my default SSH auth flow.

Cardinal Directions

There are 4 main ways Yubico supports SSH auth flows.

• Personal Identify Verification (PIV)
• OpenPGP (PGP)
• One-Time Password (OTP)
• Fast IDentity Online (FIDO2)

Each way has it's own benefits and trade-offs. A quick review of the differences will help us decide the path forward.

PIV

This flow stores and manages the asymmetric keys within the PIV application through the PKCS#11 interface. A main benefit to this flow is centralized key management. Meaning if you lost your YubiKey, it's easily replaced with another as the keys are centrally managed.

As an end user looking to have a single key stored on my YubiKey this is not the right approach for me.

Additional Docs

PGP

This auth flow is good for developers who wish to use the same key for authentication and code/commit signing, while already having PGP or GnuPG already setup. Somewhat similar to PIV but supporting a different set of standards. There are some limits with master/sub keys, but it is easy to backup the private key. Additionally due to PGP/GPG wide support, this flow works on Windows.

As someone who want's the private key stored on the YubKey itself, and does not need to auth to Windows, this is not the right approach for me.

Additional Docs

OTP

Server admins can use the Yubico PAM module to enable the use of Yubikey OTP codes. This is not considered as secure as asymmetric keys, and Yubico indicates this is not a preferred method because of EOL timelines.

Due to the lack of future support, and the requirement to have admin configuration on the server side, this is not the right approach for me.

Additional Docs

FIDO2

FIDO2 is a newer standard, being rapidly adopted in the form of Passkeys and WebAuthn flows. Passkeys have become increasingly popular and are well supported on Android, Desktop Linux, and MacOS. This flow allows a user to use a secret stored on the YubiKey to generate the private key and derive a public key. Additionally the YubiKey can be secured with a PIN code, protecting it from loss or theft. There are some gotchas for MacOS and a lack of support for Windows at the time. However the main draw is that it continues to work side by side with existing FIDO2/WebAuthn sites.

via GIPHY

Given the Private Key never actually leaves the YubiKey and I can easily manage it locally, while accessing the systems I need makes this the right approach for me.

Additional Docs

FIDO2 was his Name-o

Using the official docs (linked above) as our reference, we can quickly get our systems setup with this new SSH auth flow using FIDO2 based keys, secured with a PIN.

Software Requirements

• YubiKey firmware >= 5.2.3
• OpenSSH >= 8.3
• Yubico Authenticator >= 7.1
• A terminal

MacOS Shenanigans

While the current version of OpenSSH shipped with MacOS does support security keys, it does not have FIDO2 support. The easiest and most secure way to fix this is to install the full distribution of openssh from the brew repository.

brew upgrade
brew install openssh

Linux Friction

Ensure that you have libfido2 is installed on your system.

Passkey Prep

Launch the Yubico Authenticator app with your YubiKey plugged into your system.

On the left hand side of the app, select the icon in the middle that looks like a humanoid with a key, to select the Passkeys configuration screen.

Set a pin.

NOTE: I was also able to set a PIN on the YubiKey using Pixel phone by plugging it into the USB-C port.

The Keymaker

Use ssh-keygen to generate a new security-key compatible ssh key

ssh-keygen -t ed25519-sk -O resident -O verify-required -C "Main Key"

Lets review these flags.

• -t ed25519-sk - Sets the key type to use the ed25519 elliptical curve, with the sub type of a security key
• -O resident - Specifies that we wish to store the private key on the security key
• -O verify-required - Specifies that we wish to have the private key unlocked only with a PIN
• -C "Main Key" - Add a note to your key so that you can identify it if you create more than one

Once you execute the command you will encounter an interactive flow on the command line.

• You will be first prompted for your PIN
• Next you will be asked to touch the YubiKey to confirm you are physically present
• You will Asked for a password to lock the private key.
• Leave it blank for a better XP. The private key file just points to the real private key stored on your YubiKey and can not be used by itself. By setting a pin we are securing access to the private key
• Set the path for the local copies.
• Default path in .ssh/ should be fine and not override your previous keys as the new keys will have _sk in the name

NOTE: I have not tested setting a password on the private key for this flow. The private key on your machine is only a pointer to the smartkey

Set the file permission in order satisfy certain security checks

chmod 600 ~/.ssh/id_ed25519_sk

Additional keys can be added with a given UID

ssh-keygen -t ed25519-sk -O resident -O application=ssh:<UID> -C "My Comment"

Master and Commander

SSH Agent

There is no need to use an SSH agent, because you will no longer be storing an unlocked copy of the private key in memory. Instead you will use the PIN + tap to unlock the private key. This is a significantly improved security model as the private key then can not be dumped from memory.

If you currently are using an agent, this will end up being the default key for most situations. To avoid this, disable your SSH agent software.

Remote Access

You will now need to distribute the id_ed25519_sk.pub key to the servers you wish to access.

When connecting to the server, you can explicitly pass the correct ssh key with the -i flag

ssh <user>@<host> -i .ssh/id_ed25519_sk

• First you will be prompted for you PIN
• Next you will be asked to tap your YubiKey to confirm your physical presence

Login will complete and you should be dropped into the new environment.

Github

Github now supports using -sk type ssh keys that can be generated and stored on a security key.

More information from Github's blog.

Arch + KDE

I ran into an issue with my desktop.

sign_and_send_pubkey: signing failed for ED25519-SK "/home/<USERNAME>/.ssh/id_ed25519_sk" from agent: agent refused operation

I thought I had the ssh agent disabled. However, this system being as old as it is, has had multiple configurations through out it's life and it's quite possible that some cruft I have configured in the past is the source of my issue. Regardless, this workaround sorted it for me until ssh-agent can support sk keys with a PIN.

I added the following to my ~/.ssh/config

Host *
 IdentityAgent none
 IdentitiesOnly yes

This tells SSH to ignore ssh-agent even if it is running.

Grand Finale

via GIPHY

Now for the best part. The YubiKey is secured and the public key distributed, we are ready to use this on another box.

With the YubiKey plugged into a new host, we care able to again derive the public key.

cd ~/.ssh/
ssh-keygen -K
Enter PIN for authenticator:
You may need to touch your authenticator to authorize key download.
Enter passphrase for "id_ed25519_sk_rk" (empty for no passphrase):
Enter same passphrase again:
Saved ED25519-SK key to id_ed25519_sk_rk

The keys downloaded from the FIDO2 security key and given a new extension to help identify them. I drop the _rk extensions from the files after downloading them from the YubiKey.

NOTE: If you know what the _rk extension stands for I would love to know! The source code wasn't a huge help for me there.

From here you plug into the host you wish to connect from, and you are ready to go. If that box gets trashed, or needs to be sent for repair, you still have access to your servers and services, no need to generate and distribute a new key. Additionally if the machine is stolen or decommissioned you don't have to worry about accidentally handing out private keys.

The Catch?

via GIPHY

Don't loose your YubiKey unless you have a backup authentication flow. This could be having another private key stored in your password manager, using KVM solutions, virtual consoles, or HTTP based login flows. However those too should be MFA protected and if your YubiKey is the only MFA flow for your HTTP auth, then you are back at the same issue of single point of auth failure. Something to consider given the context you plan to operate in.