SSH 101 with Felix

Things everybody assumes you already know about ssh
(secure shell, though nobody calls it that)

Felix Schüren
2025-11-09 DENOG17 Essen

Background illustration created with Artbreeder

SSH 101 - Felix

Intro

  • Which of you...
    • use ssh already?
    • use port forwarding?
    • use ProxyJump?
    • use ssh certificates?
    • use Windows? MacOS? Linux? Something else?

Timeline ideas

  • passwordless login
  • port forwarding
  • jumphosts, agent forwarding, proxyjump
  • host_keys / mitm
  • ssh_config

ideas

  • Im laufenden Lab auf einem jumphost die host_keys ändern, um Warnungen zu produzieren
  • agent_forwarding abgreifen und mich als einen der User einloggen
  • Hinterher aufzeigen (frei nach "I've been here the whole time!"), dass man cert-based logins benutzt hat
  • ggfs Stacy Fakename, Joanna Fakename als Easter eggs?
  • wie baue ich eine gute ssh_config
  1. How?
  2. Host Keys
  3. Key-based auth
  4. SSH Agent
  5. Agent forwarding

Illustration created with Artbreeder

how?

The essence behind SSH is asymmetric cryptography
(public/private key cryptography)

  1. you give others a "servant key" (public key)
    • can only be used to encrypt data
  2. you keep the master key (private key)
    • this decrypts the data that was encrypted with 1.
    • keep this very secure, never give it away.

host keys

Very similar concept to user/client keys, the host/server also has a private/public key pair

  • live in /etc/ssh/, typically called ssh_host_$TYPE_key
  • be mindful of this when templating/cloning VMs or containers

known_hosts and you

The authenticity of host 'schmargonrog.example.com' can't be established.
ED25519 key fingerprint is SHA256:H49twPDi0au3WObFIUmrbUSqc2j8uYzb2BCDigttvbw.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:s7Z+oc2AUQUqNH91OPkqzL0VXbe2fAoF+p+robhrCv0.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:71
  remove with:
  ssh-keygen -f "/root/.ssh/known_hosts" -R "clab-f2-spine2"
[...]
Host key verification failed.

how does it keep track of known hosts?

~/.ssh/known_hosts (personal) or /etc/ssh/ssh_known_hosts (global)

  • lists hostname, ssh key type and public key for hosts you have previously connected to
    server1.example.com ssh-ed25519 AAAAC3NzaC1lZ...
    server1.example.com ssh-rsa AAAAB3NzaC1yc2E...
    server1.example.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLX...
    
  • some Linux distros default to HashKnownHosts yes, which replaces the hostname with a hashed version
    |1|32Q6...=|Ul+EoN...= ssh-ed25519 AAAAC3NzaC1l...
    

To recap:

  • temp pub/priv keys used to establish connection & share the session secret
    • from this point on, all comms on the ssh session are protected by this session secret
  • known_hosts important to prevent man-in-the-middle attacks

Photo by Adam Winger on Unsplash

Part 2

The Two Pillars

  • Key based auth
  • ssh agent

Photo by Cajeo Zhang on Unsplash

Key-based authentication

  • A private (secret, only for you) key
    • never give this to anyone else
    • never copy this to another machine
  • A public (everyone can have it) key
    • can be generated from the private key
  • typically, id_rsa and id_rsa.pub
    • or id_ed25519, id_dsa, ...

how to generate a key pair

  • ssh-keygen -t rsa -b 4096
    • will create ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub
  • ssh-keygen -t ed25519
    • id_ed25519, essentially, it's id_$TYPE
  • please, do set a passphrase for the key. I will explain how to make it not annoying.

and now? aka authorized_keys

You now put the public key into the ~/.ssh/authorized_keys file of the user/host combination you want to have access to.

# cat ~/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA6Gc/53b0ZBGL/ORF5hIa61hTPTAsrjnkxXl3wawsHT felix@home
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEx6jIZeHFxh1NDJOoHGQLg0ViqqFNKGd5ofTRvHb4Fh backuppc@backup

(Remember, do not give your private key to anyone. Never.)

example login

having added the contents of my ~/.ssh/id_ed25519.pub file to the user root on my server db.example.com, I can now do

ssh -i ~/.ssh/id_ed25519 root@db.example.com

My local SSH client will ask me for my passphrase (to unlock the private key into memory), then use the private key to authenticate me, allowing me to log in.

SSH agent

  • Stores unlocked private keys in memory
  • only enter your passphrase when adding the key to the in-memory store
  • configurable lifetime
    • I would recommend 8-10 hours, so that during normal working hours you enter your passphrase once per day.
  • Painless passwordless logins!
  • can either manually add keys to it using ssh-add or set AddKeysToAgent yes in your ssh config

Agent forwarding

  • Exposes your private key cache (the "agent") to the system you're connecting to
    • SSH_AUTH_SOCK environment variable
# echo $SSH_AUTH_SOCK
/tmp/ssh-XXXXiA23DL/agent.1337770
  • allows you to access your private keys from an intermediary system
    • but everyone with root rights on the intermediary can use your SSH_AUTH_SOCK!

Finished

  • keep your private key very private
  • be very careful with agent forwarding
  • be paranoid with known_hosts
  • check out the manpage of the ssh client config: man ssh_config
  • visit the advanced workshop

Photo by Jonatan David on Unsplash

Teaser: Ich wollte eh auf der DENOG Workshop fuer ssh machen, gebe hier mal Teaser, Rest koennt ihr dann euch selber angucken, ...

Praxis-Fails einbauen - neue hostkeys, passwort eingeben muessen, passphrase, ... jetzt muss ich meinen private key auf den jumphost kopieren, nein doch nicht, .... teaser-problem fuer proxyjump

optional proxyjump, falls ich zu schnell durchgehe bzw generell mal fragen, ob weiteres Interesse besteht

fuer lab: ein ssh-container-labyrinth, proxyjumps aussenrum und dann zum Schluss mit einem einzelnen ssh-kommando dateien aus der Mitte nach aussen... oder so.

https://www.openssh.com/legacy.html
KexAlgorithms: the key exchange methods that are used to generate per-connection keys
HostkeyAlgorithms: the public key algorithms accepted for an SSH server to authenticate itself to an SSH client
Ciphers: the ciphers to encrypt the connection
MACs: the message authentication codes used to detect traffic modification
ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 user@legacyhost
ssh -oHostKeyAlgorithms=+ssh-dss user@legacyhost

This is aimed at people using SSH in their daily work - it does not deal with details of the SSH protocol itself.

pubkey can also check signatures, but that's less important in ssh use.

This enables one-way secure communication - others can now send data to you that can not be read or changed on the way to you. That's great, but you want to communicate bi-directionally...

Now both parties (client and server) have a way to receive encrypted messages, and as mentionend use this to share a session secret (for efficiency and speed) for running sessions. Under the hood, SSH will use temporary pub/priv keys to establish the connection, then switch to symmetric encryption where the shared secret is randomly generated between the SSH parties and transmitted securely through the magic of pub/priv crypto.

you will have seen a message popping up when you connect to a new host for the first time, displaying some form of

Another message you might have seen is this one:

In both cases, you should think about what is going on. First case is pretty normal, you're connecting to a host that you have no previous information on. It's usually okay to say yes here. For the extra paranoid, verify the host key fingerprint through a secure side channel (phone? DNS? photo? etc...) before continuing the connection. Don't follow the second message blindly say yes, think about *why* the host key might have changed. Did you recently do a dist-upgrade, did someone reinstall the machine without backing up/restoring the host keys, or was there a major breaking change with ssh during a system upgrade run? In any case, be vigilant.

If you said "yes" on the prior page, how does your local ssh client actually save this information? There are two spaces, with the global one usually not being used.

(this is what happens when you answer "yes" on the first connection to a host and )

This makes some deleting entries a lot harder than before, and it might break some tab-completion magic that might use the known_hosts file

You might want to roll out the known_hosts centrally.

Right, that's the basic how. Let's get to some things that make your life so much easier, yet that are often misunderstood.

The most widely used yet often misunderstood features of SSH. This is why you are here.

One of the core failures of the whole SSH ecosystem is the default naming convention for private/public keys. Accidentally cat'ing your private key is way too easy. rsa, dsa, ed25519 are different variants/types of encryption. When in doubt, use RSA. (I would always go with ED25519 for modern deployments, though it might not work on systems that are older than 2020ish)

Securing your private key with a passphrase (which just means "long password") is a good precaution, it means somebody that gets access to your private key cannot immediately use it.

Okay, now you have your key, and you can give the public side of it to others. If others want to securely encrypt something that **only** you can decrypt, they use your **public** key to encrypt. You then use your **private** key to decrypt.

In order to use this for authentication instead of a password, the SSH server will essentially encrypt something with your public key and ask you to send the decrypted text back (to prove that you are indeed in posession of the private key part)

*READABLE* && pubkey → *CRYPTED* && privkey → *READABLE*

Yes, this feels identical to logging in with a password, except I was encouraged to make my passphrase longer and more annoying to type. Great job, you've made using ssh more annoying.

Should probably more accurately be called the SSH private key cache. It's well integrated into openssh, and normal ssh connection attempts will try to use your local ssh agent automatically.

On macos, you can use the system keychain to store the passphrase, where they will live together with all the other saved passwords, protected by a master password. It's ```UseKeychain yes ``` in your .ssh/config, and I would recommend to use that if you're on a mac.

On windows, putty has "pageant" to serve as private key cache, it works identically, though by default does not set a lifetime (you do need to re-enter passphrase after rebooting)

This is a must-have if you're serious about using key-based auth. You can get by with using keys without passphrase, but please don't, it carries a heavy price in terms of identity theft and potential damage to your reputation. As a sidenote, with more and more machines ending up in an MDM (mobile device management) "friendly rootkit" scenario, your private keys, passphrases and logins etc are essentially all compromised (to your MDM admins), making (digital) identity inside your company all but meaningless, but that is a topic for another talk... or hit me up later, I will talk at length about the problems I see with MDM :)

you have essentially compromised your keys to every admin of the intermediary system. Be very, very careful where you agent forward to. Come visit my advanced SSH workshop to learn a better way.

This covers what I consider the SSH 101