---
theme: sparta
_class: lead
size: 16:9
paginate: true
_footer: "SSH 101 - Felix"
marp: true
transition: fade
lang: en
backgroundColor: #fff
xbackgroundImage: url('https://marp.app/assets/hero-background.svg')
backgroundImage: ![bg](./img/artbreeder-composer-2024-07-15T00_14_43.169Z.jpeg)
---

<!-- _class: title-academic -->

![bg left:33% h:105% brightness:.7 opacity:.5](./img/artbreeder-image-2025-10-25T11_33_45.435Z.jpeg)

<div class="title"         > SSH essentials</div>
<div class="subtitle"      > Stuff everybody assumes you already know about ssh </div> 
<div class="author"        > Felix Schüren </div>
<div class="date"          > 2025-11-09</div>
<div class="organization"  > DENOG 17, Essen</div>

---
# Intro
## about me
I've been doing networks for over 30 years by now. And DevOps stuff before we called it that. I work for Plusnet GmbH as a Solution Architect these days, but I have been a sysadmin, developer, network engineer, network architect and systems architect among other things. 

---
# Intro continued
## about you
- Which of you...
  * use ssh already?
  * use port forwarding?
  * use ProxyJump?
  * use ssh certificates?
  * use Windows? MacOS? Linux? Something else?

---
# Clients
- Linux, Mac, Windows
  - I would always prefer OpenSSH
  - in a pinch, `git bash` portable is helpful. or `putty` portable, but be careful where you download it.
## but I need a GUI
- Maybe. I believe that a well-maintained ssh_config is usually better. And if you have too many hosts, you want to use ansible etc anyway.
- useful shells will offer tab-completion for ssh_config etc.


---
# access the lab
<!-- for the first steps, this will be the standin for your personal machine - we will integrate your actual machine later after covering some of the fundamentals. -->
- `ssh username@sshworkshop.telefrag.com -p4000+$last_octet`

---
# access our first system
## jumphost1
  * `ssh jumphost1`
    * `uptime`
    * `whoami`
    * `fireworks`
      * `CTRL-C`

---
# why is my terminal messed up?
  * escape codes, binary garbage
  * unix-like shells: `reset`
  * powershell: `[Console]::ResetColor()`

--- 
# run commands remotely
  * make sure you're still logged into your linux$N lab machine
  * `ssh jumphost1 uptime`
  * `ssh jumphost1 whoami`
  * `ssh jumphost1 fireworks`
    * `ssh jumphost1`
      * `alias`
      * `logout` or `CTRL-D`
  * `ssh jumphost1 "fireworks.sh Fireworks"`
  * `ssh -t jumphost1 "fireworks.sh Fireworks"`
    * `CTRL-C`
      * `reset`

---
# I hate typing passwords. 
  * copy-paste
    * risky (who never pasted credentials into the wrong chat?)
  * key-based auth!
    * what are pub/privkeys
  * `ssh-keygen`
    * passphrase? what, why, how?
    * ED25519, RSA, ...

--- 
# run commands remotely redux
  * make sure you're still logged into your linux$N lab machine
  * `ssh jumphost1 uptime`
  * `ssh jumphost1 whoami`
  * `ssh jumphost1 -t fireworks`
  * Yay, we've made it worse. Now we have to remember/copy-paste a passphrase instead of a password.

---
# ssh-agent
  * make sure agent is running
    * normally, your GUI should do this for you, but this is a crudely simulated personal machine :D
    * `eval (ssh-agent -s)`
      * SSH_AGENT_PID
      * SSH_AUTH_SOCK
      * nice helper: `keychain`
        * cleverly finds out if there is already an agent running, updates ENV vars if so, otherwise, starts one
  * stores your unlocked private keys
  * `ssh-add /path/to/keyfile` - defaults to `~/.ssh/id_*`
    * `ssh-add`
    * asks for passphrase
  * re-try the commands or logins

---
# milestone One
- password login vs key-based
- run commands remotely
- ssh-agent
- terminals, PTY and `-t`


---
# jumphost2
  * shared user: dbadmin
  * I need your public keys
  * stupidly left my keycollector private key (NEVER DO THIS) in `/var/lib/misc/`
    * seriously, never leave your private keys lying around.
    * this is just for the sake of labs & stuff
  * `scp  -i /var/lib/misc/keyuploader-identity /v /etc/passwd keycollector@192.168.0.254:$USER.passwd`
  * copy, chmod, try again.

---
# break - 10 mins
  - give me time to copy keys to jumphost2 etc. 
  * this is why ssh key management is inherently difficult etc


--- 
# running remote commands advanced
  * `ssh jumphost1 'if [ -f "/etc/passwd" ]; then echo "Yes"; tail -1 /etc/passwd; else echo "No"; fi'`
  * `ssh user@host 'bash -s' < local_script.sh`
  * (instead of copying, running, deleting local_script.sh)
  * you should really start looking into ansible if you've reached this state

---
# Timeline ideas
- clients
<!-- On mac & linux, use openssh via a terminal app. It's usually preinstalled, if not, it comes with the base os packages. Same for windows (10+) - Use Terminal app + openssh. -->
- passwordless login
- port forwarding
- jumphosts, agent forwarding, proxyjump
- host_keys / mitm
- ssh_config
- vs code
- identity files (multiple identities)

---
# ~/.ssh permissions etc
<!-- on some OSes (mostly Redhat derivatives) if key-based logins don't work, it's ofen to do with the permissions of the .ssh folder or the authorized_keys file. -->

---
# ideas
- Im laufenden Lab auf einem jumphost die host_keys ändern, um Warnungen zu produzieren
  - vorher jumphost2 den link zu jumphost1 auf 100% packet loss einstellen
    - in containerlab-extension jumphost2 rechtsklicken, packetloss fuer eth0 einstellen
      - ssh escape chars lernen um aus der session rauszukommen
        - enter, ~.
  - jumphost2: `rm /etc/ssh/ssh_host_*; dpkg-reconfigure openssh-server; killall -HUP sshd`
  - link impairment: clear all
- agent_forwarding abgreifen und mich als einen der User einloggen
- wie baue ich eine gute ssh_config
- change my term local to C to avoid error messages:
  - `LC_ALL=C LANG=C LC_CTYPE=C`
---
![bg right](./img/artbreeder-composer-2024-07-14T22_49_42.899Z.jpeg)

1. [How?](#how)
2. [Host Keys](#host-keys)
3. [Key-based auth](#key-based-authentication)
4. [SSH Agent](#ssh-agent)
5. [Agent forwarding](#agent-forwarding)

<span style="color: darkgrey; font-size: 0.4em"><a href="https://www.artbreeder.com/image/f1f2dd42df55daa49ddcea6db222">Illustration</a> created with <a href="https://www.artbreeder.com/">Artbreeder</a></span>

---
# SSH Agent
## Windows
```
Set-Service   ssh-agent -StartupType Automatic
Start-Service ssh-agent
```
## Mac only
To add your ssh keys to MacOS Keychain, use `UseKeychain yes` in your ssh_config.

## Linux
Generally, ssh-agent "just works" nowadays, especially on Linux GUI environments. But pay attention, you can easily run too many agents and get confused which one has your keys.
  * this is usually how tmux & co have problems with ssh-agent - due to stale environment vars.


---
# ssh_config
- on Windows:
  `%USERPROFILE%\.ssh\config`
  - `PS C:\Users\felix>` `notepad.exe .\.ssh\config`
- on Mac/Linux: `~/.ssh/config`

---
# SSH Agent cont'd
<!-- ssh agent usually works through the SSH_AUTH_SOCK and SSH_AGENT_PID variables. These variables are usually what's wrong with ssh-agent in a tmux or similar scenario. -->
```
root@clab:~/labs/ssh-workshop# ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-TzLLcOVtmlcY/agent.342525; export SSH_AUTH_SOCK;
SSH_AGENT_PID=342526; export SSH_AGENT_PID;
echo Agent pid 342526;
```
-> `eval $(ssh-agent -s)`
<!-- When ssh-agent is started, it prints the shell commands required to set its environment variables, which in turn can be evaluated in the calling shell -->

---
# Never use `ForwardAgent yes`.
*ANYONE* with access to the SSH_AUTH_SOCK file can use the socket. (Generally, root. The directory is locked down as far as it will go, but root...)
<!-- Live demo of how easy it is. Login as fred with AgentForward to a jumphost, demonstrate how the AgentForward works & helps fred to log in to webserver1, then have a second shell with root on the jumphost, demonstrate how root can't ssh to webserver1, manually set the correct SSH_AUTH_SOCK and watch how he suddenly can. -->


---
# ssh identities
- ssh_config:
```
IdentityFile ~/.ssh/my-special-key
IdentitiesOnly yes
```
---
# how?
The essence behind SSH is asymmetric cryptography
<span style="color: darkgrey; font-size: 0.5em">(public/private key cryptography)</span>

1. you give others a "servant key" (public key)
   - can only be used to encrypt data<!-- pubkey can also check signatures, but that's less important in ssh use.  -->
2. you keep the master key (private key)
   - this decrypts the data that was encrypted with 1.
   - keep this **very secure**, never give it away.

<!-- This enables one-way secure communication - others can now send data to you that
can not be read or changed on the way to you. That's great, but you want to communicate bi-directionally... -->

---
# host keys
Very similar concept to user/client keys, the host/server also has a private/public key pair
- server (`sshd`) keys live in ```/etc/ssh/```, typically called ```ssh_host_$TYPE_key```
- they are the "identity" of this particular ssh instance, they should be different on different machines
- be mindful of this when templating/cloning VMs or containers
<!-- Now both parties (client and server) have a way to receive encrypted messages, and as mentionend use this to share a session secret (for efficiency and speed) for running sessions.
Under the hood, SSH will use temporary pub/priv keys to establish the connection, then switch to symmetric encryption where the shared secret is randomly generated between the 
SSH parties and transmitted securely through the magic of pub/priv crypto. -->

---
## known_hosts and you
<!-- you will have seen a message popping up when you connect to a new host for the first time, displaying some form of -->
```
The authenticity of host 'schmargonrog.example.com' can't be established.
ED25519 key fingerprint is SHA256:H49twPDi0au3WObFIUmrbUSqc2j8uYzb2BCDigttvbw.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
```
<!-- Another message you might have seen is this one: -->
```
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:s7Z+oc2AUQUqNH91OPkqzL0VXbe2fAoF+p+robhrCv0.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:71
  remove with:
  ssh-keygen -f "/root/.ssh/known_hosts" -R "clab-f2-spine2"
[...]
Host key verification failed.
```
<!-- In both cases, you should think about what is going on. First case is pretty normal, you're connecting to a host that you have no previous information on. It's usually okay to say yes here. For the extra paranoid, verify the host key fingerprint through a secure side channel (phone? DNS? photo? etc...) before continuing the connection.

Don't follow the second message blindly say yes, think about *why* the host key might have changed. Did you recently do a dist-upgrade, did someone reinstall the machine without backing up/restoring the host keys, or was there a major breaking change with ssh during a system upgrade run? In any case, be vigilant. 
-->

---
## how does it keep track of known hosts?
<!-- If you said "yes" on the prior page, how does your local ssh client actually save this information? There are two spaces, with the global one usually not being used. -->
```~/.ssh/known_hosts``` (personal) or ```/etc/ssh/ssh_known_hosts``` (global)
- lists hostname, ssh key type and public key for hosts you have previously connected to
  ```
  server1.example.com ssh-ed25519 AAAAC3NzaC1lZ...
  server1.example.com ssh-rsa AAAAB3NzaC1yc2E...
  server1.example.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLX...
  ```
  <!--- (this is what happens when you answer "yes" on the first connection to a host and )--> 
- some Linux distros default to ```HashKnownHosts yes```, which replaces the hostname with a hashed version
  ```
  |1|32Q6...=|Ul+EoN...= ssh-ed25519 AAAAC3NzaC1l...
  ```
<!-- This makes some deleting entries a lot harder than before, and it might break some tab-completion magic that might use the known_hosts file -->
<!-- You might want to roll out the known_hosts centrally. -->
---
![bg left 70%](./img/adam-winger-GIFlfKX23rc-unsplash.jpg)
<style scoped>
ul {
  font-size: smaller;
}
</style>
To recap:
- temp pub/priv keys used to establish connection & share the session secret
  - from this point on, all comms on the ssh session are protected by this session secret
- ```known_hosts``` important to prevent man-in-the-middle attacks 
<!-- Right, that's the basic how. Let's get to some things that make your life so much easier, yet that are often misunderstood. -->


<span style="color: darkgrey; font-size: 0.4em">Photo by <a href="https://unsplash.com/@awcreativeut?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Adam Winger</a> on <a href="https://unsplash.com/photos/girl-writing-on-the-ground-using-a-spray-an-GIFlfKX23rc?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Unsplash</a></span>
  
---
# Part 2
### The Two Pillars
![bg right](./img/cajeo-zhang-EbOAPSCwjnU-unsplash.jpg)
- Key based auth
- ssh agent
<!-- The most widely used yet often misunderstood features of SSH. This is why you are here.-->
<span style="color: darkgrey; font-size: 0.4em">Photo by <a href="https://unsplash.com/@cajeo?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Cajeo Zhang</a> on <a href="https://unsplash.com/photos/gray-rock-formation-during-daytime-EbOAPSCwjnU?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Unsplash</a></span>
  
---
# Key-based authentication

- A private (secret, **only for you**) key
  - **never** give this to anyone else
  - **never** copy this to another machine
- A public (everyone can have it) key
  - <small>can be generated from the private key</small>
- typically, ```id_rsa``` and ```id_rsa.pub```
  - <small>or ```id_ed25519```, ```id_dsa```, ...</small>
<!--
One of the core failures of the whole SSH ecosystem is the default naming convention for private/public keys. Accidentally cat'ing your private key is way too easy.

rsa, dsa, ed25519 are different variants/types of encryption. When in doubt, use RSA.

(I would always go with ED25519 for modern deployments, though it might not work on systems that are older than 2020ish)

-->
---
## how to generate a key pair
- ```ssh-keygen -t rsa -b 4096```
  - will create ```~/.ssh/id_rsa``` and ```~/.ssh/id_rsa.pub```
- ```ssh-keygen -t ed25519```
  - &rarr;```id_ed25519```, essentially, it's ```id_$TYPE```
- please, do set a passphrase for the key. I will explain how to make it not annoying.

<!-- Securing your private key with a passphrase (which just means "long password") is a good precaution, it means somebody that gets access to your private key cannot immediately use it. -->

---
## and now? aka ```authorized_keys```
<!--Okay, now you have your key, and you can give the public side of it to others. If others want to securely encrypt something that **only** you can decrypt, they use your **public** key to encrypt. You then use your **private** key to decrypt.-->
You now put the **public** key into the ```~/.ssh/authorized_keys``` file of the user/host combination you want to have access to.

```
# cat ~/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA6Gc/53b0ZBGL/ORF5hIa61hTPTAsrjnkxXl3wawsHT felix@home
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEx6jIZeHFxh1NDJOoHGQLg0ViqqFNKGd5ofTRvHb4Fh backuppc@backup
```

<!-- In order to use this for authentication instead of a password, the SSH server will essentially encrypt something with your public key and ask you to send the decrypted text back (to prove that you are indeed in posession of the private key part) -->

<span style="color: orange; font-size: 0.5em">(Remember, do not give your private key to anyone. Never.)</span>
<!--*READABLE* && pubkey &rarr; *CRYPTED* && privkey &rarr; *READABLE*-->

---
## example login
having added the contents of my ```~/.ssh/id_ed25519.pub``` file to the user ```root``` on my server ```db.example.com```, I can now do
```
ssh -i ~/.ssh/id_ed25519 root@db.example.com
```
My local SSH client will ask me for my passphrase (to unlock the private key into memory), then use the private key to authenticate me, allowing me to log in.

<!-- Yes, this feels identical to logging in with a password, except I was encouraged to make my passphrase longer and more annoying to type. Great job, you've made using ssh more annoying. -->

---

# SSH agent
- Stores unlocked private keys in memory
- only enter your passphrase when adding the key to the in-memory store
- configurable lifetime
  - I would recommend 8-10 hours, so that during normal working hours you enter your passphrase once per day.
- Painless passwordless logins!
- can either manually add keys to it using ```ssh-add``` or set ```AddKeysToAgent yes``` in your ssh config
<!-- Should probably more accurately be called the SSH private key cache. It's well integrated into openssh, and normal ssh connection attempts will try to use your local ssh agent automatically. -->
<!-- On macos, you can use the system keychain to store the passphrase, where they will live together with all the other saved passwords, protected by a master password. It's ```UseKeychain yes
``` in your .ssh/config, and I would recommend to use that if you're on a mac. -->
<!-- On windows, putty has "pageant" to serve as private key cache, it works identically, though by default does not set a lifetime (you do need to re-enter passphrase after rebooting) -->

<!-- This is a must-have if you're serious about using key-based auth. You can get by with using keys without passphrase, but please don't, it carries a heavy price in terms of identity theft and potential damage to your reputation. As a sidenote, with more and more machines ending up in an MDM (mobile device management) "friendly rootkit" scenario, your private keys, passphrases and logins etc are essentially all compromised (to your MDM admins), making (digital) identity inside your company all but meaningless, but that is a topic for another talk... or hit me up later, I will talk at length about the problems I see with MDM :) -->


---
# Agent forwarding
- Exposes your private key cache (the "agent") to the system you're connecting to
  - ```SSH_AUTH_SOCK``` environment variable
```
# echo $SSH_AUTH_SOCK
/tmp/ssh-XXXXiA23DL/agent.1337770
```
- allows you to access your private keys from an intermediary system
  - but everyone with root rights on the intermediary can use your SSH_AUTH_SOCK!

<!-- you have essentially compromised your keys to every admin of the intermediary system. Be very, very careful where you agent forward to. Come visit my advanced SSH workshop to learn a better way.-->

---
![bg left 70%](./img/jonatan-david-K7MkzR7rBmM-unsplash.jpg)
# Finished
<style scoped>
ul {
  font-size: smaller;
}
</style>
- keep your private key very private
- be very careful with agent forwarding
- be paranoid with ```known_hosts```
- check out the manpage of the ssh client config: ```man ssh_config```
- visit the advanced workshop
<!-- -->

<span style="color: darkgrey; font-size: 0.4em">Photo by <a href="https://unsplash.com/@bboy_mocachino?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Jonatan David</a> on <a href="https://unsplash.com/photos/a-blurry-photo-of-a-street-with-a-stop-sign-K7MkzR7rBmM?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash">Unsplash</a></span>

<!-- This covers what I consider the SSH 101 -->

---
Teaser: Ich wollte eh auf der DENOG Workshop fuer ssh machen, gebe hier mal Teaser, Rest koennt ihr dann euch selber angucken, ...


Praxis-Fails einbauen - neue hostkeys, passwort eingeben muessen, passphrase, ... jetzt muss ich meinen private key auf den jumphost kopieren, nein doch nicht, .... teaser-problem fuer proxyjump

optional proxyjump, falls ich zu schnell durchgehe bzw generell mal fragen, ob weiteres Interesse besteht

fuer lab: ein ssh-container-labyrinth, proxyjumps aussenrum und dann zum Schluss mit einem einzelnen ssh-kommando dateien aus der Mitte nach aussen... oder so.

---
https://www.openssh.com/legacy.html
KexAlgorithms: the key exchange methods that are used to generate per-connection keys
HostkeyAlgorithms: the public key algorithms accepted for an SSH server to authenticate itself to an SSH client
Ciphers: the ciphers to encrypt the connection
MACs: the message authentication codes used to detect traffic modification
`ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 user@legacyhost`
`ssh -oHostKeyAlgorithms=+ssh-dss user@legacyhost`