felix/sshworkshop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
56dda858c01feb03151d74c1082e424eb4f044c9
sshworkshop/presentation/ssh-basics.md

21 KiB
Raw Blame History

theme, _class, size, paginate, _footer, marp, transition, lang, backgroundColor, xbackgroundImage, backgroundImage
theme _class size paginate _footer marp transition lang backgroundColor xbackgroundImage backgroundImage
sparta lead 16:9 true SSH 101 - Felix true fade en url('https://marp.app/assets/hero-background.svg')

bg left:33% h:105% brightness:.7 opacity:.5

SSH essentials
Stuff everybody assumes you already know about ssh
Felix Schüren
2025-11-09
DENOG 17, Essen

Intro

about me

I've been doing networks for over 30 years by now. And DevOps stuff before we called it that. I work for Plusnet GmbH as a Solution Architect these days, but I have been a sysadmin, developer, network engineer, network architect and systems architect among other things.

Intro continued

about you

  • Which of you...
    • use ssh already?
    • use port forwarding?
    • use ProxyJump?
    • use ssh certificates?
    • use Windows? MacOS? Linux? Something else?

Clients

  • Linux, Mac, Windows
    • I would always prefer OpenSSH
    • in a pinch, git bash portable is helpful. or putty portable, but be careful where you download it.

but I need a GUI

  • Maybe. I believe that a well-maintained ssh_config is usually better. And if you have too many hosts, you want to use ansible etc anyway.
  • useful shells will offer tab-completion for ssh_config etc.

access the lab

  • ssh username@sshworkshop.telefrag.com -p4000+$last_octet

access our first system

jumphost1

  • ssh jumphost1
    • uptime
    • whoami
    • fireworks
      • CTRL-C

why is my terminal messed up?

  • escape codes, binary garbage
  • unix-like shells: reset
  • powershell: [Console]::ResetColor()

run commands remotely

  • make sure you're still logged into your linux$N lab machine
  • ssh jumphost1 uptime
  • ssh jumphost1 whoami
  • ssh jumphost1 fireworks
    • ssh jumphost1
      • alias
      • logout or CTRL-D
  • ssh jumphost1 "fireworks.sh Fireworks"
  • ssh -t jumphost1 "fireworks.sh Fireworks"
    • CTRL-C
      • reset

I hate typing passwords.

  • copy-paste
    • risky (who never pasted credentials into the wrong chat?)
  • key-based auth!
    • what are pub/privkeys
  • ssh-keygen
    • passphrase? what, why, how?
    • ED25519, RSA, ...

run commands remotely redux

  • make sure you're still logged into your linux$N lab machine
  • ssh jumphost1 uptime
  • ssh jumphost1 whoami
  • ssh jumphost1 -t fireworks
  • Yay, we've made it worse. Now we have to remember/copy-paste a passphrase instead of a password.

ssh-agent

  • make sure agent is running
    • normally, your GUI should do this for you, but this is a crudely simulated personal machine :D
    • eval (ssh-agent -s)
      • SSH_AGENT_PID
      • SSH_AUTH_SOCK
      • nice helper: keychain
        • cleverly finds out if there is already an agent running, updates ENV vars if so, otherwise, starts one
  • stores your unlocked private keys
  • ssh-add /path/to/keyfile - defaults to ~/.ssh/id_*
    • ssh-add
    • asks for passphrase
  • re-try the commands or logins

milestone One

  • password login vs key-based
  • run commands remotely
  • ssh-agent
  • terminals, PTY and -t

jumphost2

  • shared user: dbadmin
  • I need your public keys
  • stupidly left my keycollector private key (NEVER DO THIS) in /var/lib/misc/
    • seriously, never leave your private keys lying around.
    • this is just for the sake of labs & stuff
  • scp -i /var/lib/misc/keyuploader-identity /v /etc/passwd keycollector@192.168.0.254:$USER.passwd
  • copy, chmod, try again.

break - 10 mins

  • give me time to copy keys to jumphost2 etc.
  • this is why ssh key management is inherently difficult etc

running remote commands advanced

  • ssh jumphost1 'if [ -f "/etc/passwd" ]; then echo "Yes"; tail -1 /etc/passwd; else echo "No"; fi'
  • ssh user@host 'bash -s' < local_script.sh
  • (instead of copying, running, deleting local_script.sh)
  • you should really start looking into ansible if you've reached this state

Timeline ideas

  • clients
  • passwordless login
  • port forwarding
  • jumphosts, agent forwarding, proxyjump
  • host_keys / mitm
  • ssh_config
  • vs code
  • identity files (multiple identities)

~/.ssh permissions etc

ideas

  • Im laufenden Lab auf einem jumphost die host_keys ändern, um Warnungen zu produzieren
    • vorher jumphost2 den link zu jumphost1 auf 100% packet loss einstellen
      • in containerlab-extension jumphost2 rechtsklicken, packetloss fuer eth0 einstellen
        • ssh escape chars lernen um aus der session rauszukommen
          • enter, ~.
    • jumphost2: rm /etc/ssh/ssh_host_*; dpkg-reconfigure openssh-server; killall -HUP sshd
    • link impairment: clear all
  • agent_forwarding abgreifen und mich als einen der User einloggen
  • wie baue ich eine gute ssh_config
  • change my term local to C to avoid error messages:
    • LC_ALL=C LANG=C LC_CTYPE=C

bg right

  1. How?
  2. Host Keys
  3. Key-based auth
  4. SSH Agent
  5. Agent forwarding

Illustration created with Artbreeder

SSH Agent

Windows

Set-Service   ssh-agent -StartupType Automatic
Start-Service ssh-agent

Mac only

To add your ssh keys to MacOS Keychain, use UseKeychain yes in your ssh_config.

Linux

Generally, ssh-agent "just works" nowadays, especially on Linux GUI environments. But pay attention, you can easily run too many agents and get confused which one has your keys.

  • this is usually how tmux & co have problems with ssh-agent - due to stale environment vars.

ssh_config

  • on Windows: %USERPROFILE%\.ssh\config
    • PS C:\Users\felix> notepad.exe .\.ssh\config
  • on Mac/Linux: ~/.ssh/config

SSH Agent cont'd

root@clab:~/labs/ssh-workshop# ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-TzLLcOVtmlcY/agent.342525; export SSH_AUTH_SOCK;
SSH_AGENT_PID=342526; export SSH_AGENT_PID;
echo Agent pid 342526;

-> eval $(ssh-agent -s)

Never use ForwardAgent yes.

ANYONE with access to the SSH_AUTH_SOCK file can use the socket. (Generally, root. The directory is locked down as far as it will go, but root...)

ssh identities

  • ssh_config:
IdentityFile ~/.ssh/my-special-key
IdentitiesOnly yes

how?

The essence behind SSH is asymmetric cryptography (public/private key cryptography)

  1. you give others a "servant key" (public key)
    • can only be used to encrypt data
  2. you keep the master key (private key)
    • this decrypts the data that was encrypted with 1.
    • keep this very secure, never give it away.

host keys

Very similar concept to user/client keys, the host/server also has a private/public key pair

  • server (sshd) keys live in /etc/ssh/, typically called ssh_host_$TYPE_key
  • they are the "identity" of this particular ssh instance, they should be different on different machines
  • be mindful of this when templating/cloning VMs or containers

known_hosts and you

The authenticity of host 'schmargonrog.example.com' can't be established.
ED25519 key fingerprint is SHA256:H49twPDi0au3WObFIUmrbUSqc2j8uYzb2BCDigttvbw.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:s7Z+oc2AUQUqNH91OPkqzL0VXbe2fAoF+p+robhrCv0.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:71
  remove with:
  ssh-keygen -f "/root/.ssh/known_hosts" -R "clab-f2-spine2"
[...]
Host key verification failed.

how does it keep track of known hosts?

~/.ssh/known_hosts (personal) or /etc/ssh/ssh_known_hosts (global)

  • lists hostname, ssh key type and public key for hosts you have previously connected to 
    server1.example.com ssh-ed25519 AAAAC3NzaC1lZ...
server1.example.com ssh-rsa AAAAB3NzaC1yc2E...
server1.example.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLX...
  • some Linux distros default to HashKnownHosts yes, which replaces the hostname with a hashed version 
    |1|32Q6...=|Ul+EoN...= ssh-ed25519 AAAAC3NzaC1l...

bg left 70%

<style scoped> ul { font-size: smaller; } </style>

To recap:

  • temp pub/priv keys used to establish connection & share the session secret
    • from this point on, all comms on the ssh session are protected by this session secret
  • known_hosts important to prevent man-in-the-middle attacks

Photo by Adam Winger on Unsplash

Part 2

The Two Pillars

bg right

  • Key based auth
  • ssh agent

Photo by Cajeo Zhang on Unsplash

Key-based authentication

  • A private (secret, only for you) key
    • never give this to anyone else
    • never copy this to another machine
  • A public (everyone can have it) key
    • can be generated from the private key
  • typically, id_rsa and id_rsa.pub
    • or id_ed25519, id_dsa, ...

how to generate a key pair

  • ssh-keygen -t rsa -b 4096
    • will create ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub
  • ssh-keygen -t ed25519
    • →id_ed25519, essentially, it's id_$TYPE
  • please, do set a passphrase for the key. I will explain how to make it not annoying.

and now? aka authorized_keys

You now put the public key into the ~/.ssh/authorized_keys file of the user/host combination you want to have access to.

# cat ~/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA6Gc/53b0ZBGL/ORF5hIa61hTPTAsrjnkxXl3wawsHT felix@home
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEx6jIZeHFxh1NDJOoHGQLg0ViqqFNKGd5ofTRvHb4Fh backuppc@backup

(Remember, do not give your private key to anyone. Never.)

example login

having added the contents of my ~/.ssh/id_ed25519.pub file to the user root on my server db.example.com, I can now do

ssh -i ~/.ssh/id_ed25519 root@db.example.com

My local SSH client will ask me for my passphrase (to unlock the private key into memory), then use the private key to authenticate me, allowing me to log in.

SSH agent

  • Stores unlocked private keys in memory
  • only enter your passphrase when adding the key to the in-memory store
  • configurable lifetime
    • I would recommend 8-10 hours, so that during normal working hours you enter your passphrase once per day.
  • Painless passwordless logins!
  • can either manually add keys to it using ssh-add or set AddKeysToAgent yes in your ssh config

Agent forwarding

  • Exposes your private key cache (the "agent") to the system you're connecting to
    • SSH_AUTH_SOCK environment variable
# echo $SSH_AUTH_SOCK
/tmp/ssh-XXXXiA23DL/agent.1337770
  • allows you to access your private keys from an intermediary system
    • but everyone with root rights on the intermediary can use your SSH_AUTH_SOCK!

bg left 70%

Finished

<style scoped> ul { font-size: smaller; } </style>
  • keep your private key very private
  • be very careful with agent forwarding
  • be paranoid with known_hosts
  • check out the manpage of the ssh client config: man ssh_config
  • visit the advanced workshop

Photo by Jonatan David on Unsplash

Teaser: Ich wollte eh auf der DENOG Workshop fuer ssh machen, gebe hier mal Teaser, Rest koennt ihr dann euch selber angucken, ...

Praxis-Fails einbauen - neue hostkeys, passwort eingeben muessen, passphrase, ... jetzt muss ich meinen private key auf den jumphost kopieren, nein doch nicht, .... teaser-problem fuer proxyjump

optional proxyjump, falls ich zu schnell durchgehe bzw generell mal fragen, ob weiteres Interesse besteht

fuer lab: ein ssh-container-labyrinth, proxyjumps aussenrum und dann zum Schluss mit einem einzelnen ssh-kommando dateien aus der Mitte nach aussen... oder so.

https://www.openssh.com/legacy.html KexAlgorithms: the key exchange methods that are used to generate per-connection keys HostkeyAlgorithms: the public key algorithms accepted for an SSH server to authenticate itself to an SSH client Ciphers: the ciphers to encrypt the connection MACs: the message authentication codes used to detect traffic modification ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 user@legacyhost ssh -oHostKeyAlgorithms=+ssh-dss user@legacyhost

Reference in New Issue View Git Blame Copy Permalink