POSTS

When an OpenSSH MitM Isn't One

November 22, 2018

What do you do if you know that a server’s OpenSSH host keys didn’t change, but you see a MITM warning from ssh?

To put it into other words, is an ssh client warning about a MITM a real MITM if the server host keys didn’t change? Astute observers would say, maybe. Let me be a bit more precise then.

Assuming that:

the client and server software weren’t compromised
the ssh host keys didn’t change
the client is connecting to the right host (no dns, dhcp issues etc)
the ssh client’s known_hosts file wasn’t tampered with

Is that then an indication of a MITM attack?

Not quite. Here is a scenario where there is a perfectly reasonable explanation for what’s happening. But first…

Host keys, plural

Back in the days the OpenSSH host key mainly ment an RSA key. But, as with all things crypto, new types were added. Among others, ecdsa and then ed25519.

The way host key fingerprints are stored in the known_hosts file also changed over the years. The hostname got hashed for security reasons and the fingerprint algorithm also changed by default from md5 to sha256.

Nowadays, it’s quite common an sshd to have 3-4 different host keys, depending on whether the ancient dss is still supported.

Which key is checked, then?

This is negotiated between the ssh client and the server, driven by the client. The client offers a list to the server, from which the server selects the first one that exists on the server and transmits the host key with the selected key type to the client.

debug2: local client KEXINIT proposal
debug2: KEX algorithms: [snip]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
(...)
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [snip]
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519

…so in the case of this output, the client and server agree on ecdsa-sha2-nistp256.

What if keys need to change?

To deal with key rotation and transitioning from one type of key to the next, OpenSSH 6.8 added experimental host key rotation support, where after a successful authentication, all of the available host keys are transmitted to the client which can then decide to store it in it’s known_hosts file and upgrade from older to better key types. This is not yet enabled by default.

Putting it all together

Here are the relevant details from the ssh connection producing the MITM banner:

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:KmNo0s/JvzZKpQIiHJ2OVebXU3CKk50m8ALfamBNHL4
debug3: record_hostkey: found key type RSA in file /home/foo/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from [XXX]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:KmNo0s/JvzZKpQIiHJ2OVebXU3CKk50m8ALfamBNHL4.
Please contact your system administrator.
Add correct host key in /home/foo/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/foo/.ssh/known_hosts:2
ECDSA host key for [XXX] has changed and you have requested strict checking.
Host key verification failed.

Do you see what’s happening here?

The client ssh version got upgraded or the client ssh configuration changed, bringing with it a different client host key preference order. Instead of preferring an RSA host key, it now prefers ecdsa. Nothing changed on the server. The client has seen that there is already a host key in the known_hosts file, with a different type and fingerprint and promptly terminated the connection.

This is arguably the right, safe behaviour. It also means that until host key rotation support is enabled, encountering a different type of host key results in a MITM warning, without any actual MITM taking place, if the client changed their mind about the preferred type of host key over time.

openssh