Cannot establish SSH session: kex error : no match for method mac algo client->server

[airween]

We have a Debian 12 OS and a Zabbix 6 instance (6.0.26).

We have a Mikrotik device and an item uses a trigger which is an ssh_agent.

The item reports this message:

Cannot establish SSH session: kex error : no match for method mac algo client->server: server [hmac-sha1,hmac-md5], client [[email protected],[email protected],hmac-sha2-256,hmac-sha2-512]

But when I try the SSH connection from CLI it works as I expect:

ssh [email protected] -i /home/zabbix/.ssh/mikrotik

In the documentation I found that ssh_agent supports SSH options:

https://www.zabbix.com/documentation/6.0/en/manual/config/items/itemtypes/ssh_checks#item-configuration

See the table below:

Key | Unique (per host) item key in the format ssh.run[unique short description,<ip>,<port>,<encoding>,<ssh options>] | ... ssh options (supported since version 6.0.25; require libssh of version 0.9.0 and higher or libssh2)



This is how I try to use ssh:

ssh.run[ping_{$PING_IP1},,,,"KexExchange=diffie-hellman-group-exchange-sha256;MACs=hmac-sha1"]

As I wrote we use Zabbix 6.0.26.

libssh and libssh2 versions:

libssh-4:amd64 0.10.6-0+deb12u1
libssh2-1:amd64 1.10.0-3+b1

but I still get the message above when I try to run the trigger.

Am I failed something?

[MRedbourne]

I'm rather making an assumption here. But your CLI SSH through Debian is likely running OpenSSH (relatively permissive by default). Zabbix comes packaged with libssh/libssh2. They're not the same. As the error indicates, the client and server aren't able to negotiate a mutual HMAC (MAC Hash). What MAC Algos are supported on Mikrotik? Do you have strong-crypto turned on? Manual:IP/SSH - MikroTik Wiki.

I would find out what your Mikrotik will support for KEX, Ciphers, HMAC, etc, and then pass those as SSH options in your ssh.run[...] action. Default to the highest level of mutual support.

[airween]

Hi MRedbourne,

thanks for your reply!

I'm rather making an assumption here. But your CLI SSH through Debian is likely running OpenSSH (relatively permissive by default). Zabbix comes packaged with libssh/libssh2. They're not the same. As the error indicates, the client and server aren't able to negotiate a mutual HMAC (MAC Hash).

Oh, so Zabbix uses its own SSH - thanks.

What MAC Algos are supported on Mikrotik? Do you have strong-crypto turned on? Manual:IP/SSH - MikroTik Wiki.

We tried both of them (strong and without strong).

I would find out what your Mikrotik will support for KEX, Ciphers, HMAC, etc, and then pass those as SSH options in your ssh.run[...] action. Default to the highest level of mutual support.

Here is how I checked:

# nmap --script ssh2-enum-algos -sV -p 22 1.2.1.4
Starting Nmap 7.93 ( https://nmap.org ) at 2024-02-15 13:37 CET
Nmap scan report for 1.2.1.4
Host is up (0.0024s latency).

PORT   STATE SERVICE VERSION
22/tcp open  ssh     MikroTik RouterOS sshd (protocol 2.0)
| ssh2-enum-algos:
|   kex_algorithms: (4)
|       diffie-hellman-group-exchange-sha256
|       diffie-hellman-group-exchange-sha1
|       diffie-hellman-group14-sha1
|       diffie-hellman-group1-sha1
|   server_host_key_algorithms: (3)
|       ssh-dss
|       rsa-sha2-256
|       ssh-rsa
|   encryption_algorithms: (9)
|       aes128-ctr
|       aes192-ctr
|       aes256-ctr
|       aes128-cbc
|       aes192-cbc
|       aes256-cbc
|       blowfish-cbc
|       3des-cbc
|       none
|   mac_algorithms: (2)
|       hmac-sha1
|       hmac-md5
|   compression_algorithms: (1)
|_      none
Service Info: OS: Linux; Device: router; CPE: cpe:/o:mikrotik:routeros

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 1.07 seconds

So I chose one. I also checked the client side, but now I see it does not count.

# ssh -Q mac
hmac-sha1
hmac-sha1-96
hmac-sha2-256
hmac-sha2-512
hmac-md5
hmac-md5-96
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
​

Btw: why does not have my settings any effect? I passed the options (MAXs and KEX) as the example shows in documentation.

[cyber]

Cannot establish SSH session: kex error : no match for method mac algo client->server:
server [hmac-sha1,hmac-md5],
client [[email protected],[email protected],hmac-sha2-256,hmac-sha2-512]

I assume Server is your mikrotik and client is Zabbix. As you see in your own error message, your server knows 2 algorithms, but your client does not use those... sha1 and md5 are quite long been considered as weak crypto and are possibly just not enabled in your Zabbix server... I suspect that even if you try to use specific algorithm through item config, it does not work, if not enabled systemwide. And if you manage to enable it systemwide, you don't need to specify it in item parameters..

[MRedbourne]

I agree with Cyber here. So, a couple things.

1. Make sure strong-crypto is turned ON.
   1. SHA-256/SHA2 should be mutually supported HMACs.
   2. Client: [email protected]
      Client: [email protected]
      Server: use sha256 for hashing instead of sha1
2. When you enable strong-crypto, re-scan the Mikrotik router. If the new HMACs aren't showing up, you've missed something. if they are showing up, and Zabbix still won't SSH into it, copy/paste the newly generated error.

I would perhaps advise you also contract Mikrotik about the SSH configuration on your Mikrotik equipment. Although I'm sure many users here use Mikrotik, this is a Zabbix support forum, not a Mikrotik support forum. I briefly googled mikrotik service control (eg: their equivalent to systemd), but haven't seen anything. In fact, (and this is really dated), back in '06 you couldn't manually restart individual services, you had to kick the entire router... Which seems a little asinine.

[airween]

Hi cyber,

thanks for your reply!

I assume Server is your mikrotik and client is Zabbix.

yes,

As you see in your own error message, your server knows 2 algorithms, but your client does not use those... sha1 and md5 are quite long been considered as weak crypto and are possibly just not enabled in your Zabbix server...

Sure, but how can I enable it?

I suspect that even if you try to use specific algorithm through item config, it does not work, if not enabled systemwide. And if you manage to enable it systemwide, you don't need to specify it in item parameters..

I see, but now I'm a bit confused: Zabbix uses libssh2 not the system's openssh. I can configure system's openssh, but can't libssh2 (or can I? How?)

Btw I'm not sure I can enable it system-wide, because there are many other hosts, so first I just would try to use it only for this host (and make no conflict with other hosts).

As I see, libssh2 still supports the mentioned protocols:

https://libssh2.org/ - see "Capabilities and features" -> "MAC hashes": hmac-sha2-512, [email protected], hmac-sha2-256, [email protected], hmac-sha1, hmac-sha1-96, hmac-md5, hmac-md5-96, hmac-ripemd160 ([email protected]) - or am I wrong?

[cyber]

that libssh2 is no issue here... your ssh config matters... "ssh -Q kex" should show you available key exchange algorithms.. You should look around here https://manpages.debian.org/buster/o...nfig.5.en.html probably.

[airween]

Hi cyber,

thanks again for your help!

that libssh2 is no issue here... your ssh config matters... "ssh -Q kex" should show you available key exchange algorithms.. You should look around here https://manpages.debian.org/buster/o...nfig.5.en.html probably.

right. Btw I checked the system-wide ssh_config, and this setting was already there:

KexAlgorithms diffie-hellman-group1-sha1

Now I added `MACs` option too, and the error message has changed, I get this one:

Public key try failed: The key algorithm 'ssh-rsa' is not allowed to be used by PUBLICKEY_ACCEPTED_TYPES configuration option

but I don't find any relevant solution for that. Here are my modifications:

PubkeyAcceptedKeyTypes +ssh-rsa
HostKeyAlgorithms +ssh-rsa
KexAlgorithms diffie-hellman-group1-sha1
MACs hmac-sha1,hmac-md5
PubkeyAcceptedAlgorithms +ssh-rsa

Do you have any suggestion? Thanks.

[cyber]

TBH no...I have never needed to configure any ssh server/client for specific algorithms...

[airween]

Okay, many thanks for your help.

[tim.mooney]

Are you sure you know which of the two possible libraries zabbix_server is linked against? It seems like you've been assuming libssh2, but I don't know if that's true. Use 'ldd' on the zabbix_server binary and find which ssh library shows up, and then use your package management commands to figure out which package that's part of.

Have you tried briefly increasing your zabbix_server logging to debug level and looking for messages in the log from the function

ssh_set_options()

? Those messages should tell you if your ssh options are being successfully applied before the poller executes ssh_run().

Remember to turn your logging back to normal after you've captured log entries for the ssh run for that item, otherwise you'll have huge logs.

[airween]

Hi tim.mooney,

thanks for your reply,

Are you sure you know which of the two possible libraries zabbix_server is linked against? It seems like you've been assuming libssh2, but I don't know if that's true. Use 'ldd' on the zabbix_server binary and find which ssh library shows up, and then use your package management commands to figure out which package that's part of.

I downloaded it from Zabbix "official" repository.
It uses more libssh library:

# ldd /usr/sbin/zabbix_server | grep ssh
libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f75cede0000)
libssh2.so.1 => /lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f75cdf3f000)

Have you tried briefly increasing your zabbix_server logging to debug level and looking for messages in the log from the function

ssh_set_options()

? Those messages should tell you if your ssh options are being successfully applied before the poller executes ssh_run().

Remember to turn your logging back to normal after you've captured log entries for the ssh run for that item, otherwise you'll have huge logs.

I already tried it with debuglevel 5, but I can't find any pattern you wrote (ssh_set_option).

[tim.mooney]

I downloaded it from Zabbix "official" repository.
It uses more libssh library:

# ldd /usr/sbin/zabbix_server | grep ssh
libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f75cede0000)
libssh2.so.1 => /lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f75cdf3f000)

That's a surprise, and it seems wrong to me. They don't build zabbix_server that way for RHEL (it's only linked against libssh on RHEL 8, at least. I didn't check RHEL 9.)

I'm not sure what to make of ssh_set_options not appearing in the debug log output. It's possible I misunderstood the code there, or it's possible it's not getting called for some reason (there are code reasons why it wouldn't be called, but I didn't expect any of them to be true for your versions).

Do you see ssh_run appear in the debug output you have?

[airween]

Hi tim.mooney,

thanks again.

That's a surprise, and it seems wrong to me. They don't build zabbix_server that way for RHEL (it's only linked against libssh on RHEL 8, at least. I didn't check RHEL 9.)

Here is what I see:

# apt-cache show zabbix-server-mysql
Package: zabbix-server-mysql
Source: zabbix
Version: 1:6.0.26-1+debian12
Architecture: amd64
Maintainer: Zabbix Packager <[email protected]>

I'm not sure what to make of ssh_set_options not appearing in the debug log output. It's possible I misunderstood the code there, or it's possible it's not getting called for some reason (there are code reasons why it wouldn't be called, but I didn't expect any of them to be true for your versions).

Do you see ssh_run appear in the debug output you have?

Yes, there is:

2095311:20240227:233459.630 key:'ssh.run[ping_192.168.7.2]'
...
2095311:20240227:233459.631 In get_value() key:'ssh.run[ping_192.168.7.2]'
2095311:20240227:233459.632 In ssh_run()
2095311:20240227:233459.804 ssh_run() supported authentication methods: password, publickey
2095311:20240227:233459.805 End of ssh_run():NOTSUPPORTED
2095311:20240227:233459.805 Item [:ssh.run[ping_192.168.7.2]] error: Public key try failed: The key algorithm 'ssh-rsa' is not allowed to be used by PUBLICKEY_ACCEPTED_TYPES configuration option
2095311:20240227:233459.805 End of get_value():NOTSUPPORTED
2095311:20240227:233459.805 End of zbx_trapper_item_test_run():FAIL

A small note: the monitored device's IP is not this (192.168.7.2). The Mikrotik device IP address is 10.2.1.4, and SSH wants to run a command: "ping 192.168.7.2".