/usr/bin/ssh and /sbin/sshd cksums are actually the servers.

[joshuamcdo]

Zabbix Version : 2.0.16
O/S : Amazon linux
Database : mySQL (RDS)
Keys from template :
vfs.file.cksum[/usr/sbin/sshd]
vfs.file.cksum[/usr/bin/ssh]

I recently did some SSL, ssh and other system updates on our zabbix server. I restarted apache and the zabbix-server service and a few minutes later I received 200+ alerts.

Basically every single host alerted to a cksum change for /sbin/sshd and /usr/bin/ssh .

I found this curious and started comparing the last 500 values with the current value, and oddly enough none of them matched. On a hunch I decided to check the servers cksums for the same bins and to my surprise, they matched.


Is this a bug? A known issue? I inherited this and always assumed that the keys were stock form.

Thanks in advance,
J

[joshuamcdo]

Re: /usr/bin/ssh and /sbin/sshd cksums are actually the servers.

Hmm.. Bump?

This seems like something that should concern many as I don't currently know of any way to "screw this up", meaning it could be a for serious bug thing.

J

[scout]

If you updated ssh and sshd programs on the server then chksum should differ because previously value was different than newly installed/ updated ssh/ sshd programs.

Can you check chksum of old ssh/ sshd program on the server and then on the different server of new sshd?

I.e. downgrade on one system ssh and compare chksum on both systems

[joshuamcdo]

Re:Re:Re: /usr/bin/ssh and /sbin/sshd cksums are actually the servers.

So.. Maybe I am not explaining this correctly.

Let's say I have the following config.

zabbix-server.c0m <-Main point of monitoring for Zabbix.

I have instances

webhost01.c0m webhost02.c0m webhost03.c0m

I apply the Linux_OS template to them because they are all running amazon linux.
In that template, the cksums of the /usr/bin/ssh and /sbin/sshd are supposed to be monitored and sent back to the zabbix server along with many other values that are in fact monitored.

One day I decided to update security related packages on zabbix-server.com, and nothing else.

Within minutes I am slammed with 200+ alerts stating that the cksum values for /usr/bin/ssh and /sbin/sshd have changed on webhost01.c0m webhost02.c0m webhost03.c0m . But I didn't touch the packages on those hosts. They have remained the same.

When I look closer, the values in history never matched the values on webhost01.c0m webhost02.c0m webhost03.c0m, they match the values on the zabbix-server.c0m. I shouldn't be getting alerts for webhost01.c0m webhost02.c0m webhost03.c0m when I only updated the packages on zabbix-server.c0m .

Does that make more sense?

Thanks in advance.

J

[joshuamcdo]

/usr/bin/ssh and /sbin/sshd cksums are actually the servers.

^bump^
^
(coughs)

[joshuamcdo]

RE:RE:RE:RE:/usr/bin/ssh and /sbin/sshd cksums are actually the servers.

Anyone? (cough cough cough)

[joshuamcdo]

...Bump...

[joshuamcdo]

Am I missing important information that is required to garner a reply? This seems like a potentially critical issue.

J

[Pedro.Almeida]

Have no idea on how you have your templates setup but:

[root@xxxx ~]# zabbix_get -s 127.0.0.1 -k vfs.file.cksum[/usr/sbin/sshd]
2113100602
[root@xxxx~]# zabbix_get -s 10.123.7.15 -k vfs.file.cksum[/usr/sbin/sshd]
3295297830

I've configured a vfs.file.cksum[/etc/passwd] (Zabbix Active) and here's what I get:

MariaDB [zabbix]> select distinct hosts.hostid, value, items.itemid, items.name  from history_uint, items, hosts where history_uint.itemid=items.itemid and items.hostid=hosts.hostid and items.templateid=393019;
+--------+------------+--------+----------------+
| hostid | value      | itemid | name           |
+--------+------------+--------+----------------+
|  10437 | 1001666856 | 393020 | Checksum of 1 |
|  10941 |  698600492 | 393021 | Checksum of 1 |
|  11048 |  571406797 | 393022 | Checksum of 1 |
|  11087 | 3465574447 | 393023 | Checksum of 1 |
|  11248 | 1534279434 | 393024 | Checksum of 1 |
|  11401 | 2564005511 | 393026 | Checksum of 1 |
|  11601 |  875718216 | 393028 | Checksum of 1 |
|  11639 | 1595096379 | 393029 | Checksum of 1 |
|  11640 | 1595096379 | 393030 | Checksum of 1 |
|  11436 |  571406797 | 393031 | Checksum of 1 |
|  11653 | 3531279159 | 393032 | Checksum of 1 |
|  11668 |  986837484 | 393033 | Checksum of 1 |
|  11849 |  890927533 | 393034 | Checksum of 1 |
|  11850 | 1443653487 | 393035 | Checksum of 1 |
|  11851 | 2496271503 | 393036 | Checksum of 1 |
|  10557 |  633173968 | 393037 | Checksum of 1 |
|  11896 | 4199427646 | 393038 | Checksum of 1 |
|  11954 | 2730576235 | 393039 | Checksum of 1 |
|  11996 | 1507060331 | 393040 | Checksum of 1 |
+--------+------------+--------+----------------+
19 rows in set (0.01 sec)

(Just removed the '$' before 1 on the post as the forum was complaining about too many links)

There are only two repeated values for hosts that are in fact clones.
From what I stand it's being calculated at the agent, not at the server.

[joshuamcdo]

So I think I see the problem..

The hosts that are throwing this error when the packages on the server are upgraded are all active connections.. They have their IP set to 127.0.0.1.... I thought this was the correct course of action but will defer myself back to the documentation.

Thanks,
J

[joshuamcdo]

I looked at the docs and it does say to set the ip to 127.0.0.1. However, what I am not seeing is anything that states cksums of files are supported in active mode. I enabled level 4 debugging and the only things I saw the agent delivering were CPU stats, a custom log metric, a customer AWS ECS agent metric and that's it.. This makes me wonder if any of the memory usage metric information that has been reported is even real...

This is a real problem.

J

[joshuamcdo]

Anyone? Surely someone has something to add to this good bad or indifferent doesn't really matter to me. As long as it helps me understand what in the actual is going on here.

Thanks,
J

[al.netrusov]

Can you please attach host configuration screenshot?

Thanks.