Filesystem discovery not working

[cunningdavid]

We have a server with a filesystem that's not being discovered as expected. The filesystem doesn't show up on the Zabbix server's "Latest data" page for the client, and we can't figure out why.

The Zabbix server has a discovery rule with:

Type: Zabbix agent (active) -- note that the client is behind NAT and cannot be directly accessed from the server
Key: vfs.fs.discovery
Filters: {#FSTYPE} matches ext

Here's the filesystem on the Zabbix client:

/dev/mapper/u01--vg-u01--lv on /u01 type ext4 (rw,relatime)

The Zabbix client (version 5.0.17) can see the filesystem as follows:

root@myserver:~# /usr/sbin/zabbix_agentd -t vfs.fs.discovery
vfs.fs.discovery [s|[{"{#FSNAME}":"/sys","{#FSTYPE}":"sysfs"},{"{#FSNAME}":"/proc","{#FSTYPE}":"proc"},{"{#FSNAME}":"/dev","{#FSTYPE}":"devtmpfs"},{"{#FSNAME}":"/dev/pts","{#FSTYPE}":"devpts"},{"{#FSNAME}":"/run","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/","{#FSTYPE}":"ext4"},{"{#FSNAME}":"/sys/kernel/security","{#FSTYPE}":"securityfs"},{"{#FSNAME}":"/dev/shm","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/run/lock","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/sys/fs/cgroup","{#FSTYPE}":"cgroup2"},{"{#FSNAME}":"/sys/fs/pstore","{#FSTYPE}":"pstore"},{"{#FSNAME}":"/sys/fs/bpf","{#FSTYPE}":"bpf"},{"{#FSNAME}":"/proc/sys/fs/binfmt_misc","{#FSTYPE}":"autofs"},{"{#FSNAME}":"/dev/hugepages","{#FSTYPE}":"hugetlbfs"},{"{#FSNAME}":"/dev/mqueue","{#FSTYPE}":"mqueue"},{"{#FSNAME}":"/sys/kernel/debug","{#FSTYPE}":"debugfs"},{"{#FSNAME}":"/sys/kernel/tracing","{#FSTYPE}":"tracefs"},{"{#FSNAME}":"/sys/fs/fuse/connections","{#FSTYPE}":"fusectl"},{"{#FSNAME}":"/sys/kernel/config","{#FSTYPE}":"configfs"},{"{#FSNAME}":"/run/credentials/systemd-sysusers.service","{#FSTYPE}":"ramfs"},{"{#FSNAME} ":"/snap/lxd/24322","{#FSTYPE}":"squashfs"},{"{#FSNAME}":"/boot","{#FSTYPE}":"ext4"},{"{#FSNAME}":"/u02","{#FSTYPE}":"ext4"},{"{#FSNAME}":"/u01","{#FSTYPE}":"ext4"},{"{#FSNAME}":"/proc/sys/fs/binfmt_misc","{#FSTYPE}":"binfmt_misc"},{"{#FSNAME }":"/run/snapd/ns","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/run/snapd/ns/lxd.mnt","{#FSTYPE}":"nsfs"},{"{#FSNAME}":"/snap/core20/1852","{#FSTYPE}":"squashfs"},{"{#FSNAME}":"/snap/snapd/18933","{#FSTYPE}":"squashfs"},{"{#FSNAME}":"/snap/snapd/19122","{#FSTYPE}":"squashfs"},{"{#FSNAME}":"/snap/core20/1879","{#FSTYPE}":"squashfs"},{"{#FSNAME}":"/run/user/1001","{#FSTYPE}":"tmpfs"}]]

And the filesystem also appears in /var/log/zabbix-agent/zabbix_agentd.log when the log level is set to debug. So why it the filesystem not appearing on the Zabbix server?

Thanks in advance for any help.

[cyber]

Only that one FS is ignored or all of them?
Is there any data at all coming in from that agent?

[cunningdavid]

There are two filesystems that should be discovered, and neither of them are. They are /dev/mapper/u01--vg-u01--lv on /u01 and /dev/mapper/u02--vg-u02--lv on /u02. All the other Zabbix data is coming through to the server successfully. Any ideas?

[cyber]

Based on filter "ext" you should also find "/" and "/boot" from that host... As your zabbix agent reports all data in test mode, then either that data is not reaching your server or there is some more filters or overrides, that prevents creating of items..

[cunningdavid]

The / filesystem is already monitored by static (non discovered) item, so I assumed that it wouldn't be duplicated. But yes /boot should be discovered as well.

Can you see any flaws in my discovery rule? Thank you.

[cyber]

Nothing related to that host in server logs? Something about not being able to communicate or anything else? I still have somekind of gut feeling, it is related to communication...

I cannot see anything wrong that would stick out from the things you provided..

[cunningdavid]

I enabled debug logging on the server and we see the "vfs.fs.discovery" data there. It is in a slightly different format than some other clients.

The problem client is running version 5.0.17 and gets logged by the server like this:

31905:20230518:073522.794 trapper got '{"request":"agent data","session":"42d60dedf6bc2716f4c0e9275907cbcb" ,"data":[{"host":"example.com","key":"vfs.fs.discovery", vfs .fs.discovery","value":"[{\"{#FSNAME}\":\"/sys\",\"{#FSTYPE}\":\"sysfs\"},{\"{#FSNAME}\":\"/proc\",\"{#FSTYPE}\":\"proc\"},{\"{#FSNAME}\":\"/dev\",\"{#FSTYPE}\":\"devtmpfs\"},{\"{#FSNAME}\":\ "/dev/pts\",\"{#FSTYPE}\":\"devpts\"},{\"{#FSNAME}\":\"/run\",\"{#FSTYPE}\":\"tmpfs\"},{\"{#FSNAME}\":\"/\",\"{#FSTYPE}\":\"ext4\"},{\"{#FSNAME}\":\"/sys/kernel/security\",\"{#FSTYPE}\":\"securityfs\"},{\"{#FSNA ME}\":\"/dev/shm\",\"{#FSTYPE}\":\"tmpfs\"},{\"{#FSNAME}\":\"/run/lock\",\"{#FSTYPE}\":\"tmpfs\"},{\"{#FSNAME}\":\"/sys/fs/cgroup\",\"{#FSTYPE}\":\"cgroup2\"},{\"{#FSNAME}\" :\"/sys/fs/pstore\",\"{#FSTYPE}\":\"pstore\"},{\"{#FSNAME}\": \"/sys/fs/bpf\",\"{#FSTYPE}\":\"bpf\"},{\"{#FSNAME}\":\"/proc/sys/fs/binfmt_misc\",\"{#FSTYPE}\":\"autofs\"},{\"{#FSNAM E}\":\"/dev/hugepages\",\"{#FSTYPE}\":\"hugetlbfs\"},{\"{#FSNA ME}\":\"/dev/mqueue\",\"{#FSTYPE}\":\"mqueue\"},{\"{#FSNAME}\": \"/sys/kernel/debug\",\"{#FSTYPE}\":\"debugfs\"},{\"{#FSNAME}\": \"/sys/kernel/tracing\",\"{#FSTYPE}\":\"tracefs\"},{\"{#FSNAME}\ ":\"/sys/fs/fuse/connections\",\"{#FSTYPE}\":\"fusectl\"},{\"{#FSNA ME}\":\"/sys/kernel/config\",\"{#FSTYPE}\":\"configfs\"},{\"{#FSNAME}\ ":\"/run/credentials/systemd-sysusers.service\",\"{#FSTYPE}\":\"ramfs\"},{\"{#F SNAME}\":\"/snap/lxd/24322\",\"{#FSTYPE}\":\"squashfs\"},{\"{#FSNAME}\" :\"/boot\",\"{#FSTYPE}\":\"ext4\"},{\"{#FSNAME}\":\"/u02\",\"{#FSTYPE}\":\"ext4\"},{\"{#FSNAME}\":\"/u01\",\"{#FSTYPE}\":\"ext4\"},{\"{#FSNAME}\":\"/proc/sys/fs/binfmt_misc\",\"{#FSTYPE}\":\"binfmt_misc\"},{\"{# FSNAME}\":\"/run/snapd/ns\",\"{#FSTYPE}\":\"tmpfs\"},{\"{#FSNAME}\":\"/run/snapd/ns/lxd.mnt\",\"{#FSTYPE}\":\"nsfs\"},{\"{#FSNAME}\":\ "/snap/snapd/18933\",\"{#FSTYPE}\":\"squashfs\"},{\"{#FSNAME}\" :\"/snap/snapd/19122\",\"{#FSTYPE}\":\"squashfs\"},{\"{#FSNAME}\" :\"/snap/core20/1879\",\"{#FSTYPE}\":\"squashfs\"},{\"{#FSNAME}\": \"/snap/core20/1891\",\"{#FSTYPE}\":\"squashfs\"}]",...

Whereas another client running version 2.2.23 gets logged like this:

30240:20230518:072518.327 trapper got '{
"request":"agent data",
"data":[
...
​ {
"host":"other.com",
"key":"vfs.fs.discovery",
"value":"{\n\t\"data\":[\n\t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/sys\",\n\t\t\t\"{#FSTYPE}\":\"sysfs\"},\n\t\t{\n\t \t\t\"{#FSNAME}\":\"\\\/proc\",\n\t\t\t\"{#FSTYPE}\":\"proc\"},\n\t\t{\n\t \t\t\"{#FSNAME}\":\"\\\/dev\",\n\t\t\t\"{#FSTYPE}\":\"devtmpfs\"},\n\t\t{\ n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/kernel\\\/security\",\n\t\t\t\"{#FSTYPE}\":\"securityfs\"},\ n\t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/dev\\\/shm\",\n\t\t\t\"{#FSTYPE}\":\"tmpfs\"},\n\t\t{\n\t \t\t\"{#FSNAME}\":\"\\\/dev\\\/pts\",\n\t\t\t\"{#FSTYPE}\":\"devpts\"},\n\t\t{\n\ t\t\t\"{#FSNAME}\":\"\\\/run\",\n\t\t\t\"{#FSTYPE}\":\"tmpfs\"},\n\t\t{\n\t \t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\",\n\t\t\t\"{#FSTYPE}\":\"tmpfs\"},\n\t\t{\ n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/systemd\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t {\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/pstore\",\n\t\t\t\"{#FSTYPE}\":\"pstore\"},\n\t\t{ \n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/memory\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t{ \n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/cpu,cpuacct\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n \t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/net_cls,net_prio\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\ "},\n\t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/blkio\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t{\ n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/pids\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t{\n \t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/devices\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t {\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/freezer\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t {\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/hugetlb\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t {\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/perf_event\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\ t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/cgroup\\\/cpuset\",\n\t\t\t\"{#FSTYPE}\":\"cgroup\"},\n\t\t{ \n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/kernel\\\/config\",\n\t\t\t\"{#FSTYPE}\":\"configfs\"},\n\t\ t{\n\t\t\t\"{#FSNAME}\":\"\\\/\",\n\t\t\t\"{#FSTYPE}\":\"xfs\"},\n\t\t{\n\t\t\t\ "{#FSNAME}\":\"\\\/proc\\\/sys\\\/fs\\\/binfmt_misc\",\n\t\t\t\"{#FSTYPE}\":\"autofs\"},\n \t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/dev\\\/mqueue\",\n\t\t\t\"{#FSTYPE}\":\"mqueue\"},\n\t\t{ \n\t\t\t\"{#FSNAME}\":\"\\\/dev\\\/hugepages\",\n\t\t\t\"{#FSTYPE}\":\"hugetlbfs\"},\ n\t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/kernel\\\/debug\",\n\t\t\t\"{#FSTYPE}\":\"debugfs\"},\n\t\t{ \n\t\t\t\"{#FSNAME}\":\"\\\/sys\\\/fs\\\/fuse\\\/connections\",\n\t\t\t\"{#FSTYPE}\":\"fusectl\"},\ n\t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/boot\",\n\t\t\t\"{#FSTYPE}\":\"xfs\"},\n\t\t{\n\t\ t\t\"{#FSNAME}\":\"\\\/var\\\/lib\\\/nfs\\\/rpc_pipefs\",\n\t\t\t\"{#FSTYPE}\":\"rpc_pipefs\"} ,\n\t\t{\n\t\t\t\"{#FSNAME}\":\"\\\/run\\\/user\\\/1001\",\n\t\t\t\"{#FSTYPE}\":\"tmpfs\"},\n\t\t{\n\ t\t\t\"{#FSNAME}\":\"\\\/run\\\/user\\\/42\",\n\t\t\t\"{#FSTYPE}\":\"tmpfs\"},\n\t\t{\n\t\ t\t\"{#FSNAME}\":\"\\\/run\\\/user\\\/0\",\n\t\t\t\"{#FSTYPE}\":\"tmpfs\"}]}",
"clock":1684394717,
"ns":319037955}],

The server itself is running version 4.0.39. Could the problem be version incompatibility, since the client is running a newer version than the server?

It would be nice if the server logged its interpretation of the "vfs.fs.discovery" data, but that doesn't seem to happen even at log level 5.

Thanks.

[cyber]

yea.. you should not run components with higher version than your server... Things are often backwards compatible, but not forward ...

[cunningdavid]

I see, thank you for that.

[icunginebula]

There could be several reasons why the filesystem is not appearing on the Zabbix server despite being visible on the Zabbix client. Here are some troubleshooting steps you can take to investigate the issue:

Verify connectivity: Ensure that the Zabbix server can communicate with the Zabbix client over the network. Since you mentioned that the client is behind NAT, confirm that the necessary network configurations (such as port forwarding) are correctly set up to allow communication between the server and the client.

Check firewall settings: Make sure that there are no firewall rules blocking the communication between the Zabbix server and the Zabbix client. Check both the server-side and client-side firewalls to ensure that the required ports are open.

Confirm Zabbix agent configuration: Double-check the Zabbix agent configuration file on the client side (zabbix_agentd.conf) to ensure that it includes the necessary settings for filesystem discovery. Verify that the EnableRemoteCommands and LogRemoteCommands parameters are set appropriately to allow the Zabbix server to execute remote commands on the client.

Review Zabbix server logs: Check the Zabbix server logs (zabbix_server.log) for any relevant error messages or warnings related to the filesystem discovery. Look for entries that may provide clues about why the filesystem is not appearing.

Verify Zabbix server discovery rule configuration: Review the discovery rule configuration on the Zabbix server to ensure it is correctly set up. Check the filter expression ({#FSTYPE} matches ext) and confirm that it matches the filesystem type of the target filesystem on the client.

Confirm Zabbix server item configuration: Make sure that the Zabbix server has the corresponding item configured to collect data from the discovered filesystems. Verify that the item key and other settings are accurately defined.

By following these troubleshooting steps, you should be able to narrow down the issue and identify any configuration or connectivity problems that may be causing the filesystem to not appear on the Zabbix server's "Latest data" page.