Zabbix server logs

[yatta]

Hi,
I just installed Zabbix 6.0 on Suse and trying to add Linux servers.
But in zabbix_server.log file, there are lots of entries like below;

3290:20220316:140211.365 item "Zabbix server:vfs.file.contents[/sys/block/sdcm/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.366 item "Zabbix server:vfs.file.contents[/sys/block/sdcl/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.366 item "Zabbix server:vfs.file.contents[/sys/block/sdck/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.366 item "Zabbix server:vfs.file.contents[/sys/block/sdcj/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.366 item "Zabbix server:vfs.file.contents[/sys/block/sdci/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.366 item "Zabbix server:vfs.file.contents[/sys/block/sdch/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.366 item "Zabbix server:vfs.file.contents[/sys/block/sdcg/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.366 item "Zabbix server:vfs.fs.size[/PWS,total]" became not supported: Cannot obtain filesystem information: [2] No such file or directory
3290:20220316:140211.367 item "Zabbix server:vfs.file.contents[/sys/block/sdcs/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.367 item "Zabbix server:vfs.file.contents[/sys/block/sdcr/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.367 item "Zabbix server:vfs.file.contents[/sys/block/sdcq/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.367 item "Zabbix server:vfs.file.contents[/sys/block/sdcp/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.367 item "Zabbix server:vfs.file.contents[/sys/block/sdco/stat]" became not supported: Cannot open file: [2] No such file or directory
3290:20220316:140211.367 item "Zabbix server:vfs.file.contents[/sys/block/sdcn/stat]" became not supported: Cannot open file: [2] No such file or directory

Any idea what are these?
Thanks!

[Glencoe]

My guess is that you have either broken sysfs (very unlikely) or block device discovery discovered many devices that have been removed since then (more likely)

1. What distro / kernel version do you run?
2. What do commands mount and ls -l /sys/block say?
3. What does zabbix_agentd -t vfs.dev.discovery say?

[yatta]

My guess is that you have either broken sysfs (very unlikely) or block device discovery discovered many devices that have been removed since then (more likely)

1. What distro / kernel version do you run?
2. What do commands mount and ls -l /sys/block say?
3. What does zabbix_agentd -t vfs.dev.discovery say?

Hello!

1. SLES SP3

2.
zabbix:~ # ls -l /sys/block
total 0
lrwxrwxrwx 1 root root 0 Mar 17 11:26 dm-0 -> ../devices/virtual/block/dm-0
lrwxrwxrwx 1 root root 0 Mar 17 11:26 dm-1 -> ../devices/virtual/block/dm-1
lrwxrwxrwx 1 root root 0 Mar 17 11:26 dm-2 -> ../devices/virtual/block/dm-2
lrwxrwxrwx 1 root root 0 Mar 17 11:26 sda -> ../devices/pci0000:00/0000:00:10.0/host33/target33:0:0/33:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Mar 17 11:26 sdb -> ../devices/pci0000:00/0000:00:10.0/host33/target33:0:1/33:0:1:0/block/sdb


zabbix:~ # mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,size=4096k,nr_inodes=65536,mode= 755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=7680000k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode =000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=1628612k,nr_inodes=819200,mo de=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024, mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=system d)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
/dev/mapper/system-root on / type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,max proto=5,direct,pipe_ino=15834)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/mapper/system-root on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/.snapshots)
/dev/mapper/system-root on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/boot/grub2/x86_64-efi)
/dev/mapper/system-root on /home type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/home)
/dev/mapper/system-root on /tmp type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/tmp)
/dev/mapper/system-root on /opt type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/opt)
/dev/mapper/system-root on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/boot/grub2/i386-pc)
/dev/mapper/system-root on /var/lib/libvirt/images type btrfs (rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/libvirt/images)
/dev/mapper/system-root on /srv type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/srv)
/dev/mapper/system-root on /usr/local type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/usr/local)
/dev/mapper/system-root on /var/crash type btrfs (rw,relatime,space_cache,subvolid=268,subvol=/@/var/crash)
/dev/mapper/system-root on /var/cache type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/var/cache)
/dev/mapper/system-root on /var/opt type btrfs (rw,relatime,space_cache,subvolid=277,subvol=/@/var/opt)
/dev/mapper/system-root on /var/lib/machines type btrfs (rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/machines)
/dev/mapper/system-root on /var/lib/mailman type btrfs (rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/mailman)
/dev/mapper/system-root on /var/lib/mariadb type btrfs (rw,relatime,space_cache,subvolid=272,subvol=/@/var/lib/mariadb)
/dev/mapper/system-root on /var/log type btrfs (rw,relatime,space_cache,subvolid=276,subvol=/@/var/log)
/dev/mapper/system-root on /var/lib/named type btrfs (rw,relatime,space_cache,subvolid=274,subvol=/@/var/lib/named)
/dev/mapper/system-root on /var/lib/pgsql type btrfs (rw,relatime,space_cache,subvolid=275,subvol=/@/var/lib/pgsql)
/dev/mapper/system-root on /var/lib/mysql type btrfs (rw,relatime,space_cache,subvolid=273,subvol=/@/var/lib/mysql)
/dev/mapper/system-root on /var/spool type btrfs (rw,relatime,space_cache,subvolid=278,subvol=/@/var/spool)
/dev/mapper/system-root on /var/tmp type btrfs (rw,relatime,space_cache,subvolid=279,subvol=/@/var/tmp)
/dev/mapper/vgmysql-vol_mysql on /opt/mysql type ext4 (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/etc/auto.install on /install type autofs (rw,relatime,fd=6,pgrp=1389,timeout=60,minproto=5, maxproto=5,direct,pipe_ino=30686)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=814304k,nr_inodes=2 03576,mode=700)


3.
zabbix:~ # zabbix_agentd -t vfs.dev.discovery
vfs.dev.discovery [s|[{"{#DEVNAME}":"dm-2","{#DEVTYPE}":"disk"},{"{#DEVNAME}":"dm-1","{#DEVTYPE}":"disk"},{"{#DEVNAME}":"dm-0","{#DEVTYPE}":"disk"},{"{#DEVNAME}":"sda1","{ #DE VTYPE}":"partition"},{"{#DEVNAME}":"sdb","{#DEVTYP E}":"disk"},{"{#DEVNAME}":"sda","{#DEVTYPE}":"di sk "}]]

[Glencoe]

Ok, so these block devices (sdcr,sdcs,sdcq etc) are not being discovered anymore. I don't know what happened on your system but at some point there were many these devices which then have been picked by Zabbix during block device discovery. These are not in the system anymore but Zabbix is still trying to poll these because "Keep lost resources" is set to 30 days in the template.

You have the following options:

1. Delete these items Zabbix server:vfs.file.contents[/sys/block/*/stat] that are in unsupported state.
2. Wait until "Keep lost resources" expired. This may be a long wait though.
3. Set "Keep lost resources" to whatever is more reasonable for you (for me, 1 day works well, but ymmv) and wait until the discovery rule fires next time (or invoke it manually with Execute now)

[yatta]

Thanks for the information!