zabbix_get timeout on some items

[dignus]

I'm getting timeouts using zabbix_get on some items.

Situation:
Server: Zabbix proxy &zabbix_get installed on debian 11 (6.0.12-1-debian11)
Client: zabbix agent2 installed on CentOS 7 (zabbix-agent2-6.0.12-release1.el7.x86_64)

When I execute a zabbix_get for uptime for example, I receive immediate response:

root@proxy:~# zabbix_get -s 10.29.55.103 -k system.uptime
28144426

Other items work fine as well. However, when I want to discover filesystems, I'm getting a timeout:

root@proxy:~# zabbix_get -s 10.29.55.103 -k vfs.fs.discovery
zabbix_get [3693099]: Timeout while executing operation

However, on the client side I see the following in the agent logs:

2023/01/12 13:12:16.904023 received passive check request: 'vfs.fs.discovery' from '10.29.55.6'
2023/01/12 13:12:16.904108 [1] processing update request (1 requests)
2023/01/12 13:12:16.904115 [1] registering new client
2023/01/12 13:12:16.904169 [1] adding new request for key: 'vfs.fs.discovery'
2023/01/12 13:12:16.904192 [1] created direct exporter task for plugin 'VfsFs' itemid:0 key 'vfs.fs.discovery'
2023/01/12 13:12:16.904251 executing direct exporter task for key 'vfs.fs.discovery'
2023/01/12 13:12:16.904503 executed direct exporter task for key 'vfs.fs.discovery'
2023/01/12 13:12:16.904521 sending passive check response: '[{"{#FSNAME}":"/","{#FSTYPE}":"rootfs"},{"{#FSNAME}":"/sys","{#FSTYPE}":"sysfs"},{"{#FSNAME}":"/proc","{#FSTYPE}":"proc"},{"{#FSNAME}":"/dev","{#FSTYPE}":"devtmpfs"},{"{#FSNAME}":"/sys/kernel/security","{#FSTYPE}":"securityfs"},{"{#FSNAME}":"/dev/shm","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/dev/pts","{#FSTYPE}":"devpts"},{"{#FSNAME}":"/run","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/sys/fs/cgroup","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/sys/fs/cgroup/systemd","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/pstore","{#FSTYPE}":"pstore"},{"{#FSNAME}":"/sys/firmware/efi/efivars","{#FSTYPE}":"efivarfs"},{"{#FSNAME}":"/sys/fs/cgroup/blkio","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/memory","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/cpu,cpuacct","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/devices","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/cpuset","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/net_cls,net_prio","{#FSTYPE}":"cgroup"},{"{#FSNAME }":"/sys/fs/cgroup/freezer","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/pids","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/perf_event","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/hugetlb","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/kernel/config","{#FSTYPE}":"configfs"},{"{#FSNAME}":"/","{#FSTYPE}":"xfs"},{"{#FSNAME}":"/sys/kernel/debug","{#FSTYPE}":"debugfs"},{"{#FSNAME}":"/proc/sys/fs/binfmt_misc","{#FSTYPE}":"autofs"},{"{#FSNAME}":"/dev/mqueue","{#FSTYPE}":"mqueue"},{"{#FSNAME}":"/run/storpool/hugetlb","{#FSTYPE}":"hugetlbfs"},{"{#FSNAME}":"/dev/hugepages","{#FSTYPE}":"hugetlbfs"},{"{#FSNAME}":"/boot","{#FSTYPE}":"xfs"},{"{#FSNAME}":"/boot/efi","{#FSTYPE}":"vfat"},{"{#FSNAME}":"/run/user/0","{#FSTYPE}":"tmpfs"}]' to '10.29.55.6'

Retrieving the same key locally on the client does give me the correct response:

[root@client ~]# zabbix_agent2 -t vfs.fs.discovery
vfs.fs.discovery [s|[{"{#FSNAME}":"/","{#FSTYPE}":"rootfs"},{"{#FSNAME}":"/sys","{#FSTYPE}":"sysfs"},{"{#FSNAME}":"/proc","{#FSTYPE}":"proc"},{"{#FSNAME}":"/dev","{#FSTYPE}":"devtmpfs"},{"{#FSNAME}":"/sys/kernel/security","{#FSTYPE}":"securityfs"},{"{#FSNAME}":"/dev/shm","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/dev/pts","{#FSTYPE}":"devpts"},{"{#FSNAME}":"/run","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/sys/fs/cgroup","{#FSTYPE}":"tmpfs"},{"{#FSNAME}":"/sys/fs/cgroup/systemd","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/pstore","{#FSTYPE}":"pstore"},{"{#FSNAME}":"/sys/firmware/efi/efivars","{#FSTYPE}":"efivarfs"},{"{#FSNAME}":"/sys/fs/cgroup/blkio","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/memory","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/cpu,cpuacct","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/devices","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/cpuset","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/net_cls,net_prio","{#FSTYPE}":"cgroup"},{"{#FSNAME }":"/sys/fs/cgroup/freezer","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/pids","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/perf_event","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/fs/cgroup/hugetlb","{#FSTYPE}":"cgroup"},{"{#FSNAME}":"/sys/kernel/config","{#FSTYPE}":"configfs"},{"{#FSNAME}":"/","{#FSTYPE}":"xfs"},{"{#FSNAME}":"/sys/kernel/debug","{#FSTYPE}":"debugfs"},{"{#FSNAME}":"/proc/sys/fs/binfmt_misc","{#FSTYPE}":"autofs"},{"{#FSNAME}":"/dev/mqueue","{#FSTYPE}":"mqueue"},{"{#FSNAME}":"/run/storpool/hugetlb","{#FSTYPE}":"hugetlbfs"},{"{#FSNAME}":"/dev/hugepages","{#FSTYPE}":"hugetlbfs"},{"{#FSNAME}":"/boot","{#FSTYPE}":"xfs"},{"{#FSNAME}":"/boot/efi","{#FSTYPE}":"vfat"},{"{#FSNAME}":"/run/user/0","{#FSTYPE}":"tmpfs"}]]​

When stracing the zabbix_get command it's stuck here:

socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 3
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
alarm(30) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(10050), sin_addr=inet_addr("10.29.55.103")}, 16) = 0
write(3, "ZBXD\1\20\0\0\0\0\0\0\0vfs.fs.discovery", 29) = 29
read(3,

Looks like the data the agent is sending back, never arrives at the server I run zabbix_get from.

So... network communication seems OK, I can retrieve other items. The agent's output for this key is also good when executed from localhost. However.. not for fs discovery. Any ideas what's going wrong?

Issue resolved: MTU misconfiguration in the path from 1 to the other machine.

[Markku]

Issue resolved: MTU misconfiguration in the path from 1 to the other machine.

Thanks for confirming, I would have suggested verifying the MTU

Markku