Hi all...
I recently upgraded my zabbix 6 LTS to 7, and now since that upgrade I am seeing sever instability with hosts being able to be checked.
From a UI perspective this results in an active check being fine, but interface for the dns host name reporting issues:
If I dig into my zabbix server logs, I can see alot of issues like this...
Specifically I can see the error "Cannot resolve address: nodename nor servname provided, or not known'", which ties in with what is reported in the UI. My google fu points me to a DNS issue, however my DNS seems to be working fine from the terminal of my Zabbix rig. At no point do I get an error when pinging based on the FQDN.
I did some searches and found another post which seems to have similar issues to me...
Everything in that thread looks the same, however I am not using systemd_resolver. My internal DNS server is DNSMasq, and that seems to be working fine, so I can't seem to understand why Zabbix 7 seems to hate on my setup over Zabbix 6.
More logs on another example...
I recently upgraded my zabbix 6 LTS to 7, and now since that upgrade I am seeing sever instability with hosts being able to be checked.
From a UI perspective this results in an active check being fine, but interface for the dns host name reporting issues:
If I dig into my zabbix server logs, I can see alot of issues like this...
Code:
7015:20261027:135855.007 trapper got request 'active check heartbeat' 7015:20261027:135855.007 In zbx_ipc_socket_write() 7015:20261027:135855.007 End of zbx_ipc_socket_write():SUCCEED 7015:20261027:135855.007 zbx_setproctitle() title:'trapper #4 [processed data in 0.000129 sec, waiting for connection]' 7045:20261027:135855.007 End of zbx_ipc_service_recv():1 7045:20261027:135855.007 query [txnlev:1] [begin;] 7045:20261027:135855.007 query [txnlev:1] [update interface set disable_until=1793109591 where interfaceid=35;.update interface set available=2,error='Get value from agent failed: Cannot resolve address: nodename nor servname provided, or not known',disable_until=1793109593 where interfaceid=38;.] 7045:20261027:135855.008 query [txnlev:1] [commit;] 7045:20261027:135855.024 In zbx_ipc_service_recv() timeout:1.000 6990:20261027:135855.038 zbx_setproctitle() title:'http poller #1 [got 0 values in 0.000239 sec, getting values]' 6990:20261027:135855.038 In process_httptests() 6990:20261027:135855.038 End of process_httptests() 6990:20261027:135855.038 zbx_setproctitle() title:'http poller #1 [got 0 values in 0.000170 sec, idle 5 sec]' 7004:20261027:135855.071 zbx_setproctitle() title:'self-monitoring [processing data]' 7004:20261027:135855.071 In collect_selfmon_stats() 7004:20261027:135855.071 In zbx_timekeeper_collect() 7004:20261027:135855.071 End of zbx_timekeeper_collect() 7004:20261027:135855.071 End of collect_selfmon_stats() 7004:20261027:135855.071 zbx_setproctitle() title:'self-monitoring [processed data in 0.000222 sec, idle 1 sec]' 7017:20261027:135855.085 zbx_setproctitle() title:'icmp pinger #1 [getting values]' 7017:20261027:135855.085 In get_pinger_hosts() 7017:20261027:135855.085 In zbx_dc_config_get_poller_items() poller_type:3 7017:20261027:135855.085 End of zbx_dc_config_get_poller_items():0 7017:20261027:135855.085 End of get_pinger_hosts():0 7017:20261027:135855.085 In process_pinger_hosts() 7017:20261027:135855.085 End of process_pinger_hosts() 7017:20261027:135855.085 In zbx_dc_config_get_poller_nextcheck() poller_type:3 7017:20261027:135855.085 End of zbx_dc_config_get_poller_nextcheck():-1 7017:20261027:135855.085 zbx_setproctitle() title:'icmp pinger #1 [got 0 values in 0.000322 sec, idle 5 sec]' 6991:20261027:135855.101 End of zbx_ipc_async_socket_recv():0 6991:20261027:135855.101 zbx_setproctitle() title:'browser poller #1 [got 0 values in 0.000188 sec, getting values]' 6991:20261027:135855.101 In get_values() 6991:20261027:135855.101 In zbx_dc_config_get_poller_items() poller_type:11 6991:20261027:135855.101 End of zbx_dc_config_get_poller_items():0 6991:20261027:135855.101 In zbx_dc_config_get_poller_nextcheck() poller_type:11 6991:20261027:135855.101 End of zbx_dc_config_get_poller_nextcheck():-1 6991:20261027:135855.101 End of get_values():0 6991:20261027:135855.101 zbx_setproctitle() title:'browser poller #1 [got 0 values in 0.000194 sec, idle 5 sec]' 6991:20261027:135855.101 In zbx_ipc_async_socket_recv() timeout:5 7049:20261027:135855.110 zbx_setproctitle() title:'agent poller #1 [got 5 values, queued 5 in 5 sec, awaiting 0]' 7049:20261027:135855.110 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:135855.110 End of zbx_ipc_async_socket_recv():0 7049:20261027:135855.111 interfaces num:1 7049:20261027:135855.111 In poller_update_interfaces() num:1 7049:20261027:135855.111 In zbx_activate_item_interface() interfaceid:33 itemid:49192 type:0 version:70020 7049:20261027:135855.111 resuming Zabbix agent checks on host "home-svr": connection restored 7049:20261027:135855.111 End of zbx_activate_item_interface() 7049:20261027:135855.111 In zbx_ipc_socket_write() 7049:20261027:135855.111 End of zbx_ipc_socket_write():SUCCEED 7049:20261027:135855.111 End of poller_update_interfaces() 7049:20261027:135855.111 In zbx_dc_config_get_poller_items() poller_type:8 7045:20261027:135855.111 End of zbx_ipc_service_recv():1 7045:20261027:135855.111 In zbx_ipc_service_recv() timeout:1.000 7049:20261027:135855.111 End of zbx_dc_config_get_poller_items():2 7049:20261027:135855.111 In zbx_prepare_items() num:2 7049:20261027:135855.111 In substitute_key_macros_impl() data:'vfs.fs.inode[/data,pfree]' 7049:20261027:135855.111 End of substitute_key_macros_impl():SUCCEED data:'vfs.fs.inode[/data,pfree]' 7049:20261027:135855.111 In substitute_simple_macros_impl() data:'10050' 7049:20261027:135855.111 End substitute_simple_macros_impl() data:'10050' 7049:20261027:135855.111 In substitute_simple_macros_impl() data:'4s' 7049:20261027:135855.111 End substitute_simple_macros_impl() data:'4s' 7049:20261027:135855.111 In substitute_key_macros_impl() data:'zfs.get.fsinfo[rpool/data/subvol-105-disk-0,used]' 7049:20261027:135855.111 End of substitute_key_macros_impl():SUCCEED data:'zfs.get.fsinfo[rpool/data/subvol-105-disk-0,used]' 7049:20261027:135855.111 In substitute_simple_macros_impl() data:'10050' 7049:20261027:135855.111 End substitute_simple_macros_impl() data:'10050' 7049:20261027:135855.111 In substitute_simple_macros_impl() data:'4s' 7049:20261027:135855.111 End substitute_simple_macros_impl() data:'4s' 7049:20261027:135855.111 End of zbx_prepare_items() 7049:20261027:135855.111 queue processing_num:0 pending:0 7049:20261027:135855.111 In async_initiate_queued_checks() 7049:20261027:135855.111 In zbx_async_check_agent() itemid:47144 key:'vfs.fs.inode[/data,pfree]' host:'homeassistant' addr:'homeassistant.home.local' conn:'unencrypted' 7049:20261027:135855.111 In zbx_is_ip4() ip:'homeassistant.home.local' 7049:20261027:135855.111 End of zbx_is_ip4():FAIL 7049:20261027:135855.111 In zbx_is_ip6() ip:'homeassistant.home.local' 7049:20261027:135855.112 End of zbx_is_ip6():FAIL 7049:20261027:135855.112 In zbx_is_ip4() ip:'homeassistant.home.local' 7049:20261027:135855.112 End of zbx_is_ip4():FAIL 7049:20261027:135855.112 In zbx_is_ip6() ip:'homeassistant.home.local' 7049:20261027:135855.112 End of zbx_is_ip6():FAIL 7049:20261027:135855.112 End of zbx_async_check_agent():SUCCEED 7049:20261027:135855.112 In zbx_async_check_agent() itemid:47335 key:'zfs.get.fsinfo[rpool/data/subvol-105-disk-0,used]' host:'home-svr' addr:'home-svr.home.local' conn:'unencrypted' 7049:20261027:135855.112 In zbx_is_ip4() ip:'home-svr.home.local' 7049:20261027:135855.112 End of zbx_is_ip4():FAIL 7049:20261027:135855.112 In zbx_is_ip6() ip:'home-svr.home.local' 7049:20261027:135855.112 End of zbx_is_ip6():FAIL 7049:20261027:135855.112 In zbx_is_ip4() ip:'home-svr.home.local' 7049:20261027:135855.112 End of zbx_is_ip4():FAIL 7049:20261027:135855.112 In zbx_is_ip6() ip:'home-svr.home.local' 7049:20261027:135855.112 End of zbx_is_ip6():FAIL 7049:20261027:135855.112 End of zbx_async_check_agent():SUCCEED 7049:20261027:135855.112 End of async_initiate_queued_checks(): num:2 7049:20261027:135855.112 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:135855.112 End of zbx_ipc_async_socket_recv():0 7049:20261027:135855.112 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:135855.112 End of zbx_ipc_async_socket_recv():0 7050:20261027:135855.114 zbx_setproctitle() title:'snmp poller #1 [got 0 values, queued 0 in 5 sec, awaiting 0]' 7050:20261027:135855.114 In zbx_ipc_async_socket_recv() timeout:0 7050:20261027:135855.114 End of zbx_ipc_async_socket_recv():0 6999:20261027:135855.114 End of zbx_ipc_async_socket_recv():0 6999:20261027:135855.114 zbx_setproctitle() title:'java poller #2 [got 0 values in 0.000184 sec, getting values]' 6999:20261027:135855.114 In get_values()
I did some searches and found another post which seems to have similar issues to me...
Everything in that thread looks the same, however I am not using systemd_resolver. My internal DNS server is DNSMasq, and that seems to be working fine, so I can't seem to understand why Zabbix 7 seems to hate on my setup over Zabbix 6.
More logs on another example...
Code:
7049:20261027:141137.112 In zbx_async_check_agent() itemid:47837 key:'proc.num' host:'Frigate' addr:'frigate.home.local' conn:'unencrypted' 7049:20261027:141137.112 In zbx_is_ip4() ip:'frigate.home.local' 7049:20261027:141137.113 End of zbx_is_ip4():FAIL 7049:20261027:141137.113 In zbx_is_ip6() ip:'frigate.home.local' 7049:20261027:141137.113 End of zbx_is_ip6():FAIL 7049:20261027:141137.113 In zbx_is_ip4() ip:'frigate.home.local' 7049:20261027:141137.113 End of zbx_is_ip4():FAIL 7049:20261027:141137.113 In zbx_is_ip6() ip:'frigate.home.local' 7049:20261027:141137.113 End of zbx_is_ip6():FAIL 7049:20261027:141137.113 End of zbx_async_check_agent():SUCCEED 7049:20261027:141137.113 End of async_initiate_queued_checks(): num:1 7049:20261027:141137.113 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:141137.113 End of zbx_ipc_async_socket_recv():0 7049:20261027:141137.115 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:141137.115 End of zbx_ipc_async_socket_recv():0 7048:20261027:141137.115 In zbx_ipc_async_socket_recv() timeout:0 7048:20261027:141137.115 End of zbx_ipc_async_socket_recv():0 7050:20261027:141137.115 In zbx_dc_config_get_poller_items() poller_type:9 7050:20261027:141137.115 End of zbx_dc_config_get_poller_items():0 7050:20261027:141137.115 queue processing_num:0 pending:0 7048:20261027:141137.115 In zbx_dc_config_get_poller_items() poller_type:7 7048:20261027:141137.115 End of zbx_dc_config_get_poller_items():0 7048:20261027:141137.115 queue processing_num:0 pending:0 7050:20261027:141137.115 In zbx_ipc_async_socket_recv() timeout:0 7050:20261027:141137.115 End of zbx_ipc_async_socket_recv():0 7048:20261027:141137.116 In zbx_ipc_async_socket_recv() timeout:0 7048:20261027:141137.116 End of zbx_ipc_async_socket_recv():0 7049:20261027:141137.120 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:141137.120 End of zbx_ipc_async_socket_recv():0 7049:20261027:141137.120 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:141137.120 End of zbx_ipc_async_socket_recv():0 7049:20261027:141137.124 In async_dns_event() result:-2 7049:20261027:141137.124 cannot resolve DNS name: nodename nor servname provided, or not known 7049:20261027:141137.124 In async_event() 7049:20261027:141137.125 In agent_task_process() step 'init' 'timeout' event:1 itemid:47837 addr: 7049:20261027:141137.125 In process_async_result() itemid:47837 key:'proc.num' host:'Frigate' addr:'frigate.home.local' 7049:20261027:141137.125 finished processing itemid:47837 processing:0 7049:20261027:141137.125 End of process_async_result():TIMEOUT_ERROR 7049:20261027:141137.125 End of async_event():stop 7049:20261027:141137.125 End of async_dns_event() 7049:20261027:141137.125 In zbx_ipc_async_socket_recv() timeout:0 7049:20261027:141137.125 End of zbx_ipc_async_socket_recv():0 7049:20261027:141137.125 interfaces num:1 7049:20261027:141137.125 requeue num:1 7049:20261027:141137.125 In poller_update_interfaces() num:1 7049:20261027:141137.125 In zbx_deactivate_item_interface() interfaceid:31 itemid:47837 type:0 7049:20261027:141137.125 Zabbix agent item "proc.num" on host "Frigate" failed: another network error, wait for 15 seconds 7049:20261027:141137.125 zbx_deactivate_item_interface() errors_from:1793110282 available:1 7049:20261027:141137.125 End of zbx_deactivate_item_interface() 7049:20261027:141137.125 In zbx_ipc_socket_write() 7049:20261027:141137.125 End of zbx_ipc_socket_write():SUCCEED