Cannot connect to [0.0.0.0:10050]: [4] Interrupted system call

[dieselfluxcapacitor]

Running Zabbix 6.4 on Ubuntu 20.04 LTS

I have many Windows clients running Zabbix Agent 2 version 6.4.6 that are exhibiting two issues: their availability in the Zabbix frontend is consistently flapping, and remote commands executed against the passive agent return with the error "Cannot connect to [0.0.0.0:10050]: [4] Interrupted system call" (where 0.0.0.0 is redacted IP address).

To troubleshoot, I've tried the following:

• changing the update interval frequency of the Windows: Zabbix agent availability item
• lowering the Timeout parameter on the Zabbix Server from 3 to 2 seconds
• increasing the Timeout parameter on the Zabbix Server from 3 to 5 seconds
• increasing the number of unreachable poller data collector instances on the Zabbix Server
• disabling TLS/PSK encrypted communication between sever and agent
• disabling layer 7 packet inspection in the network firewall
• enabling verbose logging on the client and server and examining the logs for hints

So far, nothing has lead me to any clues as to how to resolve this issue. Has anyone else run into this issue and resolved it? I see there have been others on the forums who've complained, but I don't see any resolutions posted

1. [4] Interrupted system call - ZABBIX Forums
2. cannot connect to [[x.x.x.x]:10050]: [4] Interrupted system call - ZABBIX Forums
3. performance testing - Zabbix can't get value from Agent (Interrupted system call) - Stack Overflow
4. Agent Failed / Interrupted System Call (4) : zabbix (reddit.com)

[vijayk]

Did you check with debug logs in agent?

[dieselfluxcapacitor]

Did you check with debug logs in agent?

I see that the client dropped out multiple times overnight. Let's focus on the instances at 3:00 and 5:00

In my client log I see that the agent just restarted around 3:00. Here's the details of the log:

2023/10/10 03:00:06.280829 Zabbix Agent 2 stopped. (6.4.7)
2023/10/10 03:00:39.269900 Starting Zabbix Agent 2 (6.4.7)
2023/10/10 03:00:39.282973 OpenSSL library (OpenSSL 3.0.11 19 Sep 2023) initialized
2023/10/10 03:00:39.282973 using configuration file: C:\Program Files\Zabbix Agent\zabbix_agent2.conf
...
2023/10/10 03:00:39.288719 lowering the plugin WindowsPerfInstance capacity to 1 as the configured capacity 100 exceeds limits
2023/10/10 03:00:39.288719 using plugin 'WindowsPerfInstance' (built-in) providing following interfaces: exporter
2023/10/10 03:00:39.288719 using plugin 'WindowsPerfMon' (built-in) providing following interfaces: exporter, collector, runner
2023/10/10 03:00:39.288719 using plugin 'WindowsServices' (built-in) providing following interfaces: exporter
2023/10/10 03:00:39.288719 using plugin 'Wmi' (built-in) providing following interfaces: exporter
2023/10/10 03:00:39.288719 using plugin 'ZabbixAsync' (built-in) providing following interfaces: exporter
2023/10/10 03:00:39.288719 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2023/10/10 03:00:39.288719 lowering the plugin ZabbixSync capacity to 1 as the configured capacity 100 exceeds limits
2023/10/10 03:00:39.288719 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2023/10/10 03:00:41.419994 Plugin communication protocol version is 6.4.0
2023/10/10 03:00:41.420514 Zabbix Agent2 hostname: [CLC-01A-L3]
2023/10/10 03:00:42.599065 [101] cannot connect to [0.0.0.0:10051]: dial tcp: lookup 0.0.0.0: no such host
2023/10/10 03:00:42.599065 [101] active check configuration update from host [CLC-01A-L3] started to fail
2023/10/10 03:00:42.599627 [101] cannot connect to [0.0.0.0:10051]: dial tcp: lookup 0.0.0.0: no such host
2023/10/10 03:00:42.599627 [101] sending of heartbeat message for [CLC-01A-L3] started to fail
2023/10/10 03:00:55.526590 [101] active check configuration update from [0.0.0.0:10051] is working again
2023/10/10 03:01:43.523935 [101] sending of heartbeat message to [0.0.0.0:10051] is working again
2023/10/10 05:21:47.682186 [101] cannot connect to [0.0.0.0:10051]: dial tcp :0->10.128.254.36:10051: i/o timeout
2023/10/10 05:21:47.682186 [101] active check configuration update from host [CLC-01A-L3] started to fail
2023/10/10 05:21:53.673900 [101] active check configuration update from [0.0.0.0:10051] is working again
2023/10/10 05:32:52.677863 [101] cannot connect to [0.0.0.0:10051]: dial tcp :0->10.128.254.36:10051: i/o timeout
2023/10/10 05:32:52.677863 [101] active check configuration update from host [CLC-01A-L3] started to fail
2023/10/10 05:32:58.686397 [101] active check configuration update from [0.0.0.0:10051] is working again​

This is what I see server-side for the same time period:

2431:20231010:030019.683 Zabbix agent item "agent.hostname" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2372:20231010:030039.412 temporarily disabling Zabbix agent checks on host "CLC-01A-L3": interface unavailable
  2448:20231010:030050.173 enabling Zabbix agent checks on host "CLC-01A-L3": interface became available
  2085:20231010:030142.996 Zabbix agent item "perf_counter_en["\Memory\Page Faults/sec"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2162:20231010:030158.199 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2307:20231010:030214.266 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2284:20231010:030233.131 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2369:20231010:030308.007 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2322:20231010:030323.015 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2275:20231010:030343.279 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2459:20231010:030418.556 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2272:20231010:030433.047 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2298:20231010:030438.035 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2309:20231010:030453.051 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2403:20231010:030453.667 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2297:20231010:030528.380 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2441:20231010:030543.105 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2329:20231010:030608.483 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2441:20231010:030623.038 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2173:20231010:030726.082 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2455:20231010:030742.399 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2103:20231010:030744.032 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2336:20231010:030802.138 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2194:20231010:030832.634 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2465:20231010:030907.008 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2363:20231010:030922.074 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2233:20231010:030943.182 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2344:20231010:031018.543 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2341:20231010:031033.023 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2408:20231010:031038.469 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2076:20231010:031054.081 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2325:20231010:031058.406 Zabbix agent item "perf_counter_en["\System\Threads"]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2439:20231010:031113.779 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2301:20231010:031133.015 temporarily disabling Zabbix agent checks on host "CLC-01A-L3": interface unavailable
  2297:20231010:031143.487 enabling Zabbix agent checks on host "CLC-01A-L3": interface became available
  2355:20231010:031208.091 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2352:20231010:031223.130 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2121:20231010:031254.067 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2302:20231010:031312.071 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2114:20231010:031334.300 Zabbix agent item "perf_counter_en["\Memory\Free System Page Table Entries"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2449:20231010:031352.543 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2145:20231010:031353.248 Zabbix agent item "vm.memory.size[total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2328:20231010:031428.404 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2352:20231010:031435.524 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2319:20231010:031503.401 Zabbix agent item "proc.num[]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2309:20231010:031533.888 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2070:20231010:031555.293 Zabbix agent item "system.uptime" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2381:20231010:031630.071 Zabbix agent item "system.uptime" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2404:20231010:031633.084 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2357:20231010:031710.532 Zabbix agent item "perf_counter_en["\Paging file(_Total)\% Usage"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2283:20231010:031726.048 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2182:20231010:031727.451 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2391:20231010:031742.018 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2185:20231010:031833.156 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2269:20231010:031908.544 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2469:20231010:031923.079 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2315:20231010:031928.428 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2405:20231010:031943.240 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2092:20231010:031944.059 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2318:20231010:032002.524 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2283:20231010:032003.644 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2381:20231010:032033.664 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2222:20231010:032056.271 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Write",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2403:20231010:032112.640 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2197:20231010:032134.264 Zabbix agent item "service.info["CiscoSCMS",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2329:20231010:032152.577 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2072:20231010:032233.107 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2299:20231010:032255.487 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2299:20231010:032328.548 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2270:20231010:032343.022 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2218:20231010:032344.080 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2458:20231010:032402.109 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2126:20231010:032433.648 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2433:20231010:032508.055 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Write",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2311:20231010:032523.155 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2127:20231010:032548.365 Zabbix agent item "system.sw.os" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2338:20231010:032623.086 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2437:20231010:032623.464 Zabbix agent item "perf_counter_en["\Paging file(_Total)\% Usage"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2360:20231010:032645.375 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2389:20231010:032718.087 Zabbix agent item "perf_counter_en["\Paging file(_Total)\% Usage"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2317:20231010:032733.071 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2119:20231010:032835.687 Zabbix agent item "service.info["NiniteAgent",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2447:20231010:032853.056 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2234:20231010:032854.088 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2418:20231010:032912.197 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2143:20231010:032944.204 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2385:20231010:033002.400 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2093:20231010:033034.000 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2362:20231010:033052.069 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2096:20231010:033134.444 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% Interrupt Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2420:20231010:033152.201 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2308:20231010:033229.440 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% Privileged Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2336:20231010:033304.004 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Write",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2441:20231010:033322.560 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2365:20231010:033324.004 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2332:20231010:033342.716 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2163:20231010:033343.056 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2435:20231010:033418.031 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2345:20231010:033433.046 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2100:20231010:033528.467 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2450:20231010:033543.043 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2227:20231010:033634.220 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2159:20231010:033650.511 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk Write Queue Length",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2461:20231010:033652.480 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2089:20231010:033733.089 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2275:20231010:033808.080 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2292:20231010:033823.071 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2074:20231010:033850.098 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk Write Queue Length",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2406:20231010:033912.130 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2064:20231010:033932.419 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2363:20231010:034007.072 Zabbix agent item "agent.ping" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2425:20231010:034022.300 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2186:20231010:034134.272 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% Privileged Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2264:20231010:034152.071 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2099:20231010:034154.172 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2263:20231010:034229.844 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2314:20231010:034232.223 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2176:20231010:034254.173 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2318:20231010:034309.464 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2312:20231010:034312.197 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2205:20231010:034334.072 Zabbix agent item "service.info["CiscoSCMS",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2378:20231010:034409.608 Zabbix agent item "agent.ping" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2262:20231010:034412.199 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2281:20231010:034444.243 Zabbix agent item "agent.ping" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2417:20231010:034519.524 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2259:20231010:034522.401 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2142:20231010:034557.326 Zabbix agent item "perf_counter_en["\Paging file(_Total)\% Usage"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2405:20231010:034612.094 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2095:20231010:034633.534 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2317:20231010:034708.063 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2426:20231010:034728.028 temporarily disabling Zabbix agent checks on host "CLC-01A-L3": interface unavailable
  2304:20231010:034843.377 enabling Zabbix agent checks on host "CLC-01A-L3": interface became available
  2464:20231010:034903.427 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2357:20231010:034933.444 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2173:20231010:034954.022 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2418:20231010:035029.426 Zabbix agent item "agent.ping" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2453:20231010:035032.069 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2075:20231010:035128.456 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2396:20231010:035143.119 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2072:20231010:035143.347 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2432:20231010:035218.228 Zabbix agent item "system.sw.os" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2277:20231010:035233.365 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2298:20231010:035238.212 Zabbix agent item "service.info["Spooler",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2356:20231010:035253.045 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2074:20231010:035254.076 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2278:20231010:035312.576 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2258:20231010:035334.332 Zabbix agent item "perf_counter_en["\Memory\Free System Page Table Entries"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2309:20231010:035352.732 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2228:20231010:035354.108 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2428:20231010:035429.520 Zabbix agent item "system.uptime" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2433:20231010:035432.067 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2097:20231010:035525.932 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2308:20231010:035543.108 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2231:20231010:035544.375 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2344:20231010:035619.180 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2419:20231010:035622.212 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2407:20231010:035654.316 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2409:20231010:035712.608 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2101:20231010:035734.003 Zabbix agent item "perf_counter_en["\Memory\Free System Page Table Entries"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2357:20231010:035752.256 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2387:20231010:035829.724 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2323:20231010:035851.775 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2350:20231010:035924.003 Zabbix agent item "service.info["CiscoSCMS",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2368:20231010:035942.304 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored​

And here's what I see around the 5:00 mark:

2410:20231010:050008.088 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2313:20231010:050023.166 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2353:20231010:050043.383 Zabbix agent item "proc.num[]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2308:20231010:050118.204 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2368:20231010:050133.125 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2209:20231010:050156.128 Zabbix agent item "vm.memory.size[used]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2341:20231010:050212.628 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2227:20231010:050233.376 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2314:20231010:050308.286 Zabbix agent item "system.uptime" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2341:20231010:050323.061 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2093:20231010:050347.358 Zabbix agent item "perf_counter_en["\System\Threads"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2428:20231010:050402.131 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2221:20231010:050434.300 Zabbix agent item "service.info["CiscoSCMS",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2318:20231010:050452.289 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2244:20231010:050454.188 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2264:20231010:050512.288 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2123:20231010:050533.152 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2416:20231010:050608.135 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2341:20231010:050623.453 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2291:20231010:050648.446 Zabbix agent item "perf_counter_en["\Memory\Free System Page Table Entries"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2418:20231010:050703.535 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2425:20231010:050733.637 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2202:20231010:050757.372 Zabbix agent item "perf_counter_en["\Paging file(_Total)\% Usage"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2396:20231010:050812.031 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2195:20231010:050833.083 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2261:20231010:050908.198 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2295:20231010:050923.032 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2285:20231010:050943.163 Zabbix agent item "service.info["NiniteAgent",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2248:20231010:050958.005 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2403:20231010:051033.118 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2320:20231010:051033.217 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2347:20231010:051108.224 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Write",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2469:20231010:051123.052 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2307:20231010:051128.163 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2265:20231010:051143.096 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Write",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2468:20231010:051143.111 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2305:20231010:051223.007 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2283:20231010:051253.889 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2443:20231010:051318.483 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2373:20231010:051333.038 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2226:20231010:051334.368 Zabbix agent item "perf_counter_en["\Memory\Free System Page Table Entries"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2448:20231010:051352.068 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2359:20231010:051353.851 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2300:20231010:051423.489 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2237:20231010:051453.318 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Read",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2364:20231010:051528.311 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% DPC Time"]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2386:20231010:051543.105 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2415:20231010:051548.273 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",dropped]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2350:20231010:051603.568 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Read",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2384:20231010:051633.793 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2123:20231010:051654.087 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2267:20231010:051712.288 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2146:20231010:051733.708 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2319:20231010:051808.012 Zabbix agent item "system.uptime" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2273:20231010:051823.239 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2342:20231010:051843.325 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2465:20231010:051913.793 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2417:20231010:051938.600 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2394:20231010:051953.426 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2256:20231010:051954.089 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2355:20231010:052013.750 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2362:20231010:052029.603 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2367:20231010:052035.456 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2356:20231010:052108.531 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2298:20231010:052123.057 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2130:20231010:052143.074 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2363:20231010:052218.280 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2327:20231010:052238.243 temporarily disabling Zabbix agent checks on host "CLC-01A-L3": interface unavailable
  2289:20231010:052813.024 enabling Zabbix agent checks on host "CLC-01A-L3": interface became available
  2434:20231010:052833.328 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% Interrupt Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2262:20231010:052908.008 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Write",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2318:20231010:052923.025 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2066:20231010:052944.049 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2398:20231010:053019.232 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2294:20231010:053034.036 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2364:20231010:053114.020 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2267:20231010:053132.480 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2202:20231010:053132.684 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% DPC Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2343:20231010:053154.116 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2230:20231010:053234.088 Zabbix agent item "service.info["CiscoSCMS",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2395:20231010:053252.068 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2256:20231010:053334.121 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% User Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2426:20231010:053352.511 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2426:20231010:053429.567 Zabbix agent item "system.cpu.load[all,avg15]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2310:20231010:053444.073 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2326:20231010:053504.283 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2389:20231010:053522.399 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2170:20231010:053544.167 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2403:20231010:053602.120 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2130:20231010:053604.102 Zabbix agent item "perf_instance_en.discovery[PhysicalDisk]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2431:20231010:053622.053 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2224:20231010:053643.049 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2297:20231010:053718.427 Zabbix agent item "net.if.in["{47A09CB2-AFC4-458E-8D45-C70E8EA02050}",errors]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2438:20231010:053733.053 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2120:20231010:053754.056 Zabbix agent item "system.swap.size[,total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2319:20231010:053812.068 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2255:20231010:053833.283 Zabbix agent item "service.info["CiscoOrbital",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2341:20231010:053855.230 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2071:20231010:053925.399 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2320:20231010:053940.024 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2139:20231010:053940.504 Zabbix agent item "perf_counter_en["\Memory\Free System Page Table Entries"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2349:20231010:054002.623 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2120:20231010:054050.480 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk Write Queue Length",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2367:20231010:054112.451 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2149:20231010:054132.633 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% User Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2345:20231010:054207.007 Zabbix agent item "vm.memory.size[total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2281:20231010:054222.027 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2110:20231010:054254.112 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2402:20231010:054329.240 Zabbix agent item "perf_counter_en["\System\Context Switches/sec"]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2308:20231010:054332.227 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2222:20231010:054357.767 Zabbix agent item "perf_counter_en["\Paging file(_Total)\% Usage"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2284:20231010:054412.368 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2170:20231010:054532.280 Zabbix agent item "perf_counter_en["\Processor Information(_total)\% Interrupt Time"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2103:20231010:054548.148 Zabbix agent item "system.sw.os" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2438:20231010:054602.496 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2464:20231010:054623.113 Zabbix agent item "system.sw.os" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2352:20231010:054658.248 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk Read Queue Length",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2423:20231010:054718.083 temporarily disabling Zabbix agent checks on host "CLC-01A-L3": interface unavailable
  2298:20231010:054833.024 enabling Zabbix agent checks on host "CLC-01A-L3": interface became available
  2274:20231010:054853.400 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk Read Queue Length",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2266:20231010:054923.456 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2146:20231010:054958.478 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2368:20231010:055013.299 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2202:20231010:055034.087 Zabbix agent item "service.info["CiscoSCMS",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2306:20231010:055052.669 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2217:20231010:055054.056 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\% Idle Time",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2409:20231010:055129.523 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Writes/sec",60]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2461:20231010:055132.069 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2230:20231010:055153.244 Zabbix agent item "vm.memory.size[total]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2369:20231010:055228.481 Zabbix agent item "vm.memory.size[total]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2463:20231010:055243.071 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2455:20231010:055248.088 Zabbix agent item "service.info["QualysAgent",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2399:20231010:055323.027 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2377:20231010:055323.051 Zabbix agent item "service.info["Zabbix Agent 2",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2365:20231010:055353.280 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2285:20231010:055418.149 Zabbix agent item "service.info["QualysAgent",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2432:20231010:055433.034 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2122:20231010:055434.072 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2464:20231010:055452.447 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2111:20231010:055528.378 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2316:20231010:055543.330 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2343:20231010:055603.351 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2267:20231010:055638.088 Zabbix agent item "service.info["CiscoAMP",state]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2326:20231010:055653.261 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2328:20231010:055658.004 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Disk Reads/sec",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2379:20231010:055713.025 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2121:20231010:055744.039 Zabbix agent item "wmi.getall[root\cimv2,"select Name,Description,NetConnectionID,Speed,AdapterTypeId,NetConnectionStatus,GUID from win32_networkadapter where PhysicalAdapter=True and NetConnectionStatus>0"]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2170:20231010:055800.508 Zabbix agent item "agent.ping" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2387:20231010:055802.527 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2161:20231010:055834.363 Zabbix agent item "service.info["CiscoSCMS",state]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2279:20231010:055852.927 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2139:20231010:055853.111 Zabbix agent item "perf_counter_en["\PhysicalDisk(0 C:)\Avg. Disk sec/Read",60]" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds
  2345:20231010:055928.301 Zabbix agent item "perf_counter_en["\Memory\Cache Bytes"]" on host "CLC-01A-L3" failed: another network error, wait for 15 seconds
  2444:20231010:055935.299 resuming Zabbix agent checks on host "CLC-01A-L3": connection restored
  2080:20231010:055958.248 Zabbix agent item "agent.version" on host "CLC-01A-L3" failed: first network error, wait for 15 seconds​

So I understand that for some reason these checks are timing out and that's causing the agent to become unavailable (or vice versa). However, I'm not sure what needs to be done to rectify this issue. I appreciate any help you can provide!

[vijayk]

2023/10/10 03:00:42.599065 [101] cannot connect to [0.0.0.0:10051]: dial tcp: lookup 0.0.0.0: no such host
2023/10/10 03:00:42.599065 [101] active check configuration update from host [CLC-01A-L3] started to fail
2023/10/10 03:00:42.599627 [101] cannot connect to [0.0.0.0:10051]: dial tcp: lookup 0.0.0.0: no such host
2023/10/10 03:00:42.599627 [101] sending of heartbeat message for [CLC-01A-L3] started to fail

Error says active check goes failed.

Can you confirm the Hostname and ServerActive IP in .conf file in agent ?

[dieselfluxcapacitor]

The weird thing is that this host never had an active agent template applied, so I'm not sure why it was trying to query for active checks. Of course it will fail - there aren't any checks to be found!

[dieselfluxcapacitor]

Thanks for all the help. Ultimately, I determined that there were too many issues with the server (and I needed to reconfigure the solution architecture anyway) so I ended up blowing it away and starting over. Fortunately, on the second attempt, I am no longer seeing these issues.

[vijayk]

Thanks for all the help. Ultimately, I determined that there were too many issues with the server (and I needed to reconfigure the solution architecture anyway) so I ended up blowing it away and starting over. Fortunately, on the second attempt, I am no longer seeing these issues.

Nice to hear that issue is no remain.

[dieselfluxcapacitor]

I agree. I think the other change that helped greatly was moving away from using the passive Zabbix agent. Instead, I've shifted all possible hosts over to active monitoring which has drastically improved the performance of my Zabbix server.

[vijayk]

That's great.