Zabbix Agent crashing on Windows Server 2022 (all version from 5.4 - 6.4 affected)

[jan.messerer]

Good evening,

we recently installed our first Windows Server 2022 servers and ran into some issues with the Zabbix agents on these servers. The service is starting up and running for some time and then suddently stops working after a while.
Until now we were unable to find any pattern, but we are able to replicate this behavior on all Windows Server 2022 servers.

The issue occurs with all Zabbix agent versions from 5.4 - 6.4.2, the agentd log shows the following error message during service startup:

1395344:20230425:212844.001 Starting Zabbix Agent [PRD-DE-AIRPORT-DUS-DUSAPSVPOS001]. Zabbix 6.4.2 (revision 83c2f617387).
1395344:20230425:212844.003 **** Enabled features ****
1395344:20230425:212844.003 IPv6 support: YES
1395344:20230425:212844.003 TLS support: YES
1395344:20230425:212844.004 **************************
1395344:20230425:212844.004 using configuration file: C:\Program Files\Zabbix Agent\zabbix_agentd.conf
1395344:20230425:212844.006 In init_collector_data()
1395344:20230425:212844.007 logical CPU count 8
1395344:20230425:212844.007 End of init_collector_data()
1395344:20230425:212844.008 In init_perf_collector()
1395344:20230425:212844.009 In zbx_init_builtin_counter_indexes()
1395344:20230425:212844.009 In zbx_get_all_counter_names()
1395344:20230425:212844.122 End of zbx_get_all_counter_names()
1395344:20230425:212844.831 End of zbx_init_builtin_counter_indexes():SUCCEED
1395344:20230425:212844.832 In set_object_names()
1393232:20230425:212845.376 Zabbix Agent received request:4.
1393232:20230425:212845.390 Zabbix Agent received request:4.
1393232:20230425:212845.909 Zabbix Agent received request:4.
1393232:20230425:212845.922 Zabbix Agent received request:4.
1395344:20230425:212847.491 Unhandled exception 6d9 detected at 0x00007FFEEC5BEF5C. Crashing ...
1395344:20230425:212847.491 ====== Fatal information: ======
1395344:20230425:212847.491 Program counter: 0xec5bef5c
1395344:20230425:212847.492 === Registers: ===
1395344:20230425:212847.492 r8 = 7ffeee9fc94d = 140732901869901 = 140732901869901
1395344:20230425:212847.492 r9 = 1e0028 = 1966120 = 1966120
1395344:20230425:212847.493 r10 = b0 = 176 = 176
1395344:20230425:212847.493 r11 = 161afa0 = 23179168 = 23179168
1395344:20230425:212847.493 r12 = 6d9 = 1753 = 1753
1395344:20230425:212847.494 r13 = 0 = 0 = 0
1395344:20230425:212847.494 r14 = 176efe0 = 24571872 = 24571872
1395344:20230425:212847.494 r15 = 156d770 = 22468464 = 22468464
1395344:20230425:212847.494 rdi = 7ffed981a028 = 140732547571752 = 140732547571752
1395344:20230425:212847.495 rsi = 0 = 0 = 0
1395344:20230425:212847.495 rbp = 0 = 0 = 0
1395344:20230425:212847.495 rbx = 6d9 = 1753 = 1753
1395344:20230425:212847.496 rdx = 0 = 0 = 0
1395344:20230425:212847.496 rax = 1756c30 = 24472624 = 24472624
1395344:20230425:212847.496 rcx = 7ffeee120228 = 140732892578344 = 140732892578344
1395344:20230425:212847.497 rsp = 156d4f0 = 22467824 = 22467824
1395344:20230425:212847.497 efl = 202 = 514 = 514
1395344:20230425:212847.497 csgsfs = 332b5300 = 858477312 = 858477312
1395344:20230425:212847.497 === Backtrace: ===
1395344:20230425:212847.608 26: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(RaiseException+0x6c) [0xec5bef5c]
1395344:20230425:212847.609 25: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(RpcRaiseException+0x50) [0xee046770]
1395344:20230425:212847.609 24: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(RpcRaiseException+0x14) [0xee046734]
1395344:20230425:212847.609 23: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(Ndr64AsyncClientCall+0x395 2) [0xee11ba12]
1395344:20230425:212847.610 22: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(NdrClientCall3+0xf0) [0xee11cf40]
1395344:20230425:212847.611 21: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(EnumPrintersW+0xd6) [0xd97c9186]
1395344:20230425:212847.611 20: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PerfOpen+0x2b6) [0xd98085a6]
1395344:20230425:212847.612 19: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PerfCollect+0x10e) [0xd98082be]
1395344:20230425:212847.612 18: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(EnumServicesStatusExA+0x16 af) [0xee4b7c1f]
1395344:20230425:212847.613 17: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(EnumServicesStatusExA+0xed 2) [0xee4b7442]
1395344:20230425:212847.613 16: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(EnumServicesStatusExA+0xc9 3) [0xee4b7203]
1395344:20230425:212847.614 15: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(EnableTrace+0x27c6) [0xee4b16e6]
1395344:20230425:212847.614 14: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PerfRegQueryValue+0x3b) [0xee4b1ebb]
1395344:20230425:212847.615 13: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(GetCurrentThread+0xbe0) [0xec6365a0]
1395344:20230425:212847.615 12: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(RegQueryValueExW+0x108) [0xec5c7ca8]
1395344:20230425:212847.616 11: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PdhGetFormattedCounterArra yW+0x1f85) [0xdb3e79d5]
1395344:20230425:212847.616 10: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PdhSetCounterScaleFactor+0 x1496) [0xdb3f5166]
1395344:20230425:212847.617 9: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PdhTranslate009CounterW+0x 4575) [0xdb3f1bb5]
1395344:20230425:212847.617 8: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PdhWriteRelogSample+0xc15) [0xdb417c15]
1395344:20230425:212847.618 7: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PdhEnumObjectsHW+0xa2) [0xdb419e12]
1395344:20230425:212847.618 6: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(PdhEnumObjectsW+0xeb) [0xdb419f4b]
1395344:20230425:212847.619 5: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x400acbd5]
1395344:20230425:212847.619 4: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x400ac8ff]
1395344:20230425:212847.620 3: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x400ad439]
1395344:20230425:212847.620 2: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(BuildSecurityDescriptorFor SharingAccessEx+0xb32) [0xecb58e32]
1395344:20230425:212847.621 1: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0xedaa4de0]
1395344:20230425:212847.621 0: C:\Program Files\Zabbix Agent\zabbix_agentd.exe(RtlUserThreadStart+0x2b) [0xeea5e40b]
1395344:20230425:212847.627 ================================
1395344:20230425:212847.817 In zbx_get_all_counter_names()
1395344:20230425:212847.825 End of zbx_get_all_counter_names()
1395344:20230425:212847.826 In zbx_get_all_counter_names()
1395344:20230425:212847.834 End of zbx_get_all_counter_names()
1395344:20230425:212847.956 End of set_object_names():SUCCEED
1395344:20230425:212847.957 End of init_perf_collector():SUCCEED
1395344:20230425:212847.959 OpenSSL library (version OpenSSL 1.1.1t 7 Feb 2023) initialized
1395344:20230425:212847.960 agent #0 started [main process]
1382964:20230425:212847.962 agent [NODE="1"]Home[/NODE] started [collector]
1374256:20230425:212847.962 agent #4 started[listener #3]
1392860:20230425:212847.963 agent #3 started[listener #2]
1380520:20230425:212847.963 agent [NODE="2"]Forum[/NODE] started[listener #1]​

Referring to the log the records suddently stop but no additional details are recorded. Debug level is set to 5 in agend.conf.

The same agent + configuration are working fine on Windows 2016 / Windows 2019 Servers.

Does anyone has a clue what is wrong here? I am happy for all suggestions