Zabbix agent 2. Windows performance counter buffer error.

[AleksejEgorov]

Hey there. We have Zabbix agent 2 (7.0.0) on different windows hosts, and sometimes agent fails with this error in zabbix_agent2.log

2024/08/26 14:55:31.402274 cannot load os dependent items: Failed to load objects and default counters: There is more data to return than will fit in the supplied buffer. Allocate a larger buffer and call the function again.

Sometimes, you can restart service immediately, other times it fails again and again, but starts after 4,5,6... attempts.
Research gives me only perfmon error 0x800007D2 (PDH_MORE_DATA) here: https://learn.microsoft.com/en-us/wi...dh-error-codes and some similar issues with elastic search, but nothing more.
Any ideas?

[kamil1]

Hey,
could you please switch DebugLevel to 5 so that we are better informed what is going on there.

Thanks

[AleksejEgorov]

Hey,
could you please switch DebugLevel to 5 so that we are better informed what is going on there.

Thanks

Thank you for your reply. I will try on one or two machines, but the issue may not occured for days.

[landon_l9]

I got this error and Zabbix Agent 2 wouldn't start, kept repeating this error.

I ran

C:\Windows\SysWOW64\lodctr /r

and after a few minutes I was able to start the service again.

[zabbixio]

I have the same problem on a Windows 2022 server. Other Win 2022 servers work just fine but not one of them.

[Jeriho]

Hello! I have the same trouble, except one thing - I always can start zabbix service at first attempt.
Windows 2022, fully patched. All affected servers are RDS hosts. Other Windows 2022 hosts (like DC, FS, PS) and Linux aren`t affected.
Zabbix Agent 2 6.0.33
Error in Russian:
cannot load os dependent items: Failed to load objects and default counters: Размер предоставленного буфера мал для ожидаемых данных. Увеличьте размер буфера и вызовите функцию заново.
Error in English:
cannot load os dependent items: Failed to load objects and default counters: There is more data to return than will fit in the supplied buffer. Allocate a larger buffer and call the function again.
I do not understand what the mentioned "Buffer" is. Is it zabbixs own buffer or some windows setting?
I`ve found some mentions, that increasing defaults
BufferSend=5
BufferSize=100
in zabbix-agent config can help, but "BufferSend=3600 BufferSize=65535" doesn`t work, same as "BufferSend=10 BufferSize=300"
Latest config:
LogFile=C:\Program Files\Zabbix Agent 2\zabbix_agent2.log
LogFileSize=8
Server=my-server-ip
ServerActive=my-server-ip
BufferSend=10
BufferSize=300
HostMetadata=string-to-autoregister-agent
Plugins.ZabbixSync.System.Capacity=1 # to fix previous errors `cause it doesn`t work with any other value
Plugins.WindowsPerfInstance.System.Capacity=1 # to fix previous errors `cause it doesn`t work with any other value

Debug level 5 log right after server reboot and before crash:

2024/10/01 03:03:19.076571 Starting Zabbix Agent 2 (6.0.33)
2024/10/01 03:03:19.113064 Calling C function "tls_new_context()"
2024/10/01 03:03:19.117415 Calling C function "tls_new_context()"
2024/10/01 03:03:19.117930 Calling C function "tls_version()"
2024/10/01 03:03:19.117930 OpenSSL library (OpenSSL 3.0.11 19 Sep 2023) initialized
2024/10/01 03:03:19.117930 Calling C function "tls_describe_ciphersuites()"
2024/10/01 03:03:19.118166 Calling C function "free()"
2024/10/01 03:03:19.118166 default context ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
2024/10/01 03:03:19.118166 Calling C function "tls_describe_ciphersuites()"
2024/10/01 03:03:19.118166 Calling C function "free()"
2024/10/01 03:03:19.118166 psk context ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
2024/10/01 03:03:19.118166 using configuration file: C:\Program Files\Zabbix Agent 2\zabbix_agent2.conf
2024/10/01 03:03:19.118166 using plugin 'Agent' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.118692 using plugin 'Ceph' (built-in) providing following interfaces: exporter, runner, configurator
2024/10/01 03:03:19.118738 using plugin 'Cpu' (built-in) providing following interfaces: exporter, collector, runner
2024/10/01 03:03:19.118738 using plugin 'DNS' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.118738 using plugin 'File' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.118738 using plugin 'Log' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.118738 using plugin 'MQTT' (built-in) providing following interfaces: watcher, configurator
2024/10/01 03:03:19.118738 using plugin 'Memcached' (built-in) providing following interfaces: exporter, runner, configurator
2024/10/01 03:03:19.118738 using plugin 'Memory' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.118738 using plugin 'Modbus' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.118738 using plugin 'Mysql' (built-in) providing following interfaces: exporter, runner, configurator
2024/10/01 03:03:19.119278 using plugin 'NetIf' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119278 using plugin 'Oracle' (built-in) providing following interfaces: exporter, runner, configurator
2024/10/01 03:03:19.119278 using plugin 'Proc' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119278 using plugin 'Redis' (built-in) providing following interfaces: exporter, runner, configurator
2024/10/01 03:03:19.119278 using plugin 'Smart' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.119278 using plugin 'Swap' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119278 using plugin 'SystemRun' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.119278 using plugin 'TCP' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.119278 using plugin 'UDP' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.119278 using plugin 'Uname' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119278 using plugin 'Uptime' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119278 using plugin 'Users' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.119819 using plugin 'VFSDir' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119819 using plugin 'VMemory' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119819 using plugin 'VfsFs' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.119819 using plugin 'WebCertificate' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.119819 using plugin 'WebPage' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.119819 using plugin 'WindowsEventlog' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.120367 using plugin 'WindowsPerfInstance' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.120367 using plugin 'WindowsPerfMon' (built-in) providing following interfaces: exporter, collector, runner
2024/10/01 03:03:19.120367 using plugin 'WindowsServices' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.120367 using plugin 'Wmi' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.120367 using plugin 'ZabbixAsync' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.120367 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2024/10/01 03:03:19.120367 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2024/10/01 03:03:19.120920 [Modbus] Config is valid
2024/10/01 03:03:21.136211 cannot load os dependent items: Failed to load objects and default counters: Размер предоставленного буфера мал для ожидаемых данных. Увеличьте размер буфера и вызовите функцию заново.

[Jeriho]

Looks like changing monitoring type to "active" has resolved the problem. There are such errors on servers with template "Windows ... by zabbix agent" And no errors on those wich has templates changed to "Windows ... by zabbix agent active".
Also, I`ve noticed, that the agent crash often happened on services discovery or services status sending to serever. On RDS there is huge amount of per-user services, and data buffer should be really big in this case.
So as data parsing and normalization run on zabbix-agent side, there's no need anymore for big buffer to send them to server. But it would be nice to have a real solution, and not workaround like this.
We'll wait for answer from zabbix team.

[Roffel]

We have the same issue with active checks. Agent 2 just fails to start with the last line in the log as shown in previous posts. The error is in German, so I guess it comes from the OS. Increasing debug level or buffer size didn't help so far. Multiple restarts sometimes help. We have switched to Agent 1 for now and that seems to work fine.

[mmc.jasonb]

We've just started rolling out Agent 2 this week, replacing Agent 1, and we're also seeing this sporadically on a small percentage of hosts.

I don't have much to add other than to confirm this is still an issue on version 7.0.5 (both Agent and Server).

Also, since it was mentioned above as a potential resolution, I should say that all of our hosts are already using the "Windows by Zabbix agent active" template (version 7.0-0), it does not prevent the issue occurring.

[hrhrhrhr]

It seems the problem is only in newer Agent 2 Versions. With Agent 2 Version 6.0.23 and before we never had the problem. Updated to 7.0.5 and get the error on 1% of the Windows hosts. After multiple Agent restarts Agent is working.

[Roffel]

The problem is here in function LocateObjectsAndDefaultCounters(): https://github.com/zabbix/zabbix/blo...pdh_windows.go

If any of the called PDH (Performance Data Helper) functions result in an error, that error is returned. In our case, that's the OS language dependent text: "There is more data to return than will fit in the supplied buffer. Allocate a larger buffer and call the function again."

The above is called from function loadOSDependentItems(): https://github.com/zabbix/zabbix/blo...nt2_windows.go

...which prepends the text "failed to load objects and default counters" and finally "cannot load os dependent items" before exiting.

[cedric.marcoux]

Hello, i have the same issue with 6.0.33 to 35. Still working on 6.0.27

[zimeon]

We encountered this but with 7.0.5 agent 2 in active mode. There seems to be a bug relating to it, and it will be fixed on version 7.0.10. https://support.zabbix.com/projects/...lter=allissues