Agent 5 - Failed to initialized builtin counter's on virtualized Win2019 on Hyper-V

[mcGeorge]

Hi, am a beginner of Zabbix. I installed the Zabbix server version 5 successful and the Zabbix agent on some hardware servers. I am really happy in general about the results, but...

I installed the same agent with the same configuration on some virtualized Windows 2019 server. Everything looked good on the first view. Then I started to dig deeper into the results on the Zabbix server and noticed that I miss 2 values: "CPU utilization" and "CPU jumps".
I checked the log files on the clients and found the following error messages on each server:

7128:20200519:072751.068 Starting Zabbix Agent [XXXX-SQL-01.XXXX.LOCAL]. Zabbix 5.0.0 (revision 9665d62db0).
7128:20200519:072751.069 **** Enabled features ****
7128:20200519:072751.070 IPv6 support: YES
7128:20200519:072751.071 TLS support: NO
7128:20200519:072751.072 **************************
7128:20200519:072751.073 using configuration file: c:\AdminVM\Zabbix\zabbix_agentd.conf
7128:20200519:072751.132 Failed to initialize builtin counter: System
7128:20200519:072751.133 Failed to initialize builtin counter: Processor
7128:20200519:072751.134 Failed to initialize builtin counter: Processor Information
7128:20200519:072751.136 Failed to initialize builtin counter: Terminal Services
7128:20200519:072751.137 Failed to initialize builtin counter: Processor Queue Length
7128:20200519:072751.138 Failed to initialize builtin counter: System Up Time
7128:20200519:072751.139 Failed to initialize builtin counter: % Processor Time
7128:20200519:072751.140 Failed to initialize builtin counter: % Processor Time
7128:20200519:072751.141 Failed to initialize builtin counter: Total Sessions
7128:20200519:072751.142 agent #0 started [main process]
7696:20200519:072751.143 agent #4 started[listener #3]
7096:20200519:072751.144 agent #2 started[listener #1]
4288:20200519:072751.145 agent #3 started[listener #2]
7304:20200519:072751.146 agent #5 started [active checks #1]
760:20200519:072751.146 agent #1 started [collector]
760:20200519:072751.226 init_cpu_collector(): cannot make counterpath for "\\": [0xC0000BBD] A required argument is missing or not correct.
7696:20200519:073445.592 check_counter_path(): cannot make counterpath for "\\": [0xC0000BBD] A required argument is missing or not correct.
6896:20200519:073504.770 Zabbix Agent stopped. Zabbix 5.0.0 (revision 9665d62db0).

After that, I started the agent with the --print option and found the following messages:

system.cpu.util[all,system,avg1] [m|ZBX_NOTSUPPORTED] [Collector is not started.]
system.cpu.load[all,avg1] [m|ZBX_NOTSUPPORTED] [Collector is not started.]
system.cpu.discovery [m|ZBX_NOTSUPPORTED] [Collector is not started.]
system.uptime [m|ZBX_NOTSUPPORTED] [Cannot obtain system information.]
perf_counter[\System\Processes] [m|ZBX_NOTSUPPORTED] [Invalid performance counter format.]
perf_counter_en[\System\Processes] [m|ZBX_NOTSUPPORTED] [Invalid performance counter format.]

The rest is working fine. What is wrong and how can I get the missing data?

Thanks in advance

Georg

[mcGeorge]

After some tests, I found a solution.

If the agent is not able to load the builtin counters there is obviously a problem with the performance counter. In my case - I missed some processor data.

I checked the Perfmon counter which starts with 'Pro' the result doesn't contain the 'Processor' counter:

PS C:\Windows\system32> Get-Counter -ListSet * | Where-Object CounterSetName -Match 'Pro*' | Select-Object -Property CounterSetName

CounterSetName
--------------
Processor Information
Per Processor Network Activity Cycles
Per Processor Network Interface Card Activity
GPU Process Memory
IPsec DoS Protection
SQL Server 2017 XTP Phantom Processor
Hyper-V Hypervisor Root Virtual Processor
Hyper-V Hypervisor Logical Processor

I run the following command in an elevated command line

C:\Windows\system32>LODCTR /R

and checked it again. (You have to restart Powershell for getting the changes)

PS C:\Windows\system32> Get-Counter -ListSet * | Where-Object CounterSetName -Match 'Pro*' | Select-Object -Property CounterSetName

CounterSetName
--------------
Processor Information
Per Processor Network Activity Cycles
Per Processor Network Interface Card Activity
GPU Process Memory
IPsec DoS Protection
Hyper-V Hypervisor Root Virtual Processor
Hyper-V Hypervisor Logical Processor
.NET Data Provider for Oracle
.NET Data Provider for SqlServer
Security Per-Process Statistics
User Input Delay per Process
Processor
Process
Print Queue

A lot more performance counter and the missing 'Processor' counter.

A restart of the Zabbix agent and it was fixed.

Hopefully that will help others with the same issue.
George

[dougthor42]

Oh hot damn! I made an account just to say thank you for the solution.

In my case, I had to run lodctr /r in both the system32 and syswow64 directories.

I also ran WINMGMT.EXE /RESYNCPERF as instructed by the Manually Rebuild Performance Counters docs, but I'm not sure if it was actually needed.

[markfree]

That helped me as well. Thank you.

[Drakeson]

THX A LOT MAN!!!

[galavichid]

Thank you very much!

[paxas]

thanks man, this post is legendary

[vvictor]

Thank you a lot for posting!!!

Your resolution didn't work for me, BUT, it completly guided me for the cause of the problem, the PerfCounts!

In my case, there was a registry in RegEdit that disabled Performace Counters, and the Zabbix Agent was unable to retrieve PerfCounters data.

RegEdit path:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfOS\Performance

(there is others PerfCounters services, like: PerfDisk, PerfHost, PerfNet and PerfProc - mine had only in the PerfOS)

Registry Key:

Disable Performance Counters

- it may be

0

, if it be with

1

, PerfCounters will stay disabled.

Some logs example of my Zabbix Agent:

init_cpu_collector(): cannot make counterpath for "\Processor": [0xC0000BBD] A required argument is missing or not correct.
active check "perf_counter_en["\Memory\Cache Bytes"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\Memory\Free System Page Table Entries"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\Memory\Page Faults/sec"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\Memory\Pages/sec"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\Memory\Pool Nonpaged Bytes"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\Paging file(_Total)\% Usage"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\System\Context Switches/sec"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\System\Processor Queue Length"]" is not supported: Invalid performance counter format.
active check "perf_counter_en["\System\Threads"]" is not supported: Invalid performance counter format.
active check "system.cpu.util" is not supported: Collector is not started.
zbx_check_counter_path(): cannot make counterpath for "\System": [0xC0000BBD] A required argument is missing or not correct.
active check "system.uptime" is not supported: Cannot obtain system information.
zbx_check_counter_path(): cannot make counterpath for "\System": [0xC0000BBD] A required argument is missing or not correct.
active check "system.uptime" is not supported: Cannot obtain system information.

[mconfortini]

It worked for my servers, it was simple and easy.

Thanks!