Ad Widget

**tim.mooney** · 15-04-2020, 02:20

So, when you query the SMTP server counters successfully from a remote windows system, can you show the exact command(s) you use to do that?

Can you also show the exact zabbix_get that you're trying from the Zabbix server, and what it returns?

**snmpguru8** · 16-04-2020, 01:39

Originally posted by tim.mooney

So, when you query the SMTP server counters successfully from a remote windows system, can you show the exact command(s) you use to do that?

Can you also show the exact zabbix_get that you're trying from the Zabbix server, and what it returns?

Hi Tim - thanks so much for the response. Here is an example of using a remote Powershell Session (Enter-PSSession) to issue the command against the SMTP server and obtain the metric:

Code:

enter-pssession server1
[server1]: PS C:\Windows\system32> (Get-Counter -Counter "\SMTP Server(_Total)\Messages Received/sec")

Timestamp                 CounterSamples
---------                 --------------
4/15/2020 5:19:46 PM      \\server1\smtp server(_total)\messages received/sec :
                          3

Similarly, the results are there when Invoke-Command is used via Powershell:

Code:

PS C:\Windows\system32> Invoke-Command server1{Get-Counter -Counter "\Processor Information(_Total)\% Processor Time"}


Readings       : \\server1\processor information(_total)\% processor time :
                 0.0114286937003105

Now here is the Zabbix_get that I am trying ( a few examples ):

Code:

[rootn@zabbix ~]$ zabbix_get -s server1-k "perf_counter[\Processor Information(_Total)\% Processor Time]"  
0.023084

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\SMTP Server(_Total)\Local Queue Length]"        
ZBX_NOTSUPPORTED: Invalid performance counter format.

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\SMTP Server(_Total)\Remote Retry Queue Length]"
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\SMTP Server(_Total)\Inbound Connections Current]"
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\LogicalDisk(C:)\% Free Space]"                  
67.191353

You'll see that it returns correctly for Processor and LogicalDisk counters, but not SMTP.

Here is a snippet from the agent log:

Code:

[4620]:  Requested [perf_counter[\SMTP Server(_Total)\Inbound Connections Current]]
[4620]:  In get_perf_counter_value_by_path() path:\SMTP Server(_Total)\Inbound Connections Current interval:1
[4620]:  get_perf_counter_value_by_path(): unable to add PerfCounter '\SMTP Server(_Total)\Inbound Connections Current': [0xC0000BB8] The specified object was not found on the computer.
[4620]:  Sending back [ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.]

What's interesting is that the counters are indeed present on the computer, as shown by typeperf -qx (truncated list):

Code:

typepref -qx
\Processor Information(_Total)\% Processor Time
\ICMP\Messages/sec
\ICMP\Messages Received/sec
\LogicalDisk(C:)\% Free Space
\LogicalDisk(D:)\% Free Space
\PhysicalDisk(1 D:)\Disk Writes/sec
\SMTP Server(_Total)\Bytes Sent Total
\SMTP Server(SMTP 1)\Bytes Sent Total
\SMTP Server(_Total)\Bytes Sent/sec
\SMTP Server(SMTP 1)\Bytes Sent/sec
\SMTP Server(_Total)\Bytes Received Total
\SMTP Server(SMTP 1)\Bytes Received Total
\SMTP Server(_Total)\Messages Received/sec
\SMTP Server(SMTP 1)\Messages Received/sec
\SMTP Server(_Total)\Avg Recipients/msg Received
\SMTP Server(SMTP 1)\Avg Recipients/msg Received
\SMTP Server(_Total)\% Recipients Local
\SMTP Server(SMTP 1)\% Recipients Local
\SMTP Server(_Total)\% Recipients Remote
\SMTP Server(_Total)\Inbound Connections Current
\SMTP Server(_Total)\Remote Retry Queue Length
\SMTP Server(_Total)\Local Queue Length

I have tried rebuilding the performance counter library files as outlined here:
https://support.microsoft.com/en-us/...library-values

But that has no impact, and still, the SMTP counters work just fine when querying remotely via Powershell.

This is also happening for our IIS counters on Windows 2012 R2.
examples:

perf_counter[\W3SVC_W3WP(_Total)\Requests / Sec]

perf_counter[\Web Service(_Total)\Total Post Requests]

perf_counter["\Web Service(_Total)\Current Connections",300]

Please let me know what additional information I can provide.

Thank you

**tim.mooney** · 16-04-2020, 02:40

Originally posted by snmpguru8

Now here is the Zabbix_get that I am trying ( a few examples ):

Code:

[rootn@zabbix ~]$ zabbix_get -s server1-k "perf_counter[\Processor Information(_Total)\% Processor Time]"
0.023084

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\SMTP Server(_Total)\Local Queue Length]"
ZBX_NOTSUPPORTED: Invalid performance counter format.

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\SMTP Server(_Total)\Remote Retry Queue Length]"
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\SMTP Server(_Total)\Inbound Connections Current]"
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.

[rootn@zabbix ~]$ zabbix_get -s server1 -k "perf_counter[\LogicalDisk(C:)\% Free Space]"
67.191353

Ok, thanks for that, it helps me understand it a bit more.

The first thing that comes to mind is issues with special characters and/or quoting. On Windows, the \ character is just a separator of sorts, but on UNIX and Linux it's the "escape the next character" character. Also, as in PowerShell, there are different behaviors for something that is quoted with double quotes (") vs. something that is quoted with single quotes (').

With that in mind, first I would like to verify whether the processor info perf counter still works when the quoting is changed to like this:

Code:

# zabbix_get -s server1-k 'perf_counter["\Processor Information(_Total)\% Processor Time"]'

In other words, single quotes around the entire item key, but double quotes around the argument to perf_counter[].

So, does that still work?

If that works, then do these work:

Code:

zabbix_get -s server1 -k 'perf_counter["\SMTP Server(_Total)\Local Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\SMTP Server(_Total)\Remote Retry Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\SMTP Server(_Total)\Inbound Connections Current"]'

?

If that still doesn't work, the next thing I would try is adding an additional \ in front of each \ within the counters, like this:

Code:

zabbix_get -s server1 -k 'perf_counter["\\SMTP Server(_Total)\\Local Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\\SMTP Server(_Total)\\Remote Retry Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\\SMTP Server(_Total)\\Inbound Connections Current"]'

What that is (in theory) doing is escaping special properties of the \ character, so \\ together results in the second \ being treated as a plain old character, not something special.

Please post the results of these tests.

BTW, I think it's suspicious that one of your attempts returns "Invalid performance counter format" but the other ones fail with the "Cannot obtain performance information from collector". That implies that there are two different problems happening. Even if it does turn out one of them is related to quoting or \ characters, there might still be a separate problems for some of these counters.

**snmpguru8** · 16-04-2020, 04:50

Originally posted by tim.mooney

Ok, thanks for that, it helps me understand it a bit more.

The first thing that comes to mind is issues with special characters and/or quoting. On Windows, the \ character is just a separator of sorts, but on UNIX and Linux it's the "escape the next character" character. Also, as in PowerShell, there are different behaviors for something that is quoted with double quotes (") vs. something that is quoted with single quotes (').

With that in mind, first I would like to verify whether the processor info perf counter still works when the quoting is changed to like this:

Code:

# zabbix_get -s server1-k 'perf_counter["\Processor Information(_Total)\% Processor Time"]'

In other words, single quotes around the entire item key, but double quotes around the argument to perf_counter[].

So, does that still work?

If that works, then do these work:

Code:

zabbix_get -s server1 -k 'perf_counter["\SMTP Server(_Total)\Local Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\SMTP Server(_Total)\Remote Retry Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\SMTP Server(_Total)\Inbound Connections Current"]'

?

If that still doesn't work, the next thing I would try is adding an additional \ in front of each \ within the counters, like this:

Code:

zabbix_get -s server1 -k 'perf_counter["\\SMTP Server(_Total)\\Local Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\\SMTP Server(_Total)\\Remote Retry Queue Length"]'
zabbix_get -s server1 -k 'perf_counter["\\SMTP Server(_Total)\\Inbound Connections Current"]'

What that is (in theory) doing is escaping special properties of the \ character, so \\ together results in the second \ being treated as a plain old character, not something special.

Please post the results of these tests.

BTW, I think it's suspicious that one of your attempts returns "Invalid performance counter format" but the other ones fail with the "Cannot obtain performance information from collector". That implies that there are two different problems happening. Even if it does turn out one of them is related to quoting or \ characters, there might still be a separate problems for some of these counters.

Great suggestions - thanks, Tim.

Code:

# zabbix_get -s server1-k 'perf_counter["\Processor Information(_Total)\% Processor Time"]'

In other words, single quotes around the entire item key, but double quotes around the argument to perf_counter[].

So, does that still work?

If that works, then do these work:

Yes, this works:

Code:

[root@zabbix ~]$ zabbix_get -s server1-k 'perf_counter["\Processor Information(_Total)\% Processor Time"]'        
0.011076

But unfortunately, the suggested single quote SMTP counter checks do not:

Code:

zabbix_get -s server1-k 'perf_counter["\SMTP Server(_Total)\Local Queue Length"]'        
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.
zabbix_get -s server1-k 'perf_counter["\SMTP Server(_Total)\Remote Retry Queue Length"]'
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.
zabbix_get -s server1-k 'perf_counter["\SMTP Server(_Total)\Inbound Connections Current"]'
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.

And attempts with the additional slash:

Code:

zabbix_get -s server1-k 'perf_counter["\\SMTP Server(_Total)\\Local Queue Length"]'      
ZBX_NOTSUPPORTED: Invalid performance counter format.
zabbix_get -s server1-k 'perf_counter["\\SMTP Server(_Total)\\Remote Retry Queue Length"]'
ZBX_NOTSUPPORTED: Invalid performance counter format.
zabbix_get -s server1-k 'perf_counter["\\SMTP Server(_Total)\\Inbound Connections Current"]'
ZBX_NOTSUPPORTED: Invalid performance counter format.

And trying the previously working Processor counter with two slashes:

Code:

zabbix_get -s server1 -k 'perf_counter["\\Processor Information(_Total)\\% Processor Time"]'
ZBX_NOTSUPPORTED: Invalid performance counter format.

**tim.mooney** · 16-04-2020, 07:04

Ok, the "Cannot obtain performance information from collector" vs. "Invalid performance counter format" means we're getting somewhere. So doubling the \ isn't necessary, the correct format is the first way way.

Regarding the "Cannot obtain performance information from collector", have you tried the solution from this thread:

We’ll be back soon!

https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/53118-performance-counters-don-t-work-in-zabbix

**snmpguru8** · 16-04-2020, 16:40

Originally posted by tim.mooney

Ok, the "Cannot obtain performance information from collector" vs. "Invalid performance counter format" means we're getting somewhere. So doubling the \ isn't necessary, the correct format is the first way way.

Regarding the "Cannot obtain performance information from collector", have you tried the solution from this thread:

We’ll be back soon!

https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/53118-performance-counters-don-t-work-in-zabbix

That's an interesting find. I have reinstalled and even tried installing a different version of the Zabbix agent to no avail.
I just tried restarting the agent and running through it again, but still the SMTP or IIS specific counters are not responding:

Code:

zabbix_get -s server1-k 'perf_counter["\Processor Information(_Total)\% Processor Time"]'
0.012129
zabbix_get -s server1-k 'perf_counter["\SMTP Server(_Total)\Inbound Connections"]'       
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.

**snmpguru8** · 17-04-2020, 19:17

Originally posted by cyber

Might be stupid question, but as I have limited knowledge of windows and perfmon... Are all performance counters available to all users? Zabbix agent on windows usually runs under local system account... and if you do your queries by hand, you have your own credentials.

Thanks, cyber. This was a really great idea, but unfortunately it did not move me closer to a resolution. As the system user, I can query the metrics via system account with psexec:

C:\Windows\system32>whoami
nt authority\system

C:\Windows\system32>typeperf "\SMTP Server(_Total)\Local Queue Length"

"(PDH-CSV 4.0)","\\server1\SMTP Server(_Total)\Local Queue Length"
"04/17/2020 11:15:51.770","0.000000"

**tim.mooney** · 17-04-2020, 22:59

I had high hopes that cyber's idea was the solution for the problem.

Since we have the source code for the perfmon item and it has some debugging already built in, I think the next thing to try is

truncate or remove the existing log file on the agent system in question. If there isn't a lot file, modify your zabbix_agentd.conf so that this agent writes to a log file.
enable debugging on the agent in question. You can either do that with the DebugLevel setting in the agent config file (uncomment and set it to 5) or by using the command line argument for runtime control to increase the log level.
use "zabbix_get" from the Zabbix server to get 1 counter that will succeed, such as "\Processor Information(_Total)\% Processor Time"
use "zabbix_get" from the Zabbix server to get 1 counter in the SMTP "area" that always fails.
sanitize any IP addresses or other stuff that got logged that you don't want to reveal, but post the rest of the logs here.
remember to turn debugging back down to normal on the client system in question.

With the debug messages to compare between a success and a failure, I should be able to narrow down where it's failing. That may provide some inspiration about why it's failing.

PS: in the perfmon code (zabbix-4.4.7/src/libs/zbxwin32/perfmon.c) near the start of the there is a static data structure defined for the builtin counters:

Code:

static struct builtin_counter_ref
{
    unsigned long   pdhIndex;
    wchar_t     eng_name[PDH_MAX_COUNTER_NAME];
}
builtin_counter_map[] =
{
    { 0, L"System" },
    { 0, L"Processor" },
    { 0, L"Processor Information" },
    { 0, L"% Processor Time" },
    { 0, L"Processor Queue Length" },
    { 0, L"System Up Time" },
    { 0, L"Terminal Services" },
    { 0, L"Total Sessions" }
};

Between that and other code in the file that references builtin counters, I'm a little worried that the main issue is that you're querying counters that are not part of the built-in set. However, seeing the debug logs from successful vs. failed should help narrow down where the problem is.

**snmpguru8** · 17-04-2020, 23:28

Attached debug log

Attached Files

snmpguru8_zabbixagentlog.zip (49.8 KB, 31 views)

**snmpguru8** · 17-04-2020, 23:36

Here are some relevant snippets from the debug log (attached in above post):

Code:

  4508:20200417:152047.123 Requested [perf_counter["\SMTP Server(_Total)\Local Queue Length"]]
  4508:20200417:152047.124 In PERF_COUNTER()
  4508:20200417:152047.125 In get_perf_counter_value_by_path() path:\SMTP Server(_Total)\Local Queue Length interval:1
  4508:20200417:152047.126 In add_perf_counter() counter:'\SMTP Server(_Total)\Local Queue Length' interval:1
  4508:20200417:152047.127 add_perf_counter(): unable to add PerfCounter '\SMTP Server(_Total)\Local Queue Length': [0xC0000BB8] The specified object was not found on the computer.
  4508:20200417:152047.127 End of add_perf_counter(): FAIL
  4508:20200417:152047.128 End of get_perf_counter_value_by_path():FAIL
  4508:20200417:152047.129 End of PERF_COUNTER():unknown
  4508:20200417:152047.130 Sending back [ZBX_NOTSUPPORTED: Invalid performance counter format.]

and another SMTP counter:

Code:

  4376:20200417:152052.301 Requested [perf_counter["\SMTP Server(_Total)\Remote Retry Queue Length"]]
  4376:20200417:152052.302 In PERF_COUNTER()
  4376:20200417:152052.303 In get_perf_counter_value_by_path() path:\SMTP Server(_Total)\Remote Retry Queue Length interval:1
  4376:20200417:152052.303 In add_perf_counter() counter:'\SMTP Server(_Total)\Remote Retry Queue Length' interval:1
  4376:20200417:152052.304 add_perf_counter(): unable to add PerfCounter '\SMTP Server(_Total)\Remote Retry Queue Length': [0xC0000BB8] The specified object was not found on the computer.
  4376:20200417:152052.305 End of add_perf_counter(): FAIL
  4376:20200417:152052.306 End of get_perf_counter_value_by_path():FAIL
  4376:20200417:152052.306 End of PERF_COUNTER():unknown
  4376:20200417:152052.307 Sending back [ZBX_NOTSUPPORTED: Invalid performance counter format.]

Vs a successfully CPU query:

Code:

  3868:20200417:152055.043 Requested [perf_counter["\Processor Information(_Total)\% Processor Time"]]
  3868:20200417:152055.045 In PERF_COUNTER()
  3868:20200417:152055.047 In get_perf_counter_value_by_path() path:\Processor Information(_Total)\% Processor Time interval:1
  3868:20200417:152055.048 End of get_perf_counter_value_by_path():SUCCEED
  3868:20200417:152055.050 End of PERF_COUNTER():SUCCEED
  3868:20200417:152055.052 Sending back [62.311693]

Thanks so much for the help

**tim.mooney** · 19-04-2020, 01:40

The successful vs. unsuccessful code path isn't quite the same, because the running zabbix agentd had previously been queried for the "\Processor Information(_Total)\% Processor Time" and had loaded that counter successfully. That's why the successful case doesn't show the call to "add_perf_counter_value_by_path" and then add_perf_counter -- it didn't need to because it had previously loaded that particular counter.

Still, there's enough there to show the exact call that's failing, and the Windows PDH error code it's returning ([0xC0000BB8] The specified object was not found on the computer.)

Keep in mind also that I've been looking at the source code for 4.4.7, but you're using some version of 4.0, so the code might be a little different, but based on log messages probably not much.

The Zabbix docs specific to perf_counter have some interesting information in them. Is the Windows server in question installed with English as its default locale? I assume so, but it's apparently relevant when trying to resolve perf counters by name or path.

Also, I think it would be worth trying the example on that page, to look up what the internal ID is for these counters specific to this host and then try using that, instead of the path, to see if the results change. Can you try that, and report back what you find?

**snmpguru8** · 19-04-2020, 02:22

Originally posted by tim.mooney

The successful vs. unsuccessful code path isn't quite the same, because the running zabbix agentd had previously been queried for the "\Processor Information(_Total)\% Processor Time" and had loaded that counter successfully. That's why the successful case doesn't show the call to "add_perf_counter_value_by_path" and then add_perf_counter -- it didn't need to because it had previously loaded that particular counter.

Still, there's enough there to show the exact call that's failing, and the Windows PDH error code it's returning ([0xC0000BB8] The specified object was not found on the computer.)

Keep in mind also that I've been looking at the source code for 4.4.7, but you're using some version of 4.0, so the code might be a little different, but based on log messages probably not much.

The Zabbix docs specific to perf_counter have some interesting information in them. Is the Windows server in question installed with English as its default locale? I assume so, but it's apparently relevant when trying to resolve perf counters by name or path.

Also, I think it would be worth trying the example on that page, to look up what the internal ID is for these counters specific to this host and then try using that, instead of the path, to see if the results change. Can you try that, and report back what you find?

Hi Tim,

Thanks for the feedback.It does appear that English is the locale:

Code:

PS C:\temp> Get-WinSystemLocale

LCID             Name             DisplayName
----             ----             -----------
1033             en-US            English (United States)

I also grabbed the internal IDs for the counters and had interesting results. From the local machine, via command line, typeperf for the counter Ids fails:

Code:

PS C:\temp> typeperf.exe "4\24"


Error: No valid counters.

But attempting from zabbix for the same common counter is successful:

Code:

zabbix_get -s server1-k 'perf_counter["\4\24"]'
1021124608.000000

But unfortunately, the same cannot be done for the SMTP counter IDs in this case (truncated ID list):

Code:

2730
SMTP Server
2732
Bytes Sent Total
2734
Bytes Sent/sec
2736
Bytes Received Total
2738
Bytes Received/sec
2740
Bytes Total
2742
Bytes Total/sec
2744
Message Bytes Sent Total
2746
Message Bytes Sent/sec
2748
Message Bytes Received Total
2750
Message Bytes Received/sec
2752
Message Bytes Total
2754

And then:

Code:

zabbix_get -s server1-k 'perf_counter["\2730\2732"]'
ZBX_NOTSUPPORTED: Cannot obtain performance information from collector.

This is become more bizarre.

**heshammhafez** · 23-12-2020, 11:35

Hello Guys,

Did anyone find a solution for this issue ?

**rsghotra07** · 23-06-2021, 08:31

Originally posted by snmpguru8

Zabbix 4.0

Zabbix cannot detect the Perfmon counters for the SMTP Server role, all items are "Not Supported"

- I have a Server 2016 Core instance with the SMTP Server role installed.
- "typeperf" commands for SMTP Server counters work fine
- "zabbix_get" from the Zabbix server fails
- Agent debug log shows "The specified object is not found on the system."
- Firewall is off
- From a remote Windows system, I can query the SMTP Server perfmon counters
- Zabbix can query other perfmon counters fine, such as Processor metrics, for instance
- I have tried to reset and rebuild the perflib database
- I have tried to use the SMTP Server counter IDs instead of names to no avail

This is also occurring on a standard 2012 R2 server with the IIS/W3SVC template

I found this old thread from 2011 which stated (https://www.zabbix.com/forum/zabbix-...-not-supported):
"Ok. I found the issue, system was a 64bit machine. Apparently some counters do not work for zabbix when you install a 32bit version unto a 64bit machine.

Thanks for all the help."

I don't know if this is useful, but this issue has me pulling my hair out.

CORRECT FORMAT:

-k 'perf_counter[""\Process(xyz)\Handle Count"]'

Ad Widget

Windows Application perf counters returning "Not Supported"

Windows Application perf counters returning "Not Supported"

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment