zabbix_get and zabbix_agentd OK, web ui says not supported

[gmichels]

Hi all,

I've been trying to solve a strange issue I have lately... Basically there are a few items which the web ui says "not supported", however manually running both zabbix_get on the server and zabbix_agentd on the client return expected values.

$ zabbix_get -sX.X.X.X -k"perf_counter[\Processor(_Total)\% Processor Time]"
0.786965

>zabbix_agentd.exe -t "perf_counter[\Processor(_Total)\% Processor Time]"
perf_counter[\Processor(_Total)\% Processor Time]  [d|0.005760]

Server is version 1.8.1
Client is version 1.8.7

I can get that perf_counter from several other servers, there's just a handful which are having issues.

I have the ability to try other client versions, though upgrading the server is in the roadmap, but not in the near future.

Any ideas on how to solve this?

Thanks

[tchjts1]

Did you hover your mouse over the "Not supported" link to see if it gave you details about why it wasn't supported? Possibly using the incorrect value 'Unsigned" vs "Float"? Should be a float value for that.

[gmichels]

Did you hover your mouse over the "Not supported" link to see if it gave you details about why it wasn't supported? Possibly using the incorrect value 'Unsigned" vs "Float"? Should be a float value for that.

Thanks for your reply.

I tried hovering over the "not supported" link but nothing pops up. This is server version 1.8.1, I'm not sure if that feature is present. The thing is, this is a templated item that is working on several other boxes (it is set as float), so it should be working. And there are other templated items that work on other boxes but not on this one (and another similar one) as well.

I'm not that experienced with Zabbix, but at first I noticed the return from zabbix_agentd is "[d|0.005760]", and I was wondering if the "d" is for "decimal". If so, the agent would be returning the wrong type for the perf_counter. But I issued the same command on another box which the perf_counters are working fine in Zabbix, and the result is also "d", so I'm kind of clueless on this.

Thanks for any information that might help me track down the problem.

[tchjts1]

Any difference between the OS's where it is working and the one that is not working? ie - 32bit vs 64bit? If that's the case, be sure you have the right agent installed.

[gmichels]

Any difference between the OS's where it is working and the one that is not working? ie - 32bit vs 64bit? If that's the case, be sure you have the right agent installed.

We have a really mixed environment here, ranging from Windows 2000 to Windows 2008 R2. These particular boxes are 2008 R2, but we have the same templated perf_counters working on other 2008 R2 (and 2008, 2003, 2000). As the agent is version 1.8.x, they're all 32 bits, even on 64 bits boxes.

On these particular problematic boxes, I can get some information from perf_counters, such as:

perf_counter[\ASP.NET Apps v2.0.50727(__Total__)\Requests Executing]
perf_counter[\Process(dllhost)\Private Bytes]
perf_counter[\MSMQ Service\Total bytes in all queues]
and others

And several others, they all appear as active in Zabbix web ui. Just a few perf counters will show as not supported, such as:

perf_counter[\Processor(_Total)\% Processor Time]
perf_counter[\Memory\Available KBytes]
perf_counter[\LogicalDisk(C:)\% Free Space]
and others

Another funny thing is that I get a "No such service (255)" for one service_state check (also templated and working on several other boxes), however while checking another service, I get the the expected reply.

This is a brand new machine, had just entered production, so I'm really clueless.

[tchjts1]

As the agent is version 1.8.x, they're all 32 bits, even on 64 bits boxes.

So, is it 64bit boxes where the perf_counters are not working correctly?
Just for troubleshooting purposes, I would download the latest 2.0.6 Windows agent and put the 64bit agent on one and see if that corrects the issue. I think it might work OK with Zabbix server-side being at 1.8... enough for troubleshooting anyway.
http://www.zabbix.com/download.php

Otherwise, I would go to the specific box where you are having issues and run this from DOS: typeperf -qx > perfcounter.txt

Then search through that text file and see if the counters you are trying to use are actually present. I know you are getting the value through zabbix_get, but I would look anyway. The fact that you are getting it through zabbix_get makes me think you have a 32bit agent on a 64bit server.

[gmichels]

So, is it 64bit boxes where the perf_counters are not working correctly?

These particular boxes are 64bit, but I have several other boxes which are 64bit and have the perf_counters working fine. So I really don't think this is related to the agent being 32bit, since it works on pretty much all 64bit boxes we have.

Just for troubleshooting purposes, I would download the latest 2.0.6 Windows agent and put the 64bit agent on one and see if that corrects the issue. I think it might work OK with Zabbix server-side being at 1.8... enough for troubleshooting anyway.
http://www.zabbix.com/download.php

I'm not sure if I'll be able to do that. This is production, nothing can break, so I'd rather not take any big chances. I have uninstalled and installed other agent versions, but all in the 1.8.x series. We mostly use 1.8.4 or 1.8.7 at the moment.

Otherwise, I would go to the specific box where you are having issues and run this from DOS: typeperf -qx > perfcounter.txt

Then search through that text file and see if the counters you are trying to use are actually present. I know you are getting the value through zabbix_get, but I would look anyway. The fact that you are getting it through zabbix_get makes me think you have a 32bit agent on a 64bit server.

Yes, the text files has all the counters. I didn't know about the typeperf trick, I always used performance monitor to actually check if the counters were available, but indeed they are.

What would be the big issue of having a 32bit agent on a 64bit server? AFAIK this place never had a 64bit agent running on a 64bit server and Zabbix has been running for a couple years now.

Thanks!

[tchjts1]

What would be the big issue of having a 32bit agent on a 64bit server?

Thanks!

I'm not saying it would be a big issue. Just possibly that some of your counters may not report in correctly. And I am not saying this is the solution to your problem.

I do seem to recall that some other folks had this same issue where some counters came back as not supported... because they had 32 bit agents on 64bit OS's.

"typeperf" isn't a trick. Simply a standard Windows command that will put the counters from PerMon into a text based file. Much easier to sort through than having to load up PerfMon.

But I am out of ideas. The fact that you can get values using zabbix_get, but are "not supported" in the GUI... are baffling.

[gmichels]

I'm not saying it would be a big issue. Just possibly that some of your counters may not report in correctly. And I am not saying this is the solution to your problem.

I do seem to recall that some other folks had this same issue where some counters came back as not supported... because they had 32 bit agents on 64bit OS's.

"typeperf" isn't a trick. Simply a standard Windows command that will put the counters from PerMon into a text based file. Much easier to sort through than having to load up PerfMon.

But I am out of ideas. The fact that you can get values using zabbix_get, but are "not supported" in the GUI... are baffling.

Thanks for your help so far. I'm going to keep trying and if I ever find a solution to this issue, I'll update the thread.

[tchjts1]

What is the log on your Zabbix server indicating for these unsupported items? zabbix_server.log

[gmichels]

What is the log on your Zabbix server indicating for these unsupported items? zabbix_server.log

This:

  1169:20130501:112224.044 Item [HOST:perf_counter[\Processor(_Total)\% Privileged Time]] error: Not supported by Zabbix Agent
  1169:20130501:112224.044 Parameter [HOST:perf_counter[\Processor(_Total)\% Privileged Time]] is not supported by agent Old status [0]
  1170:20130501:112224.776 Item [HOST:perf_counter[\Processor(_Total)\% Processor Time]] error: Not supported by Zabbix Agent
  1170:20130501:112224.777 Parameter [HOST:perf_counter[\Processor(_Total)\% Processor Time]] is not supported by agent Old status [0]
  1171:20130501:112225.001 Item [HOST:perf_counter[\Processor(_Total)\% User Time]] error: Not supported by Zabbix Agent
  1171:20130501:112225.001 Parameter [HOST:perf_counter[\Processor(_Total)\% User Time]] is not supported by agent Old status [0]

But the log file does have some weird entries:

  1222:20130501:112404.507 Sending list of active checks to [X.X.X.X] failed: host [HOST.DOMAIN] not found

I have the host setup to connect to the IP address and not the DNS name, but the DNS name does resolve normally on the zabbix server.

[tchjts1]

Ahhh. So these items that are failing... are they Zabbix agent (Active) items?

If that's the case, the Hostname= value in zabbix_agentd.conf has to be an exact match (including case sensitive) as to the name you have in the GUI.
If you make any changes to that conf file, you need to restart the agent.

[gmichels]

Ahhh. So these items that are failing... are they Zabbix agent (Active) items?

If that's the case, the Hostname= value in zabbix_agentd.conf has to be an exact match (including case sensitive) as to the name you have in the GUI.
If you make any changes to that conf file, you need to restart the agent.

They're not active checks, they're passive.



And anyway, the hostname is the same on both host config and zabbix_agentd.conf, including case.

I guess the "sending list of active checks" in the log means "sending list of enabled checks", not "zabbix agent (active)" checks. Wild guess

[inhayn]

gmichels,

Have you found a solution for your problem?

I had the same problem and what i did was change the item in Frontend:
[\Processor(_Total)\% Privileged Time] to ["\Processor(_Total)\% Privileged Time"].

It works for me.

[gmichels]

Our Zabbix 1.8 installation was a total mess, so when we upgraded to 2.0 we started fresh, and I didn't have any issues since.

I still use the same perfcounter without quotes and it works fine (in 2.2 now).