LLD with trapper

[mdhart]

I am trying to setup monitoring for activemq destinations using LLD. For various reasons I can't use the Java gateway. I have a discovery script that is configured as a UserParameter:

UserParameter=activemq.destination.discovery[*],/usr/local/zabbix/activemq_monitor.py discovery

The template discovery rule key is:

activemq.destination.discovery

and the item prototype is of type "Zabbix trapper" with the key:

activemq.destination[{#DESTNAME},QueueSize]

and AFAIK the discovery portion works properly, as I see the items in the UI under Monitoring -> Latest Data but with the Last Check set to never. I see lots of activity in the logs when the discovery is run with the correct destinations being configured and no obvious errors.

The monitoring script runs every minute, and when it does I see these lines in the server log, but no indication why it failed:

7198:20121217:164204.564 Trapper got [{
"request":"sender data",
"data":[
{
"host":"mdhpeernode01",
"key":"activemq.destination.peernode-ops.QueueSize",
"value":"0"}]}] len 142
7198:20121217:164204.564 In recv_agenthistory()
7198:20121217:164204.564 In process_hist_data()
7198:20121217:164204.564 In process_mass_data()
7198:20121217:164204.564 End of process_mass_data()
7198:20121217:164204.564 End of process_hist_data():SUCCEED
7198:20121217:164204.564 In zbx_send_response()
7198:20121217:164204.564 zbx_send_response() '{
"response":"success",
"info":"Processed 0 Failed 1 Total 1 Seconds spent 0.000130"}'

Any ideas how to debug this further?

thanks
mike

[danrog]

I've ran into this before with other keys because '-'s....try changing

peernode-ops

to

peernode_ops

And run a manual test against it

[mdhart]

Thanks, I have queues with both dashes and underscores, neither work. A bit more information... in the DB the items seem to get populated properly for the host in question, the four items below is what I expect along with the discovery line.

mysql> select i.name as item_name, i.key_, i.status as item_status, h.name as host_name, h.status as host_status from items i, hosts h where i.key_ like 'activemq.destination%' and i.hostid = h.hostid and h.name = 'mdhpeernode01';
+-------------------+----------------------------------------------------+-------------+---------------+-------------+
| item_name | key_ | item_status | host_name | host_status |
+-------------------+----------------------------------------------------+-------------+---------------+-------------+
| Destination | activemq.destination.discovery | 0 | mdhpeernode01 | 0 |
| Queue Size for $1 | activemq.destination[peernode-ops,QueueSize] | 0 | mdhpeernode01 | 0 |
| Queue Size for $1 | activemq.destination[props_queue,QueueSize] | 0 | mdhpeernode01 | 0 |
| Queue Size for $1 | activemq.destination[rtkbox-ops-results,QueueSize] | 0 | mdhpeernode01 | 0 |
| Queue Size for $1 | activemq.destination[signals_topic,QueueSize] | 0 | mdhpeernode01 | 0 |
| Queue Size for $1 | activemq.destination[{#DESTNAME},QueueSize] | 0 | mdhpeernode01 | 0 |
+-------------------+----------------------------------------------------+-------------+---------------+-------------+

I've added some debugging lines into zabbix_server and recompiled, so far I can see that when the trapper gets the information from zabbix_sender, it attempts to save it, and at one point in processing calls DCfind_item() (in DCconfig_get_item_by_key() in libs/zbxdbcache/dbconfig.c) which returns NULL. I'm quickly approaching my ability to read C to further debug this.

thanks
mike

[mdhart]

I should also note i'm running 2.0.4 on both agent and server.

[qix]

I wonder if LLD will create trapper items properly? If it creates the items as agent items instead, it could cause some problems.

Can you maybe spot it under the item config for the host or else you can surely find it in the database tables.

[mdhart]

From what I can see it does create them properly as trapper items... assuming that the item type is described by the enum zbx_item_type_t in include/common.h, then zabbix types are 0 and trapper items are 2.


mysql> select itemid, type, name, key_ from items where key_ like 'activemq.destination[%'
+--------+------+-------------------+----------------------------------------------------+
| itemid | type | name | key_ |
+--------+------+-------------------+----------------------------------------------------+
| 24693 | 2 | Queue Size for $1 | activemq.destination[{#DESTNAME},QueueSize] |
<snipped for brevity>
| 24705 | 2 | Queue Size for $1 | activemq.destination[peernode-ops,QueueSize] |
+--------+------+-------------------+----------------------------------------------------+


thanks
mike

[qix]

One more idea: is the type of info that you are sending correct for the items? Float, signed, character? You could set it to text to verify.

[mdhart]

Thanks for the suggestions, unfortunately that makes no difference either, still no data showing up. Is there a way to tell what's in the config cache on the server short of writing something to dump it out? From my debugging it looks suspiciously like the trapper can't find the key in the cache.

cheers
mike

[mdhart]

I figured it out, thanks everyone for your help!

I was sending data using the zabbix_sender like this:

zabbix_sender -c /path/agent.conf -k activemq.destination.peernode-ops.QueueSize -o 1


However that's incorrect, the key should be:

zabbix_sender -c /path/agent.conf -k "activemq.destination[peernode-ops,QueueSize]" -o 1


And then it works at it should. Hopefully this will help someone else in the future.

cheers!
mike