Succeccfully polled SNMP V2 item data does not show up in latest data

[b52]

Hi,

I have a problem to see some snmp v2 data in zabbix. I created an item to poll some snmp v2 values, but these values are never listed in latest data of that host, and I have no idea why they are missing.

We use version 4.0.14 on the Zabbix Server and on the Proxy. The Server-Proxy communication works, as I receive other valid values of that proxy in the same time.

The (discovered) Item of the Host:

Discovered by     Cisco Meraki Device
Type             SNMPv2 agent
Key              MerakiDeviceIndexSerial.[224.203.188.146.57.133]
Host interface     XXXX : 16100
SNMP OID         1.3.6.1.4.1.29671.1.1.4.1.8.224.203.188.146.57.133
SNMP Community  {$SNMP_COMMUNITY}
Port             {$SNMP_PORT}
Type of information Character (or Text)
Update interval    300s
Do not keep History
Show value       As is
Enabled
no Preprocessing

If I trigger this item with "check now" the SNMP poller requests that oid and receives a valid value...

Proxy Log with increased logging:

  2784:20191105:141900.805 SNMP [XXXXXXX@XXXXXX:16100]
  2784:20191105:141900.813 End of zbx_snmp_open_session()
  2784:20191105:141900.813 In zbx_snmp_process_standard()
  2784:20191105:141900.813 In zbx_snmp_translate() OID:'1.3.6.1.4.1.29671.1.1.4.1.8.224.203.188.146.57.133'
  2784:20191105:141900.813 End of zbx_snmp_translate() oid_translated:'1.3.6.1.4.1.29671.1.1.4.1.8.224.203.188.146.57.133'
  2784:20191105:141900.813 In zbx_snmp_get_values() num:1 level:0
  2784:20191105:141900.823 zbx_snmp_get_values() snmp_synch_response() status:0 s_snmp_errno:0 errstat:0 mapping_num:1
  2784:20191105:141900.823 In zbx_snmp_set_result() type:4
  2784:20191105:141900.823 In zbx_snmp_get_octet_string()
  2784:20191105:141900.823 zbx_snmp_get_octet_string() full value:'STRING: "XXXX-XXXX-XXXX"' hint:'(null)'
  2784:20191105:141900.823 End of zbx_snmp_get_octet_string():'XXXX-XXXX-XXXX'
  2784:20191105:141900.823 End of zbx_snmp_set_result():SUCCEED
  2784:20191105:141900.824 End of zbx_snmp_get_values():SUCCEED
  2784:20191105:141900.824 End of zbx_snmp_process_standard():SUCCEED
  2784:20191105:141900.824 In zbx_snmp_close_session()
  2784:20191105:141900.824 End of zbx_snmp_close_session()
  2784:20191105:141900.824 End of get_values_snmp()
  2784:20191105:141900.824 In zbx_activate_item_host() hostid:10713 itemid:122118 type:4
  2784:20191105:141900.824 End of zbx_activate_item_host()
  2784:20191105:141900.824 End of get_values():1
  2784:20191105:141900.824 __zbx_zbx_setproctitle() title:'poller #10 [got 1 values in 0.039356 sec, idle 1 sec]'
  2775:20191105:141900.855 __zbx_zbx_setproctitle() title:'poller #1 [got 0 values in 0.000386 sec, getting values]'

But this string never shows up in latest data of that host. Do you have any idea what might be reason or how I may debug that further?
Thank you!

Regards,
b52

[valentins.grins]

Hello,

What is the status of SNMP items in Zabbix frontend? Are they enabled or "Not supported"?
Do you see any queue for zabbix proxy in Administration->Queue?

[b52]

Hi valentins.grins,

Thanks. The items are enabled and without any error messages. The Queue of this proxy looks good:
5s 0
10s 1
30s 68
1m 210
5m 0
10m 0

Any Ideas why it should poll the oid and discard the value?

Best regards,
b52

[valentins.grins]

Did you tried snmpwalk from proxy to the host? What is the output ?
Please try "snmpwalk -v 2c -c public <IP of your CISO> 1.3.6.1.4.1.29671.1.1.4.1.8.224.203.188.146.57.133 ".

Your proxy is active or passive?

What DB do you have on proxy? Can you check if you have this item in proxy DB database? For mysql there will be following query:

select * from zabbix.items where key_ LIKE '%MerakiDeviceIndexSerial%';

[b52]

Thanks valentins.grins!

Snmp seems not to be the problem here. I can snmpget/snmpwalk the value from the proxy and also the zabbix-proxy queries the snmp value successfully, as you see in the logs. The snmp output is the String "XXXX-XXXX-XXXX".

The Proxy uses a sqlite3 DB and is in active mode.

sqlite> select * from items where key_ LIKE '%MerakiDeviceIndexSerial%' and snmp_oid LIKE '%224.203.188.146.57.133%';
122118|4|{$SNMP_COMMUNITY}|1.3.6.1.4.1.29671.1.1.4.1.8.224.203.188.146.57.133|10713||MerakiDeviceIndexSerial.[224.203.188.146.57.133]|60s|90d|365d|0|1||||0|||||0||||||0|||||0|4|534|{$SNMP_PORT}||0|30d|0|0|0||0|||3s||||200|1|0|||0|0|0||||0|0|0

The Item is in the database...

sqlite> select * from proxy_history where itemid=122118;
36154138|122118|1573049514|0||0|XXXX-XXXX-XXXX|0|533879264|0|0|0|0
36154372|122118|1573049574|0||0|XXXX-XXXX-XXXX|0|636526086|0|0|0|0
36154641|122118|1573049634|0||0|XXXX-XXXX-XXXX|0|794441813|0|0|0|0
36154867|122118|1573049694|0||0|XXXX-XXXX-XXXX|0|796643706|0|0|0|0
36155152|122118|1573049757|0||0|XXXX-XXXX-XXXX|0|977174445|0|0|0|0
36155386|122118|1573049814|0||0|XXXX-XXXX-XXXX|0|553305786|0|0|0|0
36155627|122118|1573049874|0||0|XXXX-XXXX-XXXX|0|809851791|0|0|0|0
36155840|122118|1573049934|0||0|XXXX-XXXX-XXXX|0|866566951|0|0|0|0
36156147|122118|1573049995|0||0|XXXX-XXXX-XXXX|0|32385389|0|0|0|0
36156370|122118|1573050054|0||0|XXXX-XXXX-XXXX|0|247420835|0|0|0|0
36156638|122118|1573050114|0||0|XXXX-XXXX-XXXX|0|508068812|0|0|0|0
36156879|122118|1573050174|0||0|XXXX-XXXX-XXXX|0|538353070|0|0|0|0
36157121|122118|1573050234|0||0|XXXX-XXXX-XXXX|0|761883006|0|0|0|0
36157365|122118|1573050295|0||0|XXXX-XXXX-XXXX|0|277745057|0|0|0|0
36157604|122118|1573050355|0||0|XXXX-XXXX-XXXX|0|34437935|0|0|0|0
36157839|122118|1573050414|0||0|XXXX-XXXX-XXXX|0|811836195|0|0|0|0
36158080|122118|1573050474|0||0|XXXX-XXXX-XXXX|0|956619770|0|0|0|0
36158310|122118|1573050534|0||0|XXXX-XXXX-XXXX|0|646596334|0|0|0|0

...and even the values are in the database Strange...

After I activate "Show items without data" (I missed that option before) in hosts Latest data, the items are listed, but of course without data.

[valentins.grins]

Well, try to increase log_level for trapper process on server.

# zabbix_server -R log_level_increase=trapper

and then grep zabbix server log for this item id(122118). If you see there that data is received then it make sense to check if it is actually in database of zabbix-server.

[b52]

Thanks!

Well, the server receives these items...

 10573:20191106:173619.802 trapper got '{"request":"proxy data","host":"XXXXXX.YYYYYYY","session":"5ac9006a9faca4c3856318f2921cc9af","history data":[{"id":36183923,"itemid":28835,"clock":1573058075,"ns":692986246,"value":"0.000000"},{"id":36183924,"itemid":122133,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183925,"itemid":122115,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183926,"itemid":122128,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183927,"itemid":122130,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183928,"itemid":122113,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183929,"itemid":122127,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183930,"itemid":122106,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183931,"itemid":122145,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183932,"itemid":122096,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183933,"itemid":122132,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183934,"itemid":122099,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183935,"itemid":122141,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183936,"itemid":122124,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183937,"itemid":122116,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183938,"itemid":122101,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183939,"itemid":122102,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183940,"itemid":122104,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183941,"itemid":122143,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183942,"itemid":122139,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183943,"itemid":122119,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183944,"itemid":122129,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183945,"itemid":122125,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183946,"itemid":122137,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183947,"itemid":122123,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183948,"itemid":122121,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183949,"itemid":122098,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183950,"itemid":122111,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183951,"itemid":122120,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183952,"itemid":122108,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183953,"itemid":122097,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183954,"itemid":122117,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183955,"itemid":122142,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183956,"itemid":122105,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183957,"itemid":122114,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183958,"itemid":122100,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183959,"itemid":122136,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183960,"itemid":122144,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183961,"itemid":122135,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183962,"itemid":122146,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183963,"itemid":122103,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183964,"itemid":122138,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183965,"itemid":122140,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183966,"itemid":122107,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183967,"itemid":122109,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183968,"itemid":122126,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183969,"itemid":122122,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183970,"itemid":122095,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183971,"itemid":122112,"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183972,"itemid":122110,"c
lock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"},{"id":36183973,"itemid":[COLOR=#FF0000]122118[/COLOR],"clock":1573058075,"ns":720681593,"value":"XXXX-XXXX-XXXX"}],"version":"4.0.14","clock":1573058077,"ns":147543180}'
 10573:20191106:173619.802 In zbx_recv_proxy_data()
 10573:20191106:173619.802 In process_proxy_data()
 10573:20191106:173619.802 log_client_timediff(): timestamp from json 1573058077 seconds and 147543180 nanosecond, delta time from json 102 seconds and 646481461 nanosecond
 10573:20191106:173619.802 In process_proxy_history_data_33()
 10573:20191106:173619.802 In parse_history_data_33()
 10573:20191106:173619.802 End of parse_history_data_33():SUCCEED processed:51/51
 10573:20191106:173619.803 In process_history_data()
 10573:20191106:173619.803 In zbx_preprocess_item_value()
 10573:20191106:173619.803 End of zbx_preprocess_item_value()
 10573:20191106:173619.803 In zbx_preprocess_item_value()
 10573:20191106:173619.803 End of zbx_preprocess_item_value()
......
 10573:20191106:173619.805 In zbx_preprocess_item_value()
 10573:20191106:173619.805 End of zbx_preprocess_item_value()
 10573:20191106:173619.805 In zbx_ipc_socket_write()
 10573:20191106:173619.805 End of zbx_ipc_socket_write():SUCCEED
 10573:20191106:173619.805 End of process_history_data() processed:51
 10573:20191106:173619.805 End of process_proxy_history_data_33():SUCCEED
 10573:20191106:173619.805 End of process_proxy_data():SUCCEED
 10573:20191106:173619.805 query [txnlev:0] [select t.taskid,t.type,t.clock,t.ttl,c.command_type,c.execute_on,c.port,c.authtype,c.username,c.password,c.publickey,c.privatekey,c.command,c.alertid,c.parent_taskid,c.hostid,cn.itemid from task t left join task_remote_command c on t.taskid=c.taskid left join task_check_now cn on t.taskid=cn.taskid where t.status=1 and t.proxy_hostid=10315 and (t.ttl=0 or t.clock+t.ttl>1573058179) order by t.taskid]
 10573:20191106:173619.806 End of zbx_recv_proxy_data():SUCCEED
 10573:20191106:173619.806 __zbx_zbx_setproctitle() title:'trapper #1 [processed data in 0.012005 sec, waiting for connection]'

Do you know how to track this issue on the server? I don't know the database table structure of the server.

[dimir]

[b52]

Thanks dimir!

Uh, I totaly misinterpreted that "Do not keep History"-Setting

I thought it means: Do not keep historical values instead keep only the last value.

Manual says, "item history is not stored".

I think the option name is confusing and should be more clear. The settings name "History storage period" and the word history is about the past, thus all the values collected in the past, and doesn't say anything about the actual (last) value. Since I don't care about the historical serial numbers of a device, they usually don't change, I set the "History storage period" to "Do not keep History". This says "do not keep historical values" but it means "values are not stored at all". Of course I am interested in the last collected value. This option should be renamed to "do not store values".

Further it would be a nice feature to add an option "keep last value only".

At least I learned some debugging possibilities of zabbix

b52

[dimir]

Great proposal. You are welcome to create a feature request in https://support.zabbix.com :-)

[b52]

https://support.zabbix.com/browse/ZBX-16882

Since the option name is confusing/wrong I opened a Problem report instead of a feature request.