Elasticsearch Issues with Trapper data

[mdiorio]

Hi,

If I can get this to work, this is the best use for Zabbix and Elastic monitoring.
I get Zabbix agent results back for master_node and cluster health status, but none of the Zabbix Trapper items are returning any data.

In the es_stats_zabbix.log I'm seeing the following:

 2016-03-28 11:21:22,202 DEBUG es_stats_zabbix batch:131 Batch mode with named batch: thirty_seconds
 2016-03-28 11:21:22,203 DEBUG es_stats_zabbix batch:137 Batch config args: {'key3': 'health[initializing_shards]', 'key2': 'health[relocating_shards]', 'key1': 'health[unassigned_shards]', 'server': 'ghq-1pmonap01.globalspec.net', 'host': 'ghq-1delasticnode01.globalspec.net', 'key4': 'health[delayed_unassigned_shards]', 'port': '10051'}
 2016-03-28 11:21:22,203 DEBUG es_stats_zabbix batch:143 Batch keys: {'key3': 'health[initializing_shards]', 'key2': 'health[relocating_shards]', 'key1': 'health[unassigned_shards]', 'key4': 'health[delayed_unassigned_shards]'}
 2016-03-28 11:21:22,204 DEBUG es_stats_zabbix.utils parse_key:142 API: health Node: None Key: initializing_shards
 2016-03-28 11:21:22,204 DEBUG es_stats_zabbix.utils parse_key:142 API: health Node: None Key: relocating_shards
 2016-03-28 11:21:22,204 DEBUG es_stats_zabbix.utils parse_key:142 API: health Node: None Key: unassigned_shards
 2016-03-28 11:21:22,204 DEBUG es_stats_zabbix.utils parse_key:142 API: health Node: None Key: delayed_unassigned_shards
 2016-03-28 11:21:22,204 DEBUG es_stats_zabbix batch:150 API-separated keys: {'nodeinfo': [], 'clusterstate': [], 'health': [('health', None, 'initializing_shards'), ('health', None, 'relocating_shards'), ('health', None, 'unassigned_shards'), ('health', None, 'delayed_unassigned_shards')], 'nodestats': [], 'clusterstats': []}
 2016-03-28 11:21:22,205 DEBUG urllib3.util.retry from_int:155 Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
 2016-03-28 11:21:22,208 DEBUG urllib3.connectionpool _make_request:383 "GET /_cluster/health HTTP/1.1" 200 390
 2016-03-28 11:21:22,208 INFO elasticsearch log_request_success:63 GET http://127.0.0.1:9200/_cluster/health [status:200 request:0.003s]
 2016-03-28 11:21:22,209 DEBUG elasticsearch log_request_success:65 > None
 2016-03-28 11:21:22,209 DEBUG elasticsearch log_request_success:66 < {"cluster_name":"artemis_dev","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":62,"active_shards":62,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":62,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}
 2016-03-28 11:21:22,210 DEBUG es_stats_zabbix batch:165 Metrics: [Metric('ghq-1delasticnode01.globalspec.net', 'health[initializing_shards]', 0), Metric('ghq-1delasticnode01.globalspec.net', 'health[relocating_shards]', 0), Metric('ghq-1delasticnode01.globalspec.net', 'health[unassigned_shards]', 62), Metric('ghq-1delasticnode01.globalspec.net', 'health[delayed_unassigned_shards]', 0)]
 **2016-03-28 11:21:22,214 DEBUG zbxsender send_to_zabbix:58 Got response from Zabbix: {u'info': u'processed: 0; failed: 4; total: 4; seconds spent: 0.000018', u'response': u'success'}
 2016-03-28 11:21:22,214 INFO zbxsender send_to_zabbix:59 processed: 0; failed: 4; total: 4; seconds spent: 0.000018
 **2016-03-28 11:21:22,214 DEBUG es_stats_zabbix batch:167 Result = True
 2016-03-28 11:21:22,215 INFO es_stats_zabbix batch:170 Job completed.

In zabbix server log set to debug, I do see:

9185:20160328:112635.725 __zbx_zbx_setproctitle() title:'trapper #4 [processing data]'
 9185:20160328:112635.725 trapper got '{
 "request":"sender data",
 "data":[
 {
 "host":"ghq-1delasticnode01.globalspec.net",
 "key":"health[initializing_shards]",
 "value":0,
 "clock":1459178782.36},
 {
 "host":"ghq-1delasticnode01.globalspec.net",
 "key":"health[relocating_shards]",
 "value":0,
 "clock":1459178782.36},
 {
 "host":"ghq-1delasticnode01.globalspec.net",
 "key":"health[unassigned_shards]",
 "value":62,
 "clock":1459178782.36},
 {
 "host":"ghq-1delasticnode01.globalspec.net",
 "key":"health[delayed_unassigned_shards]",
 "value":0,
 "clock":1459178782.36}]
 }

Which tells me that it's getting the data, but when I view the last data for the host, there is no data there.

Any ideas? I've only been using Zabbix for two weeks as I'm evaluating new monitoring solutions, and this is really my first major hurdle for it.

Thanks.


Zabbix Version 3.0.1
es_stats_zabbix version: 0.1.3
Elasticsearch version: 2.2.0

[kloczek]

Which tells me that it's getting the data, but when I view the last data for the host, there is no data there.

Do you have in list of registered hosts ghq-1delasticnode01.globalspec.net and/or in list of monitored items on this host item with health[initializing_shards] and other key definition as "zabbix trapper" items?

[mdiorio]

Sure do. Monitors using zabbix agent on the elastic node host return data successfully. It's only the zabbix trapper items that don't contain data.

[blackspiraldancer]

Same exact behavior here. Single checks are OK, but no trapping.

[urusha]

fix

Python zbxsend uses non-integer timestamp, which zabbix-server doesn't accept.
One way to fix this is changing

clock

to

int(clock)

in

metrics_data.append

in zbxsend.py
Here is the issue about it: https://github.com/pistolero/zbxsend/issues/12