Debug LLD with trapper item prototype

[linbo]

In my test environment I try to setup lvs monitor with zabbix LLD.

Zabbix 3.2.0
OS: Ubuntu 14.04.1 LTS

# Discovery rule
1. key: lvs.service
2. Type: Zabbix agent
3. Two Filters with empty regular expression

{#SERVICE_HOST}
    {#SERVICE_PORT}

    # cat /etc/zabbix/zabbix_agentd.conf.d/userparameter_lvs.conf
    UserParameter=lvs.service, python /etc/zabbix/script/lvs_monitor/dump_service.py


    # python /etc/zabbix/script/lvs_monitor/dump_service.py
    {"data": [{"{#SERVICE_HOST}": "10.211.55.12", "{#SERVICE_PORT}": "7777"}, {"{#SERVICE_HOST}": "10.211.55.11", "{#SERVICE_PORT}": "7778"}]}

# Item prototypes 1 works fine
Key: net.tcp.service[tcp,{#SERVICE_HOST},{#SERVICE_PORT}]
Type: Zabbix agent

# Item prototypes 2 doesn't work, no item created in Configuration -> Hosts -> Items page
Key: lvs_service[inbps,{$SERVICE_HOST},{$SERVICE_PORT}]
Type: Zabbix trapper

A crontab use zabbix_sender to send data to zabbix_server

# cat /tmp/lvs_service_stat
    precise64 lvs_service[inpps,10.211.55.12,7777] 1485221745 0
    precise64 lvs_service[inpps,10.211.55.11,7778] 1485221745 0

In zabbix console, discovery rule show error:

Cannot create item: item with the same key "lvs_service[inbps,{$SERVICE_HOST},{$SERVICE_PORT}]" already exists.
Cannot create item: item with the same key "lvs_service[inbps,{$SERVICE_HOST},{$SERVICE_PORT}]" already exists.

After some time, the error message missing, and Zabbix Server log

trapper got '{
        "request":"sender data",
        "data":[
            {
                "host":"precise64",
                "key":"lvs_service[inpps,10.211.55.12,7777]",
                "value":"0",
                "clock":1485221821},
            {
                "host":"precise64",
                "key":"lvs_service[inpps,10.211.55.11,7778]",
                "value":"0",
                "clock":1485221821}],
        "clock":1485221821}'
     16152:20170124:093701.030 In recv_agenthistory()
     16152:20170124:093701.030 In process_hist_data()
     16152:20170124:093701.030 In process_mass_data()
     16152:20170124:093701.031 End of process_mass_data()
     16152:20170124:093701.031 End of process_hist_data():SUCCEED
     16152:20170124:093701.031 In zbx_send_response()
     16152:20170124:093701.031 zbx_send_response() '{"response":"success","info":"processed: 0; failed: 2; total: 2; seconds spent: 0.000102"}'

How to debug LLD issue?

Thanks,
Linbo

[nobodysu]

Key: lvs_service[inbps,{$SERVICE_HOST},{$SERVICE_PORT}]

needs to be

Key: lvs_service[inbps,{#SERVICE_HOST},{#SERVICE_PORT}]

[linbo]

Thank you.

I correct the key name, now item show in zabbix console without issue. But still no data.

# cat /tmp/lvs_service_stat
precise64 lvs_service[inpps,10.211.55.12,7777] 1485390541 0
precise64 lvs_service[inpps,10.211.55.11,7778] 1485390541 0
# zabbix_sender -z 10.211.55.4 -T -i /tmp/lvs_service_stat -vv
zabbix_sender [13922]: DEBUG: answer [{"response":"success","info":"processed: 0; failed: 2; total: 2; seconds spent: 0.000102"}]
Info from server: "processed: 0; failed: 2; total: 2; seconds spent: 0.000102"
sent: 2; skipped: 0; total: 2

The zabbix server log:

 2146:20170126:082958.280 In substitute_key_macros() data:'lvs_service[inbps,{#SERVICE_HOST},{#SERVICE_PORT}]'
  2146:20170126:082958.280 In substitute_lld_macros() data:'{#SERVICE_HOST}'
  2146:20170126:082958.280 End of substitute_lld_macros():SUCCEED data:'10.211.55.11'
  2146:20170126:082958.280 In substitute_lld_macros() data:'{#SERVICE_PORT}'
  2146:20170126:082958.280 End of substitute_lld_macros():SUCCEED data:'7778'
  2146:20170126:082958.280 End of substitute_key_macros():SUCCEED data:'lvs_service[inbps,10.211.55.11,7778]'
  2146:20170126:082958.280 In substitute_key_macros() data:'lvs_service[inbps,{#SERVICE_HOST},{#SERVICE_PORT}]'
  2146:20170126:082958.280 In substitute_lld_macros() data:'{#SERVICE_HOST}'
  2146:20170126:082958.280 End of substitute_lld_macros():SUCCEED data:'10.211.55.12'
  2146:20170126:082958.280 In substitute_lld_macros() data:'{#SERVICE_PORT}'
  2146:20170126:082958.280 End of substitute_lld_macros():SUCCEED data:'7777'
  2146:20170126:082958.280 End of substitute_key_macros():SUCCEED data:'lvs_service[inbps,10.211.55.12,7777]'
  2146:20170126:082958.280 In substitute_key_macros() data:'net.tcp.service[tcp,{#SERVICE_HOST},{#SERVICE_PORT}]'
  2146:20170126:082958.280 In substitute_lld_macros() data:'{#SERVICE_HOST}'
  2146:20170126:082958.280 End of substitute_lld_macros():SUCCEED data:'10.211.55.11'
  2146:20170126:082958.280 In substitute_lld_macros() data:'{#SERVICE_PORT}'
  2146:20170126:082958.280 End of substitute_lld_macros():SUCCEED data:'7778'
  2146:20170126:082958.280 End of substitute_key_macros():SUCCEED data:'net.tcp.service[tcp,10.211.55.11,7778]'
  2146:20170126:082958.280 In substitute_key_macros() data:'net.tcp.service[tcp,{#SERVICE_HOST},{#SERVICE_PORT}]'
  2146:20170126:082958.280 In substitute_lld_macros() data:'{#SERVICE_HOST}'
  2146:20170126:082958.280 End of substitute_lld_macros():SUCCEED data:'10.211.55.12'
  2146:20170126:082958.281 In substitute_lld_macros() data:'{#SERVICE_PORT}'
  2146:20170126:082958.281 End of substitute_lld_macros():SUCCEED data:'7777'
  2146:20170126:082958.281 End of substitute_key_macros():SUCCEED data:'net.tcp.service[tcp,10.211.55.12,7777]'
  2146:20170126:082958.281 In lld_item_update()
  2146:20170126:082958.281 In substitute_lld_macros() data:'LVS Service $2:$3 status'
  2146:20170126:082958.281 End of substitute_lld_macros():SUCCEED data:'LVS Service $2:$3 status'


2153:20170126:082959.375 __zbx_zbx_setproctitle() title:'trapper #4 [processing data]'
  2153:20170126:082959.375 trapper got '{
    "request":"sender data",
    "data":[
        {
            "host":"precise64",
            "key":"lvs_service[inpps,10.211.55.12,7777]",
            "value":"0",
            "clock":1485390541},
        {
            "host":"precise64",
            "key":"lvs_service[inpps,10.211.55.11,7778]",
            "value":"0",
            "clock":1485390541}],
    "clock":1485390599}'
  2153:20170126:082959.375 In recv_agenthistory()
  2153:20170126:082959.375 In process_hist_data()
  2153:20170126:082959.375 In process_mass_data()
  2153:20170126:082959.375 End of process_mass_data()
  2153:20170126:082959.375 End of process_hist_data():SUCCEED
  2153:20170126:082959.375 In zbx_send_response()
  2153:20170126:082959.375 zbx_send_response() '{"response":"success","info":"processed: 0; failed: 2; total: 2; seconds spent: 0.000096"}'
  2153:20170126:082959.375 End of zbx_send_response():SUCCEED
  2153:20170126:082959.375 End of recv_agenthistory()
  2153:20170126:082959.375 __zbx_zbx_setproctitle() title:'trapper #4 [processed data in 0.000571 sec, waiting for connection]'

[linbo]

Sorry my bad, I use wrong key name, now fixed it and everything works fine.

Thank you very much!