Hi Zabbix experts, I have a deployment of zabbix in production, and a lab system for debug.
I have a userparameter in the agent that kicks off a python script that will pull output from a voip subsystem (broadsoft) for telephony fraud. The command gets run from my python script, output gets parsed and i want to create LLD items for the host. Later in the script the values will get sent via zabbix_sender, however i cannot get the items created dynamically and i have checked all my IO and logic.
About a week of troubleshooting has me against the ropes and I hope the community can assist.
(I am about to prove the usability and use case for zabbix versus an onsite vendor deployment, and this may prove the tipping point to convince management we should go zabbix)
1. OS:
root@zimtest-mtn-co-za:/var/log/zabbix# uname -a
Linux zimtest-mtn-co-za 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
2. Version of zabbix:
root@zimtest-mtn-co-za:/var/log/zabbix# zabbix_server --version
zabbix_server (Zabbix) 4.2.6
Revision d3bb18bbc4 26 August 2019, compilation time: Aug 26 2019 14:32:04
Copyright (C) 2019 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.
This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit whatsapp web 192.168.0.1 routerlogin
Compiled with OpenSSL 1.1.0g 2 Nov 2017
Running with OpenSSL 1.1.1 11 Sep 2018
3. My zabbix agent userparameter on the host defined in Zabbix:
UserParameter=mtnvoip.fraud.lld,/usr/lib/zabbix/externalscripts/pbixer.py
4. Attached screen snapshots of my LLD and discovery items.
5. Agent debugs on the zabbix server and agent host (same machine in LAB/DEV). I have disabled any other items on the host so i can get as much of a clean uncluttered debug as possible.I have validated the json sent with json lint.
1201:20190930:160622.781 EXECUTE_STR() command:'/usr/lib/zabbix/externalscripts/pbixer.py' len:23698 cmd_result:'{"data": ["{ #ITEMNA'
1201:20190930:160622.781 Sending back [{"data": ["{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}",
<SNIP><SNIP>
"{ #ITEMNAME:GROUP 104374-018 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103199-001 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-04 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-002 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-06 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-01 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-006 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-03 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-02 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-007 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP MTN-BS-PLK SERVICEPROVIDER MTN}", "{ #ITEMNAME:GROUP hyt001-001 SERVICEPROVIDER hyt001}", "{ #ITEMNAME:GROUP 101793-423 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 101793-220 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 104374-020 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103966-005 SERVICEPROVIDER 103966}", "{ #ITEMNAME:GROUP 104374-021 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 100350-002 SERVICEPROVIDER 100350}", "{ #ITEMNAME:GROUP 103199-008 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104374-022 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 1-1tib1o-122 SERVICEPROVIDER 1-1tib1o}", "{ #ITEMNAME:GROUP 104374-023 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104043-09 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104374-026 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104374-027 SERVICEPROVIDER 104374}", "{ #ITEMNAME:maxsource}", "{ #ITEMNAME:maxnum}", "{ #ITEMNAME:maxdest}"]}]
1201:20190930:160622.781 zbx_setproctitle() title:'listener #8 [waiting for connection]'
1205:20190930:160622.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160622.872 End of send_buffer():SUCCEED
1205:20190930:160622.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160623.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160623.267 In update_cpustats()
1193:20190930:160623.267 End of update_cpustats()
1193:20190930:160623.267 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160623.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160623.710 End of send_buffer():SUCCEED
1206:20190930:160623.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160623.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160623.719 End of send_buffer():SUCCEED
1204:20190930:160623.719 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
1205:20190930:160623.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160623.872 End of send_buffer():SUCCEED
1205:20190930:160623.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160624.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160624.267 In update_cpustats()
1193:20190930:160624.267 End of update_cpustats()
1193:20190930:160624.267 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160624.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160624.710 End of send_buffer():SUCCEED
1206:20190930:160624.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160624.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160624.719 End of send_buffer():SUCCEED
1204:20190930:160624.719 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
1205:20190930:160624.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160624.872 End of send_buffer():SUCCEED
1205:20190930:160624.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160625.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160625.267 In update_cpustats()
1193:20190930:160625.268 End of update_cpustats()
1193:20190930:160625.268 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160625.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160625.710 End of send_buffer():SUCCEED
1206:20190930:160625.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160625.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160625.719 End of send_buffer():SUCCEED
I have a userparameter in the agent that kicks off a python script that will pull output from a voip subsystem (broadsoft) for telephony fraud. The command gets run from my python script, output gets parsed and i want to create LLD items for the host. Later in the script the values will get sent via zabbix_sender, however i cannot get the items created dynamically and i have checked all my IO and logic.
About a week of troubleshooting has me against the ropes and I hope the community can assist.
(I am about to prove the usability and use case for zabbix versus an onsite vendor deployment, and this may prove the tipping point to convince management we should go zabbix)
1. OS:
root@zimtest-mtn-co-za:/var/log/zabbix# uname -a
Linux zimtest-mtn-co-za 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
2. Version of zabbix:
root@zimtest-mtn-co-za:/var/log/zabbix# zabbix_server --version
zabbix_server (Zabbix) 4.2.6
Revision d3bb18bbc4 26 August 2019, compilation time: Aug 26 2019 14:32:04
Copyright (C) 2019 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.
This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit whatsapp web 192.168.0.1 routerlogin
Compiled with OpenSSL 1.1.0g 2 Nov 2017
Running with OpenSSL 1.1.1 11 Sep 2018
3. My zabbix agent userparameter on the host defined in Zabbix:
UserParameter=mtnvoip.fraud.lld,/usr/lib/zabbix/externalscripts/pbixer.py
4. Attached screen snapshots of my LLD and discovery items.
5. Agent debugs on the zabbix server and agent host (same machine in LAB/DEV). I have disabled any other items on the host so i can get as much of a clean uncluttered debug as possible.I have validated the json sent with json lint.
1201:20190930:160622.781 EXECUTE_STR() command:'/usr/lib/zabbix/externalscripts/pbixer.py' len:23698 cmd_result:'{"data": ["{ #ITEMNA'
1201:20190930:160622.781 Sending back [{"data": ["{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}",
<SNIP><SNIP>
"{ #ITEMNAME:GROUP 104374-018 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103199-001 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-04 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-002 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-06 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-01 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-006 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-03 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-02 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-007 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP MTN-BS-PLK SERVICEPROVIDER MTN}", "{ #ITEMNAME:GROUP hyt001-001 SERVICEPROVIDER hyt001}", "{ #ITEMNAME:GROUP 101793-423 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 101793-220 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 104374-020 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103966-005 SERVICEPROVIDER 103966}", "{ #ITEMNAME:GROUP 104374-021 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 100350-002 SERVICEPROVIDER 100350}", "{ #ITEMNAME:GROUP 103199-008 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104374-022 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 1-1tib1o-122 SERVICEPROVIDER 1-1tib1o}", "{ #ITEMNAME:GROUP 104374-023 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104043-09 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104374-026 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104374-027 SERVICEPROVIDER 104374}", "{ #ITEMNAME:maxsource}", "{ #ITEMNAME:maxnum}", "{ #ITEMNAME:maxdest}"]}]
1201:20190930:160622.781 zbx_setproctitle() title:'listener #8 [waiting for connection]'
1205:20190930:160622.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160622.872 End of send_buffer():SUCCEED
1205:20190930:160622.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160623.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160623.267 In update_cpustats()
1193:20190930:160623.267 End of update_cpustats()
1193:20190930:160623.267 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160623.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160623.710 End of send_buffer():SUCCEED
1206:20190930:160623.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160623.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160623.719 End of send_buffer():SUCCEED
1204:20190930:160623.719 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
1205:20190930:160623.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160623.872 End of send_buffer():SUCCEED
1205:20190930:160623.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160624.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160624.267 In update_cpustats()
1193:20190930:160624.267 End of update_cpustats()
1193:20190930:160624.267 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160624.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160624.710 End of send_buffer():SUCCEED
1206:20190930:160624.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160624.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160624.719 End of send_buffer():SUCCEED
1204:20190930:160624.719 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
1205:20190930:160624.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160624.872 End of send_buffer():SUCCEED
1205:20190930:160624.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160625.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160625.267 In update_cpustats()
1193:20190930:160625.268 End of update_cpustats()
1193:20190930:160625.268 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160625.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160625.710 End of send_buffer():SUCCEED
1206:20190930:160625.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160625.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160625.719 End of send_buffer():SUCCEED