LLD creation for large amount of items using python script and userparameters

[kgbeast]

Hi Zabbix experts, I have a deployment of zabbix in production, and a lab system for debug.
I have a userparameter in the agent that kicks off a python script that will pull output from a voip subsystem (broadsoft) for telephony fraud. The command gets run from my python script, output gets parsed and i want to create LLD items for the host. Later in the script the values will get sent via zabbix_sender, however i cannot get the items created dynamically and i have checked all my IO and logic.
About a week of troubleshooting has me against the ropes and I hope the community can assist.
(I am about to prove the usability and use case for zabbix versus an onsite vendor deployment, and this may prove the tipping point to convince management we should go zabbix)

1. OS:
root@zimtest-mtn-co-za:/var/log/zabbix# uname -a
Linux zimtest-mtn-co-za 5.0.0-29-generic #31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

2. Version of zabbix:
root@zimtest-mtn-co-za:/var/log/zabbix# zabbix_server --version
zabbix_server (Zabbix) 4.2.6
Revision d3bb18bbc4 26 August 2019, compilation time: Aug 26 2019 14:32:04

Copyright (C) 2019 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/).

Compiled with OpenSSL 1.1.0g 2 Nov 2017
Running with OpenSSL 1.1.1 11 Sep 2018

3. My zabbix agent userparameter on the host defined in Zabbix:
UserParameter=mtnvoip.fraud.lld,/usr/lib/zabbix/externalscripts/pbixer.py

4. Attached screen snapshots of my LLD and discovery items.

5. Agent debugs on the zabbix server and agent host (same machine in LAB/DEV). I have disabled any other items on the host so i can get as much of a clean uncluttered debug as possible.I have validated the json sent with json lint.

1201:20190930:160622.781 EXECUTE_STR() command:'/usr/lib/zabbix/externalscripts/pbixer.py' len:23698 cmd_result:'{"data": ["{ #ITEMNA'
1201:20190930:160622.781 Sending back [{"data": ["{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}",
<SNIP><SNIP>
"{ #ITEMNAME:GROUP 104374-018 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103199-001 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-04 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-002 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-06 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-01 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-006 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-03 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-02 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-007 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP MTN-BS-PLK SERVICEPROVIDER MTN}", "{ #ITEMNAME:GROUP hyt001-001 SERVICEPROVIDER hyt001}", "{ #ITEMNAME:GROUP 101793-423 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 101793-220 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 104374-020 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103966-005 SERVICEPROVIDER 103966}", "{ #ITEMNAME:GROUP 104374-021 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 100350-002 SERVICEPROVIDER 100350}", "{ #ITEMNAME:GROUP 103199-008 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104374-022 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 1-1tib1o-122 SERVICEPROVIDER 1-1tib1o}", "{ #ITEMNAME:GROUP 104374-023 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104043-09 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104374-026 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104374-027 SERVICEPROVIDER 104374}", "{ #ITEMNAME:maxsource}", "{ #ITEMNAME:maxnum}", "{ #ITEMNAME:maxdest}"]}]
1201:20190930:160622.781 zbx_setproctitle() title:'listener #8 [waiting for connection]'
1205:20190930:160622.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160622.872 End of send_buffer():SUCCEED
1205:20190930:160622.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160623.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160623.267 In update_cpustats()
1193:20190930:160623.267 End of update_cpustats()
1193:20190930:160623.267 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160623.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160623.710 End of send_buffer():SUCCEED
1206:20190930:160623.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160623.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160623.719 End of send_buffer():SUCCEED
1204:20190930:160623.719 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
1205:20190930:160623.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160623.872 End of send_buffer():SUCCEED
1205:20190930:160623.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160624.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160624.267 In update_cpustats()
1193:20190930:160624.267 End of update_cpustats()
1193:20190930:160624.267 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160624.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160624.710 End of send_buffer():SUCCEED
1206:20190930:160624.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160624.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160624.719 End of send_buffer():SUCCEED
1204:20190930:160624.719 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
1205:20190930:160624.872 In send_buffer() host:'10.80.52.121' port:10051 entries:0/4096
1205:20190930:160624.872 End of send_buffer():SUCCEED
1205:20190930:160624.872 zbx_setproctitle() title:'active checks #2 [idle 1 sec]'
1193:20190930:160625.267 zbx_setproctitle() title:'collector [processing data]'
1193:20190930:160625.267 In update_cpustats()
1193:20190930:160625.268 End of update_cpustats()
1193:20190930:160625.268 zbx_setproctitle() title:'collector [idle 1 sec]'
1206:20190930:160625.710 In send_buffer() host:'127.0.0.1' port:10051 entries:0/4096
1206:20190930:160625.710 End of send_buffer():SUCCEED
1206:20190930:160625.710 zbx_setproctitle() title:'active checks #3 [idle 1 sec]'
1204:20190930:160625.719 In send_buffer() host:'10.217.109.114' port:10051 entries:0/4096
1204:20190930:160625.719 End of send_buffer():SUCCEED




6. Server debugs on the zabbix server and agent host (same machine in LAB/DEV). I have disabled any other items on the host so i can get as much of a clean uncluttered debug as possible. I have validated the json sent with json lint

1719:20190930:160622.781 get value from agent result: '{"data": ["{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}", "{ #ITEMNAME:[email protected]:}",
<SNIP><SNIP>
"{ #ITEMNAME:GROUP 104374-018 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103199-001 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-04 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-002 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-06 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-01 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-006 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104043-03 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104043-02 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 103199-007 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP MTN-BS-PLK SERVICEPROVIDER MTN}", "{ #ITEMNAME:GROUP hyt001-001 SERVICEPROVIDER hyt001}", "{ #ITEMNAME:GROUP 101793-423 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 101793-220 SERVICEPROVIDER 101793}", "{ #ITEMNAME:GROUP 104374-020 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 103966-005 SERVICEPROVIDER 103966}", "{ #ITEMNAME:GROUP 104374-021 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 100350-002 SERVICEPROVIDER 100350}", "{ #ITEMNAME:GROUP 103199-008 SERVICEPROVIDER 103199}", "{ #ITEMNAME:GROUP 104374-022 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 1-1tib1o-122 SERVICEPROVIDER 1-1tib1o}", "{ #ITEMNAME:GROUP 104374-023 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104043-09 SERVICEPROVIDER 104043}", "{ #ITEMNAME:GROUP 104374-026 SERVICEPROVIDER 104374}", "{ #ITEMNAME:GROUP 104374-027 SERVICEPROVIDER 104374}", "{ #ITEMNAME:maxsource}", "{ #ITEMNAME:maxnum}", "{ #ITEMNAME:maxdest}"]}'
1719:20190930:160622.781 End of get_value_agent():SUCCEED
1719:20190930:160622.781 End of get_value():SUCCEED
1719:20190930:160622.781 In zbx_activate_item_host() hostid:10269 itemid:28695 type:0
1719:20190930:160622.781 End of zbx_activate_item_host()
1719:20190930:160622.781 In zbx_preprocess_item_value()
1719:20190930:160622.781 End of zbx_preprocess_item_value()
1719:20190930:160622.781 In zbx_ipc_socket_write()
1719:20190930:160622.781 End of zbx_ipc_socket_write():SUCCEED
1719:20190930:160622.781 End of get_values():1
1719:20190930:160622.781 zbx_setproctitle() title:'poller #5 [got 1 values in 6.891772 sec, idle 5 sec]'
1735:20190930:160622.781 zbx_ipc_service_recv() code:2 size:23769 data:17 70 00 00 00 00 00 00 | 04 01 00 00 00 00 00 01 | de 0b 92 5d 29 f3 95 2e | 01 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 93 5c 00 | 00 7b 22 64 61 74 61 22 | 3a 20 5b 22 7b 20 23 49 | 54 45 4d 4e 41 4d 45 3a | 32 37 38 37 37 34 33 34 | 33 35 36 40 6d 74 6e 2e | 63 6f 2e 7a 61 3a 7d 22 | 2c 20 22 7b 20 23 49 54 | 45 4d 4e 41 4d 45 3a 32 | 37 38 37 37 35 30 31 30
1735:20190930:160622.781 End of zbx_ipc_service_recv():1
1735:20190930:160622.781 In preprocessor_add_request()
1735:20190930:160622.781 In preprocessor_sync_configuration()
1735:20190930:160622.781 In DCconfig_get_preprocessable_items()
1735:20190930:160622.781 End of DCconfig_get_preprocessable_items() items:0
1735:20190930:160622.781 End of preprocessor_sync_configuration() item config size: 0, history cache size: 0
1735:20190930:160622.781 In preprocessor_enqueue() itemid: 28695
1735:20190930:160622.781 In zbx_ipc_socket_write()
1735:20190930:160622.781 End of zbx_ipc_socket_write():SUCCEED
1735:20190930:160622.781 In preprocessor_enqueue_dependent() itemid: 28695
1735:20190930:160622.781 End of preprocessor_enqueue_dependent()
1735:20190930:160622.781 End of preprocessor_enqueue()
1741:20190930:160622.781 zbx_ipc_service_recv() code:1200 size:23724 data:17 70 00 00 00 00 00 00 | 93 5c 00 00 7b 22 64 61 | 74 61 22 3a 20 5b 22 7b | 20 23 49 54 45 4d 4e 41 | 4d 45 3a 32 37 38 37 37 | 34 33 34 33 35 36 40 6d | 74 6e 2e 63 6f 2e 7a 61 | 3a 7d 22 2c 20 22 7b 20 | 23 49 54 45 4d 4e 41 4d | 45 3a 32 37 38 37 37 35 | 30 31 30 37 31 40 6d 74 | 6e 2e 63 6f 2e 7a 61 3a | 7d 22 2c 20 22 7b 20 23 | 49 54 45 4d 4e 41 4d 45 | 3a 32 37 38 37 37 34 31 | 33 37 30 37 40 6d 74 6e
1741:20190930:160622.782 End of zbx_ipc_service_recv():1
1741:20190930:160622.782 In lld_queue_request()
1741:20190930:160622.782 queuing discovery rule:28695
1741:20190930:160622.782 End of lld_queue_request()
1741:20190930:160622.782 In zbx_ipc_client_send() clientid:2
1741:20190930:160622.782 End of zbx_ipc_client_send():SUCCEED
1741:20190930:160622.782 In zbx_ipc_service_recv() timeout:1
1743:20190930:160622.782 zbx_ipc_socket_read() code:1100 size:23724 data:17 70 00 00 00 00 00 00 | 93 5c 00 00 7b 22 64 61 | 74 61 22 3a 20 5b 22 7b | 20 23 49 54 45 4d 4e 41 | 4d 45 3a 32 37 38 37 37 | 34 33 34 33 35 36 40 6d | 74 6e 2e 63 6f 2e 7a 61 | 3a 7d 22 2c 20 22 7b 20 | 23 49 54 45 4d 4e 41 4d | 45 3a 32 37 38 37 37 35 | 30 31 30 37 31 40 6d 74 | 6e 2e 63 6f 2e 7a 61 3a | 7d 22 2c 20 22 7b 20 23 | 49 54 45 4d 4e 41 4d 45 | 3a 32 37 38 37 37 34 31 | 33 37 30 37 40 6d 74 6e
1743:20190930:160622.782 End of zbx_ipc_socket_read():SUCCEED
1735:20190930:160622.782 In preprocessor_assign_tasks()
1743:20190930:160622.782 In lld_process_task()
1743:20190930:160622.782 processing discovery rule:28695
1743:20190930:160622.782 In lld_process_discovery_rule() itemid:28695
1743:20190930:160622.782 query [txnlev:0] [select hostid,key_,evaltype,formula,lifetime from items where itemid=28695]
1735:20190930:160622.782 In preprocessor_get_next_task()
1735:20190930:160622.782 End of preprocessor_get_next_task()
1735:20190930:160622.782 End of preprocessor_assign_tasks()
1735:20190930:160622.782 End of preprocessor_add_request()
1735:20190930:160622.782 In zbx_ipc_service_recv() timeout:1
1743:20190930:160622.782 In substitute_simple_macros() data:'30d'
1743:20190930:160622.782 query [txnlev:0] [select item_conditionid,macro,value,operator from item_condition where itemid=28695]
1743:20190930:160622.782 In zbx_lld_macro_paths_get()
1743:20190930:160622.782 query [txnlev:0] [select lld_macro,path from lld_macro_path where itemid=28695 order by lld_macro]
1743:20190930:160622.782 End of zbx_lld_macro_paths_get()
1743:20190930:160622.782 In lld_rows_get()
1743:20190930:160622.782 End of lld_rows_get():SUCCEED
1743:20190930:160622.782 In lld_update_items()
1743:20190930:160622.782 In lld_item_prototypes_get()

[kgbeast]

I solved my own problem. It seems like zabbix is very specific in how the json is presened to it in an LLD discovery. If you do a jsondump from python it refuses to ingest the correctly formatted json. I had to dump all the LLD counters into a list, transform to a string, and then append / prepend the json formatting manually. Interesting to see no reply or zero interest from the community in the 2 weeks this has been posted.I have another issue that is seeing my 4.2 distro of zabbix on it's arse, the biggest hurdle for me in getting zabbix to compete in my corporate environment against other packages is the sad lack of response unless you already have paid for the support.
If anyone bumps into the issue, here is the code that fixed it for me (steer clear of python json.dump)

json_me = list(filter(None,create_lld_list)) #remove empty items from the list
for i,item in enumerate(json_me):
# format the LLD string for the elements
if 'USER' in item:
json_me[i] = '{ "{#USERITEM}":"' + item + '" }'
if 'GROUP' in item:
json_me[i] = '{ "{#GROUPITEM}":"' + item + '" }'
json_edited = ', '.join(json_me)
#THIS PRINT OUTPUT FOR ZABBIX AGENT TO PICK UP AND CREATE LLD ITEMS
print('{ "data": [')
print(json_edited)
print(']}')

[kloczek]

I see that you have some syntax issues in generated json.
To test is that syntax is just use:

/usr/lib/zabbix/externalscripts/pbixer.py | jq .

[talha123]

I am using the same sceneirio, But I am not able to get values of the created item in latest data.

Although I also added pre-processing in javascript in my item