Ubiquiti UniFi + zabbix

[corsino]

Hi Sadman,

I sent you a private message. I need just an explanation to have all sites monitoring working.


Kind regards,

Fabio Corsino

[sadman]

Благодаря усилиям наших иностранных товарищей, которые не покладая рук тестируют скрипт и подкидывают разнообразную информацию, Miner жирнеет не по годам, а по дням.

На сегодняшний момент:
- изменен алгоритм кеширования, в связи с чем ожидается уменьшение вероятности выпадения элементов данных в состояние unsupported.
- в LLD добавлена поддержка USW - UniFi Switch. Самого устройства у меня нет, поэтому темплейт тестировать не на чем.
- формат составных ключей расширен. Теперь можно задавать фильтр, по которому будет отобран элемент массива JSON-ответа контроллера. Ранее это представляло проблему, так как элементы не именуются, а применять в качестве ссылки индекс было бы неверно. Отсутствие такой возможности делало наблюдение за портами UniFi Switch невозможным.
- проект завел себе страничку на github, описание на русском и ломанном английском.

Забирать скрипт тут: https://github.com/zbx-sadman/unifi_miner
Читать описание: https://github.com/zbx-sadman/unifi_...ual-on-Russian и https://github.com/zbx-sadman/unifi_...ual-on-English

Примеры применения добавленных возможностей

# Получение LLD для USW
./unifi_miner.pl -o usw

# Получение количества гостевых сетей, поддерживаемых конкретной беспроводной точкой
./unifi_miner.pl -o uap -i 5523fd299932508ffaf3b400 -k "vap_table.[is_guest=1].name" -a count

# Получение значение ESSID гостевой беспроводной сети имеющей шифрование WEP, поднятой на точке с заданным MAC
./unifi_miner.pl -o uap -m 00:27:22:d8:33:23 -k "vap_table.is_guest=1&is_wep=1].essid" -a get

# Получение значения ключа port_poe, относящегося к порту #13 коммутатора с заданным id
./unifi_miner.pl -o usw -i 5571c343e4b0c5f69fea8d2c -k "port_table.[port_idx=13].port_poe" -a get

# Получение значения ключа port_poe, относящегося к порту коммутатора, имеющему алиас "Port 1"
./unifi_miner.pl -o usw -i 5571c343e4b0c5f69fea8d2c -k "port_table.[name=Port 1].port_poe" -a get

Примеры соответствующих UserParameter для агента zabbix приведены в кратком руководстве.

[bbrendon]

This is really cool. I'd love to get some of this data into zabbix.

But after trying it, I think this needs to be re-architected. I'm not exactly sure how yet, but I just found out by trying to implement it that it doesn't scale well.

The two main problems I have found.
1. Sites aren't supported.
2. zabbix agents != unifi controllers or sites

Ideally you would have one controller with all sites in it which equals to one host in zabbix OR each unifi site equal one host in zabbix.

I think the best way to do this is with external checks or with a combination of external checks and zabbix_sender (aka trapper).

To get around #2 I suppose you could run two zabbix agents on one host with seperate config files. One for the host itself and one for the Unifi controller/site. Thinking about it more, this would require one agent instance per unifi site which makes it a bad choice.

Anyway, I wasn't expecting this to be this hard to implement. It appears to be 99% done at least but I'm finding it needs a lot of help. I might dabble with it more if I have time.

[bbrendon]

I converted the zabbix template to external checks. I've attached it. Its better in the way the data is presented in zabbix and also supports sites. Ultimately though, it overloads the server.

I think a real solution has to use cron+zabbix_sender for bulk data collection and external checks only for discovery.

I can't use it because it doesn't scale. Each individual metric requires an execution of perl which is very inefficient. I applied it to a few sites and the server load went too high. I was going to write up some instructions but since I don't like it and needs an overhaul, you are on your own except for a few hints.

1. use the unifi_miner script and hard code username, password, location.
2. put the script in the zabbix external script dir
3. import the template
4. create a host in zabbix and use the SITE macro.

[sadman]

Thank for your remarks, bbrendon.

I think you will agree that best way for Zabbix & UniFi integration is using loadable module, which get, with some kind of magic, values of every metric without downloading huge JSON. But no module is exist, no magic way is exist too and vendor share only shell scripts, which fetch data from Controller. So, i think that any external app, which started by Zabbix agent or Zabbix server take the same resources, but on different hosts. Or is it not so?

BTW, in my case, monitoring of the controller, which contain one six-UAPs site (~200 items) take no more 20% of CPU load (i use Virtual Machine for UniFi Controller). It's OK for me. And i can't check how do they feel the big system, because no devices here.

So, i haven't undoubtedly the best method of delivering output of Perl script to Zabbix. My template created as an example. Everyone can choose their own way to integration console application and Zabbix.

I hope you will share real ideas to increase performance UniFi Miner.

[bbrendon]

Maybe you don't understand how zabbix sender works? With the way you have it now, each metric requires one execution of your script. So for a few APs, it probably takes about 100 executions. With zabbix sender, you can do it with one execution.

Zabbix sender allows you to format a txt file in an item/key data format and gobbles up everything at once

I agree the current scripts work, but they will only work for 20 or so access points before the load becomes a problem.

I would improve your script but I don't have the time to devote to rewriting it in this way especially since my perl is very rusty. Most scripts I do now are easy bash things.

[bbrendon]

I poked around the underpinnings of zabbix and it all looks possible. You can even do discovery using trapper items.

I think the best thing to do would be to map a Unifi site to a zabbix host. Your script would have to output the hostname configured in zabbix so that would have to be a parameter. For example this command:

# default is the site name in unifi controller
# unifi1 is the hostname in zabbix
unifi_miner.pl -s default -h unifi1 > /tmp/sendToTrapper

example contents of sendToTrapper:

unifi1 num_sta[536977580cf29e4ac5aef442] 3
unifi1 rx_bytes[536977580cf29e4ac5aef442] 323923
....
...1000 or so more lines of statistics

Here is a proof of concept Item Prototype. I added this to the existing template and ran zabbix sender against it successfully.

Name: Clients connected to UAP "{#ALIAS}"
Type: Trapper
Key: num_sta[{#ID}]

The sendToTrapper data would get into zabbix like such

zabbix_sender -z localhost -i /tmp/sendToTrapper

I didn't test discovery using trapper item types, but apparently it also works. See here.

https://www.zabbix.com/forum/showthread.php?t=42834

What do you think? It looks like it would be relatively easy to modify your script to output a txt file like sendToTrapper above.

I'll make the templates!

[bbrendon]

Here is some code that sort of works. Still learning python and I'm not really sure how to enumerate the lists properly.

#!/usr/bin/env python

import argparse
import pickle
import pprint

from unifi.controller import Controller

parser = argparse.ArgumentParser()
parser.add_argument('-c', '--controller', default='unifi', help='the controller address (default "unifi")')
parser.add_argument('-u', '--username', default='admin', help='the controller username (default("admin")')
parser.add_argument('-p', '--password', default='', help='the controller password')
parser.add_argument('-b', '--port', default='8443', help='the controller port (default "8443")')
parser.add_argument('-v', '--version', default='v2', help='the controller base version (default "v2")')
parser.add_argument('-s', '--siteid', default='default', help='the site ID, UniFi >=3.x only (default "default")')
args = parser.parse_args()

c = Controller(args.controller, args.username, args.password, args.port, args.version, args.siteid)

aps = c.get_aps()
ap_names = dict([(ap['mac'], ap['name']) for ap in aps])
clients = c.get_clients()
clients.sort(key=lambda x: -x['rssi'])

for ap in aps:
    print "beginning1..."

    for i in ap:
        print "hostname "+ i + "[" + ap['_id'] + "]" + " " + str(ap[i])
    print "end1...."

example output

hostname scanning[53681ed80cf290bb08f6fb78] False
hostname vwire_table[53681ed80cf290bb08f6fb78] []
hostname version[53681ed80cf290bb08f6fb78] 3.2.1.2601
hostname wlan_overrides[53681ed80cf290bb08f6fb78] []
hostname ip[53681ed80cf290bb08f6fb78] 10.2.0.104
hostname ng_ast_be_xmit[53681ed80cf290bb08f6fb78] 1662
hostname site_id[53681ed80cf290bb08f6fb78] 53681e330cf290bb08f6fb67
hostname inform_url[53681ed80cf290bb08f6fb78] http://64 0/inform
hostname ng_ast_cst[53681ed80cf290bb08f6fb78] 4
hostname radio_table[53681ed80cf290bb08f6fb78] [{u'antenna_gain': u'6', u'name': u'wifi0', u'builtin_ant_gain': 0, u'ht': u'20', u'max_txpower': 23, u'min_txpower': 5, u'builtin_antenna': True, u'tx_power': u'auto', u'radio': u'ng', u'channel': u'0', u'tx_power_mode': u'high'}]
hostname ng-state[53681ed80cf290bb08f6fb78] RUN
hostname na_tx_rates[53681ed80cf290bb08f6fb78] []
hostname serial[53681ed80cf290bb08f6fb78] 24A43C6C3AF0
hostname rx_bytes-d[53681ed80cf290bb08f6fb78] 276
hostname _uptime[53681ed80cf290bb08f6fb78] 553333
hostname wlangroup_id_na[53681ed80cf290bb08f6fb78] 53681e330cf290bb08f6fb6b
hostname downlink_table[53681ed80cf290bb08f6fb78] []
hostname x[53681ed80cf290bb08f6fb78] 520.666666667
hostname uptime[53681ed80cf290bb08f6fb78] 553333
hostname ng-tx_power[53681ed80cf290bb08f6fb78] 23
hostname guest_token[53681ed80cf290bb08f6fb78] 7C2FFBF0FA1DBD8218162B8D300E8ED4
hostname ng_tx_rates[53681ed80cf290bb08f6fb78] [144444, -1]
hostname ng-gain[53681ed80cf290bb08f6fb78] 0
hostname inform_ip[53681ed80cf290bb08f6fb78] 3.3.9.10
hostname isolated[53681ed80cf290bb08f6fb78] False
hostname adopted[53681ed80cf290bb08f6fb78] True
hostname state[53681ed80cf290bb08f6fb78] 1
hostname ng_tx_packets[53681ed80cf290bb08f6fb78] 0
...

[sadman]

I have a few questions:

1) How about Item's "Update interval (in sec)" - it will be N sec for all metrics?
2) How about LLD filter using?
3) How about count/sum/etc operations?
4) How about metrics in nested object like vap_table, port_table
5) How about nuking Zabbix with over 9000 items per minute, when i need for guest/user numbers only.
6) How about filter-keys?
7) How about maping info from stat/sta?

[bbrendon]

I'm not sure I understand all the questions but I'll try.

1) How about Item's "Update interval (in sec)" - it will be N sec for all metrics?
A: This is determined by how often zabbix sender runs. If you want different intervals for different metrics, then the script would need to be able to output them seperately. This could be done by intelligence in the script (say grouping different metrics types and having a -g1,-g2,-g3 type param) or simply with grep/awk.

2) How about LLD filter using?
A: Not sure what you're asking but LLD works based on someone's testing in the link I posted before.

3) How about count/sum/etc operations?
A: Not sure what you're asking. You can use the zabbix functions or create your own additional metrics using the script.

4) How about metrics in nested object like vap_table, port_table
A: I saw these. I'm not sure what data is in there and haven't looked it it closely but i'm guessing it would be something like:
vap_table.bssid
vap_table.rx_packets

There might be some script intelligence needed to appropriately handle something here. Not sure.

5) How about nuking Zabbix with over 9000 items per minute, when i need for guest/user numbers only.
This sounds like it relates to #1 above. Also, whatever metric is disabled in zabbix will be ignored by zabbix. So for example if you send the whole output of the script, zabbix will ignore the lines where a zabbix key doesn't exist.

6) How about filter-keys?
A: Not sure what you mean.

7) How about maping info from stat/sta?
A: I haven't dug into the data much yet so I'm not sure what this is.

[sadman]

Ok.

1) Then we have two point of control - inside script & zabbix. Manual sorting/regrouping metrics, awk-ing, debugging- hard way to administration the system. What is profit for user?

2) I want to control all USW ports, which is UP. Your solution - grep as in 1)?

3) For example: i want to get a number of unadopted devices and send alarm if its > 0. Or: I want to get incoming traffic for site (sum rx_bytes from all uaps). What zabbix functions help me to sum all rx_bytes[{#ID}] Items?

5) Recieve first and ignore then? Trappers process made is busy, SQL request to checking key existence bump the server and other overheads?

6) Example from the life: someone want to know how many pending guests on the site and link trigger to that Item. Pending guest - is guest, which is connected to UAP, but not authorized thru guest portal. I can do: ./unifi_miner.pl ... -o user -k "[authorized=0].is_guest" -a sum. What way to get number of pending guests with zabbix_sender?

7) see 6) - connected user's info stored into stat/sta and updates permanently. If u have big UAPs installation, then u have a HUGE heap of users. And flood Zabbix with big massives of metrics, what saves to SQL, but needed only for one `sum` operation.

...and two Miner is better that one. Let's compare your and mine

[bbrendon]

1. The profit for the user is that they can monitor lots of Unifi APs with probably no noticeable load on the server.

2. Not sure what a USW Port is.

3. Zabbix has a way to sum multiple metrics, but I'm not sure it'll work for discovered items. If you want to Sum the xfer of the site and unifi doesn't have the metric, it might be necessary to add the code to the script.

5. Not sure what you're getting at, sorry. If zabbix doesn't have a item enabled and its sent to it with zabbix_sender, it gets ignored.

6. You probably can't. zabbix_sender only sends data you tell it to Zabbix.

7. Yea, like I said before. In order for this to work you'd probably have to have the script do some thinking.

I don't for-see me ever making anything as comprehensive as your script. I just want something that I can use. Maybe the answer right now is two scripts. One for bulk data and one for processed data. This wouldn't be ideal, but it would at least be usable for me.

[sadman]

I'm not sure that awk'ing and grepping will be cheapest, that perl scripting.

U point to https://www.zabbix.com/documentation...ypes/aggregate for sumarize various item? I see that this function for items on many hosts, but not for many items on one host.

USW is UniFi switches. UVP is VoIP devices. USG is Gateways... and all in one site, in one devices table. 48-ports USW have > 1000 metrics, for example.

[bbrendon]

I agree awk/grep etc isn't cheapest. I just mentioned it as an option.

I think it would be a calculated item. I'm not sure I've used these before myself.

https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/calculated

[sadman]

I think it would be a calculated item. I'm not sure I've used these before myself.
https://www.zabbix.com/documentation...pes/calculated

I want to see expression of calculated item for opened list of Items, which can be unsupported and sometimes not exists.

I have a suggestion for u. I think, that can append handling of -i (or -f) and -o options for input and output files. Input files must contain a key list, values of which must be delivered to Zabbix. Output is file in zabbix_sender format. Your part of work is searching of method (but not only theoretica) for making input file. May be with Zabbix API or other tools.

P.S. ...file with keys in UniFi Miner, of course. With reference to object and may be site... may be CSV-like style.