zabbix and mysql cluster for write scaling

[rumpel1984]

I am doing initial exploration for zabbix monitoring 100000+ notes, I want to build it with scalability in mind.

My research suggests the main limitation is database IO. I want to find out if I were to setup a mysqlcluster with ndbcluster datatypes if zabbix will/can utiliize this.

Thanks!

[kloczek]

I am doing initial exploration for zabbix monitoring 100000+ notes, I want to build it with scalability in mind.

My research suggests the main limitation is database IO. I want to find out if I were to setup a mysqlcluster with ndbcluster datatypes if zabbix will/can utiliize this.

Thanks!

Try to use ZFS as backend FS.
COW semantics will allow you reduce write IOs by factor 3-4 in comparison to any standard Linux FSes.
If you will be doing monitoring on such scale most important is organize descent proxies infrastructure.
Number of proxies with frequency of transferring monitoring data will be you main factor determining number of physical IOs.

As long as you will have buffered data over proxies you can switch ZFS volume settings to sync=disabled. Why? Because if even whole system will be suddenly restarted within proxy buffer time window all historic monitoring data will be resynced when zabbix server will be up again.
Enable ZFS compression (at last lzjb) is next factor allowing significantly reduce number of physical IOs.

With so big number of hosts you may have at last 10^2 bigger number of items to monitor. In such case quite important will be to have enough memory to have enough big zabbix cashes.

[rumpel1984]

That helps, I have not heard that one before, Thanks!

[syndeysider]

+1 ZFS + PCIe SLC SSD's for History and Trends Data

[kloczek]

+1 ZFS + PCIe SLC SSD's for History and Trends Data

First: there is no at the moment any more SLC SSD chips. All latest SSDs are using MLC.
Second: if someone is going to use ZFS instead straight using such SSD as storage better is using it as ZIL and L2ARC devices. 1-2GB as ZIL and all remaining space as L2ARC should be OK.

If zabbix server is well tuned zabbix (enough big caches) zabbix server will generate to database backend mainly write IOs. If in the zpool are slow spindles + ZIL this will allow finish write IOs as quickly as possible )below latency of spindles).
L2ARC device in zpool will automatically collect MFU data.

Nevertheless in case using ZFS quite important is enough big size of ARC. Specially if someone is going to use biggest possible recordsize (1MB) with compression.
Zabbix automatically is trying to buffers last hour raw data in own caches to calculate every hour trends data. If size of zabbix cache is enough such operations will not generate additional selects queries to calculate trends.

Another part of the data which should be quickly accessible are data used on generate most of usually observed graphs.

In my zabbix installation I'm trying to have completely buffered in ARC current day whole history* tables partitions so my commendation is to have ARC size around daily written data in history* partitions.

Using biggest possible ZFS recordsize=1MB have to consequences:

• bigger compression ratio
• less ARC space is used to keep MFR/MRU blocks hashes (effectively ratio between ZFS data and metadata in ARC is bigger)