Evolution of Zabbix to ElasticSearch or opentsdb like storage ?

[mcarbonneaux]

is juste an idea of evolution of zabbix... a sugestion...

now they are solution on large storage for serie of information that are more performante than sql database, and they are also requestable like database but in general with rest api.

http://opentsdb.net/

https://www.elastic.co/fr/products/elasticsearch

while be interesting that Zabbix evolve to have the possibility to store trend and history to that type of storage to extra scale.

this type solution are super performant with big volume of data, and can scale with big data solution like Hadoop and hbase if the original storage are not suffisant and scale as the Hadoop data cluster scale...


and all the reste of information rest in sql database only trend and history are to be in this storage.

or can be selective by item/host/template/group... in that way you can select the storage in function of as to many information or not...

[kloczek]

Main bottleneck of zabbix DB backend i latency of SQL queries.
This latency is caused by interacting with slow storage.
Ergo: if you will put the same amount of RAM as cache area speed of SQL DB backend and something like elastic search will be almost the same.
Problem with elastic search is that it cannot provide the same level of functionality as typical SQL engine.

So .. sorry kid by you are totally wrong.
Try to have look on https://kloczek.wordpress.com/2016/0...ql-db-backend/

[jan.garaj]

This one is not new idea and it was already discussed on LI - http://s9.postimg.org/npc8dz0m5/influxdb_zabbix.png

[kloczek]

This one is not new idea and it was already discussed on LI - http://s9.postimg.org/npc8dz0m5/influxdb_zabbix.png

And in meantime nothing changed
High rate of DB write operation (inserts and/or updates) can be handed only by caching enough data in memory because every update or insert needs informations about where b-tree or other structures needs to be updated. This is not obvious that DB writes generates on storage layer reads IOs as well
Such fact has nothing to do with type of DB engine.

[Colttt]

This one is not new idea and it was already discussed on LI - http://s9.postimg.org/npc8dz0m5/influxdb_zabbix.png

sad to see that this will be discussed on LI and not here in the forum

[jan.garaj]

FYI: Current comparison of time series databases https://docs.google.com/spreadsheets...HxM/edit#gid=0
Blog post: https://blog.dataloop.io/top10-open-...ries-databases

[kloczek]

FYI: Current comparison of time series databases https://docs.google.com/spreadsheets...HxM/edit#gid=0
Blog post: https://blog.dataloop.io/top10-open-...ries-databases

"The way I’d like to see benchmarking work is that you get to choose the highest powered server you can possibly lay your hands on. Then set up a reasonable real world test, like firing in 6000 metrics per agent over 500 agents at 1 second granularity (3 million metrics per second) and seeing if queries still work. Benchmarking reads is even more subjective so I wussied out and just went for fast, moderate and slow."

1) On MySQL to have such effect (very high write speed with very high read speed) is matter of using proper transaction isolation (READ-COMMITTED which is nit default).

2) Have so high NVPS like 3mln but completely without any triggers is not a big deal. Problems begins with growing ratio of number of metrics which needs to be evaluated against triggers and after ths triggers changes needs to be evaluated against actions. Here is waaaay bigger obstacle. Every change in trigger state generates update query which is problematic especially as long as beneath DB engine is not FS with COW semantics like ZFS. I don't know any DB engine which can handle well updates in case horizontally scaled engines. Ergo: single node DB engine is only solution.

[Colttt]

as i mentioned here use another DB solve not the problem.. zabbix must reduced uselessdata, this has the effect db-size is smaller -> faster queries
next step drop some DB-support and use more DB specific features

[Ctex]

I'll try set it up with MariaDB and TokuDB engine, which also has a fsync time window you can adjust, on another application i have hundreds of inserts / sec without making a dent in performance, and thats for a table 10s millions of rows with 7 indexes.
That is of course if Zabbix doesn't depend on InnoDB specific features.