Zabbix and Elasticsearch housekeeper

[Rodney]

I´m studying Zabbix 3.4.10 which Elasticsearch as historical database and I´m dealing with the following problem:

What is the best solution to delete old data in the Elasticsearch repository?

As an example, I´d like to keep just 7 days of data in the log, dbl, uint and text indexes on the Elastichsearch.

After reading the Elasticsearch documentation (I´m new on it!), I found that the best way to delete ids of a index that has many ids is delete the index itself. It seems to drop a table on a SQL database.

But howto make this without loosing data?

It is possible to create "rollover index" in Elasticsearch, but when a new index is created, based on a date rule, for example, the alias point to the new index, that is empty.

The solution is your application make searches with the "*" charater at the end of index name, for example:



GET /dbl*/values ...



Any idea as Zabbix deals with this problem or a better way to clean old Zabbix data in Elasticsearch?



Thanks in advance!

[kloczek]

BTW ElasticSearch.
Someone have some has some numbers about speed when ES backend is used?
Personally I suppose that ES backend will be slower as SQL has better caching algorithms and new data in ES files IIRC needs to be written to some temporary files before compacting everything.
As usually trends data takes only few % of whole database with even 2-3 years of trends data using ES to store those data should be even more pointless.

[telder]

This is actually a very valid question since those of us that use Elasticsearch to store the historical data are doing so because we are operationlizing that data for reporting that Zabbix lacks as well as Machine learning, etc. Right now Zabbix just plops the data in Elasticsearch without any easy way to manage the data whereas they need to change over to time-based indices for users to have a valid way to manage their data, much in the same way that administrators create MySQL DB partitions to easily manage the amount of data in their databases.

[stephane.papin]

Hi,
You've to look for CURATOR extension from ES. Ut's a external tools to clean ES.index.
Best Regards.