Zabbix 4.2.5 with Elasticsearch 6.1.4 High utilization of History Syncer / LLD worker

[Zhankin]

My setup:

Zabbix Server + 9 Proxy
All devices SNMP
Backend Elasticsearch



High utilization of History Syncer Internal Processes, and Utilization of LLD worker





zabbix.conf

• name: ZBX_HISTORYSTORAGEURL
  value: http://es6-lb.elk-6:9200
• name: ZBX_HISTORYSTORAGETYPES
  value: "uint,dbl,str,log,text"
• name: ZBX_STARTDBSYNCERS
  value: "60"
• name: ZBX_CACHESIZE
  value: 4G
• name: ZBX_HISTORYCACHESIZE
  value: 2G
• name: ZBX_HISTORYINDEXCACHESIZE
  value: 2G
• name: ZBX_TRENDCACHESIZE
  value: 2G
• name: ZBX_VALUECACHESIZE
  value: 4G
• name: ZBX_HOUSEKEEPINGFREQUENCY
  value: "0"

I can't figure out how I can reduce the load, where to look for the problem

[kloczek]

I really do not understand why people want to use ES storage backend.
If someone is thinking that this can be faster because it serves some data out of memory just do the same with SQL backend by giving exactly the same memory for SQL engine.
Adding to the whole stack yet another component like ES makes only everything more complicated and sensitive on single error (HA few cascaded components decreases with sum of each components probability of the errors).

[rudolphk]

Zhankin Were you able to resolve this? I am having the same issue.

[rudolphk]

I've made some progress on my issue, so I'll post my updates here in case someone else stumbles upon this post.

1. Make sure your HistoryStorageURL parameter points to a properly configured load balancer or VIP. If you point this parameter to a single node in the ES cluster, then all of the HistorySyncer processes will connect to a single node in your cluster and hammer it.
2. The previous admin who installed Zabbix 4.0 set the StartDBSyncer to 32 (OP has his set to 60). This was FAR too high for a small 3 data node cluster. You could see from the zabbix server logs that I was getting a ton of HTTP 503 errors which indicated that ES was overloaded, and looking at _cat/thread_pool was showing a large number of search rejects. I set my DBSyncers to 6 and that gave me plenty of throughput
3. Zabbix does not support trends with Elasticsearch. I had three months of data and about 2.5 billion documents in ES. I reduced my retention down to 1 month (800M documents) and performance instantly improved. It appears that the HistorySyncer process issues a TON of queries when it starts up. Many of these query take the form of dt <= <some epoch>. If you have a lot of historical data, those queries are really going to bog you down. I've turned off the Housekeeper process since I'm using Elasticsearch Curator for pruning my indexes.

[Cynthiaa]

It will not work in case when "Show warning if Zabbix server is down" global configuration option is enabled and zabbix frontend (any menu) is kept opened or when for example pages on Monitoring menu perform background information refreshes myprepaidcenter