Problem with Zabbix History Write Cache

[JulioZanette]

Hello all from Community

About a mont ago, due to company decision, we moved our zabbix server enviroment to Amazon AWS. Since that we are experiencing a problem with History Write Cache value dropping on irregular times.


------------------------------------------------------
Our scenraio is :

Zabbix Server AWS Site
Zabbix Server: Amazon Linux AMI - m3.medium
Zabbix FrontEnd: Amazon Linux AMI - m3.medium
ZabbixDB: MySQL 5.6.27 - db.m3.medium

And 2 Proxyes in diferent locations
Zabbix Proxy 1: CentOS 6.4 - 2 CPU´s - 6GB Ram and MySQL
Zabbix Proxy 2: CentOS 6.8 - 4 CPU´s - 4GB Ram and MySQL

Zabbix Server Conf Parameters is set to:

StartTimers=1
StartPollers=60
StartTrappers=5
StartPingers=20
StartEscalators=5
StartDBSyncers=10
SenderFrequency=60
StartJavaPollers=20
StartDiscoverers=10
StartHTTPPollers=10
StartPollersUnreachable=20
CacheUpdateFrequency=300
CacheSize=256M
TrendCacheSize=256M
ValueCacheSize=256M
HistoryCacheSize=256M
HistoryIndexCacheSize=256M
TrapperTimeout=300
UnavailableDelay=30
UnreachableDelay=30
UnreachablePeriod=60
LogSlowQueries=6000
HousekeepingFrequency=8
MaxHousekeeperDelete=1000000

#Status of Zabbix

Number of hosts (enabled/disabled/templates) 335 251 / 7 / 77
Number of items (enabled/disabled/not supported) 76328 13086 / 63242 / 0
Number of triggers (enabled/disabled [problem/ok]) 5908 3957 / 1951 [5 / 3952]
Number of users (online) 30 9
Required server performance, new values per second 152.18

#Last Mont Graphs


This Graph, show what is happening to the history write cache. As i could remember before we moved to AWS, this cache was alwys above 99%.
Every time i do a restart to Zabbix Server is flushs the cache and the then the cache start to drop again.





The other graphs shows all good to me, as we have a avg of 354+Kvps and the other proccesses is low from 10%

------------------------------------------------------

Since the problem started, i have read tons of treads and tutorials in how to tune Zabbix and other related infos. Like: thread=22941 and thread=45128

Also, i have adjusted Zabbix parameters trying to solve the problem. Like increasing the HistoryCacheSize and other cache values to 512M or 1024M. Pollers and other processes to a variety of combinations. I also asked our DBA to tune some parameters in SQL. But till now, i have not found a solution to the problem or what is causing it.

Since Zabbix server only receive data from Proxyes, the log is pretty clear, the proxyes logs show noting abnormal.

I am realy out of options in how to solve this.

So this is why i started this tread. Does any buddy could give it a try?

[sperr0w]

Check your syncers processes utilization.
HistoryWriteCache grows if syncers not push data from cache to database.

StartDBSyncers=10 it is too high for your installation. StartDBSyncers=4 by default it is ok.

Also in my opinion you dont need to change

SenderFrequency=60
and
CacheUpdateFrequency=300

from default falues without the real needs

[kloczek]

Usually when free Zabbix History Write Cache drops significantly below 100% it means that it is problem with write speed to DB backend.

[sperr0w]

Yes, DB is first you need to check. if DB is OK,
check DB zombie sessions from Zabbix
If all in DB is ok, check syncers in zabbix,
It it VMWare, check CPU ready on Zabbix Server
Also check CPU iowait.

[JulioZanette]

Hello Kloczek and Sperr0w (also angelhc if he find this out)

Thanks to your advice i have found that my problem is right related to the DB side.

My solution was (sound of drums).... Optimize (tune) the zabbix database on AWS RDS.

I gave a try to this nice tutorial: https://www.percona.com/blog/2014/11...-mysql-zabbix/ (thanks to Mr. Mikalauskas).

The parameters i have changed (what helped me a litle) was:

innodb_flush_log_at_trx_commit = 0
innodb_support_xa = 0

And the the real magic happened when i disabled the:

sync_binlog=0 (as i dont realy need this feature)

By the time i am happily writing this post my zabbix cache is nice like this:



Hope it continue just like it is and this post could help someone else in trouble.

Best Regards all

[JulioZanette]

Ohhhh, i forgot to say that what is causing my problem was the "heavily" write I/Ops i have on my RDS instance!

Maybe increasing the I/Ops could solve the problem... maybe!.