Zabbix server and mysql DB performance issue...

[kanchan]

Hi,

We are monitoring our 2 DCs by using Zabbix 2.0.9 without proxy. Status of Zabbix server till now is as below :

MonIT server is running Yes
Number of hosts (monitored/not monitored/templates) 1454 (1128 / 53 / 273 )
Number of items (monitored/disabled/not supported) 123065 (89830 / 20073 / 13162)
Number of triggers (enabled/disabled)[problem/unknown/ok] 67781 (62208 / 5573) [1129 / 0 / 61079]
Number of users (online) 158 (6)
Required server performance, new values per second 940.21

Zabbix Server Configuration :

1. Server Type : VM
2. Allocated Memory : 25 GB
3. CPU : 8 vcpu
4. DB : mysql Ver 14.14 Distrib 5.6.19, for Linux (x86_64) using EditLine wrapper
5. OS : Red Hat Enterprise Linux Server release 6.4 (Santiago)


We are using Zabbix since last 6 years without any issue and last year we have upgraded Zabbix server from 1.8.3 to 2.0.9.

As business is growing, every day we are adding hosts in Zabbix and now we have frequently (once in a week) started experiencing below issues on Zabbix server :

1. Queries related to ids, event and history tables take long time. Please have a look on enclosed mtop screenshots.
2. If we check "netstat -plant | grep 10051 | grep -i establish" on Zabbix server, we does not get pid number for the respetcive client connection which is not a healthy status.
3. Whenever above 2 things occurs, Zabbix stops working and through error " Zabbix server is not running : the information displayed may not be current".
4. Zabbix alerter, history syncer and housekeeper processes are busy more than 75 %

Above situation happens atleats for 30 min once in a week and automatically get fix once DB clears all long pending queries.

Please note we have partitioned history_uint table and this table is getting served from SSD hard drive, rest of the tables are getting serve from SAS drive.

Please find enclosed Zabbix Server and mysql configuration for your reference.
It will be great if anyone suggest how to fix this DB and Zabbix server performance issue. Do we need to partition / archive event table as well ?

Please suggest..

Thanks,
Kanchan

[kanchan]

Dear All,

Please provide some suggestions...

[kanchan]

Hi...

Today again I faced the same issue....

This time number of established connection for zabbix_server was hardly 2 when I ran netstat command...once DB got stable I restarted zabbix server service couple of time then only connection started establishing....

It will be great if someone give me some clue ...

Thanks

[kanchan]

Hi,

Please provide some suggestions....

[kloczek]

Hi,

Please provide some suggestions....

1) go ASAP over upgrades to 2.2.x and 2.4.x.
2) stop using passive monitoring
3) move behind proxy/proxies monitoring of all hosts and use server only to communicate with proxies.
4) do you have any IO stats of the system where is running server DB backend?
5) upgrade DB backend to MySQL 5.6.