History Syncers 100% busy

[sperr0w]

Hello!

My inrfastructure:

Zabbix 3.0.3 on VM 16 CPU 16Gb RAM
two active proxy on VMs 4 CPU 8GB RAM
Zabbix Database on VM 12 CPU, 16Gb RAM

5000 hosts
6000NVPS


Last 2 weeks after 2 days downtime of my sever I get problem with Zabbix syncers.

Zabbix server config:

Pollers 600
Start DB cyncers 9

After restart all syncers is busy for 100% and I see queue of data waiting to send to server on proxy is increasing (data senders on proxy is also busy 100%).

I think that main problem is in server. I tried to change Start DBSyncers from 8 to 40 without any schanges. Wint StartDBSyncers=18, after 6-7 hours system come to stable state, but on sever graph DBsyncers busy was from 80 to 100%.

Pollers also busy almost 100%, but 80% of data come from proxy, and dont collects on main server.

On server syncers looks like this:

zabbix 19808 18116 5 11:02 ? 00:01:44 /usr/local/sbin/zabbix_server: history syncer #1 [synced 13394 items in 62.118624 sec, syncing history]
zabbix 19809 18116 6 11:02 ? 00:01:53 /usr/local/sbin/zabbix_server: history syncer #2 [synced 12452 items in 61.538994 sec, syncing history]
zabbix 19810 18116 5 11:02 ? 00:01:51 /usr/local/sbin/zabbix_server: history syncer #3 [synced 13428 items in 62.275340 sec, syncing history]
zabbix 19811 18116 6 11:02 ? 00:01:53 /usr/local/sbin/zabbix_server: history syncer #4 [synced 12397 items in 60.349846 sec, syncing history]
zabbix 19812 18116 5 11:02 ? 00:01:50 /usr/local/sbin/zabbix_server: history syncer #5 [synced 17081 items in 78.937150 sec, syncing history]
zabbix 19813 18116 5 11:02 ? 00:01:51 /usr/local/sbin/zabbix_server: history syncer #6 [synced 10970 items in 61.221723 sec, syncing history]
zabbix 19814 18116 5 11:02 ? 00:01:42 /usr/local/sbin/zabbix_server: history syncer #7 [synced 26974 items in 99.706424 sec, syncing history]
zabbix 19815 18116 6 11:02 ? 00:01:56 /usr/local/sbin/zabbix_server: history syncer #8 [synced 26751 items in 98.977624 sec, syncing history]
zabbix 19816 18116 5 11:02 ? 00:01:44 /usr/local/sbin/zabbix_server: history syncer #9 [synced 11761 items in 61.998643 sec, syncing history]


Where is my bottleneck and where to start investigation?

How is it possible to solve this issue with syncers busy and what syncers count are you use for 4-6 NVPS??

[Colttt]

first I've no experience with so much nvps, but take a look at this article: http://blog.zabbix.com/scalable-zabb...400-nvps/2615/

don't use so much DB-syncers 4-6 is ok (mostly). I guess you bottleneck is you Database-server, and 16GB-RAM for a database server isn't so much, do you also tune you my.cnf?

[sperr0w]

Hello!

I know, that one syncer it is about 100 nvps and for me 4-6 syncers must be enough, but in rela right now I tried 4 syncers on servers and it was very sadly.

history write cache was full and syncers busy was 100%.

I think about database as about bottleneck, I use Oracle DB acrially, not MySQL. But on database side our DBA say that there is no any problem, no problem with iowait, not CPU or RAM problems.

To get more performance hardware for database, I need to show, that problem it is actially here.

Here is some stats from DB.

root@xxxxx ~ # vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
5 1 392412 2070096 472524 791492 1 1 1522 1585 0 0 26 4 62 9 0
root@xxxxx ~ # free
total used free shared buffers cached
Mem: 16331988 14237984 2094004 0 472524 791492
-/+ buffers/cache: 12973968 3358020
Swap: 16777208 392412 16384796

[nick0909]

While not really Zabbix related, are you sure you need 16 CPUs on your Zabbix VM? Is the system really pushing all those CPUs all the time? Can you see the CPU load for the server? If you have too many CPUs assigned to your VMs they have to wait longer to get them all ready to execute, and they actually perform worse. If this is VMware, can you see your CPU wait time vs ready time? You might actually gain performance by lowering the CPUs assigned to those VMs if your ready time is high.

[Colttt]

Here is some stats from DB.

thats unreadable, please use code-tags!

and please post the output of on both on you db server and you zabbixserver
vmstat 1 10
iostat -x 1 10