Zabbix DB Syncers issue

[artem]

Hi. I have pretty big installation of zabbix 2.4 (5k hosts, about 1k NVPS). And there is an issue with performance: every 3 hours gaps emerge in graphs. It happens when DB Syncer process rises 100%. So I decided to increase syncers count and set StartDBSyncers=10 as opposed to default 4. But it seems zabbix starts only 4 instances anyway:

$ ps -ef | grep zabbix | grep syncer
zabbix   23444 23203  1 Sep11 ?        01:04:35 /usr/local/sbin/zabbix_server: configuration syncer [synced configuration in 47.880271 sec, idle 1000 sec]
zabbix   23801 23203  1 Sep11 ?        00:40:50 /usr/local/sbin/zabbix_server: history syncer #1 [synced 807 items in 1.667508 sec, idle 4 sec]
zabbix   23802 23203  1 Sep11 ?        00:44:35 /usr/local/sbin/zabbix_server: history syncer #2 [synced 647 items in 1.149725 sec, syncing history]
zabbix   23803 23203  1 Sep11 ?        00:53:42 /usr/local/sbin/zabbix_server: history syncer #3 [synced 1000 items in 2.235152 sec, idle 2 sec]
zabbix   23804 23203  1 Sep11 ?        00:43:09 /usr/local/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000002 sec, idle 5 sec]

Am I doing something wrong or this behavior is expected? Any ideas how to mitigate gaps in graphs?

[kloczek]

Hi. I have pretty big installation of zabbix 2.4 (5k hosts, about 1k NVPS).

In case of zabbix server number of hosts is not relevant. More important is NVPS and 1k it is lower band of mid size zabbix

And there is an issue with performance: every 3 hours gaps emerge in graphs. It happens when DB Syncer process rises 100%. So I decided to increase syncers count and set StartDBSyncers=10 as opposed to default 4. But it seems zabbix starts only 4 instances anyway:

$ ps -ef | grep zabbix | grep syncer
zabbix   23444 23203  1 Sep11 ?        01:04:35 /usr/local/sbin/zabbix_server: configuration syncer [synced configuration in 47.880271 sec, idle 1000 sec]
zabbix   23801 23203  1 Sep11 ?        00:40:50 /usr/local/sbin/zabbix_server: history syncer #1 [synced 807 items in 1.667508 sec, idle 4 sec]
zabbix   23802 23203  1 Sep11 ?        00:44:35 /usr/local/sbin/zabbix_server: history syncer #2 [synced 647 items in 1.149725 sec, syncing history]
zabbix   23803 23203  1 Sep11 ?        00:53:42 /usr/local/sbin/zabbix_server: history syncer #3 [synced 1000 items in 2.235152 sec, idle 2 sec]
zabbix   23804 23203  1 Sep11 ?        00:43:09 /usr/local/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000002 sec, idle 5 sec]

Am I doing something wrong or this behavior is expected? Any ideas how to mitigate gaps in graphs?

DB syncers are processes used on pushing data from from zabbix caches to DB. If time to time you have high usage of those processes it means that something is pushing a lot of data to zabbix. Most frequently something like this happens when zbx srv looses contact with proxies and on reconnect proxy or proxies are pushing all collected data to server.
Some people are increasing ProxyOfflineBuffer to more than default 1h not aware that if proxy will loose connectivity it resynces/pushes all buffered data and if time window will be to big it may cause overloading zabbix server on reconnect.
Cause of loosing connectivity in such cases it is another aspect (not related to zabbix) and if you are loosing connections between prxies and srv regularly you should diagnose this.

Questions:
- are you using proxies?
- if yes: are you using active or passive proxies?
- if not: are you using active or passive agents and items types?
- are you using default agents configuration(s)?

[artem]

Questions:
- are you using proxies?
- if yes: are you using active or passive proxies?
- if not: are you using active or passive agents and items types?
- are you using default agents configuration(s)?

I'm not using proxies. Vast majority of my agents are passive, and I am using lots of snmp items. All agents are almost default.

Isn't it possible to gain sufficient performance without proxies?

And still, why I can only see 4 history syncer processes, when 10 is configured?

Thanks.

[kloczek]

I'm not using proxies. Vast majority of my agents are passive, and I am using lots of snmp items. All agents are almost default.

FYI: SNMP monitoring is done always by server/proxy.

Isn't it possible to gain sufficient performance without proxies?

If your number of monitored hosts will be growing sooner or later you will be forced to use proxies.

And still, why I can only see 4 history syncer processes, when 10 is configured?

Are you really sure that you've added StartDBSyncers=10 in correct file? (IMO it is most possible cause)
And again: if only time to time you are see syncers saturation you should be more focused on investigate cause of this issue than increasing StartDBSyncers.
If you have bottleneck in you DB backend increase StartDBSyncers will not solve you issue. With high probability it will even cause that it will be worse than it is now if syncer cannot move monitoring data from history cache to DB. Try to check your server log do you see inserts interrupted by timeout or logged as slow queries.

[artem]

FYI: SNMP monitoring is done always by server/proxy.

Sure it is. I meant SNMP items in server.

And again: if only time to time you are see syncers saturation you should be more focused on investigate cause of this issue than increasing StartDBSyncers.
If you have bottleneck in you DB backend increase StartDBSyncers will not solve you issue. With high probability it will even cause that it will be worse than it is now if syncer cannot move monitoring data from history cache to DB. Try to check your server log do you see inserts interrupted by timeout or logged as slow queries.

I got it, thanks for advice.

[mushero]

Sounds more like DB issue where DB sync gets slower and backed up, then runs out of time/processes - remember your triggers also are evaluated in the DB Syncer so if they get behind then your alerts are also way behind - this was a huge issue for us before went to SSDs on the DBs.

So suggest you watch the DB situation at these peaks/problem times.

Steve