Syncing History Data takes more than 36 hours

[jvella]

We are running zabbix 1.8.2 to monitor 69 hosts and approx 4000 items. Performance is generally good, however, every now and again the number of items in the queue waiting to be processed gets to several thousand entries (ie the zabbix server virtually stops processing items). Looking in the Zabbix Server log file shows that the server is "Syncing History Data".

The first couple of times it happened I rebooted the server and it seemed to come good for a while. This time, I let the syncing run its course and it took more than 36 hours of "Syncing History Data"!

Obviously this is not how it should be. Our database is mysql and is only about 2GB in size.

Anyone with any clues as to why this happens?

[richlv]

what's your new values per second ?
monitor your iowait on the system. what is it normally and during the slowdown ?
monitor your sql queries per second. what is the average amount ?

[jvella]

The number of values per second is just under 30.

The Mysql queries per second spikes at the time of the slow down (see attached graph) and it seems to be a fairly regular time (ie approx 6 am when it occurs)

Can you tell me how to determine the iowait figures that you asked for?

thank.

[richlv]

in top output, you can see that as "wa" valuea.

you can also run "vmstat 1 2" - "wa" column value in the second line is the one you need

is the db on a virtual machine by any chance ?

[MrKen]

C'mon richlv, what about Zabbix

system.cpu.util[,iowait,avg1]

Put that in the CPU Utilization stacked graph with user, system, and idle time.

[richlv]

hehe, good point
i was mostly referring to quick debugging, but yeah - in general it is a good idea to monitor iowait to see hot it changes over time

[jvella]

Richlv,

No, the db is not in a virtual machine.
I have added an item recording iowait on the Zabbix Server and I will let you know the results.

Thanks.

[jvella]

Can anyone tell me what is sctually happening when the zabbix server log says "syncing history data"?

[richlv]

child nodes are sending history and trends data to master node

[jvella]

Why would it be doing this as I don't have a distributed Zabbix configuration and no zabbix proxies!

[jvella]

richlv,

Finally got to record the change in iowait.
Normal iowait is 1 - 2.
iowait during "history sync" is approx 30.

While this is happening, the zabbix dashboard indicates that zabbix is stopped (even though it is actually still running) and the item queue builds up.

[jvella]

These issues seem to occur at approx 6 am every day. Is there anything significant about this time? Does Zabbix schedule something for this time?
I have checked all of my cron jobs and there is nothing external that is scheduled for this time.