zabbix proxy huge data lag

[insider]

Hello,
We have distributed monitoring with Zabbix.
4 proxies, each monitoring approx. 400 hosts, send data to server. Server has 26GB RAM, 8CPU Intel Xeon and attached iscsi drive. Both proxies and server use MySQL database, server version is 5.5, proxies — 5.0. Main database is 3TB large, we keep all history for whole year.
Zabbix version 2.6 is installed on proxies and server.
Network is 1Gbps.

The problem is whether main Zabbix server is rebooted or database is stopped we experience continuous data lag from proxies. And this lag is growing and growing until we completely drop and recreate database on proxies. Nothing suspicious in messages and either zabbix_proxy.log or zabbix_server.log

Please help troubleshooting this occasion.
What can be done to find and destroy issue of this lag?

[tchjts1]

On your proxies, in zabbix_proxy.conf, what do you have this value set to?

### Option: ProxyOfflineBuffer
#       Proxy will keep data for N hours in case if no connectivity with Zabbix Server.
#       Older data will be lost.
#
# Mandatory: no
# Range: 1-720
# Default:
# [B]ProxyOfflineBuffer=1[/B]

At my previous implementation of Zabbix, I had 14 proxies and had this value set to 24, thinking it was a great way to keep data if the Zabbix App or DB server went down. What I found out was... that it takes Zabbix about an hour to process 2 hours worth of data held by a proxy.

So, if we were down for 24 hours, it was going to take 12 hours to process the data held by a proxy, before it even got to the data to where the Zabbix App or DB server came back up.

So now I leave it set to the default of 1.

By the way, at that implementation, we were using very high end, high horsepower servers.

[insider]

Whoa? It's 48! I will change it to see if it helps.
Here is almost full proxy config, exluding db details and pid/sock options.

ProxyOfflineBuffer=48
ConfigFrequency=60
DataSenderFrequency=1
StartPollers=64
StartIPMIPollers=2
StartPollersUnreachable=10
StartTrappers=10
StartPingers=4
StartDiscoverers=0
CacheSize=512M
HistoryTextCacheSize=128M
Timeout=5
ExternalScripts=/etc/zabbix/externalscripts
LogSlowQueries=1000

And server config in case it may be useful.

StartPollers=80
StartIPMIPollers=10
StartPollersUnreachable=80
StartTrappers=20
StartPingers=10
StartDiscoverers=0
DisableHousekeeping=1
CacheSize=512M
CacheUpdateFrequency=60
StartDBSyncers=8
HistoryCacheSize=256M
TrendCacheSize=256M
HistoryTextCacheSize=256M
AlertScriptsPath=/etc/zabbix/alertscripts
ExternalScripts=/etc/zabbix/externalscripts
ProxyConfigFrequency=120
ProxyDataFrequency=30

[insider]

solved

We've found a bottleneck. It was radi6. When Zabbix server starts receiving data from all proxies at a time disk utilization becomes ~100% and not going down until part of proxies is stopped.

Thanks for the help.