Proxies Not keeping up

[cozancin]

We are running a Zabbix installation with 320 hosts, 29000 items, and a Required Server Performance of 192.8. We have 4 proxies running.

All has been running well until lately. We have started noticing that the data from the proxies will start falling behind. Initially we thought this was a proxy issue. Recently we discovered that by restarting the Zabbix Server, temporarily fixed this lag for a few hours. Eventually, all proxy data begin to lag behind.

Has anyone else seen this issue before?

[heaje]

We've had a somewhat related problem in the past where Zabbix proxies were slow to send data to the server. However, in those cases we had to increase the number of pollers on each proxy and then restart those. It may or may not be related to your issue, but I thought I'd post it just in case it helps at all.

[LenR]

Do you have the zabbix server performance template installed? Check the Housekeeping busy values, do they go up when you server gets behind? If any of the various processes are too busy, some of them can have more processes set in zabbix_server.conf. Check the update interval on the items, many of the defaults are too frequent.

Check the queue, are all proxies falling behind or just some?

You might have general performance issues, check the server load on both proxy and master, use something like iperf to check net bandwidth between the two. Tune you database and it's hardware. If mysql, use file per table if possible.

We are 500 hosts, 75000 items, 609 nvps, 2 proxies, 100 & 61 hosts on them, rest monitored by the main Z server. Proxies are VM's, main is a Dell 2950, db is local RAID 5. (It's what I had....) It is "adequate".

[culrich]

I'm running into the same issue.

I've got 3 proxies all doing SNMP. Each with roughly 220vps.

All are running FreeBSD 9.0 on xen using 2.0.3 and sqlite local databases. 4 CPUs, 4gb of ram, SAN backed storage (and in one case the sqlite database is on a ramdisk)


The server and web ui are running on another VM (freebsd 9.0, 4gb ram, 4 CPUs).


The database is postgresql 9.1.6 running on a monster server with 48gb of ram and the storage is a fusionIO card. The database server doesn't seem very heavily loaded judging by iostat or top.


The proxies seem to queue up the data then send 10-15 minutes worth of data in chunks. Sometimes one or two are behind, sometimes none are behind, sometimes all three are behind.

iperf doesn't indicate any problems with the network.

server:
StartPollers=5
StartPollersUnreachable=5
StartTrappers=40
StartPingers=5
StartDiscoverers=5
StartHTTPPollers=5
CacheSize=256M
StartDBSyncers=30
HistoryCacheSize=128M
TrendCacheSize=16M
HistoryTextCacheSize=32M
FpingLocation=/usr/local/sbin/fping
HousekeepingFrequency=4

Proxy:
ProxyMode=0
Server=10.22.65.33
HostnameItem=system.hostname
LogFile=/tmp/zabbix_proxy.log
LogFileSize=5
DebugLevel=3
DBName=/tmp/zab/zabbix.db
ProxyLocalBuffer=0
ProxyOfflineBuffer=720
HeartbeatFrequency=60
ConfigFrequency=360
DataSenderFrequency=1
StartPollers=60
StartTrappers=60
StartPingers=60
StartDiscoverers=30
JavaGateway=127.0.0.1
JavaGatewayPort=10052
CacheSize=800M
HistoryCacheSize=80M
HistoryTextCacheSize=160M
Timeout=30
TrapperTimeout=300
FpingLocation=/usr/local/sbin/fping
StartDBSyncers=8


The queue backlog looks like

http://www.freeimagehosting.net/u5638

Any suggestions?

[bashman]

Take a look at https://support.zabbix.com/browse/ZBX-6249