Proxy not syncing to Master (1.8.4 for both)

[mgoodman]

Ok, I've read dozens of different posts about this, but I still can't quite figure this out.

I have a simple setup of a master server and a single remote active proxy. About 75% of the hosts/items are monitored via the master, about 25% via the proxy.

Number of hosts (monitored/not monitored/templates) 436 307 / 66 / 63
Number of items (monitored/disabled/not supported) 2656 2311 / 38 / 307
Number of triggers (enabled/disabled)[problem/unknown/ok] 1153 958 / 195 [32 / 341 / 585]

I am using a combination of agents (win/lin), simple checks and ping checks (no SNMP, no IPMI, no active checks).

My item queue for ALL items monitored by the proxy was several days behind. I rebuilt the database (MySQL InnoDB) and events started flowing into the master. After 12 hours, it was already several hours behind. I've made several tweaks to the environment, restarted the services and MySQL on both the master and proxy, but can't get it to catch up. All checks (711 of them) are queued for more than 10 minutes (at least several hours).

Here are some relevant settings from the Master:
###
zabbix_server.conf
###
StartPollers=30
StartPollersUnreachable=2
StartTrappers=20
StartPingers=2
CacheSize=32M

###
my.cnf
###
innodb_file_per_table
max_connections = 400
default-storage-engine=innodb
innodb_log_file_size = 250M
innodb_flush_log_at_trx_commit = 2


And from the Proxy:
###
zabbix_proxy.conf
###
ProxyMode=0
ProxyLocalBuffer=0
ProxyOfflineBuffer=120
ConfigFrequency=900
DataSenderFrequency=1
StartPollers=30
StartPollersUnreachable=2
StartTrappers=20
StartPingers=2
CacheSize=32M
StartDBSyncers=16

###
my.cnf
###
innodb_file_per_table
max_connections = 300
default-storage-engine=innodb
innodb_log_file_size = 250M
innodb_flush_log_at_trx_commit = 2

Some other notes:
- Time is identical (same time zone also), using the same NTP source
- Both hosts are resolved properly (both added to /etc/hosts also, to eliminate DNS issues)
- Both hosts can access each other over 10050 and 10051
- Both proxy and master are using the same version of Zabbix (1.8.4) and MySQL (Ver 14.12 Distrib 5.0.77, for redhat-linux-gnu (x86_64) using readline 5.1)

I'm not seeing any abnormalities in the logs. Data is being collected in the proxy MySQL database in real-time, as I can see that in the database directly.

Any ideas on why this is lagging? Any way to catch it up without deleting proxy history? E.g. is there any way to force a sync of proxy data? I don't mind pausing monitoring on the master for a couple minutes while this occurs. I just don't want to have to rebuild the database again.

Thanks,
Michael

[mgoodman]

I thought for a bit that it could be the backups running on the database, so I disabled those. Cleared the database, got the queue up to normal, and after less than one day it is already delayed by about an hour. Any help would be greatly appreciated.

[mgoodman]

Ok, I think the problem is entirely with the proxy. When I stop the proxy and then restart it, I can see that events start flowing in, but then stop. I restart it again, then they start flowing in again, and then stop again. I can tell because I am watching a specific data point in "Latest Data" and I can see the time gradually catching up.

At this point, I can imagine a workaround is that I can restart the zabbix-proxy service every 15 minutes. But that's a really crappy fix.

Anyone know why the zabbix proxy stops sending data to the master database? How to tweak this? I don't think it's on the master side, as performance is fine, but also because restarting the proxy starts the data flowing in again...so it appears there is an issue on the proxy side...

[mgoodman]

Still can't figure this out...anyone have any ideas?

Some insight as to how the proxy sends data to the master would help. The zabbix proxy sends data to the master on port 10051, and then the master writes to its database? Is that right? It isn't directly to the master SQL database, is it?

[mgoodman]

Well, no responses...hopefully someone else can benefit from this though:

I ran tcpdump on the master to see if there were some anomalies and noticed there were a few ICMP admin prohibited packets being sent from the proxy. That is basically a graceful error message from the proxy saying that it has denied certain packets.

It turns out that there was a firewall blocking UDP 10051 (although it allowed TCP). So, it worked for a little bit, then broke. I'm not sure what exactly is happening that allows for *some* of the data to come in, and then get blocked later, then come in again (so basically the data flow appears to switch transmission protocols).

Maybe the developers can comment on this behavior?

Since resolving the firewall issues yesterday, everything seems to be fine. I'll write back in if it comes back or turns out to be something else.