Zabbix Proxy does not send zabbix trapper data to Server

[torre76]

Hello,
I have a strange problem with Zabbix Proxy and Zabbix Trapper.

My environment is:

- Machine A, Zabbix Server 1.6.9, DB Backend Postgres 8.2 with 50 host controlled
- Machine B, Zabbix Proxy 1.6.9, DB Backend Postgres 8.2, with about 30 host controlled

A lot of our checks are based on Zabbix Trapper, custom scripts on various hosts invoke zabbix_sender with the appropriate parameters.
And everything worked fine on a single server installation but, since our farm is growing, we decided to share the server worload using a proxy.
While I was migrating the host from server to the new proxy (i.e. changing the server configuration on the zabbix_agent and reconfiguring the host on server), everything worked like a charm until I loaded into the proxy a small number of hosts.
When I reached about 25 hosts on the proxy, suddently all the zabbix trappers (which was working perfectly) were not sent to the zabbix server (and thus a lot of trigger raised up).

I tried to check what was happening, turning the DebugLevel to 4 on the server and the proxy. What I found is:

- Proxy received and handled correctly the trap
- Proxy saved into proxy_history table the alert
- The data sent to the server, thru the history trapper, does not contain the trap data sent to the proxy.

The only things I changed from the standard Proxy configuration is the DataSend time (1) and the number of thread (about 50 pollers, 50 trappers) while into the server i raised the trapper thread to 25.
What I don't understand is why a moment before everything worked fine while now all the traps are missing.

If I revert the configuration (i. e. the hosts are connected directly to the server) everything works fine.
I tried to load only 20 servers to the proxy, but this situation does not change.

Any suggestion to workaround this situation?
Thank you in advance.

[torre76]

Status update

Today I found that all the test trap I send to Zabbix Proxy was delivered to the server with about 45minutes - 1h of delay.
All the traps are registered into the history of Zabbix Server Now.

This morning I moved again some server to test the configuration and everything works like a charm (i.e. traps are sent from proxy to server instantly).

In my case is not acceptable to have such a delay on trapper deliveries, because I have SLA on some servers that are very restrictive, and I cannot use proxy if this situation persists.

Any idea on that strange situation, what could cause this delay in delivering messages?

[Calypso]

I notice the same when I turn on eventlog monitoring of Windows servers (>300) on my proxy. Eventlog monitoring is also a "active" check, and it appears that active checks are handled by the trapper.

Of those 300 servers I use lots of passive checks. Works like a charm, and the proxy is sending the values to the server fast.

Turning on quite a number of active checks (like I said, eventlog monitoring in my case) cause this delay. Looking into it I noticed that apparantly the proxy sends his data to the zabbix server through trappers also. It looks like there are so many trappers "needed" by the incoming values of the proxy that they take all "space" (read: trappers) for sending the data from the proxy to the main server. This normale shouldn't be a big problem, but it also appears that after such a situation (e.g. an information burst in your eventlogs or something) when there aren't all trappers on the proxy used, the proxy doesn't regain access to the server.

What I did to test this looking how many trappers are opened on the proxy and the server (netstat -a | grep -i trapper | grep -i "IP from proxy" | wc -l) to get rough values. When I max the number of trappers (say 250) on the proxy (on the server too), I can see say 60 trappers from proxy to server. When I turn on eventlog monitoring the number of trappers on the proxy rise after a while (hundreds), and the number of trappers to the server decrease (to a few). The number of trappers on the proxy varies - when it comes to say 100, I still see only 1 or 2 trappers active on the server.

Only when I disable eventlog monitoring, everything turns back to normal...

Is there a logical explanation? I've tried several things (several values of trappers, serveral values of timeout for the trappers), but the situation remains the same.

[DSon]

Interesting..

I've just posted something regarding proxy data stopping in another thread, which was an indirect follow up to the issue I reported via the tracker some months ago.

The response I received back then was "this doesn't happen".

If my proxy stops sending data, and it does this regularly, I always notice in the zabbix server logs many "unable to resolve trigger" type messages.

After an hour of no new data from any of my proxies, the situation seems to get worse because then none of my triggers will evaluate (they are nearly all time sensitive e.g. "last 5 mins, value = xyz" etc).

The only way I can get things going again is by disabling all the triggers on the zabbix server, then rebooting all the proxies (sometimes having to reboot the server too, due to excessive timeouts on the database).

And before anyone suggests it is a DB issue, the server (and DB) configuration have been perfomance optimised on several occasions, as the DB size grew.

Anyway I would be interested to experiment with the items I have for event log monitoring (they are active checks, as you point out).

Next time I experience no data from my proxies, I'll try disabling all the templated items relating to event log monitoring, see if it resolves the problem..

I'll post back on here next time..

Danny.


PS. I have 3 proxies, each monitoring about 1000 items (not that much?)