zabbix_sender triggered by user parameter error

[rwa]

I have a user parameter that triggers a shell script on the monitored machine which builds a file of multiple mysql server data and then sends it via zabbix_sender to the server. The user parameter then returns the current result of show processlist to the item that called it.

So I have a Zabbix agent item which is a single user parameter, then roughly a dozen Zabbix trappers that receive the results from zabbix_sender. This has worked well for the first eight servers I set up. All of the monitored machines are OS X and all agents and server are version 1.6. Every server is set up identically. The Zabbix server is Ubuntu.

The two most recent monitored machines have an odd error. Initially they will work fine, and they will enter into the zabbix_agentd log:

zabbix_sender [61090]: DEBUG: Answer [{
        "response":"success",
        "info":"Processed 14 Failed 0 Total 14 Seconds spent 0.042351"}]

During this time the data is received by the server. The Zabbix agent item is checked every fifteen minuts (causing the script to run each time), and after an hour or so I begin to see this in the logs:

zabbix_sender [64637]: DEBUG: Send value error: Cannot connect to [zabbix.xxx.lan:10051] [Permission denied]

And none of the trappers receive data. Nothing has changed on host or monitored machine, nothing restarted. At this point the script will see this output from the command it runs:

Sending failed. Use option -vv for more detailed output.

and that's the case even though the zabbix_sender is called with the -vv flag. At this point I can manually run (as the zabbix user) from the monitored machine the exact zabbix_sender command that the script runs and it will work fine. If I use zabbix_get from the server to fetch the user parameter (and thus trigger the script) it fails.

Restarting the agent fixes this, but only for an hour. And again, it's only on two machines, the other eight have had no problems.

Any ideas? Thanks a lot.

[rwa]

This is odd, but it turns out commands called by zabbix_agentd were losing their ability to do dns lookups after several hours.

I added a test to the script here to ping google. After several hours it would not be able to find the host, though everything worked fine on that machine from the command line. Changing the zabbix_sender connection to use an IP fixed things.

Also, this only happened on OS X 10.5 machines. Our Tiger machines don't show this behavior.

I don't think it's purely an OS thing as ping, traceroute and zabbix_sender work fine from the command line...the issue is when they are called by zabbix_agentd.