My zabbix server cannot get data

[insanelee]

About 2 days ago, my zabbix server suddenly received few data, as you can see from the attachment.
I checked the history data from mysql, it did show little data. All my zabbix server, agents, zabbix items are active and working.
What's the problem?

[insanelee]

When I ping the agent machine, it returns "time=0.000 ms".

[Davidus]

Can you explain the problem in bit more details?

[Antonio Carlos Ribeiro]

Can you explain the problem in bit more details?

I´m experiencing almost the same. Zabbix stops collecting from all my 36 windows agents a few hours after rebooting the server. And I´m having this issue for months, with 1.8.3, 1.8.4 and I tried everything. Last week I got out from a VMWare infrastructure to HyperV, reinstalled the whole server, migrating from Debian Lenny (5.0) to Squeeze (6.0) and I Dropped all my hosts configuration starting from scratch and recreating one by one. And nothing.

Yesterday I´ve upgrade it so now I have:

Zabbix Server 1.8.5
Zabbix Agents 1.8.2 (It could be that difference?!)
Debian 6 - kernel 2.6.32-5-686
MySQL 5.1.49-3
Apache/2.2.16
Running under a HyperV VM
Dual-Core AMD Opteron(tm) Processor 8222 stepping 03

I´ve rebooted the server yesterday, around 8h30pm, after upgrading Zabbix and, as you can see, today at 11h30am it stopped collecting data from all my servers.



But now, after restarting just Zabbix Server, I got in the lastest values of processes of the same host:

2011.May.05 16:48:00 84
2011.May.05 16:47:29 84
2011.May.05 16:46:17 84
2011.May.05 16:45:52 84
2011.May.05 16:45:05 84
2011.May.05 16:44:15 84 --> Zabbix was restarted here!
2011.May.05 11:29:19 80
2011.May.05 11:25:51 80
2011.May.05 11:22:24 82
2011.May.05 11:18:31 83
2011.May.05 11:15:11 81
2011.May.05 11:12:08 81
2011.May.05 11:08:34 82

But... Zabbix Server keeps collecting from my switches, via SNMP.

Any thoughts?

Thanks is advance.

Antonio Carlos

[Davidus]

Can you post at least one log of zabbix agent, and also zabbix server log file
usually they contain information we have to pay attention on

[Antonio Carlos Ribeiro]

Davidus,

Here are the logs ZabbixLogs.zip.

4 Files, 2 from server and agent before and 2 after restarting zabbix server.

Look for the host ALf100 (or AL100 or 172.20.11.63), it stopped around 4h04pm.

To "bypass" the problem, my server has cron job to restart zabbix server every 2 hours...

Thanks a lot.

[Antonio Carlos Ribeiro]

I´m using a "workaround" and it´s working for now: stopping and starting the zabbix_server process every hour. Here´s my script (/etc/scripts/restart_zabbix_server.sh):

############################
PROCESS_ID=`ps -e | grep zabbix_server | awk -F" " '{print $1}' 2> /dev/null`

for ID in $PROCESS_ID
do
kill -9 $ID

if [ $? -ne 0 ]
then
echo "Failed to Kill Process with PID = [ $ID ]"
fi
done

#restarting Zabbix

/usr/local/sbin/zabbix_server
############################

And in the crontab:

15 * * * * /etc/scripts/restart_zabbix_server.sh

############################

It´s working for know but it´s a really bad solution...

[Antonio Carlos Ribeiro]

Zabbix stops collecting info from hosts

Bump!

Guys, I´m kinda throwing in the towel here...

The workaround script above works, but it doesn´t.
I cannot predict when it will stop collecting, so I´m restarting the server several times a day. But...

Restarting every hour sometimes gives me a big hole in all graphics.
Restarting every 5 minutes gives me a swiss cheese in all graphics and make it very difficult to look at.

Zabbix works fine here, but I would like to not have to restart the thing every 5 minutes, because, due the frequency, sometimes after a restart it gives me no info about a failing hardware.

Any thoughts on this?

Thanks!

Antonio Carlos

[richlv]

which version of zabbix server do you have now ? are you using any external checks ?

[Antonio Carlos Ribeiro]

which version of zabbix server do you have now ? are you using any external checks ?

Just updated to 1.8.8. Same behaviour.

[richlv]

if you grep server logfile for that hostname, do you get any errors logged ?