PID issues!

[moo]

Hi all,

I'm pretty sure the solution to this problem is going to be really simple (it's happened to me before but can't for the life of me remember what I did!)

Basically, logged onto the Zabbix FE this morning to find that the zabbix server wasn't running.

So hopped onto the box that it should be running from & found the zabbix-agent threads running fine but no zabbix-server.

So, attempted to start the service and got;

 * WARNING:  "zabbix-server" has already been started.

Ummmmmm, ok, so I try and stop it and get;

No process in pidfile `/var/run/zabbix/zabbix_server.pid' found running; none killed.

Argh! so anyway much thanks to Alexei on twitter - I was advised to delete the PID - however when I attempt to delete I get;

cannot remove `/var/run/zabbix/zabbix_agentd.pid': No such file or directory

Yup, the zabbix/ is empty!

So we have no PID, no processes, and the seemingly impossible task of restarting or starting the server process!

Couple other things of note - I would reboot this thing as an easy way out however it's a mission critical server & must remain up pretty much constantly!

Cheers in advance guys, and happy Friday!

[moo]

Ah, heres the most recent log also from /var/log/zabbix/zabbix_server.log

 25300:20110729:010227.553 One child process died (PID:25307). Exiting ...
 25300:20110729:010230.014 Syncing history data...
 25300:20110729:010231.772 Syncing history data...done.
 25300:20110729:010231.773 Syncing trends data...
 25300:20110729:010253.577 Syncing trends data...done.
 25300:20110729:010253.586 Zabbix Server stopped. Zabbix 1.8.2 (revision 11211).

[xibbaz]

Ensure zabbix_server is not running (ps -ef | grep zabbix_server) and if not, simply delete the pid file. This has more to do in how this was implemented in the /etc/init.d/zabbix-server startup file which is dependent on the distribution you are using.

[moo]

Hey, thanks for the reply xibbaz

As mentioned - there is no process thread for zabbix_server - there is also no PID so I can't delete this.

Also (sorry for not mentioning earlier!) this is a *very* customised Gentoo server.

This is all pretty odd tbh as the zabbix server has been up for at least 3 months or more & has randomly stopped with no warning and not much info as to why it's decided to take a break :S


Thx, Laura.

[moo]

Just a thought to anyone who might have more ideas - could the cause of this be a dodgy unsupported item returning blanks/errors to the server?

Still not got the server back up & running - will provide beer & chocolate to anyone that can help! :P

[moo]

Also I've just realised the last error when trying to delete the PID I've done the agent rather than server - however it's of no consequence really as the /var/run/zabbix/ dir is empty of any PID anyway :/

[moo]

SOLVED!

Thanks to my awesome collague - for future reference if anyones running a zab server from gentoo - it creates a link in /var/lib/init.d/started/ - if zab crashes then this link also has to be deleted before the service can be started back up again.

Not sure if this is the case for other distros, not had a problem with agents on any other box so

Cheers anyway guys