Restart one instance on a Linux cluster

[LarryDeen]

I'm coming along well using Zabbix but trying to figure out how to monitor and restart a failed Linux service is not going well.
I have probably read and followed 20 different articles and most seem to have slightly different variations with nothing working in the end so I thought it's time to post and ask here.

Can someone please walk me through the process of doing this. I'll keep this post handy until I know it inside out I promise .

I have a cluster of web servers, all Centos 8 that serve web pages.
Server01 must run a custom service that does a tally to update some things in the DB.
If the service fails on server01, I'd like to have zabbix automatically restart it.

This is the first thing I very much need to get working and is mainly why I started looking at and using zabbix.

As a bonus and not so much a requirement
Is there a way that zabbix could be used in the following way as well.
If server01 goes away and zabbix cannot reach it, could it immediately start the same service on that second server.
It would have to keep track of which server is running the service so that it does not start another one on another server.

The server and the agents are 5.0.7.

[LarryDeen]

How do we know when our post is visible or not?

[Zdenek_OMNISENSUIT]

Hello.

At first about your web application etc. and HA solution.
You should not use the monitoring tool for keep your service in HA.
(yes, zabbix can start the service but your case is a little bit different with different needs)
About the linux server/cluster please try look at Pacemaker, Corosync, HA proxy and VIP.
It can help you a lot.
Then you can check with zabbix abailability of your linux hosts, services running on them etc.
For your web service you can keep eye on the VIP for this service for example and both processes on the servers etc.
So, for example - pacemaker will handle your application process to run and HA proxy will handle communication to your database.
Your process will run on server01 and if this process crash pcmkr will try start it but if not possible pcmkr will run this process on server02 etc.


-------------------------
omnisensuit.com

[LarryDeen]

Hi,

Well, as I mentioned, the second part is a bonus only. First I'd like to learn how to even get zabbix to monitor and restart a service. Something I've not been able to figure out even in days of reading so far.

[Zdenek_OMNISENSUIT]

Hello.

For restart linux service you must setup zabbix agent with enable remote commands.
Than setup zabbix (agent) user to be able restart process (setup rights) - for example:
visudo
# allows 'zabbix' user to restart apache without password.
zabbix ALL=NOPASSWD: /etc/init.d/apache restart
Than setup Action on the Trigger watching the process to restart it when down:
Actions -> Create new Action -> Select your Trigger (watching Apache process in this example) -> Operations -> Add -> Remote command -> Current host -> Commands:
sudo /etc/init.d/apache restart


-------------------------
omnisensuit.com

[LarryDeen]

Yes, I have this working from the command line on the zabbix server. In fact, same as the mysql question I have, in both cases, I can read values and run commands to remote hosts but never from the zabbix server GUI. I'll take another look at this as well as your reply to my mysql question. I must be missing something small.

[LarryDeen]

Hi, sorry for the long delay, didn't get an email notification about your reply. All good on the mysql, I just needed to switch to agent2 on the host and the same on the server. Now it's working fine.
Thanks for your input, it's appreciated.

[cyber]

Permissions? Do you execute commands from command line with same user running zabbix? Just randomly throwing things in the air here..