Zabbix server service restarting every 10 seconds?

[ESDadmin]

Hey everyone,

I've been struggling to find out why this is happening to me. I'm not necessarily new to linux, but it isn't my daily OS.

I've installed Zabbix 3.4 on CentOS 7 and it was working great for some time. I'd say about 4 hours into configuring hosts and whatnot, the server service crashed.

Long story short... when I run systemctl status zabbix-server.service I get this:

[root@nm01 ~]# systemctl status zabbix-server.service
● zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: resources) since Thu 2017-10-26 15:05:58 PDT; [B]9s ago[/B]
  Process: 20497 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
  Process: 24665 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 20491 (code=exited, status=1/FAILURE)

Oct 26 15:05:58 nm01 systemd[1]: Failed to start Zabbix Server.
Oct 26 15:05:58 nm01 systemd[1]: Unit zabbix-server.service entered failed state.
Oct 26 15:05:58 nm01 systemd[1]: zabbix-server.service failed.
[root@nm01 ~]# systemctl status zabbix-server.service
● zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
   Active: activating (start) since Thu 2017-10-26 15:06:08 PDT; [B]1ms ago[/B]
  Process: 20497 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
 Main PID: 20491 (code=exited, status=1/FAILURE);         : 24674 (zabbix_server)
   CGroup: /system.slice/zabbix-server.service
           └─24674 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf

Oct 26 15:06:08 nm01 systemd[1]: Starting Zabbix Server...
[root@nm01 ~]# systemctl status zabbix-server.service
● zabbix-server.service - Zabbix Server
   Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: resources) since Thu 2017-10-26 15:06:08 PDT; [B]416ms ago[/B]
  Process: 20497 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE)
  Process: 24674 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 20491 (code=exited, status=1/FAILURE)

Oct 26 15:06:08 nm01 systemd[1]: Failed to start Zabbix Server.
Oct 26 15:06:08 nm01 systemd[1]: Unit zabbix-server.service entered failed state.
Oct 26 15:06:08 nm01 systemd[1]: zabbix-server.service failed.

What I find odd is that every 10 seconds, the auto-restart counter resets itself. I'm not sure what is going on and I've been googling so much that I'm getting overwhelmed since none of the threads I'm finding are working for me..

Help!!

[kaspars.mednis]

Hi !

Look at /var/log/zabbix/zabbix_server.log, there you may see the reason why is server crashing...

Regards,
Kaspars

[ESDadmin]

Hi kaspars,

Sorry about that, I was kicking myself for not including it in the original post I checked out the log and saw 100's of repeating lines. They are:

zabbix_server [12648]: cannot open log: cannot create semaphore set: [28] No space left on device

I find this odd because if I see this on my machine:

[root@nm01 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  3.8G   47G   8% /
devtmpfs                 3.9G     0  3.9G   0% /dev
tmpfs                    3.9G     0  3.9G   0% /dev/shm
tmpfs                    3.9G   41M  3.8G   2% /run
tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1               1014M  246M  769M  25% /boot
/dev/mapper/centos-home   90G   38M   90G   1% /home
tmpfs                    782M   12K  782M   1% /run/user/42
tmpfs                    782M     0  782M   0% /run/user/0
tmpfs                    782M   32K  782M   1% /run/user/1000

How can I run out of space when I'm barely scratching the surface as it is?

[kaspars.mednis]

Hello !

is SELINUX ON ? it may be a SELINUX issue...

getenforce

If it is enforcing, you can temporary switch off SELINUX with

setenforce 0

for testing purposes, if your company policy on this system allows this

Regards,
Kaspars

[ESDadmin]

SELinux is on. (I thought I remember putting in rules for Zabbix..)

[root@nm01 ~]# getenforce
Enforcing

After running setenforce 0 , it's now permissive:

[root@nm01 ~]# getenforce
Permissive

I was still having trouble starting the server, so I took at look at any processes that were for zabbix server and I found one for the ICMP pinger. Once I force killed that process, the server service immediately started back up.

I rebooted the server and sure enough, zabbix server service did not want to start. I temporary switched off SELinux like before and now it's working.

Guess that means I just need to get some better rules in place for SELinux then huh? I did a few google searches and I'm finding mixed results on what to enter in.. Could you help me out?

[kaspars.mednis]

This solutions looks pretty clear and nice to me

http://yallalabs.com/linux/how-to-install-zabbix-3-4-server-on-centos-7-rhel-7/

Regards,
Kaspars

[ESDadmin]

Thank you very much for the link. I'll enter in these three SELinux commands and see where that gets me

Take care!