zabbix server not starting

[james.shaw]

Hi all,

I hope everyone is taking care and safe.

Ok, so I have a new build of Zabbix 4.4 and I have never encountered so many problems before with a new build Zabbix, I did quite a few installs with 3.* and they all worked seamlessly but selinux has and is screwing my latest build up.

Here is my build:
CentOS8
PHP7.2
MySQL 8
Apache

To start off, I cannot find the zabbix-server.pid file anywhere, its not in /run/zabbix nor is it in /var/run/zabbix, I tried changing the location it looks for the pid in the zabbix service file to /var/run/zabbix/zabbix-server.pid and still nothing, the agent is in both directories but not the server pid?

I used the following script to add selinux modules:

module zabbix_server_add 1.1;

require {
type zabbix_var_run_t;
type tmp_t;
type zabbix_t;
class sock_file { create unlink write };
class unix_stream_socket connectto;
class process setrlimit;
class capability dac_override;
}

#============= zabbix_t ============== #
!!!! This avc is allowed in the current policy (Edit by James; please bare in mind self: process need to go together, had to put a space due to emojis)
allow zabbix_t self: process setrlimit;

#!!!! This avc is allowed in the current policy
allow zabbix_t self:unix_stream_socket connectto;

#!!!! This avc is allowed in the current policy
allow zabbix_t tmp_t:sock_file { create unlink write };

#!!!! This avc is allowed in the current policy
allow zabbix_t zabbix_var_run_t:sock_file { create unlink write };

#!!!! This avc is allowed in the current policy
allow zabbix_t self:capability dac_override;

And also did the following:

checkmodule -M -m -o zabbix_server_add.mod zabbix_server_add.te
semodule_package -m zabbix_server_add.mod -o zabbix_server_add.pp
semodule -i zabbix_server_add.pp

setsebool -P httpd_can_network_connect 1
setsebool -P httpd_can_connect_zabbix 1
setsebool zabbix_can_network on

And still nothing:

[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# service zabbix-server start
Redirecting to /bin/systemctl start zabbix-server.service
^C
[root@localhost ~]# cd /run/zabbix/
[root@localhost zabbix]# ll
total 4
-rw-rw-r--. 1 zabbix zabbix 3 Mar 30 16:29 zabbix_agentd.pid
[root@localhost zabbix]# php -v

All the correct firewall rules are in place, on the server so I don't understand where I am going wrong now.

Any help would be really appreciated.

Kind Regards,

James

[james.shaw]

Just bumping to see if I can get assistance, to be honest, I am trying to build a zabbix monitoring platform to take over icinga as I said its much better, has better features, monitors more devices etc. Not really getting off to a good start here :'( haha

[tim.mooney]

Any time you add SELinux to the mix, you increase the difficulty. I'm not saying you shouldn't use SELinux, just that the problem isn't purely Zabbix, it's probably Zabbix with SELinux.

First, don't use "service" for starting stuff any more. It's been deprecated since RHEL/CentOS 7.0 released. Yes, it's still there and trying to be helpful, but it's just adding an additional layer.

So, try this instead:

# systemctl enable zabbix-server.service
# systemctl start zabbix-server.service
# systemctl status -l zabbix-server.service

Include the results here and someone will likely be able to help.

[james.shaw]

This is from journalctl:

Apr 02 09:06:39 localhost.localdomain systemd[1]: Starting Zabbix Server...
Apr 02 09:06:39 localhost.localdomain systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: No such file or directory
Apr 02 09:06:39 localhost.localdomain systemd[1]: Started Zabbix Server.
Apr 02 09:06:39 localhost.localdomain systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
Apr 02 09:06:39 localhost.localdomain systemd[1]: zabbix-server.service: Failed with result 'exit-code'.

There is a symlink between the 2 server pid destinations:

[root@localhost ~]# grep -i '^pidfile' /etc/zabbix/zabbix_server.conf /usr/lib/systemd/system/zabbix-server.service; ls -dl /var/run
/etc/zabbix/zabbix_server.conf:PidFile=/var/run/zabbix/zabbix_server.pid
/usr/lib/systemd/system/zabbix-server.service:PIDFile=/run/zabbix/zabbix_server.pid
lrwxrwxrwx. 1 root root 6 Mar 30 16:47 /var/run -> ../run

So, to me its like zabbix isn't creating the server pid?

PS SELinux is now disabled and still nothing.

Thank you in advance :-)

[tim.mooney]

I *think* that error message "Can't open PID file /run/zabbix/zabbix_server.pid (yet?)" is a bit of a red herring. It's something that I hope will be fixed in the future. I think it's a problem with the systemd service file.

However, I don't think that's what's causing your server to fail to start. systemd complained about the pid file, but it noticed that the main process exited with a return code of 1.

Are there any log entries in your zabbix_server.log?

[james.shaw]

I *think* that error message "Can't open PID file /run/zabbix/zabbix_server.pid (yet?)" is a bit of a red herring. It's something that I hope will be fixed in the future. I think it's a problem with the systemd service file.

However, I don't think that's what's causing your server to fail to start. systemd complained about the pid file, but it noticed that the main process exited with a return code of 1.

Are there any log entries in your zabbix_server.log?

Yeah, it's not the problem, thing is I forgot I changed the name of the db and didn't grant the correct privileges to the correct database, now thats done it starts up fine but I have now hit a new snag.

My Zabbix web server is running apache2, but when I try to navigate to server ip /zabbix I get a 404 Not Found nginx 1.12.2 error, which I don't understand why cause nginx isn't on the server and I follow Zabbix guide with installing it through Apache..... Why has Zabbix got soooooooooo confusing over time ahha.there nothing in apache error log, message log, php log or zabbix logs pointing to what may be causing this so I am stuck again

Glad I dont call my self an expert xD

[tim.mooney]

Can you post the exact error that your browser reports?

Also, what does

rpm -qa | egrep zabbix

output on your server?

[james.shaw]

Can you post the exact error that your browser reports?

Also, what does

rpm -qa | egrep zabbix

output on your server?

So the output from the command:

[root@zabbix zabbix]# rpm -qa | egrep zabbix
zabbix-release-4.4-1.el8.noarch
zabbix-server-mysql-4.4.7-1.el8.x86_64
zabbix-web-4.4.7-1.el8.noarch
zabbix-web-mysql-4.4.7-1.el8.noarch
zabbix-agent-4.4.7-1.el8.x86_64
zabbix-apache-conf-4.4.7-1.el8.noarch
[root@zabbix zabbix]#

And see please the attached image.

[tim.mooney]

The package list looks reasonable, and most importantly it doesn't include 'zabbix-nginx-conf'.

What does

netstat -tlnp | egrep ':80|:443'

output? The 'netstat' command must be run as root or via sudo, to get the last column "PID/Program name"

[james.shaw]

The package list looks reasonable, and most importantly it doesn't include 'zabbix-nginx-conf'.

What does

netstat -tlnp | egrep ':80|:443'

output? The 'netstat' command must be run as root or via sudo, to get the last column "PID/Program name"

Looks like its trying to listen on IPv6

[root@zabbix zabbix]# netstat -tlnp | egrep ':80|:443'
tcp6 0 0 :::80 :::* LISTEN 905/httpd

hmmmmmm

edit: Sorted that part now, but still same issue unfortunately.

[tim.mooney]

Ok, but is the process id that's listening on port 80 actually apache httpd, or is it nginx? Based on your error message, I would expect that it's most likely nginx that's actually listening.

# systemctl list-units | egrep 'http|nginx'

[james.shaw]

Ok, but is the process id that's listening on port 80 actually apache httpd, or is it nginx? Based on your error message, I would expect that it's most likely nginx that's actually listening.

# systemctl list-units | egrep 'http|nginx'

Nope, httpd bud.

[root@zabbix ~]# systemctl list-units | egrep 'http|nginx'
httpd.service loaded active running The Apache HTTP Server

Thats what I dont get the most. Lol.

[tim.mooney]

Ok. httpd tcp connections are short lived so it's somewhat tricky to catch them in netstat output. Luckily the default logging config for apache httpd on RHEL & CentOS logs accesses, so, do this:

On your Zabbix server:

tail -f /var/log/httpd/access_log

then try connect to the Zabbix web front end with a browser again.

Does your IP and browser details get logged in the access log when you do so?

Does anything get logged in /var/log/httpd/error_log as the same time?

[james.shaw]

Ok. httpd tcp connections are short lived so it's somewhat tricky to catch them in netstat output. Luckily the default logging config for apache httpd on RHEL & CentOS logs accesses, so, do this:

On your Zabbix server:

tail -f /var/log/httpd/access_log

then try connect to the Zabbix web front end with a browser again.

Does your IP and browser details get logged in the access log when you do so?

Does anything get logged in /var/log/httpd/error_log as the same time?

So it's like its not hitting the webserver at all. Nothing in the logs, IPs are all set up correctly as I can see traffic going through.

[tim.mooney]

Have you tried a different browser? Or from a completely different client?

Are you comfortable using tcpdump on the server, to watch for traffic on port 80?

Based on nothing getting logged in the access_log, despite Apache httpd running, it seems like there's a problem even getting to the server.