Error during zabbix server restart on CentOS8 .

[spithani]

hi - I need your help/advice over the error I am getting below on CentOS8 , I am using PostgreSQL , I am not understanding if its a zabbix error related or pgsql related. I am using the latest version of zabbix i.e. 5 and I have followed the instructions as given in the zabbix installation page .
My VM is running on Oracle VirtualBox . I googled through but there are bit old post related to it and nothing which can help & no zabbix server log is created . Below info from messages file.

Appreciate any pointers in this regard.

-- Unit zabbix-server.service has begun starting up.
May 14 04:42:20 moni zabbix_server[2151]: /usr/sbin/zabbix_server: /usr/pgsql-12/lib/libpq.so.5: no version information available (required by /usr/sbin/zabbix_server)
May 14 04:42:20 moni zabbix_server[2151]: zabbix_server [2151]: Cannot initialize IPC services: Cannot access path "/var/run/zabbix": [13] Permission denied.

File is available
[root@moni ~]# ls -l /usr/pgsql-12/lib/libpq.so.5
lrwxrwxrwx. 1 root root 13 Feb 17 05:56 /usr/pgsql-12/lib/libpq.so.5 -> libpq.so.5.12

[tim.mooney]

Those are two probably unrelated problems.

The first problem is that there's something wrong with either your Zabbix build (are you using official packages?) or your PostgreSQL install.

What does this command output:

rpm -qa | egrep -i 'zab|post'

The 2nd problem is that there seems to be a permission problem with the runtime directory zabbix needs.

When Zabbix server is not running, does the /var/run/zabbix directory exist? What are the permissions on it?

ls -ld /var/run/zabbix

[spithani]

Tim appreciate your response , here is the requested info . I have installed zabbix from the official packages

[root@moni ~]# rpm -qa | egrep -i 'zab|post'
zabbix-release-5.0-1.el8.noarch
zabbix-web-5.0.0-1.el8.noarch
zabbix-apache-conf-5.0.0-1.el8.noarch
postgresql12-server-12.2-2PGDG.rhel8.x86_64
zabbix-web-pgsql-5.0.0-1.el8.noarch
zabbix-server-pgsql-5.0.0-1.el8.x86_64
postgresql12-12.2-2PGDG.rhel8.x86_64
zabbix-agent-5.0.0-1.el8.x86_64
postgresql12-libs-12.2-2PGDG.rhel8.x86_64

I have identified the issue with /var/run/zabbix eariler , i.e. the directory wasn't existent earlier so I have created one manually ( that was after the I noticed the error not before )

[root@moni ~]# ls -ld /var/run/zabbix
drwxr-xr-x. 2 zabbix zabbix 60 May 14 03:23 /var/run/zabbix

[tim.mooney]

So in your original question, you said you were using Postgres for your database. Where did those postgres packages come from? I'm pretty sure those are not the CentOS 8 AppStream/Module packages for Postgres 12. That might be why you're getting the message from zabbix_server about "no version info available".

What output do you get from the following command:

systemctl status -l php-fpm.service

[spithani]

I have installed postgres from its website(below FYR) . Would installing postgres in this manner creates a prb ? If that is the case how do I need to install ?

https://www.postgresql.org/download/linux/redhat/

[root@moni ~]# systemctl status -l php-fpm.service
● php-fpm.service - The PHP FastCGI Process Manager
Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2020-05-14 03:23:04 EDT; 1 day 8h ago
Main PID: 518 (php-fpm)
Status: "Processes active: 0, idle: 10, Requests: 0, slow: 0, Traffic: 0req/sec"
Tasks: 11 (limit: 5046)
Memory: 42.0M
CGroup: /system.slice/php-fpm.service
├─518 php-fpm: master process (/etc/php-fpm.conf)
├─757 php-fpm: pool www
├─758 php-fpm: pool www
├─759 php-fpm: pool www
├─760 php-fpm: pool www
├─761 php-fpm: pool www
├─762 php-fpm: pool zabbix
├─763 php-fpm: pool zabbix
├─764 php-fpm: pool zabbix
├─765 php-fpm: pool zabbix
└─766 php-fpm: pool zabbix

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

[ripperSK]

Hi spithani . I have encountered this problem as well.

To fix my situation I made sure PidFile full path matched in systemd .service script and in .conf file:

[root@sldamgprx01p ~]# grep -i pidfile /usr/lib/systemd/system/zabbix-proxy-prod.service
PIDFile=/var/run/zabbix/zabbix_proxy-prod.pid

[root@sldamgprx01p ~]# grep -i pidfile /etc/zabbix/zabbix_proxy-prod.conf
PidFile=/var/run/zabbix/zabbix_proxy-prod.pid

If you need to make changes in your

/usr/lib/systemd/system/zabbix*service

Unit file don't forget to daemon-reload systemd:

[root@sldamgprx01p ~]# systemctl daemon-reload

To see what's going on in systemd you can issue:

[root@sldamgprx01p ~]# journalctl -xe -u zabbix-agent.service

hope it helps .

[MichaelDBA]

I had to modify the zabbix-server.service file since it had PIDFile=/run/zabbix/zabbix_server.pid and so I changed it to /var/run/zabbix/zabbix_server.pid.
I restarted the systemd daemon (systemctl daemon-reload), but it made no difference when I tried to restart zabbix-server:
systemctl start zabbix-server
-->
Oct 23 07:58:58 michaelv2 zabbix_server[3153]: /usr/sbin/zabbix_server: /usr/pgsql-13/lib/libpq.so.5: no version information available (required by /usr/sbin/zabbix_server)
Oct 23 07:58:58 michaelv2 zabbix_server[3153]: zabbix_server [3153]: Cannot initialize IPC services: Cannot access path "/var/run/zabbix": [13] Permission denied.

[max.ch.88]

May 14 04:42:20 moni zabbix_server[2151]: zabbix_server [2151]: Cannot initialize IPC services: Cannot access path "/var/run/zabbix": [13] Permission denied.

Hi! What state is SELinux in? The getenforce command can show this.

[MichaelDBA]

As noted above, I also have this same problem. I can get around the problem by putting 777 permissions on /var/run/zabbix, but not 755.

I am not using the zabbix proxy service, and I followed these instructions verbatim: https://www.zabbix.com/download?zabb...esql&ws=apache

I am using an existing PostgreSQL installation for PostgreSQL version 13 on CentOS 8.

I had to modify the zabbix-server.service file since it had PIDFile=/run/zabbix/zabbix_server.pid and so I changed it to /var/run/zabbix/zabbix_server.pid.
I restarted the systemd daemon (systemctl daemon-reload), but it made no difference when I tried to restart zabbix-server:
systemctl start zabbix-server
-->
Oct 23 07:58:58 michaelv2 zabbix_server[3153]: /usr/sbin/zabbix_server: /usr/pgsql-13/lib/libpq.so.5: no version information available (required by /usr/sbin/zabbix_server)
Oct 23 07:58:58 michaelv2 zabbix_server[3153]: zabbix_server [3153]: Cannot initialize IPC services: Cannot access path "/var/run/zabbix": [13] Permission denied.

The other services started OK:
# ps -ef | grep 'zabbix\|httpd\|php'
-->
root 1705 1 0 07:20 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
root 1706 1 0 07:20 ? 00:00:00 php-fpm: master process (/etc/php-fpm.conf)
apache 1707 1706 0 07:21 ? 00:00:00 php-fpm: pool www
apache 1708 1706 0 07:21 ? 00:00:00 php-fpm: pool www
apache 1709 1706 0 07:21 ? 00:00:00 php-fpm: pool www
apache 1710 1706 0 07:21 ? 00:00:00 php-fpm: pool www
apache 1711 1706 0 07:21 ? 00:00:00 php-fpm: pool www
apache 1712 1706 0 07:21 ? 00:00:00 php-fpm: pool zabbix
apache 1713 1706 0 07:21 ? 00:00:00 php-fpm: pool zabbix
apache 1714 1706 0 07:21 ? 00:00:00 php-fpm: pool zabbix
apache 1715 1706 0 07:21 ? 00:00:00 php-fpm: pool zabbix
apache 1716 1706 0 07:21 ? 00:00:00 php-fpm: pool zabbix
apache 1717 1705 0 07:21 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 1718 1705 0 07:21 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 1719 1705 0 07:21 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 1720 1705 0 07:21 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
zabbix 1943 1 0 07:21 ? 00:00:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix 1944 1943 0 07:21 ? 00:00:00 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
zabbix 1945 1943 0 07:21 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
zabbix 1946 1943 0 07:21 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
zabbix 1947 1943 0 07:21 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
zabbix 1948 1943 0 07:21 ? 00:00:00 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

[tim.mooney]

I had to modify the zabbix-server.service file since it had PIDFile=/run/zabbix/zabbix_server.pid and so I changed it to /var/run/zabbix/zabbix_server.pid.
I restarted the systemd daemon (systemctl daemon-reload), but it made no difference when I tried to restart zabbix-server:
systemctl start zabbix-server

I'm not trying to pick on you Michael, so please don't take this the wrong way, but I try stomp out incorrect information when I see it on the forums, so it isn't picked up by less-experienced users. With that in mind:

At least on current RHEL/CentOS/ScientificLinux/etc., there is no reason to change /run to /var/run in zabbix-server.service because they are the same place:

$ ls -al /var/run
lrwxrwxrwx. 1 root root 6 Mar 26 2020 /var/run -> ../run/
$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 (Ootpa)

Since /var/run is a symlink to /run, /var/run/zabbix is exactly the same directory as /run/zabbix.

Next, you almost never want to directly modify the systemd service file for a service. systemd provides ways to override and augment the settings from the stock service files, without modifying the service file. If you modify the service file directly, then every time you update the package, your changes are lost. Do some web searches for things like "systemd override" for more info. For example:

https://unix.stackexchange.com/quest...-file-settings


With that out of the way, I would say that the original post in this thread has nothing to do with the problem that you are seeing.

Here's what I would ask you to do, to try debug what's going on. This assumes that it's OK for you to reboot your Zabbix server.

1. disable 'zabbix-server' (just during testing) so that it doesn't attempt to automatically start after reboot: systemctl disable zabbix-server.service
2. reboot your Zabbix server.
3. Once your Zabbix server comes back, with the zabbix_server still shut down, run this:

   ls -al /var/run/zabbix ; sudo -u zabbix touch /var/run/zabbix/test-file; sudo -u zabbix rm -f /var/run/zabbix/test-file

4. What output do you get from the commands in Step 3? Does /var/run/zabbix exist after reboot? It should, because it should be getting automatically recreated at each reboot by systemd, by virtue of the config file in /etc/tmpfiles.d/ . See the man pages for tmpfiles.d(5) and systemd-tmpfiles(5) for more info.
5. Were you able to create the "test-file" in /var/run/zabbix when running "touch" as the zabbix user? If not, what message did you get? Is SELinux enabled on your Zabbix server?
6. Assuming everything works in step 3, if you then enable and start zabbix-server.service, does it start up correctly, or does it still generate a permission denied error?

Knowing what happens in each of these steps will hopefully point us in a direction to determine what's really going on.

[MichaelDBA]

Hi Tim. First of all thank you for taking the extra time to document all the stuff you put here. I am impressed by your response. I missed the symlink thing, thanks for pointing that out. After more finagling around, I found out how to get past the 2 errors I was getting:

"Oct 23 07:58:58 michaelv2 zabbix_server[3153]: /usr/sbin/zabbix_server: /usr/pgsql-13/lib/libpq.so.5: no version information available (required by /usr/sbin/zabbix_server)"

Someone in the IRC channel recommended that I reinstall PostgreSQL 13 again, which I did, and that made this error go away. I do not understand how it fixed the problem, but it did. I had previously installed PG 13 with the official PG repo, and then installed Zabbix.


"Oct 23 07:58:58 michaelv2 zabbix_server[3153]: zabbix_server [3153]: Cannot initialize IPC services: Cannot access path "/var/run/zabbix": [13] Permission denied."
This error was totally related to SELinux. This error went away after I followed the steps verbatim here, https://bestmonitoringtools.com/how-...inux_on_Zabbix

Thanks again for your kind response.

[shred00]

So it's acceptable to you to simply disable a very important security feature of your OS just to run Zabbix? Zabbix should be shipping a selinux policy module that doesn't require this.

[cyber]

So it's acceptable to you to simply disable a very important security feature of your OS just to run Zabbix? Zabbix should be shipping a selinux policy module that doesn't require this.

At least for v6 there is selinux rpm included. What it exactly contains, I have not yet found, but I have not searched too hard also..
And that page really did not tell to disable anything, but adjust according to needs...

[shred00]

It's good news that selinux is being addressed directly in v6. There is also https://support.zabbix.com/browse/ZBXSEC-74 which also hopefully gets into v6 to alleviate another security gap. Indeed, I hope that the selinux module in v6 does not actually allow the DAC_OVERRIDE and instead ZBXSEC-74 is used to correct the problem.

[morsing]

So it's acceptable to you to simply disable a very important security feature of your OS just to run Zabbix? [...]