cannot start zabbix-server with zabbix 3.4.11

[rk325]

We recently updated to zabbix 3.4.11, and can only start zabbix-agent and zabbix-java-gateway. When trying to start zabbix-server I get the error:

Job for zabbix-server.service failed because a configured resourse limit was exceeded...

zabbix_server.log continuously reports:

"cannot open log: cannot create semaphore set: [28] No space left on device"

/var/log/messages reports:

"Starting zabbix Server...
PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Failed to start zabbix Server
Unit zabbix-server.service entered failed state."

I tried the solutions posted in this forum that mention issues with SELinux, about creating a policy for allowing Zabbix to run, but I cannot find any entries about "denied" or AVC in audit.log for comm=zabbix-server. Even after I set the SELinux to Permissive and restart my VM. I even disabled SELinux completely, restarted my VM, and still cannot get to install zabbix-server. zabbix_server.log and messages keep reporting the same errors I quoted before.

I also try to delete zabbix semaphores (another solution in this forum) but nothing. I simply cannot start zabbix-server nor can I see any AVC denied entry for it in the log files.

The rpms I have installed are:

zabbix-get-3.4.11-1el7.x86_64
zabbix-sender-3.4.11-1el7.x86_64
zabbix-server-pgsql-3.4.11-1el7.x86_64
zabbix-web-3.4.11-1el7.x86_64
zabbix-web-pgsql-3.4.11-1el7.x86_64
zabbix-java-gateway-3.4.11-1el7.x86_64
zabbix-agent-3.4.11-1el7.x86_64

The same rpms I had on my previous Zabbix 3.2.4 (but with that version, 3.2.4) and SELinux in "enforced" mode. zabbix ran with no issues there.

Any help will be appreciated.

[Atsushi]

If you enable SELinux and run it, you may need to perform the operation of setting the policy multiple times if startup fails.
For example, suppose that you added a policy because you can not start up because the read permission of the target directory is insufficient.
After that, when you try to start up, the next time you need write permission to that directory, you may need to set the policy again.

Zabbix 3.4 requires more policy settings than 3.2.
Check the log again to see what kind of error is being output.

The "No space left on device" error is probably because the shared memory area was allocated more than once because the start failed multiple times, and the space became gone.
It will be improved by terminating all Zabbix related processes and deleting it or rebooting the OS.

[rk325]

Hello Atsuchi. I was not sure my original post got posted here.
Anyway the actual message in zabbix_server.log was : cannot open log: cannot create semaphore set: [28] No space left on device

There is no AVC message in the audit.log file, so I am investigating the messages in zabbix_server.log.

I deleted all zabbix semaphores and start zabbix-server one more time. It did not start, but zabbix_server.log started to report:

28215:20180801:150156.400 Starting Zabbix Server. Zabbix 3.4.11 (revision 82160).
28215:20180801:150156.401 ****** Enabled features ******
28215:20180801:150156.401 SNMP monitoring: YES
28215:20180801:150156.401 IPMI monitoring: YES
28215:20180801:150156.401 Web monitoring: YES
28215:20180801:150156.401 VMware monitoring: YES
28215:20180801:150156.401 SMTP authentication: YES
28215:20180801:150156.402 Jabber notifications: YES
28215:20180801:150156.402 Ez Texting notifications: YES
28215:20180801:150156.402 ODBC: YES
28215:20180801:150156.402 SSH2 support: YES
28215:20180801:150156.402 IPv6 support: YES
28215:20180801:150156.402 TLS support: YES
28215:20180801:150156.402 ******************************
28215:20180801:150156.402 using configuration file: /etc/zabbix/zabbix_server.conf
28215:20180801:150156.445 current database version (mandatory/optional): 03040000/03040001
28215:20180801:150156.446 required mandatory version: 03040000
28215:20180801:150156.446 optional patches were found
28215:20180801:150156.446 starting automatic database upgrade
28215:20180801:150156.448 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR: constraint "c_sessions_1" of relation "sessions" does not exist
[alter table sessions drop constraint c_sessions_1]
28215:20180801:150156.449 database upgrade failed

Then I added the constraint "c_sessions_1" to table sessions in the zabbix db, and start zabbix-server one more time. It is still hanign to return to me a promp from that start.

zabbix_server.log started reporting other messages:
: : :
29596:20180801:151101.630 server #288 started [poller #54]
29597:20180801:151101.641 server #289 started [poller #55]
29595:20180801:151101.682 server #287 started [poller #53]
29598:20180801:151101.692 server #290 started [poller #56]
29599:20180801:151101.703 server #291 started [poller #57]
29518:20180801:151102.232 server #227 started [java poller #196]
29612:20180801:151102.666 server #294 started [poller #60]
29614:20180801:151102.741 server #296 started [poller #62]
29611:20180801:151102.741 server #293 started [poller #59]
29613:20180801:151102.752 server #295 started [poller #61]
29610:20180801:151102.754 server #292 started [poller #58]
29064:20180801:151103.062 cannot connect to alert manager service: Cannot connect to service "alerter": [2] No such file or directory.
29065:20180801:151103.067 cannot connect to alert manager service: Cannot connect to service "alerter": [2] No such file or directory.
29063:20180801:151103.071 cannot connect to alert manager service: Cannot connect to service "alerter": [2] No such file or directory.
29058:20180801:151104.125 One child process died (PID:29064,exitcode/signal:1). Exiting ...
29625:20180801:151105.166 server #297 started [poller #63]
29058:20180801:151106.155 syncing history data...
29058:20180801:151106.155 syncing history data done
29058:20180801:151106.155 syncing trend data...
29058:20180801:151106.156 syncing trend data done
29058:20180801:151106.156 Zabbix Server stopped. Zabbix 3.4.11 (revision 82160).
zabbix_server [29625]: [file:'log.c',line:246] lock failed: [22] Invalid argument

It stopped reporting messages; stopped there on the last "lock failed" message.

What do you think?

Thanks!

[Atsushi]

cannot create semaphore set: [28] No space left on device

This error occurs because it is not able to solve the error that zabbix_server can not be started due to another problem, so it seems that the vacancy has disappeared trying to secure semaphores more than once.

29064:20180801:151103.062 cannot connect to alert manager service: Cannot connect to service "alerter": [2] No such file or directory.

If this error occurs, you probably have failed to create the socket.

Have you added socket creation policy in SELinux?
If you have not done it, please check the log in /var/log/audit/audit.log and add the policy.

[rk325]

Yes, I have. As a matter of fact I tried today to install zabbix 3.4.11 on a brand new VM that never had zabbix installed before (restored to a previous snapshot) but before I run Puppet to do the installation, I did all the fixes to this new server, like adding the SELinux policies. Still zabbix-server cannot start, but at least I do not get a long list of zabbix semaphores; however this time zabbix_server.log ended with:

7354:20180807:162104.889 server #356 started [poller #122]
6732:20180807:162105.021 cannot connect to alert manager service: Cannot connect to service "alerter": [2] No such file or directory.
6734:20180807:162105.027 cannot connect to alert manager service: Cannot connect to service "alerter": [2] No such file or directory.
6733:20180807:162105.032 cannot connect to alert manager service: Cannot connect to service "alerter": [2] No such file or directory.
7368:20180807:162105.088 server #361 started [poller #127]
7369:20180807:162105.099 server #362 started [poller #128]
7374:20180807:162105.109 server #367 started [poller #133]
7370:20180807:162105.119 server #363 started [poller #129]
7372:20180807:162105.131 server #365 started [poller #131]
7373:20180807:162105.141 server #366 started [poller #132]
7371:20180807:162105.152 server #364 started [poller #130]
7375:20180807:162105.163 server #368 started [poller #134]
6723:20180807:162105.262 One child process died (PID:6732,exitcode/signal:1). Exiting ...
7380:20180807:162106.534 server #369 started [poller #135]
6723:20180807:162107.320 syncing history data...
6723:20180807:162107.320 syncing history data done
6723:20180807:162107.320 syncing trend data...
6723:20180807:162107.321 syncing trend data done
6723:20180807:162107.321 Zabbix Server stopped. Zabbix 3.4.11 (revision 82160).
zabbix_server [7380]: [file:'log.c',line:246] lock failed: [22] Invalid argument

Searching for [file:'log.c',line:246] lock failed: [22] Invalid argument I found this link https://www.zabbix.com/documentation...x/other_issues.
But here I get lost: what does it mean "Stop using zabbix account for anything else than Zabbix processes, create a dedicated account for other things." ?

I defined a zabbix user (with "zabbix" as its name), a password and use these 2 in my zabbix_server.conf that Puppet uses to build the initial zabbix db. I do not psql or ssh to any other instance of this database using this zabbix user or any user. How do I know which user is used by my Zabbix processes? Is it the one defined in zabbix_server.conf ?
Does it mean I need to create a different user for Puppet to create the zabboix db? Still this "new" user will have to have login capabilities, right?

Please help me understand this.

Thanks,

[Atsushi]

You must check the log in /var/log/audit/audit.log again.