"Zabbix server is not running" was displayed and it server was actually stopped.

[ABC Loud]

(Update and fixed some sentences on 16th Aug)

Hello all.

I am writing to ask your support to solve the problem on my zabbix server.

Please let me explain what happened first.
I have a zabbix server (ver.4.0.7 on SUSELinux 15) and it has already been running for more than 2 years.
On last Sunday, suddenly, Zabbix server stopped working and the message Zabbix server is not running" was displayed.


Here is the detail below. I need to know why it was happened.

Date:
08/08/2021 (the server was stopped)
09/08/2021 (the server was recovered manually by teammate)

Situation:
After logged on Zabbix console, my teammates noticed that zabbix server is actually not working.
They tried to reboot zabbix server, however, they could not reboot it.
They killed each process, and reboot the server finally.

Log:
We checked zabbix_server.log and the last log before stopping working is "/var/log/zabbix/zabbix_server.log": [13] Permission denied".
I suppose that it might be relevant to log rotation, since zabbix server log's rotation was working at that timing.
However, it is not clear that log rotation failure might make zabbix server unavailable.
Also, log rotation itself has already been running for almost 2 years, but such error has never been happened as long as I know.

Question:
Would you have any suggestion of reason why zabbix server suddenly stopped?
Is it possible that log rotation failure causes zabbix status error?
Or are there any other possible causes to make zabbix server not running?

I have already checked resource status of the server, and usage of CPU and memory were not high (CPU usage was around 20%, and that of memory was around 70%),
so I suggest OS resource would not be the cause.


Thank you and kind regards.

[johndoe2374]

Hello. Check permissions on your /var/log/zabbix directory and zabbix_server.log file. If logrotate is used for log rotation, it should assign access mode and owner/group to new log files (you can check at logrotate's settings in file /etc/logrotate.d/zabbix-server - "create 0664 zabbix zabbix"). Also I think you can try to rotate logs without logrotate daemon or crontab job for it by using LogFileSize option in Zabbix server's config.

[johndoe2374]

Also can be a SELinux issue, if you didn't disable it or set to permissive mode.

[Andre Fidder]

I had the same issue last Sunday. In my case it turned out that the issue was caused by an external script that takes care of the log rotation (once a week).