Zabbix Server/Agent Log-Rotation

[c.h.]

On our zabbix 5.0.20 server (Ubuntu 18.04, from Zabbix's deb packages), /etc/zabbix/zabbix_server.conf has LogFileSize=0 and there's /etc/logrotate.d/zabbix-server-pgsql which rotates the log weekly, keeping 12 log files (~ 3 months).

I've just modified the logrotation to be daily, keeping 30 log files, and then tested it:

$ sudo bash
# logrotate -f /etc/logrotate.d/zabbix-server-pgsql
# lsof -p $(cat $(grep -hoP '^PidFile=\K.*' /etc/zabbix/zabbix_server.conf)) | grep zabbix_server.log

zabbix_se 991396 zabbix 1w REG 252,5 268534 12455876 /var/log/zabbix/zabbix_server.log.1
zabbix_se 991396 zabbix 2w REG 252,5 268534 12455876 /var/log/zabbix/zabbix_server.log.1

# fuser /var/log/zabbix/zabbix_server.log{,.1}

/var/log/zabbix/zabbix_server.log: 991437 991441 991444 991445 991446 991447 991449 991451 991452 991453 991454 991455 991456 991457 991458 991459 991460 991461 991462 991463 991464 991465 991481 991482 991484 991485 991486
/var/log/zabbix/zabbix_server.log.1: 991396 991442 991443 991450

The server is still writing to the old log file, even though it's been rotated. Most other child processes are writing to the new log file.

So, I added a post-rotate script to the logrotate file like this (the man page didn't show a 'reopen log files' option, maybe config_cache_reload will do it?):

     postrotate
         /usr/sbin/zabbix_server -R config_cache_reload
     endscript

# fuser /var/log/zabbix/zabbix_server.log{,.1}

/var/log/zabbix/zabbix_server.log: 983428 983431 983434 983435 983436 983437 983438 983440 983442 983443 983445 983446 983447 983448 983449 983450 983451 983452 983453 983465 983466 983467 983468 983469 983472 983473 983474
/var/log/zabbix/zabbix_server.log.1: 983385 983432 983433 983439

In short, reloading the config_cache didn't change a thing.

Now I've taken out the postrotate script and the `create 0640 zabbix zabbix` line, and added `copytruncate`.

When I ran logrotate (in this two-logfile state), things got weird (zabbix_server_log.1 is deleted, but zabbix_server is still writing to it). `systemctl restart zabbix-server` gets it back to normal again.

Final working version of /etc/logrotate.d/zabbix-server-pgsql:

/var/log/zabbix/zabbix_server.log {
    rotate 30
    daily
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

I do recommend that the zabbix_server command line gets an option like Nginx has: `zabbix_server -R reopen` or even `kill -USR1 $pidfile`.