v6.4.9 AllowRoot=1 Debian-12.4: failed to open log file: [24] Too many open files

[spezialist]

Dear support,

I have a dedicated application server (frontend and mariadb are on other servers) Zabbix server 6.4.9 under Debian-12.4 with 28GB RAM with Zabbix HA option enabled, which works fine with StartPollers=600 and AllowRoot=0 (as a regular user zabbix).

But as soon as I increase the parameter to StartPollers=700, then after successfully starting zabbix-server, the server runs for ~10 minutes at most and stops with these errors:

zabbix_server [424407]: failed to open log file: [24] Too many open files
zabbix_server [424407]: failed to write [cannot accept incoming IPC connection: [24] Too many open files] into log file
424893:20231212:132232.463 cannot write to IPC socket: Broken pipe
424893:20231212:132232.463 cannot send data to preprocessing service
424299:20231212:132232.512 One child process died (PID:424407,exitcode/signal:1). Exiting ...
424300:20231212:132232.512 HA manager has been paused
425661:20231212:132232.660 cannot write to IPC socket: Broken pipe
425661:20231212:132232.660 cannot send data to preprocessing service
425662:20231212:132232.719 cannot write to IPC socket: Broken pipe
425662:20231212:132232.719 cannot send data to preprocessing service
425660:20231212:132232.719 cannot write to IPC socket: Broken pipe
425660:20231212:132232.719 cannot send data to preprocessing service
425658:20231212:132232.755 cannot write to IPC socket: Broken pipe
425658:20231212:132232.756 cannot retrieve alert results
424873:20231212:132232.774 cannot write to IPC socket: Broken pipe
424873:20231212:132232.774 cannot send data to preprocessing service
zabbix_server [424299]: Error waiting for process with PID 424407: [10] No child processes
424300:20231212:132235.780 HA manager has been stopped

And, repeatedly increasing the limits for a regular zabbix user did absolutely nothing to help. Then I switched zabbix-server to run as root superuser using the AllowRoot=1 option to ensure that there were no limits for the user. But unfortunately this didn't help either and after ~10 minutes zabbix-server stopped with exactly the same error as above. And during the experiment I checked the limits for running zabbix-server processes like this:

# prlimit -p 515350
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0         0 bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max number of file locks held      unlimited unlimited locks
MEMLOCK    max locked-in-memory address space   8388608   8388608 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0
NOFILE     max number of open files                1024    524288 files
NPROC      max number of processes               111588    111588 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals         111588    111588 signals
STACK      max stack size                       8388608 unlimited bytes

At the same time I checked how many files are open in the OS and what is the maximum possible number of files according to the RAM size:

# cat /proc/sys/fs/file-max
9223372036854775807
# cat /proc/sys/fs/nr_open
1048576
# lsof | wc -l
202146

I mean, visually everything looks fine.

So as a last point I decided to ask my question here about this problem. Maybe there are some nuances? Because even running processes as root does not help and the error about limiting the number of open files persists :-(.

Very thanks for the support.

[spezialist]

SOLVED

As I wrote above, modifying file /etc/security/limits.conf for a regular zabbix user does not solve the problem. But the SOLUTION turns out to be published here https://www.zabbix.com/forum/in-russian/447695-сервер-постоянно-перезапускается?p=447966#post447966, i.e. in my case I wrote in file /etc/systemd/system/zabbix-server.service.d/override.conf and the problem was successfully solved:

[Service]
LimitNOFILE=8192​

P.S.: it would be very nice if the official Zabbix documentation would describe or mention this nuance, as the solution is not as obvious as it seems at first glance.