Upgrading 3.4.7: [24] Too many open files] into log file

[hernan]

Hello team,

After upgrade to 3.4.7 the server crash with the following error message:

**************
zabbix_server [25699]: failed to open log file: [24] Too many open files
zabbix_server [25699]: failed to write [cannot accept incoming IPC connection: [24] Too many open files] into log file
25709:20180226:152850.200 cannot read preprocessing service request
25706:20180226:152850.201 cannot read preprocessing service request
25708:20180226:152850.201 cannot read preprocessing service request
25707:20180226:152850.201 cannot read preprocessing service request
25705:20180226:152850.203 cannot read preprocessing service request
25703:20180226:152850.203 cannot read preprocessing service request
25704:20180226:152850.204 cannot read preprocessing service request
**************

Can you help me to fix this problem?

Thanks in advance.

Regards.

[elmorekevin]

Same problem here on 3.4.6

Were you able to find a solution?

[vedroid]

The same problem after upgrade from 3.2.4 to 3.4.7

[kloczek]

This has nothing to do with zabbix: 1) check zabbix user limits settings 2) check IPC memory used by zabbix user (probably something still is used after prev crash of the zabbix processes) 3) check your SELinux settings

[vedroid]

This has nothing to do with zabbix

But setting new option (there was no such option in 3.2.4) "StartPreprocessors=100" in config instead of default StartPreprocessors=3 resolved my problem. About 24 ours zabbix 3.4.7 works fine after update from 3.2.4

[kloczek]

If proves really nothing. Just try to go back to StartPreprocessors=3.

If you had some IPC shared memory blocks allocated left by not fully killed 3.2.4 processes you may observe such effects as well.
(Really try to learn how to diagnose the issues. For example how t use strace)

[ricofranco]

Dear sirs, we got this same error but with a Zabbix Server version 5.015 . Anybdoy can help me ?

[tim.mooney]

As Kloczek has responded previously, this isn't really a Zabbix problem, it's a (minor) problem with how your Zabbix server's operating system is configured. It's easy to fix, but you have to provide more information about your environment before anyone could guide you to the best fix.

The Zabbix server processes run as a normal (service) user, probably named 'zabbix'. That's good design; it's the way it should be.

Normal users on Linux and Unix systems (including service accounts like the 'zabbix' user) have resource limits in place, so that a single account cannot accidentally or maliciously starve other processes on the system of resources.

One of these resource limits that's frequently put in place is a limit on the number of open files a user can have (systemwide).

If you log into your Zabbix server as a normal user (not root) and run the shell built-in (for whatever shell you use, probably bash but it might be something else, but almost every shell calls this builtin the same thing):

ulimit -a

You'll probably see output something like:

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127579
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

The line that says "open files" and then for me shows the default limit is 1024 is probably (but not guaranteed to be) the same as the limit that the 'zabbix' user would have. It's this limit that is probably preventing Zabbix from opening more files.

How you fix that depends on your environment, though. That's why you should provide details about your environment when you ask a technical question like this.

If you're on a relatively recent Linux distro and Zabbix server is being started via 'systemd', then fixing it probably involves using

sudo systemctl edit zabbix-server.service

and adding the line

LimitNOFILE=2048

In the '[Service]' section. This leaves the stock zabbix-server.service file unmodified but augments it with your addition. This should survive updates to the Zabbix server package(s).

If your server that's running Zabbix isn't using systemd to start the Zabbix service, then there are other methods to fix the open file resource limit, but you would have to provide some details about your environment.