Zabbix Too Many Open Files

[Parasin]

Hi all,

I have a small Zabbix installation monitoring only 7 hosts. The Zabbix 2.4.4 Server is running on CentOS 6.

I have started receiving the error:

Cannot open /proc/*: [24] Too many open files

Which causes many of my Zabbix Server items to go to a Not Supported state.

I have checked the Zabbix logs and did not find any useful information on Debug level 3 or 4.

item "Zabbix server:net.if.discovery" became not supported: Cannot open /proc/net/dev: [24] Too many open files
 16267:20150511:092800.703 
item "Zabbix server:vfs.fs.discovery" became not supported: Cannot open /proc/mounts: [24] Too many open files

Anyone have ideas on how to resolve???


I have tried increasing the max file limit by using:

ulimit -n 65535

and also changing the /etc/security/limits.conf file to allow max of 65536 files.

[aib]

Read this article

https://rtcamp.com/tutorials/linux/increase-open-files-limit/

To check your current limit, use command

cat /proc/sys/fs/file-max

[Parasin]

I have already done this, my current limit is set to 70,000 with the key file-max.

Thank you for your reply

[aib]

I have already done this, my current limit is set to 70,000 with the key file-max.

Thank you for your reply

OK, you increased limits. Did it solve your problem?

[Parasin]

It solves the problem temporarily. But after a few days, the problem comes back again with the same message.

Does Zabbix have a with problem not closing sockets that I am unaware of? Or is this perhaps related to CentOS itself?


I'll keep an eye on this, as I just increased the file-max limit again and restarted the server. If it comes up again, I'll update the post.


Why does Zabbix require so many files to be open for so little items monitored?

[tchjts1]

I have already done this, my current limit is set to 70,000 with the key file-max.

Thank you for your reply

Are you sure it is 70,000 and not 700,000?

When I check mine on my RHEL Zabbix app server I get this:

$ cat /proc/sys/fs/file-max
783778

And my ulimit is:

$ ulimit -n
1024

[aib]

My limit is higher! I'm a winner!

# cat /proc/sys/fs/file-max
[B]1582435[/B]

[tchjts1]

My limit is higher! I'm a winner!

# cat /proc/sys/fs/file-max
[B]1582435[/B]

You are indeed the winner!

Your prize is a free copy of Zabbix source code for version 2.4.5 and you may claim your prize here: http://www.zabbix.com/download.php

Happy compiling!

[Parasin]

This is the exact output of the command on my Zabbix Server.

[root@ ~]# cat /proc/sys/fs/file-max
70000

And when I use ulimit -n:

[root@ ~]# ulimit -n
65536

Should I increase this limit? It seems like you both have much higher tolerances for your installations.

[tchjts1]

Your file-max seems low. But then again, I am not a Linux SA. I know just enough to be dangerous.

I do know that DB's and web server programs need more breathing room than a stock OS setup gives you in the open-files department.

Personally if it were me, I would try a setting with 200000 and see if everything was still working smoothly.

I am assuming your setup is the Zabbix App server and web server on one box and your DB on a separate box?

[Parasin]

I only have one box running the DB, Zabbix App Server, and Web Server at the moment; my Zabbix installation is very small and will most likely contain less than 20 hosts.


I have a distributed monitoring setup with proxies doing the bulk of the work.

[Parasin]

Resolved

I found the source of my problem.

One of my zabbix_agentd.conf files had the HostName listed as

Hostname=127.0.0.1

My Zabbix server was constantly trying to find the agent and was never able to, as the Hostname in the Zabbix GUI was titled "Zabbix Server Agent".

Once I corrected the zabbix_agentd.conf file and restarted the agent, all of my "Too Many Open Files" errors disappeared after a few minutes.