Zabbix poller processes more than 75% busy

[gbiondi]

Hi,

i have installed on fresh copy of CentOS 7, Zabbix 3.2
I have about 80 host queried via snmp located worldwide with response time varied from 2 ms to 400ms..
I (I hope..) tuned the 'Timeout' value to 10 and have played with StartPoller from 5 (default) to 80, but after some time the result is 'Zabbix poller process..'.
I have read some docs and all indicates to change StartPoller..
Where I wrong?

All the best.

GB

[kloczek]

You need to move away from passive monitoring (which does not scale well).
Start reading about active one.

[gbiondi]

Hi,

your answer is too cripto for me.. in other hand? I don't install zabbix agent on my hosts.. in fact this host are customer-side and I can't install nothing (I hope understand you).

[kloczek]

Try to have look for "zabbix active vs passive" on google.

[nobodysu]

You have increased 'StartPollers' in zabbix_server.conf and that did nothing?

[arta]

I have the same problem.

When I increase StartPollers in zabbix_server.conf , I get the following error upon restarting Zabbix-Server:

 root@zabbix ~]# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Fri 2020-01-24 13:11:43 CET; 6s ago
Process: 12654 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 12660 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=1/FAILURE)
Main PID: 12020 (code=exited, status=0/SUCCESS)  Jan 24 13:11:43
zabbix systemd[1]: Failed to start Zabbix Server. Jan 24 13:11:43
zabbix systemd[1]: Unit zabbix-server.service entered failed state. Jan 24 13:11:43 zabbix systemd[1]: zabbix-server.service failed.

FYI, the only thing I changes in zabbix_server.conf is the parameter StartPollers, nothing else. when I add a # before the line, it starts without problem.
Any thoughts?

[dimir]

arta , please see the end of /var/log/zabbix/zabbix_server.log for details why it failed to start. You can paste it on a paste bin or attach log here if you don't understand something. The only thing I see how increasing number of pollers can prevent Zabbix server from starting is:

- the value of StartPollers you set is not within range 0-1000
- number of OS processes reached limit

To check maximum allowed processes on your system, the lowest value of:

$ ulimit -u
63531
$ cat /proc/sys/kernel/pid_max
32768

To get number of currently running processes:

$ pgrep -c ''

[dimir]

gbiondi, you say

after some time the result is 'Zabbix poller process..'

what do you mean?

[arta]

Hello @dimir

It does not matter what value I set, any number between 1 to 1000, it fails the zabbix-server. When I set to 0 or hash it out # it works fine.

 [root@zabbix ~]# ulimit -u
15232  

 [root@zabbix ~]# cat /proc/sys/kernel/pid_max
32768    
 [root@zabbix ~]# pgrep -c ''
211

Here is what the log says:

  
 [root@zabbix ~]# tail -20 /var/log/zabbix/zabbix_server.log  22136:20200124:192722.458 server #211 started [poller #198]  22126:20200124:192722.468 server #201 started [poller #188]  22063:20200124:192722.468 [Z3001] connection to database 'zabbix' failed: [1040] Too many connections  22063:20200124:192722.468 Cannot connect to the database. Exiting...  22135:20200124:192722.469 server #210 started [poller #197]  22130:20200124:192722.479 server #205 started [poller #192]  22060:20200124:192722.480 [Z3001] connection to database 'zabbix' failed: [1040] Too many connections  22060:20200124:192722.480 Cannot connect to the database. Exiting...  22124:20200124:192722.490 server #199 started [poller #186]  22117:20200124:192722.501 server #192 started [poller #179]  22118:20200124:192722.512 server #193 started [poller #180]  22121:20200124:192722.523 server #196 started [poller #183]  22052:20200124:192722.528 [Z3001] connection to database 'zabbix' failed: [1040] Too many connections  22052:20200124:192722.528 Cannot connect to the database. Exiting...  22106:20200124:192722.530 [Z3001] connection to database 'zabbix' failed: [1040] Too many connections  22106:20200124:192722.530 Cannot connect to the database. Exiting...  21776:20200124:192722.533 One child process died (PID:21997,exitcode/signal:1). Exiting ...  21776:20200124:192722.683 syncing trend data...  21776:20200124:192722.683 syncing trend data done  21776:20200124:192722.683 Zabbix Server stopped. Zabbix 4.4.4 (revision 3131fdac04).

This is how my /etc/my.cnf looks like:

  
 [root@zabbix ~]# cat /etc/my.cnf [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 # Settings user and group are ignored when systemd is used. # If you need to run mysqld under a different user or group, # customize your systemd unit file for mariadb according to the # instructions in http://fedoraproject.org/wiki/Systemd  [mysqld_safe] log-error=/var/log/mariadb/mariadb.log pid-file=/var/run/mariadb/mariadb.pid  # # include all files from the config directory # !includedir /etc/my.cnf.d  max_connextions = 3000 [root@zabbix ~]#

Thank you

[dimir]

connection to database 'zabbix' failed: [1040] Too many connections 22106:20200124:192722.530 Cannot connect to the database. Exiting...

There's your problem. Your MySQL server does not accept the number of connections that Zabbix server needs. Each process needs a connection. See here: https://dev.mysql.com/doc/refman/5.5...nnections.html