Failed to connect to database: Error: Too many connections [1040]

[dreas]

Zabbix server is regularly shutting itself down suddenly.

 14168:20071002:133506 Host [s_se_mx3]: first network error, wait for 15 seconds
 14168:20071002:133506 Parameter [system.cpu.util[,idle,avg1]] will be checked after 40 seconds on host [s_se_mx3]
 14170:20071002:133507 Timeout while answering request
 14170:20071002:133507 Get value from agent failed. Error: Cannot connect to [mx3.spamexperts.com:10050] [Interrupted system call]
 14170:20071002:133507 Host [s_se_mx3]: first network error, wait for 15 seconds
 14170:20071002:133507 Parameter [vfs.fs.inode[/,free]] will be checked after 240 seconds on host [s_se_mx3]
Oct 02 13:35:45 database sendEmail[27204]: Email was sent successfully!
 14180:20071002:133602 Executing housekeeper
 14180:20071002:133626 Deleted 68249 records from history and trends
 14170:20071002:133847 Timeout while answering request
 14170:20071002:133847 Get value from agent failed. Error: ZBX_TCP_READ() failed [Interrupted system call]
 14170:20071002:133847 Host [s_se_mx5]: first network error, wait for 15 seconds
 14170:20071002:133847 Parameter [greylistd.status] will be checked after 300 seconds on host [s_se_mx5]
Oct 02 13:43:45 database sendEmail[28431]: Email was sent successfully!
Oct 02 13:44:15 database sendEmail[28500]: Email was sent successfully!
Oct 02 13:45:15 database sendEmail[28639]: Email was sent successfully!
Oct 02 13:45:16 database sendEmail[28641]: Email was sent successfully!
Oct 02 13:48:46 database sendEmail[29237]: Email was sent successfully!
 14170:20071002:134850 Timeout while answering request
 14170:20071002:134850 Get value from agent failed. Error: ZBX_TCP_READ() failed [Interrupted system call]
 14170:20071002:134850 Host [s_se_mx5]: first network error, wait for 15 seconds
 14170:20071002:134850 Parameter [greylistd.status] will be checked after 300 seconds on host [s_se_mx5]
 14178:20071002:134916 Failed to connect to database: Error: Too many connections [1040]
 14155:20071002:134916 One child process died. Exiting ...
 14155:20071002:134918 ZABBIX Server stopped

There do not really seem to be issues with MySQL however. Anyway it should not kill the zabbix_server? Any suggestions?

[Alexei]

There do not really seem to be issues with MySQL however. Anyway it should not kill the zabbix_server? Any suggestions?

This IS MySQL issue. It kill itself because ZABBIX processes relies on availability of MySQL. If one ZABBIX process has access to MySQL and another has not, it creates problems.

However if MySQL is down it is not a problem because all ZABBIX processes are in the same situation and they are just waiting when MySQL is back again, but this is completely different case.

[dreas]

So what is determining "Error: Too many connections"? Can't zabbix_server just wait if there are too many connections? Is this the number of maximum SQL connections as determined by the MySQL settings?

[Alexei]

SCan't zabbix_server just wait if there are too many connections? Is this the number of maximum SQL connections as determined by the MySQL settings?

Wait for how long? Yes, I am pretty sure it is the maximum number of allowed connections defined in my.cfg.

[dreas]

Wait for how long?

Until it can connect again.

[randy]

I was unable to keep zabbix_server running for more than an hour until I changed MySQL's wait_timeout.

Rather than increase /etc/my.cnf 'max_connections', first check the currently configured db server value of the 'wait_timeout' variable by running

$> mysqladmin -u root -p variables

If the value of 'wait_timeout' is greater than a couple of minutes, and if you have nobody connecting from a remote location, reduce it. In my case, the only connecting clients are webservers with quad-core Xeons on a gigabit LAN, so I reduced the value to 62 seconds. Running phpinfo on the webservers showed that they were all set to timeout in 60 seconds anyhow.

This change reduced my actual Max_used_connections from 257 to 74.

Check actual 'Max_used_connections' with:
mysql -u root -p
mysql> show status ... or ...
mysql> show status like '%conn%'; ... to get fewer lines of output

Depending on the risk of connection re-use, other folks have set the 'wait_timeout' value from 10 to 600 seconds. The safest values is at least twice the MSL, though in practice, the risk is very very small on fast networks of fast computers.

(see http://www.tcpipguide.com/free/t_TCP...mination-4.htm )

[dreas]

We reduced the load on our MySQL server and the issue has not reoccurred. I am not sure if setting wait_timeout so low is a good solution since that seems more like a quick fix for something else that is going wrong.

[hernan]

Hello Team,

I ran "show status like '%conn%';" in my status with the results on the jpg files.



Can you help me to know with this the correct procedure to change the value on the database server justo to resolve the issue?

Thanks in advance.

Regards.

[AndrewSummer]

Thanks for your Help!

Had the same error after adjusting the Poller Settings.

These two links also helped a lot:

http://pensacola-tech.com/pensacola/2010/08/20/unable-to-connect-to-database-too-many-connections/

https://www.electrictoolbox.com/update-max-connections-mysql/


Past used connections:
#mysqladmin -uadmin -p extended-status | grep Max_used_connections

Current connections limit settings can be found with:
#mysqladmin -uadmin -p variables | grep 'max.*connections'



With Ubuntu 16.04 I set the max_connections = 200 in /etc/mysql/my.cnf

Cheers.

[Peter Franca]

AndrewSummer just helped me to fix the same issue. Same behaver, after updating the poller settings on /etc/zabbix/zabbix_server.conf I got the same error. Fixing the "max_connections = 200" fixed the issue.