Ad Widget

**alixen** · 03-03-2010, 19:24

Hi,

I have exactly the same problem:

Code:

 21143:20100303:173418.394 [Z3005] Query failed: [2006] MySQL server has gone away [begin;]
 21143:20100303:173418.394 [Z3005] Query failed: [2006] MySQL server has gone away [update hosts set snmp_errors_from=1267634058,snmp_disable_until=1267634073 where hostid=10341]
 21143:20100303:173418.394 Database is down. Retrying in 10 seconds

It appears more or less once a day.

I am a little bit worried since this a new Zabbix 1.8.1 server that is monitoring only a small subset of our hosts for testing purposes.
I don't think that it will replace our 1.6 server until this kind of problem is solved.

Regards,
Alixen

**sloth101** · 03-03-2010, 20:29

Seeing the same in 1.8.1 build with mysql Ver 14.12 Distrib 5.0.75, for debian-linux-gnu (x86_64) using readline 5.2

Host 10061 is a zabbix proxy same version.

Code:

  6150:20100228:122546.646 [Z3005] Query failed: [2006] MySQL server has gone away [begin;]
  6150:20100228:122546.646 [Z3005] Query failed: [2006] MySQL server has gone away [update hosts set errors_from=1267377946,disable_until=1267377961 where hostid=10061]
  6151:20100302:092257.342 [Z3005] Query failed: [2006] MySQL server has gone away [begin;]
  6151:20100302:092257.342 [Z3005] Query failed: [2006] MySQL server has gone away [update hosts set disable_until=1267539792 where hostid=10061]
  6148:20100303:103019.676 [Z3005] Query failed: [2006] MySQL server has gone away [begin;]
  6148:20100303:103019.676 [Z3005] Query failed: [2006] MySQL server has gone away [update hosts set errors_from=1267630219,disable_until=1267630234 where hostid=10061]
  6157:20100303:122947.811 [Z3005] Query failed: [2006] MySQL server has gone away [begin;]
  6157:20100303:122947.811 [Z3005] Query failed: [2006] MySQL server has gone away [select t.triggerid,i.itemid from triggers t,functions f,items i where t.triggerid=f.triggerid and f.itemid=i.itemid and t.status in (0) and t.value not in (2) and i.itemid in (23867)]

**sloth101** · 05-03-2010, 17:44

It's a bit hard to believe that their are only three of us seeing this issue.

**Murz** · 17-03-2010, 09:36

Same problem on Debian Lenny and zabbix-mysql:

Code:

  4395:20100317:113623.764 Database is down. Retrying in 10 seconds
  4394:20100317:113726.789 [Z3005] Query failed: [2006] MySQL server has gone away [select hostid from hosts where host='xxx' and proxy_hostid=0 and hostid between 000000000000000 and 099999999999999]
  4387:20100317:114452.758 [Z3005] Query failed: [2006] MySQL server has gone away [begin;]
  4387:20100317:114452.758 [Z3005] Query failed: [2006] MySQL server has gone away [select t.triggerid,i.itemid from triggers t,functions f,items i where t.triggerid=f.triggerid and f.itemid=i.itemid and t.status in (0) and t.value not in (2) and i.itemid in (19492)]

**IPDed** · 18-03-2010, 12:49

me too!
CentOS release 5.3 (Final)
MySQL 5.0.77
Zabbix 1.8

149 hosts
12179 objects
3148 triggers

However:
Hardware = Windows 2008 Hypervisor - clustered using Compellent SAN
RAM: 4G
Processor= 2
VM network adaptor

The files have been pretty mangled back and forth with a lot of experimentation- please comment freely...

my.cnf

Code:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
max_connections=300
innodb_buffer_pool_size=2GB
innodb_flush_method= O_DIRECT
thread_concurrency=4
wait_timeout=100
connect_timeout=10
skip-bdb
innodb_file_per_table=1
innodb_thread_concurrency       = 2     #Should match number of processors
max_allowed_packet=16M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

zabbix_server.conf

Code:

 This is a configuration file for Zabbix Server process
# To get more information about Zabbix,
# visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: NodeID
#       Unique NodeID in distributed setup.
#       0 - standalone server
#
# Mandatory: no
# Range: 0-999
# Default:
# NodeID=0

### Option: ListenPort
#       Listen port for trapper.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# ListenPort=10051

### Option: SourceIP
#       Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: LogFile
#       Name of log file.
#
# Mandatory: no
# Default:
# LogFile=

LogFile=/tmp/zabbix_server.log

### Option: LogFileSize
#       Maximum size of log file in MB.
#       0 - disable automatic log rotation.
#
# Mandatory: no
# Default:
# LogFileSize=1

### Option: DebugLevel
#       Specifies debug level
#       0 - no debug
#       1 - critical information
#       2 - error information
#       3 - warnings
#       4 - for debugging (produces lots of information)
#
# Mandatory: no
# Default:
# DebugLevel=3

### Option: PidFile
#       Name of PID file.
#
# Mandatory: no
# Default:
# PidFile=/tmp/zabbix_server.pid

### Option: DBHost
#       Database host name.
#       If set to localhost, socket is used for MySQL.
#
# Mandatory: no
# Default:
# DBHost=localhost

### Option: DBName
#       Database name.
#       For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored.
#
# Mandatory: yes
# Default:
# DBName=

DBName=zabbix

### Option: DBUser
#       Database user. Ignored for SQLite.
#
# Default:
# DBUser=

DBUser=root

### Option: DBPassword
#       Database password. Ignored for SQLite.
#       Comment this line if no password is used.
#
# Mandatory: no
# Default:
# DBPassword=

### Option: DBSocket
#       Path to MySQL socket.
#
# Mandatory: no
# Default:
# DBSocket=/tmp/mysql.sock

### Option: DBPort
#       Database port when not using local socket. Ignored for SQLite.
#
# Mandatory: no
# Range: 1024-65535
# Default:
# DBPort=3306 # for MySQL

############ ADVANCED PARAMETERS ################

### Option: StartPollers
#       Number of pre-forked instances of pollers.
#       You shouldn't run more than 30 pollers normally.
#
# Mandatory: no
# Range: 0-255
# Default:
StartPollers=30

### Option: StartIPMIPollers
#       Number of pre-forked instances of IPMI pollers.
#
# Mandatory: no
# Range: 0-255
# Default:
# StartIPMIPollers=0

### Option: StartPollersUnreachable
#       Number of pre-forked instances of pollers for unreachable hosts.
#
# Mandatory: no
# Range: 0-255
# Default:
# StartPollersUnreachable=1

### Option: StartTrappers
#       Number of pre-forked instances of trappers
#
# Mandatory: no
# Range: 0-255
# Default:
#StartTrappers=20

### Option: StartPingers
#       Number of pre-forked instances of ICMP pingers.
#
# Mandatory: no
# Range: 0-255
# Default:
#StartPingers=5

### Option: StartDiscoverers
#       Number of pre-forked instances of discoverers.
#
# Mandatory: no
# Range: 0-255
# Default:
# StartDiscoverers=1

### Option: StartHTTPPollers
#       Number of pre-forked instances of HTTP pollers.
#
# Mandatory: no
# Range: 0-255
# Default:
#StartHTTPPollers=5

### Option: ListenIP
#       Listen interface for trapper.
#       Trapper will listen on all network interfaces if this parameter is missing.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0

# ListenIP=127.0.0.1

### Option: HousekeepingFrequency
#       How often Zabbix will perform housekeeping procedure (in hours).
#       Housekeeping is removing unnecessary information from history, alert, and alarms tables.
#       If PostgreSQL is used, suggested value is 24, as it performs VACUUM.
#
# Mandatory: no
# Range: 1-24
# Default:
HousekeepingFrequency=1

### Option: DisableHousekeeping
#       If set to 1, disables housekeeping.
#
# Mandatory: no
# Range: 0-1
# Default:
# DisableHousekeeping=0

### Option: SenderFrequency
#       How often Zabbix will try to send unsent alerts (in seconds).
#
# Mandatory: no
# Range: 5-3600
# Default:
# SenderFrequency=30

### Option: CacheSize
#       Size of configuration cache, in bytes.
#       Shared memory size for storing hosts and items data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
CacheSize=1G

### Option: CacheUpdateFrequency
#       How often Zabbix will perform update of configuration cache, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
CacheUpdateFrequency=300

### Option: HistoryCacheSize
#       Size of history cache, in bytes.
#       Shared memory size for storing history data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
HistoryCacheSize=256M

### Option: TrendCacheSize
#       Size of trend cache, in bytes.
#       Shared memory size for storing trends data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
TrendCacheSize=256M

### Option: HistoryTextCacheSize
#       Size of text history cache, in bytes.
#       Shared memory size for storing character, text or log history data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
# HistoryTextCacheSize=16M

### Option: NodeNoEvents
#       If set to '1' local events won't be sent to master node.
#
# Mandatory: no
# Range: 0-1
# Default:
# NodeNoEvents=0

### Option: NodeNoHistory
#       If set to '1' local history won't be sent to master node.
#
# Mandatory: no
# Range: 0-1
# Default:
# NodeNoHistory=0

### Option: Timeout
#       Specifies how long we wait for agent, SNMP device or external check (in seconds).
#
# Mandatory: no
# Range: 1-30
# Default:
Timeout=15

### Option: TrapperTimeout
#       Specifies how many seconds trapper may spend processing new data.
#
# Mandatory: no
# Range: 1-300
# Default:
# TrapperTimeout=300

# TrapperTimeout=5

### Option: UnreachablePeriod
#       After how many seconds of unreachability treat a host as unavailable
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnreachablePeriod=45

### Option: UnavailableDelay
#       How often host is checked for availability during the unavailability period.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnavailableDelay=60

### Option: UnreachableDelay
#       How often host is checked for availability during the unreachability period
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnreachableDelay=15

### Option: AlertScriptsPath
#       Location of custom alert scripts
#
# Mandatory: no
# Default:
# AlertScriptsPath=/home/zabbix/bin/

### Option: ExternalScripts
#       Location of external scripts
#
# Mandatory: no
# Default:
# ExternalScripts=/etc/zabbix/externalscripts

### Option: FpingLocation
#       Location of fping.
#       Make sure that fping binary has root ownership and SUID flag set!
#
# Mandatory: no
# Default:
# FpingLocation=/usr/sbin/fping

### Option: Fping6Location
#       Location of fping6.
#       Make sure that fping binary has root ownership and SUID flag set
#
# Mandatory: no
# Default:
# Fping6Location=/usr/sbin/fping6

### Option: SSHKeyLocation
#       Location of public keys for SSH checks
#
# Mandatory: no
# Default:
# SSHKeyLocation=

### Option: TmpDir
#       Temporary directory.
#
# Mandatory: no
# Default:
# TmpDir=/tmp

### Option: Include
#       You may include individual files or all files in a directory in the configuration file.
#
# Mandatory: no
# Default:
# Include=

# Include=/etc/zabbix/zabbix_server.general.conf
# Include=/etc/zabbix/zabbix_server/

**exkg** · 18-03-2010, 13:40

Hi,

And about your mysql logs ?

[]s,
Luciano

**IPDed** · 18-03-2010, 16:32

Originally posted by exkg

Hi,

And about your mysql logs ?

[]s,
Luciano

I'm not seeing anything meaningful at the moment.

Code:

100318 10:44:23  mysqld started
100318 10:44:23 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
100318 10:44:23 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
100318 10:44:26  InnoDB: Started; log sequence number 82 3444586613
100318 10:44:26 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

**IPDed** · 24-03-2010, 13:03

Bump

Advice elsewhere on consistency checking the DB has helped performance but it's still giving the same errors- just less frequently.

Can anyone help?

Cheers
Ed

**jcesario** · 24-03-2010, 16:13

This is because some of the threads zabbix is using hit the MySQL wait_timeout.

To verify this is the issue, on your MySQL instance.

SHOW GLOBAL VARIABLES LIKE 'wait_timeout';

SHOW PROCESSLIST - find your zabbix threads. See if they all get restarted and trigger an error when the threads hit the wait_timeout in the SLEEP state.

The only solution ive found is that I crank the wait_timeout up to an insane number.

the zabbix_server should be using some kind of keepalive on its persistent connections.

On the upside. I do not think this causes any harm at all, as the php-mysql library should resend whatever query it was trying to execute when it has to reconnect like that. I think...

EDIT
Except that this error normally occurs on the zabbix_server daemon. Which obviously does not use the PHP-MySQL library. Still, the C lib for mysql should have the same behavior.

**IPDed** · 25-03-2010, 10:14

Thank you,

as you might have guess I have little experience in MySQL optimisation and most websites are for use in web applications so the opposite of Zabbix.

The metrics for the box are not showing it as very loaded - I was wondering if there was an imbalance of Zabbix daemons to service requests against the MySQL settings.

I have no experience with Zabbix so don't know what kind of ratio is 'normal'

Thanks for you help,

Ed

**aib** · 04-02-2015, 18:56

Some time ago, after updating Zabbix server to version 2.4.3, I started to get in zabbix_server.log files the same error messages

Code:

10490:20150204:112859.376 [Z3005] query failed: [2006] MySQL server has gone away [begin;]
 10492:20150204:113029.388 [Z3005] query failed: [2006] MySQL server has gone away [begin;]
 10493:20150204:113159.400 [Z3005] query failed: [2006] MySQL server has gone away [begin;]

Any advice?

**ingus.vilnis** · 05-02-2015, 16:02

Hello Aleksey,

I would say it is for the same reason jcesario wrote here: https://www.zabbix.com/forum/showpos...8&postcount=10

Can you check what exact processes do you have running under PIDs 10490, 10492, 10493?

Let me give you an example - let's say you have 100 pollers started but they are not utilized completely since you only have 10 items. So each poller grabs some tasks, processes them and sleeps until gets new tasks. But some pollers may not be receiving any tasks at all so they wait until MySQL wait_timeout is reached and you get the log line.

Bumping up the wait_timeout - I don't know it that is a good idea. Default 28800 seconds for MySQL should be enough. I'd check the poller graph first and see if the amount could be reduced.

If that is not the case, it will be interesting to do some further troubleshooting.

Best Regards,
Ingus

**aib** · 06-02-2015, 16:21

Thank you for answer, Ingus.
Right now I cannot say anything about that processes (10490, 10492, 10493) because zabbix server already restarted.

I will let you know later when I will have the same issue.

**mkash28** · 08-04-2016, 17:49

i had the same issue:

I had the same issue: Increasing following limits helped:
-/etc/my.cnf-
max_connections=1000 <-- higher then default
wait_timeouts=45000 <-- was 28800

- /etc/zabbix/zabbix_server.conf-

Timeout=30
StartPollers=500

restart mysqld
restart zabbix-server

Ad Widget

MYSQl Gone AWAY ??

MYSQl Gone AWAY ??

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment