[Z3005] query failed: [2013] Lost connection to MySQL server during query

[whit3h0rse]

Recently I have upgraded Zabbix from 2.2.5 to 2.4.2 and since than I have noticed mysql error in log that seams to me doesn't not affect Zabbix in visible way - there is no DATA loss in items or crashes of server.

Here is a example of mentioned error:
4855:20141214:180748.245 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select hostid,key_,state,evaltype,formula,error,lifetime from items where itemid=1273939]
5109:20141214:180811.509 [Z3005] query failed: [2013] Lost connection to MySQL server during query [begin;]

I have tried everything with mysql twaeks like:
connect_timeout = 60
wait_timeout = 28800

etc. , but with no luck, server perfomance is ideal, pollers are below 60%, server load is 0.2...


MY mysql confg:
# MyISAM #
key-buffer-size = 2M
# CACHES AND LIMITS #
tmp-table-size = 96M
max-heap-table-size = 96M
query-cache-size = 96M
query-cache-limit = 6M
query_cache_type = 1
max-connections = 1350
connect_timeout = 60
wait_timeout = 28800
max_connect_errors = 1000
table_cache = 4096
thread-cache-size = 48
table-definition-cache = 2048
table-open-cache = 2048
thread_concurrency = 4
sort_buffer_size = 768k
join_buffer_size = 512k
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb_file_format = Barracuda
innodb-log-file-size = 128M
innodb-log-buffer-size = 128M
innodb-flush-log-at-trx-commit = 2
innodb-file-per-table = 1
innodb_support_xa = 0
innodb-buffer-pool-size = 4000M
innodb_lock_wait_timeout = 200
innodb_old_blocks_time = 1000
innodb_open_files = 2048
innodb_stats_on_metadata = OFF
innodb_additional_mem_pool_size = 16M


So I was wondering is this error affect Zabbix in any way? Can I safley ignore that error?
Does anyone know what changed in new Zabbix that can cause such error?

[kloczek]

1. Check mysqld logs
2. When you see lost connection try to telnet on port 3306 of your mysql DB backend

You may have some network issue.

[whit3h0rse]

1. Check mysqld logs
2. When you see lost connection try to telnet on port 3306 of your mysql DB backend

You may have some network issue.

I already checked mysql log and it is always emty, also I have tried to spam constatnt connection from Zabbix server to Mysql server and it always connect on port 3306.

Error is spaming me every minute, and it is always the same, just diffrent item ID:

[2013] Lost connection to MySQL server during query [select hostid,key_,state,evaltype,formula,error,lifetime from items where itemid=XY
Now and than I get also:
[2013] Lost connection to MySQL server during query [begin;]

Usage of network link is fine, around 1% and latency is below 10ms so I don't know how to further troubleshoot this error. Can anyone advise me? Thx.

[whit3h0rse]

I have recompiled Zabbix and rebooted server but that didn't help.

Another clue; error is almost always present except when I restart Zabbix service, than log and queue of items is empty for exactly 40 minutes, after that queue starts growing and errors " [2013] Lost connection to MySQL server during query" starts spaming in log.

And that happens every time I restart Zabbix; that proves that it is not a network problem, it is something in Zabbix - can anyone make some connections out of this?

[jan.garaj]

Try:

max_allowed_packet=64M

This settings doesn't make sense for me (zabbix doesn't use ~64MB queries), but this one helped me.

Or you can live with this error. Zabbix server is reconnecting, so it's fine.

[whit3h0rse]

max_allowed_packet 64 Mbps is recommended for this kind of issue, I also tried everything else from 4Mbps to 256 Mbps and it does not fix the error.

I have upgraded Zabbix to 2.4.3 but the error still persists

[whit3h0rse]

Solved, it was problem on firewall, just moved databaze server to the same network!

[Lodazz]

I have the same issue, but I am using the MariaDB in the localhost and it is still failing to every minute.

slow query: 928.339728 sec, "select hostid,status,tls_accept,tls_issuer,tls_subject,tl s_psk_identity from hosts where host='Zabbix server' and status in (0,1) and flags<>2 and proxy_hostid is null"

[Lodazz]

I keep getting those two errors every few minutes:

[Z3005] query failed: [2013] Lost connection to MySQL server during query [select hostid,status,tls_accept,tls_issuer,tls_subject,tl s_psk_identity from hosts where host='Zabbix' and status in (0,1) and flags<>2 and proxy_hostid is null]

slow query: 928.408303 sec, "select hostid,status,tls_accept,tls_issuer,tls_subject,tl s_psk_identity from hosts where host='Zabbix' and status in (0,1) and flags<>2 and proxy_hostid is null"

I just did a fresh install with Zabbix in one CentOS 7 and MariaDB in another CentOS 7, but I still have the same exactly issues. All of them have resources more than enough, for now I just have those two hosts on Zabbix (Zabbix itself and the MariaDB server).