Zabbix critical issue, agents keep crashing

[ortz]

Hi,

In the last couple of days from time to time suddenly all of the agents connected to Zabbix server crash.
I used to have this problem and I migrated to remote database with better performance, couple of days ago I returned to the old configuration with server & database on the same machine (but I upgraded hardware).

Zabbix configuration stands for single zabbix server with 2 CPUs with 17.5GB RAM and IO-Optimized disk for the MySQL database.

When the problem occurs I see a lot of the following lines in Zabbix-server.log:

24405:20131014:063109.598 Zabbix agent item [vfs.fs.size[/,free]] on host [SERVER1] failed: another network error, wait for 15 seconds

I noticed these couple of lines in the Zabbix-server.log also:

24457:20131014:062004.375 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update triggers set lastchange=1381724210,value=0 where triggerid=14565;
24405:20131014:062345.022 [Z3005] query failed: [2006] MySQL server has gone away [begin;]

But in mysql.log everything seems just fine.

Zabbix status and some parameters:


List of issues displayed in Zabbix:


List of graphs from Zabbix Server template (the problem starts around 6:20):



zabbix_server.conf

ListenPort=10051
LogFile=/var/log/zabbix/zabbix_server.log
PidFile=/var/run/zabbix/zabbix_server.pid
DBHost=localhost
DBName=zabbix
DBUser=USERNAME
DBPassword=PASSWORD
StartPollers=50
StartPollersUnreachable=5
HousekeepingFrequency=12
MaxHousekeeperDelete=20000
DisableHousekeeping=1
CacheSize=256M
CacheUpdateFrequency=90
StartDBSyncers=12
HistoryCacheSize=128M
TrendCacheSize=16M
HistoryTextCacheSize=16M
SSHKeyLocation=/usr/src/zabbix
Timeout=30

Note that I don't see any problem within the MySQL Log.
my.cnf (MySQL Configuration)

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
query_cache_size=128M
join_buffer_size=256K
table_cache=256
innodb_buffer_pool_size=8G
max_connections=256
wait_timeout=14400
interactive_timeout=14400
thread_cache_size=8
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

We are really frustrated from this issue, if anyone could help it would be great.

Thanks in advance,
Or.