Database stops updating

[mellis]

I am not sure if this is the right place but here goes.
I have setup a fresh Zabbix 3.4.3 on a CentOS 7.4 64bit, with PHP 5.6.31, MySQL 5.7.2. I use the yum repository. Did nto have any issues with this and the Zabbix Server started up, again no issues.

I imported my host and templates so I now have 420 host, 41832 item and 4630 triggers. I then edited the zabbix_server.conf to give it the processes and cache that I thought was in the range that should work. restarted the server and repeated a coupe time to tune it in. All looked good and soon was down to 8 triggers showing up. Then after a bout 20 mins it jumped up to 420 triggers. All services were still running and the database was still available. Did a zabbix_server restart and it dropped back to 8 triggers after 5 mins.

This keep happening each time I would make a small change in the zabbix_server.conf.

I turned on the MySQL logging and found this error:
Aborted connection 11861 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)

I run down this rabbit hole and made all kinds of adjustments to the MySQL. But have not really improved the system much.

In the server graphs after restart I see values, as the system runs they slowly start dropping out one by one after about 20 mins and by 30 mins there are no new updates.

Anyone have an idea where to go on this.

I have several 3.2.x systems running with about the same load with no issues.

[vesper1978]

Can you post the changes you made to your zabbix_server.conf

[mellis]

My Config Changes

Zabbix_Server.Conf

LogFileSize=4
DebugLevel=5

StartPollers=102
StartPreprocessors=72
StartPollersUnreachable=14
StartTrappers=64
StartEscalators=2
StartAlerters=4
MaxHousekeeperDelete=8400
CacheSize=72M
StartDBSyncers=24
HistoryCacheSize=64M
HistoryIndexCacheSize=32M
TrendCacheSize=42M
ValueCacheSize=64M
Timeout=30
TrapperTimeout=300

my.cnf

open_files_limit = 4800
max_connections = 500
innodb_log_buffer_size = 32M
innodb_log_file_size = 2047M
innodb_buffer_pool_instances = 18
innodb_buffer_pool_size=18G
max_allowed_packet = 1024M
join_buffer_size = 1M

log_error = /var/log/mariadb/mariadb.log
log_warnings = 2
log_error_verbosity = 3
slow_query_log
query_cache_size = 0
query_cache_type = 0

skip-name-resolve=1

[kaspars.mednis]

Here is similar issue..

https://www.percona.com/blog/2016/05/16/mysql-got-an-error-reading-communication-packet-errors/

another thing - you have in config

StartDBSyncers=24

try to decrease them... in most cases the default 4 DB Syncers is absolutely enough , too much dbsyncers can cause issues with db
not every time bigger is better

Regards,
Kaspars

[mellis]

Result from reducing StartDBSyncers

I did reduce the StartDBSyncers to 4 the system stayed up for about 40 mins.

Attached is some graphs of the processes at the time of failing.

[kaspars.mednis]

Sorry, but i can't see any attached graphs

can you attach /var/log/mariadb/mariadb.log ? maybe it shows some error messges

and zabbix graphs for 1 day period Zabbix cache usage, Zabbix internal procesess busy may help a lot

Regards,
Kaspars

[mellis]

Uploads

I am have an issue with the uploads, seems I am exceeding my quote
But have 0 files uploaded.

OK got one graph and the mariiadb.log up there.

[kaspars.mednis]

Whats the name of the graph ?
what is the legend for red and violet lines ?

Kaspars

[mellis]

Graph

This is the internal Processes.

violet is the configuration syncer
red is the preprocesser manager

I believe, sorry for the bad pics,

[kaspars.mednis]

Anyway, all those processes must be less than 75% busy, better keep them 30%-50% busy max

Try to increase corresponding process count in zabbix_server.conf, and restart the zabbix-server process

Will look at your sql log little later

Kaspars

[mellis]

Processes hitting 100%

I have been able to get most of the processes down under 75%, but the configuration syncer keeps jumping to 100%

[kaspars.mednis]

Hello !

What is the current value of wait_timeout in MySQL ?

show global variables like 'wait_timeout';

if it's default 28800, try to increase to 86400 in MySQL config

wait_timeout = 86400

Regards,
Kaspars

[vesper1978]

I'd suggest reading https://www.percona.com/blog/2016/05...packet-errors/

as you are getting a lot of:

2017-10-27T14:03:08.257149Z 308 [Note] Aborted connection 308 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.257236Z 313 [Note] Aborted connection 313 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.258679Z 279 [Note] Aborted connection 279 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.261161Z 271 [Note] Aborted connection 271 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.263547Z 280 [Note] Aborted connection 280 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.263818Z 302 [Note] Aborted connection 302 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.264889Z 382 [Note] Aborted connection 382 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.265597Z 384 [Note] Aborted connection 384 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.268844Z 385 [Note] Aborted connection 385 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)
2017-10-27T14:03:08.269272Z 390 [Note] Aborted connection 390 to db: 'zabbix' user: 'zabbix' host: 'localhost' (Got an error reading communication packets)

[mellis]

Problem Solved

I want to thank everyone for the help. After much database tuning and hair pulling it turned out to be a faulty cable in the datacenter.

Again thanks to everyone.