Problem zabbix history syncer processes

[wprado]

I have a problem with the processes of history Syncer, normally start synchronizing the values in the database, but after a few minutes the timing of the process becomes increasingly slow, this is the queue server grow and shoot all triggers NODATA currently have to do a restart of Zabbix Server every 10 minutes.

Zabbix Server
Dell R420 Intel (R) Xeon (R) CPU @ 2.50GHz E5-2450 v2 - 32 processors, 128 GB of RAM
Zabbix Server 2.2.11 3800 nvps, 330895 items, 2952 hosts and 90000 triggers.

Database
Intel (R) Xeon (R) CPU @ 2.50GHz 24 0 E5-2640 processors, 64 GB of RAM
MySQL 5.6.19-log MySQL Community Server (GPL), partitioned tables history, history_unit, trends, trends_units
Disabled housekeeping, three months history of history and history_unit, 9 months of historical trends and trends_unit
*
Below the settings:

zabbix_server.conf
############ GENERAL PARAMETERS ################

LogFile = / var / log / zabbix / zabbix_server.log
DebugLevel = 3
#include = / Opt / zabbix-server / server_include
PidFile = / var / run / zabbix / zabbix_server.pid
FpingLocation = / usr / sbin / fping
ExternalScripts = / usr / lib / zabbix / externalscripts
AlertScriptsPath = / usr / lib / zabbix / alertscripts
Timeout = 30
LogFileSize = 1
ListenIP = 0.0.0.0
ListenPort = 10051
AllowRoot = 0
############ DATABASE PARAMETERS ################

Dbhost = IP
DBName = zabbix
Dbuser = zabbix_app_usr
DBPassword = PASSWORD
DbPort = 3306

############ ADVANCED PARAMETERS ################
StartPollers = 20
StartPollersUnreachable = 50
StartTrappers = 700
StartPingers = 20
StartDiscoverers = 50
StartHTTPPollers = 50
CacheSize = 4G
StartDBSyncers = 80
HistoryCacheSize = 2G
TrendCacheSize = 2G
HistoryTextCacheSize = 2G
ValueCacheSize = 20G
SenderFrequency = 30
UnreachablePeriod = 45
UnavailableDelay = 60
UnreachableDelay = 60
StartTimers = 500

# VMWare Tunning
StartVMwareCollectors = 100
VMwareCacheSize = 128M
VMwareFrequency = 60


Database

my.cnf

[client]
port = 3306
socket = /var/run/mysql/mysql.sock
# disable-auto-rehash = 1
# default-character-set = utf8
#reconnect = 1
#WAIT


[mysqld]
port = 3306
socket = /var/run/mysql/mysql.sock

bind-address = IP
datadir = / var / lib / mysql / datadir

federated

lower-case-table-names
skip-name-resolve
transaction-isolation = READ-COMMITTED
binlog_format = ROW

log-slave-updates
slave-skip-errors = 1062,1452,1032,1236

relay-log = napsao-nix-mysql-zBX-3-relay-bin

log-error = / var / lib / mysql / log / napsao-nix-mysql-zBX-5.err
log-bin = / var / lib / mysql / bin_log / napsao-nix-mysql-zBX-3
# slow-query-log = / var / lib / mysql / log / napsao-nix-mysql-zBX-5_slow_query.log
tmpdir = / var / lib / mysql / temporary

max_binlog_size = 8MB

wait_timeout = 31104000
interactive_timeout = 31104000

key_buffer = 256M
max_allowed_packet = 51200M
sort_buffer_size = 4M
read_buffer_size = 4M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 32M
thread_cache_size = 50
query_cache_size = 128M

thread_concurrency = 4

server-id = 5
auto_increment_increment = 2
auto_increment_offset = 2

innodb_flush_method = O_DIRECT
innodb_data_home_dir = / var / lib / mysql / innodb /
innodb_file_per_table
innodb_file_io_threads = 18
innodb_flush_log_at_trx_commit = 2
innodb_log_group_home_dir = / var / lib / mysql / redolog /
innodb_buffer_pool_size = 42062M
innodb_log_file_size = 50M
innodb_log_buffer_size = 8M
innodb_lock_wait_timeout = 500

max_connections = 5000

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
in the self-rehash

[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 64M
sort_buffer_size = 64M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

Attached some zabbix performance charts.

Can anyone help me with the problem?

[ingus.vilnis]

Hello and welcome to Zabbix forums!

Here is what could possibly be causing this:

• Disk performance is hitting he limit
• Network performance to DB is bad or limited
• DB tuning can be improved

Few ideas for you where to start:

• Check the disk performance with any of the generic tools like "atop", "sar" etc.
• Check the network performance.
• See if you can find anything useful here regarding DB tuning
• https://www.percona.com/blog/2014/11...-mysql-zabbix/
• https://www.zabbix.com/forum/showthread.php?t=9925

As for Zabbix settings (zabbix_server.conf):

• Enable LogSlowQueries=3000 to see which queries are slow and how bad this is.
• StartDBSyncers = 8 because each DB syncer is capable of processing ~1000 nvps so having more than you need might actually decrease the performance.

At least this could be the starting point. After that maybe you will actually find that the problem is hidden somewhere else.

Good luck with this!

Best regards,
Ingus