***sorry for the repost, lost connection due to cloudflare, and seemed that i also lost a big part of my text and couldnt edit post***
Hey there, hope that someone can help me with some pointers where to look at, i don't have all the log files of the event, so i cant bee that specific.
We have a small zabbix server running for quite some time, its hosted on a Ubuntu 24.04.2 LTS; MySQL ; Zabbix version 7.0.7, Running as VM on a Windows Hyper-V.
About 140 hosts monitored; about 10.500 active items; with a nvps of about 50.
The server has been running fine, no problems at all, low io, nice usage, nothing to be concerend about.
But when i was on holiday, suddenly on Sunday the 19th of July roughly between 17:00 and 19:00 our server IO Spiked and basically never returned to normal again.
IO Wait times of 50% or more and that without stopping.
Stopping the zabbix server, didn't lower it instantly; eventually it would drop. Shutting down MySQL; would stop the IO load.
Starting the MySQL server didn't start the IO spikes back again and would run normally.
After starting the zabbix server, the IO spike did return after some time. As far as i could see, the History syncer was busy more then other processes.
As far as i could see the log files from zabbix dont throw any real big issues or errors, not on startup or anything like that.
i did enable the slow log query and that does show some more queries, but i can't really tell if that causes it, or is a result of.
To the best of our knowledge in the recent past before the event, no new hosts have been added, no old hosts have been removed,
and no real changes where made. Even the host system was let alone, i mostly do those things, but not this time.
Does anyone have some clue's where to look at?
I don't think 'performance' is in place here, because of the sudden event and the load before was not that high in any way.
On the hardware host also everything seems to be normal, no broken disks, no array faults, only high IO usage since that event.
(the machine is hosting 1 other VM, but no that VM isn't causing the IO load on the main host).
Is there some option i could turn on to show more why or what is happening. In General we dont keep that much of a history, to make the server not too overloaded.
single items stored roughly 3days to max 7days (mostly depending on poll interval); trends stored mostly max 180days;
i added some screenshots, roughly 24hour graps, Prior to the event, during the event, after the event (with more detail) and after the mysql (+reboot) change.
Hope that someone can point me in some direction where to look at.
DB backup's are done 4 times a week, zabbix server is shut down, DB backup is taken, mysql is restarted, zabbix server is start up again.
This is also running for some years and was not running during this event, that would run at night time.
Hey there, hope that someone can help me with some pointers where to look at, i don't have all the log files of the event, so i cant bee that specific.
We have a small zabbix server running for quite some time, its hosted on a Ubuntu 24.04.2 LTS; MySQL ; Zabbix version 7.0.7, Running as VM on a Windows Hyper-V.
About 140 hosts monitored; about 10.500 active items; with a nvps of about 50.
The server has been running fine, no problems at all, low io, nice usage, nothing to be concerend about.
But when i was on holiday, suddenly on Sunday the 19th of July roughly between 17:00 and 19:00 our server IO Spiked and basically never returned to normal again.
IO Wait times of 50% or more and that without stopping.
Stopping the zabbix server, didn't lower it instantly; eventually it would drop. Shutting down MySQL; would stop the IO load.
Starting the MySQL server didn't start the IO spikes back again and would run normally.
After starting the zabbix server, the IO spike did return after some time. As far as i could see, the History syncer was busy more then other processes.
As far as i could see the log files from zabbix dont throw any real big issues or errors, not on startup or anything like that.
i did enable the slow log query and that does show some more queries, but i can't really tell if that causes it, or is a result of.
To the best of our knowledge in the recent past before the event, no new hosts have been added, no old hosts have been removed,
and no real changes where made. Even the host system was let alone, i mostly do those things, but not this time.
Does anyone have some clue's where to look at?
I don't think 'performance' is in place here, because of the sudden event and the load before was not that high in any way.
On the hardware host also everything seems to be normal, no broken disks, no array faults, only high IO usage since that event.
(the machine is hosting 1 other VM, but no that VM isn't causing the IO load on the main host).
Is there some option i could turn on to show more why or what is happening. In General we dont keep that much of a history, to make the server not too overloaded.
single items stored roughly 3days to max 7days (mostly depending on poll interval); trends stored mostly max 180days;
i added some screenshots, roughly 24hour graps, Prior to the event, during the event, after the event (with more detail) and after the mysql (+reboot) change.
Hope that someone can point me in some direction where to look at.
DB backup's are done 4 times a week, zabbix server is shut down, DB backup is taken, mysql is restarted, zabbix server is start up again.
This is also running for some years and was not running during this event, that would run at night time.
Code:
[mysqld] user = mysql bind-address = 127.0.0.1 mysqlx-bind-address = 127.0.0.1 key_buffer_size = 16M myisam-recover-options = BACKUP innodb_buffer_pool_size=4G innodb_flush_log_at_trx_commit=2 log_error = /var/log/mysql/error.log slow_query_log = 1 slow_query_log_file = /var/log/mysql/mysql-slow.log long_query_time = 2 max_binlog_size = 100M binlog_expire_logs_seconds = 604800
Code:
ListenPort=10051 LogType=file LogFile=/var/log/zabbix/zabbix_server.log LogFileSize=5 DebugLevel=3 PidFile=/tmp/zabbix_server.pid DBHost=localhost DBName=zabbix DBUser=zabbix DBPassword=supersecret DBPort=3306 StartPollers=10 StartIPMIPollers=0 StartPreprocessors=6 StartPollersUnreachable=10 StartTrappers=6 StartPingers=30 StartDiscoverers=1 StartHTTPPollers=2 StartTimers=2 StartEscalators=2 StartAlerters=2 StartJavaPollers=0 StartVMwareCollectors=0 SNMPTrapperFile=/opt/zabbix/tmp/zabbix_traps.tmp StartSNMPTrapper=1 ListenIP=0.0.0.0 HousekeepingFrequency=1 MaxHousekeeperDelete=5000 CacheSize=64M StartDBSyncers=2 HistoryCacheSize=512M HistoryIndexCacheSize=64M Timeout=30 TrapperTimeout=300 UnreachablePeriod=35 UnavailableDelay=60 UnreachableDelay=15 ExternalScripts=/opt/zabbix/share/zabbix/externalscripts FpingLocation=/usr/sbin/fping Fping6Location=/usr/sbin/fping6 LogSlowQueries=3000 TmpDir=/tmp StartProxyPollers=1 ProxyConfigFrequency=3600 ProxyDataFrequency=60 AllowRoot=0 User=zabbix-srv
Comment