History cache issues after removing an un-related process

[HaveDill]

In the past we previously had shared a server with the Zabbix database and a Splunk indexer. We finally moved the splunk indexer off this server and onto a newer one last week.

However, now my zabbix server seems to be unable to offload the history cache effectively. Causing a lot of headaches. I dont see any mysql errors that stick out and am not sure what too look into to diagnose this. Can anyone point me in the right direction?


Here's my configs.

LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=50
LogType=file

CacheSize=1G
HistoryCacheSize=512M
HistoryIndexCacheSize=256M
TrendCacheSize=256M
ValueCacheSize=512M

Timeout=30
#DisableHousekeeping=1
DebugLevel=4

#Too many syncers will choke the server
StartDBSyncers=4


#Trapper stuff
#These trappers also collect ACTIVE AGENT ITEMS!
StartTrappers=150
SNMPTrapperFile=/tmp/zabbix_traps.tmp



StartIPMIPollers=15
StartEscalators=4
StartPollers=275
#StartDiscoverers=5
StartHTTPPollers=5
StartPingers=100
StartPollersUnreachable=100
StartTimers=30
SenderFrequency=30

LogSlowQueries=5000

AlertScriptsPath=/etc/zabbix/alertscripts
ExternalScripts=/etc/zabbix/externalscripts

[kloczek]

Write caches are below 100% only when it is not possible instantly write data to DB backend.

[HaveDill]

My DBA has checked the database and sees no slow queries or locks occuring. We're puzzled. I even disabled 471 hosts, and still my cache doesn't seem to free up. My DB I/O CPU'wait times are below 1%

Attached is a screenshot after i disabled 471 hosts;

https://i.imgur.com/mH5OhBk.png

Here is the I/O stats i collect from the DB over the past week. I see no major issues off hand.

https://imgur.com/a/BcivCvl

[kloczek]

My DBA has checked the database and sees no slow queries or locks occuring. We're puzzled. I even disabled 471 hosts, and still my cache doesn't seem to free up. My DB I/O CPU'wait times are below 1%[/url]

CPU utilisation has nothing to do with SQL queries types stats or IO statistics.
I've not been asking did your DBA checked the database.
IO stats shows data for very short period of time.
If you have high history syncers utilisation it may mean that you have issue with inserts/update queries and/or write IO speed on storage layer.
Using in server setup StartPollers=275 means that you are monitoring your hoosts using passive agent setup and you have no separated collection monitoring data only over the proxy/proxies.
LogSlowQueries=5000 mans that for yu slow query it is anything which takes more than 5s which is relatively very long time.
StartDBSyncers=4 may suggest that you have only 2-4 CPU cores on the host with zabbix server. With StartTrappers=150 it may mean that you have here congestion trying to write over to few DB syncers processes.

[HaveDill]

So interestingly enough, as i rebuild my Physical DB onto a g8 from a g7, and have my current DB as a VM. The issues were identical. BUT then when i turned my zabbix debug mode down from 4, it seems im getting my performance back, and my write cache is able to free itself again.

I guess when the work occured last week, i had in my configs to change the debug level to 4, which in turn caused all these issues. So not a DB problem, which is good at least