Zabbix processes are 100% busy

[hbm]

Hello,

I have a situation where most of the zabbix process will be too busy frequently and it create a large number of fake alerts at same time..

These are the mail process/cache related usage
----------------------------------------------------
process,poller,avg,busy]): 100 %
More than 150 items having missing data for more than 10 minutes. -- zabbix[queue,10m]): 178
zabbix[process,trapper,avg,busy]): 96.3 %
zabbix[vcache,cache,mode]): Low memory (1)
Zabbix history write cache, % free [wcache,history,pfree]): 24.91 %

The history Write cache is almost 100% most of the time. ...

Here is my server configuration
------------------------------------
LogFileSize=1024
DebugLevel=3
PidFile=/var/run/zabbix_server.pid
DBHost=XXXXX
DBName=XXXX
DBUser=XXXX
DBPassword=XXXX
DBPort=60000
StartPollers=100
StartIPMIPollers=5
StartPollersUnreachable=100
StartTrappers=30
StartPingers=50
StartDiscoverers=5
StartTimers=25
StartEscalators=5
StartVMwareCollectors=5
VMwareFrequency=60
VMwarePerfFrequency=60
VMwareCacheSize=256M
VMwareTimeout=90
StartSNMPTrapper=0
CacheSize=512M
HistoryCacheSize=512M
HistoryIndexCacheSize=512M
TrendCacheSize=512M
ValueCacheSize=512M
Timeout=30
TrapperTimeout=300
UnreachablePeriod=900
LogSlowQueries=30000
User=zabbix
TLSCAFile=/server-TLSCAFile.crt
TLSCertFile=/server-TLSCertFile.crt
TLSKeyFile=/server-TLSKeyFile.key

I checked the CPU/Memory of both zabbix and DB (Db2) servers and doesn't have any issue of resources contention

Any recommendation to this issue would be greatly appreciated..

Thanks
hbm

[glebs.ivanovskis]

Performance drops significantly when you have this:
zabbix[vcache,cache,mode]): Low memory (1)
You need to increase ValueCacheSize.

[hbm]

Thanks for the reply .. do you think any negative impact if I set the cache value for all these from 512MB to 1024MB ..?

CacheSize=512M
HistoryCacheSize=512M
HistoryIndexCacheSize=512M
TrendCacheSize=512M
ValueCacheSize=512M

The hard hit ones are value cache and history Write Cache .. I also see the pollar process topping 100%
Zabbix busy poller processes, in % (:zabbix[process,poller,avg,busy]): 100 %
Zabbix busy trapper processes, in % ([process,trapper,avg,busy]): 95.44 %

Any suggestions would be greatly appreciated .

Thanks ...

[glebs.ivanovskis]

I believe just increasing ValueCacheSize will resolve all your problems, do it ASAP. Increasing other caches shouldn't have any negative effect (except increased memory footprint), but, as I said, you don't really need it.

[hbm]

Hi Increased the cache values to 2GB and added some server memory as well.. Everything worked fine for about an hour then its stopped the data updates .. no alerts were sending and no data updates were happening .. I back out my changes I did and restarted the zabbix back.

dont know exactly why the zabbix data updates stoped .. any suggestion would be greatly appreciated.

[allexpetrov]

hi hbm!

Any errors in server log ?

Regards,
Alexey!

[hbm]

Not anything specific ...

[smuck]

I have same issue and it started after updating zabbix-server to 3.4.1.
Before update everything was running smooth and fast.

Trapper/processes/pinger is also 100% busy and zabbix server is switching between down and running and not getting values.

First i though it was because the history_uint table was too big (280GB). but after making new table with 0 row that didnt make any different.

StartPingers=550
StartPollers=450
StartTimers=default
MaxHousekeeperDelete=500000
CacheSize=1G
HistoryCacheSize=512M
HistoryIndexCacheSize=1G
TrendCacheSize=512M
ValueCacheSize=2G
Value per second 900, items 55000, hosts 1200, triggers 14000