Value Cache and Trend Write Cache always full

[abjornson]

Hello,

As my zabbix has grown, I've usually been able to monitor the zabbix server performance graphs, and adjust parameters to keep up. These two have me stumped.

For at least a month - I've noticed that both of these caches seemed to gradually fill over time.

Initially, Value cache would fill up and then hover around 15-20% full. After a zabbix server reboot - the cache would go back to empty, and then over the course of a week would gradually fill back up again.

I then started to see similar behavior with the trend write cache. Filled up gradually after a reboot...after about a week, stabilized and - stayed around 15-20% full.

This has recently started happening much more quickly where this same filling up of the caches now happens in only about 1.5 days after a reboot.

I know i can continue increasing cache sizes, I've done this several times. My concern, though, is that though this might make the filling of the cache size take longer - it seems like there would still be an issue of this "slow leak" filling up the caches.

Does anyone have any suggestions on what i might be able to tune to allow the value cache and trend write cache to "keep up" rather than always gradually filling up?

Thanks!

[kloczek]

Does anyone have any suggestions on what i might be able to tune to allow the value cache and trend write cache to "keep up" rather than always gradually filling up?

If you have history/trends cache usage or those caches are constantly not 100% free it means that zabbix server process is not able enough fast push data to DB backend. In other words your DB backend shows first symptoms that it is not enough fast for inserts queries.

[abjornson]

Thanks kloczek,

If my problem was slow db backend, wouldn't i see the problem in other ways too, like a deep queue of items to be processed?

If i look at my "zabbix server performance" graph, i see that the green graph of values processed is always much higher than the red zabbix queue graph.

[kloczek]

Thanks kloczek,

If my problem was slow db backend, wouldn't i see the problem in other ways too, like a deep queue of items to be processed?

If i look at my "zabbix server performance" graph, i see that the green graph of values processed is always much higher than the red zabbix queue graph.

If you are using mysql as DB backend you can try

$ while true; do mysqladmin processlist | grep insert; sleep 1; clear; done

or

$ watch "mysqladmin processlist | grep insert"

to observe low many inserts queries are running and up to how many seconds they are running.
You may try to have look on "mysqladmin processlist | grep -v Sleep" o check what kind of queries are running on your DB backend and how often on this you have running inserts.

[abjornson]

Got some advice today on IRC, and it seems like my ValueCache and TrendWriteCache values - though I'd increased from default a few times - my values were still considered "small"

The advice I got was to keep increasing them as needed, and this seems to be working.

When I reported this problem, my values were as follows:

• kernel.shmmax 128MB
• TrendWriteCache 4MB
• valueCacheSize 32MB

I upsized these three values to:

• kernel.shmmax 512MB
• TrendWriteCache 16MB
• valueCacheSize 128MB

They seem to be holding relatively steady now. All my shared memory configuration was done based on:

https://www.zabbix.org/wiki/How_to/configure_shared_memory