zabbix cache values make sudden jumps and ... how to deal with unsupported triggers:

[db100]

hi there,

i am using zabbix 7.0 and sometimes valu cache makes very high jumps and never gets back to low values.

i have got like 1G of value cache available by server configuration and sometimes i see sudden jumps of like 30% occupation. After the cache has jumped it will never return back to lower values until the server saturates and i need to force restart.

This happens every 3-4 hours or so.

After a jump the diaginfo looks like this:


== value cache diagnostic information ==
Items:13801 values:27998336 mode:0 time:0.006725
Memory:
size: free:610164848 used:462866336
chunks: free:406 used:43986 min:32 max:610087632
buckets:
32:146
40:1
48:122
56:1
80:15
96:1
112:2
160:1
176:1
208:1
256+:115
Top.values:
itemid:48918 values:249517 request.values:3
itemid:48697 values:249443 request.values:3
itemid:48696 values:249436 request.values:3
itemid:51423 values:249188 request.values:3
itemid:51422 values:249180 request.values:3
itemid:48919 values:249164 request.values:3
itemid:51420 values:248776 request.values:3
itemid:51625 values:248674 request.values:3
itemid:48906 values:248442 request.values:3
itemid:48684 values:248316 request.values:3
itemid:48694 values:248186 request.values:3
itemid:51410 values:248161 request.values:3
itemid:49476 values:248048 request.values:3
itemid:49475 values:248017 request.values:3
itemid:51626 values:247932 request.values:3
itemid:51629 values:247906 request.values:3
itemid:51277 values:247758 request.values:3
itemid:51613 values:247709 request.values:3
itemid:51426 values:247676 request.values:3
itemid:48700 values:247270 request.values:3
itemid:48922 values:247105 request.values:3
itemid:49463 values:246921 request.values:3
itemid:51265 values:246782 request.values:3
itemid:51278 values:246742 request.values:3
itemid:48702 values:246590 request.values:3
Top.request.values:
itemid:66278 values:3230 request.values:3148
itemid:71611 values:3166 request.values:3049
itemid:71609 values:2570 request.values:2535
itemid:71610 values:2094 request.values:2046
itemid:66279 values:2089 request.values:1989
itemid:66277 values:1689 request.values:1631
itemid:71612 values:1404 request.values:1366
itemid:66280 values:1334 request.values:1323
itemid:66276 values:1345 request.values:1298
itemid:23620 values:710 request.values:599
itemid:68061 values:305 request.values:263
itemid:68063 values:290 request.values:251
itemid:68062 values:244 request.values:218
itemid:68060 values:231 request.values:217
itemid:68064 values:190 request.values:168
itemid:68059 values:182 request.values:165
itemid:68058 values:144 request.values:122
itemid:52994 values:126 request.values:112
itemid:69278 values:121 request.values:108
itemid:69274 values:120 request.values:108
itemid:69282 values:119 request.values:108
itemid:69288 values:112 request.values:108
itemid:50848 values:127 request.values:107
itemid:69285 values:121 request.values:107
itemid:105745 values:117 request.values:107
==

as you can see there is definitely some issue here, since the items with most values in the cache are the ones accessed less frequently (only 3 times or so). and the counts of cached values are also suspiciously close to one another .

here you see an example of such jumps (light blue line):



As you can see from the diaginfo above, allo itemids are close to one another, as they belong to hosts that have been created at similar times. All triggers of theses hosts are in "unknown" state, because in their expressions there is an invalid expression (100 * null) which is there on purpose, to disable that trigger on the basis of a MACRO value.

I could not find anywhere in the doc the way Zabbix handles unsupported triggers so i am guessing here that they should be completely ignored. Or is the problem that they are not ignored and values are being cached for those triggers anyway ?



please let me know how i can further debug this quite annoying issues,

[cyber]

I cannot close anything.. you could use "set as answer" but as you commented on my useless post, then it would be wrong to set that post as answer.. Thats why it is always better to (quote and) answer instead of comment.. Funny feature of this forum software...

[PavelZ]

Checkout this bug report https://support.zabbix.com/browse/ZBX-25994 (already fixed in v7.0.10 from 25 Feb)

[db100]

finally !!!

i will try to update ASAP to the 7.0.10 version then

also: it is not clear from the upgrade documentation whether it is necessary to manually run the schema.sql migration for timescaledb within zabbix KUBERNETES installations.

i see that there is an initdb container that would create the db if it does not exist, but will it also run the migrations ? and if not where is the shcema.sql file located in the container ?

please this is a very much needed information from the documentation

cheers

[cyber]

You should not need any schema creation during upgrades. First start of new version server will perform all required DB upgrade steps... if there is any. It is default behaviour. Even when upgrading major versions

[db100]

unfortunately i did upgrare to 7.0.10 in general i have noticed an overall improvement (reduced mem consumptions) but the value cache issue still remains:



is there anything i could do further to debug this situation ? i will try to provide an updated diaginfo report as soon as i can

[db100]

i can confirm that the value cache is strangely populated for low-requested items. below the diaginfo. any more places i could look for more debugging info?

== value cache diagnostic information ==
Items:18970 values:48531969 mode:0 time:0.014523
Memory:
size: free:276927288 used:795528616
chunks: free:9309 used:71038 min:24 max:275388592
buckets:
24:1
32:515
40:5
48:337
56:4
64:5657
80:1
96:1
112:12
128:4
136:1
144:1
160:2
176:18
192:2
208:1
224:2
240:3
256+:2742
Top.values:
itemid:50569 values:293362 request.values:3
itemid:50583 values:293079 request.values:3
itemid:50570 values:292838 request.values:3
itemid:49580 values:292279 request.values:3
itemid:49039 values:292111 request.values:3
itemid:50584 values:292082 request.values:3
itemid:49038 values:292071 request.values:3
itemid:50582 values:291915 request.values:3
itemid:50581 values:291901 request.values:3
itemid:50585 values:291708 request.values:3
itemid:49581 values:291539 request.values:3
itemid:49031 values:291536 request.values:3
itemid:49032 values:291367 request.values:3
itemid:49030 values:291346 request.values:3
itemid:50565 values:291200 request.values:3
itemid:49586 values:290844 request.values:3
itemid:49034 values:290522 request.values:3
itemid:49033 values:290516 request.values:3
itemid:49035 values:290474 request.values:3
itemid:50940 values:290196 request.values:3
itemid:50934 values:289639 request.values:3
itemid:50935 values:289631 request.values:3
itemid:50931 values:289253 request.values:3
itemid:50933 values:289169 request.values:3
itemid:50941 values:289157 request.values:3
Top.request.values:
itemid:66278 values:3546 request.values:3480
itemid:71611 values:3153 request.values:2970
itemid:66279 values:2680 request.values:2642
itemid:71609 values:2580 request.values:2448
itemid:71610 values:2206 request.values:2084
itemid:66277 values:1646 request.values:1600
itemid:71612 values:1412 request.values:1365
itemid:66280 values:1321 request.values:1317
itemid:66276 values:1347 request.values:1299
itemid:23620 values:926 request.values:599
itemid:53520 values:401 request.values:384
itemid:53519 values:386 request.values:361
itemid:91429 values:267 request.values:274
itemid:91432 values:282 request.values:273
itemid:91430 values:288 request.values:267
itemid:91431 values:276 request.values:264
itemid:68061 values:240 request.values:238
itemid:68063 values:211 request.values:219
itemid:53521 values:187 request.values:204
itemid:68062 values:194 request.values:188
itemid:68064 values:179 request.values:187
itemid:68060 values:194 request.values:182
itemid:68059 values:174 request.values:159
itemid:68058 values:127 request.values:118
itemid:69278 values:128 request.values:108
==


here my current utilizations

[db100]

here the utilizations>

[db100]

will soon try to upgrade to v 7.0.14 where there should be more debugging material for the value cache. will then post more content. the issue is still relevant though

[cyber]

7.0.18 is out, why go to 7.0.14? Everything in .14 is also in .18 and maybe some possible bugs also fixed (or introduced.. )

[db100]

7.0.18 is out, why go to 7.0.14? Everything in .14 is also in .18 and maybe some possible bugs also fixed (or introduced.. )

i did upgrade to 7.0.18, lets see if that brings relief

[cyber]

It really does not help you, but I looked at config cache behaviour over last 3 months... no spikes ever... larger drop (from 28 to 24%) is from the moment, when I did upgrade from 6 to 7...

My gut feeling says (but that might be too hot curry aswell...), that config cache fluctuations are more likely connected to some discoveries, which find some things, then some of it gets removed some ends up in not supported in some reason...

[db100]

the flucutations are not on the config cache but on the value cache.

the problem stopped suddenly after the update from 7.0.10 to 7.0.18 i believe the issue might have been related to this: https://support.zabbix.com/browse/ZB...SRqNjAkbDAkaDA.

which was fixed in 7.0.14. The value cache was going wild on trapper items, so perhaps thats the reason.

[cyber]

Might as well be that... I never had those versions with a loaded server. so I never experienced such behaviour...

[db100]

ok how can i close this post ? is it possible in this blog ?

if i cannot, then please cyber go on and close it if you wish

[cyber]

I cannot close anything.. you could use "set as answer" but as you commented on my useless post, then it would be wrong to set that post as answer.. Thats why it is always better to (quote and) answer instead of comment.. Funny feature of this forum software...