Race conditions in calculated items?

[vanessa]

So, I have this setup.

Item A: walk[oid]
Item B: is dependent on item A, and extracts a particular value from the walk in item A.
Item C: is also dependent on item A, and also extracts another particular value from the walk in item A.
Item D: is a calculated item, equals B+C.

It is important that D is consistent, meaning: values of B+C must come from the same walk moment. If the timing is a bit wrong when D is evaluated (due to delays and what not), then B and C may be the results of two different walks.

How do you resolve this in Zabbix?

[markfree]

The "walk" item should receive the SNMP values concatenated as list.
I believe that the master item should capture the value and send it to the dependent items pretty much at the same time. So, item B and C should have the same timestamp.

[vanessa]

The "walk" item should receive the SNMP values concatenated as list.
I believe that the master item should capture the value and send it to the dependent items pretty much at the same time. So, item B and C should have the same timestamp.

I interpret the risk as unlikely but possible.

[markfree]

How tight is your timing tolerance?
Sec, Ms, ns?

You can see the actual item timestamp in Zabbix database.

[troffasky]

I am not sure that this is a Zabbix "problem". Zabbix asks the agent for the values, but all Zabbix can know is when it was sent the values, not at what point the agent collected them. Using a walk[] item is your best chance of getting values as close together in time as possible.

[ISiroshtan]

I think core of the question here is the fact that calculated items are checked on it's own timers and not tied to underlying items. So if any of dependent items pre-processing will be delayed due to pre-processing queue or just slowness of worker/system, it's possible that one item used in calculation would be already updated while other would not yet be updated, resulting in inconsistent value.

[vanessa]

What ISiroshtan said above. It can't be helped. Well, you can hack around and JavaScript it, but it does not have a first class solution, like in e.g. icinga.

[cyber]

So set you calculated item to be calculated on a schedule... m/1 for example.. that should (at least in my mind) reduce the possibility (to what I don't believe anyway) some more... taking one random time out and replacing it with fixed...

I dont know the internals,, but master and its dependents get same timestamps (also nanoseconds), it can be (and it can be also my imagination), that all that writing of data will take place all at once for master and dependents... so I have serious doubt you are able to hit that gap between 2 walks and get dfferent values from different walks.... And those are also kept in memory caches, so there can also be something that prevents calculations during cache updates...

just went and checked it from DB
this is one "master item"

select itemid,clock,ns from history_text where itemid = 10827315 limit 10;
  itemid  |   clock    |    ns
----------+------------+-----------
 10827315 | 1730160015 | 302458694
 10827315 | 1730160075 | 913172953
 10827315 | 1730160135 | 250103623
 10827315 | 1730160195 | 156043084
 10827315 | 1730160255 | 163160105
 10827315 | 1730160315 | 623419135
 10827315 | 1730160376 |  41944209
 10827315 | 1730160435 | 975299479
 10827315 | 1730160495 | 825321052
 10827315 | 1730160555 | 425507835

and this is one of dependents, getting value from above item...

select itemid,clock,ns from history where itemid = 10827317 limit 10;
  itemid  |   clock    |    ns
----------+------------+-----------
 10827317 | 1730160015 | 302458694
 10827317 | 1730160075 | 913172953
 10827317 | 1730160135 | 250103623
 10827317 | 1730160195 | 156043084
 10827317 | 1730160255 | 163160105
 10827317 | 1730160315 | 623419135
 10827317 | 1730160376 |  41944209
 10827317 | 1730160435 | 975299479
 10827317 | 1730160495 | 825321052
 10827317 | 1730160555 | 425507835
​

[ISiroshtan]

I'm not sure if it's possible or not. On my current workplace there are stories of pre-processing randomly delaying data (presumably was happening on older Zabbix versions), but I never was able to witness this marvel so I see it mostly as gremlin tales

That said, I never had described problem with dependent items (tho I never had situation where I would be combining dependent and calculated items together while having potential 1 min skew on one of the values as source of false-positive alert). But I did have this kind of issue with dependent items and trigger. I would split master item into 2 different items and both of them are to be used in calculation inside of a trigger. Because whenever item inside of trigger is updated with new value trigger is recalculated, it did trigger false-positives with 0 duration. When 1st item updated - trigger fired - 2nd item updated - trigger closed. That was on Zbx 4.x, have no idea if internal logic was changed anywhere and whether it's still possible for same to happen on modern versions.

[cyber]

yes.. having 2 different (dependent) items in one expression can do that, as each trigger is recalculatd when any of items gets new value... even if they are with same timestamp, still, each one gets new value-> recalculate... but calcualted item is different thing...

[ISiroshtan]

Fair enough
After your previous message I'm somewhat curious to add javascript preprocessing into dependent item with something like "sleep 2" (essentially just delaying pre-processing) and see if dependent item timestamp would be different from master item or not... but not interested enough to actually spin up the lab env for it