Throttling Item Checks.
I am looking for a (hopefully simple) way to throttle, or stagger item checks over time... Reason being, we have some new firewalls that are not happy with the amount of checks that are being sent...
So for example... Currently if I enable some basic items (interface, memory, CPU etc.) I end up with 366 items that are being checked every 60 seconds.... At the same time if I set up a continuous ping normal latency is <1ms, but every 60 seconds I have 1 or 2 pings with a latency of, on average, ~600-700ms.... Which we have decided is acceptable to collect the data we have deemed critical.... However, if I enable more checks, say my total is now 1000 checks, then those 2 high latency pings now become 10 to 15... Every 60 seconds....
Now, just to be clear, this is 100% a Cisco issue in my opinion... But Cisco has so far proved less than helpful (shocking, I know), so I am looking for an alternative solution.
My troubleshooting so far has shown me that Zabbix is configured to send all these checks at the same time, every sixty seconds. What I am hoping to accomplish is to essentially spread this out over the 60 seconds.... So for example... If I have 60 checks, rather than sending them all at once, I'd like to send 1 per second... Or if 120 checks, 2 per second... The idea being to spread the latency out over the entire 60 seconds rather than the first 10 to 15 seconds....
I know that this is somewhat possible using custom intervals, but when several items are created via LLD I still end up with large groups of checks being performed all at once, rather than spreading them out....
I have also considered attempting to script this, modifying custom intervals based on the number of checks and the update interval, but am unsure if I'll be able to modify checks created via a templates LLD.
Any thoughts or ideas are much appreciated....
So for example... Currently if I enable some basic items (interface, memory, CPU etc.) I end up with 366 items that are being checked every 60 seconds.... At the same time if I set up a continuous ping normal latency is <1ms, but every 60 seconds I have 1 or 2 pings with a latency of, on average, ~600-700ms.... Which we have decided is acceptable to collect the data we have deemed critical.... However, if I enable more checks, say my total is now 1000 checks, then those 2 high latency pings now become 10 to 15... Every 60 seconds....
Now, just to be clear, this is 100% a Cisco issue in my opinion... But Cisco has so far proved less than helpful (shocking, I know), so I am looking for an alternative solution.
My troubleshooting so far has shown me that Zabbix is configured to send all these checks at the same time, every sixty seconds. What I am hoping to accomplish is to essentially spread this out over the 60 seconds.... So for example... If I have 60 checks, rather than sending them all at once, I'd like to send 1 per second... Or if 120 checks, 2 per second... The idea being to spread the latency out over the entire 60 seconds rather than the first 10 to 15 seconds....
I know that this is somewhat possible using custom intervals, but when several items are created via LLD I still end up with large groups of checks being performed all at once, rather than spreading them out....
I have also considered attempting to script this, modifying custom intervals based on the number of checks and the update interval, but am unsure if I'll be able to modify checks created via a templates LLD.
Any thoughts or ideas are much appreciated....