Cisco Meraki dashboard by HTTP - Hitting API rate limit

[ottow]

We have a large Meraki infrastructure, and are trying to monitor using the official Cisco Meraki dashboard by HTTP template.
Unfortunately, it does not work very well. The template is not honoring the API rate limit (https://developer.cisco.com/meraki/api-v1/rate-limit/) and floods the API with queries, resulting in transient errors for every device similar to the one shown below.

We recently upgraded from 7.0.4 to 7.2.4, and hoped the issues had been fixed in the newer version but sadly not.

Does anyone have a solution for this?

[markfree]

The template doesn't seem to have a way to control the rate limit.
Perhaps you could try changing master items update intervals so that fewer requests are made to your meraki organization.

Or can you adjust the API rate limit?

[ottow]

We've already tried tweaking the update interval, it does not help unfortunately.
Merkai/Cisco offers no way to adjust the API rate limit, it is fixed and the same for everyone.

[markfree]

And did you try changing master items update intervals?

[ottow]

Yes, it makes no difference.

[markfree]

Does Cisco provide Meraki support for this?

[ottow]

Yes, please see https://developer.cisco.com/meraki/api-v1/rate-limit/.

[markfree]

I meant actual support that you can call for help.

Anyhow, I don't think the Zabbix template for Cisco Meraki generates that many requests per second.
Have you checked to see if other devices are querying your Meraki organization?
Maybe there's something else consuming your API requests.

[ottow]

Yes, there is Cisco support available to call, but they refer back to the application support (Zabbix) as the message "API limit exceeded for your organization" is a clear indication the app is causing the issue by sending too many requests.
There are no other services querying the Meraki API under our account.

FYI, monitoring thru Meraki API worked just fine when we was using New Relic, whose implementation honors the rate limit and the subsequent 429 response code to delay further queries.
The issues arose when we switched to Zabbix, and the provided Meraki template.
We have 800+ Meraki devices in our environment.
With 5 items each, that's 4000+ metrics being pulled by the template on each update.
The limit for the API is 10 calls/sec, with a burst of 30 for the first 2 seconds.

There needs to be a rate limit on the template, or a function to honor the 429 response code.

[markfree]

I'm not familiar with New Relic.

Anyway, Meraki template is actually a set of 3 templates. Check all master items intervals. Like I said,

Perhaps you could try changing master items update intervals so that fewer requests are made to your meraki organization.

Also, check the "Meraki device" macros for "timespan" and "interval". They may help you in this scenario.

[ottow]

Increasing the timespans and intervals makes the errors show up less frequently, but does not solve the issue. The Meraki API still gets overloaded with too many requests.
We also tested disabling the triggers for "There are errors in 'Get *' metric", which made visible 100's of false alerts for "Status is not online", for random devices. These where previously drowned out by the Get metric errors, and makes the entire solution quite unreliable.

The template, as currently designed, is not suitable for monitoring large-scale Meraki production environments.

[xaviermace]

While I 100% agree with the statement that the template really needs to be recoded to account for the rate limiting (and I've had this discussion with Zabbix support), I would also state that the out of box template is collecting a lot of info you probably don't care about and could get you under the rate limit if you disable it. We're a little over 1,000 Meraki devices right now but critically it is split between multiple Meraki orgs which means usually hitting the 100 requests per second per source IP limits rather than the 10 requests per second per Meraki organization limit. IE the Administrators Discovery and SAML Discovery are IMO largely wasted API calls, especially if you're not using SAML.

Obviously doesn't fix the core problem, which you're 100% right about but could at least bandaid it so it's still usable for you. It has for us at least. I'd also say Meraki customers need to fight this with Meraki too as 10 requests a second is a ridiculously low number for an organization of any size. Two items per device, 1,000 devices for a total of 2,000 requests means if you're a single Meraki org, that means you're taking over 3 minutes to complete a polling cycle for all your devices because of that stupidly low rate limit.

[ottow]

We did disable most of the Organization Discovery items, keeping only those we need but that clearly did not cut down the calls enough. All our devices are in one single Meraki org.

[markfree]

Is it possible to split your org into multiple ones?

[ottow]

No, splitting into multiple orgs is not an option for us.

[PedroDiogo]

Hello,
Anyone have a guide to configure Meraki integration?
I create an HOST, with Meraki Dashboard template by HTTP, define API Key Macro, but not working. Documentation appears to be simple but i can do it.

Anyone ca help me?

Thanks