Zabbix to monitor 5000+ Meraki devices and 1000+ vm on different cloud tenant

[nicfra]

Hello,

in our company we're thinking to migrate our monitoring system to Zabbix. In our system we've 5000+ Meraki devices and 1000+ vm on different cloud tenant, in different regions (Azure, AWS, GCP, Oracle; in EMEA, AMERICA, APAC).



First of all we're having problems with Meraki: i'm trying to monitor my Meraki organization with Zabbix using the template "Cisco Meraki dashboard by HTTP".

It started to discover my device but now it's stucked at 1000 devices since a couple of days.

In my organization I've more than 5000 devices that I would like to monitor.

I've read that there're some limitations due to how much info the API dumps out and some problem can occurs if you've more than 1000 devices, but I didn't find nothing official so I would like to understand if it's true and if there's a solution.

Maybe if it's possible to solve this working with Meraki support on the backend because I know that each Meraki organization has a call budget of 10 requests per second and also, a concurrency limit of 10 concurrent requests per IP is enforced; but I didn't understood if the limitation of discovering new devices is related to this limit.

The power of this template should be that just putting my Meraki API key automatically all my network devices will be discovered and monitored at once, but seems like it's stucked at 1000 device.



Also I've tryed to work with the template "Azure Virtual Machine by HTTP" and "AWS EC2 by HTTP" but it seems to be required a "resource ID" or "EC2 instance ID" that is unique for every virtual machine. In this way if I want to monitor my 1000+ virtual machine I should manually find the resource id from Azure and manually put it inside Zabbix.

I'm wondering if there's a way, just like with Meraki template, to let Zabbix gather all the data from something like the "subscription ID", or to automatize this process with a script, or something similar; because doing it for 1000+ vm will be really time expensive.

I would also like to understand if there's an integration of Zabbix also for Oracle and GCP to achieve this task, to automatically monitor and discover all my vm/db that belongs to the different clouds; because I don't want to manually configure the monitoring for 1000+ hosts.



Thanks in advance for your support!

[Hamardaban]

To understand why all devices are not detected, you need to understand HOW detection works. There are many ways to detect and you know better how it is done in the template you are using.
I will say one thing - there are no built-in restrictions on all sorts of “quantities”.
IMHO before working with a large system, you need at lees to read the documentation, but it's better to take zabbix courses.

[cyber]

As you say, "there might be a restriction in API.." That is the Meraki API, so you need to dig into that part in Meraki docs..
You may need to look into initial script that retrieves data, reconstruct steps and API calls manually and maybe try (with Postman for example) to get the data and see, if these limits are true...
Debugging is not a simple task..

[frede]

The Meraki API have rate limits.

The Zabbix Meraki Templates does not respect these limits. - you need to fix the templates and scripts yourself. or you get a lot of errors in zabbix.


From: https://developer.cisco.com/meraki/api/rate-limit/

Per organization


Each Meraki organization has a call budget of 10 requests per second, regardless of the number of API applications interacting with that organization.

• A burst of 10 additional requests are allowed in the first second, yielding a maximum of 30 requests in the first 2 seconds.
• The rate limiting technique is based off of the token bucket model.

This budget is shared across all API applications in the organization that leverage API authentication. You can check the recent API activity for the given organization to understand if you are sharing the budget with other applications. Per source IP address


Each source IP address making API requests has a call budget of 100 requests per second, regardless of the number of API clients working from that IP address.

[nicfra]

Hello,

thank you all for your answers.
This seems related to the pagination limit on meraki, that is limiting the output to 1000 entries.
I'm working to try to solve this following this article: https://developer.cisco.com/meraki/a...on/#pagination
Specifically I'm trying to work with the template "Cisco Meraki organization by HTTP"; I've tryed to modify the item "meraki: Get list of the networks" that contains a script inside it; i've changed my function adding: url += '?total_pages=all'; or adding: url += '?total_pages=-1'; but none of them was working.

I've tryed to change this part of the script, the rest is all the dafult of the template:

function getHttpData(url) {
url += '?total_pages=all';
response = request.get(url);
Zabbix.log(4, '[ Meraki API ] [ ' + url + ' ] Received response with status code ' + request.getStatus() + ': ' + response);

Do you have any suggestion about it?


Any input for the second question?

"Also I've tryed to work with the template "Azure Virtual Machine by HTTP" and "AWS EC2 by HTTP" but it seems to be required a "resource ID" or "EC2 instance ID" that is unique for every virtual machine. In this way if I want to monitor my 1000+ virtual machine I should manually find the resource id from Azure and manually put it inside Zabbix.

I'm wondering if there's a way, just like with Meraki template, to let Zabbix gather all the data from something like the "subscription ID", or to automatize this process with a script, or something similar; because doing it for 1000+ vm will be really time expensive.

I would also like to understand if there's an integration of Zabbix also for Oracle and GCP to achieve this task, to automatically monitor and discover all my vm/db that belongs to the different clouds; because I don't want to manually configure the monitoring for 1000+ hosts."

Thank you

[cyber]

"Azure by HTTP" contains multiple different discoveries, including VM disco. It should find your subscription based VM-s and attach that "Azure Virtual Machine by HTTP" to each of them..