Integrating Zabbix with Zabbix

[brucebay]

I am trying to find the best way to send alerts from lower-environment zabbix servers to one centralized zabbix system that integrates all the alerts and events from all other zabbix servers that send alerts to it.
is it viable? can we send alerts or even metrics collected from multiple zabbix servers to a centralized zabbix server?
if yes, is there a guide or instruction to do so?

thank you

[tim.mooney]

There are certainly tools like zabbix_sender to push arbitrary monitoring data into a Zabbix server, but that would still require that your "one ring to rule them all" Zabbix server would need to be a strict superset (as in set theory) of the hosts, items, triggers etc from the complete set of your clients. That's going to be incredibly unwieldy to maintain. What happens when one of the subordinate Zabbix servers gets some new hosts and items added?

My first thought is that if I just wanted a dashboard that showed a consolidated view of problems in multiple different Zabbix environments, I would build that dashboard in some external tool, and either have the subordinate Zabbix servers push just alert data to the central dashboard (via some kind of Webhook) or I would periodically poll each of the subordinate Zabbix servers via the API and look for problem events. Then your dashboard system doesn't need to know anything about the hosts or items collected from any of the subordinate environments.

Hopefully someone else here has a better idea, or has some ingenious method to avoid some of the problems I've outlined.

[brucebay]

Thank you for your comments and thoughts on this issue, Tim!
the point is there are values that are generated each second and get dumped into the /var/log/messages, I can poll and graph them using an external command script in a lower environment Zabbix server. for the sake of centralizing and support policies I need to send the alerts and events to the centralized Zabbix server (which cannot poll every second due to resource limitation). so I am looking for a way to send these alerts/events to that server from another zabbix machine.
can you help me understand how can I use Zabbix API for this purpose? or any other tool?
by the way, if I use Grafana for alerting, can I send alerts from grafana to a Zabbix Machine?

[tim.mooney]

Ok, the additional information you've provided changes things a little.

Rather than an external script and really frequent polling, why not investigate the log[] or logrt[] items (there's a Windows eventlog[] item that does something similar for Windows clients, if you have any of those that need the same treatment).

log[] and logrt[] items must be "Zabbix agent (active)" items, which moves the vast majority of the work onto the zabbix-agentd that would be running on the client. Your central server doesn't need to poll every second, because the agent is always watching the log, and when matches are detected, it pushes them to the Zabbix server (rather than requiring the Zabbix server to pull them). By doing it this way, you've offloaded most of the processing to the clients (and it's still not heavyweight there, depending on how fast the log file grows and how often the thing(s) you're interested in appear in the log).

In addition, if you have lots of clients in a few different datacenters, like if you're part of a multinational company and have servers to monitor in a few different worldwide locations, you can deploy Zabbix proxies to help offload some of the processing and make different datacenters a bit more resilient in the face of things like a brief network partition, etc.

Take a look at those two (unrelated) options. There's a good chance that they will meet your needs, without any need for using the API.

I've only used Grafana for very pretty graphs, not for alerting; my environment's one source of "problem" truth is Zabbix. It's possible to pull data out of Zabbix and display it in Grafana, but I don't know what would be involved in pushing data from Grafana to Zabbix. In my environment, we do collect some additional metrics using other tools (Graphite + collectd + telegraf, mainly) outside of Zabbix, primarily because it's much more efficient to store with Graphite than it is with Zabbix's relational DB, but that data is not for alerting.