PoC: Distributed Monitoring based on Low-Level-Discovery

[BDiE8VNy]

Lack of "Next generation Distributed Monitoring" to have at least one central Zabbix-Server that could correlate Events/Values from remote Zabbix-Server is a real pitty.

Recently I wondered how far one could get when using Low-Level-Discovery to achieve something like that. This post is about giving this a try.
Please be aware that the following scenario is just a Proof-of-Concept and not intended to be used seriously.

The aim

• One central Zabbix server (upstream or US)
• Multiple regular Zabbix server (downstream or DS)
• Propatation of only a selection of DS hosts, triggers and items to US Zabbix server
• Less as possible customization/coding
• Avoidance of DS customizations/extensions

The idea

• Discover DS hosts that are member of a dedicated host group. (Selection of hosts)
• Discover DS triggers belonging to discovered hosts. Discover only DS triggers that make use of items belonging to a dedicated application. (Selection of triggers)
• Discover DS items belonging to discovered hosts. Discover only DS items that belong to a dedicated application. (Selection of items)
• Update periodically values of DS triggers and items on US.

To keep things simple DS triggers are represented by dedicated items on US. Each of these item hold the respective DS trigger value.

The implementation

Template "Remoting-Zabbix server":

• LLD-Rule "Remote host discovery"
• Item "Remoting items update"
• Item "Remoting triggers update"

Template "Remoting-Zabbix host":

• LLD-Rule "Remote trigger discovery"
• LLD-Rule "Remote uint item discovery"

External checks (deposited on US Zabbix server):

• remoting.host.discovery
• remoting.item.discovery
• remoting.item.sender
• remoting.trigger.discovery
• remoting.trigger.sender

Process flow:

1. A host per DS Zabbix server is manually added on US Zabbix server and gets the template "Remoting-Zabbix server" assigned.
2. "Remote host discovery" adds hosts on US for each host found on DS that is member of a specific host group.
3. Each discovered host has the template "Remoting-Zabbix host" assigned.
4. "Remote trigger discovery" adds an item for each desired DS trigger to hold the current value (OK/PROBLEM, resp. 0/1). Beside of that a corresponding trigger is added with the same name of DS trigger that fires when the underlying item is in PROBLEM/1 state.
5. "Remote uint item discovery" adds an item for each desired DS item. Since value types can't be set by LLD macros, a dedicated LLD rule per item value type is needed.
6. "Remoting triggers update" queries all relevant DS trigger values and updates the corresponding items on US in one run (e.g. once per minute)
7. "Remoting items update" queries the last value of all relevant DS items and updates the corresponding items on US in one run (e.g. every five minutes)

Conclusion
Pros:

• Works almost out-of-the-box
• All values are queried at once and should provide a more or less consistent snapshot
• Items can be updated on a larger interval to be more space efficient on US.

Cons:

• Triggers have actually a relation to items but DS triggers are considered on per host basis. So one DS trigger may be based on items from several hosts but will result in a item/trigger pair per host on US.
• Since trigger severity in prototypes can't be set by LLD macros, US triggers have either a common severity or need separate discoveries like it is done for item discovery.
• Macros in trigger names might not be (correctly) resolvable since having no history available and being based on the origin trigger expression.
• Several details like item unit can't be propagated to US.

It was a nice trial but finally I believe the way to go (beside of (co-)sponsoring "Next gen DM") is by using the API for US side as well.
Only the API allows to create entities in an efficient, consistent and complete way.