Monitoring Multiple IPs on a Single Host with Circular Trigger Dependencies

[Marland]

VERSION: I’m currently using Zabbix Server version 4.4.10.


WHAT: I’m monitoring up/down on a client’s perimeter firewalls, and the on call FW Engineers have been getting escalations for FW down when it’s a circuit issue. I’m trying to find a way to adapt our monitoring to cut down on these false positives.


HISTORY: In the past it’s been set up to ping the external IP of the firewall, and the gateway router outside that firewall. Then a trigger dependency was created so if the gateway router is down then the firewall trigger is suppressed. There have recently been some architectural changes with my client that made this setup obsolete.


PLAN: My thought is to set up ping monitors for the internal and external interfaces of the firewalls, and to only alert if both interfaces become unreachable. The logic being that if one interface is up then the firewall is up, and the outage is most likely a circuit issue.


TESTING: I’ve done some testing with a single multi-homed host to see how this might work.

• First I tried adding a second ping monitor to the same firewall Host in Zabbix. I cloned the existing check, and renamed it to ping-monitor-2. When I clicked ‘Add’ I got the following error:

ERROR: Details Cannot add item

Item with key "icmpping[,4]" already exists on "FW1”

• I decided to set up 2 hosts in Zabbix (FW1-EXT, FW1-INT).
• Then I set up a trigger to fire an alert notification if the interface is unreachable for 5 minutes.
• The next step was to create trigger dependencies for both interfaces. (The plan was to make each trigger dependent on the other, and this is where it errors out. Error listed below.)

Example:

Host 1
Name: FW1-EXT
Items: Ping Monitor (Simple check, icmpping[,4])
Triggers: {HOST.NAME} ICMP Availability
Expression: icmpping[,4].avg(#5)}=0
Dependencies: FW1-INT icmp availability

Host 2
Name: FW1-INT
Items: Ping Monitor (Simple check, icmpping[,4])
Triggers: {HOST.NAME} ICMP Availability
Expression: icmpping[,4].avg(#5)}=0
Dependencies: FW1-EXT icmp availability

ERROR: Details Cannot update trigger
Cannot create circular dependencies.

I have been searching these forums, Reddit, and Google, but I haven’t found any way to set this up the way I need to. I may be overthinking it, or maybe there’s another way I’m not familiar with. Any advice on how to get this working would be appreciated.


Thanks,
Marland

[Markku]

Hi, what is the scenario where FW1-EXT is not pinging but FW1-INT is pinging?

Markku

[Marland]

Hello,

FW-EXT not pinging: Internet circuit is down.
FW-INT is pinging: MPLS circuit is up.

Thanks

[Noobz]

"
ERROR: Details Cannot add item

Item with key "icmpping[,4]" already exists on "FW1”
"
... just read the words; change the key of the second ping monitor, preferably so the index is the index of the interface you are pinging.

I'm really not sure what the question is - you want to know if the device is reachable at all, or only via one or the other IP?

[Marland]

Hey Noobz Thanks for the response. You jogged my memory that the first parameter in the icmpping key is '<target>'. Once I realized that I was able to get two ping monitors set up on one host, and that allowed me to set up a trigger condition using AND to account for both monitors.

Fwiw My question was something like "How best can I configure my Zabbix Server so that I can both the internal and external interfaces of the firewall, and to fire an alert only in the case that both are unreachable?"

I now have it working. Thank you!

[[email protected]]

Hi Marland,

If there are 2 IP addresses for the firewall (i.e. Internal and External), I would make the External dependent on the Internal.

ICMP Ping dependencies I think should be defined in order of network connectivity path from the monitoring server i.e. to Zabbix -> Internal -> External?

Either way, creating the dependencies like mentioned (FW1-EXT is dependent on FW1-INT) will produce one alert in order of the defined preference:

* FW1-INT=down, FW1-EXT=up results in one event for FW1-INT
* FW1-INT=down, FW1-EXT=down results in one event for FW1-INT
* FW1-INT=up, FW1-EXT=down results in one event for FW1-EXT

You could also utilising 'tagging' in the triggers and enable correlation to close new events where there is an existing event that matches the trigger tags?

Hope this helps

Cheers
James

[Marland]

Hey James,

Thanks for the reply. I got it working now. See my comment to Noobz above for details.

What I was trying to set up is more like:

* FW1-INT=down, FW1-EXT=up no event is fired
* FW1-INT=down, FW1-EXT=down one event is fired for FW1 is Unreachable
* FW1-INT=up, FW1-EXT=down no event is fired

Regards,
Marland

[[email protected]]

Hi Marland,

I get it now.

How we do this is monitor the device management ip to determine whether it is up and monitoring interfaces on the devices (IF-MIB) to determine whether the interface is up and throughput exists etc...

Cheers
James

[Marland]

Thanks to everyone who responded. Here is the working solution I came up with based on your advice:

I disabled the second host I created, and then adapted the first Host like so:

Example:

Host Name: FW1

Items: Ping Monitor (Simple check, icmpping[10.x.x.1,4])

Ping Monitor (Simple check, icmpping[192.168.x.1,4])

Triggers: {HOST.NAME} ICMP Availability

Trigger Expression: {10.x.x.1:icmpping[10.x.x.1,4].avg(#5)}=0 and {10.x.x.1:icmpping[192.168.x.1,4].last(#5)}=0

Now when one interface is unreachable no alert is fired, but when both interfaces are unreachable I get an alert based on the trigger conditions.