Setup different alert values for auto discovered similar elements

[Glukas]

We have big number of Cisco Switches 9000 series (9200, 9300, etc).
There is official Cisco CISCO-ENVMON-MIB by SNMP template which we use on them.
The template discovers temperature of 3 different sensors (Inlet, Outlet and Hotspot), but has single warning and critical threshold macro's ({$TEMP_WARN} and {$TEMP_CRIT}) used for all 3 sensors.
While per vendor, there are separate warning and critical temperature values for each one:

Inlet Temperature Value: 28 Degree Celsius
Temperature State: GREEN
Yellow Threshold : 46 Degree Celsius
Red Threshold : 56 Degree Celsius

Outlet Temperature Value: 34 Degree Celsius
Temperature State: GREEN
Yellow Threshold : 105 Degree Celsius
Red Threshold : 125 Degree Celsius

Hotspot Temperature Value: 44 Degree Celsius
Temperature State: GREEN
Yellow Threshold : 105 Degree Celsius
Red Threshold : 125 Degree Celsius

Since same {$TEMP_WARN} = 46 Degree required for Inlet, used for all 3, once Hotspot goes above it, we get alert, although alert threshold for Hotspot is 105 Degree.

How we can set template discovery macros so we will have separate Warn/Crit macros for each of these 3 discovered sensors?

Thanks in Advance!

[wagnerjeske]

Hello Glukas!

If I understand correctly what you need, you will need to create different macros for each group of sensors.

Create something like:

{$INLET_TEMP_CRIT}
{$INLET_TEMP_WARN}

{$OUTLET_TEMP_CRIT}
{$OUTLET_TEMP_WARN}

{$HOTSPOT_TEMP_CRIT}
{$HOTSPOT_TEMP_WARN}

After creating these macros, define the temperature values.

Then, clone the existing trigger rules and adjust them with these new macros.

I believe this is what you need.

[ISiroshtan]

wagnerjeske I think it will be bit more complicated than that. As mentioned in original post it uses discovery, not just static item/trigger configuration. And one can not just edit discovered items, for that one would need to edit the trigger prototype and re-run the discovery. But if all sensors discovered by same discovery it's not as simple as just tweaking single trigger prototype, you'd need to teach discovery to distinguish sensor groups (or understand how it does that, if such configuration is already in place)

Glukas could you give a link to exact template you using? Looking at this template for example, it utilizes the macros context feature to distinguish between few types of sensors (seems like only CPU sensors get special treatment) and pointing them to a different macros for thresholds. If your template does the same, you should be able to check discovered trigger expression and see if it might have macro with context that is not define on template/host. It might be something like (important part highlighted):

'avg(/Cisco IOS by SNMP/sensor.temp.value[ciscoEnvMonTemperatureValue.X],5m)>{$TEMP_CRIT:"Inlet"} and ... (more_conditions_here)'

If it does have something like that, you can check if host has the macro {$TEMP_CRIT:"Inlet"} if it does - that temperature would be used for threshold.
If it does not - Zabbix will use non context Macro {$TEMP_CRIT}. And you can just define on host (or template) a new context macro {$TEMP_CRIT:"Inlet"} with whatever value you want as a new threshold and it will be used for all sensors that fall in same category(configured to use same context macros)

But once again, it all fully depends on what exact template you using and how the discovery is setup in there.

[Glukas]

Right, ISiroshtan, it is more complicated due to discovery.
The template name is Cisco CISCO-ENVMON-MIB by SNMP which comes with 6.0 distribution.
It is listed here: https://git.zabbix.com/projects/ZBX/...lease%2F6.0
Many thanks in advance!

[ISiroshtan]

I don't have credentials to access link you provided. But I found same named template from Zabbix 5.0 and it seems to use same logic I described before.

Could you check list of discovered items(sensors) and paste here trigger expression used in sensors for which you want to change the threshold?

[Glukas]

Items:


Triggers:


I can easily add {$TEMP_CRIT:"Inlet"}, {$TEMP_CRIT:"Outlet"} and {$TEMP_CRIT:"Hotspot"}
But I don't get how do I connect their values once they discovered?

Many thanks in advance!

[ISiroshtan]

Huh, that is lame. It seems to not be assigning context for threshold. If you see in trigger prototype you have {$TEMP_CRIT_:""}, so it does not try to assign different contexts for different sensors.

The one way I think is viable to explore: the CISCO-ENVMON-MIB exposes one extra OID for sensors called "ciscoEnvMonTemperatureStatusDescr" (1.3.6.1.4.1.9.9.13.1.3.1.2). It's description:

Textual description of the testpoint being instrumented.
This description is a short textual label, suitable as a
human-sensible identification for the rest of the
information in the entry.

Sadly I don't have Cisco which exposes this MIB in hand reach so I can not see what exactly values are in this OID. Can you snmpwalk one of your Ciscos on sensor table (1.3.6.1.4.1.9.9.13.1.3) and show some examples of values returned by ciscoEnvMonTemperatureStatusDescr?

[Glukas]

root@zabbix:~# snmpwalk -v1 -c big-secret xxx.xxx.xxx.xxx 1.3.6.1.4.1.9.9.13.1.3
SNMPv2-SMI::enterprises.9.9.13.1.3.1.2.1010 = STRING: "Switch 1 - Inlet Temp Sensor"
SNMPv2-SMI::enterprises.9.9.13.1.3.1.2.1011 = STRING: "Switch 1 - Outlet Temp Sensor"
SNMPv2-SMI::enterprises.9.9.13.1.3.1.2.1012 = STRING: "Switch 1 - HotSpot Temp Sensor"
SNMPv2-SMI::enterprises.9.9.13.1.3.1.3.1010 = Gauge32: 22
SNMPv2-SMI::enterprises.9.9.13.1.3.1.3.1011 = Gauge32: 29
SNMPv2-SMI::enterprises.9.9.13.1.3.1.3.1012 = Gauge32: 38
SNMPv2-SMI::enterprises.9.9.13.1.3.1.4.1010 = INTEGER: 56
SNMPv2-SMI::enterprises.9.9.13.1.3.1.4.1011 = INTEGER: 125
SNMPv2-SMI::enterprises.9.9.13.1.3.1.4.1012 = INTEGER: 125
SNMPv2-SMI::enterprises.9.9.13.1.3.1.5.1010 = INTEGER: 0
SNMPv2-SMI::enterprises.9.9.13.1.3.1.5.1011 = INTEGER: 0
SNMPv2-SMI::enterprises.9.9.13.1.3.1.5.1012 = INTEGER: 0
SNMPv2-SMI::enterprises.9.9.13.1.3.1.6.1010 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.13.1.3.1.6.1011 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.13.1.3.1.6.1012 = INTEGER: 1
SNMPv2-SMI::enterprises.9.9.13.1.3.1.7.1010 = INTEGER: 22
SNMPv2-SMI::enterprises.9.9.13.1.3.1.7.1011 = INTEGER: 29
SNMPv2-SMI::enterprises.9.9.13.1.3.1.7.1012 = INTEGER: 38

[ISiroshtan]

Aha, ok, we should be able to make it work. Probably...

Firstly: I would advise to create a copy of existing template and do all changes on copy. Then assign Copy to one host and verify if all works as you want. This way you have untouched original template and another one where you testing changes.

So what you could try to do is modify the trigger prototype, from:

{Template Module Cisco CISCO-ENVMON-MIB SNMP:sensor.temp.value[ciscoEnvMonTemperatureValue.{#SNMPINDEX}].avg(5m)}>{$TEMP_CRIT:""}
or
{Template Module Cisco CISCO-ENVMON-MIB SNMP:sensor.temp.status[ciscoEnvMonTemperatureState.{#SNMPINDEX}].last(0)}={$TEMP_CRIT_STATUS}
or
{Template Module Cisco CISCO-ENVMON-MIB SNMP:sensor.temp.status[ciscoEnvMonTemperatureState.{#SNMPINDEX}].last(0)}={$TEMP_DISASTER_STATUS}

to

{Template Module Cisco CISCO-ENVMON-MIB SNMP:sensor.temp.value[ciscoEnvMonTemperatureValue.{#SNMPINDEX}].avg(5m)}>{$TEMP_CRIT:"{#SNMPVALUE}"}
or
{Template Module Cisco CISCO-ENVMON-MIB SNMP:sensor.temp.status[ciscoEnvMonTemperatureState.{#SNMPINDEX}].last(0)}={$TEMP_CRIT_STATUS}
or
{Template Module Cisco CISCO-ENVMON-MIB SNMP:sensor.temp.status[ciscoEnvMonTemperatureState.{#SNMPINDEX}].last(0)}={$TEMP_DISASTER_STATUS}

After this change, if you go to host where template assigned and sensor discovered triggers should be expanded to something like:

>{$TEMP_CRIT:"Switch 1 - Inlet Temp Sensor"}

etc.


If this worked, on template level you can add macros like "{$TEMP_CRIT:regex:".*Inlet.*"}" and set threshold value for inlet sensors. Can repeat same for other sensors you want to have a custom threshold. (this part might have some errors, did not touch regexp context macros for quite some time now)

[Glukas]

Many thanks ISiroshtan!
It worked just as you explained.
The only difference was the regex indeed - per Zabbix manual it is very simple now, {$MACRO:regex:"regular expression"}, so "{$TEMP_WARN:regex:"Inlet"}" with "{$TEMP_CRIT:regex:"Inlet"}" and similar pair for two others did the job.
They are case sensitive, but this is all.

Again, many thanks for all your help!