Zabbix 5.4.8 False High Bandwidth Monitoring

[nicnatros]

I'm experiencing the same thing in a completely different environment. I'm running version 6.2.0 in a home lab. I have an active agent installed on my Asus home router. My WAN utilization looks mostly accurate other than these impossible spikes that happen once a day or so. On a 1 Gbps interface I get spikes up to 39 Gbps. I know the data from the router isn't faulty because I was previously monitoring it with Home Assistant/Influxdb/Grafana and never had spikes that were technically impossible to achieve.

[ltep]

How is this issue progressing?
I see a simular issue on some Fortinet switchports. They sometimes report "High bandwidth usage". The graph then shows a high value but for a very short time. For example 300Gbps while the switchport is 1Gb.
The Zabbix version is 6.2.4. The interval for the Bits received and sent in the template is set to a 1minute interval.

[tikkami]

Tuning of two constants mentioned there might be worth of trying --> https://support.zabbix.com/browse/ZBXNEXT-4428

[MrGuga]

Bear in mind that ifOutUcastPkts is a 32 bit counter, which probably is too small for a 10Gbps interface in high-usage.
Have your already tried reading ifHCOutUcastPkts that is the same thing but with a 64 bit counter?

[ltep]

I'm using the 64-bit counter. Will see if it's resolved with [ZBXNEXT-4428] optimized SNMP discovery and data collection, added GetBulk support. Thanks.

[iconicnetworks]

Just a quick update from us here. We now known of this issue cross vendor, cross release version too. We're going to apply some resource in January to try and look at this with more detail again, but it's impacting Cisco Nexus 9K's and Juniper 5130-32C's and 5120-48Y's from what we've seen so far. Although others are reporting similar issues with other vendors.

[hr-leinonen]

Hi,

I wonder if this same kind of problem than incorrect high bandwidth usage on Cisco VLAN interface?
We have Cisco 9300 and Cisco 9400 SW's and Zabbix gives high bandwidth usage alarm in VLAN interfaces.

Eg. switch VLAN3000 (net.if.speed) says 1Gbps and net.if.in value is 2.62Gbps

Zabbix version is 6.0.12.

Edit: snmpwalk for VLAN interface (oid 1.3.6.1.2.1.31.1.1.1.15.514) says IF-MIB::ifHighSpeed.514 = Gauge32: 1000 and it seems to be "feature".


Regards,

Ville

[Markku]

I have a Cisco Catalyst 9500 here and it says for a Vlan123 with SNMP:

IF-MIB::ifHighSpeed.149 = Gauge32: 1000

and that value is megabits per second --> 1 Gbps

Apparently that's where the template gets the net.if.speed value, and as you noticed, that's not really useful.

One idea to fix that is to edit the template and use LLD override to disable the trigger prototype for interfaces that are "VlanXXX". The physical interfaces are alerting you anyway when they are highly utilized, so maybe you don't need that artificial VLAN interface trigger.

Or you can just manually disable all the triggers for the VLAN interfaces.

Markku

[Markku]

For some reason the changes in edited posts don't come through even though the page is reloaded. Duh. Not the only technical problem in this site.

[hr-leinonen]

For quick Fix I just filtered out Vlan interface for discovery, probably I do prototype tunning, when I have more time.

Regards,

Ville

[RadicalEd]

Hello,

we're also having the issue with a Nexus Switch. But currently I doubt that this issue is caused by Zabbix. I've created a item that simply stores the RAW-Data of ifHCInOctets without any calculation (No multiplication and 'change per second' function). And within this RAW Data I can see that, suddenly between two checks, the value greatly increase. I also verified this value with snmpget.
My only idea would be, that ifHCInOctets can't be used for such a calculation or the Switch itself is doing something wrong here.

[Markku]

we're also having the issue with a Nexus Switch

What's your device type and NX-OS version? Interested to see.

Markku

[RadicalEd]

We're running the 9000 series here with NXOS 9.3(10).

[Markku]

We have cloudscale-FX-series (like 93180YC-FX = Cisco silicon, not Broadcom etc), and haven't seen problems, with 9.3(6) or 10.2(4). How often do you get strange numbers? I mean, I cannot tell for sure that we don't ever get strange numbers with SNMP but at least they don't show up in the graphs or our triggers.

Markku

[RadicalEd]

The issue occurs completely random.

Here is an example of the RAW Data

DATE	VALUE	DIFFERENCE
2023-01-27 10:05:14	110699124462315338	+16397895118
2023-01-27 10:04:14	110699108064420220	+906784004653766
2023-01-27 10:03:14	109792324059766454	+6514130890
2023-01-27 10:02:14	109792317545635564

On this 40 GbE Interface, this "hiccup" would be a traffic of ~ 70 Tbps.

Since the value constantly increase, I really don't see an issue on the side of Zabbix here. If Zabbix would make a mistake here, the following value after a "hiccup" would be smaller as the previous one. But the numerical sequence is "flawless".

I also found this article How To Calculate Bandwidth Utilization Using SNMP - Cisco and Zabbix is exactly following this guide, except that they're using ifHCInOctets instead of ifInOctets.

[Markku]

What is your exact switch model?

Markku

[RadicalEd]

It's a 9336C-FX2