Sudden dops & spikes on interface graphs

[t4h3r0]

Hi everyone,
New to the forum and relatively new to Zabbix.

I recently finished deploying a distributed Zabbix HA cluster with proxies.
Monitoring Cisco NCS-5500 & NCS-540X series devices using "Cisco IOS by SNMP", ZTE ZXCTN 6120H series devices with a cloned version of "Cisco IOS by SNMP" named "ZTE ROSNG by SNMP" and Huawei ATN 910 series devices with "Huawei VRP by SNMP" templates. Minimal minor changes have been made to tamplates.

On the Cisco and ZTE devices, interface traffic graphs are exhibiting a recurring sudden drops & spikes cycles or rubber-banding pattern sharp drops followed by immediate spikes every 30 to 40 minutes. The actual physical traffic is perfectly smooth.
This is only happening on the Cisco and less dramatically on ZTE hosts. The Huawei devices are rendering perfectly smooth, accurate graphs under the exact same network conditions. Increasing the polling interval from 1m to 2m reduced the amplitude of the spikes but did not solve the root issue.

I would genuinely appreciate any insights, configuration tweaks or pointers you might have. Thanks in advance.

[Rudlafik]

Look at the CPU and RAM consumption of the measured active network elements. The CPU values should not exceed 60% and RAM 80% of the consumed power in the long term. Then the active element cannot keep up with the traffic.

[t4h3r0]

Look at the CPU and RAM consumption of the measured active network elements. The CPU values should not exceed 60% and RAM 80% of the consumed power in the long term. Then the active element cannot keep up with the traffic.

CPU load is under 5% and RAM Uasage is around 30% to 40%

These Cisco devices have considarably more powerful resources than the ZTE or Huawei devices

[irontmp]

Hi everyone,
New to the forum and relatively new to Zabbix.

I recently finished deploying a distributed Zabbix HA cluster with proxies.
Monitoring Cisco NCS-5500 & NCS-540X series devices using "Cisco IOS by SNMP", ZTE ZXCTN 6120H series devices with a cloned version of "Cisco IOS by SNMP" named "ZTE ROSNG by SNMP" and Huawei ATN 910 series devices with "Huawei VRP by SNMP" templates. Minimal minor changes have been made to tamplates.

On the Cisco and ZTE devices, interface traffic graphs are exhibiting a recurring sudden drops & spikes cycles or rubber-banding pattern sharp drops followed by immediate spikes every 30 to 40 minutes. The actual physical traffic is perfectly smooth.
This is only happening on the Cisco and less dramatically on ZTE hosts. The Huawei devices are rendering perfectly smooth, accurate graphs under the exact same network conditions. The issue is as unrelated to Download Subway Surfers Mod APK as it is to any other non-networking topic, which is why I've ruled out external factors. Increasing the polling interval from 1m to 2m reduced the amplitude of the spikes but did not solve the root issue.

I would genuinely appreciate any insights, configuration tweaks or pointers you might have. Thanks in advance.

I'd start by checking for missed SNMP polls or interface counter issues. That kind of rubber banding often happens when a poll is delayed or a counter isn't read correctly. Since Huawei devices are fine under the same setup, I'd compare the SNMP settings and interface items between the Cisco and Huawei templates to see what's different.

[t4h3r0]

I'd start by checking for missed SNMP polls or interface counter issues. That kind of rubber banding often happens when a poll is delayed or a counter isn't read correctly. Since Huawei devices are fine under the same setup, I'd compare the SNMP settings and interface items between the Cisco and Huawei templates to see what's different.

I also suspect that its being caused by SNMP reply delay from device or counter misbehaving on the device..
I'm pretty sure the spikes and drops are being caused by the zabbix tamplates > Discovery Rules > Item Prototypes


This preprocessing step..

cause if the value is received after even a few seconds late or early the calculation will misinterpret the value

but could not figure out how this can be countered

you can check the templates here https://drive.google.com/drive/folde...tI?usp=sharing

and device SNMP config
##########################################
# Cisco SNMP Configuration #
##########################################
snmp-server user XXXXX infra v3 auth md5 encrypted 000C04150A561B105C
snmp-server view SNMP_VIEW1 internet included
snmp-server view SNMP_VIEW1 1.3.6.1.2.1.14 excluded
snmp-server view SNMP_VIEW1 1.3.6.1.2.1.191 excluded
snmp-server group infra v3 auth read SNMP_VIEW1

##########################################
# Huawei SNMP Configuration #
##########################################
snmp-agent
snmp-agent local-engineid 800007DB0394D2BC691CDE
snmp-agent community read cipher %^%#Tki`9,S!*("a|Y9D~-#/H_g1X*Ft*$:1#iN.!QVRVRtf7@*B6DiuuG6=zi.<'{WaK;9|3M 6F/)$hAPcV%^%# alias __CommunityAliasName_01_40655
#
snmp-agent sys-info location XXXX
snmp-agent sys-info version v3
snmp-agent group v3 XXXX privacy read-view iso-view write-view iso-view notify-view iso-view
snmp-agent group v3 XXXXatn privacy read-view iso-view write-view iso-view notify-view iso-view
snmp-agent group v3 SNMP_VIEW1 authentication read-view SNMP_VIEW1 notify-view SNMP_VIEW1
snmp-agent target-host host-name __targetHost_1_9234 trap address udp-domain 10.0.X.X params securityname XXXX@Huawei1 v3 privacy private-netmanager
snmp-agent target-host host-name __targetHost_2_47746 trap address udp-domain 10.0.X.X params securityname XXXX@Huawei1 v3 privacy private-netmanager
#
snmp-agent mib-view included iso iso
snmp-agent mib-view included iso-view iso
snmp-agent mib-view included SNMP_VIEW1 iso
snmp-agent usm-user v3 XXXX
snmp-agent usm-user v3 XXXX group SNMP_VIEW1
snmp-agent usm-user v3 XXXX authentication-mode md5 cipher %^%#5E\E$l0>l4/_#(RMa<TCEti;7J@4CBP`}PB)"RgU%^%#
snmp-agent usm-user v3 XXXX@Huawei1
snmp-agent usm-user v3 XXXX@Huawei1 group XXXXatn
snmp-agent usm-user v3 XXXX@Huawei1 authentication-mode sha2-512 cipher %^%#_zKt=/_]r.VC{mIO\C"@!&h~O'g.W:|LY08BeS0V%^%#
snmp-agent usm-user v3 XXXX@Huawei1 privacy-mode aes128 cipher %^%#{16BBR,FpFPd4)<9v)V"QP5QDnx{>UVPc8U%8(K7%^%#
#
snmp-agent trap source LoopBack0
#
snmp-agent notification-log enable
#
snmp-agent protocol source-status all-interface
undo snmp-agent protocol source-status ipv6 all-interface
#
undo snmp-agent proxy protocol source-status all-interface
undo snmp-agent proxy protocol source-status ipv6 all-interface
#
snmp-agent trap enable
#
return

[Rudlafik]

Same problem when you switch off SNMPv3 and set v1 sorry v2?

[t4h3r0]

Same problem when you switch off SNMPv3 and set v1?

did not try this one yet.. i'll try it and share findings

[Rudlafik]

Sorry, no v1 but v2

[t4h3r0]

did not try this one yet.. i'll try it and share findings

observed for the last 12 hours with SNMP V2C.. same same