Hi guys, I'm seeking help on a strange issue I'm seeing on a network device.
I've started monitoring these two border router that have two interfaces (one per router) that connects to outside.
Due to the higher throughput I didn't get a good reading (spikes to 35 TBbps down to 0, then several points missing, etc). I thought it was due to the 32 bit counter issue, and actually that fixed the problem for every interface - except for the ONE interface that makes the most traffic. That keeps on having issues.
Having all interfaces in ifHCInOctets / ifHCOutOctets, these are the correct value readings on br2:
it1-b1-br2: Outgoing traffic on interface 10GigabitEthernet2/3
2016-11-08 09:04:45 1478592285 560601648
2016-11-08 09:03:45 1478592225 535431648
2016-11-08 09:02:45 1478592165 396923704
2016-11-08 09:01:45 1478592105 269858808
2016-11-08 09:00:48 1478592048 301126344
2016-11-08 08:59:45 1478591985 354442088
2016-11-08 08:58:45 1478591925 403048032
2016-11-08 08:57:45 1478591865 315130440
2016-11-08 08:56:45 1478591805 383136960
2016-11-08 08:55:48 1478591748 461519936
2016-11-08 08:54:45 1478591685 444935976
2016-11-08 08:53:45 1478591625 433555600
2016-11-08 08:52:45 1478591565 530143272
2016-11-08 08:51:45 1478591505 334078984
2016-11-08 08:50:45 1478591445 541515168
2016-11-08 08:49:45 1478591385 439207440
2016-11-08 08:48:45 1478591325 336012896
2016-11-08 08:47:45 1478591265 334556960
2016-11-08 08:46:45 1478591205 309981736
2016-11-08 08:45:45 1478591145 304087200
2016-11-08 08:44:45 1478591085 313422328
Which is totally reasonable. Now, onto the problematic interface on br1:
it1-b1-br1: Incoming traffic on interface 10GigabitEthernet2/3
2016-11-08 09:04:05 1478592245 40237648
2016-11-08 09:03:04 1478592184 3336562590104
2016-11-08 09:02:04 1478592124 0
2016-11-08 09:00:04 1478592004 3377570794752
2016-11-08 08:58:04 1478591884 0
2016-11-08 08:57:04 1478591824 3180867689152
2016-11-08 08:55:04 1478591704 22177200
2016-11-08 08:54:04 1478591644 35539976
2016-11-08 08:53:05 1478591585 29158496
2016-11-08 08:52:04 1478591524 26451224
2016-11-08 08:51:04 1478591464 3369066427168
2016-11-08 08:49:07 1478591347 20999456
I tried SNMPGETting and manually calculating the delta on counter values, and I had no issue in console (30 seconds span). Except for the first speed, calculated on the delta to 0, the others are fine; you can see the counter value as "raw value".
Speed is 3218807 Mbps (raw value 264763264371028, raw delta 264763264371028)
Speed is 214 Mbps (raw value 264764109060469, raw delta 844689441)
Speed is 259 Mbps (raw value 264765128304809, raw delta 1019244340)
Speed is 267 Mbps (raw value 264766182119175, raw delta 1053814366)
Unlink-and-clear the template and re-link it? Check.
Create another host and reproduce the issue? Check.
Any hint?
Thanks!
I've started monitoring these two border router that have two interfaces (one per router) that connects to outside.
Due to the higher throughput I didn't get a good reading (spikes to 35 TBbps down to 0, then several points missing, etc). I thought it was due to the 32 bit counter issue, and actually that fixed the problem for every interface - except for the ONE interface that makes the most traffic. That keeps on having issues.
Having all interfaces in ifHCInOctets / ifHCOutOctets, these are the correct value readings on br2:
it1-b1-br2: Outgoing traffic on interface 10GigabitEthernet2/3
2016-11-08 09:04:45 1478592285 560601648
2016-11-08 09:03:45 1478592225 535431648
2016-11-08 09:02:45 1478592165 396923704
2016-11-08 09:01:45 1478592105 269858808
2016-11-08 09:00:48 1478592048 301126344
2016-11-08 08:59:45 1478591985 354442088
2016-11-08 08:58:45 1478591925 403048032
2016-11-08 08:57:45 1478591865 315130440
2016-11-08 08:56:45 1478591805 383136960
2016-11-08 08:55:48 1478591748 461519936
2016-11-08 08:54:45 1478591685 444935976
2016-11-08 08:53:45 1478591625 433555600
2016-11-08 08:52:45 1478591565 530143272
2016-11-08 08:51:45 1478591505 334078984
2016-11-08 08:50:45 1478591445 541515168
2016-11-08 08:49:45 1478591385 439207440
2016-11-08 08:48:45 1478591325 336012896
2016-11-08 08:47:45 1478591265 334556960
2016-11-08 08:46:45 1478591205 309981736
2016-11-08 08:45:45 1478591145 304087200
2016-11-08 08:44:45 1478591085 313422328
Which is totally reasonable. Now, onto the problematic interface on br1:
it1-b1-br1: Incoming traffic on interface 10GigabitEthernet2/3
2016-11-08 09:04:05 1478592245 40237648
2016-11-08 09:03:04 1478592184 3336562590104
2016-11-08 09:02:04 1478592124 0
2016-11-08 09:00:04 1478592004 3377570794752
2016-11-08 08:58:04 1478591884 0
2016-11-08 08:57:04 1478591824 3180867689152
2016-11-08 08:55:04 1478591704 22177200
2016-11-08 08:54:04 1478591644 35539976
2016-11-08 08:53:05 1478591585 29158496
2016-11-08 08:52:04 1478591524 26451224
2016-11-08 08:51:04 1478591464 3369066427168
2016-11-08 08:49:07 1478591347 20999456
I tried SNMPGETting and manually calculating the delta on counter values, and I had no issue in console (30 seconds span). Except for the first speed, calculated on the delta to 0, the others are fine; you can see the counter value as "raw value".
Speed is 3218807 Mbps (raw value 264763264371028, raw delta 264763264371028)
Speed is 214 Mbps (raw value 264764109060469, raw delta 844689441)
Speed is 259 Mbps (raw value 264765128304809, raw delta 1019244340)
Speed is 267 Mbps (raw value 264766182119175, raw delta 1053814366)
Unlink-and-clear the template and re-link it? Check.
Create another host and reproduce the issue? Check.
Any hint?
Thanks!
Comment