Discovery job issues. Single host ends up with a ton of IP addresses

[netsaint]

Hi,

thanks for your time reading my post. Long time user, first time poster.

I use Zabbix discovery on a large metro network. I have a lot of subnets, and the hosts on that subnets are layer2 switches. These switches have at least one management IP addresses, sometimes multiple. 1,500 switches or so in total. I had an issue when using multiple discovery jobs (one for each switch management subnet), Zabbix uniqueness function seemed to not work when doing multiple jobs. I ended up with the same switch discovered multiple times, even so my uniqueness is on the snmpName. Now, I do one discovery job and that seems to work better. Here a picture of that job, you can see I check on sysName and sysObjectID, and sysName is my uniqueness criteria and what I end up for hostname and visible name:




My issue now is that I have a couple of switches (right now one switch only, but I seen 2-3 switches do that), and they end up with a lot of IP addresses. I assume they came in via discovery. Hundreds of IP addresses. These IP's are valid on my n network, but belong to other switches. Not sure how or why. Here a picture of what I mean, see all those IP's, there is another 100 or so if I scroll down:






And to complete the picture, here is my discovery actions. Basically I look at the value received for the Enterprise number in sysObjectID and depending on what I find there, I apply a certain template.




Has anybody seen something like this. I just end up deleting all the IP addresses under the host every time I see it, but I can't figure out why it adds all those or how to stop it.

Thank you

[cyber]

Giving so many ranges to one discovery job, that takes ages? its just one process running that one disco. Splitting up like you had it before, should be much faster...

But any chance there is bunch of cloned devices with similar config, so they are considered as one based on sysName? Just throwing ideas around here... I have never used network discovery, I find it useless. Just inserting those devices directly and not relying on disco seems better...

[netsaint]

Hi cyber, thanks for you answer.

I had the discovery jobs split by subnets before, so about 20 discovery jobs. The issue I seen then was that the device uniqueness criteria was not really working, so discovery started to create duplicates of a switch that it had already in the system under a different IP address, even so the name of the switch was the same. I ended up with multiple copies of the same switch, and Zabbix would just add a "_2" to the name. I resolved this with just having a single discovery job and it seems to do a better job of evaluating and honoring the device uniqueness criteria. In regard to how long it takes, actually it seems pretty quick. Looking at my Zabbix server graphs, I think discovery takes about 10 minutes. Maybe the initial discovery took long, but now it seems to be quick.



The device names or sysName is unique across the network.

Still not sure why a single host in my installation with about 1700+ ends up with ~ 200 IP addresses during the discovery, while all others seems to work just fine. I tried already once to simple disabled that hosts, see if this makes a difference. I think what the system did then was to simply choose/select another hosts that then gets 100s of IP addresses attributed after a discovery. Really weird. I think it's a bug, but I have not seen anybody else with it.

I will disable the host that has all those IP's and see if another hosts start getting extra IP addresses added during the next scan.

Thank you.

[netsaint]

I seem to be the only one with this issue. Zabbix works fine, but I have one host that is associated with a lot of IP addresses. Today I looked at it, I got > 300 IP addresses on my network associated to this one host. Now, if I choose an IP address this host is associated with and investigate, that IP address is also associated to the correct host in Zabbix and its a switch like any other switch I have on the network. So, in effect I got one record/host that is associated with 300+ IP addresses, but those IPs are also correctly associated with their correct switch. I have that 300+ IP address host disabled. Every now and then I manually delete all the IP addresses, just so that in a few days this hosts is again associated with a bunch of IP addresses. I suspect something with the discovery process is going wrong, maybe some logic issue that then associates an IP scanned with that one hosts. I also deleted that host before, but the issue just shifts to another hosts. Weird, but no idea how to troubleshoot this further. I hope some upgrade down the road changes something and knocks out that bug. Cheers.

[gferradas]

Silly question but if you do a trace from the zbx server to one of the duplicate ips you go to the right switch or to the switch with +100 ips?

[netsaint]

Not silly. Yes, when I trace route to some random IP addresses that are associated with the host, they first of all ping, and it is the right switch. Here some pings, then SNMPwalks. I do put some XXX in the names, to keep a little bit anonymity.



root@zabbixbox:/home/swulf# traceroute 10.90.0.20

traceroute to 10.90.0.20 (10.90.0.20), 30 hops max, 60 byte packets
1 _gateway (10.80.10.1) 0.594 ms 0.594 ms 0.640 ms
2 10.80.253.4 (10.80.253.4) 1.188 ms 1.253 ms 1.340 ms
3 10.80.250.1 (10.80.250.1) 1.730 ms 1.542 ms 1.865 ms
4 10.90.0.20 (10.90.0.20) 1.434 ms 1.522 ms 1.629 ms

root@zabbixbox:/home/swulf# traceroute 10.90.0.42

traceroute to 10.90.0.42 (10.90.0.42), 30 hops max, 60 byte packets
1 _gateway (10.80.10.1) 0.613 ms 0.598 ms 0.648 ms
2 10.80.253.4 (10.80.253.4) 1.131 ms 1.133 ms 1.122 ms
3 10.80.250.1 (10.80.250.1) 2.472 ms 2.151 ms 2.319 ms
4 10.90.0.42 (10.90.0.42) 2.729 ms 2.375 ms 3.053 ms

root@zabbixbox:/home/swulf# traceroute 10.90.56.41
traceroute to 10.90.56.41 (10.90.56.41), 30 hops max, 60 byte packets
1 _gateway (10.80.10.1) 1.330 ms 1.343 ms 1.436 ms
2 10.80.253.4 (10.80.253.4) 3.256 ms 3.258 ms 3.246 ms
3 10.80.250.5 (10.80.250.5) 5.044 ms 4.941 ms 5.062 ms
4 10.90.56.41 (10.90.56.41) 12.379 ms 12.352 ms 12.361 ms


root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.0.20 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: poweralert-061036411053

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.0.42 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: PLW_X460_XXX_ESC

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.56.41 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: JN2300_XXX_Baptist_XXX

Now, those three sample IP addresses I pulled randomly do exists in Zabbix with the correct host as well. So, if I search for 10.90.56.41 for example under Data Collection -> Hosts, Zabbix will list the correct host as well as that host that pulls in 100+ IP addresses:





Here a picture of that host record that has 100+ IPs associated, notice the scroll bar on the right, it is small so you can imagine how many IP's are there. And it is not just of the discovery subnets that IP addresses are attributed to this host, but across all subnets I discover. I "cleaned it up" and the last discovery just added IP's back to that host. Weird.




So weird.

[william-cl]

usually if it is adding the ips to one host is because the hostname is the same. Maybe your discovery is interpreting all the devices as the same name. For example I had 2 servers with the different hostnames but the file had hostname = <same name> so both ip's appeared on the same host

[netsaint]

thank you william-cl . What file do you refer to that had the hostname the same? When I do a snmpwalk on the hostname of each of those IP addresses, they are all different.

My uniqueness criteria is set to this SNMPv2 agent "1.3.6.1.2.1.1.5.0" which is SNMPv2-MIB::sysName.0 .

Here what I posted before, taking 3 random IP addresses, the system name is all different:

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.0.20 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: poweralert-061036411053

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.0.42 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: PLW_X460_XXX_ESC

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.56.41 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: JN2300_XXX_Baptist_XXX zz0.3anvzbvy3jgzz

I doesn't seem to affect operations, as I said, those hosts are discovered under their own name, so I got two records for 100+ IP addresses, the actual real one and that the host that for whatever reason 100+ IP's stick to as well. In previous troubleshooting I also just removed that one host completely and zabbix would stick 100+ extra IP's to some other host. Wicked for sure.

[william-cl]

Since you are using snmp, there is no file to configure sadly.


Looking through the discovery and your tags, its adding your device in the picture to several tags of your registrations. Should the registration be doing that off of the condition you created?

[belwavenoc]

thank you william-cl . What file do you refer to that had the hostname the same? When I do a snmpwalk on the hostname of each of those IP addresses, they are all different.

My uniqueness criteria is set to this SNMPv2 agent "1.3.6.1.2.1.1.5.0" which is SNMPv2-MIB::sysName.0 .

Here what I posted before, taking 3 random IP addresses, the system name is all different:

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.0.20 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: poweralert-061036411053

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.0.42 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: PLW_X460_XXX_ESC

root@zabbixbox:/home/swulf# snmpwalk -v2c -cXXX 10.90.56.41 SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: JN2300_XXX_Baptist_XXX zz0.3anvzbvy3jgzz

I doesn't seem to affect operations, as I said, those hosts are discovered under their own name, so I got two records for 100+ IP addresses, the actual real one and that the host that for whatever reason 100+ IP's stick to as well. In previous troubleshooting I also just removed that one host completely and zabbix would stick 100+ extra IP's to some other host. Wicked for sure.

I am having the same issue. It even applies discovered IP's to a host that isn't even monitored.

[belwavenoc]

select * from dservices where dhostid=1;

I had 7k+ invalid entries that were set to dhostid=1 when the wasn't a dhostid=1 at all in the dhosts table. I have deleted all of those services at this point in an attempt to correct this. Will post another update in a few hours.