Network interface mis-treated in Zabbix (7.4.1)

[pzen]

I discovered a problem with how Zabbix 7.4.1 handles network nic names.
In a trivial case, say a virtual machine or a physical machine with only one integrated (motherboard) NIC, the names are handled correctly.
But for my server with a mix of integrated and PCI-card extended, the names are of different length and this does not seem to work.

In my case i have: eno1, enp1s0f0 and enp1s0f1. But Zabbix does not see eno1 nor properly identifies the other two.
In Zabbix I see enp1s0 as the only NIC, and it does not work. (Neither does this WYSIWYG editor :-) )
I then tried to add my own tailored item, but got an error "Cannot find information for this network interface in /proc/net/dev".

Investigating, I see the nics are seen in the /proc/net/dev - file as:

  eno1: #### ####
enp1s0f0: #### ####
enp1s0f1: #### ####

Paying attention to the formatting it seems that the indent is inconsistent, which could explain the failure to detect.
I have experienced similar problems in my own code, where tools made assumptions regarding the length of the NIC-names.
The first entry is aligned with an old assumption about the length of NIC-name (so is 'lo:' ) and the two other is overflowing with 2.

By the way, the NIC from any extention-card nowadays has the format similar to enpXsYfZ, which appears to be related to addressing a PCI-slot.

Given that, I need a quick work around, since I'm new to Zabbix, and the structure of the documentation makes it hard for me to read (sight problem).

How do I get the metrics I need? I need rate of traffic in and out for the NIC's above.

Do I try to add my own item? Change something in templates or what is a good approach?

Tanx in advance...

[pzen]

The problem has been resolved by switching from DNS to IP when specifying the Agent interface.
As it turns out the default values unchanged are sticky (127.0.0.1:10050) and will be used if exist, even when I update with DNS myhost.my.net:10050.
I have not verified, but from my observations from the problem as well as what resolved it, it has to be.

> *sigh* The WebUI is a typical hacker-UI, which exposes the design behind the curtains, to the point that you have to read the code to understand the UI.

Anyhow, by switching to stating IP instead of DNS-name I suddenly got the right data alltogether.

So, the workflow and the critical part:

First I installed the zabbix-agent on the Debian 12 host I wanted to monitor, updated the /etc/zabbix/zabbix_agentd.conf to contain Server=192.168.37.42.

After restarting the agent, I attempt to add the server in Data collection/Hosts, this is the critical point:

Because both the IP and the dns-name is set, the IP is saved and used. This is not obvious later in the WebUI...

Now the discovery takes over and starts collecting information from the host, but not as I expect.
It collects from the Zabbix server (127.0.0.1), not the one stated as guru.example.net. The effective Agent interface is 127.0.0.1:10050, not guru.example.net:10050.

I lost a full day on this, thankfully everyone is still on vacation, so this time no disaster....

My resolution was to remove the dns-name and update the IP, and Connect To, then save and restart everything.

Oh, and there seemed to be issues with remnants in the database some time along the way, I think the updates are inconsistent in some cases...

Once update, I triggered the discovery, navigate the list of hosts in Data collection/Hosts and find the discovery items....

Now I pressed [Execute now]...... this should have been 1h tops, not 1d+ with a clear WebUI.....

[troffasky]

What template are you using?

[troffasky]

Try this:

# zabbix_agent2 -t net.if.discovery
net.if.discovery [s|[{"{#IFNAME}":"lo"},{"{#IFNAME}":"enp1s0"}]]

[pzen]

What template are you using?

Ah, sorry for missing details...

I use Linux by Zabbix agent, target host is a Debian 12 on HW with multiple NICS and our own SDN config.
Configuration as provided by template discovery

[pzen]

Try this:

# zabbix_agent2 -t net.if.discovery
net.if.discovery [s|[{"{#IFNAME}":"lo"},{"{#IFNAME}":"enp1s0"}]]

Sorry, I'm new to Zabbix and don't see how to apply the instruction....

But I did now notice there is a zabbix-agent and zabbix-agent2. Are they complementary or does the second replace the first??

[Markku]

But I did now notice there is a zabbix-agent and zabbix-agent2. Are they complementary or does the second replace the first??

https://www.zabbix.com/documentation/current/en/manual/concepts/agent2

So they are from different code bases, and agent 2 has somewhat more capabilities. It's up to you which one to use.

Sorry, I'm new to Zabbix and don't see how to apply the instruction....

If you have Zabbix agent: run the command /usr/sbin/zabbix_agentd -t net.if.discovery

If you have Zabbix agent 2: run the command /usr/sbin/zabbix_agent2 -t net.if.discovery

Markku

[troffasky]

Are they complementary or does the second replace the first??

Good question. I use agent2 on Linux because it can monitor systemd services. But I cannot say with any certainty why both need to exist.

[pzen]

https://www.zabbix.com/documentation/current/en/manual/concepts/agent2

So they are from different code bases, and agent 2 has somewhat more capabilities. It's up to you which one to use.



If you have Zabbix agent: run the command /usr/sbin/ -t net.if.discovery

If you have Zabbix agent 2: run the command /usr/sbin/zabbix_agent2 -t net.if.discovery

Markku

OK, got it. As a minimalist i pick zabbix_agentd, since I don't need more features nor more programming languages involved.

Either way, running the command on the host I'm having problems with gave the right interfaces, but they do not appear when I try to add items ot when trying to create graph or dashboard for networking...

How do I re-run discovery to make sure all metrics are available?

Or, is it the template that's wrong? Where do I file a bug-report?

[Markku]

What do you mean by "they do not appear when I try to add items"?

The template works like this:

• It discovers the interfaces
• For every discovered interface, it creates items and graphs for the interface
• You don't have to create any interface items manually

Are you still saying that the discovery item sees the interfaces when testing manually but there are no items created for metrics of those interfaces?

Markku

[troffasky]

On all the factory templates, interface discovery [which is a type of "Low Level Discovery"] runs regularly [eg, hourly].
Your problem is probably going to be filtering by interface name or type. Unfortunately I don't know of any way to log or debug LLD filtering so you will have to do some guessing.
On the template, drill in to Discovery Rules > Network interface discovery > Filters. You will see filters that match some macros.
Then look at what is defined under those macros {$NET.IF.....}. You can edit this at template level or override at host level, the choice is yours.

[pzen]

What do you mean by "they do not appear when I try to add items"?

The template works like this:

• It discovers the interfaces
• For every discovered interface, it creates items and graphs for the interface
• You don't have to create any interface items manually

Are you still saying that the discovery item sees the interfaces when testing manually but there are no items created for metrics of those interfaces?

Markku

OK, to be clear......

I installed zabbix-agent on the host I want to monitor. After a while I found information in "Monitorin/Latest data", which I could use to show CPU-count, and utilization and more. However, in the WebUI the network interfaces are screwed up. My host has eno1, enp1s0f0 and enp1s0f1, the last two from a PCI-slot with 2x10G interfaces. When trying to create a dashboard with "Top hosts" I can clearly get Memory and CPU using the discovered and populated items. (I.e: vm.memory.size[total]). But when looking for network interfaces the only item-key i found was for net.if.in["enp1s0"], which does not exist. It's the name of the PIC-slot, ignoring which NIC, and there's no values collected. In other words, I get one fake and three missing network interfaces.

When I run the suggested command `zabbix_agentd -t net.if.discovery` i get what looks like the right info.

[s|[{"{#IFNAME}":"lo"},{"{#IFNAME}":"eno1"},{"{#IFNAME}":"enp1s0f0"},{"{#IFNAME}":"enp1s0f1"},{"{#IFNAME}":"docker0"}]]

But this seems to be corrupted somewhere along the way to the items available via WebUI.

Note! This problem does not occur on a VM with one NIC (enp1s0 == virtual) or one NIC on the motherboard (i.e. eno1).

My main concern with the environment right now is network load passing through this particular hosts.

I was hoping to get this to work with some kind of work around, but Zabbix is new to me...

[troffasky]

I found information in "Monitorin/Latest data", which I could use to show CPU-count, and utilization and more.

To be clear, on the host where you get

[s|[{"{#IFNAME}":"lo"},{"{#IFNAME}":"eno1"},{"{#IFNAME }":"enp1s0f0"},{"{#IFNAME}":"enp1s0f1"},{"{#IFNAME }":"docker0"}]]

what do you see in Latest Data if you search for 'Bits'?

[Markku]

Ok so you have problems in creating graphs or other widgets in the dashboard, not in creating items. Maybe show a screenshot about the thing you are doing/trying and where you see incorrect item in some list?

Markku

[pzen]

----8<----

[pzen]

The problem has been resolved by switching from DNS to IP when specifying the Agent interface.
As it turns out the default values unchanged are sticky (127.0.0.1:10050) and will be used if exist, even when I update with DNS myhost.my.net:10050.
I have not verified, but from my observations from the problem as well as what resolved it, it has to be.

> *sigh* The WebUI is a typical hacker-UI, which exposes the design behind the curtains, to the point that you have to read the code to understand the UI.

Anyhow, by switching to stating IP instead of DNS-name I suddenly got the right data alltogether.

So, the workflow and the critical part:

First I installed the zabbix-agent on the Debian 12 host I wanted to monitor, updated the /etc/zabbix/zabbix_agentd.conf to contain Server=192.168.37.42.

After restarting the agent, I attempt to add the server in Data collection/Hosts, this is the critical point:

Because both the IP and the dns-name is set, the IP is saved and used. This is not obvious later in the WebUI...

Now the discovery takes over and starts collecting information from the host, but not as I expect.
It collects from the Zabbix server (127.0.0.1), not the one stated as guru.example.net. The effective Agent interface is 127.0.0.1:10050, not guru.example.net:10050.

I lost a full day on this, thankfully everyone is still on vacation, so this time no disaster....

My resolution was to remove the dns-name and update the IP, and Connect To, then save and restart everything.

Oh, and there seemed to be issues with remnants in the database some time along the way, I think the updates are inconsistent in some cases...

Once update, I triggered the discovery, navigate the list of hosts in Data collection/Hosts and find the discovery items....

Now I pressed [Execute now]...... this should have been 1h tops, not 1d+ with a clear WebUI.....

[Markku]

Sounds like a bug if the server uses IP even if DNS is selected. Could you open an issue (or search for an existing one) in https://support.zabbix.com?

Markku