nstat not returning values to zabbix

[min0sys]

Hi guys,
I`m trying to use zabbix in this configuration :
#server1:

zabbix-agent   4.0.5-1+xenial
Ubuntu 16.04.5 LTS
Linux version 4.4.0-142-generic (buildd@lgw01-amd64-033) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)
nstat -version : nstat utility, iproute2-ss151103

# server2:

zabbix-agent  4.0.5-1+trusty
Ubuntu 14.04.5 LTS
Linux version 3.13.0-164-generic (buildd@lcy01-amd64-015) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4) ) #214-Ubuntu SMP
nstat -version : nstat utility, iproute2-ss131122

In both cases, I`m using user parameter conf :

cat /etc/zabbix/zabbix_agentd.d/userparameter_tcpip.conf
UserParameter=tcpip.stats[*],nstat -az|grep "$1\ "|awk '{print $$2}'

And this is the result from zabbix :
Server1: getting value of tcpip.stats[TcpExtTCPKeepAlive]:

root@server1:~# nstat -az | grep -i TCPKeepAlive
TcpExt[B]TCPKeepAlive[/B]              40813              0.0

root@server1:~# cat /proc/net/netstat |grep -i TCPKeepAlive
TcpExt: [B]TCPKeepAlive[/B]

root@server1:~# zabbix_agentd -t tcpip.stats[TcpExtTCPKeepAlive]
tcpip.stats[TcpExtTCPKeepAlive]               [t|40816]

Server2: not getting values of tcpip.stats[TcpExtTCPKeepAlive]:

root@server2:~# nstat -az | grep -i TCPKeepAlive
root@server2:~# cat /proc/net/netstat |grep -i TCPKeepAlive
root@server2:~# zabbix_agentd -t tcpip.stats[TcpExtTCPKeepAlive]
tcpip.stats[TcpExtTCPKeepAlive]               [t|]

From my understanding, this might be related to version of kernel and/or iproute utility. What I`m not sure about, why zabbix insist on checking parameter which doesn`t exist in /proc/net/netstat , while using userparameter file?


@Admins, please check this, I`m getting on main page, when clicking on read more :
Invalid Page URL. If this is an error and the page should exist, please contact the system administrator and tell them how you got this message.



Thank you

[max.ch.88]

Zabbix agent does not know anything about your system settings. Exception handling is your script's task.

[min0sys]

Hi max.ch.88 ; I might not get it correct.
My script is only getting data from command :

nstat -az

, and parse/assign value. And I can confirm, it is working on servers, which have newer versions of kernel/iproute2 utility.
What I`m trying to say is, that if I run simple command from shell :

nstat -az

I will not get the value. Same is for

cat /proc/net/netstat

. Value is not there.
That is correct, and expected behaviour on systems with older iproute2 utility, and this is all what my script do (run the nstat command, generate output, read the output, parse values/strings) What I do not understand, why zabbix still generating alerts for values which doesn`t exists at all? 90% of values are OK, monitored, zabbix recognize them from nstat output. But the rest is generating alerts. I tried to look everywhere, but so far, didn`t find out, why.

[max.ch.88]

... What I`m not sure about, why zabbix insist on checking parameter which doesn`t exist in /proc/net/netstat , while using userparameter file?

Hi.
Zabbix doesn't insist on checking parameter. It does what you told it - nstat -az|grep "TCPKeepAlive\ "|awk '{print $$2}'. And if you received nothing when executed nstat -az|grep "TCPKeepAlive\ "|awk '{print $$2}' manually, obviously you will receive nothing using zabbix.

root@server2:~# zabbix_agentd -t tcpip.stats[TcpExtTCPKeepAlive]
tcpip.stats[TcpExtTCPKeepAlive] [t|]

The value cannot be empty. This is an error for zabbix. The value must be at least 0. You need exception handling
a=$(nstat -az|grep "$1\ "|awk '{print $$2}'); if [[ -z $a ]]; then echo 0; else echo $a; fi

[min0sys]

Hi max.ch.88, I completely agree with you on this. If I receive nothing from nstat -az ; or recieve nothing when do cat /etc/proc/netstat, then alert should not be generated. In this case I would be more than happy if it will work this way as you wrote.
But the alert is generated. TcpExtTCPKeepAlive was just example. My userparameter_tcpip.conf command generate those alerts to zabbix:

So I went to server, and check the unsupported values for key tcpip.stats + example of working TCPExt values (server name was removed):
I hope, this will provide much better view on this. I was told that Low Level Discovery should help here, but this is very last option here. I want to understand this behaviour first.

[max.ch.88]

Do you want to understand why the alert is generated? Why do the items have status "not supported"?
Because the items have "Type of information" - Numeric. The zabbix server expects the numeric value, but receives empty value.
tcpip.stats[TcpExtTCPKeepAlive] [t|]
tcpip.stats[TcpExtTCPSynRetrans] [t|]
tcpip.stats[TcpExtTCPFastOpenActiveFail] [t|]
tcpip.stats[TcpExtTCPFromZeroWindowAdv] [t|]
Why does the server receive empty value?
Because your system doesn't have those counters.
#nstat -az | grep -P "TcpExtTCPKeepAlive|TcpExtTCPSynRetrans|TcpExt TCPF astOpenActiveFail|TcpExtTCPFromZeroWindowAdv"
empty value
What could you do?
Solution 1: disable those items for the host.
Solution 2: repair or update the kernel and/or the iproute utility on the host.