Zabbix get reporting different value on agent

[mislav-eu]

Hi.

I'm using zabbix server 3.4.6
Zabbix agent is on version 3.0.7 (latest from jessie backports).

I've noticed some services being down and zabbix didn't alert me, so I asked myself why is that. Doing some troubleshooting I've noticed that zabbix_get on host with checking item is returning different value compared to the same command executed directly on zabbix agent.

Example:
- zabbix agent have user param defined item:
UserParameter=check.nfs.process,ps auxw | grep nfs | wc -l

If I execute that command directly on zabbix agent it returns value - e.g. 2, but if I run on zabbix server zabbix_get
# zabbix_get -s AGENT_IP -p 10050 -k check.nfs.process

I will get value 3 (one more). Why does this happen? I've other checks with grep-ing string and it works just fine.

How to troubleshoot this difference?

[dimir]

Don't do it this way, because you will catch your own process:

$ ps auxw | grep yada
john       32105  0.0  0.0  14108   936 pts/5    S+   12:56   0:00 grep yada

Use pgrep:

$ pgrep -lfc yada
0
$ pgrep -lfc mplayer
1

[mislav-eu]

Yeah, I forgot to mention that:
1) if I have only one grep, then I also add grep -v "grep" - this will exclude grep search itself
2) I usually have multiple strings seaches, so own process is not cached

So this should be a problem in my case, I guess?

P.s. thanks for the pgrep tip, I will look into that right away!

[dimir]

I think own process issue is what happens on the agent side.

[mislav-eu]

Indeed it is. I've just confirmed that!

zabbix_get is not aware of this and it takes cached grep value. One must use either pgrep or add grep -v "grep" into user param (seems like I've been missing in this one specific parameter).

Thanks for the help!