Zabbix check does not return values

[aigo15108]

Can someone tell me why;

My discovery rule lists services in this format (from the top);

{
"data": [
{
"{#SERVICE}": "auditd.service"
},
{
"{#SERVICE}": "chronyd.service"
},
{
"{#SERVICE}": "crond.service"
},
and so on ...
]
}

In the item prototype (after the discovery) the key is set as check.services[{#SERVICE}]

Yet in the 'Latest data' I see the list of services without data. In fact the list is seen only when 'Show items without data' is checked.

From the command line, however, I get this (as an example),

zabbix_get -s 192.168.102.138 -k check.services[zabbix-agent.service]

returns 0 (as expected, the service is running. 1 would mean it is not running. It is an INT)

So what's happening, can someone tell me?

Thank you in advance.

[dimir]

What type of item, the item prototype is? Could you show the screenshot of its configuration?

[aigo15108]

It is Zabbix agent (active)

[aigo15108]

I suspect that the variable {#SERVICE} is read in (and understood by zabbix) quite fine, it just there are no values that come out of it.

[dimir]

Is your agent configured to work in active mode?

Another thing, check agent and server log, are there any errors about getting/sending the list of active checks?

[aigo15108]

No hints whatsoever. Update interval is set to 1 minute. Restarted both about 5 minutes ago.

[root@monitored zabbix]# tail -f zabbix_agentd.log
3662:20200320:143552.356 IPv6 support: YES
3662:20200320:143552.356 TLS support: YES
3662:20200320:143552.356 **************************
3662:20200320:143552.356 using configuration file: /etc/zabbix/zabbix_agentd.conf
3662:20200320:143552.356 agent #0 started [main process]
3663:20200320:143552.357 agent #1 started [collector]
3664:20200320:143552.358 agent #2 started[listener #1]
3665:20200320:143552.363 agent #3 started[listener #2]
3666:20200320:143552.365 agent #4 started[listener #3]
3667:20200320:143552.367 agent #5 started [active checks #1]


4899:20200320:143639.462 Got signal [signal:15(SIGTERM),sender_pid:20380,sender_uid:0,r eason:0]. Exiting ...
4915:20200320:143639.462 syncing history data in progress...
4915:20200320:143639.475 syncing history data... 100.000000%
4915:20200320:143639.475 syncing history data done
4899:20200320:143639.599 syncing trend data...
4899:20200320:143639.614 syncing trend data done
4899:20200320:143639.615 Zabbix Server stopped. Zabbix 4.4.0 (revision cfac660b25).
20388:20200320:143639.645 Starting Zabbix Server. Zabbix 4.4.0 (revision cfac660b25).
20388:20200320:143639.645 ****** Enabled features ******
20388:20200320:143639.645 SNMP monitoring: YES
20388:20200320:143639.645 IPMI monitoring: YES
20388:20200320:143639.645 Web monitoring: YES
20388:20200320:143639.645 VMware monitoring: YES
20388:20200320:143639.645 SMTP authentication: YES
20388:20200320:143639.645 ODBC: YES
20388:20200320:143639.645 SSH2 support: YES
20388:20200320:143639.645 IPv6 support: YES
20388:20200320:143639.645 TLS support: YES
20388:20200320:143639.645 ******************************
20388:20200320:143639.645 using configuration file: /etc/zabbix/zabbix_server.conf
20388:20200320:143639.656 current database version (mandatory/optional): 04040000/04040000
20388:20200320:143639.656 required mandatory version: 04040000
20388:20200320:143639.707 server #0 started [main process]
20389:20200320:143639.717 server #1 started [configuration syncer #1]
20390:20200320:143639.717 server #2 started [housekeeper #1]
20391:20200320:143639.718 server #3 started [timer #1]
20392:20200320:143639.718 server #4 started [http poller #1]
20393:20200320:143639.718 server #5 started [discoverer #1]
20394:20200320:143639.722 server #6 started [discoverer #2]
20395:20200320:143639.724 server #7 started [discoverer #3]
20396:20200320:143639.726 server #8 started [discoverer #4]
20397:20200320:143639.729 server #9 started [discoverer #5]
20403:20200320:143639.731 server #15 started [history syncer #1]
20405:20200320:143639.732 server #17 started [history syncer #3]
20406:20200320:143639.732 server #18 started [history syncer #4]
20408:20200320:143639.732 server #20 started [proxy poller #1]
20411:20200320:143639.736 server #23 started [poller #1]
20404:20200320:143639.739 server #16 started [history syncer #2]
20400:20200320:143639.739 server #12 started [discoverer #8]
20413:20200320:143639.745 server #25 started [poller #3]
20402:20200320:143639.748 server #14 started [discoverer #10]
20407:20200320:143639.750 server #19 started [escalator #1]
20409:20200320:143639.753 server #21 started [self-monitoring #1]
20412:20200320:143639.753 server #24 started [poller #2]
20401:20200320:143639.756 server #13 started [discoverer #9]
20410:20200320:143639.759 server #22 started [task manager #1]
20414:20200320:143639.761 server #26 started [poller #4]
20415:20200320:143639.767 server #27 started [poller #5]
20416:20200320:143639.793 server #28 started [unreachable poller #1]
20418:20200320:143639.796 server #30 started [trapper #2]
20419:20200320:143639.799 server #31 started [trapper #3]
20417:20200320:143639.802 server #29 started [trapper #1]
20420:20200320:143639.804 server #32 started [trapper #4]
20398:20200320:143639.813 server #10 started [discoverer #6]
20399:20200320:143639.855 server #11 started [discoverer #7]
20421:20200320:143639.916 server #33 started [trapper #5]
20426:20200320:143639.918 server #38 started [alerter #3]
20423:20200320:143639.918 server #35 started [alert manager #1]
20422:20200320:143639.919 server #34 started [icmp pinger #1]
20425:20200320:143639.919 server #37 started [alerter #2]
20424:20200320:143639.919 server #36 started [alerter #1]
20435:20200320:143639.959 server #43 started [lld manager #1]
20436:20200320:143639.971 server #44 started [lld worker #1]
20431:20200320:143640.054 server #39 started [preprocessing manager #1]
20439:20200320:143640.057 server #46 started [alert syncer #1]
20438:20200320:143640.163 server #45 started [lld worker #2]
20432:20200320:143641.091 server #40 started [preprocessing worker #1]
20434:20200320:143641.091 server #42 started [preprocessing worker #3]
20433:20200320:143641.196 server #41 started [preprocessing worker #2]

I scrolled up through the logs, there is not anything meaningful in there.

[dimir]

Is the host name in "Configuration -> Hosts" absolutely the same as "Hostname=" in zabbix_agentd.conf ?

[aigo15108]

Yes it is. It is just by the ip

Server=192.168.102.139
ServerActive=192.168.102.139

From zabbix-server
[root@monitoring zabbix]# ip a | grep ens33 | grep inet
inet 192.168.102.139/24 brd 192.168.102.255 scope global noprefixroute dynamic ens33
[root@monitoring zabbix]# zabbix_get -s 192.168.102.140 -k check.services['systemd-user-sessions.service']
0 # which is fine.

[moment - please wait]
This is from /etc/hosts (the last line)
192.168.102.139 monitoring monitoring.localdomain

Are you saying I should change ServerActive to monitoring instead?

[dimir]

That is absolutely irrelevant. zabbix_get is for passive agent checks, your item is of type "Zabbix agent (active)". If you don't care, set the item type in prototype to "Zabbix agent" and it will probably start to work.

What I meant is this: https://www.zabbix.com/documentation...fig/hosts/host

Look for "Host name" form field.

[aigo15108]

Changed the Host name - did not help. Changed to type to 'Zabbix agent' (not active) the data started coming in. So is there a way to make agent send to the server (when ready) and not being polled by the server instead? This is the content of my zabbix-agent.conf file - and frankly I don't think 'Host name' has anything to do with the problem I am experiencing.

PidFile=/var/run/zabbix/zabbix_agentd.pid
LogType=file
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=5
DebugLevel=3
EnableRemoteCommands=1
LogRemoteCommands=1
Server=192.168.102.139
ListenPort=10050
ListenIP=0.0.0.0
StartAgents=4
ServerActive=192.168.102.139
Hostname=monitoring.localdomain
HostnameItem=system.hostname
RefreshActiveChecks=120
BufferSend=5
BufferSize=100
MaxLinesPerSecond=20
Timeout=10
AllowRoot=1
User=zabbix
Include=/etc/zabbix/zabbix_agentd.d/*.conf
UnsafeUserParameters=1

[dimir]

When you go to frontend "Configuration -> Hosts", is the affected host named "monitoring.localdomain"?

Also, mind the fact that Zabbix server needs time to refresh it's configuration cache after you make changes in frontend (by default 1 minute) and agent needs to re-fetch the list of active items (by default 2 minutes). Basically, when an active agent requests active checks from server with non-existing host name in Zabbix configuration there will be a message about that in server log file, e. g.:

Sending list of active checks to [<IP>] failed: host [<HOST NAME>] not found

[aigo15108]

Front is where you define the agents (nodes), not the host. I thought you meant 'Host name' in the zabbix_agent.conf file, there it would be the server. And I think I got it. In this link you posted, about the host name, it reads 'Unless you specifically define a Hostname in zabbix_agentd.conf, the system hostname of agent location will be used by server for naming the host'. I made the first word (Unless) bold. That is by default it is 'Hostname=Zabbix server', and that was misleading the agent. Commenting this out appears to have solved the problem. Now I have 'Zabbix agent (active)' on all checks and I am getting the data on the front page.

Well, talking through, good hints, and I thank you for helping me out.

[dimir]

Front is where you define a host and that host should be related to an agent, if there is one. For example, there are items that do not involve an agent (Calculated item, Simple check, Trapper item etc.) but you still attach them to some host. So, the important thing to remember is that if you define a host in frontend that will contain at least one "Zabbix agent (active)" item, this host name must be equal (case-sensitive) to what you have in Zabbix agent configuration file (either directly defined as "Hostname=<something>" or using HostnameItem=<"Zabbix agent" item that returns "something"> or, if both aren't defined, the system hostname (e. g. what command "hostname" returns in GNU/Linux)).

Great video describing this in detail: https://www.youtube.com/watch?v=-QTZVWnx6i4