Monitoring systemd process on another machine

[michaelNGV]

I'm trying to set up monitoring of a systemd-managed process on another machine so I found this stackoverflow answer. I'm puzzled because the first answer is accepted although it is (so far as I can tell) incomplete and in fact useless, if you don't already know exactly what you are doing with Zabbix (and therefore don't need help from SO).
The lacuna of the solution is that it doesn't specify the type of the item. At least, that's how it seems to me, so please tell me where my reasoning is wrong.
First of all, as per the solution, I modified /etc/zabbix/zabbix_agentd.conf on the machine I wished to monitor, to include the line:

UserParameter=systemd.unit.is-active[*],systemctl is-active --quiet '$1' && echo 1 || echo 0

and then restarted the Zabbix agent process on that machine with

systemctl restart zabbix-agent

.

Then on the Zabbix UI I:

• Selected a suitable host.
• Added an interface for the machine I wanted to monitor.
• Added an Item of type Zabbix Agent with Host interface as per the previous step.

This is when I get stuck. The SO answer tells you to create an item with key systemd.unit.is-active[postfix] (the SO example uses ssh but I want to monitor Postfix) but the UI has a Select option which offers a list of item keys which does not include systemd.unit.is-active[*]. I can just ignore the list and type systemd.unit.is-active[postfix] anyway but then the Execute Now button is grayed/disabled. So it looks like the SO answer is telling you to create an item key which cannot be referenced. What is the missing part of the SO answer, or where is my understanding at fault?

[vladimir_lv]

Hi!
The list of item keys consists of predefined keys only and your self-made keys cannot be discovered.

Execute Now button is grayed/disabled

Probably, you used type as Zabbix Agent (Active), because why the button is grayed.

[michaelNGV]

Thanks for the info. re. keys.
My item is definitely Zabbix Agent.

[vladimir_lv]

Your host is disabled. ))

[cyber]

Maybe because your host is disabled?

[michaelNGV]

Good catch. I didn't set up that host. Now, the item appears to be in error:I hope it's in order to post these big screenshots.

[Hamardaban]

Try to specify the full path /bin/systemctl

[cyber]

"No such file or directory" can only come from output of your UserParameter... IIRC [2] indicates standard error output... Something related to executing systemctl?

[Markku]

Also note that ideally the item type should be "Numeric (unsigned)" in this case as it only returns 1 or 0.

Markku

[michaelNGV]

In the agent logs:

23986:20220816:150011.216 Failed to execute command "/bin/systemctl is-active --quiet 'postfix' && /bin/echo 1 || /bin/echo 0": [2] No such file or directory

so there it is. However, even the replacement

/bin/systemctl is-active 'postfix' | /bin/sed -e 's:^active$:1:' -e 's:^[^1].*:0:'

gives:

24068:20220816:151250.451 Failed to execute command "/bin/systemctl is-active 'postfix' | /bin/sed -e 's:^active$:1:' -e 's:^[^1].*:0:'": [2] No such file or directory

so I'm at a loss.

[Markku]

Is your systemctl in /bin? Use "which systemctl"

What is your Linux distribution and version?

Markku

[michaelNGV]

CentOS 7. I can run that command in the shell; it only doesn't work when Zabbix is running it. I set DebugLevel=5 but it didn't help much. The logs have:

25696:20220817:091351.402 zbx_setproctitle() title:'listener [NODE="2"]Forum[/NODE] [processing request]'
25696:20220817:091351.403 Requested [systemd.unit.is-active[postfix]]
25696:20220817:091351.403 In zbx_popen() command:'/bin/systemctl is-active 'postfix' | /bin/sed -e 's:^active$:1:' -e 's:^[^1].*:0:''
25696:20220817:091351.403 Failed to execute command "/bin/systemctl is-active 'postfix' | /bin/sed -e 's:^active$:1:' -e 's:^[^1].*:0:'": [2] No such file or directory
25696:20220817:091351.403 Sending back [ZBX_NOTSUPPORTED: [2] No such file or directory]
25696:20220817:091351.403 zbx_setproctitle() title:'listener [NODE="2"]Forum[/NODE] [waiting for connection]'

If I copy-and-paste

/bin/systemctl is-active 'postfix' | /bin/sed -e 's:^active$:1:' -e 's:^[^1].*:0:'

into my ssh command-prompt, it works as expected.

[root@smtp02 ~]# /bin/systemctl is-active 'postfix' | /bin/sed -e 's:^active$:1:' -e 's:^[^1].*:0:'
1
[root@smtp02 ~]# which systemctl
/bin/systemctl

[michaelNGV]

Found it.

UserParameterDir=/etc/zabbix/zabbix-agentd.d

but that directory did not exist. Nothing to do with the command itself. Rather unclear error message.

[ichano186]

I'd use zabbix agent2 and default systemd monitoring template provided by zabbix installation instead

https://www.zabbix.com/integrations/systemd

you can play around the filters of the discovery setting to fine tune with which services do you need to monitor