Agent2 BUG - system.run is UNSUPPORTED

[erasedhammer]

I've had shutdown scripts for certain linux hosts in the event a UPS even happens, but upon upgrade to zabbix agent2, these system.run commands are no longer supported.
But according to the documentation:
https://www.zabbix.com/documentation...estrict_checks
It appears as system.run still works according to the docs, which it clearly does not.

From zabbix_agent2.conf

AllowKey=system.run[/usr/sbin/shutdown]
DenyKey=system.run[*]

When I try to test the item:

# zabbix_agent2 -t system.run[/usr/bin/shutdown]
system.run[/usr/bin/shutdown] [m|ZBX_NOTSUPPORTED] [Unknown metric system.run]


Running the remote command via triggered script in the gui results in:

Remote Command Failed Unknown metric system.run


Why is this functionality suddenly broken?
Is there any plan to fix this?
I think this feature is extremely important, and to see it left behind like this concerns me.

[erasedhammer]

Enabling debug mode at level 5 on the agent and recording the logs for the remote command.

These are all the logs surrounding the failed system.run key

<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.000705 plugin Cpu: executing collector task
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.325024 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.325633 [1] processing update request (1 requests)
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.649458 [1] processing update request (1 requests)
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.061030 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.061763 [1] adding new request for key: 'system.cpu.util[,softirq]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.325374 received passive check request: 'vfs.file.cksum[/etc/hosts]' from 'SERVERIP'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.326231 executing direct exporter task for key 'vfs.file.cksum[/etc/hosts]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.648801 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.325729 [1] adding new request for key: 'vfs.file.cksum[/etc/hosts]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.325801 [1] created direct exporter task for plugin 'File' itemid:0 key 'vfs.file.cksum[/etc/hosts]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.649131 received passive check request: 'system.run[/usr/sbin/shutdown,nowait]' from 'SERVERIP'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.649701 sending passive check response: ZBX_NOTSUPPORTED: 'Unknown metric system.run' to 'SERVERIP'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.061385 received passive check request: 'system.cpu.util[,softirq]' from 'SERVERIP'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.062349 executing direct exporter task for key 'system.cpu.util[,softirq]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.062425 executed direct exporter task for key 'system.cpu.util[,softirq]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.326573 executed direct exporter task for key 'vfs.file.cksum[/etc/hosts]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.326919 sending passive check response: '449343216' to 'SERVERIP'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.649620 [1] cannot monitor metric "system.run[/usr/sbin/shutdown,nowait]": Unknown metric system.run
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.061647 [1] processing update request (1 requests)
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.061842 [1] created direct exporter task for plugin 'Cpu' itemid:0 key 'system.cpu.util[,softirq]'
<30>May 11 16:44:30 zabbix_agent2[1724774]: 2022/05/11 16:44:30.062780 sending passive check response: '0.100351' to 'SERVERIP'

I used the checksum of /etc/hosts to trigger the test remote command. It pulled the checksum, then promptly ran the remote command.
The server, again, reports back: Remote Command Failed Unknown metric system.run

[erasedhammer]

Some version information:

# zabbix_agent2 -V

zabbix_agent2 (Zabbix) 5.4.12
Revision 9bd5a418b8 4 April 2022, compilation time: Apr 4 2022 15:38:01

Copyright (C) 2022 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit ([URL]http://www.openssl.org/[/URL]).

Compiled with OpenSSL 1.1.1k 25 Mar 2021
Running with OpenSSL 1.1.1n 15 Mar 2022

We use the library Eclipse Paho (eclipse/paho.mqtt.golang), which is
distributed under the terms of the Eclipse Distribution License 1.0 (The 3-Clause BSD License)
available at [URL]https://www.eclipse.org/org/documents/edl-v10.php[/URL]

We use the library go-modbus (goburrow/modbus), which is
distributed under the terms of the 3-Clause BSD License
available at [URL]https://github.com/goburrow/modbus/blob/master/LICENSE[/URL]

Server is Zabbix 6.0.3 running on Debian 11 (5.10.0-9-amd64 Home SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux)
Agent is 5.4.12 running on Debian 11 (5.10.0-9-amd64 Home SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux)

[erasedhammer]

I've attempted to try other commands through system.run key.

First changing the lines in zabbix_agent2.conf for all system.run keys:

AllowKey=system.run[*]

Trying the following tests on the host using zabbix_agent2 -t command:

# zabbix_agent2 -t system.run[/usr/sbin/ifconfig]        
system.run[/usr/sbin/ifconfig]                [m|ZBX_NOTSUPPORTED] [Unknown metric system.run]

# zabbix_agent2 -t system.run[/usr/bin/lsblk]     
system.run[/usr/bin/lsblk]                    [m|ZBX_NOTSUPPORTED] [Unknown metric system.run]

# zabbix_agent2 -t system.run[/usr/bin/df]    
system.run[/usr/bin/df]                       [m|ZBX_NOTSUPPORTED] [Unknown metric system.run]

# zabbix_agent2 -t system.run[/usr/bin/uname]
system.run[/usr/bin/uname]                    [m|ZBX_NOTSUPPORTED] [Unknown metric system.run]

None of those worked.
I'm surprised no one else has noticed this problem?

[Markku]

Are you sure you are editing the correct configuration file? I mean, just add a dummy TEST line in the configuration file and restart the agent, and verify that you get an error about it.

(I haven't used agent2 or system.run so I don't have direct experience about the features otherwise.)

Markku

[erasedhammer]

Are you sure you are editing the correct configuration file? I mean, just add a dummy TEST line in the configuration file and restart the agent, and verify that you get an error about it.

(I haven't used agent2 or system.run so I don't have direct experience about the features otherwise.)

Markku

I am pretty sure it is the correct file. /etc/zabbix/zabbix_agent2.conf

From the process list:
zabbix 433 1.2 3.1 1463404 44672 ? Ssl May11 12:48 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf

$ ls -l /etc/zabbix/zabbix_agent2.conf
-rw-r--r-- 1 root root 26570 May 11 16:44 /etc/zabbix/zabbix_agent2.conf


I did try adding quotes around system.run in the allowkey, and the agent did not like that and failed to start.
Plus I am using TLS, and the psk is not configured anywhere else (there are no other ".conf" files in /etc/zabbix)

[Markku]

Since you are using the Zabbix-provided Debian packages I think, it should be very easy for you to test with a supported agent version like 6.0.4 (5.4 is out of support), and if the behaviour persists, you could open a support ticket at support.zabbix.com.

Markku

[Markku]

Correct. https://www.zabbix.com/life_cycle_and_release_policy

No. https://www.zabbix.com/documentation.../compatibility

Markku

[gelowe]

Markku , Thank you for the details
Per the link on lifecycle https://www.zabbix.com/life_cycle_and_release_policy
It states, the end of life for Zabbix 5.0 LTS is May 31,2023. Do you know why is the Agent2 backporting of fixes also is not following this cycle or am I missing something?

[Markku]

Which problem in agent2 5.0.x are you talking about that doesn't have a fix?

Markku

[erasedhammer]

Since you are using the Zabbix-provided Debian packages I think, it should be very easy for you to test with a supported agent version like 6.0.4 (5.4 is out of support), and if the behaviour persists, you could open a support ticket at support.zabbix.com.

Markku

Switched one host over to the zabbix 6.0 repository and upgraded the agent2 from 5.4 to 6.0.
Package now installed:
zabbix-agent2/unknown,now 1:6.0.4-1+debian11 amd64 [installed]

Checking the system.run
zabbix_agent2 -t system.run[/usr/bin/uname]
system.run[/usr/bin/uname] [s|Linux]

Looks like its working now!

When I was checking support for the versions, I saw 5.0 LTS and thought, yep 5.4=5.0. Didn't know 5.4 was out of support.

Thank you !

[erasedhammer]

I was just testing some random things on the agent, and it appears theres a mismatch issue between server and client on a system.run item and a "execute script" command.

I changed the allowkey on the agent to now read while commenting out the actual specific run key for the script:
...
AllowKey=system.run[*]
....

And now the Execute Command from the server works!

So theres a mismatch between agent tests and what the server sends.
I've enabled debug=5 and reviewed the logs, and while the log appears to not log the local agent tests, it did catch the executed command from the server.

The Zabbix server log entry is:
2022/08/05 10:45:21.577157 received passive check request: 'system.run[/bin/testscript.sh,nowait]' from 'X'

Which is different than /bin/testscript.sh through the agent check:
zabbix_agent2 -t system.run[/bin/testscript.sh]

I saw some reference to this at the bottom of the page in documentation about Restricting agent checks
Almost missed it!

[gelowe]

Disregard this post, could not delete

[JoMABL]

Internally triggered calls add ,wait or , nowait, therefore * is not sufficient, but:
AllowKey=system.run[your_command*,*]