7.4 - Docker Deployment - Running Scripts on Agent

[Arrace]

I'm running zabbix 7.4 in a docker swarm deployment and I'm having issues running a custom script from the host selection in both the GUI/CLI of zabbix server.


Running it from the GUI results in

Running it on CLI from the proxy results in the same error
root@zabbix_proxy_CLT:/var/lib/zabbix# zabbix_get -s <IP> -k system.run[/usr/local/bin/zabbix_iperf.sh]
ZBX_NOTSUPPORTED: Unknown metric system.run


Looks like the config that I have on the host is active

planta-dh01:~/docker/zabbix$ cat /usr/local/bin/zabbix_iperf.sh
sudo ls /​

planta-dh01:~/docker/zabbix$ sudo cat /etc/zabbix/zabbix_agent2.conf | grep AllowKey
AllowKey=system.run[/usr/local/bin/zabbix_iperf.sh,*]
​
planta-dh01:~/docker/zabbix$ ps aux | grep zabbix_agent2
1997 2197962 0.4 0.2 1256536 26556 ? Ssl Jul15 4:42 /usr/sbin/zabbix_agent2 --foreground -c /etc/zabbix/zabbix_agent2.conf
zabbix 2361067 0.2 0.2 2435704 28712 ? Ssl 09:08 0:01 /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf
user1 2384640 0.0 0.0 6676 2304 pts/0 S+ 09:23 0:00 grep --color=auto zabbix_agent2

When I run the command on the linux server using the user zabbix it returns correctly

planta-dh01:~/docker/zabbix$ sudo zabbix_agent2 -t system.run[/usr/local/bin/zabbix_iperf.sh]
system.run[/usr/local/bin/zabbix_iperf.sh] [s|bin
bin.usr-is-merged
boot
cdrom
dev
etc
home
lib
lib64
lib.usr-is-merged
lost+found
media
mnt
opt
proc
root
run
sbin
sbin.usr-is-merged
snap
srv
swap.img
sys
tmp
usr
var]
​

[PavelZ]

When I run the command on the linux server using the user zabbix it returns correctly

Now to debug such a launch, you must first run docker exec -it <NNNNN> <command>
or run : docker compose exec <agent> <command>

Since you also decided to run all this in Docker Swarm, all you have to do is build an image with all the scripts.
It is inconvenient to make volume with scripts in Swarm.

[Observer365]

Hello,
As I see you use sudo to run the zabbix_agent -t command from agent server. If it require password it could be a reason why the item become unsupported. You can add

zabbix ALL=(ALL) NOPASSWD: <your command>

to sudoers file.
The other reason that could make the key unsupported it to put some DenyKey directive before the AllowKey. Usually DenyKeys (if exists) should be after AllowKeys.
Also make sure that you agent allow connection from server/proxy in passive mode. For this Server= directive in agent configuration file should contain server/proxy IP or domain names.
If you use a proxy between server and agent then EnableRemoteCommands should be enabled in proxy configuration.

[Arrace]

Now to debug such a launch, you must first run docker exec -it
or run : docker compose exec

Since you also decided to run all this in Docker Swarm, all you have to do is build an image with all the scripts.
It is inconvenient to make volume with scripts in Swarm.

So the command was ran inside the container, I had already exec into the container prior to running the command. If I'm miss-reading something here let me know. I'm not too sure on the volume script portion. I wouldn't see why it would matter using a custom image with scripts included, or just mounting them through a volume. The way it is now using a volume I can change scripts without having to re-do the entire image. This also allows us to utilize the stock zabbix/zabbix-server or zabbix/zabbix-proxy images.

However our script is being executed by the agent installed on the server we're monitoring, so doesn't the script live on whatever server we're monitoring instead of the proxy/server?

[Arrace]

Hello,
As I see you use sudo to run the zabbix_agent -t command from agent server. If it require password it could be a reason why the item become unsupported. You can add to sudoers file.
The other reason that could make the key unsupported it to put some DenyKey directive before the AllowKey. Usually DenyKeys (if exists) should be after AllowKeys.
Also make sure that you agent allow connection from server/proxy in passive mode. For this Server= directive in agent configuration file should contain server/proxy IP or domain names.
If you use a proxy between server and agent then EnableRemoteCommands should be enabled in proxy configuration.

I'll play with editing the sudoers file
Here's our config for the agent referencing allow/deny keys. I realize that there could be a deny key outside of the commented section but that isn't the case. So we don't have a deny key ahead of the allow key in the agent's configuration file

# Plugins.Log.MaxLinesPerSecond=20

### Option: AllowKey
#       Allow execution of item keys matching pattern.
#       Multiple keys matching rules may be defined in combination with DenyKey.
#       Key pattern is wildcard expression, which support "*" character to match any number of any characters in certain position. It might be used in both key name and key arguments.
#       Parameters are processed one by one according their appearance order.
#       If no AllowKey or DenyKey rules defined, all keys are allowed.
#
# Mandatory: no
AllowKey=system.run[/usr/local/bin/zabbix_iperf.sh,*]

### Option: DenyKey
#       Deny execution of items keys matching pattern.
#       Multiple keys matching rules may be defined in combination with AllowKey.
#       Key pattern is wildcard expression, which support "*" character to match any number of any characters in certain position. It might be used in both key name and key arguments.
#       Parameters are processed one by one according their appearance order.
#       If no AllowKey or DenyKey rules defined, all keys are allowed.
#       Unless another system.run[*] rule is specified DenyKey=system.run[*] is added by default.
#
# Mandatory: no
# Default:
# DenyKey=system.run[*]

### Option: Plugins.SystemRun.LogRemoteCommands

I double checked the passive server settings and it's setup to allow connections from any of our docker hosts that run zabbix proxies/servers. The device also shows as available and is getting updated item polling


I isolated this host to a specific proxy and made sure that the enable remote commands is enabled for the proxy. I've confirmed this by running a ping check on the same host drop down context menu



This deployment is in a proof of concept environment for now, so I think I'm going to try the same setup on our production instance that's standalone zabbix server hosted on linux without any containerization or proxies and see where I get with that.

[Markku]

So you have two scripts: "iPerf - CLT" and "Ping". Ping script works, and iPerf - CLT does not. If I understood this correctly, what is the difference between the script configurations?

Markku

[Arrace]

The ping script is a default zabbix script that's ran from the proxy/server

Whereas the iPerf - CLT is a script ran on the agent's side


Almost seems like there's some sort of disconnect between the Zabbix Server and Zabbix Agent. I get the same result outside of the docker setup where I see the following debug log from the server when running the script.

1081400:20250718:074926.300 Sending [{"request":"passive checks","data":[{"key":"system.run[/usr/local/bin/zabbix_iperf.sh]","timeout":4}]}]
1081400:20250718:074926.301 get value from agent result: '{"version":"7.4.0","variant":2,"data":[{"error":"Unknown metric system.run"}]}'​

But yet I can run system.run directly on the agent
I also didn't see anything on the agent side about a script request being ran or erroring out (debug level 4)
I did modify the sudoers file to allow zabbix to run anything without password since this is a POC

sudo -u zabbix zabbix_agent2 -t system.run[/usr/local/bin/zabbix_iperf.sh]
system.run[/usr/local/bin/zabbix_iperf.sh] [s|bin
bin.usr-is-merged
boot
cdrom
dev
etc
home
lib
lib64
lib.usr-is-merged
lost+found
media
mnt
opt
proc
root
run
sbin
sbin.usr-is-merged
snap
srv
swap.img
sys
tmp
usr
var]​

[Markku]

I don't know, it works here:

markku@testzabbixagent:~$ grep ^AllowKey /etc/zabbix/zabbix_agent2.d/agent.conf
AllowKey=system.run[/usr/local/bin/zabbix_iperf.sh,*]
markku@testzabbixagent:~$ ls -l /usr/local/bin/zabbix_iperf.sh
-rwxr-xr-x 1 root root 9 Jul 18 15:31 /usr/local/bin/zabbix_iperf.sh
markku@testzabbixagent:~$ cat /usr/local/bin/zabbix_iperf.sh
hostname
markku@testzabbixagent:~$ sudo -u zabbix zabbix_agent2 -t system.run[/usr/local/bin/zabbix_iperf.sh]
system.run[/usr/local/bin/zabbix_iperf.sh]    [s|testzabbixagent]
markku@testzabbixagent:~$ /usr/sbin/zabbix_agent2 -V | head -1
zabbix_agent2 (Zabbix) 7.0.16
markku@testzabbixagent:~$

On the server:

markku@zabbix70-1:~$ zabbix_get -s 192.168.7.105 -k system.run[/usr/local/bin/zabbix_iperf.sh]
testzabbixagent
markku@zabbix70-1:~$

No sudoers configurations for zabbix user here on the agent.

(This was with 7.0.16 but I also quickly tested with a 7.4.0 zabbix_agent2 and it worked the same: with AllowKey defined in the agent the zabbix_get call worked fine on the server)

Markku

[Markku]

I also didn't see anything on the agent side about a script request being ran or erroring out (debug level 4)

Isn't that meaning that your host configuration or something else is messed up if the call never even reaches the agent?

Markku

[Arrace]

Markku,

Thanks for the assistance with this, I think I'm going fire up a couple VMs fresh and test from scratch. At least then I might see some progress on my end haha.

I'm not really sure what would be stopping this in my current environment since it seems like all the passive items work but the script doesn't reach the remote agent. But at least I can see from scratch like your test to get it working and compare with. I even copied your allow key, changed my script contents to match yours and still couldn't get the server side to work

I probably won't get around to this until next week, but I'll hopefully have some sort of update. Worse case is I can still achieve an iperf test but from our proxy/server->host instead of the remote host -> dedicated iperf server.

[PavelZ]

All my scripts run in a container.
Of course, build a custom server image.Especially if these are scripts on python or perl.