Monitor a docker host through zabbix-proxy

[aenright]

Hello Forum,

I'm trying to use the Docker by Zabbix agent 2 template to monitor some hosts. The Zabbix Server is Rocky Linux 9.8 fully updated running Server 7.0.29 LTS. The proxy host and docker hosts are running Rocky Linux 10.2 with all packages up to date.

                                 ┌───────────────────────────────────┐
┌──────────────────┐             │                                   │            
│                  │             │ ┌──────────┐     ┌───────────┐    │            
│                  │             │ │ Zabbix   │     │           │    │            
│  Zabbix Server   │◄────────────│─┤ Proxy    ├────►│ Docker 1  │    │            
│                  │             │ │          │     │           │    │            
│                  │             │ └─────┬────┘     └───────────┘    │            
└──────────────────┘             │       │          ┌───────────┐    │            
                                 │       │          │           │    │            
                                 │       └─────────►│ Docker 2  │    │            
                                 │                  │           │    │            
                                 │                  └───────────┘    │            
                                 │                                   │            
                                 └───────────────────────────────────┘

Both of the docker hosts are working as expected with the Linux by Zabbix agent active templates. Firewall is open on the hosts to allow communication FROM the proxy to the docker hosts. I am also able to successfully run zabbix_get against the docker hosts FROM the proxy:

# zabbix_get -s 192.168.122.105 -k docker.info
{"ID":"c8937113-392b-4e53-814e-7ad79cf61655","Containers":13,"ContainersRunning":10,"ContainersPaused":0,"ContainersStopped":3,"Images":12,"Driver":"overlayfs","MemoryLimit":true,"SwapLimit":true,"CpuCfsPeriod":true,"CpuCfsQuota":true,"CPUShares":true,"CPUSet":true,"PidsLimit":true,"IPv4Forwarding":true,"Debug":false,"NFd":109,"OomKillDisable":false,"NGoroutines":117,"LoggingDriver":"json-file","CgroupDriver":"systemd","NEventsListener":1,"KernelVersion":"6.12.0-211.44.1.el10_2.x86_64","OperatingSystem":"Rocky Linux 10.2 (Red Quartz)","OSVersion":"10.2","OSType":"linux","Architecture":"x86_64","IndexServerAddress":"https://index.docker.io/v1/","NCPU":4,"MemTotal":8320057344,"DockerRootDir":"/var/lib/docker","Name":"tk01.prod.fnt.intra.nbnav.net","ExperimentalBuild":false,"ServerVersion":"29.7.2","DefaultRuntime":"runc","LiveRestoreEnabled":false,"InitBinary":"docker-init","SecurityOptions":["name=seccomp,profile=builtin","name=cgroupns"],"Warnings":null}​

Despite this, I'm still getting Docker: Service is down alerts and none of the data is making it to the server:



zabbix-agent2.conf contents for the two hosts:

PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=10
Server=192.168.122.100
ServerActive=192.168.122.100
Hostname=docker01.example.com
PluginSocket=/run/zabbix/agent.plugin.sock
ControlSocket=/run/zabbix/agent.sock
TLSConnect=psk
TLSPSKIdentity=fnt-vm-tk-01
TLSPSKFile=/etc/zabbix/certs/fnt-vm-tk-01.psk
Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf
Include=/etc/zabbix/zabbix_agent2.d/*.conf​

PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=10
Server=192.168.122.100
ServerActive=192.168.122.100
Hostname=docker02.example.com
PluginSocket=/run/zabbix/agent.plugin.sock
ControlSocket=/run/zabbix/agent.sock
TLSConnect=psk
TLSPSKIdentity=fnt-vm-semui-01
TLSPSKFile=/etc/zabbix/certs/fnt-vm-semui-01.psk
Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf
Include=/etc/zabbix/zabbix_agent2.d/*.conf​

I'm not seeing anything in the respective log files on the docker hosts or the proxy.

Not sure why data isn't getting collected/transmitted to the server from the proxy when zabbix_get from the proxy can clearly collect docker.info.

Any help is greatly appreciated!

[rozols]

Hi,

Checked you provided details to me it seems that there is some misconfiguration in config or TLS encryption.

I tried to replicate the issue and managed to do so to some extent using your provided config for agents on docker server.
Agent config:

PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=0
Server=xxx.28.96.110 <--- Proxy IP
ServerActive=xxx.28.96.110 <-- Proxy IP
Hostname=docker01.example.com
PluginSocket=/run/zabbix/agent.plugin.sock
ControlSocket=/run/zabbix/agent.sock
TLSConnect=psk
TLSPSKIdentity=docker01-tk
TLSPSKFile=/etc/zabbix/certs/docker01.psk
Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf
Include=/etc/zabbix/zabbix_agent2.d/*.conf

Zabbix-Get:

zabbix_get -s xxx.28.96.112 -k docker.info
{"ID":"2e40e3f8-209f-49ac-8cfd-7ffdb82f8afc","Containers":2,"ContainersRunning":2,"ContainersPaused":0,"ContainersStopped":0,"Images":2,"Driver":"overlayfs","MemoryLimit":true,"SwapLimit":true,"CpuCfsPeriod":true,"CpuCfsQuota":true,"CPUShares":true,"CPUSet":true,"PidsLimit":true,"IPv4Forwarding":true,"Debug":false,"NFd":37,"OomKillDisable":false,"NGoroutines":60,"LoggingDriver":"json-file","CgroupDriver":"systemd","NEventsListener":0,"KernelVersion":"6.12.0-211.16.1.el10_2.0.1.x86_64","OperatingSystem":"Rocky Linux 10.2 (Red Quartz)","OSVersion":"10.2","OSType":"linux","Architecture":"x86_64","IndexServerAddress":"https://index.docker.io/v1/","NCPU":2,"MemTotal":1036238848,"DockerRootDir":"/var/lib/docker","Name":"docker01.example.com","ExperimentalBuild":false,"ServerVersion":"29.7.2","DefaultRuntime":"runc","LiveRestoreEnabled":false,"InitBinary":"docker-init","SecurityOptions":["name=seccomp,profile=builtin","name=cgroupns"],"Warnings":null}

With this config everything worked i got container information via passive checks using template (Docker by Zabbix agent 2) and server information with active checks using template (Linux by Zabbix agent active), to me it seems you have either misconfiguration on frontend or in the conf file on docker host.

Few things to checks -

• In frontend under Administration → Proxies → [your proxy] — does it show a recent "Last seen"?
• Check the hosts Encryption tab — is PSK required for "Connections to host" and "Connections from host"? If PSK is used for both options then you need to add TLSAccept=psk in agent config on docker hosts.
• In agent config the Server/ServerActive ip address is proxy IP, not Zabbix server?
• Check /var/log/zabbix/zabbix_agent2.log on a docker host right after a poll cycle — in my repro, both bugs produced clear log lines once I looked, even though you mentioned seeing nothing in your logs. If yours are truly silent, it's possible the proxies connection isn't reaching the agent at all (firewall dropping the packet before either side logs anything) rather than being actively rejected — worth checking connectivity from the proxy to the host on port 10050 directly.

[aenright]

Hi rozols,

Thanks for helping out!

Under Administration -> Proxies -> the proxy is seen with a matter of seconds:

The proxy in question is FNT Proxy.

I have 'No encryption' for 'Conections to host' and PSK for 'Connections from host':

Later tonight I'll try completely disabling PSK for one of the hosts to see if that changes the behavior

In the agent config, the Server/ServerActive is set to the IP of the proxy (and the Linux by Agent active collection is working as expected on the hosts)

I restarted the proxy hoping that would kick off a poll cycle but didn't see anything in the docker hosts zabbix_agent2.log I left a tail up on the log for a while to see if anything came through and unless I restarted the agent, I wasn't getting any log messages.

I assume zabbix_get -s <IP> -k docker.info FROM the proxy hosts connects to the docker host on port 10050 but I installed telnet and tested just to be sure. I was able to connect via telnet and after a few moments the agent closed the connection and logged an i/o timeout message in zabbix_agent2.log.

I'll post back once I disable PSK for connections from host and test again.

Best,
Arthur