Ad Widget

Collapse

Monitor a docker host through zabbix-proxy

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • aenright
    Junior Member
    • Aug 2026
    • 3

    #1

    Monitor a docker host through zabbix-proxy

    Hello Forum,

    I'm trying to use the Docker by Zabbix agent 2 template to monitor some hosts. The Zabbix Server is Rocky Linux 9.8 fully updated running Server 7.0.29 LTS. The proxy host and docker hosts are running Rocky Linux 10.2 with all packages up to date.

    Code:
                                     ┌───────────────────────────────────┐
    ┌──────────────────┐             │                                   │            
    │                  │             │ ┌──────────┐     ┌───────────┐    │            
    │                  │             │ │ Zabbix   │     │           │    │            
    │  Zabbix Server   │◄────────────│─┤ Proxy    ├────►│ Docker 1  │    │            
    │                  │             │ │          │     │           │    │            
    │                  │             │ └─────┬────┘     └───────────┘    │            
    └──────────────────┘             │       │          ┌───────────┐    │            
                                     │       │          │           │    │            
                                     │       └─────────►│ Docker 2  │    │            
                                     │                  │           │    │            
                                     │                  └───────────┘    │            
                                     │                                   │            
                                     └───────────────────────────────────┘
    Both of the docker hosts are working as expected with the Linux by Zabbix agent active templates. Firewall is open on the hosts to allow communication FROM the proxy to the docker hosts. I am also able to successfully run zabbix_get against the docker hosts FROM the proxy:

    Code:
    # zabbix_get -s 192.168.122.105 -k docker.info
    {"ID":"c8937113-392b-4e53-814e-7ad79cf61655","Containers":13,"ContainersRunning":10,"ContainersPaused":0,"ContainersStopped":3,"Images":12,"Driver":"overlayfs","MemoryLimit":true,"SwapLimit":true,"CpuCfsPeriod":true,"CpuCfsQuota":true,"CPUShares":true,"CPUSet":true,"PidsLimit":true,"IPv4Forwarding":true,"Debug":false,"NFd":109,"OomKillDisable":false,"NGoroutines":117,"LoggingDriver":"json-file","CgroupDriver":"systemd","NEventsListener":1,"KernelVersion":"6.12.0-211.44.1.el10_2.x86_64","OperatingSystem":"Rocky Linux 10.2 (Red Quartz)","OSVersion":"10.2","OSType":"linux","Architecture":"x86_64","IndexServerAddress":"https://index.docker.io/v1/","NCPU":4,"MemTotal":8320057344,"DockerRootDir":"/var/lib/docker","Name":"tk01.prod.fnt.intra.nbnav.net","ExperimentalBuild":false,"ServerVersion":"29.7.2","DefaultRuntime":"runc","LiveRestoreEnabled":false,"InitBinary":"docker-init","SecurityOptions":["name=seccomp,profile=builtin","name=cgroupns"],"Warnings":null}​
    Despite this, I'm still getting Docker: Service is down alerts and none of the data is making it to the server:



    zabbix-agent2.conf contents for the two hosts:

    Code:
    PidFile=/run/zabbix/zabbix_agent2.pid
    LogFile=/var/log/zabbix/zabbix_agent2.log
    LogFileSize=10
    Server=192.168.122.100
    ServerActive=192.168.122.100
    Hostname=docker01.example.com
    PluginSocket=/run/zabbix/agent.plugin.sock
    ControlSocket=/run/zabbix/agent.sock
    TLSConnect=psk
    TLSPSKIdentity=fnt-vm-tk-01
    TLSPSKFile=/etc/zabbix/certs/fnt-vm-tk-01.psk
    Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf
    Include=/etc/zabbix/zabbix_agent2.d/*.conf​
    Code:
    PidFile=/run/zabbix/zabbix_agent2.pid
    LogFile=/var/log/zabbix/zabbix_agent2.log
    LogFileSize=10
    Server=192.168.122.100
    ServerActive=192.168.122.100
    Hostname=docker02.example.com
    PluginSocket=/run/zabbix/agent.plugin.sock
    ControlSocket=/run/zabbix/agent.sock
    TLSConnect=psk
    TLSPSKIdentity=fnt-vm-semui-01
    TLSPSKFile=/etc/zabbix/certs/fnt-vm-semui-01.psk
    Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf
    Include=/etc/zabbix/zabbix_agent2.d/*.conf​
    I'm not seeing anything in the respective log files on the docker hosts or the proxy.

    Not sure why data isn't getting collected/transmitted to the server from the proxy when zabbix_get from the proxy can clearly collect docker.info.

    Any help is greatly appreciated!​
  • rozols
    Junior Member
    • Aug 2026
    • 9

    #2

    Hi,

    Checked you provided details to me it seems that there is some misconfiguration in config or TLS encryption.

    I tried to replicate the issue and managed to do so to some extent using your provided config for agents on docker server.
    Agent config:
    Code:
    PidFile=/run/zabbix/zabbix_agent2.pid
    LogFile=/var/log/zabbix/zabbix_agent2.log
    LogFileSize=0
    Server=xxx.28.96.110 <--- Proxy IP
    ServerActive=xxx.28.96.110 <-- Proxy IP
    Hostname=docker01.example.com
    PluginSocket=/run/zabbix/agent.plugin.sock
    ControlSocket=/run/zabbix/agent.sock
    TLSConnect=psk
    TLSPSKIdentity=docker01-tk
    TLSPSKFile=/etc/zabbix/certs/docker01.psk
    Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf
    Include=/etc/zabbix/zabbix_agent2.d/*.conf
    Zabbix-Get:
    Code:
    zabbix_get -s xxx.28.96.112 -k docker.info
    {"ID":"2e40e3f8-209f-49ac-8cfd-7ffdb82f8afc","Containers":2,"ContainersRunning":2,"ContainersPaused":0,"ContainersStopped":0,"Images":2,"Driver":"overlayfs","MemoryLimit":true,"SwapLimit":true,"CpuCfsPeriod":true,"CpuCfsQuota":true,"CPUShares":true,"CPUSet":true,"PidsLimit":true,"IPv4Forwarding":true,"Debug":false,"NFd":37,"OomKillDisable":false,"NGoroutines":60,"LoggingDriver":"json-file","CgroupDriver":"systemd","NEventsListener":0,"KernelVersion":"6.12.0-211.16.1.el10_2.0.1.x86_64","OperatingSystem":"Rocky Linux 10.2 (Red Quartz)","OSVersion":"10.2","OSType":"linux","Architecture":"x86_64","IndexServerAddress":"https://index.docker.io/v1/","NCPU":2,"MemTotal":1036238848,"DockerRootDir":"/var/lib/docker","Name":"docker01.example.com","ExperimentalBuild":false,"ServerVersion":"29.7.2","DefaultRuntime":"runc","LiveRestoreEnabled":false,"InitBinary":"docker-init","SecurityOptions":["name=seccomp,profile=builtin","name=cgroupns"],"Warnings":null}
    With this config everything worked i got container information via passive checks using template (Docker by Zabbix agent 2) and server information with active checks using template (Linux by Zabbix agent active), to me it seems you have either misconfiguration on frontend or in the conf file on docker host.

    Few things to checks -
    • In frontend under Administration → Proxies → [your proxy] — does it show a recent "Last seen"?
    • Check the hosts Encryption tab — is PSK required for "Connections to host" and "Connections from host"? If PSK is used for both options then you need to add TLSAccept=psk in agent config on docker hosts.
    • In agent config the Server/ServerActive ip address is proxy IP, not Zabbix server?
    • Check /var/log/zabbix/zabbix_agent2.log on a docker host right after a poll cycle — in my repro, both bugs produced clear log lines once I looked, even though you mentioned seeing nothing in your logs. If yours are truly silent, it's possible the proxies connection isn't reaching the agent at all (firewall dropping the packet before either side logs anything) rather than being actively rejected — worth checking connectivity from the proxy to the host on port 10050 directly.

    Comment

    • aenright
      Junior Member
      • Aug 2026
      • 3

      #3
      Hi rozols,

      Thanks for helping out!

      Under Administration -> Proxies -> the proxy is seen with a matter of seconds:
      Click image for larger version

Name:	Screenshot 2026-08-18 at 4.54.22 PM.png
Views:	27
Size:	70.5 KB
ID:	516002
      The proxy in question is FNT Proxy.

      I have 'No encryption' for 'Conections to host' and PSK for 'Connections from host':
      Click image for larger version

Name:	Screenshot 2026-08-18 at 4.56.20 PM.png
Views:	20
Size:	48.7 KB
ID:	516003
      Later tonight I'll try completely disabling PSK for one of the hosts to see if that changes the behavior

      In the agent config, the Server/ServerActive is set to the IP of the proxy (and the Linux by Agent active collection is working as expected on the hosts)

      I restarted the proxy hoping that would kick off a poll cycle but didn't see anything in the docker hosts zabbix_agent2.log I left a tail up on the log for a while to see if anything came through and unless I restarted the agent, I wasn't getting any log messages.

      I assume zabbix_get -s <IP> -k docker.info FROM the proxy hosts connects to the docker host on port 10050 but I installed telnet and tested just to be sure. I was able to connect via telnet and after a few moments the agent closed the connection and logged an i/o timeout message in zabbix_agent2.log.

      I'll post back once I disable PSK for connections from host and test again.

      Best,
      Arthur

      Comment

      • rozols
        Junior Member
        • Aug 2026
        • 9

        #4

        Hi,

        From your provided details seems like the PSK is not the issue as it is configured the same way i have in testlab, more it seems like something is failing to poll the data via the proxy.

        For more details in proxy log you could increase the debug level to 4 and then tail -f proxy log to see some errors.
        Code:
        sudo zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf -R log_level_increase
        This increases the debug level at runtime, so no need to restart proxy after.

        Few things to check -
        • in latest data for the docker items do you see some kind of error a red indication where it shows error why data is not collected or maybe on host?
        • are the items enabled on host to collect?
        • in host settings Linux by Agent active is using active checks, but Docker template is using passive checks, so interface is needed for the host, has this been setup?
        Example log entry -
        3830:20260819:094541.428 In zbx_async_check_agent() itemid:50981 key:'docker.container_stats["/test-redis"]' host:'docker01.example.com' addr:'xxx.28.96.112' conn:'unencrypted'
        3830:20260819:094541.429 End of zbx_async_check_agent():SUCCEED


        ​My host config in frontend -
        Click image for larger version

Name:	image.png
Views:	24
Size:	40.1 KB
ID:	516011
        Click image for larger version

Name:	image.png
Views:	17
Size:	16.8 KB
ID:	516012

        Comment

        • aenright
          Junior Member
          • Aug 2026
          • 3

          #5
          Ok, I'm really confused now - here is an example of the proxy seemingly collecting docker.containers successfully:
          Code:
          In zbx_async_check_agent() itemid:54189 key:'docker.containers' host:'tk01.prod.fnt.intra.nbnav.net' addr:'192.168.122.100' conn:'unencrypted'
           85428:20260819:095709.393 In zbx_is_ip4() ip:'192.168.122.100'
           85428:20260819:095709.393 End of zbx_is_ip4():SUCCEED
           85428:20260819:095709.393 In zbx_is_ip4() ip:'192.168.122.100'
           85428:20260819:095709.393 End of zbx_is_ip4():SUCCEED
           85428:20260819:095709.393 In ares_addrinfo_cb() result:0 timeouts:0
           85428:20260819:095709.393 resolved address '192.168.122.100'
           85428:20260819:095709.393 In async_event()
           85428:20260819:095709.393 In agent_task_process() step 'init' 'init' event:0 itemid:54189 addr:192.168.122.100
           85428:20260819:095709.393 In zbx_is_ip4() ip:'192.168.122.100'
           85428:20260819:095709.393 End of zbx_is_ip4():SUCCEED
           85428:20260819:095709.393 End of async_event():write
           85428:20260819:095709.393 End of ares_addrinfo_cb()
           85428:20260819:095709.393 End of zbx_async_check_agent():SUCCEED​
          Here is the host setup:

          Click image for larger version

Name:	Screenshot 2026-08-19 at 9.00.31 AM.png
Views:	22
Size:	153.3 KB
ID:	516038

          I unlink and cleared Docker by Zabbix agent 2 and re-added it to see if something was stale there but it still hasn't picke dup any data for the host dashboard as-of-yet.

          Comment

          • rozols
            Junior Member
            • Aug 2026
            • 9

            #6
            Hi,

            Provided log confirms the async request was dispatched and the connection to 192.168.122.100:10050 was set up correctly — but it stops right at connection setup, before the actual response. So this doesnt tell whether the poll actually succeeded or failed.
            A few things to check:

            Full trace for this item ID — can you grep the log for everything tagged with this item?
            Code:
            grep "itemid:54189" /var/log/zabbix/zabbix_proxy.log
            There should be a later log entry (still at debug level 4) showing the actual response being received and parsed, or an error if it wasnt.
            Item state, not just Info column, check the status and state columns for the master items (docker.info and docker.containers). Check if they are Enabled and showing "Normal" state, not "Not supported." If Not supported there should be Info column with red icon and information about error.

            Grep the proxy log for the data senders activity:
            Code:
            grep "data sender" /var/log/zabbix/zabbix_proxy.log | tail -20
            If you see something like data sender [sent 0 values in ... sec, idle ...] repeating with 0 values over time, that would mean the proxy has nothing new to send — which would point back to the poll itself not producing data, rather than an upload problem. If it shows a nonzero count that is growing, but the server frontend still shows nothing, that is actually a different problem (server-side ingestion, possibly clock skew between proxy and server).​

            Comment

            Working...