Timeout with Zabbix Agent2 using docker.data_usage key

[minsis]

I'm getting some kind of issue with docker checks with agent2. Basically it seems like the connection to the local docker socket is timing out. I'm running this as root so I dont think it would be any kind of permission issues. This only happened after I upgraded docker from 18.X to 20.X. I have other servers that were directly isntalled with 20.x which work just fine. Any help would be appreciated. I also increased the timeout for the docker plugin to 30 seconds to see if that did anything and it still times outs. Other docker tests like docker.ping work just fine.

zabbix_agent2 -t docker.data_usage -v
2021/06/11 17:15:36.261161 using plugin 'Agent' providing following interfaces: exporter
2021/06/11 17:15:36.261285 using plugin 'Ceph' providing following interfaces: exporter, runner, configurator
2021/06/11 17:15:36.261296 using plugin 'Cpu' providing following interfaces: exporter, collector, runner
2021/06/11 17:15:36.261303 using plugin 'Docker' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261311 using plugin 'File' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261321 using plugin 'Kernel' providing following interfaces: exporter
2021/06/11 17:15:36.261328 using plugin 'Log' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261334 using plugin 'MQTT' providing following interfaces: watcher, configurator
2021/06/11 17:15:36.261340 using plugin 'Memcached' providing following interfaces: exporter, runner, configurator
2021/06/11 17:15:36.261581 using plugin 'Memory' providing following interfaces: exporter
2021/06/11 17:15:36.261595 using plugin 'Modbus' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261602 using plugin 'Mongo' providing following interfaces: exporter, runner, configurator
2021/06/11 17:15:36.261618 using plugin 'Mysql' providing following interfaces: exporter, runner, configurator
2021/06/11 17:15:36.261627 using plugin 'NetIf' providing following interfaces: exporter
2021/06/11 17:15:36.261634 using plugin 'Oracle' providing following interfaces: exporter, runner, configurator
2021/06/11 17:15:36.261642 using plugin 'Postgres' providing following interfaces: exporter, runner, configurator
2021/06/11 17:15:36.261657 using plugin 'Proc' providing following interfaces: exporter, collector
2021/06/11 17:15:36.261664 using plugin 'ProcExporter' providing following interfaces: exporter
2021/06/11 17:15:36.261670 using plugin 'Redis' providing following interfaces: exporter, runner, configurator
2021/06/11 17:15:36.261677 using plugin 'Smart' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261684 using plugin 'Sw' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261690 using plugin 'Swap' providing following interfaces: exporter
2021/06/11 17:15:36.261696 using plugin 'SystemRun' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261703 using plugin 'Systemd' providing following interfaces: exporter
2021/06/11 17:15:36.261710 using plugin 'TCP' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261717 using plugin 'UDP' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261723 using plugin 'Uname' providing following interfaces: exporter
2021/06/11 17:15:36.261729 using plugin 'Uptime' providing following interfaces: exporter
2021/06/11 17:15:36.261736 using plugin 'Users' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261743 using plugin 'VFSDev' providing following interfaces: exporter, collector
2021/06/11 17:15:36.261749 using plugin 'VfsFs' providing following interfaces: exporter
2021/06/11 17:15:36.261756 using plugin 'Web' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261762 using plugin 'ZabbixAsync' providing following interfaces: exporter
2021/06/11 17:15:36.261769 using plugin 'ZabbixStats' providing following interfaces: exporter, configurator
2021/06/11 17:15:36.261776 using plugin 'ZabbixSync' providing following interfaces: exporter
2021/06/11 17:15:36.261788 [Modbus] Config is valid
2021/06/11 17:15:36.261911 starting manager
2021/06/11 17:15:37.000167 [2] processing update request (1 requests)
2021/06/11 17:15:37.000188 [2] registering new client
2021/06/11 17:15:37.000225 [2] adding new request for key: 'docker.data_usage'
2021/06/11 17:15:37.000966 [2] created direct exporter task for plugin 'Docker' itemid:0 key 'docker.data_usage'
2021/06/11 17:15:37.000982 [2] created configurator task for plugin Docker
2021/06/11 17:15:37.001005 plugin Docker: executing configurator task
2021/06/11 17:15:37.001078 executing direct exporter task for key 'docker.data_usage'
docker.data_usage [m|ZBX_NOTSUPPORTED] [Timeout occurred while gathering data.]
2021/06/11 17:15:40.001582 failed to execute direct exporter task for key 'docker.data_usage' error: 'Cannot fetch data: Get http://1.28/system/df: net/http: request canceled (Client.Timeout exceeded while awaiting headers).'
2021/06/11 17:15:40.001638 manager has been stopped

[minsis]

Bumping to see if anyone has any ideas.

[minsis]

Seriously no one can give any advice on this?

[cholywell]

Were you able to solve this? I am having the same issue. Multiple fresh Debian 11 installs with Docker 20 and Btrfs storage. I can't seem to solve this either. With about 28 containers running this bug cause the cpu to jump over 100%.

[cholywell]

I was able to narrow down the problem further. My /var/lib/docker was on a Btrfs filesystem and docker was using the Btrfs storage driver. Once I moved /var/lib/docker to an ext4 volume with the overlay2 storage driver this was resolved. No more timeouts and high cpu usage. Looks like "docker.data_usage" is broken when polling docker on Btrfs.

[splitek]

I thing bug is in GO or in Btrfs tools.

[Janhouse]

I was able to narrow down the problem further. My /var/lib/docker was on a Btrfs filesystem and docker was using the Btrfs storage driver. Once I moved /var/lib/docker to an ext4 volume with the overlay2 storage driver this was resolved. No more timeouts and high cpu usage. Looks like "docker.data_usage" is broken when polling docker on Btrfs.

I'm not so sure that it was the cause of the problem. I have docker running on m.2 nvme storage with ext4 using overlay2 storage driver and I still get the issue where zabbix-agent2 causes high CPU usage on dockerd.