Monitoring Services INSIDE Docker Containers

[dcsd]

I was wondering if someone could share their experience with monitoring services running inside docker containers.

We have a couple dozen containers running various services (nginx, php-fpm, mysql) and would like to utilize Zabbix functionality to get alerts. We have working solutions on traditional non containerized services, but are feeling a bit in the dark on how to go about doing this.

Our options are:

Install "super agent" locally on baremetal agent, that connects to the containers and performs the various checks. This poses some problems as we do not have access to the networks that the containers are running on, so would need to map external ports to the services (say mariadb:3306 for example). This could get ugly quick.

Install agents in each of the containers and perform manual configuration for each container. This seems time consuming, and a potential performance hit - Not to mention the fact we would lose the benefits of some pre-made packages (i.e. using the mariadb provided image from hub.docker.com - we would have to build our own images going forward each time a new release occurred).

Perform Server side monitoring only - Again we would have to expose specific ports on our network to gain access to those services.

Am I missing something? Information for monitoring docker containers is available on the internet but strictly for monitoring the CPU usage, Memory and so on, yet very bare on monitoring the actual services inside the containers themselves.

[jan.garaj]

I've published a few notes - https://github.com/monitoringartist/...ockerized-apps

[jnsvano]

Hello jan.garaj

I'm using your dockbix agent with zabbix_module_docker.so loadable module, seems to be it's pretty good.
But recently it began to alarm about container's swap usage, even if swap usage is very low (say, 5KB).
For example, latest data shows:
Used swap containter1 36 KB
Used swap container2 4.61 MB etc... when all swap is 8GB.
Trigger from your template looks like:
{server1:docker.mem[/container1,total_swap].avg(15m)}>75

So, where could be problem ?

[jan.garaj]

Trigger:

{server1:docker.mem[/container1,total_swap].avg(15m)}>75

= generate PROBLEM if total_swap is more than 75 B.

And that's TRUE if your containers use 36 kB or 4.61 MB.

[jnsvano]

Trigger:

{server1:docker.mem[/container1,total_swap].avg(15m)}>75

= generate PROBLEM if total_swap is more than 75 B.
And that's TRUE if your containers use 36 kB or 4.61 MB.

Was it intended or did the purpose was to alarm when it more than 75% ?

[jan.garaj]

No, my published Docker templates don't contain any triggers. That's your customisation.

[eryunt]

Hi, I am using Dockbix agent XXL inside a cluster running many Docker containers, and my own external Zabbix server.
Metrics from Dockers are correctly shown and triggered, but I furthermore need to monitor the status of some services running inside Dockers.
I think zabbix_module_docker.so could answer my needs, but I can't understand if It is automatically loaded inside Dockbix agent XXL or if I should add
-e "ZA_LoadModule=zabbix_module_docker.so" \
line when running Dockbix agent XXL Docker.

Thanks

[jan.garaj]

Nope. Docker module is already loaded in the Dockbix agent.

[eryunt]

Thank you!
Only services related to the machine where Dockbix XXL runs are monitored or also services inside every docker are supposed to be monitored?

Again thanks

[jnsvano]

jan.garaj: is there a way to get percent of swap using by container without calculated items ?

[jan.garaj]

@eryunt: that really depends on your needs, infrastructure, app design, etc.

@jan.garaj: nope, docker stats stream/cgroup stats don't provide percent values; you can write own module/script which will calculate it on the Zabbix agent side

[eryunt]

@eryunt: that really depends on your needs, infrastructure, app design, etc.

Thank you!
I asked because I can't visualize services related to my Dockers.
I used Template App systemd services.xml template.
Maybe something is missing?

[jan.garaj]

I guess you are mixing technologies. I recommend to read READMEs of both modules, which are included in the Dockbix agent:

https://github.com/monitoringartist/zabbix-docker-monitoring

https://github.com/cavaliercoder/zabbix-module-systemd

Docker container != systemd service, but systemd service can manage containers eventually and both technologies are based on cgroups.