Zabbix + kubernetes

[gofree]

so maybe no need for the script - i managed via json path preprocessing - dependedt item prototypes created for cpu and mem per container - master item gets the data in one call - assuming that there are no two containers with same name on one namespace

[gofree]

not so bad news but still needs to be fixed

memory usage can be reported in Mi and Ki ( what I observed so far ) - when the memory usage is reported in Mi obviously the preprocessing rule fails on step 6

seem that solution could be java script checking for Ei, Pi, Ti, Gi, Mi, Ki suffixes and if doind doing the counting ( multiplication to bytes ) - anybody willing to write it ? guess it will take more for me than for somebody who can do it from scratch

[ik_zelf]

Maybe just activate the script once and have it return all the key/values in one go, using zabbix_sender?

[gofree]

Maybe just activate the script once and have it return all the key/values in one go, using zabbix_sender?

potentially yes , im trying to do some basic java script if else loop - maybe in future the metric could/will be obtained via http agent and there will be no zabbix sender needed

I'm asking myself if its all not bullshit what I'm doing and trying to achieve still learned a bit about new things

[gofree]

so with a bit of googling last step of preprocessing in java script

[ik_zelf]

Pushed a new version. Added arguments for host and zabbix-server, did formatting of the metrics, prepared input for zabbix-sender

[gofree]

I'll check tomorrow. Currently the issue is solved via Java script preprocessing. But every single code can be used later on. Just one more thing as it's like lego bricks now - at the end of the day when it comes to real production environment specific config files or parameterized conf files contexts ( auth tokens) might be needed for some kubectl commands. Maybe some bits will be done via http agent and some via kubectl. Just cherry picking

[ik_zelf]

Just pushed an experiment to the metricd branch. It should send the cpu and memory metrics to zabbix_sender

[gofree]

I sent you an email. Have a look

[gofree]

a general question to everyboidy who knows

lest say we're getting the data from summary endpoint

kubectl --kubeconfig=/etc/zabbix/kube.conf get --raw /api/v1/nodes/minikube/proxy/stats/summary

out for of the pods ( output has been stripped down for one pod with one container):

{
    "podRef": {
     "name": "kube-state-metrics-d5b45d957-cdvmz",
     "namespace": "kube-system",
     "uid": "110695d9-b4ff-4df3-a5cf-b6aa5ccd74c6"
    },
    "startTime": "2019-08-27T07:33:01Z",
    "containers": [
     {
      "name": "kube-state-metrics",
      "startTime": "2019-09-09T06:19:12Z",
      "cpu": {
       "time": "2019-09-09T09:57:25Z",
       "usageNanoCores": 656186,
       "usageCoreNanoSeconds": 6948961111
      },
      "memory": {
       "time": "2019-09-09T09:57:25Z",
       "usageBytes": 32464896,
       "workingSetBytes": 7593984,
       "rssBytes": 6307840,
       "pageFaults": 4950,
       "majorPageFaults": 132
      },
      "rootfs": {
       "time": "2019-09-09T09:57:25Z",
       "availableBytes": 223552917504,
       "capacityBytes": 248352534528,
       "usedBytes": 28672,
       "inodesFree": 15166070,
       "inodes": 15466496,
       "inodesUsed": 6
      },
      "logs": {
       "time": "2019-09-09T09:57:25Z",
       "availableBytes": 223552917504,
       "capacityBytes": 248352534528,
       "usedBytes": 28672,
       "inodesFree": 15166070,
       "inodes": 15466496,
       "inodesUsed": 300426
      },
      "userDefinedMetrics": null
     }
    ],
    "cpu": {
     "time": "2019-09-09T09:57:27Z",
     "usageNanoCores": 726337,
     "usageCoreNanoSeconds": 6964871563
    },
    "memory": {
     "time": "2019-09-09T09:57:27Z",
     "usageBytes": 32980992,
     "workingSetBytes": 8110080,
     "rssBytes": 6299648,
     "pageFaults": 0,
     "majorPageFaults": 0
    },
    "network": {
     "time": "2019-09-09T09:57:31Z",
     "name": "eth0",
     "rxBytes": 9628081,
     "rxErrors": 0,
     "txBytes": 2683006,
     "txErrors": 0,
     "interfaces": [
      {
       "name": "eth0",
       "rxBytes": 9628081,
       "rxErrors": 0,
       "txBytes": 2683006,
       "txErrors": 0
      }
     ]
    },
    "volume": [
     {
      "time": "2019-09-09T06:20:02Z",
      "availableBytes": 5113786368,
      "capacityBytes": 5113798656,
      "usedBytes": 12288,
      "inodesFree": 1248477,
      "inodes": 1248486,
      "inodesUsed": 9,
      "name": "kube-state-metrics-token-zk75r"
     }
    ],
    "ephemeral-storage": {
     "time": "2019-09-09T09:57:28Z",
     "availableBytes": 223552917504,
     "capacityBytes": 248352534528,
     "usedBytes": 57344,
     "inodesFree": 15166070,
     "inodes": 15466496,
     "inodesUsed": 6
    }
   },

I was able to create a preprocessing steps with jsonpath that extract metrics for conatiners - cpu(usageNanoCores), memory(usageBytes), disk usage(rootfs.usedBytes), disk usage for logs(logs.usedBytes)

The strange thing is that actually Im not sure if the number represent what I think they do - especially for disk usage - the number are pretty low for etcd, coredns containers, better for kube-addon-manager and pretty reasonable for kibana

Am I reading it right that those number present the the usage of container ( the writable part ) as the image of the container is read only and can be mutual for for several container and so this is actually presenting the number of bytes that are used by the living containers except the image size?

[gofree]

the LLD script used for discovering containers in pods in namespaces with node where the pod is runnign is provided via script by ik_zelf - thanks for that

https://github.com/ikzelf/zbx_kubernetes/blob/master/zbx_k8s_lld.py

[doctorbal82]

This is excellent information and details gofree .

I'm currently looking into Kubernetes Monitoring (environment deployed via kubespray) using Zabbix. I am running Zabbix version 4.0.

Do you have the template that you are using in conjunction with ik_zelf scripts (https://github.com/ikzelf/zbx_kubernetes)?

Also I'm a bit confused how the zbx_k8s_metrics.py script works in the mechanism here for the template. Some initial guidance would be much appreciated.

[ik_zelf]

It is fairly easy to create a template with a discovery rule that has item prototypes for the fields that you want to track. Create a zabbix_host and couple that template. Use zbx_k8s_lld.py to send the discovery array. Next use zbx_k8s_metrics.py to send the metrics to the given zabbix_host, using the given zabbix_server or zabbix_proxy. If you just run zbx_k8s_metrics.py XXXX you can see what metrics are passed through. The output has to match the item prototypes in the template. You can put zbx_k8s_metrics.py in a crontab. Maybe it also works through as a user_parameter for the agent. I did not test that.

[gofree]

Hi,

The idea behind is worked on slightly adjusted discovery scripts, new script for container discovery, script for getting metrics from metric server and adjusted template - original ( reason is to discover multiple containers per pod )

im on zabbix version 4.2 so I did use json path preprocessing ( should work also in v4 ) in getting the metrics per container ( dependent items ) - I think this where ik_zelf is utilizing zabbix_sender in script to get the metrics zbx_k8s_metrics.py - thats how the things are done in original project which im still using as is for everything except containers - Ive chose the json approach because Im no good at scripting & trying to utilize zabbbix functions


My changes against original ( not documented in the forked repo so far ):


1. for container discovery external script is used zbx_k8s_lld.py ( discover conatainer and puts them in to context of node,namespace and pod ) thanks to ik_zelf

2. there is a master item getting the metrics from api /api/v1/nodes/minikube/proxy/stats ( possibly need to be deployed on kubernetes to fill all the data ) - via external script k8s_con_metrics_summary.sh
in the first tests I was using metric server and api metrics.k8s.io/v1beta1 ( dont think it needs to be deployed as the /api/v1/nodes/minikube/proxy/stats is used - if it doesnt provide data deploy metric server )

3. metrics per container are being collected via json path preprocessing - details in template kube_play_template.xml

Things to consider:

- maybe its all bogwash what was done but I learned a bit

- this is just a playground at the moment - not sure if this can be extended to real production but might be a path somehow for at least basic monitoring ( there is a kubernetes template in roadmap in zabbix 4.4 - nothing was spotted in the alpha versions so not sure if guys from zabbix will cover it in 4.4 - fingers crossed - see here and here. The kubernetes speach was withdrawn from zabbix summit agenda and topic in the time spot was set "to be announced" - hope for pleasant surprise )

- config files is being used for kubectl for getting the data - in case you'll have more k8s cluster this needs to be adjusted ( maybe a template per k8s cluster utilizing different configs or one config with several k8s cluster contexts )

- I tested it on minikube ( single node ) - in multiple node k8s there will have to be some work done on k8s_con_metrics_summary.sh as this is getting data from api per node kubectl --kubeconfig=/etc/zabbix/kube.conf get --raw /api/v1/nodes/minikube/proxy/stats/summary - so some autodiscovery will be needed to get the metrics from all nodes in cluster and put the info to a single file ( master item - point 2 )

- In future http agent might be used for obtaining metrics via api ( needs some configuration on k8s and some playing with http agent on zabbix - authentification token , etc. )

- Im getting some data for containers

a) cpu is milicores ( doesnt say the utilization of cpu for the container - for example the the container can have some hard treshold how many milicores can be used - if such thing will be present then maybe some metric calculation will be done for utilization=current_usage/treshold )

b) memory thats pretty OK

c) fs usage - I'm not 100% sure it providing the data in full scope kibana looks promising - I think its like this that the container is using disk on top of the read only conainer image ( so its disk usage doesnt count to the running container disk usage ) in such case also the other numbers will be looking pretty OK as theyre default container with not so many changes on top their read only images - but there might other files that might count to the fs usage per container - Im kinda trying to sell the k8s api information - see rootfs and logs section in json output in my previous post

If you have any questions I've pinged you my email in messages

[doctorbal82]

Thank you both for the prompt reply!

Thank you ik_zelf for the details on your scripts. I missed a major detail on installing the kubernetes metrics-server that was hinted throughout the thread. Hence why I am receiving the "Error from server (NotFound): the server could not find the requested resource" because the process "kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods" can't run...

Also thank you gofree for the detailed breakdown and pointers. I will explore this this coming week and see what else I can find out regarding monitoring K8S using Zabbix and post back here.

[gofree]

How are things going ?

[doctorbal82]

gofree I was able to leverage the scripts from ik_zelf by deploying the kubernetes metrics-server using kubespray and viewing the necessary stats. I am in the process of creating the items and triggers necessary for our needs (taking concepts from your template). I prefer to use the Zabbix Trapper mechanism or Zabbix Agent (active) item type by either exclusively using the zbx_k8s_metrics.py script or using a userparameter_kubernetes.conf file (e.g.). I also presently decided to take a step back and wait until I ask the Zabbix team and others at the Zabbix Summit (which I will be gratefully attending!) to see what is the preferred means of monitoring kubernetes (Prometheus exporter, HTTP Agent Type, scripts, etc.). I also tested your directions against my personal minikube instance and it worked well! So yes the information you provided is extremely useful; thank you. Container discovery works really well and leveraging that information for CPU and Mem monitoring is straight forward. It is still in beta testing phase now seeing if the information is pertinent based on internal usage. I'll update once I hear back from the Zabbix Summit

[gofree]

Fingers crossed I'll skip this year's summit, so keep your ears open.