Zabbix + kubernetes

[warp10]

Hi all,

I'm looking for about the best practice for monitoring Kubernetes with zabbix , so I have two point : monitoring the "Kube" I think it's ok, but for the applications on the "Kube" , I think Prometheus is the best way for this ? what do you think about ? some prod expériences ...
Thanks a lot

[gofree]

might be worth a shot but needs improvement 9 discovery works pretty OK , collect script some issues - see issues on github repo )

https://share.zabbix.com/virtualization/kubernetes-monitoring-lld

let me knwo how you do with it

[warp10]

Ok goffee , thanks for the links

[gofree]

any success ?

[gofree]

Just know some progress

I had some issues to get it running on centos, better succes on archlinux ( my favourite distro ) - I had to move the config files of k8s(minikube) and cert files of k8s(minikube) to location accessible to zabbix and it started working in full ( onlypartial success with redhat ). At lest it gives out something - maybe this could a way how to improve kube/openshift monitoring. So far there wasnt anything in athe alfa version of 4.4 event if it was mentioned in the roapdmap

See screens from latest data - its from minikube cluster - so not much like a productional environemnt - first test - when I deleted the pod , ne was scheduled , autodicovery rule picked up the new one nearly immediatelly ( 1- 2 min ) and removed the deleted one in simmilar time.

If anybody is curious how to setup ping it here, even more if somebody is perl guru maybe this can be improved somehow to get the cpu, mem, disk utilization of the pods.

[warp10]

Hi Gofree,

Great Job !

Thanks for the screen , how to manage with zabbix the generate UUID by kube the network , container >>> who died , because there is lot of

[gofree]

pros:

ive managed to start collecting cpu and memory metrics via metrics-server - seems possible - cpu is in milicores


cons:


got another issue - if there is a multicontainer pod only first container is being discovered - anybody skilled in perl who can look in to that ?


to do ( if possible ) :

obtain disk usage per container from API ( warp10 I can see some section with uuid there as well - is ithe one youre looking for)

kubectl get --raw /api/v1/nodes/minikube/proxy/stats/summary

[gofree]

So this is wahts going on:

tested on minikube and minishift ( point 1 - 7 seems working pretty OK )

1. containers
- pods are being discovered also with all conatiners ( fixed )
- readiness status and restart count is working

2. api services are being discovered

3. deployments

- desired replicas
- not up to date replicas
- status

4. node

- disk pressure
- memory pressure
- out of disk
- pid pressure

5. pods

- phase
- status failed, reason evicted
- status ready

6. components statuses ( etcd, scheduler,....)

7. cluster health

8. getting the cpu ,memory, storage usage per container - needs some work - anybody willing to help with json, autodiscovery....

[gofree]

For the cpu and memory metrics see metrics.json file ( memory is in Ki and cpu in milicores )

So I do have this data and waths needed is to extract cpu and memory value preferably via autodiscovery

already discovered data which can be used and are part of json

{#NAME} will refer to name mc1
{#CONATINER} will refere to name of containers 1st, 2nd
{#NAMESPACE} optionally can be used for namespace kube-system

it is possible to get the data only for the specific pod following the link in selfLink - but I guess it will be better to get it all in one api call for all pods in the specified namespace


Any ide how to make an autodiscovery rule out of this - guess it will need some json preprocessing ....so far what I tried I failed - I created an master item and 2 dependant items for one pod - the issue is that for the first run LLD the first container sends data and for the second run of LLD the second container items replace the first one and they stay there ( details of the LLD ) in the screens

1. step master item ( basically get the data on pod with container as quoted bellow )



2. step dependent item for with some json ppath preprocessing that should retrieve the number for cpu or memory usage



3. step 1st run of LLD - container 1st metrics are present



4. step second run of LLD - container 1st metrics are replaced by 2nd condainer metrics and it stays like this




I thin the LLD rule for mem and cpu can be improved or reworked to make it better - any ideas. I need a bit distance from it right now cause I can think straight on this anymore

So help is appreciated - preferably to make it work with the whole set of data in

metrics-json.txt

to exclude the need of the pulls for each pod.

{
"metadata": {
"name": "mc1",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/mc1",
"creationTimestamp": "2019-09-04T06:42:04Z"
},
"timestamp": "2019-09-04T06:41:00Z",
"window": "1m0s",
"containers": [
{
"name": "1st",
"usage": {
"cpu": "0",
"memory": "6232Ki"
}
},
{
"name": "2nd",
"usage": {
"cpu": "1m",
"memory": "4072Ki"
}
}
]
},

[ik_zelf]

if you can help me with kubectl, I can easily add a few metrics to https://github.com/ikzelf/zbx_kubernetes Currently it only does lld but it should not be that hard to add metrics. Doing so will enable k8s monitoring also for pre 4.2 zabbix versions.

[gofree]

Hi. so here we go again we met on reddit solving the partial issue of 2( n- containers ) in the pod....being said. I'll try to explain.


I was able somehow fix the issue of N pods in a container in the perl script - so that part is done ( point 1 - 7 ). I guess this part is provided via out of box k8s api. For container metrics you need something that is able to provide them - so on top of k8s you deploy metric server

For the second part if you call something like this you'll get a file with the informations about pods and containers ( cpu,mem ). So now we're talking about kinda different source of the data but it would be great if will be possible to match to the macros mentioned earlier ( so those 2 datasources might be living together in zabbix ).

macros

{#NAME} will refer to name mc1
{#CONTAINER} will refere to name of containers 1st, 2nd
{#NAMESPACE} optionally can be used for namespace kube-system

Essentially what is needed is script that will follow the macros and parse from new source ( kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods ), create zabbix item prototypes and the script should also send data for those items ( I guess via zabbix_sender ).

The trick is utilize the macros names from already working perl solution ( k8s standard api ) with new solution based on metric server for cpu,mem container metrics . Not sure if clear enough As the perl part is working good now it can wait ( and its doing more that just pod discovery ). If you think thats managable ping me an private message and we can continue more effciently As of the moment I think I see what needs to be done just I do lack the coding abilities. After the cpu and memory usage we can try look in to the conainer storage usage - another api call - but in to this one I need to look a little bit deeper.

kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods | jq

Example for one of the pods ( full output in full.txt ):

{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/pods"
  },
  "items": [
      .
      .
      POD
      .
      .
      POD
      .
      .

     {
      "metadata": {
        "name": "mc1",
        "namespace": "kube-system",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/mc1",
        "creationTimestamp": "2019-09-04T11:28:35Z"
      },
      "timestamp": "2019-09-04T11:28:00Z",
      "window": "1m0s",
      "containers": [
        {
          "name": "1st",
          "usage": {
            "cpu": "0",
            "memory": "3252Ki"
          }
        },
        {
          "name": "2nd",
          "usage": {
            "cpu": "1m",
            "memory": "4084Ki"
          }
        }
      ]
    }
  ]
}

[ik_zelf]

I found this <https://talkcloudlytome.com/raw-kubernetes-metrics-with-kubectl/>
Could it be done without metric server ?

[ik_zelf]

I added something to play with in https://github.com/ikzelf/zbx_kubernetes branch metricd. It's not working yet, first see if the statistics returned are familiar to you. Next steps are formatting the keys, something to give the zabbix host and add zabbix_sender (to a zabbix server)

[gofree]

Hi, I'll have a look tomorrow ane provide feedback - the link you posted is aslo working with metric ( as is my understanding you can get the cpu, mem only via some kind of metric server )

In general as I'm deep diving a bit all the monitoring solutions are working with kube-state-metrics, metric-server, kubernetes api, and similar. The trick is to get it , discover it properly due tot the changing nature of k8s and present it.

[gofree]

thanks, it seems to be working - metric make sense - though I have a a suggestion for you...

for now I have manage to get the metric data for cpu and mem per container in to zabbix - basically for every pod there is a kubectl call pod with some json prepropcessing ( as the namespace, pod, container ale already autodicovered - {#NAMESAPCE},{#NAME},{#CONTAINER} )

this is done as LLD - not so effective when it comes to number of kubectl call, but works ( for every discovered pod there is a kubectl call ) - see the graph



my suggestion to to this would be to have an item that just get the data once and store the data temporarily

kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods > /tmp/zabbix/something.json

next step will be working just based on this data with some script that will extract cpu and memory usage per container ( as the namespace, pod and container name are known as macros - {#NAMESPACE}, {#NAME}, {#CONATINER} )

example item prototype key : script[{#NAMESPACE},{#NAME},{#CONTAINER},mem] and script[{#NAMESPACE},{#NAME},{#CONTAINER},cpu]

this way only one call is being made to get all the data and the preprocessing will be on script locally - no so costly for zabbix server/proxy I guess and the container will be preciselly identified by already exisiting macros

what do you think ?


as of now I have this kind of setup

1. first metric fro the pods are being called ( 3rd value is just for the item key differentiation )

cat k8s_container_metrics.sh
#!/bin/bash
kubectl --kubeconfig=/etc/zabbix/kube.conf get --raw /apis/metrics.k8s.io/v1beta1/namespaces/$1/pods/$2
echo $3 >> /dev/null

item prototype





result of the call

{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "mc1",
    "namespace": "kube-system",
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/mc1",
    "creationTimestamp": "2019-09-05T06:37:05Z"
  },
  "timestamp": "2019-09-05T06:37:00Z",
  "window": "1m0s",
  "containers": [
    {
      "name": "1st",
      "usage": {
        "cpu": "0",
        "memory": "3516Ki"
      }
    },
    {
      "name": "2nd",
      "usage": {
        "cpu": "2m",
        "memory": "4Mi"
      }
    }
  ]
}

2. there is a jsonpath preprocessing for the obtained data ( if I knew how to write json path preprocessing for the whole data set based on those 3 macros it would be evene easier and no script will be needed )

basically this json path preprocessing goes through the data retrieved in step 1 and for every container ( rememebr the container name/macro {#CONTAINER} is already discovered and put in the context of namespace and pod )

similllary there is another item prottotype for container memory

if there would be a way how to cretae a json path preprocessing rule that will get from the full list that will more efficient way - mostly the work will be done by zabbix for this - not sure if thats possible though - the full output is full.txt in one of my previous posts