Kubernetes PVC Available Space left monitoring

[andreasbe]

tl;dr
Is it possible to monitor used space on k8s pvc's with zabbix already?

Hello,


I managed to setup zabbix k8s monitoring using the helm chart.

I was wondering if anyone already setup persistent volume claims monitoring as done visually by this grafana dashboard

https://grafana.com/grafana/dashboards/13646-kubernetes-persistent-volumes/

It uses among others this prometheus metric to get the available bytes

kubelet_volume_stats_available_bytes

Zabbix Template Kubernetes cluster state by HTTP Kubernetes: Get state metrics: PVC discovery or any other template does not seem to make use of pvc metrics which include available bytes

Do you know of any template which offers pvc used space discovery?
Otherwise I'll try my best to adopt it myself (Not much experience in Zabbix templates, though)

Best regards from Germany
Andreas

[jbee43]

Did you manage to figure this out?

I'm struggling to adapt the Get state metrics: PV discovery rule in Kubernetes cluster state by HTTP template as well
I see the keys, the scripts, but can't find or understand the mapping apparently

Also, for some reason, while everything else works great, the Controller manager and Scheduler are retrieving nothing, but the endpoints seem to be fine

[mahesh396]

jbee43 In my case i have updated the {$KUBE.STATE.ENDPOINT.NAME} macro to get it worked.
But it just discovers PV and PVC status and not getting available space left

[andreasbe]

Did you manage to figure this out?

Yes by switching to prometheus / grafana / alert manager for this task
We love zabbix, but k8s support was not satisfying / we could not spend enough time to get it running properly.

(Sorry I did not receive a notification about new answers, although I was subscribed all the time)

[NgRox]

It is necessary to create a custom monitoring setup by exposing port 10250, which is the Kubelet port (it comes installed on every cluster node).

You need to create a token and run a query to retrieve the used space of PVCs:
You can get this token by running:
TOKEN=$(kubectl get secret <token-name> -n <namespace> -o jsonpath="{.data.token}" | base64 -d)

Exec:
curl -k -H "Authorization: Bearer $TOKEN" https://<ip>:10250/metrics

This will return values like:


kubelet_volume_stats_used_bytes{namespace="default ",persistentvolumeclaim="myclaim3"} 1.91054843904e+11
kubelet_volume_stats_used_bytes{namespace="default ",persistentvolumeclaim="myclaim4"} 1.91027953664e+11
kubelet_volume_stats_used_bytes{namespace="default ",persistentvolumeclaim="myclaim5"} 1.91019491328e+11

From this data, you can configure a calculated item to subtract used from total space to determine the free space.