Hi,
I'm trying to get some monitoring out of the Kube-State exporter using my Zabbix.
my architecture is a Zabbix server deployed in e K8s cluster and a Zabbix proxy deployed in another K8s cluster. I want to monitor the cluster where my proxy is at.
I have created a template with master item that takes the whole kube-state exporter data and 3 discovery rules with dependent items to discover the items from the exported data.
I have given the proxy 8 CPUs and 16GB of memory.
The amount of data coming from the kube-state exporter is huge. once deployed to the proxy it creates around 3000 items, most gets disabled after some time and and it can reach up to 20k items after 1 day.
My problem is that the master item that should run every 2 minutes gets delayed between 5 and 20 minutes. Items are not being updated for hours, I assume due to the pre-processing progress. I have added LLD workers to the Zabbix (10) and pre-processors to the proxy (30) but the CPU load is getting too high - between 10 to 20 and it's still not enough to make it work in best performance (and it's not the DB for sure...)
So here are my questions:
1. Did anyone have experience in monitoring the kube-state exporter? anyone could recommend specific monitors that are used (maybe I'm trying to take the wrong metrics)?
2. Did anyone worked out the performance of monitoring a K8s cluster (not the nodes themselves) at the application/pod/container level? Examples can be pods keep crashing repeatedly over time, Pods Out of memory issues, Pod not able to get to running state, Pods on pending state for a long period of time?
Any advice and help will be great.
I'm trying to get some monitoring out of the Kube-State exporter using my Zabbix.
my architecture is a Zabbix server deployed in e K8s cluster and a Zabbix proxy deployed in another K8s cluster. I want to monitor the cluster where my proxy is at.
I have created a template with master item that takes the whole kube-state exporter data and 3 discovery rules with dependent items to discover the items from the exported data.
I have given the proxy 8 CPUs and 16GB of memory.
The amount of data coming from the kube-state exporter is huge. once deployed to the proxy it creates around 3000 items, most gets disabled after some time and and it can reach up to 20k items after 1 day.
My problem is that the master item that should run every 2 minutes gets delayed between 5 and 20 minutes. Items are not being updated for hours, I assume due to the pre-processing progress. I have added LLD workers to the Zabbix (10) and pre-processors to the proxy (30) but the CPU load is getting too high - between 10 to 20 and it's still not enough to make it work in best performance (and it's not the DB for sure...)
So here are my questions:
1. Did anyone have experience in monitoring the kube-state exporter? anyone could recommend specific monitors that are used (maybe I'm trying to take the wrong metrics)?
2. Did anyone worked out the performance of monitoring a K8s cluster (not the nodes themselves) at the application/pod/container level? Examples can be pods keep crashing repeatedly over time, Pods Out of memory issues, Pod not able to get to running state, Pods on pending state for a long period of time?
Any advice and help will be great.