Available solutions
Kubernetes Cluster by HTTP
Overview
The template to monitor Kubernetes Cluster. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes Cluster by HTTP - collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget to change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes v1.32.6
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the service account name (if a specific release name is used):
kubectl get serviceaccounts -n monitoring
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Also, see the Macros section for a list of macros used to set trigger values.
Set up the fields to filter the metrics of discovered node by names:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter in values of the chart Zabbix Helm Chart to get only the necessary metrics in raw data before they go into Zabbix.
Use the metricLabelsAllowlist and metricAnnotationsAllowList fields in KSM for advanced filtering by labels and annotations.
You can also set up evaluation periods for replica mismatch triggers (Deployments, ReplicaSets, StatefulSets) with the macro {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD}, which supports context and regular expressions. For example, you can create the following macros:
- Set the evaluation period for the Deployment "nginx-deployment" in the namespace "default" to the 3 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:default:nginx-deployment"} = #3
- Set the evaluation period for all Deployments to the 10 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"deployment:.*:.*"} = #10 or {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"^deployment.*"} = #10
- Set the evaluation period for Deployments, ReplicaSets and StatefulSets in the namespace "default" to 15 minutes:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:".*:default:.*"} = 15m
Note: that different context macros with regular expressions matching the same string can be applied in an undefined order, and simple context macros (without regular expressions) have higher priority. Read the Important notes section in Zabbix documentation for details.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc |
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$HTTP.TLS.VERIFY} | TLS certificate verification for script items and JavaScript preprocessing: "none" - disabled, "peer" - verify the certificate chain and expiration, "full" - full verification. Any other value enables full verification. To override the setting for this template only, define the macro with the context "Kubernetes Cluster", e.g. {$HTTP.TLS.VERIFY:"Kubernetes Cluster"}. |
full |
| {$KUBE.NODE.EXPORTER.PORT} | Kubernetes prometheus-node-exporter metrics endpoint port. |
9100 |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable metrics by namespace. |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered metrics by namespace. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.PV.MATCHES} | Filter of discoverable persistent volumes by name. |
.* |
| {$KUBE.LLD.FILTER.PV.NOT_MATCHES} | Filter to exclude discovered persistent volumes by name. |
CHANGE_IF_NEEDED |
| {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD} | The evaluation period range which is used for calculation of expressions in trigger prototypes (time period or value range). Can be used with context. |
#5 |
| {$KUBE.CLUSTER.CPU.UTIL.WARN} | The threshold of CPU usage in percent. |
70 |
| {$KUBE.CLUSTER.MEMORY.UTIL.WARN} | The threshold of memory usage in percent. |
70 |
| {$KUBE.CLUSTER.CPU.UTIL.CRIT} | The threshold of CPU usage in percent. |
80 |
| {$KUBE.CLUSTER.MEMORY.UTIL.CRIT} | The threshold of memory usage in percent. |
80 |
| {$KUBE.NODE.NUMBER.THRESHOLD} | The number of nodes in the cluster used in triggers. |
2 |
| {$KUBE.CLUSTER.POD.PENDING.THRESHOLD} | The number of pending pods used in triggers. |
5 |
| {$KUBE.CLUSTER.POD.FAILED.THRESHOLD} | The number of failed pods used in triggers. |
5 |
| {$KUBE.NAMESPACE.POD.PENDING.THRESHOLD} | The number of pending pods used in triggers. |
3 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
Script | kube.state.metrics |
| Get resource metrics | Collecting metrics from kubelet in |
Dependent item | kube.resource.metrics Preprocessing
|
| Get readyz | HTTP agent | kube.get.readyz Preprocessing
|
|
| Get livez | HTTP agent | kube.get.livez Preprocessing
|
|
| Get state metrics check | Check that the kube-state-metrics data has been received correctly. |
Dependent item | kube.state.metrics.check Preprocessing
|
| Namespace count | The number of namespaces. |
Dependent item | kube.namespace.count Preprocessing
|
| CronJob count | The number of cronjobs. |
Dependent item | kube.cronjob.count Preprocessing
|
| Job count | Number of jobs (generated by cronjob + job). |
Dependent item | kube.job.count Preprocessing
|
| Endpoint count | The number of endpoints. |
Dependent item | kube.endpoint.count Preprocessing
|
| Deployment count | The number of deployments. |
Dependent item | kube.deployment.count Preprocessing
|
| Service count | The number of services. |
Dependent item | kube.service.count Preprocessing
|
| StatefulSet count | The number of statefulsets. |
Dependent item | kube.statefulset.count Preprocessing
|
| Node count | The number of nodes. |
Dependent item | kube.node.count Preprocessing
|
| Node Ready count | The number of Ready nodes. |
Dependent item | kube.node.ready.count Preprocessing
|
| Node Not Ready count | The number of Not Ready nodes. |
Dependent item | kube.node.not_ready.count Preprocessing
|
| Configmap count | The number of configmaps. |
Dependent item | kube.configmap.count Preprocessing
|
| PVC count | The number of PVC. |
Dependent item | kube.persistentvolumeclaim.count Preprocessing
|
| PV count | The number of PV. |
Dependent item | kube.persistentvolume.count Preprocessing
|
| Secret count | The number of secrets. |
Dependent item | kube.secret.count Preprocessing
|
| Memory allocatable | The memory allocatable in the cluster calculated as a sum of allocatable memory on all nodes. |
Dependent item | kube.cluster.memory_allocatable Preprocessing
|
| Memory limits | The memory limits in the cluster calculated as a sum of memory limits on all pods. |
Dependent item | kube.cluster.memory_limits Preprocessing
|
| Memory requests | The memory requests in the cluster calculated as a sum of memory requests on all pods. |
Dependent item | kube.cluster.memory_requests Preprocessing
|
| Memory usage | The memory usage in the cluster calculated as a sum of working set memory on all nodes. |
Dependent item | kube.cluster.memory_usage Preprocessing
|
| Memory utilization | Cluster memory utilization calculated as a percentage of used memory from total memory. |
Calculated | kube.cluster.memory_utilization |
| CPU allocatable | The CPU allocatable in the cluster calculated as a sum of allocatable CPU on all nodes. |
Dependent item | kube.cluster.cpu_allocatable Preprocessing
|
| CPU limits | The CPU limits in the cluster calculated as a sum of CPU limits on all pods. |
Dependent item | kube.cluster.cpu_limits Preprocessing
|
| CPU requests | The CPU requests in the cluster calculated as a sum of CPU requests on all pods. |
Dependent item | kube.cluster.cpu_requests Preprocessing
|
| CPU usage | The CPU usage in the cluster calculated as a sum of CPU seconds on all nodes. |
Dependent item | kube.cluster.cpu_usage Preprocessing
|
| CPU utilization | The CPU utilization in the cluster calculated as a percentage of used CPU from allocatable CPU. |
Calculated | kube.cluster.cpu_utilization |
| Running pods | The number of running pods. |
Dependent item | kube.pod.running Preprocessing
|
| Pod pending | The number of pods in pending state. |
Dependent item | kube.pod.pending Preprocessing
|
| Pod failed | The number of pods in failed state. |
Dependent item | kube.pod.failed Preprocessing
|
| Containers running | The number of running containers in the cluster. |
Dependent item | kube.pod.containers_running Preprocessing
|
| ReplicaSets count | The number of ReplicaSets in the cluster. |
Dependent item | kube.pod.replicaset_count Preprocessing
|
| Daemonset count | The number of DaemonSets in the cluster. |
Dependent item | kube.pod.daemonset_count Preprocessing
|
| Ingress count | The number of Ingresses in the cluster. |
Dependent item | kube.pod.ingress_count Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Cluster: Readyz is unhealthy | count(/Kubernetes Cluster by HTTP/kube.get.readyz,#2,"ne","1")=2 and length(last(/Kubernetes Cluster by HTTP/kube.get.readyz))>0 |
Average | ||
| Kubernetes Cluster: Livez is unhealthy | count(/Kubernetes Cluster by HTTP/kube.get.livez,#2,"ne","1")=2 and length(last(/Kubernetes Cluster by HTTP/kube.get.livez))>0 |
Average | ||
| Kubernetes Cluster: Failed to get kube-state-metrics data | Failed to get kube-state-metrics. |
length(last(/Kubernetes Cluster by HTTP/kube.state.metrics.check))>0 |
Warning | |
| Kubernetes Cluster: No Ready nodes | There are no Ready nodes in the cluster. |
last(/Kubernetes Cluster by HTTP/kube.node.ready.count)=0 and last(/Kubernetes Cluster by HTTP/kube.node.count)>0 |
High | |
| Kubernetes Cluster: Low number of Ready nodes | The number of Ready nodes in the cluster is less than |
last(/Kubernetes Cluster by HTTP/kube.node.ready.count)< {$KUBE.NODE.NUMBER.THRESHOLD} and last(/Kubernetes Cluster by HTTP/kube.node.count)>0 |
High | Depends on:
|
| Kubernetes Cluster: High memory utilization | The cluster memory utilization has exceeded |
min(/Kubernetes Cluster by HTTP/kube.cluster.memory_utilization,5m) > {$KUBE.CLUSTER.MEMORY.UTIL.CRIT} |
Average | |
| Kubernetes Cluster: High memory utilization | The cluster memory utilization has exceeded |
min(/Kubernetes Cluster by HTTP/kube.cluster.memory_utilization,5m) > {$KUBE.CLUSTER.MEMORY.UTIL.WARN} |
Warning | Depends on:
|
| Kubernetes Cluster: High cpu utilization | Cluster CPU utilization has exceeded |
min(/Kubernetes Cluster by HTTP/kube.cluster.cpu_utilization,5m) > {$KUBE.CLUSTER.CPU.UTIL.CRIT} |
Average | |
| Kubernetes Cluster: High cpu utilization | Cluster CPU utilization is too high. The cluster might be slow to respond. |
min(/Kubernetes Cluster by HTTP/kube.cluster.cpu_utilization,5m) > {$KUBE.CLUSTER.CPU.UTIL.WARN} |
Warning | Depends on:
|
| Kubernetes Cluster: Pod pending high | The number of pending pods is high |
last(/Kubernetes Cluster by HTTP/kube.pod.pending) > {$KUBE.CLUSTER.POD.PENDING.THRESHOLD} |
Warning | |
| Kubernetes Cluster: Pod failed high | The number of failed pods is high |
last(/Kubernetes Cluster by HTTP/kube.pod.failed) > {$KUBE.CLUSTER.POD.FAILED.THRESHOLD} |
Warning |
LLD rule Namespace discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace discovery | Dependent item | kube.namespace.discovery Preprocessing
|
Item prototypes for Namespace discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}]: CPU: Total usage | The total CPU usage in the namespace calculated as a sum of non-idle CPU seconds on all containers in the namespace. |
Dependent item | kube.namespace.cpu_total[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: CPU: Limits | The CPU limits in the namespace calculated as a sum of CPU limits on all pods in the namespace. |
Dependent item | kube.namespace.limits.cpu[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: CPU: Requests | The number of requested CPU cores by a container. |
Dependent item | kube.pod.containers.requests.cpu[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: CPU: Usage, % | The CPU usage in the namespace calculated as a percentage of used CPU from allocatable CPU in the cluster. |
Calculated | kube.namespace.cpu_usage[{#NAMESPACE}] |
| Namespace [{#NAMESPACE}]: Memory: Total usage | The total memory usage in bytes in the namespace calculated as a sum of used memory on all containers in the namespace. |
Dependent item | kube.namespace.memory_usage[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: Memory: Usage, % | The memory usage in the namespace calculated as a percentage of used memory from allocatable memory in the cluster. |
Calculated | kube.namespace.memory_usage_percentage[{#NAMESPACE}] |
| Namespace [{#NAMESPACE}]: Memory: Limits | The memory limits in the namespace calculated as a sum of memory limits on all pods in the namespace. |
Dependent item | kube.namespace.limits.memory[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: Memory: Requests | The memory requests in the namespace calculated as a sum of memory requests on all pods in the namespace. |
Dependent item | kube.namespace.requests.memory[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: Pods: Running | The number of pods in running state in the namespace. |
Dependent item | kube.namespace.pod.running[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: Pods: Pending | Pod is in pending state. |
Dependent item | kube.namespace.pod.pending[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: Pods: Failed | Pod is in failed state. |
Dependent item | kube.namespace.pod.failed[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}]: Status phase | The current status phase of the namespace. |
Dependent item | kube.namespace.status_phase[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for Namespace discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Cluster: Namespace [{#NAMESPACE}]: Pods pending high | last(/Kubernetes Cluster by HTTP/kube.namespace.pod.pending[{#NAMESPACE}]) > {$KUBE.NAMESPACE.POD.PENDING.THRESHOLD:"{#NAMESPACE}"} |
High | ||
| Kubernetes Cluster: Namespace [{#NAMESPACE}]: Terminated | count(/Kubernetes Cluster by HTTP/kube.namespace.status_phase[{#NAMESPACE}],2m,,2)>=2 |
Warning |
LLD rule Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment discovery | Dependent item | kube.deployment.discovery[{#NAMESPACE}] Preprocessing
|
Item prototypes for Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment [{#DEPLOYMENT}]: CPU: Total usage | The total CPU usage in the deployment calculated as a sum of non-idle CPU seconds on all containers in the deployment. |
Dependent item | kube.deployment.cpu_total[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: CPU: Limits | The CPU limits in the deployment calculated as a sum of CPU limits on all pods in the deployment. |
Dependent item | kube.deployment.limits.cpu[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: CPU: Requests | The number of requested CPU cores by a container. |
Dependent item | kube.deployment.requests.cpu[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: CPU: Usage, % | The CPU usage in the deployment calculated as a percentage of used CPU from allocatable CPU in the cluster. |
Calculated | kube.deployment.cpu_usage[{#NAMESPACE}/{#DEPLOYMENT}] |
| Deployment [{#DEPLOYMENT}]: Memory: Total usage | The total memory usage in bytes in the deployment calculated as a sum of used memory on all containers in the deployment. |
Dependent item | kube.deployment.memory_usage[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: Memory: Usage, % | The memory usage in the deployment calculated as a percentage of used memory from allocatable memory in the cluster. |
Calculated | kube.deployment.memory_usage_percentage[{#NAMESPACE}/{#DEPLOYMENT}] |
| Deployment [{#DEPLOYMENT}]: Memory: Limits | The memory limits in the deployment calculated as a sum of memory limits on all pods in the deployment. |
Dependent item | kube.deployment.containers.limits.memory[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: Memory: Requests | The memory requests in the deployment calculated as a sum of memory requests on all pods in the deployment. |
Dependent item | kube.deployment.containers.requests.memory[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: Replicas desired | Number of desired pods for a deployment. |
Dependent item | kube.deployment.replicas_desired[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: Replicas available | The number of available replicas per deployment. |
Dependent item | kube.deployment.replicas_available[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
| Deployment [{#DEPLOYMENT}]: Replicas mismatched | The number of available replicas not matching the desired number of replicas. |
Dependent item | kube.deployment.replicas_mismatched[{#NAMESPACE}/{#DEPLOYMENT}] Preprocessing
|
Trigger prototypes for Deployment discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Cluster: Deployment [{#DEPLOYMENT}]: Deployment replicas mismatch | Deployment has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes Cluster by HTTP/kube.deployment.replicas_mismatched[{#NAMESPACE}/{#DEPLOYMENT}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:{#NAMESPACE}:{#DEPLOYMENT}"})>0 and last(/Kubernetes Cluster by HTTP/kube.deployment.replicas_desired[{#NAMESPACE}/{#DEPLOYMENT}])>=0 and last(/Kubernetes Cluster by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#DEPLOYMENT}])>=0 |
Warning |
LLD rule PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PVC discovery | Dependent item | kube.pvc.discovery Preprocessing
|
Item prototypes for PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase | The current status phase of the persistent volume claim. |
Dependent item | kube.pvc.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
Dependent item | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Bound, sum | The total amount of persistent volume claims in the Bound phase. |
Dependent item | kube.pvc.status_phase.bound.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Lost, sum | The total amount of persistent volume claims in the Lost phase. |
Dependent item | kube.pvc.status_phase.lost.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Pending, sum | The total amount of persistent volume claims in the Pending phase. |
Dependent item | kube.pvc.status_phase.pending.sum[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for PVC discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Cluster: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | count(/Kubernetes Cluster by HTTP/kube.pvc.status_phase[{#NAMESPACE}/{#NAME}],2m,,5)>=2 |
Warning |
LLD rule PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV discovery | Dependent item | kube.pv.discovery Preprocessing
|
Item prototypes for PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV [{#NAME}] Status phase | The current status phase of the persistent volume. |
Dependent item | kube.pv.status_phase[{#NAME}] Preprocessing
|
| PV [{#NAME}] Capacity bytes | A capacity of the persistent volume in bytes. |
Dependent item | kube.pv.capacity.bytes[{#NAME}] Preprocessing
|
| PV status phase: Pending, sum | The total amount of persistent volumes in the Pending phase. |
Dependent item | kube.pv.status_phase.pending.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Available, sum | The total amount of persistent volumes in the Available phase. |
Dependent item | kube.pv.status_phase.available.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Bound, sum | The total amount of persistent volumes in the Bound phase. |
Dependent item | kube.pv.status_phase.bound.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Released, sum | The total amount of persistent volumes in the Released phase. |
Dependent item | kube.pv.status_phase.released.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Failed, sum | The total amount of persistent volumes in the Failed phase. |
Dependent item | kube.pv.status_phase.failed.sum[{#SINGLETON}] Preprocessing
|
Trigger prototypes for PV discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Cluster: PV [{#NAME}]: PV has failed | count(/Kubernetes Cluster by HTTP/kube.pv.status_phase[{#NAME}],2m,,3)>=2 |
Warning |
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Status phase | Pod status phases. |
Dependent item | kube.pod.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
Dependent item | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Cluster: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod failed phase | Pod failed state. |
count(/Kubernetes Cluster by HTTP/kube.pod.status_phase[{#NAMESPACE}/{#NAME}],2m,,3)>=2 |
Average | |
| Kubernetes Cluster: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod pending phase | Pod pending state. |
count(/Kubernetes Cluster by HTTP/kube.pod.status_phase[{#NAMESPACE}/{#NAME}],2m,,1)>=2 |
Warning | |
| Kubernetes Cluster: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod unknown phase | Pod unknown state. |
count(/Kubernetes Cluster by HTTP/kube.pod.status_phase[{#NAMESPACE}/{#NAME}],2m,,4)>=2 |
Warning | |
| Kubernetes Cluster: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes Cluster by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes Cluster by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],15m))>1 |
Warning |
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Kubernetes Node by HTTP
Overview
The template to monitor Kubernetes Node by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Node by HTTP - collects metrics by HTTP agent from Prometheus metrics endpoint.
Metrics are collected by requests to prometheus-node-exporter.
Note: Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes v1.32.6
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from Prometheus metrics endpoint.
This template is used in low-level discovery and will be auto-assigned to host prototypes. Additional configuration is not required.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$KUBE.NODE.EXPORTER.PORT} | Kubernetes prometheus-node-exporter metrics endpoint port. |
9100 |
| {$KUBE.NODE.DISK.NAME.MATCHES} | Filter of discoverable metrics by filesystem. |
.* |
| {$KUBE.NODE.DISK.NAME.NOT_MATCHES} | Filter to exclude discovered metrics by filesystem. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.NETWORK.IF.NAME.MATCHES} | Filter of discoverable network interfaces. |
.* |
| {$KUBE.NODE.NETWORK.IF.NAME.NOT_MATCHES} | Filter to exclude discovered network interfaces. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.NETWORK.IF.OPERSTATUS.MATCHES} | Used for network interface discovery. Can be overridden on the host or linked template level. |
^.*$ |
| {$KUBE.NODE.NETWORK.IF.OPERSTATUS.NOT_MATCHES} | Used for network interface discovery. Can be overridden on the host or linked template level. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.NETWORK.IF.ADMINSTATUS.MATCHES} | Filter of discoverable network interfaces by administrative status. |
^.*$ |
| {$KUBE.NODE.NETWORK.IF.ADMINSTATUS.NOT_MATCHES} | Ignore the |
^down$ |
| {$KUBE.NODE.CPU.UTILIZATION.CRIT} | Critical threshold of CPU utilization expressed in %. |
90 |
| {$KUBE.NODE.LOAD_AVG_PER_CPU.MAX.WARN} | The CPU load per core is considered sustainable. If necessary, it can be tuned. |
1.5 |
| {$KUBE.NODE.MEMORY.UTIL.MAX} | Used as a threshold in the memory utilization trigger. |
90 |
| {$KUBE.NODE.NETWORK.ERRORS.WARN} | The threshold for network interface errors per second - for the Warning trigger. |
10 |
| {$IFCONTROL} | Macro for operational state of the interface for link down trigger. Can be used with interface name as context. |
1 |
| {$KUBE.NODE.FILESYSTEM.MATCHES} | Filter of discoverable filesystem mount points. |
.* |
| {$KUBE.NODE.FILESYSTEM.NOT_MATCHES} | Filter to exclude discovered filesystem mount points. |
^(/sys|/proc|/dev|/run/containerd) |
| {$KUBE.NODE.IO.UTIL.HIGH} | High I/O utilization threshold for disk. |
80 |
| {$KUBE.NODE.FS.PUSED.MAX.WARN} | Warning threshold of the filesystem utilization. In the range from 0 to 100 inclusive. |
80 |
| {$KUBE.NODE.FS.PUSED.MAX.CRIT} | Critical threshold of the filesystem utilization. In the range from 0 to 100 inclusive. |
90 |
| {$KUBE.NODE.PSI.CPU.WARN} | Warning threshold for CPU pressure (% of time processes are waiting for CPU). |
20 |
| {$KUBE.NODE.PSI.CPU.CRIT} | Critical threshold for CPU pressure (% of time processes are waiting for CPU). |
50 |
| {$KUBE.NODE.PSI.MEMORY.WARN} | Warning threshold for memory pressure (% of time processes are waiting for memory). |
10 |
| {$KUBE.NODE.PSI.MEMORY.CRIT} | Critical threshold for memory pressure (% of time processes are fully stalled for memory). |
5 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get node-exporter metrics | Collecting Kubernetes metrics from prometheus-node-exporter. |
HTTP agent | kube.node.exporter.metrics |
| CPU: Utilization | Percentage of CPU time spent in non-idle modes (calculated as 1 - idle time). |
Dependent item | kube.node.cpu_utilization Preprocessing
|
| CPU: Cores | Number of CPU cores. |
Dependent item | kube.node.cpu_core Preprocessing
|
| Memory: Total | Total memory available in bytes. |
Dependent item | kube.node.memory_total Preprocessing
|
| Memory: Available | Available memory in bytes. |
Dependent item | kube.node.memory_available Preprocessing
|
| Memory: Cache | Cache memory in bytes. |
Dependent item | kube.node.memory_cache Preprocessing
|
| Memory: Buffers | Buffers memory in bytes. |
Dependent item | kube.node.memory_buffers Preprocessing
|
| Memory: Swap total | Swap total memory in bytes. |
Dependent item | kube.node.memory_swap_total Preprocessing
|
| Memory: Swap free | Free swap total memory in bytes. |
Dependent item | kube.node.memory_swap_free Preprocessing
|
| Memory: Swap used | Used swap memory in bytes. |
Calculated | kube.node.memory_swap_used |
| Memory: Utilization | Node memory utilization calculated as a percentage of used memory from total memory. |
Calculated | kube.node.memory_utilization |
| CPU: Load average 1min | System load average over 1 minute. |
Dependent item | kube.node.load_avg_1min Preprocessing
|
| CPU: Load average 5min | System load average over 5 minutes. |
Dependent item | kube.node.load_avg_5min Preprocessing
|
| CPU: Load average 15min | System load average over 15 minutes. |
Dependent item | kube.node.load_avg_15min Preprocessing
|
| Network: Receive bytes | Network bytes received per second. |
Dependent item | kube.node.network_receive_bytes Preprocessing
|
| Network: Transmit bytes | Network bytes transmitted per second. |
Dependent item | kube.node.network_transmit_bytes Preprocessing
|
| Uptime | Node uptime. |
Dependent item | kube.node.boot_time Preprocessing
|
| Pressure: CPU waiting | CPU pressure - time processes spend waiting for CPU scheduling per second (kernel PSI metric). |
Dependent item | kube.node.cpu_pressure_waiting Preprocessing
|
| Pressure: Memory waiting | Memory pressure - time processes spend waiting for memory per second (kernel PSI metric). |
Dependent item | kube.node.memory_pressure_waiting Preprocessing
|
| Pressure: Memory stalled | Memory pressure - time processes are fully stalled waiting for memory per second (kernel PSI metric). |
Dependent item | kube.node.memory_pressure_stalled Preprocessing
|
| Pressure: I/O waiting | I/O pressure - time processes spend waiting for I/O per second (kernel PSI metric). |
Dependent item | kube.node.io_pressure_waiting Preprocessing
|
| Pressure: I/O stalled | I/O pressure - time processes are fully stalled waiting for I/O per second (kernel PSI metric). |
Dependent item | kube.node.io_pressure_stalled Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Node: High CPU utilization | CPU utilization is over {$KUBE.NODE.CPU.UTILIZATION.CRIT}% for 5 minutes. |
min(/Kubernetes Node by HTTP/kube.node.cpu_utilization, 5m) > {$KUBE.NODE.CPU.UTILIZATION.CRIT} |
High | |
| Kubernetes Node: High memory utilization | The system is running out of free memory. |
min(/Kubernetes Node by HTTP/kube.node.memory_utilization,5m)>{$KUBE.NODE.MEMORY.UTIL.MAX} |
Average | |
| Kubernetes Node: Load average is too high | The load average per CPU is too high. The system may be slow to respond. |
min(/Kubernetes Node by HTTP/kube.node.load_avg_1min,5m)/last(/Kubernetes Node by HTTP/kube.node.cpu_core)>{$KUBE.NODE.LOAD_AVG_PER_CPU.MAX.WARN} and last(/Kubernetes Node by HTTP/kube.node.load_avg_5min)>0 and last(/Kubernetes Node by HTTP/kube.node.load_avg_15min)>0 |
Average | |
| Kubernetes Node: Node has been restarted | Uptime is less than 10 minutes. |
last(/Kubernetes Node by HTTP/kube.node.boot_time)<10 |
Info | Manual close: Yes |
| Kubernetes Node: High CPU pressure | Node CPU pressure is above {$KUBE.NODE.PSI.CPU.WARN}%. Processes are spending significant time waiting for CPU scheduling, indicating CPU saturation. |
last(/Kubernetes Node by HTTP/kube.node.cpu_pressure_waiting) > {$KUBE.NODE.PSI.CPU.WARN} |
Warning | Depends on:
|
| Kubernetes Node: Critical CPU pressure | Node CPU pressure is critically high (>{$KUBE.NODE.PSI.CPU.CRIT}%). Severe CPU saturation detected. |
last(/Kubernetes Node by HTTP/kube.node.cpu_pressure_waiting) > {$KUBE.NODE.PSI.CPU.CRIT} |
High | |
| Kubernetes Node: High memory pressure | Node memory pressure is above {$KUBE.NODE.PSI.MEMORY.WARN}%. Processes are experiencing significant memory contention. |
last(/Kubernetes Node by HTTP/kube.node.memory_pressure_waiting) > {$KUBE.NODE.PSI.MEMORY.WARN} |
Warning | Depends on:
|
| Kubernetes Node: Critical memory pressure | Node memory pressure is critically high (>{$KUBE.NODE.PSI.MEMORY.CRIT}%). Processes are fully stalled waiting for memory - severe memory saturation. |
last(/Kubernetes Node by HTTP/kube.node.memory_pressure_waiting) > {$KUBE.NODE.PSI.MEMORY.CRIT} |
High |
LLD rule Storage discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Storage discovery | Discovery Node disk. |
Dependent item | kube.node.storage.discovery Preprocessing
|
Item prototypes for Storage discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Disk [{#NAME}]: Read IOps | Disk |
Dependent item | kube.node.disk_read_io_ps[{#NAME}] Preprocessing
|
| Disk [{#NAME}]: Write IOps | Disk |
Dependent item | kube.node.disk_write_io_ps[{#NAME}] Preprocessing
|
| Disk [{#NAME}]: I/O Utilization | Percentage of time the disk |
Dependent item | kube.node.disk_io_utilization[{#NAME}] Preprocessing
|
| Disk [{#NAME}]: Read bytes | Disk |
Dependent item | kube.node.disk_read_bytes[{#NAME}] Preprocessing
|
| Disk [{#NAME}]: Write bytes | Disk |
Dependent item | kube.node.disk_write_bytes[{#NAME}] Preprocessing
|
Trigger prototypes for Storage discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Node: Disk [{#NAME}]: I/O utilization is too high | Current disk |
min(/Kubernetes Node by HTTP/kube.node.disk_io_utilization[{#NAME}],5m)>={$KUBE.NODE.IO.UTIL.HIGH:"{#NAME}"} |
High |
LLD rule Filesystem discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Filesystem discovery | Discovery of filesystem mount points from node-exporter metrics. |
Dependent item | kube.node.filesystem.discovery Preprocessing
|
Item prototypes for Filesystem discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Filesystem [{#FSNAME}]: Available space | Available space on filesystem |
Dependent item | kube.node.filesystem.avail_bytes[{#FSNAME}] Preprocessing
|
| Filesystem [{#FSNAME}]: Total space | Total space on filesystem |
Dependent item | kube.node.filesystem.size_bytes[{#FSNAME}] Preprocessing
|
| Filesystem [{#FSNAME}]: Used space | Used space on filesystem |
Calculated | kube.node.filesystem.used_bytes[{#FSNAME}] |
| Filesystem [{#FSNAME}]: Utilization | Utilization of filesystem |
Calculated | kube.node.filesystem.utilization[{#FSNAME}] |
Trigger prototypes for Filesystem discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Node: Filesystem [{#FSNAME}]: Low available space | Filesystem |
min(/Kubernetes Node by HTTP/kube.node.filesystem.utilization[{#FSNAME}], 5m) > {$KUBE.NODE.FS.PUSED.MAX.WARN:"{#FSNAME}"} |
Warning | Depends on:
|
| Kubernetes Node: Filesystem [{#FSNAME}]: Critical available space | Filesystem |
min(/Kubernetes Node by HTTP/kube.node.filesystem.utilization[{#FSNAME}], 5m) > {$KUBE.NODE.FS.PUSED.MAX.CRIT:"{#FSNAME}"} |
High |
LLD rule Network interface discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Network interface discovery | Discovery of network interfaces from node-exporter metrics. |
Dependent item | kube.node.network.discovery Preprocessing
|
Item prototypes for Network interface discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Network [{#IFNAME}]: Operational status | Current operational status of network interface |
Dependent item | kube.node.network.if.oper.status[{#IFNAME}] Preprocessing
|
| Network [{#IFNAME}]: Speed | Network interface |
Dependent item | kube.node.network.if.speed[{#IFNAME}] Preprocessing
|
| Network [{#IFNAME}]: Receive bytes | Network interface |
Dependent item | kube.node.net.if.receive_bytes[{#IFNAME}] Preprocessing
|
| Network [{#IFNAME}]: Transmit bytes | Network interface |
Dependent item | kube.node.net.if.transmit_bytes[{#IFNAME}] Preprocessing
|
| Network [{#IFNAME}]: Receive packets | Network interface |
Dependent item | kube.node.net.if.receive_packets[{#IFNAME}] Preprocessing
|
| Network [{#IFNAME}]: Transmit packets | Network interface |
Dependent item | kube.node.net.if.transmit_packets[{#IFNAME}] Preprocessing
|
| Network [{#IFNAME}]: Receive errors | Network interface |
Dependent item | kube.node.net.if.receive_errors[{#IFNAME}] Preprocessing
|
| Network [{#IFNAME}]: Transmit errors | Network interface |
Dependent item | kube.node.net.if.transmit_errors[{#IFNAME}] Preprocessing
|
Trigger prototypes for Network interface discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Node: Link [{#IFNAME}]: Down | This trigger expression works as follows: |
{$IFCONTROL:"{#IFNAME}"}=1 and last(/Kubernetes Node by HTTP/kube.node.network.if.oper.status[{#IFNAME}])=0 and (last(/Kubernetes Node by HTTP/kube.node.network.if.oper.status[{#IFNAME}],#1)<>last(/Kubernetes Node by HTTP/kube.node.network.if.oper.status[{#IFNAME}],#2)) |
Average | Manual close: Yes |
| Kubernetes Node: Network [{#IFNAME}]: High receive errors | Network interface |
last(/Kubernetes Node by HTTP/kube.node.net.if.receive_errors[{#IFNAME}]) > {$KUBE.NODE.NETWORK.ERRORS.WARN:"{#IFNAME}"} |
Warning | |
| Kubernetes Node: Network [{#IFNAME}]: High error rate | It recovers when it is below 80% of the |
min(/Kubernetes Node by HTTP/kube.node.net.if.transmit_errors[{#IFNAME}],5m)>{$KUBE.NODE.NETWORK.ERRORS.WARN:"{#IFNAME}"} or min(/Kubernetes Node by HTTP/kube.node.net.if.receive_errors[{#IFNAME}],5m)>{$KUBE.NODE.NETWORK.ERRORS.WARN:"{#IFNAME}"} |
Warning | Manual close: Yes Depends on:
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes API server by HTTP
Overview
The template to monitor Kubernetes API server that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes API server by HTTP - collects metrics by HTTP agent from API server /metrics endpoint.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes API server 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.API.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: Some metrics may not be collected depending on your Kubernetes API server instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.API.SERVER.URL} | Kubernetes API server metrics endpoint URL. |
https://localhost:6443/metrics |
| {$KUBE.API.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.API.HTTP.SERVER.ERROR} | Maximum number of HTTP server requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get API instance metrics | Get raw metrics from API instance /metrics endpoint. |
HTTP agent | kubernetes.api.get_metrics Preprocessing
|
| Audit events, total | Accumulated number audit events generated and sent to the audit backend. |
Dependent item | kubernetes.api.audit_event_total Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.api.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.api.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.api.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.api.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.api.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.api.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.api.max_fds Preprocessing
|
| gRPCs client started, rate | Total number of RPCs started per second. |
Dependent item | kubernetes.api.grpc_client_started.rate Preprocessing
|
| gRPCs messages received, rate | Total number of gRPC stream messages received per second. |
Dependent item | kubernetes.api.grpc_client_msg_received.rate Preprocessing
|
| gRPCs messages sent, rate | Total number of gRPC stream messages sent per second. |
Dependent item | kubernetes.api.grpc_client_msg_sent.rate Preprocessing
|
| Request terminations, rate | Number of requests which apiserver terminated in self-defense per second. |
Dependent item | kubernetes.api.apiserver_request_terminations Preprocessing
|
| TLS handshake errors, rate | Number of requests dropped with 'TLS handshake error from' error per second. |
Dependent item | kubernetes.api.apiserver_tls_handshake_errors_total.rate Preprocessing
|
| API server requests: 5xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_500.rate Preprocessing
|
| API server requests: 4xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_400.rate Preprocessing
|
| API server requests: 3xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_300.rate Preprocessing
|
| API server requests: 0 | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_0.rate Preprocessing
|
| API server requests: 2xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_200.rate Preprocessing
|
| HTTP requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_500.rate Preprocessing
|
| HTTP requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_400.rate Preprocessing
|
| HTTP requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_300.rate Preprocessing
|
| HTTP requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_200.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API server: Too many server errors | "Kubernetes API server is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.apiserver_request_total_500.rate,5m)>{$KUBE.API.HTTP.SERVER.ERROR} |
Warning | |
| Kubernetes API server: Too many client errors | "Kubernetes API client is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.rest_client_requests_total_500.rate,5m)>{$KUBE.API.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running requests | Discovery of long-running requests by verb, resource and scope. |
Dependent item | kubernetes.api.longrunning_gauge.discovery Preprocessing
|
Item prototypes for Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running ["{#VERB}"] requests ["{#RESOURCE}"]: {#SCOPE} | Gauge of all active long-running apiserver requests broken out by verb, resource and scope. Not all requests are tracked this way. |
Dependent item | kubernetes.api.longrunning_gauge["{#RESOURCE}","{#SCOPE}","{#VERB}"] Preprocessing
|
LLD rule Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Request duration histogram | Discovery raw data and percentile items of request duration. |
Dependent item | kubernetes.api.requests_bucket.discovery Preprocessing
|
Item prototypes for Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#VERB}"] Requests bucket: {#LE} | Response latency distribution in seconds for each verb. |
Dependent item | kubernetes.api.request_duration_seconds_bucket[{#LE},"{#VERB}"] Preprocessing
|
| ["{#VERB}"] Requests, p90 | 90 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p90["{#VERB}"] |
| ["{#VERB}"] Requests, p95 | 95 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p95["{#VERB}"] |
| ["{#VERB}"] Requests, p99 | 99 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p99["{#VERB}"] |
| ["{#VERB}"] Requests, p50 | 50 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p50["{#VERB}"] |
LLD rule Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests inflight discovery | Discovery requests inflight by kind. |
Dependent item | kubernetes.api.inflight_requests.discovery Preprocessing
|
Item prototypes for Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests current: {#KIND} | Maximal number of currently used inflight request limit of this apiserver per request kind in last second. |
Dependent item | kubernetes.api.current_inflight_requests["{#KIND}"] Preprocessing
|
LLD rule gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPC completed requests discovery | Discovery grpc completed requests by grpc code. |
Dependent item | kubernetes.api.grpc_client_handled.discovery Preprocessing
|
Item prototypes for gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPCs completed: {#GRPC_CODE}, rate | Total number of RPCs completed by the client regardless of success or failure per second. |
Dependent item | kubernetes.api.grpc_client_handled_total.rate["{#GRPC_CODE}"] Preprocessing
|
LLD rule Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts discovery | Discovery authentication attempts by result. |
Dependent item | kubernetes.api.authentication_attempts.discovery Preprocessing
|
Item prototypes for Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts: {#RESULT}, rate | Authentication attempts by result per second. |
Dependent item | kubernetes.api.authentication_attempts.rate["{#RESULT}"] Preprocessing
|
LLD rule Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication requests discovery | Discovery authentication attempts by name. |
Dependent item | kubernetes.api.authenticated_user_requests.discovery Preprocessing
|
Item prototypes for Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authenticated requests: {#NAME}, rate | Counter of authenticated requests broken out by username per second. |
Dependent item | kubernetes.api.authenticated_user_requests.rate["{#NAME}"] Preprocessing
|
LLD rule Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers metrics discovery | Discovery watchers by kind. |
Dependent item | kubernetes.api.apiserver_registered_watchers.discovery Preprocessing
|
Item prototypes for Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers: {#KIND} | Number of currently registered watchers for a given resource. |
Dependent item | kubernetes.api.apiserver_registered_watchers["{#KIND}"] Preprocessing
|
LLD rule Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Etcd objects metrics discovery | Discovery etcd objects by resource. |
Dependent item | kubernetes.api.etcd_object_counts.discovery Preprocessing
|
Item prototypes for Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| etcd objects: {#RESOURCE} | Number of stored objects at the time of last check split by kind. |
Dependent item | kubernetes.api.etcd_object_counts["{#RESOURCE}"] Preprocessing
|
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Discovery workqueue metrics by name. |
Dependent item | kubernetes.api.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"] Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.api.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"] Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.api.workqueue_adds_total.rate["{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Controller manager by HTTP
Overview
The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP - collects metrics by HTTP agent from Controller manager /metrics endpoint.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes Controller manager 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.CONTROLLER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: You might need to set the --binding-address option for Controller Manager to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
Note: Some metrics may not be collected depending on your Kubernetes Controller manager instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token |
|
| {$KUBE.CONTROLLER.SERVER.URL} | Kubernetes Controller manager metrics endpoint URL. |
https://localhost:10257/metrics |
| {$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller: Get Controller metrics | Get raw metrics from Controller instance /metrics endpoint. |
HTTP agent | kubernetes.controller.get_metrics Preprocessing
|
| Leader election status | Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. |
Dependent item | kubernetes.controller.leader_election_master_status Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.controller.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.controller.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.controller.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.controller.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.controller.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.controller.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.controller.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_500.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Controller manager: Too many HTTP client errors | "Kubernetes Controller manager is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Controller manager by HTTP/kubernetes.controller.client_http_requests_500.rate,5m)>{$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Dependent item | kubernetes.controller.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"]: Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_adds_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.controller.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue unfinished work, sec | How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
Dependent item | kubernetes.controller.workqueue_unfinished_work_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue retries, rate | Total number of retries handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_retries_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue longest running processor, sec | How many seconds has the longest running processor for workqueue been running. |
Dependent item | kubernetes.controller.workqueue_longest_running_processor_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue work duration, p90 | 90 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p95 | 95 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p99 | 99 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, 50p | 50 percentiles of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p50["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p90 | 90 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p95 | 95 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p99 | 99 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, 50p | 50 percentile of how long in seconds an item stays in workqueue before being requested. If there are no requests for 5 minute, item value will be discarded. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p50["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue duration seconds bucket, {#LE} | How long in seconds processing an item from workqueue takes. |
Dependent item | kubernetes.controller.duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Queue duration seconds bucket, {#LE} | How long in seconds an item stays in workqueue before being requested. |
Dependent item | kubernetes.controller.queue_duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Kubelet by HTTP
Overview
The template to monitor Kubernetes Kubelet by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Kubelet by HTTP - collects metrics by HTTP agent from Kubelet /metrics endpoint.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
Note: Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
Note: Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.KUBELET.URL} | Kubernetes Kubelet instance URL. |
https://localhost:10250 |
| {$KUBE.KUBELET.METRIC.ENDPOINT} | Kubelet /metrics endpoint. |
/metrics |
| {$KUBE.KUBELET.CADVISOR.ENDPOINT} | cAdvisor metrics from Kubelet /metrics/cadvisor endpoint. |
/metrics/cadvisor |
| {$KUBE.KUBELET.PODS.ENDPOINT} | Kubelet /pods endpoint. |
/pods |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get kubelet metrics | Collecting raw Kubelet metrics from /metrics endpoint. |
HTTP agent | kube.kubelet.metrics |
| Get cadvisor metrics | Collecting raw Kubelet metrics from /metrics/cadvisor endpoint. |
HTTP agent | kube.cadvisor.metrics |
| Get pods | Collecting raw Kubelet metrics from /pods endpoint. |
HTTP agent | kube.pods |
| Pods running | The number of running pods. |
Dependent item | kube.kubelet.pods.running Preprocessing
|
| Containers started | The number of started containers. |
Dependent item | kube.kubelet.containers.started Preprocessing
|
| Containers ready | The number of ready containers. |
Dependent item | kube.kubelet.containers.ready Preprocessing
|
| Containers last state terminated | The number of containers that were previously terminated. |
Dependent item | kube.kublet.containers.terminated Preprocessing
|
| Containers restarts | The number of times the container has been restarted. |
Dependent item | kube.kubelet.containers.restarts Preprocessing
|
| CPU cores, total | The number of cores in this machine (available until kubernetes v1.18). |
Dependent item | kube.kubelet.cpu.cores Preprocessing
|
| Machine memory, bytes | Resident memory size in bytes. |
Dependent item | kube.kubelet.machine.memory Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kube.kubelet.virtual.memory Preprocessing
|
| File descriptors, max | Maximum number of open file descriptors. |
Dependent item | kube.kubelet.process_max_fds Preprocessing
|
| File descriptors, open | Number of open file descriptors. |
Dependent item | kube.kubelet.process_open_fds Preprocessing
|
LLD rule Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Runtime operations discovery | Dependent item | kube.kubelet.runtime_operations_bucket.discovery Preprocessing
|
Item prototypes for Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| [{#OP_TYPE}] Runtime operations bucket: {#LE} | Duration in seconds of runtime operations. Broken down by operation type. |
Dependent item | kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Runtime operations total, rate | Cumulative number of runtime operations by operation type. |
Dependent item | kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Operations, p90 | 90 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p95 | 95 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p99 | 99 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p50 | 50 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"] |
LLD rule Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pods discovery | Dependent item | kube.kubelet.pods.discovery Preprocessing
|
Item prototypes for Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s | Pods cpu load average over the last 10 seconds. |
Dependent item | kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total | System cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Usage seconds, total | Consumed cpu time. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_usage_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total | User cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| REST client requests discovery | Dependent item | kube.kubelet.rest.requests.discovery Preprocessing
|
Item prototypes for REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}] | Number of HTTP requests, partitioned by status code, method, and host. |
Dependent item | kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"] Preprocessing
|
LLD rule Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Container memory discovery | Dependent item | kube.kubelet.container.memory.cache.discovery Preprocessing
|
Item prototypes for Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache | Number of bytes of page cache memory. |
Dependent item | kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage | Maximum memory usage recorded in bytes. |
Dependent item | kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS | Size of RSS in bytes. |
Dependent item | kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap | Container swap usage in bytes. |
Dependent item | kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage | Current memory usage in bytes, including all memory regardless of when it was accessed. |
Dependent item | kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set | Current working set in bytes. |
Dependent item | kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes nodes by HTTP
Overview
The template to monitor Kubernetes nodes that work without any external scripts. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API. Install the Zabbix Helm Chart (https://git.zabbix.com/projects/ZT/repos/kubernetes-helm/browse?at=refs%2Fheads%2Fmaster) in your Kubernetes cluster.
Change the values according to the environment in the file $HOME/zabbix_values.yaml.
For example:
Enables use of Zabbix proxy
enabled: false
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the service account name. If a different release name is used.
kubectl get serviceaccounts -n monitoring
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set up the macros to filter the metrics of discovered nodes.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chart-agent.
Set up the macros to filter the metrics of discovered nodes and host creation based on host prototypes:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES}
Set up macros to filter pod metrics by namespace:
- {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable pods.
You can use the {$KUBE.NODE.FILTER.LABELS}, {$KUBE.POD.FILTER.LABELS}, {$KUBE.NODE.FILTER.ANNOTATIONS} and {$KUBE.POD.FILTER.ANNOTATIONS} macros for advanced filtering of nodes and pods by labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such a label key exists for the entity that is being filtered (it means that if you specify a key in a filter, entities which do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
Note: The discovered nodes will be created as separate hosts in Zabbix with the Linux template automatically assigned to them.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$HTTP.TLS.VERIFY} | TLS certificate verification for the script item: "none" - disabled, "peer" - verify the certificate chain and expiration, "full" - full verification. Any other value enables full verification. To override the setting for this template only, define the macro with the context "Kubernetes nodes", e.g. {$HTTP.TLS.VERIFY:"Kubernetes nodes"}. |
full |
| {$KUBE.NODES.ENDPOINT.NAME} | Kubernetes nodes endpoint name. See "kubectl -n monitoring get ep". |
zabbix-zabbix-helm-chart-agent |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES} | Filter of discoverable nodes by role. |
.* |
| {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES} | Filter to exclude discovered node by role. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.FILTER.ANNOTATIONS} | Annotations to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.NODE.FILTER.LABELS} | Labels to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.ANNOTATIONS} | Annotations to filter pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.LABELS} | Labels to filter Pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace. |
.* |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace. |
CHANGE_IF_NEEDED |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get nodes | Collecting and processing cluster nodes data via Kubernetes API. |
Script | kube.nodes |
| Get nodes check | Data collection check. |
Dependent item | kube.nodes.check Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Failed to get nodes | length(last(/Kubernetes nodes by HTTP/kube.nodes.check))>0 |
Warning |
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: Get data | Collecting and processing cluster by node [{#NAME}] data via Kubernetes API. |
Dependent item | kube.node.get[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: External IP | Typically the IP address of the node that is externally routable (available from outside the cluster). |
Dependent item | kube.node.addresses.external_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: Internal IP | Typically the IP address of the node that is routable only within the cluster. |
Dependent item | kube.node.addresses.internal_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: CPU | Allocatable CPU. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Memory | Allocatable Memory. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.allocatable.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: CPU | CPU resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Memory | Memory resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.capacity.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Disk pressure | True if pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
Dependent item | kube.node.conditions.diskpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Memory pressure | True if pressure exists on the node memory - that is, if the node memory is low; otherwise False. |
Dependent item | kube.node.conditions.memorypressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Network unavailable | True if the network for the node is not correctly configured, otherwise False. |
Dependent item | kube.node.conditions.networkunavailable[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: PID pressure | True if pressure exists on the processes - that is, if there are too many processes on the node; otherwise False. |
Dependent item | kube.node.conditions.pidpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Ready | True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
Dependent item | kube.node.conditions.ready[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Architecture | Node architecture. |
Dependent item | kube.node.info.architecture[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Container runtime | Container runtime. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
Dependent item | kube.node.info.containerruntime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kernel version | Node kernel version. |
Dependent item | kube.node.info.kernelversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kubelet version | Version of Kubelet. |
Dependent item | kube.node.info.kubeletversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: KubeProxy version | Version of KubeProxy. |
Dependent item | kube.node.info.kubeproxyversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Operating system | Node operating system. |
Dependent item | kube.node.info.operatingsystem[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: OS image | Node OS image. |
Dependent item | kube.node.info.osversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Roles | Node roles. |
Dependent item | kube.node.info.roles[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: CPU | Node CPU limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: Memory | Node Memory limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: CPU | Node CPU requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: Memory | Node Memory requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Uptime | Node uptime. |
Dependent item | kube.node.uptime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Used: Pods | Current number of pods on the node. |
Dependent item | kube.node.used.pods[{#NAME}] Preprocessing
|
Trigger prototypes for Node discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the disk size | True - pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.diskpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the node memory | True - pressure exists on the node memory - that is, if the node memory is low; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.memorypressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Network is not correctly configured | True - the network for the node is not correctly configured, otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.networkunavailable[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the processes | True - pressure exists on the processes - that is, if there are too many processes on the node; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.pidpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Is not in Ready state | False - if the node is not healthy and is not accepting pods. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.ready[{#NAME}])<>1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] has been restarted | Uptime is less than 10 minutes. |
last(/Kubernetes nodes by HTTP/kube.node.uptime[{#NAME}])<10 |
Info | |
| Kubernetes nodes: Node [{#NAME}] Used: Kubelet too many pods | Kubelet is running at capacity. |
last(/Kubernetes nodes by HTTP/kube.node.used.pods[{#NAME}])/ last(/Kubernetes nodes by HTTP/kube.node.capacity.pods[{#NAME}]) > 0.9 |
Warning |
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Get data | Collecting and processing cluster by node [{#NODE}] data via Kubernetes API. |
Dependent item | kube.pod.get[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Containers ready | All containers in the Pod are ready. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.containers_ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Initialized | All init containers have started successfully. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.initialized[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Ready | The Pod is able to serve requests and should be added to the load balancing pools of all matching Services. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Scheduled | The Pod has been scheduled to a node. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.scheduled[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Containers: Restarts | The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. |
Dependent item | kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Phase | The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase |
Dependent item | kube.pod.status.phase[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Uptime | Pod uptime. |
Dependent item | kube.pod.uptime[{#NAMESPACE}/{#POD}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}])-min(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}],15m))>1 |
Warning | |
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Kubernetes Pod not healthy | Pod has been in a non-ready state for longer than 10 minutes. |
count(/Kubernetes nodes by HTTP/kube.pod.status.phase[{#NAMESPACE}/{#POD}],10m, "regexp","^(1|4|5)$")>=9 |
High |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Scheduler by HTTP
Overview
The template to monitor Kubernetes Scheduler by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Scheduler by HTTP - collects metrics by HTTP agent from Scheduler /metrics endpoint.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes Scheduler 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.SCHEDULER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: You might need to set the --binding-address option for Scheduler to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-scheduler.yaml
Note: Some metrics may not be collected depending on your Kubernetes Scheduler instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.SCHEDULER.SERVER.URL} | Kubernetes Scheduler metrics endpoint URL. |
https://localhost:10259/metrics |
| {$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.SCHEDULER.UNSCHEDULABLE} | Maximum number of scheduling failures with 'unschedulable' used for trigger. |
2 |
| {$KUBE.SCHEDULER.ERROR} | Maximum number of scheduling failures with 'error' used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get Scheduler metrics | Get raw metrics from Scheduler instance /metrics endpoint. |
HTTP agent | kubernetes.scheduler.get_metrics Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.scheduler.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.scheduler.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.scheduler.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.scheduler.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.scheduler.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.scheduler.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.scheduler.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_500.rate Preprocessing
|
| Schedule attempts: scheduled | Number of attempts to schedule pods with result "scheduled" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.scheduled.rate Preprocessing
|
| Schedule attempts: unschedulable | Number of attempts to schedule pods with result "unschedulable" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate Preprocessing
|
| Schedule attempts: error | Number of attempts to schedule pods with result "error" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.error.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler: Too many REST Client errors | "Kubernetes Scheduler REST Client requests is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.client_http_requests_500.rate,5m)>{$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} |
Warning | |
| Kubernetes Scheduler: Too many unschedulable pods | Number of attempts to schedule pods with 'unschedulable' result is too high. 'unschedulable' means a pod could not be scheduled. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate,5m)>{$KUBE.SCHEDULER.UNSCHEDULABLE} |
Warning | |
| Kubernetes Scheduler: Too many schedule attempts with errors | Number of attempts to schedule pods with 'error' result is too high. 'error' means an internal scheduler problem. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.error.rate,5m)>{$KUBE.SCHEDULER.ERROR} |
Warning |
LLD rule Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm histogram | Discovery raw data of scheduling algorithm latency. |
Dependent item | kubernetes.scheduler.scheduling_algorithm.discovery Preprocessing
|
Item prototypes for Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm duration bucket, {#LE} | Scheduling algorithm latency in seconds. |
Dependent item | kubernetes.scheduler.scheduling_algorithm_duration[{#LE}] Preprocessing
|
| Scheduling algorithm duration, p90 | 90 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p90[{#SINGLETON}] |
| Scheduling algorithm duration, p95 | 95 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p95[{#SINGLETON}] |
| Scheduling algorithm duration, p99 | 99 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p99[{#SINGLETON}] |
| Scheduling algorithm duration, p50 | 50 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p50[{#SINGLETON}] |
LLD rule Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding histogram | Discovery raw data of binding latency. |
Dependent item | kubernetes.scheduler.binding.discovery Preprocessing
|
Item prototypes for Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding duration bucket, {#LE} | Binding latency in seconds. |
Dependent item | kubernetes.scheduler.binding_duration[{#LE}] Preprocessing
|
| Binding duration, p90 | 90 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p90[{#SINGLETON}] |
| Binding duration, p95 | 99 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p95[{#SINGLETON}] |
| Binding duration, p99 | 95 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p99[{#SINGLETON}] |
| Binding duration, p50 | 50 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p50[{#SINGLETON}] |
LLD rule e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| e2e scheduling histogram | Discovery raw data and percentile items of e2e scheduling latency. |
Dependent item | kubernetes.controller.e2e_scheduling.discovery Preprocessing
|
Item prototypes for e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#RESULT}"]: e2e scheduling seconds bucket, {#LE} | E2e scheduling latency in seconds (scheduling algorithm + binding) |
Dependent item | kubernetes.scheduler.e2e_scheduling_bucket[{#LE},"{#RESULT}"] Preprocessing
|
| ["{#RESULT}"]: e2e scheduling, p50 | 50 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p50["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p90 | 90 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p90["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p95 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p95["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p99 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p99["{#RESULT}"] |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes cluster state by HTTP
Overview
The template to monitor Kubernetes state. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes cluster state by HTTP - collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget to change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Requirements
Zabbix version: 8.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the service account name. If a different release name is used.
kubectl get serviceaccounts -n monitoring
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.STATE.ENDPOINT.NAME} with Kube state metrics endpoint name. See kubectl -n monitoring get ep. Default: zabbix-kube-state-metrics.
Note: If you wish to monitor Controller Manager and Scheduler components, you might need to set the --binding-address option for them to the address where Zabbix proxy can reach them.
For example, for clusters created with kubeadm it can be set in the following manifest files (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-scheduler.yaml
Depending on your Kubernetes distribution, you might need to adjust {$KUBE.CONTROL_PLANE.TAINT} macro (for example, set it to node-role.kubernetes.io/master for OpenShift).
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Also, see the Macros section for a list of macros used to set trigger values.
Set up the macros to filter the metrics of discovered Kubelets by node names:
- {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES}
- {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Set up macros to filter node metrics by nodename:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable namespaces.
You can use the {$KUBE.KUBELET.FILTER.LABELS} and {$KUBE.KUBELET.FILTER.ANNOTATIONS} macros for advanced filtering of kubelets by node labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such label key exists for the entity that is being filtered (it means that if you specify a key in the filter, entities that do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the kubelets on nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
You can also set up evaluation periods for replica mismatch triggers (Deployments, ReplicaSets, StatefulSets) with the macro {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD}, which supports context and regular expressions. For example, you can create the following macros:
- Set the evaluation period for the Deployment "nginx-deployment" in the namespace "default" to the 3 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:default:nginx-deployment"} = #3
- Set the evaluation period for all Deployments to the 10 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"deployment:.*:.*"} = #10 or {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"^deployment.*"} = #10
- Set the evaluation period for Deployments, ReplicaSets and StatefulSets in the namespace "default" to 15 minutes:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:".*:default:.*"} = 15m
Note: that different context macros with regular expressions matching the same string can be applied in an undefined order, and simple context macros (without regular expressions) have higher priority. Read the Important notes section in Zabbix documentation for details.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.LIVEZ.ENDPOINT} | Kubernetes API livez endpoint /livez |
/livez |
| {$KUBE.API.COMPONENTSTATUSES.ENDPOINT} | Kubernetes API componentstatuses endpoint /api/v1/componentstatuses |
/api/v1/componentstatuses |
| {$KUBE.API.READYZ.ENDPOINT} | Kubernetes API readyz endpoint /readyz |
/readyz |
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$HTTP.TLS.VERIFY} | TLS certificate verification for script items: "none" - disabled, "peer" - verify the certificate chain and expiration, "full" - full verification. Any other value enables full verification. To override the setting for this template only, define the macro with the context "Kubernetes cluster state", e.g. {$HTTP.TLS.VERIFY:"Kubernetes cluster state"}. |
full |
| {$KUBE.STATE.ENDPOINT.NAME} | Endpoint name for kube-state-metrics service (check with |
zabbix-kube-state-metrics |
| {$OPENSHIFT.STATE.ENDPOINT.NAME} | OpenShift state endpoint name. |
openshift-state-metrics |
| {$KUBE.API_SERVER.SCHEME} | Kubernetes API servers metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.API_SERVER.PORT} | Kubernetes API servers metrics endpoint port. Used in ControlPlane LLD. |
6443 |
| {$KUBE.CONTROL_PLANE.TAINT} | Taint that applies to control plane nodes. Change if needed. Used in ControlPlane LLD. |
node-role.kubernetes.io/control-plane |
| {$KUBE.CONTROLLER_MANAGER.SCHEME} | Kubernetes Controller manager metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.CONTROLLER_MANAGER.PORT} | Kubernetes Controller manager metrics endpoint port. Used in ControlPlane LLD. |
10257 |
| {$KUBE.SCHEDULER.SCHEME} | Kubernetes Scheduler metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.SCHEDULER.PORT} | Kubernetes Scheduler metrics endpoint port. Used in ControlPlane LLD. |
10259 |
| {$KUBE.KUBELET.SCHEME} | Kubernetes Kubelet metrics endpoint scheme. Used in Kubelet LLD. |
https |
| {$KUBE.KUBELET.PORT} | Kubernetes Kubelet metrics endpoint port. Used in Kubelet LLD. |
10250 |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable metrics by namespace. |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered metrics by namespace. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES} | Filter of discoverable Kubelets by nodename. |
.* |
| {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES} | Filter to exclude discovered Kubelets by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.KUBELET.FILTER.ANNOTATIONS} | Node annotations to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.KUBELET.FILTER.LABELS} | Node labels to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.PV.MATCHES} | Filter of discoverable persistent volumes by name. |
.* |
| {$KUBE.LLD.FILTER.PV.NOT_MATCHES} | Filter to exclude discovered persistent volumes by name. |
CHANGE_IF_NEEDED |
| {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD} | The evaluation period range which is used for calculation of expressions in trigger prototypes (time period or value range). Can be used with context. |
#5 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
Script | kube.state.metrics |
| Control plane LLD | Generation of data for Control plane discovery rules. |
Script | kube.control_plane.lld Preprocessing
|
| Node LLD | Generation of data for Kubelet discovery rules. |
Script | kube.node.lld Preprocessing
|
| Get component statuses | HTTP agent | kube.componentstatuses Preprocessing
|
|
| Get readyz | HTTP agent | kube.readyz Preprocessing
|
|
| Get livez | HTTP agent | kube.livez Preprocessing
|
|
| Namespace count | The number of namespaces. |
Dependent item | kube.namespace.count Preprocessing
|
| CronJob count | Number of cronjobs. |
Dependent item | kube.cronjob.count Preprocessing
|
| Job count | Number of jobs (generated by cronjob + job). |
Dependent item | kube.job.count Preprocessing
|
| Endpoint count | Number of endpoints. |
Dependent item | kube.endpoint.count Preprocessing
|
| Deployment count | The number of deployments. |
Dependent item | kube.deployment.count Preprocessing
|
| Service count | The number of services. |
Dependent item | kube.service.count Preprocessing
|
| StatefulSet count | The number of statefulsets. |
Dependent item | kube.statefulset.count Preprocessing
|
| Node count | The number of nodes. |
Dependent item | kube.node.count Preprocessing
|
LLD rule API servers discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| API servers discovery | Dependent item | kube.api_servers.discovery |
LLD rule Controller manager nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Controller manager nodes discovery | Dependent item | kube.controller_manager.discovery |
LLD rule Scheduler servers nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduler servers nodes discovery | Dependent item | kube.scheduler.discovery |
LLD rule Kubelet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubelet discovery | Dependent item | kube.kubelet.discovery |
LLD rule Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Daemonset discovery | Dependent item | kube.daemonset.discovery Preprocessing
|
Item prototypes for Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Ready | The number of nodes that should be running the daemon pod and have one or more running and ready. |
Dependent item | kube.daemonset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Scheduled | The number of nodes that run at least one daemon pod and are supposed to. |
Dependent item | kube.daemonset.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Desired | The number of nodes that should be running the daemon pod. |
Dependent item | kube.daemonset.desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Misscheduled | The number of nodes that run a daemon pod but are not supposed to. |
Dependent item | kube.daemonset.misscheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Updated number scheduled | The total number of nodes that are running updated daemon pod. |
Dependent item | kube.daemonset.updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PVC discovery | Dependent item | kube.pvc.discovery Preprocessing
|
Item prototypes for PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase | The current status phase of the persistent volume claim. |
Dependent item | kube.pvc.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
Dependent item | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Bound, sum | The total amount of persistent volume claims in the Bound phase. |
Dependent item | kube.pvc.status_phase.bound.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Lost, sum | The total amount of persistent volume claims in the Lost phase. |
Dependent item | kube.pvc.status_phase.lost.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Pending, sum | The total amount of persistent volume claims in the Pending phase. |
Dependent item | kube.pvc.status_phase.pending.sum[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for PVC discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | count(/Kubernetes cluster state by HTTP/kube.pvc.status_phase[{#NAMESPACE}/{#NAME}],2m,,5)>=2 |
Warning |
LLD rule PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV discovery | Dependent item | kube.pv.discovery Preprocessing
|
Item prototypes for PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV [{#NAME}] Status phase | The current status phase of the persistent volume. |
Dependent item | kube.pv.status_phase[{#NAME}] Preprocessing
|
| PV [{#NAME}] Capacity bytes | A capacity of the persistent volume in bytes. |
Dependent item | kube.pv.capacity.bytes[{#NAME}] Preprocessing
|
| PV status phase: Pending, sum | The total amount of persistent volumes in the Pending phase. |
Dependent item | kube.pv.status_phase.pending.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Available, sum | The total amount of persistent volumes in the Available phase. |
Dependent item | kube.pv.status_phase.available.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Bound, sum | The total amount of persistent volumes in the Bound phase. |
Dependent item | kube.pv.status_phase.bound.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Released, sum | The total amount of persistent volumes in the Released phase. |
Dependent item | kube.pv.status_phase.released.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Failed, sum | The total amount of persistent volumes in the Failed phase. |
Dependent item | kube.pv.status_phase.failed.sum[{#SINGLETON}] Preprocessing
|
Trigger prototypes for PV discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: PV [{#NAME}]: PV has failed | count(/Kubernetes cluster state by HTTP/kube.pv.status_phase[{#NAME}],2m,,3)>=2 |
Warning |
LLD rule Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment discovery | Dependent item | kube.deployment.discovery Preprocessing
|
Item prototypes for Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Paused | Whether the deployment is paused and will not be processed by the deployment controller. |
Dependent item | kube.deployment.spec_paused[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas desired | Number of desired pods for a deployment. |
Dependent item | kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Rollingupdate max unavailable | Maximum number of unavailable replicas during a rolling update of a deployment. |
Dependent item | kube.deployment.rollingupdate.max_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas | The number of replicas per deployment. |
Dependent item | kube.deployment.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas available | The number of available replicas per deployment. |
Dependent item | kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas unavailable | The number of unavailable replicas per deployment. |
Dependent item | kube.deployment.replicas_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas updated | The number of updated replicas per deployment. |
Dependent item | kube.deployment.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas mismatched | The number of available replicas not matching the desired number of replicas. |
Dependent item | kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Deployment discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Deployment replicas mismatch | Deployment has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Endpoint discovery | Dependent item | kube.endpoint.discovery Preprocessing
|
Item prototypes for Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address available | Number of addresses available in endpoint. |
Dependent item | kube.endpoint.address_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address not ready | Number of addresses not ready in endpoint. |
Dependent item | kube.endpoint.address_not_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Age | Endpoint age (number of seconds since creation). |
Dependent item | kube.endpoint.age[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: CPU allocatable | The CPU resources of a node that are available for scheduling. |
Dependent item | kube.node.cpu_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory allocatable | The memory resources of a node that are available for scheduling. |
Dependent item | kube.node.memory_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods allocatable | The pods resources of a node that are available for scheduling. |
Dependent item | kube.node.pods_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage allocatable | The allocatable ephemeral storage of a node that is available for scheduling. |
Dependent item | kube.node.ephemeral_storage_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: CPU capacity | The capacity for CPU resources of a node. |
Dependent item | kube.node.cpu_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory capacity | The capacity for memory resources of a node. |
Dependent item | kube.node.memory_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage capacity | The ephemeral storage capacity of a node. |
Dependent item | kube.node.ephemeral_storage_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods capacity | The capacity for pods resources of a node. |
Dependent item | kube.node.pods_capacity[{#NAME}] Preprocessing
|
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Pending | Pod is in pending state. |
Dependent item | kube.pod.phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Succeeded | Pod is in succeeded state. |
Dependent item | kube.pod.phase.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Failed | Pod is in failed state. |
Dependent item | kube.pod.phase.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Unknown | Pod is in unknown state. |
Dependent item | kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Running | Pod is in unknown state. |
Dependent item | kube.pod.phase.running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers terminated | Describes whether the container is currently in terminated state. |
Dependent item | kube.pod.containers_terminated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers waiting | Describes whether the container is currently in waiting state. |
Dependent item | kube.pod.containers_waiting[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers ready | Describes whether the containers readiness check succeeded. |
Dependent item | kube.pod.containers_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
Dependent item | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers running | Describes whether the container is currently in running state. |
Dependent item | kube.pod.containers_running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Ready | Describes whether the pod is ready to serve requests. |
Dependent item | kube.pod.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Scheduled | Describes the status of the scheduling process for the pod. |
Dependent item | kube.pod.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Unschedulable | Describes the unschedulable status for the pod. |
Dependent item | kube.pod.unschedulable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU limits | The limit on CPU cores to be used by a container. |
Dependent item | kube.pod.containers.limits.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory limits | The limit on memory to be used by a container. |
Dependent item | kube.pod.containers.limits.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU requests | The number of requested CPU cores by a container. |
Dependent item | kube.pod.containers.requests.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory requests | The number of requested memory bytes by a container. |
Dependent item | kube.pod.containers.requests.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is not healthy | min(/Kubernetes cluster state by HTTP/kube.pod.phase.failed[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.pending[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}],10m)>0 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],15m))>1 |
Warning |
LLD rule ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ReplicaSet discovery | Dependent item | kube.replicaset.discovery Preprocessing
|
Item prototypes for ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas | The number of replicas per ReplicaSet. |
Dependent item | kube.replicaset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Desired replicas | Number of desired pods for a ReplicaSet. |
Dependent item | kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Fully labeled replicas | The number of fully labeled replicas per ReplicaSet. |
Dependent item | kube.replicaset.fully_labeled_replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Ready | The number of ready replicas per ReplicaSet. |
Dependent item | kube.replicaset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the desired number of replicas. |
Dependent item | kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for ReplicaSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] RS [{#NAME}]: ReplicaSet mismatch | ReplicaSet has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"replicaset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| StatefulSet discovery | Dependent item | kube.statefulset.discovery Preprocessing
|
Item prototypes for StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas | The number of replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Desired replicas | Number of desired pods for a StatefulSet. |
Dependent item | kube.statefulset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Current replicas | The number of current replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Ready replicas | The number of ready replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Updated replicas | The number of updated replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the number of replicas. |
Dependent item | kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for StatefulSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet is down | (last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]) / last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}]))<>1 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet replicas mismatch | StatefulSet has not matched the number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"statefulset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PodDisruptionBudget discovery | Dependent item | kube.pdb.discovery Preprocessing
|
Item prototypes for PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods healthy | Current number of healthy pods. |
Dependent item | kube.pdb.pods_healthy[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods desired | Minimum desired number of healthy pods. |
Dependent item | kube.pdb.pods_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Disruptions allowed | Number of pod disruptions that are allowed. |
Dependent item | kube.pdb.disruptions_allowed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods total | Total number of pods counted by this disruption budget. |
Dependent item | kube.pdb.pods_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| CronJob discovery | Dependent item | kube.cronjob.discovery Preprocessing
|
Item prototypes for CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Suspend | Suspend flag tells the controller to suspend subsequent executions. |
Dependent item | kube.cronjob.spec_suspend[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Active | Active holds pointers to currently running jobs. |
Dependent item | kube.cronjob.status_active[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Last schedule | LastScheduleTime keeps information of when was the last time the job was successfully scheduled. |
Dependent item | kube.cronjob.last_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Next schedule | Next time the cronjob should be scheduled. The time after lastScheduleTime or after the cron job's creation time if it's never been scheduled. Use this to determine if the job is delayed. |
Dependent item | kube.cronjob.next_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.cronjob.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.cronjob.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.cronjob.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.cronjob.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Job discovery | Dependent item | kube.job.discovery Preprocessing
|
Item prototypes for Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.job.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.job.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.job.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.job.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component statuses discovery | Dependent item | kube.componentstatuses.discovery Preprocessing
|
Item prototypes for Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component [{#NAME}]: Healthy | Cluster component healthy. |
Dependent item | kube.componentstatuses.healthy[{#NAME}] Preprocessing
|
Trigger prototypes for Component statuses discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Component [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}],#2,"ne","True")=2 and length(last(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}]))>0 |
Warning |
LLD rule Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz discovery | Dependent item | kube.readyz.discovery Preprocessing
|
Item prototypes for Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz [{#NAME}]: Healthcheck | Result of readyz healthcheck for component. |
Dependent item | kube.readyz.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Readyz discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Readyz [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez discovery | Dependent item | kube.livez.discovery Preprocessing
|
Item prototypes for Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez [{#NAME}]: Healthcheck | Result of livez healthcheck for component. |
Dependent item | kube.livez.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Livez discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Livez [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift BuildConfig discovery | Dependent item | openshift.buildconfig.discovery Preprocessing
|
Item prototypes for OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Created | OpenShift BuildConfig Unix creation timestamp. |
Dependent item | openshift.buildconfig.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.buildconfig.generation[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Latest version | The latest version of BuildConfig. |
Dependent item | openshift.buildconfig.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Build discovery | Dependent item | openshift.build.discovery Preprocessing
|
Item prototypes for OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Created | OpenShift Build Unix creation timestamp. |
Dependent item | openshift.build.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.build.sequence.number[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Status phase | The Build phase. |
Dependent item | openshift.build.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Build discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Build [{#NAME}]: Build has failed | count(/Kubernetes cluster state by HTTP/openshift.build.status_phase[{#NAMESPACE}/{#NAME}],2m,"ge",6)>=2 |
Warning |
LLD rule OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift ClusterResourceQuota discovery | Dependent item | openshift.cluster.resource.quota.discovery Preprocessing
|
Item prototypes for OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Quota [{#NAME}] Resource [{#RESOURCE}]: Type [{#TYPE}]] | Usage about resource quota. |
Dependent item | openshift.cluster.resource.quota[{#RESOURCE}/{#NAME}/{#TYPE}] Preprocessing
|
LLD rule OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Route discovery | Dependent item | openshift.route.discovery Preprocessing
|
Item prototypes for OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Created | OpenShift Route Unix creation timestamp. |
Dependent item | openshift.route.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Status | Information about route status. |
Dependent item | openshift.route.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Route discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Route [{#NAME}] with issue: Status is false | count(/Kubernetes cluster state by HTTP/openshift.route.status[{#NAMESPACE}/{#NAME}],2m,,0)>=2 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Vendor and Community integrations are created by third parties and are not verified by Zabbix. Zabbix is not responsible for their functionality or quality.
Zabbix community templates
1 Contributed to the zabbix/community-templates repositoryDescriptionzabbix-kube-prom is a batch of Zabbix LLD templates for Zabbix server.It is used for external Kubernetes monitoring by Zabbix via Prometheus API.InstallationInstall kube-prometheus-stack into the Kubernetes cluster.Import global Zabbix Template (zabbix-kube-prom.xml) into your Zabbix server.Create or import a host identifying your Kubernetes cluster where Prometheus is deployed.Let LLD create discovered nodes as new "Zabbix hosts"Let LLD create discovered pods as new "Virtual Zabbix hostsTemplatesThe global export (zabbix-kube-prom.xml) contains following templates:TemplatesDescriptionTemplate Kube by Prom APICreates a Zabbix host for each pod and node discovered.Template Kube Node by Prom APITemplate applied to the created host (node).Template Kube Pod by Prom APITemplate applied to the created host (pod).LicensesTemplateLicenseTemplate OS Linux by PromGNU General Public License v2.0 or laterCopyright (C) 2001-2021 Zabbix SIATemplate Kube by Prom APITemplate Kube Node by Prom APITemplate Kube Pod by Prom APIGNU General Public License v3.0Copyright (C) 2021 Diagnostica Stago
4
Community repositories
3 Independent projects on GitHubMonitoring of Docker container by using Zabbix. Available CPU, mem, blkio, net container metrics and some containers config details, e.g. IP, name, ... Zabbix Docker module has native support for Docker containers (Systemd included) and should also support a few other container types (e.g. LXC) out of the box.
1190
329
Nagios-style checks against Kubernetes API. Designed for usage with Nagios, Icinga, Zabbix... Whatever
70
150
Kubernetes autodiscovery monitoring
20
302
Other sources
1Working with a postgres database container, though there is an option to provide a custom database container
Kubernetes API server by HTTP
Overview
The template to monitor Kubernetes API server that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes API server by HTTP - collects metrics by HTTP agent from API server /metrics endpoint.
Requirements
Zabbix version: 7.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes API server 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.API.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: Some metrics may not be collected depending on your Kubernetes API server instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.API.SERVER.URL} | Kubernetes API server metrics endpoint URL. |
https://localhost:6443/metrics |
| {$KUBE.API.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.API.HTTP.SERVER.ERROR} | Maximum number of HTTP server requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get API instance metrics | Get raw metrics from API instance /metrics endpoint. |
HTTP agent | kubernetes.api.get_metrics Preprocessing
|
| Audit events, total | Accumulated number audit events generated and sent to the audit backend. |
Dependent item | kubernetes.api.audit_event_total Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.api.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.api.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.api.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.api.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.api.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.api.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.api.max_fds Preprocessing
|
| gRPCs client started, rate | Total number of RPCs started per second. |
Dependent item | kubernetes.api.grpc_client_started.rate Preprocessing
|
| gRPCs messages received, rate | Total number of gRPC stream messages received per second. |
Dependent item | kubernetes.api.grpc_client_msg_received.rate Preprocessing
|
| gRPCs messages sent, rate | Total number of gRPC stream messages sent per second. |
Dependent item | kubernetes.api.grpc_client_msg_sent.rate Preprocessing
|
| Request terminations, rate | Number of requests which apiserver terminated in self-defense per second. |
Dependent item | kubernetes.api.apiserver_request_terminations Preprocessing
|
| TLS handshake errors, rate | Number of requests dropped with 'TLS handshake error from' error per second. |
Dependent item | kubernetes.api.apiserver_tls_handshake_errors_total.rate Preprocessing
|
| API server requests: 5xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_500.rate Preprocessing
|
| API server requests: 4xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_400.rate Preprocessing
|
| API server requests: 3xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_300.rate Preprocessing
|
| API server requests: 0 | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_0.rate Preprocessing
|
| API server requests: 2xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_200.rate Preprocessing
|
| HTTP requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_500.rate Preprocessing
|
| HTTP requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_400.rate Preprocessing
|
| HTTP requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_300.rate Preprocessing
|
| HTTP requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_200.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API server: Too many server errors | "Kubernetes API server is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.apiserver_request_total_500.rate,5m)>{$KUBE.API.HTTP.SERVER.ERROR} |
Warning | |
| Kubernetes API server: Too many client errors | "Kubernetes API client is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.rest_client_requests_total_500.rate,5m)>{$KUBE.API.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running requests | Discovery of long-running requests by verb, resource and scope. |
Dependent item | kubernetes.api.longrunning_gauge.discovery Preprocessing
|
Item prototypes for Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running ["{#VERB}"] requests ["{#RESOURCE}"]: {#SCOPE} | Gauge of all active long-running apiserver requests broken out by verb, resource and scope. Not all requests are tracked this way. |
Dependent item | kubernetes.api.longrunning_gauge["{#RESOURCE}","{#SCOPE}","{#VERB}"] Preprocessing
|
LLD rule Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Request duration histogram | Discovery raw data and percentile items of request duration. |
Dependent item | kubernetes.api.requests_bucket.discovery Preprocessing
|
Item prototypes for Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#VERB}"] Requests bucket: {#LE} | Response latency distribution in seconds for each verb. |
Dependent item | kubernetes.api.request_duration_seconds_bucket[{#LE},"{#VERB}"] Preprocessing
|
| ["{#VERB}"] Requests, p90 | 90 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p90["{#VERB}"] |
| ["{#VERB}"] Requests, p95 | 95 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p95["{#VERB}"] |
| ["{#VERB}"] Requests, p99 | 99 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p99["{#VERB}"] |
| ["{#VERB}"] Requests, p50 | 50 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p50["{#VERB}"] |
LLD rule Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests inflight discovery | Discovery requests inflight by kind. |
Dependent item | kubernetes.api.inflight_requests.discovery Preprocessing
|
Item prototypes for Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests current: {#KIND} | Maximal number of currently used inflight request limit of this apiserver per request kind in last second. |
Dependent item | kubernetes.api.current_inflight_requests["{#KIND}"] Preprocessing
|
LLD rule gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPC completed requests discovery | Discovery grpc completed requests by grpc code. |
Dependent item | kubernetes.api.grpc_client_handled.discovery Preprocessing
|
Item prototypes for gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPCs completed: {#GRPC_CODE}, rate | Total number of RPCs completed by the client regardless of success or failure per second. |
Dependent item | kubernetes.api.grpc_client_handled_total.rate["{#GRPC_CODE}"] Preprocessing
|
LLD rule Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts discovery | Discovery authentication attempts by result. |
Dependent item | kubernetes.api.authentication_attempts.discovery Preprocessing
|
Item prototypes for Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts: {#RESULT}, rate | Authentication attempts by result per second. |
Dependent item | kubernetes.api.authentication_attempts.rate["{#RESULT}"] Preprocessing
|
LLD rule Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication requests discovery | Discovery authentication attempts by name. |
Dependent item | kubernetes.api.authenticated_user_requests.discovery Preprocessing
|
Item prototypes for Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authenticated requests: {#NAME}, rate | Counter of authenticated requests broken out by username per second. |
Dependent item | kubernetes.api.authenticated_user_requests.rate["{#NAME}"] Preprocessing
|
LLD rule Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers metrics discovery | Discovery watchers by kind. |
Dependent item | kubernetes.api.apiserver_registered_watchers.discovery Preprocessing
|
Item prototypes for Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers: {#KIND} | Number of currently registered watchers for a given resource. |
Dependent item | kubernetes.api.apiserver_registered_watchers["{#KIND}"] Preprocessing
|
LLD rule Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Etcd objects metrics discovery | Discovery etcd objects by resource. |
Dependent item | kubernetes.api.etcd_object_counts.discovery Preprocessing
|
Item prototypes for Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| etcd objects: {#RESOURCE} | Number of stored objects at the time of last check split by kind. |
Dependent item | kubernetes.api.etcd_object_counts["{#RESOURCE}"] Preprocessing
|
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Discovery workqueue metrics by name. |
Dependent item | kubernetes.api.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"] Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.api.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"] Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.api.workqueue_adds_total.rate["{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes API server by HTTP
Overview
The template to monitor Kubernetes API server that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes API server by HTTP - collects metrics by HTTP agent from API server /metrics endpoint.
Requirements
Zabbix version: 7.2 and higher.
Tested versions
This template has been tested on:
- Kubernetes API server 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.API.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. Some metrics may not be collected depending on your Kubernetes API server instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.SERVER.URL} | Kubernetes API server metrics endpoint URL. |
https://localhost:6443/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.API.CERT.EXPIRATION} | Number of days for alert of client certificate used for trigger. |
7 |
| {$KUBE.API.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.API.HTTP.SERVER.ERROR} | Maximum number of HTTP server requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get API instance metrics | Get raw metrics from API instance /metrics endpoint. |
HTTP agent | kubernetes.api.get_metrics Preprocessing
|
| Audit events, total | Accumulated number audit events generated and sent to the audit backend. |
Dependent item | kubernetes.api.audit_event_total Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.api.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.api.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.api.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.api.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.api.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.api.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.api.max_fds Preprocessing
|
| gRPCs client started, rate | Total number of RPCs started per second. |
Dependent item | kubernetes.api.grpc_client_started.rate Preprocessing
|
| gRPCs messages received, rate | Total number of gRPC stream messages received per second. |
Dependent item | kubernetes.api.grpc_client_msg_received.rate Preprocessing
|
| gRPCs messages sent, rate | Total number of gRPC stream messages sent per second. |
Dependent item | kubernetes.api.grpc_client_msg_sent.rate Preprocessing
|
| Request terminations, rate | Number of requests which apiserver terminated in self-defense per second. |
Dependent item | kubernetes.api.apiserver_request_terminations Preprocessing
|
| TLS handshake errors, rate | Number of requests dropped with 'TLS handshake error from' error per second. |
Dependent item | kubernetes.api.apiserver_tls_handshake_errors_total.rate Preprocessing
|
| API server requests: 5xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_500.rate Preprocessing
|
| API server requests: 4xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_400.rate Preprocessing
|
| API server requests: 3xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_300.rate Preprocessing
|
| API server requests: 0 | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_0.rate Preprocessing
|
| API server requests: 2xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_200.rate Preprocessing
|
| HTTP requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_500.rate Preprocessing
|
| HTTP requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_400.rate Preprocessing
|
| HTTP requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_300.rate Preprocessing
|
| HTTP requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_200.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API server: Too many server errors | "Kubernetes API server is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.apiserver_request_total_500.rate,5m)>{$KUBE.API.HTTP.SERVER.ERROR} |
Warning | |
| Kubernetes API server: Too many client errors | "Kubernetes API client is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.rest_client_requests_total_500.rate,5m)>{$KUBE.API.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running requests | Discovery of long-running requests by verb, resource and scope. |
Dependent item | kubernetes.api.longrunning_gauge.discovery Preprocessing
|
Item prototypes for Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running ["{#VERB}"] requests ["{#RESOURCE}"]: {#SCOPE} | Gauge of all active long-running apiserver requests broken out by verb, resource and scope. Not all requests are tracked this way. |
Dependent item | kubernetes.api.longrunning_gauge["{#RESOURCE}","{#SCOPE}","{#VERB}"] Preprocessing
|
LLD rule Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Request duration histogram | Discovery raw data and percentile items of request duration. |
Dependent item | kubernetes.api.requests_bucket.discovery Preprocessing
|
Item prototypes for Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#VERB}"] Requests bucket: {#LE} | Response latency distribution in seconds for each verb. |
Dependent item | kubernetes.api.request_duration_seconds_bucket[{#LE},"{#VERB}"] Preprocessing
|
| ["{#VERB}"] Requests, p90 | 90 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p90["{#VERB}"] |
| ["{#VERB}"] Requests, p95 | 95 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p95["{#VERB}"] |
| ["{#VERB}"] Requests, p99 | 99 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p99["{#VERB}"] |
| ["{#VERB}"] Requests, p50 | 50 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p50["{#VERB}"] |
LLD rule Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests inflight discovery | Discovery requests inflight by kind. |
Dependent item | kubernetes.api.inflight_requests.discovery Preprocessing
|
Item prototypes for Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests current: {#KIND} | Maximal number of currently used inflight request limit of this apiserver per request kind in last second. |
Dependent item | kubernetes.api.current_inflight_requests["{#KIND}"] Preprocessing
|
LLD rule gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPC completed requests discovery | Discovery grpc completed requests by grpc code. |
Dependent item | kubernetes.api.grpc_client_handled.discovery Preprocessing
|
Item prototypes for gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPCs completed: {#GRPC_CODE}, rate | Total number of RPCs completed by the client regardless of success or failure per second. |
Dependent item | kubernetes.api.grpc_client_handled_total.rate["{#GRPC_CODE}"] Preprocessing
|
LLD rule Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts discovery | Discovery authentication attempts by result. |
Dependent item | kubernetes.api.authentication_attempts.discovery Preprocessing
|
Item prototypes for Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts: {#RESULT}, rate | Authentication attempts by result per second. |
Dependent item | kubernetes.api.authentication_attempts.rate["{#RESULT}"] Preprocessing
|
LLD rule Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication requests discovery | Discovery authentication attempts by name. |
Dependent item | kubernetes.api.authenticated_user_requests.discovery Preprocessing
|
Item prototypes for Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authenticated requests: {#NAME}, rate | Counter of authenticated requests broken out by username per second. |
Dependent item | kubernetes.api.authenticated_user_requests.rate["{#NAME}"] Preprocessing
|
LLD rule Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers metrics discovery | Discovery watchers by kind. |
Dependent item | kubernetes.api.apiserver_registered_watchers.discovery Preprocessing
|
Item prototypes for Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers: {#KIND} | Number of currently registered watchers for a given resource. |
Dependent item | kubernetes.api.apiserver_registered_watchers["{#KIND}"] Preprocessing
|
LLD rule Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Etcd objects metrics discovery | Discovery etcd objects by resource. |
Dependent item | kubernetes.api.etcd_object_counts.discovery Preprocessing
|
Item prototypes for Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| etcd objects: {#RESOURCE} | Number of stored objects at the time of last check split by kind. |
Dependent item | kubernetes.api.etcd_object_counts["{#RESOURCE}"] Preprocessing
|
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Discovery workqueue metrics by name. |
Dependent item | kubernetes.api.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"] Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.api.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"] Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.api.workqueue_adds_total.rate["{#NAME}"] Preprocessing
|
LLD rule Client certificate expiration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Client certificate expiration histogram | Discovery raw data of client certificate expiration |
Dependent item | kubernetes.api.certificate_expiration.discovery Preprocessing
|
Item prototypes for Client certificate expiration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Certificate expiration seconds bucket, {#LE} | Distribution of the remaining lifetime on the certificate used to authenticate a request. |
Dependent item | kubernetes.api.client_certificate_expiration_seconds_bucket[{#LE}] Preprocessing
|
| Client certificate expiration, p1 | 1 percentile of the remaining lifetime on the certificate used to authenticate a request. |
Calculated | kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}] |
Trigger prototypes for Client certificate expiration histogram
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API server: Kubernetes client certificate is expiring | A client certificate used to authenticate to the apiserver is expiring in {$KUBE.API.CERT.EXPIRATION} days. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < {$KUBE.API.CERT.EXPIRATION}*24*60*60 |
Warning | Depends on:
|
| Kubernetes API server: Kubernetes client certificate expires soon | A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < 24*60*60 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes API server by HTTP
Overview
The template to monitor Kubernetes API server that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes API server by HTTP - collects metrics by HTTP agent from API server /metrics endpoint.
Requirements
Zabbix version: 7.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes API server 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.API.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: Some metrics may not be collected depending on your Kubernetes API server instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.SERVER.URL} | Kubernetes API server metrics endpoint URL. |
https://localhost:6443/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.API.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.API.HTTP.SERVER.ERROR} | Maximum number of HTTP server requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get API instance metrics | Get raw metrics from API instance /metrics endpoint. |
HTTP agent | kubernetes.api.get_metrics Preprocessing
|
| Audit events, total | Accumulated number audit events generated and sent to the audit backend. |
Dependent item | kubernetes.api.audit_event_total Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.api.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.api.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.api.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.api.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.api.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.api.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.api.max_fds Preprocessing
|
| gRPCs client started, rate | Total number of RPCs started per second. |
Dependent item | kubernetes.api.grpc_client_started.rate Preprocessing
|
| gRPCs messages received, rate | Total number of gRPC stream messages received per second. |
Dependent item | kubernetes.api.grpc_client_msg_received.rate Preprocessing
|
| gRPCs messages sent, rate | Total number of gRPC stream messages sent per second. |
Dependent item | kubernetes.api.grpc_client_msg_sent.rate Preprocessing
|
| Request terminations, rate | Number of requests which apiserver terminated in self-defense per second. |
Dependent item | kubernetes.api.apiserver_request_terminations Preprocessing
|
| TLS handshake errors, rate | Number of requests dropped with 'TLS handshake error from' error per second. |
Dependent item | kubernetes.api.apiserver_tls_handshake_errors_total.rate Preprocessing
|
| API server requests: 5xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_500.rate Preprocessing
|
| API server requests: 4xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_400.rate Preprocessing
|
| API server requests: 3xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_300.rate Preprocessing
|
| API server requests: 0 | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_0.rate Preprocessing
|
| API server requests: 2xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_200.rate Preprocessing
|
| HTTP requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_500.rate Preprocessing
|
| HTTP requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_400.rate Preprocessing
|
| HTTP requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_300.rate Preprocessing
|
| HTTP requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_200.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API server: Too many server errors | "Kubernetes API server is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.apiserver_request_total_500.rate,5m)>{$KUBE.API.HTTP.SERVER.ERROR} |
Warning | |
| Kubernetes API server: Too many client errors | "Kubernetes API client is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.rest_client_requests_total_500.rate,5m)>{$KUBE.API.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running requests | Discovery of long-running requests by verb, resource and scope. |
Dependent item | kubernetes.api.longrunning_gauge.discovery Preprocessing
|
Item prototypes for Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running ["{#VERB}"] requests ["{#RESOURCE}"]: {#SCOPE} | Gauge of all active long-running apiserver requests broken out by verb, resource and scope. Not all requests are tracked this way. |
Dependent item | kubernetes.api.longrunning_gauge["{#RESOURCE}","{#SCOPE}","{#VERB}"] Preprocessing
|
LLD rule Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Request duration histogram | Discovery raw data and percentile items of request duration. |
Dependent item | kubernetes.api.requests_bucket.discovery Preprocessing
|
Item prototypes for Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#VERB}"] Requests bucket: {#LE} | Response latency distribution in seconds for each verb. |
Dependent item | kubernetes.api.request_duration_seconds_bucket[{#LE},"{#VERB}"] Preprocessing
|
| ["{#VERB}"] Requests, p90 | 90 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p90["{#VERB}"] |
| ["{#VERB}"] Requests, p95 | 95 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p95["{#VERB}"] |
| ["{#VERB}"] Requests, p99 | 99 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p99["{#VERB}"] |
| ["{#VERB}"] Requests, p50 | 50 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p50["{#VERB}"] |
LLD rule Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests inflight discovery | Discovery requests inflight by kind. |
Dependent item | kubernetes.api.inflight_requests.discovery Preprocessing
|
Item prototypes for Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests current: {#KIND} | Maximal number of currently used inflight request limit of this apiserver per request kind in last second. |
Dependent item | kubernetes.api.current_inflight_requests["{#KIND}"] Preprocessing
|
LLD rule gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPC completed requests discovery | Discovery grpc completed requests by grpc code. |
Dependent item | kubernetes.api.grpc_client_handled.discovery Preprocessing
|
Item prototypes for gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPCs completed: {#GRPC_CODE}, rate | Total number of RPCs completed by the client regardless of success or failure per second. |
Dependent item | kubernetes.api.grpc_client_handled_total.rate["{#GRPC_CODE}"] Preprocessing
|
LLD rule Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts discovery | Discovery authentication attempts by result. |
Dependent item | kubernetes.api.authentication_attempts.discovery Preprocessing
|
Item prototypes for Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts: {#RESULT}, rate | Authentication attempts by result per second. |
Dependent item | kubernetes.api.authentication_attempts.rate["{#RESULT}"] Preprocessing
|
LLD rule Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication requests discovery | Discovery authentication attempts by name. |
Dependent item | kubernetes.api.authenticated_user_requests.discovery Preprocessing
|
Item prototypes for Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authenticated requests: {#NAME}, rate | Counter of authenticated requests broken out by username per second. |
Dependent item | kubernetes.api.authenticated_user_requests.rate["{#NAME}"] Preprocessing
|
LLD rule Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers metrics discovery | Discovery watchers by kind. |
Dependent item | kubernetes.api.apiserver_registered_watchers.discovery Preprocessing
|
Item prototypes for Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers: {#KIND} | Number of currently registered watchers for a given resource. |
Dependent item | kubernetes.api.apiserver_registered_watchers["{#KIND}"] Preprocessing
|
LLD rule Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Etcd objects metrics discovery | Discovery etcd objects by resource. |
Dependent item | kubernetes.api.etcd_object_counts.discovery Preprocessing
|
Item prototypes for Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| etcd objects: {#RESOURCE} | Number of stored objects at the time of last check split by kind. |
Dependent item | kubernetes.api.etcd_object_counts["{#RESOURCE}"] Preprocessing
|
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Discovery workqueue metrics by name. |
Dependent item | kubernetes.api.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"] Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.api.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"] Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.api.workqueue_adds_total.rate["{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes API server by HTTP
Overview
The template to monitor Kubernetes API server that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes API server by HTTP - collects metrics by HTTP agent from API server /metrics endpoint.
Requirements
Zabbix version: 6.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes API server 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.API.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. Some metrics may not be collected depending on your Kubernetes API server instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.SERVER.URL} | Kubernetes API server metrics endpoint URL. |
https://localhost:6443/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.API.CERT.EXPIRATION} | Number of days for alert of client certificate used for trigger. |
7 |
| {$KUBE.API.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.API.HTTP.SERVER.ERROR} | Maximum number of HTTP server requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Get API instance metrics | Get raw metrics from API instance /metrics endpoint. |
HTTP agent | kubernetes.api.get_metrics Preprocessing
|
| Kubernetes API: Audit events, total | Accumulated number audit events generated and sent to the audit backend. |
Dependent item | kubernetes.api.audit_event_total Preprocessing
|
| Kubernetes API: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.api.process_virtual_memory_bytes Preprocessing
|
| Kubernetes API: Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.api.process_resident_memory_bytes Preprocessing
|
| Kubernetes API: CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.api.cpu.util Preprocessing
|
| Kubernetes API: Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.api.go_goroutines Preprocessing
|
| Kubernetes API: Go threads | Number of OS threads created. |
Dependent item | kubernetes.api.go_threads Preprocessing
|
| Kubernetes API: Fds open | Number of open file descriptors. |
Dependent item | kubernetes.api.open_fds Preprocessing
|
| Kubernetes API: Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.api.max_fds Preprocessing
|
| Kubernetes API: gRPCs client started, rate | Total number of RPCs started per second. |
Dependent item | kubernetes.api.grpc_client_started.rate Preprocessing
|
| Kubernetes API: gRPCs messages received, rate | Total number of gRPC stream messages received per second. |
Dependent item | kubernetes.api.grpc_client_msg_received.rate Preprocessing
|
| Kubernetes API: gRPCs messages sent, rate | Total number of gRPC stream messages sent per second. |
Dependent item | kubernetes.api.grpc_client_msg_sent.rate Preprocessing
|
| Kubernetes API: Request terminations, rate | Number of requests which apiserver terminated in self-defense per second. |
Dependent item | kubernetes.api.apiserver_request_terminations Preprocessing
|
| Kubernetes API: TLS handshake errors, rate | Number of requests dropped with 'TLS handshake error from' error per second. |
Dependent item | kubernetes.api.apiserver_tls_handshake_errors_total.rate Preprocessing
|
| Kubernetes API: API server requests: 5xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_500.rate Preprocessing
|
| Kubernetes API: API server requests: 4xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_400.rate Preprocessing
|
| Kubernetes API: API server requests: 3xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_300.rate Preprocessing
|
| Kubernetes API: API server requests: 0 | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_0.rate Preprocessing
|
| Kubernetes API: API server requests: 2xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_200.rate Preprocessing
|
| Kubernetes API: HTTP requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_500.rate Preprocessing
|
| Kubernetes API: HTTP requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_400.rate Preprocessing
|
| Kubernetes API: HTTP requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_300.rate Preprocessing
|
| Kubernetes API: HTTP requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_200.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API: Too many server errors | "Kubernetes API server is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.apiserver_request_total_500.rate,5m)>{$KUBE.API.HTTP.SERVER.ERROR} |
Warning | |
| Kubernetes API: Too many client errors | "Kubernetes API client is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.rest_client_requests_total_500.rate,5m)>{$KUBE.API.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running requests | Discovery of long-running requests by verb, resource and scope. |
Dependent item | kubernetes.api.longrunning_gauge.discovery Preprocessing
|
Item prototypes for Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Long-running ["{#VERB}"] requests ["{#RESOURCE}"]: {#SCOPE} | Gauge of all active long-running apiserver requests broken out by verb, resource and scope. Not all requests are tracked this way. |
Dependent item | kubernetes.api.longrunning_gauge["{#RESOURCE}","{#SCOPE}","{#VERB}"] Preprocessing
|
LLD rule Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Request duration histogram | Discovery raw data and percentile items of request duration. |
Dependent item | kubernetes.api.requests_bucket.discovery Preprocessing
|
Item prototypes for Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: ["{#VERB}"] Requests bucket: {#LE} | Response latency distribution in seconds for each verb. |
Dependent item | kubernetes.api.request_duration_seconds_bucket[{#LE},"{#VERB}"] Preprocessing
|
| Kubernetes API: ["{#VERB}"] Requests, p90 | 90 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p90["{#VERB}"] |
| Kubernetes API: ["{#VERB}"] Requests, p95 | 95 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p95["{#VERB}"] |
| Kubernetes API: ["{#VERB}"] Requests, p99 | 99 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p99["{#VERB}"] |
| Kubernetes API: ["{#VERB}"] Requests, p50 | 50 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p50["{#VERB}"] |
LLD rule Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests inflight discovery | Discovery requests inflight by kind. |
Dependent item | kubernetes.api.inflight_requests.discovery Preprocessing
|
Item prototypes for Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Requests current: {#KIND} | Maximal number of currently used inflight request limit of this apiserver per request kind in last second. |
Dependent item | kubernetes.api.current_inflight_requests["{#KIND}"] Preprocessing
|
LLD rule gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPC completed requests discovery | Discovery grpc completed requests by grpc code. |
Dependent item | kubernetes.api.grpc_client_handled.discovery Preprocessing
|
Item prototypes for gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: gRPCs completed: {#GRPC_CODE}, rate | Total number of RPCs completed by the client regardless of success or failure per second. |
Dependent item | kubernetes.api.grpc_client_handled_total.rate["{#GRPC_CODE}"] Preprocessing
|
LLD rule Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts discovery | Discovery authentication attempts by result. |
Dependent item | kubernetes.api.authentication_attempts.discovery Preprocessing
|
Item prototypes for Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Authentication attempts: {#RESULT}, rate | Authentication attempts by result per second. |
Dependent item | kubernetes.api.authentication_attempts.rate["{#RESULT}"] Preprocessing
|
LLD rule Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication requests discovery | Discovery authentication attempts by name. |
Dependent item | kubernetes.api.authenticated_user_requests.discovery Preprocessing
|
Item prototypes for Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Authenticated requests: {#NAME}, rate | Counter of authenticated requests broken out by username per second. |
Dependent item | kubernetes.api.authenticated_user_requests.rate["{#NAME}"] Preprocessing
|
LLD rule Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers metrics discovery | Discovery watchers by kind. |
Dependent item | kubernetes.api.apiserver_registered_watchers.discovery Preprocessing
|
Item prototypes for Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Watchers: {#KIND} | Number of currently registered watchers for a given resource. |
Dependent item | kubernetes.api.apiserver_registered_watchers["{#KIND}"] Preprocessing
|
LLD rule Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Etcd objects metrics discovery | Discovery etcd objects by resource. |
Dependent item | kubernetes.api.etcd_object_counts.discovery Preprocessing
|
Item prototypes for Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: etcd objects: {#RESOURCE} | Number of stored objects at the time of last check split by kind. |
Dependent item | kubernetes.api.etcd_object_counts["{#RESOURCE}"] Preprocessing
|
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Discovery workqueue metrics by name. |
Dependent item | kubernetes.api.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: ["{#NAME}"] Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.api.workqueue_depth["{#NAME}"] Preprocessing
|
| Kubernetes API: ["{#NAME}"] Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.api.workqueue_adds_total.rate["{#NAME}"] Preprocessing
|
LLD rule Client certificate expiration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Client certificate expiration histogram | Discovery raw data of client certificate expiration |
Dependent item | kubernetes.api.certificate_expiration.discovery Preprocessing
|
Item prototypes for Client certificate expiration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Certificate expiration seconds bucket, {#LE} | Distribution of the remaining lifetime on the certificate used to authenticate a request. |
Dependent item | kubernetes.api.client_certificate_expiration_seconds_bucket[{#LE}] Preprocessing
|
| Kubernetes API: Client certificate expiration, p1 | 1 percentile of the remaining lifetime on the certificate used to authenticate a request. |
Calculated | kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}] |
Trigger prototypes for Client certificate expiration histogram
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API: Kubernetes client certificate is expiring | A client certificate used to authenticate to the apiserver is expiring in {$KUBE.API.CERT.EXPIRATION} days. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < {$KUBE.API.CERT.EXPIRATION}*24*60*60 |
Warning | Depends on:
|
| Kubernetes API: Kubernetes client certificate expires soon | A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < 24*60*60 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes API server by HTTP
Overview
For Zabbix version: 6.2 and higher
The template to monitor InfluxDB by Zabbix that works without any external scripts.
Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes API server by HTTP — collects metrics by HTTP agent from API server /metrics endpoint.
This template was tested on:
- Kubernetes API server, version 1.19.10
Setup
See Zabbix template operation for basic instructions.
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.API.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values. NOTE. Some metrics may not be collected depending on your Kubernetes API server instance version and configuration.
Zabbix configuration
No specific Zabbix configuration is required.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.CERT.EXPIRATION} | Number of days for alert of client certificate used for trigger |
7 |
| {$KUBE.API.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger |
2 |
| {$KUBE.API.HTTP.SERVER.ERROR} | Maximum number of HTTP client requests failures used for trigger |
2 |
| {$KUBE.API.SERVER.URL} | instance URL |
http://localhost:8086/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token |
`` |
Template links
There are no template links in this template.
Discovery rules
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts discovery | Discovery authentication attempts by result. |
DEPENDENT | kubernetes.api.authentication_attempts.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
| Authentication requests discovery | Discovery authentication attempts by name. |
DEPENDENT | kubernetes.api.authenticated_user_requests.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
| Client certificate expiration histogram | Discovery raw data of client certificate expiration |
DEPENDENT | kubernetes.api.certificate_expiration.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: Overrides: bucket item total item |
| Etcd objects metrics discovery | Discovery etcd objects by resource. |
DEPENDENT | kubernetes.api.etcd_object_counts.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
| gRPC completed requests discovery | Discovery grpc completed requests by grpc code. |
DEPENDENT | kubernetes.api.grpc_client_handled.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
| Long-running requests | Discovery of long-running requests by verb, resource and scope. |
DEPENDENT | kubernetes.api.longrunning_gauge.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
| Request duration histogram | Discovery raw data and percentile items of request duration. |
DEPENDENT | kubernetes.api.requests_bucket.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: Overrides: bucket item total item |
| Requests inflight discovery | Discovery requests inflight by kind. |
DEPENDENT | kubernetes.api.inflight_requests.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
| Watchers metrics discovery | Discovery watchers by kind. |
DEPENDENT | kubernetes.api.apiserver_registered_watchers.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
| Workqueue metrics discovery | Discovery workqueue metrics by name. |
DEPENDENT | kubernetes.api.workqueue.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
Items collected
| Group | Name | Description | Type | Key and additional info |
|---|---|---|---|---|
| Kubernetes API | Kubernetes API: Audit events, total | Accumulated number audit events generated and sent to the audit backend. |
DEPENDENT | kubernetes.api.audit_event_total Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: Virtual memory, bytes | Virtual memory size in bytes. |
DEPENDENT | kubernetes.api.process_virtual_memory_bytes Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: Resident memory, bytes | Resident memory size in bytes. |
DEPENDENT | kubernetes.api.process_resident_memory_bytes Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: CPU | Total user and system CPU usage ratio. |
DEPENDENT | kubernetes.api.cpu.util Preprocessing: - PROMETHEUS_PATTERN: - CHANGE_PER_SECOND - MULTIPLIER: |
| Kubernetes API | Kubernetes API: Goroutines | Number of goroutines that currently exist. |
DEPENDENT | kubernetes.api.go_goroutines Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: Go threads | Number of OS threads created. |
DEPENDENT | kubernetes.api.go_threads Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: Fds open | Number of open file descriptors. |
DEPENDENT | kubernetes.api.open_fds Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: Fds max | Maximum allowed open file descriptors. |
DEPENDENT | kubernetes.api.max_fds Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: gRPCs client started, rate | Total number of RPCs started per second. |
DEPENDENT | kubernetes.api.grpc_client_started.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: gRPCs messages ressived, rate | Total number of gRPC stream messages received per second. |
DEPENDENT | kubernetes.api.grpc_client_msg_received.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: gRPCs messages sent, rate | Total number of gRPC stream messages sent per second. |
DEPENDENT | kubernetes.api.grpc_client_msg_sent.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: Request terminations, rate | Number of requests which apiserver terminated in self-defense per second. |
DEPENDENT | kubernetes.api.apiserver_request_terminations Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: TLS handshake errors, rate | Number of requests dropped with 'TLS handshake error from' error per second. |
DEPENDENT | kubernetes.api.apiserver_tls_handshake_errors_total.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: API server requests: 5xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
DEPENDENT | kubernetes.api.apiserver_request_total_500.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: API server requests: 4xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
DEPENDENT | kubernetes.api.apiserver_request_total_400.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: API server requests: 3xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
DEPENDENT | kubernetes.api.apiserver_request_total_300.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: API server requests: 0 | Counter of apiserver requests broken out for each HTTP response code. |
DEPENDENT | kubernetes.api.apiserver_request_total_0.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: API server requests: 2xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
DEPENDENT | kubernetes.api.apiserver_request_total_200.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: HTTP requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
DEPENDENT | kubernetes.api.rest_client_requests_total_500.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: HTTP requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
DEPENDENT | kubernetes.api.rest_client_requests_total_400.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: HTTP requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
DEPENDENT | kubernetes.api.rest_client_requests_total_300.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: HTTP requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
DEPENDENT | kubernetes.api.rest_client_requests_total_200.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: Long-running ["{#VERB}"] requests ["{#RESOURCE}"]: {#SCOPE} | Gauge of all active long-running apiserver requests broken out by verb, resource and scope. Not all requests are tracked this way. |
DEPENDENT | kubernetes.api.longrunning_gauge["{#RESOURCE}","{#SCOPE}","{#VERB}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: ["{#VERB}"] Requests bucket: {#LE} | Response latency distribution in seconds for each verb. |
DEPENDENT | kubernetes.api.request_duration_seconds_bucket[{#LE},"{#VERB}"] Preprocessing: - PROMETHEUS_PATTERN: |
| Kubernetes API | Kubernetes API: ["{#VERB}"] Requests, p90 | 90 percentile of response latency distribution in seconds for each verb. |
CALCULATED | kubernetes.api.request_duration_seconds_p90["{#VERB}"] Expression: bucket_percentile(//kubernetes.api.request_duration_seconds_bucket[*,"{#VERB}"],5m,90) |
| Kubernetes API | Kubernetes API: ["{#VERB}"] Requests, p95 | 95 percentile of response latency distribution in seconds for each verb. |
CALCULATED | kubernetes.api.request_duration_seconds_p95["{#VERB}"] Expression: bucket_percentile(//kubernetes.api.request_duration_seconds_bucket[*,"{#VERB}"],5m,95) |
| Kubernetes API | Kubernetes API: ["{#VERB}"] Requests, p99 | 99 percentile of response latency distribution in seconds for each verb. |
CALCULATED | kubernetes.api.request_duration_seconds_p99["{#VERB}"] Expression: bucket_percentile(//kubernetes.api.request_duration_seconds_bucket[*,"{#VERB}"],5m,99) |
| Kubernetes API | Kubernetes API: ["{#VERB}"] Requests, p50 | 50 percentile of response latency distribution in seconds for each verb. |
CALCULATED | kubernetes.api.request_duration_seconds_p50["{#VERB}"] Expression: bucket_percentile(//kubernetes.api.request_duration_seconds_bucket[*,"{#VERB}"],5m,50) |
| Kubernetes API | Kubernetes API: Requests current: {#KIND} | Maximal number of currently used inflight request limit of this apiserver per request kind in last second. |
DEPENDENT | kubernetes.api.current_inflight_requests["{#KIND}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: gRPCs completed: {#GRPC_CODE}, rate | Total number of RPCs completed by the client regardless of success or failure per second. |
DEPENDENT | kubernetes.api.grpc_client_handled_total.rate["{#GRPC_CODE}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: Authentication attempts: {#RESULT}, rate | Authentication attempts by result per second. |
DEPENDENT | kubernetes.api.authentication_attempts.rate["{#RESULT}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: Authenticated requests: {#NAME}, rate | Counter of authenticated requests broken out by username per second. |
DEPENDENT | kubernetes.api.authenticated_user_requests.rate["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: Watchers: {#KIND} | Number of currently registered watchers for a given resource. |
DEPENDENT | kubernetes.api.apiserver_registered_watchers["{#KIND}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: etcd objects: {#RESOURCE} | Number of stored objects at the time of last check split by kind. |
DEPENDENT | kubernetes.api.etcd_object_counts["{#RESOURCE}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: ["{#NAME}"] Workqueue depth | Current depth of workqueue. |
DEPENDENT | kubernetes.api.workqueue_depth["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: ["{#NAME}"] Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
DEPENDENT | kubernetes.api.workqueue_adds_total.rate["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes API | Kubernetes API: Certificate expiration seconds bucket, {#LE} | Distribution of the remaining lifetime on the certificate used to authenticate a request. |
DEPENDENT | kubernetes.api.client_certificate_expiration_seconds_bucket[{#LE}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes API | Kubernetes API: Client certificate expiration, p1 | 1 percentile of the remaining lifetime on the certificate used to authenticate a request. |
CALCULATED | kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.api.client_certificate_expiration_seconds_bucket[*],5m,1) |
| Zabbix raw items | Kubernetes API: Get API instance metrics | Get raw metrics from API instance /metrics endpoint. |
HTTP_AGENT | kubernetes.api.get_metrics Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: |
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API: Too many server errors | "Kubernetes API server is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.apiserver_request_total_500.rate,5m)>{$KUBE.API.HTTP.SERVER.ERROR} |
WARNING | |
| Kubernetes API: Too many client errors | "Kubernetes API client is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.rest_client_requests_total_500.rate,5m)>{$KUBE.API.HTTP.CLIENT.ERROR} |
WARNING | |
| Kubernetes API: Kubernetes client certificate is expiring | A client certificate used to authenticate to the apiserver is expiring in {$KUBE.API.CERT.EXPIRATION} days. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < {$KUBE.API.CERT.EXPIRATION}*24*60*60 |
WARNING | Depends on: - Kubernetes API: Kubernetes client certificate expires soon |
| Kubernetes API: Kubernetes client certificate expires soon | A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < 24*60*60 |
WARNING |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template or ask for help with it at ZABBIX forums.
Kubernetes API server by HTTP
Overview
The template to monitor Kubernetes API server that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes API server by HTTP - collects metrics by HTTP agent from API server /metrics endpoint.
Requirements
Zabbix version: 6.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes API server 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.API.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. Some metrics may not be collected depending on your Kubernetes API server instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.SERVER.URL} | Kubernetes API server metrics endpoint URL. |
https://localhost:6443/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.API.CERT.EXPIRATION} | Number of days for alert of client certificate used for trigger. |
7 |
| {$KUBE.API.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.API.HTTP.SERVER.ERROR} | Maximum number of HTTP server requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Get API instance metrics | Get raw metrics from API instance /metrics endpoint. |
HTTP agent | kubernetes.api.get_metrics Preprocessing
|
| Kubernetes API: Audit events, total | Accumulated number audit events generated and sent to the audit backend. |
Dependent item | kubernetes.api.audit_event_total Preprocessing
|
| Kubernetes API: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.api.process_virtual_memory_bytes Preprocessing
|
| Kubernetes API: Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.api.process_resident_memory_bytes Preprocessing
|
| Kubernetes API: CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.api.cpu.util Preprocessing
|
| Kubernetes API: Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.api.go_goroutines Preprocessing
|
| Kubernetes API: Go threads | Number of OS threads created. |
Dependent item | kubernetes.api.go_threads Preprocessing
|
| Kubernetes API: Fds open | Number of open file descriptors. |
Dependent item | kubernetes.api.open_fds Preprocessing
|
| Kubernetes API: Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.api.max_fds Preprocessing
|
| Kubernetes API: gRPCs client started, rate | Total number of RPCs started per second. |
Dependent item | kubernetes.api.grpc_client_started.rate Preprocessing
|
| Kubernetes API: gRPCs messages received, rate | Total number of gRPC stream messages received per second. |
Dependent item | kubernetes.api.grpc_client_msg_received.rate Preprocessing
|
| Kubernetes API: gRPCs messages sent, rate | Total number of gRPC stream messages sent per second. |
Dependent item | kubernetes.api.grpc_client_msg_sent.rate Preprocessing
|
| Kubernetes API: Request terminations, rate | Number of requests which apiserver terminated in self-defense per second. |
Dependent item | kubernetes.api.apiserver_request_terminations Preprocessing
|
| Kubernetes API: TLS handshake errors, rate | Number of requests dropped with 'TLS handshake error from' error per second. |
Dependent item | kubernetes.api.apiserver_tls_handshake_errors_total.rate Preprocessing
|
| Kubernetes API: API server requests: 5xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_500.rate Preprocessing
|
| Kubernetes API: API server requests: 4xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_400.rate Preprocessing
|
| Kubernetes API: API server requests: 3xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_300.rate Preprocessing
|
| Kubernetes API: API server requests: 0 | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_0.rate Preprocessing
|
| Kubernetes API: API server requests: 2xx, rate | Counter of apiserver requests broken out for each HTTP response code. |
Dependent item | kubernetes.api.apiserver_request_total_200.rate Preprocessing
|
| Kubernetes API: HTTP requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_500.rate Preprocessing
|
| Kubernetes API: HTTP requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_400.rate Preprocessing
|
| Kubernetes API: HTTP requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_300.rate Preprocessing
|
| Kubernetes API: HTTP requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.api.rest_client_requests_total_200.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API: Too many server errors | "Kubernetes API server is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.apiserver_request_total_500.rate,5m)>{$KUBE.API.HTTP.SERVER.ERROR} |
Warning | |
| Kubernetes API: Too many client errors | "Kubernetes API client is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes API server by HTTP/kubernetes.api.rest_client_requests_total_500.rate,5m)>{$KUBE.API.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Long-running requests | Discovery of long-running requests by verb, resource and scope. |
Dependent item | kubernetes.api.longrunning_gauge.discovery Preprocessing
|
Item prototypes for Long-running requests
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Long-running ["{#VERB}"] requests ["{#RESOURCE}"]: {#SCOPE} | Gauge of all active long-running apiserver requests broken out by verb, resource and scope. Not all requests are tracked this way. |
Dependent item | kubernetes.api.longrunning_gauge["{#RESOURCE}","{#SCOPE}","{#VERB}"] Preprocessing
|
LLD rule Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Request duration histogram | Discovery raw data and percentile items of request duration. |
Dependent item | kubernetes.api.requests_bucket.discovery Preprocessing
|
Item prototypes for Request duration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: ["{#VERB}"] Requests bucket: {#LE} | Response latency distribution in seconds for each verb. |
Dependent item | kubernetes.api.request_duration_seconds_bucket[{#LE},"{#VERB}"] Preprocessing
|
| Kubernetes API: ["{#VERB}"] Requests, p90 | 90 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p90["{#VERB}"] |
| Kubernetes API: ["{#VERB}"] Requests, p95 | 95 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p95["{#VERB}"] |
| Kubernetes API: ["{#VERB}"] Requests, p99 | 99 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p99["{#VERB}"] |
| Kubernetes API: ["{#VERB}"] Requests, p50 | 50 percentile of response latency distribution in seconds for each verb. |
Calculated | kubernetes.api.request_duration_seconds_p50["{#VERB}"] |
LLD rule Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Requests inflight discovery | Discovery requests inflight by kind. |
Dependent item | kubernetes.api.inflight_requests.discovery Preprocessing
|
Item prototypes for Requests inflight discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Requests current: {#KIND} | Maximal number of currently used inflight request limit of this apiserver per request kind in last second. |
Dependent item | kubernetes.api.current_inflight_requests["{#KIND}"] Preprocessing
|
LLD rule gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| gRPC completed requests discovery | Discovery grpc completed requests by grpc code. |
Dependent item | kubernetes.api.grpc_client_handled.discovery Preprocessing
|
Item prototypes for gRPC completed requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: gRPCs completed: {#GRPC_CODE}, rate | Total number of RPCs completed by the client regardless of success or failure per second. |
Dependent item | kubernetes.api.grpc_client_handled_total.rate["{#GRPC_CODE}"] Preprocessing
|
LLD rule Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication attempts discovery | Discovery authentication attempts by result. |
Dependent item | kubernetes.api.authentication_attempts.discovery Preprocessing
|
Item prototypes for Authentication attempts discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Authentication attempts: {#RESULT}, rate | Authentication attempts by result per second. |
Dependent item | kubernetes.api.authentication_attempts.rate["{#RESULT}"] Preprocessing
|
LLD rule Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Authentication requests discovery | Discovery authentication attempts by name. |
Dependent item | kubernetes.api.authenticated_user_requests.discovery Preprocessing
|
Item prototypes for Authentication requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Authenticated requests: {#NAME}, rate | Counter of authenticated requests broken out by username per second. |
Dependent item | kubernetes.api.authenticated_user_requests.rate["{#NAME}"] Preprocessing
|
LLD rule Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Watchers metrics discovery | Discovery watchers by kind. |
Dependent item | kubernetes.api.apiserver_registered_watchers.discovery Preprocessing
|
Item prototypes for Watchers metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Watchers: {#KIND} | Number of currently registered watchers for a given resource. |
Dependent item | kubernetes.api.apiserver_registered_watchers["{#KIND}"] Preprocessing
|
LLD rule Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Etcd objects metrics discovery | Discovery etcd objects by resource. |
Dependent item | kubernetes.api.etcd_object_counts.discovery Preprocessing
|
Item prototypes for Etcd objects metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: etcd objects: {#RESOURCE} | Number of stored objects at the time of last check split by kind. |
Dependent item | kubernetes.api.etcd_object_counts["{#RESOURCE}"] Preprocessing
|
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Discovery workqueue metrics by name. |
Dependent item | kubernetes.api.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: ["{#NAME}"] Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.api.workqueue_depth["{#NAME}"] Preprocessing
|
| Kubernetes API: ["{#NAME}"] Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.api.workqueue_adds_total.rate["{#NAME}"] Preprocessing
|
LLD rule Client certificate expiration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Client certificate expiration histogram | Discovery raw data of client certificate expiration |
Dependent item | kubernetes.api.certificate_expiration.discovery Preprocessing
|
Item prototypes for Client certificate expiration histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes API: Certificate expiration seconds bucket, {#LE} | Distribution of the remaining lifetime on the certificate used to authenticate a request. |
Dependent item | kubernetes.api.client_certificate_expiration_seconds_bucket[{#LE}] Preprocessing
|
| Kubernetes API: Client certificate expiration, p1 | 1 percentile of the remaining lifetime on the certificate used to authenticate a request. |
Calculated | kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}] |
Trigger prototypes for Client certificate expiration histogram
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes API: Kubernetes client certificate is expiring | A client certificate used to authenticate to the apiserver is expiring in {$KUBE.API.CERT.EXPIRATION} days. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < {$KUBE.API.CERT.EXPIRATION}*24*60*60 |
Warning | Depends on:
|
| Kubernetes API: Kubernetes client certificate expires soon | A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours. |
last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) > 0 and last(/Kubernetes API server by HTTP/kubernetes.api.client_certificate_expiration_p1[{#SINGLETON}]) < 24*60*60 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Controller manager by HTTP
Overview
The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP - collects metrics by HTTP agent from Controller manager /metrics endpoint.
Requirements
Zabbix version: 7.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes Controller manager 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.CONTROLLER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: You might need to set the --binding-address option for Controller Manager to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
Note: Some metrics may not be collected depending on your Kubernetes Controller manager instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token |
|
| {$KUBE.CONTROLLER.SERVER.URL} | Kubernetes Controller manager metrics endpoint URL. |
https://localhost:10257/metrics |
| {$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller: Get Controller metrics | Get raw metrics from Controller instance /metrics endpoint. |
HTTP agent | kubernetes.controller.get_metrics Preprocessing
|
| Leader election status | Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. |
Dependent item | kubernetes.controller.leader_election_master_status Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.controller.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.controller.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.controller.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.controller.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.controller.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.controller.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.controller.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_500.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Controller manager: Too many HTTP client errors | "Kubernetes Controller manager is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Controller manager by HTTP/kubernetes.controller.client_http_requests_500.rate,5m)>{$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Dependent item | kubernetes.controller.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"]: Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_adds_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.controller.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue unfinished work, sec | How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
Dependent item | kubernetes.controller.workqueue_unfinished_work_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue retries, rate | Total number of retries handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_retries_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue longest running processor, sec | How many seconds has the longest running processor for workqueue been running. |
Dependent item | kubernetes.controller.workqueue_longest_running_processor_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue work duration, p90 | 90 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p95 | 95 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p99 | 99 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, 50p | 50 percentiles of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p50["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p90 | 90 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p95 | 95 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p99 | 99 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, 50p | 50 percentile of how long in seconds an item stays in workqueue before being requested. If there are no requests for 5 minute, item value will be discarded. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p50["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue duration seconds bucket, {#LE} | How long in seconds processing an item from workqueue takes. |
Dependent item | kubernetes.controller.duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Queue duration seconds bucket, {#LE} | How long in seconds an item stays in workqueue before being requested. |
Dependent item | kubernetes.controller.queue_duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Controller manager by HTTP
Overview
The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP - collects metrics by HTTP agent from Controller manager /metrics endpoint.
Requirements
Zabbix version: 7.2 and higher.
Tested versions
This template has been tested on:
- Kubernetes Controller manager 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.CONTROLLER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. You might need to set the --binding-address option for Controller Manager to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
NOTE. Some metrics may not be collected depending on your Kubernetes Controller manager instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.CONTROLLER.SERVER.URL} | Kubernetes Controller manager metrics endpoint URL. |
https://localhost:10257/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token |
|
| {$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller: Get Controller metrics | Get raw metrics from Controller instance /metrics endpoint. |
HTTP agent | kubernetes.controller.get_metrics Preprocessing
|
| Leader election status | Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. |
Dependent item | kubernetes.controller.leader_election_master_status Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.controller.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.controller.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.controller.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.controller.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.controller.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.controller.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.controller.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_500.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Controller manager: Too many HTTP client errors | "Kubernetes Controller manager is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Controller manager by HTTP/kubernetes.controller.client_http_requests_500.rate,5m)>{$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Dependent item | kubernetes.controller.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"]: Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_adds_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.controller.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue unfinished work, sec | How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
Dependent item | kubernetes.controller.workqueue_unfinished_work_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue retries, rate | Total number of retries handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_retries_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue longest running processor, sec | How many seconds has the longest running processor for workqueue been running. |
Dependent item | kubernetes.controller.workqueue_longest_running_processor_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue work duration, p90 | 90 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p95 | 95 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p99 | 99 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, 50p | 50 percentiles of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p50["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p90 | 90 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p95 | 95 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p99 | 99 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, 50p | 50 percentile of how long in seconds an item stays in workqueue before being requested. If there are no requests for 5 minute, item value will be discarded. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p50["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue duration seconds bucket, {#LE} | How long in seconds processing an item from workqueue takes. |
Dependent item | kubernetes.controller.duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Queue duration seconds bucket, {#LE} | How long in seconds an item stays in workqueue before being requested. |
Dependent item | kubernetes.controller.queue_duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Controller manager by HTTP
Overview
The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP - collects metrics by HTTP agent from Controller manager /metrics endpoint.
Requirements
Zabbix version: 7.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes Controller manager 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.CONTROLLER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: You might need to set the --binding-address option for Controller Manager to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
Note: Some metrics may not be collected depending on your Kubernetes Controller manager instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.CONTROLLER.SERVER.URL} | Kubernetes Controller manager metrics endpoint URL. |
https://localhost:10257/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token |
|
| {$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller: Get Controller metrics | Get raw metrics from Controller instance /metrics endpoint. |
HTTP agent | kubernetes.controller.get_metrics Preprocessing
|
| Leader election status | Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. |
Dependent item | kubernetes.controller.leader_election_master_status Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.controller.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.controller.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.controller.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.controller.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.controller.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.controller.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.controller.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_500.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Controller manager: Too many HTTP client errors | "Kubernetes Controller manager is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Controller manager by HTTP/kubernetes.controller.client_http_requests_500.rate,5m)>{$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Dependent item | kubernetes.controller.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#NAME}"]: Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_adds_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.controller.workqueue_depth["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue unfinished work, sec | How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
Dependent item | kubernetes.controller.workqueue_unfinished_work_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue retries, rate | Total number of retries handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_retries_total["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue longest running processor, sec | How many seconds has the longest running processor for workqueue been running. |
Dependent item | kubernetes.controller.workqueue_longest_running_processor_seconds["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue work duration, p90 | 90 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p95 | 95 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, p99 | 99 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue work duration, 50p | 50 percentiles of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p50["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p90 | 90 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p90["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p95 | 95 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p95["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, p99 | 99 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p99["{#NAME}"] |
| ["{#NAME}"]: Workqueue queue duration, 50p | 50 percentile of how long in seconds an item stays in workqueue before being requested. If there are no requests for 5 minute, item value will be discarded. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p50["{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Workqueue duration seconds bucket, {#LE} | How long in seconds processing an item from workqueue takes. |
Dependent item | kubernetes.controller.duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
| ["{#NAME}"]: Queue duration seconds bucket, {#LE} | How long in seconds an item stays in workqueue before being requested. |
Dependent item | kubernetes.controller.queue_duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Controller manager by HTTP
Overview
The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP - collects metrics by HTTP agent from Controller manager /metrics endpoint.
Requirements
Zabbix version: 6.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes Controller manager 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.CONTROLLER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. You might need to set the --binding-address option for Controller Manager to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
NOTE. Some metrics may not be collected depending on your Kubernetes Controller manager instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.CONTROLLER.SERVER.URL} | Kubernetes Controller manager metrics endpoint URL. |
https://localhost:10257/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token |
|
| {$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller: Get Controller metrics | Get raw metrics from Controller instance /metrics endpoint. |
HTTP agent | kubernetes.controller.get_metrics Preprocessing
|
| Kubernetes Controller Manager: Leader election status | Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. |
Dependent item | kubernetes.controller.leader_election_master_status Preprocessing
|
| Kubernetes Controller Manager: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.controller.process_virtual_memory_bytes Preprocessing
|
| Kubernetes Controller Manager: Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.controller.process_resident_memory_bytes Preprocessing
|
| Kubernetes Controller Manager: CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.controller.cpu.util Preprocessing
|
| Kubernetes Controller Manager: Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.controller.go_goroutines Preprocessing
|
| Kubernetes Controller Manager: Go threads | Number of OS threads created. |
Dependent item | kubernetes.controller.go_threads Preprocessing
|
| Kubernetes Controller Manager: Fds open | Number of open file descriptors. |
Dependent item | kubernetes.controller.open_fds Preprocessing
|
| Kubernetes Controller Manager: Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.controller.max_fds Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_200.rate Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_300.rate Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_400.rate Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_500.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Controller Manager: Too many HTTP client errors | "Kubernetes Controller manager is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Controller manager by HTTP/kubernetes.controller.client_http_requests_500.rate,5m)>{$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Dependent item | kubernetes.controller.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_adds_total["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.controller.workqueue_depth["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue unfinished work, sec | How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
Dependent item | kubernetes.controller.workqueue_unfinished_work_seconds["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue retries, rate | Total number of retries handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_retries_total["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue longest running processor, sec | How many seconds has the longest running processor for workqueue been running. |
Dependent item | kubernetes.controller.workqueue_longest_running_processor_seconds["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p90 | 90 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p90["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p95 | 95 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p95["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p99 | 99 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p99["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, 50p | 50 percentiles of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p50["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p90 | 90 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p90["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p95 | 95 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p95["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p99 | 99 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p99["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, 50p | 50 percentile of how long in seconds an item stays in workqueue before being requested. If there are no requests for 5 minute, item value will be discarded. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p50["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue duration seconds bucket, {#LE} | How long in seconds processing an item from workqueue takes. |
Dependent item | kubernetes.controller.duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Queue duration seconds bucket, {#LE} | How long in seconds an item stays in workqueue before being requested. |
Dependent item | kubernetes.controller.queue_duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Controller manager by HTTP
Overview
For Zabbix version: 6.2 and higher
The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts.
Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP — collects metrics by HTTP agent from Controller manager /metrics endpoint.
This template was tested on:
- Kubernetes Controller manager, version 1.19.10
Setup
See Zabbix template operation for basic instructions.
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.CONTROLLER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values. NOTE. Some metrics may not be collected depending on your Kubernetes Controller manager instance version and configuration.
Zabbix configuration
No specific Zabbix configuration is required.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token |
`` |
| {$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger |
2 |
| {$KUBE.CONTROLLER.SERVER.URL} | Instance URL |
http://localhost:10252/metrics |
Template links
There are no template links in this template.
Discovery rules
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | DEPENDENT | kubernetes.controller.workqueue.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: Overrides: bucket item total item |
Items collected
| Group | Name | Description | Type | Key and additional info |
|---|---|---|---|---|
| Kubernetes Controller | Kubernetes Controller Manager: Leader election status | Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. |
DEPENDENT | kubernetes.controller.leader_election_master_status Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: Virtual memory, bytes | Virtual memory size in bytes. |
DEPENDENT | kubernetes.controller.process_virtual_memory_bytes Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: Resident memory, bytes | Resident memory size in bytes. |
DEPENDENT | kubernetes.controller.process_resident_memory_bytes Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: CPU | Total user and system CPU usage ratio. |
DEPENDENT | kubernetes.controller.cpu.util Preprocessing: - PROMETHEUS_PATTERN: - CHANGE_PER_SECOND - MULTIPLIER: |
| Kubernetes Controller | Kubernetes Controller Manager: Goroutines | Number of goroutines that currently exist. |
DEPENDENT | kubernetes.controller.go_goroutines Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: Go threads | Number of OS threads created. |
DEPENDENT | kubernetes.controller.go_threads Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: Fds open | Number of open file descriptors. |
DEPENDENT | kubernetes.controller.open_fds Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: Fds max | Maximum allowed open file descriptors. |
DEPENDENT | kubernetes.controller.max_fds Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
DEPENDENT | kubernetes.controller.client_http_requests_200.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Controller | Kubernetes Controller Manager: REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
DEPENDENT | kubernetes.controller.client_http_requests_300.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Controller | Kubernetes Controller Manager: REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
DEPENDENT | kubernetes.controller.client_http_requests_400.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Controller | Kubernetes Controller Manager: REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
DEPENDENT | kubernetes.controller.client_http_requests_500.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
DEPENDENT | kubernetes.controller.workqueue_adds_total["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue depth | Current depth of workqueue. |
DEPENDENT | kubernetes.controller.workqueue_depth["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue unfinished work, sec | How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
DEPENDENT | kubernetes.controller.workqueue_unfinished_work_seconds["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue retries, rate | Total number of retries handled by workqueue per second. |
DEPENDENT | kubernetes.controller.workqueue_retries_total["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue longest running processor, sec | How many seconds has the longest running processor for workqueue been running. |
DEPENDENT | kubernetes.controller.workqueue_longest_running_processor_seconds["{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p90 | 90 percentile of how long in seconds processing an item from workqueue takes, by queue. |
CALCULATED | kubernetes.controller.workqueue_work_duration_seconds_p90["{#NAME}"] Expression: bucket_percentile(//kubernetes.controller.duration_seconds_bucket[*,"{#NAME}"],5m,90) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p95 | 95 percentile of how long in seconds processing an item from workqueue takes, by queue. |
CALCULATED | kubernetes.controller.workqueue_work_duration_seconds_p95["{#NAME}"] Expression: bucket_percentile(//kubernetes.controller.duration_seconds_bucket[*,"{#NAME}"],5m,95) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p99 | 99 percentile of how long in seconds processing an item from workqueue takes, by queue. |
CALCULATED | kubernetes.controller.workqueue_work_duration_seconds_p99["{#NAME}"] Expression: bucket_percentile(//kubernetes.controller.duration_seconds_bucket[*,"{#NAME}"],5m,99) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, 50p | 50 percentiles of how long in seconds processing an item from workqueue takes, by queue. |
CALCULATED | kubernetes.controller.workqueue_work_duration_seconds_p50["{#NAME}"] Expression: bucket_percentile(//kubernetes.controller.duration_seconds_bucket[*,"{#NAME}"],5m,50) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p90 | 90 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
CALCULATED | kubernetes.controller.workqueue_queue_duration_seconds_p90["{#NAME}"] Expression: bucket_percentile(//kubernetes.controller.queue_duration_seconds_bucket[*,"{#NAME}"],5m,90) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p95 | 95 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
CALCULATED | kubernetes.controller.workqueue_queue_duration_seconds_p95["{#NAME}"] Expression: bucket_percentile(//kubernetes.controller.queue_duration_seconds_bucket[*,"{#NAME}"],5m,95) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p99 | 99 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
CALCULATED | kubernetes.controller.workqueue_queue_duration_seconds_p99["{#NAME}"] Expression: bucket_percentile(//kubernetes.controller.queue_duration_seconds_bucket[*,"{#NAME}"],5m,99) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, 50p | 50 percentile of how long in seconds an item stays in workqueue before being requested. If there are no requests for 5 minute, item value will be discarded. |
CALCULATED | kubernetes.controller.workqueue_queue_duration_seconds_p50["{#NAME}"] Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: Expression: bucket_percentile(//kubernetes.controller.queue_duration_seconds_bucket[*,"{#NAME}"],5m,50) |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Workqueue duration seconds bucket, {#LE} | How long in seconds processing an item from workqueue takes. |
DEPENDENT | kubernetes.controller.duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Controller | Kubernetes Controller Manager: ["{#NAME}"]: Queue duration seconds bucket, {#LE} | How long in seconds an item stays in workqueue before being requested. |
DEPENDENT | kubernetes.controller.queue_duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Zabbix raw items | Kubernetes Controller: Get Controller metrics | Get raw metrics from Controller instance /metrics endpoint. |
HTTP_AGENT | kubernetes.controller.get_metrics Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: |
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Controller Manager: Too many HTTP client errors | "Kubernetes Controller manager is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Controller manager by HTTP/kubernetes.controller.client_http_requests_500.rate,5m)>{$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} |
WARNING |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template or ask for help with it at ZABBIX forums.
Kubernetes Controller manager by HTTP
Overview
The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP - collects metrics by HTTP agent from Controller manager /metrics endpoint.
Requirements
Zabbix version: 6.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes Controller manager 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.CONTROLLER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. You might need to set the --binding-address option for Controller Manager to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
NOTE. Some metrics may not be collected depending on your Kubernetes Controller manager instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.CONTROLLER.SERVER.URL} | Kubernetes Controller manager metrics endpoint URL. |
https://localhost:10257/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token |
|
| {$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller: Get Controller metrics | Get raw metrics from Controller instance /metrics endpoint. |
HTTP agent | kubernetes.controller.get_metrics Preprocessing
|
| Kubernetes Controller Manager: Leader election status | Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. |
Dependent item | kubernetes.controller.leader_election_master_status Preprocessing
|
| Kubernetes Controller Manager: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.controller.process_virtual_memory_bytes Preprocessing
|
| Kubernetes Controller Manager: Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.controller.process_resident_memory_bytes Preprocessing
|
| Kubernetes Controller Manager: CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.controller.cpu.util Preprocessing
|
| Kubernetes Controller Manager: Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.controller.go_goroutines Preprocessing
|
| Kubernetes Controller Manager: Go threads | Number of OS threads created. |
Dependent item | kubernetes.controller.go_threads Preprocessing
|
| Kubernetes Controller Manager: Fds open | Number of open file descriptors. |
Dependent item | kubernetes.controller.open_fds Preprocessing
|
| Kubernetes Controller Manager: Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.controller.max_fds Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_200.rate Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_300.rate Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_400.rate Preprocessing
|
| Kubernetes Controller Manager: REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.controller.client_http_requests_500.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Controller Manager: Too many HTTP client errors | "Kubernetes Controller manager is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Controller manager by HTTP/kubernetes.controller.client_http_requests_500.rate,5m)>{$KUBE.CONTROLLER.HTTP.CLIENT.ERROR} |
Warning |
LLD rule Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Workqueue metrics discovery | Dependent item | kubernetes.controller.workqueue.discovery Preprocessing
|
Item prototypes for Workqueue metrics discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue adds total, rate | Total number of adds handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_adds_total["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue depth | Current depth of workqueue. |
Dependent item | kubernetes.controller.workqueue_depth["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue unfinished work, sec | How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. |
Dependent item | kubernetes.controller.workqueue_unfinished_work_seconds["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue retries, rate | Total number of retries handled by workqueue per second. |
Dependent item | kubernetes.controller.workqueue_retries_total["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue longest running processor, sec | How many seconds has the longest running processor for workqueue been running. |
Dependent item | kubernetes.controller.workqueue_longest_running_processor_seconds["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p90 | 90 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p90["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p95 | 95 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p95["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, p99 | 99 percentile of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p99["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue work duration, 50p | 50 percentiles of how long in seconds processing an item from workqueue takes, by queue. |
Calculated | kubernetes.controller.workqueue_work_duration_seconds_p50["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p90 | 90 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p90["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p95 | 95 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p95["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, p99 | 99 percentile of how long in seconds an item stays in workqueue before being requested, by queue. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p99["{#NAME}"] |
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue queue duration, 50p | 50 percentile of how long in seconds an item stays in workqueue before being requested. If there are no requests for 5 minute, item value will be discarded. |
Calculated | kubernetes.controller.workqueue_queue_duration_seconds_p50["{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Workqueue duration seconds bucket, {#LE} | How long in seconds processing an item from workqueue takes. |
Dependent item | kubernetes.controller.duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
| Kubernetes Controller Manager: ["{#NAME}"]: Queue duration seconds bucket, {#LE} | How long in seconds an item stays in workqueue before being requested. |
Dependent item | kubernetes.controller.queue_duration_seconds_bucket[{#LE},"{#NAME}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Kubelet by HTTP
Overview
The template to monitor Kubernetes Kubelet by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Kubelet by HTTP - collects metrics by HTTP agent from Kubelet /metrics endpoint.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
Note: Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Requirements
Zabbix version: 7.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
Note: Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.KUBELET.URL} | Kubernetes Kubelet instance URL. |
https://localhost:10250 |
| {$KUBE.KUBELET.METRIC.ENDPOINT} | Kubelet /metrics endpoint. |
/metrics |
| {$KUBE.KUBELET.CADVISOR.ENDPOINT} | cAdvisor metrics from Kubelet /metrics/cadvisor endpoint. |
/metrics/cadvisor |
| {$KUBE.KUBELET.PODS.ENDPOINT} | Kubelet /pods endpoint. |
/pods |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get kubelet metrics | Collecting raw Kubelet metrics from /metrics endpoint. |
HTTP agent | kube.kubelet.metrics |
| Get cadvisor metrics | Collecting raw Kubelet metrics from /metrics/cadvisor endpoint. |
HTTP agent | kube.cadvisor.metrics |
| Get pods | Collecting raw Kubelet metrics from /pods endpoint. |
HTTP agent | kube.pods |
| Pods running | The number of running pods. |
Dependent item | kube.kubelet.pods.running Preprocessing
|
| Containers started | The number of started containers. |
Dependent item | kube.kubelet.containers.started Preprocessing
|
| Containers ready | The number of ready containers. |
Dependent item | kube.kubelet.containers.ready Preprocessing
|
| Containers last state terminated | The number of containers that were previously terminated. |
Dependent item | kube.kublet.containers.terminated Preprocessing
|
| Containers restarts | The number of times the container has been restarted. |
Dependent item | kube.kubelet.containers.restarts Preprocessing
|
| CPU cores, total | The number of cores in this machine (available until kubernetes v1.18). |
Dependent item | kube.kubelet.cpu.cores Preprocessing
|
| Machine memory, bytes | Resident memory size in bytes. |
Dependent item | kube.kubelet.machine.memory Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kube.kubelet.virtual.memory Preprocessing
|
| File descriptors, max | Maximum number of open file descriptors. |
Dependent item | kube.kubelet.process_max_fds Preprocessing
|
| File descriptors, open | Number of open file descriptors. |
Dependent item | kube.kubelet.process_open_fds Preprocessing
|
LLD rule Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Runtime operations discovery | Dependent item | kube.kubelet.runtime_operations_bucket.discovery Preprocessing
|
Item prototypes for Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| [{#OP_TYPE}] Runtime operations bucket: {#LE} | Duration in seconds of runtime operations. Broken down by operation type. |
Dependent item | kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Runtime operations total, rate | Cumulative number of runtime operations by operation type. |
Dependent item | kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Operations, p90 | 90 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p95 | 95 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p99 | 99 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p50 | 50 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"] |
LLD rule Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pods discovery | Dependent item | kube.kubelet.pods.discovery Preprocessing
|
Item prototypes for Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s | Pods cpu load average over the last 10 seconds. |
Dependent item | kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total | System cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Usage seconds, total | Consumed cpu time. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_usage_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total | User cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| REST client requests discovery | Dependent item | kube.kubelet.rest.requests.discovery Preprocessing
|
Item prototypes for REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}] | Number of HTTP requests, partitioned by status code, method, and host. |
Dependent item | kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"] Preprocessing
|
LLD rule Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Container memory discovery | Dependent item | kube.kubelet.container.memory.cache.discovery Preprocessing
|
Item prototypes for Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache | Number of bytes of page cache memory. |
Dependent item | kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage | Maximum memory usage recorded in bytes. |
Dependent item | kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS | Size of RSS in bytes. |
Dependent item | kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap | Container swap usage in bytes. |
Dependent item | kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage | Current memory usage in bytes, including all memory regardless of when it was accessed. |
Dependent item | kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set | Current working set in bytes. |
Dependent item | kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Kubelet by HTTP
Overview
The template to monitor Kubernetes Kubelet by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Kubelet by HTTP - collects metrics by HTTP agent from Kubelet /metrics endpoint.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Requirements
Zabbix version: 7.2 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.KUBELET.URL} | Kubernetes Kubelet instance URL. |
https://localhost:10250 |
| {$KUBE.KUBELET.METRIC.ENDPOINT} | Kubelet /metrics endpoint. |
/metrics |
| {$KUBE.KUBELET.CADVISOR.ENDPOINT} | cAdvisor metrics from Kubelet /metrics/cadvisor endpoint. |
/metrics/cadvisor |
| {$KUBE.KUBELET.PODS.ENDPOINT} | Kubelet /pods endpoint. |
/pods |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get kubelet metrics | Collecting raw Kubelet metrics from /metrics endpoint. |
HTTP agent | kube.kubelet.metrics |
| Get cadvisor metrics | Collecting raw Kubelet metrics from /metrics/cadvisor endpoint. |
HTTP agent | kube.cadvisor.metrics |
| Get pods | Collecting raw Kubelet metrics from /pods endpoint. |
HTTP agent | kube.pods |
| Pods running | The number of running pods. |
Dependent item | kube.kubelet.pods.running Preprocessing
|
| Containers started | The number of started containers. |
Dependent item | kube.kubelet.containers.started Preprocessing
|
| Containers ready | The number of ready containers. |
Dependent item | kube.kubelet.containers.ready Preprocessing
|
| Containers last state terminated | The number of containers that were previously terminated. |
Dependent item | kube.kublet.containers.terminated Preprocessing
|
| Containers restarts | The number of times the container has been restarted. |
Dependent item | kube.kubelet.containers.restarts Preprocessing
|
| CPU cores, total | The number of cores in this machine (available until kubernetes v1.18). |
Dependent item | kube.kubelet.cpu.cores Preprocessing
|
| Machine memory, bytes | Resident memory size in bytes. |
Dependent item | kube.kubelet.machine.memory Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kube.kubelet.virtual.memory Preprocessing
|
| File descriptors, max | Maximum number of open file descriptors. |
Dependent item | kube.kubelet.process_max_fds Preprocessing
|
| File descriptors, open | Number of open file descriptors. |
Dependent item | kube.kubelet.process_open_fds Preprocessing
|
LLD rule Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Runtime operations discovery | Dependent item | kube.kubelet.runtime_operations_bucket.discovery Preprocessing
|
Item prototypes for Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| [{#OP_TYPE}] Runtime operations bucket: {#LE} | Duration in seconds of runtime operations. Broken down by operation type. |
Dependent item | kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Runtime operations total, rate | Cumulative number of runtime operations by operation type. |
Dependent item | kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Operations, p90 | 90 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p95 | 95 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p99 | 99 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p50 | 50 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"] |
LLD rule Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pods discovery | Dependent item | kube.kubelet.pods.discovery Preprocessing
|
Item prototypes for Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s | Pods cpu load average over the last 10 seconds. |
Dependent item | kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total | System cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Usage seconds, total | Consumed cpu time. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_usage_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total | User cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| REST client requests discovery | Dependent item | kube.kubelet.rest.requests.discovery Preprocessing
|
Item prototypes for REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}] | Number of HTTP requests, partitioned by status code, method, and host. |
Dependent item | kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"] Preprocessing
|
LLD rule Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Container memory discovery | Dependent item | kube.kubelet.container.memory.cache.discovery Preprocessing
|
Item prototypes for Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache | Number of bytes of page cache memory. |
Dependent item | kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage | Maximum memory usage recorded in bytes. |
Dependent item | kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS | Size of RSS in bytes. |
Dependent item | kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap | Container swap usage in bytes. |
Dependent item | kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage | Current memory usage in bytes, including all memory regardless of when it was accessed. |
Dependent item | kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set | Current working set in bytes. |
Dependent item | kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Kubelet by HTTP
Overview
The template to monitor Kubernetes Kubelet by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Kubelet by HTTP - collects metrics by HTTP agent from Kubelet /metrics endpoint.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
Note: Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Requirements
Zabbix version: 7.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
Note: Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.KUBELET.URL} | Kubernetes Kubelet instance URL. |
https://localhost:10250 |
| {$KUBE.KUBELET.METRIC.ENDPOINT} | Kubelet /metrics endpoint. |
/metrics |
| {$KUBE.KUBELET.CADVISOR.ENDPOINT} | cAdvisor metrics from Kubelet /metrics/cadvisor endpoint. |
/metrics/cadvisor |
| {$KUBE.KUBELET.PODS.ENDPOINT} | Kubelet /pods endpoint. |
/pods |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get kubelet metrics | Collecting raw Kubelet metrics from /metrics endpoint. |
HTTP agent | kube.kubelet.metrics |
| Get cadvisor metrics | Collecting raw Kubelet metrics from /metrics/cadvisor endpoint. |
HTTP agent | kube.cadvisor.metrics |
| Get pods | Collecting raw Kubelet metrics from /pods endpoint. |
HTTP agent | kube.pods |
| Pods running | The number of running pods. |
Dependent item | kube.kubelet.pods.running Preprocessing
|
| Containers started | The number of started containers. |
Dependent item | kube.kubelet.containers.started Preprocessing
|
| Containers ready | The number of ready containers. |
Dependent item | kube.kubelet.containers.ready Preprocessing
|
| Containers last state terminated | The number of containers that were previously terminated. |
Dependent item | kube.kublet.containers.terminated Preprocessing
|
| Containers restarts | The number of times the container has been restarted. |
Dependent item | kube.kubelet.containers.restarts Preprocessing
|
| CPU cores, total | The number of cores in this machine (available until kubernetes v1.18). |
Dependent item | kube.kubelet.cpu.cores Preprocessing
|
| Machine memory, bytes | Resident memory size in bytes. |
Dependent item | kube.kubelet.machine.memory Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kube.kubelet.virtual.memory Preprocessing
|
| File descriptors, max | Maximum number of open file descriptors. |
Dependent item | kube.kubelet.process_max_fds Preprocessing
|
| File descriptors, open | Number of open file descriptors. |
Dependent item | kube.kubelet.process_open_fds Preprocessing
|
LLD rule Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Runtime operations discovery | Dependent item | kube.kubelet.runtime_operations_bucket.discovery Preprocessing
|
Item prototypes for Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| [{#OP_TYPE}] Runtime operations bucket: {#LE} | Duration in seconds of runtime operations. Broken down by operation type. |
Dependent item | kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Runtime operations total, rate | Cumulative number of runtime operations by operation type. |
Dependent item | kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"] Preprocessing
|
| [{#OP_TYPE}] Operations, p90 | 90 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p95 | 95 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p99 | 99 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"] |
| [{#OP_TYPE}] Operations, p50 | 50 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"] |
LLD rule Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pods discovery | Dependent item | kube.kubelet.pods.discovery Preprocessing
|
Item prototypes for Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s | Pods cpu load average over the last 10 seconds. |
Dependent item | kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total | System cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Usage seconds, total | Consumed cpu time. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_usage_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total | User cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| REST client requests discovery | Dependent item | kube.kubelet.rest.requests.discovery Preprocessing
|
Item prototypes for REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}] | Number of HTTP requests, partitioned by status code, method, and host. |
Dependent item | kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"] Preprocessing
|
LLD rule Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Container memory discovery | Dependent item | kube.kubelet.container.memory.cache.discovery Preprocessing
|
Item prototypes for Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache | Number of bytes of page cache memory. |
Dependent item | kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage | Maximum memory usage recorded in bytes. |
Dependent item | kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS | Size of RSS in bytes. |
Dependent item | kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap | Container swap usage in bytes. |
Dependent item | kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage | Current memory usage in bytes, including all memory regardless of when it was accessed. |
Dependent item | kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set | Current working set in bytes. |
Dependent item | kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Kubelet by HTTP
Overview
The template to monitor Kubernetes Kubelet by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Kubelet by HTTP - collects metrics by HTTP agent from Kubelet /metrics endpoint.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Requirements
Zabbix version: 6.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.KUBELET.URL} | Kubernetes Kubelet instance URL. |
https://localhost:10250 |
| {$KUBE.KUBELET.METRIC.ENDPOINT} | Kubelet /metrics endpoint. |
/metrics |
| {$KUBE.KUBELET.CADVISOR.ENDPOINT} | cAdvisor metrics from Kubelet /metrics/cadvisor endpoint. |
/metrics/cadvisor |
| {$KUBE.KUBELET.PODS.ENDPOINT} | Kubelet /pods endpoint. |
/pods |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Get kubelet metrics | Collecting raw Kubelet metrics from /metrics endpoint. |
HTTP agent | kube.kubelet.metrics |
| Kubernetes: Get cadvisor metrics | Collecting raw Kubelet metrics from /metrics/cadvisor endpoint. |
HTTP agent | kube.cadvisor.metrics |
| Kubernetes: Get pods | Collecting raw Kubelet metrics from /pods endpoint. |
HTTP agent | kube.pods |
| Kubernetes: Pods running | The number of running pods. |
Dependent item | kube.kubelet.pods.running Preprocessing
|
| Kubernetes: Containers running | The number of running containers. |
Dependent item | kube.kubelet.containers.running Preprocessing
|
| Kubernetes: Containers last state terminated | The number of containers that were previously terminated. |
Dependent item | kube.kublet.containers.terminated Preprocessing
|
| Kubernetes: Containers restarts | The number of times the container has been restarted. |
Dependent item | kube.kubelet.containers.restarts Preprocessing
|
| Kubernetes: CPU cores, total | The number of cores in this machine (available until kubernetes v1.18). |
Dependent item | kube.kubelet.cpu.cores Preprocessing
|
| Kubernetes: Machine memory, bytes | Resident memory size in bytes. |
Dependent item | kube.kubelet.machine.memory Preprocessing
|
| Kubernetes: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kube.kubelet.virtual.memory Preprocessing
|
| Kubernetes: File descriptors, max | Maximum number of open file descriptors. |
Dependent item | kube.kubelet.process_max_fds Preprocessing
|
| Kubernetes: File descriptors, open | Number of open file descriptors. |
Dependent item | kube.kubelet.process_open_fds Preprocessing
|
LLD rule Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Runtime operations discovery | Dependent item | kube.kubelet.runtime_operations_bucket.discovery Preprocessing
|
Item prototypes for Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: [{#OP_TYPE}] Runtime operations bucket: {#LE} | Duration in seconds of runtime operations. Broken down by operation type. |
Dependent item | kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"] Preprocessing
|
| Kubernetes: [{#OP_TYPE}] Runtime operations total, rate | Cumulative number of runtime operations by operation type. |
Dependent item | kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"] Preprocessing
|
| Kubernetes: [{#OP_TYPE}] Operations, p90 | 90 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"] |
| Kubernetes: [{#OP_TYPE}] Operations, p95 | 95 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"] |
| Kubernetes: [{#OP_TYPE}] Operations, p99 | 99 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"] |
| Kubernetes: [{#OP_TYPE}] Operations, p50 | 50 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"] |
LLD rule Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pods discovery | Dependent item | kube.kubelet.pods.discovery Preprocessing
|
Item prototypes for Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s | Pods cpu load average over the last 10 seconds. |
Dependent item | kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total | System cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Usage seconds, total | Consumed cpu time. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_usage_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total | User cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| REST client requests discovery | Dependent item | kube.kubelet.rest.requests.discovery Preprocessing
|
Item prototypes for REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}] | Number of HTTP requests, partitioned by status code, method, and host. |
Dependent item | kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"] Preprocessing
|
LLD rule Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Container memory discovery | Dependent item | kube.kubelet.container.memory.cache.discovery Preprocessing
|
Item prototypes for Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache | Number of bytes of page cache memory. |
Dependent item | kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage | Maximum memory usage recorded in bytes. |
Dependent item | kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS | Size of RSS in bytes. |
Dependent item | kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap | Container swap usage in bytes. |
Dependent item | kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage | Current memory usage in bytes, including all memory regardless of when it was accessed. |
Dependent item | kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set | Current working set in bytes. |
Dependent item | kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Kubelet by HTTP
Overview
For Zabbix version: 6.2 and higher. The template to monitor Kubernetes Controller manager by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Controller manager by HTTP — collects metrics by HTTP agent from Controller manager /metrics endpoint.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}. NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
This template was tested on:
- Kubernetes, version 1.19
Setup
See Zabbix template operation for basic instructions.
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}. NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Zabbix configuration
No specific Zabbix configuration is required.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token |
`` |
| {$KUBE.KUBELET.CADVISOR.ENDPOINT} | cAdvisor metrics from Kubelet /metrics/cadvisor endpoint |
/metrics/cadvisor |
| {$KUBE.KUBELET.METRIC.ENDPOINT} | Kubelet /metrics endpoint |
/metrics |
| {$KUBE.KUBELET.PODS.ENDPOINT} | Kubelet /pods endpoint |
/pods |
| {$KUBE.KUBELET.URL} | Instance URL |
https://localhost:10250 |
Template links
There are no template links in this template.
Discovery rules
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Container memory discovery | DEPENDENT | kube.kubelet.container.memory.cache.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
|
| Pods discovery | DEPENDENT | kube.kubelet.pods.discovery Preprocessing: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
|
| REST client requests discovery | DEPENDENT | kube.kubelet.rest.requests.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: |
|
| Runtime operations discovery | DEPENDENT | kube.kubelet.runtime_operations_bucket.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: Overrides: bucket item total item |
Items collected
| Group | Name | Description | Type | Key and additional info |
|---|---|---|---|---|
| Kubernetes | Kubernetes: Get kubelet metrics | Collecting raw Kubelet metrics from /metrics endpoint. |
HTTP_AGENT | kube.kubelet.metrics |
| Kubernetes | Kubernetes: Get cadvisor metrics | Collecting raw Kubelet metrics from /metrics/cadvisor endpoint. |
HTTP_AGENT | kube.cadvisor.metrics |
| Kubernetes | Kubernetes: Get pods | Collecting raw Kubelet metrics from /pods endpoint. |
HTTP_AGENT | kube.pods |
| Kubernetes | Kubernetes: Pods running | The number of running pods. |
DEPENDENT | kube.kubelet.pods.running Preprocessing: - JSONPATH: |
| Kubernetes | Kubernetes: Containers running | The number of running containers. |
DEPENDENT | kube.kubelet.containers.running Preprocessing: - JSONPATH: |
| Kubernetes | Kubernetes: Containers last state terminated | The number of containers that were previously terminated. |
DEPENDENT | kube.kublet.containers.terminated Preprocessing: - JSONPATH: |
| Kubernetes | Kubernetes: Containers restarts | The number of times the container has been restarted. |
DEPENDENT | kube.kubelet.containers.restarts Preprocessing: - JSONPATH: |
| Kubernetes | Kubernetes: CPU cores, total | The number of cores in this machine (available until kubernetes v1.18). |
DEPENDENT | kube.kubelet.cpu.cores Preprocessing: - PROMETHEUS_PATTERN: |
| Kubernetes | Kubernetes: Machine memory, bytes | Resident memory size in bytes. |
DEPENDENT | kube.kubelet.machine.memory Preprocessing: - PROMETHEUS_PATTERN: |
| Kubernetes | Kubernetes: Virtual memory, bytes | Virtual memory size in bytes. |
DEPENDENT | kube.kubelet.virtual.memory Preprocessing: - PROMETHEUS_PATTERN: |
| Kubernetes | Kubernetes: File descriptors, max | Maximum number of open file descriptors. |
DEPENDENT | kube.kubelet.process_max_fds Preprocessing: - PROMETHEUS_PATTERN: |
| Kubernetes | Kubernetes: File descriptors, open | Number of open file descriptors. |
DEPENDENT | kube.kubelet.process_open_fds Preprocessing: - PROMETHEUS_PATTERN: |
| Kubernetes | Kubernetes: [{#OP_TYPE}] Runtime operations bucket: {#LE} | Duration in seconds of runtime operations. Broken down by operation type. |
DEPENDENT | kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"] Preprocessing: - PROMETHEUS_PATTERN: |
| Kubernetes | Kubernetes: [{#OP_TYPE}] Runtime operations total, rate | Cumulative number of runtime operations by operation type. |
DEPENDENT | kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes | Kubernetes: [{#OP_TYPE}] Operations, p90 | 90 percentile of operation latency distribution in seconds for each verb. |
CALCULATED | kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"] Expression: bucket_percentile(//kube.kublet.runtime_ops_duration_seconds_bucket[*,"{#OP_TYPE}"],5m,90) |
| Kubernetes | Kubernetes: [{#OP_TYPE}] Operations, p95 | 95 percentile of operation latency distribution in seconds for each verb. |
CALCULATED | kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"] Expression: bucket_percentile(//kube.kublet.runtime_ops_duration_seconds_bucket[*,"{#OP_TYPE}"],5m,95) |
| Kubernetes | Kubernetes: [{#OP_TYPE}] Operations, p99 | 99 percentile of operation latency distribution in seconds for each verb. |
CALCULATED | kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"] Expression: bucket_percentile(//kube.kublet.runtime_ops_duration_seconds_bucket[*,"{#OP_TYPE}"],5m,99) |
| Kubernetes | Kubernetes: [{#OP_TYPE}] Operations, p50 | 50 percentile of operation latency distribution in seconds for each verb. |
CALCULATED | kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"] Expression: bucket_percentile(//kube.kublet.runtime_ops_duration_seconds_bucket[*,"{#OP_TYPE}"],5m,50) |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s | Pods cpu load average over the last 10 seconds. |
DEPENDENT | kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total | The number of cores used for system time. |
DEPENDENT | kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total | The number of cores used for user time. |
DEPENDENT | kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}] | Number of HTTP requests, partitioned by status code, method, and host. |
DEPENDENT | kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"] Preprocessing: - PROMETHEUS_PATTERN: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache | Number of bytes of page cache memory. |
DEPENDENT | kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing: - PROMETHEUS_PATTERN: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage | Maximum memory usage recorded in bytes. |
DEPENDENT | kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing: - PROMETHEUS_PATTERN: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS | Size of RSS in bytes. |
DEPENDENT | kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing: - PROMETHEUS_PATTERN: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap | Container swap usage in bytes. |
DEPENDENT | kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing: - PROMETHEUS_PATTERN: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage | Current memory usage in bytes, including all memory regardless of when it was accessed. |
DEPENDENT | kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing: - PROMETHEUS_PATTERN: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set | Current working set in bytes. |
DEPENDENT | kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing: - PROMETHEUS_PATTERN: - DISCARD_UNCHANGED_HEARTBEAT: |
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|
Feedback
Please report any issues with the template at https://support.zabbix.com.
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums.
Kubernetes Kubelet by HTTP
Overview
The template to monitor Kubernetes Kubelet by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Kubelet by HTTP - collects metrics by HTTP agent from Kubelet /metrics endpoint.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Requirements
Zabbix version: 6.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.KUBELET.URL}, {$KUBE.API.TOKEN}.
NOTE. Some metrics may not be collected depending on your Kubernetes instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.KUBELET.URL} | Kubernetes Kubelet instance URL. |
https://localhost:10250 |
| {$KUBE.KUBELET.METRIC.ENDPOINT} | Kubelet /metrics endpoint. |
/metrics |
| {$KUBE.KUBELET.CADVISOR.ENDPOINT} | cAdvisor metrics from Kubelet /metrics/cadvisor endpoint. |
/metrics/cadvisor |
| {$KUBE.KUBELET.PODS.ENDPOINT} | Kubelet /pods endpoint. |
/pods |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Get kubelet metrics | Collecting raw Kubelet metrics from /metrics endpoint. |
HTTP agent | kube.kubelet.metrics |
| Kubernetes: Get cadvisor metrics | Collecting raw Kubelet metrics from /metrics/cadvisor endpoint. |
HTTP agent | kube.cadvisor.metrics |
| Kubernetes: Get pods | Collecting raw Kubelet metrics from /pods endpoint. |
HTTP agent | kube.pods |
| Kubernetes: Pods running | The number of running pods. |
Dependent item | kube.kubelet.pods.running Preprocessing
|
| Kubernetes: Containers started | The number of started containers. |
Dependent item | kube.kubelet.containers.started Preprocessing
|
| Kubernetes: Containers ready | The number of ready containers. |
Dependent item | kube.kubelet.containers.ready Preprocessing
|
| Kubernetes: Containers last state terminated | The number of containers that were previously terminated. |
Dependent item | kube.kublet.containers.terminated Preprocessing
|
| Kubernetes: Containers restarts | The number of times the container has been restarted. |
Dependent item | kube.kubelet.containers.restarts Preprocessing
|
| Kubernetes: CPU cores, total | The number of cores in this machine (available until kubernetes v1.18). |
Dependent item | kube.kubelet.cpu.cores Preprocessing
|
| Kubernetes: Machine memory, bytes | Resident memory size in bytes. |
Dependent item | kube.kubelet.machine.memory Preprocessing
|
| Kubernetes: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kube.kubelet.virtual.memory Preprocessing
|
| Kubernetes: File descriptors, max | Maximum number of open file descriptors. |
Dependent item | kube.kubelet.process_max_fds Preprocessing
|
| Kubernetes: File descriptors, open | Number of open file descriptors. |
Dependent item | kube.kubelet.process_open_fds Preprocessing
|
LLD rule Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Runtime operations discovery | Dependent item | kube.kubelet.runtime_operations_bucket.discovery Preprocessing
|
Item prototypes for Runtime operations discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: [{#OP_TYPE}] Runtime operations bucket: {#LE} | Duration in seconds of runtime operations. Broken down by operation type. |
Dependent item | kube.kublet.runtime_ops_duration_seconds_bucket[{#LE},"{#OP_TYPE}"] Preprocessing
|
| Kubernetes: [{#OP_TYPE}] Runtime operations total, rate | Cumulative number of runtime operations by operation type. |
Dependent item | kube.kublet.runtime_ops_total.rate["{#OP_TYPE}"] Preprocessing
|
| Kubernetes: [{#OP_TYPE}] Operations, p90 | 90 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p90["{#OP_TYPE}"] |
| Kubernetes: [{#OP_TYPE}] Operations, p95 | 95 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p95["{#OP_TYPE}"] |
| Kubernetes: [{#OP_TYPE}] Operations, p99 | 99 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p99["{#OP_TYPE}"] |
| Kubernetes: [{#OP_TYPE}] Operations, p50 | 50 percentile of operation latency distribution in seconds for each verb. |
Calculated | kube.kublet.runtime_ops_duration_seconds_p50["{#OP_TYPE}"] |
LLD rule Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pods discovery | Dependent item | kube.kubelet.pods.discovery Preprocessing
|
Item prototypes for Pods discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Load average, 10s | Pods cpu load average over the last 10 seconds. |
Dependent item | kube.pod.container_cpu_load_average_10s[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: System seconds, total | System cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_system_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: Usage seconds, total | Consumed cpu time. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_usage_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] CPU: User seconds, total | User cpu time consumed. It is calculated from the cumulative value using the |
Dependent item | kube.pod.container_cpu_user_seconds_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| REST client requests discovery | Dependent item | kube.kubelet.rest.requests.discovery Preprocessing
|
Item prototypes for REST client requests discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Host [{#HOST}] Request method [{#METHOD}] Code:[{#CODE}] | Number of HTTP requests, partitioned by status code, method, and host. |
Dependent item | kube.kubelet.rest.requests["{#CODE}", "{#HOST}", "{#METHOD}"] Preprocessing
|
LLD rule Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Container memory discovery | Dependent item | kube.kubelet.container.memory.cache.discovery Preprocessing
|
Item prototypes for Container memory discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory page cache | Number of bytes of page cache memory. |
Dependent item | kube.kubelet.container.memory.cache["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Memory max usage | Maximum memory usage recorded in bytes. |
Dependent item | kube.kubelet.container.memory.max_usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: RSS | Size of RSS in bytes. |
Dependent item | kube.kubelet.container.memory.rss["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Swap | Container swap usage in bytes. |
Dependent item | kube.kubelet.container.memory.swap["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Usage | Current memory usage in bytes, including all memory regardless of when it was accessed. |
Dependent item | kube.kubelet.container.memory.usage["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#POD}] Container [{#CONTAINER}]: Working set | Current working set in bytes. |
Dependent item | kube.kubelet.container.memory.working_set["{#CONTAINER}", "{#NAMESPACE}", "{#POD}"] Preprocessing
|
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes nodes by HTTP
Overview
The template to monitor Kubernetes nodes that work without any external scripts. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API. Install the Zabbix Helm Chart (https://git.zabbix.com/projects/ZT/repos/kubernetes-helm/browse?at=refs%2Fheads%2Frelease/7.4) in your Kubernetes cluster.
Change the values according to the environment in the file $HOME/zabbix_values.yaml.
For example:
Enables use of Zabbix proxy
enabled: false
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the service account name. If a different release name is used.
kubectl get serviceaccounts -n monitoring
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set up the macros to filter the metrics of discovered nodes.
Requirements
Zabbix version: 7.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chart-agent.
Set up the macros to filter the metrics of discovered nodes and host creation based on host prototypes:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES}
Set up macros to filter pod metrics by namespace:
- {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable pods.
You can use the {$KUBE.NODE.FILTER.LABELS}, {$KUBE.POD.FILTER.LABELS}, {$KUBE.NODE.FILTER.ANNOTATIONS} and {$KUBE.POD.FILTER.ANNOTATIONS} macros for advanced filtering of nodes and pods by labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such a label key exists for the entity that is being filtered (it means that if you specify a key in a filter, entities which do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
Note: The discovered nodes will be created as separate hosts in Zabbix with the Linux template automatically assigned to them.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$HTTP.TLS.VERIFY} | TLS certificate verification for the script item: "none" - disabled, "peer" - verify the certificate chain and expiration, "full" - full verification. Any other value enables full verification. To override the setting for this template only, define the macro with the context "Kubernetes nodes", e.g. {$HTTP.TLS.VERIFY:"Kubernetes nodes"}. |
full |
| {$KUBE.NODES.ENDPOINT.NAME} | Kubernetes nodes endpoint name. See "kubectl -n monitoring get ep". |
zabbix-zabbix-helm-chart-agent |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES} | Filter of discoverable nodes by role. |
.* |
| {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES} | Filter to exclude discovered node by role. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.FILTER.ANNOTATIONS} | Annotations to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.NODE.FILTER.LABELS} | Labels to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.ANNOTATIONS} | Annotations to filter pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.LABELS} | Labels to filter Pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace. |
.* |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace. |
CHANGE_IF_NEEDED |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get nodes | Collecting and processing cluster nodes data via Kubernetes API. |
Script | kube.nodes |
| Get nodes check | Data collection check. |
Dependent item | kube.nodes.check Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Failed to get nodes | length(last(/Kubernetes nodes by HTTP/kube.nodes.check))>0 |
Warning |
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: Get data | Collecting and processing cluster by node [{#NAME}] data via Kubernetes API. |
Dependent item | kube.node.get[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: External IP | Typically the IP address of the node that is externally routable (available from outside the cluster). |
Dependent item | kube.node.addresses.external_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: Internal IP | Typically the IP address of the node that is routable only within the cluster. |
Dependent item | kube.node.addresses.internal_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: CPU | Allocatable CPU. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Memory | Allocatable Memory. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.allocatable.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: CPU | CPU resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Memory | Memory resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.capacity.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Disk pressure | True if pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
Dependent item | kube.node.conditions.diskpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Memory pressure | True if pressure exists on the node memory - that is, if the node memory is low; otherwise False. |
Dependent item | kube.node.conditions.memorypressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Network unavailable | True if the network for the node is not correctly configured, otherwise False. |
Dependent item | kube.node.conditions.networkunavailable[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: PID pressure | True if pressure exists on the processes - that is, if there are too many processes on the node; otherwise False. |
Dependent item | kube.node.conditions.pidpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Ready | True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
Dependent item | kube.node.conditions.ready[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Architecture | Node architecture. |
Dependent item | kube.node.info.architecture[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Container runtime | Container runtime. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
Dependent item | kube.node.info.containerruntime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kernel version | Node kernel version. |
Dependent item | kube.node.info.kernelversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kubelet version | Version of Kubelet. |
Dependent item | kube.node.info.kubeletversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: KubeProxy version | Version of KubeProxy. |
Dependent item | kube.node.info.kubeproxyversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Operating system | Node operating system. |
Dependent item | kube.node.info.operatingsystem[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: OS image | Node OS image. |
Dependent item | kube.node.info.osversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Roles | Node roles. |
Dependent item | kube.node.info.roles[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: CPU | Node CPU limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: Memory | Node Memory limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: CPU | Node CPU requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: Memory | Node Memory requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Uptime | Node uptime. |
Dependent item | kube.node.uptime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Used: Pods | Current number of pods on the node. |
Dependent item | kube.node.used.pods[{#NAME}] Preprocessing
|
Trigger prototypes for Node discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the disk size | True - pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.diskpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the node memory | True - pressure exists on the node memory - that is, if the node memory is low; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.memorypressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Network is not correctly configured | True - the network for the node is not correctly configured, otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.networkunavailable[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the processes | True - pressure exists on the processes - that is, if there are too many processes on the node; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.pidpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Is not in Ready state | False - if the node is not healthy and is not accepting pods. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.ready[{#NAME}])<>1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] has been restarted | Uptime is less than 10 minutes. |
last(/Kubernetes nodes by HTTP/kube.node.uptime[{#NAME}])<10 |
Info | |
| Kubernetes nodes: Node [{#NAME}] Used: Kubelet too many pods | Kubelet is running at capacity. |
last(/Kubernetes nodes by HTTP/kube.node.used.pods[{#NAME}])/ last(/Kubernetes nodes by HTTP/kube.node.capacity.pods[{#NAME}]) > 0.9 |
Warning |
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Get data | Collecting and processing cluster by node [{#NODE}] data via Kubernetes API. |
Dependent item | kube.pod.get[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Containers ready | All containers in the Pod are ready. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.containers_ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Initialized | All init containers have started successfully. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.initialized[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Ready | The Pod is able to serve requests and should be added to the load balancing pools of all matching Services. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Scheduled | The Pod has been scheduled to a node. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.scheduled[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Containers: Restarts | The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. |
Dependent item | kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Phase | The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase |
Dependent item | kube.pod.status.phase[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Uptime | Pod uptime. |
Dependent item | kube.pod.uptime[{#NAMESPACE}/{#POD}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}])-min(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}],15m))>1 |
Warning | |
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Kubernetes Pod not healthy | Pod has been in a non-ready state for longer than 10 minutes. |
count(/Kubernetes nodes by HTTP/kube.pod.status.phase[{#NAMESPACE}/{#POD}],10m, "regexp","^(1|4|5)$")>=9 |
High |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes nodes by HTTP
Overview
The template to monitor Kubernetes nodes that work without any external scripts. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API. Install the Zabbix Helm Chart (https://git.zabbix.com/projects/ZT/repos/kubernetes-helm/browse?at=refs%2Fheads%2Frelease%2F7.2) in your Kubernetes cluster.
Change the values according to the environment in the file $HOME/zabbix_values.yaml.
For example:
-
Enables use of Zabbix proxy
enabled: false
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set up the macros to filter the metrics of discovered nodes
Requirements
Zabbix version: 7.2 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chrt-agent.
Set up the macros to filter the metrics of discovered nodes and host creation based on host prototypes:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES}
Set up macros to filter pod metrics by namespace:
- {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES}
Note, If you have a large cluster, it is highly recommended to set a filter for discoverable pods.
You can use the {$KUBE.NODE.FILTER.LABELS}, {$KUBE.POD.FILTER.LABELS}, {$KUBE.NODE.FILTER.ANNOTATIONS} and {$KUBE.POD.FILTER.ANNOTATIONS} macros for advanced filtering of nodes and pods by labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such a label key exists for the entity that is being filtered (it means that if you specify a key in a filter, entities which do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
Note, the discovered nodes will be created as separate hosts in Zabbix with the Linux template automatically assigned to them.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$KUBE.NODES.ENDPOINT.NAME} | Kubernetes nodes endpoint name. See "kubectl -n monitoring get ep". |
zabbix-zabbix-helm-chrt-agent |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES} | Filter of discoverable nodes by role. |
.* |
| {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES} | Filter to exclude discovered node by role. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.FILTER.ANNOTATIONS} | Annotations to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.NODE.FILTER.LABELS} | Labels to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.ANNOTATIONS} | Annotations to filter pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.LABELS} | Labels to filter Pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace. |
.* |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace. |
CHANGE_IF_NEEDED |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get nodes | Collecting and processing cluster nodes data via Kubernetes API. |
Script | kube.nodes |
| Get nodes check | Data collection check. |
Dependent item | kube.nodes.check Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Failed to get nodes | length(last(/Kubernetes nodes by HTTP/kube.nodes.check))>0 |
Warning |
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: Get data | Collecting and processing cluster by node [{#NAME}] data via Kubernetes API. |
Dependent item | kube.node.get[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: External IP | Typically the IP address of the node that is externally routable (available from outside the cluster). |
Dependent item | kube.node.addresses.external_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: Internal IP | Typically the IP address of the node that is routable only within the cluster. |
Dependent item | kube.node.addresses.internal_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: CPU | Allocatable CPU. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Memory | Allocatable Memory. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.allocatable.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: CPU | CPU resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Memory | Memory resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.capacity.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Disk pressure | True if pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
Dependent item | kube.node.conditions.diskpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Memory pressure | True if pressure exists on the node memory - that is, if the node memory is low; otherwise False. |
Dependent item | kube.node.conditions.memorypressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Network unavailable | True if the network for the node is not correctly configured, otherwise False. |
Dependent item | kube.node.conditions.networkunavailable[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: PID pressure | True if pressure exists on the processes - that is, if there are too many processes on the node; otherwise False. |
Dependent item | kube.node.conditions.pidpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Ready | True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
Dependent item | kube.node.conditions.ready[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Architecture | Node architecture. |
Dependent item | kube.node.info.architecture[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Container runtime | Container runtime. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
Dependent item | kube.node.info.containerruntime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kernel version | Node kernel version. |
Dependent item | kube.node.info.kernelversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kubelet version | Version of Kubelet. |
Dependent item | kube.node.info.kubeletversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: KubeProxy version | Version of KubeProxy. |
Dependent item | kube.node.info.kubeproxyversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Operating system | Node operating system. |
Dependent item | kube.node.info.operatingsystem[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: OS image | Node OS image. |
Dependent item | kube.node.info.osversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Roles | Node roles. |
Dependent item | kube.node.info.roles[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: CPU | Node CPU limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: Memory | Node Memory limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: CPU | Node CPU requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: Memory | Node Memory requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Uptime | Node uptime. |
Dependent item | kube.node.uptime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Used: Pods | Current number of pods on the node. |
Dependent item | kube.node.used.pods[{#NAME}] Preprocessing
|
Trigger prototypes for Node discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the disk size | True - pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.diskpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the node memory | True - pressure exists on the node memory - that is, if the node memory is low; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.memorypressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Network is not correctly configured | True - the network for the node is not correctly configured, otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.networkunavailable[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the processes | True - pressure exists on the processes - that is, if there are too many processes on the node; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.pidpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Is not in Ready state | False - if the node is not healthy and is not accepting pods. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.ready[{#NAME}])<>1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] has been restarted | Uptime is less than 10 minutes. |
last(/Kubernetes nodes by HTTP/kube.node.uptime[{#NAME}])<10 |
Info | |
| Kubernetes nodes: Node [{#NAME}] Used: Kubelet too many pods | Kubelet is running at capacity. |
last(/Kubernetes nodes by HTTP/kube.node.used.pods[{#NAME}])/ last(/Kubernetes nodes by HTTP/kube.node.capacity.pods[{#NAME}]) > 0.9 |
Warning |
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Get data | Collecting and processing cluster by node [{#NODE}] data via Kubernetes API. |
Dependent item | kube.pod.get[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Containers ready | All containers in the Pod are ready. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.containers_ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Initialized | All init containers have started successfully. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.initialized[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Ready | The Pod is able to serve requests and should be added to the load balancing pools of all matching Services. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Scheduled | The Pod has been scheduled to a node. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.scheduled[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Containers: Restarts | The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. |
Dependent item | kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Phase | The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase |
Dependent item | kube.pod.status.phase[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Uptime | Pod uptime. |
Dependent item | kube.pod.uptime[{#NAMESPACE}/{#POD}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}])-min(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}],15m))>1 |
Warning | |
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Kubernetes Pod not healthy | Pod has been in a non-ready state for longer than 10 minutes. |
count(/Kubernetes nodes by HTTP/kube.pod.status.phase[{#NAMESPACE}/{#POD}],10m, "regexp","^(1|4|5)$")>=9 |
High |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes nodes by HTTP
Overview
The template to monitor Kubernetes nodes that work without any external scripts. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API. Install the Zabbix Helm Chart (https://git.zabbix.com/projects/ZT/repos/kubernetes-helm/browse?at=refs%2Fheads%2Frelease%2F7.0) in your Kubernetes cluster.
Change the values according to the environment in the file $HOME/zabbix_values.yaml.
For example:
Enables use of Zabbix proxy
enabled: false
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the service account name. If a different release name is used.
kubectl get serviceaccounts -n monitoring
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set up the macros to filter the metrics of discovered nodes.
Requirements
Zabbix version: 7.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chart-agent.
Set up the macros to filter the metrics of discovered nodes and host creation based on host prototypes:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES}
Set up macros to filter pod metrics by namespace:
- {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable pods.
You can use the {$KUBE.NODE.FILTER.LABELS}, {$KUBE.POD.FILTER.LABELS}, {$KUBE.NODE.FILTER.ANNOTATIONS} and {$KUBE.POD.FILTER.ANNOTATIONS} macros for advanced filtering of nodes and pods by labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such a label key exists for the entity that is being filtered (it means that if you specify a key in a filter, entities which do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
Note: The discovered nodes will be created as separate hosts in Zabbix with the Linux template automatically assigned to them.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$HTTP.TLS.VERIFY} | TLS certificate verification for the script item: "none" - disabled, "peer" - verify the certificate chain and expiration, "full" - full verification. Any other value enables full verification. To override the setting for this template only, define the macro with the context "Kubernetes nodes", e.g. {$HTTP.TLS.VERIFY:"Kubernetes nodes"}. |
full |
| {$KUBE.NODES.ENDPOINT.NAME} | Kubernetes nodes endpoint name. See "kubectl -n monitoring get ep". |
zabbix-zabbix-helm-chart-agent |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES} | Filter of discoverable nodes by role. |
.* |
| {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES} | Filter to exclude discovered node by role. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.FILTER.ANNOTATIONS} | Annotations to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.NODE.FILTER.LABELS} | Labels to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.ANNOTATIONS} | Annotations to filter pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.LABELS} | Labels to filter Pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace. |
.* |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace. |
CHANGE_IF_NEEDED |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get nodes | Collecting and processing cluster nodes data via Kubernetes API. |
Script | kube.nodes |
| Get nodes check | Data collection check. |
Dependent item | kube.nodes.check Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Failed to get nodes | length(last(/Kubernetes nodes by HTTP/kube.nodes.check))>0 |
Warning |
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: Get data | Collecting and processing cluster by node [{#NAME}] data via Kubernetes API. |
Dependent item | kube.node.get[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: External IP | Typically the IP address of the node that is externally routable (available from outside the cluster). |
Dependent item | kube.node.addresses.external_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: Internal IP | Typically the IP address of the node that is routable only within the cluster. |
Dependent item | kube.node.addresses.internal_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: CPU | Allocatable CPU. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Memory | Allocatable Memory. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.allocatable.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: CPU | CPU resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Memory | Memory resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.capacity.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Disk pressure | True if pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
Dependent item | kube.node.conditions.diskpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Memory pressure | True if pressure exists on the node memory - that is, if the node memory is low; otherwise False. |
Dependent item | kube.node.conditions.memorypressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Network unavailable | True if the network for the node is not correctly configured, otherwise False. |
Dependent item | kube.node.conditions.networkunavailable[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: PID pressure | True if pressure exists on the processes - that is, if there are too many processes on the node; otherwise False. |
Dependent item | kube.node.conditions.pidpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Ready | True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
Dependent item | kube.node.conditions.ready[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Architecture | Node architecture. |
Dependent item | kube.node.info.architecture[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Container runtime | Container runtime. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
Dependent item | kube.node.info.containerruntime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kernel version | Node kernel version. |
Dependent item | kube.node.info.kernelversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kubelet version | Version of Kubelet. |
Dependent item | kube.node.info.kubeletversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: KubeProxy version | Version of KubeProxy. |
Dependent item | kube.node.info.kubeproxyversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Operating system | Node operating system. |
Dependent item | kube.node.info.operatingsystem[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: OS image | Node OS image. |
Dependent item | kube.node.info.osversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Roles | Node roles. |
Dependent item | kube.node.info.roles[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: CPU | Node CPU limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: Memory | Node Memory limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: CPU | Node CPU requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: Memory | Node Memory requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Uptime | Node uptime. |
Dependent item | kube.node.uptime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Used: Pods | Current number of pods on the node. |
Dependent item | kube.node.used.pods[{#NAME}] Preprocessing
|
Trigger prototypes for Node discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the disk size | True - pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.diskpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the node memory | True - pressure exists on the node memory - that is, if the node memory is low; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.memorypressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Network is not correctly configured | True - the network for the node is not correctly configured, otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.networkunavailable[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Pressure exists on the processes | True - pressure exists on the processes - that is, if there are too many processes on the node; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.pidpressure[{#NAME}])=1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Conditions: Is not in Ready state | False - if the node is not healthy and is not accepting pods. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.ready[{#NAME}])<>1 |
Warning | |
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 1 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Kubernetes nodes: Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.8 |
Average | ||
| Kubernetes nodes: Node [{#NAME}] has been restarted | Uptime is less than 10 minutes. |
last(/Kubernetes nodes by HTTP/kube.node.uptime[{#NAME}])<10 |
Info | |
| Kubernetes nodes: Node [{#NAME}] Used: Kubelet too many pods | Kubelet is running at capacity. |
last(/Kubernetes nodes by HTTP/kube.node.used.pods[{#NAME}])/ last(/Kubernetes nodes by HTTP/kube.node.capacity.pods[{#NAME}]) > 0.9 |
Warning |
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Get data | Collecting and processing cluster by node [{#NODE}] data via Kubernetes API. |
Dependent item | kube.pod.get[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Containers ready | All containers in the Pod are ready. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.containers_ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Initialized | All init containers have started successfully. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.initialized[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Ready | The Pod is able to serve requests and should be added to the load balancing pools of all matching Services. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.ready[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Conditions: Scheduled | The Pod has been scheduled to a node. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.scheduled[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Containers: Restarts | The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. |
Dependent item | kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Phase | The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase |
Dependent item | kube.pod.status.phase[{#NAMESPACE}/{#POD}] Preprocessing
|
| Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Uptime | Pod uptime. |
Dependent item | kube.pod.uptime[{#NAMESPACE}/{#POD}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}])-min(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#NAMESPACE}/{#POD}],15m))>1 |
Warning | |
| Kubernetes nodes: Node [{#NODE}] Namespace [{#NAMESPACE}] Pod [{#POD}] Status: Kubernetes Pod not healthy | Pod has been in a non-ready state for longer than 10 minutes. |
count(/Kubernetes nodes by HTTP/kube.pod.status.phase[{#NAMESPACE}/{#POD}],10m, "regexp","^(1|4|5)$")>=9 |
High |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes nodes by HTTP
Overview
The template to monitor Kubernetes nodes that work without any external scripts.
It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Install the Zabbix Helm Chart (https://git.zabbix.com/projects/ZT/repos/kubernetes-helm/browse?at=refs%2Fheads%2Frelease%2F6.4) in your Kubernetes cluster.
Change the values according to the environment in the file $HOME/zabbix_values.yaml.
For example:
-
Enables use of Zabbix proxy
enabled: false
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set up the macros to filter the metrics of discovered nodes
Requirements
Zabbix version: 6.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chrt-agent.
Set up the macros to filter the metrics of discovered nodes and host creation based on host prototypes:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES}
Set up macros to filter pod metrics by namespace:
- {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES}
Note, If you have a large cluster, it is highly recommended to set a filter for discoverable pods.
You can use the {$KUBE.NODE.FILTER.LABELS}, {$KUBE.POD.FILTER.LABELS}, {$KUBE.NODE.FILTER.ANNOTATIONS} and {$KUBE.POD.FILTER.ANNOTATIONS} macros for advanced filtering of nodes and pods by labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such a label key exists for the entity that is being filtered (it means that if you specify a key in a filter, entities which do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
Note, the discovered nodes will be created as separate hosts in Zabbix with the Linux template automatically assigned to them.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$KUBE.NODES.ENDPOINT.NAME} | Kubernetes nodes endpoint name. See "kubectl -n monitoring get ep". |
zabbix-zabbix-helm-chrt-agent |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES} | Filter of discoverable nodes by role. |
.* |
| {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES} | Filter to exclude discovered node by role. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.FILTER.ANNOTATIONS} | Annotations to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.NODE.FILTER.LABELS} | Labels to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.ANNOTATIONS} | Annotations to filter pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.LABELS} | Labels to filter Pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace. |
.* |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace. |
CHANGE_IF_NEEDED |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Get nodes | Collecting and processing cluster nodes data via Kubernetes API. |
Script | kube.nodes |
| Get nodes check | Data collection check. |
Dependent item | kube.nodes.check Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Failed to get nodes | length(last(/Kubernetes nodes by HTTP/kube.nodes.check))>0 |
Warning |
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: Get data | Collecting and processing cluster by node [{#NAME}] data via Kubernetes API. |
Dependent item | kube.node.get[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: External IP | Typically the IP address of the node that is externally routable (available from outside the cluster). |
Dependent item | kube.node.addresses.external_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: Internal IP | Typically the IP address of the node that is routable only within the cluster. |
Dependent item | kube.node.addresses.internal_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: CPU | Allocatable CPU. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Memory | Allocatable Memory. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.allocatable.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: CPU | CPU resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Memory | Memory resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.capacity.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Disk pressure | True if pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
Dependent item | kube.node.conditions.diskpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Memory pressure | True if pressure exists on the node memory - that is, if the node memory is low; otherwise False. |
Dependent item | kube.node.conditions.memorypressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Network unavailable | True if the network for the node is not correctly configured, otherwise False. |
Dependent item | kube.node.conditions.networkunavailable[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: PID pressure | True if pressure exists on the processes - that is, if there are too many processes on the node; otherwise False. |
Dependent item | kube.node.conditions.pidpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Ready | True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
Dependent item | kube.node.conditions.ready[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Architecture | Node architecture. |
Dependent item | kube.node.info.architecture[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Container runtime | Container runtime. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
Dependent item | kube.node.info.containerruntime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kernel version | Node kernel version. |
Dependent item | kube.node.info.kernelversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kubelet version | Version of Kubelet. |
Dependent item | kube.node.info.kubeletversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: KubeProxy version | Version of KubeProxy. |
Dependent item | kube.node.info.kubeproxyversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Operating system | Node operating system. |
Dependent item | kube.node.info.operatingsystem[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: OS image | Node OS image. |
Dependent item | kube.node.info.osversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Roles | Node roles. |
Dependent item | kube.node.info.roles[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: CPU | Node CPU limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: Memory | Node Memory limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: CPU | Node CPU requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: Memory | Node Memory requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Uptime | Node uptime. |
Dependent item | kube.node.uptime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Used: Pods | Current number of pods on the node. |
Dependent item | kube.node.used.pods[{#NAME}] Preprocessing
|
Trigger prototypes for Node discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Node [{#NAME}] Conditions: Pressure exists on the disk size | True - pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.diskpressure[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Pressure exists on the node memory | True - pressure exists on the node memory - that is, if the node memory is low; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.memorypressure[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Network is not correctly configured | True - the network for the node is not correctly configured, otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.networkunavailable[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Pressure exists on the processes | True - pressure exists on the processes - that is, if there are too many processes on the node; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.pidpressure[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Is not in Ready state | False - if the node is not healthy and is not accepting pods. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.ready[{#NAME}])<>1 |
Warning | |
| Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 1 |
Average | ||
| Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 1 |
Average | ||
| Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.8 |
Average | ||
| Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.8 |
Average | ||
| Node [{#NAME}]: Has been restarted | Uptime is less than 10 minutes. |
last(/Kubernetes nodes by HTTP/kube.node.uptime[{#NAME}])<10 |
Info | |
| Node [{#NAME}] Used: Kubelet too many pods | Kubelet is running at capacity. |
last(/Kubernetes nodes by HTTP/kube.node.used.pods[{#NAME}])/ last(/Kubernetes nodes by HTTP/kube.node.capacity.pods[{#NAME}]) > 0.9 |
Warning |
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NODE}] Pod [{#POD}]: Get data | Collecting and processing cluster by node [{#NODE}] data via Kubernetes API. |
Dependent item | kube.pod.get[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Containers ready | All containers in the Pod are ready. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.containers_ready[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Initialized | All init containers have started successfully. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.initialized[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Ready | The Pod is able to serve requests and should be added to the load balancing pools of all matching Services. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.ready[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Scheduled | The Pod has been scheduled to a node. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.scheduled[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Containers: Restarts | The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. |
Dependent item | kube.pod.containers.restartcount[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Status: Phase | The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase |
Dependent item | kube.pod.status.phase[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Uptime | Pod uptime. |
Dependent item | kube.pod.uptime[{#POD}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Node [{#NODE}] Pod [{#POD}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#POD}])-min(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#POD}],15m))>1 |
Warning | |
| Node [{#NODE}] Pod [{#POD}] Status: Kubernetes Pod not healthy | Pod has been in a non-ready state for longer than 10 minutes. |
count(/Kubernetes nodes by HTTP/kube.pod.status.phase[{#POD}],10m, "regexp","^(1|4|5)$")>=9 |
High |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes nodes by HTTP
Overview
For Zabbix version: 6.2 and higher.
The template to monitor Kubernetes nodes that work without any external scripts.
It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Install the Zabbix Helm Chart (https://git.zabbix.com/projects/ZT/repos/kubernetes-helm/browse?at=refs%2Fheads%2Frelease%2F6.2) in your Kubernetes cluster.
Set the {$KUBE.API.ENDPOINT.URL} such as <scheme>://<host>:<port>/api.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chrt-agent.
Set up the macros to filter the metrics of discovered nodes
This template was tested on:
- Kubernetes, version 1.19
Setup
See Zabbix template operation for basic instructions.
Install the Zabbix Helm Chart in your Kubernetes cluster.
Set the {$KUBE.API.ENDPOINT.URL} such as <scheme>://<host>:<port>/api.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chrt-agent.
Set up the macros to filter the metrics of discovered nodes:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES}
Set up the macros to filter host creation based on host prototypes:
- {$KUBE.LLD.FILTER.NODE_HOST.MATCHES}
- {$KUBE.LLD.FILTER.NODE_HOST.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE_HOST.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE_HOST.ROLE.NOT_MATCHES}
Set up macros to filter pod metrics by namespace:
- {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES}
Note, If you have a large cluster, it is highly recommended to set a filter for discoverable pods.
You can use {$KUBE.NODE.FILTER.LABELS}, {$KUBE.POD.FILTER.LABELS}, {$KUBE.NODE.FILTER.ANNOTATIONS} and {$KUBE.POD.FILTER.ANNOTATIONS} macros for advanced filtering nodes and pods by labels and annotations. Macro values are specified separated by commas and must have the key/value form with support for regular expressions in the value.
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the nodes 5-25 without the "ingress" role will be discovered.
See documentation for details:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
Note, the discovered nodes will be created as separate hosts in Zabbix with the Linux template automatically assigned to them.
Zabbix configuration
No specific Zabbix configuration is required.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.ENDPOINT.URL} | Kubernetes API endpoint URL in the format |
https://localhost:6443/api |
| {$KUBE.API.TOKEN} | Service account bearer token |
`` |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES} | Filter of discoverable nodes by role |
.* |
| {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES} | Filter to exclude discovered node by role |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE_HOST.MATCHES} | Filter of discoverable cluster nodes |
.* |
| {$KUBE.LLD.FILTER.NODE_HOST.NOT_MATCHES} | Filter to exclude discovered cluster nodes |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE_HOST.ROLE.MATCHES} | Filter of discoverable nodes hosts by role |
.* |
| {$KUBE.LLD.FILTER.NODE_HOST.ROLE.NOT_MATCHES} | Filter to exclude discovered cluster nodes by role |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace |
.* |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.FILTER.ANNOTATIONS} | Annotations to filter nodes (regex in values are supported) |
`` |
| {$KUBE.NODE.FILTER.LABELS} | Labels to filter nodes (regex in values are supported) |
`` |
| {$KUBE.NODES.ENDPOINT.NAME} | Kubernetes nodes endpoint name. See kubectl -n monitoring get ep |
zabbix-zabbix-helm-chrt-agent |
| {$KUBE.POD.FILTER.ANNOTATIONS} | Annotations to filter pods (regex in values are supported) |
`` |
| {$KUBE.POD.FILTER.LABELS} | Labels to filter Pods (regex in values are supported) |
`` |
Template links
There are no template links in this template.
Discovery rules
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Cluster node discovery | - |
DEPENDENT | kube.node_host.discovery Filter: AND- {#NAME} MATCHES_REGEX - {#NAME} NOT_MATCHES_REGEX - {#ROLES} MATCHES_REGEX - {#ROLES} NOT_MATCHES_REGEX |
| Node discovery | - |
DEPENDENT | kube.node.discovery Filter: AND- {#NAME} MATCHES_REGEX - {#NAME} NOT_MATCHES_REGEX - {#ROLES} MATCHES_REGEX - {#ROLES} NOT_MATCHES_REGEX |
| Pod discovery | - |
DEPENDENT | kube.pod.discovery Preprocessing: - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NODE} MATCHES_REGEX - {#NODE} NOT_MATCHES_REGEX - {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
Items collected
| Group | Name | Description | Type | Key and additional info |
|---|---|---|---|---|
| Kubernetes | Kubernetes: Get nodes | Collecting and processing cluster nodes data via Kubernetes API. |
SCRIPT | kube.nodes Expression: The text is too long. Please see the template. |
| Kubernetes | Get nodes check | Data collection check. |
DEPENDENT | kube.nodes.check Preprocessing: - JSONPATH: ⛔️ON_FAIL: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node LLD | Generation of data for node discovery rules. |
DEPENDENT | kube.nodes.lld Preprocessing: - JAVASCRIPT: `function parseFilters(filter) { var pairs = {}; filter.split(/\s*,\s*/).forEach(function (kv) { if (/([\w.-]+/[\w.-]+):\s*.+/.test(kv)) { var pair = kv.split(/\s*:\s*/); pairs[pair[0]] = pair[1]; } }); return pairs; } function filter(name, data, filters) { var filtered = true; if (typeof data === 'object') { Object.keys(filters).some(function (filter) { var exclude = filter.match(/^!(.+)/); if (filter in data |
| Kubernetes | Node [{#NAME}]: Get data | Collecting and processing cluster by node [{#NAME}] data via Kubernetes API. |
DEPENDENT | kube.node.get[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Addresses: External IP | Typically the IP address of the node that is externally routable (available from outside the cluster). |
DEPENDENT | kube.node.addresses.external_ip[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Addresses: Internal IP | Typically the IP address of the node that is routable only within the cluster. |
DEPENDENT | kube.node.addresses.internal_ip[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Allocatable: CPU | Allocatable CPU. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
DEPENDENT | kube.node.allocatable.cpu[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Allocatable: Memory | Allocatable Memory. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
DEPENDENT | kube.node.allocatable.memory[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Allocatable: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
DEPENDENT | kube.node.allocatable.pods[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Capacity: CPU | CPU resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
DEPENDENT | kube.node.capacity.cpu[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Capacity: Memory | Memory resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
DEPENDENT | kube.node.capacity.memory[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Capacity: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
DEPENDENT | kube.node.capacity.pods[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Conditions: Disk pressure | True if pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
DEPENDENT | kube.node.conditions.diskpressure[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NAME}] Conditions: Memory pressure | True if pressure exists on the node memory - that is, if the node memory is low; otherwise False. |
DEPENDENT | kube.node.conditions.memorypressure[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NAME}] Conditions: Network unavailable | True if the network for the node is not correctly configured, otherwise False. |
DEPENDENT | kube.node.conditions.networkunavailable[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NAME}] Conditions: PID pressure | True if pressure exists on the processes - that is, if there are too many processes on the node; otherwise False. |
DEPENDENT | kube.node.conditions.pidpressure[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NAME}] Conditions: Ready | True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
DEPENDENT | kube.node.conditions.ready[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NAME}] Info: Architecture | Node architecture. |
DEPENDENT | kube.node.info.architecture[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Info: Container runtime | Container runtime. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
DEPENDENT | kube.node.info.containerruntime[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Info: Kernel version | Node kernel version. |
DEPENDENT | kube.node.info.kernelversion[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Info: Kubelet version | Version of Kubelet. |
DEPENDENT | kube.node.info.kubeletversion[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Info: KubeProxy version | Version of KubeProxy. |
DEPENDENT | kube.node.info.kubeproxyversion[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Info: Operating system | Node operating system. |
DEPENDENT | kube.node.info.operatingsystem[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Info: OS image | Node OS image. |
DEPENDENT | kube.node.info.osversion[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Info: Roles | Node roles. |
DEPENDENT | kube.node.info.roles[{#NAME}] Preprocessing: - JSONPATH: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Node [{#NAME}] Limits: CPU | Node CPU limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
DEPENDENT | kube.node.limits.cpu[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Limits: Memory | Node Memory limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
DEPENDENT | kube.node.limits.memory[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Requests: CPU | Node CPU requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
DEPENDENT | kube.node.requests.cpu[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Requests: Memory | Node Memory requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
DEPENDENT | kube.node.requests.memory[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NAME}] Uptime | Node uptime. |
DEPENDENT | kube.node.uptime[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: |
| Kubernetes | Node [{#NAME}] Used: Pods | Current number of pods on the node. |
DEPENDENT | kube.node.used.pods[{#NAME}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NODE}] Pod [{#POD}]: Get data | Collecting and processing cluster by node [{#NODE}] data via Kubernetes API. |
DEPENDENT | kube.pod.get[{#POD}] Preprocessing: - JSONPATH: |
| Kubernetes | Node [{#NODE}] Pod [{#POD}] Conditions: Containers ready | All containers in the Pod are ready. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
DEPENDENT | kube.pod.conditions.containers_ready[{#POD}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NODE}] Pod [{#POD}] Conditions: Initialized | All init containers have started successfully. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
DEPENDENT | kube.pod.conditions.initialized[{#POD}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NODE}] Pod [{#POD}] Conditions: Ready | The Pod is able to serve requests and should be added to the load balancing pools of all matching Services. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
DEPENDENT | kube.pod.conditions.ready[{#POD}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NODE}] Pod [{#POD}] Conditions: Scheduled | The Pod has been scheduled to a node. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
DEPENDENT | kube.pod.conditions.scheduled[{#POD}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['True', 'False', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NODE}] Pod [{#POD}] Containers: Restarts | The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. |
DEPENDENT | kube.pod.containers.restartcount[{#POD}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
| Kubernetes | Node [{#NODE}] Pod [{#POD}] Status: Phase | The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase |
DEPENDENT | kube.pod.status.phase[{#POD}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: `return ['Pending', 'Running', 'Succeeded', 'Failed', 'Unknown'].indexOf(value) + 1 |
| Kubernetes | Node [{#NODE}] Pod [{#POD}] Uptime | Pod uptime. |
DEPENDENT | kube.pod.uptime[{#POD}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: - JAVASCRIPT: |
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Failed to get nodes | - |
length(last(/Kubernetes nodes by HTTP/kube.nodes.check))>0 |
WARNING | |
| Node [{#NAME}] Conditions: Pressure exists on the disk size | True - pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.diskpressure[{#NAME}])=1 |
WARNING | |
| Node [{#NAME}] Conditions: Pressure exists on the node memory | True - pressure exists on the node memory - that is, if the node memory is low; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.memorypressure[{#NAME}])=1 |
WARNING | |
| Node [{#NAME}] Conditions: Network is not correctly configured | True - the network for the node is not correctly configured, otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.networkunavailable[{#NAME}])=1 |
WARNING | |
| Node [{#NAME}] Conditions: Pressure exists on the processes | True - pressure exists on the processes - that is, if there are too many processes on the node; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.pidpressure[{#NAME}])=1 |
WARNING | |
| Node [{#NAME}] Conditions: Is not in Ready state | False - if the node is not healthy and is not accepting pods. Unknown - if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
last(/Kubernetes nodes by HTTP/kube.node.conditions.ready[{#NAME}])<>1 |
WARNING | |
| Node [{#NAME}] Limits: Total CPU limits are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.9 |
WARNING | Depends on: - Node [{#NAME}] Limits: Total CPU limits are too high |
| Node [{#NAME}] Limits: Total CPU limits are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 1 |
AVERAGE | |
| Node [{#NAME}] Limits: Total memory limits are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.9 |
WARNING | Depends on: - Node [{#NAME}] Limits: Total memory limits are too high |
| Node [{#NAME}] Limits: Total memory limits are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 1 |
AVERAGE | |
| Node [{#NAME}] Requests: Total CPU requests are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.5 |
WARNING | Depends on: - Node [{#NAME}] Requests: Total CPU requests are too high |
| Node [{#NAME}] Requests: Total CPU requests are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.8 |
AVERAGE | |
| Node [{#NAME}] Requests: Total memory requests are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.5 |
WARNING | Depends on: - Node [{#NAME}] Requests: Total memory requests are too high |
| Node [{#NAME}] Requests: Total memory requests are too high | - |
last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.8 |
AVERAGE | |
| Node [{#NAME}]: Has been restarted | Uptime is less than 10 minutes |
last(/Kubernetes nodes by HTTP/kube.node.uptime[{#NAME}])<10 |
INFO | |
| Node [{#NAME}] Used: Kubelet too many pods | Kubelet is running at capacity. |
last(/Kubernetes nodes by HTTP/kube.node.used.pods[{#NAME}])/ last(/Kubernetes nodes by HTTP/kube.node.capacity.pods[{#NAME}]) > 0.9 |
WARNING | |
| Node [{#NODE}] Pod [{#POD}]: Pod is crash looping | Pos restarts more than 2 times in the last 3 minutes. |
(last(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#POD}])-min(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#POD}],3m))>2 |
WARNING | |
| Node [{#NODE}] Pod [{#POD}] Status: Kubernetes Pod not healthy | Pod has been in a non-ready state for longer than 10 minutes. |
`count(/Kubernetes nodes by HTTP/kube.pod.status.phase[{#POD}],10m, "regexp","^(1 | 4 | 5)$")>=9` |
Feedback
Please report any issues with the template at https://support.zabbix.com.
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums.
Kubernetes nodes by HTTP
Overview
The template to monitor Kubernetes nodes that work without any external scripts.
It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Install the Zabbix Helm Chart (https://git.zabbix.com/projects/ZT/repos/kubernetes-helm/browse?at=refs%2Fheads%2Frelease%2F6.0) in your Kubernetes cluster.
Change the values according to the environment in the file $HOME/zabbix_values.yaml.
For example:
-
Enables use of Zabbix proxy
enabled: false
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set up the macros to filter the metrics of discovered nodes
Requirements
Zabbix version: 6.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.NODES.ENDPOINT.NAME} with Zabbix agent's endpoint name. See kubectl -n monitoring get ep. Default: zabbix-zabbix-helm-chrt-agent.
Set up the macros to filter the metrics of discovered nodes and host creation based on host prototypes:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES}
Set up macros to filter pod metrics by namespace:
- {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES}
Note, If you have a large cluster, it is highly recommended to set a filter for discoverable pods.
You can use the {$KUBE.NODE.FILTER.LABELS}, {$KUBE.POD.FILTER.LABELS}, {$KUBE.NODE.FILTER.ANNOTATIONS} and {$KUBE.POD.FILTER.ANNOTATIONS} macros for advanced filtering of nodes and pods by labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such a label key exists for the entity that is being filtered (it means that if you specify a key in a filter, entities which do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
Note, the discovered nodes will be created as separate hosts in Zabbix with the Linux template automatically assigned to them.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$KUBE.NODES.ENDPOINT.NAME} | Kubernetes nodes endpoint name. See "kubectl -n monitoring get ep". |
zabbix-zabbix-helm-chrt-agent |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.ROLE.MATCHES} | Filter of discoverable nodes by role. |
.* |
| {$KUBE.LLD.FILTER.NODE.ROLE.NOT_MATCHES} | Filter to exclude discovered node by role. |
CHANGE_IF_NEEDED |
| {$KUBE.NODE.FILTER.ANNOTATIONS} | Annotations to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.NODE.FILTER.LABELS} | Labels to filter nodes (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.ANNOTATIONS} | Annotations to filter pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.POD.FILTER.LABELS} | Labels to filter Pods (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.POD.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace. |
.* |
| {$KUBE.LLD.FILTER.POD.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace. |
CHANGE_IF_NEEDED |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Get nodes | Collecting and processing cluster nodes data via Kubernetes API. |
Script | kube.nodes |
| Get nodes check | Data collection check. |
Dependent item | kube.nodes.check Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Failed to get nodes | length(last(/Kubernetes nodes by HTTP/kube.nodes.check))>0 |
Warning |
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: Get data | Collecting and processing cluster by node [{#NAME}] data via Kubernetes API. |
Dependent item | kube.node.get[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: External IP | Typically the IP address of the node that is externally routable (available from outside the cluster). |
Dependent item | kube.node.addresses.external_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Addresses: Internal IP | Typically the IP address of the node that is routable only within the cluster. |
Dependent item | kube.node.addresses.internal_ip[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: CPU | Allocatable CPU. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Memory | Allocatable Memory. 'Allocatable' on a Kubernetes node is defined as the amount of compute resources that are available for pods. The scheduler does not over-subscribe 'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now. |
Dependent item | kube.node.allocatable.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Allocatable: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.allocatable.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: CPU | CPU resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Memory | Memory resource capacity. https://kubernetes.io/docs/concepts/architecture/nodes/#capacity |
Dependent item | kube.node.capacity.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Capacity: Pods | https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ |
Dependent item | kube.node.capacity.pods[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Disk pressure | True if pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
Dependent item | kube.node.conditions.diskpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Memory pressure | True if pressure exists on the node memory - that is, if the node memory is low; otherwise False. |
Dependent item | kube.node.conditions.memorypressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Network unavailable | True if the network for the node is not correctly configured, otherwise False. |
Dependent item | kube.node.conditions.networkunavailable[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: PID pressure | True if pressure exists on the processes - that is, if there are too many processes on the node; otherwise False. |
Dependent item | kube.node.conditions.pidpressure[{#NAME}] Preprocessing
|
| Node [{#NAME}] Conditions: Ready | True if the node is healthy and ready to accept pods, False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 40 seconds). |
Dependent item | kube.node.conditions.ready[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Architecture | Node architecture. |
Dependent item | kube.node.info.architecture[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Container runtime | Container runtime. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ |
Dependent item | kube.node.info.containerruntime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kernel version | Node kernel version. |
Dependent item | kube.node.info.kernelversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Kubelet version | Version of Kubelet. |
Dependent item | kube.node.info.kubeletversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: KubeProxy version | Version of KubeProxy. |
Dependent item | kube.node.info.kubeproxyversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Operating system | Node operating system. |
Dependent item | kube.node.info.operatingsystem[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: OS image | Node OS image. |
Dependent item | kube.node.info.osversion[{#NAME}] Preprocessing
|
| Node [{#NAME}] Info: Roles | Node roles. |
Dependent item | kube.node.info.roles[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: CPU | Node CPU limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Limits: Memory | Node Memory limits. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.limits.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: CPU | Node CPU requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.cpu[{#NAME}] Preprocessing
|
| Node [{#NAME}] Requests: Memory | Node Memory requests. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ |
Dependent item | kube.node.requests.memory[{#NAME}] Preprocessing
|
| Node [{#NAME}] Uptime | Node uptime. |
Dependent item | kube.node.uptime[{#NAME}] Preprocessing
|
| Node [{#NAME}] Used: Pods | Current number of pods on the node. |
Dependent item | kube.node.used.pods[{#NAME}] Preprocessing
|
Trigger prototypes for Node discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Node [{#NAME}] Conditions: Pressure exists on the disk size | True - pressure exists on the disk size - that is, if the disk capacity is low; otherwise False. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.diskpressure[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Pressure exists on the node memory | True - pressure exists on the node memory - that is, if the node memory is low; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.memorypressure[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Network is not correctly configured | True - the network for the node is not correctly configured, otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.networkunavailable[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Pressure exists on the processes | True - pressure exists on the processes - that is, if there are too many processes on the node; otherwise False |
last(/Kubernetes nodes by HTTP/kube.node.conditions.pidpressure[{#NAME}])=1 |
Warning | |
| Node [{#NAME}] Conditions: Is not in Ready state | False - if the node is not healthy and is not accepting pods. |
last(/Kubernetes nodes by HTTP/kube.node.conditions.ready[{#NAME}])<>1 |
Warning | |
| Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Node [{#NAME}] Limits: Total CPU limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 1 |
Average | ||
| Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.9 |
Warning | Depends on:
|
|
| Node [{#NAME}] Limits: Total memory limits are too high | last(/Kubernetes nodes by HTTP/kube.node.limits.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 1 |
Average | ||
| Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Node [{#NAME}] Requests: Total CPU requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.cpu[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.cpu[{#NAME}]) > 0.8 |
Average | ||
| Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.5 |
Warning | Depends on:
|
|
| Node [{#NAME}] Requests: Total memory requests are too high | last(/Kubernetes nodes by HTTP/kube.node.requests.memory[{#NAME}]) / last(/Kubernetes nodes by HTTP/kube.node.allocatable.memory[{#NAME}]) > 0.8 |
Average | ||
| Node [{#NAME}]: Has been restarted | Uptime is less than 10 minutes. |
last(/Kubernetes nodes by HTTP/kube.node.uptime[{#NAME}])<10 |
Info | |
| Node [{#NAME}] Used: Kubelet too many pods | Kubelet is running at capacity. |
last(/Kubernetes nodes by HTTP/kube.node.used.pods[{#NAME}])/ last(/Kubernetes nodes by HTTP/kube.node.capacity.pods[{#NAME}]) > 0.9 |
Warning |
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NODE}] Pod [{#POD}]: Get data | Collecting and processing cluster by node [{#NODE}] data via Kubernetes API. |
Dependent item | kube.pod.get[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Containers ready | All containers in the Pod are ready. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.containers_ready[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Initialized | All init containers have started successfully. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.initialized[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Ready | The Pod is able to serve requests and should be added to the load balancing pools of all matching Services. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.ready[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Conditions: Scheduled | The Pod has been scheduled to a node. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-conditions |
Dependent item | kube.pod.conditions.scheduled[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Containers: Restarts | The number of times the container has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. |
Dependent item | kube.pod.containers.restartcount[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Status: Phase | The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase |
Dependent item | kube.pod.status.phase[{#POD}] Preprocessing
|
| Node [{#NODE}] Pod [{#POD}] Uptime | Pod uptime. |
Dependent item | kube.pod.uptime[{#POD}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Node [{#NODE}] Pod [{#POD}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#POD}])-min(/Kubernetes nodes by HTTP/kube.pod.containers.restartcount[{#POD}],15m))>1 |
Warning | |
| Node [{#NODE}] Pod [{#POD}] Status: Kubernetes Pod not healthy | Pod has been in a non-ready state for longer than 10 minutes. |
count(/Kubernetes nodes by HTTP/kube.pod.status.phase[{#POD}],10m, "regexp","^(1|4|5)$")>=9 |
High |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Scheduler by HTTP
Overview
The template to monitor Kubernetes Scheduler by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Scheduler by HTTP - collects metrics by HTTP agent from Scheduler /metrics endpoint.
Requirements
Zabbix version: 7.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes Scheduler 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.SCHEDULER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: You might need to set the --binding-address option for Scheduler to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-scheduler.yaml
Note: Some metrics may not be collected depending on your Kubernetes Scheduler instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.SCHEDULER.SERVER.URL} | Kubernetes Scheduler metrics endpoint URL. |
https://localhost:10259/metrics |
| {$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.SCHEDULER.UNSCHEDULABLE} | Maximum number of scheduling failures with 'unschedulable' used for trigger. |
2 |
| {$KUBE.SCHEDULER.ERROR} | Maximum number of scheduling failures with 'error' used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get Scheduler metrics | Get raw metrics from Scheduler instance /metrics endpoint. |
HTTP agent | kubernetes.scheduler.get_metrics Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.scheduler.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.scheduler.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.scheduler.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.scheduler.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.scheduler.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.scheduler.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.scheduler.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_500.rate Preprocessing
|
| Schedule attempts: scheduled | Number of attempts to schedule pods with result "scheduled" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.scheduled.rate Preprocessing
|
| Schedule attempts: unschedulable | Number of attempts to schedule pods with result "unschedulable" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate Preprocessing
|
| Schedule attempts: error | Number of attempts to schedule pods with result "error" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.error.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler: Too many REST Client errors | "Kubernetes Scheduler REST Client requests is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.client_http_requests_500.rate,5m)>{$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} |
Warning | |
| Kubernetes Scheduler: Too many unschedulable pods | Number of attempts to schedule pods with 'unschedulable' result is too high. 'unschedulable' means a pod could not be scheduled. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate,5m)>{$KUBE.SCHEDULER.UNSCHEDULABLE} |
Warning | |
| Kubernetes Scheduler: Too many schedule attempts with errors | Number of attempts to schedule pods with 'error' result is too high. 'error' means an internal scheduler problem. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.error.rate,5m)>{$KUBE.SCHEDULER.ERROR} |
Warning |
LLD rule Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm histogram | Discovery raw data of scheduling algorithm latency. |
Dependent item | kubernetes.scheduler.scheduling_algorithm.discovery Preprocessing
|
Item prototypes for Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm duration bucket, {#LE} | Scheduling algorithm latency in seconds. |
Dependent item | kubernetes.scheduler.scheduling_algorithm_duration[{#LE}] Preprocessing
|
| Scheduling algorithm duration, p90 | 90 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p90[{#SINGLETON}] |
| Scheduling algorithm duration, p95 | 95 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p95[{#SINGLETON}] |
| Scheduling algorithm duration, p99 | 99 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p99[{#SINGLETON}] |
| Scheduling algorithm duration, p50 | 50 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p50[{#SINGLETON}] |
LLD rule Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding histogram | Discovery raw data of binding latency. |
Dependent item | kubernetes.scheduler.binding.discovery Preprocessing
|
Item prototypes for Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding duration bucket, {#LE} | Binding latency in seconds. |
Dependent item | kubernetes.scheduler.binding_duration[{#LE}] Preprocessing
|
| Binding duration, p90 | 90 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p90[{#SINGLETON}] |
| Binding duration, p95 | 99 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p95[{#SINGLETON}] |
| Binding duration, p99 | 95 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p99[{#SINGLETON}] |
| Binding duration, p50 | 50 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p50[{#SINGLETON}] |
LLD rule e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| e2e scheduling histogram | Discovery raw data and percentile items of e2e scheduling latency. |
Dependent item | kubernetes.controller.e2e_scheduling.discovery Preprocessing
|
Item prototypes for e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#RESULT}"]: e2e scheduling seconds bucket, {#LE} | E2e scheduling latency in seconds (scheduling algorithm + binding) |
Dependent item | kubernetes.scheduler.e2e_scheduling_bucket[{#LE},"{#RESULT}"] Preprocessing
|
| ["{#RESULT}"]: e2e scheduling, p50 | 50 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p50["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p90 | 90 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p90["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p95 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p95["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p99 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p99["{#RESULT}"] |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Scheduler by HTTP
Overview
The template to monitor Kubernetes Scheduler by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Scheduler by HTTP - collects metrics by HTTP agent from Scheduler /metrics endpoint.
Requirements
Zabbix version: 7.2 and higher.
Tested versions
This template has been tested on:
- Kubernetes Scheduler 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.SCHEDULER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. You might need to set the --binding-address option for Scheduler to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-scheduler.yaml
NOTE. Some metrics may not be collected depending on your Kubernetes Scheduler instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.SCHEDULER.SERVER.URL} | Kubernetes Scheduler metrics endpoint URL. |
https://localhost:10259/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.SCHEDULER.UNSCHEDULABLE} | Maximum number of scheduling failures with 'unschedulable' used for trigger. |
2 |
| {$KUBE.SCHEDULER.ERROR} | Maximum number of scheduling failures with 'error' used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get Scheduler metrics | Get raw metrics from Scheduler instance /metrics endpoint. |
HTTP agent | kubernetes.scheduler.get_metrics Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.scheduler.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.scheduler.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.scheduler.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.scheduler.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.scheduler.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.scheduler.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.scheduler.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_500.rate Preprocessing
|
| Schedule attempts: scheduled | Number of attempts to schedule pods with result "scheduled" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.scheduled.rate Preprocessing
|
| Schedule attempts: unschedulable | Number of attempts to schedule pods with result "unschedulable" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate Preprocessing
|
| Schedule attempts: error | Number of attempts to schedule pods with result "error" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.error.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler: Too many REST Client errors | "Kubernetes Scheduler REST Client requests is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.client_http_requests_500.rate,5m)>{$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} |
Warning | |
| Kubernetes Scheduler: Too many unschedulable pods | Number of attempts to schedule pods with 'unschedulable' result is too high. 'unschedulable' means a pod could not be scheduled. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate,5m)>{$KUBE.SCHEDULER.UNSCHEDULABLE} |
Warning | |
| Kubernetes Scheduler: Too many schedule attempts with errors | Number of attempts to schedule pods with 'error' result is too high. 'error' means an internal scheduler problem. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.error.rate,5m)>{$KUBE.SCHEDULER.ERROR} |
Warning |
LLD rule Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm histogram | Discovery raw data of scheduling algorithm latency. |
Dependent item | kubernetes.scheduler.scheduling_algorithm.discovery Preprocessing
|
Item prototypes for Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm duration bucket, {#LE} | Scheduling algorithm latency in seconds. |
Dependent item | kubernetes.scheduler.scheduling_algorithm_duration[{#LE}] Preprocessing
|
| Scheduling algorithm duration, p90 | 90 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p90[{#SINGLETON}] |
| Scheduling algorithm duration, p95 | 95 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p95[{#SINGLETON}] |
| Scheduling algorithm duration, p99 | 99 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p99[{#SINGLETON}] |
| Scheduling algorithm duration, p50 | 50 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p50[{#SINGLETON}] |
LLD rule Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding histogram | Discovery raw data of binding latency. |
Dependent item | kubernetes.scheduler.binding.discovery Preprocessing
|
Item prototypes for Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding duration bucket, {#LE} | Binding latency in seconds. |
Dependent item | kubernetes.scheduler.binding_duration[{#LE}] Preprocessing
|
| Binding duration, p90 | 90 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p90[{#SINGLETON}] |
| Binding duration, p95 | 99 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p95[{#SINGLETON}] |
| Binding duration, p99 | 95 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p99[{#SINGLETON}] |
| Binding duration, p50 | 50 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p50[{#SINGLETON}] |
LLD rule e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| e2e scheduling histogram | Discovery raw data and percentile items of e2e scheduling latency. |
Dependent item | kubernetes.controller.e2e_scheduling.discovery Preprocessing
|
Item prototypes for e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#RESULT}"]: e2e scheduling seconds bucket, {#LE} | E2e scheduling latency in seconds (scheduling algorithm + binding) |
Dependent item | kubernetes.scheduler.e2e_scheduling_bucket[{#LE},"{#RESULT}"] Preprocessing
|
| ["{#RESULT}"]: e2e scheduling, p50 | 50 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p50["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p90 | 90 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p90["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p95 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p95["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p99 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p99["{#RESULT}"] |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Scheduler by HTTP
Overview
The template to monitor Kubernetes Scheduler by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Scheduler by HTTP - collects metrics by HTTP agent from Scheduler /metrics endpoint.
Requirements
Zabbix version: 7.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes Scheduler 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.SCHEDULER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: You might need to set the --binding-address option for Scheduler to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-scheduler.yaml
Note: Some metrics may not be collected depending on your Kubernetes Scheduler instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.SCHEDULER.SERVER.URL} | Kubernetes Scheduler metrics endpoint URL. |
https://localhost:10259/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.SCHEDULER.UNSCHEDULABLE} | Maximum number of scheduling failures with 'unschedulable' used for trigger. |
2 |
| {$KUBE.SCHEDULER.ERROR} | Maximum number of scheduling failures with 'error' used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get Scheduler metrics | Get raw metrics from Scheduler instance /metrics endpoint. |
HTTP agent | kubernetes.scheduler.get_metrics Preprocessing
|
| Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.scheduler.process_virtual_memory_bytes Preprocessing
|
| Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.scheduler.process_resident_memory_bytes Preprocessing
|
| CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.scheduler.cpu.util Preprocessing
|
| Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.scheduler.go_goroutines Preprocessing
|
| Go threads | Number of OS threads created. |
Dependent item | kubernetes.scheduler.go_threads Preprocessing
|
| Fds open | Number of open file descriptors. |
Dependent item | kubernetes.scheduler.open_fds Preprocessing
|
| Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.scheduler.max_fds Preprocessing
|
| REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_200.rate Preprocessing
|
| REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_300.rate Preprocessing
|
| REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_400.rate Preprocessing
|
| REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_500.rate Preprocessing
|
| Schedule attempts: scheduled | Number of attempts to schedule pods with result "scheduled" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.scheduled.rate Preprocessing
|
| Schedule attempts: unschedulable | Number of attempts to schedule pods with result "unschedulable" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate Preprocessing
|
| Schedule attempts: error | Number of attempts to schedule pods with result "error" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.error.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler: Too many REST Client errors | "Kubernetes Scheduler REST Client requests is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.client_http_requests_500.rate,5m)>{$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} |
Warning | |
| Kubernetes Scheduler: Too many unschedulable pods | Number of attempts to schedule pods with 'unschedulable' result is too high. 'unschedulable' means a pod could not be scheduled. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate,5m)>{$KUBE.SCHEDULER.UNSCHEDULABLE} |
Warning | |
| Kubernetes Scheduler: Too many schedule attempts with errors | Number of attempts to schedule pods with 'error' result is too high. 'error' means an internal scheduler problem. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.error.rate,5m)>{$KUBE.SCHEDULER.ERROR} |
Warning |
LLD rule Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm histogram | Discovery raw data of scheduling algorithm latency. |
Dependent item | kubernetes.scheduler.scheduling_algorithm.discovery Preprocessing
|
Item prototypes for Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm duration bucket, {#LE} | Scheduling algorithm latency in seconds. |
Dependent item | kubernetes.scheduler.scheduling_algorithm_duration[{#LE}] Preprocessing
|
| Scheduling algorithm duration, p90 | 90 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p90[{#SINGLETON}] |
| Scheduling algorithm duration, p95 | 95 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p95[{#SINGLETON}] |
| Scheduling algorithm duration, p99 | 99 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p99[{#SINGLETON}] |
| Scheduling algorithm duration, p50 | 50 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p50[{#SINGLETON}] |
LLD rule Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding histogram | Discovery raw data of binding latency. |
Dependent item | kubernetes.scheduler.binding.discovery Preprocessing
|
Item prototypes for Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding duration bucket, {#LE} | Binding latency in seconds. |
Dependent item | kubernetes.scheduler.binding_duration[{#LE}] Preprocessing
|
| Binding duration, p90 | 90 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p90[{#SINGLETON}] |
| Binding duration, p95 | 99 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p95[{#SINGLETON}] |
| Binding duration, p99 | 95 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p99[{#SINGLETON}] |
| Binding duration, p50 | 50 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p50[{#SINGLETON}] |
LLD rule e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| e2e scheduling histogram | Discovery raw data and percentile items of e2e scheduling latency. |
Dependent item | kubernetes.controller.e2e_scheduling.discovery Preprocessing
|
Item prototypes for e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ["{#RESULT}"]: e2e scheduling seconds bucket, {#LE} | E2e scheduling latency in seconds (scheduling algorithm + binding) |
Dependent item | kubernetes.scheduler.e2e_scheduling_bucket[{#LE},"{#RESULT}"] Preprocessing
|
| ["{#RESULT}"]: e2e scheduling, p50 | 50 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p50["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p90 | 90 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p90["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p95 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p95["{#RESULT}"] |
| ["{#RESULT}"]: e2e scheduling, p99 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p99["{#RESULT}"] |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Scheduler by HTTP
Overview
The template to monitor Kubernetes Scheduler by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Scheduler by HTTP - collects metrics by HTTP agent from Scheduler /metrics endpoint.
Requirements
Zabbix version: 6.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes Scheduler 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.SCHEDULER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. You might need to set the --binding-address option for Scheduler to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-scheduler.yaml
NOTE. Some metrics may not be collected depending on your Kubernetes Scheduler instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.SCHEDULER.SERVER.URL} | Kubernetes Scheduler metrics endpoint URL. |
https://localhost:10259/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.SCHEDULER.UNSCHEDULABLE} | Maximum number of scheduling failures with 'unschedulable' used for trigger. |
2 |
| {$KUBE.SCHEDULER.ERROR} | Maximum number of scheduling failures with 'error' used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: Get Scheduler metrics | Get raw metrics from Scheduler instance /metrics endpoint. |
HTTP agent | kubernetes.scheduler.get_metrics Preprocessing
|
| Kubernetes Scheduler: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.scheduler.process_virtual_memory_bytes Preprocessing
|
| Kubernetes Scheduler: Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.scheduler.process_resident_memory_bytes Preprocessing
|
| Kubernetes Scheduler: CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.scheduler.cpu.util Preprocessing
|
| Kubernetes Scheduler: Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.scheduler.go_goroutines Preprocessing
|
| Kubernetes Scheduler: Go threads | Number of OS threads created. |
Dependent item | kubernetes.scheduler.go_threads Preprocessing
|
| Kubernetes Scheduler: Fds open | Number of open file descriptors. |
Dependent item | kubernetes.scheduler.open_fds Preprocessing
|
| Kubernetes Scheduler: Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.scheduler.max_fds Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_200.rate Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_300.rate Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_400.rate Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_500.rate Preprocessing
|
| Kubernetes Scheduler: Schedule attempts: scheduled | Number of attempts to schedule pods with result "scheduled" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.scheduled.rate Preprocessing
|
| Kubernetes Scheduler: Schedule attempts: unschedulable | Number of attempts to schedule pods with result "unschedulable" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate Preprocessing
|
| Kubernetes Scheduler: Schedule attempts: error | Number of attempts to schedule pods with result "error" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.error.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler: Too many REST Client errors | "Kubernetes Scheduler REST Client requests is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.client_http_requests_500.rate,5m)>{$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} |
Warning | |
| Kubernetes Scheduler: Too many unschedulable pods | Number of attempts to schedule pods with 'unschedulable' result is too high. 'unschedulable' means a pod could not be scheduled. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate,5m)>{$KUBE.SCHEDULER.UNSCHEDULABLE} |
Warning | |
| Kubernetes Scheduler: Too many schedule attempts with errors | Number of attempts to schedule pods with 'error' result is too high. 'error' means an internal scheduler problem. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.error.rate,5m)>{$KUBE.SCHEDULER.ERROR} |
Warning |
LLD rule Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm histogram | Discovery raw data of scheduling algorithm latency. |
Dependent item | kubernetes.scheduler.scheduling_algorithm.discovery Preprocessing
|
Item prototypes for Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: Scheduling algorithm duration bucket, {#LE} | Scheduling algorithm latency in seconds. |
Dependent item | kubernetes.scheduler.scheduling_algorithm_duration[{#LE}] Preprocessing
|
| Kubernetes Scheduler: Scheduling algorithm duration, p90 | 90 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p90[{#SINGLETON}] |
| Kubernetes Scheduler: Scheduling algorithm duration, p95 | 95 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p95[{#SINGLETON}] |
| Kubernetes Scheduler: Scheduling algorithm duration, p99 | 99 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p99[{#SINGLETON}] |
| Kubernetes Scheduler: Scheduling algorithm duration, p50 | 50 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p50[{#SINGLETON}] |
LLD rule Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding histogram | Discovery raw data of binding latency. |
Dependent item | kubernetes.scheduler.binding.discovery Preprocessing
|
Item prototypes for Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: Binding duration bucket, {#LE} | Binding latency in seconds. |
Dependent item | kubernetes.scheduler.binding_duration[{#LE}] Preprocessing
|
| Kubernetes Scheduler: Binding duration, p90 | 90 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p90[{#SINGLETON}] |
| Kubernetes Scheduler: Binding duration, p95 | 99 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p95[{#SINGLETON}] |
| Kubernetes Scheduler: Binding duration, p99 | 95 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p99[{#SINGLETON}] |
| Kubernetes Scheduler: Binding duration, p50 | 50 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p50[{#SINGLETON}] |
LLD rule e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| e2e scheduling histogram | Discovery raw data and percentile items of e2e scheduling latency. |
Dependent item | kubernetes.controller.e2e_scheduling.discovery Preprocessing
|
Item prototypes for e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling seconds bucket, {#LE} | E2e scheduling latency in seconds (scheduling algorithm + binding) |
Dependent item | kubernetes.scheduler.e2e_scheduling_bucket[{#LE},"{#RESULT}"] Preprocessing
|
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p50 | 50 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p50["{#RESULT}"] |
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p90 | 90 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p90["{#RESULT}"] |
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p95 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p95["{#RESULT}"] |
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p99 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p99["{#RESULT}"] |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes Scheduler by HTTP
Overview
For Zabbix version: 6.2 and higher
The template to monitor Kubernetes Scheduler by Zabbix that works without any external scripts.
Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Scheduler by HTTP — collects metrics by HTTP agent from Scheduler /metrics endpoint.
This template was tested on:
- Kubernetes Scheduler, version 1.19.10
Setup
See Zabbix template operation for basic instructions.
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.SCHEDULER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values. NOTE. Some metrics may not be collected depending on your Kubernetes Scheduler instance version and configuration.
Zabbix configuration
No specific Zabbix configuration is required.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | API Authorization Token |
`` |
| {$KUBE.SCHEDULER.ERROR} | Maximum number of scheduling failures with 'error' used for trigger |
2 |
| {$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger |
2 |
| {$KUBE.SCHEDULER.SERVER.URL} | Instance URL |
http://localhost:10251/metrics |
| {$KUBE.SCHEDULER.UNSCHEDULABLE} | Maximum number of scheduling failures with 'unschedulable' used for trigger |
2 |
Template links
There are no template links in this template.
Discovery rules
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding histogram | Discovery raw data of binding latency. |
DEPENDENT | kubernetes.scheduler.binding.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: Overrides: bucket item total item |
| e2e scheduling histogram | Discovery raw data and percentile items of e2e scheduling latency. |
DEPENDENT | kubernetes.controller.e2e_scheduling.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: Overrides: bucket item total item |
| Scheduling algorithm histogram | Discovery raw data of scheduling algorithm latency. |
DEPENDENT | kubernetes.scheduler.scheduling_algorithm.discovery Preprocessing: - PROMETHEUS_TO_JSON: - JAVASCRIPT: - DISCARD_UNCHANGED_HEARTBEAT: Overrides: bucket item total item |
Items collected
| Group | Name | Description | Type | Key and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler | Kubernetes Scheduler: Virtual memory, bytes | Virtual memory size in bytes. |
DEPENDENT | kubernetes.scheduler.process_virtual_memory_bytes Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: Resident memory, bytes | Resident memory size in bytes. |
DEPENDENT | kubernetes.scheduler.process_resident_memory_bytes Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: CPU | Total user and system CPU usage ratio. |
DEPENDENT | kubernetes.scheduler.cpu.util Preprocessing: - PROMETHEUS_PATTERN: - CHANGE_PER_SECOND - MULTIPLIER: |
| Kubernetes Scheduler | Kubernetes Scheduler: Goroutines | Number of goroutines that currently exist. |
DEPENDENT | kubernetes.scheduler.go_goroutines Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: Go threads | Number of OS threads created. |
DEPENDENT | kubernetes.scheduler.go_threads Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: Fds open | Number of open file descriptors. |
DEPENDENT | kubernetes.scheduler.open_fds Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: Fds max | Maximum allowed open file descriptors. |
DEPENDENT | kubernetes.scheduler.max_fds Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
DEPENDENT | kubernetes.scheduler.client_http_requests_200.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Scheduler | Kubernetes Scheduler: REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
DEPENDENT | kubernetes.scheduler.client_http_requests_300.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Scheduler | Kubernetes Scheduler: REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
DEPENDENT | kubernetes.scheduler.client_http_requests_400.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Scheduler | Kubernetes Scheduler: REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
DEPENDENT | kubernetes.scheduler.client_http_requests_500.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Scheduler | Kubernetes Scheduler: Schedule attempts: scheduled | Number of attempts to schedule pods with result "scheduled" per second. |
DEPENDENT | kubernetes.scheduler.scheduler_schedule_attempts.scheduled.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Scheduler | Kubernetes Scheduler: Schedule attempts: unschedulable | Number of attempts to schedule pods with result "unschedulable" per second. |
DEPENDENT | kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Scheduler | Kubernetes Scheduler: Schedule attempts: error | Number of attempts to schedule pods with result "error" per second. |
DEPENDENT | kubernetes.scheduler.scheduler_schedule_attempts.error.rate Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - CHANGE_PER_SECOND |
| Kubernetes Scheduler | Kubernetes Scheduler: Scheduling algorithm duration bucket, {#LE} | Scheduling algorithm latency in seconds. |
DEPENDENT | kubernetes.scheduler.scheduling_algorithm_duration[{#LE}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: Scheduling algorithm duration, p90 | 90 percentile of scheduling algorithm latency in seconds. |
CALCULATED | kubernetes.scheduler.scheduling_algorithm_duration_p90[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.scheduling_algorithm_duration[*],5m,90) |
| Kubernetes Scheduler | Kubernetes Scheduler: Scheduling algorithm duration, p95 | 95 percentile of scheduling algorithm latency in seconds. |
CALCULATED | kubernetes.scheduler.scheduling_algorithm_duration_p95[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.scheduling_algorithm_duration[*],5m,95) |
| Kubernetes Scheduler | Kubernetes Scheduler: Scheduling algorithm duration, p99 | 99 percentile of scheduling algorithm latency in seconds. |
CALCULATED | kubernetes.scheduler.scheduling_algorithm_duration_p99[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.scheduling_algorithm_duration[*],5m,99) |
| Kubernetes Scheduler | Kubernetes Scheduler: Scheduling algorithm duration, p50 | 50 percentile of scheduling algorithm latency in seconds. |
CALCULATED | kubernetes.scheduler.scheduling_algorithm_duration_p50[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.scheduling_algorithm_duration[*],5m,50) |
| Kubernetes Scheduler | Kubernetes Scheduler: Binding duration bucket, {#LE} | Binding latency in seconds. |
DEPENDENT | kubernetes.scheduler.binding_duration[{#LE}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: Binding duration, p90 | 90 percentile of binding latency in seconds. |
CALCULATED | kubernetes.scheduler.binding_duration_p90[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.binding_duration[*],5m,90) |
| Kubernetes Scheduler | Kubernetes Scheduler: Binding duration, p95 | 99 percentile of binding latency in seconds. |
CALCULATED | kubernetes.scheduler.binding_duration_p95[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.binding_duration[*],5m,95) |
| Kubernetes Scheduler | Kubernetes Scheduler: Binding duration, p99 | 95 percentile of binding latency in seconds. |
CALCULATED | kubernetes.scheduler.binding_duration_p99[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.binding_duration[*],5m,99) |
| Kubernetes Scheduler | Kubernetes Scheduler: Binding duration, p50 | 50 percentile of binding latency in seconds. |
CALCULATED | kubernetes.scheduler.binding_duration_p50[{#SINGLETON}] Expression: bucket_percentile(//kubernetes.scheduler.binding_duration[*],5m,50) |
| Kubernetes Scheduler | Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling seconds bucket, {#LE} | E2e scheduling latency in seconds (scheduling algorithm + binding) |
DEPENDENT | kubernetes.scheduler.e2e_scheduling_bucket[{#LE},"{#RESULT}"] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes Scheduler | Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p50 | 50 percentile of e2e scheduling latency. |
CALCULATED | kubernetes.scheduler.e2e_scheduling_p50["{#RESULT}"] Expression: bucket_percentile(//kubernetes.scheduler.e2e_scheduling_bucket[*,"{#RESULT}"],5m,50) |
| Kubernetes Scheduler | Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p90 | 90 percentile of e2e scheduling latency. |
CALCULATED | kubernetes.scheduler.e2e_scheduling_p90["{#RESULT}"] Expression: bucket_percentile(//kubernetes.scheduler.e2e_scheduling_bucket[*,"{#RESULT}"],5m,90) |
| Kubernetes Scheduler | Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p95 | 95 percentile of e2e scheduling latency. |
CALCULATED | kubernetes.scheduler.e2e_scheduling_p95["{#RESULT}"] Expression: bucket_percentile(//kubernetes.scheduler.e2e_scheduling_bucket[*,"{#RESULT}"],5m,95) |
| Kubernetes Scheduler | Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p99 | 95 percentile of e2e scheduling latency. |
CALCULATED | kubernetes.scheduler.e2e_scheduling_p99["{#RESULT}"] Expression: bucket_percentile(//kubernetes.scheduler.e2e_scheduling_bucket[*,"{#RESULT}"],5m,99) |
| Zabbix raw items | Kubernetes Scheduler: Get Scheduler metrics | Get raw metrics from Scheduler instance /metrics endpoint. |
HTTP_AGENT | kubernetes.scheduler.get_metrics Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: |
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler: Too many REST Client errors | "Kubernetes Scheduler REST Client requests is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.client_http_requests_500.rate,5m)>{$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} |
WARNING | |
| Kubernetes Scheduler: Too many unschedulable pods | "Number of attempts to schedule pods with 'unschedulable' result is too high. 'unschedulable' means a pod could not be scheduled." |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate,5m)>{$KUBE.SCHEDULER.UNSCHEDULABLE} |
WARNING | |
| Kubernetes Scheduler: Too many schedule attempts with errors | "Number of attempts to schedule pods with 'error' result is too high. 'error' means an internal scheduler problem." |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.error.rate,5m)>{$KUBE.SCHEDULER.ERROR} |
WARNING |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template or ask for help with it at ZABBIX forums.
Kubernetes Scheduler by HTTP
Overview
The template to monitor Kubernetes Scheduler by Zabbix that works without any external scripts. Most of the metrics are collected in one go, thanks to Zabbix bulk data collection.
Template Kubernetes Scheduler by HTTP - collects metrics by HTTP agent from Scheduler /metrics endpoint.
Requirements
Zabbix version: 6.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes Scheduler 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Internal service metrics are collected from /metrics endpoint. Template needs to use Authorization via API token.
Don't forget change macros {$KUBE.SCHEDULER.SERVER.URL}, {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. You might need to set the --binding-address option for Scheduler to the address where Zabbix proxy can reach it.
For example, for clusters created with kubeadm it can be set in the following manifest file (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-scheduler.yaml
NOTE. Some metrics may not be collected depending on your Kubernetes Scheduler instance version and configuration.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.SCHEDULER.SERVER.URL} | Kubernetes Scheduler metrics endpoint URL. |
https://localhost:10259/metrics |
| {$KUBE.API.TOKEN} | API Authorization Token. |
|
| {$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} | Maximum number of HTTP client requests failures used for trigger. |
2 |
| {$KUBE.SCHEDULER.UNSCHEDULABLE} | Maximum number of scheduling failures with 'unschedulable' used for trigger. |
2 |
| {$KUBE.SCHEDULER.ERROR} | Maximum number of scheduling failures with 'error' used for trigger. |
2 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: Get Scheduler metrics | Get raw metrics from Scheduler instance /metrics endpoint. |
HTTP agent | kubernetes.scheduler.get_metrics Preprocessing
|
| Kubernetes Scheduler: Virtual memory, bytes | Virtual memory size in bytes. |
Dependent item | kubernetes.scheduler.process_virtual_memory_bytes Preprocessing
|
| Kubernetes Scheduler: Resident memory, bytes | Resident memory size in bytes. |
Dependent item | kubernetes.scheduler.process_resident_memory_bytes Preprocessing
|
| Kubernetes Scheduler: CPU | Total user and system CPU usage ratio. |
Dependent item | kubernetes.scheduler.cpu.util Preprocessing
|
| Kubernetes Scheduler: Goroutines | Number of goroutines that currently exist. |
Dependent item | kubernetes.scheduler.go_goroutines Preprocessing
|
| Kubernetes Scheduler: Go threads | Number of OS threads created. |
Dependent item | kubernetes.scheduler.go_threads Preprocessing
|
| Kubernetes Scheduler: Fds open | Number of open file descriptors. |
Dependent item | kubernetes.scheduler.open_fds Preprocessing
|
| Kubernetes Scheduler: Fds max | Maximum allowed open file descriptors. |
Dependent item | kubernetes.scheduler.max_fds Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 2xx, rate | Number of HTTP requests with 2xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_200.rate Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 3xx, rate | Number of HTTP requests with 3xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_300.rate Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 4xx, rate | Number of HTTP requests with 4xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_400.rate Preprocessing
|
| Kubernetes Scheduler: REST Client requests: 5xx, rate | Number of HTTP requests with 5xx status code per second. |
Dependent item | kubernetes.scheduler.client_http_requests_500.rate Preprocessing
|
| Kubernetes Scheduler: Schedule attempts: scheduled | Number of attempts to schedule pods with result "scheduled" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.scheduled.rate Preprocessing
|
| Kubernetes Scheduler: Schedule attempts: unschedulable | Number of attempts to schedule pods with result "unschedulable" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate Preprocessing
|
| Kubernetes Scheduler: Schedule attempts: error | Number of attempts to schedule pods with result "error" per second. |
Dependent item | kubernetes.scheduler.scheduler_schedule_attempts.error.rate Preprocessing
|
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes Scheduler: Too many REST Client errors | "Kubernetes Scheduler REST Client requests is experiencing high error rate (with 5xx HTTP code). |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.client_http_requests_500.rate,5m)>{$KUBE.SCHEDULER.HTTP.CLIENT.ERROR} |
Warning | |
| Kubernetes Scheduler: Too many unschedulable pods | Number of attempts to schedule pods with 'unschedulable' result is too high. 'unschedulable' means a pod could not be scheduled. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.unschedulable.rate,5m)>{$KUBE.SCHEDULER.UNSCHEDULABLE} |
Warning | |
| Kubernetes Scheduler: Too many schedule attempts with errors | Number of attempts to schedule pods with 'error' result is too high. 'error' means an internal scheduler problem. |
min(/Kubernetes Scheduler by HTTP/kubernetes.scheduler.scheduler_schedule_attempts.error.rate,5m)>{$KUBE.SCHEDULER.ERROR} |
Warning |
LLD rule Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduling algorithm histogram | Discovery raw data of scheduling algorithm latency. |
Dependent item | kubernetes.scheduler.scheduling_algorithm.discovery Preprocessing
|
Item prototypes for Scheduling algorithm histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: Scheduling algorithm duration bucket, {#LE} | Scheduling algorithm latency in seconds. |
Dependent item | kubernetes.scheduler.scheduling_algorithm_duration[{#LE}] Preprocessing
|
| Kubernetes Scheduler: Scheduling algorithm duration, p90 | 90 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p90[{#SINGLETON}] |
| Kubernetes Scheduler: Scheduling algorithm duration, p95 | 95 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p95[{#SINGLETON}] |
| Kubernetes Scheduler: Scheduling algorithm duration, p99 | 99 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p99[{#SINGLETON}] |
| Kubernetes Scheduler: Scheduling algorithm duration, p50 | 50 percentile of scheduling algorithm latency in seconds. |
Calculated | kubernetes.scheduler.scheduling_algorithm_duration_p50[{#SINGLETON}] |
LLD rule Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Binding histogram | Discovery raw data of binding latency. |
Dependent item | kubernetes.scheduler.binding.discovery Preprocessing
|
Item prototypes for Binding histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: Binding duration bucket, {#LE} | Binding latency in seconds. |
Dependent item | kubernetes.scheduler.binding_duration[{#LE}] Preprocessing
|
| Kubernetes Scheduler: Binding duration, p90 | 90 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p90[{#SINGLETON}] |
| Kubernetes Scheduler: Binding duration, p95 | 99 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p95[{#SINGLETON}] |
| Kubernetes Scheduler: Binding duration, p99 | 95 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p99[{#SINGLETON}] |
| Kubernetes Scheduler: Binding duration, p50 | 50 percentile of binding latency in seconds. |
Calculated | kubernetes.scheduler.binding_duration_p50[{#SINGLETON}] |
LLD rule e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| e2e scheduling histogram | Discovery raw data and percentile items of e2e scheduling latency. |
Dependent item | kubernetes.controller.e2e_scheduling.discovery Preprocessing
|
Item prototypes for e2e scheduling histogram
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling seconds bucket, {#LE} | E2e scheduling latency in seconds (scheduling algorithm + binding) |
Dependent item | kubernetes.scheduler.e2e_scheduling_bucket[{#LE},"{#RESULT}"] Preprocessing
|
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p50 | 50 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p50["{#RESULT}"] |
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p90 | 90 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p90["{#RESULT}"] |
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p95 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p95["{#RESULT}"] |
| Kubernetes Scheduler: ["{#RESULT}"]: e2e scheduling, p99 | 95 percentile of e2e scheduling latency. |
Calculated | kubernetes.scheduler.e2e_scheduling_p99["{#RESULT}"] |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes cluster state by HTTP
Overview
The template to monitor Kubernetes state. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes cluster state by HTTP - collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget to change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Requirements
Zabbix version: 7.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the service account name. If a different release name is used.
kubectl get serviceaccounts -n monitoring
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.STATE.ENDPOINT.NAME} with Kube state metrics endpoint name. See kubectl -n monitoring get ep. Default: zabbix-kube-state-metrics.
Note: If you wish to monitor Controller Manager and Scheduler components, you might need to set the --binding-address option for them to the address where Zabbix proxy can reach them.
For example, for clusters created with kubeadm it can be set in the following manifest files (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-scheduler.yaml
Depending on your Kubernetes distribution, you might need to adjust {$KUBE.CONTROL_PLANE.TAINT} macro (for example, set it to node-role.kubernetes.io/master for OpenShift).
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Also, see the Macros section for a list of macros used to set trigger values.
Set up the macros to filter the metrics of discovered Kubelets by node names:
- {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES}
- {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Set up macros to filter node metrics by nodename:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable namespaces.
You can use the {$KUBE.KUBELET.FILTER.LABELS} and {$KUBE.KUBELET.FILTER.ANNOTATIONS} macros for advanced filtering of kubelets by node labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such label key exists for the entity that is being filtered (it means that if you specify a key in the filter, entities that do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the kubelets on nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
You can also set up evaluation periods for replica mismatch triggers (Deployments, ReplicaSets, StatefulSets) with the macro {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD}, which supports context and regular expressions. For example, you can create the following macros:
- Set the evaluation period for the Deployment "nginx-deployment" in the namespace "default" to the 3 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:default:nginx-deployment"} = #3
- Set the evaluation period for all Deployments to the 10 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"deployment:.*:.*"} = #10 or {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"^deployment.*"} = #10
- Set the evaluation period for Deployments, ReplicaSets and StatefulSets in the namespace "default" to 15 minutes:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:".*:default:.*"} = 15m
Note: that different context macros with regular expressions matching the same string can be applied in an undefined order, and simple context macros (without regular expressions) have higher priority. Read the Important notes section in Zabbix documentation for details.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.LIVEZ.ENDPOINT} | Kubernetes API livez endpoint /livez |
/livez |
| {$KUBE.API.COMPONENTSTATUSES.ENDPOINT} | Kubernetes API componentstatuses endpoint /api/v1/componentstatuses |
/api/v1/componentstatuses |
| {$KUBE.API.READYZ.ENDPOINT} | Kubernetes API readyz endpoint /readyz |
/readyz |
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$HTTP.TLS.VERIFY} | TLS certificate verification for script items: "none" - disabled, "peer" - verify the certificate chain and expiration, "full" - full verification. Any other value enables full verification. To override the setting for this template only, define the macro with the context "Kubernetes cluster state", e.g. {$HTTP.TLS.VERIFY:"Kubernetes cluster state"}. |
full |
| {$KUBE.STATE.ENDPOINT.NAME} | Endpoint name for kube-state-metrics service (check with |
zabbix-kube-state-metrics |
| {$OPENSHIFT.STATE.ENDPOINT.NAME} | OpenShift state endpoint name. |
openshift-state-metrics |
| {$KUBE.API_SERVER.SCHEME} | Kubernetes API servers metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.API_SERVER.PORT} | Kubernetes API servers metrics endpoint port. Used in ControlPlane LLD. |
6443 |
| {$KUBE.CONTROL_PLANE.TAINT} | Taint that applies to control plane nodes. Change if needed. Used in ControlPlane LLD. |
node-role.kubernetes.io/control-plane |
| {$KUBE.CONTROLLER_MANAGER.SCHEME} | Kubernetes Controller manager metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.CONTROLLER_MANAGER.PORT} | Kubernetes Controller manager metrics endpoint port. Used in ControlPlane LLD. |
10257 |
| {$KUBE.SCHEDULER.SCHEME} | Kubernetes Scheduler metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.SCHEDULER.PORT} | Kubernetes Scheduler metrics endpoint port. Used in ControlPlane LLD. |
10259 |
| {$KUBE.KUBELET.SCHEME} | Kubernetes Kubelet metrics endpoint scheme. Used in Kubelet LLD. |
https |
| {$KUBE.KUBELET.PORT} | Kubernetes Kubelet metrics endpoint port. Used in Kubelet LLD. |
10250 |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable metrics by namespace. |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered metrics by namespace. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES} | Filter of discoverable Kubelets by nodename. |
.* |
| {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES} | Filter to exclude discovered Kubelets by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.KUBELET.FILTER.ANNOTATIONS} | Node annotations to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.KUBELET.FILTER.LABELS} | Node labels to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.PV.MATCHES} | Filter of discoverable persistent volumes by name. |
.* |
| {$KUBE.LLD.FILTER.PV.NOT_MATCHES} | Filter to exclude discovered persistent volumes by name. |
CHANGE_IF_NEEDED |
| {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD} | The evaluation period range which is used for calculation of expressions in trigger prototypes (time period or value range). Can be used with context. |
#5 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
Script | kube.state.metrics |
| Control plane LLD | Generation of data for Control plane discovery rules. |
Script | kube.control_plane.lld Preprocessing
|
| Node LLD | Generation of data for Kubelet discovery rules. |
Script | kube.node.lld Preprocessing
|
| Get component statuses | HTTP agent | kube.componentstatuses Preprocessing
|
|
| Get readyz | HTTP agent | kube.readyz Preprocessing
|
|
| Get livez | HTTP agent | kube.livez Preprocessing
|
|
| Namespace count | The number of namespaces. |
Dependent item | kube.namespace.count Preprocessing
|
| CronJob count | Number of cronjobs. |
Dependent item | kube.cronjob.count Preprocessing
|
| Job count | Number of jobs (generated by cronjob + job). |
Dependent item | kube.job.count Preprocessing
|
| Endpoint count | Number of endpoints. |
Dependent item | kube.endpoint.count Preprocessing
|
| Deployment count | The number of deployments. |
Dependent item | kube.deployment.count Preprocessing
|
| Service count | The number of services. |
Dependent item | kube.service.count Preprocessing
|
| StatefulSet count | The number of statefulsets. |
Dependent item | kube.statefulset.count Preprocessing
|
| Node count | The number of nodes. |
Dependent item | kube.node.count Preprocessing
|
LLD rule API servers discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| API servers discovery | Dependent item | kube.api_servers.discovery |
LLD rule Controller manager nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Controller manager nodes discovery | Dependent item | kube.controller_manager.discovery |
LLD rule Scheduler servers nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduler servers nodes discovery | Dependent item | kube.scheduler.discovery |
LLD rule Kubelet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubelet discovery | Dependent item | kube.kubelet.discovery |
LLD rule Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Daemonset discovery | Dependent item | kube.daemonset.discovery Preprocessing
|
Item prototypes for Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Ready | The number of nodes that should be running the daemon pod and have one or more running and ready. |
Dependent item | kube.daemonset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Scheduled | The number of nodes that run at least one daemon pod and are supposed to. |
Dependent item | kube.daemonset.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Desired | The number of nodes that should be running the daemon pod. |
Dependent item | kube.daemonset.desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Misscheduled | The number of nodes that run a daemon pod but are not supposed to. |
Dependent item | kube.daemonset.misscheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Updated number scheduled | The total number of nodes that are running updated daemon pod. |
Dependent item | kube.daemonset.updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PVC discovery | Dependent item | kube.pvc.discovery Preprocessing
|
Item prototypes for PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase | The current status phase of the persistent volume claim. |
Dependent item | kube.pvc.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
Dependent item | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Bound, sum | The total amount of persistent volume claims in the Bound phase. |
Dependent item | kube.pvc.status_phase.bound.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Lost, sum | The total amount of persistent volume claims in the Lost phase. |
Dependent item | kube.pvc.status_phase.lost.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Pending, sum | The total amount of persistent volume claims in the Pending phase. |
Dependent item | kube.pvc.status_phase.pending.sum[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for PVC discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | count(/Kubernetes cluster state by HTTP/kube.pvc.status_phase[{#NAMESPACE}/{#NAME}],2m,,5)>=2 |
Warning |
LLD rule PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV discovery | Dependent item | kube.pv.discovery Preprocessing
|
Item prototypes for PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV [{#NAME}] Status phase | The current status phase of the persistent volume. |
Dependent item | kube.pv.status_phase[{#NAME}] Preprocessing
|
| PV [{#NAME}] Capacity bytes | A capacity of the persistent volume in bytes. |
Dependent item | kube.pv.capacity.bytes[{#NAME}] Preprocessing
|
| PV status phase: Pending, sum | The total amount of persistent volumes in the Pending phase. |
Dependent item | kube.pv.status_phase.pending.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Available, sum | The total amount of persistent volumes in the Available phase. |
Dependent item | kube.pv.status_phase.available.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Bound, sum | The total amount of persistent volumes in the Bound phase. |
Dependent item | kube.pv.status_phase.bound.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Released, sum | The total amount of persistent volumes in the Released phase. |
Dependent item | kube.pv.status_phase.released.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Failed, sum | The total amount of persistent volumes in the Failed phase. |
Dependent item | kube.pv.status_phase.failed.sum[{#SINGLETON}] Preprocessing
|
Trigger prototypes for PV discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: PV [{#NAME}]: PV has failed | count(/Kubernetes cluster state by HTTP/kube.pv.status_phase[{#NAME}],2m,,3)>=2 |
Warning |
LLD rule Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment discovery | Dependent item | kube.deployment.discovery Preprocessing
|
Item prototypes for Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Paused | Whether the deployment is paused and will not be processed by the deployment controller. |
Dependent item | kube.deployment.spec_paused[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas desired | Number of desired pods for a deployment. |
Dependent item | kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Rollingupdate max unavailable | Maximum number of unavailable replicas during a rolling update of a deployment. |
Dependent item | kube.deployment.rollingupdate.max_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas | The number of replicas per deployment. |
Dependent item | kube.deployment.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas available | The number of available replicas per deployment. |
Dependent item | kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas unavailable | The number of unavailable replicas per deployment. |
Dependent item | kube.deployment.replicas_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas updated | The number of updated replicas per deployment. |
Dependent item | kube.deployment.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas mismatched | The number of available replicas not matching the desired number of replicas. |
Dependent item | kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Deployment discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Deployment replicas mismatch | Deployment has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Endpoint discovery | Dependent item | kube.endpoint.discovery Preprocessing
|
Item prototypes for Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address available | Number of addresses available in endpoint. |
Dependent item | kube.endpoint.address_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address not ready | Number of addresses not ready in endpoint. |
Dependent item | kube.endpoint.address_not_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Age | Endpoint age (number of seconds since creation). |
Dependent item | kube.endpoint.age[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: CPU allocatable | The CPU resources of a node that are available for scheduling. |
Dependent item | kube.node.cpu_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory allocatable | The memory resources of a node that are available for scheduling. |
Dependent item | kube.node.memory_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods allocatable | The pods resources of a node that are available for scheduling. |
Dependent item | kube.node.pods_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage allocatable | The allocatable ephemeral storage of a node that is available for scheduling. |
Dependent item | kube.node.ephemeral_storage_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: CPU capacity | The capacity for CPU resources of a node. |
Dependent item | kube.node.cpu_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory capacity | The capacity for memory resources of a node. |
Dependent item | kube.node.memory_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage capacity | The ephemeral storage capacity of a node. |
Dependent item | kube.node.ephemeral_storage_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods capacity | The capacity for pods resources of a node. |
Dependent item | kube.node.pods_capacity[{#NAME}] Preprocessing
|
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Pending | Pod is in pending state. |
Dependent item | kube.pod.phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Succeeded | Pod is in succeeded state. |
Dependent item | kube.pod.phase.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Failed | Pod is in failed state. |
Dependent item | kube.pod.phase.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Unknown | Pod is in unknown state. |
Dependent item | kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Running | Pod is in unknown state. |
Dependent item | kube.pod.phase.running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers terminated | Describes whether the container is currently in terminated state. |
Dependent item | kube.pod.containers_terminated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers waiting | Describes whether the container is currently in waiting state. |
Dependent item | kube.pod.containers_waiting[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers ready | Describes whether the containers readiness check succeeded. |
Dependent item | kube.pod.containers_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
Dependent item | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers running | Describes whether the container is currently in running state. |
Dependent item | kube.pod.containers_running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Ready | Describes whether the pod is ready to serve requests. |
Dependent item | kube.pod.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Scheduled | Describes the status of the scheduling process for the pod. |
Dependent item | kube.pod.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Unschedulable | Describes the unschedulable status for the pod. |
Dependent item | kube.pod.unschedulable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU limits | The limit on CPU cores to be used by a container. |
Dependent item | kube.pod.containers.limits.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory limits | The limit on memory to be used by a container. |
Dependent item | kube.pod.containers.limits.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU requests | The number of requested CPU cores by a container. |
Dependent item | kube.pod.containers.requests.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory requests | The number of requested memory bytes by a container. |
Dependent item | kube.pod.containers.requests.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is not healthy | min(/Kubernetes cluster state by HTTP/kube.pod.phase.failed[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.pending[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}],10m)>0 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],15m))>1 |
Warning |
LLD rule ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ReplicaSet discovery | Dependent item | kube.replicaset.discovery Preprocessing
|
Item prototypes for ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas | The number of replicas per ReplicaSet. |
Dependent item | kube.replicaset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Desired replicas | Number of desired pods for a ReplicaSet. |
Dependent item | kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Fully labeled replicas | The number of fully labeled replicas per ReplicaSet. |
Dependent item | kube.replicaset.fully_labeled_replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Ready | The number of ready replicas per ReplicaSet. |
Dependent item | kube.replicaset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the desired number of replicas. |
Dependent item | kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for ReplicaSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] RS [{#NAME}]: ReplicaSet mismatch | ReplicaSet has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"replicaset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| StatefulSet discovery | Dependent item | kube.statefulset.discovery Preprocessing
|
Item prototypes for StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas | The number of replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Desired replicas | Number of desired pods for a StatefulSet. |
Dependent item | kube.statefulset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Current replicas | The number of current replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Ready replicas | The number of ready replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Updated replicas | The number of updated replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the number of replicas. |
Dependent item | kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for StatefulSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet is down | (last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]) / last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}]))<>1 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet replicas mismatch | StatefulSet has not matched the number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"statefulset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PodDisruptionBudget discovery | Dependent item | kube.pdb.discovery Preprocessing
|
Item prototypes for PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods healthy | Current number of healthy pods. |
Dependent item | kube.pdb.pods_healthy[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods desired | Minimum desired number of healthy pods. |
Dependent item | kube.pdb.pods_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Disruptions allowed | Number of pod disruptions that are allowed. |
Dependent item | kube.pdb.disruptions_allowed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods total | Total number of pods counted by this disruption budget. |
Dependent item | kube.pdb.pods_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| CronJob discovery | Dependent item | kube.cronjob.discovery Preprocessing
|
Item prototypes for CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Suspend | Suspend flag tells the controller to suspend subsequent executions. |
Dependent item | kube.cronjob.spec_suspend[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Active | Active holds pointers to currently running jobs. |
Dependent item | kube.cronjob.status_active[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Last schedule | LastScheduleTime keeps information of when was the last time the job was successfully scheduled. |
Dependent item | kube.cronjob.last_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Next schedule | Next time the cronjob should be scheduled. The time after lastScheduleTime or after the cron job's creation time if it's never been scheduled. Use this to determine if the job is delayed. |
Dependent item | kube.cronjob.next_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.cronjob.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.cronjob.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.cronjob.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.cronjob.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Job discovery | Dependent item | kube.job.discovery Preprocessing
|
Item prototypes for Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.job.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.job.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.job.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.job.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component statuses discovery | Dependent item | kube.componentstatuses.discovery Preprocessing
|
Item prototypes for Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component [{#NAME}]: Healthy | Cluster component healthy. |
Dependent item | kube.componentstatuses.healthy[{#NAME}] Preprocessing
|
Trigger prototypes for Component statuses discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Component [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}],#2,"ne","True")=2 and length(last(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}]))>0 |
Warning |
LLD rule Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz discovery | Dependent item | kube.readyz.discovery Preprocessing
|
Item prototypes for Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz [{#NAME}]: Healthcheck | Result of readyz healthcheck for component. |
Dependent item | kube.readyz.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Readyz discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Readyz [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez discovery | Dependent item | kube.livez.discovery Preprocessing
|
Item prototypes for Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez [{#NAME}]: Healthcheck | Result of livez healthcheck for component. |
Dependent item | kube.livez.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Livez discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Livez [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift BuildConfig discovery | Dependent item | openshift.buildconfig.discovery Preprocessing
|
Item prototypes for OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Created | OpenShift BuildConfig Unix creation timestamp. |
Dependent item | openshift.buildconfig.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.buildconfig.generation[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Latest version | The latest version of BuildConfig. |
Dependent item | openshift.buildconfig.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Build discovery | Dependent item | openshift.build.discovery Preprocessing
|
Item prototypes for OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Created | OpenShift Build Unix creation timestamp. |
Dependent item | openshift.build.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.build.sequence.number[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Status phase | The Build phase. |
Dependent item | openshift.build.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Build discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Build [{#NAME}]: Build has failed | count(/Kubernetes cluster state by HTTP/openshift.build.status_phase[{#NAMESPACE}/{#NAME}],2m,"ge",6)>=2 |
Warning |
LLD rule OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift ClusterResourceQuota discovery | Dependent item | openshift.cluster.resource.quota.discovery Preprocessing
|
Item prototypes for OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Quota [{#NAME}] Resource [{#RESOURCE}]: Type [{#TYPE}]] | Usage about resource quota. |
Dependent item | openshift.cluster.resource.quota[{#RESOURCE}/{#NAME}/{#TYPE}] Preprocessing
|
LLD rule OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Route discovery | Dependent item | openshift.route.discovery Preprocessing
|
Item prototypes for OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Created | OpenShift Route Unix creation timestamp. |
Dependent item | openshift.route.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Status | Information about route status. |
Dependent item | openshift.route.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Route discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Route [{#NAME}] with issue: Status is false | count(/Kubernetes cluster state by HTTP/openshift.route.status[{#NAMESPACE}/{#NAME}],2m,,0)>=2 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes cluster state by HTTP
Overview
The template to monitor Kubernetes state. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes cluster state by HTTP - collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget to change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
Requirements
Zabbix version: 7.2 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command:
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.STATE.ENDPOINT.NAME} with Kube state metrics endpoint name. See kubectl -n monitoring get ep. Default: zabbix-kube-state-metrics.
NOTE. If you wish to monitor Controller Manager and Scheduler components, you might need to set the --binding-address option for them to the address where Zabbix proxy can reach them.
For example, for clusters created with kubeadm it can be set in the following manifest files (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-scheduler.yaml
Depending on your Kubernetes distribution, you might need to adjust {$KUBE.CONTROL_PLANE.TAINT} macro (for example, set it to node-role.kubernetes.io/master for OpenShift).
NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
Also, see the Macros section for a list of macros used to set trigger values.
Set up the macros to filter the metrics of discovered Kubelets by node names:
- {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES}
- {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Set up macros to filter node metrics by nodename:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable namespaces.
You can use the {$KUBE.KUBELET.FILTER.LABELS} and {$KUBE.KUBELET.FILTER.ANNOTATIONS} macros for advanced filtering of kubelets by node labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such label key exists for the entity that is being filtered (it means that if you specify a key in the filter, entities that do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the kubelets on nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
You can also set up evaluation periods for replica mismatch triggers (Deployments, ReplicaSets, StatefulSets) with the macro {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD}, which supports context and regular expressions. For example, you can create the following macros:
- Set the evaluation period for the Deployment "nginx-deployment" in the namespace "default" to the 3 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:default:nginx-deployment"} = #3
- Set the evaluation period for all Deployments to the 10 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"deployment:.*:.*"} = #10 or {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"^deployment.*"} = #10
- Set the evaluation period for Deployments, ReplicaSets and StatefulSets in the namespace "default" to 15 minutes:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:".*:default:.*"} = 15m
Note that different context macros with regular expressions matching the same string can be applied in an undefined order, and simple context macros (without regular expressions) have higher priority. Read the Important notes section in Zabbix documentation for details.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.READYZ.ENDPOINT} | Kubernetes API readyz endpoint /readyz |
/readyz |
| {$KUBE.API.LIVEZ.ENDPOINT} | Kubernetes API livez endpoint /livez |
/livez |
| {$KUBE.API.COMPONENTSTATUSES.ENDPOINT} | Kubernetes API componentstatuses endpoint /api/v1/componentstatuses |
/api/v1/componentstatuses |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$KUBE.STATE.ENDPOINT.NAME} | Kubernetes state endpoint name. |
zabbix-kube-state-metrics |
| {$OPENSHIFT.STATE.ENDPOINT.NAME} | OpenShift state endpoint name. |
openshift-state-metrics |
| {$KUBE.API_SERVER.SCHEME} | Kubernetes API servers metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.API_SERVER.PORT} | Kubernetes API servers metrics endpoint port. Used in ControlPlane LLD. |
6443 |
| {$KUBE.CONTROL_PLANE.TAINT} | Taint that applies to control plane nodes. Change if needed. Used in ControlPlane LLD. |
node-role.kubernetes.io/control-plane |
| {$KUBE.CONTROLLER_MANAGER.SCHEME} | Kubernetes Controller manager metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.CONTROLLER_MANAGER.PORT} | Kubernetes Controller manager metrics endpoint port. Used in ControlPlane LLD. |
10257 |
| {$KUBE.SCHEDULER.SCHEME} | Kubernetes Scheduler metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.SCHEDULER.PORT} | Kubernetes Scheduler metrics endpoint port. Used in ControlPlane LLD. |
10259 |
| {$KUBE.KUBELET.SCHEME} | Kubernetes Kubelet metrics endpoint scheme. Used in Kubelet LLD. |
https |
| {$KUBE.KUBELET.PORT} | Kubernetes Kubelet metrics endpoint port. Used in Kubelet LLD. |
10250 |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable metrics by namespace. |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered metrics by namespace. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES} | Filter of discoverable Kubelets by nodename. |
.* |
| {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES} | Filter to exclude discovered Kubelets by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.KUBELET.FILTER.ANNOTATIONS} | Node annotations to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.KUBELET.FILTER.LABELS} | Node labels to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.PV.MATCHES} | Filter of discoverable persistent volumes by name. |
.* |
| {$KUBE.LLD.FILTER.PV.NOT_MATCHES} | Filter to exclude discovered persistent volumes by name. |
CHANGE_IF_NEEDED |
| {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD} | The evaluation period range which is used for calculation of expressions in trigger prototypes (time period or value range). Can be used with context. |
#5 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
Script | kube.state.metrics |
| Control plane LLD | Generation of data for Control plane discovery rules. |
Script | kube.control_plane.lld Preprocessing
|
| Node LLD | Generation of data for Kubelet discovery rules. |
Script | kube.node.lld Preprocessing
|
| Get component statuses | HTTP agent | kube.componentstatuses Preprocessing
|
|
| Get readyz | HTTP agent | kube.readyz Preprocessing
|
|
| Get livez | HTTP agent | kube.livez Preprocessing
|
|
| Namespace count | The number of namespaces. |
Dependent item | kube.namespace.count Preprocessing
|
| CronJob count | Number of cronjobs. |
Dependent item | kube.cronjob.count Preprocessing
|
| Job count | Number of jobs (generated by cronjob + job). |
Dependent item | kube.job.count Preprocessing
|
| Endpoint count | Number of endpoints. |
Dependent item | kube.endpoint.count Preprocessing
|
| Deployment count | The number of deployments. |
Dependent item | kube.deployment.count Preprocessing
|
| Service count | The number of services. |
Dependent item | kube.service.count Preprocessing
|
| StatefulSet count | The number of statefulsets. |
Dependent item | kube.statefulset.count Preprocessing
|
| Node count | The number of nodes. |
Dependent item | kube.node.count Preprocessing
|
LLD rule API servers discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| API servers discovery | Dependent item | kube.api_servers.discovery |
LLD rule Controller manager nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Controller manager nodes discovery | Dependent item | kube.controller_manager.discovery |
LLD rule Scheduler servers nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduler servers nodes discovery | Dependent item | kube.scheduler.discovery |
LLD rule Kubelet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubelet discovery | Dependent item | kube.kubelet.discovery |
LLD rule Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Daemonset discovery | Dependent item | kube.daemonset.discovery Preprocessing
|
Item prototypes for Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Ready | The number of nodes that should be running the daemon pod and have one or more running and ready. |
Dependent item | kube.daemonset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Scheduled | The number of nodes that run at least one daemon pod and are supposed to. |
Dependent item | kube.daemonset.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Desired | The number of nodes that should be running the daemon pod. |
Dependent item | kube.daemonset.desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Misscheduled | The number of nodes that run a daemon pod but are not supposed to. |
Dependent item | kube.daemonset.misscheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Updated number scheduled | The total number of nodes that are running updated daemon pod. |
Dependent item | kube.daemonset.updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PVC discovery | Dependent item | kube.pvc.discovery Preprocessing
|
Item prototypes for PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase | The current status phase of the persistent volume claim. |
Dependent item | kube.pvc.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
Dependent item | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Bound, sum | The total amount of persistent volume claims in the Bound phase. |
Dependent item | kube.pvc.status_phase.bound.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Lost, sum | The total amount of persistent volume claims in the Lost phase. |
Dependent item | kube.pvc.status_phase.lost.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Pending, sum | The total amount of persistent volume claims in the Pending phase. |
Dependent item | kube.pvc.status_phase.pending.sum[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for PVC discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | count(/Kubernetes cluster state by HTTP/kube.pvc.status_phase[{#NAMESPACE}/{#NAME}],2m,,5)>=2 |
Warning |
LLD rule PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV discovery | Dependent item | kube.pv.discovery Preprocessing
|
Item prototypes for PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV [{#NAME}] Status phase | The current status phase of the persistent volume. |
Dependent item | kube.pv.status_phase[{#NAME}] Preprocessing
|
| PV [{#NAME}] Capacity bytes | A capacity of the persistent volume in bytes. |
Dependent item | kube.pv.capacity.bytes[{#NAME}] Preprocessing
|
| PV status phase: Pending, sum | The total amount of persistent volumes in the Pending phase. |
Dependent item | kube.pv.status_phase.pending.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Available, sum | The total amount of persistent volumes in the Available phase. |
Dependent item | kube.pv.status_phase.available.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Bound, sum | The total amount of persistent volumes in the Bound phase. |
Dependent item | kube.pv.status_phase.bound.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Released, sum | The total amount of persistent volumes in the Released phase. |
Dependent item | kube.pv.status_phase.released.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Failed, sum | The total amount of persistent volumes in the Failed phase. |
Dependent item | kube.pv.status_phase.failed.sum[{#SINGLETON}] Preprocessing
|
Trigger prototypes for PV discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: PV [{#NAME}]: PV has failed | count(/Kubernetes cluster state by HTTP/kube.pv.status_phase[{#NAME}],2m,,3)>=2 |
Warning |
LLD rule Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment discovery | Dependent item | kube.deployment.discovery Preprocessing
|
Item prototypes for Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Paused | Whether the deployment is paused and will not be processed by the deployment controller. |
Dependent item | kube.deployment.spec_paused[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas desired | Number of desired pods for a deployment. |
Dependent item | kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Rollingupdate max unavailable | Maximum number of unavailable replicas during a rolling update of a deployment. |
Dependent item | kube.deployment.rollingupdate.max_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas | The number of replicas per deployment. |
Dependent item | kube.deployment.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas available | The number of available replicas per deployment. |
Dependent item | kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas unavailable | The number of unavailable replicas per deployment. |
Dependent item | kube.deployment.replicas_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas updated | The number of updated replicas per deployment. |
Dependent item | kube.deployment.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas mismatched | The number of available replicas not matching the desired number of replicas. |
Dependent item | kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Deployment discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Deployment replicas mismatch | Deployment has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Endpoint discovery | Dependent item | kube.endpoint.discovery Preprocessing
|
Item prototypes for Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address available | Number of addresses available in endpoint. |
Dependent item | kube.endpoint.address_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address not ready | Number of addresses not ready in endpoint. |
Dependent item | kube.endpoint.address_not_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Age | Endpoint age (number of seconds since creation). |
Dependent item | kube.endpoint.age[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: CPU allocatable | The CPU resources of a node that are available for scheduling. |
Dependent item | kube.node.cpu_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory allocatable | The memory resources of a node that are available for scheduling. |
Dependent item | kube.node.memory_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods allocatable | The pods resources of a node that are available for scheduling. |
Dependent item | kube.node.pods_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage allocatable | The allocatable ephemeral storage of a node that is available for scheduling. |
Dependent item | kube.node.ephemeral_storage_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: CPU capacity | The capacity for CPU resources of a node. |
Dependent item | kube.node.cpu_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory capacity | The capacity for memory resources of a node. |
Dependent item | kube.node.memory_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage capacity | The ephemeral storage capacity of a node. |
Dependent item | kube.node.ephemeral_storage_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods capacity | The capacity for pods resources of a node. |
Dependent item | kube.node.pods_capacity[{#NAME}] Preprocessing
|
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Pending | Pod is in pending state. |
Dependent item | kube.pod.phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Succeeded | Pod is in succeeded state. |
Dependent item | kube.pod.phase.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Failed | Pod is in failed state. |
Dependent item | kube.pod.phase.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Unknown | Pod is in unknown state. |
Dependent item | kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Running | Pod is in unknown state. |
Dependent item | kube.pod.phase.running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers terminated | Describes whether the container is currently in terminated state. |
Dependent item | kube.pod.containers_terminated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers waiting | Describes whether the container is currently in waiting state. |
Dependent item | kube.pod.containers_waiting[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers ready | Describes whether the containers readiness check succeeded. |
Dependent item | kube.pod.containers_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
Dependent item | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers running | Describes whether the container is currently in running state. |
Dependent item | kube.pod.containers_running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Ready | Describes whether the pod is ready to serve requests. |
Dependent item | kube.pod.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Scheduled | Describes the status of the scheduling process for the pod. |
Dependent item | kube.pod.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Unschedulable | Describes the unschedulable status for the pod. |
Dependent item | kube.pod.unschedulable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU limits | The limit on CPU cores to be used by a container. |
Dependent item | kube.pod.containers.limits.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory limits | The limit on memory to be used by a container. |
Dependent item | kube.pod.containers.limits.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU requests | The number of requested CPU cores by a container. |
Dependent item | kube.pod.containers.requests.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory requests | The number of requested memory bytes by a container. |
Dependent item | kube.pod.containers.requests.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is not healthy | min(/Kubernetes cluster state by HTTP/kube.pod.phase.failed[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.pending[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}],10m)>0 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],15m))>1 |
Warning |
LLD rule ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ReplicaSet discovery | Dependent item | kube.replicaset.discovery Preprocessing
|
Item prototypes for ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas | The number of replicas per ReplicaSet. |
Dependent item | kube.replicaset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Desired replicas | Number of desired pods for a ReplicaSet. |
Dependent item | kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Fully labeled replicas | The number of fully labeled replicas per ReplicaSet. |
Dependent item | kube.replicaset.fully_labeled_replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Ready | The number of ready replicas per ReplicaSet. |
Dependent item | kube.replicaset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the desired number of replicas. |
Dependent item | kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for ReplicaSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] RS [{#NAME}]: ReplicaSet mismatch | ReplicaSet has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"replicaset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| StatefulSet discovery | Dependent item | kube.statefulset.discovery Preprocessing
|
Item prototypes for StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas | The number of replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Desired replicas | Number of desired pods for a StatefulSet. |
Dependent item | kube.statefulset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Current replicas | The number of current replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Ready replicas | The number of ready replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Updated replicas | The number of updated replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the number of replicas. |
Dependent item | kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for StatefulSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet is down | (last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]) / last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}]))<>1 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet replicas mismatch | StatefulSet has not matched the number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"statefulset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PodDisruptionBudget discovery | Dependent item | kube.pdb.discovery Preprocessing
|
Item prototypes for PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods healthy | Current number of healthy pods. |
Dependent item | kube.pdb.pods_healthy[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods desired | Minimum desired number of healthy pods. |
Dependent item | kube.pdb.pods_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Disruptions allowed | Number of pod disruptions that are allowed. |
Dependent item | kube.pdb.disruptions_allowed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods total | Total number of pods counted by this disruption budget. |
Dependent item | kube.pdb.pods_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| CronJob discovery | Dependent item | kube.cronjob.discovery Preprocessing
|
Item prototypes for CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Suspend | Suspend flag tells the controller to suspend subsequent executions. |
Dependent item | kube.cronjob.spec_suspend[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Active | Active holds pointers to currently running jobs. |
Dependent item | kube.cronjob.status_active[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Last schedule | LastScheduleTime keeps information of when was the last time the job was successfully scheduled. |
Dependent item | kube.cronjob.last_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Next schedule | Next time the cronjob should be scheduled. The time after lastScheduleTime or after the cron job's creation time if it's never been scheduled. Use this to determine if the job is delayed. |
Dependent item | kube.cronjob.next_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.cronjob.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.cronjob.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.cronjob.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.cronjob.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Job discovery | Dependent item | kube.job.discovery Preprocessing
|
Item prototypes for Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.job.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.job.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.job.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.job.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component statuses discovery | Dependent item | kube.componentstatuses.discovery Preprocessing
|
Item prototypes for Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component [{#NAME}]: Healthy | Cluster component healthy. |
Dependent item | kube.componentstatuses.healthy[{#NAME}] Preprocessing
|
Trigger prototypes for Component statuses discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Component [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}],#2,"ne","True")=2 and length(last(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}]))>0 |
Warning |
LLD rule Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz discovery | Dependent item | kube.readyz.discovery Preprocessing
|
Item prototypes for Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz [{#NAME}]: Healthcheck | Result of readyz healthcheck for component. |
Dependent item | kube.readyz.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Readyz discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Readyz [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez discovery | Dependent item | kube.livez.discovery Preprocessing
|
Item prototypes for Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez [{#NAME}]: Healthcheck | Result of livez healthcheck for component. |
Dependent item | kube.livez.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Livez discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Livez [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift BuildConfig discovery | Dependent item | openshift.buildconfig.discovery Preprocessing
|
Item prototypes for OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Created | OpenShift BuildConfig Unix creation timestamp. |
Dependent item | openshift.buildconfig.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.buildconfig.generation[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Latest version | The latest version of BuildConfig. |
Dependent item | openshift.buildconfig.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Build discovery | Dependent item | openshift.build.discovery Preprocessing
|
Item prototypes for OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Created | OpenShift Build Unix creation timestamp. |
Dependent item | openshift.build.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.build.sequence.number[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Status phase | The Build phase. |
Dependent item | openshift.build.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Build discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Build [{#NAME}]: Build has failed | count(/Kubernetes cluster state by HTTP/openshift.build.status_phase[{#NAMESPACE}/{#NAME}],2m,"ge",6)>=2 |
Warning |
LLD rule OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift ClusterResourceQuota discovery | Dependent item | openshift.cluster.resource.quota.discovery Preprocessing
|
Item prototypes for OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Quota [{#NAME}] Resource [{#RESOURCE}]: Type [{#TYPE}]] | Usage about resource quota. |
Dependent item | openshift.cluster.resource.quota[{#RESOURCE}/{#NAME}/{#TYPE}] Preprocessing
|
LLD rule OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Route discovery | Dependent item | openshift.route.discovery Preprocessing
|
Item prototypes for OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Created | OpenShift Route Unix creation timestamp. |
Dependent item | openshift.route.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Status | Information about route status. |
Dependent item | openshift.route.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Route discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Route [{#NAME}] with issue: Status is false | count(/Kubernetes cluster state by HTTP/openshift.route.status[{#NAMESPACE}/{#NAME}],2m,,0)>=2 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes cluster state by HTTP
Overview
The template to monitor Kubernetes state. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes cluster state by HTTP - collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget to change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Requirements
Zabbix version: 7.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the service account name. If a different release name is used.
kubectl get serviceaccounts -n monitoring
Get the generated service account token using the command:
kubectl get secret zabbix-zabbix-helm-chart -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.STATE.ENDPOINT.NAME} with Kube state metrics endpoint name. See kubectl -n monitoring get ep. Default: zabbix-kube-state-metrics.
Note: If you wish to monitor Controller Manager and Scheduler components, you might need to set the --binding-address option for them to the address where Zabbix proxy can reach them.
For example, for clusters created with kubeadm it can be set in the following manifest files (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-scheduler.yaml
Depending on your Kubernetes distribution, you might need to adjust {$KUBE.CONTROL_PLANE.TAINT} macro (for example, set it to node-role.kubernetes.io/master for OpenShift).
Note: Some metrics may not be collected depending on your Kubernetes version and configuration.
Also, see the Macros section for a list of macros used to set trigger values.
Set up the macros to filter the metrics of discovered Kubelets by node names:
- {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES}
- {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Set up macros to filter node metrics by nodename:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable namespaces.
You can use the {$KUBE.KUBELET.FILTER.LABELS} and {$KUBE.KUBELET.FILTER.ANNOTATIONS} macros for advanced filtering of kubelets by node labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such label key exists for the entity that is being filtered (it means that if you specify a key in the filter, entities that do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the kubelets on nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
You can also set up evaluation periods for replica mismatch triggers (Deployments, ReplicaSets, StatefulSets) with the macro {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD}, which supports context and regular expressions. For example, you can create the following macros:
- Set the evaluation period for the Deployment "nginx-deployment" in the namespace "default" to the 3 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:default:nginx-deployment"} = #3
- Set the evaluation period for all Deployments to the 10 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"deployment:.*:.*"} = #10 or {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"^deployment.*"} = #10
- Set the evaluation period for Deployments, ReplicaSets and StatefulSets in the namespace "default" to 15 minutes:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:".*:default:.*"} = 15m
Note: that different context macros with regular expressions matching the same string can be applied in an undefined order, and simple context macros (without regular expressions) have higher priority. Read the Important notes section in Zabbix documentation for details.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.READYZ.ENDPOINT} | Kubernetes API readyz endpoint /readyz |
/readyz |
| {$KUBE.API.LIVEZ.ENDPOINT} | Kubernetes API livez endpoint /livez |
/livez |
| {$KUBE.API.COMPONENTSTATUSES.ENDPOINT} | Kubernetes API componentstatuses endpoint /api/v1/componentstatuses |
/api/v1/componentstatuses |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$HTTP.TLS.VERIFY} | TLS certificate verification for script items: "none" - disabled, "peer" - verify the certificate chain and expiration, "full" - full verification. Any other value enables full verification. To override the setting for this template only, define the macro with the context "Kubernetes cluster state", e.g. {$HTTP.TLS.VERIFY:"Kubernetes cluster state"}. |
full |
| {$KUBE.STATE.ENDPOINT.NAME} | Endpoint name for kube-state-metrics service (check with |
zabbix-kube-state-metrics |
| {$OPENSHIFT.STATE.ENDPOINT.NAME} | OpenShift state endpoint name. |
openshift-state-metrics |
| {$KUBE.API_SERVER.SCHEME} | Kubernetes API servers metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.API_SERVER.PORT} | Kubernetes API servers metrics endpoint port. Used in ControlPlane LLD. |
6443 |
| {$KUBE.CONTROL_PLANE.TAINT} | Taint that applies to control plane nodes. Change if needed. Used in ControlPlane LLD. |
node-role.kubernetes.io/control-plane |
| {$KUBE.CONTROLLER_MANAGER.SCHEME} | Kubernetes Controller manager metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.CONTROLLER_MANAGER.PORT} | Kubernetes Controller manager metrics endpoint port. Used in ControlPlane LLD. |
10257 |
| {$KUBE.SCHEDULER.SCHEME} | Kubernetes Scheduler metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.SCHEDULER.PORT} | Kubernetes Scheduler metrics endpoint port. Used in ControlPlane LLD. |
10259 |
| {$KUBE.KUBELET.SCHEME} | Kubernetes Kubelet metrics endpoint scheme. Used in Kubelet LLD. |
https |
| {$KUBE.KUBELET.PORT} | Kubernetes Kubelet metrics endpoint port. Used in Kubelet LLD. |
10250 |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable metrics by namespace. |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered metrics by namespace. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES} | Filter of discoverable Kubelets by nodename. |
.* |
| {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES} | Filter to exclude discovered Kubelets by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.KUBELET.FILTER.ANNOTATIONS} | Node annotations to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.KUBELET.FILTER.LABELS} | Node labels to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.PV.MATCHES} | Filter of discoverable persistent volumes by name. |
.* |
| {$KUBE.LLD.FILTER.PV.NOT_MATCHES} | Filter to exclude discovered persistent volumes by name. |
CHANGE_IF_NEEDED |
| {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD} | The evaluation period range which is used for calculation of expressions in trigger prototypes (time period or value range). Can be used with context. |
#5 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
Script | kube.state.metrics |
| Control plane LLD | Generation of data for Control plane discovery rules. |
Script | kube.control_plane.lld Preprocessing
|
| Node LLD | Generation of data for Kubelet discovery rules. |
Script | kube.node.lld Preprocessing
|
| Get component statuses | HTTP agent | kube.componentstatuses Preprocessing
|
|
| Get readyz | HTTP agent | kube.readyz Preprocessing
|
|
| Get livez | HTTP agent | kube.livez Preprocessing
|
|
| Namespace count | The number of namespaces. |
Dependent item | kube.namespace.count Preprocessing
|
| CronJob count | Number of cronjobs. |
Dependent item | kube.cronjob.count Preprocessing
|
| Job count | Number of jobs (generated by cronjob + job). |
Dependent item | kube.job.count Preprocessing
|
| Endpoint count | Number of endpoints. |
Dependent item | kube.endpoint.count Preprocessing
|
| Deployment count | The number of deployments. |
Dependent item | kube.deployment.count Preprocessing
|
| Service count | The number of services. |
Dependent item | kube.service.count Preprocessing
|
| StatefulSet count | The number of statefulsets. |
Dependent item | kube.statefulset.count Preprocessing
|
| Node count | The number of nodes. |
Dependent item | kube.node.count Preprocessing
|
LLD rule API servers discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| API servers discovery | Dependent item | kube.api_servers.discovery |
LLD rule Controller manager nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Controller manager nodes discovery | Dependent item | kube.controller_manager.discovery |
LLD rule Scheduler servers nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduler servers nodes discovery | Dependent item | kube.scheduler.discovery |
LLD rule Kubelet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubelet discovery | Dependent item | kube.kubelet.discovery |
LLD rule Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Daemonset discovery | Dependent item | kube.daemonset.discovery Preprocessing
|
Item prototypes for Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Ready | The number of nodes that should be running the daemon pod and have one or more running and ready. |
Dependent item | kube.daemonset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Scheduled | The number of nodes that run at least one daemon pod and are supposed to. |
Dependent item | kube.daemonset.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Desired | The number of nodes that should be running the daemon pod. |
Dependent item | kube.daemonset.desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Misscheduled | The number of nodes that run a daemon pod but are not supposed to. |
Dependent item | kube.daemonset.misscheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Updated number scheduled | The total number of nodes that are running updated daemon pod. |
Dependent item | kube.daemonset.updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PVC discovery | Dependent item | kube.pvc.discovery Preprocessing
|
Item prototypes for PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase | The current status phase of the persistent volume claim. |
Dependent item | kube.pvc.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
Dependent item | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Bound, sum | The total amount of persistent volume claims in the Bound phase. |
Dependent item | kube.pvc.status_phase.bound.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Lost, sum | The total amount of persistent volume claims in the Lost phase. |
Dependent item | kube.pvc.status_phase.lost.sum[{#NAMESPACE}] Preprocessing
|
| Namespace [{#NAMESPACE}] PVC status phase: Pending, sum | The total amount of persistent volume claims in the Pending phase. |
Dependent item | kube.pvc.status_phase.pending.sum[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for PVC discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | count(/Kubernetes cluster state by HTTP/kube.pvc.status_phase[{#NAMESPACE}/{#NAME}],2m,,5)>=2 |
Warning |
LLD rule PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV discovery | Dependent item | kube.pv.discovery Preprocessing
|
Item prototypes for PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV [{#NAME}] Status phase | The current status phase of the persistent volume. |
Dependent item | kube.pv.status_phase[{#NAME}] Preprocessing
|
| PV [{#NAME}] Capacity bytes | A capacity of the persistent volume in bytes. |
Dependent item | kube.pv.capacity.bytes[{#NAME}] Preprocessing
|
| PV status phase: Pending, sum | The total amount of persistent volumes in the Pending phase. |
Dependent item | kube.pv.status_phase.pending.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Available, sum | The total amount of persistent volumes in the Available phase. |
Dependent item | kube.pv.status_phase.available.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Bound, sum | The total amount of persistent volumes in the Bound phase. |
Dependent item | kube.pv.status_phase.bound.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Released, sum | The total amount of persistent volumes in the Released phase. |
Dependent item | kube.pv.status_phase.released.sum[{#SINGLETON}] Preprocessing
|
| PV status phase: Failed, sum | The total amount of persistent volumes in the Failed phase. |
Dependent item | kube.pv.status_phase.failed.sum[{#SINGLETON}] Preprocessing
|
Trigger prototypes for PV discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: PV [{#NAME}]: PV has failed | count(/Kubernetes cluster state by HTTP/kube.pv.status_phase[{#NAME}],2m,,3)>=2 |
Warning |
LLD rule Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment discovery | Dependent item | kube.deployment.discovery Preprocessing
|
Item prototypes for Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Paused | Whether the deployment is paused and will not be processed by the deployment controller. |
Dependent item | kube.deployment.spec_paused[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas desired | Number of desired pods for a deployment. |
Dependent item | kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Rollingupdate max unavailable | Maximum number of unavailable replicas during a rolling update of a deployment. |
Dependent item | kube.deployment.rollingupdate.max_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas | The number of replicas per deployment. |
Dependent item | kube.deployment.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas available | The number of available replicas per deployment. |
Dependent item | kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas unavailable | The number of unavailable replicas per deployment. |
Dependent item | kube.deployment.replicas_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas updated | The number of updated replicas per deployment. |
Dependent item | kube.deployment.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas mismatched | The number of available replicas not matching the desired number of replicas. |
Dependent item | kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Deployment discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Deployment replicas mismatch | Deployment has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Endpoint discovery | Dependent item | kube.endpoint.discovery Preprocessing
|
Item prototypes for Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address available | Number of addresses available in endpoint. |
Dependent item | kube.endpoint.address_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address not ready | Number of addresses not ready in endpoint. |
Dependent item | kube.endpoint.address_not_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Age | Endpoint age (number of seconds since creation). |
Dependent item | kube.endpoint.age[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node [{#NAME}]: CPU allocatable | The CPU resources of a node that are available for scheduling. |
Dependent item | kube.node.cpu_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory allocatable | The memory resources of a node that are available for scheduling. |
Dependent item | kube.node.memory_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods allocatable | The pods resources of a node that are available for scheduling. |
Dependent item | kube.node.pods_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage allocatable | The allocatable ephemeral storage of a node that is available for scheduling. |
Dependent item | kube.node.ephemeral_storage_allocatable[{#NAME}] Preprocessing
|
| Node [{#NAME}]: CPU capacity | The capacity for CPU resources of a node. |
Dependent item | kube.node.cpu_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Memory capacity | The capacity for memory resources of a node. |
Dependent item | kube.node.memory_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Ephemeral storage capacity | The ephemeral storage capacity of a node. |
Dependent item | kube.node.ephemeral_storage_capacity[{#NAME}] Preprocessing
|
| Node [{#NAME}]: Pods capacity | The capacity for pods resources of a node. |
Dependent item | kube.node.pods_capacity[{#NAME}] Preprocessing
|
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Pending | Pod is in pending state. |
Dependent item | kube.pod.phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Succeeded | Pod is in succeeded state. |
Dependent item | kube.pod.phase.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Failed | Pod is in failed state. |
Dependent item | kube.pod.phase.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Unknown | Pod is in unknown state. |
Dependent item | kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Running | Pod is in unknown state. |
Dependent item | kube.pod.phase.running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers terminated | Describes whether the container is currently in terminated state. |
Dependent item | kube.pod.containers_terminated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers waiting | Describes whether the container is currently in waiting state. |
Dependent item | kube.pod.containers_waiting[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers ready | Describes whether the containers readiness check succeeded. |
Dependent item | kube.pod.containers_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
Dependent item | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers running | Describes whether the container is currently in running state. |
Dependent item | kube.pod.containers_running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Ready | Describes whether the pod is ready to serve requests. |
Dependent item | kube.pod.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Scheduled | Describes the status of the scheduling process for the pod. |
Dependent item | kube.pod.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Unschedulable | Describes the unschedulable status for the pod. |
Dependent item | kube.pod.unschedulable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU limits | The limit on CPU cores to be used by a container. |
Dependent item | kube.pod.containers.limits.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory limits | The limit on memory to be used by a container. |
Dependent item | kube.pod.containers.limits.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU requests | The number of requested CPU cores by a container. |
Dependent item | kube.pod.containers.requests.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory requests | The number of requested memory bytes by a container. |
Dependent item | kube.pod.containers.requests.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is not healthy | min(/Kubernetes cluster state by HTTP/kube.pod.phase.failed[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.pending[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}],10m)>0 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],15m))>1 |
Warning |
LLD rule ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ReplicaSet discovery | Dependent item | kube.replicaset.discovery Preprocessing
|
Item prototypes for ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas | The number of replicas per ReplicaSet. |
Dependent item | kube.replicaset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Desired replicas | Number of desired pods for a ReplicaSet. |
Dependent item | kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Fully labeled replicas | The number of fully labeled replicas per ReplicaSet. |
Dependent item | kube.replicaset.fully_labeled_replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Ready | The number of ready replicas per ReplicaSet. |
Dependent item | kube.replicaset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the desired number of replicas. |
Dependent item | kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for ReplicaSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] RS [{#NAME}]: ReplicaSet mismatch | ReplicaSet has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"replicaset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| StatefulSet discovery | Dependent item | kube.statefulset.discovery Preprocessing
|
Item prototypes for StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas | The number of replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Desired replicas | Number of desired pods for a StatefulSet. |
Dependent item | kube.statefulset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Current replicas | The number of current replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Ready replicas | The number of ready replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Updated replicas | The number of updated replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the number of replicas. |
Dependent item | kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for StatefulSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet is down | (last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]) / last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}]))<>1 |
High | ||
| Kubernetes cluster state: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet replicas mismatch | StatefulSet has not matched the number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"statefulset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PodDisruptionBudget discovery | Dependent item | kube.pdb.discovery Preprocessing
|
Item prototypes for PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods healthy | Current number of healthy pods. |
Dependent item | kube.pdb.pods_healthy[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods desired | Minimum desired number of healthy pods. |
Dependent item | kube.pdb.pods_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Disruptions allowed | Number of pod disruptions that are allowed. |
Dependent item | kube.pdb.disruptions_allowed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods total | Total number of pods counted by this disruption budget. |
Dependent item | kube.pdb.pods_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| CronJob discovery | Dependent item | kube.cronjob.discovery Preprocessing
|
Item prototypes for CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Suspend | Suspend flag tells the controller to suspend subsequent executions. |
Dependent item | kube.cronjob.spec_suspend[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Active | Active holds pointers to currently running jobs. |
Dependent item | kube.cronjob.status_active[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Last schedule | LastScheduleTime keeps information of when was the last time the job was successfully scheduled. |
Dependent item | kube.cronjob.last_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Next schedule | Next time the cronjob should be scheduled. The time after lastScheduleTime or after the cron job's creation time if it's never been scheduled. Use this to determine if the job is delayed. |
Dependent item | kube.cronjob.next_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.cronjob.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.cronjob.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.cronjob.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.cronjob.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Job discovery | Dependent item | kube.job.discovery Preprocessing
|
Item prototypes for Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.job.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.job.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.job.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.job.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component statuses discovery | Dependent item | kube.componentstatuses.discovery Preprocessing
|
Item prototypes for Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component [{#NAME}]: Healthy | Cluster component healthy. |
Dependent item | kube.componentstatuses.healthy[{#NAME}] Preprocessing
|
Trigger prototypes for Component statuses discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Component [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}],#2,"ne","True")=2 and length(last(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}]))>0 |
Warning |
LLD rule Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz discovery | Dependent item | kube.readyz.discovery Preprocessing
|
Item prototypes for Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz [{#NAME}]: Healthcheck | Result of readyz healthcheck for component. |
Dependent item | kube.readyz.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Readyz discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Readyz [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez discovery | Dependent item | kube.livez.discovery Preprocessing
|
Item prototypes for Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez [{#NAME}]: Healthcheck | Result of livez healthcheck for component. |
Dependent item | kube.livez.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Livez discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Livez [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift BuildConfig discovery | Dependent item | openshift.buildconfig.discovery Preprocessing
|
Item prototypes for OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Created | OpenShift BuildConfig Unix creation timestamp. |
Dependent item | openshift.buildconfig.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.buildconfig.generation[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Latest version | The latest version of BuildConfig. |
Dependent item | openshift.buildconfig.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Build discovery | Dependent item | openshift.build.discovery Preprocessing
|
Item prototypes for OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Created | OpenShift Build Unix creation timestamp. |
Dependent item | openshift.build.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.build.sequence.number[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Build [{#NAME}]: Status phase | The Build phase. |
Dependent item | openshift.build.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Build discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Build [{#NAME}]: Build has failed | count(/Kubernetes cluster state by HTTP/openshift.build.status_phase[{#NAMESPACE}/{#NAME}],2m,"ge",6)>=2 |
Warning |
LLD rule OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift ClusterResourceQuota discovery | Dependent item | openshift.cluster.resource.quota.discovery Preprocessing
|
Item prototypes for OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Quota [{#NAME}] Resource [{#RESOURCE}]: Type [{#TYPE}]] | Usage about resource quota. |
Dependent item | openshift.cluster.resource.quota[{#RESOURCE}/{#NAME}/{#TYPE}] Preprocessing
|
LLD rule OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Route discovery | Dependent item | openshift.route.discovery Preprocessing
|
Item prototypes for OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Created | OpenShift Route Unix creation timestamp. |
Dependent item | openshift.route.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Namespace [{#NAMESPACE}] Route [{#NAME}]: Status | Information about route status. |
Dependent item | openshift.route.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Route discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes cluster state: Route [{#NAME}] with issue: Status is false | count(/Kubernetes cluster state by HTTP/openshift.route.status[{#NAMESPACE}/{#NAME}],2m,,0)>=2 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes cluster state by HTTP
Overview
The template to monitor Kubernetes state. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes cluster state by HTTP - collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget to change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
Requirements
Zabbix version: 6.4 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command:
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.STATE.ENDPOINT.NAME} with Kube state metrics endpoint name. See kubectl -n monitoring get ep. Default: zabbix-kube-state-metrics.
NOTE. If you wish to monitor Controller Manager and Scheduler components, you might need to set the --binding-address option for them to the address where Zabbix proxy can reach them.
For example, for clusters created with kubeadm it can be set in the following manifest files (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-scheduler.yaml
Depending on your Kubernetes distribution, you might need to adjust {$KUBE.CONTROL_PLANE.TAINT} macro (for example, set it to node-role.kubernetes.io/master for OpenShift).
NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
Also, see the Macros section for a list of macros used to set trigger values.
Set up the macros to filter the metrics of discovered Kubelets by node names:
- {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES}
- {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Set up macros to filter node metrics by nodename:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable namespaces.
You can use the {$KUBE.KUBELET.FILTER.LABELS} and {$KUBE.KUBELET.FILTER.ANNOTATIONS} macros for advanced filtering of kubelets by node labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such label key exists for the entity that is being filtered (it means that if you specify a key in the filter, entities that do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the kubelets on nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
You can also set up evaluation periods for replica mismatch triggers (Deployments, ReplicaSets, StatefulSets) with the macro {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD}, which supports context and regular expressions. For example, you can create the following macros:
- Set the evaluation period for the Deployment "nginx-deployment" in the namespace "default" to the 3 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:default:nginx-deployment"} = #3
- Set the evaluation period for all Deployments to the 10 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"deployment:.*:.*"} = #10 or {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"^deployment.*"} = #10
- Set the evaluation period for Deployments, ReplicaSets and StatefulSets in the namespace "default" to 15 minutes:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:".*:default:.*"} = 15m
Note that different context macros with regular expressions matching the same string can be applied in an undefined order, and simple context macros (without regular expressions) have higher priority. Read the Important notes section in Zabbix documentation for details.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.READYZ.ENDPOINT} | Kubernetes API readyz endpoint /readyz |
/readyz |
| {$KUBE.API.LIVEZ.ENDPOINT} | Kubernetes API livez endpoint /livez |
/livez |
| {$KUBE.API.COMPONENTSTATUSES.ENDPOINT} | Kubernetes API componentstatuses endpoint /api/v1/componentstatuses |
/api/v1/componentstatuses |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$KUBE.STATE.ENDPOINT.NAME} | Kubernetes state endpoint name. |
zabbix-kube-state-metrics |
| {$OPENSHIFT.STATE.ENDPOINT.NAME} | OpenShift state endpoint name. |
openshift-state-metrics |
| {$KUBE.API_SERVER.SCHEME} | Kubernetes API servers metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.API_SERVER.PORT} | Kubernetes API servers metrics endpoint port. Used in ControlPlane LLD. |
6443 |
| {$KUBE.CONTROL_PLANE.TAINT} | Taint that applies to control plane nodes. Change if needed. Used in ControlPlane LLD. |
node-role.kubernetes.io/control-plane |
| {$KUBE.CONTROLLER_MANAGER.SCHEME} | Kubernetes Controller manager metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.CONTROLLER_MANAGER.PORT} | Kubernetes Controller manager metrics endpoint port. Used in ControlPlane LLD. |
10257 |
| {$KUBE.SCHEDULER.SCHEME} | Kubernetes Scheduler metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.SCHEDULER.PORT} | Kubernetes Scheduler metrics endpoint port. Used in ControlPlane LLD. |
10259 |
| {$KUBE.KUBELET.SCHEME} | Kubernetes Kubelet metrics endpoint scheme. Used in Kubelet LLD. |
https |
| {$KUBE.KUBELET.PORT} | Kubernetes Kubelet metrics endpoint port. Used in Kubelet LLD. |
10250 |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable metrics by namespace. |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered metrics by namespace. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES} | Filter of discoverable Kubelets by nodename. |
.* |
| {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES} | Filter to exclude discovered Kubelets by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.KUBELET.FILTER.ANNOTATIONS} | Node annotations to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.KUBELET.FILTER.LABELS} | Node labels to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.PV.MATCHES} | Filter of discoverable persistent volumes by name. |
.* |
| {$KUBE.LLD.FILTER.PV.NOT_MATCHES} | Filter to exclude discovered persistent volumes by name. |
CHANGE_IF_NEEDED |
| {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD} | The evaluation period range which is used for calculation of expressions in trigger prototypes (time period or value range). Can be used with context. |
#5 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
Script | kube.state.metrics |
| Kubernetes: Control plane LLD | Generation of data for Control plane discovery rules. |
Script | kube.control_plane.lld Preprocessing
|
| Kubernetes: Node LLD | Generation of data for Kubelet discovery rules. |
Script | kube.node.lld Preprocessing
|
| Kubernetes: Get component statuses | HTTP agent | kube.componentstatuses Preprocessing
|
|
| Kubernetes: Get readyz | HTTP agent | kube.readyz Preprocessing
|
|
| Kubernetes: Get livez | HTTP agent | kube.livez Preprocessing
|
|
| Kubernetes: Namespace count | The number of namespaces. |
Dependent item | kube.namespace.count Preprocessing
|
| Kubernetes: CronJob count | Number of cronjobs. |
Dependent item | kube.cronjob.count Preprocessing
|
| Kubernetes: Job count | Number of jobs (generated by cronjob + job). |
Dependent item | kube.job.count Preprocessing
|
| Kubernetes: Endpoint count | Number of endpoints. |
Dependent item | kube.endpoint.count Preprocessing
|
| Kubernetes: Deployment count | The number of deployments. |
Dependent item | kube.deployment.count Preprocessing
|
| Kubernetes: Service count | The number of services. |
Dependent item | kube.service.count Preprocessing
|
| Kubernetes: StatefulSet count | The number of statefulsets. |
Dependent item | kube.statefulset.count Preprocessing
|
| Kubernetes: Node count | The number of nodes. |
Dependent item | kube.node.count Preprocessing
|
LLD rule API servers discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| API servers discovery | Dependent item | kube.api_servers.discovery |
LLD rule Controller manager nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Controller manager nodes discovery | Dependent item | kube.controller_manager.discovery |
LLD rule Scheduler servers nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduler servers nodes discovery | Dependent item | kube.scheduler.discovery |
LLD rule Kubelet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubelet discovery | Dependent item | kube.kubelet.discovery |
LLD rule Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Daemonset discovery | Dependent item | kube.daemonset.discovery Preprocessing
|
Item prototypes for Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Ready | The number of nodes that should be running the daemon pod and have one or more running and ready. |
Dependent item | kube.daemonset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Scheduled | The number of nodes that run at least one daemon pod and are supposed to. |
Dependent item | kube.daemonset.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Desired | The number of nodes that should be running the daemon pod. |
Dependent item | kube.daemonset.desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Misscheduled | The number of nodes that run a daemon pod but are not supposed to. |
Dependent item | kube.daemonset.misscheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Updated number scheduled | The total number of nodes that are running updated daemon pod. |
Dependent item | kube.daemonset.updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PVC discovery | Dependent item | kube.pvc.discovery Preprocessing
|
Item prototypes for PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase | The current status phase of the persistent volume claim. |
Dependent item | kube.pvc.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
Dependent item | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC status phase: Bound, sum | The total amount of persistent volume claims in the Bound phase. |
Dependent item | kube.pvc.status_phase.bound.sum[{#NAMESPACE}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC status phase: Lost, sum | The total amount of persistent volume claims in the Lost phase. |
Dependent item | kube.pvc.status_phase.lost.sum[{#NAMESPACE}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC status phase: Pending, sum | The total amount of persistent volume claims in the Pending phase. |
Dependent item | kube.pvc.status_phase.pending.sum[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for PVC discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | count(/Kubernetes cluster state by HTTP/kube.pvc.status_phase[{#NAMESPACE}/{#NAME}],2m,,5)>=2 |
Warning |
LLD rule PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV discovery | Dependent item | kube.pv.discovery Preprocessing
|
Item prototypes for PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: PV [{#NAME}] Status phase | The current status phase of the persistent volume. |
Dependent item | kube.pv.status_phase[{#NAME}] Preprocessing
|
| Kubernetes: PV [{#NAME}] Capacity bytes | A capacity of the persistent volume in bytes. |
Dependent item | kube.pv.capacity.bytes[{#NAME}] Preprocessing
|
| Kubernetes: PV status phase: Pending, sum | The total amount of persistent volumes in the Pending phase. |
Dependent item | kube.pv.status_phase.pending.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Available, sum | The total amount of persistent volumes in the Available phase. |
Dependent item | kube.pv.status_phase.available.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Bound, sum | The total amount of persistent volumes in the Bound phase. |
Dependent item | kube.pv.status_phase.bound.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Released, sum | The total amount of persistent volumes in the Released phase. |
Dependent item | kube.pv.status_phase.released.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Failed, sum | The total amount of persistent volumes in the Failed phase. |
Dependent item | kube.pv.status_phase.failed.sum[{#SINGLETON}] Preprocessing
|
Trigger prototypes for PV discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: PV [{#NAME}]: PV has failed | count(/Kubernetes cluster state by HTTP/kube.pv.status_phase[{#NAME}],2m,,3)>=2 |
Warning |
LLD rule Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment discovery | Dependent item | kube.deployment.discovery Preprocessing
|
Item prototypes for Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Paused | Whether the deployment is paused and will not be processed by the deployment controller. |
Dependent item | kube.deployment.spec_paused[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas desired | Number of desired pods for a deployment. |
Dependent item | kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Rollingupdate max unavailable | Maximum number of unavailable replicas during a rolling update of a deployment. |
Dependent item | kube.deployment.rollingupdate.max_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas | The number of replicas per deployment. |
Dependent item | kube.deployment.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas available | The number of available replicas per deployment. |
Dependent item | kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas unavailable | The number of unavailable replicas per deployment. |
Dependent item | kube.deployment.replicas_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas updated | The number of updated replicas per deployment. |
Dependent item | kube.deployment.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas mismatched | The number of available replicas not matching the desired number of replicas. |
Dependent item | kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Deployment discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Deployment replicas mismatch | Deployment has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Endpoint discovery | Dependent item | kube.endpoint.discovery Preprocessing
|
Item prototypes for Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address available | Number of addresses available in endpoint. |
Dependent item | kube.endpoint.address_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address not ready | Number of addresses not ready in endpoint. |
Dependent item | kube.endpoint.address_not_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Age | Endpoint age (number of seconds since creation). |
Dependent item | kube.endpoint.age[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Node [{#NAME}]: CPU allocatable | The CPU resources of a node that are available for scheduling. |
Dependent item | kube.node.cpu_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Memory allocatable | The memory resources of a node that are available for scheduling. |
Dependent item | kube.node.memory_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Pods allocatable | The pods resources of a node that are available for scheduling. |
Dependent item | kube.node.pods_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Ephemeral storage allocatable | The allocatable ephemeral storage of a node that is available for scheduling. |
Dependent item | kube.node.ephemeral_storage_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: CPU capacity | The capacity for CPU resources of a node. |
Dependent item | kube.node.cpu_capacity[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Memory capacity | The capacity for memory resources of a node. |
Dependent item | kube.node.memory_capacity[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Ephemeral storage capacity | The ephemeral storage capacity of a node. |
Dependent item | kube.node.ephemeral_storage_capacity[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Pods capacity | The capacity for pods resources of a node. |
Dependent item | kube.node.pods_capacity[{#NAME}] Preprocessing
|
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Pending | Pod is in pending state. |
Dependent item | kube.pod.phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Succeeded | Pod is in succeeded state. |
Dependent item | kube.pod.phase.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Failed | Pod is in failed state. |
Dependent item | kube.pod.phase.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Unknown | Pod is in unknown state. |
Dependent item | kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Running | Pod is in unknown state. |
Dependent item | kube.pod.phase.running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers terminated | Describes whether the container is currently in terminated state. |
Dependent item | kube.pod.containers_terminated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers waiting | Describes whether the container is currently in waiting state. |
Dependent item | kube.pod.containers_waiting[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers ready | Describes whether the containers readiness check succeeded. |
Dependent item | kube.pod.containers_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
Dependent item | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers running | Describes whether the container is currently in running state. |
Dependent item | kube.pod.containers_running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Ready | Describes whether the pod is ready to serve requests. |
Dependent item | kube.pod.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Scheduled | Describes the status of the scheduling process for the pod. |
Dependent item | kube.pod.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Unschedulable | Describes the unschedulable status for the pod. |
Dependent item | kube.pod.unschedulable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU limits | The limit on CPU cores to be used by a container. |
Dependent item | kube.pod.containers.limits.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory limits | The limit on memory to be used by a container. |
Dependent item | kube.pod.containers.limits.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU requests | The number of requested CPU cores by a container. |
Dependent item | kube.pod.containers.requests.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory requests | The number of requested memory bytes by a container. |
Dependent item | kube.pod.containers.requests.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is not healthy | min(/Kubernetes cluster state by HTTP/kube.pod.phase.failed[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.pending[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}],10m)>0 |
High | ||
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],15m))>1 |
Warning |
LLD rule ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ReplicaSet discovery | Dependent item | kube.replicaset.discovery Preprocessing
|
Item prototypes for ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas | The number of replicas per ReplicaSet. |
Dependent item | kube.replicaset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Desired replicas | Number of desired pods for a ReplicaSet. |
Dependent item | kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Fully labeled replicas | The number of fully labeled replicas per ReplicaSet. |
Dependent item | kube.replicaset.fully_labeled_replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Ready | The number of ready replicas per ReplicaSet. |
Dependent item | kube.replicaset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the desired number of replicas. |
Dependent item | kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for ReplicaSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] RS [{#NAME}]: ReplicaSet mismatch | ReplicaSet has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"replicaset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| StatefulSet discovery | Dependent item | kube.statefulset.discovery Preprocessing
|
Item prototypes for StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas | The number of replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Desired replicas | Number of desired pods for a StatefulSet. |
Dependent item | kube.statefulset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Current replicas | The number of current replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Ready replicas | The number of ready replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Updated replicas | The number of updated replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the number of replicas. |
Dependent item | kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for StatefulSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet is down | (last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]) / last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}]))<>1 |
High | ||
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet replicas mismatch | StatefulSet has not matched the number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"statefulset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PodDisruptionBudget discovery | Dependent item | kube.pdb.discovery Preprocessing
|
Item prototypes for PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods healthy | Current number of healthy pods. |
Dependent item | kube.pdb.pods_healthy[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods desired | Minimum desired number of healthy pods. |
Dependent item | kube.pdb.pods_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Disruptions allowed | Number of pod disruptions that are allowed. |
Dependent item | kube.pdb.disruptions_allowed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods total | Total number of pods counted by this disruption budget. |
Dependent item | kube.pdb.pods_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| CronJob discovery | Dependent item | kube.cronjob.discovery Preprocessing
|
Item prototypes for CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Suspend | Suspend flag tells the controller to suspend subsequent executions. |
Dependent item | kube.cronjob.spec_suspend[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Active | Active holds pointers to currently running jobs. |
Dependent item | kube.cronjob.status_active[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Last schedule | LastScheduleTime keeps information of when was the last time the job was successfully scheduled. |
Dependent item | kube.cronjob.last_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Next schedule | Next time the cronjob should be scheduled. The time after lastScheduleTime or after the cron job's creation time if it's never been scheduled. Use this to determine if the job is delayed. |
Dependent item | kube.cronjob.next_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.cronjob.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.cronjob.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.cronjob.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.cronjob.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Job discovery | Dependent item | kube.job.discovery Preprocessing
|
Item prototypes for Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.job.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.job.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.job.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.job.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component statuses discovery | Dependent item | kube.componentstatuses.discovery Preprocessing
|
Item prototypes for Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Component [{#NAME}]: Healthy | Cluster component healthy. |
Dependent item | kube.componentstatuses.healthy[{#NAME}] Preprocessing
|
Trigger prototypes for Component statuses discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Component [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}],#3,,"True")<2 and length(last(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}]))>0 |
Warning |
LLD rule Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz discovery | Dependent item | kube.readyz.discovery Preprocessing
|
Item prototypes for Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Readyz [{#NAME}]: Healthcheck | Result of readyz healthcheck for component. |
Dependent item | kube.readyz.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Readyz discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Readyz [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}],#3,,"ok")<2 and length(last(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez discovery | Dependent item | kube.livez.discovery Preprocessing
|
Item prototypes for Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Livez [{#NAME}]: Healthcheck | Result of livez healthcheck for component. |
Dependent item | kube.livez.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Livez discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Livez [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}],#3,,"ok")<2 and length(last(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift BuildConfig discovery | Dependent item | openshift.buildconfig.discovery Preprocessing
|
Item prototypes for OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Created | OpenShift BuildConfig Unix creation timestamp. |
Dependent item | openshift.buildconfig.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.buildconfig.generation[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Latest version | The latest version of BuildConfig. |
Dependent item | openshift.buildconfig.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Build discovery | Dependent item | openshift.build.discovery Preprocessing
|
Item prototypes for OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Namespace [{#NAMESPACE}] Build [{#NAME}]: Created | OpenShift Build Unix creation timestamp. |
Dependent item | openshift.build.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] Build [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.build.sequence.number[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] Build [{#NAME}]: Status phase | The Build phase. |
Dependent item | openshift.build.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Build discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| OpenShift: Build [{#NAME}]: Build has failed | count(/Kubernetes cluster state by HTTP/openshift.build.status_phase[{#NAMESPACE}/{#NAME}],2m,"ge",6)>=2 |
Warning |
LLD rule OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift ClusterResourceQuota discovery | Dependent item | openshift.cluster.resource.quota.discovery Preprocessing
|
Item prototypes for OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Quota [{#NAME}] Resource [{#RESOURCE}]: Type [{#TYPE}]] | Usage about resource quota. |
Dependent item | openshift.cluster.resource.quota[{#RESOURCE}/{#NAME}/{#TYPE}] Preprocessing
|
LLD rule OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Route discovery | Dependent item | openshift.route.discovery Preprocessing
|
Item prototypes for OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Namespace [{#NAMESPACE}] Route [{#NAME}]: Created | OpenShift Route Unix creation timestamp. |
Dependent item | openshift.route.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] Route [{#NAME}]: Status | Information about route status. |
Dependent item | openshift.route.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Route discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| OpenShift: Route [{#NAME}] with issue: Status is false | count(/Kubernetes cluster state by HTTP/openshift.route.status[{#NAMESPACE}/{#NAME}],2m,,0)>=2 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
Kubernetes cluster state by HTTP
Overview
For Zabbix version: 6.2 and higher. The template to monitor Kubernetes state that work without any external scripts. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes cluster state by HTTP — collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values. NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
This template was tested on:
- Kubernetes, version 1.19
Setup
See Zabbix template operation for basic instructions.
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use Authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.STATE.ENDPOINT.NAME} with Kube state metrics endpoint name. See kubectl -n monitoring get ep. Default: zabbix-kube-state-metrics.
Also, see the Macros section for a list of macros used to set trigger values. NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
Set up the macros to filter the metrics of discovered worker nodes:
- {$KUBE.LLD.FILTER.WORKER_NODE.MATCHES}
- {$KUBE.LLD.FILTER.WORKER_NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Set up macros to filter node metrics by nodename:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Note, If you have a large cluster, it is highly recommended to set a filter for discoverable namespaces.
Zabbix configuration
No specific Zabbix configuration is required.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.COMPONENTSTATUSES.ENDPOINT} | Kubernetes API componentstatuses endpoint /api/v1/componentstatuses |
/api/v1/componentstatuses |
| {$KUBE.API.LIVEZ.ENDPOINT} | Kubernetes API livez endpoint /livez |
/livez |
| {$KUBE.API.READYZ.ENDPOINT} | Kubernetes API readyz endpoint /readyz |
/readyz |
| {$KUBE.API.TOKEN} | Service account bearer token |
`` |
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://localhost:6443 |
| {$KUBE.API_SERVER.PORT} | Kubernetes API servers metrics endpoint port. Used in ControlPlane LLD. |
6443 |
| {$KUBE.API_SERVER.SCHEME} | Kubernetes API servers metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.CONTROLLER_MANAGER.PORT} | Kubernetes Controller manager metrics endpoint port. Used in ControlPlane LLD. |
10252 |
| {$KUBE.CONTROLLER_MANAGER.SCHEME} | Kubernetes Controller manager metrics endpoint scheme. Used in ControlPlane LLD. |
http |
| {$KUBE.KUBELET.PORT} | Kubernetes Kubelet manager metrics endpoint port. Used in Kubelet LLD. |
10250 |
| {$KUBE.KUBELET.SCHEME} | Kubernetes Kubelet manager metrics endpoint scheme. Used in Kubelet LLD. |
https |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable pods by namespace |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered pods by namespace |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.WORKER_NODE.MATCHES} | Filter of discoverable worker nodes by nodename |
.* |
| {$KUBE.LLD.FILTER.WORKER_NODE.NOT_MATCHES} | Filter to exclude discovered worker nodes by nodename |
CHANGE_IF_NEEDED |
| {$KUBE.SCHEDULER.PORT} | Kubernetes Scheduler manager metrics endpoint port. Used in ControlPlane LLD. |
10251 |
| {$KUBE.SCHEDULER.SCHEME} | Kubernetes Scheduler manager metrics endpoint scheme. Used in ControlPlane LLD. |
http |
| {$KUBE.STATE.ENDPOINT.NAME} | Kubernetes state endpoint name |
zabbix-kube-state-metrics |
Template links
There are no template links in this template.
Discovery rules
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| API servers discovery | - |
DEPENDENT | kube.api_servers.discovery |
| Component statuses discovery | - |
DEPENDENT | kube.componentstatuses.discovery Preprocessing: - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT |
| Controller manager nodes discovery | - |
DEPENDENT | kube.controller_manager.discovery |
| CronJob discovery | - |
DEPENDENT | kube.cronjob.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| Daemonset discovery | - |
DEPENDENT | kube.daemonset.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| Deployment discovery | - |
DEPENDENT | kube.deployment.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| Endpoint discovery | - |
DEPENDENT | kube.endpoint.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| Job discovery | - |
DEPENDENT | kube.job.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| Kubelet discovery | - |
DEPENDENT | kube.kubelet.discovery Filter: AND- {#NAME} MATCHES_REGEX - {#NAME} NOT_MATCHES_REGEX |
| Livez discovery | - |
DEPENDENT | kube.livez.discovery Preprocessing: - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT |
| Node discovery | - |
DEPENDENT | kube.node.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAME} MATCHES_REGEX - {#NAME} NOT_MATCHES_REGEX |
| Pod discovery | - |
DEPENDENT | kube.pod.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| PodDisruptionBudget discovery | - |
DEPENDENT | kube.pdb.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| PVC discovery | - |
DEPENDENT | kube.pvc.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| Readyz discovery | - |
DEPENDENT | kube.readyz.discovery Preprocessing: - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT |
| Replicaset discovery | - |
DEPENDENT | kube.replicaset.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
| Scheduler servers nodes discovery | - |
DEPENDENT | kube.scheduler.discovery |
| Statefulset discovery | - |
DEPENDENT | kube.statefulset.discovery Preprocessing: - PROMETHEUS_TO_JSON - JAVASCRIPT - DISCARD_UNCHANGED_HEARTBEAT Filter: AND- {#NAMESPACE} MATCHES_REGEX - {#NAMESPACE} NOT_MATCHES_REGEX |
Items collected
| Group | Name | Description | Type | Key and additional info |
|---|---|---|---|---|
| Kubernetes | Kubernetes: Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
SCRIPT | kube.state.metrics Expression: The text is too long. Please see the template. |
| Kubernetes | Kubernetes: Control plane LLD | Generation of data for Control plane discovery rules. |
SCRIPT | kube.control_plane.lld Preprocessing: - DISCARD_UNCHANGED_HEARTBEAT: Expression: The text is too long. Please see the template. |
| Kubernetes | Kubernetes: Node LLD | Generation of data for Kubelet discovery rules. |
SCRIPT | kube.node.lld Preprocessing: - DISCARD_UNCHANGED_HEARTBEAT: Expression: The text is too long. Please see the template. |
| Kubernetes | Kubernetes: Get component statuses | - |
HTTP_AGENT | kube.componentstatuses Preprocessing: - CHECK_NOT_SUPPORTED ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Get readyz | - |
HTTP_AGENT | kube.readyz Preprocessing: - JAVASCRIPT: |
| Kubernetes | Kubernetes: Get livez | - |
HTTP_AGENT | kube.livez Preprocessing: - JAVASCRIPT: |
| Kubernetes | Kubernetes: Namespace count | The number of namespaces. |
DEPENDENT | kube.namespace.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: CronJob count | Number of cronjobs. |
DEPENDENT | kube.cronjob.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Job count | Number of jobs(generated by cronjob + job). |
DEPENDENT | kube.job.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Endpoint count | Number of endpoints. |
DEPENDENT | kube.endpoint.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Deployment count | The number of deployments. |
DEPENDENT | kube.deployment.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Service count | The number of services. |
DEPENDENT | kube.service.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Statefulset count | The number of statefulsets. |
DEPENDENT | kube.statefulset.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node count | The number of nodes. |
DEPENDENT | kube.node.count Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Ready | The number of nodes that should be running the daemon pod and have one or more running and ready. |
DEPENDENT | kube.daemonset.ready[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Scheduled | The number of nodes running at least one daemon pod and are supposed to. |
DEPENDENT | kube.daemonset.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Desired | The number of nodes that should be running the daemon pod. |
DEPENDENT | kube.daemonset.desired[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Misscheduled | The number of nodes running a daemon pod but are not supposed to. |
DEPENDENT | kube.daemonset.misscheduled[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Updated number scheduled | The total number of nodes that are running updated daemon pod. |
DEPENDENT | kube.daemonset.updated[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase: Available | Persistent volume claim is currently in Active phase. |
DEPENDENT | kube.pvc.status_phase.active[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase: Lost | Persistent volume claim is currently in Lost phase. |
DEPENDENT | kube.pvc.status_phase.lost[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase: Bound | Persistent volume claim is currently in Bound phase. |
DEPENDENT | kube.pvc.status_phase.bound[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase: Pending | Persistent volume claim is currently in Pending phase. |
DEPENDENT | kube.pvc.status_phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
DEPENDENT | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Status phase: Pending, sum | Persistent volume claim is currently in Pending phase. |
DEPENDENT | kube.pvc.status_phase.pending.sum[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Status phase: Active, sum | Persistent volume claim is currently in Active phase. |
DEPENDENT | kube.pvc.status_phase.active.sum[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Status phase: Bound, sum | Persistent volume claim is currently in Bound phase. |
DEPENDENT | kube.pvc.status_phase.bound.sum[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Status phase: Lost, sum | Persistent volume claim is currently in Lost phase. |
DEPENDENT | kube.pvc.status_phase.lost.sum[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Paused | Whether the deployment is paused and will not be processed by the deployment controller. |
DEPENDENT | kube.deployment.spec_paused[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas desired | Number of desired pods for a deployment. |
DEPENDENT | kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Rollingupdate max unavailable | Maximum number of unavailable replicas during a rolling update of a deployment. |
DEPENDENT | kube.deployment.rollingupdate.max_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas | The number of replicas per deployment. |
DEPENDENT | kube.deployment.replicas[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas available | The number of available replicas per deployment. |
DEPENDENT | kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas unavailable | The number of unavailable replicas per deployment. |
DEPENDENT | kube.deployment.replicas_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas updated | The number of updated replicas per deployment. |
DEPENDENT | kube.deployment.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address available | Number of addresses available in endpoint. |
DEPENDENT | kube.endpoint.address_available[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address not ready | Number of addresses not ready in endpoint. |
DEPENDENT | kube.endpoint.address_not_ready[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Age | Endpoint age (number of seconds since creation). |
DEPENDENT | kube.endpoint.age[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - JAVASCRIPT: |
| Kubernetes | Kubernetes: Node [{#NAME}]: CPU allocatable | The CPU resources of a node that are available for scheduling. |
DEPENDENT | kube.node.cpu_allocatable[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node [{#NAME}]: Memory allocatable | The Memory resources of a node that are available for scheduling. |
DEPENDENT | kube.node.memory_allocatable[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node [{#NAME}]: Pods allocatable | The Pods resources of a node that are available for scheduling. |
DEPENDENT | kube.node.pods_allocatable[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node [{#NAME}]: Ephemeral storage allocatable | The allocatable ephemeral-storage of a node that is available for scheduling. |
DEPENDENT | kube.node.ephemeral_storage_allocatable[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node [{#NAME}]: CPU capacity | The capacity for CPU resources of a node. |
DEPENDENT | kube.node.cpu_capacity[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node [{#NAME}]: Memory capacity | The capacity for Memory resources of a node. |
DEPENDENT | kube.node.memory_capacity[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node [{#NAME}]: Ephemeral storage capacity | The ephemeral-storage capacity of a node. |
DEPENDENT | kube.node.ephemeral_storage_capacity[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Node [{#NAME}]: Pods capacity | The capacity for Pods resources of a node. |
DEPENDENT | kube.node.pods_capacity[{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Pending | Pod is in pending state. |
DEPENDENT | kube.pod.phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Succeeded | Pod is in succeeded state. |
DEPENDENT | kube.pod.phase.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Failed | Pod is in failed state. |
DEPENDENT | kube.pod.phase.failed[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Unknown | Pod is in unknown state. |
DEPENDENT | kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Running | Pod is in unknown state. |
DEPENDENT | kube.pod.phase.running[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers terminated | Describes whether the container is currently in terminated state. |
DEPENDENT | kube.pod.containers_terminated[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers waiting | Describes whether the container is currently in waiting state. |
DEPENDENT | kube.pod.containers_waiting[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers ready | Describes whether the containers readiness check succeeded. |
DEPENDENT | kube.pod.containers_ready[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
DEPENDENT | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers running | Describes whether the container is currently in running state. |
DEPENDENT | kube.pod.containers_running[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Ready | Describes whether the pod is ready to serve requests. |
DEPENDENT | kube.pod.ready[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Scheduled | Describes the status of the scheduling process for the pod. |
DEPENDENT | kube.pod.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Unschedulable | Describes the unschedulable status for the pod. |
DEPENDENT | kube.pod.unschedulable[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU limits | The limit on CPU cores to be used by a container. |
DEPENDENT | kube.pod.containers.limits.cpu[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory limits | The limit on memory to be used by a container. |
DEPENDENT | kube.pod.containers.limits.memory[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU requests | The number of requested cpu cores by a container. |
DEPENDENT | kube.pod.containers.requests.cpu[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory requests | The number of requested memory bytes by a container. |
DEPENDENT | kube.pod.containers.requests.memory[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Replicaset [{#NAME}]: Replicas | The number of replicas per ReplicaSet. |
DEPENDENT | kube.replicaset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Replicaset [{#NAME}]: Desired replicas | Number of desired pods for a ReplicaSet. |
DEPENDENT | kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Replicaset [{#NAME}]: Fully labeled replicas | The number of fully labeled replicas per ReplicaSet. |
DEPENDENT | kube.replicaset.fully_labeled_replicas[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Replicaset [{#NAME}]: Ready | The number of ready replicas per ReplicaSet. |
DEPENDENT | kube.replicaset.ready[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Statefulset [{#NAME}]: Replicas | The number of replicas per StatefulSet. |
DEPENDENT | kube.statefulset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Statefulset [{#NAME}]: Desired replicas | Number of desired pods for a StatefulSet. |
DEPENDENT | kube.statefulset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Statefulset [{#NAME}]: Current replicas | The number of current replicas per StatefulSet. |
DEPENDENT | kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Statefulset [{#NAME}]: Ready replicas | The number of ready replicas per StatefulSet. |
DEPENDENT | kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Statefulset [{#NAME}]: Updated replicas | The number of updated replicas per StatefulSet. |
DEPENDENT | kube.statefulset.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods healthy | Current number of healthy pods. |
DEPENDENT | kube.pdb.pods_healthy[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods desired | Minimum desired number of healthy pods. |
DEPENDENT | kube.pdb.pods_desired[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Disruptions allowed | Number of pod disruptions that are allowed. |
DEPENDENT | kube.pdb.disruptions_allowed[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods total | Total number of pods counted by this disruption budget. |
DEPENDENT | kube.pdb.pods_total[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Suspend | Suspend flag tells the controller to suspend subsequent executions. |
DEPENDENT | kube.cronjob.spec_suspend[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - DISCARD_UNCHANGED_HEARTBEAT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Active | Active holds pointers to currently running jobs. |
DEPENDENT | kube.cronjob.status_active[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Last schedule | LastScheduleTime keeps information of when was the last time the job was successfully scheduled. |
DEPENDENT | kube.cronjob.last_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - JAVASCRIPT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Next schedule | Next time the cronjob should be scheduled. The time after lastScheduleTime, or after the cron job's creation time if it's never been scheduled. Use this to determine if the job is delayed. |
DEPENDENT | kube.cronjob.next_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: - JAVASCRIPT: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Failed | The number of pods which reached Phase Failed and the reason for failure. |
DEPENDENT | kube.cronjob.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Succeeded | The number of pods which reached Phase Succeeded. |
DEPENDENT | kube.cronjob.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion succeeded | Number of job has completed its execution. |
DEPENDENT | kube.cronjob.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion failed | Number of job has failed its execution. |
DEPENDENT | kube.cronjob.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Failed | The number of pods which reached Phase Failed and the reason for failure. |
DEPENDENT | kube.job.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Succeeded | The number of pods which reached Phase Succeeded. |
DEPENDENT | kube.job.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion succeeded | Number of job has completed its execution. |
DEPENDENT | kube.job.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion failed | Number of job has failed its execution. |
DEPENDENT | kube.job.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing: - PROMETHEUS_PATTERN: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Component [{#NAME}]: Healthy | Cluster component healthy. |
DEPENDENT | kube.componentstatuses.healthy[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Readyz [{#NAME}]: Healthcheck | Result of readyz healthcheck for component. |
DEPENDENT | kube.readyz.healthcheck[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
| Kubernetes | Kubernetes: Livez [{#NAME}]: Healthcheck | Result of livez healthcheck for component. |
DEPENDENT | kube.livez.healthcheck[{#NAME}] Preprocessing: - JSONPATH: ⛔️ON_FAIL: |
Triggers
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | - |
min(/Kubernetes cluster state by HTTP/kube.pvc.status_phase.pending[{#NAMESPACE}/{#NAME}],2m)>0 |
WARNING | |
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Deployment replicas mismatch | - |
(last(/Kubernetes cluster state by HTTP/kube.deployment.replicas[{#NAMESPACE}/{#NAME}])-last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}]))<>0 |
WARNING | |
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is not healthy | - |
min(/Kubernetes cluster state by HTTP/kube.pod.phase.failed[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.pending[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}],10m)>0 |
HIGH | |
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | - |
(last(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],#3))>2 |
WARNING | |
| Kubernetes: Namespace [{#NAMESPACE}] RS [{#NAME}]: ReplicasSet mismatch | - |
(last(/Kubernetes cluster state by HTTP/kube.replicaset.replicas[{#NAMESPACE}/{#NAME}])-last(/Kubernetes cluster state by HTTP/kube.replicaset.ready[{#NAMESPACE}/{#NAME}]))<>0 |
WARNING | |
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatfulSet is down | - |
(last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]) / last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}]))<>1 |
HIGH | |
| Kubernetes: Namespace [{#NAMESPACE}] RS [{#NAME}]: Statefulset replicas mismatch | - |
(last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas[{#NAMESPACE}/{#NAME}])-last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]))<>0 |
WARNING | |
| Kubernetes: Component [{#NAME}] is unhealthy | - |
count(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}],#3,,"True")<2 and length(last(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}]))>0 |
WARNING | |
| Kubernetes: Readyz [{#NAME}] is unhealthy | - |
count(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}],#3,,"ok")<2 and length(last(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}]))>0 |
WARNING | |
| Kubernetes: Livez [{#NAME}] is unhealthy | - |
count(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}],#3,,"ok")<2 and length(last(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}]))>0 |
WARNING |
Feedback
Please report any issues with the template at https://support.zabbix.com.
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums.
Kubernetes cluster state by HTTP
Overview
The template to monitor Kubernetes state that work without any external scripts. It works without external scripts and uses the script item to make HTTP requests to the Kubernetes API.
Template Kubernetes cluster state by HTTP - collects metrics by HTTP agent from kube-state-metrics endpoint and Kubernetes API.
Don't forget change macros {$KUBE.API.URL} and {$KUBE.API.TOKEN}. Also, see the Macros section for a list of macros used to set trigger values.
NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
Requirements
Zabbix version: 6.0 and higher.
Tested versions
This template has been tested on:
- Kubernetes 1.19.10
Configuration
Zabbix should be configured according to the instructions in the Templates out of the box section.
Setup
Install the Zabbix Helm Chart in your Kubernetes cluster. Internal service metrics are collected from kube-state-metrics endpoint.
Template needs to use authorization via API token.
Set the {$KUBE.API.URL} such as <scheme>://<host>:<port>.
Get the generated service account token using the command:
kubectl get secret zabbix-service-account -n monitoring -o jsonpath={.data.token} | base64 -d
Then set it to the macro {$KUBE.API.TOKEN}.
Set {$KUBE.STATE.ENDPOINT.NAME} with Kube state metrics endpoint name. See kubectl -n monitoring get ep. Default: zabbix-kube-state-metrics.
NOTE. If you wish to monitor Controller Manager and Scheduler components, you might need to set the --binding-address option for them to the address where Zabbix proxy can reach them.
For example, for clusters created with kubeadm it can be set in the following manifest files (changes will be applied immediately):
- /etc/kubernetes/manifests/kube-controller-manager.yaml
- /etc/kubernetes/manifests/kube-scheduler.yaml
Depending on your Kubernetes distribution, you might need to adjust {$KUBE.CONTROL_PLANE.TAINT} macro (for example, set it to node-role.kubernetes.io/master for OpenShift).
NOTE. Some metrics may not be collected depending on your Kubernetes version and configuration.
Also, see the Macros section for a list of macros used to set trigger values.
Set up the macros to filter the metrics of discovered Kubelets by node names:
- {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES}
- {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES}
Set up macros to filter metrics by namespace:
- {$KUBE.LLD.FILTER.NAMESPACE.MATCHES}
- {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES}
Set up macros to filter node metrics by nodename:
- {$KUBE.LLD.FILTER.NODE.MATCHES}
- {$KUBE.LLD.FILTER.NODE.NOT_MATCHES}
Note: If you have a large cluster, it is highly recommended to set a filter for discoverable namespaces.
You can use the {$KUBE.KUBELET.FILTER.LABELS} and {$KUBE.KUBELET.FILTER.ANNOTATIONS} macros for advanced filtering of kubelets by node labels and annotations.
Notes about labels and annotations filters:
- Macro values should be specified separated by commas and must have the key/value form with support for regular expressions in the value (
key1: value, key2: regexp). - ECMAScript syntax is used for regular expressions.
- Filters are applied if such label key exists for the entity that is being filtered (it means that if you specify a key in the filter, entities that do not have this key will not be affected by the filter and will still be discovered, and only entities containing that key will be filtered by the value).
- You can also use the exclamation point symbol (
!) to invert the filter (!key: value).
For example: kubernetes.io/hostname: kubernetes-node[5-25], !node-role.kubernetes.io/ingress: .*. As a result, the kubelets on nodes 5-25 without the "ingress" role will be discovered.
See the Kubernetes documentation for details about labels and annotations:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
- https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
You can also set up evaluation periods for replica mismatch triggers (Deployments, ReplicaSets, StatefulSets) with the macro {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD}, which supports context and regular expressions. For example, you can create the following macros:
- Set the evaluation period for the Deployment "nginx-deployment" in the namespace "default" to the 3 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:default:nginx-deployment"} = #3
- Set the evaluation period for all Deployments to the 10 last values:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"deployment:.*:.*"} = #10 or {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:"^deployment.*"} = #10
- Set the evaluation period for Deployments, ReplicaSets and StatefulSets in the namespace "default" to 15 minutes:
{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:regex:".*:default:.*"} = 15m
Note that different context macros with regular expressions matching the same string can be applied in an undefined order, and simple context macros (without regular expressions) have higher priority. Read the Important notes section in Zabbix documentation for details.
Macros used
| Name | Description | Default |
|---|---|---|
| {$KUBE.API.URL} | Kubernetes API endpoint URL in the format |
https://kubernetes.default.svc.cluster.local:443 |
| {$KUBE.API.READYZ.ENDPOINT} | Kubernetes API readyz endpoint /readyz |
/readyz |
| {$KUBE.API.LIVEZ.ENDPOINT} | Kubernetes API livez endpoint /livez |
/livez |
| {$KUBE.API.COMPONENTSTATUSES.ENDPOINT} | Kubernetes API componentstatuses endpoint /api/v1/componentstatuses |
/api/v1/componentstatuses |
| {$KUBE.API.TOKEN} | Service account bearer token. |
|
| {$KUBE.HTTP.PROXY} | Sets the HTTP proxy to |
|
| {$KUBE.STATE.ENDPOINT.NAME} | Kubernetes state endpoint name. |
zabbix-kube-state-metrics |
| {$OPENSHIFT.STATE.ENDPOINT.NAME} | OpenShift state endpoint name. |
openshift-state-metrics |
| {$KUBE.API_SERVER.SCHEME} | Kubernetes API servers metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.API_SERVER.PORT} | Kubernetes API servers metrics endpoint port. Used in ControlPlane LLD. |
6443 |
| {$KUBE.CONTROL_PLANE.TAINT} | Taint that applies to control plane nodes. Change if needed. Used in ControlPlane LLD. |
node-role.kubernetes.io/control-plane |
| {$KUBE.CONTROLLER_MANAGER.SCHEME} | Kubernetes Controller manager metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.CONTROLLER_MANAGER.PORT} | Kubernetes Controller manager metrics endpoint port. Used in ControlPlane LLD. |
10257 |
| {$KUBE.SCHEDULER.SCHEME} | Kubernetes Scheduler metrics endpoint scheme. Used in ControlPlane LLD. |
https |
| {$KUBE.SCHEDULER.PORT} | Kubernetes Scheduler metrics endpoint port. Used in ControlPlane LLD. |
10259 |
| {$KUBE.KUBELET.SCHEME} | Kubernetes Kubelet metrics endpoint scheme. Used in Kubelet LLD. |
https |
| {$KUBE.KUBELET.PORT} | Kubernetes Kubelet metrics endpoint port. Used in Kubelet LLD. |
10250 |
| {$KUBE.LLD.FILTER.NAMESPACE.MATCHES} | Filter of discoverable metrics by namespace. |
.* |
| {$KUBE.LLD.FILTER.NAMESPACE.NOT_MATCHES} | Filter to exclude discovered metrics by namespace. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.NODE.MATCHES} | Filter of discoverable nodes by nodename. |
.* |
| {$KUBE.LLD.FILTER.NODE.NOT_MATCHES} | Filter to exclude discovered nodes by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.LLD.FILTER.KUBELET_NODE.MATCHES} | Filter of discoverable Kubelets by nodename. |
.* |
| {$KUBE.LLD.FILTER.KUBELET_NODE.NOT_MATCHES} | Filter to exclude discovered Kubelets by nodename. |
CHANGE_IF_NEEDED |
| {$KUBE.KUBELET.FILTER.ANNOTATIONS} | Node annotations to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.KUBELET.FILTER.LABELS} | Node labels to filter Kubelets (regex in values are supported). See the template's README.md for details. |
|
| {$KUBE.LLD.FILTER.PV.MATCHES} | Filter of discoverable persistent volumes by name. |
.* |
| {$KUBE.LLD.FILTER.PV.NOT_MATCHES} | Filter to exclude discovered persistent volumes by name. |
CHANGE_IF_NEEDED |
| {$KUBE.REPLICA.MISMATCH.EVAL_PERIOD} | The evaluation period range which is used for calculation of expressions in trigger prototypes (time period or value range). Can be used with context. |
#5 |
Items
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Get state metrics | Collecting Kubernetes metrics from kube-state-metrics. |
Script | kube.state.metrics |
| Kubernetes: Control plane LLD | Generation of data for Control plane discovery rules. |
Script | kube.control_plane.lld Preprocessing
|
| Kubernetes: Node LLD | Generation of data for Kubelet discovery rules. |
Script | kube.node.lld Preprocessing
|
| Kubernetes: Get component statuses | HTTP agent | kube.componentstatuses Preprocessing
|
|
| Kubernetes: Get readyz | HTTP agent | kube.readyz Preprocessing
|
|
| Kubernetes: Get livez | HTTP agent | kube.livez Preprocessing
|
|
| Kubernetes: Namespace count | The number of namespaces. |
Dependent item | kube.namespace.count Preprocessing
|
| Kubernetes: CronJob count | Number of cronjobs. |
Dependent item | kube.cronjob.count Preprocessing
|
| Kubernetes: Job count | Number of jobs (generated by cronjob + job). |
Dependent item | kube.job.count Preprocessing
|
| Kubernetes: Endpoint count | Number of endpoints. |
Dependent item | kube.endpoint.count Preprocessing
|
| Kubernetes: Deployment count | The number of deployments. |
Dependent item | kube.deployment.count Preprocessing
|
| Kubernetes: Service count | The number of services. |
Dependent item | kube.service.count Preprocessing
|
| Kubernetes: StatefulSet count | The number of statefulsets. |
Dependent item | kube.statefulset.count Preprocessing
|
| Kubernetes: Node count | The number of nodes. |
Dependent item | kube.node.count Preprocessing
|
LLD rule API servers discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| API servers discovery | Dependent item | kube.api_servers.discovery |
LLD rule Controller manager nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Controller manager nodes discovery | Dependent item | kube.controller_manager.discovery |
LLD rule Scheduler servers nodes discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Scheduler servers nodes discovery | Dependent item | kube.scheduler.discovery |
LLD rule Kubelet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubelet discovery | Dependent item | kube.kubelet.discovery |
LLD rule Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Daemonset discovery | Dependent item | kube.daemonset.discovery Preprocessing
|
Item prototypes for Daemonset discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Ready | The number of nodes that should be running the daemon pod and have one or more running and ready. |
Dependent item | kube.daemonset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Scheduled | The number of nodes that run at least one daemon pod and are supposed to. |
Dependent item | kube.daemonset.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Desired | The number of nodes that should be running the daemon pod. |
Dependent item | kube.daemonset.desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Misscheduled | The number of nodes that run a daemon pod but are not supposed to. |
Dependent item | kube.daemonset.misscheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Daemonset [{#NAME}]: Updated number scheduled | The total number of nodes that are running updated daemon pod. |
Dependent item | kube.daemonset.updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PVC discovery | Dependent item | kube.pvc.discovery Preprocessing
|
Item prototypes for PVC discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Status phase | The current status phase of the persistent volume claim. |
Dependent item | kube.pvc.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC [{#NAME}] Requested storage | The capacity of storage requested by the persistent volume claim. |
Dependent item | kube.pvc.requested.storage[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC status phase: Bound, sum | The total amount of persistent volume claims in the Bound phase. |
Dependent item | kube.pvc.status_phase.bound.sum[{#NAMESPACE}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC status phase: Lost, sum | The total amount of persistent volume claims in the Lost phase. |
Dependent item | kube.pvc.status_phase.lost.sum[{#NAMESPACE}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PVC status phase: Pending, sum | The total amount of persistent volume claims in the Pending phase. |
Dependent item | kube.pvc.status_phase.pending.sum[{#NAMESPACE}] Preprocessing
|
Trigger prototypes for PVC discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: NS [{#NAMESPACE}] PVC [{#NAME}]: PVC is pending | count(/Kubernetes cluster state by HTTP/kube.pvc.status_phase[{#NAMESPACE}/{#NAME}],2m,,5)>=2 |
Warning |
LLD rule PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PV discovery | Dependent item | kube.pv.discovery Preprocessing
|
Item prototypes for PV discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: PV [{#NAME}] Status phase | The current status phase of the persistent volume. |
Dependent item | kube.pv.status_phase[{#NAME}] Preprocessing
|
| Kubernetes: PV [{#NAME}] Capacity bytes | A capacity of the persistent volume in bytes. |
Dependent item | kube.pv.capacity.bytes[{#NAME}] Preprocessing
|
| Kubernetes: PV status phase: Pending, sum | The total amount of persistent volumes in the Pending phase. |
Dependent item | kube.pv.status_phase.pending.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Available, sum | The total amount of persistent volumes in the Available phase. |
Dependent item | kube.pv.status_phase.available.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Bound, sum | The total amount of persistent volumes in the Bound phase. |
Dependent item | kube.pv.status_phase.bound.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Released, sum | The total amount of persistent volumes in the Released phase. |
Dependent item | kube.pv.status_phase.released.sum[{#SINGLETON}] Preprocessing
|
| Kubernetes: PV status phase: Failed, sum | The total amount of persistent volumes in the Failed phase. |
Dependent item | kube.pv.status_phase.failed.sum[{#SINGLETON}] Preprocessing
|
Trigger prototypes for PV discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: PV [{#NAME}]: PV has failed | count(/Kubernetes cluster state by HTTP/kube.pv.status_phase[{#NAME}],2m,,3)>=2 |
Warning |
LLD rule Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Deployment discovery | Dependent item | kube.deployment.discovery Preprocessing
|
Item prototypes for Deployment discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Paused | Whether the deployment is paused and will not be processed by the deployment controller. |
Dependent item | kube.deployment.spec_paused[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas desired | Number of desired pods for a deployment. |
Dependent item | kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Rollingupdate max unavailable | Maximum number of unavailable replicas during a rolling update of a deployment. |
Dependent item | kube.deployment.rollingupdate.max_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas | The number of replicas per deployment. |
Dependent item | kube.deployment.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas available | The number of available replicas per deployment. |
Dependent item | kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas unavailable | The number of unavailable replicas per deployment. |
Dependent item | kube.deployment.replicas_unavailable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas updated | The number of updated replicas per deployment. |
Dependent item | kube.deployment.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Replicas mismatched | The number of available replicas not matching the desired number of replicas. |
Dependent item | kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Deployment discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Deployment [{#NAME}]: Deployment replicas mismatch | Deployment has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.deployment.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"deployment:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.deployment.replicas_available[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Endpoint discovery | Dependent item | kube.endpoint.discovery Preprocessing
|
Item prototypes for Endpoint discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address available | Number of addresses available in endpoint. |
Dependent item | kube.endpoint.address_available[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Address not ready | Number of addresses not ready in endpoint. |
Dependent item | kube.endpoint.address_not_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Endpoint [{#NAME}]: Age | Endpoint age (number of seconds since creation). |
Dependent item | kube.endpoint.age[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Node discovery | Dependent item | kube.node.discovery Preprocessing
|
Item prototypes for Node discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Node [{#NAME}]: CPU allocatable | The CPU resources of a node that are available for scheduling. |
Dependent item | kube.node.cpu_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Memory allocatable | The memory resources of a node that are available for scheduling. |
Dependent item | kube.node.memory_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Pods allocatable | The pods resources of a node that are available for scheduling. |
Dependent item | kube.node.pods_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Ephemeral storage allocatable | The allocatable ephemeral storage of a node that is available for scheduling. |
Dependent item | kube.node.ephemeral_storage_allocatable[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: CPU capacity | The capacity for CPU resources of a node. |
Dependent item | kube.node.cpu_capacity[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Memory capacity | The capacity for memory resources of a node. |
Dependent item | kube.node.memory_capacity[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Ephemeral storage capacity | The ephemeral storage capacity of a node. |
Dependent item | kube.node.ephemeral_storage_capacity[{#NAME}] Preprocessing
|
| Kubernetes: Node [{#NAME}]: Pods capacity | The capacity for pods resources of a node. |
Dependent item | kube.node.pods_capacity[{#NAME}] Preprocessing
|
LLD rule Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Pod discovery | Dependent item | kube.pod.discovery Preprocessing
|
Item prototypes for Pod discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Pending | Pod is in pending state. |
Dependent item | kube.pod.phase.pending[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Succeeded | Pod is in succeeded state. |
Dependent item | kube.pod.phase.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Failed | Pod is in failed state. |
Dependent item | kube.pod.phase.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Unknown | Pod is in unknown state. |
Dependent item | kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}] Phase: Running | Pod is in unknown state. |
Dependent item | kube.pod.phase.running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers terminated | Describes whether the container is currently in terminated state. |
Dependent item | kube.pod.containers_terminated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers waiting | Describes whether the container is currently in waiting state. |
Dependent item | kube.pod.containers_waiting[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers ready | Describes whether the containers readiness check succeeded. |
Dependent item | kube.pod.containers_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers restarts | The number of container restarts. |
Dependent item | kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers running | Describes whether the container is currently in running state. |
Dependent item | kube.pod.containers_running[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Ready | Describes whether the pod is ready to serve requests. |
Dependent item | kube.pod.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Scheduled | Describes the status of the scheduling process for the pod. |
Dependent item | kube.pod.scheduled[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Unschedulable | Describes the unschedulable status for the pod. |
Dependent item | kube.pod.unschedulable[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU limits | The limit on CPU cores to be used by a container. |
Dependent item | kube.pod.containers.limits.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory limits | The limit on memory to be used by a container. |
Dependent item | kube.pod.containers.limits.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers CPU requests | The number of requested CPU cores by a container. |
Dependent item | kube.pod.containers.requests.cpu[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Containers memory requests | The number of requested memory bytes by a container. |
Dependent item | kube.pod.containers.requests.memory[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for Pod discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is not healthy | min(/Kubernetes cluster state by HTTP/kube.pod.phase.failed[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.pending[{#NAMESPACE}/{#NAME}],10m)>0 or min(/Kubernetes cluster state by HTTP/kube.pod.phase.unknown[{#NAMESPACE}/{#NAME}],10m)>0 |
High | ||
| Kubernetes: Namespace [{#NAMESPACE}] Pod [{#NAME}]: Pod is crash looping | Containers of the pod keep restarting. This most likely indicates that the pod is in the CrashLoopBackOff state. |
(last(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}])-min(/Kubernetes cluster state by HTTP/kube.pod.containers_restarts[{#NAMESPACE}/{#NAME}],15m))>1 |
Warning |
LLD rule ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| ReplicaSet discovery | Dependent item | kube.replicaset.discovery Preprocessing
|
Item prototypes for ReplicaSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas | The number of replicas per ReplicaSet. |
Dependent item | kube.replicaset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Desired replicas | Number of desired pods for a ReplicaSet. |
Dependent item | kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Fully labeled replicas | The number of fully labeled replicas per ReplicaSet. |
Dependent item | kube.replicaset.fully_labeled_replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Ready | The number of ready replicas per ReplicaSet. |
Dependent item | kube.replicaset.ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] ReplicaSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the desired number of replicas. |
Dependent item | kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for ReplicaSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] RS [{#NAME}]: ReplicaSet mismatch | ReplicaSet has not matched the expected number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"replicaset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.replicas_desired[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.replicaset.ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| StatefulSet discovery | Dependent item | kube.statefulset.discovery Preprocessing
|
Item prototypes for StatefulSet discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas | The number of replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Desired replicas | Number of desired pods for a StatefulSet. |
Dependent item | kube.statefulset.replicas_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Current replicas | The number of current replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Ready replicas | The number of ready replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Updated replicas | The number of updated replicas per StatefulSet. |
Dependent item | kube.statefulset.replicas_updated[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: Replicas mismatched | The number of ready replicas not matching the number of replicas. |
Dependent item | kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for StatefulSet discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet is down | (last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}]) / last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_current[{#NAMESPACE}/{#NAME}]))<>1 |
High | ||
| Kubernetes: Namespace [{#NAMESPACE}] StatefulSet [{#NAME}]: StatefulSet replicas mismatch | StatefulSet has not matched the number of replicas during the specified trigger evaluation period. |
min(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_mismatched[{#NAMESPACE}/{#NAME}],{$KUBE.REPLICA.MISMATCH.EVAL_PERIOD:"statefulset:{#NAMESPACE}:{#NAME}"})>0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas[{#NAMESPACE}/{#NAME}])>=0 and last(/Kubernetes cluster state by HTTP/kube.statefulset.replicas_ready[{#NAMESPACE}/{#NAME}])>=0 |
Warning |
LLD rule PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| PodDisruptionBudget discovery | Dependent item | kube.pdb.discovery Preprocessing
|
Item prototypes for PodDisruptionBudget discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods healthy | Current number of healthy pods. |
Dependent item | kube.pdb.pods_healthy[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods desired | Minimum desired number of healthy pods. |
Dependent item | kube.pdb.pods_desired[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Disruptions allowed | Number of pod disruptions that are allowed. |
Dependent item | kube.pdb.disruptions_allowed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] PodDisruptionBudget [{#NAME}]: Pods total | Total number of pods counted by this disruption budget. |
Dependent item | kube.pdb.pods_total[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| CronJob discovery | Dependent item | kube.cronjob.discovery Preprocessing
|
Item prototypes for CronJob discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Suspend | Suspend flag tells the controller to suspend subsequent executions. |
Dependent item | kube.cronjob.spec_suspend[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Active | Active holds pointers to currently running jobs. |
Dependent item | kube.cronjob.status_active[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Last schedule | LastScheduleTime keeps information of when was the last time the job was successfully scheduled. |
Dependent item | kube.cronjob.last_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Next schedule | Next time the cronjob should be scheduled. The time after lastScheduleTime or after the cron job's creation time if it's never been scheduled. Use this to determine if the job is delayed. |
Dependent item | kube.cronjob.next_schedule_time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.cronjob.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.cronjob.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.cronjob.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] CronJob [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.cronjob.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Job discovery | Dependent item | kube.job.discovery Preprocessing
|
Item prototypes for Job discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Failed | The number of pods which reached the Failed phase and the reason for failure. |
Dependent item | kube.job.status_failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Succeeded | The number of pods which reached the Succeeded phase. |
Dependent item | kube.job.status_succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion succeeded | Number of jobs the execution of which has been completed. |
Dependent item | kube.job.completion.succeeded[{#NAMESPACE}/{#NAME}] Preprocessing
|
| Kubernetes: Namespace [{#NAMESPACE}] Job [{#NAME}]: Completion failed | Number of jobs the execution of which has failed. |
Dependent item | kube.job.completion.failed[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Component statuses discovery | Dependent item | kube.componentstatuses.discovery Preprocessing
|
Item prototypes for Component statuses discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Component [{#NAME}]: Healthy | Cluster component healthy. |
Dependent item | kube.componentstatuses.healthy[{#NAME}] Preprocessing
|
Trigger prototypes for Component statuses discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Component [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}],#2,"ne","True")=2 and length(last(/Kubernetes cluster state by HTTP/kube.componentstatuses.healthy[{#NAME}]))>0 |
Warning |
LLD rule Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Readyz discovery | Dependent item | kube.readyz.discovery Preprocessing
|
Item prototypes for Readyz discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Readyz [{#NAME}]: Healthcheck | Result of readyz healthcheck for component. |
Dependent item | kube.readyz.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Readyz discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Readyz [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.readyz.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Livez discovery | Dependent item | kube.livez.discovery Preprocessing
|
Item prototypes for Livez discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| Kubernetes: Livez [{#NAME}]: Healthcheck | Result of livez healthcheck for component. |
Dependent item | kube.livez.healthcheck[{#NAME}] Preprocessing
|
Trigger prototypes for Livez discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| Kubernetes: Livez [{#NAME}] is unhealthy | count(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}],#2,"ne","ok")=2 and length(last(/Kubernetes cluster state by HTTP/kube.livez.healthcheck[{#NAME}]))>0 |
Warning |
LLD rule OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift BuildConfig discovery | Dependent item | openshift.buildconfig.discovery Preprocessing
|
Item prototypes for OpenShift BuildConfig discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Created | OpenShift BuildConfig Unix creation timestamp. |
Dependent item | openshift.buildconfig.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.buildconfig.generation[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] BuildConfig [{#NAME}]: Latest version | The latest version of BuildConfig. |
Dependent item | openshift.buildconfig.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
LLD rule OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Build discovery | Dependent item | openshift.build.discovery Preprocessing
|
Item prototypes for OpenShift Build discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Namespace [{#NAMESPACE}] Build [{#NAME}]: Created | OpenShift Build Unix creation timestamp. |
Dependent item | openshift.build.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] Build [{#NAME}]: Generation | Sequence number representing a specific generation of the desired state. |
Dependent item | openshift.build.sequence.number[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] Build [{#NAME}]: Status phase | The Build phase. |
Dependent item | openshift.build.status_phase[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Build discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| OpenShift: Build [{#NAME}]: Build has failed | count(/Kubernetes cluster state by HTTP/openshift.build.status_phase[{#NAMESPACE}/{#NAME}],2m,"ge",6)>=2 |
Warning |
LLD rule OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift ClusterResourceQuota discovery | Dependent item | openshift.cluster.resource.quota.discovery Preprocessing
|
Item prototypes for OpenShift ClusterResourceQuota discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Quota [{#NAME}] Resource [{#RESOURCE}]: Type [{#TYPE}]] | Usage about resource quota. |
Dependent item | openshift.cluster.resource.quota[{#RESOURCE}/{#NAME}/{#TYPE}] Preprocessing
|
LLD rule OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift Route discovery | Dependent item | openshift.route.discovery Preprocessing
|
Item prototypes for OpenShift Route discovery
| Name | Description | Type | Key and additional info |
|---|---|---|---|
| OpenShift: Namespace [{#NAMESPACE}] Route [{#NAME}]: Created | OpenShift Route Unix creation timestamp. |
Dependent item | openshift.route.created.time[{#NAMESPACE}/{#NAME}] Preprocessing
|
| OpenShift: Namespace [{#NAMESPACE}] Route [{#NAME}]: Status | Information about route status. |
Dependent item | openshift.route.status[{#NAMESPACE}/{#NAME}] Preprocessing
|
Trigger prototypes for OpenShift Route discovery
| Name | Description | Expression | Severity | Dependencies and additional info |
|---|---|---|---|---|
| OpenShift: Route [{#NAME}] with issue: Status is false | count(/Kubernetes cluster state by HTTP/openshift.route.status[{#NAMESPACE}/{#NAME}],2m,,0)>=2 |
Warning |
Feedback
Please report any issues with the template at https://support.zabbix.com
You can also provide feedback, discuss the template, or ask for help at ZABBIX forums
