How to monitor external k8s cluster?

[elgoch0]

Hi,


I'm trying to monitor k8s cluster A from my Zabbix instance (installed in k8s cluster Z), I followed the steps indicated at https://www.zabbix.com/integrations/kubernetes and setting values for:
- {$KUBE.API.URL} = https://$IP_OR_DNSNAME_OF_A_CONTROL_PLANE:6443/api

- {$KUBE.STATE.ENDPOINT.NAME} = zabbix-agent-kube-state-metrics

- {$KUBE.API.TOKEN} --> obtained doing:

kubectl -n zabbix get secret -o jsonpath={.data.token} zabbix-service-account | base64 -d

for the hosts created to set the templates Kubernetes cluster state by HTTP and Kubernetes nodes by HTTP

but I get this error:

250:20240124:172134.576 [ Kubernetes ] ERROR: Request failed with status code 403: {"kind
":"Status","apiVersion":"v1","metadata":{},"status ":"Failure","message":"v1 \"nodes\" is for
bidden: User \"system:serviceaccount:zabbix:zabbix-service-account\" cannot get resource \"v
1\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"name":"nodes","
kind":"v1"},"code":403}

I know TOKEN is right because:

$ curl -X GET https://$IP_OR_DNSNAME_OF_A_CONTROL_PLANE:6443/api --header "Authorization: Bearer $TOKEN" --insecure
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "$IP:6443"
}
]
}
​

but

curl -X GET https://$IP_OR_DNSNAME_OF_A_CONTROL_PLANE:6443/endpoints --header "Authorization: Bearer $TOKEN" --insecure
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User \"system:serviceaccount:zabbix:zabbix-service-account
\" cannot get path \"/endpoints\"",
"reason": "Forbidden",
"details": {},
"code": 403

​

My guess is some missing grants in service account system:serviceaccount:zabbix:zabbix-service-account, but even if I tried to grant admin permissions I got the same result


Any idea what am I missing here?

Thanks!

[elgoch0]

I made some progress, Cluster node monitoring using the template "Kubernetes nodes by HTTP" is working now, the fix was basically:
1. Zabbix Agent2 v6.4.6 (I tried v6.4.10 but it did some weird things with the monitoring of the databases)
2. Set macros
- {$KUBE.API.TOKEN}
- {$KUBE.API.URL} --> should be similar to:

https://$CP_NODE:6443

- {$KUBE.NODES.ENDPOINT.NAME} --> get the value using:

kubectl -n $ZABBIX_AGENT_NAMESPACE get ep

(use the one that looks like "zabbix-agent-zabbix-helm-chrt-agent")


but Main cluster components monitoring using the template "Kubernetes cluster state by HTTP" is still not working, very likely because endpoint:

zabbix-agent-kube-state-metrics 10.42.6.179:8080

is not reachable from outside the cluster (a proxy might be my way to go in this case)