Kubernetes: Failed to get nodes | Last value: Error: alloc failed.

[sarang]

Hello Team,

I have installed Zabbix Kubernetes helm chart(v6.0) on AKS cluster. I'm able to get latest data for Kubernetes cluster state whereas no data for Kubernetes Nodes. For Kubernetes nodes, getting last value as Error: alloc failed & problem opened as Kubernetes: Failed to get nodes. Please help me resolve the issue.

Note: I have disabled proxy from helm chart & zabbix-agent is reporting to proxy of my own Zabbix tenant outside of kube cluster.

Zabbix-proxy logs:
82883:20230515:104437.224 cannot send list of active checks to "****": host [aks-dev-vm0000] not found
82887:20230515:104437.350 cannot send list of active checks to "****": host [aks-dev**-*****-vm**0000*y] not found
82887:20230515:104437.408 cannot send list of active checks to "****": host [aks-dev**-*****-vms**0000**] not found

Zabbix Agent logs:
2023/05/15 15:44:14.837646 failed to accept an incoming connection: connection from "****" rejected, allowed hosts: "******"
2023/05/15 15:44:24.837072 failed to accept an incoming connection: connection from "*****" rejected, allowed hosts: "*****"
2023/05/15 15:44:34.837052 failed to accept an incoming connection: connection from "*****" rejected, allowed hosts: "******"

Thanks and Regards,
Sarang

[cyber]

Agent says, connections come from the place which is not allowed in its config... I guess if agent config has allow list (Server/ServerActive), then you should add your proxy there..
Proxy says, it cannot send any config to host that is asking for some, as it does not know anything about this... So, I think, in hosts config in zabbix, it is not marked as "monitored by this proxy". Or host does not exist in that zabbix..

[jennyfoster]

Hello,

I am just new here but not new here. I just want to say that Hello Sarang,

Based on the information you provided, it appears that there are issues with the Zabbix agent and Zabbix proxy communicating with the Kubernetes nodes in your AKS cluster. Let's troubleshoot these issues step by step:

1. Zabbix Proxy Logs: The log messages indicate that the Zabbix proxy is unable to find the hosts (Kubernetes nodes) in question. Make sure that the hosts are properly configured and added to Zabbix. You can verify this by checking the following:
   • Ensure that the host names in Zabbix match the actual host names of the Kubernetes nodes.
   • Confirm that the hosts are in the correct host group and are associated with the Zabbix proxy you're using.
   • Check the Zabbix proxy configuration and make sure it is properly set up to communicate with the Zabbix server.
2. Zabbix Agent Logs: The Zabbix agent logs show connection rejections from certain IP addresses. This could be due to incorrect configuration or firewall rules. Here's what you can do:
   • Verify that the Zabbix agent configuration (zabbix_agentd.conf) on the Kubernetes nodes is correctly set up to report to the Zabbix proxy's IP or hostname.
   • Check the firewall rules (if any) on the Kubernetes nodes and ensure that they allow incoming connections from the Zabbix proxy IP or hostname.
3. Network Connectivity: Ensure that there is proper network connectivity between the Zabbix proxy, Zabbix agent, and the Kubernetes nodes. Check for any network-related issues such as firewall restrictions, network routing problems, or DNS resolution issues that may be preventing communication.
4. Zabbix Helm Chart Configuration: Double-check your Zabbix Helm chart configuration. Ensure that you have correctly specified the Zabbix proxy and Zabbix server information in the Helm values file, and that the necessary configuration options are properly set for monitoring Kubernetes nodes.
5. Zabbix Version Compatibility: Confirm that the Zabbix version you are using is compatible with the Zabbix Kubernetes integration. Check the Zabbix documentation or the Helm chart repository for any specific requirements or known issues related to your Zabbix version. Home Bargains Employee Portal

[sarang]

cyber and jennyfoster Following your suggestions, I did some changes in values.yml. The changes are below. However I'm not able to get rid of Error: alloc failed. Please help me with next steps. Thanks in advance!

Kubernetes Helm repo: https://git.zabbix.com/projects/ZT/r...se/values.yaml

zabbixAgent:
enabled: true
env:
- name: ZBX_SERVER_HOST
value: <ProxyIP>
- name: ZBX_PASSIVESERVERS
value: 0.0.0.0/0
- name: ZBX_PASSIVE_ALLOW
value: true
- name: ZBX_ACTIVE_ALLOW
value: true

Agent Logs(****** is kube Node):
2023/05/17 18:25:27.569644 failed to process an incoming connection from ******: EOF
2023/05/17 18:25:37.569602 failed to process an incoming connection from ******: EOF
2023/05/17 18:25:47.569614 failed to process an incoming connection from *******: EOF

Proxy Logs:
440652:20230517:142803.619 [ Kubernetes ] ERROR: Error: alloc failed.
440638:20230517:142803.621 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.
440638:20230517:142803.621 [ Kubernetes discovery ] ERROR: Incorrect JSON. Check debug log for more information.

I am refereeing this document to configure hosts in UI: https://blog.zabbix.com/monitoring-k...-zabbix/25055/

Thanks!