Issues monitoring Kafka using JMX (Zabbix Java Gateway)

[onallion]

Hey guys,

I've been given a task to monitor Kafka via Zabbix:

• I installed Zabbix Java Gateway as per the instructions, on the same server as the Zabbix Server.
• Enabled JMX via port 12345 on one of my Kafka hosts by adding the following lines to its config:
  

  # JMX settings
  if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "
  fi

• Restarted Kafka
• Allowed port 12345 from Zabbix Server to a Kafka host in the FW
• Checked Telnet from Zabbix Server to Kafka host - connection established
• Added JMX interface to the host in Zabbix frontend
• Linked template to host (template from here: https://github.com/helli0n/kafka-monitoring)

Unfortunately, in the Zabbix Server log I am seeing errors like the following:

JMX agent item "jmx["kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent","Value"]" on host "kafkahost" failed: another network error, wait for 15 seconds

And then:

temporarily disabling JMX agent checks on host "Sandbox-Kafka-hrc1": host unavailable

I then check the frontend and the host has "JMX" in red with the error "ZBX_TCP_READ() timed out".

I'm kind of at a loss as to why this is not working, and don't know how to continue checking further. Any help would be appreciated!

[onallion]

In case anyone runs into this, I've solved it.

The line in the configuration file should be:

if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.rmi.port=8989"
fi

(Note for the port you can choose anything that is available, and you will need to allow in your FW).

Then edit kafka-server-start.sh and add the following:

export JMX_PORT=${JMX_PORT:-12345}

(For this you can also use any available port, but 12345 is the default one Zabbix uses for JMX interface so I just used it).

The problem was that unless you define "Dcom.sun.management.jmxremote.rmi.port=" it will use a random one.

[kaloyan.s]

Thank you onallion.
It is very useful.

Cheers,