zabbix javagate container(4.2.5) Connection timed out

[heeroz]

I want to use the lastest zabbix container wiht host network to monitor kafka.
I create zabbix containers by following commands.
#mysql container
docker run --name mysql-server -t \
--net host \
--restart=always \
-v /mysql/data:/var/lib/mysql \
-v /mysql/backup:/mysql/backup \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
-d mysql:5.7 \
--character-set-server=utf8 --collation-server=utf8_bin

#java-gateway container
docker run --name zabbix-java-gateway -t \
--net host \
--restart=always \
-d zabbix/zabbix-java-gateway:latest

#zabbix server container
docker run --name zabbix-server-mysql -t \
--net host \
--restart=always \
--add-host=mysql-server:127.0.0.1 \
--add-host=mysql:127.0.0.1 \
--add-host=zabbix-java-gateway:127.0.0.1 \
--add-host=zabbix-server-mysql:127.0.0.1 \
--add-host=zabbix-server:127.0.0.1 \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
-e ZBX_JAVAGATEWAY_ENABLE="true" \
-e ZBX_JAVAGATEWAY="zabbix-java-gateway" \
-d zabbix/zabbix-server-mysql:latest

#nginx container
docker run --name zabbix-web-nginx-mysql -t \
--net host \
--restart=always \
--add-host=mysql-server:127.0.0.1 \
--add-host=mysql:127.0.0.1 \
--add-host=zabbix-java-gateway:127.0.0.1 \
--add-host=zabbix-server-mysql:127.0.0.1 \
--add-host=zabbix-server:127.0.0.1 \
-e DB_SERVER_HOST="mysql-server" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix_pwd" \
-e MYSQL_ROOT_PASSWORD="root_pwd" \
-d zabbix/zabbix-web-nginx-mysql:latest

Zabbix Server has enabled Gateway and Listen to 0.0.0.0
Kafka can run without problem and port 12345 is used for jmx.
I can access kafka jmx by using jconsole or javagateway container(4.2.3).

According to the readme of this template.

https://github.com/helli0n/kafka-monitoring

I add jmx_discovery and JMXDiscovery-0.0.1.jar to /usr/sbin/zabbix_java/ext_lib/
javagateway use these jar in the process.

# docker exec -it $(docker ps | grep gateway | awk '{print $1}') bash
bash-5.0# ps -ef
PID USER TIME COMMAND
1 zabbix 0:01 /usr/bin/java -server -Dlogback.configurationFile=/etc/zabbix/zabbix_java_gateway_logback.xml -classpath /usr/sbin/zabbix_java/lib:/usr/sbin/zabbix_java/lib/android-json-4.3_r3.1.jar:/usr/sbin/zabbix_java/lib/logback-classic-0.9.27.jar:/usr/sbin/zabbix_java/lib/logback-core-0.9.27.jar:/usr/sbin/zabbix_java/lib/slf4j-api-1.6.1.jar:/usr/sbin/zabbix_java/lib/JMXDiscovery-0.0.1.jar:/usr/sbin/zabbix_java/bin/zabbix-java-gateway-4.2.5.jar:/usr/sbin/zabbix_java/ext_lib/JMXDiscovery-0.0.1.jar com.zabbix.gateway.JavaGateway
84 root 0:00 bash
89 root 0:00 ps -ef


I got folloing gateway Errors.
# docker logs -f $(docker ps | grep gateway | awk '{print $1}')
2019-08-26 15:20:14.411 [pool-1-thread-4] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"Connection timed out: service:jmx:rmi:\/\/\/jndi\/rmi:\/\/192.168.10.30:12345\/jmxrmi"}
2019-08-26 15:20:14.411 [pool-1-thread-4] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection
2019-08-26 15:20:14.426 [pool-1-thread-5] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection
2019-08-26 15:20:14.426 [pool-1-thread-5] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header
2019-08-26 15:20:14.426 [pool-1-thread-5] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 8 bytes of data length
2019-08-26 15:20:14.426 [pool-1-thread-5] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 249 bytes of request data
2019-08-26 15:20:14.426 [pool-1-thread-5] DEBUG c.z.gateway.BinaryProtocolSpeaker - received the following data in request: {"request":"java gateway jmx","username":"{$JMX_USER}","password":"{$JMX_PA SS}","jmx_endpoint":"service:jmx:rmi:///jndi/rmi://192.168.10.31:12345/jmxrmi","keys":["jmx["java.lang:type=GarbageCollector,name=G1 Old Generation","CollectionCount"]"]}
2019-08-26 15:20:14.426 [pool-1-thread-5] DEBUG com.zabbix.gateway.SocketProcessor - dispatched request to class com.zabbix.gateway.JMXItemChecker
2019-08-26 15:20:14.426 [pool-1-thread-5] DEBUG c.z.g.ZabbixJMXConnectorFactory - connecting to JMX agent at 'service:jmx:rmi:///jndi/rmi://192.168.10.31:12345/jmxrmi'
2019-08-26 15:20:17.427 [pool-1-thread-5] WARN com.zabbix.gateway.SocketProcessor - error processing request, item "jmx["java.lang:type=GarbageCollector,name=G1 Old Generation","CollectionCount"]" failed: Connection timed out: service:jmx:rmi:///jndi/rmi://192.168.10.31:12345/jmxrmi
2019-08-26 15:20:17.427 [pool-1-thread-5] DEBUG com.zabbix.gateway.SocketProcessor - error caused by
com.zabbix.gateway.ZabbixException: Connection timed out: service:jmx:rmi:///jndi/rmi://192.168.10.31:12345/jmxrmi
at com.zabbix.gateway.JMXItemChecker.getValues(JMXIte mChecker.java:128) ~[zabbix-java-gateway-4.2.5.jar:na]
at com.zabbix.gateway.SocketProcessor.run(SocketProce ssor.java:62) ~[zabbix-java-gateway-4.2.5.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1149) [na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) [na:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2019-08-26 15:20:17.428 [pool-1-thread-5] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"Connection timed out: service:jmx:rmi:\/\/\/jndi\/rmi:\/\/192.168.10.31:12345\/jmxrmi"}
2019-08-26 15:20:17.428 [pool-1-thread-5] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection


I create zabbix by same commands three months ago.
I have no idea why jmx cannnot by be accessed by javagateway this time.
Any advice will be help.
Thank you in advance.

[heeroz]

Here is some additional information.

I have 2 zabbix container servers.
One was created with 4.2.3 three months ago.
Another one was create with 4.2.5 this week.
Selinux and firewall are disabled.

In 4.2.3 zabbix server. I stop 4.2.3 javagateway and deploy 4.2.5 javagateway container.
jmx can be accessed.

Maybe some wrong in 4.2.5 zabbix server container.
# cat zabbix_get_jmx
#!/usr/bin/env bash
ZBXGET="/usr/bin/zabbix_get"
if [ $# != 5 ]
then
echo "Usage: $0 <JAVA_GATEWAY_HOST> <JAVA_GATEWAY_PORT> <JMX_SERVER> <JMX_PORT> <KEY>"
exit;
fi
QUERY="{"request": "java gateway jmx","conn": "$3","port": $4,"keys": ["$5"]}"
echo $QUERY
$ZBXGET -s $1 -p $2 -k "$QUERY"

# ./zabbix_get_jmx localhost 10052 192.168.10.7 12345 'jmx[java.lang:type=Threading,DaemonThreadCount]'
{"request": "java gateway jmx","conn": "192.168.10.7","port": 12345,"keys": ["jmx[java.lang:type=Threading,DaemonThreadCount]"]}
{"response":"failed","error":"No value for jmx_endpoint"}