JMX Monitoring "Permission Denied" error

[ifernandez]

Hello everyone!

I am trying to set up JMX Jenkins monitoring through Zabbix. I found a Jenkins plugin that adds a template with some discovery rules (https://github.com/hudson2-plugins/z...gration-plugin), and I have built it and installed it in Jenkins. It's set up pointing to the appropiate host, where the Zabbix server is installed.

I have googled quite a bit regarding JMX and Java monitoring and all that. Here is how I start up my Jenkins instance (in my Windows machine)

%BASE%\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.20.197.20 -jar "%BASE%\jenkins.war" --httpPort=8080

(formatting is bad but all that is one single line )

Here is what my JMX interface configuration looks like, for my machine:



Here is what my zabbix_server.conf file has to say about Java Gateways and such:

JavaGateway=easy-ifernande1
JavaGatewayPort=12345
StartJavaPollers=5

Here is the error I see:


And from the logs, with some more lines wrapped around (log level 4):

4434:20150917:111659.971 End of substitute_key_macros():SUCCEED data:'jmx["org.hudsonci.plugin.jmxmonitoring:type=Jenkins","Executors"]'
  4434:20150917:111659.971 In substitute_simple_macros() data:'12345'
  4434:20150917:111659.971 In substitute_simple_macros() data:EMPTY
  4434:20150917:111659.971 In substitute_simple_macros() data:EMPTY
  4434:20150917:111659.971 In get_values_java() host:'EASY-IFERNANDE1' addr:'easy-ifernande1.dice.ad.ea.com' num:1
  4434:20150917:111659.974 getting Java values failed: cannot connect to [[easy-ifernande1]:12345]: [13] Permission denied
  4434:20150917:111659.974 End of get_values_java()
  4434:20150917:111659.974 In deactivate_host() hostid:10133 itemid:25077 type:16
  4434:20150917:111659.974 query [txnlev:1] [begin;]
  4434:20150917:111659.975 query [txnlev:1] [update hosts set jmx_disable_until=1442485079,jmx_error='cannot connect to [[easy-ifernande1]:12345]: [13] Permission denied' where hostid=10133]

I have installed the zabbix_java_gateway package, but I don't know if that's necessary, if it has to be running at the same time, etc. etc.; Furthermore, when installing it created a .service file, but as much as I try to enable it with systemctl I can't see it on the list.

As an example, here is the config of one of the items included in the template from the Jenkins plugin:


This is my zabbix version:

Zabbix server v2.4.6 (revision 54796) (10 August 2015)
Compilation time: Aug 10 2015 17:18:14

I don't believe it's the Jenkins plugin, since I can see MBeans with information on them. I can telnet both from my machine to the zabbix URL, and from the zabbix server machine to my machine on port 12345.

Any suggestions? I will accept both suggestions to fix this and also suggestions to put Jenkins monitoring in Zabbix, I won't be picky

Cheers!

[LenR]

We have macros in our JMX template like {$JMX_USERNAME} and {$JMX_PASSWORD}. When those aren't correct, that is the error we get. That id is used to login to the Java stack with authority to get the stats.

See

http://www.edugarage.com/display/BBDN/How+to+deploy+Zabbix+agents+for+Learn+Template+Suite

[ifernandez]

Ok! It seems the first problem was the fact that since 2.0.0, Zabbix includes the Java Gateway Agent by default.

I have disabled the "external" Java Gateway, and am using the server one now instead. It now receives data but it seems that the JSON data that is sent from the Jenkins server gets lost somewhere.

 24695:20150921:083918.917 End of DCconfig_get_poller_items():1
 24695:20150921:083918.917 In substitute_key_macros() data:'jmx["org.hudsonci.plugin.zabbix:type=Discovery","Nodes"]'
 24695:20150921:083918.917 End of substitute_key_macros():SUCCEED data:'jmx["org.hudsonci.plugin.zabbix:type=Discovery","Nodes"]'
 24695:20150921:083918.917 In substitute_simple_macros() data:'12345'
 24695:20150921:083918.917 In substitute_simple_macros() data:EMPTY
 24695:20150921:083918.917 In substitute_simple_macros() data:EMPTY
 24695:20150921:083918.917 In get_values_java() host:'EASY-IFERNANDE1' addr:'easy-ifernande1.dice.ad.ea.com' num:1
[B] 24695:20150921:083918.918 JSON before sending [{"request":"java gateway jmx","conn":"easy-ifernande1.dice.ad.ea.com","port":12345,"keys":["jmx[\"org.hudsonci.plugin.zabbix:type=Discovery\",\"Nodes\"]"]}]
 24695:20150921:083919.118 JSON back [][/B]
 24695:20150921:083919.119 getting Java values failed: Cannot open received JSON
 24695:20150921:083919.119 End of get_values_java()
 24695:20150921:083919.119 In deactivate_host() hostid:10135 itemid:25165 type:16
 24695:20150921:083919.119 query [txnlev:1] [begin;]
 24695:20150921:083919.119 query [txnlev:1] [update hosts set jmx_disable_until=1442821219,jmx_error='Cannot open received JSON' where hostid=10135]
 24695:20150921:083919.120 query [txnlev:1] [commit;]

Is there any way of debugging this?

Cheers!

P.S.: LenR, I can't open the link?

[LenR]

Odd on the link, search "blackboard zabbix java", it was my first hit.

[ifernandez]

Ok, I fixed this!! Here are my mistakes:

1) You NEED the Java Gateway. In my case, I built it from source (instead of yum) following the steps in the wiki.
2) I configured the Java Gateway like this (settings.sh file):

• LISTEN_IP=FDQN of the local machine
• LISTEN_PORT=10052

3) In zabbix_server.conf

• JavaGateway=FQDN of the local machine
• JavaGatewayPort=10052

4) Start both services
5) Magic

Cheers!

[mark.chan]

i just solve it by change the SELinux status

[root@localhost lib]# setenforce 0
[root@localhost lib]# getenforce
Permissive

and restart the server, and the jmx logo turns to GREEN!

FINALLY, I got it by DISABLE the SELinux!