VMWare monitoring, Timeout was reached

[mikkelrobin]

Hi.

Im trying to monitor my VMWare installation using Zabbix 2.2.1 - I have tested the following on my Zabbix Installation running in production, witch is an upgrade from 2.0.9 and on a fresh installed Zabbix 2.2.1 Appliance - With the same results.

My VMWare vCenter is a 5.5 running on a Virtual Appliance.

When follow the guide from here: https://www.zabbix.com/documentation.../vm_monitoring

Im just getting "Timeout was reached" on the Items on the host.

The funny thing is, that it seems like the collector is running fine, but the item is not able to update.


Please help


Here is some debug info:

Logs:

 27435:20140107:110415.909 vmware collector #1 [updated 0, removed 0 VMware services in 0.000005 sec, querying VMware services]
 27435:20140107:110415.909 vmware collector #1 [updated 0, removed 0 VMware services in 0.000005 sec, idle 5 sec]
 27430:20140107:110416.920 item [vcenter:vmware.fullname[{$URL}]] became not supported: Timeout was reached
 27435:20140107:110420.910 vmware collector #1 [updated 0, removed 0 VMware services in 0.000005 sec, querying VMware services]
 27435:20140107:110420.910 vmware collector #1 [updated 0, removed 0 VMware services in 0.000009 sec, idle 5 sec]
 27403:20140107:110424.125 In substitute_key_macros() data:'vmware.eventlog[{$URL}]'
 27403:20140107:110424.126 End of substitute_key_macros():SUCCEED data:'vmware.eventlog[https://192.168.3.70/sdk]'
 27403:20140107:110424.126 End substitute_simple_macros() data:'vmware'
 27403:20140107:110424.126 In get_value() key:'vmware.eventlog[{$URL}]'
 27403:20140107:110424.126 In get_value_simple() key_orig:'vmware.eventlog[{$URL}]' addr:'192.168.3.70'
 27403:20140107:110424.126 In zbx_vmware_get_service() vmware@https://192.168.3.70/sdk
 27403:20140107:110424.126 End of zbx_vmware_get_service():SUCCEED
 27403:20140107:110424.126 Item [vcenter:vmware.eventlog[{$URL}]] error: Timeout was reached
 27435:20140107:110425.911 vmware collector #1 [updated 0, removed 0 VMware services in 0.000009 sec, querying VMware services]
 27435:20140107:110425.911 vmware collector #1 [updated 0, removed 0 VMware services in 0.000010 sec, idle 5 sec]
 27428:20140107:110426.925 item [vcenter:vmware.eventlog[{$URL}]] became not supported: Timeout was reached
 27435:20140107:110430.911 vmware collector #1 [updated 0, removed 0 VMware services in 0.000010 sec, querying VMware services]
 27435:20140107:110430.912 vmware collector #1 [updated 0, removed 0 VMware services in 0.000005 sec, idle 5 sec]

VMWare part of config file:

### Option: StartVMwareCollectors
#       Number of pre-forked vmware collector instances.
#
# Mandatory: no
# Range: 0-250
# Default:
StartVMwareCollectors=1

### Option: VMwareFrequency
#       How often Zabbix will connect to VMware service to obtain a new data.
#
# Mandatory: no
# Range: 10-86400
# Default:
VMwareFrequency=60

### Option: VMwareCacheSize
#       Size of VMware cache, in bytes.
#       Shared memory size for storing VMware data.
#       Only used if VMware collectors are started.
#
# Mandatory: no
# Range: 256K-2G
# Default:
VMwareCacheSize=8M

[steveboyson]

Just a thought:

Are you sure that $USERNAME and $PASSWORD macros are set correctly? Is this user really permitted to query the ESX host?

Quite often this is an authentication issue.

[mikkelrobin]

Auth review

Hi.

I have checked and double checked that too

It seems like the collector is getting data but the problem is the item itself.

 27403:20140107:114619.822 In substitute_key_macros() data:'vmware.fullname[{$URL}]'
 27403:20140107:114619.823 End of substitute_key_macros():SUCCEED data:'vmware.fullname[http://192.168.3.70/sdk]'
 27403:20140107:114619.824 End substitute_simple_macros() data:'vmware'
 27403:20140107:114619.824 In get_value() key:'vmware.fullname[{$URL}]'
 27403:20140107:114619.824 In get_value_simple() key_orig:'vmware.fullname[{$URL}]' addr:'192.168.3.70'
 27403:20140107:114619.824 In zbx_vmware_get_service() vmware@http://192.168.3.70/sdk
 27403:20140107:114619.825 End of zbx_vmware_get_service():SUCCEED
 27403:20140107:114619.825 Item [vcenter:vmware.fullname[{$URL}]] error: Couldn't connect to server
 27428:20140107:114622.545 item [vcenter:vmware.fullname[{$URL}]] became not supported: Couldn't connect to server
 27435:20140107:114629.029 End of vmware_service_get_event_session():FAIL
 27435:20140107:114629.030 End of vmware_service_get_event_data():FAIL
 27435:20140107:114629.031 End of vmware_service_update():FAIL
 27435:20140107:114629.031 In vmware_service_update() vmware@http://192.168.3.70/sdk
 27435:20140107:114629.031 In vmware_service_authenticate() url:'http://192.168.3.70/sdk' username:'vmware'
 27435:20140107:114629.444 End of vmware_service_authenticate():SUCCEED
 27435:20140107:114629.444 In vmware_service_get_hv_list()
 27435:20140107:114630.450 vmware_service_get_hv_list() page.data:'<?xml version="1.0" encoding="UTF-8"?>

 27435:20140107:114612.881 End of vmware_service_get_perfcounter_refreshrate():SUCCEED
 27435:20140107:114613.904 End of vmware_service_get_vm_stats():SUCCEED
 27435:20140107:114613.905 End of vmware_service_create_vm():SUCCEED
 27435:20140107:114613.905 In vmware_service_create_vm() vmid:'vm-242'
 27435:20140107:114613.905 In vmware_service_get_vm_data() vmid:'vm-242'
 27435:20140107:114613.916 End of vmware_service_get_vm_data():SUCCEED
 27435:20140107:114613.918 In vmware_service_get_vm_stats()
 27435:20140107:114613.918 In vmware_service_get_perfcounter_refreshrate()
 27435:20140107:114613.922 vmware_service_get_perfcounter_refreshrate() refresh_rate:20
 27435:20140107:114613.922 End of vmware_service_get_perfcounter_refreshrate():SUCCEED
 27435:20140107:114614.939 End of vmware_service_get_vm_stats():SUCCEED
 27435:20140107:114614.940 End of vmware_service_create_vm():SUCCEED
 27435:20140107:114614.940 In vmware_service_create_vm() vmid:'vm-234'
 27435:20140107:114614.940 In vmware_service_get_vm_data() vmid:'vm-234'
 27435:20140107:114614.948 End of vmware_service_get_vm_data():SUCCEED
 27435:20140107:114614.949 In vmware_service_get_vm_stats()
 27435:20140107:114614.950 In vmware_service_get_perfcounter_refreshrate()
 27435:20140107:114614.952 vmware_service_get_perfcounter_refreshrate() refresh_rate:20

[steveboyson]

Item [vcenter:vmware.fullname[{$URL}]] error: Couldn't connect to server

... seems as it can not connect to your ESX host.

Could you just do a test:

curl -k -I https://your-server/sdk

and see what is coming back?
It should return something like "400 Bad Request"

If not, there is a network and/or firewall issue.

[mikkelrobin]

Just to be sure, i have enabled both http and https on the vCenter to ensure this was not a SSL issue.

But everything seems OK, on both http and https.

root@zabbix:~# curl -I -k "http://192.168.3.70/sdk"
HTTP/1.1 400 Bad Request
Date: Tue, 7 Jan 2014 12:03:08 GMT
Connection: close
Content-Type: text; charset=plain
Content-Length: 0

root@zabbix:~# curl -I -k "https://192.168.3.70/sdk"
HTTP/1.1 400 Bad Request
Date: Tue, 7 Jan 2014 12:03:14 GMT
Connection: close
Content-Type: text; charset=plain
Content-Length: 0

root@zabbix:~#

I agree on the connection issue, but the collector is somehow able to pull XML data:

 27435:20140107:114613.905 End of vmware_service_create_vm():SUCCEED
 27435:20140107:114613.905 In vmware_service_create_vm() vmid:'vm-242'
 27435:20140107:114613.905 In vmware_service_get_vm_data() vmid:'vm-242'

[steveboyson]

Hmmm, then I have no more ideas. Are you really sure that your user 'vmware" is allowed to access?

Are you speaking to your ESX servers directly or are you connecting to your vcenter? That is a difference since VMWare's SSO (single sign on) is somewhat picky.

BTW, there is a new version of vcenter available (5.5.0.b).

To make it sure I would recommend to test a different user and try to connect directly to the ESX hosts (without vcenter).

[mikkelrobin]

Well, i have already been thinking about SSO, but i have a PHP Class that uses the SDK, and that is working fine with the same user. But i have also tried with another user, without success.

I have tried connection directly to a ESX host, and that is working fine.

I have not noticed the new vCenter version - I will upgrade as see if it do any difference.

[steveboyson]

Oh, directly to ESX it is working? That is at least an advance!

So I am not sure if Zabbix' vSphere Monitoring generally supports vCenter. AFAIK there is only direct ESX querying supported but I can be wrong.

[mikkelrobin]

Yeah, but in the doc it says:

{$URL} - VMware service (vCenter or vSphere) SDK URL (https://servername/sdk).

But maybe vCenter ver 5.5 is not yet supported.

[Proth]

some problem! you somehow managed to solve this problem?

[mikkelrobin]

Yes, and no!

Im not able to query my vCenter, so i have added the template on each ESXi host, and that is working.

Long way from the perfect solution, but at least i have my vmware env. monitored now.

I would love to open up a bug report to the Zabbix guys, but I'm not able to say if its Zabbix, the vSphere API, or my config, that's the problem.

[Proth]

Well, it's difficult to explain in English, but maybe I can give some useful information.
My problem is similar to yours, but same not work with templates for ESXi.

Identified failures zabbix authentication on a Web service VMware
Tested options:

connection with a domain account and SSO - equally unsuccessful
checking of the connection point by using VMware soapUI - it works

The problem is that zabbix can not authenticate in vSphere due to the fact that in vSphere 5.1 to change the method of authentication. To authenticate, you need a token.
I do not have any ideas to solve this problem. The token can be substituted with hands, but when will the session it will have to do again.

LOGS

5220:20140116:140324.669 vmware collector #4 [updated 0, removed 0 VMware services in 0.000002 sec, querying VMware services]
5220:20140116:140324.669 In vmware_service_update() monitoring@system-domain@https://server_name/sdk
5220:20140116:140324.669 In vmware_service_authenticate() url:'https://server_name/sdk' username:'monitoring@system-domain'
5220:20140116:140324.669 In vmware_service_authenticate(): try to detect the service type first using vCenter service manager object
5220:20140116:140324.669 In vmware_service_authenticate(): answer: <?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring>
Error returned by expat parser: not well-formed (invalid token)

yes, I think it zabbix will work with the version vSphere 5.0, which is sufficient only for authentication username and password

[mikkelrobin]

For auth on each ESXi host, I'm setting the macros directly on the host:

{$USER} => root (the local root account directly on the ESXi host)
{$PASSWORD} => your pw
{$URL} => https://<IP or DNS for the host>/sdk

I don't know if this help you in any way?

[Proth]

I'll try and find other methods to achieve success, but if it is not in zabbix code, I'm afraid we'll just have to wait for the fresh version in which the developers will fix this incompatibility

[dodgetimes2]

I'm having this same exact problem on Zabbix 2.2.1 trying to monitor vCenter 5.1.0. I've tried using both the root account as well as a read-only zabbix user account to connect to vCenter. After turning up the debug level on my Zabbix server, I see that I'm getting the "Timeout was reached" error when trying to connect to vCenter. I've tried using curl to check the URL and I am able to get a 400 when hitting the SDK URL. Anything else I can check?