VMware monitoring with Zabbix, Python & vPoller

[noeand]

Hey guys,

Just wanted to share with you a different way of monitoring your VMware environment using Zabbix integration with vPoller.

Documentation on setting up Zabbix with vPoller can be found on the link below.

http://unix-heaven.org/node/115

Hope you find it useful!

[steveboyson]

Looks quite promising, althougj I'm a bit intimidated by the python (and dependancies) install.

I will definetly give it a try in the next week. Right now we have a big bunch of scripts in perl, python etc. which periodically query the vSphere API (or WBEM) from the "vMA" appliance. Quite costly but it works ...

Does vPoller continiously poll the vSphere environment and keeps data instantly "accessible" to clients or are requests send one-by-one and in-time to the vCenter/ESXi host?

Anyway, thanks very much for your work!

[noeand]

Oh, I've just noticed that I've sent a wrong link to the Zabbix + vPoller documentation

The correct link for setting up Zabbix with vPoller is this one:

* http:/unix-heaven.org/node/114

And this documentation here describes how you can import your vSphere objects as regular Zabbix hosts instead:

* http://unix-heaven.org/node/115

Now, to the questions

Looks quite promising, althougj I'm a bit intimidated by the python (and dependancies) install.

If you are already running a GNU/Linux system chances are you already have Python, so it's just a matter of getting some additional modules installed

You can also install the Python stuff in a virtualenv, so you don't make changes globally on your system, which is also a great way for testing things out.

Does vPoller continiously poll the vSphere environment and keeps data instantly "accessible" to clients or are requests send one-by-one and in-time to the vCenter/ESXi host?

vPoller runs asynchronous Workers which take care of polling properties from the vSphere environment. The workers process any requests simultaneously, which means you don't have to wait for one request to be completed in order to start the with the next one.

The actual polling of vSphere properties by the Workers is done by getting a vSphere View of a subset from the Inventory and then using a PropertyCollector in order to collect the requested properties. This way we effectively retrieve only properties from objects we request and do not traverse the whole vSphere Inventory, which could be a real issue in large environments.

In front of every pool of vPoller Workers we have a vPoller Proxy which load-balances requests to all Workers and allows vPoller to scale. Adding more Workers to a pool also means adding more processing power to vPoller.

Let me know if you have any questions and I'll try to answer them

[steveboyson]

Hey, thank you very much again!

While calling myself an experienced Linux admin (doing that since mid '90s) I am still (or again) not convinced with side-installing packages besides the systems's packaging system. That is: it would be nice to have Debian or RPM packages in place to ease the installation.
But I am completely fine without packages on test systems, just when it comes to rollout to production I tend to behave quite lazy ;-)

I understood the basic architecture of vPoller with the workers and proxies. What I am interested is if the workers keep some sort of a "copy" of the whole vSphere inventory/performance/... tree or if they execute the requests as they are coming.

There are some very costly API calls, for example the stats for each individual VMDK file (thin vs. thick, eager vs. lazy, saved bytes through thin provisioning, locks, snapshots, etc.).

While as an example our script needs about 4 minutes to query 3 ESX hosts for ~35 VMs with (in summary) ~60 VMDKs attached I wonder if vPoller just has this data instantly available and constantly "pre-polls" in the background for the next incoming request or if that request is executed just on demand.
(Did not check if that specific call is implemented in vPoller, though).

Also, digging through all the WBEM classes or getting the datastore IOPs and latencies, CPU stats and the like takes a fair amount of time (and memory and CPU cycles) on our vMA.
That's why I was asking - I would like to have that more "speedy"

Thank you!

[noeand]

While calling myself an experienced Linux admin (doing that since mid '90s) I am still (or again) not convinced with side-installing packages besides the systems's packaging system. That is: it would be nice to have Debian or RPM packages in place to ease the installation.
But I am completely fine without packages on test systems, just when it comes to rollout to production I tend to behave quite lazy ;-)

I totally agree with you that having package installs is the preferred way for deploying stuff on production. It makes things easier to manage, especially if you are running some Configuration Management system out there and want to roll out everything nice and easy

This has been on my todo list for some time, but unfortunately haven't found time to actually do it. I could work on this in the next days, but cannot say for sure that they will be ready, though.

I understood the basic architecture of vPoller with the workers and proxies. What I am interested is if the workers keep some sort of a "copy" of the whole vSphere inventory/performance/... tree or if they execute the requests as they are coming.

They execute requests as they are coming, keeping a local cache would probably make the Workers more greedy when it comes to resources, e.g. eat much more memory. This might change in the future, but for now vPoller execute requests as they come in.

While as an example our script needs about 4 minutes to query 3 ESX hosts for ~35 VMs with (in summary) ~60 VMDKs attached I wonder if vPoller just has this data instantly available and constantly "pre-polls" in the background for the next incoming request or if that request is executed just on demand.
(Did not check if that specific call is implemented in vPoller, though).

Wow, 4 minutes seems like a lot of time

Largest test I've done with vPoller was with ~100 ESXi hosts and ~600 VMs with ~30 items for each ESXi host and ~30 items for each VM with a 60 second poll interval for an item. The number of Workers running there were 8 and the time for a single request to be processed was 50-100 milliseconds each. The vCenter itself was fine as well with the number of requests fired at it

Of course these were my tests, and yours could vary. Best way to check performance is actually test it and see how it works for you.

Also, digging through all the WBEM classes or getting the datastore IOPs and latencies, CPU stats and the like takes a fair amount of time (and memory and CPU cycles) on our vMA.
That's why I was asking - I would like to have that more "speedy"

Thing is that with vPoller we are trying to poll a property as fast as possible, so it can be added to Zabbix and avoid any timeouts during that API call. So far everything works well here for me and I'm pretty pleased with the speed, but as mentioned earlier it's best to test it out for yourself and see if it fits your needs.

If you need more info, just ask

[steveboyson]

Thanks for clarification. As I mentioned before I will start some testing in the next week and will of course keep you informed how it works in our rather small environment.

[noeand]

Sure, let me know if you run into any troubles

[gavind]

Please update us Steve on the results.

[steveboyson]

Hmmm. zeromq in version 4.x is not available as packages for debian or ubuntu.
Installation went fine, anyway, as described in the docs.
I added "--prefix=/usr/local" to "./configure" just to be safe.

me@mars:/usr/local/vPoller/py-vpoller# ls -l /usr/local/lib/|grep zm
-rw-r--r--  1 root root  8106058 Mai  2 21:26 libzmq.a
-rwxr-xr-x  1 root root      947 Mai  2 21:26 libzmq.la
lrwxrwxrwx  1 root root       15 Mai  2 21:26 libzmq.so -> libzmq.so.4.0.0
lrwxrwxrwx  1 root root       15 Mai  2 21:26 libzmq.so.4 -> libzmq.so.4.0.0
-rwxr-xr-x  1 root root  3231507 Mai  2 21:26 libzmq.so.4.0.0

Installing py-vpoller tends to be much more complicated.
I created a virtualenv python directorym which was successfull.
I was not able to find the git link to py-vpoller, however by using

git clone git://github.com/dnaeon/py-vpoller.git

in the previously created virtualenv I got it downloaded.

Changed into that directory and

python setup.py install

and that gave at the end:

...
Searching for pyzmq>=14.1.1
Reading http://pypi.python.org/simple/pyzmq/
Reading http://github.com/zeromq/pyzmq
Reading http://github.com/zeromq/pyzmq/downloads
Reading http://github.com/zeromq/pyzmq/releases
Best match: pyzmq 14.2.0
Downloading https://pypi.python.org/packages/source/p/pyzmq/pyzmq-14.2.0.zip#md5=b3744ba372d70db9ce888fb4cd535db4
Processing pyzmq-14.2.0.zip
Running pyzmq-14.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-uslJFN/pyzmq-14.2.0/egg-dist-tmp-QLQae5
no previously-included directories found matching 'docs/build'
no previously-included directories found matching 'docs/gh-pages'
warning: no previously-included files found matching 'bundled/zeromq/src/Makefile*'
warning: no previously-included files found matching 'bundled/zeromq/src/platform.hpp'
warning: no previously-included files found matching 'setup.cfg'
warning: no previously-included files found matching 'zmq/libzmq*'
warning: no previously-included files matching '__pycache__/*' found anywhere in distribution
warning: no previously-included files matching '.deps/*' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '.git*' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
warning: no previously-included files matching '.mailmap' found anywhere in distribution
warning: no previously-included files matching 'Makefile.am' found anywhere in distribution
warning: no previously-included files matching 'Makefile.in' found anywhere in distribution
Using zmq-prefix /usr/local (found via pkg-config).
************************************************
Configure: Autodetecting ZMQ settings...
    Custom ZMQ dir:
[COLOR="Red"]build/temp.linux-x86_64-2.7/scratch/tmp/easy_install-uslJFN/pyzmq-14.2.0/temp/timer_createHP4171.o: In function `main':
timer_createHP4171.c:(.text+0x15): undefined reference to `timer_create'[/COLOR]
collect2: ld gab 1 als Ende-Status zurÃŒck
    ZMQ version detected: 4.0.5
************************************************
zip_safe flag not set; analyzing archive contents...
zmq.__init__: module references __file__
zmq.backend.cffi._cffi: module references __file__
Adding pyzmq 14.2.0 to easy-install.pth file

Installed /usr/local/vPoller/lib/python2.7/site-packages/pyzmq-14.2.0-py2.7-linux-x86_64.egg
Finished processing dependencies for py-vpoller==0.2.0

I'm not sure if it makes sense to proceed from that point.

[steveboyson]

Anyway, having paths different from the examples is not the right idea so far, at least this is what I'am suspceting.

vconnector-cli -d /usr/local/vPoller/vconnector/vconnector.db -o /usr/local/vPoller/log/vconnector.log -H vcenter.XXXXX -U XXXXX -P XXXXX add

gives:

Traceback (most recent call last):
  File "/usr/local/vPoller/bin/vconnector-cli", line 5, in <module>
    pkg_resources.run_script('py-vpoller==0.2.0', 'vconnector-cli')
  File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 182, in <module>
    main()
  File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 167, in main
    host=args['--host'],
  File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 65, in add_update_agent
    host=host,
  File "/usr/local/vPoller/local/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/vpoller/connector.py", line 355, in add_update_agent
    self.cursor.execute('INSERT OR REPLACE INTO hosts VALUES (?,?,?,?)', (host, user, pwd, enabled))
sqlite3.OperationalError: no such table: hosts

The SQLite file gets created, though, but is empty (0 bytes).
The log say that everything is ok:

cat /usr/local/vPoller/log/vconnector.log
2014-05-02 22:09:55,266 - INFO - vconnector-cli[11029]: Adding/updating vSphere Agent vcenter.XXXXX in database

I will stop here and wait for some comments... ;-)

[noeand]

Installing in a Python virtualenv should be as easy as doing this:

$ virtualenv vpoller-venv
$ source vpoller-venv/bin/activate
$ git clone https://github.com/dnaeon/py-vpoller.git
$ cd py-vpoller
$ python setup.py install

This should get you all the dependencies installed as well.

As for this error from the ZeroMQ build:

************************************************
Configure: Autodetecting ZMQ settings...
    Custom ZMQ dir:
build/temp.linux-x86_64-2.7/scratch/tmp/easy_install-uslJFN/pyzmq-14.2.0/temp/timer_createHP4171.o: In function `main':
timer_createHP4171.c:(.text+0x15): undefined reference to `timer_create'
collect2: ld gab 1 als Ende-Status zurÃŒck
    ZMQ version detected: 4.0.5
************************************************

You should not worry about it much, because it will successfully build and install the module.

Here's a quick way to confirm that you've successfully installed ZeroMQ:

$ python
>>> import zmq
>>> print zmq.zmq_version()

[noeand]

Anyway, having paths different from the examples is not the right idea so far, at least this is what I'am suspceting.

vconnector-cli -d /usr/local/vPoller/vconnector/vconnector.db -o /usr/local/vPoller/log/vconnector.log -H vcenter.XXXXX -U XXXXX -P XXXXX add

gives:

Traceback (most recent call last):
  File "/usr/local/vPoller/bin/vconnector-cli", line 5, in <module>
    pkg_resources.run_script('py-vpoller==0.2.0', 'vconnector-cli')
  File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 182, in <module>
    main()
  File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 167, in main
    host=args['--host'],
  File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 65, in add_update_agent
    host=host,
  File "/usr/local/vPoller/local/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/vpoller/connector.py", line 355, in add_update_agent
    self.cursor.execute('INSERT OR REPLACE INTO hosts VALUES (?,?,?,?)', (host, user, pwd, enabled))
sqlite3.OperationalError: no such table: hosts

The SQLite file gets created, though, but is empty (0 bytes).
The log say that everything is ok:

cat /usr/local/vPoller/log/vconnector.log
2014-05-02 22:09:55,266 - INFO - vconnector-cli[11029]: Adding/updating vSphere Agent vcenter.XXXXX in database

I will stop here and wait for some comments... ;-)

Okay, I see what the issue is - the database file is not initialized.

Please, execute this command before trying to add a vCenter first:

$ vconnector-cli init

Or in your case it would be:

$ vconnector-cli -d /usr/local/vPoller/vconnector/vconnector.db -o /usr/local/vPoller/log/vconnector.log init

After that you should be able to add your vCenter. I will update the documentation tomorrow, because it seems this one is not mentioned there

I guess it's time for me to get these Debian & RHEL packages ready in order to ease installation and testing

[steveboyson]

Ok, with "vconnector-client init" it's working and the SQlite db file gets installed and populated with the requested entries.

Time to move on ;-)

Thanks for your help!

P.S. Python's virtualenv was not the problem as mentioned before ...

[steveboyson]

vpoller-proxy starts fine and creates listeners as defined in the conf file. vpoller-worker does not start and emits an "address already in use" error.

A short test gave that the webmin service was active on :10000. Stopping that helped ;-) and vpoller-worker started.

I did some quick tests and the basic "-m ..." commands do work well. What I do not get is the status output, probably again due to path/logfile/pid issues.

i.e.

vpoller-worker -p /usr/local/vPoller/run/vpoller-worker.pid -f /usr/local/etc/vpoller.conf -o /usr/local/vPoller/log/vpoller-worker.log -e tcp://localhost:10000 status

fails.

Regarding that, it would be nice if the programs would respect some environment variables, let's say
- VPOLLER_PID=/my/prefered/path/$(basename $PROG).pid
- VPOLLER_LOG=/my/prefered/path/$(basename $PROG).log
- VPOLLER_CONF=/my/preferred/path/vpoller.conf
and so on.

For the construct "$(basename $PROG)" it would insert vpoller-{proxy|worker|client} as needed, of course.

I have some problems on reprobe the shown examples on http://unix-heaven.org/node/103 (it is presumably "host.get" and not "host.poll" as shown in the screens). A better documentation for the methods and their optional or required parameters would be really nice.

Also, some of the outputs are not really usefull or throw errors:

vpoller-client -o /usr/local/vPoller/log/vpoller-client.log -V vcenter -m host.get -n esx1 -p runtime.bootTime
{
    "msg": "Cannot serialize result: datetime.datetime(2014, 4, 23, 12, 11, 59, 769750, tzinfo=<pyVmomi.Iso8601.TZInfo object at 0x17bcdd0>) is not JSON serializable",
    "success": 1
}

What I see is the almost lightning speed of vPoller and it's friends. Usually, login to a vSphere environment takes up to 5 seconds. As vpoller-worker is already logged in the results come back in a very amazing speed!

So, as a first conclusion: it looks quite promising regarding speed, but lacks some documentation and most likely needs some more work.
I would love to see vPoller becoming a mature way of vSphere monitoring and will definitely stay on track.

Thank you for your very good work!

[noeand]

vpoller-proxy starts fine and creates listeners as defined in the conf file. vpoller-worker does not start and emits an "address already in use" error.

A short test gave that the webmin service was active on :10000. Stopping that helped ;-) and vpoller-worker started.

I did some quick tests and the basic "-m ..." commands do work well. What I do not get is the status output, probably again due to path/logfile/pid issues.

i.e.

vpoller-worker -p /usr/local/vPoller/run/vpoller-worker.pid -f /usr/local/etc/vpoller.conf -o /usr/local/vPoller/log/vpoller-worker.log -e tcp://localhost:10000 status

fails.

Until the Debian and RHEL packages for vPoller are ready I thought it would be good to have a bootstrap script which would install and configure everything for you nice and easy.

You can find the the bootstrap script at the link below which you can use as a quick way to install and configure everything in one shot:

* https://github.com/dnaeon/py-vpoller...rap-vpoller.sh
* https://github.com/dnaeon/py-vpoller#installation

As for the "status" vPoller Worker requests -- there was a leftover from the refactoring, which is now fixed. You should be able to get the "status" information now

Regarding that, it would be nice if the programs would respect some environment variables, let's say
- VPOLLER_PID=/my/prefered/path/$(basename $PROG).pid
- VPOLLER_LOG=/my/prefered/path/$(basename $PROG).log
- VPOLLER_CONF=/my/preferred/path/vpoller.conf
and so on.

For the construct "$(basename $PROG)" it would insert vpoller-{proxy|worker|client} as needed, of course.

Yes, that would be nice to have. Could you please submit a bug issue at the issue tracker and I'll take care of it?