VMware monitoring with Zabbix, Python & vPoller

[noeand]

I have some problems on reprobe the shown examples on http://unix-heaven.org/node/103 (it is presumably "host.get" and not "host.poll" as shown in the screens). A better documentation for the methods and their optional or required parameters would be really nice.

Yes, this page actually needs some updates to it as these were the vPoller methods used in pre 0.2.0 version. The new methods were discussed in the changelog for next releases here:

* http://unix-heaven.org/node/104

Also, some of the outputs are not really usefull or throw errors:

vpoller-client -o /usr/local/vPoller/log/vpoller-client.log -V vcenter -m host.get -n esx1 -p runtime.bootTime
{
    "msg": "Cannot serialize result: datetime.datetime(2014, 4, 23, 12, 11, 59, 769750, tzinfo=<pyVmomi.Iso8601.TZInfo object at 0x17bcdd0>) is not JSON serializable",
    "success": 1
}

That one looks like a candidate for another bug issue as well With the pyVmomi integration we've had a lot of refactoring and code cleanup, but as you can see some things got forgotten. vPoller tries to serialize all data into a human-readable way, but as with vSphere data this is usually vSphere objects sometime we need to perform some extra work in order to present the data in a nice way. Luckily this is an easy one to fix

What I see is the almost lightning speed of vPoller and it's friends. Usually, login to a vSphere environment takes up to 5 seconds. As vpoller-worker is already logged in the results come back in a very amazing speed!

So, as a first conclusion: it looks quite promising regarding speed, but lacks some documentation and most likely needs some more work.
I would love to see vPoller becoming a mature way of vSphere monitoring and will definitely stay on track.

Thank you for your very good work!

Right now the most accurate information regarding documentation is the vPoller table of supported methods at the Github README.md documentation:

* https://github.com/dnaeon/py-vpoller...ster/README.md

And the most accurate documentation with examples to look at is the agent.py module, which provides information for each method in it's docstrings:

* https://github.com/dnaeon/py-vpoller...oller/agent.py

Also the most complete list of using vPoller methods for monitoring you can find at the Zabbix templates for vPoller here:

* https://github.com/dnaeon/py-vpoller...-templates.xml

I know this needs more documentation, especially for the end-users who don't need to look at the module in order to find the properties they need to send, so I'll try to document that one as well in a better way.

Also, please feel free to submit bug issues to the Github issue tracker for any issues you might experience, so we can make vPoller better

Thanks!

[steveboyson]

Hmmmm, next problem (and not your fault):
I planned to implement vPoller on the "vSphere Management Assistant", we have 5.5 in place. Unfortunately the wise guys from Vmware decided to have that appliance based on SuSE Linux Enterprise 11, Pacthlevel 2.

And that wonderfull piece of a crippled software environment has only Python 2.6(.8) installed. Further, there are no repositories configured (as there are no "free" repos for SLES - you'll need a support contract for that).

Which in fact means that pyvmomi will not run nor will vPoller do unless I decide to "handport" a decent release of python onto that system.
Bummer.

So my next choice is either have a new VM with a current Linux distribution or go straight to our Zabbix server. So far I did basic testing on Ubuntu 12.04 and Opensuse-12.3 (our Zabbix server) - each with good results.

I need to think about that ;-)

[vnline]

I use the bootstrap-vpoller.sh script setup up vpoller .Use vconnector-cli command add host and start vpoller-proxy but can not pass,it's stuck,nothing output.How can I to resolve the issue?

vconnector-cli output:
+-------------+---------------+------------+-----------+
| Hostname | Username | Password | Enabled |
+=============+===============+============+====== =====+
| 172.16.1.xx | administrator | xxxxxxxx| 1 |
+-------------+---------------+------------+-----------+

vpoller work start error:

[root@xxx ~]# vpoller-worker start
[2] 3783
[root@xxx ~]# Traceback (most recent call last):
File "/usr/bin/vpoller-worker", line 5, in <module>
pkg_resources.run_script('py-vpoller==0.2.0', 'vpoller-worker')
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 461, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1194, in run_script
execfile(script_filename, namespace, namespace)
File "/usr/lib/python2.6/site-packages/py_vpoller-0.2.0-py2.6.egg/EGG-INFO/scripts/vpoller-worker", line 42, in <module>
from vpoller.worker import VPollerWorker
File "/usr/lib/python2.6/site-packages/py_vpoller-0.2.0-py2.6.egg/vpoller/worker.py", line 38, in <module>
from vpoller.agent import VSphereAgent
File "/usr/lib/python2.6/site-packages/py_vpoller-0.2.0-py2.6.egg/vpoller/agent.py", line 285
'result': [ {prop:getattr(self.si.content.about, prop, None) for prop in properties} ]
^
SyntaxError: invalid syntax

[steveboyson]

In my previous post I mentioned that Python 2.6 is not supported. You will need at least 2.7 for pyvmomi and ZeroMQ.

[noeand]

Hey guys,

So, I spent some time getting packages ready and I'm happy to say that packages for Debian GNU/Linux systems are ready

vPoller along with it's dependencies have been packaged for Debian systems using python-stdeb.

At the link below you can find the latest Debian GNU/Linux packages of vPoller, pyVmomi, Python ZeroMQ bindings, etc.

* http://files.unix-heaven.org/vpoller-packages/

The packages were built on the latest Debian 7.4 release and chances are they would install on a Ubuntu system as well, though I've only tested them on Debian.

Please note that the vPoller C Client is not included in the python-vpoller package, so if you need it you will have to build it yourself for now.

Github documentation has also been updated which now mentions from where to get the Debian packages as well.

So, in order to install the packages, simply grab the packages from the location above and use dpkg to install them.

If you happen to have an APT internal repo which you use, simply adding the vPoller packages there and using apt-get should pull all dependencies for you and install them.

Hope you find them useful!

[noeand]

Hmmmm, next problem (and not your fault):
I planned to implement vPoller on the "vSphere Management Assistant", we have 5.5 in place. Unfortunately the wise guys from Vmware decided to have that appliance based on SuSE Linux Enterprise 11, Pacthlevel 2.

And that wonderfull piece of a crippled software environment has only Python 2.6(.8) installed. Further, there are no repositories configured (as there are no "free" repos for SLES - you'll need a support contract for that).

Which in fact means that pyvmomi will not run nor will vPoller do unless I decide to "handport" a decent release of python onto that system.
Bummer.

So my next choice is either have a new VM with a current Linux distribution or go straight to our Zabbix server. So far I did basic testing on Ubuntu 12.04 and Opensuse-12.3 (our Zabbix server) - each with good results.

I need to think about that ;-)

Just an idea - since I'm already having some Zabbix Proxies installed I usually install vPoller Proxy/Workers on the Zabbix Proxy itself.

That way I don't touch the Zabbix Server/Frontend, just to make sure I don't put some load on it (even though the load won't be that much, but still)

Oh, and make sure you check out the Debian packages if you are about to deploy vPoller on a Debian system. They'll probably make your life easier during the deployment phase

[vnline]

I have about 60 vpshere hosts and about 300+ vms,when I add the vcent to zabbix monitor by vpoller.I got "Time out excute script" error for zabbix server and zabbix server starting 200 vpoller-clients to collecting data,this make zabbix server load too high!

[noeand]

I have about 60 vpshere hosts and about 300+ vms,when I add the vcent to zabbix monitor by vpoller.I got "Time out excute script" error for zabbix server and zabbix server starting 200 vpoller-clients to collecting data,this make zabbix server load too high!

Hi,

Can you please provide the following information?

* What is the number of Zabbix items you get for these hosts/vms?
* What is the number of vPoller Workers you are running?
* What are the hardware specs where you run Zabbix and vPoller?
* What is the number of Zabbix Pollers you have?

Also, please try using the Zabbix vPoller C Client templates and let me know how it goes.

I'm running vPoller with the default Zabbix Timeout settings, but depending on the environment size I run different number of vPoller Workers, so I can have more processing power.

Best way to find the optimal number of vPoller Workers required for your environment is to time the requests, e.g. start Zabbix with vPoller monitoring and then run time vpoller-client .... If it takes a lot of time, then simply add a vPoller Worker to the pool and repeat that.

Pretty soon you should be able to see what would be the required vPoller Workers for your environment.

I'm running vPoller on a number of sites in different sizes (some larger than yours) and don't have a timeout issue.

Also having a Zabbix Proxy where you can possibly run vPoller is a better option, since you won't put load on your Zabbix Server.

Let me know if you need any assistance on this one.

[vnline]

Hi,

Can you please provide the following information?

* What is the number of Zabbix items you get for these hosts/vms?
* What is the number of vPoller Workers you are running?
* What are the hardware specs where you run Zabbix and vPoller?
* What is the number of Zabbix Pollers you have?

Also, please try using the Zabbix vPoller C Client templates and let me know how it goes.

I'm running vPoller with the default Zabbix Timeout settings, but depending on the environment size I run different number of vPoller Workers, so I can have more processing power.

Best way to find the optimal number of vPoller Workers required for your environment is to time the requests, e.g. start Zabbix with vPoller monitoring and then run time vpoller-client .... If it takes a lot of time, then simply add a vPoller Worker to the pool and repeat that.

Pretty soon you should be able to see what would be the required vPoller Workers for your environment.

I'm running vPoller on a number of sites in different sizes (some larger than yours) and don't have a timeout issue.

Also having a Zabbix Proxy where you can possibly run vPoller is a better option, since you won't put load on your Zabbix Server.

Let me know if you need any assistance on this one.

Thank your reply

1、
2、Just one vpoller worker,because I not found how to start up more than one vpoller worker?
3、my zabbix hardware : 8 cpus,8G RAM,150G RAID0+1,it's running on vsphere.
4、my zabbix server config as below

StartPollers=200
StartIPMIPollers=50
StartPollersUnreachable=100
StartTrappers=200
StartPingers=300
StartDiscoverers=10
StartHTTPPollers=100
StartTimers=10
StartSNMPTrapper=1
HousekeepingFrequency=24
MaxHousekeeperDelete=5000
CacheSize=256M
StartDBSyncers=80
HistoryCacheSize=512M
TrendCacheSize=64M
HistoryTextCacheSize=128M
ValueCacheSize=128M
Timeout=3
UnreachablePeriod=15
UnavailableDelay=15
UnreachableDelay=15
AlertScriptsPath=/data/zabbix/bin
ExternalScripts=/data/zabbix/externalscripts
FpingLocation=/usr/local/sbin/fping
LogSlowQueries=1000
TmpDir=/tmp
ProxyConfigFrequency=3600
ProxyDataFrequency=1

[noeand]

2、Just one vpoller worker,because I not found how to start up more than one vpoller worker?

You can run any number of vPoller Workers by just specifying a different config.

So for example if you have this default example /etc/vpoller/vpoller.conf config file:

[proxy]
frontend = tcp://*:10123
backend  = tcp://*:10124
mgmt     = tcp://*:9999

[worker]
db       = /var/lib/vconnector/vconnector.db
proxy    = tcp://localhost:10124
mgmt     = tcp://*:10000

When you start the vpoller-proxy and vpoller-worker services you would have vPoller running with 1 Worker.

Here is how to add a second worker to your pool. Create a /etc/vpoller/worker-2.conf file with the following contents:

[worker]
db       = /var/lib/vconnector/vconnector.db
proxy    = tcp://localhost:10124
mgmt     = tcp://*:10001

Please note that the only change here in [worker] section from the default config file is the mgmt endpoint, so that each Worker binds to it's own management endpoint.

Now for starting the second vPoller Worker you can execute this command:

$ vpoller-worker -d -f /etc/vpoller/worker-2.conf

For adding more Workers just create a config file for each new Worker and then start it up.

I am using a custom init.d script here which takes care of starting up all vPoller Workers, but need to clean it up a bit first and then will publish it so you could use it instead of starting the workers manually. For now at least you could use this way in order to discover the optimal number of Workers for your environment.

3、my zabbix hardware : 8 cpus,8G RAM,150G RAID0+1,it's running on vsphere.

That should be more than enough.

4、my zabbix server config as below

StartPollers=200
StartIPMIPollers=50
StartPollersUnreachable=100
StartTrappers=200
StartPingers=300
StartDiscoverers=10
StartHTTPPollers=100
StartTimers=10
StartSNMPTrapper=1
HousekeepingFrequency=24
MaxHousekeeperDelete=5000
CacheSize=256M
StartDBSyncers=80
HistoryCacheSize=512M
TrendCacheSize=64M
HistoryTextCacheSize=128M
ValueCacheSize=128M
Timeout=3
UnreachablePeriod=15
UnavailableDelay=15
UnreachableDelay=15
AlertScriptsPath=/data/zabbix/bin
ExternalScripts=/data/zabbix/externalscripts
FpingLocation=/usr/local/sbin/fping
LogSlowQueries=1000
TmpDir=/tmp
ProxyConfigFrequency=3600
ProxyDataFrequency=1

You have a lot of Zabbix Pollers configured. I'd recommend using the vPoller C Client for that amount of pollers. It is "lighter" than the Python client and generally much faster.

When you add your vCenter host to Zabbix make sure you also attach the "Template VMware vSphere - vPoller with C Client" template.

Let me know if there's anything unclear or you need assistance.

[Mython]

You can run any number of vPoller Workers by just specifying a different config.

So for example if you have this default example /etc/vpoller/vpoller.conf config file:

[proxy]
frontend = tcp://*:10123
backend  = tcp://*:10124
mgmt     = tcp://*:9999

[worker]
db       = /var/lib/vconnector/vconnector.db
proxy    = tcp://localhost:10124
mgmt     = tcp://*:10000

When you start the vpoller-proxy and vpoller-worker services you would have vPoller running with 1 Worker.

Here is how to add a second worker to your pool. Create a /etc/vpoller/worker-2.conf file with the following contents:

[worker]
db       = /var/lib/vconnector/vconnector.db
proxy    = tcp://localhost:10124
mgmt     = tcp://*:10001

Please note that the only change here in [worker] section from the default config file is the mgmt endpoint, so that each Worker binds to it's own management endpoint.

Now for starting the second vPoller Worker you can execute this command:

$ vpoller-worker -d -f /etc/vpoller/worker-2.conf

For adding more Workers just create a config file for each new Worker and then start it up.

I am using a custom init.d script here which takes care of starting up all vPoller Workers, but need to clean it up a bit first and then will publish it so you could use it instead of starting the workers manually. For now at least you could use this way in order to discover the optimal number of Workers for your environment.



That should be more than enough.



You have a lot of Zabbix Pollers configured. I'd recommend using the vPoller C Client for that amount of pollers. It is "lighter" than the Python client and generally much faster.

When you add your vCenter host to Zabbix make sure you also attach the "Template VMware vSphere - vPoller with C Client" template.

Let me know if there's anything unclear or you need assistance.

"vpoller-worker -d -f /etc/vpoller/worker-2.conf START"

btw, it´s working great..

70 ESXi with 600vm, now without delays...

Thanks

[noeand]

"vpoller-worker -d -f /etc/vpoller/worker-2.conf START"

btw, it´s working great..

70 ESXi with 600vm, now without delays...

Thanks

Hi,

I see that you are using the "old way" of starting vPoller Workers from the example command you've provided above

I would recommend that you upgrade your vPoller setup as the new releases come with lots of improvements + a loadable vPoller module for Zabbix!

[Mython]

Hi,

I see that you are using the "old way" of starting vPoller Workers from the example command you've provided above

I would recommend that you upgrade your vPoller setup as the new releases come with lots of improvements + a loadable vPoller module for Zabbix!

Mmm rare, where do I can find the most updated documents to proceed with the installation of the release that you mentioned...
Thks in advance...

[noeand]

Mmm rare, where do I can find the most updated documents to proceed with the installation of the release that you mentioned...
Thks in advance...

Check the vPoller documentation at:

http://vpoller.readthedocs.org/en/latest/

Go over the "Installation" & "Configuration" sections of the documentation.

Once ready with that head to the "vPoller Integration with Zabbix" documentation:

http://vpoller.readthedocs.org/en/latest/vpoller-zabbix.html

Let me know if you run into troubles or even better submit an issue at the Github repo of vPoller:

https://github.com/dnaeon/py-vpoller

[oitss]

Is it possible to add method to query vim.StorageResourceManager.StoragePerformanceSumma ry for getting information about datastore performance parameters ?