Ad Widget

Collapse

VMware monitoring with Zabbix, Python & vPoller

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • noeand
    Junior Member
    • Apr 2014
    • 24

    #1

    VMware monitoring with Zabbix, Python & vPoller

    Hey guys,

    Just wanted to share with you a different way of monitoring your VMware environment using Zabbix integration with vPoller.

    Documentation on setting up Zabbix with vPoller can be found on the link below.



    Hope you find it useful!
  • steveboyson
    Senior Member
    • Jul 2013
    • 582

    #2
    Looks quite promising, althougj I'm a bit intimidated by the python (and dependancies) install.

    I will definetly give it a try in the next week. Right now we have a big bunch of scripts in perl, python etc. which periodically query the vSphere API (or WBEM) from the "vMA" appliance. Quite costly but it works ...

    Does vPoller continiously poll the vSphere environment and keeps data instantly "accessible" to clients or are requests send one-by-one and in-time to the vCenter/ESXi host?

    Anyway, thanks very much for your work!

    Comment

    • noeand
      Junior Member
      • Apr 2014
      • 24

      #3
      Oh, I've just noticed that I've sent a wrong link to the Zabbix + vPoller documentation

      The correct link for setting up Zabbix with vPoller is this one:

      * http:/unix-heaven.org/node/114

      And this documentation here describes how you can import your vSphere objects as regular Zabbix hosts instead:

      * http://unix-heaven.org/node/115

      Now, to the questions

      Looks quite promising, althougj I'm a bit intimidated by the python (and dependancies) install.
      If you are already running a GNU/Linux system chances are you already have Python, so it's just a matter of getting some additional modules installed

      You can also install the Python stuff in a virtualenv, so you don't make changes globally on your system, which is also a great way for testing things out.

      Does vPoller continiously poll the vSphere environment and keeps data instantly "accessible" to clients or are requests send one-by-one and in-time to the vCenter/ESXi host?
      vPoller runs asynchronous Workers which take care of polling properties from the vSphere environment. The workers process any requests simultaneously, which means you don't have to wait for one request to be completed in order to start the with the next one.

      The actual polling of vSphere properties by the Workers is done by getting a vSphere View of a subset from the Inventory and then using a PropertyCollector in order to collect the requested properties. This way we effectively retrieve only properties from objects we request and do not traverse the whole vSphere Inventory, which could be a real issue in large environments.

      In front of every pool of vPoller Workers we have a vPoller Proxy which load-balances requests to all Workers and allows vPoller to scale. Adding more Workers to a pool also means adding more processing power to vPoller.

      Let me know if you have any questions and I'll try to answer them
      Last edited by noeand; 01-05-2014, 08:48.

      Comment

      • steveboyson
        Senior Member
        • Jul 2013
        • 582

        #4
        Hey, thank you very much again!

        While calling myself an experienced Linux admin (doing that since mid '90s) I am still (or again) not convinced with side-installing packages besides the systems's packaging system. That is: it would be nice to have Debian or RPM packages in place to ease the installation.
        But I am completely fine without packages on test systems, just when it comes to rollout to production I tend to behave quite lazy ;-)

        I understood the basic architecture of vPoller with the workers and proxies. What I am interested is if the workers keep some sort of a "copy" of the whole vSphere inventory/performance/... tree or if they execute the requests as they are coming.

        There are some very costly API calls, for example the stats for each individual VMDK file (thin vs. thick, eager vs. lazy, saved bytes through thin provisioning, locks, snapshots, etc.).

        While as an example our script needs about 4 minutes to query 3 ESX hosts for ~35 VMs with (in summary) ~60 VMDKs attached I wonder if vPoller just has this data instantly available and constantly "pre-polls" in the background for the next incoming request or if that request is executed just on demand.
        (Did not check if that specific call is implemented in vPoller, though).

        Also, digging through all the WBEM classes or getting the datastore IOPs and latencies, CPU stats and the like takes a fair amount of time (and memory and CPU cycles) on our vMA.
        That's why I was asking - I would like to have that more "speedy"

        Thank you!

        Comment

        • noeand
          Junior Member
          • Apr 2014
          • 24

          #5
          While calling myself an experienced Linux admin (doing that since mid '90s) I am still (or again) not convinced with side-installing packages besides the systems's packaging system. That is: it would be nice to have Debian or RPM packages in place to ease the installation.
          But I am completely fine without packages on test systems, just when it comes to rollout to production I tend to behave quite lazy ;-)
          I totally agree with you that having package installs is the preferred way for deploying stuff on production. It makes things easier to manage, especially if you are running some Configuration Management system out there and want to roll out everything nice and easy

          This has been on my todo list for some time, but unfortunately haven't found time to actually do it. I could work on this in the next days, but cannot say for sure that they will be ready, though.

          I understood the basic architecture of vPoller with the workers and proxies. What I am interested is if the workers keep some sort of a "copy" of the whole vSphere inventory/performance/... tree or if they execute the requests as they are coming.
          They execute requests as they are coming, keeping a local cache would probably make the Workers more greedy when it comes to resources, e.g. eat much more memory. This might change in the future, but for now vPoller execute requests as they come in.

          While as an example our script needs about 4 minutes to query 3 ESX hosts for ~35 VMs with (in summary) ~60 VMDKs attached I wonder if vPoller just has this data instantly available and constantly "pre-polls" in the background for the next incoming request or if that request is executed just on demand.
          (Did not check if that specific call is implemented in vPoller, though).
          Wow, 4 minutes seems like a lot of time

          Largest test I've done with vPoller was with ~100 ESXi hosts and ~600 VMs with ~30 items for each ESXi host and ~30 items for each VM with a 60 second poll interval for an item. The number of Workers running there were 8 and the time for a single request to be processed was 50-100 milliseconds each. The vCenter itself was fine as well with the number of requests fired at it

          Of course these were my tests, and yours could vary. Best way to check performance is actually test it and see how it works for you.

          Also, digging through all the WBEM classes or getting the datastore IOPs and latencies, CPU stats and the like takes a fair amount of time (and memory and CPU cycles) on our vMA.
          That's why I was asking - I would like to have that more "speedy"
          Thing is that with vPoller we are trying to poll a property as fast as possible, so it can be added to Zabbix and avoid any timeouts during that API call. So far everything works well here for me and I'm pretty pleased with the speed, but as mentioned earlier it's best to test it out for yourself and see if it fits your needs.

          If you need more info, just ask

          Comment

          • steveboyson
            Senior Member
            • Jul 2013
            • 582

            #6
            Thanks for clarification. As I mentioned before I will start some testing in the next week and will of course keep you informed how it works in our rather small environment.

            Comment

            • noeand
              Junior Member
              • Apr 2014
              • 24

              #7
              Sure, let me know if you run into any troubles

              Comment

              • gavind
                Member
                • Mar 2013
                • 59

                #8
                Please update us Steve on the results.

                Comment

                • steveboyson
                  Senior Member
                  • Jul 2013
                  • 582

                  #9
                  Hmmm. zeromq in version 4.x is not available as packages for debian or ubuntu.
                  Installation went fine, anyway, as described in the docs.
                  I added "--prefix=/usr/local" to "./configure" just to be safe.

                  Code:
                  me@mars:/usr/local/vPoller/py-vpoller# ls -l /usr/local/lib/|grep zm
                  -rw-r--r--  1 root root  8106058 Mai  2 21:26 libzmq.a
                  -rwxr-xr-x  1 root root      947 Mai  2 21:26 libzmq.la
                  lrwxrwxrwx  1 root root       15 Mai  2 21:26 libzmq.so -> libzmq.so.4.0.0
                  lrwxrwxrwx  1 root root       15 Mai  2 21:26 libzmq.so.4 -> libzmq.so.4.0.0
                  -rwxr-xr-x  1 root root  3231507 Mai  2 21:26 libzmq.so.4.0.0
                  Installing py-vpoller tends to be much more complicated.
                  I created a virtualenv python directorym which was successfull.
                  I was not able to find the git link to py-vpoller, however by using
                  Code:
                  git clone git://github.com/dnaeon/py-vpoller.git
                  in the previously created virtualenv I got it downloaded.

                  Changed into that directory and
                  Code:
                  python setup.py install
                  and that gave at the end:

                  Code:
                  ...
                  Searching for pyzmq>=14.1.1
                  Reading http://pypi.python.org/simple/pyzmq/
                  Reading http://github.com/zeromq/pyzmq
                  Reading http://github.com/zeromq/pyzmq/downloads
                  Reading http://github.com/zeromq/pyzmq/releases
                  Best match: pyzmq 14.2.0
                  Downloading https://pypi.python.org/packages/source/p/pyzmq/pyzmq-14.2.0.zip#md5=b3744ba372d70db9ce888fb4cd535db4
                  Processing pyzmq-14.2.0.zip
                  Running pyzmq-14.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-uslJFN/pyzmq-14.2.0/egg-dist-tmp-QLQae5
                  no previously-included directories found matching 'docs/build'
                  no previously-included directories found matching 'docs/gh-pages'
                  warning: no previously-included files found matching 'bundled/zeromq/src/Makefile*'
                  warning: no previously-included files found matching 'bundled/zeromq/src/platform.hpp'
                  warning: no previously-included files found matching 'setup.cfg'
                  warning: no previously-included files found matching 'zmq/libzmq*'
                  warning: no previously-included files matching '__pycache__/*' found anywhere in distribution
                  warning: no previously-included files matching '.deps/*' found anywhere in distribution
                  warning: no previously-included files matching '*.so' found anywhere in distribution
                  warning: no previously-included files matching '*.pyd' found anywhere in distribution
                  warning: no previously-included files matching '.git*' found anywhere in distribution
                  warning: no previously-included files matching '.DS_Store' found anywhere in distribution
                  warning: no previously-included files matching '.mailmap' found anywhere in distribution
                  warning: no previously-included files matching 'Makefile.am' found anywhere in distribution
                  warning: no previously-included files matching 'Makefile.in' found anywhere in distribution
                  Using zmq-prefix /usr/local (found via pkg-config).
                  ************************************************
                  Configure: Autodetecting ZMQ settings...
                      Custom ZMQ dir:
                  [COLOR="Red"]build/temp.linux-x86_64-2.7/scratch/tmp/easy_install-uslJFN/pyzmq-14.2.0/temp/timer_createHP4171.o: In function `main':
                  timer_createHP4171.c:(.text+0x15): undefined reference to `timer_create'[/COLOR]
                  collect2: ld gab 1 als Ende-Status zurÃŒck
                      ZMQ version detected: 4.0.5
                  ************************************************
                  zip_safe flag not set; analyzing archive contents...
                  zmq.__init__: module references __file__
                  zmq.backend.cffi._cffi: module references __file__
                  Adding pyzmq 14.2.0 to easy-install.pth file
                  
                  Installed /usr/local/vPoller/lib/python2.7/site-packages/pyzmq-14.2.0-py2.7-linux-x86_64.egg
                  Finished processing dependencies for py-vpoller==0.2.0
                  I'm not sure if it makes sense to proceed from that point.

                  Comment

                  • steveboyson
                    Senior Member
                    • Jul 2013
                    • 582

                    #10
                    Anyway, having paths different from the examples is not the right idea so far, at least this is what I'am suspceting.

                    Code:
                    vconnector-cli -d /usr/local/vPoller/vconnector/vconnector.db -o /usr/local/vPoller/log/vconnector.log -H vcenter.XXXXX -U XXXXX -P XXXXX add
                    gives:
                    Code:
                    Traceback (most recent call last):
                      File "/usr/local/vPoller/bin/vconnector-cli", line 5, in <module>
                        pkg_resources.run_script('py-vpoller==0.2.0', 'vconnector-cli')
                      File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script
                        self.require(requires)[0].run_script(script_name, ns)
                      File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script
                        execfile(script_filename, namespace, namespace)
                      File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 182, in <module>
                        main()
                      File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 167, in main
                        host=args['--host'],
                      File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 65, in add_update_agent
                        host=host,
                      File "/usr/local/vPoller/local/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/vpoller/connector.py", line 355, in add_update_agent
                        self.cursor.execute('INSERT OR REPLACE INTO hosts VALUES (?,?,?,?)', (host, user, pwd, enabled))
                    sqlite3.OperationalError: no such table: hosts
                    The SQLite file gets created, though, but is empty (0 bytes).
                    The log say that everything is ok:
                    Code:
                    cat /usr/local/vPoller/log/vconnector.log
                    2014-05-02 22:09:55,266 - INFO - vconnector-cli[11029]: Adding/updating vSphere Agent vcenter.XXXXX in database
                    I will stop here and wait for some comments... ;-)

                    Comment

                    • noeand
                      Junior Member
                      • Apr 2014
                      • 24

                      #11
                      Installing in a Python virtualenv should be as easy as doing this:

                      Code:
                      $ virtualenv vpoller-venv
                      $ source vpoller-venv/bin/activate
                      $ git clone https://github.com/dnaeon/py-vpoller.git
                      $ cd py-vpoller
                      $ python setup.py install
                      This should get you all the dependencies installed as well.

                      As for this error from the ZeroMQ build:

                      Code:
                      ************************************************
                      Configure: Autodetecting ZMQ settings...
                          Custom ZMQ dir:
                      build/temp.linux-x86_64-2.7/scratch/tmp/easy_install-uslJFN/pyzmq-14.2.0/temp/timer_createHP4171.o: In function `main':
                      timer_createHP4171.c:(.text+0x15): undefined reference to `timer_create'
                      collect2: ld gab 1 als Ende-Status zurÃŒck
                          ZMQ version detected: 4.0.5
                      ************************************************
                      You should not worry about it much, because it will successfully build and install the module.

                      Here's a quick way to confirm that you've successfully installed ZeroMQ:

                      Code:
                      $ python
                      >>> import zmq
                      >>> print zmq.zmq_version()

                      Comment

                      • noeand
                        Junior Member
                        • Apr 2014
                        • 24

                        #12
                        Originally posted by steveboyson
                        Anyway, having paths different from the examples is not the right idea so far, at least this is what I'am suspceting.

                        Code:
                        vconnector-cli -d /usr/local/vPoller/vconnector/vconnector.db -o /usr/local/vPoller/log/vconnector.log -H vcenter.XXXXX -U XXXXX -P XXXXX add
                        gives:
                        Code:
                        Traceback (most recent call last):
                          File "/usr/local/vPoller/bin/vconnector-cli", line 5, in <module>
                            pkg_resources.run_script('py-vpoller==0.2.0', 'vconnector-cli')
                          File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 499, in run_script
                            self.require(requires)[0].run_script(script_name, ns)
                          File "/usr/local/vPoller/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 1235, in run_script
                            execfile(script_filename, namespace, namespace)
                          File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 182, in <module>
                            main()
                          File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 167, in main
                            host=args['--host'],
                          File "/usr/local/vPoller/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/EGG-INFO/scripts/vconnector-cli", line 65, in add_update_agent
                            host=host,
                          File "/usr/local/vPoller/local/lib/python2.7/site-packages/py_vpoller-0.2.0-py2.7.egg/vpoller/connector.py", line 355, in add_update_agent
                            self.cursor.execute('INSERT OR REPLACE INTO hosts VALUES (?,?,?,?)', (host, user, pwd, enabled))
                        sqlite3.OperationalError: no such table: hosts
                        The SQLite file gets created, though, but is empty (0 bytes).
                        The log say that everything is ok:
                        Code:
                        cat /usr/local/vPoller/log/vconnector.log
                        2014-05-02 22:09:55,266 - INFO - vconnector-cli[11029]: Adding/updating vSphere Agent vcenter.XXXXX in database
                        I will stop here and wait for some comments... ;-)
                        Okay, I see what the issue is - the database file is not initialized.

                        Please, execute this command before trying to add a vCenter first:

                        Code:
                        $ vconnector-cli init
                        Or in your case it would be:

                        Code:
                        $ vconnector-cli -d /usr/local/vPoller/vconnector/vconnector.db -o /usr/local/vPoller/log/vconnector.log init
                        After that you should be able to add your vCenter. I will update the documentation tomorrow, because it seems this one is not mentioned there

                        I guess it's time for me to get these Debian & RHEL packages ready in order to ease installation and testing

                        Comment

                        • steveboyson
                          Senior Member
                          • Jul 2013
                          • 582

                          #13
                          Ok, with "vconnector-client init" it's working and the SQlite db file gets installed and populated with the requested entries.

                          Time to move on ;-)

                          Thanks for your help!

                          P.S. Python's virtualenv was not the problem as mentioned before ...

                          Comment

                          • steveboyson
                            Senior Member
                            • Jul 2013
                            • 582

                            #14
                            vpoller-proxy starts fine and creates listeners as defined in the conf file. vpoller-worker does not start and emits an "address already in use" error.

                            A short test gave that the webmin service was active on :10000. Stopping that helped ;-) and vpoller-worker started.

                            I did some quick tests and the basic "-m ..." commands do work well. What I do not get is the status output, probably again due to path/logfile/pid issues.

                            i.e.
                            Code:
                            vpoller-worker -p /usr/local/vPoller/run/vpoller-worker.pid -f /usr/local/etc/vpoller.conf -o /usr/local/vPoller/log/vpoller-worker.log -e tcp://localhost:10000 status
                            fails.

                            Regarding that, it would be nice if the programs would respect some environment variables, let's say
                            - VPOLLER_PID=/my/prefered/path/$(basename $PROG).pid
                            - VPOLLER_LOG=/my/prefered/path/$(basename $PROG).log
                            - VPOLLER_CONF=/my/preferred/path/vpoller.conf
                            and so on.

                            For the construct "$(basename $PROG)" it would insert vpoller-{proxy|worker|client} as needed, of course.

                            I have some problems on reprobe the shown examples on http://unix-heaven.org/node/103 (it is presumably "host.get" and not "host.poll" as shown in the screens). A better documentation for the methods and their optional or required parameters would be really nice.

                            Also, some of the outputs are not really usefull or throw errors:
                            Code:
                            vpoller-client -o /usr/local/vPoller/log/vpoller-client.log -V vcenter -m host.get -n esx1 -p runtime.bootTime
                            {
                                "msg": "Cannot serialize result: datetime.datetime(2014, 4, 23, 12, 11, 59, 769750, tzinfo=<pyVmomi.Iso8601.TZInfo object at 0x17bcdd0>) is not JSON serializable",
                                "success": 1
                            }
                            What I see is the almost lightning speed of vPoller and it's friends. Usually, login to a vSphere environment takes up to 5 seconds. As vpoller-worker is already logged in the results come back in a very amazing speed!

                            So, as a first conclusion: it looks quite promising regarding speed, but lacks some documentation and most likely needs some more work.
                            I would love to see vPoller becoming a mature way of vSphere monitoring and will definitely stay on track.

                            Thank you for your very good work!

                            Comment

                            • noeand
                              Junior Member
                              • Apr 2014
                              • 24

                              #15
                              vpoller-proxy starts fine and creates listeners as defined in the conf file. vpoller-worker does not start and emits an "address already in use" error.

                              A short test gave that the webmin service was active on :10000. Stopping that helped ;-) and vpoller-worker started.

                              I did some quick tests and the basic "-m ..." commands do work well. What I do not get is the status output, probably again due to path/logfile/pid issues.

                              i.e.
                              Code:
                              vpoller-worker -p /usr/local/vPoller/run/vpoller-worker.pid -f /usr/local/etc/vpoller.conf -o /usr/local/vPoller/log/vpoller-worker.log -e tcp://localhost:10000 status
                              fails.
                              Until the Debian and RHEL packages for vPoller are ready I thought it would be good to have a bootstrap script which would install and configure everything for you nice and easy.

                              You can find the the bootstrap script at the link below which you can use as a quick way to install and configure everything in one shot:

                              * https://github.com/dnaeon/py-vpoller...rap-vpoller.sh
                              * https://github.com/dnaeon/py-vpoller#installation

                              As for the "status" vPoller Worker requests -- there was a leftover from the refactoring, which is now fixed. You should be able to get the "status" information now

                              Regarding that, it would be nice if the programs would respect some environment variables, let's say
                              - VPOLLER_PID=/my/prefered/path/$(basename $PROG).pid
                              - VPOLLER_LOG=/my/prefered/path/$(basename $PROG).log
                              - VPOLLER_CONF=/my/preferred/path/vpoller.conf
                              and so on.

                              For the construct "$(basename $PROG)" it would insert vpoller-{proxy|worker|client} as needed, of course.
                              Yes, that would be nice to have. Could you please submit a bug issue at the issue tracker and I'll take care of it?

                              Comment

                              Working...