Extend loadable modules

[tatapoum]

Hi all,

I was trying to find some information on the loadable modules that appeared with Zabbix 2.2, and I came accross this proof of concept :

https://github.com/BlueSkyDetector/mruby_module_for_zabbix_agent

This is interesting because to my mind, the loadable modules is a great idea when you need performance, but it's way too complicated for most people to code in C. This little piece of code shows how to extend zabbix agents or servers with a ruby interpreter.
So you can create a bunch of monitoring scripts in ruby instead of C. These scripts won't need a fork() to run so I think you can achieve very good performance.

This approach could also be used to run python or perl scripts for example.

[kloczek]

Hi all,

I was trying to find some information on the loadable modules that appeared with Zabbix 2.2, and I came accross this proof of concept :

https://github.com/BlueSkyDetector/mruby_module_for_zabbix_agent

This is interesting because to my mind, the loadable modules is a great idea when you need performance, but it's way too complicated for most people to code in C. This little piece of code shows how to extend zabbix agents or servers with a ruby interpreter.
So you can create a bunch of monitoring scripts in ruby instead of C. These scripts won't need a fork() to run so I think you can achieve very good performance.

This approach could also be used to run python or perl scripts for example.

Monitoring probes should be lightweight as possible.
Using ruby to implement such probes (even implemented as loadable interface) IMO is completely orthogonal to such target.
IMO it is less POC but more proof that always is possible to implement something using WrongWay(tm)

[tatapoum]

I don't agree.
ruby or python for example offer a large library of modules. We extend Zabbix with a few external scripts because it doesn't really support some modern or complex protocols (WBEM, CIM-S, WMI, APIs to Xen or HyperV, etc). There are many use cases where an agent is not possible or desirable (network/SAN devices, appliances, etc). And the server-to-agent security model is very light with Zabbix (no authentication, no encryption, etc). In a regulated environment, this is important. All the above protocols provide authentication and encryption if necessary.

However, the drawback is the high load on the server due to all these processes. So integrating a simple language runtime into the server or agent can be useful.

Nagios does this with the embedded perl interpreter for example.

[vulogov]

Here is a module, if you prefer Python over Ruby

https://github.com/vulogov/pyzabbix-agent

[kloczek]

Hi all,

I was trying to find some information on the loadable modules that appeared with Zabbix 2.2, and I came accross this proof of concept :

https://github.com/BlueSkyDetector/mruby_module_for_zabbix_agent

This is interesting because to my mind, the loadable modules is a great idea when you need performance, but it's way too complicated for most people to code in C. This little piece of code shows how to extend zabbix agents or servers with a ruby interpreter.
So you can create a bunch of monitoring scripts in ruby instead of C. These scripts won't need a fork() to run so I think you can achieve very good performance.

This approach could also be used to run python or perl scripts for example.

Using something like ruby in something which by definition should be lightweight is IMO big contradiction. It sounds more like dry water

IMO it would really better to have support for something like lua.

[kloczek]

I don't agree.
ruby or python for example offer a large library of modules. We extend Zabbix with a few external scripts because it doesn't really support some modern or complex protocols (WBEM, CIM-S, WMI, APIs to Xen or HyperV, etc).

You may not agree with what I wrote but so far you've not mentioned anything which may undermine what I wrote.

Using complex protocols in monitoring may be another contradiction.
If something is using complicated protocol you don't need to follow this path. Really it is only your choice .. nothing more.
KISS principle is always valid.
Without this you will have something which I call kind of quantum effect of monitoring where observation is affecting observed object.

In monitoring and testing theory you have yet another effect which is sometimes called "death by thousands cuts". Single small cut is not dangerous but thousands may kill even elephant. This is why on maintaining some code in long term keeping almost the same speed of all building blocks is so important. This is why some companies maintaining in long term some critical code are looking co close on speed of the code and treats speed degradation as bug which needs to be investigated and fixed.

Present time hardware is constantly faster and faster. Problem is with not peak speed of the CPUs but with all bottlenecks sitting for example in L1/L2 CPU caches or overhead of data synchronization between core caches.
CPUs are constantly faster but these two things are still not made from rubber.
Bigger binary of the agent enlarged by memory used by complicated embedded scripting language like rubby will affect at some point what you are monitoring. There is no miracles ..

There are many use cases where an agent is not possible or desirable (network/SAN devices, appliances, etc). And the server-to-agent security model is very light with Zabbix (no authentication, no encryption, etc). In a regulated environment, this is important. All the above protocols provide authentication and encryption if necessary.

I don't know single SAN or network device which does not support SNMP.
SNMP is de facto (not de iure) standard. On monitoring high speed devices you have additionally sFlow (at the moment even some switches made for some home of small offices network environments supports sFlow). Maybe it will be not bad idea to have sFlow support in zabbix as well in future.

SNMPv3 provides high level of security.
Is anything wrong with monitoring such devices like yours over SNMPv3?
zabbix in latest versions supports quite well SNMP.
After introducing SNMP bulk queries seems fundament of such monitoring is IMO solid enough.
In SNMPv2 MIB you can find few OIDs providing king of self monitoring of SNMP agent. In these OIDs are exposed in/out SNMP packets/transactions traffic data. After switching to zabbix version supporting SNMP bulk queries I've observed enough drop down of SNMP overhead to not worry longer about using SNMP.

[vulogov]

SNMP isn't cure for all monitoring needs. There are plenty of other things, which you do need to monitor and which doesn't speak SNMP. Custom applications, webserver farms, cluster software and filesystems and so on. Also, I met good amount of evidences that sometimes you can get more data through CLI or Web interface or other non-SNMP means than from classic SNMP.

So, if you are tending just a network infrastructure, good chance that you will be just fine with SNMP. Otherwise, I wouldn't be so definite.

[kloczek]

SNMP isn't cure for all monitoring needs. There are plenty of other things, which you do need to monitor and which doesn't speak SNMP. Custom applications, webserver farms, cluster software and filesystems and so on. Also, I met good amount of evidences that sometimes you can get more data through CLI or Web interface or other non-SNMP means than from classic SNMP.

Again: on all these cases IMO better would be use one protocol like sFlow and add to the zabbix ability to understand this protocol.
Please have look on https://code.google.com/p/mod-sflow/ as an example.
You can find easily more examples allowing using sFlow to monitor something.

For example all of the known for me clustered hood software has ability to talk over SNMP. Probably other software if exist and does not support SNMP is so on early development stage that developers didn't yet implemented proper monitoring interface basing on industry standard protocols (both SNMP and sFlow are defined in RFCs).

You mention about monitoring something over http. As long as something like this is possible to monitor something over http/https probably you can use straight zabbix agent working on some host. Remember that for single host in zabbix monitoring you can define multiple interfaces. For example if you need to monitor host A you can define that zabbix agent (active) items monitoring will done over host B with working on it zabbix agent.

As long as wee are talking about monitoring something over http as an example in attachment you can find one of my simplified a bit templates allowing monitor apache without any backend script collecting monitoring data over http using native apache mod_status abilities.
Only requirement of this template is use zabbix agent >= 2.2.x. It does not need any backend scripts or so on zabbix agent side like you can find in many other publicly available zabbix apache templates.

However now you are changing scope of original subject trying now to redefine it
Now it is not only the case of SAN and/or network devices.
Can you agree that with exactly such devices zabbix provides almost what is needed?

However you not answered on my question about is it something wrong with SNMP as protocol
Using standard net-snmp you can expose any data you want and as long as you have ability to install some software on monitored object/application/device and you can monitor such thing securely. net-snmp allows extend agent side over local mux interface so not all parts of agent side needs to be embedded straight into snmpd. net-snmp has IMO decent documentation about implementing such extensions. Knowing this really sometimes I cannot understand why some people are reinventing the wheel defining even own monitoring protocol (this sentence is not about zabbix .. zabbix here is kind of exceptions because his needs are way beyond what SNMP provides ).

So, if you are tending just a network infrastructure, good chance that you will be just fine with SNMP. Otherwise, I wouldn't be so definite.

I'm not tending to tell anything. It was only my plain replay on your definition of your needs pointing additionally that in your opinion SNMP is not enough in context of secure monitoring

[vulogov]

Man, you really do like SNMP and sFlow. Although, there is nothing particularly evil in those protocols, you can use them for whatever purposes you can find suitable for particular situation, I am not a big fan of "this is the only solution, because this protocol is great" approach. Those protocols been designed with network devices in mind. Which means, they may or may not be suitable for any other task.

But I will not be the one who will try to change you mind over this issue.

[kloczek]

Man, you really do like SNMP and sFlow.

Wrong impression. Read what I wrote not between the lines but straight.
I know only enough to point that some sentences contains false not even facts but opinions. Only this and nothing more.
Please do not try to divert this exchange of comments to some personal area
To be honest with you .. my personal opinion about SNMP is completely opposite. I don't like SNMP because many vendors not uses it properly which makes something so simple like monitoring over SNMP in many cases harder or messier than it should be.
Small example: in SNMPv2 MIB is defined part of subtree where should be exposed physical layer of network interfaces (if device has something like this). For some reasons some vendors are choosing exactly this MIB to expose logical layer and physical layer is exposed in vendor specific MIB(s). Example of such misunderstanding are Big F5 devices.

Although, there is nothing particularly evil in those protocols, you can use them for whatever purposes you can find suitable for particular situation, I am not a big fan of "this is the only solution, because this protocol is great" approach. Those protocols been designed with network devices in mind. Which means, they may or may not be suitable for any other task.

But I will not be the one who will try to change you mind over this issue.

So let me write what is my impression what you wrote
You have develop zabbix loadable module which uses rubby not because something is missing or wrong with zabbix agent, or used by zabbix protocols (like SNMPv3), but probably because you have some libraries (probably even enough well tested) of rubby code allowing to "talk" with some other components which you want/must monitor.
Probably you even had something like JFDIN [1] ..
My impression is that it is not purely technical decision but slightly emotional (probably you even like rubby which I hate). Isn't it?
IMO it is nothing wrong with such conclusion .. if it is correct conclusion
Your application may be perfectly even working for you.
However what I'm trying to tell you is that this approach may have some serious issues and/or limitations and/or impact on using your approach widely.
Just try to stress your monitoring interface to find out kind of limitations of your approach to evaluate are these boundaries will be passed in time period when your application will be used on production or not.
If final conclusion will be negative you can or should even ignore what I wrote and you can stamp your code using ItWorksForMe(tm). Why? Because now you probably have something working and from now you can star something new.

As I wrote personally I don't like or even hate SNMP. Working on monitoring is not even my primary job profile
However when it comes to my job I'm using it without thinking longer than blink of an eye. Why? Only because in most cases ItWorks(tm). It is really so simple ..

[1] http://www.urbandictionary.com/define.php?term=JFDI

[vulogov]

Ruby module is not mine. Mine is a Python module. Just to clarify.

[kloczek]

Ruby module is not mine. Mine is a Python module. Just to clarify.

Mea culpa, mea maxima culpa ..

Have a good day/evening/night

[vulogov]

No problem, I just do not want to take a credit for someone else work.

Also, just to illustrate, that the SNMP/sFlow ain't acceptable in all cases, I bring an example from one of my past projects, where I was contracted to create a monitoring solution for an complex distributed messaging application. The hardware side of the problem didn't presented much of the issues: I used SNMP. Abou the systems, I got data through Zabbix agent, which were been just simpler than to do the same through SNMP. But the real poser was with application. It was built around cluster of MQ servers and all data (including the metrics) flow through that MQ servers. You may say it was a bad design. I say that the way it was. If Zabbix had a loadable modules then, I wouldn't be needed to create complex, XML speaking structure just to interface with an application and get the data I needed. With loadable modules I would be easier, because Python interpreter is reasonably fast when executing the code, but fairly slow during the startup.

Also, since you mention that creating a monitoring solution is not you primary task, I will permit me-self to note, that often, monitoring people brought "onboard" on later stages of the project implementation. Again, this is not good, this is not bad, this is simple fact of life. In dozen my most recent projects, only few has working on design of how to monitor then from earlier stages. Again, that the way the world often are.

And the linguistic choice, often is not "emotional" one. Again, we are limited to what company (or specific project) is agreed to permit. Sometimes it is nothing but ether C or Bash. Sometimes Perl, sometimes Python or Ruby. There are system architects (and architect boards) which imposing such guidelines.

So, it was an interesting discussion. Have a smashing day!

[kloczek]

First: small correction what I wrote. I've pointed on wrong MIB. I wrote:

"Small example: in SNMPv2 MIB is defined part of subtree where should be exposed physical layer of network interfaces"

Of course it is not SNMPv2 MIB but IF MIB.

http://support.ipmonitor.com/mibs/IF-MIB/tree.aspx

Also, just to illustrate, that the SNMP/sFlow ain't acceptable in all cases, I bring an example from one of my past projects, where I was contracted to create a monitoring solution for an complex distributed messaging application. The hardware side of the problem didn't presented much of the issues: I used SNMP. Abou the systems, I got data through Zabbix agent, which were been just simpler than to do the same through SNMP. But the real poser was with application. It was built around cluster of MQ servers and all data (including the metrics) flow through that MQ servers. You may say it was a bad design. I say that the way it was. If Zabbix had a loadable modules then, I wouldn't be needed to create complex, XML speaking structure just to interface with an application and get the data I needed. With loadable modules I would be easier, because Python interpreter is reasonably fast when executing the code, but fairly slow during the startup.

I wrote that (in ideal scenario) it would be good to have zabbix LUA module.
What do you think about mixing it with http://lua-users.org/wiki/LuaXml?

I'm not sure about reasons in your case of using MQ. Could you please drop some hints about why exactly MQ on transport layer? Maybe some significant irregularities in rate of date processed by zabbix or something else?

Also, since you mention that creating a monitoring solution is not you primary task, I will permit me-self to note, that often, monitoring people brought "onboard" on later stages of the project implementation.

I've spend significant time working on monitoring in the past but (with full respect) IMO monitoring it is a little boring subject

I know a lot or enough about monitoring and for example when I've started my current job I've propose that I'll start my cleanups from bottom .. from monitoring infrastructure. In last more than year I've spend a lot of time working with OPS guys not only building/improving monitoring infrastructure but also working as senior guiding or teaching about some general monitoring methodology or advising how monitoring should be evolving.
(I'm writing this only to explain a bit what may be behind some of my opinions or my point of view)

About month ago I've started reviewing last area of our monitoring which is SNMP monitoring. I've not been touching this area in last 6-7 years and after such long time with still solid basic knowledge about SNMP now I have completely different "optics" looking on this subject.
I'm waiting at the moment on formal approve in my company that I can publish few results of my work on SNMP monitoring using zabbix.
I want to publish my kind of SNMP zabbix guideline and my current stash of MIB based templates and few examples of using them on assembly exact device specific monitoring.
I cannot promise anything about exact date because not everything depends on me. Hopefully I'll be able to publish this to the end of this year. Hopefully shortly you will have opportunity to critique result of my work :P

(I've been able to convince few people in my company about such move because we still have significant amount of work to finish in context of SNMP monitoring (and I want to move away from this subject ASAP). It is possible that after publishing what I did other people will be able to help us finish our work quicker or improve what we have now .. everything still in kind of WIP state)

Again, this is not good, this is not bad, this is simple fact of life. In dozen my most recent projects, only few has working on design of how to monitor then from earlier stages. Again, that the way the world often are.

It is not only your problem. Starting my current job I choose start my cleanups from monitoring because I like to see precisely results of my primary work. I see constantly wasted a lot of developers or architects power/time only because people are not introducing monitoring own work on early stages.

And the linguistic choice, often is not "emotional" one.

I'm not native English speaker and I've choose phrase "emotional decision" as opposite to "cold/technical decision". Wrong wording. It does not have even proper meaning in my native language (which is Polish).

Again, we are limited to what company (or specific project) is agreed to permit. Sometimes it is nothing but ether C or Bash. Sometimes Perl, sometimes Python or Ruby. There are system architects (and architect boards) which imposing such guidelines.

So it was kind of JFDI

[vulogov]

I wrote that (in ideal scenario) it would be good to have zabbix LUA module.
What do you think about mixing it with http://lua-users.org/wiki/LuaXml?

Agree on that, but let it be a project for another daring soul. LUA is fine choice for small stuff (In My Humble Opinion), but if it worthy to employ "the language", you better use some more powerful one. Python/Ruby/Perl/CLIPS comes to mind. Me, personally, preferring OO programming for many reasons.

I'm not sure about reasons in your case of using MQ. Could you please drop some hints about why exactly MQ on transport layer? Maybe some significant irregularities in rate of date processed by zabbix or something else?

It wasn't my choice. All app has been built around distributed XML-based MQ. If you wanna "talk" to the app, or subscribe on channels to which app pushes the various metrics, you have to "speak" XML. Since data flow was very irregular (even if it shall be regular in theory), and firing Python interpreter in Zabbix agent through UserParam is expensive, I end up creating Python query scripts, which pulled data from the system and pushes it to memcached and then C applications called by Zabbix agent, which reads the data from memcached. Now, with module, having custom scripts inside an agent it's as easy as 1-2-3 and if I would solve the same problem now, it would be simpler in design.

I've spend significant time working on monitoring in the past but (with full respect) IMO monitoring it is a little boring subject

It is a job on the border of system/network/etc administration and software development. The real challenge is that often you are an SA, developer, tester and tech writer at the same time. I wouldn't call this particularly boring.

I cannot promise anything about exact date because not everything depends on me. Hopefully I'll be able to publish this to the end of this year. Hopefully shortly you will have opportunity to critique result of my work :P

I will be waiting for this with great anticipation.

(I've been able to convince few people in my company about such move because we still have significant amount of work to finish in context of SNMP monitoring (and I want to move away from this subject ASAP). It is possible

I feel you. Spent some time doing HP PSP SNMP templating and data acquisition. I don't wanna touch this again unless I have to. SNMP is boring, custom apps is a challenge and challenge is never boring.

I'm not native English speaker and I've choose phrase "emotional decision"

Same here, although I can understand some Polish, but I can not speak. Therefore, better stick with English, for better and for worst

In fact, "emotional decision" as it worded do not have too much meaning in Russian as well. But in English, it is different story. Sorry that I didn't understand you.