Extend loadable modules

[kloczek]

Agree on that, but let it be a project for another daring soul. LUA is fine choice for small stuff (In My Humble Opinion), but if it worthy to employ "the language", you better use some more powerful one. Python/Ruby/Perl/CLIPS comes to mind. Me, personally, preferring OO programming for many reasons.

Just entered into google search "zabbix lua" and I found this:

https://support.zabbix.com/browse/ZBXNEXT-1444

!@#$ <censored> .. why I didn't this earlier?
This looks like good sky fold for what I'm working on now

[mushero]

As a company with dozens of custom bash scripts for Zabbix, and another few dozen agent changes pending, plus several SANs and other things that don't support SNMP, I can tell you this is VERY useful.

We were ready to code lots of direct agent changes until the modules came along, so now we can use those, but still takes C programming skills which we have but in limited supply. Using Python makes more sense for a lot of what we do in many cases (for simple Linux stuff we'll probably still use modules).

Many/most of our production systems are 16+ cores, dozens of GB of RAM, etc. so running a python script a few times every 300 seconds is not a serious challenge, given the advantage that I can actually get the monitoring, protocols, logic, and more that I want in real systems. Obviously one has to pay attention to the weight and effects, but valuable in simply enabling many things that were too hard before.

There is a security element in that compiling things into the agent strikes me as safer than running easy-to-change python/ruby code on the server, i.e. I can control it better.

But these are all just tools and options - modules are a huge improvement, and adding an external language like Ruby, Lua, Python, etc. is even better for many real-uses.

[vulogov]

I am working on similar module, to extend not the agent, but the server. The module API are the same, but I think the protocol between C and Python shall be slightly different in the server case. Stay tuned.

[kloczek]

But these are all just tools and options - modules are a huge improvement, and adding an external language like Ruby, Lua, Python, etc. is even better for many real-uses.

Agree with this part.
Still don't understand why something like SAN does not support SNMP and/or "why not SNMP?"

[mushero]

Agree with this part.
Still don't understand why something like SAN does not support SNMP and/or "why not SNMP?"

It's an IBM SAN and believe me, we've tried - the management console will do some SNMP it seems, but the box itself will not - just proprietary stuff, and painful.

But lots of other stuff we monitor like services, JMX, MySQL, HAProxy, custom apps, databases, 3rd party stuff, etc. that need these types of extensions.

[kloczek]

It's an IBM SAN and believe me, we've tried - the management console will do some SNMP it seems, but the box itself will not - just proprietary stuff, and painful.

But lots of other stuff we monitor like services, JMX, MySQL, HAProxy, custom apps, databases, 3rd party stuff, etc. that need these types of extensions.

JMX data can be monitored over SNMP SMUX bridge. Zabbix supports own way of collectiong JMX data.
Mysql does not need anything. I'm monitoring it without any backend scripts. Just mysql command without running additionally awk/grep/perl/etc. If you want I can share my own mysql template showing how to do this.
HA Proxy IIRC has snmp SMUX/AgentX plugin.

If your IBM SAN device exposes SAN stats over SNMP and IBM provides MIB for this device it should be no problem ..

[mushero]

JMX data can be monitored over SNMP SMUX bridge. Zabbix supports own way of collectiong JMX data.
Mysql does not need anything. I'm monitoring it without any backend scripts. Just mysql command without running additionally awk/grep/perl/etc. If you want I can share my own mysql template showing how to do this.
HA Proxy IIRC has snmp SMUX/AgentX plugin.

If your IBM SAN device exposes SAN stats over SNMP and IBM provides MIB for this device it should be no problem ..

Yes, Zabbix has a way to do JMX, but messy and ours is much better with direct access with no daemons or anything - we built it years ago and directly calls JMX from the agent/script.

We also have a great mySQL interface we built that can get any variable, any status, parses error and innodb.status files; far better than any other tool or template we've seen. But happy to look at how you do it to learn more!

Unfortunately, the SAN does not expose anything over SNMP that we can tell and IBM confirmed I think; you have to use their tools and GUI; seems absurd to me as it's new and high-end.

[kloczek]

Yes, Zabbix has a way to do JMX, but messy and ours is much better with direct access with no daemons or anything - we built it years ago and directly calls JMX from the agent/script.

We also have a great mySQL interface we built that can get any variable, any status, parses error and innodb.status files; far better than any other tool or template we've seen. But happy to look at how you do it to learn more!

Unfortunately, the SAN does not expose anything over SNMP that we can tell and IBM confirmed I think; you have to use their tools and GUI; seems absurd to me as it's new and high-end.

In attachment you can find two templates: "Service MySQL" and "Service MySQL slave".
"Service MySQL" uses only system.run[] to execute mysql queries constructed in specific way to output only sampled data so filtering output of the mysql command is not needed.
On monitoring innodb mysql is good to change innodb_stats_on_metadata=0. It gives a little less accurate data but in the same time monitoring overhead and latency of the queries used by these templates are way smaller.

[mushero]

Thanks and interesting.

Note we can't really put our customers production passwords into Zabbix (too dangerous) and never allow/use system.run as too dangerous in our world - we run systems in 100+ data centers globally on untrusted links, and even then we can't expose our customers' systems to allowing Zabbix to run arbitrary commands - but very useful for more common single company LANs.

In our system the key is just something like mysql_realtime(bytes_sent) and we get the value. Far easier than having the huge string in each key, also more easily parsed for our DB integrations that read DB data directly for rule engines and predictive models, etc.

JMX works the same way, and runs on the host via the agent, just jmx(java.heap_max) sort of thing, pulls any JMX key with full security (which we setup when we enable JMX, with host/user/password in Java). Far, far simpler than what Zabbix's JMX gateway does and needs (not to mention security issues in distributed environments).

[kloczek]

Thanks and interesting.

Note we can't really put our customers production passwords into Zabbix (too dangerous) and never allow/use system.run as too dangerous in our world - we run systems in 100+ data centers globally on untrusted links, and even then we can't expose our customers' systems to allowing Zabbix to run arbitrary commands - but very useful for more common single company LANs.

I understand this.
Mysql in exactly this case is terrible here because to monitor slave you must give almost full access to all databases. Using similar approach n case Oracle is possible to give very precise access to only engine metrics.
In this case it problem of the mysql.

In our system the key is just something like mysql_realtime(bytes_sent) and we get the value. Far easier than having the huge string in each key, also more easily parsed for our DB integrations that read DB data directly for rule engines and predictive models, etc.

It is only matter of abstraction of the queries to create shorter keys. I have some work done here. Only problem which I found with current modules API is that it is not possible to access to host/template macros within module extensions code.
My template only shows kind of low level way of accessing to mysql metrics in some very minimalists way reducing to minimum additional steps on processing output of the queries.

[mushero]

I understand this.
Mysql in exactly this case is terrible here because to monitor slave you must give almost full access to all databases. Using similar approach n case Oracle is possible to give very precise access to only engine metrics.
In this case it problem of the mysql.

Not with 5.1 or later as can have very minimal user get all variables and status, so we use that. But we still can't centralize it as too dangerous for the customers and still security bugs in MySQL that make it unsafe to have a valid user. So our user/pw is local to the Zabbix user in .my.cnf on the server.

[kloczek]

Not with 5.1 or later as can have very minimal user get all variables and status, so we use that. But we still can't centralize it as too dangerous for the customers and still security bugs in MySQL that make it unsafe to have a valid user. So our user/pw is local to the Zabbix user in .my.cnf on the server.

Still had no time to experiment with 5.5 and new auth interface which promises use kerberos, so it should be possible to specify that on host X for example zabbix user should be able to login in mysql without using password.

[LynxChaus]

I wrote that (in ideal scenario) it would be good to have zabbix LUA module.

Where ? In server or in agent?

[kloczek]

Where ? In server or in agent?

Probably both.

[LynxChaus]

Probably both.

For what zabbix version?