new Zabbix-implementation to improve automation

[daniel.h]

Hello all!

I am currently looking into re-writing a Zabbix implementation monitoring besides others a data warehouse to automate it further.

What we have today
There are Ruby-scripts on the Zabbix server (!) that connect to the database to be monitored and execute queries there. These queries return information about e.g. the consistency of the data (NULL-values, existence of FKs, number of rows inserted in a certain period) and about ETL-jobs executed (which are logged inside the DB). This information is then written - by the same Ruby-script - to files locally on the Zabbix server, forming the data to existing items.

What we want
Well, the whole thing should be automated to a greater extend than today; new tables or ETL-jobs should be added once and then just magically appear all the way in the Web-frontend (excuse me if the names aren't always right, I'm new to Zabbix).
So I'm thinking of having a few main objects to monitor, that would e.g. be a (dimension or fact-) table or an ETL-job. Each of these objects need one generic implementation and - for each new "instance", so each new table or Job - some metadata. Example: if a developer adds a new ETL-job, he adds the metadata needed. From there, all's picked up by the monitoring implementation, items and triggers are created once and the data is gathered and sent regularly.

Questions
1. Makes this solution (metadata + generic script) some sense for you or would you tackle the problem differently?
2. What do you think about the current implementation with scripts on the server connection to the DB? What would be a better architecture? How do you monitor data inside a database?

I hope to hit the right level of detail with these questions; I'm looking for some ideas or opinions and not for you to do my job. Just wanna make sure I'm on the right path, I guess.

[tim.mooney]

I can only speak to part of this, but the automatic discovery that you want is covered in the Low Level Discovery (LLD) documentation , especially the section quite a ways down called "Creating custom LLD rules".

LLD is very powerful, but the tradeoff is that it's also complex. You'll want to carefully read and re-read the entirety of the LLD docs and search for examples of LLD stuff other people have done here and elsewhere on the Internet. You indicate that you're new to Zabbix, so make certain you understand the basics first, as LLD is a somewhat advanced topic.

Everything you're trying to do can definitely be done with Zabbix, but you'll need to do some scripting to get the kind of integration you want. The good news is that some of what you already have may be reusable, if you can get it to generate the necessary format JSON, instead of write to files on the server.