Best way to expose application data and metrics?

[TTvince]

Apologies in advance for the newbie question - we're still early days with Zabbix...

I have a network with dozens of small VPN-connected devices, each running several custom apps and a Zabbix Agent. Our custom apps all generate interesting management data that I want to integrate with Zabbix, and I'm trying to understand the most efficient way to do this.

As an example, one of our applications running on the endpoint devices periodically updates an XML status file, something like this:

<Health>Okay at hh:mm:ss.SSS</Health>
<Transactions>nnnnnn</Transactions>
<LastEventTime>YYYYmmdd@HH:MM:SS</LastEventTime>
<Uptime>nnnnn seconds</Uptime>
<Pid>2521</Pid>
<ClientRequests>nnnnn</ClientRequests>
<Errors>nnnn</Errors>
<CPUTtime>nnnn.nnn seconds</CPUTime>
<Cores>4</Cores>
<MemPeak>981.5MB</MemPeak>
<MemInUse>145.90MB</MemInUse>
<MemMax>2731MB</MemMax>
<LastMessage>Initialization Complete</LastMessage>
...

We want to put all of these metrics in Zabbix so that we can monitor the app network wide using Zabbix triggers, graphs and so forth.

We've played around with scripts via "system.run()" - it lets us pluck metrics out of the status file, but it seems like it requires a transaction for every metric on every device, and we end up running an inefficient shell script thousands of times across the network to get the data we want...that hardly seems optimal. We also have a crude command-line program that fetches the information above from any of our devices, so we can run our program on the Zabbix server to fetch metrics like the ones above from any endpoint. It's still the same issue, except we end up running thousands of transactions on the Zabbix Server rather than the agent endpoint. Both seem like poor choices.

Given the sophistication we see elsewhere in Zabbix, we're thinking there must be an efficient way to instrument a custom app in Zabbix, and any hints we could get on the best way to do this are definitely appreciated.

[gofree]

1. utilize zabbix proxy not zabbix server
2. use http agent with preprocessing with throttling rules | https://www.zabbix.com/documentation.../preprocessing and https://www.zabbix.com/documentation...itemtypes/http
3. dont run the check too often ( generally 5 min interval is enough )

[tim.mooney]

It sounds to me like the problem is that you have no efficient way to expose the metrics from these applications. That's more of an issue with the application than anything.

If you read up on dependent items, it might be possible to collect all the items together, instead of separate invocations of a script for each. That might help.

Since you have the Zabbix agent on each client, you could alternately define custom UserParameter= for each custom item and use active checks. That method still means that the script is run many times, but it's not the Zabbix server that is initiating the run. Each agent is doing that individually and pushing the data to the Zabbix server.

When the problem is that it's expensive to collect and parse values from an application, another method that sometimes works is to use an intermediate program to cache the values in another format. For example, run a cron job or scheduled task that collects the data from the application using whatever method is appropriate, and then writes each value to a separate cache file somewhere (assumes you have a writable filesystem), like /var/cache/YOUR_APP/METRIC_1 . Then Zabbix just needs to use something like vfs.file.contents[] on each path to get the metric.

It's also possible to extend the Zabbix agent using loadable modules, probably written in C or C++. If you have programmers on staff that know one of those languages, they may be able to create a more efficient item using e.g. XML-parsing libraries.

[TTvince]

It sounds to me like the problem is that you have no efficient way to expose the metrics from these applications. That's more of an issue with the application than anything.

If you read up on dependent items, it might be possible to collect all the items together, instead of separate invocations of a script for each. That might help.

Since you have the Zabbix agent on each client, you could alternately define custom UserParameter= for each custom item and use active checks. That method still means that the script is run many times, but it's not the Zabbix server that is initiating the run. Each agent is doing that individually and pushing the data to the Zabbix server.

When the problem is that it's expensive to collect and parse values from an application, another method that sometimes works is to use an intermediate program to cache the values in another format. For example, run a cron job or scheduled task that collects the data from the application using whatever method is appropriate, and then writes each value to a separate cache file somewhere (assumes you have a writable filesystem), like /var/cache/YOUR_APP/METRIC_1 . Then Zabbix just needs to use something like vfs.file.contents[] on each path to get the metric.

It's also possible to extend the Zabbix agent using loadable modules, probably written in C or C++. If you have programmers on staff that know one of those languages, they may be able to create a more efficient item using e.g. XML-parsing libraries.

Thanks for the feedback.

I'm not opposed to writing better monitoring into the apps - it's our own application, and making it more manageable is an important goal. We had an idea similar to your "cache values in another format" suggestion that involved exposing our metrics using JMX, SNMP or IPMI, but each of these seems like a lot of work. Perhaps doing a custom Zabbix module would be easier - although it would certainly lock us into Zabbix. I'm a bit surprised there isn't more Zabbix capability in this area - you'd think the need to push a bunch of application metrics into Zabbix would be a common one that should have an elegant solution.

[tim.mooney]

I'm not opposed to writing better monitoring into the apps - it's our own application, and making it more manageable is an important goal. We had an idea similar to your "cache values in another format" suggestion that involved exposing our metrics using JMX, SNMP or IPMI, but each of these seems like a lot of work. Perhaps doing a custom Zabbix module would be easier

My guess is the custom loadable module would be the most difficult. That would be a last resort, if I were in your shoes.

- although it would certainly lock us into Zabbix. I'm a bit surprised there isn't more Zabbix capability in this area - you'd think the need to push a bunch of application metrics into Zabbix would be a common one that should have an elegant solution.

There is a way to push data to Zabbix. Read up on pushing data to Zabbix with trapper items for more info on that. But if the application doesn't have a way to expose ("surface") that data, you're still effectively going to be running something to collect the data from the application and something to submit the data. It all comes back to how efficiently your application makes that data available.

[gofree]

Ifits your application the easiest way would be to expose enpoint with metrics ( maybe youre already doing that ) and scrape it with http agent ( using preprocessing )

Already you have xml which can be parsed with zabbix preprocessing, even better and more usable would be if you expose the data in json ( defacto standard nowadays ). If you dont want to be zabbix locked in you can do it as prometheus exporter - zabbix can read that too. Zabbix module is honestly would be no go way for me as there are better possibilities - especially if you own the app and already have xml status file/endpoint.