Best design for custom agent with 1000+ items

[dlethe]

I've got an app that does a lot of monitoring and tuning for disk/RAID related metrics. The app maintains a large number of database (SQLITE) tables with varying internal polling intervals, and it is important to minimize the overhead of obtaining some of the data.

Having said that, I want to create a mechanism to query everything, and interact with the app's internal database that is reasonably efficient, if user wants to get a few hundred items (or more) from my application's internal sqlite database at every polling interval.

Everything is in a sqlite database, but it is just unworkable to call a sqlite3 command-line hundreds of times per polling interval. Is there an elegant, easy way to accomplish this?

If I take the zabbix_agent program, which is threaded and add hooks into the database then this looks like the best way, but highest degree of difficulty.

Can zabbix_get & zabbix_sender be used efficiently to minimize overhead?

[Alexei]

zabbix_sender is highly efficient when it comes to sending thousands of values. It may read data from a file and send everything to the server using just one TCP connection. A combination of a custom script (called from a crontab?) with zabbix_sender is a good start.

[dlethe]

So then create a daemon that invokes zabbix_sender and collects a large amount of data, say 50 - 100 fields per target device at a time, all in one pop, and then let the zabbix server parse all the information and shove it into the server database.

This will certainly have the desired effect of low overhead on the computer running the agent, but not sure if the data would be usable by all the built-ins w/o a lot of individual calls to zabbix_sender. Example:

For any moment in time, my software records info below (this is only part but gives you an idea), and I want to measure efficiency of file system and see if fragmentation, or file system block size , software RAID chunks, compression, whatever needs to be changed. So you measure bytes read/written as returned by the built-in metrics at the kernel, but compare to what the disks actually do, as this is the only way to truly tune (or see if load balancing is working right, or you have bus saturation, RAID settings weak, or you have hardware issues affecting I/O etc ...)

So precision in terms of timestamps are vital, and I need to tie all of this together with the millisecond timestamp..
Logical blocks sent to initiators: 279976695
Logical blocks received from initiators: 911418518
Logical blocks read from cache, sent to initiators: 3204995
Number of read and write commands <= current segment size: 2555856
Number of read and write commands > current segment size: 7609
Write errors corrected with possible delays: 0
Total Write errors: 0
Write errors corrected: 0
Times correction algorithm processed (on Writes): 0
Bytes processed (on Writes): 470068374016
Unrecovered errors (on Writes): 0
Read errors corrected without substantial delay: 1076715412
Read errors corrected with possible delays: 0
Total Read errors: 0
Read errors corrected: 1076715412
Times correction algorithm processed (on Reads): 0
Bytes processed (on Reads): 143348067840
Unrecovered errors (on Reads): 0
Total Non-medium errors: 34
SAS Phy #0 (50-00-C5-00-34-25-B7-39) - Invalid dwords: 255
SAS Phy #0 (50-00-C5-00-34-25-B7-39) - Running disparity errors: 255
SAS Phy #0 (50-00-C5-00-34-25-B7-39) - Loss of dword syncs: 2
SAS Phy #0 (50-00-C5-00-34-25-B7-39) - Reset problems: 0
==========
Each of these would be in the database as individual values per device with a timestamp. I can certainly put this all in a single database table per device.

If I am looking at load balancing or helping user figure out if they could use another HBA because aggregate bytes transferred on all the disks is near the limit of the physical interface, then I want to be able to get it in database with timestamps of when the data was collected from the disk(s), not obtained from the agent.

What would be a good way to get it in the database so everything has the correct and same timestamp for all of the above information, or maybe 2-3x that much info if they really want to look deeper?

Thanks

[eskytthe]

You can specify local timestamps in the zabbix_sender input file.
http://www.zabbix.com/documentation/.../zabbix_sender.

Another thing: I think you can increase the number of agents process running at your hosts to speed up / make parallel transfer of data, (if I remember correct). You may also need to increase zabbix server processes.

Br
Erik