I'm trying to figure out the best way to monitor dynamic resources, expending the least effort while getting the best reporting.
My example is something along these lines of:
I'm going to receive snippets of log files from 50 machines.
And these 50 servers monitor around 500 different kinds of fish.
Any given server may respond for 10 different kinds of fish -- sometimes more, sometimes less. These fish will be distributed dynamicly across the fifty servers.
So, for instance, server A might produce snippets like:
trout spawning is at 50
trout dead are 3
trout time is 100
salmon spawning is at 10
salmon dead are 4
salmon restarted
An hour later, server A may no longer be responsible for trout. Server C may be responsible for trout.
So, I can regex against those values and create triggers.
But I need to do the following:
Get statistics per server, and get statistics per fish.
Since I have 500 fish, and this number can increase or shrink, change at will -- where tomorrow we suddenly have cuttlefish in the mix -- can someone think of an easy way to create graphs and values based upon that changing list? Present a graph with the collective salmon that died?
But I also need to be able to alarm when the salmon reaches 150, send an alert with the server associated with that salmon count.
I guess, in short, if you have an item that floats between many different hosts, how do you consolidate that data into a unified view?
My example is something along these lines of:
I'm going to receive snippets of log files from 50 machines.
And these 50 servers monitor around 500 different kinds of fish.
Any given server may respond for 10 different kinds of fish -- sometimes more, sometimes less. These fish will be distributed dynamicly across the fifty servers.
So, for instance, server A might produce snippets like:
trout spawning is at 50
trout dead are 3
trout time is 100
salmon spawning is at 10
salmon dead are 4
salmon restarted
An hour later, server A may no longer be responsible for trout. Server C may be responsible for trout.
So, I can regex against those values and create triggers.
But I need to do the following:
Get statistics per server, and get statistics per fish.
Since I have 500 fish, and this number can increase or shrink, change at will -- where tomorrow we suddenly have cuttlefish in the mix -- can someone think of an easy way to create graphs and values based upon that changing list? Present a graph with the collective salmon that died?
But I also need to be able to alarm when the salmon reaches 150, send an alert with the server associated with that salmon count.
I guess, in short, if you have an item that floats between many different hosts, how do you consolidate that data into a unified view?