Remote Web Monitoring

[anrstone]

I have been working on creating some code to give Site Confidence (see: http://www.siteconfidence.com/home.aspx ) style reporting within Zabbix. To do this I have written some code to use Curl and the Tidy libraries to get information about page size / number of objects on the page / dowloand times / lookup times etc. etc. I'm currently working up journey's and the like but that's a few weeks off.

The key driver for this was to be able to have the web-monitoring well away from the Zabbix Server at various remote locations and to be able to record significantly more information about the page performance than is currently available within the Zabbix Web functionality.

If anyone's interested in the code please leave a post and I can send it over and if anyone's keen to develop this into a better more integrated solution (e.g. by extending the existing agent model I would be happy to support that etc.)

Nick

[richlv]

is that information somehow pushed into zabbix, or is zabbix extended to gather it, or... ?

[anrstone]

So at the moment I'm pushing it into zabbix - the plan (the bit I'm working on next) is to somehow extend the agent model to bolt this in and then see if the interface can be changed to generate the necessary config options


I still need to do work on the capture part as I really need to look at how pages will actually render in a real browser. So for instance if a CSS file specifies a background image for a feature - clearly it's not appropriate to drag all of those down but it may be the case that some are required to generate a specific page and so to get real stats some page interpretation is required.

[richlv]

depending on what your goal is, best approach would probably to implement this as part of zabbix web monitoring (which eventually could be run through proxies)

[anrstone]

Not really what I wanted to do - but might be OK. I'm not sure I want to have to run a proxy to perform this test but I'll give that some thought. I guess I would like a bit more flexibility to be able to run this as an agent which gives me more platform choice for the remote testing location.

That all said I've not looked heavily at Proxies yet so I'll finish off the testing part to gather the data I need and then see what the best step is from there.

There are a few things with using Zabbix for this anyway that I probably need to think about as Ideally I would like to produce more comprehensive graphing of test results than I believe is possible in 1.8 e.g. for each test I ideally want to be able to produce a graph similar to this:

[richlv]

ooooh. now that's detailed. making this easy to get with zabbix would be exciting. i'll be following your progress

[danrog]

I'd be interested in something like this too

[unknown_man]

Not really what I wanted to do - but might be OK. I'm not sure I want to have to run a proxy to perform this test but I'll give that some thought. I guess I would like a bit more flexibility to be able to run this as an agent which gives me more platform choice for the remote testing location.

That all said I've not looked heavily at Proxies yet so I'll finish off the testing part to gather the data I need and then see what the best step is from there.

There are a few things with using Zabbix for this anyway that I probably need to think about as Ideally I would like to produce more comprehensive graphing of test results than I believe is possible in 1.8 e.g. for each test I ideally want to be able to produce a graph similar to this:

[ATTACH]2497[/ATTACH]

could you show us how you did to get this graph? thanks

[anrstone]

So I've got to the point where I have the data (more or less) the bit I'm struggling with is how to get it all in to Zabbix in a sensible way. I can pass back detailed summary information which show's the same detail at the page level and I can also pass back an ascii style list showing the data but I kind of think (and I'd really appreciate people's views) that it would be useful if you could pass back a list as a single record and then use that as the basis for some kind of graph within Zabbix... It would mean having a new option(s) within the Zabbix UI to display a single result as graph / chart.

With regard to how I got the example picture - i'm afraid I screen grabbed something from another system we have here but I doubt it would be hard to create to be honest - the trick is having the data in the first place.

[richlv]

i'd suggest pushing each datapoint into individual item. that would fit zabbix model best.
then on top of item data new reports could be added (maybe by using bar reports as a basis - or not)

[anrstone]

I did think of doing that - the only problem with it is that we monitor some 80 sites for both HTTP / HTTPS at page and journey level and at individual server and load balancer level. This means we would be getting huge amounts of results which are not necessarily related to each other.

So I guess the issue will be grouping the data so that we can do this. By which I mean each test may produce this level of data and as these will run evry 5/10 mins if not more frequently we will (a) have a lot of data and (b) have a lot of data that we will struggle to separate out if they are sent as you suggest.

Now a way around this is to have the concept of a test number (or similar) so if Zabbix (and I haven't looked at this or noticed it anywhere) receives a group of data points then we could assign an extra parameter to indicate that they are related in some way - is this possible?

e.g. we would send up a result set like

serverX, Text1, ParamA, value1
serverX, Text1, ParamB, value2
serverX, Text1, ParamC, value3
serverX, Text1, ParamD, value4
serverX, Text1, ParamE, value5
serverX, Text1, ParamF, value6

again this doesn't quite do what's described in the graph as in fact each bar is made up of 4 values so there is more complexity than could be covered with my simple example above - what do you think?

[richlv]

for item "grouping" you can use zabbix hosts, or on a host level - applications.
you would still be pushing data in one value type per one item, but you can get it back for specific host or specific application on a specific host.
note that applications are just that - a way to group items

[anrstone]

Interestingly that's kind of what I'm doing now - I still have the issue that for each "application" I get many sets of results - so the bit I haven't solved is how to get around that issue in order to group one set of results together to produce this type of graph....

[richlv]

maybe encode that additional information in item names ? then you can grab data by that easily

[anrstone]

Yes - that's probably a good approach. I was kind of doing that by making the item names of the form X.Y.Z. which at least allows me to capture the N values per result but hasn't quite solved the problem of splitting up each test result set in the web interface - not sure how to do that yet...