Web Check delays and frequency

[bignose]

Hi,

I have about 22 "one step" web checks. They each check a web app once every 10 seconds. If the return code is anything except 200, then it fires a trigger, which I action.

Between 23:15 and 23:28 on Saturday night I got about a bazillion actions to my phone. So I've been digging thru the history_uint table to get some answers and I can see that around the times above, web checks were not happening every 10 seconds. They were happening more like every 2 to 3 minutes.

What kinds of things can cause such a slow down?

I have StartHTTPPollers=3 but I'm not sure if this is too low or not, or if that is the right settings.

I'm tying to do some co-relation now with other data to see if something else was slowing down the machine at that time.

[bignose]

I should mention here are the errors I'm seeing in my server logs:

7100:20120714:232346.407 Web scenario step [xxxx:Visit site] error: error doing curl_easy_perform: Timeout was reached
7102:20120714:232347.227 Web scenario step [yyyyyy:Visit site] error: error doing curl_easy_perform: SSL connect error

The monitors work most of the time. Occasionally they fail tho with these errors.

Also, I looked at my queue monitor and around the times I listed, there were over 1000 items sitting in the queue for up to 15 minutes.

So it sounds like my problem was just extremely slow IO for that 15 minutes.