consistant alert emails, no log entries

[racooper]

I have a very odd issue that I haven't had any luck tracking down.

There is a service check net.tcp.service[https] that fires every 90 seconds on 40 hosts. Every day at 2:30 AM and 2:30 PM, this check fires a trigger that the service is down for approximately 10 of these hosts. It's the same hosts every day, and they are all located on different phyiscal or virtual servers, in different subnets.

Today, I disabled the trigger. at 2:30 this afternoon, I again received emails, but on only three of the hosts (again, sites on three different servers).

There is nothing in the Trigger logs, Event logs, zabbix_server.log, anywhere that I can find for these events. I don't have any idea why they're continually being sent out.

We are running 2.0.2 installed on CentOS 6.3 from the kodai packages.

I would appreciate it if anyone has any ideas on where I can look to track this down. It's really getting on my and my fellow admin's nerves.

[heaje]

How long do the alerts stick around? Just until the next check? One thing I'd try is to set up a traceroute to port 443 on a couple of the servers to see what that shows (from the zabbix server/proxy used to monitor the servers). Have that run every 10-15 seconds or so just before, during, and after the problem. That would at least help narrow down where the issue could be (network or elsewhere).

This just sounds too darn rhythmic to be an actual Zabbix issue (especially since it is the exact same servers every 12 hours).