No Data spam, System performance issue?

[mrogers-9898]

Hi all,

I've been using the same Zabbix server for almost 2 years. I believe it started off as a 1.6.something and now is a 1.8.1.

Mostly it’s been smooth sailing.

95% of my hosts are on remote sites, I use active agent items to get my data. I use a nodata(300) trigger on one of my items to determine if the hosts are online.

Zabbix is running on Debian Etch with MySQL

Here is a status of Zabbix.

Number of hosts (monitored/not monitored/templates) 206 95 / 45 / 66
Number of items (monitored/disabled/not supported) 8650 6225 / 2168 / 257
Number of triggers (enabled/disabled)[true/unknown/false] 3901 3117 / 784 [123 / 24 / 2970]
Required server performance, new values per second 21.5125 -

Occasionally in the past, it would 'go nuts' and we'd get flooded with nodata alerts, the trigger would flap on and off over time. If we waited a short period, 10min or so, it would just sort itself out. Previously this was only a rare occurrence. The more hosts I add however, the more often it is happening. Lately I've had to turn off the trigger completely (we got 1,500 no data alerts from 95 hosts in 30 min...)

I can see that the data does actually appear to come into Zabbix fine. The connection isn’t severed, clients can communicate with the server.

There are no gaps in the monitoring data, it does all appear to get in there eventually.

so; the question is; where do I start looking for the actual cause of this problem? What metrics should I be measuring and comparing?

Could my server disk I/O be too poor for the amount of hosts/items I have? CPU under spec? how could I go about verify that?

Thanks!

[mrogers-9898]

I wonder if I simply have a config issue, the hardware I have should be sufficent for the number of hosts I have?

Xeon(R) CPU X3320 @ 2.50GHz
3 GB RAM
10K SAS (simple volume) hosting zabbix, web, db.

[ghoz]

did you create all the indexes while upgrading your zabbix installation ?
I think i remember seeing an index creation requiered when updating...

[mrogers-9898]

I do remember running the update scripts on the database during the 1.6 > 1.8 upgrade.

I didnt have any problems at all with the upgrade, it went really smooth.

[mrogers-9898]

When I get my "no data" spam, my queues are going nuts.

When the counter is at the bottom of the peaks, those items queued seem legit (as they're hosts that cannot be reached).

the top of the peak, the queue is filled with items from all my hosts.

How could I go about seeing what is causing items to get queued up like this?

[mrogers-9898]

fixed.

While it seems my system has enough cpu/ram for my items/hosts my MySQL database's memory usage was not configured properly, thus it was only using a very small percentage of the available ram.

Resolving this sorted out my queues and flappy no data triggers.