iowait, Is this normal

[FABinNC]

I am seeing a consistent 5% CPU iowait on my Zabbix server. Although this isn't terribly high, I am curious if this is normal or if there is some tuning I should be aware of. This is a new installation on 2.0.6 with only 22 hosts (932 items). I am planning much more, but I want to understand this before moving forward. Thanks.

How much iowait is typical?

[nilie]

Being normal or not depends on how your server is behaving. If your server is reporting 5% iowait doing what it is supposed to do and under a reasonable load there is no reason to worry yet, however if you have that same 5% when the server is doing almost nothing then you might have to take a look at it right away.
If you are saying your server monitors about 1000 items then you should be prepared to review your hardware pretty soon. Iowait could be an indication that the disk channel of your server can't cope with the volume of data transferred to disk. Although the number of items is not that high, maybe the polling frequency is, so the server has to write performance stats very often.
Basically if the situation becomes intolerable, you have two approaches. The first is to upgrade hardware (faster disks) and the second is to ease the load on the disk channel by reducing the volume of data written (retaining only pertinent items and reducing history) or increasing the amount of time it takes to write the same data (increasing the polling interval).
Just to have an idea, what hardware are you using for storage (local, network, RAID level, disk rpm etc.) ?

[SupportGuy]

Sizing info ?

In complement to the previous post, a littel bit more info can help :
- server characteristics, ie, proc nb/freq., memory (RAM and swap), storage type.
- which database
- datafile size

[FABinNC]

I really don't have a lot of information on the hardware characteristics. It is running Fedora 17 under VMWare on an ESX box. The backend is MySQL. I don't have the version handy but suffice it to say it is fairly current.

The overall tone of these replies indicates this may be hardware related and not a normal behavior of the Zabbix framework. Should I be talking to my hardware guys?

[tchjts1]

What is your memory and swap usage looking like on your DB server? Can you give us screenshots of those for, say, a 1 month period like my below screenshots?

I have a similar setup as yours, except it is RHEL on VM under ESX. I also had high IO wait time. Part of the solution was to modify the "swappiness" setting under RHEL. Fedora also has this, I believe.

See this post regarding what I did to resolve my issue:

https://www.zabbix.com/forum/showthread.php?t=38575

.

[FABinNC]

Thanks for that tip. Swappiness was set at the default of 60. I set it to zero. Although I doubt it will make any difference as my free swap has not dipped below 98% for the three weeks this server has been up and running.

I should note I do see periods of time (like right now) where the iowait goes to near zero for awhile. Then it jumps up to 5 to 10% and hangs there fairly consistently for a long period of time. I cannot correlate these events with anything specific just yet...

[tchjts1]

One other thing you should look at are the graphs of the Zabbix internal processes and see what they are looking like. Maybe there are some tuning opportunities you could find from the information they provide.

See this post regarding those graphs:

https://www.zabbix.com/forum/showpost.php?p=133808&postcount=4