Disk I/O is overloaded on Zabbix server

[LePascal]

Hello,

I'm using zabbix 2.0, build fron the appliance on zabbix.com.

The problem : I notice alert "Disk I/O is overloaded on Zabbix server", every hour, during 1 or 2 mn since 2 days


21 Feb 2014 14:02:21 Zabbix server Disk I/O is overloaded on Zabbix server Warning 15m 17s No
-
21 Feb 2014 14:00:21 Zabbix server Disk I/O is overloaded on Zabbix server Warning 2m No
-
21 Feb 2014 13:01:21 Zabbix server Disk I/O is overloaded on Zabbix server OK Warning 59m No
-
21 Feb 2014 13:00:21 Zabbix server Disk I/O is overloaded on Zabbix server PROBLEM Warning 1m No
-
21 Feb 2014 12:02:21 Zabbix server Disk I/O is overloaded on Zabbix server OK Warning 58m No
-
21 Feb 2014 11:59:21 Zabbix server Disk I/O is overloaded on Zabbix server PROBLEM Warning 3m No
-
21 Feb 2014 11:00:21 Zabbix server Disk I/O is overloaded on Zabbix server OK Warning 59m No
-
21 Feb 2014 10:59:21 Zabbix server Disk I/O is overloaded on Zabbix server PROBLEM Warning 1m No
-
21 Feb 2014 10:00:21 Zabbix server Disk I/O is overloaded on Zabbix server OK Warning 59m No
-

What can be done to solve this ?

Thank you

[pc99096]

run it on a more powerful hardware?

[steveboyson]

This is the housekeeper, check it's intervall in zabbix_server.conf.

There are a lot of tips regarding housekeeper settings, starting with completely disabling it, running it in lesser intervalls (which then running longer, of course) up to using partitioning the database.

I assume this is a medium-to-large installation? If not, your hardware really might be not powerful enough.

[LePascal]

Thanks for answering my problem.

The installation size is :

98 hosts
15 items per second
keep history : 90 days

I think it will growth until 300 hosts.

I don't know the hardware configuration because the zabbox server is not on my harware. I just know that the zabbix server is a VM on Vmware. I will ask for the exact configuration today.

What hardware configuration should be done for this installation size ?

[pc99096]

more/faster hard drives should help

[LePascal]

more/faster hard drives should help

It may be true, but i want to understand how i should build my zabbix server regarding the installation size I noticed up. Have you any informations about it ?

Thanks.

[pc99096]

https://www.zabbix.com/documentation/1.8/manual/installation/requirements

[steveboyson]

15 values per second is a very light installation. That should be running an nearly any hardware (or VM host).

How does your zabbix internal performance look like?
CPU load, iostat, how many disk IOPs, load of ESX host, all that stuff.

See these:

https://www.zabbix.com/documentation/2.0/manual/appendix/performance_tuning

https://www.zabbix.com/forum/showthread.php?t=25760

[LePascal]

CPU load : max is 1.2 on the last 7 days

I don't know how to retains the followings datas :
iostat ?
disk IOPs ?

What are thne indicators for these datas ?

I can see that the free swap size is 0

[steveboyson]

iostat is a linux command, usually to be found in the "sysstat" package.
It shows (besides others) the latencies and IOwait counters for a particular disk.

Having an iowait value of more then 80% for a longer period shows a lack of resources in disk IO.

See http://en.wikipedia.org/wiki/Iostat and other docs available in the net.
---
Disk IOPs is a counter which measures the current number of disk input/output operations (per second). You can see these in the vSphere Client, under your datastores. Also they are exportable via Perl or Python vSphere API. Look at the performance metrics of you host and your VMs.

If your ESX hosts show a large amount of IOPs and your disks are to slow then you might run into problems.
Also, the values vor costop and cpuready might be interesting (host level).

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008885

---
"Free swap" does not really matter, as long as you put enough RAM into your VM. 1,5G is a good start.

Pls. see also the other tuning tips. It's hard to do an analysis by not knowing anything about your environment.

[LePascal]

OK

Thanks for the time you spend. I think it will help us to understand the problem.

I will make a return when we find the solution.

many thanks.

[steveboyson]

Actually, the described behaviour is quite common while housekeeper is running.
And it is getting worse with increasing number of hosts (and items, events, ...).

We did define a mechanism called "hysteresis" on these particular triggers:
({TRIGGER.VALUE} = 0 & {zabbix:system.cpu.util[,iowait].avg(5m)}>70) | ({TRIGGER.VALUE} = 1 & {zabbix:system.cpu.util[,iowait].avg(5m)}>40)

This will at least prevent the events from appearing.

https://www.zabbix.com/documentation/2.0/manual/config/triggers/expression#hysteresis

You might want to set the trigger shown above as long your zabbix installation feels good besides the hourly hickups.

[LePascal]

We change the VM parameters :

RAM from 512 kb to 1 Mb
vCPU : from 1 to 2 vCPU

All is fine now.

[aib]

We change the VM parameters :

RAM from 512 kb to 1 Mb
vCPU : from 1 to 2 vCPU

All is fine now.

Cannot believe that anything nowadays can work in One Megabyte(!) of RAM.
Did you mean One Gigabyte, right?

[LePascal]

Oups !

Of course RAM is 1 GB