Zabbix 1.4.2 with very high disk IO

[johanpre44]

I have zabbix monitoring about 112 hosts, with a total of 1300 enabled items. most of these items have an interval of more than 60 seconds (maybe 20% of 1300 items is at 30 seconds or less). I went through the posts relating to DB tuning and eventually I changed the DB/Table type to MyISAM, this did make a difference, but my Disk IO is still very high.

My server is a dell pe860 with a Perc5 Raid controller and 2 SAS disks (15k RPM) in a mirrored configuration. I'm running Fedora 7 64 bit on this host (the opne source drivers for the disk subsystem was developed by Dell, but it is community supported only).

My concern is the iostat "await" (or iowait in top) that seems to be running between 40% and 60% most of the time (this changed by about 10% less when I changed the DB from InnoDB to MyISAM, I actually expected much more of a difference). This machine is doing avg of 1.0 MB wrights per second (this number was about 6 MB when it was with InnoDB).

Is it a problem if the iowait is so high? The MB written per second is kind of low (I think ) for a DB applications that adds a couple of hundred records every minute.

Any ideas on this issue? I'm rather new to monitoring the Health of a server in terms of disk IO.

[jeenam]

dur

Try changing the intervals of your checks. Most default to 30 seconds but don't really need to be run that often. An interval of 300 seconds for non-critical services will lower your I/O.

[nelsonab]

Another option may be to put the INNODB file on a different physical disk than the main OS and the INNODB log file. This will reduce your physical wait time as the DB will be only really have to compete against itself for disk access if the main DB file is placed on a different drive and controller.

We have a similar situation on our servers. I've also been somewhat aggressive in tuning down the templates, network checks every 30-60 sec, cpu checks every 30-60 sec, values that don't change often every 30min-1hr and so forth. The biggest change for us happened when we tuned down the default cpu and network settings.

[skullone]

I had high disk IO also at first. MySQL was responsible for all of it, so I tuned InnoDB and got the IO to really low levels (30-40KB writes/sec, or about 10 iops a sec)

Here is the meat of the my.cnf:


[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-locking
key_buffer = 256M
max_allowed_packet = 1M
table_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 2

#innodb_data_home_dir = /usr/local/mysql/data/
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /usr/local/mysql/data/
#innodb_log_arch_dir = /usr/local/mysql/data/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high

innodb_file_per_table
innodb_buffer_pool_size=256M
innodb_log_buffer_size=8M
## Produce informations about wrong informations in tables.
#innodb_log_file_size=1M
innodb_support_xa=0
innodb_flush_log_at_trx_commit=0
innodb_checksums=0
innodb_doublewrite=0

query_cache_type=1
query_cache_size=10M

[shohn]

Works

Using the atop tool, I noted that disk IO seemed to hold at about 90-95% almost constantly and the server was sluggish.

I used the my.cnf noted above and moved the collection of non key metrics to 5 minutes instead of every 30 seconds.


Disk IO went down considerably as a result.

[johanpre44]

Fantastic!

Thanks for the response, using atop pinpointed the problem straight away, and it was not even MySQL at all. The problem was with Syslog (or more specifically all the firewall activity logged). Quickly adjusting the iptables log levels instantly cut about 40% of the IO Wait time (bringing the Disk IO down from 100% to about 20% avg).

[Cynchie]

Hi,

I am experiencing a similar problem: xvda5: Disk read/write request responses are too high (read > 20 ms for 15m or write > 20 ms for 15m) where the disk write is too high. I am running Zabbix v4.4 with PostgreSQL as my database. What do I need to do?

Kindly advise.