Disk IO skyrockets with 1.8.3 (And maybe Ubuntu 10.04 and MySQL 5.1.41)

[QwErTy_LoGiC]

Hey all,

I did a completely new install for Zabbix 1.8.3 and restored the 1.8.2 database. With 1.8.2 the avg. CPU load was around 0.80 and avg. utilization was around 3-4 percent.

With 1.8.3 I was pleasantly surprised to see the load average drop to 0.22 and cpu avg. util. drop around 2%.

However, I nearly spat my morning coffee when I saw disk IOs... The disk IO were a solid, neverchanging 30/iops to a whopping 298/iops average...

Now, mind you I also upgraded the os from Ubuntu 9.10 to 10.04 and MySQL 5.1.37 to 5.1.41.

Anyone seeing a jump in the disk IO?

I can't, for lack of better word, pin this on zabbix since too many variables were changed at once. But the physical server and disk layout is exactly the same, the my.cnf for MySQL is still the same also.

Cheers!

[QwErTy_LoGiC]

Probably the same issue...

http://www.zabbix.com/forum/showthread.php?t=18668

[richlv]

do you have a statistic for your setup of queries against the db ?
which process is generating the most disk access for you ?

[QwErTy_LoGiC]

Stats

Zabbix required performance is at ~70

In phpadmin I was at 25k requests per hour with 1.8.2, with 1.8.3 I'm at 12k requests per hour.

With iotop, mysql dominates the IO, almost entirely in writes.

[QwErTy_LoGiC]

Whoops

Sorry, thats 25k requests per minute with 1.8.2 and 12k requests per minute with 1.8.3...

[walterheck]

Can you see if the process that is generating that disk i/o is the zabbix server or mysql?

[QwErTy_LoGiC]

Ok...My Bad... I think...

Ok guys, I think I nailed it... I'll let 24hours run by before I celebrate.

I think I messed up with my restored DB... It restored fine but I forgot to run the mysql update on it...

I stopped everything, let MySQL calm down, ran the mysql_update started everything back up and I'm back to my 30iops... at least for now.

I'll let you guys know tomorrow if everything is stable.

[Alexei]

Let Zabbix run for at least a couple of hours before judging about its performance. Zabbix keeps lots of data in memory, some time and resources are required to populate the memory cache.

[QwErTy_LoGiC]

Survey says!???

Allright, we're 12 hours past the the mysql_upgrade and everything is rock solid...

That is, plus the CPU Utilization & Load performance gain! Nice to see my server purring again.

Thanks for all the input!

Cheers