1.4 Performance - Makes MySQL Very Busy?

[dlindquist]

I've just installed and been playing with 1.4, and my setup looks something like this:

"server"
- MySQL + Zabbix (Server + Agent + PHPFrontend)
- "Template_Linux"

"desktop"
- Zabbix (Agent)
- "Template_Linux"

So, only two hosts, both Template_Linux, one running all the "server"-type software for it.

My problem is that when all this is running, my "server" seems to stay at a constant 0.40 load. Some quick investigation revealed that MySQL is processing something like 35-40 queries per second, apparently coming from Zabbix. (Shutting down zabbix-server dropped the usage off again.)

I'm running the 1.4 release, on Gentoo.

Is this type of MySQL load "normal" for Zabbix? Or is something wrong here?

I've made one "modification" to the install package -- I've replaced "InnoDB" with "MyISAM" in all of the database build scripts (the reason being that the "InnoDB" tables seem to last for about a day before they start reporting "table is full" errors).

Thanks in advance for any help!

[pasqu]

Hi, I have the same problem but I have a clustered Mysql (2+1).
Zabbix point to the load-balacer that do round-robin on two node.

I have seriously problem with performance too.

If anyone have some trick/solution ....

[Alexei]

I know two tricks:

- monitor only what you really need to monitor
- do not keep items' refresh rate too low

In other words, modify standart templates to your needs!

[dlindquist]

That's more of a "solve the symptoms" rather than "solve the problem". ;-)

The question still remains... Is this "normal" for zabbix? Or is this abnormal?

I'm using the standard templates, which are set at 30 second refresh (pretty low, overall), and I'm monitoring only the two machines, with only the Template_Linux items. If this is "normal" for such a setup, I shudder to think how this solution will scale!

[Alexei]

MySQL load is quite normal. I cannot comment on CPU load due to lack of hadrware details.

[Markus]

Hi

I had some serious performance problems with PostgreSQL. Change Zabbix Server logging to level 4 and watch the log file. I had a lot of selects/updates on the 'ids' table (1800 selects and 600 updates per minute while only monitoring a single host) which made the database very, very slow.
Alexei is currently looking into this problem. It would be interesting though if "my' problem is not only related to PostgreSQL but also occurs with MySQL.

Markus

[dlindquist]

MySQL load is quite normal. I cannot comment on CPU load due to lack of hadrware details.

Hi Alexei,

So the constant "35 queries per second" type of load that I am seeing is expected?

I'm running an Athlon Thunderbird 1.2 GHz with 512M RAM.

I was expecting to see "bursts" of activity for a very short period of time around each 30s mark as the updates were done... I wasn't expecting to see such consistent load on the DB for such a small monitoring set.

By simple scaling, this would seem to indicate that I'd likely see constant, 100% load trying to monitor as few as 5 or 6 hosts -- and to monitor the 30 or so machines at my work would load the fastest machine on the market at 100%... Does this sound right?

TIA!

[WeBRainstorm]

Scalability

Hi guys, I'm also having same symptoms with a brand new zabbix 1.4

My tests say that, each host I add, 30 query / secs are added.
This means that, for monioting 50 hosts (as I wish) I should handle a traffic of 1500 query / secs.

I already have a zabbix sercer (1.2) with 40 hosts loaded and it generates 50 query / secs which is affordable for me, but upgrading to 1.4 (which has a LOT of interesting features) will increase my load to approx 1000 query/secs

Hope something will be done to optimize database load

Regards

Edoardo

[sauron]

the reason being that the "InnoDB" tables seem to last for about a day before they start reporting "table is full" errors

Find innodb_data_file_path and remove max:128M

[WeBRainstorm]

Considerations

Hi all, sorry for cross-posting to the users ML, here are my considerations

QPS are linearly increasing with every host I add in 1.4 (with a multiplier of 30); it wasn't so with 1.2

I'm really concerned about this linear relation (QPS = 30 * hosts)
because I really think it deeply impact on zabbix scalability.
(They say zabbix 1.2 was used to monitor 5000 hosts, it means 150.000 QPS with zabbix 1.4... it seems to me a lot of work for MySQL if compared to version 1.2)

I didn't have time to look deeply at zabbix code but I think that this problem should be taken into account and some optimization should be made to lower the query rate.

I think a linear relation linking qps to hosts should be avoided.
Of course this cannot be achieved for INSERTs or UPDATEs that are related to hosts items but probably SELECTs can be optimized to return result sets valids for multiple hosts/items

Zabbix 1.2 was such a wonderful tool and 1.4 features are so interesting that would make zabbix the leader of monitoring tools, but I don't see zabbix developers taking care of the problem...

If it can help... if I look at Queue page I have a table of zeros... don't know why...

I hope this matter will be investigated to make zabbix 1.4 a better tool

Tnx in advance

Regards

[Alexei]

Just curious, did you use the default templates?

I mean, please, be more mathematically precise and do not speculate by definitions like "one host requires 30 QPS".

Processing of a single value normally takes 6-8 SQL queries. MySQL running on a decent hardware is able to process thousands of QPS, which translates to 300-500 new values per second, or 18-30K of value per minute. I leave a task of preparing of a 3D diagram of refresh rate/number of items per host/number of monitored hosts to you

[WeBRainstorm]

Just curious, did you use the default templates?

Yes, just for trying... I still didn't customize my template to take advantage from the new multi template linking

I mean, please, be more mathematically precise and do not speculate by definitions like "one host requires 30 QPS".

Sorry Alexei, I mean with the default template for Linux hosts

Processing of a single value normally takes 6-8 SQL queries. MySQL running on a decent hardware is able to process thousands of QPS, which translates to 300-500 new values per second, or 18-30K of value per minute. I leave a task of preparing of a 3D diagram of refresh rate/number of items per host/number of monitored hosts to you

I promise I'll investigate deeply, but... just curious, why it didn't happen with 1.2 ?

Tnx for answer

Edoardo

[Alexei]

I promise I'll investigate deeply, but... just curious, why it didn't happen with 1.2 ?

There is no difference in a number of executed SQL statements per new value between 1.1.x and 1.4. If a distributed monitoring is enabled, it adds an additional insert statement, nothing more.

[tronite]

I know two tricks:

- monitor only what you really need to monitor
- do not keep items' refresh rate too low

In other words, modify standart templates to your needs!

This sounds quiet reasonable, that one might only run what they need infomation for and leave the rest.

[bbrendon]

I'm not going to be scientific, but my zabbix 1.1.7 system runs between 75-300 qps and averages at about 80-85 qps

Zabbix has 76 hosts, 1127 items active, 574 triggers active.

It also has website databases, spamassassin, and other stuff on the same mysql server.

When the system had 768 MB of RAM, performance was poor. It now has 1.5 GB and works fine.

It will be interesting to see what happens with 1.4.