Scaling the web front end

[jcesario]

Im just curious, how do people deal with the slowness of the web front end in larger deployments.

For example, we monitor close to 16k unique items (these items are not able to be broken down or partitioned even more than they already are)

A typical host for us has about 1k of items associated with it.

What ive found is that doing even trivial tasks in zabbix becomes unbearable. 5 or 6 minutes just to access an items page.

We are already using PHP in FastCGI mode on Lighttpd with APC.

So just curious, how are others scaling their web interface?

[untergeek]

What's your frontend hardware? What's the DB on the backend?

[jcesario]

DB:
MySQL 5.0 running the InnoDB plugin
8 core i7 L5520 (16 cpus)
48GB memory (36GB of this allocated to the innodb buffer pool)
32 x 15K drives in a RAID-10 array

Frontend
8 core i7 L5520 (16 cpus)
24GB memory
Lighttpd running PHP 5.2 with FastCGI
Xcache opcode cacher.

[untergeek]

Whoa... That's some serious hardware!

I guess you're hitting a rendering wall. PHP can query pretty fast, but if you're trying to put over 1K items per server, that's a large number. That's up to 20 pages of information in the configuration pages at the default 50 lines per page!

My recommendation would be to slim it down, make "sub" hosts or something, e.g. run separate instances of zabbix_agent on different ports so you can offload some of that immense number of items per server.

[jcesario]

Yup, pretty much already did all that. separate zabbix_agent though we havent done.

[jcesario]

Ive tried partitioning the items into the smallest "host groups" possible. it just doesnt work anymore.

[jcesario]

btw, the agent and poller are absolutely stress free. The DB is nearly idle as well.

The only part choking is the PHP frontend, in fact ive started resorting to try to hack together some sprocs to manipulate everything i need right on the DB.

[untergeek]

Hmmm. How many connections does the php script open? As many as it wants? As many as it can? Only a few?

We've noticed some improvement with simultaneous viewing of highly detailed screens by increasing the OCI concurrent connections in php.ini (we're on oracle). But, I wonder if the script could be optimized to run these sorts of queries in parallel. It would make situations like yours (and ours) run much faster, I would think.

[jcesario]

Were on MySQL. I honestly think whatever the web frontend is doing it probably spends more time building and tearing down connections to the DB than the queries actually take.

The nice thing about the DB for right now is history_uint and items are both still so small we can float our entire data set.

We also mount a nice 8GB tmpfs partition on the mysql tmpdir to ensure that any tmp tables or massive sorts still end up being done in memory.

What I see being the bottleneck is when, for example, You goto Latest Data, youll see DB go (for the most part) idle. However 2 PHP-CGI processes will be pegged to 100% CPU util. Now we still have plenty of CPU leftover to fill but I think this is more of a limitation of PHP than anything else.

I do agree that further parallelization would be a benefit. I also think some pretty severe pagination would be in order as well. I need to crack open the web frontend to confirm but I have a feeling its making some nasty arrays/objects in there during the creation of the page.

[tchjts1]

I would be very interested in this as well. We also have high-end hardware, with DB residing on the SAN and using file-per-table partitioning on MySql.

One particular screen of graphs we have contains about 80 graphs. When you open the screen, go fix your coffee and let it draw. Change the time period you want to view and let it redraw yet again...

[jcesario]

Well it seems like we have some very high end zabbix people in this thread. Im assuming we all have the support contracts, perhaps we should be voicing this to our reps.

Or even better, figuring out how to patch it and committing it back?

[Aly]

What versions of ZABBIX you are using?

[tchjts1]

What versions of ZABBIX you are using?

My current production version is 1.6.6

We are waiting for the release of 1.8.2 to upgrade prod.

[Aly]

In 1.8 paged view is introduced, also very helpful will be new item filter and global search.
You shouldn't get any performance related problems to view your items in 1.8.

BTW nice hardware out there, jcesario

[jcesario]

Thanks :-). Were taking zabbix pretty seriously over here.

Were waiting for 1.8.2 as well. We did a POC of moving to 1.8.1 and it didnt turn out well :-(