Multiply the speed of your zabbix server

[dugrands]

Hi all. Like many I have battled with the speed of zabbix and the number of hosts monitored and trigers increased. In particular, the overview page which is the one we mostly use would take minutes to load and we were desperate for a solution.

After reading many posts here (most of which blamed postgres) and tweaking many sql queries and even getting a more powerful server we made one simple but important observation. MOST ZABBIX SERVER FRONT ENDS ARE SLOW BECAUSE OF THE SIZE OF THE PAGES.

Yes, my overview page in one of my servers was approaching 1 Mb in size and since the server was off site, that would mean a lot of time downloading the page.

OK. The solution! COMPRESSION. I just activated the deflate module in apache and the size (believe it or not) dropped to just 20kb for the once 1mb page.

I hope the zabbix people will recognize this and probably include a note in their manual because surely it'll save many adminstrators the headaches we had to go through.

Happy monitoring!

Dugrands.

[zabbix_zen]

Zabbix webpage rendering speed - Apache deflate module

Alexei,
Any side effect activating this?

[Alexei]

Alexei,
Any side effect activating this?

I think it is absolutely safe.

On a related note, we do lots of changes in 1.8 in order to ensure that all generated HTML content is as small as possible. All pages will be optimized in speed and size. Anyway the advice may benefit all Zabbix versions, good idea indeed.

[nelsonab]

One thing to possibly consider is that compressing the output stream adds a little bit to the processor load. I don't know what kind of impact one might see with it as it's somewhat of a chicken and egg problem. Faster downloads mean faster completion for Apache processes which might mean more spikes in CPU but overall better performance for a loaded system.

I think it would be interesting to see before and after CPU loads for a heavily used frontend server.

[Alexei]

One thing to possibly consider is that compressing the output stream adds a little bit to the processor load.

I believe that generation of 1MB of HTML code by PHP is at least 100x (1000x?) slower operation than compressing the resulting data. I wouldn't expect any visible impact.

[teferi]

Can you please supply the sample configuration you use? I'm trying to use that, but as I'm new to mod_deflate - It would be great if you supply some config =))

[nelsonab]

Holy smokes!!!

I decided to give it a go on my system and wow are things a lot snappier.

I'm running OpenSuSE and I added the following to the vhost entry that all sites sit in:

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/x-js text/css

I've seen other documentation say it should be inside the Location tag but it looks like it can go at a higher level than that in the heirarchy.

[globifrosch]

the debian way:

a2enmod deflate
vi /etc/apache2/mods-available/deflate.conf -> add your text mime types (text/css application/javascript application/xml)
/etc/init.d/apache2 restart

[dugrands]

One thing to possibly consider is that compressing the output stream adds a little bit to the processor load. I don't know what kind of impact one might see with it as it's somewhat of a chicken and egg problem. Faster downloads mean faster completion for Apache processes which might mean more spikes in CPU but overall better performance for a loaded system.

I think it would be interesting to see before and after CPU loads for a heavily used frontend server.

Actually surprisingly the processor loads were reduced from 3 - 5 to 0.3 - 1 on our 8 gig dual core, raid 10 postgres setup monitoring about 200 hosts and still increasing. As Alexei mentioned, the increase in server load because of compression in only minimal at worst but the fast completion of apache processes is what really matters. Also note that since the pages loaded slowly users would start refreshing the page manually and very frequently when the page seems to have failed leading to more resources used on the server side.

Talking of refreshing, has anyone tried squid with zabbix yet? I haven't tried it yet because my server is superfast right now and doesn't really need more hacks.

**By the way, I also have the same apache configuration as that of Nelsonab though my server is CentOS 5.3. If it doesn't work, make sure the module is installed and loaded. The line to load the module in my apache is;
LoadModule deflate_module modules/mod_deflate.so
***

Happy monitoring!

Dugrands.

[skogan]

Compression definitely helps, but I found that the biggest killer on the overview screen is vtext.php: these little images are reloaded fairly frequently. Has anybody been able to enable caching of vtext.php-generated images?

[Palmertree]

For best results, use php compression turned on in your php.ini file and use eAccelerator (http://eaccelerator.net/) which will cache php scripts so that they are not recompiled every time. eAccelerator has a compression as well. Mine is set to "9".

Other speed improvements:
1. Put your frontend on a dedicate server.
2. Optimize your Apache settings
3. MySQL client connection set to at lest one hour so appache does not have to reconnect over and over
4. For the experts, convert your js (javascripts) and css (style sheets) to php files so they are cached.
Requires code changes in main code to rename all .js and .css to .js.php and .css.php and add the following to the js and css files at the top of each file type:

js files:

<?php

        // send the requisite header information and character set
        header ("content-type: application/x-javascript");

        // check cached credentials and reprocess accordingly
        header ("cache-control: must-revalidate");

        // set variable for duration of cached content
        $offset = 60 * 60 * 6;

        // set variable specifying format of expiration header
        $expire = "expires: " . gmdate ("D, d M Y H:i:s", time() + $offset) . " GMT";

        // send cache expiration header to the client broswer
        header ($expire);

?>

css files:

<?php

        // send the requisite header information and character set
        header ("content-type: text/css");

        // check cached credentials and reprocess accordingly
        header ("cache-control: must-revalidate");

        // set variable for duration of cached content
        $offset = 60 * 60 * 6;

        // set variable specifying format of expiration header
        $expire = "expires: " . gmdate ("D, d M Y H:i:s", time() + $offset) . " GMT";

        // send cache expiration header to the client broswer
        header ($expire);

?>

[bbrendon]

I've been using expires and deflate modules in apache along with APC.

I've done some experimenting with busy websites and these combinations and they can dramatically improve performance to the end user and improve server load as well.

[bbrendon]

4. For the experts, convert your js (javascripts) and css (style sheets) to php files so they are cached.

Why not cache them on the browser? That seems like a better solution.

[Palmertree]

Why not cache them on the browser? That seems like a better solution.

Do an http watch and you will notice that javascripts and css sheets will be pulled in frequently between pages. Also, they do not get compressed in there native format unless converted to php. With this fix, you can change the expired time to 6 hours or whaver you like. We have seen a very good turn around on performance doing this. Especially on slow connections.

[bondbig]

i've installed ngix as a front-end to apache in my zabbix installation and also have tuned my.cnf to let innodb use more RAM and my CPU utilisation lowed from ~9-10% (4 CPUs, 1600 values for 16 hosts) to 2-3%.
nginx just caches static files (js, css, images) and gzips pages.