Scaling the web front end

[jcesario]

Well after some extensive tuning of lighttpd I had gotten the frontend to at least be reasonably responsive again. However, the nonresponsive zabbix problem has crept up in our 1.6.8 instance now.

Aly, Its not just viewing items that times out. Once we hit some magical critical mass point, zabbix simply stops responding to modifications/additions/removals to items AT ALL.

We cannot add a single item to a host, nor can we remove or modify any items. This basically renders zabbix completely useless.

Whenever trying to add or remove templates from hosts It simply spins and spins. Then eventually stops processing and does nothing. No exhausted memory ( I cranked my PHP max memory up to 2GB, no max execution time (set to 6 hours).

Big fail.

Were currently downgraded to 1.6.8, but this is also the issue we have with 1.8.1 - Aly, can you confirm behavior like this is not going to be present in 1.8.2 ?

[jcesario]

For S&G I flipped over to pg 8.4 to see if innodb shoddy concurrency was causing issues.

Replicated the exact same issue. Whenever dealing with large number of hosts or items (hosts in the hundres and items in the 10s of thousands). The Zabbix front end likes to just sit there and spin its wheels for a while and then just quit out without completing whatever action.

Able to replicate doing the following things
Adding 200 hosts to a template with 996 items in it.
Adding 200 hosts to a specific host group.
Removing or linking a template against 5/10/100/200 hosts when the template has multiple thousands of items in it.

Just FYI all of our items are SNMPv2

[jcesario]

Bump. Bueller ?

[Alexei]

We are about to release 1.8.2. Jcesario, may I ask you to download and try the latest nightly build and tell us about your experience?

It is here: http://www.zabbix.com/developers.php

Note that you may install the newest front-end in a different location, no need to break your existing 1.8.1. I would be very interested in you feedback.

The hardware you use is capable of monitoring of thousands of devices without breaking a sweat. It must be some minor issue, misconfiguration or inefficiency which makes the front-end very slow in your environment. It would be nice to have it resolved prior to 1.8.2.

[jcesario]

Will do Alexei. Ill deploy the nightlies sometime today and get back to the forums.

Assuming I still have the same issue what kind of information would you like me to gather for diagnosis?

[Alexei]

Assuming I still have the same issue what kind of information would you like me to gather for diagnosis?

Let's wait for performance results. Actually some stats from your system would be handy: total number of hosts and average number of items/triggers/applications per host, also number of values per second from the Dashboard.

[jcesario]

Setting up 1.8.2 nightly (10966) right now. Ill let you know how it goes in a few hours.

[jcesario]

When doing an import of my xml templates:

Mar 22 11:34:13 p3plmysqlweb02 php-cgi: PHP Fatal error: require_once() [<a href='function.require'>function.require</a>]: Failed opening required 'include/classes/class.domdocument.php' (include_path='.:/usr/share/pear:/usr/share/php') in /var/www/html/zabbix182/include/config.inc.php on line 62

[jcesario]

just for reference:


Current value Required Recommended
PHP version 5.2.6 5.0 5.3.0 Ok
PHP memory limit 2048M 128M 256M Ok
PHP post max size 1024M 16M 32M Ok
PHP upload max filesize 1024M 2M 16M Ok
PHP max execution time 1800 300 600 Ok
PHP max input time 600 300 600 Ok
PHP timezone America/Denver Ok
PHP databases support MySQL
PostgreSQL Ok
PHP BC math yes Ok
PHP MB string yes Ok
PHP Sockets yes Ok
PHP GD 2.0.34 2.0 2.0.34 Ok
GD PNG Support yes Ok
libxml module 2.6.26 2.6.15 2.7.6 Ok
ctype module yes Ok


# grep -P "^error_reporting" /etc/php.ini
error_reporting = E_ALL

[jcesario]

When doing an import of my xml templates:

Mar 22 11:34:13 p3plmysqlweb02 php-cgi: PHP Fatal error: require_once() [<a href='function.require'>function.require</a>]: Failed opening required 'include/classes/class.domdocument.php' (include_path='.:/usr/share/pear:/usr/share/php') in /var/www/html/zabbix182/include/config.inc.php on line 62

btw that was my bad. forgot to install php xml extension - but shouldnt the installer pick that up now?

[jcesario]

Same behavior as before. When trying to add or manipulate a large number of items against hosts the frontend simply stops processing after a few moments and nothing is changed.

Attempt to add single template (comprised of 12 other templates) tallying 2000 SNMPv2 items.

Attempted to add this template to host group comprised of 163 hosts.

There are no PHP or MySQL errors returned from this, as well as nothing displayed on the frontend.

Number of hosts (monitored/not monitored/templates) 691 449 / 1 / 241
Number of items (monitored/disabled/not supported) 0 0 / 0 / 0
Number of triggers (enabled/disabled)[true/unknown/false] 0 0 / 0 [0 / 0 / 0]
Number of users (online) 2 1
Required server performance, new values per second 0 -

[jcesario]

Same behavior on latest nightly.

[Alexei]

There are no PHP or MySQL errors returned from this, as well as nothing displayed on the frontend.

It looks like PHP memory settings in php.ini are too low for your setup. Try to increase it to 512MB.

[jcesario]

Its currently at 2GB????

Also, if I was hitting the max memory limitation for PHP. There would be memory exhaustion errors thrown in the error_log. I have my PHP debug level set to E_ALL

[nelsonab]

I wish there was some functionality in the frontend for debugging/profiling log files. It would be nice if you could at least say "this was the last function called" to help track down bugs like this.