Zabbix web interface issue with Chrome

[mschlegel]

I have not been able to completely isolate the source of this issue, but it does seem to happen fairly regularly anytime more than one http request is in process simultaneously. It seems to happen easiest with the dashboard view, but I have had it happen on other pages as well.

Under these circumstances, I am seeing the web interface stop responding completely until the web server is restarted and the following shows up in a 'show full processlist' (trimmed for clarity):

| Query | freeing items | UPDATE ids SET nextid=nextid+1 WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid' |
| Query | Updating | UPDATE ids SET nextid=nextid+1 WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid' |
| Query | Updating | UPDATE ids SET nextid=nextid+1 WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid' |
| Query | Updating | UPDATE ids SET nextid=nextid+1 WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid' |
| Query | Updating | UPDATE ids SET nextid=nextid+1 WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid' |
| Query | freeing items | UPDATE ids SET nextid=nextid+1 WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid' |


After having the interface lock up like this a couple times, I setup a separate mysql username for the web interface to assist in troubleshooting. All of these queries are coming from the web interface.

I have not seen this happen when using IE7 with the web interface. Perhaps IE7 is not sending the additional requests to fill in the layout as quickly?

PHP options are configured to match the requirements checker during setup.

Zabbix-1.8

[wakko]

I've also been having this same problem.

I don't think it's browser-specific. I use Firefox for all of my browsing, but I've also tested Chrome and IE8, and all of them cause zabbix to DOS the database.

Chasing the call-chain in the code, I see this:

1. get_dbid() in include/db.inc.php has the offending query on line 749.

2. the only files calling get_dbid() for updating profileid are:

insert_profile() from include/profiles.inc.php
update_node_profile() from include/nodes.inc.php

3. I doubt the dashboard is inserting profiles, so I'm guessing that update_node_profile() is probably causing the problem.

4. update_node_profile() is only used in init_nodes() from include/nodes.inc.php

5. init_nodes() is used by:

include/page_header.php
get_current_nodeid() from include/nodes.inc.php

6. page_header and get_current_nodeid() are used pretty much everywhere.


My gut feeling is that the AJAX on the dashboard is generating too many get_dbid() queries too fast, and the DB can't cope.

Filed bug: https://support.zabbix.com/browse/ZBX-1657

[mschlegel]

Happen to have any updates on this? I don't see any updates on the bug and the bugid isn't mentioned on the 1.8.1 release notes. I have been avoiding the dashboard view, which is the only page I've had trigger a UI hang, but that really isn't a sustainable solution.

[Alexei]

We are still unable to repeat this problem in our environment. Working on this...

[mschlegel]

Could this possibly be related to something in the distributed monitoring code? I haven't seen this happen as frequently when I have a specific node selected. I can't say 100% for sure that it hasn't happened at all.

I'll try keeping a dashboard page open on a specific node and see if I run into any hangups today. It seemed pretty quick to happen most of the time.

[wakko]

Is it possible to store the client's state in a cookie, rather than in the database?

It seems to me that a large part of this problem can be solved if you're not writing to the database every time I click a new page in the UI.

Maybe I haven't spent enough time looking at the code to understand what's going on, but there doesn't seem to be a reason for the profile_id to be updated 5 times just because I look at the dashboard.

[mschlegel]

For reference, I did just get the same ID loop triggered with a specific node being selected.

php 5.1.6
Apache 2.2.3 (Centos 5.3)
Zabbix 1.8

[wax66]

Same problem here. Web interface gets hung within a minute of restarting Apache.

No slow queries that I see (25 seconds max, and that's very rare), restarting the Zabbix server doesn't seem to do any good, but restarting Apache clears it up until I hit the interface a bit.

Using:
Zabbix 1.8.1 in DM mode with 1 master, 2 slaves
/usr/libexec/mysqld Ver 5.0.45-log for redhat-linux-gnu on x86_64 (Source distribution)
PHP 5.1.6
Apache/2.2.3
CentOS release 5.2 (Final)

Just recently upgraded from 1.6.6.

It looks like nextid is incrementing at about 3000 times a second on the zabbix db. I wonder if that's a good thing, since it'll run out of space in about a year.

[wax66]

So far I haven't found any browsers that don't pound the database with "UPDATE ids SET nextid=nextid+1 WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid'" when loading the Dashboard.

My basic troubleshooting steps have been:

Restart apache, load web browser, log in, go to Dashboard, while Dashboard is loading check for "UPDATE ids".

I also check how much nextid has been updated with "SELECT nextid FROM ids WHERE nodeid=1 AND table_name='profiles' AND field_name='profileid';" after each page load.

The Dashboard can increment it over 18000 times from one page load that will take around 45 seconds to complete, but all the other pages increment it only a few times. Most of them only increment nextid only once. Pages such as Overview, Latest Data, Config->Web, Config->Hosts, etc, all work fine.

I'll go dig through the code and see if I can figure out why the dashboard kills the frontend.

BTW, for those looking for a 'workaround', if I don't load the dashboard, I can use the frontend endlessly.

[Alexei]

Wax66, what database, version and storage engine (InnoDB, MyISAM) you have?

[wax66]

mysql 5.0.45-log for redhat-linux-gnu on x86_64 (Source distribution) using InnoDB is the storage engine.

Looks like it may be localized to update_node_profile.

Thanks!
-Ron

[nelsonab]

I think this is related to the issue I'm seeing.

This is from a fresh install of Zabbix 1.8.1 on OpenSuSE 11.2

apache2-mod_php5-5.3.0-2.4.3.x86_64
php5-5.3.0-2.4.3.x86_64
mysql-5.1.36-6.7.2.x86_64

Running innodb

I'm seeing this for every page within Zabbix. It runs great for a while then, WHAM.... frozen. It will only load the menu bar and that's it.

[nelsonab]

Actually digging around a little more. I'm seeing it whenever I try to edit a graph. I can't $#@#$$@#ing do it! Everything else is working fine, I just can't #$#!@$!@#ing edit graphs! Template linked or not...

WTF?!

[Alexei]

mysql 5.0.45-log for redhat-linux-gnu on x86_64 (Source distribution) using InnoDB is the storage engine.

Looks like it may be localized to update_node_profile.

Thanks!
-Ron

We are working on this issue next week. It might be related to specific versions and/or types of database engines. I am sure we will fix it soon.

[nelsonab]

Good news Alexei!

If there is any low level debug information you need, let us I know how to get it for you.