Issues changing database that server talks to

[zillions]

Hi,
We're in the process of testing the restore of a database backup in Zabbix, and are running into a strange problem. In short, we connect the Zabbix front end to the restored backend, and the frontend still shows all the info from the old DB, not the new one. I feel like somewhere there is a cache or a reference to the old database that we need to wipe or overwrite, but where?

Here's the full story:
We have a prod Zabbix instance, and a dev instance for testing. We are moving the backend to shared storage attached to a new backend server, and wanted to test to make sure everything worked. We restored a backup of the database to the new backend server, and then connected it to the frontend of the dev server.
In the UI we turned off housekeeper, updated zabbix_server.conf with all the new database info, restarted the apache and zabbix server services, and then went to the UI.

At that point the UI was up, but all the references (number of hosts, template list, etc...) all were still for the dev instance (the old DB). Suspecting that old database maintained a connection, we tailed binlog, and sure enough, it was. We then stopped the mysql service on the old database server, and restarted all the services again on the frontend server. We can see the logs showing various connections and reads/writes, however the UI is not updating.

We tried reloading the configuration cache: http://blog.zabbix.com/reloading-con...ion-cache/528/

We tried changing the password in the config file to something that wouldn't work, and sure enough that showed in the logs (so we know the password and account are working).

I'm not sure where else to look. There are no references to the server in my.cnf on the mysql server, and it seems like there is a cache somewhere of the old DB. The UI just isn't reflecting what changed.
I should note that there is no overlap of hosts between both databases. The dev database which we disabled only has about 28 hosts, and the prod database that we're trying to connect to has over 3000 hosts.
I should also note that none of the prod hosts are pointing at the dev server as their server, so no data should be coming to the dev server. I was just expecting to at least see the host count or template list have the right numbers.
I did point one prod host at the dev server (but purposely left the UI alone as the database already has the host), but that didn't change/fix anything.

Also, the frontend server in dev (that we're testing on), is not the same name as the prod server.
The database names are different, however we updated the configs to account for that.

Any suggestions?

[tchjts1]

Did you update zabbix.conf.php?

$DB["TYPE"] = 'MYSQL';
$DB["SERVER"] = 'My DB IP Here';
$DB["PORT"] = '0';
$DB["DATABASE"] = 'zabbix';
$DB["USER"] = 'zabbix';
$DB["PASSWORD"] = 'MySecretPasswordHere';

[zillions]

I didn't, let me try that! Thank you! I'll post back the results!

[zillions]

Hi tchjts1,
Thanks again for your suggestion, that was exactly what the problem was.
I'd never actually opened/modified that file directly before, and didn't realize (or didn't remember) that the UI connects to the DB separately from the main server engine.

Sure enough, I updated that file, and restarted httpd and zabbix-server and everything came up right away.

Thank you very much!

[tchjts1]

Glad that fixed the issue. Another tip for when you upgrade from version to version of the source code - keep a copy of zabbix.conf.php locally or in a separate dir on the Zabbix server where it won't get overwritten during an upgrade, then put it back in place after your upgrade is done.

Otherwise you will have to go through the configuration steps again to reconfigure it.

[zillions]

Thanks!
We actually manage that file through puppet and the variables are generated dynamically so we don't have to change/set anything.
In this case I was testing a DB restore on new storage, so it was a temporary test. I turned puppet off, but at that point it had already set those values. The puppet stuff was setup a while ago so I didn't even remember that file.

Thanks again!