Switching to distributed monitoring

[Albéric de Pertat]

Hi,

I am running two Zabbix servers (1.8.14 - Ubuntu 12.04, database engine is MySQL) and after some time I have decided to switch my setup to the distributed setup. I gave an ID of 1 and 2 to my nodes and then ran (on both nodes)

zabbix_server -n 2 -c /etc/zabbix/zabbix_server.conf
zabbix_server -n 1 -c /etc/zabbix/zabbix_server.conf

just as the docs say. It goes on for about an hour or so then fails with the following message :

Converting tables ..........................zabbix_server [27880]: Error: [Z3005] query failed: [1062] Duplicate entry '200000000000001' for key 'PRIMARY' [update events set eventid=eventid+200000000000000 where eventid>0]
..............................................................Conversion failed.

The zabbix_server.log doesn't say anything about the whole process, even though I have upped the Debug value to 4.

I tried this with the main zabbix_server service running as well as stopped, but it still fails in either case.

The Web GUI still says my setup is not configured for distributed monitoring and no event is received anymore by the nodes.

So I can't seem to be able to rollback the changes that have been made and yet I can't complete the switch to the distributed setup.

Is there any way I can manually update what needs to be updated in the database directly and resume monitoring tasks, or at the very least roll back to the previous situation (two autonomous servers) ?

Thanks in advance

[heaje]

You'd probably be better off restoring from a backup before the changes if you have one. I believe the changes for nodes are a one-way trip.

[Albéric de Pertat]

I finally manage to pinpoint the problem in the MySQL server (Percona XtraDB really): it was crashing when the conversion mechanism was trying to update the ids in the history table. I tried to boost my VM's memory to 4G but the server would still crash.

Luckily for me, the Zabbix server was not yet deployed so I could truncate the history and trends tables (delete would crash my server as well).

As for the state my database was left in, some tables had lingering ids with values like 100100000... so I just removed them directly through the MySQL client. So you're right, restoring from a backup is faster, but I wanted to fully understand what was going on because I plan on using Zabbix intensely in the future.

After the truncate, I ran the magic command (zabbix_server -n 1) again and it did the trick.

Cheers

[Stingles]

I'm running into this same problem, but we want to keep the history. Is there a solution to this besides truncating the table? The error message I'm getting is as follows:

zabbix_server [13920]: Error: [Z3005] query failed: [1451] Cannot delete or update a parent row: a foreign key constraint fails (`zabbix`.`alerts`, CONSTRAINT `c_alerts_2` FOREIGN KEY (`eventid`) REFERENCES `events` (`eventid`) ON DELETE CASCADE) [update events set eventid=eventid+100000000000000 where eventid>0]

Thanks in advance

[Colttt]

maybe you cann delete this entry..

or make a backup of the DB.. delete the DB and after the zabbixprocess you can restore the database..