Migrated to new server - proxy problems

[tob]

Hi there,

I'm migrating a zabbix server to a new hardware setup. Not to complicated, since the setup was already behind a loadbalancer VIP. So basically:

1. Dumped all mysql data on old server.
2. Moved and imported the mysql data on the new server.
3. New server running fine, everything seems to work (we accept the fact that we lost a couple of hours worth of data because of the migration)
4. Point the LB VIP to the new internal address.

At this point the new server started to complain about the proxies, on the server side (debug level 4):

---
13319:20110804:115006.628 History data from active proxy on [192.168.231.105] failed: proxy [zXXX3] not found
13319:20110804:115006.628 In zbx_send_response()
13319:20110804:115006.628 zbx_send_response() '{
"response":"failed",
"info":"<F1>"}'
13319:20110804:115006.628 End of zbx_send_response():SUCCEED
13319:20110804:115006.628 End of recv_proxyhistory()

13299:20110804:115006.645 Discovery data from active proxy on [10.31.3.250] failed: proxy [zXXX1] not found
13299:20110804:115006.645 In zbx_send_response()
13299:20110804:115006.645 zbx_send_response() '{
"response":"failed"}'
13299:20110804:115006.645 End of zbx_send_response():SUCCEED
13299:20110804:115006.645 End of recv_discovery_data():FAIL
---

On the proxy side:

---
19007:20110804:114654.961 In zbx_recv_response()
19007:20110804:114654.969 zbx_recv_response() '{
"response":"failed",
"info":"<F1>"}'
19007:20110804:114654.969 End of zbx_recv_response():FAIL
19007:20110804:114654.969 End of put_data_to_server():FAIL


28483:20110804:132328.610 proxy_get_history_data() lastid:4
28483:20110804:132328.611 End of proxy_get_history_data():4
28483:20110804:132328.611 In connect_to_server() [192.168.XXX.XXX]:10051 [timeout:600]
28483:20110804:132328.614 In put_data_to_server() datalen:301
28483:20110804:132328.614 In send_data_to_server() [{
"request":"auto registration",
"host":"XXX1",
"data":[
{
"clock":1312456834,
"host":"XXX-speech"},
{
"clock":1312456859,
"host":"XXX1"},
{
"clock":1312456954,
"host":"master-speech"},
{
"clock":1312456980,
"host":"XXX3"}],
"clock":1312457008}]
28483:20110804:132328.614 In zbx_recv_response()
28483:20110804:132328.620 zbx_recv_response() '{
"response":"failed"}'
28483:20110804:132328.620 End of zbx_recv_response():FAIL
28483:20110804:132328.620 End of put_data_to_server():FAIL
28483:20110804:132328.620 End of autoreg_host_sender()
28483:20110804:132328.620 Datasender spent 0.014011 seconds while processing 0 values.
---

Ugh! So this is strange since the DB is exactly the same, proxy config did not change - and are configured the right way on the server (which also didn't change).

I even tried to configure a new proxy, but this seems to give the same issue's.

The only strange thing was when I tried to disable the hosts under a proxy, the frontend gave:



• Error in query [INSERT INTO events (eventid, source, object, objectid, clock, value, acknowledged) VALUES (19882996,0,0,25201,1312457520,2,0)] [Duplicate entry '19882996' for key 'PRIMARY']
• Updated status of host "blabla1.intra.ZZZ.YYY"
• Error in query [INSERT INTO events (eventid, source, object, objectid, clock, value, acknowledged) VALUES (19882997,0,0,26262,1312457520,2,0)] [Duplicate entry '19882997' for key 'PRIMARY']
• Updated status of host "blabla2.intra.ZZZ.YYY"


Not sure if this last error is directly related though...

Both server and proxies are running 1.8.5, of course I checked the network/firewalls etc... It really seems to be a application (in mysql?) issue.

Running out of options atm, anyone a clue?

[Mojah]

• Error in query [INSERT INTO events (eventid, source, object, objectid, clock, value, acknowledged) VALUES (19882996,0,0,25201,1312457520,2,0)] [Duplicate entry '19882996' for key 'PRIMARY']
• Updated status of host "blabla1.intra.ZZZ.YYY"
• Error in query [INSERT INTO events (eventid, source, object, objectid, clock, value, acknowledged) VALUES (19882997,0,0,26262,1312457520,2,0)] [Duplicate entry '19882997' for key 'PRIMARY']
• Updated status of host "blabla2.intra.ZZZ.YYY"

This may very well be your problem. It looks like your auto increment values may be wrong, which now seem to generate an ID that already exists in your database, thus violating the database integrity check of the primary key.

How did you migrate your server? A mysqldump of the entire database?

[tob]

I did a mysqldump of the zabbix database, then imported all again... (mind you, this took a good 17 hrs...) another option would be to copy all the bin files directly....

[Mojah]

I did a mysqldump of the zabbix database, then imported all again... (mind you, this took a good 17 hrs...) another option would be to copy all the bin files directly....

Find the max() eventid in the tabel "events" and then run the following query:
ALTER TABLE events AUTO_INCREMENT = xxx;

Where you change "xxx" with the current max() value eventid + 1.

[tob]

Alright, tried that:

mysql> ALTER TABLE events AUTO_INCREMENT = 19858966;
Query OK, 15126895 rows affected (3 min 25.74 sec)
Records: 15126895 Duplicates: 0 Warnings: 0

But:

23025:20110808:111500.510 History data from active proxy on [192.168.xx.30] failed: proxy [xxxpxi] not found
23025:20110808:111500.784 History data from active proxy on [192.168.xxx.1] failed: proxy [xxx3] not found
23048:20110808:111501.045 History data from active proxy on [10.31.xxx.250] failed: proxy [xxxpxi1] not found
23048:20110808:111501.049 Discovery data from active proxy on [10.31.xxx.250] failed: proxy [xxxpxi1] not found
23048:20110808:111501.353 History data from active proxy on [192.168.xxx.1] failed: proxy [xxxxi1] not found
22959:20110808:111501.567 History data from active proxy on [192.168.xxx.30] failed: proxy [xxxpxi] not found

The mysql data is 4+ days old now, so I'll have to start again - if I get this working, that is

[Mojah]

If you put your Zabbix Server on most verbose, do you see any more MySQL related errors?

[tob]

Nope, not really.. around the proxies I see stuff like:



23615:20110808:130411.223 Query without transaction detected
23615:20110808:130411.223 Query [txnlev:0] [delete from history_text where itemid=47354 and clock<1301528175]
23612:20110808:130411.223 In recv_proxyhistory()
23612:20110808:130411.223 Query [txnlev:0] [select hostid from hosts where host='Xpxi3' and status in (5) and hostid between 1000000000000000 and 1099999999999999]
23612:20110808:130411.224 History data from active proxy on [192.168.243.1] failed: proxy [Xpxi3] not found
23612:20110808:130411.224 In zbx_send_response()
23612:20110808:130411.224 zbx_send_response() '{
"response":"failed",
"info":"?"}'
23612:20110808:130411.224 End of zbx_send_response():SUCCEED
23612:20110808:130411.224 End of recv_proxyhistory()


23525:20110808:130411.444 Trapper got [{
"request":"history data",
"host":"Xpxi1",
"data":[
{
"host":"sw3-xxxxxs",
"key":"icmppingsec",
"clock":1312801411,
"value":"0.003977"},
(SNIP)

"host":"10.128.1.xxx",
"key":"cisco.serialnumber",
"clock":1312801446,
"value":"\"FCZXXXXXXXX\""}],
"clock":1312801451}] len 66438
23504:20110808:130411.746 In recv_proxyhistory()
23504:20110808:130411.747 Query [txnlev:0] [select hostid from hosts where host='xxxpxi1' and status in (5) and hostid between 1000000000000000 and 1099999999999999]
23504:20110808:130411.747 History data from active proxy on [10.31.3.250] failed: proxy [xxxpxi1] not found
23504:20110808:130411.747 In zbx_send_response()
23504:20110808:130411.747 zbx_send_response() '{
"response":"failed",
"info":"?"}'
23504:20110808:130411.747 End of zbx_send_response():SUCCEED
23504:20110808:130411.747 End of recv_proxyhistory()
23615:20110808:130411.751 In delete_history(history_str,47582,7,1312801373)


23504:20110808:130411.761 Trapper got [{
"request":"discovery data",
"host":"xxxpxi1",
"data":[
{
"clock":1312801419,
"drule":3,
"dcheck":16,
"type":12,
"ip":"10.128.1.116",
"status":1},
(SNIP)

{
"clock":1312801447,
"drule":3,
"dcheck":0,
"type":-1,
"ip":"10.128.1.128"}],
"clock":1312801451}] len 2596
23504:20110808:130411.761 In recv_discovery_data()
23504:20110808:130411.761 Query [txnlev:0] [select hostid from hosts where host='xxxpxi1' and status in (5) and hostid between 1000000000000000 and 1099999999999999]
23504:20110808:130411.761 Discovery data from active proxy on [10.31.3.250] failed: proxy [xxxpxi1] not found
23504:20110808:130411.761 In zbx_send_response()
23504:20110808:130411.761 zbx_send_response() '{
"response":"failed"}'

[Mojah]

Afraid I'm out of ideas then for now, you can hope someone who's already done a migration like this can help you out.

We've done this as well (migrated MySQL to a new server with several proxies) but never experienced anything like this.

[tob]

So, as a quick follow-up... problem solved ;-)

Turns out dns resolving (well, the resolving to out internal systems) wasn't working properly. For now I added the proxies to /etc/hosts.. this fixed the problem!

tob