Zabbix upgrade failure from 3.2 to 3.4

[irfad]

Server info:

Operating system:
Ubuntu 16.04.4 LTS

Current Zabbix server version:
zabbix_server (Zabbix) 3.2.11
Revision 76339 27 December 2017, compilation time: Dec 27 2017 15:21:29

Zabbix NVPS:
1245

Database:
MySQL 5.7

Problem 1
We tried to upgrade our current zabbix version to 3.4 and initially got below database related errors

1867:20181031:060213.293 [Z3005] query failed: [1091] Can't DROP 'c_dservices_2'; check that column/key exists [alter table dservices drop foreign key c_dservices_2]

For this we ran below query and it got fixed

ALTER TABLE dservices ADD CONSTRAINT c_dservices_2 FOREIGN KEY (dcheckid) REFERENCES dchecks (dcheckid) ON DELETE CASCADE;

Problem 2
During the upgrade we got another error like below

22286:20181031:122343.855 [Z3005] query failed: [1452] Cannot add or update a child row: a foreign key constraint fails (`zabbix`.`alerts`, CONSTRAINT `c_alerts_5` FOREIGN KEY (`p_eventid`) REFERENCES `events` (`eventid`) ON DELETE CASCADE) [update alerts set p_eventid=1042260 where eventid=1042312;

we tried to delete this record from alerts table and it got cleared, but same error appeared with a different event id and so on. Since we have large number of events this was a difficult and time consuming task to check each time upgrade fails and we delete these records one by one.

We tried two methods to get around this
Option 1
We tried to turn off foreign key check.

Option 2
We removed trigger data from events and alerts table table

delete from alerts;
delete from events where source=0;

With both options at each attempts (we're testing this upgrade with a snapshot of production) , database upgrade completed to 100% and right after that we got below same error from the logs

12460:20181102:115613.150 completed 66% of database upgrade
 12460:20181102:115613.235 completed 83% of database upgrade
 12460:20181102:115613.236 completed 100% of database upgrade
 12460:20181102:115613.236 database upgrade fully completed
 12460:20181102:115626.508 slow query: 13.218710 sec, "select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.snmp_community,i.snmp_oid,i.port,i.snmpv3_securityname,i.snmpv3_securitylevel,i.snmpv3_authpassphrase,i.snmpv3_privpassphrase,i.ipmi_sensor,i.delay,i.trapper_hosts,i.logtimefmt,i.params,i.state,i.authtype,i.username,i.password,i.publickey,i.privatekey,i.flags,i.interfaceid,i.snmpv3_authprotocol,i.snmpv3_privprotocol,i.snmpv3_contextname,i.lastlogsize,i.mtime,i.history,i.trends,i.inventory_link,i.valuemapid,i.units,i.error,i.jmx_endpoint,i.master_itemid from items i,hosts h where i.hostid=h.hostid and h.status in (0,1) and i.flags<>2"
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.
zabbix_server [12460]: ERROR [file:dbconfig.c,line:1999] Something impossible has just happened.

Then server process started and stopped with below error

 25387:20181031:153429.034 syncing history data...
 25387:20181031:153429.034 syncing history data done
 25387:20181031:153429.034 syncing trend data...
 25387:20181031:153429.034 syncing trend data done
 25387:20181031:153429.036 Zabbix Server stopped. Zabbix 3.4.14 (revision 84877).
zabbix_server [6976]: [file:'log.c',line:246] lock failed: [22] Invalid argument

I'd be great if anyone could help us to get it fixed, or has anyone already experienced the same problem before and fixed it?

[irfad]

could anyone shed some light?

[ingus.vilnis]

Hi,
https://support.zabbix.com/browse/ZBX-12657 and

https://support.zabbix.com/browse/ZBX-14445

These two could have traces. Please have a look there and see if that is your case.