Ad Widget

Collapse

Database upgrade fail - 4.2.3 to 4.4.10 or 5.0.1 - same result

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • tizer001
    Junior Member
    • Jul 2020
    • 2

    #1

    Database upgrade fail - 4.2.3 to 4.4.10 or 5.0.1 - same result

    Anyone help/advise on an upgrade issue please.

    I'm in the process of building a new Zabbix server as my current hardware and Linux distro is pretty old.

    I'm trying to move from :-
    OpenSuse 13.1 (Bottle)
    MySQL - mysql-community-server-5.6.28-7.19.1.x86_64
    Zabbix Server 4.2.3 - Complied from Source

    Moving to (new hardware) :-
    openSUSE Leap 15.1
    MariaDB - mariadb-10.2.32-lp151.2.15.1.x86_64
    Zabbix Server 5.0.1 - compiled from source

    All my attempts so far have failed. Many database restores onto the new server but it always fails during the database upgrade. So far, if I get a fresh database restore and fire up a freshly compiled Zabbix Server 4.2.3 on the new server it starts up fine with no issue. If I try running 4.4.10 it will fail during the database upgrade. If I run 5.0.1 it will also fail.

    Maybe relevant, maybe not. My current zabbix instance has seen quite a few upgrades over the years, all from compiled source.

    Jul 10 2015 zabbix-2.2.5
    Feb 18 2016 zabbix-3.0.0
    Mar 9 2016 zabbix-3.0.1
    Apr 19 2017 zabbix-3.2.4
    Aug 25 2017 zabbix-3.4.0
    Aug 30 2017 zabbix-3.4.1
    May 7 2019 zabbix-4.2.1
    Jun 13 2019 zabbix-4.2.3


    Log output from running 4.4.10 (note that running 5.0.1 also produces the same errors)

    17175:20200707:105334.109 Starting Zabbix Server. Zabbix 4.4.10 (revision 4db30afc70).
    17175:20200707:105334.109 ****** Enabled features ******
    17175:20200707:105334.109 SNMP monitoring: YES
    17175:20200707:105334.109 IPMI monitoring: YES
    17175:20200707:105334.109 Web monitoring: YES
    17175:20200707:105334.109 VMware monitoring: YES
    17175:20200707:105334.109 SMTP authentication: YES
    17175:20200707:105334.109 ODBC: NO
    17175:20200707:105334.109 SSH support: NO
    17175:20200707:105334.109 IPv6 support: NO
    17175:20200707:105334.109 TLS support: NO
    17175:20200707:105334.109 ******************************
    17175:20200707:105334.109 using configuration file: /usr/local/etc/zabbix_server.conf
    17175:20200707:105334.109 In zbx_load_modules()
    17175:20200707:105334.109 End of zbx_load_modules():SUCCEED
    17175:20200707:105334.109 In init_database_cache()
    17175:20200707:105334.109 In zbx_mem_create() param:'HistoryCacheSize' size:16777216
    17175:20200707:105334.109 valid user addresses: [0x7f1f394db168, 0x7f1f3a4daff0] total size: 16776840
    17175:20200707:105334.109 End of zbx_mem_create()
    17175:20200707:105334.109 In zbx_mem_create() param:'HistoryIndexCacheSize' size:4194304
    17175:20200707:105334.109 valid user addresses: [0x7f1f390db178, 0x7f1f394daff0] total size: 4193912
    17175:20200707:105334.109 End of zbx_mem_create()
    17175:20200707:105334.109 In init_trend_cache()
    17175:20200707:105334.109 In zbx_mem_required_size() size:0 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
    17175:20200707:105334.109 End of zbx_mem_required_size() size:414
    17175:20200707:105334.109 In zbx_mem_create() param:'TrendCacheSize' size:4194304
    17175:20200707:105334.109 valid user addresses: [0x7f1f38cdb168, 0x7f1f390daff0] total size: 4193928
    17175:20200707:105334.109 End of zbx_mem_create()
    17175:20200707:105334.109 End of init_trend_cache()
    17175:20200707:105334.109 End of init_database_cache()
    17175:20200707:105334.109 In init_configuration_cache() size:268435456
    17175:20200707:105334.109 In zbx_mem_create() param:'CacheSize' size:268435456
    17175:20200707:105334.109 valid user addresses: [0x7f1f28cdb168, 0x7f1f38cdaff0] total size: 268435080
    17175:20200707:105334.109 End of zbx_mem_create()
    17175:20200707:105334.109 End of init_configuration_cache()
    17175:20200707:105334.109 In init_selfmon_collector()
    17175:20200707:105334.109 init_selfmon_collector() size:14568
    17175:20200707:105334.109 End of init_selfmon_collector() collector:0x7f1f3fd4b000
    17175:20200707:105334.109 In zbx_vmware_init()
    17175:20200707:105334.109 In zbx_mem_required_size() size:0 chunks_num:1 descr:'vmware cache size' param:'VMwareCacheSize'
    17175:20200707:105334.109 End of zbx_mem_required_size() size:421
    17175:20200707:105334.109 In zbx_mem_create() param:'VMwareCacheSize' size:8388187
    17175:20200707:105334.109 valid user addresses: [0x7f1f284db170, 0x7f1f28cdae50] total size: 8387808
    17175:20200707:105334.109 End of zbx_mem_create()
    17175:20200707:105334.109 End of zbx_vmware_init()
    17175:20200707:105334.109 In zbx_vc_init()
    17175:20200707:105334.109 In zbx_mem_required_size() size:0 chunks_num:1 descr:'value cache size' param:'ValueCacheSize'
    17175:20200707:105334.109 End of zbx_mem_required_size() size:419
    17175:20200707:105334.109 In zbx_mem_create() param:'ValueCacheSize' size:8388608
    17175:20200707:105334.110 valid user addresses: [0x7f1f27cdb168, 0x7f1f284daff0] total size: 8388232
    17175:20200707:105334.110 End of zbx_mem_create()
    17175:20200707:105334.110 End of zbx_vc_init()
    17175:20200707:105334.110 In zbx_db_get_database_type()
    17175:20200707:105334.110 In DBconnect() flag:0
    17175:20200707:105334.110 End of DBconnect():0
    17175:20200707:105334.110 query [txnlev:0] [select userid from users limit 1]
    17175:20200707:105334.110 there is at least 1 record in "users" table
    17175:20200707:105334.110 End of zbx_db_get_database_type():ZBX_DB_SERVER
    17175:20200707:105334.110 In DBcheck_version()
    17175:20200707:105334.110 In DBconnect() flag:0
    17175:20200707:105334.110 End of DBconnect():0
    17175:20200707:105334.110 query [txnlev:0] [show tables like 'dbversion']
    17175:20200707:105334.110 query [txnlev:0] [select mandatory,optional from dbversion]
    17175:20200707:105334.111 current database version (mandatory/optional): 04020000/04020000
    17175:20200707:105334.111 required mandatory version: 04040000
    17175:20200707:105334.111 optional patches were found
    17175:20200707:105334.111 starting automatic database upgrade
    17175:20200707:105334.111 query [txnlev:1] [begin;]
    17175:20200707:105334.111 query [txnlev:1] [alter table `autoreg_host` modify `host` varchar(128) default '' not null]
    17175:20200707:105334.117 query [txnlev:1] [update dbversion set mandatory=4030000,optional=4030000]
    17175:20200707:105334.123 query [txnlev:1] [commit;]
    17175:20200707:105334.123 completed 1% of database upgrade
    17175:20200707:105334.123 query [txnlev:1] [begin;]
    17175:20200707:105334.123 query [txnlev:1] [alter table `proxy_autoreg_host` modify `host` varchar(128) default '' not null]
    17175:20200707:105334.130 query [txnlev:1] [update dbversion set mandatory=4030001,optional=4030001]
    17175:20200707:105334.137 query [txnlev:1] [commit;]
    17175:20200707:105334.137 completed 3% of database upgrade
    17175:20200707:105334.137 query [txnlev:1] [begin;]
    17175:20200707:105334.137 query [txnlev:1] [alter table `host_discovery` modify `host` varchar(128) default '' not null]
    17175:20200707:105334.517 query [txnlev:1] [update dbversion set mandatory=4030002,optional=4030002]
    17175:20200707:105334.524 query [txnlev:1] [commit;]
    17175:20200707:105334.525 completed 5% of database upgrade
    17175:20200707:105334.525 query [txnlev:1] [begin;]
    17175:20200707:105334.525 query [txnlev:1] [create table item_rtdata (
    `itemid` bigint unsigned not null,
    `lastlogsize` bigint unsigned default '0' not null,
    `state` integer default '0' not null,
    `mtime` integer default '0' not null,
    `error` varchar(2048) default '' not null,
    primary key (itemid)
    ) engine=innodb]
    17175:20200707:105334.525 [Z3005] query failed: [1050] Table 'item_rtdata' already exists [create table item_rtdata (
    `itemid` bigint unsigned not null,
    `lastlogsize` bigint unsigned default '0' not null,
    `state` integer default '0' not null,
    `mtime` integer default '0' not null,
    `error` varchar(2048) default '' not null,
    primary key (itemid)
    ) engine=innodb]
    17175:20200707:105334.525 query [create table item_rtdata (
    `itemid` bigint unsigned not null,
    `lastlogsize` bigint unsigned default '0' not null,
    `state` integer default '0' not null,
    `mtime` integer default '0' not null,
    `error` varchar(2048) default '' not null,
    primary key (itemid)
    ) engine=innodb] failed, setting transaction as failed
    17175:20200707:105334.525 query [txnlev:1] [rollback;]
    17175:20200707:105334.525 database upgrade failed
    17175:20200707:105334.525 End of DBcheck_version():FAIL



    Any help/pointers much appreciated.

    Thanks.
  • tim.mooney
    Senior Member
    • Dec 2012
    • 1427

    #2
    Hi tizer001

    My site is like yours: the zabbix database has been upgraded many times over the years. I didn't realize how much the upgraded database schema had diverged from a fresh zabbix schema of the same version, until I needed to debug another database problem (the collation change I needed to do as part of the 3.2.x release had some problems in the process (see https://support.zabbix.com/browse/ZBX-17357 )).

    If your database is like mine, it turns out that even though the upgrades had been successful over the years, the database schema has accumulated some differences from what a fresh database would be for the exact same version.

    You probably have two options: either import your existing data (only, not schema) into a freshly created schema for the version you're currently at (4.2.3) or identify the places where your existing schema is different from a pristine schema for your version, and then alter your existing schema to match what it should be.

    I didn't want the down-time it would require for the export & load with a fresh schema, so what I ended up doing was the second option: fix my existing schema to be what it should be.

    To identify all of the differences between existing and fresh, I created a new, empty zabbix database on a different host, using the create.sql.gz for the version I started with. In your case, you want the create.sql.gz from 4.2.3.

    Once I had a new, pristine DB on a different host, I needed a way to compare the schema and find any differences. There is a 'mysqldiff' tool from Oracle that Zabbix support told me about, but I had already found a way to export just the schema in a standard format using the 'mysqldump' command, so I used that to export the two schemas. Then I used the GNU diff tool to find all the differences. One type of difference (constraints that were the same but listed in a slightly different order) could be ignored, but the other types I fixed.

    Once I had identified the differences, the process was to figure out the SQL command I needed to apply to my schema to get it to match the expected schema. That involved a bunch of web searching and some scripting, so I didn't have to manually type all of the ALTER statements.

    Comment

    • tizer001
      Junior Member
      • Jul 2020
      • 2

      #3
      Thanks for the detailed reply Tim, very much appreciated.
      I had a funny feeling it wasn't going to be a quick and simple fix.

      I'll give the mysqldiff tool a try out of curiosity and see how things go.

      Cheers

      Comment

      Working...