Ad Widget

Collapse

Best Order To Upgrade Zabbix Cluster?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • alexw-z
    Member
    • Dec 2021
    • 36

    #1

    Best Order To Upgrade Zabbix Cluster?


    I've inherited a bit of a monstrosity of a Zabbix cluster, comprised of the following:

    1x Zabbix server
    1x MariaDB Database node
    2x Web Front Ends
    2x HA nodes which sit in front of the two web nodes running HAProxy.

    The boxes are running 4.x, and I'm looking to upgrade the cluster as a whole to 4.4.10, before exporting the DB to a newer platform/server and continuing the upgrade to 6.0 (implementing a second native node for HA as I go).

    This is a pre-prod cluster (the Prod box funnily enough is a stand alone box) so downtime/stopping every service first isn't an issue. but I'm wondering if there's any advised order for performing upgrade.

    Relevant installed software is as follows:

    Main Zabbix Server
    zabbix-agent 4.2.6
    zabbix-frontend-php 4.2.6
    zabbig-get 4.2.6
    zabbix-java-gateway 4.2.6
    zabbix-release 4.0-3
    zabbix-server-mysql 4.2.6
    percona-zabbix-templates 1.7-2

    Web Nodes
    zabbix-agent 4.2.8
    zabbix-frontend-php 4.2.8
    zabbig-get 4.2.8
    zabbix-java-gateway 4.2.8
    zabbix-release 4.2.8

    MariaDB Node
    zabbix-agent 4.0.11
    zabbix-get 4.0.11
    zabbix-sender 4.0.11

    HA Nodes
    zabbix-agent 4.0.30
    zabbix-get 4.0.30
    zabbix-sender 4.0.30
    haproxy for web nodes

    As this is just a stop gap solution, I suspect I might even get away with just upgrading the main server and the two web nodes. I might not even have to stop the database service, although I'm inclined to just to halt it during the upgrade in order to err on the safe side.

    Is anybody able to confirm which boxes and which order it would be best to upgrade in please?

    Thanks!
  • Markku
    Senior Member
    Zabbix Certified SpecialistZabbix Certified ProfessionalZabbix Certified Expert
    • Sep 2018
    • 1781

    #2
    First of all you need to read and understand all the upgrade notes between 4.2.x and 6.0.12: https://www.zabbix.com/documentation...l/installation (and from that on)

    What I would do is roughly this (I'll just plain and simple assume that the operating systems and the database is some old version anyway):
    1. Install new database server and import the current database there
    2. Install new Zabbix server and start it against the new database server
    3. Install the new frontend(s) and other additional stuff on new server(s)
    4. Shutdown the old processes, change the old server IP addresses and change the new server IP addresses.
    Zabbix server process will automatically upgrade the database from 4.2 to 6.0 when started (see zabbix_server.log for any errors). Do note that you will still need to check your zabbix_server.conf to be correctly set up with Zabbix 6.0.x. Maybe there are some other required or optional changes that you need to execute manually in the database, the upgrade notes will tell you that.

    Markku

    Comment

    • cyber
      Senior Member
      Zabbix Certified SpecialistZabbix Certified Professional
      • Dec 2006
      • 4806

      #3
      I'm not sure you will benefit from doing that upgrade to 4.4.10 in between and then going to 6... And all those agent, get, sender versions are kind of irrelevant in all this process. Markku pretty much said it all, what you need to do..

      Comment

      • alexw-z
        Member
        • Dec 2021
        • 36

        #4
        Thanks for the replies guys. I've read the release notes already, and the software version numbers are just for completeness really, I'm not so worried about agent & tool versions.

        The old server is indeed an older version of Ubuntu (14 or 16 from Memory), and the new platform will be Alma 8 (hence the kinda convoluted 2 step upgrade process to get around some dependency issues). The new box will be stand alone initially, with a second HA node added at a later date.

        My understanding was that it would be best to upgrade to 6.0 from the latest version of 4.4, and that the versions should match.

        Is it safe to port a 4.2.x database into a new box running 4.4.10? I would have thought that would have the potential to cause problems.

        Comment

        • cyber
          Senior Member
          Zabbix Certified SpecialistZabbix Certified Professional
          • Dec 2006
          • 4806

          #5
          since v2 of zabbix there is possibility to directly upgrade from whatever version to latest. That's why I said, you don't really benefit from doing double upgrades... It should not really matter if you go to 6.0 from 4.2 or 4.4. I don't know how big your DB is, but you can export all your tables as csv and import to new DB version (whatever Mariadb version you may choose there), and recreate old version DB schema.... So you would have new platform with old DB content. Your new server will do all the necessary conversions, when you run it first time. You may need to run some additional conversion patches (primary key adding and some double precision mods on tables) later, but those are in release notes (also read the inbetween versions (5.x) release notes.. ).

          Comment

          • alexw-z
            Member
            • Dec 2021
            • 36

            #6
            Just checked back on my notes (I'm resuming this project after being blocked for a few months), I'm blocked from upgrading directly to 6.0 on the current box as version of Ubuntu is EOL and doesn't have the necessary versions of PHP and/or MySQL/Maria.

            I'm aware technically I could do a distro upgrade but I've never done them (I'm a CentOS guy), and read varying reports about them - especially given the original boxes were built by my predecessor and some of his configs are a bit... idiosyncratic. Another primary goal is to move the service away from Ubuntu as a platform and onto a box I've designed and built myself. That's the reason for the two step process, so I can get to a sensible version to incorporate a platform jump.

            I've actually done a dry run of a sandboxed copy of the prod box (like I said it's just stand alone so the solution was simpler), and got that up to 6.0 with just a couple of minor tweaks needed, but as I said I'm in this weird situation where the pre-prod instance is more complex than the prod one (I was initially led to believe that the pre-prod instance was a not in service POC thing, hence why I've ended up evaluating pre-prod after doing a dry run on prod, I appreciate some of this approach sounds a little bit backwards ), so I'm just trying to iron out whether there's a preferred upgrade order where multiple nodes are a factor. Otherwise I'm fairly comfortable with my plan.

            Comment

            • cyber
              Senior Member
              Zabbix Certified SpecialistZabbix Certified Professional
              • Dec 2006
              • 4806

              #7
              yep, thats true, your old platform will not support latest versios. But as you plan to change it anyway.. All you need is DB contents basically.. Set up parallel instance, copy over DB contents and run it.. I recently did the same to go from 4.4 to 6.. multiple corosync/pacemaker proxy clusters, HA frontends and DB-s.. yada-yada... the main task was to transfer database and run it on new platform. I needed to go from RH7 to 8, but even going from ubuntu to RH /Centos or something else would have been pretty much the same... Get parallel (and up to date) platform going and insert data. I would do dist upgrades only on some desktop host.. better to start clean on servers.

              But if you really want to do that 4.4 there... just introduce new repos to those hosts (http://repo.zabbix.com/zabbix/4.4/ub...abbix-release/ and something according to your version of buntu). Then stop everything and make sure it will not start automatically. Then upgrade everything. Going from 4.2 to 4.4 should be pretty painless. Then start up DB, after that server, check server logs, that it does all needed tasks and stays running. After that start up frontend(s). Then you can already go back and restore automatic startups, where needed etc.

              Comment

              • alexw-z
                Member
                • Dec 2021
                • 36

                #8
                Brilliant - thanks for the replies/confirmation

                Comment

                Working...