Ad Widget

Collapse

Services->Services tab broken after 4.0 - 6.4 upgrade

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Tod Sandman
    Junior Member
    • Jul 2023
    • 9

    #1

    Services->Services tab broken after 4.0 - 6.4 upgrade

    I am testing an upgrade from Zabbix 4.0 LTS to the latest 6.4: Zabbix RPMs from the official Zabbix repo (zabbix-server-pgsql-6.4.5, zabbix-nginx-conf-6.4.5); Red Hat Enterprise Linux: 7.9 -> 9.2; Postgresql 12 -> 15; PHP 8.0.27.

    After fresh installation I dropped the zabbix database, pg_dump'ed the Zabbix database from the 4.0 to 6.4 server, fired up Zabbix, and the database was successfully upgraded with no error messages. I successfully applied double precision, history* primary key, and timescaledb updates.

    The only issue I see is accessing Services->Services; I immediately get an error page:


    This page isn’t working
    zbx1.uss.rice.edu is currently unable to handle this request.
    HTTP ERROR 500

    The nginx error log shows:

    2023/08/08 17:46:17 [error] 294307#294307: *5895 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught TypeError: CControllerServiceListGeneral::extendProblemEvents (): Argument Home ($services) must be of type array, bool given, called in /usr/share/zabbix/app/controllers/CControllerServiceList.php on line 145 and defined in /usr/share/zabbix/app/controllers/CControllerServiceListGeneral.php:290
    Stack trace:
    #0 /usr/share/zabbix/app/controllers/CControllerServiceList.php(145): CControllerServiceListGeneral::extendProblemEvents ()
    Home /usr/share/zabbix/include/classes/mvc/CController.php(476): CControllerServiceList->doAction()
    Forum /usr/share/zabbix/include/classes/core/ZBase.php(634): CController->run()
    #3 /usr/share/zabbix/include/classes/core/ZBase.php(233): ZBase->processRequest()
    #4 /usr/share/zabbix/include/config.inc.php(25): ZBase->run()
    #5 /usr/share/zabbix/zabbix.php(22): require_once('...')
    Special {main}
    thrown in /usr/share/zabbix/app/controllers/CControllerServiceListGeneral.php on line 290" while reading response header from upstream, client: 168.7.56.225, server: server_name, request: "GET /zabbix.php?action=service.list HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm/zabbix.sock:", host: "zbx1.uss.rice.edu", referrer: "https://zbx1.uss.rice.edu/zabbix.php?action=sla.list"

    I have tried with multiple browsers and users, including a new user created with full superuser access. I have repeated the OS installation and database dump/restore process from scratch, and I get the same result.


    I see a very similar issue posted at



    but the user closed the problem report with this comment:

    This issue was solved by exporting and importing events table. This table was not exported by mistake. This issue can be closed. Sorry for the confusion.

    I am not sure I understand the comment, but my events table was successfully copied over from the 4.0 database and converted to 6.4.

    Any ideas how to resolve this?
  • Tod Sandman
    Junior Member
    • Jul 2023
    • 9

    #2
    I started from scratch this time installing the latest Zabbix 6.0 LTO (zabbix-server-pgsql-6.0.20 etal) and get the same exact behavior.

    Comment

    • Tod Sandman
      Junior Member
      • Jul 2023
      • 9

      #3
      I cranked up Postgresql logging and get no error whatsoever. I maxed Zabbix logging and don't see anything that stands out. The only thing I can find that may correspond to me clicking the Serices->Services tab is
      95568:20230810:100126.095 zbx_setproctitle() title:'service manager Home [processed 0 events, updated 0 event tags, deleted 0 problems, synced 0 service updates, idle 5.131184 sec during 5.131651 sec]'
      95568:20230810:100126.095 services : 221 (1009 slots)
      95568:20230810:100126.095 service tags : 221 (1009 slots)
      95568:20230810:100126.095 services links : 202 (1009 slots)
      95568:20230810:100126.095 service problem tags : 30 (1009 slots)
      95569:20230810:100126.095 poller_type:255 location:0
      95568:20230810:100126.095 service problem tag index : 1 (1009 slots)
      95568:20230810:100126.096 service problems index : 2 (1009 slots)
      95568:20230810:100126.096 service matches : 0 (1009 slots)

      Comment

      • Tod Sandman
        Junior Member
        • Jul 2023
        • 9

        #4
        The services table does appear intact and seems to have been converted correctly from the 4.0 schema to the 6.0 (or 6.4) schema. The table itself has changed quite a bit between 4 and 6, but I cannot see anything wrong with the new services table. Below is a an example of a 4.0 entry and corresponding 6.0 entry:

        zabbix=> select * from services where serviceid = 207;
        -[ RECORD 1 ]------
        serviceid | 207
        name | WSCMS10
        status | 0
        algorithm | 1
        triggerid |
        showsla | 0
        goodsla | 99.9000
        sortorder | 0


        zabbix=> select * from services where serviceid = 207;
        -[ RECORD 1 ]-----+---------------------------------
        serviceid | 207
        name | WSCMS10
        status | -1
        algorithm | 2
        sortorder | 0
        weight | 0
        propagation_rule | 0
        propagation_value | 0
        description |
        uuid | a89b29daadc3443090e70e314ad50049
        created_at | 946684800


        Does anyone see anything obviously wrong or have any suggestions what else I should look at?

        Comment

        • tim.mooney
          Senior Member
          • Dec 2012
          • 1427

          #5
          My site isn't on 6.4, but I have a vague recollection of seeing other people with a similar issue post to the forums in the past few months.

          Did Services->Services work on your fresh 6.4 install, before you imported your 4.0 database with real world data? You're approaching it like it's something with your database that's causing the issue, and that's a very reasonable assumption, but at this point you've apparently ruled out the really obvious stuff with the DB. I'm just wondering if it might be something else?

          You've checked that you have all the PHP extensions installed that are listed as required for Zabbix 6.4 frontend? RHEL packages many of them in php-common, but you need a few specific additional packages to get all the requirements. It would be good to rule that out as a possible source of the front-end issue.

          Comment

          • Tod Sandman
            Junior Member
            • Jul 2023
            • 9

            #6
            Thanks for replying. Yes, Services->Services works fine out the box for both 6.0 and 6.4. It's only after I copy the old 4.0 data over and let Zabbix covert it.

            I will have to dig more regarding php packages. These are what I have (on my Zabbix-6.0 install):

            php-common-8.0.27-1.el9_1.x86_64
            php-ldap-8.0.27-1.el9_1.x86_64
            php-xml-8.0.27-1.el9_1.x86_64
            php-mbstring-8.0.27-1.el9_1.x86_64
            php-pdo-8.0.27-1.el9_1.x86_64
            php-pgsql-8.0.27-1.el9_1.x86_64
            php-bcmath-8.0.27-1.el9_1.x86_64
            php-fpm-8.0.27-1.el9_1.x86_64
            php-gd-8.0.27-1.el9_1.x86_64
            php-cli-8.0.27-1.el9_1.x86_64

            I am running a dnf install php\* install; if that does not changes anything, I will look more closely at the PHP requirements you mentioned. Thanks again.

            Comment

            • tim.mooney
              Senior Member
              • Dec 2012
              • 1427

              #7
              Since RHEL includes some of the extensions in php-common, I often find it easiest to use

              Code:
              php -m
              from the command line to get a sorted list of loaded modules/extensions.

              Based upon your list, it doesn't look like it's a missing extension that's causing the front-end issue, so we probably are back at looking at the database.

              How many records do you have in your services table? If it's just a few, would it be possible to review your services in your 4.0 install via the web interface, delete them (via the web interface) in the 4.0 install, then do the upgrade, then re-create them new at 6.x? Yes, it's tedious and it shouldn't be necessary, but if I were in your situation it would probably be the next thing I would look at. My guess here is that although the table schema is getting converted correctly, you have some pre-existing data in one of the service-related tables (might not be the main "services" table) that wouldn't be valid if you tried to insert it today.

              If you have many services defined and it would be a lot of work to recreate them all, then this approach isn't worth doing.

              Comment

              • Tod Sandman
                Junior Member
                • Jul 2023
                • 9

                #8
                I have 221 services defined (I actually did not create any of these myself). The php\* install took me down a rabbit hole, but php -m shows I have everything required.

                I deleted all events with a service ID < 100 (64 items), and the issue persisted. I then deleted all < 200 (91 items), leaving only 65 services, and the issue went away. So as you suggested it certainly seems like an issue with one or more of my services.

                I am going to have to back burner this upgrade for a while as I jump to an unrelated project, but eventually I will hone in on the event or events causing the issue. Thanks for all you help.

                Comment

                • tim.mooney
                  Senior Member
                  • Dec 2012
                  • 1427

                  #9
                  I was looking at the 6.0.0 upgrade notes for unrelated reasons and noticed the section at the very end, about the changes to Service monitoring that happen during the 6.0.x upgrade. That section and the link to details about what takes place during the upgrade may be useful to you for identifying likely service entries you have that may be getting converted incorrectly.

                  Comment

                  • Tod Sandman
                    Junior Member
                    • Jul 2023
                    • 9

                    #10
                    Thanks. Once I hone in on one of the services causing the issue, those details will certainly be helpful. I'm off on a completely un-related project now and will have to cycle back to this, most like at the end of September.

                    Comment

                    Working...