Ad Widget

Collapse

zabbix database error connection refused when accessing login page

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • pgp82
    Junior Member
    • Nov 2020
    • 23

    #1

    zabbix database error connection refused when accessing login page

    Hi All,
    This is my first post and new to zabbix. Great software and I have been adding few nodes and it was working until today. When I try to login I get "database error: connection refused" error on the login page. I have logged into zabbix VM and can see mysql is out of space. Would really appreciate if someone could let me know how to fix this issue?

    mysql service status:
    mysqld.service - MySQL 8.0 database server
    Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/mysqld.service.d
    └─override.conf
    Active: failed (Result: exit-code) since Fri 2020-11-20 02:30:08 UTC; 22min ago
    Process: 1109 ExecStopPost=/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
    Process: 1107 ExecStart=/usr/libexec/mysqld --basedir=/usr (code=exited, status=1/FAILURE)
    Process: 1069 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS)
    Process: 1044 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
    Main PID: 1107 (code=exited, status=1/FAILURE)
    Status: "Server startup in progress"
    Error: 28 (No space left on device)

    when I run df -h i can see that sda5 which is 100%.

    df -h
    Filesystem Size Used Avail Use% Mounted on
    devtmpfs 3.9G 0 3.9G 0% /dev
    tmpfs 3.9G 0 3.9G 0% /dev/shm
    tmpfs 3.9G 8.5M 3.9G 1% /run
    tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
    /dev/sda2 4.0G 997M 3.1G 25% /
    /dev/sda1 488M 80M 373M 18% /boot
    /dev/sda3 1014M 40M 975M 4% /tmp
    /dev/sda5 4.5G 4.5G 20K 100% /var/lib/mysql
    tmpfs 787M 0 787M 0% /run/user/0

    I have expanded the VM hard drive but not sure how to expand sql portion to have more space. I'm running zabbix appliance which is running on centos.

    Thank you.
  • tim.mooney
    Senior Member
    • Dec 2012
    • 1427

    #2
    See post #10 in this thread, and the other forum posts they reference: https://www.zabbix.com/forum/zabbix-...e-or-directory

    Comment

    • pgp82
      Junior Member
      • Nov 2020
      • 23

      #3
      Originally posted by tim.mooney
      See post #10 in this thread, and the other forum posts they reference: https://www.zabbix.com/forum/zabbix-...e-or-directory
      Thank you tim.mooney for the quick reply. I tried following the article however I'm stuck after couple of steps. In my case, I have expanded the disks. So i'm following the steps outlined below;


      - start fdisk /dev/sda
      u (don't mind the depricated warning)
      n
      p
      (leave all the questions to default answer)
      w
      REBOOT
      pvcreate /dev/sda3
      vgextend zabbix-vg /dev/sda3
      lvextend -l 100%FREE /dev/mapper/zabbix--vg-root
      resize2fs /dev/mapper/zabbix--vg-root


      So in my case below is the output of the commands.

      [root@appliance ~]# fdisk /dev/sda

      Welcome to fdisk (util-linux 2.32.1).
      Changes will remain in memory only, until you decide to write them.
      Be careful before using the write command.


      Command (m for help): u
      Changing display/entry units to cylinders (DEPRECATED!).

      Command (m for help): n
      All primary partitions are in use.
      Adding logical partition 6
      No free sectors available.

      Command (m for help): p

      Disk /dev/sda: 30 GiB, 32212254720 bytes, 62914560 sectors
      Geometry: 255 heads, 2 sectors/track, 3916 cylinders
      Units: cylinders of 510 * 512 = 261120 bytes
      Sector size (logical/physical): 512 bytes / 512 bytes
      I/O size (minimum/optimal): 512 bytes / 512 bytes
      Disklabel type: dos
      Disk identifier: 0xf8386911

      Device Boot Start End Cylinders Size Id Type
      /dev/sda1 * 5 2061 2057 512M 83 Linux
      /dev/sda2 2061 18509 16449 4G 83 Linux
      /dev/sda3 18509 22621 4113 1G 83 Linux
      /dev/sda4 22621 41121 18501 4.5G 5 Extended
      /dev/sda5 22625 41121 18497 4.5G 83 Linux

      Command (m for help): w
      The partition table has been altered.
      Syncing disks.


      Where I'm stuck is at the pvcreate command. I'm guessing in my case I need to use sda6? as command "n" added a logical partition 6? however when I try to run pvcreate i get below.

      [root@appliance ~]# pvcreate /dev/sda6
      Device /dev/sda6 not found.

      I'm not sure if I'm doing the right thing here. Appreciate the help.

      Comment

      • pgp82
        Junior Member
        • Nov 2020
        • 23

        #4
        just wondering if anyone is able to help here? sorry i'm not that familiar with centos.

        Comment

        • tech
          Junior Member
          • Jan 2017
          • 5

          #5
          @pgp82 why use only 4.5 GB for mysql ? If have unallocated space so you can extend this otherwise not.

          Comment

          • pgp82
            Junior Member
            • Nov 2020
            • 23

            #6
            Originally posted by tech
            @pgp82 why use only 4.5 GB for mysql ? If have unallocated space so you can extend this otherwise not.
            Hi tech I simply downloaded zabbix appliance for hyper-v through https://www.zabbix.com/download_appliance. And I believe that's how the partition being done? i have expanded the virtual machine hard disk to 30GB now. However following above guide to extend the mysql partition is where I get stuck.

            If I do fdisk -l virtual drive being detected as 30GB.

            [root@appliance ~]# fdisk -l
            Disk /dev/sda: 30 GiB, 32212254720 bytes, 62914560 sectors
            Units: sectors of 1 * 512 = 512 bytes
            Sector size (logical/physical): 512 bytes / 512 bytes
            I/O size (minimum/optimal): 512 bytes / 512 bytes
            Disklabel type: dos
            Disk identifier: 0xf8386911

            Device Boot Start End Sectors Size Id Type
            /dev/sda1 * 2048 1050623 1048576 512M 83 Linux
            /dev/sda2 1050624 9439231 8388608 4G 83 Linux
            /dev/sda3 9439232 11536383 2097152 1G 83 Linux
            /dev/sda4 11536384 20971519 9435136 4.5G 5 Extended
            /dev/sda5 11538432 20971519 9433088 4.5G 83 Linux


            Below is the output of df -h

            [root@appliance ~]# df -h
            Filesystem Size Used Avail Use% Mounted on
            devtmpfs 3.9G 0 3.9G 0% /dev
            tmpfs 3.9G 0 3.9G 0% /dev/shm
            tmpfs 3.9G 8.5M 3.9G 1% /run
            tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
            /dev/sda2 4.0G 1008M 3.1G 25% /
            /dev/sda1 488M 80M 373M 18% /boot
            /dev/sda3 1014M 40M 975M 4% /tmp
            /dev/sda5 4.5G 4.5G 20K 100% /var/lib/mysql
            tmpfs 787M 0 787M 0% /run/user/0


            Comment

            • pgp82
              Junior Member
              • Nov 2020
              • 23

              #7
              anyone able to help me with this issue?

              Comment

              • tim.mooney
                Senior Member
                • Dec 2012
                • 1427

                #8
                First, my apologies. After reviewing what you've posted, it appears that the new appliance seems to have changed things a bit. They don't appear to be using the Linux logical volume manager (LVM) as an intermediate storage layer on the newest appliances, so the parts of the old threads that referenced LVM commands like 'pvcreate', 'lvextend', etc. aren't relevant to the current appliance.

                To make certain that LVM isn't involved, can you run

                Code:
                vgdisplay -v
                as root? You should either get an error or no output, which confirms that LVM isn't involved on the recent appliance. If it outputs information about volumes, extents, etc. then please post that information.

                Assuming that the logical volume manager is no longer used on the appliance, then extending the partition containing the mysql bits requires fewer commands, but there's one step that is a lot more dangerous. If you mess it up, you may as well start over with a new appliance.

                Next question: is the 'parted' command installed, and if so, what version is it?

                Comment

                • pgp82
                  Junior Member
                  • Nov 2020
                  • 23

                  #9
                  Originally posted by tim.mooney
                  First, my apologies. After reviewing what you've posted, it appears that the new appliance seems to have changed things a bit. They don't appear to be using the Linux logical volume manager (LVM) as an intermediate storage layer on the newest appliances, so the parts of the old threads that referenced LVM commands like 'pvcreate', 'lvextend', etc. aren't relevant to the current appliance.

                  To make certain that LVM isn't involved, can you run

                  Code:
                  vgdisplay -v
                  as root? You should either get an error or no output, which confirms that LVM isn't involved on the recent appliance. If it outputs information about volumes, extents, etc. then please post that information.

                  Assuming that the logical volume manager is no longer used on the appliance, then extending the partition containing the mysql bits requires fewer commands, but there's one step that is a lot more dangerous. If you mess it up, you may as well start over with a new appliance.

                  Next question: is the 'parted' command installed, and if so, what version is it?
                  Hi tim.mooney thank you for the reply and the clarification. When I try vgdisplay -v it says no volume groups found. I have parted installed and the version is 3.2.

                  Comment

                  • tim.mooney
                    Senior Member
                    • Dec 2012
                    • 1427

                    #10
                    Originally posted by pgp82
                    When I try vgdisplay -v it says no volume groups found. I have parted installed and the version is 3.2.
                    Thanks for the confirmation that the new appliance just uses traditional partitions, with no LVM involved.

                    With no LVM, and traditional filesystems directly on partitions, there are two logical steps you need to take:
                    1. resize the partition. With older tools this used to be a little dangerous, because you had to delete the existing partition and create it again starting at the same place but with a larger size. With parted 3.x, you can now resize the partition without the error-prone deletion step.
                    2. once the partition has been resized, you need to resize (grow) the filesystem that sits on top of it, to make use of the new space. This step is easy, once you have the partition fixed up.

                    The one part that I'm unsure of, and I don't have a convenient system to test with, is whether it's necessary to resize both partition 4 (the extended partition) and partition 5 (the partition with /var/lib/mysql), or if it's enough to just resize partition 5.

                    Here's what I recommend you do:
                    1. Since you're running it as a VM, shut the appliance down and take a snapshot, so you can recover if the partitioning/resizing goes wrong.
                    2. Boot the appliance up and log in as root and execute
                      Code:
                      	parted -s /dev/sda resizepart 5 100%
                    3. If step 2 succeeds with no error messages, skip to step 4. If it doesn't, try this:
                      1. Code:
                        parted -s /dev/sda resizepart 4 100%
                        , followed by
                      2. Code:
                        parted -s /dev/sda resizepart 5 100%
                        .
                      3. If those fail, don't go any further. Report whatever error messages you got and revert to the snapshot your took in step 1.
                    4. Assuming one of the two partition resize procedures worked, the last step is to resize the filesystem that's sitting on top of it.
                      1. If /var/lib/mysql is filesystem type ext3 or ext4, then use:
                        Code:
                        resizes2fs /dev/sda5
                      2. If /var/lib/mysql is filesystem type xfs, then use:
                        Code:
                        xfs_growfs /dev/sda5
                    Be sure you have a good VM snapshot before you embark on this path.


                    Last edited by tim.mooney; 02-12-2020, 06:39.

                    Comment

                    • pgp82
                      Junior Member
                      • Nov 2020
                      • 23

                      #11
                      Originally posted by tim.mooney

                      Thanks for the confirmation that the new appliance just uses traditional partitions, with no LVM involved.

                      With no LVM, and traditional filesystems directly on partitions, there are two logical steps you need to take:
                      1. resize the partition. With older tools this used to be a little dangerous, because you had to delete the existing partition and create it again starting at the same place but with a larger size. With parted 3.x, you can now resize the partition without the error-prone deletion step.
                      2. once the partition has been resized, you need to resize (grow) the filesystem that sits on top of it, to make use of the new space. This step is easy, once you have the partition fixed up.

                      The one part that I'm unsure of, and I don't have a convenient system to test with, is whether it's necessary to resize both partition 4 (the extended partition) and partition 5 (the partition with /var/lib/mysql), or if it's enough to just resize partition 5.

                      Here's what I recommend you do:
                      1. Since you're running it as a VM, shut the appliance down and take a snapshot, so you can recover if the partitioning/resizing goes wrong.
                      2. Boot the appliance up and log in as root and execute
                        Code:
                        	parted -s /dev/sda resizepart 5 100%
                      3. If step 2 succeeds with no error messages, skip to step 4. If it doesn't, try this:
                        1. Code:
                          parted -s /dev/sda resizepart 4 100%
                          , followed by
                        2. Code:
                          parted -s /dev/sda resizepart 5 100%
                          .
                        3. If those fail, don't go any further. Report whatever error messages you got and revert to the snapshot your took in step 1.
                      4. Assuming one of the two partition resize procedures worked, the last step is to resize the filesystem that's sitting on top of it.
                        1. If /var/lib/mysql is filesystem type ext3 or ext4, then use:
                          Code:
                          resizes2fs /dev/sda5
                        2. If /var/lib/mysql is filesystem type xfs, then use:
                          Code:
                          xfs_growfs /dev/sda5
                      Be sure you have a good VM snapshot before you embark on this path.

                      Hi tim.mooney Thank you for your quick reply. I have run the first command and below is what I get. not sure If I should further progress at this stage?

                      [root@appliance ~]# parted -s /dev/sda resizepart 5 100%
                      Warning: Partition /dev/sda5 is being used. Are you sure you want to continue?
                      [root@appliance ~]#

                      FYI - I have taken a snapshop of the VM.

                      Comment

                      • tim.mooney
                        Senior Member
                        • Dec 2012
                        • 1427

                        #12
                        It should be OK for the partition to be in use -- the filesystem won't notice until you do the later step to resize it.

                        If you want to be extra sure, you could do this:
                        1. temporarily stop 'zabbix-server' and 'httpd', so that nothing is accessing the database. That's probably done via:
                          1. Code:
                            systemctl stop zabbix-server.service
                          2. Code:
                            systemctl stop httpd.service
                        2. once nothing is accessing the database, stop it too:
                          Code:
                          systemctl stop mysqld.service
                        3. once the database is shut down, nothing should be accessing /var/lib/mysql. You can verify that with
                          Code:
                          fuser -c /var/lib/mysql
                        4. Assuming nothing was accessing /var/lib/mysql, you can temporarily unmount it:
                          Code:
                          umount /var/lib/mysql
                        With /var/lib/mysql unmounted, try the parted resize again. You can safely do the 'resize2fs' or the 'xfs_growfs' (whichever one is appropriate) with it unmounted too.

                        Then just reboot.

                        Comment

                        • pgp82
                          Junior Member
                          • Nov 2020
                          • 23

                          #13
                          Originally posted by tim.mooney
                          It should be OK for the partition to be in use -- the filesystem won't notice until you do the later step to resize it.

                          If you want to be extra sure, you could do this:
                          1. temporarily stop 'zabbix-server' and 'httpd', so that nothing is accessing the database. That's probably done via:
                            1. Code:
                              systemctl stop zabbix-server.service
                            2. Code:
                              systemctl stop httpd.service
                          2. once nothing is accessing the database, stop it too:
                            Code:
                            systemctl stop mysqld.service
                          3. once the database is shut down, nothing should be accessing /var/lib/mysql. You can verify that with
                            Code:
                            fuser -c /var/lib/mysql
                          4. Assuming nothing was accessing /var/lib/mysql, you can temporarily unmount it:
                            Code:
                            umount /var/lib/mysql
                          With /var/lib/mysql unmounted, try the parted resize again. You can safely do the 'resize2fs' or the 'xfs_growfs' (whichever one is appropriate) with it unmounted too.

                          Then just reboot.
                          Hi tim.mooney Thank again for your speedy and well explained responses. I have run above command and tried to run 'xfs_growfs' command since var/lib/mysql is filesystem type xfs. But I get below error.

                          [root@appliance ~]# xfs_growfs /dev/sda5
                          xfs_growfs: /dev/sda5 is not a mounted XFS filesystem
                          [root@appliance ~]#

                          FYI - Parted output below.

                          [root@appliance ~]# parted -l /dev/sda
                          Model: Msft Virtual Disk (scsi)
                          Disk /dev/sda: 32.2GB
                          Sector size (logical/physical): 512B/512B
                          Partition Table: msdos
                          Disk Flags:

                          Number Start End Size Type File system Flags
                          1 1049kB 538MB 537MB primary ext4 boot
                          2 538MB 4833MB 4295MB primary xfs
                          3 4833MB 5907MB 1074MB primary xfs
                          4 5907MB 10.7GB 4831MB extended
                          5 5908MB 10.7GB 4830MB logical xfs

                          Comment

                          • tim.mooney
                            Senior Member
                            • Dec 2012
                            • 1427

                            #14
                            Huh, Ok, sorry for giving you bad advice again.

                            My org doesn't use xfs much, so I was just going off what the man page said. Try re-mounting /var/lib/mysql:

                            Code:
                            mount /var/lib/mysql
                            and try the grow command again.

                            Also, congratulations on getting the partition fixed up. For my edification, which option did you have to use? parted on just partition 5, or parted on both 4 and 5?

                            Comment

                            • pgp82
                              Junior Member
                              • Nov 2020
                              • 23

                              #15
                              Originally posted by tim.mooney
                              Huh, Ok, sorry for giving you bad advice again.

                              My org doesn't use xfs much, so I was just going off what the man page said. Try re-mounting /var/lib/mysql:

                              Code:
                              mount /var/lib/mysql
                              and try the grow command again.

                              Also, congratulations on getting the partition fixed up. For my edification, which option did you have to use? parted on just partition 5, or parted on both 4 and 5?
                              tim.mooney I have run the above command and the grow command. Below is the output. Just want to make sure all ok before rebooting.

                              [root@appliance ~]# xfs_growfs /dev/sda5
                              meta-data=/dev/sda5 isize=512 agcount=4, agsize=294784 blks
                              = sectsz=512 attr=2, projid32bit=1
                              = crc=1 finobt=1, sparse=1, rmapbt=0
                              = reflink=1
                              data = bsize=4096 blocks=1179136, imaxpct=25
                              = sunit=0 swidth=0 blks
                              naming =version 2 bsize=4096 ascii-ci=0, ftype=1
                              log =internal log bsize=4096 blocks=2560, version=2
                              = sectsz=512 sunit=0 blks, lazy-count=1
                              realtime =none extsz=4096 blocks=0, rtextents=0

                              I only ran the parted on just partition 5. Didn't need to do parted on partition 4. Again really appreciative of your help and detailed responses

                              Comment

                              Working...