Database Error - Error connecting to database: No such file or directory

[agvonline]

Hi

New to Zabbix...

I recently deployed a pre-built Zabbix appliance for Hyper-V (v4.4). It uses Ubuntu.

All was well, and it's worked great for the last 4 weeks. But this morning, it displays when I navigate to http://hostname/zabbix:

Database Error - Error connecting to database: No such file or directory

Can anyone help?

I found this link https://github.com/linux-on-ibm-z/do...-Zabbix-server which suggests I need to change the database host name from localhost to 127.0.0.1. But where do I do this?

Thanks

A

[provels]

Not to threadjack, but I have the same issue today after 5 months of running fine. Mine is a Hyper-V 4.4.6 appliance. Plenty of diskspace, just a small home network. No idea what happened, it just broke. Were you able to solve your problem? Thanks.

[tim.mooney]

I recently deployed a pre-built Zabbix appliance for Hyper-V (v4.4). It uses Ubuntu.

All was well, and it's worked great for the last 4 weeks. But this morning, it displays when I navigate to http://hostname/zabbix:

Database Error - Error connecting to database: No such file or directory

If the pre-built image worked fine for several weeks but now is not working, it's not because the database hostname is wrong. Whatever thread you found isn't relevant to your problem.

When a MySQL or MySQL-like database is running and there's other software (like the Zabbix web interface or the Zabbix server) on the same system, there are two ways that the other software can connect to the database:

1. over the network. This is the only option you have when your database is on a different system from your Zabbix server or your Zabbix web frontend
2. via a local socket which exists as a path in the filesystem. This is generally the preferred method when both the application software and the database are on the same server.

The "No such file or directory" error message you're seeing from the web interface probably means that the web interface can't find the socket file it's probably trying to use.

It would be interesting to know whether it's only the web interface that can't access the database or whether the zabbix_serverd software also is having trouble accessing the database. You should look in the /var/log/zabbix/zabbix_server.log and look to see if the server has logged any recent messages (near the bottom of the file) about database connection issues.

If both of them are having trouble, you should check to see if the MySQL database server is running. It will appear in the process list as 'mysqld' or perhaps 'mysqld_safe', and will most likely be running as the 'mysql' user.

If it's not running, you need to look to see if there are any /var/log/mysql.log or /var/log/mysql/*.log files that may contain a clue as to what happened to your database.

[provels]

Thanks very much for your reply, Tim. My install is a 4.4.6 appliance VM on Hyper-V monitoring only 4-5 devices in a home hobby environment. I have also installed a parallel virgin 4.4.6 VM but without my hosts so I can compare things like files and permissions. I'm using WinSCP. I took a look at the various log files on the failed server and I am unable to open them for viewing with a permissions error but they open fine on the fresh one.

Pardon my ignorance, but where exactly is the DB located on the appliances? I tried to restart mysqld, but it returned "mysqld.service not found".

While browsing the logs, I came across the /var/log/unattended-upgrades folder. The last line of the unattended-upgrades.log says to check the unattended-upgrades-dpkg.log.
Am I correct in assuming Ubuntu applies patches to itself automatically, like Windows does? If so, I was not aware of that. Here is the unattended-upgrades-dpkg.log file. It seems to show that I had run out of disk. The VHD is a 10GB disk with about 4GB free.

unattended-upgrades-dpkg.txt Pretty sure this is the same day the DB failed.

I know this is pretty disjointed, but if my DB still exists, maybe I can port it to the second server.

[tim.mooney]

Thanks very much for your reply, Tim. My install is a 4.4.6 appliance VM on Hyper-V monitoring only 4-5 devices in a home hobby environment. I have also installed a parallel virgin 4.4.6 VM but without my hosts so I can compare things like files and permissions. I'm using WinSCP. I took a look at the various log files on the failed server and I am unable to open them for viewing with a permissions error but they open fine on the fresh one.

Interesting. If you select one of the log files on the problem VM and then click the "Properties" button in the WinSCP tab (at least in Explorer mode it's on the top "ribbon", if you're using Commander UI mode you might have to hunt for it), what does it say for "Owner", "Group", and the Octal value for "Permissions", and how does that compare to the pristine VM?

Pardon my ignorance, but where exactly is the DB located on the appliances? I tried to restart mysqld, but it returned "mysqld.service not found".

I've never actually used the appliance, so I'm not very familiar with it. Assuming it's using either MySQL or MariaDB, in either case the root of the database directory is likely /var/lib/mysql . Under there you should find (permissions will be restricted so you'll probably need to be root or use a shell and 'sudo') a directory named 'zabbix', and within that directory should be each table for the database. It's possible they've done something to relocate where the database is, but that's not likely. If it's not under /var/lib/mysql, you'll want to look at /etc/my.cnf and look for the 'datadir' setting.

While browsing the logs, I came across the /var/log/unattended-upgrades folder. The last line of the unattended-upgrades.log says to check the unattended-upgrades-dpkg.log.
Am I correct in assuming Ubuntu applies patches to itself automatically, like Windows does? If so, I was not aware of that. Here is the unattended-upgrades-dpkg.log file. It seems to show that I had run out of disk. The VHD is a 10GB disk with about 4GB free.

unattended-upgrades-dpkg.txt Pretty sure this is the same day the DB failed.

I know this is pretty disjointed, but if my DB still exists, maybe I can port it to the second server.

I don't know how they set up the appliance, but having it apply updates automatically is certainly a possibility (if it has access to the internet) and does seem to explain the issue.

I'm sure your database is still present, though there may be some corruption. If this were an extremely busy transactional system then recovery might be a long shot, but on a small home system, it may just auto-recover, once you can get it started up. If it doesn't, it should log something that either tells you what to run to try salvage things OR is enough of an error message that you can web search on it.

Before you'll know what to do about your database, you probably need to do some cleanup to free some space and potentially to allow the partially-completed update to finish.

You probably first want to do some web searches for something like "Ubuntu apt cache clean" or similar, to find the procedure to clean up packages that are cached by the package management system. Those are taking up space in /var/cache/apt/ , which is almost certainly shared with /var/lib/mysql (same volume).

You can also remove any old log files that you don't need. The ones that are previous log files (not actively being written to) will be in /var/log and will have names ending in .log.# or .log.#.gz (if they've been compressed).

If you can free enough space (safely!) to get the database to start, you'll then know what, if any, damage you have to deal with there.

[provels]

TL;DR - Cleared the apt cache and restarted the server and the DB started up.

-----------------------------------------

Cache wasn't really that large, 140MB. No data since it shutdown on 9/2, but no harm in that. A little sluggish at the beginning, but maybe it was repairing the DB. I'm no SQL guru (really? lol). Still can't open any /var/log/zabbix log files, though. Don't know how those permissions could have changed. Wonder if somehow the failed update had done something, but doesn't seem likely. Suppose I should change them to match the new machine's 664 octal? Guessing it wouldn't hurt.

EDIT - Just noticed the Apache log files have the same 640 octal as the Zabbix logs, but I can open the Apache logs, go figure.

Thanks again for all your help, Tim.

-----------------------------------------

More detail here:

Log file permissions

Old machine
Group - zabbix
Owner - zabbix
Octal - 640

New machine
Group - zabbix [112]
Owner - zabbix [108]
Octal - 664

Attempting to change permissions from WinSCP fails (I suppose because I'm logged in as "appliance" user, but I can sudo in PuTTY to change.

The whole GZipped zabbix/log folder is only 4MB going back to last June (probably the original install).

datadir = /var/lib/mysql

-----------------------------------------

Just leaving these here if it helps someone in the future.
About Ubuntu auto-updates - https://itsfoss.com/auto-updates-ubuntu/
About freeing disk space - https://www.omgubuntu.co.uk/2016/08/...pace-on-ubuntu, https://itsfoss.com/free-up-space-ubuntu-linux/

[charliemav]

This worked for me as well! thanks!

[provels]

Just ran into another problem. Server running apparently fine, but when I run "sudo apt-get autoremove" , I receive the following error. I thought it might be because I was using a dynamically expanding virtual disk, so I converted it to fixed. There is 5GB free of 10GB total. This is the same package that failed to install when the server went down. How is this possible?

For the time being I have turned off automatic updates.

appliance@zabbix:~$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
linux-headers-4.4.0-189-generic : Depends: linux-headers-4.4.0-189 but it is not installed
E: Unmet dependencies. Try using -f.
appliance@zabbix:~$
appliance@zabbix:~$ sudo apt-get install linux-headers-4.4.0-189
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
linux-headers-4.4.0-189
0 upgraded, 1 newly installed, 0 to remove and 31 not upgraded.
7 not fully installed or removed.
Need to get 9,988 kB of archives.
After this operation, 71.2 MB of additional disk space will be used.
E: You don't have enough free space in /var/cache/apt/archives/.

[tim.mooney]

I don't know apt-based systems as well as I know RPM-based systems, but I would suspect that both packages were going to be updated and as part of the out-of-space issue the database believes one of them is now missing.

I honestly doubt that anything installed on the system requires any of the linux header packages. If you can't expand the filesystem to make more space available, I would probably try

sudo apt-get remove linux-header-4.4.0-189-generic

If removing that wants to also remove other packages that depend upon linux-header-4.4.0-189-generic, then say "N" when prompted, to cancel the operation.

[provels]

Thanks again for your help. The remove command didn't do anything because that was the pkg that failed to install during autp-update.

While messing around creating a screen, I received a message saying the auditlog was full. I suspect the "auditlog", is a mysql log and not a server log. I did find that the default server logfile size was set to '0' (so unlimited), so I changed that to '1' and restarted. After a while the server came up normally and I was able to add a new screen w/o error.

I retried sudo apt-get install linux-headers-4.4.0-189 but it again said "not enough space in /var/cache/apt/archives/". Yet, there is currently only 28K in /var/cache/apt/archives.

Maybe this?

appliance@zabbix:/$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 470M 0 470M 0% /dev
tmpfs 99M 3.0M 96M 3% /run
/dev/mapper/zabbix--vg-root 5.1G 4.8G 2.3M 100% /
tmpfs 492M 0 492M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 492M 0 492M 0% /sys/fs/cgroup
/dev/sda1 472M 126M 323M 28% /boot

I found info on expanding a disk (https://www.geoffstratton.com/expand...isk-ubuntu-lvm) but I'll be doing more reading before I try it.

EDIT - Found this https://www.zabbix.com/forum/zabbix-...n-hyper-v-2008 (last post)

I'm sorry, Tim, but I seem to know just enough about Linux to put my foot in my mouth (and then shoot it...). But thanks again for all your help.

[tim.mooney]

appliance@zabbix:/$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 470M 0 470M 0% /dev
tmpfs 99M 3.0M 96M 3% /run
/dev/mapper/zabbix--vg-root 5.1G 4.8G 2.3M 100% /
tmpfs 492M 0 492M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 492M 0 492M 0% /sys/fs/cgroup
/dev/sda1 472M 126M 323M 28% /boot

I found info on expanding a disk (https://www.geoffstratton.com/expand...isk-ubuntu-lvm) but I'll be doing more reading before I try it.

EDIT - Found this https://www.zabbix.com/forum/zabbix-...n-hyper-v-2008 (last post)

I think those URLs have procedures that look like they'll work. I actually think @KoenDeMoel's post #2 is the best, because he uses the 'u' option with fdisk and more importantly he covers two different scenarios depending on how you added storage in your VM environment (expanded the existing virtual disk OR added a 2nd virtual disk).

Keep in mind that there are essentially 4 steps that need to happen:

1. You need to present more disk space from your virtualization environment (for you, Hyper-V) to the VM, either by growing the one existing virtual disk OR by adding a second virtual disk
2. Once the new storage has been presented and the Linux VM can see it, in this case you want to partition it. That's where the "fdisk" comes in. As @KoenDeMoel explained, if you expanded the size of the existing disk, then the easiest method is to create a new partition on that expanded disk, which because of how the appliance is partitioned already, would become partition 3 on device sda (so: /dev/sda3). Note that you run fdisk on the entire device (/dev/sda), but in this scenario you're creating the new partition /dev/sda3.
   1. if you presented an entirely separate virtual disk to the Linux VM, rather than growing the single virtual disk, then you use KoenDeMol 's 2nd fdisk procedure, starting with "If you have added a new disk to the VM". In that case, you're not partitioning the free space at the end of an expanded /dev/sda, you're instead creating the first and only partition on a separate disk /dev/sdb
3. Once the partitioning is done and you've rebooted, the next 3 commands are all related to the Logical Volume Manager (LVM):
   1. First, you use "pvcreate" to "claim" the new partition as an LVM physical volume. This effectively adds the partition to LVM's general "pool of available space". Again, which partition you claim with pvcreate depends on whether you expanded the one virtual disk or added a new, second virtual disk.
   2. Next, the "vgextend" adds the storage on the LVM physical volume you claimed in the previous step to the volume group named "zabbix-vg". On a big LVM install you may have lots of different LVM physical volumes and you may have multiple different volume groups. On this appliance there's (probably! I haven't looked) only the one volume group ("zabbix-vg"), but you still need to tell LVM that you're assigning the physical volume to that specific volume group before it can be used.
   3. The previous command added more available space to the volume group, but that doesn't automatically expand any particular logical volume, since there can be multiple logical volumes within a volume group. To actually tell a particular logical volume to expand to use some additional space in the volume group's free space, you use the "lvextend" command.
4. You might think that after doing the "lvextend" you're done, because you've grown the storage available to the volume /dev/mapper/zabbix--vg-root . With traditional Linux filesystems, though, the filesystem itself doesn't dynamically expand to use additional space that becomes available "underneath it" when the underlying volume is grown. When the filesystem was created, it was created with a particular size and on-disk structure, and that doesn't automatically change just because the logical volume it resides upon grew. To finish the job and make the filesystem use the newly available space, you have to use the "resize2fs" command (note: that originated with ext2 type filesystems, but it works with ext3 and ext4 as well).

Newer storage managers (ZFS, BtrFS) blur the distinction between the volume and the filesystem, but with LVM and ext2/ext3/ext4, the logical volume and the filesystem on top of it are two distinct entities, and you need to expand (or shrink) them each separately.

There are other details I've skipped that apply to other environments but not this appliance, but the procedure that KoenDeMol outlined is a good one.

[provels]

Thanks for the reply. It worked, but I had to add a '+' before the '100%' (which I had seen in another article about extending, but no idea what it does).
I had tried it yesterday, but I must have had something wrong. I then tried to use gparted (which extended the volume easily, but then every command I tried to extend the FS failed.

So the way I understand it, I extended the virtual disk, created a new partition in the empty space, added the new partition to the old partition that the OS sees as a single volume, then extended the FS over the whole of the new volume. Whew.

It looks like I'm all set (until next time... lol). I have my server, database, my graphs (missing a little data, but so what). I was able to manually install the Linux headers pkg w/o issue, and then ran unattended-upgrades which went fine and cleaned up some obsolete stuff. I'll keep a much closer eye on disk space from now on.

[COLOR=#c0392b][B]Filesystem Size Used Avail Use% Mounted on[/B][/COLOR]
udev 470M 0 470M 0% /dev
tmpfs 99M 1.8M 97M 2% /run
[COLOR=#c0392b][B]/dev/mapper/zabbix--vg-root 11G 4.8G 5.0G 50% /[/B][/COLOR]
tmpfs 492M 0 492M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 492M 0 492M 0% /sys/fs/cgroup
/dev/sda1 472M 126M 323M 28% /boot

Thanks again for all your help, Tim. Closing my 50 tabs for today. Drinks on me! =b

Peder

[Pfannenwender]

Hello,

if done this, and it worked!

https://medium.com/@elierherrera96/h...e-79ac2eac8279

after this: 1. Prüfe, welche MySQL- oder MariaDB-Pakete installiert sind:

bash

Kopieren


Bearbeiten


rpm -qa | grep -E 'mysql|mariadb'
2. Falls MariaDB installiert ist, probiere diesen Startbefehl:

bash

Kopieren


Bearbeiten


systemctl start mariadb
3. Falls MySQL installiert ist, versuche:

bash

Kopieren


Bearbeiten


systemctl start mysqld