Zabbix database is not available.

[Maissa DRIDI]

Hello,

I'm currently facing an issue with my Zabbix Database and. I'm encountering the following error:

"MySQL database "zabbix" on "localhost" is not available: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'"



Here are some details about my setup:

- Zabbix version: 6.4.7
- MySQL version: Ver 8.0.35-0ubuntu0.22.04.1 for Linux on x86_64
- Operating System: Ubuntu 22.04.3 LTS

I have already tried the following steps:

1. Checked MySQL service status: active
2. Verified MySQL socket path and configuration.
3. Restarted MySQL service
4. Checked MySQL user permissions for Zabbix.
5. Reviewed firewall settings for MySQL port.
6. Ensured there is enough disk space on the server.

[Vermizz]

Hi,
you can connect to the database using the credentials zabbix uses from file: /etc/zabbix/zabbix_server.conf ? Check this value in file:
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=<password>

Also check the information in the logs:
/var/log/mysql/<something>
/var/log/syslog
/var/log/zabbix/zabbix_serverd.log

[Maissa DRIDI]

Hi,

First thank you for your help,

So i checked the database configuration in /etc/zabbix/zabbix_server.conf and i connect to the database using DBUser=zabbix and DBPassword=<password> and i successfully logged in,

Also i checked the log file in /var/log/mysql/error.log it was empty
in the log file /var/log/syslog i didn't find any error relative to the mysql
but in the zabbix server log /var/log/zabbix/zabbix_server.log i found this warning "WARNING: MYSQL_OPT_RECONNECT is deprecated and will be removed in a future version"

[MRedbourne]

Have you checked your AVC denials (if you use SE Linux/AppAmor)? I'm not an Ubuntu person, however I believe that Ubuntu uses app armor. From the documentation by SUSE, it looks like you can use ausearch (same as RHEL/CentOS). audit.log file that has the apparmor (AVC) entries that ausearch can't read. | Support | SUSE

Execute these two commands:

systemctl restart zabbix-server
ausearch -m avc -ts recent

Is there anything present? Maybe grep /var/log/audit for "mysqld" if ausearch doesn't return anything.

cat /var/log/audit | egrep -i "mysqld|zabbix".

If that returns results, look up your equivalent to audit2allow. It's a tool that will analyze SE Linux issues and create new policies to allow them that you can import.

[Vermizz]

Hi,
Like a MRedbourne say check apparmor. For test you can stop it "systemctl stop apparmor" and restart zabbix-server.

Does the file /var/run/mysqld/mysqld.sock exist in the location: /var/run/mysqld/ ?
If yes check if the file: /etc/zabbix/zabbix_server.conf contains the correct parameter for: DBSocket=

Please send information about what the command returns:
ps aux |grep mysql
and
ls -al /var/run/mysqld/

Did you install zabbix server from the official manual? https://www.zabbix.com/download?zabb...l&ws=apache

[Maissa DRIDI]

Hello,

First i disable the policy related to the mysql and i still get the same error. So i stopped the "apparmor" and restart the zabbix server and nothing has changed the error keep showing up every 15 min.


Yes the file /var/run/mysqld/mysqld.sock exist and i did configure the DBSocket=/var/run/mysqld/mysqld.sock in /etc/zabbix/zabbix_server.conf but nothing has changed.



tha's the output for ps aux |grep mysql :

mysql 952 6.4 9.2 2894220 750528 ? Ssl 11:20 8:42 /usr/sbin/mysqld
root 3266 0.0 0.0 6608 2292 pts/1 S+ 13:34 0:00 grep --color=auto mysql


tha's the output for ls -al /var/run/mysqld/ :
total 8
drwxr-xr-x 2 mysql mysql 120 Nov 20 11:20 .
drwxr-xr-x 35 root root 980 Nov 20 13:09 ..
srwxrwxrwx 1 mysql mysql 0 Nov 20 11:20 mysqld.sock
-rw------- 1 mysql mysql 4 Nov 20 11:20 mysqld.sock.lock
srwxrwxrwx 1 mysql mysql 0 Nov 20 11:20 mysqlx.sock
-rw------- 1 mysql mysql 4 Nov 20 11:20 mysqlx.sock.lock


yes i did install it from the official manual.

[MRedbourne]

You shouldn't need to define the socket if it's in a default location. We don't define a specific socket location on either of our Zabbix instances (we're running a cluster of 6.4.8 servers and another 6.2 server that's standalone).

You mentioned stopping the apparmor service. However, did you actually check ausearch as indicated? It makes little sense that you could connect to MySQL locally using your shell, but Zabbix can't do it using it's user. Usually this is a result of AppArmor/SELinux blocking the access because it violates AVC. I'm assuming your environment is a single Zabbix Server (Non-HA) running the DB locally on the server?

[Maissa DRIDI]

Hi, and thank you for your help,


So I did change the config file and comment the socket directory (as it was before) and followed this steps:

1. check the apparmor service: it was stopped
2. restart zabbix server
3. check the ausearch by using this command "ausearch -m avc -ts recent" : the output was "<no matches>"
4. i checked the log file for the audit by using this command "cat /var/log/audit/audit.log | egrep -i "mysqld|zabbix" " : the output was

type=SERVICE_STOP msg=audit(1700479180.433:77): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-web-service comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_STOP msg=audit(1700479180.437:79): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-agent comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_STOP msg=audit(1700479189.077:92): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1700479228.493:43): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-web-service comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1700479228.665:62): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-agent comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1700479231.277:68): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_STOP msg=audit(1700485923.534:155): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1700485923.678:156): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=USER_CMD msg=audit(1700485999.781:158): pid=3006 uid=0 auid=0 ses=8 subj=unconfined msg='cwd="/run/mysqld" cmd=6E616E6F202F6574632F7A61626269782F7A6162626978 5F7365727665722E636F6E66 exe="/usr/bin/sudo" terminal=pts/1 res=success'UID="root" AUID="root"
type=SERVICE_STOP msg=audit(1700486386.244:169): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1700486386.388:170): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_STOP msg=audit(1700551870.858:638): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=SERVICE_START msg=audit(1700551870.990:639): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=zabbix-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"

[Maissa DRIDI]

and yes my environment is a single Zabbix Server, running the DB locally on the server

[MRedbourne]

That looks OK; no AVC denials.

Re: the "MYSQL_OPT_RECONNECT" alert you're seeing in logs. This is known to Zabbix developers and is tracked under ZBX-23145 and has been observed with Zabbix 5.0, 6.0, 6.4 and 7.0 (beta). I haven't encountered it yet, but we also have a managed MySQL instance in Azure that hosts our DB. We don't track minor versions of it. It shouldn't be impacting service function right now.

When you created the Zabbix DB and the associated Zabbix account it uses, when you created the credentials for it, which command did you use? Did you copy and paste them? If you run these two commands on the MySQL instances as root, what do you get back?

select Host,User,Update_Priv,Insert_Priv from mysql.user;
show grants for '%username%'@'localhost';

Notes:
- %username% should be replaced with your DB username in use by the Zabbix Configuration.
- localhost should be replaced by the indicated host in the first command results for the specified user.

Here's what my 6.4.8 DB looks like in Azure (with some information removed, like other users, subnets in use, etc. Note that because our DB is a hosted solution (and/or lives on a different server), we specify the locations the credentials can be used from. You should be using 'localhost' if the connections originates locally.



What happens if you run this command on your regular bash console? time tcpdump -i any port 3306

Let that command run for a few minutes and post a screenshot of the MySQL results (please scrub any sensitive information) and the tcpdump command.