Ad Widget

Collapse

Mysql process consuming high cpu after server crash

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • arsalan.zahoor
    Junior Member
    • Apr 2023
    • 5

    #1

    Mysql process consuming high cpu after server crash

    Hi Team,

    we have 2x Zabbix server running in HA via crm for many years but few days back due to RAM crunch server got crashed. This made things messy for us now mysql is continously consuming high cpu and when ever we take mysqldump the zabbix server process goes down. Need your support to figure this out i have checked the processlist and its stuck in history tablel update i guess.


    Server version: 5.5.60-MariaDB MariaDB Server
    zabbix_server --version
    zabbix_server (Zabbix) 3.0.22
    Revision 84876 14 September 2018, compilation time: Sep 14 2018 07:34:13



    top - 11:37:16 up 8 days, 19:52, 1 user, load average: 10.49, 11.29, 11.82
    Tasks: 445 total, 5 running, 440 sleeping, 0 stopped, 0 zombie
    %Cpu(s): 74.3 us, 4.2 sy, 0.0 ni, 20.9 id, 0.2 wa, 0.0 hi, 0.4 si, 0.0 st
    KiB Mem : 28893100 total, 617948 free, 17739936 used, 10535216 buff/cache
    KiB Swap: 4190204 total, 4122620 free, 67584 used. 9090916 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    4902 mysql 20 0 19.4g 14.2g 9036 S 750.0 51.4 56527:47 mysqld
    22333 zabbix 20 0 3019252 103880 60960 S 0.3 0.4 4:25.38 zabbix_server
    22334 zabbix 20 0 3019128 103688 60996 S 0.0 0.4 4:23.19 zabbix_server
    22329 zabbix 20 0 3018868 103444 60932 S 0.0 0.4 4:23.40 zabbix_server
    22322 zabbix 20 0 3017408 100896 59864 S 0.0 0.3 4:20.26 zabbix_server
    5224 root rt 0 189688 93472 70796 S 0.7 0.3 90:54.48 corosync



    | 2041003 | zabbix | localhost | zabbix | Query | 15181 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2043287 | zabbix | localhost | zabbix | Query | 14528 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2045543 | zabbix | localhost | zabbix | Query | 14152 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2047089 | zabbix | localhost | zabbix | Query | 13786 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2048369 | zabbix | localhost | zabbix | Query | 13409 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2050568 | zabbix | localhost | zabbix | Query | 12761 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2052824 | zabbix | localhost | zabbix | Query | 12394 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2054335 | zabbix | localhost | zabbix | Query | 12029 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2056896 | zabbix | localhost | zabbix | Query | 11435 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2058605 | zabbix | localhost | zabbix | Query | 11048 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2059223 | zabbix | localhost | zabbix | Query | 10676 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2060900 | zabbix | localhost | zabbix | Query | 10292 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
    | 2062773 | zabbix | localhost | zabbix | Query | 9901 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |​
  • arsalan.zahoor
    Junior Member
    • Apr 2023
    • 5

    #2
    MariaDB [(none)]> SELECT TABLE_SCHEMA, TABLE_NAME,(INDEX_LENGTH+DATA_LENGTH)/(1024*1024) AS SIZE_MB, TABLE_ROWS FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA NOT IN ("mysql", "information_schema") ORDER BY SIZE_MB DESC LIMIT 10;
    +--------------+--------------+-------------+------------+
    | TABLE_SCHEMA | TABLE_NAME | SIZE_MB | TABLE_ROWS |
    +--------------+--------------+-------------+------------+
    | zabbix | history_uint | 212552.2031 | 2226280917 |
    | zabbix | history | 31372.2813 | 379889501 |
    | zabbix | history_log | 11833.0469 | 26884077 |
    | zabbix | trends_uint | 4288.3125 | 57214829 |
    | zabbix | history_str | 1778.0781 | 22601337 |
    | zabbix | trends | 1022.2813 | 12898724 |
    | zabbix | alerts | 223.1406 | 462513 |
    | zabbix | events | 215.0469 | 1042869 |
    | zabbix | items | 39.6250 | 69808 |
    | zabbix | auditlog | 19.0469 | 120969 |
    +--------------+--------------+-------------+------------+​


    cat /etc/my.cnf
    [mysqld]
    datadir=/var/lib/mysql
    socket=/var/lib/mysql/mysql.sock
    # Disabling symbolic-links is recommended to prevent assorted security risks
    symbolic-links=0
    # Settings user and group are ignored when systemd is used.
    # If you need to run mysqld under a different user or group,
    # customize your systemd unit file for mariadb according to the
    # instructions in http://fedoraproject.org/wiki/Systemd

    user=mysql
    log-bin=/zabbix/log/mysql/bin/mysql-bin
    expire_logs_days=7
    max_connections = 1500
    innodb_buffer_pool_size=12G
    innodb_thread_concurrency=8
    innodb_log_file_size=256M
    innodb_file_per_table
    innodb_log_buffer_size=8388608
    innodb_io_capacity=500
    innodb_flush_method=O_DIRECT
    innodb_buffer_pool_instances=4
    thread_cache_size=4

    [mysqld_safe]
    log-error=/zabbix/log/mysql/error/mysqld.log
    pid-file=/var/run/mysqld/mysqld.pid

    #
    # include all files from the config directory
    #
    !includedir /etc/my.cnf.d

    Comment

    • arsalan.zahoor
      Junior Member
      • Apr 2023
      • 5

      #3
      Hi Guys, your support is requested i am new with zabbix and stuck with this.

      Comment

      • tim.mooney
        Senior Member
        • Dec 2012
        • 1427

        #4
        From what you've described so far, this isn't a Zabbix problem, it's a mysql/MariaDB problem.

        Have you enabled mysql logging? If so, have you looked at mysql's logs? Is there a chance you have some lingering corruption after the crash?

        Comment

        • arsalan.zahoor
          Junior Member
          • Apr 2023
          • 5

          #5
          Hi Tim,

          Thank you for the response i was not able to find any thing in the DB logs and I was able to recover zabbix server but mysqld proccess is consuming high CPU. In the mysql process list i can find 29 following queries which are stuck and waiting for something. I have checked the item.id it does not exist in the history table i have doubt on housekeeper. Is there any way i can stop this query and manually drop the data in the history table with the same id.

          | 2062773 | zabbix | localhost | zabbix | Query | 9901 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |​

          Thanks
          Arsalan

          Comment

          • tim.mooney
            Senior Member
            • Dec 2012
            • 1427

            #6
            Even with nothing logged, this still really sounds like some kind of database corruption.

            You should have a command called "mysqlcheck", if it's not installed then there may be a 'mariadb-utils' or 'mariadb-server-utils' or similar package that you need to install.

            Read the man page for 'mysqlcheck' and then run it in "extended" check mode against either your 'zabbix' database or all databases. It's basically just doing "CHECK TABLE <tablename here>" in a loop, but it's a convenient way to check *all* your tables.

            If any of the tables do not report "OK", then look at the "repair" option to mysqlcheck.

            Let us know what the check operation reports.

            Comment

            • arsalan.zahoor
              Junior Member
              • Apr 2023
              • 5

              #7
              I will run mysql check today and will post the output Thanks

              Comment

              Working...