Hi Team,
we have 2x Zabbix server running in HA via crm for many years but few days back due to RAM crunch server got crashed. This made things messy for us now mysql is continously consuming high cpu and when ever we take mysqldump the zabbix server process goes down. Need your support to figure this out i have checked the processlist and its stuck in history tablel update i guess.
Server version: 5.5.60-MariaDB MariaDB Server
zabbix_server --version
zabbix_server (Zabbix) 3.0.22
Revision 84876 14 September 2018, compilation time: Sep 14 2018 07:34:13
top - 11:37:16 up 8 days, 19:52, 1 user, load average: 10.49, 11.29, 11.82
Tasks: 445 total, 5 running, 440 sleeping, 0 stopped, 0 zombie
%Cpu(s): 74.3 us, 4.2 sy, 0.0 ni, 20.9 id, 0.2 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 28893100 total, 617948 free, 17739936 used, 10535216 buff/cache
KiB Swap: 4190204 total, 4122620 free, 67584 used. 9090916 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4902 mysql 20 0 19.4g 14.2g 9036 S 750.0 51.4 56527:47 mysqld
22333 zabbix 20 0 3019252 103880 60960 S 0.3 0.4 4:25.38 zabbix_server
22334 zabbix 20 0 3019128 103688 60996 S 0.0 0.4 4:23.19 zabbix_server
22329 zabbix 20 0 3018868 103444 60932 S 0.0 0.4 4:23.40 zabbix_server
22322 zabbix 20 0 3017408 100896 59864 S 0.0 0.3 4:20.26 zabbix_server
5224 root rt 0 189688 93472 70796 S 0.7 0.3 90:54.48 corosync
| 2041003 | zabbix | localhost | zabbix | Query | 15181 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2043287 | zabbix | localhost | zabbix | Query | 14528 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2045543 | zabbix | localhost | zabbix | Query | 14152 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2047089 | zabbix | localhost | zabbix | Query | 13786 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2048369 | zabbix | localhost | zabbix | Query | 13409 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2050568 | zabbix | localhost | zabbix | Query | 12761 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2052824 | zabbix | localhost | zabbix | Query | 12394 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2054335 | zabbix | localhost | zabbix | Query | 12029 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2056896 | zabbix | localhost | zabbix | Query | 11435 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2058605 | zabbix | localhost | zabbix | Query | 11048 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2059223 | zabbix | localhost | zabbix | Query | 10676 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2060900 | zabbix | localhost | zabbix | Query | 10292 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2062773 | zabbix | localhost | zabbix | Query | 9901 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
we have 2x Zabbix server running in HA via crm for many years but few days back due to RAM crunch server got crashed. This made things messy for us now mysql is continously consuming high cpu and when ever we take mysqldump the zabbix server process goes down. Need your support to figure this out i have checked the processlist and its stuck in history tablel update i guess.
Server version: 5.5.60-MariaDB MariaDB Server
zabbix_server --version
zabbix_server (Zabbix) 3.0.22
Revision 84876 14 September 2018, compilation time: Sep 14 2018 07:34:13
top - 11:37:16 up 8 days, 19:52, 1 user, load average: 10.49, 11.29, 11.82
Tasks: 445 total, 5 running, 440 sleeping, 0 stopped, 0 zombie
%Cpu(s): 74.3 us, 4.2 sy, 0.0 ni, 20.9 id, 0.2 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 28893100 total, 617948 free, 17739936 used, 10535216 buff/cache
KiB Swap: 4190204 total, 4122620 free, 67584 used. 9090916 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4902 mysql 20 0 19.4g 14.2g 9036 S 750.0 51.4 56527:47 mysqld
22333 zabbix 20 0 3019252 103880 60960 S 0.3 0.4 4:25.38 zabbix_server
22334 zabbix 20 0 3019128 103688 60996 S 0.0 0.4 4:23.19 zabbix_server
22329 zabbix 20 0 3018868 103444 60932 S 0.0 0.4 4:23.40 zabbix_server
22322 zabbix 20 0 3017408 100896 59864 S 0.0 0.3 4:20.26 zabbix_server
5224 root rt 0 189688 93472 70796 S 0.7 0.3 90:54.48 corosync
| 2041003 | zabbix | localhost | zabbix | Query | 15181 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2043287 | zabbix | localhost | zabbix | Query | 14528 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2045543 | zabbix | localhost | zabbix | Query | 14152 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2047089 | zabbix | localhost | zabbix | Query | 13786 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2048369 | zabbix | localhost | zabbix | Query | 13409 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2050568 | zabbix | localhost | zabbix | Query | 12761 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2052824 | zabbix | localhost | zabbix | Query | 12394 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2054335 | zabbix | localhost | zabbix | Query | 12029 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2056896 | zabbix | localhost | zabbix | Query | 11435 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2058605 | zabbix | localhost | zabbix | Query | 11048 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2059223 | zabbix | localhost | zabbix | Query | 10676 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2060900 | zabbix | localhost | zabbix | Query | 10292 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
| 2062773 | zabbix | localhost | zabbix | Query | 9901 | Sorting result | SELECT h.* FROM history h WHERE (h.itemid BETWEEN '27919' AND '27923' OR h.itemid BETWEEN '27968' AN | 0.000 |
Comment