Ad Widget

Collapse

Zabbix 5.0.23 - Timeouts - big times DB Queries. 502/504 errors

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • damiancancela
    Junior Member
    • May 2020
    • 14

    #1

    Zabbix 5.0.23 - Timeouts - big times DB Queries. 502/504 errors

    Hello, i have Zabbix 5.0.23 version using Server and Proxies. Also im using two isolated hosts for the Frontend (httpd 2.4 and php-fpm 7.2).

    MySQL version 8.0.15

    Since a couple of weeks im getting some strange behavior, PHP pool is dynamic and is starting to grow the amount of php processes. For what I saw, happens when frontend try to check for Problems in problem page or from Problem widget. What happens from Problems page is that sometimes users get 502 http error or from widget Problem the loading weal keeps spinning



    If i check the DB current processes im seeing queries that could be running for more than 4000 sec, same query for what i can see.



    Like:



    SELECT COUNT(DISTINCT a.alertid) AS rowscount,a.eventid FROM alerts a WHERE EXISTS (SELECT NULL FROM events e,functions f,items i,hosts_groups hgg JOIN rights r ON r.id=hgg.groupid AND r.groupid=22 WHERE a.eventid=e.eventid AND e.objectid=f.triggerid AND f.itemid=i.itemid AND i.hostid=hgg.hostid GROUP BY f.triggerid HAVING MIN(r.permission)>0 AND MAX(r.permission)>='2') AND EXISTS (SELECT NULL FROM actions aa WHERE a.actionid=aa.actionid AND aa.eventsource='0') AND (a.userid IS NULL OR EXISTS (SELECT NULL FROM users_groups ug WHERE ug.userid=a.userid AND ug.usrgrpid=22)) AND a.eventid IN (171549,194437,194454,194817,208536,208725,209682, 209705,209724,210026,213358,214582,214596,214872,2 17713,217740,217887,217904,218302,219519,234691,23 5012,235013,235064,235077,235089,235102,235114,571 682,571904,574575,574576,574865,574882,574908,5749 20,769019,1225248,1226534,1226554,1226578,1226595, 1227052,1227053,1227420,1392400,1392412,1392433,13 93034,1395466,1395484,1395504,1395522,1395596,1395 597,1744261,1744265,1745075,1745088,1745105,174512 3,1745158,1745160,1766651,1766673,1766691,1766708, 1766748,1766749,5185522,5185992,5188826,5188973,51 89014,5189118,5189317,5189318,5335130,5335132,5335 146,5335149,5335153,5335155,6332586,6332852,633296 7,6418691,6962830,6963514,6963754,6965597,6965666, 6965705,6965751,6966198,6966200,6971792,7795968,77 96210,7796698,7828035,7828036,7828743,7828828,7828 854,7828919,7932716,10228697,10228699,10228701,112 56798,11256805,11263503,11263505,11264675,11264711 ,11264747,11264775,11264801,11264873,12445280,1276 3819,14473153,14569966,14578805,14579527,16130617, 16130622,16132529,16132575,16132610,16132643,16132 685,16132716,16132759,16132787,16133016,16133058,1 6133092,16133130,16133165,16133196,16133233,161332 71,16136920,16142526,16142565,16142599,16142638,16 142672,16142706,17587540,17589846,17590024,1759020 4,17590363,18506404,18507662,18975552,19595605,200 93620,20200813,21114017,21807745,21943848,21971016 ,21976022,21976024,21976028,21976291,21977800,2205 5301,22136008,22238711,22238712,22338318,22352482, 22360609,22381539,22422079,22436726,22439318,22441 113,22449664,22452164,22453390,22456240,22457243,2 2461914,22461947,22461994,22462008,22462177,224630 04,22463568,22463813,22463861,22463919,22464139,22 464190) GROUP BY a.eventid



    These queries come from the Frontend hosts, the workaround im doing is restarting httpd and php services, that will clear the queries but eventually the same situation will happen again.



    Nothing has been changed, maybe more users are using the tool but not more than that. Strange that only happens with users with User or Admin privileges, SuperAdmin looks good.


    Actually this problem started to happen some weeks ago with version 5.0.7, investigating i found a thread (similar to my issue) that was fixed in v 5.0.15 https://support.zabbix.com/browse/ZBX-16822

    For that reason ive upgraded into latest 5.0.23 but issue is still there.




    Do you know what could be happening? attaching a screenshot from MySQL workbench, you can see there that is same query running from different PHP connections.

    Click image for larger version  Name:	MySQL Workbench.png Views:	0 Size:	78.2 KB ID:	446139



    Thanks
  • damiancancela
    Junior Member
    • May 2020
    • 14

    #2
    PHP conf if helps

    [zabbix]
    user = apache
    group = apache

    listen = /var/opt/rh/rh-php72/run/php-fpm/zabbix.sock
    listen.acl_users = apache
    listen.allowed_clients = 127.0.0.1

    pm = dynamic
    pm.max_children = 50
    pm.start_servers = 5
    pm.min_spare_servers = 5
    pm.max_spare_servers = 35
    pm.max_requests = 200

    php_value[session.save_handler] = files
    php_value[session.save_path] = /var/opt/rh/rh-php72/lib/php/session/

    php_value[max_execution_time] = 300
    php_value[memory_limit] = 128M
    php_value[post_max_size] = 16M
    php_value[upload_max_filesize] = 2M
    php_value[max_input_time] = 300
    php_value[max_input_vars] = 10000
    ; php_value[date.timezone] = Europe/Riga

    Comment

    • damiancancela
      Junior Member
      • May 2020
      • 14

      #3
      Hello, were you able to see this? Someone had this issue or any idea?

      Thank you

      Comment

      • tim.mooney
        Senior Member
        • Dec 2012
        • 1427

        #4
        I haven't experienced this issue at my site, that I know of. We're still on Zabbix 5.0.20, but I'll probably upgrade us to 5.0.24 soon. Like you, I'm also using rh-php72 (on RHEL 7.9), but I'm using rh-mariadb102-mariadb-server-10.2.33 as the database backend, not MySQL 8.0. Did your MySQL packages come from mysql.com, or are you getting them from RedHat/CentOS/Oracle/etc. ? The Red Hat MySQL 8.0 package (rh-mysql80-mysql-server) is version 8.0.26, so if you're using packages from your distro vendor, you might be a few versions behind. That may be worth looking into.

        It might be worth creating a bug report in support.zabbix.com, explaining that you're still seeing query issues with this query and referencing the closed ticket.

        You don't mention if you've done any MySQL database tuning. How much RAM does your Zabbix + MySQL server have, and how much have you dedicated to the InnoDB buffer pool?

        Comment

        • damiancancela
          Junior Member
          • May 2020
          • 14

          #5
          Originally posted by tim.mooney
          I haven't experienced this issue at my site, that I know of. We're still on Zabbix 5.0.20, but I'll probably upgrade us to 5.0.24 soon. Like you, I'm also using rh-php72 (on RHEL 7.9), but I'm using rh-mariadb102-mariadb-server-10.2.33 as the database backend, not MySQL 8.0. Did your MySQL packages come from mysql.com, or are you getting them from RedHat/CentOS/Oracle/etc. ? The Red Hat MySQL 8.0 package (rh-mysql80-mysql-server) is version 8.0.26, so if you're using packages from your distro vendor, you might be a few versions behind. That may be worth looking into.

          It might be worth creating a bug report in support.zabbix.com, explaining that you're still seeing query issues with this query and referencing the closed ticket.

          You don't mention if you've done any MySQL database tuning. How much RAM does your Zabbix + MySQL server have, and how much have you dedicated to the InnoDB buffer pool?
          Hi, actually the environment is in AWS, for that reason we are using DB RDS AWS service for the MySQL, with MySQL 8.0.15 engine version. Instance class db.r5.4xlarge which give 16vCPU and 128GB RAM. Database size 70 GB.
          Unfortunately im not experience with DBs so there is no MySQL tunning. Do you think could be related? Because is strange, normally all queries are processed fast but specifically these ones reach to 5000 time value (only workaround is kill the queries and restart php services)

          Zabbix Server/Proxies/Frontend services were installed using zabbix RPMs from RHEL 7 https://repo.zabbix.com/zabbix/5.0/rhel/7/x86_64/


          Thanks in advance, I've submitted an issue https://support.zabbix.com/browse/ZBX-21222

          Comment

          Working...