Ad Widget

Collapse

Terrible DB performance after upgrade 2.0.9 -> 2.2.1. Zabbix unusable

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • chojin
    Member
    Zabbix Certified Specialist
    • Jul 2011
    • 64

    #1

    Terrible DB performance after upgrade 2.0.9 -> 2.2.1. Zabbix unusable

    I was running Zabbix 2.0.9 with current stats:
    Code:
    Number of hosts (monitored/not monitored/templates)	963	854 / 48 / 61
    Number of items (monitored/disabled/not supported)	47135	42624 / 2024 / 2487
    Number of triggers (enabled/disabled) [problem/ok]	30999	25519 / 5480 [623 / 24896]
    Required server performance, new values per second	641.24	-
    Zabbix server and the frontend are running on a Vmware virtual machine and the Percona MySQL database is running on a seperate 4-core 2Ghz server with 4Gb of RAM. The database files are located on Flash disks in a fibre connected SAN..

    When using v2.0.9 this al worked correctly. MySQL running between 50 and 80% cpu usage and performing about 250 IOPS on the database diskpool with peaks of max 550 IOPS.

    After the upgrade to 2.2.1, the MySQL process now consumes between 250 to 400% CPU usage (hence 100% on all cores) and we now have 4500 - 5500 IOPS in the database diskpool..

    And even with that zabbix servers starts complaining about transaction locks timing out and the value cache being fully used (meanwhile adjusted to 1GB size, but still fully used). All active agents complain about timeouts connecting to the server and even the PHP frontend is only able to connect to the zabbix server process once every few minutes..

    bottom line.. zabbix is currently completely unusable..

    Did something go wrong during the upgrade? I can't imagine that it is normal that the database activity is increased 5 to 6 times just by upgrading .. ?
  • chojin
    Member
    Zabbix Certified Specialist
    • Jul 2011
    • 64

    #2
    This is a graph from our SAN displaying the IOPS on the database flashdiskpool, before and after the upgrade:
    Attached Files

    Comment

    • chojin
      Member
      Zabbix Certified Specialist
      • Jul 2011
      • 64

      #3
      I have been able to solve this problem by
      - upgrading the database server hardware to 2 dual-core CPU's 3Ghz and 32 GB RAM ..
      - complete table optimization / db rebuild
      - upgrade Percona mysql 5.5 -> 5.6
      this way the history sync processes where able to complete after about half a minute (still with all CPU's to 100% and still a few lock timeouts) and now the IOPS usage seems to be better than before the upgrade.. and zabbix is working normally again.

      Comment

      • Colttt
        Senior Member
        Zabbix Certified Specialist
        • Mar 2009
        • 878

        #4
        thanks for this!

        but my opinion is that is not the reason why going the IOPS so high..

        maybe create a bug-report? (possibly there is a wrong SQL-query?)
        Debian-User

        Sorry for my bad english

        Comment

        • wrocha
          Junior Member
          • Aug 2013
          • 27

          #5
          It looks like a problem over snmp dynamic queries.
          Look at this post. I snnifed the traffic with tcpdump and every querie zabbix does with dynamic indexes it tries to build the whole snmp OID table.

          Comment

          Working...