Ad Widget

Collapse

stress test - slow queries

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • gde
    Junior Member
    Zabbix Certified Specialist
    • Mar 2011
    • 5

    #1

    stress test - slow queries

    Hello,

    We are in a phase of stress testing zabbix in our environnement. We're doing it in particular way by creating bogus hosts/items/triggers just made for that (we've created 1000 hosts with 100 items and 200 triggers each). All in all, here's what looks like :

    Number of hosts 1062
    Number of items 100159
    Number of triggers 200089
    we're testing by using a script, on the zabbix server, that loops around zabbix_sender. The script launch a zabbix_sender for a certain item, puts a random value (float), and goes on the next item of the same host. Once every items, of a single host, have received a value, the script moves on to the next host. It does so until a fixed time given in paramater.

    We've noticed that by simply doing a test on the first 500 hosts for 5 minutes, we get a bunch of slow queries. here's an excerpt :

    27106:20110926:120928.425 Slow query: 120.509592 sec, "update ids set nextid=nextid+256 where nodeid=0 and table_name='events' and field_name='eventid'"
    27101:20110926:120928.498 Slow query: 66.452517 sec, "update ids set nextid=nextid+256 where nodeid=0 and table_name='events' and field_name='eventid'"
    27107:20110926:120931.416 Slow query: 62.458832 sec, "update ids set nextid=nextid+256 where nodeid=0 and table_name='events' and field_name='eventid'"
    27123:20110926:120934.875 Slow query: 108.504381 sec, "update ids set nextid=nextid+256 where nodeid=0 and table_name='events' and field_name='eventid'"
    our database is a postgresql 9.0.4 hosted on a seperate machine from zabbix server.

    strangely, this doesn't happen when we test the first 100 hosts.

    These results have an impact on zabbix. From the frontend, we see a queue for items that are supposed to be checked by zabbix_agent (or internal) that gets ever increasing until the script stop.
    We've also noticed that once the queries are done (which happens a couple minutes after the end of the test, obviously) all graphs get updated.

    we've noticed this bug report, but it's marked as fixed as of version 1.8.3 (we're using 1.8.6) and there aren't any information on how to reproduce it or how it was fixed (and what caused the problem).

    has anybody run into this problem or have any information on that old bug that might explain the slow queries ?

    thanks
  • xsbr
    Junior Member
    Zabbix Certified Specialist
    • Oct 2009
    • 25

    #2
    Originally posted by gde
    Hello,

    We are in a phase of stress testing zabbix in our environnement. We're doing it in particular way by creating bogus hosts/items/triggers just made for that (we've created 1000 hosts with 100 items and 200 triggers each). All in all, here's what looks like :



    we're testing by using a script, on the zabbix server, that loops around zabbix_sender. The script launch a zabbix_sender for a certain item, puts a random value (float), and goes on the next item of the same host. Once every items, of a single host, have received a value, the script moves on to the next host. It does so until a fixed time given in paramater.

    We've noticed that by simply doing a test on the first 500 hosts for 5 minutes, we get a bunch of slow queries. here's an excerpt :



    our database is a postgresql 9.0.4 hosted on a seperate machine from zabbix server.

    strangely, this doesn't happen when we test the first 100 hosts.

    These results have an impact on zabbix. From the frontend, we see a queue for items that are supposed to be checked by zabbix_agent (or internal) that gets ever increasing until the script stop.
    We've also noticed that once the queries are done (which happens a couple minutes after the end of the test, obviously) all graphs get updated.

    we've noticed this bug report, but it's marked as fixed as of version 1.8.3 (we're using 1.8.6) and there aren't any information on how to reproduce it or how it was fixed (and what caused the problem).

    has anybody run into this problem or have any information on that old bug that might explain the slow queries ?

    thanks
    It's really an annoying problem. Like a workaround, I patched the Zabbix Server to avoid to get the next EventID from ID table. So I alter Events table to AUTO_INCREMENT. This solve my problem.

    Comment

    • gde
      Junior Member
      Zabbix Certified Specialist
      • Mar 2011
      • 5

      #3
      Hey xsbr !

      Thanks for your reply! I was thinking it was a bug, but I didn't want to believe it.

      You seem to have encountered it a while ago, this is quite odd it hasn't been fixed yet. I'm wondering if the fixed tag on that bug report doesn't mean it has been fixed for 2.0 only…

      Anyway, Have you considered posting/releasing your patch ? has zabbix is being used in larger and larger environnements, I think the community would benefit greatly from it.

      Comment

      • frankymryao
        Member
        • Oct 2011
        • 52

        #4
        Hi buddy, we once faced the same question and we solved it by modify the source code of zabbix 1.8.8. I think this thread is useful for you.

        Comment

        • Rawlings
          Junior Member
          • Sep 2012
          • 24

          #5
          Hello

          how do you make your stress test ?

          kind Regards

          Comment

          Working...