Ad Widget

Collapse

Zabbix history syncer process high: zbx_vc_add_value():FAIL

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • waydeechen
    Junior Member
    • Jun 2014
    • 16

    #1

    Zabbix history syncer process high: zbx_vc_add_value():FAIL

    one day, after zabbix server restarted, zabbix history process and time process became very high, and there are lots of zbx_vc_add_value():FAIL in the debug log






    10450:20150722:140627.773 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.773 In zbx_vc_add_value() itemid:6304788 value_type:0 timestamp:1437544913.379362204
    10450:20150722:140627.773 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.773 In zbx_vc_add_value() itemid:11936897 value_type:3 timestamp:1437544912.384566333
    10450:20150722:140627.773 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.773 In zbx_vc_add_value() itemid:12618248 value_type:0 timestamp:1437544910.372289360
    10450:20150722:140627.773 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.773 In zbx_vc_add_value() itemid:11714611 value_type:3 timestamp:1437544910.373642337
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:2775087 value_type:3 timestamp:1437544910.384277682
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714615 value_type:0 timestamp:1437544910.384306786
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714616 value_type:0 timestamp:1437544910.384321232
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714617 value_type:0 timestamp:1437544910.384335106
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714618 value_type:3 timestamp:1437544910.384418332
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714619 value_type:0 timestamp:1437544910.384441559
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714620 value_type:0 timestamp:1437544910.384447919
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714621 value_type:0 timestamp:1437544910.384453858
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.774 In zbx_vc_add_value() itemid:11714622 value_type:0 timestamp:1437544910.384459886
    10450:20150722:140627.774 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:11714623 value_type:0 timestamp:1437544910.384466053
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:11714624 value_type:0 timestamp:1437544910.384471481
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:11714625 value_type:0 timestamp:1437544910.384477675
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:11714628 value_type:3 timestamp:1437544910.384490124
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:11714629 value_type:0 timestamp:1437544910.384497866
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:12346148 value_type:0 timestamp:1437544910.384512878
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:12346149 value_type:0 timestamp:1437544910.384526983
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:12346150 value_type:0 timestamp:1437544910.384538898
    10450:20150722:140627.775 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.775 In zbx_vc_add_value() itemid:12346151 value_type:0 timestamp:1437544910.384550352
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346152 value_type:0 timestamp:1437544910.384561889
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346153 value_type:0 timestamp:1437544910.384573627
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346154 value_type:0 timestamp:1437544910.384585913
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346155 value_type:3 timestamp:1437544910.384593363
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346162 value_type:0 timestamp:1437544910.384600973
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346176 value_type:3 timestamp:1437544910.384608075
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346156 value_type:3 timestamp:1437544910.384615849
  • snariom38
    Junior Member
    • Apr 2014
    • 21

    #2
    same problem

    Originally posted by waydeechen
    one day, after zabbix server restarted, zabbix history process and time process became very high, and there are lots of zbx_vc_add_value():FAIL in the debug log

    timestamp:1437544910.384585913
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346155 value_type:3 timestamp:1437544910.384593363
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346162 value_type:0 timestamp:1437544910.384600973
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346176 value_type:3 timestamp:1437544910.384608075
    10450:20150722:140627.776 End of zbx_vc_add_value():FAIL
    10450:20150722:140627.776 In zbx_vc_add_value() itemid:12346156 value_type:3 timestamp:1437544910.384615849
    I have the same problem, anyone found a solution ?

    I forgot to say this problem is appeared after an upgrade to version 2.2 -> 2.4 -> 3.2.

    Maybe I've made a little mistake, because at the end of the migration the setup documentation indicate "to wait before restart server during the database upgrade" and unfortunetly I have restarted immediately after the update.
    But after the restart I must wait 10 minutes before zabbix was operational, and in the same time i saw some extra activity on mysql, then I supposed it was the database upgrade. And only after that i could access zabbix frontend...but now I have this error message
    Last edited by snariom38; 02-02-2017, 15:35. Reason: lack of information

    Comment

    • batchenr
      Senior Member
      • Sep 2016
      • 440

      #3
      Originally posted by snariom38
      I have the same problem, anyone found a solution ?
      try to run this :
      mysql -e "select itemid,value_type,name,key_ from zabbixdb.items where itemid='ITEMVALUE';"
      change ITEMID and zabbixdb to you zabbix db name
      and in the ITEMVALUE put this item id you see at the log and put the info here.

      did you try to fix db ? restart ?

      Comment

      • snariom38
        Junior Member
        • Apr 2014
        • 21

        #4
        thank you for your help @batchenr

        the result of your command :
        +--------+------------+--------------------------------+-----------------------+
        | itemid | value_type | name | key_ |
        +--------+------------+--------------------------------+-----------------------+
        | 23760 | 0 | Free inodes on $1 (percentage) | vfs.fs.inode[/,pfree] |
        | 31410 | 0 | Switch59-trk(45,47)Out | trk1Out |
        | 41460 | 3 | Used disk space on $1 | vfs.fs.size[C:,used] |
        +--------+------------+--------------------------------+-----------------------+

        for the itemid 31410, i've an idea about the problem but for 23760 and 41460 I think it's an autodiscovery rules who can not access vmware

        Comment

        • batchenr
          Senior Member
          • Sep 2016
          • 440

          #5
          Originally posted by snariom38
          thank you for your help @batchenr

          the result of your command :
          +--------+------------+--------------------------------+-----------------------+
          | itemid | value_type | name | key_ |
          +--------+------------+--------------------------------+-----------------------+
          | 23760 | 0 | Free inodes on $1 (percentage) | vfs.fs.inode[/,pfree] |
          | 31410 | 0 | Switch59-trk(45,47)Out | trk1Out |
          | 41460 | 3 | Used disk space on $1 | vfs.fs.size[C:,used] |
          +--------+------------+--------------------------------+-----------------------+

          for the itemid 31410, i've an idea about the problem but for 23760 and 41460 I think it's an autodiscovery rules who can not access vmware
          if you see thous items as unsupported at the zabbix then disable them

          Comment

          • kloczek
            Senior Member
            • Jun 2006
            • 1771

            #6
            From source code:
            Code:
             * Function: zbx_vc_add_value                                                 *
             *                                                                            *
             * Purpose: adds an item value to the value cache                             *
             * Parameters: itemid     - [IN] the item id                                  *
             *             value_type - [IN] the value type (see ITEM_VALUE_TYPE_* defs)  *
             *             timestamp  - [IN] the value timestamp                          *
             *             value      - [IN] the value to add                             *
             *                                                                            *
             * Return value:  SUCCEED - the item value was added successfully             *
             *                FAIL    - failed to add item value to cache (not fatal      *
             *                          failure - cache might be in low memory mode)      *
            If this function fails it means that value cache free space is in low state.
            Usually it means that performance of inserting new values to DB engine is not enough.
            You must improve (write) performance of your DB engine.
            http://uk.linkedin.com/pub/tomasz-k%...zko/6/940/430/
            https://kloczek.wordpress.com/
            zapish - Zabbix API SHell binding https://github.com/kloczek/zapish
            My zabbix templates https://github.com/kloczek/zabbix-templates

            Comment

            • snariom38
              Junior Member
              • Apr 2014
              • 21

              #7
              Originally posted by kloczek
              From source code:


              If this function fails it means that value cache free space is in low state.
              Usually it means that performance of inserting new values to DB engine is not enough.
              You must improve (write) performance of your DB engine.
              Do you have some advice to improve mysql write performance ?

              Comment

              • kloczek
                Senior Member
                • Jun 2006
                • 1771

                #8
                Originally posted by snariom38
                Do you have some advice to improve mysql write performance ?
                Yes I have. Do not try to improve anything before you will add proper monitoring of the DB backend.
                Because you are using MySQL as DB engine fill free to use my template which I've posted here few times:

                When you will import this template look into template description where are all details how to use it.
                http://uk.linkedin.com/pub/tomasz-k%...zko/6/940/430/
                https://kloczek.wordpress.com/
                zapish - Zabbix API SHell binding https://github.com/kloczek/zapish
                My zabbix templates https://github.com/kloczek/zabbix-templates

                Comment

                • snariom38
                  Junior Member
                  • Apr 2014
                  • 21

                  #9
                  Hi, sorry for the answering delay
                  I followed your advice and after installed the mysql template and an observation period, I obtained this measures.




                  I do not known the "good ratio" for zabbix/mysql with a small infrastructure* but it seems good right now, isn't it ?

                  I thank you all for the time and the patience you did have with my case

                  *small infrastructure = zabbix with 77 hosts with 30 items and 2 triggers for each host in average

                  Comment

                  Working...