Ad Widget

Collapse

Please increase VMwareCacheSize configuration parameter

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Mr.Fox
    Junior Member
    • Aug 2018
    • 7

    #1

    Please increase VMwareCacheSize configuration parameter

    Hello everyone,

    I'm a trying to troubleshoot my Zabbix server.
    It's been working great for some time and recently had a crash due to insufficient cache size for vmware. I followed what the log file said and increased the cache size.
    About 8 h later the server crashed again with the same error.

    Code:
    32168:20180828:031614.789 __mem_malloc: skipped 340 asked 9677816 skip_min 272 skip_max 1551896
     32168:20180828:031614.789 [file:vmware.c,line:86] zbx_mem_realloc(): out of memory (requested 9677816 bytes)
     32168:20180828:031614.789 [file:vmware.c,line:86] zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
     32088:20180828:031614.838 One child process died (PID:32168,exitcode/signal:1). Exiting ...
     32088:20180828:031616.840 syncing history data...
     32088:20180828:031616.850 syncing history data done
     32088:20180828:031616.850 syncing trend data...
     32088:20180828:031618.185 syncing trend data done
     32088:20180828:031618.185 Zabbix Server stopped. Zabbix 3.4.7 (revision 77720).
     14290:20180828:031628.858 Starting Zabbix Server. Zabbix 3.4.7 (revision 77720).
    When i check the graph with cache sizes it shows about 2% usage of the vmware cache.

    What could be causing this ?

    I am monitoring 409 hosts for 10200 objects (166 vps)

    Thanks in advance,

    Mr. Fox
  • Kirin2
    Junior Member
    • Jul 2018
    • 28

    #2

    How much have your increased cache size? Mb it's not enough? Also mb you need to check another paramaters related with cache size(I'm not sure, but you can try)

    Comment

    • dimir
      Zabbix developer
      • Apr 2011
      • 1080

      #3
      Could you please attach the graph screenshot, which includes the time of crash (time bounds before and after the crash)?

      Comment

      • Mr.Fox
        Junior Member
        • Aug 2018
        • 7

        #4
        Thank you for your replies.

        Originally posted by Kirin2
        How much have your increased cache size? Mb it's not enough? Also mb you need to check another paramaters related with cache size(I'm not sure, but you can try)
        I increased it from 4M to 32M then 128M and even 256M I think. What parameters are you thinking about ?



        Originally posted by dimir
        Could you please attach the graph screenshot, which includes the time of crash (time bounds before and after the crash)?

        Click image for larger version  Name:	InkedGVs4fE2_LI.jpg Views:	1 Size:	68.6 KB ID:	365169

        Also it's probably usefull to mention that i coudln't restart zabbix-server after the crash. I probably had to kill every zabbix server process but I restarted the machine to go faster.

        Edit : decreased caches size yesterday and it seem to have taken longer to crash.
        Last edited by Mr.Fox; 31-08-2018, 10:07.

        Comment

        • Mr.Fox
          Junior Member
          • Aug 2018
          • 7

          #5
          Up.
          My problem is still there, I also noticed the server was actually silently crashing sometimes but would automatically restart on it's own.

          Comment

          • dimir
            Zabbix developer
            • Apr 2011
            • 1080

            #6
            I don't see vmware cache usage graph where it shows 2%.

            Comment

            • Mr.Fox
              Junior Member
              • Aug 2018
              • 7

              #7
              Originally posted by dimir
              I don't see vmware cache usage graph where it shows 2%.
              Last one, Zabbix VMware cache, % free 99,55%

              Comment

              • dimir
                Zabbix developer
                • Apr 2011
                • 1080

                #8
                Ah, right, it's free. Is there enough memory on the server?

                Comment

                • Mr.Fox
                  Junior Member
                  • Aug 2018
                  • 7

                  #9
                  About 3GB of available memory and 140 MB of free memory.

                  Comment

                  • Mr.Fox
                    Junior Member
                    • Aug 2018
                    • 7

                    #10
                    Seems like the problem was coming from monitoring vCenter with the template. I noticed that the vCenter had significant higher CPU usage (+3Ghz) so I went back to monitoring via the ESXis and it's been steady for more than a week with lesser CPU usage on the ESXis.

                    Any idea on why that is ?

                    Comment

                    • LarryD
                      Junior Member
                      • Mar 2014
                      • 5

                      #11
                      Originally posted by Mr.Fox
                      Seems like the problem was coming from monitoring vCenter with the template.
                      I'm having the same problem... can you elaborate on what you mean by this statement?

                      Comment

                      • Mr.Fox
                        Junior Member
                        • Aug 2018
                        • 7

                        #12
                        Originally posted by LarryD

                        I'm having the same problem... can you elaborate on what you mean by this statement?
                        The problem i had was monitoring a vCenter with the built-in vmware template. When switching to monitoring via ESXi the problem went away.
                        I also found vCenter monitoring was using about 2-3Ghz of CPU usage on the vCenter.

                        Bumping up the vmcachesize way above what seemed to be needed utimately did the trick. What was wierd is that the cache size was 99.99% free but still it looked like sometime a very big chunk of data would come and made the poller crash (that wasn't happening with ESXi monitoring).
                        I increased the vm memory to 4GB and then increased the vmwarecachesize to 2G.
                        => vCenter CPU usage dropped to normal level and cpu load of the Zabbix did the same.

                        Hope that helps

                        Comment


                        • niumar
                          niumar commented
                          Editing a comment
                          Hi! I have exactly the same problem with zabbix-server and in the same conditions. Did your changes on vcenter solve the problem?
                      • vso
                        Zabbix developer
                        • Aug 2016
                        • 190

                        #13
                        Hello! If you could please apply following patch, it could provide some insights.
                        Code:
                        Index: src/libs/zbxmemory/memalloc.c
                        ===================================================================
                        --- src/libs/zbxmemory/memalloc.c    (revision 85057)
                        +++ src/libs/zbxmemory/memalloc.c    (working copy)
                        @@ -667,6 +667,9 @@
                                         file, line, __function_name, (zbx_fs_size_t)size);
                                 zabbix_log(LOG_LEVEL_CRIT, "[file:%s,line:%d] %s(): please increase %s configuration parameter",
                                         file, line, __function_name, info->mem_param);
                        +        zabbix_increase_log_level();
                        +        zabbix_increase_log_level();
                        +        zbx_mem_dump_stats(info);
                                 exit(EXIT_FAILURE);
                             }
                        
                        @@ -700,6 +703,9 @@
                                         file, line, __function_name, (zbx_fs_size_t)size);
                                 zabbix_log(LOG_LEVEL_CRIT, "[file:%s,line:%d] %s(): please increase %s configuration parameter",
                                         file, line, __function_name, info->mem_param);
                        +        zabbix_increase_log_level();
                        +        zabbix_increase_log_level();
                        +        zbx_mem_dump_stats(info);
                                 exit(EXIT_FAILURE);
                             }

                        Comment


                        • niumar
                          niumar commented
                          Editing a comment
                          Hi vso! I applied your patch and the first log I receave after start zabbix server was:

                          29016:20181001:133122.911 Error of query maxQueryMetrics: 'config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted..

                          The daemon is still running and I'm waiting it to die to reply you the log.

                        • niumar
                          niumar commented
                          Editing a comment
                          Hi again vso!
                          Well the server died. I'm not sure if the patch has increased the log. These are all de messagens logged after the server start.

                          29016:20181001:133122.911 Error of query maxQueryMetrics: 'config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted..
                          28989:20181001:140112.105 executing housekeeper
                          28989:20181001:140113.584 housekeeper [deleted 0 hist/trends, 0 items/triggers, 2058 events, 10 problems, 0 sessions, 0 alarms, 0 audit items in 1.453030 sec, idle for 1 hour(s)]
                          29016:20181001:142122.336 Error of query maxQueryMetrics: 'config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted..
                          29016:20181001:142123.242 __mem_malloc: skipped 0 asked 152 skip_min 18446744073709551615 skip_max 0
                          29016:20181001:142123.244 [file:vmware.c,line:87] zbx_mem_malloc(): out of memory (requested 151 bytes)
                          29016:20181001:142123.244 [file:vmware.c,line:87] zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
                          28976:20181001:142123.276 One child process died (PID:29016,exitcode/signal:1). Exiting ...
                          zabbix_server [28976]: Error waiting for process with PID 29016: [10] No child processes
                          28976:20181001:142123.454 syncing history data...
                          28976:20181001:142123.454 syncing history data done
                          28976:20181001:142123.454 syncing trend data...
                          28976:20181001:142123.941 syncing trend data done
                          28976:20181001:142124.099 Zabbix Server stopped. Zabbix 4.0.0rc3 (revision 85283).
                      • vso
                        Zabbix developer
                        • Aug 2016
                        • 190

                        #14
                        I am sorry but it looks like you have not applied patch or did not compile

                        Comment


                        • niumar
                          niumar commented
                          Editing a comment
                          Sorry, i forgot to edit the first function. Now i think it's right...

                          2161:20181001:152408.339 Error of query maxQueryMetrics: 'config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted..
                          2129:20181001:155357.224 executing housekeeper
                          2129:20181001:155403.632 housekeeper [deleted 0 hist/trends, 0 items/triggers, 1864 events, 10 problems, 0 sessions, 0 alarms, 0 audit items in 6.386429 sec, idle for 1 hour(s)]
                          2158:20181001:161128.135 Error of query maxQueryMetrics: 'config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted..
                          2158:20181001:161128.210 __mem_malloc: skipped 0 asked 24 skip_min 18446744073709551615 skip_max 0
                          2158:20181001:161128.211 [file:vmware.c,line:87] zbx_mem_malloc(): out of memory (requested 24 bytes)
                          2158:20181001:161128.211 [file:vmware.c,line:87] zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
                          2158:20181001:161128.211 === memory statistics for vmware cache size ===
                          2158:20181001:161128.211 min chunk size: 18446744073709551615 bytes
                          2158:20181001:161128.211 max chunk size: 0 bytes
                          2158:20181001:161128.211 memory of total size 8387808 bytes fragmented into 85957 chunks
                          2158:20181001:161128.211 of those, 0 bytes are in 0 free chunks
                          2158:20181001:161128.211 of those, 7012512 bytes are in 85957 used chunks
                          2158:20181001:161128.211 ================================
                          2111:20181001:161128.223 One child process died (PID:2158,exitcode/signal:1). Exiting ...
                          zabbix_server [2111]: Error waiting for process with PID 2158: [10] No child processes
                          2111:20181001:161128.304 syncing history data...
                          2111:20181001:161128.312 syncing history data... 100.000000%
                          2111:20181001:161128.312 syncing history data done
                          2111:20181001:161128.312 syncing trend data...
                          2111:20181001:161128.524 syncing trend data done
                          2111:20181001:161128.534 Zabbix Server stopped. Zabbix 4.0.0rc3 (revision 85283).
                      • vso
                        Zabbix developer
                        • Aug 2016
                        • 190

                        #15
                        What is your size of VMwareCacheSize configuration parameter?

                        Comment


                        • niumar
                          niumar commented
                          Editing a comment
                          It is the default value (8MB), but I have already tried values of 16M, 32M and 64M.
                      Working...