Ad Widget

Collapse

How to tell if Zabbix has memory leak?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • drose12
    Junior Member
    • Apr 2007
    • 27

    #1

    How to tell if Zabbix has memory leak?

    1.4.2 from SVN.
    CentOS 4.5, 2.6.9-55ELsmp
    3GB RAM, 6GB swap

    Been running only for a day or so, but I see that the zabbix_server processes are starting to take up quite a bit of memory...280MB VSZ and 118MB RSS, but only for 10 of these:

    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

    zabbix 9659 0.0 0.0 7220 896 ? SN Aug01 0:00 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9664 0.0 0.0 9548 2200 ? SN Aug01 1:46 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9665 0.0 0.0 9548 2180 ? SN Aug01 1:46 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9666 0.0 0.0 9548 2040 ? SN Aug01 2:10 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9667 0.0 0.0 9564 2236 ? SN Aug01 2:03 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9668 0.0 0.0 9548 2212 ? SN Aug01 1:52 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9669 0.0 0.0 9548 2224 ? SN Aug01 2:12 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9670 0.0 0.0 9548 2224 ? SN Aug01 2:29 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9671 0.0 0.0 9548 2188 ? SN Aug01 2:24 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9672 0.0 0.0 9548 1984 ? SN Aug01 1:40 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9673 0.0 0.0 9548 1996 ? SN Aug01 2:02 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9674 0.0 0.0 9548 2008 ? SN Aug01 1:51 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9675 0.0 0.0 9548 2156 ? SN Aug01 2:21 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf

    zabbix 9676 0.0 3.8 280184 118536 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9677 0.0 3.7 279012 116704 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9679 0.0 3.7 278920 116604 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9680 0.0 3.7 279524 117324 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9684 0.0 3.7 278936 116372 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9685 0.0 3.7 277736 116228 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9687 0.0 3.7 280392 117620 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9689 0.0 3.7 279308 116704 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9691 0.0 3.7 278360 117040 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9692 0.0 3.7 279836 116788 ? SN Aug01 0:23 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf


    zabbix 9694 0.0 0.0 7220 964 ? SN Aug01 0:02 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9698 0.0 0.0 7220 896 ? SN Aug01 0:01 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9700 0.0 0.0 7224 860 ? SN Aug01 0:20 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9702 0.0 0.0 7220 648 ? SN Aug01 0:00 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9710 0.0 0.0 9548 1976 ? SN Aug01 0:03 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9716 0.0 0.0 7220 644 ? SN Aug01 0:01 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9718 0.0 0.0 7340 1972 ? SN Aug01 0:09 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9720 0.0 0.0 7332 1936 ? SN Aug01 0:09 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9722 0.0 0.0 7332 1940 ? SN Aug01 0:09 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9724 0.0 0.0 7332 1920 ? SN Aug01 0:31 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9726 0.0 0.0 7220 632 ? SN Aug01 0:01 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 9728 0.0 0.0 9512 1496 ? SN Aug01 0:00 /usr/local/bin/zabbix_server -c /etc/zabbix/zabbix_server.conf

    My swap use is trending up...

    see attachment...
    Attached Files
    Last edited by drose12; 03-08-2007, 17:20.
  • Palmertree
    Senior Member
    • Sep 2005
    • 746

    #2
    I was just about to post the same problem. I have to restart zabbix_server ounce a day because the memory for the zabbix_server process slowly increases by the day.

    Comment

    • Saftnase
      Member
      • Jul 2006
      • 30

      #3
      dito can confirm this behavior, but need to restart my zabbix server only every 5 days

      Comment

      • JonB
        Member
        • Oct 2006
        • 63

        #4
        I also had the same issue with 1.4.1 on 2 CentOS 4.3 based servers. In both cases the agents had been compiled with the --static option.

        Once I recompiled without the --static option the problem went away.

        Jon

        Comment

        • Palmertree
          Senior Member
          • Sep 2005
          • 746

          #5
          Interesting....

          I wonder why the --static option has an affect on the memory consumption.

          Comment

          • Alexei
            Founder, CEO
            Zabbix Certified Trainer
            Zabbix Certified SpecialistZabbix Certified Professional
            • Sep 2004
            • 5654

            #6
            Originally posted by Palmertree
            Interesting....

            I wonder why the --static option has an affect on the memory consumption.
            It sounds like a memory leak in a third party static library, not in ZABBIX code itself.
            Alexei Vladishev
            Creator of Zabbix, Product manager
            New York | Tokyo | Riga
            My Twitter

            Comment

            • Palmertree
              Senior Member
              • Sep 2005
              • 746

              #7
              Are there any good linux utilites out there to help pin point the problem?

              Comment

              • Alexei
                Founder, CEO
                Zabbix Certified Trainer
                Zabbix Certified SpecialistZabbix Certified Professional
                • Sep 2004
                • 5654

                #8
                Originally posted by Palmertree
                Are there any good linux utilites out there to help pin point the problem?
                Hmm, ZABBIX? I would be interested to see ZABBIX server memory usage for a period of time, not usage of swap space.
                Alexei Vladishev
                Creator of Zabbix, Product manager
                New York | Tokyo | Riga
                My Twitter

                Comment

                • drose12
                  Junior Member
                  • Apr 2007
                  • 27

                  #9
                  Originally posted by Alexei
                  Hmm, ZABBIX? I would be interested to see ZABBIX server memory usage for a period of time, not usage of swap space.
                  The swap space use graph was the results of the zabbix processes taking up all the memory. The funny thing on my system is that I have 36 zabbix_server processes, but not all of them are consuming memory. And the parent process looks fine. Without knowing which pid to watch, how can I go about graphing the memory usage of 35 processes? I have 10 right now which are 185mb each right now. Looks like I require a restart. BTW I did not compile my server with the static option.

                  Comment

                  • bbrendon
                    Senior Member
                    • Sep 2005
                    • 870

                    #10
                    I think I'm leaking too. Not sure yet, but I found this.



                    Interesting.

                    ps_mem.py reports:
                    224.0 MiB + 2.3 MiB = 226.3 MiB zabbix_server (20)

                    proc.mem[zabbix_server] reports: 1,342,836,736

                    1.3 Gigabytes vs 226 megabytes ?!??!?

                    WOW.
                    Last edited by bbrendon; 15-08-2007, 08:32.
                    Unofficial Zabbix Expert
                    Blog, Corporate Site

                    Comment

                    • pixelbeat
                      Junior Member
                      • Aug 2007
                      • 1

                      #11
                      Originally posted by infinity005
                      I think I'm leaking too. Not sure yet, but I found this.



                      Interesting.

                      ps_mem.py reports:
                      224.0 MiB + 2.3 MiB = 226.3 MiB zabbix_server (20)

                      proc.mem[zabbix_server] reports: 1,342,836,736

                      1.3 Gigabytes vs 226 megabytes ?!??!?

                      WOW.

                      Note ps_mem.py reports the RAM used.
                      I.E. any memory swapped out to disk is not reported.

                      In summary ps_mem.py reports the current amount of
                      RAM used per program, rather than the current amount
                      of memory used per process, as reported by traditional tools

                      Comment

                      • bbrendon
                        Senior Member
                        • Sep 2005
                        • 870

                        #12
                        Whoa. Did I just take that script from your website? And you're here?

                        Creepy. That was un-intentional.

                        ...

                        Wait. You have 1 post! Are you monitoring your weblogs in realtime? Are you a stalker?

                        Please brain dump on us with all your knowledge while you're stalking. Thanks!
                        Last edited by bbrendon; 15-08-2007, 19:00.
                        Unofficial Zabbix Expert
                        Blog, Corporate Site

                        Comment

                        • bbrendon
                          Senior Member
                          • Sep 2005
                          • 870

                          #13
                          Enough fun. Back to the topic at hand.
                          My zabbix server has a LOT of stuff running on it. So hopefully others can help with this.

                          I created a proc.mem[zabbix_server], and so far (12 hrs) it has increased from 1342836736 to 1376583680

                          Difference: +33,746,944

                          I'm running Debian etch, amd64.

                          How can we find the library causing the leaks? I don't think I have enough data to say zabbix is leaking, but based on my swap data collection and the time I'm restarting the zabbix_server, it seems zabbix IS leaking.
                          Unofficial Zabbix Expert
                          Blog, Corporate Site

                          Comment

                          • drose12
                            Junior Member
                            • Apr 2007
                            • 27

                            #14
                            Originally posted by infinity005
                            Enough fun. Back to the topic at hand.
                            My zabbix server has a LOT of stuff running on it. So hopefully others can help with this.

                            I created a proc.mem[zabbix_server], and so far (12 hrs) it has increased from 1342836736 to 1376583680
                            I've created mine as well...I'll post my results after a day or so as I've just restarted zabbix to show the trend.

                            Comment

                            • bbrendon
                              Senior Member
                              • Sep 2005
                              • 870

                              #15
                              So this is getting to be a problem. I though our server was going to crash and burn today. Luckily it came back after about 5 mins. When it was finally responding, i typed "uptime"

                              load average: 44.41, 29.11, 14.10

                              So it appears a simple stop of zabbix caused the load to go from a normal 1-2 to 44

                              Attaching my proc.mem[zabbix_server]

                              Help!
                              Attached Files
                              Unofficial Zabbix Expert
                              Blog, Corporate Site

                              Comment

                              Working...