zabbix server Memory Fill Up

[richlv]

Shouldn't a notice be attached to the "1.6.2 stable" link in the download section to this patch? Can 1.6.2 really be considered stable with a known memory leak?

personally, i'd vote for 1.6.2.1.
this could be a quite damning first experience.

[ggiesen]

I'm inclined to agree, 1.6.2 was my first install of Zabbix as we evaluated the product, and we nearly abandoned it because of this.

[Alexei]

personally, i'd vote for 1.6.2.1.
this could be a quite damning first experience.

The memory leak happens only in case if Zabbix is unable to calculate a trigger expression. This does not happen normally, thus the memory leak does not affect majority of the users. I think it is absolutely non-visible for small installations.

[ajszi]

Thanks very much to all of you, problem solved here as well.
Zabbix rulz!

[bioposi]

Still leaking for me, unfortunately

Dear all,

I would have like to be as lucky as you are because my installation is still leaking memory, even after having applied the patch (redhat RHEL5U3, zabbix 1.6.2 + db.c patch, mysql db backend, no snmp support, with libcurl and openipmi, agentd on 101 linux monitored hosts in passive mode).

I even tried to compile without net-snmp 5.4.2.1, as the situation was worse with the snmp support enabled. I feel very frustrated to see this great potential wasted due to this annoying leak of server memory.

I have worked to solve all the bad server acquisitions from the agents as it seems that this is where the issue is, guess what: no success.
Having reached this point, I see no way out currently.

Any solution out there ?
Thanks in advance.
Cheers.
Bio/

[vhm3]

Worked for me!

The memory leak happens only in case if Zabbix is unable to calculate a trigger expression. This does not happen normally, thus the memory leak does not affect majority of the users. I think it is absolutely non-visible for small installations.

I'm running a few installations of Zabbix, one rather small, and another a little bit larger. The larger of my 2 instances, did not have this problem (RHEL 4, Zabbix 1.6.2), while my small instance (FreeBSD 7.1, Zabbix 1.6.2) did. I applied the patch, and no longer leak memory.

Thank you all!

Chip

[Alexei]

Dear all,

I would have like to be as lucky as you are because my installation is still leaking memory, even after having applied the patch (redhat RHEL5U3, zabbix 1.6.2 + db.c patch, mysql db backend, no snmp support, with libcurl and openipmi, agentd on 101 linux monitored hosts in passive mode).

I even tried to compile without net-snmp 5.4.2.1, as the situation was worse with the snmp support enabled. I feel very frustrated to see this great potential wasted due to this annoying leak of server memory.

I have worked to solve all the bad server acquisitions from the agents as it seems that this is where the issue is, guess what: no success.
Having reached this point, I see no way out currently.

Any solution out there ?
Thanks in advance.
Cheers.
Bio/

Please double (triple) check that you are running patched binaries. Also, if you have some time to experiment, install fresh Ubuntu distro with the same Zabbix code. No memory leaks in Ubuntu would prove that there is something wrong with RedHat's libraries.

[bioposi]

Thanks Alexei for having taken some time to answer.

I will give a try to a RHEL4U3 to see if the problem happens. I do not have too much time at the moment, but I'll try an Ubuntu asap.
I would have prefer to remain on redhat as it is the corporate distro of my company.

I'll post an update if I found something pertinent.
Thanks again.
Bio/

[Alexei]

I would have prefer to remain on redhat as it is the corporate distro of my company.

It is absolutely ok to stay on redhat. I asked for the Ubuntu experiment because I know it works with Zabbix fine out-of-the-box.

[jsuhre]

The memory leak happens only in case if Zabbix is unable to calculate a trigger expression. This does not happen normally, thus the memory leak does not affect majority of the users. I think it is absolutely non-visible for small installations.

I am in the process of evaluating Zabbix in our lab. Our setup has a central Zabbix server running the server process, agentd (passive), mysql, and ui. We then added a proxy running the proxy process, agentd (active), and mysql to monitor 7 CISCO routers using the default snmpv2 template. Everything else was using default configurations and this caused the server to crash and burn in a spectacular fashion.

As a new user this was definitely a let down. With that said I'm very happy with the active response here in the forum by both the community and developers! But I do think this should be fixed in a current release because I think it would turn away new users not inclined to dig a little further for answers.

I've attached a graph of our memory usage on the server before and after the patch was applied. Server is Ubuntu 8.04.

Just my $0.02! Keep up the good work!

[bioposi]

At last, some progress !

Thanks to you all, I was able to set up a machine that has not crashed yet ;-)
However, I copy you here a graph of it free memory trend over two days of usage. As you will see, the server consumes all the free mem it has, then reach a bottom and remains at this level.
Strange behaviour indeed, I was expecting that the running Zabbix server with it database had consumed a certain level of free memory then stop. No, It eat all that is available, putting blocks in cached memory ... ?
Anyone else has such profile on it server ?
For your information, I built the same server on an Ubuntu 8.10 (zbx 1.6.2, snmp support, active agents on ~100 linux hosts) and it behaves the same ...

[jsuhre]

Thanks to you all, I was able to set up a machine that has not crashed yet ;-)
However, I copy you here a graph of it free memory trend over two days of usage. As you will see, the server consumes all the free mem it has, then reach a bottom and remains at this level.
Strange behaviour indeed, I was expecting that the running Zabbix server with it database had consumed a certain level of free memory then stop. No, It eat all that is available, putting blocks in cached memory ... ?
Anyone else has such profile on it server ?
For your information, I built the same server on an Ubuntu 8.10 (zbx 1.6.2, snmp support, active agents on ~100 linux hosts) and it behaves the same ...

It's not uncommon for a linux system to show only a small amount of free memory. But in most cases it's ok because as you pointed out the "free" spaced is being used for caching.

Typically as long as there isn't a lot of swap activity (thrashing) your ok. You can also use 'ps auxw' to get more info on memory usage of individual processes. Before I applied this patch I would show a zabbix_server process that consumed a very large percentage of available system memory (indicating a leak).

[bioposi]

It's not uncommon for a linux system to show only a small amount of free memory. But in most cases it's ok because as you pointed out the "free" spaced is being used for caching.

Typically as long as there isn't a lot of swap activity (thrashing) your ok. You can also use 'ps auxw' to get more info on memory usage of individual processes. Before I applied this patch I would show a zabbix_server process that consumed a very large percentage of available system memory (indicating a leak).

Thanks Jsuhre. This sounds logical for me now according to what you explained. Specially,
I monitored the server using 'top' and 'vmstat' and saw that the amount of memory per Zabbix daemon didn't changed.
So, I encourage anyone to apply the patch in msg#25 as it brings an efficient fix for a potential issue. I wishes also that it appears in the 'patch' section of the forum to be universally deployed. I expect that this code be part of the coming releases.

Thanks to all and Happy Zabbix

[otheus]

Great job, Zabbix!

Last week, we (tiscover.com) installed 1.6.2 for testing purposes. After about 5 hours of running, the server would consume all of swap, causing "bad things" to happen. As it was noted on this forum, we also had many SNMP - bad-value warnings. On the server, we are running RHEL5 (and RHEL4 and 5.2 servers being monitored).

I can confirm that 1.6.3 appears to fix these problems. Using the following command, we can see that memory appears relatively stable:

 ps f -C zabbix_server -o pid,vsz,args

... and after a day, there is no increase, unlike with 1.6.2.

[Kerrygeek]

At the risk of sounding pretty ignorant, what do I need to do to apply this patch? Is it something I just run from the command line or do I need to be logged into mysql to run it from there? I'm a hardware guy and don't know much about software... I see today that my 1.6.2 server seems to be eating memory, it's hitting swap for the first time and has been growing all day.

Tnx,
Kerry