zabbix server Memory Fill Up

[[email protected]]

Hi,
i have zabbix 1.6 and all work fine (20 host monitored mixed linux/win).
Only problem is memory utilization.
in my server i have 2gb of physical memory,zabbix fill up it in one week.
I have hot add 2gb of virtual memory (in a file) zabbix fill up it in one week.
I have add another 2gb of swap file and zabbix fill up too.
I need to schedule one restart of zabbix-server to free memory.
Why?
Thank's

[tekknokrat]

there are some memleak issues with server which were fixed in zabbix-1.6.1.

[ZhidkovAN]

there are some memleak issues with server which were fixed in zabbix-1.6.1.

But in 1.6.2 appeared again

[pace]

Same deal with me too 1.6 had a memory leak 1.6.1 fixed it, 1.6.2 has it worse than 1.6 had it.


pace

[tekknokrat]

did you already opened some tickets on that?
would be courious about your setup, is also proxy in use?

[rmrf]

Have the same problem with 1.6.2. Losing about 150MB per day. Will downgrade to 1.6.1 this weekend.

[Alexei]

Thanks for the thread. I cannot confirm the memory leak. Give us some time to try to reproduce it in our test environment.

[rmrf]

I've made some investigations and found out next things:
1 - It's a Cached memory stably increases approx. 4K per 5 seconds, as I see in /proc/meminfo
2 - It's not depend on running agent or apache with frontend. (I have all on the same machine). Have started only mysqld and cached memory began to increase just after zabbix_server have started (no httpd or zabbix_agent was started).
3 - I've tried do downgrade from 1.6.2 to 1.6.1 - have the same result.

My system:
Ubuntu 2.6.24-19-server
Mysql Ver 5.1.30 for pc-linux-gnu on i686
httpd-2.2.11
php-5.2.8
zabbix-1.6.2
all compiled from sources.

For now I've resolved problem next way: dropping memory caches daily with script below, running by cron.

#!/bin/bash
/bun/sync
/bin/echo 3 > /proc/sys/vm/drop_caches

[hulting74]

Better solution

Hi

Better solution to have the memory NOT to swap out the program from memory.

echo 0 > /proc/sys/vm/swappiness
(default setting = echo 60 > /proc/sys/vm/swappiness)

You can check you present setting with

sysctl -A | grep vm

and check for vm.swappiness.

Have fun

[hughmcl]

Memory filling up fast on 1.6.2.

I installed 1.6.2 the other day, and the server memory is filling up really fast. The zabbix_server processes slowly want more and more memory until all the physical memory and swap space is used up, when it then crashes and frees up all the memory.

You can see from the top snapshots that the same process slowly sucks up more and more memory.

We are monitoring about 30 servers, and 112 web checks.

CentOS 5.2 with all the latest patches and kernel.
3.5 GB Physical memory, 6GB Swap space.

Here are a couple of snapshots:-

14672 zabbix 20 5 63180 38m 1672 S 2 1.2 0:08.21 zabbix_server
14686 zabbix 20 5 9264 3372 2188 S 2 0.1 0:46.68 zabbix_server
14670 zabbix 20 5 62568 38m 1652 S 2 1.2 0:08.07 zabbix_server
14671 zabbix 20 5 63180 39m 1656 S 2 1.2 0:08.07 zabbix_server
14673 zabbix 21 5 62072 38m 1660 S 2 1.2 0:08.10 zabbix_server
14677 zabbix 21 5 8968 1688 852 S 1 0.1 0:06.23 zabbix_server
14678 zabbix 20 5 8968 1688 852 S 1 0.1 0:05.34 zabbix_server


2 minutes later:-

14686 zabbix 20 5 9668 3672 2188 S 15 0.1 0:48.92 zabbix_server
14672 zabbix 20 5 65336 40m 1672 S 2 1.2 0:08.53 zabbix_server
14673 zabbix 20 5 64104 39m 1660 S 2 1.2 0:08.45 zabbix_server
14671 zabbix 20 5 65604 40m 1656 S 1 1.2 0:08.42 zabbix_server
5768 zabbix 20 5 4288 484 392 S 1 0.0 0:37.39 zabbix_agentd
14670 zabbix 20 5 64792 40m 1652 S 1 1.2 0:08.41 zabbix_server
14674 zabbix 20 5 65032 40m 1660 S 1 1.2 0:08.66 zabbix_server

[Alexei]

We spent more than two days doing extensive testing of Zabbix Server and Proxy this week. We still cannot confirm any memory leaks.

Anyone having the memory leak please provide me with the following information:

1. Value for DBStartSyncers in zabbix_server.conf
2. Maximum memory usage you've seen for a single zabbix_server process. Output of "top" or "ps" is highly appreciated.
3. Item types you monitor (SNMP, Agent, Agent active, simple checks, etc).

Note that it is normal for Zabbix Server to increase memory usage slowly to around 65MB and then stabilize. The value also includes shared memory used by Zabbix Server, the memory is SHARED between all server processes.

Thank you.

[hughmcl]

1) StartDBSyncers=0 ( I added this and restarted zabbix_server, option
wasn't in the 1.6.2 config files or the original 1.6 i installed).


2) top - 18:03:14 up 19:45, 5 users, load average: 0.90, 0.81, 0.70
Tasks: 171 total, 1 running, 169 sleeping, 1 stopped, 0 zombie
Cpu(s): 12.9%us, 6.5%sy, 3.0%ni, 72.5%id, 4.2%wa, 0.2%hi, 0.8%si, 0.0%st
Mem: 3369668k total, 1235752k used, 2133916k free, 17744k buffers
Swap: 6225896k total, 507744k used, 5718152k free, 204748k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3836 mysql 15 0 794m 331m 3328 S 69 10.1 482:22.06 mysqld
1219 zabbix 20 5 156m 106m 1664 S 6 3.2 0:25.40 zabbix_server
1216 zabbix 20 5 159m 109m 1668 S 5 3.3 0:23.71 zabbix_server
1237 zabbix 20 5 9668 3672 2188 S 4 0.1 2:36.44 zabbix_server
1217 zabbix 20 5 153m 104m 1656 S 2 3.2 0:25.74 zabbix_server
1224 zabbix 20 5 8840 1540 876 S 1 0.0 0:04.62 zabbix_server
1223 zabbix 20 5 8840 1564 892 S 0 0.0 0:04.84 zabbix_server

This is current snapshot. The processes are still growing in usage.

A few minutes later:-

top - 18:08:15 up 19:50, 5 users, load average: 0.64, 0.75, 0.70
Tasks: 171 total, 1 running, 169 sleeping, 1 stopped, 0 zombie
Cpu(s): 7.2%us, 3.7%sy, 4.3%ni, 74.6%id, 9.8%wa, 0.1%hi, 0.3%si, 0.0%st
Mem: 3369668k total, 1357172k used, 2012496k free, 23072k buffers
Swap: 6225896k total, 507736k used, 5718160k free, 268020k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3836 mysql 15 0 794m 331m 3328 S 39 10.1 484:15.92 mysqld
1237 zabbix 20 5 9668 3672 2188 S 12 0.1 2:52.26 zabbix_server
1218 zabbix 20 5 168m 115m 1664 S 3 3.5 0:27.73 zabbix_server
1216 zabbix 20 5 173m 119m 1668 S 2 3.6 0:26.16 zabbix_server
1215 zabbix 20 5 171m 117m 1652 S 1 3.6 0:26.23 zabbix_server
1217 zabbix 20 5 167m 115m 1656 S 1 3.5 0:28.26 zabbix_server
1220 zabbix 20 5 8840 1508 852 S 1 0.0 0:05.05 zabbix_server
1221 zabbix 21 5 8840 1492 836 S 0 0.0 0:05.29 zabbix_server
1222 zabbix 20 5 8840 1516 836 S 0 0.0 0:04.99 zabbix_server


Growth rate appears to be about 0.5MB/min for the processes.

a couple of minutes later:-

top - 18:10:45 up 19:53, 5 users, load average: 0.94, 0.75, 0.70
Tasks: 171 total, 1 running, 169 sleeping, 1 stopped, 0 zombie
Cpu(s): 3.9%us, 1.9%sy, 1.7%ni, 91.4%id, 1.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 3369668k total, 1400944k used, 1968724k free, 25252k buffers
Swap: 6225896k total, 507736k used, 5718160k free, 285128k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3836 mysql 15 0 794m 331m 3328 S 20 10.1 485:09.43 mysqld
1237 zabbix 20 5 9256 3364 2188 S 4 0.1 3:00.11 zabbix_server
1216 zabbix 20 5 180m 124m 1668 S 1 3.8 0:27.44 zabbix_server
1218 zabbix 20 5 174m 119m 1664 S 1 3.6 0:29.00 zabbix_server
1215 zabbix 20 5 178m 122m 1652 S 0 3.7 0:27.43 zabbix_server
1219 zabbix 20 5 177m 121m 1664 S 0 3.7 0:29.18 zabbix_server
1220 zabbix 20 5 8840 1508 852 S 0 0.0 0:05.28 zabbix_server
1224 zabbix 20 5 8840 1540 876 S 0 0.0 0:05.34 zabbix_server


3) We use Zabbix Agent, SNMP v1, SNMP v2, externalScripts, custom Scripts and web monitoring checks.


Some more info:-

uname -a:-
Linux xxxx.xxx.xxx 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 12:03:43 EST 2008 i686 i686 i386 GNU/Linux

CentOS 5.2 with patches installed.
mysql-5.0.67
php-5.2.6
apache httpd-2.2.9
zabbix_server 1.6.2

[kmitbo]

zabbix_server process memory grow on my system has started after messages like
" 30516:20090119:153001 [Z3005] Query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update items set lastclock=1232371801,prevvalue=lastvalue,prevorgva lue=NULL,lastvalue='1213904.000000' where itemid=26906;" appeared in zabbix_server.log.
And this not depend on StartDBSyncers value.

[Alexei]

zabbix_server process memory grow on my system has started after messages like
" 30516:20090119:153001 [Z3005] Query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update items set lastclock=1232371801,prevvalue=lastvalue,prevorgva lue=NULL,lastvalue='1213904.000000' where itemid=26906;" appeared in zabbix_server.log.
And this not depend on StartDBSyncers value.

The warning may appear ONLY if StartDBSyncers is set to 1! FYI The deadlock issue has been resolved in pre 1.6.3.

[hulting74]

Has found no good working solution

Hi

Seems my work has not given any good results, still eating memory. :-(

1. DBSyncers set as default.
2. See attachments
3. Monitoring mainly Windows servers (Zabbix Agent, Web monitoring checks, mainly)

Attachments from top, free, and ps:

P.s the suggestions "drop_caches" and "swappiness" seem to work temporarily, but not a permanent solution.

/S