Zabbix 3.2.0 Rev 62485. Out of Memory

[jruizjimenez]

Hi all

We have upgraded our Ubuntu 14.04 LTS server to "Ubuntu 16.04.1 LTS" and Zabbix to Zabbix 3.2.1. (Although it shows Zabbix 3.2.0 (revision 62485) and current database version (mandatory/optional): 03020000/03020000 in log). Upgrade was done with packages.

The server goes down every few days as it runs out of free memory. Server is dedicated to Zabbix, data bases is in another host.

Zabbix processes (zabbix_server) take more and more memory every minute.


Please take a look at the free memory graph, before upgrade was always around 3 GB, after upgrades goes to 0 every few hours.


Can you please suggest a test to gather more information.

Thanks.

[andris]

Hi!

Let Zabbix server monitor its own memory leaks

Here are my items for monitoring "data" segments by-process:

server alerter "data" segment:               proc.mem[zabbix_server,zabbix,sum,zabbix_server: alerter,data]
server configuration syncer "data" segment:  proc.mem[zabbix_server,zabbix,sum,zabbix_server: configuration syncer,data]
server DB watchdog "data" segment:           proc.mem[zabbix_server,zabbix,sum,zabbix_server: db watchdog,data]
server discoverer "data" segment:            proc.mem[zabbix_server,zabbix,sum,zabbix_server: discoverer,data]
server escalator "data" segment:             proc.mem[zabbix_server,zabbix,sum,zabbix_server: escalator,data]
server history syncer "data" segment:        proc.mem[zabbix_server,zabbix,sum,zabbix_server: history syncer,data]
server housekeeper "data" segment:           proc.mem[zabbix_server,zabbix,sum,zabbix_server: housekeeper,data]
server http poller "data" segment:           proc.mem[zabbix_server,zabbix,sum,zabbix_server: http poller,data]
server icmp pinger "data" segment:           proc.mem[zabbix_server,zabbix,sum,zabbix_server: icmp pinger,data]
server ipmi poller "data" segment:           proc.mem[zabbix_server,zabbix,sum,zabbix_server: ipmi poller,data]
server main process "data" segment:          proc.mem[zabbix_server,zabbix,sum,zabbix_server($| ),data]
server poller "data" segment:                proc.mem[zabbix_server,zabbix,sum,zabbix_server: poller,data]
server proxy poller "data" segment:          proc.mem[zabbix_server,zabbix,sum,zabbix_server: proxy poller,data]
server self-monitoring "data" segment:       proc.mem[zabbix_server,zabbix,sum,zabbix_server: self-monitoring,data]
server timer "data" segment:                 proc.mem[zabbix_server,zabbix,sum,zabbix_server: timer,data]
server trapper "data" segment:               proc.mem[zabbix_server,zabbix,sum,zabbix_server: trapper,data]
server unreachable poller "data" segment:    proc.mem[zabbix_server,zabbix,sum,zabbix_server: unreachable poller,data]

If a "data" segment is growing it could be a memory leak.
Try "ps -fu zabbix | grep zabbix_server" to see what Zabbix processes you have and tweak items for your environment to make sure all Zabbix server processes are covered.
Hopefully you will get data to find out which type of process is leaking memory.
In a similar way you can set up items to monitor memory leaks on proxy and agents.

Andris

[jruizjimenez]

Thanks Andris

I will add the items.
So far I believe that pollers are the processes that grow without limit.

After 12 hours:

USER       PID  %CPU %MEM  VSZ RSS     TTY   STAT START   TIME COMMAND
zabbix     996  0.4 17.2 952708 697036 ?       S    08:33   3:19 /usr/sbin/zabbix_server: poller #3 [got 37 values in 1.004115 sec, idle 1 sec]
zabbix     995  0.4 16.8 941616 683012 ?       S    08:33   3:19 /usr/sbin/zabbix_server: poller #2 [got 26 values in 0.905711 sec, idle 1 sec]
zabbix     997  0.4 16.8 936556 680732 ?       S    08:33   3:19 /usr/sbin/zabbix_server: poller #4 [got 17 values in 0.907179 sec, idle 1 sec]
zabbix     994  0.4 16.5 931344 671660 ?       S    08:33   3:19 /usr/sbin/zabbix_server: poller #1 [got 26 values in 0.911027 sec, idle 1 sec]
zabbix     998  0.4 16.5 935312 670556 ?       S    08:33   3:19 /usr/sbin/zabbix_server: poller #5 [got 32 values in 0.900032 sec, idle 1 sec]

After a service zabbix-server restart:

USER       PID  %CPU %MEM  VSZ RSS     TTY   STAT START   TIME COMMAND
zabbix    9787  2.0  0.3 252084 14592 ?        S    20:54   0:00 /usr/sbin/zabbix_server: poller #1 [got 58 values in 1.943747 sec, getting values]
zabbix    9790  4.2  0.3 252084 14488 ?        S    20:54   0:00 /usr/sbin/zabbix_server: poller #4 [got 43 values in 1.869551 sec, getting values]
zabbix    9791  2.6  0.3 252016 14424 ?        S    20:54   0:00 /usr/sbin/zabbix_server: poller #5 [got 56 values in 2.010855 sec, getting values]
zabbix    9788  2.6  0.3 252152 14340 ?        S    20:54   0:00 /usr/sbin/zabbix_server: poller #2 [got 61 values in 2.045364 sec, getting values]
zabbix    9784  1.8  0.3 242224 14064 ?        S    20:54   0:00 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
zabbix    9789  2.6  0.2 247804 11408 ?        S    20:54   0:00 /usr/sbin/zabbix_server: poller #3 [got 59 values in 1.918113 sec, getting values]

Regards