Zabbix Agent for Windows - memory leaking

[aib]

Hello!

A couple of days ago I had to restart the old server which ate all memory.
After some investigation I found out that it was Zabbix Agent who used more than 400 Mb of RAM.

I started to collect data about memory utilization of Zabbix Agent application

[LIST][*]perf_counter[\Process(zabbix_agentd.e)\Private Bytes][*]perf_counter[\Process(zabbix_agentd.e)\Virtual Bytes][/LIST]

On the graph we can see how Zabbix Agent slowly increase the amount of memory which it use.

After first day I upgraded Zabbix Agent from version 2.2.1 to 2.2.3 but nothing was changed.

I checked the different servers with different OS'e (from Windows XP to Windows 2008) and typically Zabbix Agent use 4-10 MB of RAM and this amount of memory doesn't grow up over time.

Server OS : Windows 2000 SP4, virtual environment
Zabbix Agent Version: 2.2.1 or 2.2.3
Server Memory: 654,828 bytes

How I can fix it (if it possible to fix) ?

[ingus.vilnis]

Hello Aleksey

I suspect that it is related to Windows Log monitoring but it has not been confirmed yet. So far only Windows 32 bit server machines seem to be affected but it is hard to reproduce this issue.

Is it possible that you share your template or show a screenshot of monitored Items for this host?

Best Regards,
Ingus

[aib]

Thank you for a quick answer.

Unfortunately, I don't have any items for Windows Log Monitoring (you can see it in attached template)

And you are right - it is the 32-bit version of Microsoft Windows 2000.

If you like, I can do some experiments with Item and monitoring settings. (ex. disable all Items and left on only one Item per time)

So far (after two days of work) Zabbix Agent use 56.34 Mb of Private Memory which is 15 times more than it used after restart (3.76 Mb)

[aib]

Hi, Ingus!

Do you have any news/updates/hints about strange behavior of my Agent?

Right now I have to restart it every week to keep Memory usage under 100 Mb.

Could you advice me some tuning options or debug operations which can help me to reduce the memory leaking?

[ingus.vilnis]

Hello Aleksey!

Unfortunately I don't have any positive news yet. (Therefore I apologize for not replying to your previous post!)

However there is bug report created on this very issue (https://support.zabbix.com/browse/ZBX-8236). Please don't take into account that this ticket is on Windows 2008. 32-bit versions are what matters.

I will let you know if there will be any news about this problem!

With Best Regards,
Ingus

[aib]

Thank you for your information!
I'll keep my fingers crossed.

[tchjts1]

Until they get a fix, why not just downgrade the agent to something like 2.0.9 or similar?

[aib]

Thank you for advice, tchjts1

Agent was downgraded to version 2.0.9
I will see how it works.

[aib]

It doesn't help.
After downgrading to version 2.0.9 I still have the same behavior of Zabbix Agent.
He is eating more and more memory every hour.
After 24 hours of working his memory usage grow up from 3.48 Mb up to 54.8 Mb.

[ingus.vilnis]

Wow, this is really strange!

Anyhow, let's wait for the ZBX to be solved! I will post here if there will be any news from my side.

[ingus.vilnis]

Hello Aleksey!

Are you still experiencing problems with Zabbix agent memory leak on Windows? If so, could you please try zabbix_agentd.exe version from here (https://www.dropbox.com/s/k89lix1ypi...agentd.exe.zip)?

In case the memory usage still grows, is it possible for you to troubleshoot it by disabling items on your problematic host filtering items by key type?
Something like this:

• first disable all items with key "perf_counter", then check memory usage
• if no positive changes, disable all items with "vfs.fs.size", then check memory usage again
• etc, etc, until you probably get to the point that the memory usage is not growing anymore.

In such way we could at least get the item key or type which is probably causing the leak.

Best Regards,
Ingus

[aib]

Thank you, Ingus!
I will check it.

Just FYI - I tried to disable the different keys to find the bad one. Even with agent.version and agent.ping enabled, the memory grow up.

Will see how it works with new version.

[aib]

First news:
- after one hour of collecting information from agent* keys (agent.hostname, agent.ping, agent.version) NO any changes in amount of using memory was noticed.

- then I added some system* keys :
system.cpu.load[,avg1]
system.cpu.load[,avg5]
system.cpu.load[,avg15]
system.cpu.load[percpu,avg1]
system.cpu.load[percpu,avg5]
system.cpu.load[percpu,avg15]
system.cpu.num
system.cpu.util[all]
system.swap.size[,free]
system.swap.size[,total]
system.uname
system.uptime

- after two hours the amount of memory increased from 7,156 kb to 7,332 kb and still growing up.

- after four hours the amount of memory increased from 7,156 kb to 7,680 kb and still growing up.

- after 22 hours the amount of memory increased from 7,156 kb to 9,132 kb and stopped.

Then I disabled almost all checks and enabled only:
- agent.hostname
- agent.ping
- agent.version
- system.uname
- system.uptime

So far - after four hours - the amount of memory grown up from 6,544 kb to 6,876 kb.

Then, I added two perf_counter's
- perf_counter[\Process(zabbix_agentd.e)\Private Bytes]
- perf_counter[\Process(zabbix_agentd.e)\Virtual Bytes]

And the memory starts leaking quicker.
After 18 hours the amount of memory grown up from 7,460 kb to 16,344 kb.

[ingus.vilnis]

Hi Aleksey,

Thank you for sharing such detailed results! However the news are still not good.

This is an important issue and I really appreciate your help on testing this! However it is hard to reproduce and even harder to find out the real reason for this problem. I will keep you updated when there will be some fix or at least another version to try.

Best Regards,
Ingus

[aib]

Thank you, Ingus!

I worry about that because one day I had my Windows 2000 server shutted down.
When I checked "Why?" he told me that happens because of lack of memory.
Zabbix_agent used more than 400 mb of memory and killed server.

so far I can say that system.* keys has something which use memory but didn't release it.