Zabbix 7 possible memory leak?

[cptneko]

Hello everyone!

We have recently moved over to Zabbix 7, and since then there is a noticeable, constant, decrease in available memory on the server. A day ago it hit the point where it just "went downhill" and by the end, zabbix_server had been consuming over 17 gigabytes of memory.


The migration took place on 06-07, we are using Timescaledb, and this isn't a particularly big installation of around 900 hosts with NVPS around 650. PG and zabbix_server are on the same computer, with no HA. There hadn't been any change to it except the migration to pg16 (from 15) and zabbix 6 to 7.

How could I start diagnosing this issue?

[seacreature]

Since the upgrade to Zabbix 7 we're also seeing a constant decrease in available memory on our Zabbix server. We make use of MariaDB and only have 38 hosts in Zabbix.



This memory seems to be largely consumed by the zabbix_server process (sits at ~35% in top).

If there is anything I can do to help with debugging this, please let me know!

[cptneko]

Still an issue, after the previous post it is currently eating 16GB of memory, which is slowly climbing. All seem to be private memory, not shared so assuming not the caches are doing it.

[Markku]

I suggest voting for and following ZBX-24712 (shortlink to "Possible memory leak in zabbix_server process")

Markku

[kbcetn]

Markku - that issue is closed so voting for it is not possible. Yet, like others, I'm still having the issue as well.

└──$ free -mt
total used free shared buff/cache available
Mem: 15989 8571 193 72 7223 6996
Swap: 4095 3385 710
Total: 20085 11957 903
└──$ sudo systemctl restart zabbix-server.service
└──$ free -mt
total used free shared buff/cache available
Mem: 15989 1778 9781 66 4429 13795
Swap: 4095 326 3769
Total: 20085 2104 13551
└──$

Note that Zabbix-Server had only been running for 19 days.

[Markku]

Markku - that issue is closed so voting for it is not possible. Yet, like others, I'm still having the issue as well.

As shown in the issue, it was fixed in 7.0.1rc1 (rc = release candidate), and 7.0.1 was released yesterday.

Markku

[[email protected]]

Hello

I have Zabbix version 6.0.21 , i downloaded Kubernetes Templates (Kubernetes API server by HTTP, Kubernetes cluster state by HTTP, Kubernetes Kubelet by HTTP, Kubernetes nodes by HTTP and Kubernetes Scheduler by HTTP) and assign them to Kuebrnetes host, when i enable it it leads to memory leak as shown below - Please check advise :



Here some outputs from zabbix logs:

ly exists at this OID
zabbix_server.log:170946:20240918:190338.602 item "Kubectl - Microservice - Live:hrStorageDescr[/run/user/1004]" became not supported: No Such Instance currently exists
at this OID
zabbix_server.log:170946:20240918:190338.606 item "Kubectl - Microservice - Live:hrStorageSize[/run/user/1004]" became not supported: No Such Instance currently exists a
t this OID
zabbix_server.log:170969:20240918:193200.977 item "Microservice Live Cluster:kube.pod.get[mongobackup-job-incremental-28777860-59qk5]" became not supported: Preprocessin
g failed for: {"nodes":[{"filternode":{"metadata":{"name":"microservice-live-9191fd-master-0","uid":"4a13a2fe-5...
zabbix_server.log:171019:20240918:194809.657 item "Kubectl - Microservice - Live:hrStorageSizeInBytes[/run/user/1004]" became not supported: Cannot evaluate function: it
em "/Kubectl - Microservice - Live/hrStorageSize[/run/user/1004]" is not supported at "last(//hrStorageSize[/run/user/1004]) * last(//hrStorageAllocationUnits[/run/user/
1004])".
zabbix_server.log:171006:20240918:200200.895 item "Microservice Live Cluster:kube.pod.get[mongobackup-job-incremental-28777890-nh979]" became not supported: Preprocessin
g failed for: {"nodes":[{"filternode":{"metadata":{"name":"microservice-live-9191fd-master-0","uid":"4a13a2fe-5...
zabbix_server.log:171021:20240918:203301.296 item "Microservice Live Cluster:kube.pod.get[mongobackup-job-incremental-28777920-wwvhd]" became not supported: Preprocessin
g failed for: {"nodes":[{"filternode":{"metadata":{"name":"microservice-live-9191fd-master-0","uid":"4a13a2fe-5...
zabbix_server.log:171023:20240918:210301.661 item "Microservice Live Cluster:kube.pod.get[mongobackup-job-incremental-28777950-2clhg]" became not supported: Preprocessin
g failed for: {"nodes":[{"filternode":{"metadata":{"name":"microservice-live-9191fd-master-0","uid":"4a13a2fe-5...
zabbix_server.log:170966:20240918:211654.191 slow query: 14.610175 sec, "insert into history_str (itemid,clock,ns,value) values (1869489,1726683510,632866667,'IT-WEB1'),
(1869565,1726683512,171303956,'TR69'),(1869906,172 6683511,145775873,'rnt-3gacc1.idm.net.lb'),(1869907,1726683512,165889233, 'rnt-3gacc2.idm.net.lb'),(3278165,1726683512,1
66500853,'rspamd15'),(4047102,1726683511,146385153 ,'ns12.idm.net.lb'),(4047226,1726683512,169217854, 'ns13.idm.net.lb'),(4275016,1726683512,175862539,' untangle.example.co
m'),(4288337,1726683513,188147785,'rspamd11'),(526 2654,1726683512,245863447,'microservice-live-9191fd-worker-4'),(5264436,1726683511,533249364,'microservice-live-9191fd-
worker-6'),(5284597,1726683513,187578770,'system');
zabbix_server.log:170962:20240918:211654.294 slow query: 14.373905 sec, "insert into history_str (itemid,clock,ns,value) values (1869499,1726683521,179671079,'dnstest'),
(1869518,1726683521,219284771,'srv'),(1869646,1726 683520,220172163,'EM-N1.emcl.idm.net.lb'),(1926307,1726683520,269037093 ,'PowerBI'),(3145436,1726683520,279169450,'rspam
d9'),(3183534,1726683520,370691805,'claudiusfe'),( 3186336,1726683521,245162585,'FE-DB'),(4269528,1726683520,257784570,'RSPAMD-corp-new-2'),(5262687,1726683520,264819175,
'microservice-live-9191fd-worker-5');
zabbix_server.log:170988:20240918:211654.311 slow query: 14.374360 sec, "insert into history_str (itemid,clock,ns,value) values (1869607,1726683521,255914890,'Traverse11
-WS'),(1869647,1726683521,281626843,'EM-N2.emcl.idm.net.lb'),(1869756,1726683521,271046938 ,'econet-lb-backend-1'),(1869757,1726683522,275075264,'econet-lb-backend-2'),(4
034476,1726683521,518153242,'nx-AHV2'),(4295264,1726683521,518153242,'Still working.'),(5264832,1726683521,708470009,'microser vice-live-9191fd-worker-8');
zabbix_server.log:170965:20240918:211654.402 slow query: 14.440370 sec, "insert into history_str (itemid,clock,ns,value) values (1557235,1726683531,276894742,'ST10000NM0
016-1TT101'),(1557236,1726683531,281665463,'ST10000NM0 17B-2TL103'),(1557237,1726683531,286432785,'ST10000NM0 01G-2MW103'),(1557238,1726683531,284606647,'ST10000NM0 086-2AA
101'),(1557239,1726683531,290514307,'ST10000NM0016-1TT101'),(1557240,1726683531,277481905,'ST10000NM0 01G-2MW103'),(1557241,1726683531,277491836,'ST10000NM0 01G-2MW103'),(
1557242,1726683531,276931992,'ST10000NM0086-2AA101'),(1557243,1726683529,321272025,'ST10000NM0 016-1TT101'),(1557244,1726683530,114893679,'ST10000NM0 016-1TT101'),(1557245
,1726683531,285437043,'ST10000NM0016-1TT101'),(1557246,1726683531,274380120,'ST10000NM0 086-2AA101'),(1557247,1726683531,275669978,'ST10000NM0 086-2AA101'),(1557248,172668
3531,288420904,'ST10000NM0016-1TT101'),(1557249,1726683531,274620532,'ST10000NM0 01G-2MW103'),(1557250,1726683531,286433367,'ST10000NM0 016-1TT101'),(1557251,1726683531,27
9244553,'ST10000NM001G-2MW103'),(1557252,1726683531,287411903,'WD101EFBX-68B0AN0'),(1557253,1726683531,283509158,'WD102KFBX-68M95N0'),(1557254,1726683529,321841413,'ST10
000NM0086-2AA101'),(1557255,1726683531,281627936,'ST10000NM0 086-2AA101'),(1557256,1726683531,278064186,'ST10000NM0 086-2AA101'),(1557301,1726683531,277441835,'SATA'),(15 5
7302,1726683531,282923164,'SATA'),(1557303,1726683 531,290576366,'SATA'),(1557304,1726683529,32153720 9,'SATA'),(1557305,1726683531,278332577,'SATA'),(1 557306,1726683531,2
74180826,'SATA'),(1557307,1726683531,274676637,'SA TA'),(1557308,1726683531,284544892,'SATA'),(155730 9,1726683531,283475506,'SATA'),(1557310,1726683531 ,278247939,'SATA'),
(1557311,1726683531,274087579,'SATA'),(1557312,172 6683531,281830998,'SATA'),(1557313,1726683531,2885 95936,'SATA'),(1557314,1726683531,284419526,'SATA' ),(1557315,17266835
31,282595259,'SATA'),(1557316,1726683531,277329878 ,'SATA'),(1557317,1726683531,274145578,'SATA'),(15 57318,1726683531,274311291,'SATA'),(1557319,172668 3530,115573098,'SAT
A'),(1557320,1726683531,282643487,'SATA'),(1557321 ,1726683531,291449113,'SATA'),(1557322,1726683531, 287386115,'SATA'),(1650680,1726683531,279244167,'S T10000NM0086-2AA101
'),(1650681,1726683529,321975197,'ST10000NM0086-2AA101'),(1650682,1726683529,321203028,'ST10000NM0 086-2AA101'),(1650683,1726683531,290473807,'ST10000NM0 086-2AA101'),(165
0684,1726683531,281441001,'ST10000NM0086-2AA101'),(1650685,1726683530,114763360,'ST10000NM0 086-2AA101'),(1650686,1726683531,277684303,'ST10000NM0 086-2AA101'),(1650687,17
26683529,322403657,'ST10000NM0086-2AA101'),(1650688,1726683531,281696405,'ST10000NM0 086-2AA101'),(1650689,1726683531,277563031,'WUH721414A L5204'),(1650690,1726683530,115
034148,'WUH721414AL5204'),(1650691,1726683531,2803 77319,'WUH721414AL5204'),(1650716,1726683531,28134 4580,'SATA'),(1650717,1726683531,284506834,'SATA') ,(1650718,172668353
1,278196372,'SATA'),(1650719,1726683531,291488193, 'SATA'),(1650720,1726683531,274851632,'SATA'),(165 0721,1726683531,277556703,'SATA'),(1650722,1726683 531,277090450,'SATA
'),(1650723,1726683529,322011808,'SATA'),(1650724, 1726683531,278233876,'SATA'),(1650725,1726683531,2 81512128,'SAS'),(1650726,1726683531,277622056,'SAS '),(1650727,1726683
529,321479120,'SAS'),(1666535,1726683529,321385155 ,'WUH721414AL5204'),(1666536,1726683531,280372369, 'ST10000NM0086-2AA101'),(1666541,1726683531,275426133,'SAS'),(166 6542
,1726683531,274729962,'SATA'),(1869508,1726683533, 272664692,'WebSQL'),(1869715,1726683530,114900280, 'RackStation'),(1933522,1726683528,573763178,'nse2 .cyberia.net.lb'),(
1943105,1726683531,210003608,'Synology'),(1943342, 1726683531,210003608,'ST4000VN008-2DR166'),(1943343,1726683531,210003608,'ST4000VN00 8-2DR166'),(1943348,1726683531,2100
03608,'SATA'),(1943349,1726683531,210003608,'SATA' ),(3141731,1726683529,321470486,'WUH721414AL5204') ,(3141732,1726683529,321395906,'WUH721414AL5204'), (3141733,1726683529
,322987960,'ST10000NM001G-2MW103'),(3141734,1726683531,288349786,'WUH721414A L5204'),(3141735,1726683531,277389975,'WUH721414AL 5204'),(3141736,1726683531,275936481,'WUH72
1414AL5204'),(3141737,1726683531,284388249,'WUH721 414AL5204'),(3141738,1726683531,280336774,'ST10000 NM0016-1TT101'),(3141739,1726683531,274403695,'SUV500M896 0G'),(314174
0,1726683531,279167745,'SUV500M8960G'),(3141761,17 26683531,282785827,'SAS'),(3141762,1726683530,1148 94698,'SAS'),(3141763,1726683531,275573429,'SATA') ,(3141764,172668353
1,281478038,'SAS'),(3141765,1726683531,288379067,' SAS'),(3141766,1726683531,282838763,'SAS'),(314176 7,1726683531,276892124,'SAS'),(3141768,1726683529, 321976473,'SATA'),(
3141769,1726683531,278064636,'SATA'),(3141770,1726 683530,114712830,'SATA'),(3178390,1726683529,25413 1056,'atlanta'),(3186266,1726683528,443804609,'eco netlb-production-ku
be-7'),(4283879,1726683529,321660289,'454541'),(42838 80,1726683531,279431425,'182'),(4283881,1726683531 ,279397616,'9895'),(4283882,1726683529,322523650,' 61'),(4283883,17
26683530,114926520,'0'),(4283885,1726683531,280537 969,'1992201354'),(4283886,1726683531,274213038,'1 541816609'),(4283887,1726683529,322362316,'1188795 877'),(4283888,1726
683531,276995430,'1605824408'),(4389670,1726683531 ,210003608,'212971'),(4389671,1726683531,210003608 ,'339'),(4389672,1726683531,210003608,'841'),(4389 673,1726683531,2100
03608,'1'),(4389674,1726683531,210003608,'0'),(438 9675,1726683531,210003608,'596163383'),(5262522,17 26683530,171178138,'microservice-live-9191fd-worker-0'),(5262588,1726
683528,432299336,'microservice-live-9191fd-worker-2');

[Daham Kumarapathirana]

I'm Still experiencing this issue in 2025 - November. Any Updates for a Fix ?