Zabbix 4.4.10 trapper refuses connections

[rpjeff]

Zabbix version 4.4.10 installed via rpm on CentOS 7.8.2003 in GCP with Cloud SQL Postgres.


Parameter Value Details
Zabbix server is running Yes localhost:10051
Number of hosts (enabled/disabled/templates) 6673 6640 / 6 / 27
Number of items (enabled/disabled/not supported) 544213 530436 / 456 / 13321
Number of triggers (enabled/disabled [problem/ok]) 403372 396375 / 6997 [9496 / 386879]
Number of users (online) 31 5
Required server performance, new values per second 98.78


After a period of time (about 3 days) the trapper stops accepting connections and the log show that Zabbix has been unable to run external scripts due to "[12] Cannot allocate memory". The memory seem to decrease over a number of days from when the server is restarted. The other thing, which may or may not be related, is that database CPU is high before hand. Where's a good place to start investigating this?

[rpjeff]

This is starting to resemble a memory leak. The graph below shows the free memory on the VM between restarts of the zabbix server. As can be seen the available memory decreases over three days. Are there any known memory leaks with this version of zabbix (4.4.10 installed via rpm zabbix-server-pgsql-4.4.10-1.el7.x86_64 from the zabbix repo)?

[anonamoose]

Did this ever get resolved?

I used valgrind and this was happening:

==10236== Invalid write of size 8
==10236== at 0x15FE52: ??? (in /usr/sbin/zabbix_server_mysql)
==10236== by 0x160A55: pinger_thread (in /usr/sbin/zabbix_server_mysql)
==10236== by 0x2D59C8: zbx_thread_start (in /usr/sbin/zabbix_server_mysql)
==10236== by 0x14EF25: MAIN_ZABBIX_ENTRY (in /usr/sbin/zabbix_server_mysql)
==10236== by 0x2C6B05: daemon_start (in /usr/sbin/zabbix_server_mysql)
==10236== by 0x14E387: main (in /usr/sbin/zabbix_server_mysql)
==10236== Address 0x1ffebc6078 is on thread 1's stack


Looking into that online, some suggestions where that C/C++ code may not be doing garbage collection. No matter how much memory I give the server, it just slowly consumes it until it's all used up. A mariadb.service restart will free up the memory and then it begins to fill again.

So far I've tried a bunch of different tweaks to php-fpm configuration files to no effect. I was hoping that it was the process manager not killing off workers but it seems that isn't the issue. I am currently on mariadb 10.4 and Centos 7.8.

Thanks.