High Memory Usage Following V5 Upgrade...

[Cossy]

Dear Forum Contributors,

We have recently upgraded our Zabbix environment from v4 to v5 and we now see a perfectly linear increase in memory consumption right up until it hits 100% and things start failing.

If I execute a top -o %MEM command, I can see that most of the memory is consumed by php-fpm processes...

top - 12:19:34 up 21 days, 23:47, 1 user, load average: 0.08, 0.15, 0.28
Tasks: 247 total, 1 running, 246 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.7 us, 0.5 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 3663144 total, 123608 free, 3409056 used, 130480 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 60108 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18893 zabbix 20 0 566272 206736 5988 S 0.0 5.6 0:08.46 /usr/sbin/zabbix_server: housekeeper [deleted 55718 hist/trends, 0 items/triggers, 5 events,+
21696 apache 20 0 453720 110120 3644 S 1.7 3.0 21:33.90 php-fpm: pool zabbix
21697 apache 20 0 452888 109220 3624 S 0.0 3.0 21:38.78 php-fpm: pool zabbix
21699 apache 20 0 452772 109048 3588 S 0.0 3.0 21:35.72 php-fpm: pool zabbix
21707 apache 20 0 452500 108848 3612 S 0.0 3.0 21:30.38 php-fpm: pool zabbix
21698 apache 20 0 452980 108624 3648 S 1.0 3.0 21:41.25 php-fpm: pool zabbix
23616 apache 20 0 421148 79484 3604 S 0.0 2.2 14:12.66 php-fpm: pool zabbix
23620 apache 20 0 423444 79420 3632 S 0.0 2.2 14:19.32 php-fpm: pool zabbix
23618 apache 20 0 422772 79228 3684 S 0.0 2.2 14:06.94 php-fpm: pool zabbix
23617 apache 20 0 422828 79092 3604 S 0.0 2.2 14:08.63 php-fpm: pool zabbix
23614 apache 20 0 422764 78980 3596 S 0.0 2.2 14:11.37 php-fpm: pool zabbix
23613 apache 20 0 422448 78812 3616 S 0.0 2.2 14:12.39 php-fpm: pool zabbix
23615 apache 20 0 422496 78748 3560 S 0.0 2.1 14:09.79 php-fpm: pool zabbix
23609 apache 20 0 420316 78712 3620 S 0.0 2.1 14:09.27 php-fpm: pool zabbix
23608 apache 20 0 420180 78544 3612 S 0.0 2.1 14:15.99 php-fpm: pool zabbix
23607 apache 20 0 422088 78312 3644 S 0.0 2.1 14:19.09 php-fpm: pool zabbix
23619 apache 20 0 420544 78236 3668 S 0.0 2.1 14:13.18 php-fpm: pool zabbix
23610 apache 20 0 422360 78080 3592 S 0.0 2.1 14:14.04 php-fpm: pool zabbix
23611 apache 20 0 421544 77836 3600 S 0.0 2.1 14:11.84 php-fpm: pool zabbix
23612 apache 20 0 420152 77736 3636 S 0.0 2.1 14:12.60 php-fpm: pool zabbix
23606 apache 20 0 421740 76976 3640 S 0.0 2.1 14:12.41 php-fpm: pool zabbix
18904 zabbix 20 0 403632 55572 17376 S 0.0 1.5 0:22.29 /usr/sbin/zabbix_server: history syncer #6 [processed 0 values, 0 triggers in 0.000016 sec, +
10851 root 24 4 758236 54800 2504 S 0.7 1.5 116:12.91 /var/awslogs/bin/python /var/awslogs/bin/aws logs push --config-file /var/awslogs/etc/awslog+
18903 zabbix 20 0 404112 54108 15500 S 0.0 1.5 0:21.51 /usr/sbin/zabbix_server: history syncer #5 [processed 0 values, 0 triggers in 0.000020 sec, +
18907 zabbix 20 0 404040 53772 15236 S 0.3 1.5 0:22.93 /usr/sbin/zabbix_server: history syncer #9 [processed 0 values, 0 triggers in 0.000026 sec, +
18905 zabbix 20 0 401728 52320 16016 S 0.0 1.4 0:22.28 /usr/sbin/zabbix_server: history syncer #7 [processed 0 values, 0 triggers in 0.000020 sec, +
18909 zabbix 20 0 401780 51496 15192 S 0.0 1.4 0:21.59 /usr/sbin/zabbix_server: history syncer #11 [processed 0 values, 0 triggers in 0.000018 sec,+
18899 zabbix 20 0 401640 51300 15128 S 0.0 1.4 0:22.58 /usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000025 sec, +
18908 zabbix 20 0 400012 49712 15176 S 0.0 1.4 0:22.56 /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000011 sec,+
25712 apache 20 0 389472 47700 3672 S 0.0 1.3 7:19.43 php-fpm: pool zabbix
25707 apache 20 0 387304 45564 3624 S 0.0 1.2 7:24.42 php-fpm: pool zabbix
25718 apache 20 0 387068 45376 3632 S 1.0 1.2 7:27.78 php-fpm: pool zabbix
25716 apache 20 0 386896 45244 3640 S 0.0 1.2 7:25.37 php-fpm: pool zabbix
18901 zabbix 20 0 392996 45208 17720 S 0.0 1.2 0:25.20 /usr/sbin/zabbix_server: history syncer #3 [processed 0 values, 2 triggers in 0.001140 sec, +
25714 apache 20 0 388796 45164 3680 S 0.0 1.2 7:28.40 php-fpm: pool zabbix
18906 zabbix 20 0 395276 45116 15344 S 0.0 1.2 0:22.58 /usr/sbin/zabbix_server: history syncer #8 [processed 30 values, 13 triggers in 0.013190 sec+
25713 apache 20 0 386556 44928 3636 S 0.0 1.2 7:26.52 php-fpm: pool zabbix
25696 apache 20 0 386768 44640 3684 S 0.0 1.2 7:26.34 php-fpm: pool zabbix
25717 apache 20 0 388872 44568 3672 S 0.0 1.2 7:30.02 php-fpm: pool zabbix
18910 zabbix 20 0 394668 44476 15236 S 0.0 1.2 0:23.14 /usr/sbin/zabbix_server: history syncer #12 [processed 0 values, 0 triggers in 0.000013 sec,+
25710 apache 20 0 388152 44420 3672 S 0.0 1.2 7:23.89 php-fpm: pool zabbix
25697 apache 20 0 386584 44356 3672 S 0.0 1.2 7:25.93 php-fpm: pool zabbix
25687 apache 20 0 388872 44320 3696 S 1.7 1.2 7:29.82 php-fpm: pool zabbix
25705 apache 20 0 387096 44244 3648 S 0.0 1.2 7:30.22 php-fpm: pool zabbix
25711 apache 20 0 388544 44144 3620 S 0.0 1.2 7:26.52 php-fpm: pool zabbix


I found the following post that describes my situation exactly and I have modified the '/etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf' file to include the following entry and restarted the zabbix service but that does not appear to have resolved my problem.

pm.max_requests = 100

https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/402946-zabbix-5-0-high-memory-utilization-php-fpm

Does anyone have any other ideas?

Zabbix v6 appears to be out now, could that be an option?

Regards,
Cossy

[Cossy]

This issue was resolved by setting the following configuration values and restarting the rh-php72-php-fpm.service.

cat /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
;THIS FILE IS MANAGED BY PUPPET
[zabbix]
user = apache
group = apache

listen = /var/opt/rh/rh-php72/run/php-fpm/zabbix.sock
listen.acl_users = apache
listen.allowed_clients = 127.0.0.1

pm = ondemand
pm.max_children = 10
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 100
php_value[session.save_handler] = files
php_value[session.save_path] = /var/opt/rh/rh-php72/lib/php/session/

php_value[max_execution_time] = 30
php_value[memory_limit] = 64M
php_value[post_max_size] = 16M
php_value[upload_max_filesize] = 2M
php_value[max_input_time] = 30
php_value[always_populate_raw_post_data] = -1
php_value[max_input_vars] = 10000
;Set correct timezone
php_value[date.timezone] = Europe/London