Zabbix 5.0 High memory utilization php-fpm

[johng]

We had a Zabbix 4.4 server running for about a month. We decided to drop the DB and do a fresh install for Zabbix 5.0.1 server on AWS t3.small(2GB memory). It ran out of memory in a few days. I relaunched the server as t3.medium(4GB memory). It ran out of memory. Does anyone know why php-fpm is using so much memory?

The Zabbix server is CentOS 7. The database is external RDS mysql.

[root@zabbixserver01 log]# rpm -qa | grep  zabbix*
zabbix-server-mysql-5.0.1-1.el7.x86_64
zabbix-get-5.0.1-1.el7.x86_64
zabbix-web-deps-scl-5.0.1-1.el7.noarch
zabbix-apache-conf-scl-5.0.1-1.el7.noarch
zabbix-web-5.0.1-1.el7.noarch
zabbix-web-mysql-scl-5.0.1-1.el7.noarch
zabbix-agent-5.0.1-1.el7.x86_64

[root@zabbixserver01 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           3729        3207         150         159         371         114
Swap:             0           0           0
[root@zabbixserver01 ~]# systemctl restart rh-php72-php-fpm
[root@zabbixserver01 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           3729         725        2623         159         381        2592
Swap:             0           0           0

[root@zabbixserver01 log]# more /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
[zabbix]
user = apache
group = apache

listen = /var/opt/rh/rh-php72/run/php-fpm/zabbix.sock
listen.acl_users = apache
listen.allowed_clients = 127.0.0.1

pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35

php_value[session.save_handler] = files
php_value[session.save_path] = /var/opt/rh/rh-php72/lib/php/session/

php_value[max_execution_time] = 300
php_value[memory_limit] = 128M
php_value[post_max_size] = 16M
php_value[upload_max_filesize] = 2M
php_value[max_input_time] = 300
php_value[max_input_vars] = 10000
php_value[date.timezone] = America/New_York

I created a phpinfo.php file. That seems to verify the memory_limit is 128M.

<?php
    phpinfo();
?>

[ingus.vilnis]

Try pm = ondemand instead of dynamic. Restart php-fpm to apply. Observe in longer run if the memory is still consumed.

[davlanas]

Same problem with my zabbix server. Operation system same CentOS7. I update zabbix server to 5.0.2 version? but problem not resolved. I setup pm in zabbix.conf (/etc/opt/rh/rh-php72/php-fpm.d) in ondemand, but problem stay. I minimaze children process from 50 to 10 and memory limit from 128M to 64M, but problem saved. What i do not right?

[davlanas]

Same i change php_value[max_execution_time] = 300 to 30 and php_value[max_input_time] = 300 to 30.

[tim.mooney]

You don't need to change max_execution_time or max_input_time, those were fine.

My site has seen some memory issues with RHEL 7's rh-php72 FPM workers under certain conditions with other PHP applications. It appeared that the workers would (under some unknown condition) use much more virtual memory than they should be able to based on their memory_limit.

What we've done to help avoid the problem is to set pm.max_requests, to force workers to die off after they've served so many requests. This way, if they have somehow allocated more virtual memory than they should be able to, they are forced to die off and free the memory.

Also, the Zabbix front-end probably doesn't need to keep around so many spare servers. You could likely set pm.max_spare_servers to something like 10 or 15 and not notice any difference in performance of the web interface.

[davlanas]

What number of max.request you set?

[tim.mooney]

We were seeing the very high memory use from rh-php72 on a very busy application, so we set pm.max_requests to 1000. It was high enough so that FPM workers weren't dying immediately but low enough so they weren't staying around for days.

For the Zabbix front end, which should be much lower traffic, you can likely get by with a lower value. It looks like from your more recent post that you're trying 100.

Remember too that the PHP-FPM pool of workers can optionally be configured to respond to a special URL (like Apache httpd's /server-info and /server-status locations) to give you some status information about your worker pool, including whether you've ever reached your "max children". You can use this to help determine if you have enough workers, how many are idle, etc. You could even monitor it with Zabbix, if you wanted. :-)

[lawdt]

hi also have this problem. same config - centos 7 and zabbix 5, rhel's php-fpm 7.2

[davlanas]

Ok! I setup pm.max_requests = 100 and we'll see.

[davlanas]

I have been testing my config for 5 hours and the graph looks good.
[zabbix]
user = apache
group = apache

listen = /var/opt/rh/rh-php72/run/php-fpm/zabbix.sock
listen.acl_users = apache
listen.allowed_clients = 127.0.0.1

pm = ondemand
pm.max_children = 10
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 100

php_value[session.save_handler] = files
php_value[session.save_path] = /var/opt/rh/rh-php72/lib/php/session/

php_value[max_execution_time] = 30
php_value[memory_limit] = 64M
php_value[post_max_size] = 16M
php_value[upload_max_filesize] = 2M
php_value[max_input_time] = 30
php_value[max_input_vars] = 10000
php_value[date.timezone] = Asia/Novokuznetsk
About 1,700 nodes are monitored. The web-interface does not slow down, but all other changes did not give any result at all.

[davlanas]

Problem solved
pm.max_requests = 100
And now memmory usage 0.9GB

[stevensedory]

Problem solved
pm.max_requests = 100
And now memmory usage 0.9GB

Thank you!!

[Tariq Hasan]

Hi
I have the same issue but don't know where to start.
I used an AWS AMI from the marketplace for the Server & DB.
Memory is leaking constantly.
Any ideas would be appreciated - please help - I am stuck.

[Tariq Hasan]

Hi
I have the same issue but don't know where to start.
I used an AWS AMI from the marketplace for the Server & DB.
Memory is leaking constantly.
Any ideas would be appreciated - please help - I am stuck.
Regards
Tariq

[tim.mooney]

Hi
I have the same issue but don't know where to start.
I used an AWS AMI from the marketplace for the Server & DB.
Memory is leaking constantly.
Any ideas would be appreciated - please help - I am stuck.
Regards
Tariq

Are you certain it's the same issue? Is it the php-fpm processes that are using far more virtual memory than they should be, or is it some other component? If it's php-fpm specifically, then as described earlier in the thread, you can modify the PHP-FPM worker pool config file to set pm.max_requests = 500 or something similar.

Where that config file is located depends on your distro and other things. It should be in a directory called php-fpm.d . On RHEL 7.x with php 7.2 from software collections library (SCL), The directory is /etc/opt/rh/rh-php72/php-fpm.d/ and the config file is the www.conf in that directory.

[Tariq Hasan]

Are you certain it's the same issue? Is it the php-fpm processes that are using far more virtual memory than they should be, or is it some other component? If it's php-fpm specifically, then as described earlier in the thread, you can modify the PHP-FPM worker pool config file to set pm.max_requests = 500 or something similar.

Where that config file is located depends on your distro and other things. It should be in a directory called php-fpm.d . On RHEL 7.x with php 7.2 from software collections library (SCL), The directory is /etc/opt/rh/rh-php72/php-fpm.d/ and the config file is the www.conf in that directory.

Hi Tim
Thanks for the reply
I have created the php file exactly as above.
Here is my memory usage graph in zabbix for the server - the hourly dips are when I restart the Sophos MTDD (Malicious Traffic Detection Daemon) - then it slowly increases again.

[Cossy]

Hi,

We have a similar issue with a linear increase in memory utilisation right up until it hits 100% and things start crashing.

Here's our zabbix.conf file with the highlighted additional line for pm.max_requests:

# cat /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
;THIS FILE IS MANAGED BY PUPPET
[zabbix]
user = apache
group = apache

listen = /var/opt/rh/rh-php72/run/php-fpm/zabbix.sock
listen.acl_users = apache
listen.allowed_clients = 127.0.0.1

pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.max_requests = 100

php_value[session.save_handler] = files
php_value[session.save_path] = /var/opt/rh/rh-php72/lib/php/session/

php_value[max_execution_time] = 300
php_value[memory_limit] = 128M
php_value[post_max_size] = 16M
php_value[upload_max_filesize] = 2M
php_value[max_input_time] = 300
php_value[always_populate_raw_post_data] = -1
php_value[max_input_vars] = 1000
;Set correct timezone
php_value[date.timezone] = Europe/London


Does anyone have any further suggestions as to what might be the underlying cause of the issue?

Regards,
Cossy

[Atsushi]

Check out the past posts in this thread. You are still using dynamic as the pm setting. Try changing to ondemand and restarting php-fpm.