Ad Widget

Collapse

Zabbix 5.0 High memory utilization php-fpm

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Cossy
    Junior Member
    • Mar 2022
    • 16

    #16
    Here's the output from a top command:

    [root@zabbix-i03c9f7f44ae7137f8 ~]# top -o %MEM
    top - 12:27:12 up 21 days, 23:55, 1 user, load average: 0.00, 0.04, 0.18
    Tasks: 247 total, 1 running, 246 sleeping, 0 stopped, 0 zombie
    %Cpu(s): 1.7 us, 0.3 sy, 0.0 ni, 97.5 id, 0.3 wa, 0.0 hi, 0.2 si, 0.0 st
    KiB Mem : 3663144 total, 150856 free, 3411808 used, 100480 buff/cache
    KiB Swap: 0 total, 0 free, 0 used. 72356 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    18893 zabbix 20 0 566272 206736 5988 S 0.0 5.6 0:08.46 /usr/sbin/zabbix_server: housekeeper [deleted 55718 hist/trends, 0 items/triggers, 5 events,+
    21696 apache 20 0 453832 109528 3672 S 0.0 3.0 21:35.34 php-fpm: pool zabbix
    21698 apache 20 0 453140 109528 3656 S 0.0 3.0 21:42.70 php-fpm: pool zabbix
    21697 apache 20 0 452952 109216 3616 S 0.0 3.0 21:40.14 php-fpm: pool zabbix
    21699 apache 20 0 452764 109180 3660 S 0.0 3.0 21:37.02 php-fpm: pool zabbix
    21707 apache 20 0 452500 108900 3640 S 0.7 3.0 21:31.60 php-fpm: pool zabbix
    23612 apache 20 0 422252 80488 3676 S 0.0 2.2 14:13.92 php-fpm: pool zabbix
    23620 apache 20 0 423444 79688 3668 S 0.0 2.2 14:20.69 php-fpm: pool zabbix
    23616 apache 20 0 421208 79596 3644 S 0.0 2.2 14:13.89 php-fpm: pool zabbix
    23618 apache 20 0 422888 79316 3676 S 0.0 2.2 14:08.06 php-fpm: pool zabbix
    23619 apache 20 0 420716 79012 3664 S 1.0 2.2 14:14.38 php-fpm: pool zabbix
    23613 apache 20 0 422436 78844 3636 S 0.0 2.2 14:13.73 php-fpm: pool zabbix
    23610 apache 20 0 422564 78820 3664 S 0.0 2.2 14:15.51 php-fpm: pool zabbix
    23609 apache 20 0 420396 78804 3612 S 0.0 2.2 14:10.52 php-fpm: pool zabbix
    23608 apache 20 0 420236 78628 3652 S 0.0 2.1 14:17.25 php-fpm: pool zabbix
    23617 apache 20 0 422828 78620 3644 S 0.0 2.1 14:10.12 php-fpm: pool zabbix
    23614 apache 20 0 422712 78548 3684 S 0.0 2.1 14:12.71 php-fpm: pool zabbix
    23615 apache 20 0 422496 78120 3632 S 0.0 2.1 14:11.09 php-fpm: pool zabbix
    23611 apache 20 0 421724 77932 3612 S 0.0 2.1 14:13.00 php-fpm: pool zabbix
    23607 apache 20 0 422228 77824 3648 S 0.0 2.1 14:20.50 php-fpm: pool zabbix
    23606 apache 20 0 421876 76996 3648 S 0.0 2.1 14:13.78 php-fpm: pool zabbix
    18904 zabbix 20 0 403632 55568 17372 S 0.0 1.5 0:22.36 /usr/sbin/zabbix_server: history syncer #6 [processed 0 values, 0 triggers in 0.000026 sec, +
    10851 root 24 4 758236 54772 2476 S 0.0 1.5 116:14.29 /var/awslogs/bin/python /var/awslogs/bin/aws logs push --config-file /var/awslogs/etc/awslog+
    18903 zabbix 20 0 404112 54108 15500 S 0.0 1.5 0:21.56 /usr/sbin/zabbix_server: history syncer #5 [processed 0 values, 0 triggers in 0.000011 sec, +
    18907 zabbix 20 0 404040 53772 15236 S 0.0 1.5 0:22.98 /usr/sbin/zabbix_server: history syncer #9 [processed 0 values, 0 triggers in 0.000011 sec, +
    18905 zabbix 20 0 401728 52424 16120 S 0.0 1.4 0:22.37 /usr/sbin/zabbix_server: history syncer #7 [processed 20 values, 19 triggers in 0.030571 sec+
    18909 zabbix 20 0 401780 51496 15192 S 0.0 1.4 0:21.66 /usr/sbin/zabbix_server: history syncer #11 [processed 1 values, 2 triggers in 0.011925 sec,+
    18899 zabbix 20 0 401640 51296 15124 S 0.0 1.4 0:22.64 /usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000012 sec, +
    18908 zabbix 20 0 400012 49712 15176 S 0.0 1.4 0:22.64 /usr/sbin/zabbix_server: history syncer #10 [processed 20 values, 5 triggers in 0.017287 sec+
    25712 apache 20 0 387440 45848 3676 S 0.0 1.3 7:20.77 php-fpm: pool zabbix
    25707 apache 20 0 387296 45700 3688 S 0.0 1.2 7:25.74 php-fpm: pool zabbix
    25705 apache 20 0 387224 45588 3648 S 0.0 1.2 7:31.55 php-fpm: pool zabbix
    25717 apache 20 0 388984 45456 3672 S 0.0 1.2 7:31.39 php-fpm: pool zabbix
    25687 apache 20 0 388944 45436 3696 S 0.0 1.2 7:31.41 php-fpm: pool zabbix
    18901 zabbix 20 0 392996 45208 17720 S 0.0 1.2 0:25.26 /usr/sbin/zabbix_server: history syncer #3 [processed 7 values, 9 triggers in 0.014482 sec, +
    25714 apache 20 0 388808 45140 3672 S 0.0 1.2 7:29.72 php-fpm: pool zabbix
    18906 zabbix 20 0 395276 45116 15344 S 0.0 1.2 0:22.67 /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000014 sec, +
    25697 apache 20 0 386708 44984 3676 S 0.0 1.2 7:27.20 php-fpm: pool zabbix
    25711 apache 20 0 388600 44924 3652 S 0.0 1.2 7:28.09 php-fpm: pool zabbix
    25718 apache 20 0 387020 44920 3696 S 0.0 1.2 7:29.37 php-fpm: pool zabbix
    25704 apache 20 0 386368 44840 3688 S 0.0 1.2 7:28.04 php-fpm: pool zabbix
    25706 apache 20 0 388268 44556 3668 S 0.0 1.2 7:29.85 php-fpm: pool zabbix
    25696 apache 20 0 386776 44500 3712 S 1.0 1.2 7:27.73 php-fpm: pool zabbix
    18910 zabbix 20 0 394668 44476 15236 S 0.0 1.2 0:23.18 /usr/sbin/zabbix_server: history syncer #12 [processed 2 values, 0 triggers in 0.012019 sec,+
    25710 apache 20 0 388104 44440 3676 S 0.0 1.2 7:25.12 php-fpm: pool zabbix
    25716 apache 20 0 386996 44300 3680 S 0.0 1.2 7:26.71 php-fpm: pool zabbix

    Does anyone have any further suggestions as to what might be the underlying cause of the issue?

    Regards,
    Cossy

    Comment

    • tim.mooney
      Senior Member
      • Dec 2012
      • 1427

      #17
      I'm pretty sure the memory utilization issues are a bug in either PHP 7.2 or in Red Hat's build of PHP 7.2. It could also be in one of the libraries that FPM links with.

      Beyond tuning pm.max_requests = 100, as you have, the next thing I would suggest is what I said in an earlier post: you probably don't need 35 spare servers for the Zabbix front end. pm.max_spare_servers should be an integer multiple of pm.min_spare_servers, so try setting pm.max_spare_servers = 10 and restart your worker pool.

      As I mentioned earlier, if you want a little bit of visibility into the usage of that worker pool, it's also possible to enable a status URL for the pool, so you can see how many workers you've had active (max) at one time, what each worker is doing, etc.

      Comment

      • Cossy
        Junior Member
        • Mar 2022
        • 16

        #18
        Hi Tim,

        Thanks for the reply and suggestions.

        I have amended the zabbix.conf file as suggested and will restart the entire Zabbix service as I do not know how to restart the worker pool.

        cat /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
        ;THIS FILE IS MANAGED BY PUPPET
        [zabbix]
        user = apache
        group = apache

        listen = /var/opt/rh/rh-php72/run/php-fpm/zabbix.sock
        listen.acl_users = apache
        listen.allowed_clients = 127.0.0.1

        pm = ondemand
        pm.max_children = 10
        pm.start_servers = 5
        pm.min_spare_servers = 5
        pm.max_spare_servers = 10
        pm.max_requests = 100
        pm.max_servers = 10
        php_value[session.save_handler] = files
        php_value[session.save_path] = /var/opt/rh/rh-php72/lib/php/session/

        php_value[max_execution_time] = 30
        php_value[memory_limit] = 64M
        php_value[post_max_size] = 16M
        php_value[upload_max_filesize] = 2M
        php_value[max_input_time] = 30
        php_value[always_populate_raw_post_data] = -1
        php_value[max_input_vars] = 10000
        ;Set correct timezone
        php_value[date.timezone] = Europe/London

        Also, the php-fpm service doesnt look particularly healthy either, is this normal and expected or could this have something to do with the issue?

        systemctl status php-fpm.service
        ● php-fpm.service - The PHP FastCGI Process Manager
        Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; disabled; vendor preset: disabled)
        Active: failed (Result: exit-code) since Thu 2022-03-17 14:06:56 GMT; 17h ago
        Process: 1775 ExecStart=/usr/sbin/php-fpm --nodaemonize (code=exited, status=78)
        Main PID: 1775 (code=exited, status=78)

        Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: Starting The PHP FastCGI Process Manager...
        Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 php-fpm[1775]: [17-Mar-2022 14:06:56] ERROR: unable to bind listening socket for address '127.0.0.1:9000': Ad...use (98)
        Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 php-fpm[1775]: [17-Mar-2022 14:06:56] ERROR: FPM initialization failed
        Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: php-fpm.service: main process exited, code=exited, status=78/n/a
        Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: Failed to start The PHP FastCGI Process Manager.
        Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: Unit php-fpm.service entered failed state.
        Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: php-fpm.service failed.
        Hint: Some lines were ellipsized, use -l to show in full.

        Thanks again.

        Cossy

        Comment

        • tim.mooney
          Senior Member
          • Dec 2012
          • 1427

          #19
          Originally posted by Cossy
          I have amended the zabbix.conf file as suggested and will restart the entire Zabbix service as I do not know how to restart the worker pool.

          cat /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
          ;THIS FILE IS MANAGED BY PUPPET
          [zabbix]
          user = apache
          group = apache

          listen = /var/opt/rh/rh-php72/run/php-fpm/zabbix.sock
          listen.acl_users = apache
          listen.allowed_clients = 127.0.0.1

          pm = ondemand
          pm.max_children = 10
          pm.start_servers = 5
          pm.min_spare_servers = 5
          pm.max_spare_servers = 10
          pm.max_requests = 100
          pm.max_servers = 10
          php_value[session.save_handler] = files
          php_value[session.save_path] = /var/opt/rh/rh-php72/lib/php/session/

          php_value[max_execution_time] = 30
          php_value[memory_limit] = 64M
          php_value[post_max_size] = 16M
          php_value[upload_max_filesize] = 2M
          php_value[max_input_time] = 30
          php_value[always_populate_raw_post_data] = -1
          php_value[max_input_vars] = 10000
          ;Set correct timezone
          php_value[date.timezone] = Europe/London

          Also, the php-fpm service doesnt look particularly healthy either, is this normal and expected or could this have something to do with the issue?

          systemctl status php-fpm.service
          ● php-fpm.service - The PHP FastCGI Process Manager
          Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; disabled; vendor preset: disabled)
          Active: failed (Result: exit-code) since Thu 2022-03-17 14:06:56 GMT; 17h ago
          Process: 1775 ExecStart=/usr/sbin/php-fpm --nodaemonize (code=exited, status=78)
          Main PID: 1775 (code=exited, status=78)

          Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: Starting The PHP FastCGI Process Manager...
          Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 php-fpm[1775]: [17-Mar-2022 14:06:56] ERROR: unable to bind listening socket for address '127.0.0.1:9000': Ad...use (98)
          Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 php-fpm[1775]: [17-Mar-2022 14:06:56] ERROR: FPM initialization failed
          Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: php-fpm.service: main process exited, code=exited, status=78/n/a
          Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: Failed to start The PHP FastCGI Process Manager.
          Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: Unit php-fpm.service entered failed state.
          Mar 17 14:06:56 zabbix-i03c9f7f44ae7137f8 systemd[1]: php-fpm.service failed.
          Hint: Some lines were ellipsized, use -l to show in full.
          Both in this post and your previous one you show a PHP-FPM pool configuration file that references "rh-php72" in a couple places, which immediately made me assume you were using Red Hat (or derivative) software collection library (SCL) rh-php72 components.

          However, now that you show your output from "systemctl status", something is very strange. If you're using the php-fpm that comes with SCL rh-php72, the service name is not 'php-fpm', it's 'rh-php72-php-fpm.service'. It's like you're mixing two different versions of PHP-FPM on the system. Did you try Remi's PHP install at one point?

          On my RHEL 7.x server where the Zabbix front-end is currently running:

          Code:
          $ sudo systemctl list-units | egrep php
          rh-php72-php-fpm.service loaded active running The PHP FastCGI Process Manager
          
          $ sudo systemctl status php-fpm.service
          Unit php-fpm.service could not be found.
          
          $ sudo systemctl status rh-php72-php-fpm.service
          * rh-php72-php-fpm.service - The PHP FastCGI Process Manager
          Loaded: loaded (/usr/lib/systemd/system/rh-php72-php-fpm.service; enabled; vendor preset: disabled)
          Active: active (running) since Wed 2022-02-09 17:37:22 CST; 1 months 6 days ago
          Main PID: 1682 (php-fpm)
          Status: "Processes active: 0, idle: 10, Requests: 2570787, slow: 0, Traffic: 0.5req/sec"
          Tasks: 11
          Memory: 114.2M
          CGroup: /system.slice/rh-php72-php-fpm.service
          |- 1580 php-fpm: pool www
          |- 1682 php-fpm: master process (/etc/opt/rh/rh-php72/php-fpm.conf...
          |- 6567 php-fpm: pool www
          |-21267 php-fpm: pool www
          |-21379 php-fpm: pool www
          |-24674 php-fpm: pool www
          |-24820 php-fpm: pool www
          |-27902 php-fpm: pool www
          |-28631 php-fpm: pool www
          |-29792 php-fpm: pool www
          `-31974 php-fpm: pool www
          
          Feb 09 17:37:21 monitor.it.ndsu.edu systemd[1]: Starting The PHP FastCGI Pro....
          Feb 09 17:37:22 monitor.it.ndsu.edu systemd[1]: Started The PHP FastCGI Proc....
          Hint: Some lines were ellipsized, use -l to show in full.
          (Note: we call our php-fpm worker pool "www" rather than "zabbix", to match other PHP apps in our environment, but this is showing the worker pool for the Zabbix front-end).

          Can you clarify how you got to the point of having some config from SCL rh-php72 and some config that clearly isn't from SCL?

          Comment

          • Cossy
            Junior Member
            • Mar 2022
            • 16

            #20
            Hi Tim,

            Thanks for taking the time to reply and sharing your observations.

            I am very new to Zabbix so please bear with me.

            We do indeed have a 'rh-php72-php-fpm.service' so I guess I was looking at the wrong service - sorry. I will also ensure that the redundant php-fpm service is removed from systemd to avoid any future misunderstandings.

            Code:
            systemctl status rh-php72-php-fpm.service
            ● rh-php72-php-fpm.service - The PHP FastCGI Process Manager
            Loaded: loaded (/usr/lib/systemd/system/rh-php72-php-fpm.service; enabled; vendor preset: disabled)
            Active: active (running) since Wed 2022-02-23 12:38:53 GMT; 3 weeks 3 days ago
            Main PID: 11338 (php-fpm)
            Status: "Processes active: 0, idle: 40, Requests: 2824139, slow: 0, Traffic: 0.8req/sec"
            CGroup: /system.slice/rh-php72-php-fpm.service
            ├─11338 php-fpm: master process (/etc/opt/rh/rh-php72/php-fpm.conf)
            ├─11339 php-fpm: pool www
            ├─11340 php-fpm: pool www
            ├─11341 php-fpm: pool www
            ├─11342 php-fpm: pool www
            ├─11343 php-fpm: pool www
            ├─23616 php-fpm: pool zabbix
            ├─23617 php-fpm: pool zabbix
            ├─23618 php-fpm: pool zabbix
            ├─23619 php-fpm: pool zabbix
            ├─23620 php-fpm: pool zabbix
            ├─25687 php-fpm: pool zabbix
            ├─25696 php-fpm: pool zabbix
            ├─25697 php-fpm: pool zabbix
            ├─25704 php-fpm: pool zabbix
            ├─25705 php-fpm: pool zabbix
            ├─25706 php-fpm: pool zabbix
            ├─25707 php-fpm: pool zabbix
            ├─25710 php-fpm: pool zabbix
            ├─25711 php-fpm: pool zabbix
            ├─25712 php-fpm: pool zabbix
            ├─25713 php-fpm: pool zabbix
            ├─25714 php-fpm: pool zabbix
            ├─25716 php-fpm: pool zabbix
            ├─25717 php-fpm: pool zabbix
            ├─25718 php-fpm: pool zabbix
            ├─29974 php-fpm: pool zabbix
            ├─29975 php-fpm: pool zabbix
            ├─29976 php-fpm: pool zabbix
            ├─29977 php-fpm: pool zabbix
            ├─29978 php-fpm: pool zabbix
            ├─29979 php-fpm: pool zabbix
            ├─29980 php-fpm: pool zabbix
            ├─29981 php-fpm: pool zabbix
            ├─29983 php-fpm: pool zabbix
            ├─29984 php-fpm: pool zabbix
            ├─29985 php-fpm: pool zabbix
            ├─29986 php-fpm: pool zabbix
            ├─29987 php-fpm: pool zabbix
            ├─29990 php-fpm: pool zabbix
            └─29991 php-fpm: pool zabbix
            If I look at the conf file...

            cat /etc/opt/rh/rh-php72/php-fpm.conf
            Code:
            ;;;;;;;;;;;;;;;;;;;;;
            ; FPM Configuration ;
            ;;;;;;;;;;;;;;;;;;;;;
            
            ; All relative paths in this configuration file are relative to PHP's install
            ; prefix.
            
            ; Include one or more files. If glob(3) exists, it is used to include a bunch of
            ; files from a glob(3) pattern. This directive can be used everywhere in the
            ; file.
            include=/etc/opt/rh/rh-php72/php-fpm.d/*.conf
            
            ;;;;;;;;;;;;;;;;;;
            ; Global Options ;
            ;;;;;;;;;;;;;;;;;;
            
            [global]
            ; Pid file
            ; Default Value: none
            pid = /var/opt/rh/rh-php72/run/php-fpm/php-fpm.pid
            
            ; Error log file
            ; If it's set to "syslog", log is sent to syslogd instead of being written
            ; in a local file.
            ; Default Value: /var/opt/rh/rh-php72/log/php-fpm.log
            error_log = /var/opt/rh/rh-php72/log/php-fpm/error.log
            
            ; syslog_facility is used to specify what type of program is logging the
            ; message. This lets syslogd specify that messages from different facilities
            ; will be handled differently.
            ; See syslog(3) for possible values (ex daemon equiv LOG_DAEMON)
            ; Default Value: daemon
            ;syslog.facility = daemon
            
            ; syslog_ident is prepended to every message. If you have multiple FPM
            ; instances running on the same server, you can change the default value
            ; which must suit common needs.
            ; Default Value: php-fpm
            ;syslog.ident = php-fpm
            
            ; Log level
            ; Possible Values: alert, error, warning, notice, debug
            ; Default Value: notice
            ;log_level = notice
            
            ; If this number of child processes exit with SIGSEGV or SIGBUS within the time
            ; interval set by emergency_restart_interval then FPM will restart. A value
            ; of '0' means 'Off'.
            ; Default Value: 0
            ;emergency_restart_threshold = 0
            
            ; Interval of time used by emergency_restart_interval to determine when
            ; a graceful restart will be initiated. This can be useful to work around
            ; accidental corruptions in an accelerator's shared memory.
            ; Available Units: s(econds), m(inutes), h(ours), or d(ays)
            ; Default Unit: seconds
            ; Default Value: 0
            ;emergency_restart_interval = 0
            
            ; Time limit for child processes to wait for a reaction on signals from master.
            ; Available units: s(econds), m(inutes), h(ours), or d(ays)
            ; Default Unit: seconds
            ; Default Value: 0
            ;process_control_timeout = 0
            
            ; The maximum number of processes FPM will fork. This has been designed to control
            ; the global number of processes when using dynamic PM within a lot of pools.
            ; Use it with caution.
            ; Note: A value of 0 indicates no limit
            ; Default Value: 0
            ;process.max = 128
            
            ; Specify the nice(2) priority to apply to the master process (only if set)
            ; The value can vary from -19 (highest priority) to 20 (lowest priority)
            ; Note: - It will only work if the FPM master process is launched as root
            ; - The pool process will inherit the master process priority
            ; unless specified otherwise
            ; Default Value: no set
            ;process.priority = -19
            
            ; Send FPM to background. Set to 'no' to keep FPM in foreground for debugging.
            ; Default Value: yes
            daemonize = yes
            
            ; Set open file descriptor rlimit for the master process.
            ; Default Value: system defined value
            ;rlimit_files = 1024
            
            ; Set max core size rlimit for the master process.
            ; Possible Values: 'unlimited' or an integer greater or equal to 0
            ; Default Value: system defined value
            ;rlimit_core = 0
            
            ; Specify the event mechanism FPM will use. The following is available:
            ; - select (any POSIX os)
            ; - poll (any POSIX os)
            ; - epoll (linux >= 2.5.44)
            ; Default Value: not set (auto detection)
            ;events.mechanism = epoll
            
            ; When FPM is built with systemd integration, specify the interval,
            ; in seconds, between health report notification to systemd.
            ; Set to 0 to disable.
            ; Available Units: s(econds), m(inutes), h(ours)
            ; Default Unit: seconds
            ; Default value: 10
            ;systemd_interval = 10
            
            ;;;;;;;;;;;;;;;;;;;;
            ; Pool Definitions ;
            ;;;;;;;;;;;;;;;;;;;;
            
            ; Multiple pools of child processes may be started with different listening
            ; ports and different management options. The name of the pool will be
            ; used in logs and stats. There is no limitation on the number of pools which
            ; FPM can handle. Your system will tell you anyway :)
            
            ; See /etc/opt/rh/rh-php72/php-fpm.d/*.conf
            Also, we are running the following version of Red Hat...

            Code:
            # cat /etc/os-release
            NAME="Red Hat Enterprise Linux Server"
            VERSION="7.9 (Maipo)"
            ID="rhel"
            ID_LIKE="fedora"
            VARIANT="Server"
            VARIANT_ID="server"
            VERSION_ID="7.9"
            PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
            ANSI_COLOR="0;31"
            CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
            HOME_URL="https://www.redhat.com/"
            BUG_REPORT_URL="https://bugzilla.redhat.com/"
            
            REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
            REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
            REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
            REDHAT_SUPPORT_PRODUCT_VERSION="7.9"
            top command by memory utilisation...

            Code:
            top - 17:14:00 up 24 days, 4:42, 2 users, load average: 0.06, 0.05, 0.05
            Tasks: 240 total, 1 running, 239 sleeping, 0 stopped, 0 zombie
            %Cpu(s): 0.2 us, 0.3 sy, 0.2 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
            KiB Mem : 3663144 total, 183468 free, 3194460 used, 285216 buff/cache
            KiB Swap: 0 total, 0 free, 0 used. 236924 avail Mem
            
            PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
            19504 zabbix 20 0 567204 204008 2320 S 0.0 5.6 0:08.01 /usr/sbin/zabbix_server: housekeeper [deleted 5 hist/trends, 0 items/triggers, 38 events, 0+
            23618 apache 20 0 451368 107992 3904 S 0.0 2.9 19:57.56 php-fpm: pool zabbix
            23619 apache 20 0 449148 107612 3908 S 0.0 2.9 20:04.49 php-fpm: pool zabbix
            23616 apache 20 0 447688 106280 3904 S 0.0 2.9 20:02.63 php-fpm: pool zabbix
            23620 apache 20 0 488980 106232 3900 S 0.0 2.9 20:11.35 php-fpm: pool zabbix
            23617 apache 20 0 449224 105772 3904 S 0.0 2.9 19:58.30 php-fpm: pool zabbix
            366 root 20 0 285396 89008 88324 S 0.0 2.4 39:30.11 /usr/lib/systemd/systemd-journald
            25705 apache 20 0 415888 74340 3904 S 0.0 2.0 13:19.80 php-fpm: pool zabbix
            25711 apache 20 0 416768 73320 3904 S 0.0 2.0 13:19.78 php-fpm: pool zabbix
            25712 apache 20 0 416388 73212 4340 S 0.0 2.0 13:07.57 php-fpm: pool zabbix
            25717 apache 20 0 415812 72520 3904 S 0.0 2.0 13:22.20 php-fpm: pool zabbix
            25716 apache 20 0 413820 72432 3908 S 0.0 2.0 13:15.47 php-fpm: pool zabbix
            25687 apache 20 0 415604 72252 3904 S 0.0 2.0 13:20.95 php-fpm: pool zabbix
            25707 apache 20 0 413484 72204 3908 S 0.0 2.0 13:15.84 php-fpm: pool zabbix
            25718 apache 20 0 413408 71968 3904 S 0.0 2.0 13:17.22 php-fpm: pool zabbix
            25697 apache 20 0 413188 71788 3904 S 0.0 2.0 13:16.63 php-fpm: pool zabbix
            25713 apache 20 0 415188 71784 3904 S 0.0 2.0 13:19.63 php-fpm: pool zabbix
            25704 apache 20 0 412928 71604 3908 S 0.0 2.0 13:16.03 php-fpm: pool zabbix
            25710 apache 20 0 414908 71428 3904 S 0.0 1.9 13:15.93 php-fpm: pool zabbix
            25696 apache 20 0 413596 71248 3904 S 0.0 1.9 13:19.46 php-fpm: pool zabbix
            25706 apache 20 0 414624 71212 3904 S 0.0 1.9 13:20.07 php-fpm: pool zabbix
            25714 apache 20 0 415268 71076 3904 S 0.0 1.9 13:17.28 php-fpm: pool zabbix
            30284 root 20 0 1043508 65824 50876 S 0.0 1.8 5:46.59 /opt/journald-cloudwatch-logs/journald-cloudwatch-logs /opt/journald-cloudwatch-logs/journa+
            19510 zabbix 20 0 417056 59916 8372 S 0.0 1.6 0:14.92 /usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000036 sec,+
            10851 root 24 4 758236 54936 2624 S 0.3 1.5 126:47.82 /var/awslogs/bin/python /var/awslogs/bin/aws logs push --config-file /var/awslogs/etc/awslo+
            10168 root 20 0 740108 52196 49416 S 0.0 1.4 12:20.63 /usr/sbin/rsyslogd -n
            19514 zabbix 20 0 403196 45828 8328 S 0.0 1.3 0:14.43 /usr/sbin/zabbix_server: history syncer #5 [processed 0 values, 0 triggers in 0.000021 sec,+
            19515 zabbix 20 0 400992 43668 8332 S 0.0 1.2 0:15.27 /usr/sbin/zabbix_server: history syncer #6 [processed 0 values, 0 triggers in 0.000010 sec,+
            19518 zabbix 20 0 400348 43232 8380 S 0.0 1.2 0:14.47 /usr/sbin/zabbix_server: history syncer #9 [processed 0 values, 0 triggers in 0.000015 sec,+
            19519 zabbix 20 0 400296 43192 8408 S 0.0 1.2 0:15.11 /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000025 sec+
            29983 apache 20 0 382012 41192 4460 S 0.0 1.1 4:22.46 php-fpm: pool zabbix
            29978 apache 20 0 382036 41168 4456 S 0.0 1.1 4:19.48 php-fpm: pool zabbix
            29979 apache 20 0 425492 41056 4460 S 0.0 1.1 4:21.08 php-fpm: pool zabbix
            19520 zabbix 20 0 398096 41016 8360 S 0.3 1.1 0:15.10 /usr/sbin/zabbix_server: history syncer #11 [processed 0 values, 0 triggers in 0.000014 sec+
            19521 zabbix 20 0 397908 40844 8388 S 0.3 1.1 0:14.57 /usr/sbin/zabbix_server: history syncer #12 [processed 0 values, 0 triggers in 0.000023 sec+
            19517 zabbix 20 0 398076 40792 8252 S 0.0 1.1 0:14.28 /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000021 sec,+
            19511 zabbix 20 0 397768 40524 8292 S 0.0 1.1 0:14.34 /usr/sbin/zabbix_server: history syncer #2 [processed 17 values, 7 triggers in 0.012475 sec+
            29985 apache 20 0 382144 40436 4456 S 0.0 1.1 4:21.60 php-fpm: pool zabbix
            19513 zabbix 20 0 397604 40424 8356 S 0.0 1.1 0:14.72 /usr/sbin/zabbix_server: history syncer #4 [processed 0 values, 0 triggers in 0.000010 sec,+
            29986 apache 20 0 423444 39540 4460 S 0.0 1.1 4:18.13 php-fpm: pool zabbix
            29990 apache 20 0 380240 39420 4456 S 0.0 1.1 4:21.08 php-fpm: pool zabbix
            29987 apache 20 0 423444 39220 4456 S 0.0 1.1 4:19.55 php-fpm: pool zabbix
            29976 apache 20 0 379960 39156 4460 S 0.0 1.1 4:19.76 php-fpm: pool zabbix
            29975 apache 20 0 381820 39112 4456 S 0.0 1.1 4:18.81 php-fpm: pool zabbix
            29974 apache 20 0 379868 38988 4456 S 0.0 1.1 4:16.07 php-fpm: pool zabbix
            Process Tree...

            Code:
            [root@zabbix-i03c9f7f44ae7137f8 ~]# pstree -a
            systemd --switched-root --system --deserialize 21
            ├─agetty --keep-baud 115200,38400,9600 ttyS0 vt220
            ├─agetty --noclear tty1 linux
            ├─amazon-ssm-agen
            │ ├─ssm-agent-worke
            │ │ └─11*[{ssm-agent-worke}]
            │ └─10*[{amazon-ssm-agen}]
            ├─atd -f
            ├─auditd
            │ ├─audispd
            │ │ └─{audispd}
            │ └─{auditd}
            ├─awslogs-agent-l /var/awslogs/bin/awslogs-agent-launcher.sh
            │ └─aws /var/awslogs/bin/aws logs push --config-file /var/awslogs/etc/awslogs.conf --additional-configs-dir /var/awslogs/etc/config
            │ └─8*[{aws}]
            ├─chronyd
            ├─consul agent -config-dir /etc/consul
            │ └─10*[{consul}]
            ├─crond -n
            │ └─crond -n
            ├─dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
            │ └─{dbus-daemon}
            ├─dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H ip-10-99-41-123 eth0
            ├─dhclient -q -cf /etc/dhcp/dhclient-eth1.conf -lf /var/lib/dhclient/dhclient--eth1.lease -pf /var/run/dhclient-eth1.pid -H zabbix-i03c9f7f44ae7137f8 eth1
            ├─dnsmasq -k
            ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ ├─httpd -DFOREGROUND
            │ └─httpd -DFOREGROUND
            ├─irqbalance --foreground
            ├─journald-cloudw /opt/journald-cloudwatch-logs/journald-cloudwatch-logs.conf
            │ └─9*[{journald-cloudw}]
            ├─lvmetad -f
            ├─master -w
            │ ├─pickup -l -t fifo -u
            │ └─qmgr -l -t fifo -u
            ├─node_exporter --collector.filesystem.ignored-mount-points ^/(sys|proc|dev|host|etc|tmp)($|/) --collector.filesystem.ignored-fs-types^(sys|proc|au
            │ └─10*[{node_exporter}]
            ├─oddjobd -n -p /var/run/oddjobd.pid -t 300
            ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ ├─php-fpm
            │ └─php-fpm
            ├─polkitd --no-debug
            │ └─6*[{polkitd}]
            ├─rhnsd
            ├─rsyslogd -n
            │ └─2*[{rsyslogd}]
            ├─sshd -D
            │ ├─sshd
            │ │ └─sshd
            │ │ └─sftp-server
            │ ├─sshd
            │ │ └─sshd
            │ │ └─bash
            │ │ └─sudo -i
            │ │ └─bash
            │ │ └─pstree -a
            │ └─sshd
            │ └─sshd
            │ └─bash
            ├─sssd -i --logger=files
            │ ├─sssd_be --domain ad.nonlive.global.aws.aevi.com --uid 0 --gid 0 --logger=files
            │ ├─sssd_nss --uid 0 --gid 0 --logger=files
            │ ├─sssd_pam --uid 0 --gid 0 --logger=files
            │ └─sssd_ssh --uid 0 --gid 0 --logger=files
            ├─systemd-journal
            ├─systemd-logind
            ├─systemd-udevd
            ├─tuned -Es /usr/sbin/tuned -l -P
            │ └─4*[{tuned}]
            ├─zabbix_agentd --foreground -c /etc/zabbix/zabbix_agentd.conf
            │ ├─zabbix_agentd
            │ ├─zabbix_agentd
            │ ├─zabbix_agentd
            │ ├─zabbix_agentd
            │ └─zabbix_agentd
            └─zabbix_server --foreground -c /etc/zabbix/zabbix_server.conf
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            ├─zabbix_server
            └─zabbix_server
            Last edited by Cossy; 19-03-2022, 19:21.

            Comment

            • Cossy
              Junior Member
              • Mar 2022
              • 16

              #21
              I've restarted the rh-php72-php-fpm.service as I have previously only been restarting the Zabbix service. Im assuming that only restarting the Zabbix service wasnt the correct thing way of implementing the changes to the /etc/opt/rh/rh-php72/php-fpm.conf' so hopefully now we will see an improvement in memory utilisation. I will continue to monitor and update you tomorrow.

              Comment

              • Cossy
                Junior Member
                • Mar 2022
                • 16

                #22
                My problem appears to be resolved following the 'rh-php72-php-fpm.service' restart.

                Clearly restarting just the Zabbix service was the wrong thing to do.

                Thanks tim.mooney

                Regards,
                ​​​​​​​Cossy

                Comment

                • mikerobert890
                  Junior Member
                  • Mar 2022
                  • 2

                  #23
                  Hey friend i am facing the same issue and I also hope to get great help from this community replies. You people are great.

                  Comment

                  Working...