After Upgrading to Zabbix 6.0 LTS - Latest data (very very slow)

[colloque]

Hello,
I have migrate zabbix 5.4 to 6.0 LTS from i have som difficulties to see latest data (sometimes Gateway Timeout
appears).

After upgrade zabbix-server and zabbix_agentd2 I do the prerequisite database modification (https://www.zabbix.com/documentation...eys#postgresql)

Do you have any ideas ?

Thanks in advance for your help

[Glencoe]

Please provide more detail. Is it slow always or with specific parameters? Or maybe without parameters? Are there any error messages in Apache/Nginx error.log?

[colloque]

Hello,
Thanks for your quick answer


You will find below the informations:

For information It's always slow for "latest data" only


(/var/log/httpd/error_log)


[Mon Feb 21 11:24:07.974685 2022] [proxy:error] [pid 3064839:tid 140650020325120] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /run/php-fpm/zabbix.sock (*) failed
[Mon Feb 21 11:24:07.974725 2022] [proxy_fcgi:error] [pid 3064839:tid 140650020325120] [client x.x.x.x:53543] AH01079: failed to make connection to backend: httpd-UDS, referer: http://zabbix.xxxxxxxxxxxx...%5D%5Btag%5D=& tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D =&show_tags=3&tag_name_format=0&tag_priority=&fi lt er_name=&filter_show_counter=0&filter_custom_time= 0&sort=name&sortorder=ASC&show_details=0&action= la test.view&hostids%5B%5D=11673


[Mon Feb 21 11:50:37.239110 2022] [proxy_fcgi:error] [pid 3064837:tid 140650397800192] (70007)The timeout specified has expired: [client 172.16.20.17:64972] AH01075: Error dispatching request to : (polling), referer: http://zabbix.xxxxxxx/...earch=xxxxxxxx

xxxxxx (server name)


(zabbix_server.conf)

LogFileSize=50
PidFile=/var/run/zabbix/zabbix_server.pid
SocketDir=/var/run/zabbix

DBName=zabbix

DBUser=zabbix
DBPassword=xxxxxxx


StartPollers=70
StartIPMIPollers=5
StartPreprocessors=8
StartPollersUnreachable=70
StartHistoryPollers=15
StartTrappers=40
StartPingers=20
StartDiscoverers=20
StartHTTPPollers=40
StartTimers=5
StartEscalators=20
StartAlerters=5
StartVMwareCollectors=15

VMwareCacheSize=512M

SNMPTrapperFile=/var/log/snmptrap/snmptrap.log

StartSNMPTrapper=1

HousekeepingFrequency=1

MaxHousekeeperDelete=5000

CacheSize=3G

HistoryCacheSize=512M

HistoryIndexCacheSize=256M

TrendCacheSize=256M

ValueCacheSize=512M

Timeout=10

LogSlowQueries=3000

StartLLDProcessors=15

StatsAllowedIP=127.0.0.1

StartReportWriters=2

WebServiceURL=http://xxxxxxxxxx:10053/report

StartODBCPollers=10



Thanks yet

[Glencoe]

Any errors in php-fpm.log?

[colloque]

I have just below

[21-Feb-2022 11:24:08] NOTICE: fpm is running, pid 3148147
[21-Feb-2022 11:24:08] NOTICE: ready to handle connections
[21-Feb-2022 11:24:08] NOTICE: systemd monitor interval set to 10000ms

...
Nothing in 11h50



For information : system information (in attrachement)


Thanks

[Glencoe]

It seems that not every error is configured to show in the php-fpm.log by default. Please try to add catch_workers_output = yes to your php-fpm.conf, restart php-fpm and reproduce the problem.

[colloque]

Thanks

I don't have catch_workers_output parameter into php-fpm 7.2.24
I had to put log_level in debug mode (initial value = notice)


You will find the result log in the attachement

[Glencoe]

That's why I said "Please try to *add* catch_workers_output = yes"

[colloque]

Thanks

It's not working

systemctl restart php-fpm.service
Job for php-fpm.service failed because the control process exited with error code.
See "systemctl status php-fpm.service" and "journalctl -xe" for details.

xxxxxxxx php-fpm[3299679]: [21-Feb-2022 19:28:13] ERROR: [/etc/php-fpm.conf:138] unknown entry 'catch_workers_output'
xxxxxxxx php-fpm[3299679]: [21-Feb-2022 19:28:13] ERROR: failed to load configuration file '/etc/php-fpm.conf'
xxxxxxxx php-fpm[3299679]: [21-Feb-2022 19:28:13] ERROR: FPM initialization failed
xxxxxxxx systemd[1]: php-fpm.service: Main process exited, code=exited, status=78/CONFIG
xxxxxxxx systemd[1]: php-fpm.service: Failed with result 'exit-code'.



Thank.

[Glencoe]

It must be defined at the pool level. What pools do you have? These may be included in the main php-fpm.conf and reside in php-fpm.d directory.

[colloque]

Thanks for your answer

Below the modification and result


# cat /etc/php-fpm.d/www.conf |grep catch
catch_workers_output = yes

# ls -lrt /etc/php-fpm.d/www.conf
-rw-r--r--. 1 root root 19414 21 févr. 19:50 /etc/php-fpm.d/www.conf


# Restart
● php-fpm.service - The PHP FastCGI Process Manager
Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-02-21 20:21:22 CET; 9min ago
Main PID: 15581 (php-fpm)


# ls -lrt /var/log/php-fpm/
total 20
-rw-------. 1 root root 56 23 janv. 03:33 error.log-20220130
-rw-------. 1 root root 56 30 janv. 03:46 error.log-20220206
-rw-------. 1 root root 56 6 févr. 03:31 error.log-20220213
-rw-------. 1 root root 1307 19 févr. 09:50 error.log-20220220
-rw-------. 1 root root 187 21 févr. 20:21 error.log


# more error.log
[21-Feb-2022 20:21:22] NOTICE: fpm is running, pid 15581
[21-Feb-2022 20:21:22] NOTICE: ready to handle connections
[21-Feb-2022 20:21:22] NOTICE: systemd monitor interval set to 10000ms

for information I have had the error "Gateway timeOut"

Thanks yet

[colloque]

Some informations:


When i passe by Monitoring --> Latest data --> Host Groups (no matter the groups) next Apply :: It's Quick (however i don't see all servers of group)

When i passe by Monitoring --> Latest data --> Hosts (no matter the hosts) -next Apply -> it' very slow, same impossible


Thanks

[Glencoe]

It's weird there is nothing in the log. Since there is a timeout calling php-fpm from Nginx I hoped the fpm log will give some hint.

Is your problem anything like described here? https://support.zabbix.com/browse/ZBX-20644

[colloque]

Hello,
Yes the problem describe in ZBX-20644 it's this one with in more the same comportment via DashBoard --> Click on host then --> latest data.

Thanks you.

[Glencoe]

How many items do your hosts have? Those that are slow in the latest data when selected individually?

[colloque]

Yes it's when the host is selected individually.

For example

- With 661 Items and i have got the error Bad Gateway (see attachment)
- With 91 item for another Hosts i have seen the latest data after 01:15 minute
- With 3 items about 10 secondes for another

Thanks

[Glencoe]

This is not normal. What times did you see on 5.4?

[colloque]

I did not time because it was quick

[colloque]

For information, i have changed the value of shared_buffers in my postgresql.conf file (of 4GB to 8GB) now it's quick.

Thanks for your help

[Glencoe]

Thanks, this is helpful. This means the new latest data queries consume considerably more memory.