We have a multi-site Zabbix system, I have notice that the GUI seemed to slow down over the last month. Poking around and I found in the apache logs a number of PHP Fatal errors. It seems that the server is having problems loading some required files.
I ran these down and and they are in fact at the location that the log says it can not load them from. Looking and testing a bit more and it turns out that we are generating three to four errors on each screen I try loading. From time to time I will get an Error 500, but a refresh will load the page.
We are running Zabbix 4.4.7, PHP 5.4.16, MySQL 8.0.19 on a Oracle Linux 7.7, in a Hyper-V with 4 cpu and 8G memory for the Web Server. The database and Zabbix server run on different VM's One other clue is that it seems at least one of the cpu's is always at 100%.
This is a sample of the error log
[Sun May 10 03:26:03.600506 2020] [lbmethod_heartbeat:notice] [pid 7399] AH02282: No slotmem from mod_heartmonitor
[Sun May 10 03:26:03.601893 2020] [ssl:warn] [pid 7399] AH01909: RSA certificate configured for 10.96.110.21:443 does NOT include an ID which matches the server name
[Sun May 10 03:26:03.896989 2020] [mpm_prefork:notice] [pid 7399] AH00163: Apache/2.4.6 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Sun May 10 03:26:03.897007 2020] [core:notice] [pid 7399] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sun May 10 06:04:02.335468 2020] [:error] [pid 112635] [client 10.96.110.10:46530] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/APIException.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...view&ddreset=1
[Sun May 10 07:09:04.386319 2020] [:error] [pid 113900] [client 10.96.110.10:51271] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/html/CTag.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...er.list&page=2
[Sun May 10 07:09:35.429308 2020] [:error] [pid 113900] [client 10.96.110.10:51335] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/services/CHost.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...cation=&select =&show_without_data=1&filter_set=1
[Sun May 10 07:13:19.001171 2020] [:error] [pid 112637] [client 10.96.110.10:51686] PHP Fatal error: Class 'CHostInterface' not found in /usr/share/zabbix/include/classes/core/CRegistryFactory.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...87&groupid=416
[Sun May 10 07:22:01.626409 2020] [:error] [pid 112633] [client 10.96.110.10:52460] PHP Fatal error: Class 'CJson' not found in /usr/share/zabbix/app/views/popup.generic.php on line 664, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:25:34.430505 2020] [:error] [pid 126242] [client 10.96.110.10:52766] PHP Warning: require(/usr/share/zabbix/include/classes/api/CApiServiceFactory.php): failed to open stream: Not a directory in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:25:34.430542 2020] [:error] [pid 126242] [client 10.96.110.10:52766] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/CApiServiceFactory.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:26:10.854442 2020] [:error] [pid 112635] [client 10.96.110.10:52805] PHP Warning: require(/usr/share/zabbix/include/classes/api/API.php): failed to open stream: Not a directory in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:26:10.854477 2020] [:error] [pid 112635] [client 10.96.110.10:52805] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/API.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:35:29.828746 2020] [:error] [pid 126843] [client 10.96.110.10:53691] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/html/CLink.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:36:28.219417 2020] [:error] [pid 112637] [client 10.96.110.10:53758] PHP Warning: require_once(/usr/share/zabbix/include/classes/core/CAutoloader.php): failed to open stream: Not a directory in /usr/share/zabbix/include/classes/core/ZBase.php on line 22, referer: https://fdc-zabpzy01p.l1id.local/zab....php?ddreset=1
[Sun May 10 07:36:28.219449 2020] [:error] [pid 112637] [client 10.96.110.10:53758] PHP Fatal error: require_once(): Failed opening required '/usr/share/zabbix/include/classes/core/CAutoloader.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/ZBase.php on line 22, referer: https://fdc-zabpzy01p.l1id.local/zab....php?ddreset=1
[Sun May 10 07:37:17.323876 2020] [:error] [pid 112636] [client 10.96.110.10:53870] PHP Fatal error: require_once(): Failed opening required '/usr/share/zabbix/include/page_footer.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/latest.php on line 807, referer: https://fdc-zabpzy01p.l1id.local/zab...cation=&select =&show_without_data=1&filter_set=1
[Sun May 10 07:43:09.441902 2020] [:error] [pid 126240] [client 10.96.110.10:54423] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/html/CLabel.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...cation=&select =&show_without_data=1&filter_set=1
[Sun May 10 07:53:29.341880 2020] [:error] [pid 128510] [client 10.96.110.10:55367] PHP Fatal error: Class 'CApiClientResponse' not found in /usr/share/zabbix/include/classes/api/clients/CLocalApiClient.php on line 66, referer: https://fdc-zabpzy01p.l1id.local/zab...view&ddreset=1
[Sun May 10 07:54:53.757043 2020] [:error] [pid 126240] [client 10.96.110.10:55484] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/parsers/CValidationRule.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...view&ddreset=1
I ran these down and and they are in fact at the location that the log says it can not load them from. Looking and testing a bit more and it turns out that we are generating three to four errors on each screen I try loading. From time to time I will get an Error 500, but a refresh will load the page.
We are running Zabbix 4.4.7, PHP 5.4.16, MySQL 8.0.19 on a Oracle Linux 7.7, in a Hyper-V with 4 cpu and 8G memory for the Web Server. The database and Zabbix server run on different VM's One other clue is that it seems at least one of the cpu's is always at 100%.
This is a sample of the error log
[Sun May 10 03:26:03.600506 2020] [lbmethod_heartbeat:notice] [pid 7399] AH02282: No slotmem from mod_heartmonitor
[Sun May 10 03:26:03.601893 2020] [ssl:warn] [pid 7399] AH01909: RSA certificate configured for 10.96.110.21:443 does NOT include an ID which matches the server name
[Sun May 10 03:26:03.896989 2020] [mpm_prefork:notice] [pid 7399] AH00163: Apache/2.4.6 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Sun May 10 03:26:03.897007 2020] [core:notice] [pid 7399] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sun May 10 06:04:02.335468 2020] [:error] [pid 112635] [client 10.96.110.10:46530] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/APIException.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...view&ddreset=1
[Sun May 10 07:09:04.386319 2020] [:error] [pid 113900] [client 10.96.110.10:51271] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/html/CTag.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...er.list&page=2
[Sun May 10 07:09:35.429308 2020] [:error] [pid 113900] [client 10.96.110.10:51335] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/services/CHost.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...cation=&select =&show_without_data=1&filter_set=1
[Sun May 10 07:13:19.001171 2020] [:error] [pid 112637] [client 10.96.110.10:51686] PHP Fatal error: Class 'CHostInterface' not found in /usr/share/zabbix/include/classes/core/CRegistryFactory.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...87&groupid=416
[Sun May 10 07:22:01.626409 2020] [:error] [pid 112633] [client 10.96.110.10:52460] PHP Fatal error: Class 'CJson' not found in /usr/share/zabbix/app/views/popup.generic.php on line 664, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:25:34.430505 2020] [:error] [pid 126242] [client 10.96.110.10:52766] PHP Warning: require(/usr/share/zabbix/include/classes/api/CApiServiceFactory.php): failed to open stream: Not a directory in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:25:34.430542 2020] [:error] [pid 126242] [client 10.96.110.10:52766] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/CApiServiceFactory.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:26:10.854442 2020] [:error] [pid 112635] [client 10.96.110.10:52805] PHP Warning: require(/usr/share/zabbix/include/classes/api/API.php): failed to open stream: Not a directory in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:26:10.854477 2020] [:error] [pid 112635] [client 10.96.110.10:52805] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/api/API.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:35:29.828746 2020] [:error] [pid 126843] [client 10.96.110.10:53691] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/html/CLink.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...dashboardid=40
[Sun May 10 07:36:28.219417 2020] [:error] [pid 112637] [client 10.96.110.10:53758] PHP Warning: require_once(/usr/share/zabbix/include/classes/core/CAutoloader.php): failed to open stream: Not a directory in /usr/share/zabbix/include/classes/core/ZBase.php on line 22, referer: https://fdc-zabpzy01p.l1id.local/zab....php?ddreset=1
[Sun May 10 07:36:28.219449 2020] [:error] [pid 112637] [client 10.96.110.10:53758] PHP Fatal error: require_once(): Failed opening required '/usr/share/zabbix/include/classes/core/CAutoloader.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/ZBase.php on line 22, referer: https://fdc-zabpzy01p.l1id.local/zab....php?ddreset=1
[Sun May 10 07:37:17.323876 2020] [:error] [pid 112636] [client 10.96.110.10:53870] PHP Fatal error: require_once(): Failed opening required '/usr/share/zabbix/include/page_footer.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/latest.php on line 807, referer: https://fdc-zabpzy01p.l1id.local/zab...cation=&select =&show_without_data=1&filter_set=1
[Sun May 10 07:43:09.441902 2020] [:error] [pid 126240] [client 10.96.110.10:54423] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/html/CLabel.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...cation=&select =&show_without_data=1&filter_set=1
[Sun May 10 07:53:29.341880 2020] [:error] [pid 128510] [client 10.96.110.10:55367] PHP Fatal error: Class 'CApiClientResponse' not found in /usr/share/zabbix/include/classes/api/clients/CLocalApiClient.php on line 66, referer: https://fdc-zabpzy01p.l1id.local/zab...view&ddreset=1
[Sun May 10 07:54:53.757043 2020] [:error] [pid 126240] [client 10.96.110.10:55484] PHP Fatal error: require(): Failed opening required '/usr/share/zabbix/include/classes/parsers/CValidationRule.php' (include_path='.:/usr/share/pear:/usr/share/php') in /usr/share/zabbix/include/classes/core/CAutoloader.php on line 59, referer: https://fdc-zabpzy01p.l1id.local/zab...view&ddreset=1