Sudden erratic behavior of our zabbix 5.4 installation...

[Plexus]

Hi folks,

returning from lunch I had to find out, that 10 minutes after starting my break our zabbix began to strange without any direct external influences. The monitoring itself still works fine, the values are recorded and reported as usual and also Grafana displays all graphs flawlessly. The first indicator for an existing problem was zabbix vue´s (chrome plugin) refusal of work - instead a communication error was reported. Opening our zabbix WebGUI the problem-widget of our default dashbord wasn´t loading on both of our webservers. After deleting the problem-widget and reinserting the same one it worked again. With this working dashboard I got aware of the php_errorlog on those machines announcing as follows:

[14-Jul-2022 15:41:31 Europe/Berlin] PHP Notice: Undefined index: items in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[14-Jul-2022 15:41:31 Europe/Berlin] PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[14-Jul-2022 15:41:31 Europe/Berlin] PHP Notice: Undefined index: items in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[14-Jul-2022 15:41:31 Europe/Berlin] PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[14-Jul-2022 15:41:31 Europe/Berlin] PHP Notice: Undefined index: items in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[14-Jul-2022 15:41:31 Europe/Berlin] PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[14-Jul-2022 15:41:31 Europe/Berlin] PHP Notice: Undefined index: items in /usr/share/zabbix/include/classes/screens/CScreenProblem.php on line 591
[14-Jul-2022 15:41:31 Europe/Berlin] PHP Fatal error: Uncaught TypeError: Argument 1 passed to CMacrosResolverHelper::resolveItemNames() must be of the type array, null given, called in /usr/share/zabbix/include/classes/screens/CScreenProblem.php on line 591 and defined in /usr/share/zabbix/include/classes/macros/CMacrosResolverHelper.php:538
Stack trace:
#0 /usr/share/zabbix/include/classes/screens/CScreenProblem.php(591): CMacrosResolverHelper::resolveItemNames()
#1 /usr/share/zabbix/app/controllers/CControllerWidgetProblemsView.php(69): CScreenProblem::makeData()
#2 /usr/share/zabbix/include/classes/mvc/CController.php(390): CControllerWidgetProblemsView->doAction()
#3 /usr/share/zabbix/include/classes/core/ZBase.php(525): CController->run()
#4 /usr/share/zabbix/include/classes/core/ZBase.php(209): ZBase->processRequest()
#5 /usr/share/zabbix/include/config.inc.php(25): ZBase->run()
#6 /usr/share/zabbix/zabbix.php(22): require_once('/usr/share/zabb...')
#7 {main}
thrown in /usr/share/zabbix/include/classes/macros/CMacrosResolverHelper.php on line 538

We are using zabbix 5.4.12 with postgreSQL13, both installed on debian 10. Those Webservers for the GUI are running nginx (1.14.2) with php7.4_fpm (7.4.30) installed on Debian 10 also.

Thanks an advance for any hints how to get rid of those problems.

Greetings,
Plex

[Markku]

communication error was reported

If I understand correctly the Zabbix Vue extension, it connects to Zabbix API. What do you see in the Zabbix server logs?

Markku

[Plexus]

Yes, zabbix Vue seems to get it´s values from the Webinterface/API.

After checking the zabbix-server logs this morning I found this fatal error happening at the time problems started:

2986:20220714:154118.429 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR: insert or update on table "functions" violates foreign key constraint "c_functions_1"
DETAIL: Key (itemid)=(203040) is not present in table "items".
[insert into functions (functionid,itemid,triggerid,name,parameter) values (98583,203040,80161,'find','$,,"regexp","^available$"'),(98584,202957,80162,'find','$,,"regexp","^available$"'),(98585,203041,80163,'find','$,,"regexp","^up$"'),(98586,202958,80164,'find','$,,"regexp","^up$"'),(98587,203042,80165,'find','$,,"regexp","^available$"'),(98588,202959,80166,'find','$,,"regexp","^available$"');

So zabbix tried to insert into functions table but due to a missing item with ID 203040, which would have been the foreign key for the first valueset, this failed. I checked the current state of the DB and found this particular functions with another valid itemID - TriggerID, name and parameter were still the same. Guess those were automatic generated items from discovery rules.

This morning I reset our DB from a backup some hours before those problems occured. At least no more fatal errors on the zabbix-server itself but still we do have problems connecting zabbix Vue and fatal errors on the Webinterface/API machines and endless sides in the php-errorlog:

[15-Jul-2022 14:24:16 Europe/Berlin] PHP Notice: Undefined index: items in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[15-Jul-2022 14:24:16 Europe/Berlin] PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2784
[15-Jul-2022 14:24:16 Europe/Berlin] PHP Notice: Undefined index: items in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2821
[15-Jul-2022 14:24:16 Europe/Berlin] PHP Warning: count(): Parameter must be an array or an object that implements Countable in /usr/share/zabbix/include/classes/macros/CMacrosResolver.php on line 2821
[15-Jul-2022 14:24:16 Europe/Berlin] PHP Notice: Undefined index: items in /usr/share/zabbix/include/classes/screens/CScreenProblem.php on line 591
[15-Jul-2022 14:24:16 Europe/Berlin] PHP Fatal error: Uncaught TypeError: Argument 1 passed to CMacrosResolverHelper::resolveItemNames() must be of the type array, null given, called in /usr/share/zabbix/include/classes/screens/CScreenProblem.php on line 591 and defined
in /usr/share/zabbix/include/classes/macros/CMacrosResolverHelper.php:538
Stack trace:
#0 /usr/share/zabbix/include/classes/screens/CScreenProblem.php(591): CMacrosResolverHelper::resolveItemNames()
[NODE="1"]Home[/NODE] /usr/share/zabbix/app/controllers/CControllerWidgetProblemsView.php(69): CScreenProblem::makeData()
[NODE="2"]Forum[/NODE] /usr/share/zabbix/include/classes/mvc/CController.php(390): CControllerWidgetProblemsView->doAction()
#3 /usr/share/zabbix/include/classes/core/ZBase.php(525): CController->run()
#4 /usr/share/zabbix/include/classes/core/ZBase.php(209): ZBase->processRequest()
#5 /usr/share/zabbix/include/config.inc.php(25): ZBase->run()
[NODE="6"]Special[/NODE] /usr/share/zabbix/zabbix.php(22): require_once('/usr/share/zabb...')
#7 {main}
thrown in /usr/share/zabbix/include/classes/macros/CMacrosResolverHelper.php on line 538

Edit:
after returning from lunch the dashboard widget of our zabbix webinterface is hanging again... seems zabbix don´t want me to have my lunchtime break

[Markku]

Btw are you sure you are running the same Zabbix component versions (5.4.12) on the Zabbix server and on the web frontends? Just an idea, if someone had run "apt upgrade" on the Zabbix server (for example) but the web frontends are still on another minor Zabbix version.

Markku

[Plexus]

I had this idea yesterday also and upgraded every affected component from 5.4.4 to 5.4.12 just to be sure...