Zabbix server does not start anymore: Cannot connect to service "rtc"

[polx]

Hello all,
I am running zabbix 6.0.12 (zabbix-server-mysql/focal,now 1:6.0.12-1+ubuntu20.04 amd64) on ubuntu Linux 5.15.0-58-generic and since a few days, it displays that the zabbix server is not running.
Indeed sudo /etc/init.d/zabbix-server restart responds it cannot start ("the service did not take the steps required by its unit configuration").

To try to understand it, I've been running "to front" and I obtain the following:
$ sudo /usr/sbin/zabbix_server -f -R log_level_increase -c /etc/zabbix/zabbix_server.conf
zabbix_server [7355]: Cannot perform runtime control command: Cannot connect to service "rtc": [111] Connection refused.

I dug into what rtc can be... whether it's the hardware clock or some zabbix internal. The string above indeed appears into the zabbix source but I could not understand which socket has a connection refused.

Can it be there a service that we have let, through the updates of Ubuntu, disappear?

The web-service works and displays data but it's all old data (nothing more in latest data). The agent seems to run (according to ps).

thanks very much for your hints.

[polx]

aaaaah! That was the right hint. Can one tell me how to convert these dates?

So... I was reading /var/log/zabbix-server/zabbix_server.log which is indeed from may.
Reading /var/log/zabbix/zabbix_server.log gave me the necessary clue:
1550867:20230123:121723.863 [Z3005] query failed: [1419] You do not have the SUPER privilege and binary logging is enabled

And indeed, searching around gave me the missing instruction: doing SET GLOBAL log_bin_trust_function_creators = 1; in MySQL and restarting kicked my zabbix.
I'll safely remove /var/log/zabbix-server/zabbix_server.log !

That was a tunnel walk! Thank you very much for the light in the tunnel.

paul

PS: Is there a way to "mark as resolved" ?

[tim.mooney]

This was for Zabbix proxy, but seems likely to be a similar issue:

https://support.zabbix.com/browse/ZBX-20708

[Markku]

What does zabbix_server.log actually say when trying to start the process? Also, what did it say when the process was unexpectedly stopped a few days ago?

Markku

[polx]

> This was for Zabbix proxy, but seems likely to be a similar issue:
> https://support.zabbix.com/browse/ZBX-20708

The solution sounds to be for PostgresSQL or? We use MySQL.

> What does zabbix_server.log actually say when trying to start the process? Also, what did it say when the process was unexpectedly stopped a few days ago?

I just stops with a non-zero result. Here is the tail of the log. I think that I see nothing critical in there.
To find a log where this started to fail, I'd have to crawl through the backups. It seems longer than I thought (I know it's a shame but it was a stressy christmas time).

1868216:20220531:163015.589 enabling Zabbix agent checks on host "Zabbix server": host became available
1868219:20220531:163108.162 cannot send list of active checks to "127.0.0.1": host [devic] not found
1868225:20220531:163308.177 cannot send list of active checks to "127.0.0.1": host [devic] not found
1868228:20220531:163508.191 cannot send list of active checks to "127.0.0.1": host [devic] not found
1868192:20220531:171729.918 executing housekeeper
1868192:20220531:171729.922 housekeeper [deleted 0 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit items in 0.002507 sec, idle for 1 hour(s)]
1868192:20220531:181730.167 executing housekeeper
1868192:20220531:181730.171 housekeeper [deleted 0 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit items in 0.002674 sec, idle for 1 hour(s)]
1868192:20220531:191730.415 executing housekeeper
1868192:20220531:191730.415 housekeeper [deleted 0 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit items in 0.000523 sec, idle for 1 hour(s)]
1868228:20220531:193822.632 cannot send list of active checks to "127.0.0.1": host [Zabbix server] not found
1868190:20220531:194009.433 Got signal [signal:15(SIGTERM),sender_pid:1934034,sender_uid:0 ,reason:0]. Exiting ...
1868196:20220531:194009.433 syncing history data in progress...
1868197:20220531:194009.444 syncing history data... 100.000000%
1868197:20220531:194009.444 syncing history data done
1868190:20220531:194009.447 syncing trend data...
1868190:20220531:194009.455 syncing trend data done
1868190:20220531:194009.455 Zabbix Server stopped. Zabbix 4.0.17 (revision a528a0a4bc).

Thanks for your responses!

paul

[cyber]

This looks like from end of May last year? Really ? Very long christmas period, I would say..
Ok... it was stopped for moths, but if you try to start it now, what is written to logs then?

[polx]

aaaaah! That was the right hint. Can one tell me how to convert these dates?

So... I was reading /var/log/zabbix-server/zabbix_server.log which is indeed from may.
Reading /var/log/zabbix/zabbix_server.log gave me the necessary clue:
1550867:20230123:121723.863 [Z3005] query failed: [1419] You do not have the SUPER privilege and binary logging is enabled

And indeed, searching around gave me the missing instruction: doing SET GLOBAL log_bin_trust_function_creators = 1; in MySQL and restarting kicked my zabbix.
I'll safely remove /var/log/zabbix-server/zabbix_server.log !

That was a tunnel walk! Thank you very much for the light in the tunnel.

paul

PS: Is there a way to "mark as resolved" ?

[cyber]

You can press "set as answer".. I guess Markku was the first one to point you to read logfile.. But then again, your own post describes the most of problem and gives also the resolution..

[polx]

For the posterity, I think that my reply gives more details so I chose mine.
I hope I have not robbed credit points from anyone in an unjust manner.
paul