Zabbix server won't stop (sometimes), 7.0.18, Ubuntu 24.04.3, psql 17.2

[Linwood]

I have zabbix running on a number of clients and am embarrassed to say it's my own home network where I have a problem I can't find.


When doing a reboot of ubuntu, it will hang forever. Sometimes. Not always.

I cannot find any resource issues - plenty of disk and memory and CPU. When it comes back up it runs fine.

This last time I did a "systemctl stop zabbix-server" and then quite literally stopped (kill -9) every pid that had "zabbix" in the name and then did the reboot and it hung as shown below.

What I'd like to do is change that unlimited to something like 3 minutes or so. Or figure out why it is hanging, but so far I find nothing -- zabbix was running fine and responding fine, and polling when I did the shutdown.

I'd love to find the root cause, but in the meantime anyone know how to persistently change the timeout on that stop line from unlimited?

I can change /lib/systemd/system/zabbix-server.service but it's going to get overwritten (or it should) by updates, right?

Should it REALLY be infinity?

[troffasky]

Yes I have also seen this.
7.0.x, proxy and server, Ubuntu 22.04, MariaDB.
Probably happened 2-3 times in total over past year. Only solution is to hard-reset the VM. It doesn't just happen on shutdown, also when installing updates.

[Linwood]

So doesn't sound specific to Postgresql then. Any zabbix team members following, is there any reason the distro service file uses infinite?

[cyber]

Have you ever looked into zabbix own log from that time ? I have seen cases, when history sync during shutdown takes ages... you can literally see it going 0.xx% per sec, as it writes that progress to log... And it may take 30-40 minutes... even more.

[Aer0]

I have the same problem at my test system which has quite high load. Not sure about the cause (i have not seen log-file), but I'm thinking to make a "#DefaultTimeoutStopSec=" for force stop.

[Linwood]

Have you ever looked into zabbix own log from that time ? I have seen cases, when history sync during shutdown takes ages... you can literally see it going 0.xx% per sec, as it writes that progress to log... And it may take 30-40 minutes... even more.

Oddly enough it looks to me like it DID stop (2nd screen shot), but look (1st screen shot) what I found in the syslog.

I have no idea what that means. Let me do a couple of reboots.

[Linwood]

I just did a reboot, and there is NOTHING in the syslog showing zabbix server stopped.

However, with zabbix running I did a systemctl stop zabbix-server, and that invalid argument is in the result (and it did stop). So I think that's wrong, not sure what but maybe something in the distro incompatible with this systemd, but it does not at least by itself keep it from stopping.

But sometimes it won't stop. Those of you with it also failing, hunt around next time, see if you can find clues. It sounds like it's infrequent but fairly widespread given the responses (and how little traffic you get here normally).

[cyber]

Some syntax error in service file? Bad encoding?

systemd-analyze verify yourname.service

[Linwood]

Learned a new command, thank you. No errors shown. If you do a systemctl stop zabbix-server and check syslog is there an error there?

To my knowledge this is the file from the distro, though I think at this instant I've changed it from infinite to 5m, but the original error was before I changed it. I would expect it to give the error on any system at similar versions.

[troffasky]

Bitten by this again today. Tried to shut down VM, stuck waiting per @Linwood's first screenshot.

In the journal I can see lots of services being stopped, then nothing for 30 minutes, then it times out.

Nov 17 15:05:44 proxy systemd[1]: Stopped target Network is Online.
Nov 17 15:05:44 proxy systemd[1]: Stopped target Host and Network Name Lookups.
Nov 17 15:05:44 proxy systemd[1]: NetworkManager-wait-online.service: Deactivated successfully.
Nov 17 15:05:44 proxy systemd[1]: Stopped Network Manager Wait Online.
Nov 17 15:35:11 proxy systemd[1]: reboot.target: Job reboot.target/start timed out.
Nov 17 15:35:11 proxy systemd[1]: Timed out starting System Reboot.
Nov 17 15:35:11 proxy systemd[1]: reboot.target: Job reboot.target/start failed with result 'timeout'.
Nov 17 15:35:11 proxy systemd[1]: Forcibly rebooting: job timed out
Nov 17 15:35:11 proxy systemd[1]: Shutting down.

It's weird, but you would not know that zabbix-proxy is the culprit here as it doesn't seem to log anything about it in the journal!

Service file has

TimeoutSec=infinity

configured in it. Changed to 300.

There are references to this issue in this ancient bug:

https://support.zabbix.com/browse/ZBX-15602

[troffasky]

Upgraded a proxy from Ubuntu 20.04, to 22.04 to 24.04 today. Hit by this at every shutdown! On the 22.04 shutdown step, even though TimeoutSec=infinity was in the service file, it actually said it was only going to wait 5 minutes.