Centos 8 Zabbix Agent only works after "systemctl restart zabbix-agent" Please help

[sho1sho1]

Hi all,

I am new to Zabbix. So far, I have tried to setup agent auto-registration. My environment has mainly Linux Centos 8 hosts. In the zabbix_agentd.conf, I added Active Server IP address and Hostmetadataitem for auto-registration to link Template OS Linux Zabbix Agent Active. For some reason, the hosts do not get auto-registered until I do "systemctl restart zabbix-agent". When the zabbix-agent restart, Zabbix frontend instantly auto-registers the hosts. The auto-registration does not work when zabbix-agent is started on boot.

The next issue that is similar to the above is system uptime. For some odd reason, when a host is rebooted, the uptime does not get updated and Zabbix frontend still thinks the host is up. It is not until the host boots into Centos 8 and I manually do "systemctl restart zabbix-agent", then the uptime on Zabbix frontend will update.

I have firewalld disabled and stopped on the Zabbix Server. Security Linux for Zabbix Server and Zabbis Hosts are on permissive setting.
I have firewalld disabled and stopped on the Zabbix Hosts and the issue is still there.

Is this a bug or am I doing something wrong?

Please help!

Thanks in advance!

[tim.mooney]

You've verified that the zabbix-agent.service is actually enabled, and that it is starting at boot?

If you reboot one of your CentOS 8 clients and then log in and type

systemctl status -f zabbix-agent.service

what does it show for "Loaded" and "Active"?

[sho1sho1]

This is on one of the host that just booted to Centos 8. I have rebooted and power cycled this host about a dozen times. It has not auto-registered on the Zabbix front-end yet. However, if I do a manual systemctl restart zabbix-agent, it will all of a sudden come alive.

uname -a shows Linux 71nodes-69.testbench.local 4.18.0-80.el8.x86_64 #1 SMP Tue Jun 4 09:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@71nodes-69 ~]# systemctl status -f zabbix-agent.service
● zabbix-agent.service - Zabbix Agent
Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-02-19 19:33:01 CST; 1h 9min ago
Process: 2173 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 2200 (zabbix_agentd)
Tasks: 6 (limit: 65535)
Memory: 11.9M
CGroup: /system.slice/zabbix-agent.service
├─2200 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
├─2203 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
├─2204 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
├─2205 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
├─2206 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
└─2207 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: User lookup succeeded: uid=976 gid=974
Feb 19 19:33:01 localhost.localdomain systemd[2173]: zabbix-agent.service: Executing: /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: Child 2173 belongs to zabbix-agent.service.
Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: Control process exited, code=exited status=0
Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: Got final SIGCHLD for state start.
Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: New main PID 2200 belongs to service, we are happy.
Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: Main PID loaded: 2200
Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: Changed start -> running
Feb 19 19:33:01 localhost.localdomain systemd[1]: zabbix-agent.service: Job zabbix-agent.service/start finished, result=done
Feb 19 19:33:01 localhost.localdomain systemd[1]: Started Zabbix Agent.

[sho1sho1]

Here are the logs, not sure why it shows started to fail and then shows is working again...

[root@71nodes-69 ~]# cat /var/log/zabbix/zabbix_agentd.log
2286:20200219:231452.750 Starting Zabbix Agent [localhost.localdomain]. Zabbix 4.4.5 (revision b93f5c4fc0).
2286:20200219:231452.750 **** Enabled features ****
2286:20200219:231452.750 IPv6 support: YES
2286:20200219:231452.750 TLS support: YES
2286:20200219:231452.750 **************************
2286:20200219:231452.750 using configuration file: /etc/zabbix/zabbix_agentd.conf
2286:20200219:231452.751 agent #0 started [main process]
2287:20200219:231452.751 agent #1 started [collector]
2288:20200219:231452.751 agent #2 started[listener #1]
2289:20200219:231452.751 agent #3 started[listener #2]
2290:20200219:231452.751 agent #4 started[listener #3]
2291:20200219:231452.752 agent #5 started [active checks #1]
2291:20200219:231452.753 active check configuration update from [192.168.1.38:10051] started to fail (cannot connect to [[192.168.1.38]:10051]: [101] Network is unreachable)
2291:20200219:171501.983 the system time has been pushed back, adjusting active check schedule
2291:20200219:171554.038 active check configuration update from [192.168.1.38:10051] is working again
2286:20200219:193135.563 Got signal [signal:15(SIGTERM),sender_pid:40287,sender_uid:976 ,reason:0]. Exiting ...
2286:20200219:193135.564 Zabbix Agent stopped. Zabbix 4.4.5 (revision b93f5c4fc0).
2200:20200219:193301.153 Starting Zabbix Agent [localhost.localdomain]. Zabbix 4.4.5 (revision b93f5c4fc0).
2200:20200219:193301.185 **** Enabled features ****
2200:20200219:193301.185 IPv6 support: YES
2200:20200219:193301.185 TLS support: YES
2200:20200219:193301.185 **************************
2200:20200219:193301.185 using configuration file: /etc/zabbix/zabbix_agentd.conf
2200:20200219:193301.185 agent #0 started [main process]
2203:20200219:193301.186 agent #1 started [collector]
2204:20200219:193301.186 agent #2 started[listener #1]
2205:20200219:193301.186 agent #3 started[listener #2]
2206:20200219:193301.186 agent #4 started[listener #3]
2207:20200219:193301.186 agent #5 started [active checks #1]
2207:20200219:193301.187 active check configuration update from [192.168.1.38:10051] started to fail (cannot connect to [[192.168.1.38]:10051]: [101] Network is unreachable)
2207:20200219:193401.250 active check configuration update from [192.168.1.38:10051] is working again

[dimir]

Here are the logs, not sure why it shows started to fail and then shows is working again...

[root@71nodes-69 ~]# cat /var/log/zabbix/zabbix_agentd.log
[...]
2291:20200219:231452.752 agent #5 started [active checks #1]
2291:20200219:231452.753 active check configuration update from [192.168.1.38:10051] started to fail (cannot connect to [[192.168.1.38]:10051]: [101] Network is unreachable)
2291:20200219:171501.983 the system time has been pushed back, adjusting active check schedule
2291:20200219:171554.038 active check configuration update from [192.168.1.38:10051] is working again

Looks like the time on your host is out of sync. You should use ntpd to keep the system time in sync.

[sho1sho1]

Ah, let me enable ntpd and see.
I'll report back later!
Thanks!

[tim.mooney]

RHEL 8 doesn't include ntpd any longer, so I'm guessing that is the same for CentOS 8. The supported option for time sync is chronyd. It still speaks the NTP protocol, but it's a much different implementation than traditional ntpd.