Zabbix Problems and "Problems by severity" section not working
Good Day!
As seen in the attached, a client is complaining that since yesterday, they noticed that the Dashboards and Monitoring-> Problems section no longer seems to work (specifically “Problems” and “Problems by severity”) - see attached:
The user claims that no changes were done on the admin node so not sure why they are not seeing problems described in those sections.
What was already tried:
[root@xxxxxADMIN ~]# systemctl status zabbix-server.service
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:50 CET; 6min ago
Process: 1278 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 1288 (zabbix_server)
Memory: 220.3M
CGroup: /system.slice/zabbix-server.service
├─ 1288 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
├─ 2327 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.863163 sec, idle 60 sec]
├─ 2328 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
├─ 2329 /usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.000490 sec, idle 59 sec]
├─ 2330 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000592 sec, idle 5 sec]
├─ 2331 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000545 sec, performing discovery]
├─ 2332 /usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000027 sec, idle 1 sec]
├─ 2334 /usr/sbin/zabbix_server: history syncer #2 [processed 244 values, 0 triggers in 0.022097 sec, idle 1 sec]
├─ 2336 /usr/sbin/zabbix_server: history syncer #3 [processed 292 values, 7 triggers in 0.055103 sec, idle 1 sec]
├─ 2338 /usr/sbin/zabbix_server: history syncer #4 [processed 455 values, 1 triggers in 0.050551 sec, idle 1 sec]
├─ 2340 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.006683 sec, idle 3 sec]
├─ 2342 /usr/sbin/zabbix_server: java poller #1 [got 0 values in 0.000032 sec, idle 5 sec]
├─ 2343 /usr/sbin/zabbix_server: java poller #2 [got 0 values in 0.000031 sec, idle 5 sec]
├─ 2345 /usr/sbin/zabbix_server: java poller #3 [got 0 values in 0.000030 sec, idle 5 sec]
├─ 2346 /usr/sbin/zabbix_server: java poller #4 [got 0 values in 0.000023 sec, idle 5 sec]
├─ 2347 /usr/sbin/zabbix_server: java poller #5 [got 0 values in 0.000032 sec, idle 5 sec]
├─ 2348 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000029 sec, idle 5 sec]
├─ 2349 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000024 sec, idle 1 sec]
├─ 2350 /usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.000427 sec, idle 5 sec]
├─ 2352 /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000026 sec, idle 1 sec]
├─ 2354 /usr/sbin/zabbix_server: poller #2 [got 995 values in 0.297186 sec, idle 1 sec]
├─ 2356 /usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000028 sec, idle 1 sec]
├─ 2359 /usr/sbin/zabbix_server: poller #4 [got 0 values in 0.000029 sec, idle 1 sec]
├─ 2361 /usr/sbin/zabbix_server: poller #5 [got 0 values in 0.000031 sec, idle 1 sec]
├─ 2364 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000153 sec, idle 1 sec]
├─ 2365 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.000377 sec, waiting for connection]
├─ 2366 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.000088 sec, waiting for connection]
├─ 2367 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.000480 sec, waiting for connection]
├─ 2368 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.000101 sec, waiting for connection]
├─ 2369 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.000084 sec, waiting for connection]
├─ 2370 /usr/sbin/zabbix_server: icmp pinger #1 [pinging hosts]
├─ 2371 /usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.009546 sec during 5.009643 sec]
├─ 2372 /usr/sbin/zabbix_server: alerter #1 [sent 0, failed 0 alerts, idle 111.262759 sec during 116.265273 sec]
├─ 2374 /usr/sbin/zabbix_server: alerter #2 [sent 0, failed 0 alerts, idle 111.265718 sec during 116.333905 sec]
├─ 2378 /usr/sbin/zabbix_server: alerter #3 [sent 0, failed 0 alerts, idle 10.072499 sec during 15.076396 sec]
├─ 2380 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 4476 values, idle 5.657378 sec during
5.678010 sec]
├─ 2381 /usr/sbin/zabbix_server: preprocessing worker #1 started
├─ 2383 /usr/sbin/zabbix_server: preprocessing worker #2 started
├─ 2384 /usr/sbin/zabbix_server: preprocessing worker #3 started
├─12832 sh -c /usr/sbin/fping -C10 2>&1 </tmp/zabbix_server_2370.pinger;/usr/sbin/fping6 -C10 2>&1
</tmp/zabbix_server_2370.pinger;
└─12833 /usr/sbin/fping -C10
Mar 01 12:18:50 xxxxxADMIN systemd[1]: Starting Zabbix Server...
Mar 01 12:18:50 xxxxxADMIN systemd[1]: Started Zabbix Server.
[root@xxxxxADMIN ~]# systemctl status zabbix-java-gateway.service
● zabbix-java-gateway.service - Zabbix Java Gateway
Loaded: loaded (/usr/lib/systemd/system/zabbix-java-gateway.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:52 CET; 6min ago
Process: 1253 ExecStart=/usr/sbin/zabbix_java_gateway (code=exited, status=0/SUCCESS)
Main PID: 1259 (java)
Memory: 1.3G
CGroup: /system.slice/zabbix-java-gateway.service
└─1259 java -server -Dlogback.configurationFile=/etc/zabbix/zabbix_java_gateway_logback.xml -classpath
lib:lib/android-json-4.3_r3.1.jar:lib/jboss-as-client-all.jar:lib/logback-classic...
Mar 01 12:18:50 xxxxxxADMIN systemd[1]: Starting Zabbix Java Gateway...
Mar 01 12:18:52 xxxxxxADMIN systemd[1]: Started Zabbix Java Gateway.
[root@xxxxxxADMIN ~]# systemctl status zabbix-agent.service
● zabbix-agent.service - Zabbix Agent
Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:50 CET; 6min ago
Process: 1336 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 1351 (zabbix_agentd)
Memory: 5.6M
CGroup: /system.slice/zabbix-agent.service
├─1351 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
├─1364 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
├─1366 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
├─1367 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
├─1368 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
├─1370 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
└─1371 /usr/sbin/zabbix_agentd: active checks #2 [idle 1 sec]
Mar 01 12:18:50 xxxxxxADMIN systemd[1]: Starting Zabbix Agent...
Mar 01 12:18:50 xxxxxxADMIN systemd[1]: Started Zabbix Agent.
[root@xxxxxxADMIN zabbix]# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:52 CET; 20min ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 1866 (httpd)
Status: "Total requests: 1028; Current requests/sec: -6.33; Current traffic: 0 B/sec"
Memory: 7.3G
CGroup: /system.slice/httpd.service
├─ 1866 /usr/sbin/httpd -DFOREGROUND
├─16996 /usr/sbin/httpd -DFOREGROUND
├─21232 /usr/sbin/httpd -DFOREGROUND
├─22416 /usr/sbin/httpd -DFOREGROUND
├─23802 /usr/sbin/httpd -DFOREGROUND
├─24622 /usr/sbin/httpd -DFOREGROUND
├─25412 /usr/sbin/httpd -DFOREGROUND
├─26024 /usr/sbin/httpd -DFOREGROUND
├─26033 /usr/sbin/httpd -DFOREGROUND
├─26487 /usr/sbin/httpd -DFOREGROUND
├─26555 /usr/sbin/httpd -DFOREGROUND
├─26556 /usr/sbin/httpd -DFOREGROUND
├─27799 /usr/sbin/httpd -DFOREGROUND
├─27806 /usr/sbin/httpd -DFOREGROUND
├─27807 /usr/sbin/httpd -DFOREGROUND
├─27885 /usr/sbin/httpd -DFOREGROUND
├─27926 /usr/sbin/httpd -DFOREGROUND
├─27934 /usr/sbin/httpd -DFOREGROUND
├─28072 /usr/sbin/httpd -DFOREGROUND
├─28073 /usr/sbin/httpd -DFOREGROUND
├─28074 /usr/sbin/httpd -DFOREGROUND
└─28075 /usr/sbin/httpd -DFOREGROUND
Mar 01 12:18:51 xxxxxxADMIN systemd[1]: Starting The Apache HTTP Server...
Mar 01 12:18:52 xxxxxxADMIN httpd[1866]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name,
using xx.xxx.xx.xxx. Set the 'ServerName' directive g... this message
Mar 01 12:18:52 ALBEEAADMIN systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.
==============================================
So it seems from the output above that things should be running/monitoring fine. What I was going to ask/send them was in order to narrow this issue down what type of user was running/logged into Zabbix? Is it a guest account? A regular user? Is this happening with every zabbix user or just one specifically? Just out of curiosity, if it is a Guest account maybe they can check/set permissions to access the host information registered as a Guests user group (i.e. Select Administration -> User groups 2. Guests group click 3. click Permissions tab)?
I also was going to ask them to provide the latest zabbix server logs when reproducing the issue seen in the picture above (such as the zabbix_server.log & zabbix_agentd.log)?
Does anyone have any additional thoughts on what to check/restart?
Thank you!
As seen in the attached, a client is complaining that since yesterday, they noticed that the Dashboards and Monitoring-> Problems section no longer seems to work (specifically “Problems” and “Problems by severity”) - see attached:
The user claims that no changes were done on the admin node so not sure why they are not seeing problems described in those sections.
What was already tried:
- Zabbix and httpd services restarted
- Admin node server restarted several times without success
- all related services are up and running
[root@xxxxxADMIN ~]# systemctl status zabbix-server.service
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:50 CET; 6min ago
Process: 1278 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 1288 (zabbix_server)
Memory: 220.3M
CGroup: /system.slice/zabbix-server.service
├─ 1288 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
├─ 2327 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.863163 sec, idle 60 sec]
├─ 2328 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
├─ 2329 /usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.000490 sec, idle 59 sec]
├─ 2330 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000592 sec, idle 5 sec]
├─ 2331 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000545 sec, performing discovery]
├─ 2332 /usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000027 sec, idle 1 sec]
├─ 2334 /usr/sbin/zabbix_server: history syncer #2 [processed 244 values, 0 triggers in 0.022097 sec, idle 1 sec]
├─ 2336 /usr/sbin/zabbix_server: history syncer #3 [processed 292 values, 7 triggers in 0.055103 sec, idle 1 sec]
├─ 2338 /usr/sbin/zabbix_server: history syncer #4 [processed 455 values, 1 triggers in 0.050551 sec, idle 1 sec]
├─ 2340 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.006683 sec, idle 3 sec]
├─ 2342 /usr/sbin/zabbix_server: java poller #1 [got 0 values in 0.000032 sec, idle 5 sec]
├─ 2343 /usr/sbin/zabbix_server: java poller #2 [got 0 values in 0.000031 sec, idle 5 sec]
├─ 2345 /usr/sbin/zabbix_server: java poller #3 [got 0 values in 0.000030 sec, idle 5 sec]
├─ 2346 /usr/sbin/zabbix_server: java poller #4 [got 0 values in 0.000023 sec, idle 5 sec]
├─ 2347 /usr/sbin/zabbix_server: java poller #5 [got 0 values in 0.000032 sec, idle 5 sec]
├─ 2348 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000029 sec, idle 5 sec]
├─ 2349 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000024 sec, idle 1 sec]
├─ 2350 /usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.000427 sec, idle 5 sec]
├─ 2352 /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000026 sec, idle 1 sec]
├─ 2354 /usr/sbin/zabbix_server: poller #2 [got 995 values in 0.297186 sec, idle 1 sec]
├─ 2356 /usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000028 sec, idle 1 sec]
├─ 2359 /usr/sbin/zabbix_server: poller #4 [got 0 values in 0.000029 sec, idle 1 sec]
├─ 2361 /usr/sbin/zabbix_server: poller #5 [got 0 values in 0.000031 sec, idle 1 sec]
├─ 2364 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000153 sec, idle 1 sec]
├─ 2365 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.000377 sec, waiting for connection]
├─ 2366 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.000088 sec, waiting for connection]
├─ 2367 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.000480 sec, waiting for connection]
├─ 2368 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.000101 sec, waiting for connection]
├─ 2369 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.000084 sec, waiting for connection]
├─ 2370 /usr/sbin/zabbix_server: icmp pinger #1 [pinging hosts]
├─ 2371 /usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.009546 sec during 5.009643 sec]
├─ 2372 /usr/sbin/zabbix_server: alerter #1 [sent 0, failed 0 alerts, idle 111.262759 sec during 116.265273 sec]
├─ 2374 /usr/sbin/zabbix_server: alerter #2 [sent 0, failed 0 alerts, idle 111.265718 sec during 116.333905 sec]
├─ 2378 /usr/sbin/zabbix_server: alerter #3 [sent 0, failed 0 alerts, idle 10.072499 sec during 15.076396 sec]
├─ 2380 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 4476 values, idle 5.657378 sec during
5.678010 sec]
├─ 2381 /usr/sbin/zabbix_server: preprocessing worker #1 started
├─ 2383 /usr/sbin/zabbix_server: preprocessing worker #2 started
├─ 2384 /usr/sbin/zabbix_server: preprocessing worker #3 started
├─12832 sh -c /usr/sbin/fping -C10 2>&1 </tmp/zabbix_server_2370.pinger;/usr/sbin/fping6 -C10 2>&1
</tmp/zabbix_server_2370.pinger;
└─12833 /usr/sbin/fping -C10
Mar 01 12:18:50 xxxxxADMIN systemd[1]: Starting Zabbix Server...
Mar 01 12:18:50 xxxxxADMIN systemd[1]: Started Zabbix Server.
[root@xxxxxADMIN ~]# systemctl status zabbix-java-gateway.service
● zabbix-java-gateway.service - Zabbix Java Gateway
Loaded: loaded (/usr/lib/systemd/system/zabbix-java-gateway.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:52 CET; 6min ago
Process: 1253 ExecStart=/usr/sbin/zabbix_java_gateway (code=exited, status=0/SUCCESS)
Main PID: 1259 (java)
Memory: 1.3G
CGroup: /system.slice/zabbix-java-gateway.service
└─1259 java -server -Dlogback.configurationFile=/etc/zabbix/zabbix_java_gateway_logback.xml -classpath
lib:lib/android-json-4.3_r3.1.jar:lib/jboss-as-client-all.jar:lib/logback-classic...
Mar 01 12:18:50 xxxxxxADMIN systemd[1]: Starting Zabbix Java Gateway...
Mar 01 12:18:52 xxxxxxADMIN systemd[1]: Started Zabbix Java Gateway.
[root@xxxxxxADMIN ~]# systemctl status zabbix-agent.service
● zabbix-agent.service - Zabbix Agent
Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:50 CET; 6min ago
Process: 1336 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 1351 (zabbix_agentd)
Memory: 5.6M
CGroup: /system.slice/zabbix-agent.service
├─1351 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
├─1364 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
├─1366 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
├─1367 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
├─1368 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
├─1370 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
└─1371 /usr/sbin/zabbix_agentd: active checks #2 [idle 1 sec]
Mar 01 12:18:50 xxxxxxADMIN systemd[1]: Starting Zabbix Agent...
Mar 01 12:18:50 xxxxxxADMIN systemd[1]: Started Zabbix Agent.
[root@xxxxxxADMIN zabbix]# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-03-01 12:18:52 CET; 20min ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 1866 (httpd)
Status: "Total requests: 1028; Current requests/sec: -6.33; Current traffic: 0 B/sec"
Memory: 7.3G
CGroup: /system.slice/httpd.service
├─ 1866 /usr/sbin/httpd -DFOREGROUND
├─16996 /usr/sbin/httpd -DFOREGROUND
├─21232 /usr/sbin/httpd -DFOREGROUND
├─22416 /usr/sbin/httpd -DFOREGROUND
├─23802 /usr/sbin/httpd -DFOREGROUND
├─24622 /usr/sbin/httpd -DFOREGROUND
├─25412 /usr/sbin/httpd -DFOREGROUND
├─26024 /usr/sbin/httpd -DFOREGROUND
├─26033 /usr/sbin/httpd -DFOREGROUND
├─26487 /usr/sbin/httpd -DFOREGROUND
├─26555 /usr/sbin/httpd -DFOREGROUND
├─26556 /usr/sbin/httpd -DFOREGROUND
├─27799 /usr/sbin/httpd -DFOREGROUND
├─27806 /usr/sbin/httpd -DFOREGROUND
├─27807 /usr/sbin/httpd -DFOREGROUND
├─27885 /usr/sbin/httpd -DFOREGROUND
├─27926 /usr/sbin/httpd -DFOREGROUND
├─27934 /usr/sbin/httpd -DFOREGROUND
├─28072 /usr/sbin/httpd -DFOREGROUND
├─28073 /usr/sbin/httpd -DFOREGROUND
├─28074 /usr/sbin/httpd -DFOREGROUND
└─28075 /usr/sbin/httpd -DFOREGROUND
Mar 01 12:18:51 xxxxxxADMIN systemd[1]: Starting The Apache HTTP Server...
Mar 01 12:18:52 xxxxxxADMIN httpd[1866]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name,
using xx.xxx.xx.xxx. Set the 'ServerName' directive g... this message
Mar 01 12:18:52 ALBEEAADMIN systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.
==============================================
So it seems from the output above that things should be running/monitoring fine. What I was going to ask/send them was in order to narrow this issue down what type of user was running/logged into Zabbix? Is it a guest account? A regular user? Is this happening with every zabbix user or just one specifically? Just out of curiosity, if it is a Guest account maybe they can check/set permissions to access the host information registered as a Guests user group (i.e. Select Administration -> User groups 2. Guests group click 3. click Permissions tab)?
I also was going to ask them to provide the latest zabbix server logs when reproducing the issue seen in the picture above (such as the zabbix_server.log & zabbix_agentd.log)?
Does anyone have any additional thoughts on what to check/restart?
Thank you!
Comment