Data Randomly stopped being recorded

[sfereday]

Good (whatever) all.
The setup:
CentOS Linux release 7.9.2009
VM running on qemu 1.5.3
6 CPU
18GB Mem
Zabbix version 5.0.3

Last night around 0300 several data points stopped recording data.


I had this issue before, but i thought it was an IO error. I didn't find IO errors like I did before.

I restarted the the mariadb and checked the status:
$ mysqladmin -u zabbix -p status
Uptime: 915 Threads: 26 Questions: 53833 Slow queries: 0 Opens: 127 Flush tables: 2 Open tables: 153 Queries per second avg: 58.833

Not all of the servers reporting to Zabbix are doing this, and not all data points on a single server are doing this.
I'm been searching forums and will continue, but I haven't found anything that has pointed me in the right direction yet. Anyone seen this before?

[sfereday]

Also, my performance figures.

UID PID PPID C STIME TTY TIME CMD
zabbix 8677 1 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
zabbix 8684 8677 0 08:25 ? 00:00:01 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.273671 sec, idle 60 sec]
zabbix 8688 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: housekeeper [deleted 135365 hist/trends, 0 items/triggers, 21 events, 0 sessions, 0 alarms, 0 audit items, 0 records in 44.482344 sec, idl
zabbix 8689 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.001556 sec, idle 59 sec]
zabbix 8691 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: http poller #1 [got 1 values in 0.034013 sec, idle 5 sec]
zabbix 8692 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000589 sec, idle 60 sec]
zabbix 8694 8677 0 08:25 ? 00:00:01 /usr/sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000053 sec, idle 1 sec]
zabbix 8696 8677 0 08:25 ? 00:00:01 /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000060 sec, idle 1 sec]
zabbix 8697 8677 0 08:25 ? 00:00:01 /usr/sbin/zabbix_server: history syncer #3 [processed 0 values, 0 triggers in 0.000053 sec, idle 1 sec]
zabbix 8700 8677 0 08:25 ? 00:00:01 /usr/sbin/zabbix_server: history syncer #4 [processed 45 values, 41 triggers in 0.112212 sec, idle 1 sec]
zabbix 8702 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.001341 sec, idle 3 sec]
zabbix 8703 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000031 sec, idle 5 sec]
zabbix 8704 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000056 sec, idle 1 sec]
zabbix 8705 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.000437 sec, idle 5 sec]
zabbix 8706 8677 0 08:25 ? 00:00:03 /usr/sbin/zabbix_server: poller #1 [got 13 values in 0.163156 sec, idle 1 sec]
zabbix 8707 8677 0 08:25 ? 00:00:03 /usr/sbin/zabbix_server: poller #2 [got 15 values in 0.199561 sec, idle 1 sec]
zabbix 8708 8677 0 08:25 ? 00:00:03 /usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000039 sec, idle 1 sec]
zabbix 8709 8677 0 08:25 ? 00:00:03 /usr/sbin/zabbix_server: poller #4 [got 44 values in 0.333630 sec, idle 1 sec]
zabbix 8711 8677 0 08:25 ? 00:00:03 /usr/sbin/zabbix_server: poller #5 [got 0 values in 0.000066 sec, idle 1 sec]
zabbix 8712 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000043 sec, idle 5 sec]
zabbix 8716 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.007571 sec, waiting for connection]
zabbix 8717 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.008205 sec, waiting for connection]
zabbix 8718 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.000458 sec, waiting for connection]
zabbix 8719 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.000472 sec, waiting for connection]
zabbix 8720 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.007484 sec, waiting for connection]
zabbix 8721 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: icmp pinger #1 [got 0 values in 0.000038 sec, idle 5 sec]
zabbix 8722 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.005953 sec during 5.006117 sec]
zabbix 8723 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: alerter #1 [sent 0, failed 0 alerts, idle 123.460696 sec during 123.608042 sec]
zabbix 8724 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: alerter #2 [sent 0, failed 0 alerts, idle 623.927037 sec during 624.068679 sec]
zabbix 8725 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: alerter #3 [sent 0, failed 0 alerts, idle 1104.252690 sec during 1104.424240 sec]
zabbix 8726 8677 0 08:25 ? 00:00:05 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 157 values, idle 5.586641 sec during 5.589754 sec]
zabbix 8727 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: preprocessing worker #1 started
zabbix 8728 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: preprocessing worker #2 started
zabbix 8729 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: preprocessing worker #3 started
zabbix 8730 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: lld manager #1 [processed 2 LLD rules during 5.331810 sec]
zabbix 8731 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: lld worker #1 [processed 1 LLD rules, idle 8.334342 sec during 8.335105 sec]
zabbix 8732 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 19.937393 sec during 19.938192 sec]
zabbix 8733 8677 0 08:25 ? 00:00:00 /usr/sbin/zabbix_server: alert syncer [queued 0 alerts(s), flushed 0 result(s) in 0.001023 sec, idle 1 sec]

[sfereday]

My zabbix queue

[sfereday]

And My Zabbix Server Statistics via the Zabbix Server Template

[sfereday]

The last two images would be when the server is working it seems. The issue has randomly stopped.

[sfereday]

Cleared out the queue errors now.