After the upgrade to 6.0, all of my mysql-proxies and the main server all say "More than 100 items having missing data for 10 minutes." The problem is that two of the proxies only have 3 hosts and 9 items, yet they still show 100 items missing data error. Also, there is all of the data. I thought and update might fix it, the update made the error go away, but then it came back within an hour. The proxy log, only show configuration data coming in from the server. There are no errors indicated.
-
-
Have you checked that all the Zabbix components have the same synchronized time?
Markku -
Yes, the time is correct. I went through and checked each one.
Comment
-
What does Administration - Queue - Queue details say?
MarkkuComment
-
There is nothing in the queue.
Attached FilesComment
-
-
That does not look like the Queue details page. This is how it looks like here:
MarkkuComment
-
You are certainly correct, but I do have three other proxies and they fill up the page that you are referring to. All of my proxies are acting strangely, but the others are not quite as strange. Anyway, I do have items in the queue details but none from those two proxies.Comment
-
So, you are saying that (let's pick one of your "strange" proxies) the proxy has a long queue (= the proxy item that has the "More than 100 items having missing data for more than 10 minutes" trigger fired) but on Zabbix GUI the Queue overview by proxy table doesn't show anything in the "More than 10 minutes" column for that proxy and that the Queue details does not show any items for that proxy.
How are your proxies monitored?
For example, I have a proxy "hel-server-1", and I have a host "Proxy-hel-server-1" that has the Zabbix Proxy template assigned and that is monitored by the proxy "hel-server-1".
MarkkuComment
-
I have a five proxies. P1, P2, DMZ, A, and M and the host. All proxies are monitored by the host with the "Remote Zabbix proxy health, MySQL by Zabbix, and Linux by Zabbix" templates. Right now I have six alerts, one from each saying that there are more than 100 items missing data for more than 10 minutes. These alerts have been around for sometime after the upgrade to 6, but since the update to 6.0.4 reset them, they have all been there for about 6 days and 6 hours and some change.
I've included a screen shot of the queue overview. It shows the 0's in A, M, and Server. And about 50 in the DMZ. P1 and P2 should be generating an error, and I am still hunting down why this is happening. It was double that number a few days ago.Attached FilesComment
-
-
So clearly the proxies have queued items in that picture. What did you mean by "there is all the data" in your first post, how did you verify it if Zabbix says it cannot get all the data?
You will find your problem when you isolate the problem case enough. For example, troubleshoot one proxy only at a time, disable all monitored hosts but one only (on that proxy), and disable most of the items on that one host. And so on.
In my experience the usual way to get into long queues is to disconnect the active Zabbix agents (by having the agent not start automatically at reboot, or making a firewall change that blocks the agent, etc).
MarkkuComment
-
I looked at my first post, I am not sure where I was going with "there is all the data." I think it was a phrase that was left behind in editing.
You have some good ideas about troubleshooting, and I will work on the longer queues. But that doesn't fix the original problem. A proxy that only pings three web sites can't have 100 items missing data.Comment
-
I'd recommend you to also carefully check how you actually monitor the proxies, how the monitoring items (for the proxies) are configured and how they are getting their data. I already presented you how I monitor the proxies (using the Zabbix-provided proxy template), your current way of monitoring is something different. "Remote Zabbix proxy health" does not sound like a Zabbix-provided template so that's something else.
MarkkuComment
-
Those proxy templates probably came with my original install of Zabbix and were just never updated. I looked at the code and they were very similar to the latest ones, but I went ahead and removed the old ones for the proxies and the main server. The errors went away and then came back about 10 minutes later. I fully admit that I am not great about updating templates, but usually they just keep working. But it was worth a shot.
Comment
-
1. Are you still saying that the proxy reports a long queue in your proxy monitoring but you don't see any items in the Queue details list for that proxy?
2. If you only have three items monitored by the proxy, what does the latest data values list show you for those three items? (= Does the data arrive to the server or not)
3. Which Zabbix versions are you running for the server and the proxy?
MarkkuComment
-
I seem to have fixed it. I shutdown all of the proxies and waited until the server recognized those proxies were down. I then powered on each proxy from the smallest to the largest (# of hosts) until everything was back up.
During this process, I found and issue with the two proxies with virtually nothing on them. The Zabbix agent on those were using another proxy instead of the main host as it's upstream device. I fixed them to point to the main host. I could see how this could have affected the proxy process.
With all this the errors have gone away. Interestingly, the two queues that monitor the most hosts, are over 100 items for 10 minutes, but now no longer show an error, when perhaps they should.
But to answer your questions: 1. Yes, if I understand you. Error but 0 data. 2. Items are up to date. 3. All are the latest versions.Comment
Comment