Zabbix 1.8.11 Suddenly Stops Sending Notifications

[jashley]

For several months Zabbix was sending notifications just fine including my escalations. 5 days ago it stopped. No change was made to any of the Action configurations; I CAN send an SMTP message with telnet from my Zabbix server and I see NO actions when I look in the audit trail and nothing in my SMTP logs on my Exchange server.

Today I went into my windows template and changed the Ping Down from a 0 to a 1... this caused all of the servers to appear to be down on the dashboard but still NO email notifications.

Now I am to the point where I am going to remove my Actions and redo them but I thought this isn't Windows. I shouldn't have to do that. It is running on Ubuntu 12.04.2 with no security updates available.

Please help!!

[burn1024]

Did you try sending from localhost?

[jashley]

I'm not sure what you mean.. I was able to send an SMTP message using telnet. Also, this was working 100% before a few days ago. I'm very frustrated. I have been running Zabbix since 2007 and it has been up to now a set-it-and-forget-it program.

[burn1024]

I mean, did you try to use an SMTP server on localhost (exim, postfix, etc), not your remote Exchange?

[jashley]

Ahhh. As it was said before... This was working 100% before 6/20 using an Exchange server as the SMTP server.

So the issue is that it was working fine. Now it isn't. I have run Zabbix for years and never had to troubleshoot it now I am paying for it. I guess I need assistance on how to track this down.

[burn1024]

Ahhh. As it was said before... This was working 100% before 6/20 using an Exchange server as the SMTP server.

So the issue is that it was working fine. Now it isn't. I have run Zabbix for years and never had to troubleshoot it now I am paying for it. I guess I need assistance on how to track this down.

so, did you try to use an SMTP server on localhost (exim, postfix, etc), not your remote Exchange?

[jashley]

Ok.. So no I didn't. There are NO actions in the audit so it isn't even trying to send anything. In the "last 20 actions" screen there are no actions. I need help understanding why an item gets triggered in that screen but only intermittently sends an alert. I know everyone reading this thinks something must have changed but I am an IT manager with no time to tinker with something that has worked for me for years. So I am looking for some methodical troubleshooting techniques to get this tracked down.

I love Zabbix and will ALWAYS use it. I just need to get over this hump.

[jashley]

Any help would be appreciated.
Update: I got some alerts today but still not 100%

[tchjts1]

See this post and in particular, the last paragraph regarding the graphs for Zabbix internal performance. What are your performance metrics looking like over a day (or a week)?

https://www.zabbix.com/forum/showthread.php?t=41219

.

[jashley]

Thank you.. I have seen that thread but I do not have those graphs to look at. Was there an add-on?

[tchjts1]

It was introduced in 1.8.5
It's not really an add-on. It is simply a template. Check in your templates and see if you have one called "Template App Zabbix Server" and attach it to your Zabbix server.

If you don't have it, I am attaching it here. Import it and attach it.

[jashley]

That's great. I would love to get this in there however when I import the template it does not show up in my template list.

[tchjts1]

Try manually creating a template and name it exactly as this: Template App Zabbix Server

Don't create any items or triggers. Just leave it as that. Then do your import again, but before you click on the "Import" button, uncheck all boxes under "Update existing". Then under "Add Missing" try it with only these boxes checked: Templates, Items, Triggers, Graphs. Leave everything else unchecked.

Import.

[jashley]

That sounded like a great idea but it did not work. I even tried to create one of the items after it failed to see if it would update that item and still no joy.

This morning as usual I get my hourly alerts of only 2 of my 10 or so low disk space warnings. I have 2 other NEW alerts of low disk space that I am NOT getting a email about. I don't understand why it is so random.

Please help. I am at a loss.

[tchjts1]

Well, without the Zabbix App performance data, you will simply need to do some basic troubleshooting.

- How many hosts are you monitoring?
- Have you recently added more hosts for monitoring?
- On your Zabbix App server, are you seeing anything suspicious in zabbix_server.log? Maybe a lot of "First network error, retrying in 15 seconds"?
- In your zabbix_server.conf file, what values do you have for:
"StartPollers="
"StartPollersUnreachable="
"StartTrappers="
"Timeout="

For one of the hosts where you are not receiving alerts, but think you should be, in the Zabbix GUI, go to Monitoring --> Latest Data --> <host> --> "Filesystems" (or "Volume") category and find "Free disk space <drive> (percentage) or whatever item you are triggering on. Now, all the way out to the right, click on "Graph". Then at the top right in the dropdown, change "Graph" to "Values". Now change your zoom period to include a time where you should have gotten alerts. Do you have steady data coming in during that period? Are you missing chunks of data?

If there is no data for Zabbix to evaluate, it is not going to generate an alert.