Zabbix Sending PROBLEM and OK emails same time : False Alert?

[zabbixfk]

Hello All,

I have encountered kind of weired situation for 2 trigger configured.

Below are the trigger configurations.

1). Trigger to check zabbix agent reachablity. It checks if the zabbix agent in added host is not reachable for 5 mins, if yes then send the alert.

Name: Zabbix Agent is not reachable on {HOST.HOST} -> {HOST.IP} for 5 minutes.
Expression:{IT-Servers-Linux-:agent.ping.nodata(300)}=1
Multiple PROBLEM events generation : blank / unchecked.
Description: : Zabbix agent in {HOST.HOST} is not reachable for 5 mins.
URL:[/B] blank
Severity: Warning


2). Checks if the configured host is reachable using icmpping item.

Name: Ping reachablity check using icmp ping failed to {HOST.HOST}
Expression:

{IT-Servers-Linux-:icmpping[,5,,,300].last(0)}=0

Description: Ping reachability failed to {HOST.HOST} : {HOST.IP}
Severity: High

Both (1) and (2) triggers are getting triggered frequently. Hence i have disabled (1) - i.e to check zabbix agent reachablity.
But for (2) - ping check : i do not understand why its complaining. It was working fine till couple of days back.
When it triggers (2), it sends PROBLEM and OK together, in PROBLEM i get 0, and OK will have 1 as body. During the email ( i have configured email under Configuration -> Actions ), when i log in to zabbix server and check, the host for which email came will be pinging fine. Because of this, i am forced to ignore ping alterts, which resulted ignoring genuine alerts.

I have about 430+ hosts (linux, windows, swiches, f/w's etc) are being monitored using icmpping. Out of which about 390 are connected/monitored via
zabbix master server and rest of the 40+ hosts are distributed to 2 zabbix proxy servers ( under Configuration -> Host -> Select Monitored via proxy : also in host config file , put proxy ip address as zabbix server address).

I have

StartPingers=50

in zabbix master conf. I have also installed fping. In proxy servers , zabbix_proxy.conf has StartPingers=10.

I have 16GB ram in master server, 8 core machine with 300GB HDD ( raid 10). Max mysql connections are 160, innodb_buffer_pool_size=8G ( can't increase more, h/w problems)

Do i need to increase more StartPingers ? Or Do i need to add more proxy servers and distribute hosts?

Any pointers are greatly helpful.

Thanks.

[jan.garaj]

Triggers generates event only. They don't send any notifications. Actions are responsible for sending of emails, so check your actions.
It looks that you defined email actions without condition Trigger value = PROBLEM (see my attachment), but I'm not sure, because OK and PROBLEM emails contain different values. I recommend you to check your actions at first.

[zabbixfk]

Zabbix Sending PROBLEM and OK emails same time : False Alert?

Thank you for the reply. Yes i know that actions send emails not triggers. I have configured actions for above triggers as sending email. Well the action will send only when there is a trigger , pls see below image.


As per my understanding, PROBLEM and OK values are different, because of the trigger event ( as per trigger). Whenever there is a event which has unexpected value, PROBLEM event will trigger, and when it falls back to proper/expected value OK event will trigger and if actions are configured for that trigger, it sends email.

And in my case, i see color change in dashboard against the host ( i.e whenever PROBLEM/OK events triggered)

Please correct me if i am wrong. Do you think i need to change my trigger config/actions?

Thanks.

[jan.garaj]

Observe how one of the conditions is Trigger value = PROBLEM. Since actions are evaluated for every event, and a trigger switching from PROBLEM to OK is an event in and on itself, if you don't specify this condition the action will be executed both when the trigger switches to PROBLEM and when the trigger switches back to OK. Depending on how you have constructed your default message and what operations you intend to to with your actions, this may very well be what you intended and Zabbix would behave exactly as expected.

So if you don't have Trigger value = PROBLEM I'll expect 3 emails:
1.) PROBLEM
2.) OK (recovery) + 3.) again PROBLEM - because action don't have the conditions Trigger value = PROBLEM

For debugging try to add into your email message some macros:
{TRIGGER.STATUS} - current trigger status: OK,PROBLEM, and also UNKNOWN is possible - maybe your trigger is in UNKNOWN state for short time - check also your Monitoring -> Events how it looks
{ITEM.LOG.DATE1} {ITEM.LOG.TIME1} log datetime of item

[zabbixfk]

Zabbix Sending PROBLEM and OK emails same time : False Alert?

Thank you very much for the reply.

I have added {ITEM.LOG.DATE1} {ITEM.LOG.TIME1} to action body, and i could see that, both PROBLEM and OK emails are getting same time/date.

Now, i went to Monitoring -> Events, and select the host which is sending me ping related PROBLEM/OK emails ( in other words triggers/events ) frequently, i can see that mostly every 2-3 mins its saying PROBLEM/OK.

This applies to most of hosts added. So looks like, some problem with my pinger configuration? i have set StartPollers=50 and StartPingers=150 in zabbix_server.conf, i have max_connections=200 ( which reaches mostly up to 170), and 12GB RAM, out of which 8G is innodb_buffer_pool_size and Maximum possible memory usage: 9.9G (85% of installed RAM).

I suspect, Zabbix is simply generating trigger ( ping failed ), though ping is fine during the time of the trigger generated ( i checked using manual fping ).
Also i am seriously concerned about number of emails floating : my mailbox quota is getting full

Any pointers are greatly helpful.

[jan.garaj]

{IT-Servers-Linux-:icmpping[,5,,,300].last(0)}=0

Try adequate command in your Zabbix server command line:

/usr/sbin/fping -t 300 -C 5 <pinged_host>

Is 300ms timeout enough for your network?