Action on discovery does not work

[kanour]

Hi.

I have configured some actions for discovery, but they seem to not work. Can you advise me what is wrong with my rules?

Discovery works:
172.16.0.0/16 3600 SSH,TCP,SNMPv2 agent,ICMP Ping

Actions seem to not work:

Action 1
Uptime/Downtime >= "600"
Discovery status = "Up"
Service type = "ICMP Ping"
Host IP = "172.16.11.0/24"

Link to template "Template_Ping"
Add to group "User VLAN"
Add host

Action 2
Uptime/Downtime >= "600"
Discovery status = "Up"
Service type = "SSH"
Host IP = "172.16.0.1/29"

Link to template "Template_Ping"
Add to group "Apaclab"
Add host


One more question:
Is it possible to create action condition with combination of AND and OR? Eg: (A and B) or (C and D).

[kanour]

Can anybody point me how I can debug this issue. Is there any real time log which shows me what is the problem and why the Action Condition does not work.

[kanour]

Anyone can outline how to debug the discovery action rules. I tried all combinations but I am still not able to move discovered machines to particular groups.

Thank you.

[untergeek]

I was not aware that Discovery Actions would allow you to specify a subnet instead of a host IP. If that's the case, I don't see anything necessarily wrong.

It depends on ordering and item type for AND and OR operations. Experiment and see if you can convince it to work for you.

[kanour]

Hi.

The problem is that I already tried all possible combinations. I would like to approach it differently - configure debug logging to see why the action fails. It is probably condition issue.

Is there any way how to test the condition manually and see what is the problem? Or set debug logging and watch why the action fails?

[alixen]

Hi,

Can anybody point me how I can debug this issue. Is there any real time log which shows me what is the problem and why the Action Condition does not work.

I have also tried to use automatic discovery and I find that Zabbix don't give enough feedback on what is going on.

In case you missed it, there are two places where you'll get some information:

• Monitoring -> Discovery : shows hosts discovered
• Monitoring -> Events -> Source : Discovery (in dropdown menu) : shows events fired by discovery

I have not tried next tip yet but I will do it next time I need automatic discovery:
Create a simple action associated to discovery, valid on any condition, that sends an email with all discovery specific macros:
{DISCOVERY.DEVICE.IPADDRESS}
{DISCOVERY.DEVICE.STATUS}
{DISCOVERY.DEVICE.UPTIME}
{DISCOVERY.RULE.NAME}
{DISCOVERY.SERVICE.NAME}
{DISCOVERY.SERVICE.PORT}
{DISCOVERY.SERVICE.STATUS}
{DISCOVERY.SERVICE.UPTIME}

Hope this helps
Alixen

[alixen]

Hi,

As I suggested in previous post, I have set up an action with:

Name : Discovery - Any event
Event source : Discovery
Default subject: Discovery - {DISCOVERY.DEVICE.IPADDRESS}
Default message:

Date-time                  : {DATE}-{TIME}
Event                      : {EVENT.DATE}-{EVENT.TIME} 

DISCOVERY.DEVICE.IPADDRESS : {DISCOVERY.DEVICE.IPADDRESS}
DISCOVERY.DEVICE.STATUS    : {DISCOVERY.DEVICE.STATUS}
DISCOVERY.DEVICE.UPTIME    : {DISCOVERY.DEVICE.UPTIME}
DISCOVERY.RULE.NAME        : {DISCOVERY.RULE.NAME}
DISCOVERY.SERVICE.NAME     : {DISCOVERY.SERVICE.NAME}
DISCOVERY.SERVICE.PORT     : {DISCOVERY.SERVICE.PORT}
DISCOVERY.SERVICE.STATUS   : {DISCOVERY.SERVICE.STATUS}
DISCOVERY.SERVICE.UPTIME   : {DISCOVERY.SERVICE.UPTIME}

Status: Enabled
Conditions : None
Action operations : Send message to User "admin"

I am getting mails with:

Subject: Discovery - xxx.xxx.2.99
Body:
Date-time                  : 2010.12.08-12:59:52
Event                      : 2010.12.08-12:59:50 

DISCOVERY.DEVICE.IPADDRESS : xxx.xxx.2.99
DISCOVERY.DEVICE.STATUS    : UP
DISCOVERY.DEVICE.UPTIME    : 14951d 11h 59m
DISCOVERY.RULE.NAME        : My Network
DISCOVERY.SERVICE.NAME     : HTTP
DISCOVERY.SERVICE.PORT     : 80
DISCOVERY.SERVICE.STATUS   : UP
DISCOVERY.SERVICE.UPTIME   : 0m

or

Subject: Discovery - xxx.xxx.2.99
Body:
Date-time                  : 2010.12.08-12:58:09
Event                      : 2010.12.08-12:57:16 

DISCOVERY.DEVICE.IPADDRESS : xxx.xxx.2.99
DISCOVERY.DEVICE.STATUS    : UP
DISCOVERY.DEVICE.UPTIME    : 0m
DISCOVERY.RULE.NAME        : My network

I post these examples here since I think it may be useful for discovery debugging.

Regards,
Alixen

[fmrapid]

Alixen that is a very neat trick for debugging Discovery. You should post it up on the wiki.

Cheers,

fmrapid

[kanour]

Hi.

I created the following action for testing:
Uptime/Downtime >= "600"
Discovery status = "Up"
Service type = "ICMP Ping"
Host IP = "192.168.40.2-254"

I also implemented the messages as you suggested. Let's see how it goes.

[kanour]

Hi.

So I created simple rule:
Uptime/Downtime >= "600"
Discovery status = "Up"
Service type = "ICMP Ping"

It started to work after some time (not instaltly).

Now I am receiving emails with:
Date-time : 2010.12.22-17:28:02
Event : 2010.12.22-17:28:01
DISCOVERY.DEVICE.IPADDRESS : 172.16.1.50
DISCOVERY.DEVICE.STATUS : UP
DISCOVERY.DEVICE.UPTIME : 27d 22h 14m
DISCOVERY.RULE.NAME : apaclab
DISCOVERY.SERVICE.NAME : ICMP Ping
DISCOVERY.SERVICE.PORT : 0
DISCOVERY.SERVICE.STATUS : UP
DISCOVERY.SERVICE.UPTIME : 27d 22h 14m

Now I am going to set some other Actions to see if those will work.


Can someone explain why the started to work after some time and not instantly?

(by the way thanks for the tip with the email)

[kanour]

This is interesting. I disable the action (above), waited 30 seconds and enabled again and it did not send any notification yet (12 minutes now).

[untergeek]

Discovery doesn't happen instantaneously, not even after a rule is changed. Zabbix keeps track of when a discovery rule was last executed and doesn't run again until the next time.

As my experience has demonstrated (your mileage may vary), Discovery also doesn't happen all at once, but rather from one IP to the next, depending on the number of "discoverers" you have set in the zabbix_server.conf.

I have my discovery rules run every 30 minutes, but it can take up to 2 or 3 hours before a new host is added. Why? Not sure, but that's been my experience. I've seen some network crawls just…hang…and that's been my assumption with my Zabbix delays–that it's just hanging too.

You could run NMAP and see what you find there.

[kanour]

Futher testing shows, that it somehow works. But it is very unpredictable.

I have got several hosts on 172.16.0.0/16 subnet.

Having this rule:

Uptime/Downtime >= "600"
Discovery status = "Up"
Host IP = "172.16.10.0/24,172.16.11.0/24,172.16.12.0/24,172.16.13.0/24,172.16.14.0/24,172.16.15.0/24,172.16.16.0/24,172.16.17.0/24,172.16.18.0/24,172.16.19.0/24,172.16.20.0/24,172.16.21.0/24,172.16.22.0/24,172.16.23.0/24,172.16.24.0/24,172.16.25.0/24,172.16.26.0/24"

Link to template "Template_Ping"
Add to group "APACLAB VLAN USERS"
Send message to User "kanour"

10 hosts were added to the group, but then it stoped. I do not know why. I have around 500 hosts on the network. They are all discovered and shows up (green).

I would expect some logic behind it. Is there anywere more detailed explanation how it works to predict the behaviour. I think it simply does not work correctly.
I would expect that the 500 discovered hosts will be added to the group quite quickly, because they are discovered and they are up. But that is not happening.

[untergeek]

How many discoverers are configured in zabbix_server.conf?

StartDiscoverers=???

If you're trying to discover that many hosts, you'll want more than 10, perhaps 25 to 50. It all depends on what you're polling for and how frequently.

Next, you define the polling frequency in the discovery rule. The discovery action will not run more frequently than you have defined in the discovery rule.

I have actually had discovery "lock-up" on me and not recover until I restarted the zabbix_server processes. Perhaps you need to increase the number of discoverers.

[kanour]

I had the default. I configured StartDiscoverers=30 now. Lets see how it goes.