Trigger Trouble

[thefuzz4]

So I have this trigger that I created

{Template App Splunk Forwarder Serviceroc.num[,root,all,splunkd].last(0)}=0

All I want it to do is if there are 0 splunkd processes running let the trigger fire so the action can be ran to go restart the forwarder.

Problem is that in my logs with debug level 5 turned on this is what I see

38974:20160610:093458.274 In evaluate_function() function:'havok01roc.num[,root,all,splunkd].last(0)'

So I can see its trying to evaluate it. Zabbix does indeed tell me that the forwarder is down. I just don't think that my trigger is written correctly in order to catch this state.

I think that once I have this thing working correctly the action should fire. If not I'll visit that next. I've read through the wiki and searched to no end to try and find a nice example of what I'm attempting to pull off here. Thank you all for your help with this in advance and I'm sure its something simple and stupid.

[LenR]

Does latest data for the item show the number of running procs and does it go to 0 when none are running?

[thefuzz4]

Good Evening Len,

Yes in the graph it will show that the number went from 2 to 0 and the dashboard does show that the process is not running.

[thefuzz4]

So when I'm intentionally attempting to trigger this thing

I have the debug level set at 5 and I see this in the logs

100548:20160613:160431.452 In evaluate_function() function:'havok01:proc.num[,root,all,splunkd].last(0)'
1. Splunk Forwarder Process (havok01:proc.num[,root,all,splunkd]): 0
1. Splunk Forwarder Process (havok01:proc.num[,root,all,splunkd]): 0
1. Splunk Forwarder Process (havok01:proc.num[,root,all,splunkd]): 0
100549:20160613:160501.026 In evaluate_function() function:'havok01:proc.num[,root,all,splunkd].last(0)'

Which does tell me that hey the trigger sees it but for whatever reason the action will not fire.

This is what the action is looking for

Maintenance status not in maintenance
Template = Template App Splunk Forwarder Service
Trigger value = PROBLEM
Trigger severity >= Warning

[LenR]

I've never used the Template condition before, you might remove that for testing.

Can you post what you have for the operation?

Also for testing, change your operation to just send an email. Make sure the user in the email has permissions for the host. (Our most frequent problem with actions is permissions)

[thefuzz4]

Ok so here is the trigger conditions

Maintenance status not in maintenance
Trigger value = PROBLEM
Trigger severity >= Warning

Here is the action

Send message to user groups: Zabbix administrators via all media

I know that it should be sending an email out now

Looking at the actions result on the dashboard just shows my default notification. It still didn't fire though .

Thank you so much for your help with this. I'm sure there is something silly that I'm just overlooking here with this.

[LenR]

More basic questions, sorry if they are obvious things you've already done.

1) Other action emails work?

2) The trigger appears on the dashboard with Warning or higher severity?

[thefuzz4]

Yeah the default message email works that came with the install.

No nothing shows up on the dashboard for this action/trigger that anything happened.

[LenR]

That sounds like the problem is in the trigger then.

Maybe try just proc.num[splunkd] for the item.

[thefuzz4]

So something like this?

Service:proc.num[splunkd].last(0)}=0

[LenR]

Yes, I actually have that trigger, save for the template name. I have one case where it triggered, but it's several months ago. It probably triggered at reboot, it was 0 for 1 sample cycle, so it wouldn't have tripped any operation in my action.

[thefuzz4]

Can you share your expression. I'm trying to save it but I keep getting

Incorrect trigger expression. Check expression part starting from "Service:proc.num[splunkd].last(0)=0".

[LenR]

Here's the template - https://file.io/qezl0j

Did you change the item first? Simplify the item key, the trigger should change automatically.

[thefuzz4]

Woohoo that did the trick by changing the item key. Thank you so much for your help with this mess

[LenR]

I still don't understand why the original item proc.num[,root,all,splunkd] didn't work. That should have been "all" processes, running as root, in any state, with splunkd in the command line.

I can test on my splunkd running host with zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf -t proc.num[,root,all,splunkd] and I get 4. With proc.num[splunkd], I get 3. I suspect the 4 includes the zabbix_agentd running with the parm :-)