Remote Commands doesn't work properly

[marcelotada]

Hello everybody!

Lets go to facts:

1. The trigger was created successfully generating events;
2. The action was created with no conditions to catch all events;
3. An action operation was created with a Remote Comand with the expression bellow:
a. "Zabbix server:/monitor/zabbix/zabbix-alarm-logger-1.0/bin/teste.sh"
Where "Zabbix server" is the host of "zabbix servers" (same of own zabbix server")
b. "teste.sh" is a shell script that create a log file writing a "hello world" inside.
4. The event happens with action operation status "executed";
5. The script is manually executable with zabbix user normally but nothing happens with the event execution.
6. The zabbix agentd is running properly;

So, whats going wrong? What can I do to solve or at least workaround this issue?

Regards.

[bcarpio]

Make sure the zabbix_agentd.conf is setup to allow remote commands

### Option: EnableRemoteCommands
# Whether remote commands from Zabbix server are allowed.
# 0 - not allowed
# 1 - allowed
#
# Mandatory: no
# Default:
EnableRemoteCommands=1

[marcelotada]

Hi bcarpio, thanks for your reply.

Yes, the EnableRemoteCommands is set to 1.

The most weird is that the remote command did runs properly sometimes in some thousands of events. (which each one should execute the script)

So what is wrong? What can I do to trace the problem? I have no idea since all looks correct.

Thanks

[bcarpio]

Turn on debug

Turn on debug mode in zabbix_server.conf and zabbix_agentd.conf then trigger the action this should provide enough info to figure out what is wrong. There will be a ton of info in the logs but you should be able to grep out the trigger name or script.

[marcelotada]

I'll turn debug mode on but I'm afraind cant do that in a production environment because there are a lot of systems been monitored.

I've founded some posts that points me to think there's a bug on zabbix but I have no sure because that is about run on a proxy, something not used here. The zabbix server and agentd is local.

http://www.zabbix.com/forum/showthread.php?t=19486

or

https://support.zabbix.com/browse/ZBXNEXT-936?page=com.atlassian.jira.plugin.system.issuetabpanels:changehistory-tabpanel#issue-tabs

Thanks.

[marcelotada]

I've been monitoring zabbix server log and agentd log after turned on both in debug mode.

The agentd.log shows me the lines bellow:

25429:20111219:180008.340 Requested [system.run["/home/zabbix/teste.sh","nowait"]]
25429:20111219:180008.340 Executing command '/home/zabbix/teste.sh'
25429:20111219:180008.341 Sending back [1]

Everthing looks normal but the script "teste" wasn't executed exactly what happens on our production environment. The log, even in a debug (4) mode doesn't show any problem at all.

However, we can see that server is sending the command to agent and the agent that aren't executing and doesn't log any error.

Can we report a bug to the zabbix dev team? Or relly on their support?

[frankymryao]

try to make a new trigger. We once faced this weird condition, too. All methods were tried but make no sense. After some time, it did work by making a some new trigger.

[marcelotada]

Yes but I already create a new trigger, many others actually with the same behavior.

All of it execute the remote command sending to the agentd but this one is that doesn't do anything.

What can I do to test the agentd? There are some kind of shell-based execution?

Thanks

[frankymryao]

Yes but I already create a new trigger, many others actually with the same behavior.

All of it execute the remote command sending to the agentd but this one is that doesn't do anything.

What can I do to test the agentd? There are some kind of shell-based execution?

Thanks

change your script to 'echo "test" >> /tmp/test.log'. check it will work or not. Perhaps the cause is your script.

[marcelotada]

change your script to 'echo "test" >> /tmp/test.log'. check it will work or not. Perhaps the cause is your script.

The script works fine when executed manually...

Thanks

[frankymryao]

The script works fine when executed manually...

Thanks

OK. The issue may be these:

1. zabbix not run script
2. zabbix run script, but didn't make sense

[marcelotada]

Hello everbody!

I finally solve the problem. I don't know exactly what was the root cause but following the steps I've maded:

- Re-configuration of config files zabbix-agentd.conf and zabbix-server.conf with Server as IP or hostname replacing 127.0.0.1; Hostname as a name without spaces (before was "Zabbix server")
- Restart both services
- Reconfiguration on zabbix server console the new host name, IP addr, connect to;
- The triggers were recreated and actions reconfigured too.

Them the script started works properly. However, sometimes it works sometimes not and I had to adjust too the script been executed which doesn't accept some spetial chars like "(, ), >, <".

Thank you everbody that help given to me ideas to debug and solve the problem.

[elcamlost]

Same issue ((

I'm just started to use Zabbix and think i have pretty same issue.

I create action with params .

Everyting seems right, when i disable proftpd service manually trigger start to work and perform predefined action

That's what i see in zabbix_server.log with debug enabled (string with db query removed)

22073:20120725:170537.464 In execute_operations()
22073:20120725:170537.464 In check_operation_conditions(): operationid [10]
22073:20120725:170537.465 End of check_operation_conditions():SUCCEED
22073:20120725:170537.465 Conditions match our event. Execute operation.
22073:20120725:170537.465 In substitute_simple_macros() data:NULL
22073:20120725:170537.465 In substitute_simple_macros() data:'{HOSTNAME}:sudo service proftpd restart'
22073:20120725:170537.465 In DBget_trigger_value()
22073:20120725:170537.465 End of trigger_get_N_functionid():SUCCEED
22073:20120725:170537.465 End of DBget_trigger_value():SUCCEED
22073:20120725:170537.465 End substitute_simple_macros() data:'vermont.ergeslab.ru:sudo service proftpd restart'
22073:20120725:170537.465 In add_command_alert()
22073:20120725:170537.465 In DCget_nextid() table:'alerts' num:1
22073:20120725:170537.465 End of DCget_nextid() table:'alerts' [10:10]
22073:20120725:170537.465 In run_commands()
22073:20120725:170537.465 In get_next_command(command_list:vermont.ergeslab.ru: sudo service proftpd restart)
22073:20120725:170537.465 End get_next_command(alias:vermont.ergeslab.ru,is_grou p:0,command:sudo service proftpd restart)
22073:20120725:170537.465 In run_remote_command(hostname:vermont.ergeslab.ru,co mmand:sudo service proftpd restart)
22073:20120725:170537.466 In get_value_agent() host:'vermont.ergeslab.ru' addr:'127.0.0.1' key:''
22073:20120725:170537.466 Sending [system.run["sudo service proftpd restart","nowait"]
]
22073:20120725:170537.471 get value from agent result: '1'
22073:20120725:170537.471 End run_remote_command(result:0)
22073:20120725:170537.471 In get_next_command(command_list
22073:20120725:170537.471 Result get_next_command [EOL]
22073:20120725:170537.471 End run_commands()

That's what i see in zabbix_agent.log

22122:20120725:170537.466 Processing request.
22122:20120725:170537.466 Requested [system.run["sudo service proftpd restart","nowait"]]
22122:20120725:170537.466 Executing command 'sudo service proftpd restart'
22122:20120725:170537.467 Sending back [1]

but actually nothing restarted ((

Any ideas how can i solve this problem?

[elcamlost]

Really? No Replies? It's very sadly ((

[mfecteau]

Timeout

I'm posting my solution to the problem I got with remote commands not executing.

By default, I think (someone correct me if I'm wrong) that a remote command has only 5 seconds to execute. If it takes longer, by default, Zabbix will kill the process. We need to change the "Timeout" value (default=5sec, max=60sec) in zabbix_agentd.conf and zabbix_server.conf.

Another solution is to start the command in a screen session, so the command could not be killed by Zabbix when it times out. Here's an exemple of a remote command :

ZABBIX_HOSTNAME:screen -dm -S MY_CUSTOM_SCREEN_SESSION /root/timeout.sh -t 180 my_command

/root/timeout.sh is an optional script to timeout my command after 180 seconds (a timeout after 60 seconds is not possible in zabbix_server.conf). I find it useful when I need to start a script that takes time to execute that will make an automatic call with pjsua (softphone) on a landphone when a critial alert has not been acked after a period of time. The timeout.sh script could be downloaded here :

http://www.bashcookbook.com/bashinfo/source/bash-4.0/examples/scripts/timeout3