Custom scripts to monitor applications

[static0verdrive]

Hi all!

We are still loving Zabbix for our monitoring solution. We are currently running 4.0.4 in production, with proxies and additional servers in the regions so the admins there can monitor their own hosts, but we at HQ can monitor everything.

Everything works well out-of-the-box for standard checks, but now we are being asked to have the Zabbix server call custom health/check scripts on the hosts so an admin can be alerted if an application isn't working as expected. In an effort to reduce the workload (and required expertise with respect to custom applications we provide servers for, but have no actual hands-on experience with ourselves) we've asked the admins and developers to create their own scripts to check the health of their apps. After all, they are the experts on those apps, unlike the rest of us in the infrastructure team.

I am hoping someone can assist me in figuring out how to have the Zabbix server call a custom script as such, and alert if needed based on the output. What is the "Best Practice" and the required steps to configure the Zabbix server to run a script and alert if there are errors found? For example, the team responsible for an app we have on Windows wrote a simple PowerShell script:

# Pseudocode for C:\Scripts\check.ps1
if port available
echo "No issues found"
exit (0)
else
echo "Not responding"
exit (1)


With that in place on the host, here's what I tried to do on the Zabbix server under Administration > Scripts > "Create script" :
Name: Test Windows Check
Type: Script
Execute on: Zabbix agent
Commands: C:\Windows\System32\WindowsPowerShell\v1.0\powersh ell.exe -ExecutionPolicy Bypass C:\Scripts\check.ps1 -RunType $True -Patch C:\
...

Questions:
1. Is Administration > Scripts > "Create script" the correct place to configure it?
2. Do I need to create a host group for each server in order to have different scripts on the hosts?
3. How do I schedule this to run every 15 minutes? (Do I need to create a corresponding item for the script? Do I need to use the UserParameter= line in the zabbix_agentd.conf on the host?)
4. How do I send an alert if it exited with a non-zero (hopefully in more complicated ones I can create let's say an email if exit=1, an SMS if exit=2, and so on). Would this be done via triggers or events?

I'm feeling a bit lost despite having a good grasp of the basics for Zabbix installation and maintenance. I noticed in https://www.zabbix.com/forum/zabbix-...rom-a-cron-job that someone is using zabbix_sender, but even that doesn't seem to be as straight-forward as I thought at first, and that isn't part of the agent package but a separate install. I think the -k is for a key, and he still created an item for it. Where does one define the key? In the UserParameter= line of the zabbix_agentd.conf on the host? Or on the Zabbix server somewhere? I think I need a full rundown, as the wiki has left me with more questions than when I first started trying to get these working!


Thanks in advance for any assistance you can provide, and as always, keep up the amazing work.

[static0verdrive]

I also just came across this link and maybe this is more of what I need? I think I'm even more confused now...
https://www.zabbix.com/documentation...s/media/script

edit: This appears to be more for what to do if a problem was found rather than logic to determine if an action is needed..?

[Atsushi]

1. Is Administration > Scripts > "Create script" the correct place to configure it?

If you want to register a script to retrieve the value of an item for monitoring, the place to set it is not that location.

If you want to run the script on Windows to be monitored, you need to place the script on Windows. In order to call it, it is necessary to use UserParameter or to use sysytem.run[].

2. Do I need to create a host group for each server in order to have different scripts on the hosts?

Scripts must be placed on each Windows server.
Although it is possible to assign the same template with the same script name, it is advisable to recognize what is the script for administrative purposes and to separate the monitoring template for each role.

3. How do I schedule this to run every 15 minutes? (Do I need to create a corresponding item for the script? Do I need to use the UserParameter= line in the zabbix_agentd.conf on the host?)

Register script call processing as an item and set the update interval to 15 minutes.

4. How do I send an alert if it exited with a non-zero (hopefully in more complicated ones I can create let's say an email if exit=1, an SMS if exit=2, and so on). Would this be done via triggers or events?

When handling as Zabbix item value, please output it as character (or numeric) to standard output instead of exit code. Then set the trigger for that item. Please refer to the thing set in the template for the setting of the trigger.
However, if you want to execute different actions depending on the value, you will need to set multiple triggers and actions.


Manual: 4 User parameters

https://www.zabbix.com/documentation/4.0/manual/config/items/userparameters

[static0verdrive]

Thank you very much for your reply Atsushi - it has helped immeasurably!

== Here's what I did in case it helps others in the future ==

On the application host, create the script and configure/restart the zabbix agent:
1. Create C:\Scripts\check.ps1 to check the health of a custom application, which will echo the number of the error found (or 0 if no errors found).
2. Edit C:\Zabbix\zabbix_agentd.conf to uncomment the UserParameter= line and append to that line as follows:
UserParameter=healthCheck,C:\Windows\System32\Wind owsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass C:\Scripts\check.ps1 -RunType $True -Patch C:\
3. Restart the zabbix agent service.

On the Zabbix server, create the item and trigger:
4. Under Configuration > Hosts, select "Items" next to the application host in question.
5. Click "Create Item" (top right). Set the Type to "Zabbix agent" (but I will have to experiment to see if "Zabbix agent(active)" would be better / more appropriate). Ensure the key matches the key provided in #2 above (healthCheck in this case). Fill in the other required fields such as "Update interval" (how often to perform this check) and save the item.
6. You should be back on the Configuration > Hosts screen. Select "Triggers" (next to the application host in question).
7. Click "Create Trigger" (top right). In the first "Name" field, enter a meaningful error or problem description ("check.ps1 failed to find the required file" in this case). Set the desired Severity ("Average" in my case).
8. Next to the "Expression" box, click the "Add" button to locate and select the item created in #5 above.
9. The "Condition" box appears with the new/custom item populated in the first "Item" box. Set the result at the bottom to the result which indicates a problem was found (in my case, not equal to zero, therefore "<>" in the drop-down menu and "0" in the textbox.

=====

The above works, but I have a couple of quick question for the gurus to advance my understanding:

- When creating the item, under "Applications" should I choose "Zabbix agent" or create a new one using the "New Application" field just above?

- Currently I left "Show value" on "As is" (but it appears I can create value mappings for what each error number represents). Where are these displayed? In the Zabbix log? Right now I do not see the value returned at all on the Problem/Severity dashboard or in the list of problems specific to that host, but I would love to include what the issue is based on value mapping.

Cheers!
-static-

[Atsushi]

- When creating the item, under "Applications" should I choose "Zabbix agent" or create a new one using the "New Application" field just above?

I think that application settings should be used to group the items. The setting of this application is used for filtering at item list display, tree display with latest data, etc.

If you want to group and manage them in the same way as those registered in the existing application, select the existing application. If you want to group and manage as individual business application, create an application for that business application.

- Currently I left "Show value" on "As is" (but it appears I can create value mappings for what each error number represents). Where are these displayed? In the Zabbix log? Right now I do not see the value returned at all on the Problem/Severity dashboard or in the list of problems specific to that host, but I would love to include what the issue is based on value mapping.

The result of the value mapping should have been used when displaying the latest data or specifying a macro that refers to the value of the item. As an example, take a look at the Agent ping value of the Zabbix agent.

Monitoring -> Latest data -> Zabbix agent -> Agent ping

The setting of value mapping can be found in the following screen.

Administration -> General -> Value mapping

You can use existing ones or define new ones.