Create/Monitor hosts dynamically

[manushnandan]

Hi all,

I'm new to the forum so please forgive me if I break any rules.

Currently we are in effort to monitor microservices using Zabbix. The scenario is that many instance of same application(Java) can run in same machine and when these instance comes up this has to be monitored by Zabbix without manually creating host in Zabbix. Our Java application runs in docker container, this application also have metrics that will be exposed through JMX and monitored in Zabbix.

We tried with Low Level Discovery but this would just allow me to have dynamic items/triggers but really what we would need is another JMX interface for each port to be created for each instance. There could be scenarios where we will run multiple instance on same host.

Below is one of the way which I tried, and it worked. Although it worked I haven't tested in all cases and I am not sure if this is perfect solution to deal this scenario since I started learning Zabbix few weeks back also I am looking for a more generic way from Zabbix side for doing this. If someone have any idea better than this please help me out.

Zabbix version - 3.0

Create a discovery shell script that will monitor with (ps -ef | grep ${SEARCH_PAT} | grep -Po ${REGEXP_PAT} | tail) command to check if any process comes up. This script will ouput service name and service port(JMX port) in JSON. I have item prototype attached to this discovery script which will create actual item. This item will call user defined script in the monitored host and creates host and apply given template to it. By this way I will create discovery rule for each service.

Discovery script : Key : discover.services[ProcessName]
//arg1 - process name to search
UserParameter.conf : UserParameter=discover.services[*],/var/lib/zabbix/discover_services.sh "$1"
Output of userparameter script: {"data":[ {"{#SERVICE_NAME}":"ItemName","{#SERVICE_SEARCH_PA TTERN1}":"ProcessName","{#SERVICE_PORT}":"10051"} ]}
Item prototype : create.host[{HOST.IP},{HOST.NAME},{#SERVICE_PORT},DMP_Servers, DMP_HTTP_Receiver1,HTTPReceiver]
// arg1 - hostip for jmx interface in newly created host
// arg2 - hostname - going to be the newly created hostname. Hostname format - Hostname+"_"+ProcessName+"_"Port
// arg3 - jmx interface port for monitoring
// arg4 - groupname
// arg5 - templatename that has to be applied for the newly created host
// arg6 - Process Name
UserParameter.conf : UserParameter=create.host[*],python /var/lib/zabbix/bin/hostadd.py "$1" "$2" "$3" "$4" "$5" "$6"
/var/lib/zabbix/bin/hostadd.py python script will first authenticate itself and gets a authorization key which will be used for following API calls.
Get the group id from given group name.
Get the template id from given template name and create new host with groupid and templateid.

*This python script uses Zabbix API to get all details required and to add host.
*It seems we can't have multiple requests in single api call to fetch all details. So each api call will be called seperately.

[manushnandan]

May be another Solution

As another solution can Zapcat (http://www.kjkoster.org/zapcat/Zapca...ix_Bridge.html) be used for above scenario along with discovery logic in Zabbix?

Each Java application has to start Zapcat inside the container. By doing so each java application will behave as one standard Zabbix agent and by using automatic network discovery functionality to discover hosts. In the ‘Discover rules’ attributes, Checks can be set as ‘Zabbix agent’.

For each discovered host, define one action to link one template doing jmx checking with this host.

Will this solution works perfectly?

[morten.moesgaard]

LLD with host prototype ??

Have you considered using the "Host Prototypes" ?
we have created a synthetic Host with the discovery of Projects on rancher.
you could with an advantage use that function, because then you can "create" and "destroy" hosts.
we have something similar to the first script you use, with the discovery part,
But via that script you could also assign templates.

[manushnandan]

Query on host prototype

Thanks for you suggestion.

what does synthetic host means?

Did not try with host prototype, but is it possible to create new host with different JMX interface depending upon output from discovery script(I see that the Agent and JMX interfaces are in readonly mode), even if i can create new host with different JMX interface is it possible to assign different kind of template depending upon output from discovery script(maybe through regex) eventually there is going to be different kind of process(docker container) running in particular machine which needs to be assigned with their own template.

[kedaly]

Did you ever get a solution for this?
We are also building microservices that we would like to monitor with Zabbix. Since our services are inside Docker Swarm, Zabbix can't pole them directly, so we need to build out an active agent.