Automating server checks via Puppet (or any CMS): brainstorm needed

[Mojah]

Hi,

I'm jealous of the Nagios users out there for a very simple reason: their configs are all text-based and *very* easy to manage with a tool like Puppet/Chef/Ansible/CFEngine/...

What I would like to do is:
- have hosts added automatically as soon as it's added in our CMS
- have checks added automatically to hosts as soon as I configure a service or module in our CMS (ie.: easy checks like HTTP/Ping and more advanced like web-content checks per vhost)
- have checks removed automatically when a module is removed from our CMS

In Nagios it's relatively easy: you generate text-files from the CMS, reload Nagios, done. Purge the directory where your configs are and all obsolete configs are automatically deleted.

In Zabbix, it's more tricky since everything is stored in the database. Has anyone found a proper, scalable, solution to use the Zabbix API for this? Does this work? Is your zabbix frontend overloaded with API requests like this?

Looking for more feedback on how to implement this in a scalable fashion.

M.

[Colttt]

you can use the API, or you can use the autodiscovery..

[Mojah]

you can use the API, or you can use the autodiscovery..

Obviously :-)

Autodiscovery is too limited to do per-service checks or per-service vhosts and for the API I wonder if it can handle this kind of traffic (or if it was even designed for this?).

Has anyone implemented it in either way?

[Colttt]

in zabbix 2.2 or 2.4 the complete frontend work only with the API..

https://www.zabbix.com/documentation/2.0/manual/appendix/api/api

maybe you can check if port XX open then its a Webserver and add the Template "Webserver" and add group Websrv..

[f.koch]

hi,

we assign base templates based on parts of the hostname atm. The specific templates manually.
But we migrated our environment from spacewalk to puppet last year , and now i think about to reimplement the auto assign vit a custom puppet type and the zabbix api. This way we can also assign specific templates, and have the logic in the puppet classes.

The last time i see the frontend overloaded was in the early 1.8 days, and the reason was a bug.

regards fl0

[dale]

I use auto-discovery rules/checks to find servers and services then discovery actions to provision templates.

For example. Here's a couple of checks out of several for a discovery rule called "BigIP Scan":
* Zabbix agent "proc.num[ns-slapd]"
* Zabbix agent "proc.num[memcached]"

I then have discovery actions to provision checks for each type of server/service.
Here is a discovery action called "Provision LDAP Servers":

Conditions:
* Discovery check = "BigIP Scan:Zabbix agent "proc.num[ns-slapd]""
* Received value >= "1"

Operations:
* Add to group "LDAP Servers"
* Link to template "Template_LDAP_Servers"

I like this because I don't necessarily have puppet-like capabilities on every server I'm monitoring, and this keeps things consistent and hopefully transparent.

[dennisj]

What I would like to see is the addition of an idempotent function to create/set object that would work in the following way:

item.set(item-data)

If the item with the given key doesn't exists on the host create it.
If the item already exists and all attributes are identical do nothing.
If the item already exists and some attributes are different update the item in the db.

Such a function should then be created for all object types e.g. items, triggers, hosts, etc.
Alternatively a parameter could be added to the create function that indicates that if the item already exists it should be updated instead (sort of like the "on duplicate key update" functionality of MySQL).

This would make it much easier create e.g. a puppet type that creates items, assigns templates, etc. in an idempotent way.

[danrog]

Hi Mojah, can you give specific examples of a service you are trying to discover? I use low level discovery to find a hosts services that is supposed to be running (and this works well across our entire estate of 1500+ hosts [discovery runs once a day for this one, but we have others that run hourly - primarily network/SNMP device related]). We then manage those services (along with the discovery scripts like the one below) with puppet. I'm sure if it hasn't be done yet, someone will create puppet module/lib for the Zabbix API or hooks to the Zabbix API Ruby console tool.

# Lists all chkconfig compatible installed services plus a custom set of services not chkconfig compatible out of the box (e.g. tripwire)
UserParameter=discovery.linux.services[*],(chkconfig --list; for i in $1; do if [ -r /etc/init.d/$$i ]; then echo "$$i 3:on"; fi; done) | /bin/awk '/3:on/{ c="\n"; printf "\"{#SERVICE}\":\""$$1"\""c}' | /bin/awk 'BEGIN{printf "{\"data\":["};{printf c"{"$$0"}";c=","};END{print "]}"}' 

As for v-host type stuff, you can do the same sort of thing with a custom script that creates LLD formatted output.

Regarding the API, we use it as the only source of data for some custom web pages (events, triggers, history up to an hour and IT Services) and there have been no issues (a few of the pages inside Zabbix 2.0 use the API fully today). I would suggest you monitor your mysql server for slow queries and test out the API to see if it will meet your needs.

Obviously :-)

Autodiscovery is too limited to do per-service checks or per-service vhosts and for the API I wonder if it can handle this kind of traffic (or if it was even designed for this?).

Has anyone implemented it in either way?