Zabbix for large environment setups

[qix]

dont use hard-disks!! use SSD! its much faster has a very doof io-performace and its enterprise-ready!

Has anyone tried flashcache with a Zabbix DB yet?

https://github.com/facebook/flashcache

It could be a good solution to cache I/O and still use HDD's to store the historical data. I'm curious if performance will benefit from it in large scale installs.

[Colttt]

it is testet in a german magazin (admin-magazin.de) and they are said it is quiet very good perfomarnce, but if the database greate than the SSD the performace goes down! but it is much better than a HDD

http://www.admin-magazin.de/Das-Heft/2012/04/SSDs-als-Cache-fuer-Festplattenspeicher/(language)/ger-DE

[fazie]

Hello,

We've got such configuration:
DB: 24 x 2533Mhz Intel Xeon, 6xSSD OCZ Vertex 240GB raid 10, 48GB RAM
Server: 24 x 2533Mhz Intel Xeon, 2xSAS HDD, 48GB RAM
17 x Proxy: VM, 1-4 cores, 1-4GB RAM

hosts: 2200
items: 411383
triggers: 180174
nvps: 5216.4
update qps ~ 5k
select qps ~ 2k

We plan aprox twice more items on the same hardware.

[fazie]

Just for update.

Currently status:
nvps: 7366.83
hosts: 3724
items: 588556
triggers: 225739

Hardware is the same.
MySQL avg qps ~ 12k

[heaje]

I would love to know what kind of IOPs some of your large installations require. I'm in the middle of requesting the storage for a large installation at work, and having the IOPs would help tremendously to convince the storage team of what I need .

[fazie]

This is our results for 7k+ nvps (in attachment)

This huge spike was caused by my trigger, which was counting something each 60 seconds for last 24h for few houndreds of items
SSD disks are really very powerful, if you have money to buy them, don't hesitate, they can read 0.5G/s!

[heaje]

@fazie - Thanks for putting up those graphs. That helps considerably. Would you be alright with putting up a copy of your configuration files for Zabbix Server and Proxy? It'd also be great to see a copy of your MySQL my.cnf file as well. If you're not comfortable posting them to the forum, feel free to send me a private message.

[lukemacneil]

I use zabbix in a large environment.

Number of hosts (monitored/not monitored/templates) 7718
Number of items (monitored/disabled/not supported) 365383
Number of triggers (enabled/disabled)[problem/unknown/ok] 288473
Required server performance, new values per second 861.46

I use 1 zabbix server and 3 proxies. All blades in the same chassis.
All running mysql 5.5 for the backend.

I modified the agent startup script with my own logic for discovery.

This generally works very well, with the exception of the Availability Report.
That chokes to death.

[qix]

I'm curious to what discoveries you are doing.
Can you tell us more about that?

[heaje]

I'd be curious about that as well.

Also, I'd be interested in seeing the size of people's databases (GB/TB) in some of these large environments posted in this thread (specifically MySQL). Yes, I know that entirely depends on polling intervals, but getting a decent idea of size gives me a point of reference for my large environment.

[innocent]

Anyone here who has a guide, howto, implementation plan or something else for setting up a zabbix server (2.04) with multiple proxies (in my case 5). I'd like to know whats the best (or good) way to configure a setup like that.

Things I'm curious about as well is using proxies in active or passive mode and the best way for discovery of new servers monitored by proxies.

I want to create a setup that detects new servers and automatically configures the monitoring for a new found server by a proxy.

zabbix agents will probably only use passive checks and the database for the server and proxies is probably going to be postgres 9.2.

[lukemacneil]

I've disabled zabbix discovery, and instead modified the zabbix agent init script to take over that function.

At startup, the agent startup script sends a curl request up to a php script on one of my proxies. It sends the IP address and environment of the client.

The php script then returns the DNS name of that IP directly from my DNS zone files, and queries zabbix for the monitoring proxy... and sends them back down to the agent.

Then the startup script modifies the /etc/zabbix/zabbix_agentd.conf file with the appropriate hostname and proxy.

If the host does not currently exist in the zabbix database, it uses the API to create it, and assign it to a proxy.

The agent portion of this code looks like this:

  echo -n "Starting zabbix agentd "
   . /cvsrx/admin/functions/common.fun
   fDetermine_Str_Environment

   if [[ $(hostname) == 'newstore' ]]
   then
   echo "Refusing to start while hostname is newstore"
   exit 0
   fi

   IP=$(ifconfig eth0 | grep 'inet addr' | awk '{print $2}' | cut -d":" -f2)

   if [[ $ENVIRONMENT == "TEST" ]]
   then
   ENV='t'
   else
   ENV='p'
   fi

   ## Identify the active proxy and hostname for this host.
   PROXYLINE=$(curl -x "" -s -d "ip=${IP}&environment=${ENV}" http://x.x.x.x/zabbix/getproxy.php)
   if [[ $PROXYLINE == "1" ]]
   then
   echo "Unable to retrieve host information from x.x.x.x"
   exit 0
   fi

   HOST=$(echo $PROXYLINE | cut -d":" -f1)
   PROXY=$(echo $PROXYLINE | cut -d":" -f2)
   [[ -n $PROXY ]] && sed -i "s/^Server=.*/Server=${PROXY}/" /etc/zabbix/zabbix_agentd.conf
   [[ -n $HOST ]] && sed -i "s/^Hostname=.*/Hostname=${HOST}/" /etc/zabbix/zabbix_agentd.conf

   ## Start daemon with startproc(8). If this fails
   ## the return value is set appropriately by startproc.
   mkdir -p `dirname $PidFile` 2> /dev/null
   chown zabbix.zabbix `dirname $PidFile`
   chown -R zabbix /var/log/zabbix
   /sbin/startproc -p $PidFile $ZABBIX_AGENTD_BIN [ -c /etc/zabbix/zabbix_agentd.conf ]

I've found this to work well, as it allows me to adjust how the new system is configured in zabbix, while maintaining the flexibility of being able to make chain wide configuration changes by modifying code centrally. Adding/Removing a proxy, or reassigning hosts requires only modifying the proxies in the Zabbix frontend, and restarting the agents. I use the $ENVIRONMENT to determine what hostgroups/templates should be applied to the host.

I save 3 days history and 3 days trend data, unless there is a specific need to monitor a production issue over time... as the intention of our monitoring solution is alerting, not reporting. Some items are monitored in 5 minute increments, some, like md5sums are in 12 hour increments.

My MySQL database is currently 36G.

[innocent]

Thanks,

Nice solution i will have a look at it.

Maybe it can be combined with puppet as well.

[prakhar]

Need suggestions..

Hi,

I am trying to load test zabbix for my deployment needs. I am running zabbix2.0.3 with postgresql 8.4 on a single server whose specifications are as follows:
2 Blades (2x4 CPU, 2GHZ Sandybridge, 96GB 1336 RAM, 4TB storage (Raid0), 4x10Gig Ethernet)

I will be using zabbix to monitor 5000 items per host with around 100 host.

After tuning zabbix and postgres to some extent i have been able to monitor 75000 items (update interval: 60) with passive agents, the current NVPS being ~1250.

Though the avg busy poller = 55% and avg histrory sync = 45% there are lot of items in queue. with the avg being = 50K items. As i am increasing the number of items being monitored the queue avg also increases..

What could be a practicle max for NVPS with such configuration and passive agents. And please give suggestions on how can i achieve the target monitoring needs.

thanks,
Prakhar..

[qix]

@ lukemacneil : Thanks for the info! Sounds like a great way to do proxy management.
You mentioned something about the API creating the host if it doesn't exist yet.
I assume that is handled by the PHP script you've mentioned?