3000+ Server monitoring planning

[rangeshgupta]

Hi all,

We have to monitor the 3000+ server with the help of zabbix.

So we are planning for distributed monitoring.

There are two type of distributed monitoring in zabbix:

1. Proxy
2. Node

I had understand the concept.

But we also want an expert advise for setting up the environment.

Below are some information you may required for suggestion.

1. approx 40 items per second for 10 hosts
2. All items are based on SNMP
3. 10-15 triggers per host

My zabbix server configuration:-

Memory: 16GB
Storage: 900 GB
CPU : 8 core * Intel(R) Xeon(R) @ 2.33GHz


Thanks

[walterheck]

If it's all under your supervision, you should probably go with proxies, as nodes have a bit of overhead (every node is it's own zabbix server).

That said, if you are using proxies, each proxy needs a database to store the data it gathers as well. That means that each node needs a reasonably powerful server as well, depending on how much data it proxies.

[QuakENT]

3000+ host on that hw, with so many items? hm, I guess, that you will need 2+ DB-mysql cluster for start, with lots of ram and 15000rpm SAS. I thing that zabbix server daemon could by handle with 8xintel core on such enviroment but not with db on the same server. Dont even try

[NOB]

If it's all under your supervision, you should probably go with proxies, as nodes have a bit of overhead (every node is it's own zabbix server).

That said, if you are using proxies, each proxy needs a database to store the data it gathers as well. That means that each node needs a reasonably powerful server as well, depending on how much data it proxies.

Hi

Be very careful with that number of items and hosts !

It might work with a single server, but we have a LOT of problems with a
medium amount of zabbix_sender processes sending data.
Sooner or later the server won't accept any items
send by zabbix_sender (timeout in zabbix_sender is 60 seconds).
This can be "fixed" by stopping and starting the zabbix_server
processes for a few hours. Then it'll start again.

The same happens with a small amount of items monitored by
a Zabbix proxy.
No surprise, it's the same code, regardless whether an agent sends
data for active items, a proxy sends data for its agents or
data send by zabbix_sender.

We use 1.6.8 now but the suspicious code is the same up to 1.8.1
and the changes in 1.8.2 are not that massive to let us migrate
to that version.
It is not in the part where massive changes were done for 1.8.x, i.e.
the reading of every item and trigger for every received data.
This is now in a RAM cache. The problem is in the sync'ing of the data
between nodes. As soon as you use nodes, this code is executed.

So, as walterheck states, if you just use Proxies it might work.
However, Proxies don't cascade, so we didn't have a choice
and you might not have, too.

Another weak point is the method of replicating the data from
one node to the other. It can drive you crazy, esp.
if you are changing a lot of things in, say, a template including
deletion and later new addition of the same item, ...
Suddenly all items are gone or just partly copied to the slave node, etc.

We use 15k SAS disks (HP Hardware, enough RAM for the small number of hosts: 600 and items).
and use MySQL as the DB. CPU load is negligible: less than 1 most of the time.

Good luck

Norbert.

[tchjts1]

This thread may be of interest to you. We monitor a little over 1,000 hosts with Zabbix 1.8.2 : http://www.zabbix.com/forum/showthread.php?t=17762

You can see some stats and graphs from our Zabbix app and proxy server. We have 13 proxies located globally as we have monitored servers in the US, Hong Kong, Turkey, Japan, Korea, India, and UK