Zabbix Proxy -> Zabbix Proxy -> Zabbix server?

[suprdave]

Hello!

I am running Zabbix in a *very* large environment for monitoring, and unfortunately even with using proxies our main Zabbix server is still getting overwhelmed with the incoming data. Is there a way of creating a "chain" of proxies to cut down on the incoming network connections to the main zabbix server? For example, like I have in the title:

Nodes -> Zabbix proxy 1 -> Zabbix proxy 2 -> Main zabbix server?


Thanks for the help!

[richlv]

nope, that's not supported. i also believe it wouldn't help in your case - although you have not mentioned how many proxies do you have. in most cases bottleneck won't be the incoming connections but the db i/o speed on the server end.

you might also want to post in the zabbix forum that's dedicated to large environments - http://www.zabbix.com/forum/forumdisplay.php?f=23

make sure to note total amount of active items, new values per second and proxies you are using right now.

[suprdave]

Thank you! We have around 100 proxies handling about 8000 total nodes. The database seems to be keeping up fine at this point. Looks like we just need to restructure the proxies to have a lower count of proxies, and each proxy handling a larger number of nodes.

Will post in the large environment forum as well for further discussion. Thanks!

[richlv]

with 100 proxies your trappers might be unable to deal with all the incoming connections. how many trappers do you have started ? which version of zabbix is that ?

[suprdave]

Hi!

We have 100 trappers started on the main zabbix server:

### Option: StartTrappers
# Number of pre-forked instances of trappers
#
# Mandatory: no
# Range: 0-255
# Default:
StartTrappers=100



.. and 1 trapper started on each proxy. Each proxy handles roughly 80 nodes. This is Zabbix 1.8.4.

[richlv]

how did you determine that the amount of incoming connections is to blame ?
if you upgrade to 1.8.5, you could monitor how busy the trappers are and thus try to figure out whether that might indeed be an issue.

[suprdave]

Hi Rich,

We believe it's a traffic issue since the database appears to be operating fine. We can't really come up with any other logical reason. However..... after increasing the trapper count to 100 recently, we actually haven't had any further pager storms or hosts showing down when they aren't.

Is this the correct method of solving this issue? Is this a scalable solution, or would the better option be to still lower the amount of proxies, and have more hosts per proxy, so that the trappers aren't receiving data from such a large amount of proxies? The amount of hosts are only going to increase, and we don't want to back ourselves into a wall.

[richlv]

i don't see anything wrong with such an approach, but i might be missing something of course.
what i would strongly suggest, upgrading to 1.8.5 and monitoring busy trapper processes. try to find an amount that can cope with the load but isn't too high above that threshold.