How to handle more than 10 hosts?

[tchjts1]

I understand that out of the box, Zabbix server is configured to handle 10 hosts. What values need to be tweaked in zabbix_server.conf and MySql conf to handle hundreds of hosts? Say for instance, 500 hosts.

Thanks

[xs-]

who gave you that idea?
zabbix isn't limited or 'pre-configured' for anything.

If you want to monitor 500+ hosts, you do want a separate database server and, depending on how many ppl will use it, a separate webserver for the frontend part.
(its best to keep the frontend off the database server, you can place the zabbix server on either)

You also want to up the values of the specific worker threads, depending on your needs (i.e. active agents, snmp polls, discovery, etc).
Oh and unless you are an over-experienced oracle or postgresql dba, go for mysql5 as the database.

[tchjts1]

who gave you that idea?
zabbix isn't limited or 'pre-configured' for anything.

Page 43 of the Zabbix manual gave me that idea. The excerpt here -

Step
10
Configure /etc/zabbix/zabbix_server.conf

For small installations (up to ten monitored hosts), default parameters are sufficient. However, you should change default parameters to maximize performance from ZABBIX. See section [Performance tuning] for more details.
You may take misc/conf/zabbix_server.conf as example.

[xs-]

Hmm, ok good point

Well, the above applies. Ofcourse you want some sturdy hardware to go with that.
You also might want to up the item intervals a but, default check range from 5 to 30 secs, which might be a bit often. I personally use 600-3600 interval depending on the item.

[tchjts1]

I do appreciate your replies. The manual is quite skimpy on specific examples for a lot of items.

[nelsonab]

The two things you will want to tweak are the database and the number of server threads for things like pollers and trappers and so forth. You'll probably want to start tuning MySQL when you get to about 15-20 hosts. It's not a requirement, but giving it more RAM and tweaking the Innodb options does help. As you grow from there, you'll want to move to striped RAID setup (ie RAID 5) and you'll want to move MySQL to another server.

Also increasing the number of pollers is common. That however is more dependent of the number of items and how often they are checked. Here queue length *can* be helpful. If you're chugging along and you start to see you queue length increase over time it's likely you don't have enough poller threads to check the items. Increasing this one early on does not hurt as the server does not take that many resources. However increasing this early on will likely not yield a performance improvement.

[tchjts1]

Thanks Nelson.

BTW, I see you are also in Ca. I am in the bay area.

Jeff

[nelsonab]

Nice, I posted in the offtopic about a meetup in the Bay Area, add to the thread, when one of us gets motivated enough we'll know who to invite. :-)

[cstackpole]

Just to emphasis previous points, the problem I have had everytime when I scaled up past ~20 hosts has been within MySQL not Zabbix. Running MySQL/frontend on a seperate computer is a good idea if you have the ability.

If you are new to MySQL and still learning about it then...

I highly recommend you to look into mysql tuning.

http://hackmysql.com/mysqlreport

http://www.day32.com/MySQL/

Those two scripts can be difficult to comprehend but please trust me when I say that once you understand what they are telling you they will dramatically help increase your performance. Both scripts are well documented.

When dealing with MySQL, the one bit of advice that I plead and beg you to take away is: NEVER make more then one change to MySQL at a time and ALWAYS document what change you made (previous and current value).

It will be slow going, but one change can go horribly wrong or blissfully right. If you made 5 changes it will be difficult to find out which one helped and which one hindered. Especially if one change helped a bit and a second change hurt a lot.

You can avoid many headaches by using those scripts and by making only one change at a time.

As for scaling Zabbix, I have made changes like increase trappers. For the most part, however, I have never had to make a dramatic change to "scale" Zabbix.

Have fun!

[fableman]

I run everything on same server just added a quad(4) core CPU.

Number of hosts (monitored/not monitored/templates/deleted) 160(140/2/18/0)
Number of items (monitored/disabled/not supported)[trapper] 1196(948/53/195)[0]
Number of triggers (enabled/disabled)[true/unknown/false] 455(448/7)[1/48/399]


And I even have my own scripts for pings and packetloss, radio unit monitoring and more. And the server keep up really good without any problems.


mysqld is around: CPU: 25% and Memory: 1.8%
zabbix_server is around CPU: 1 % and Memory: 0.2 %



top - 11:33:55 up 41 days, 23:29, 1 user, load average: 0.31, 0.33, 0.34
Tasks: 191 total, 1 running, 190 sleeping, 0 stopped, 0 zombie
Cpu0 : 11.7%us, 1.0%sy, 0.0%ni, 82.6%id, 4.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 5.6%us, 1.7%sy, 0.3%ni, 91.7%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 1.0%us, 1.0%sy, 0.7%ni, 97.0%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st

Mem: 2065968k total, 2007444k used, 58524k free, 182848k buffers
Swap: 2031608k total, 80k used, 2031528k free, 1473996k cached



from my zabbix_server.conf

# Number of pre-forked instances of pollers for unreachable hosts
# Default value is 1
# This parameter must be between 0 and 255
#StartPollersUnreachable=1

# Number of pre-forked instances of trappers
# Default value is 5
# This parameter must be between 0 and 255
#StartTrappers=5

# Number of pre-forked instances of ICMP pingers
# Default value is 1
# This parameter must be between 0 and 255
#StartPingers=1

# Number of pre-forked instances of discoverers
# Default value is 1
# This parameter must be between 0 and 255
#StartDiscoverers=1

# Number of pre-forked instances of HTTP pollers
# Default value is 1
# This parameter must be between 0 and 255
#StartHTTPPollers=1

# Listen port for trapper. Default port number is 10051. This parameter
# must be between 1024 and 32767

#ListenPort=10051

# Listen interface for trapper. Trapper will listen all network interfaces
# if this parameter is missing.

#ListenIP=127.0.0.1

# How often ZABBIX will perform housekeeping procedure
# (in hours)
# Default value is 1 hour
# Housekeeping is removing unnecessary information from
# tables history, alert, and alarms
# This parameter must be between 1 and 24

#HousekeepingFrequency=1

# How often ZABBIX will try to send unsent alerts
# (in seconds)
# Default value is 30 seconds
SenderFrequency=30

# Uncomment this line to disable housekeeping procedure
#DisableHousekeeping=1

# Specifies debug level
# 0 - debug is not created
# 1 - critical information
# 2 - error information
# 3 - warnings (default)
# 4 - for debugging (produces lots of information)

DebugLevel=3

# Specifies how long we wait for agent response (in sec)
# Must be between 1 and 30
Timeout=15

# Specifies how many seconds trapper may spend processing new data
# Must be between 1 and 30
#TrapperTimeout=5

# After how many seconds of unreachability treat a host as unavailable
UnreachablePeriod=45

# How ofter check host for availability during the unreachability period
UnavailableDelay=15

# How ofter check host for availability during the unavailability period
UnavailableDelay=60

# Name of PID file

PidFile=/var/run/zabbix/zabbix.pid

# Name of log file
# If not set, syslog is used

LogFile=/var/log/zabbix/zabbix_server.log

# Maximum size of log file in MB. Set to 0 to disable automatic log rotation.
#LogFileSize=1

# Location for custom alert scripts
#AlertScriptsPath=/var/lib/zabbix/
AlertScriptsPath=/etc/zabbix/externalscripts/
# Location of external scripts
ExternalScripts=/etc/zabbix/externalscripts

# Location of 'fping. Default is /usr/sbin/fping
# Make sure that fping binary has root permissions and SUID flag set
FpingLocation=/usr/sbin/fping

# Frequency of ICMP pings (item keys 'icmpping' and 'icmppingsec'). Defauls is 60 seconds.
#PingerFrequency=60

# Database host name
# Default is localhost

#DBHost=localhost

# Database name
# SQLite3 note: path to database file must be provided. DBUser and DBPassword are ignored.
DBName=zabbix

# Database user

DBUser=zabbix

# Database password
# Comment this line if no password used

DBPassword=2349cn349683n968qw3nx9qw348m9x34863x8m3 890y6048y6

# Connect to MySQL using Unix socket?

#DBSocket=/var/lib/mysql/mysql.sock