Ad Widget

**cheezus** · 26-01-2012, 05:08

Server specs: 4 x 2.4ghz westmere cores, 8gb memory, 4 x ocz vertex 2 ssds in raid 0.

my.cnf:
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0

[mysqld_safe]
log_slow_queries=/var/log/mysql.slow.log
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
max_connections = 1000
max_connect_errors = 10
max_allowed_packet = 16M
max_heap_table_size = 64M

sort_buffer_size = 8M
join_buffer_size = 8M
query_cache_size = 64M
query_cache_limit = 2M

tmp_table_size = 64M
key_buffer_size = 32M
read_buffer_size = 2M

read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_max_extra_sort_file_size = 10G

myisam_repair_threads = 1
myisam_recover

innodb_additional_mem_pool_size = 16M

innodb_buffer_pool_size = 3G
thread_concurrency = 4
thread_cache_size = 4
table_cache = 1024

**cheezus** · 26-01-2012, 05:09

zabbix_server.conf:

# This is a configuration file for Zabbix Server process
# To get more information about Zabbix,
# visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

# Zabbix 1.8 specific configuration parameters

CacheSize=256M
TrendCacheSize=64M
CacheUpdateFrequency=300
LogSlowQueries=5000

### Option: NodeID
# Unique NodeID in distributed setup.
# 0 - standalone server
#
# Mandatory: no
# Range: 0-999
# Default:
# NodeID=0

### Option: ListenPort
# Listen port for trapper.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# ListenPort=10051

### Option: SourceIP
# Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: LogFile
# Name of log file.
# If not set, syslog is used.
#
# Mandatory: no
# Default:
# LogFile=

LogFile=/tmp/zabbix_server.log

### Option: LogFileSize
# Maximum size of log file in MB.
# 0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
# LogFileSize=1

### Option: DebugLevel
# Specifies debug level
# 0 - no debug
# 1 - critical information
# 2 - error information
# 3 - warnings
# 4 - for debugging (produces lots of information)
#
# Mandatory: no
# Range: 0-4
# Default:
# DebugLevel=3

### Option: PidFile
# Name of PID file.
#
# Mandatory: no
# Default:
# PidFile=/tmp/zabbix_server.pid

### Option: DBHost
# Database host name.
# If set to localhost, socket is used for MySQL.
#
# Mandatory: no
# Default:
# DBHost=localhost

### Option: DBName
# Database name.
# For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored.
#
# Mandatory: yes
# Default:
# DBName=

DBName=zabbix

### Option: DBSchema
# Schema name. Used for IBM DB2.
#
# Mandatory: no
# Default:
# DBSchema=

### Option: DBUser
# Database user. Ignored for SQLite.
#
# Mandatory: no
# Default:
# DBUser=

DBUser=zabbix

### Option: DBPassword
# Database password. Ignored for SQLite.
# Comment this line if no password is used.
#
# Mandatory: no
# Default:
DBPassword=asdf7asd90f87asd9f87ad0f9as7dfas9f7asd0 9fa87sdf9a8sd7fas9df

### Option: DBSocket
# Path to MySQL socket.
#
# Mandatory: no
# Default:
# DBSocket=/tmp/mysql.sock

### Option: DBPort
# Database port when not using local socket. Ignored for SQLite.
#
# Mandatory: no
# Range: 1024-65535
# Default (for MySQL):
# DBPort=3306

############ ADVANCED PARAMETERS ################

### Option: StartPollers
# Number of pre-forked instances of pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartPollers=100

### Option: StartIPMIPollers
# Number of pre-forked instances of IPMI pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartIPMIPollers=2

### Option: StartPollersUnreachable
# Number of pre-forked instances of pollers for unreachable hosts (including IPMI).
#
# Mandatory: no
# Range: 0-1000
# Default:
StartPollersUnreachable=2

### Option: StartTrappers
# Number of pre-forked instances of trappers.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartTrappers=5

### Option: StartPingers
# Number of pre-forked instances of ICMP pingers.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartPingers=5

### Option: StartDiscoverers
# Number of pre-forked instances of discoverers.
#
# Mandatory: no
# Range: 0-250
# Default:
# StartDiscoverers=1

### Option: StartHTTPPollers
# Number of pre-forked instances of HTTP pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartHTTPPollers=1

### Option: ListenIP
# List of comma delimited IP addresses that the trapper should listen on.
# Trapper will listen on all network interfaces if this parameter is missing.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0

# ListenIP=127.0.0.1

### Option: HousekeepingFrequency
# How often Zabbix will perform housekeeping procedure (in hours).
# Housekeeping is removing unnecessary information from history, alert, and alarms tables.
#
# Mandatory: no
# Range: 1-24
# Default:
# HousekeepingFrequency=1

### Option: MaxHousekeeperDelete
# The table "housekeeper" contains "tasks" for housekeeping procedure in the format:
# [housekeeperid], [tablename], [field], [value].
# No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value])
# will be deleted per one task in one housekeeping cycle.
# SQLite3 does not use this parameter, deletes all corresponding rows without a limit.
# If set to 0 then no limit is used at all. In this case you must know what you are doing!
#
# Mandatory: no
# Range: 0-1048576
# Default:
# MaxHousekeeperDelete=500

### Option: DisableHousekeeping
# If set to 1, disables housekeeping.
#
# Mandatory: no
# Range: 0-1
# Default:
# DisableHousekeeping=0

### Option: SenderFrequency
# How often Zabbix will try to send unsent alerts (in seconds).
#
# Mandatory: no
# Range: 5-3600
# Default:
# SenderFrequency=30

### Option: CacheSize
# Size of configuration cache, in bytes.
# Shared memory size for storing hosts and items data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
# CacheSize=8M

### Option: CacheUpdateFrequency
# How often Zabbix will perform update of configuration cache, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# CacheUpdateFrequency=60

### Option: StartDBSyncers
# Number of pre-forked instances of DB Syncers
#
# Mandatory: no
# Range: 1-100
# Default:
# StartDBSyncers=4

### Option: HistoryCacheSize
# Size of history cache, in bytes.
# Shared memory size for storing history data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
# HistoryCacheSize=8M

### Option: TrendCacheSize
# Size of trend cache, in bytes.
# Shared memory size for storing trends data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
# TrendCacheSize=4M

### Option: HistoryTextCacheSize
# Size of text history cache, in bytes.
# Shared memory size for storing character, text or log history data.
#
# Mandatory: no
# Range: 128K-1G
# Default:
# HistoryTextCacheSize=16M

### Option: NodeNoEvents
# If set to '1' local events won't be sent to master node.
# This won't impact ability of this node to propagate events from its child nodes.
#
# Mandatory: no
# Range: 0-1
# Default:
# NodeNoEvents=0

### Option: NodeNoHistory
# If set to '1' local history won't be sent to master node.
# This won't impact ability of this node to propagate history from its child nodes.
#
# Mandatory: no
# Range: 0-1
# Default:
# NodeNoHistory=0

### Option: Timeout
# Specifies how long we wait for agent, SNMP device or external check (in seconds).
#
# Mandatory: no
# Range: 1-30
# Default:
# Timeout=3

### Option: TrapperTimeout
# Specifies how many seconds trapper may spend processing new data.
#
# Mandatory: no
# Range: 1-300
# Default:
# TrapperTimeout=300

### Option: UnreachablePeriod
# After how many seconds of unreachability treat a host as unavailable.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnreachablePeriod=45

### Option: UnavailableDelay
# How often host is checked for availability during the unavailability period, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnavailableDelay=60

### Option: UnreachableDelay
# How often host is checked for availability during the unreachability period, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnreachableDelay=15

### Option: AlertScriptsPath
# Location of custom alert scripts
#
# Mandatory: no
# Default:
# AlertScriptsPath=/home/zabbix/bin/

### Option: ExternalScripts
# Location of external scripts
#
# Mandatory: no
# Default:
# ExternalScripts=/etc/zabbix/externalscripts

### Option: FpingLocation
# Location of fping.
# Make sure that fping binary has root ownership and SUID flag set.
#
# Mandatory: no
# Default:
FpingLocation=/usr/sbin/fping

### Option: Fping6Location
# Location of fping6.
# Make sure that fping6 binary has root ownership and SUID flag set.
# Make empty if your fping utility is capable to process IPv6 addresses.
#
# Mandatory: no
# Default:
# Fping6Location=/usr/sbin/fping6

### Option: SSHKeyLocation
# Location of public keys for SSH checks
#
# Mandatory: no
# Default:
# SSHKeyLocation=

### Option: LogSlowQueries
# How long a database query may take before being logged (in milliseconds).
# 0 - don't log slow queries.
#
# Mandatory: no
# Range: 1-3600000
# Default:
# LogSlowQueries=0

### Option: TmpDir
# Temporary directory.
#
# Mandatory: no
# Default:
# TmpDir=/tmp

### Option: Include
# You may include individual files or all files in a directory in the configuration file.
#
# Mandatory: no
# Default:
# Include=

# Include=/etc/zabbix/zabbix_server.general.conf
# Include=/etc/zabbix/zabbix_server/

### Option: StartProxyPollers
# Number of pre-forked instances of pollers for passive proxies.
#
# Mandatory: no
# Range: 0-250
# Default:
# StartProxyPollers=1

### Option: ProxyConfigFrequency
# How often Zabbix Server sends configuration data to a Zabbix Proxy in seconds.
# This parameter is used only for proxies in the passive mode.
#
# Mandatory: no
# Range: 1-3600*24*7
# Default:
# ProxyConfigFrequency=3600

### Option: ProxyDataFrequency
# How often Zabbix Server requests history data from a Zabbix Proxy in seconds.
# This parameter is used only for proxies in the passive mode.
#
# Mandatory: no
# Range: 1-3600
# Default:
# ProxyDataFrequency=1

**dave_t** · 07-10-2013, 19:02

a few pointers to help resolve the "missing data for > 10 mins" mystery

hopefully you resolved your issue, but if not - or if anyone else is sufferring with this "hard to diagnose" issue, here is what I did.

* firstly - update your zabbix server to at minimum 1.8.10 *

There are some useful internal items that have been included in the Template_Zabbix_Server (which, funnily enough you should apply to your zabbix server)

This template comes with a really useful pre-defined graph called
"zabbix data gathering process busy %" which should give you an indication whether any of your pre-forked instances (pollers/pingers/trappers etc.) are struggling.

....for my zabbix_server.conf, I've changed the following from the defaults:

StartPollers=40
StartPollersUnreachable=10
StartPingers=10
HousekeepingFrequency=1
Timeout=30

NOTE: If you have cause to increase the number of pre-forked instances, please bear in mind your zabbix users ulimit, and and overriding system limits.
for example, my zabbix server runs on AIX, so I needed to consider the output of
# lsattr -El sys0 -a maxuproc
maxuproc 4000 Maximum number of PROCESSES allowed per user True
#

With regards to the monitored host, you should also consider the number of agents you have configured in zabbix_agentd.conf, as there may be insufficient.
In other words, you may be asking for more information from your monitored host than the agent can respond to in a timely manner.

# Number of pre-forked instances of zabbix_agentd.
# Default value is 5
# This parameter must be between 1 and 16
StartAgents=16

The other thing to bear in mind is that the zabbix agent runs at a lower priority than other processes on your monitored host, so if you are getting strange results / missing data etc., from one particular host it may be worth logging on to the host in question and running some lower level diagnostics to determine if the monitored host is overloaded.

Something else to consider is the payload on the backend zabbix database, especially around whether you are keeping too much detailed history.
Whenever the zabbix housekeeper kicks in, it has to trawl through the granular history data and (amongst lots of other things) summarise the data into trends and purge unwanted data.

If you set the housekeeper frequency too low, and have an excessive amount of data that your housekeeper needs to get through, you'll possibly cause a bottleneck which may have adverse effects on your zabbix environment.

You'll see the effect of overloading the housekeeper in the graphs that come with the Template_Zabbix_server (check the "internal busy process busy %" graph).

Finally, although this won't be the cause of the "missing data for > 10 mins" problem, it's still worth a mention....

Consider how you are backing up your database.

My current config (421 hosts, 18,117 items, 3584 triggers with the majority of items being collected at 179 second intervals or higher, and less than 10% with a history of more than 6 days) is running a full mysqldump every day at 06:00, which takes 15 minutes to generate a 8gig file (I know it seems a lot, but we're storing a lot of TEXT type data (that we want to keep)). This causes a 15 minute 'hang' which I can tolerate for now (at least while I'm planning the migration to zabbix 2.x, and mysql replication).

I'm sure that there are lots more things you can consider that relate to your environment, and they're all in the manual, but thought I'd share some of my experiences whilst investigating the "missing data for > 10 mins" mystery

Hope this helps.....

Dave

Ad Widget

There are items missing data for more than 10 minutes.

There are items missing data for more than 10 minutes.

Comment

Comment

Comment