Configuration for Performance

[ahanson]

I have a 1.4.1 installation of Zabbix up and running. It runs alright, but it doesn't poll as quickly as I want it to. I have 86 hosts actively monitored with about 76 items on each. That's just an average, some have more some have less. There are quite a few things I poll every 30 seconds, but the system seems to not poll but once every 5 minutes or so. This I got from the "Last 500 Values" screen.

The zabbix server, apache, and mysql are all running on one machine. It's a dual-xeon 3.06GHz w/ 2GB of RAM machine.

My question is this: in the zabbix server configuration, should I be starting more instances of pollers or something to make sure that as close as possible, every 30 seconds a value is taken? All of my checks are divided between the zabbix agent and snmpv1 (with a few simple check pings).

[Alexei]

ZABBIX 1.4.x performance secrets:

- use ZABBIX agent active checks whenever possible
- tune ZABBIX database to use most of your memory resources, don't use default settings
- tune number of pollers and trappers

The 76 items per each host means 6536 parameters for monitoring, or 220 checks per second on average if 30 sec refresh rate is used. I am sure your system is capable of doing at least twice more.

[ahanson]

With only the 2GB of RAM, I'm going to see about throwing more in, just because it's easy to do and there's no reason not to.

That being said, I have changed the amount of pollers to 30, and MySQL InnoDB Buffer Pool Size is set to consume 1GB of RAM, as this machine isn't dedicated to MySQL. I expect the 50% ratio to continue with the additional RAM. Everything else in MySQL is kept as a default. Should I also be tweaking any other InnoDB settings?

[ahanson]

In regard to the active checking, I did this for a bit with log file monitoring, but any zabbix agent check can be an active check? Does this require any changes to the client's zabbix configuration file, or do I just change it on the server?

Thanks again for all of your help on this.

[bbrendon]

In regard to the active checking, I did this for a bit with log file monitoring, but any zabbix agent check can be an active check? Does this require any changes to the client's zabbix configuration file, or do I just change it on the server?

I'll chime in real quick.
Not anything can be an active check. Agent items can be active. Simple, internal, etc can not. Yes, you have to put a hostname and server in the agent configuration file. I think that should be it.

[fmtaylor2]

Performance FYI

These are stats from my query engine (zabbix_server). I replicate this database to another (identical) server to run the web fronend (circular replication). It is a 4 processor dual core box with 16GB of RAM, and raid 10 on the database drives. If I knew how to tune it better I would.....


Number of hosts (monitored/not monitored/templates/deleted) 238(195/31/12/0)
Number of items (monitored/disabled/not supported)[trapper] 25353(14057/5936/5360)[1]
Number of triggers (enabled/disabled)[true/unknown/false] 13991(7312/6679)[40/61/7211]


top - 08:50:38 up 43 days, 22:05, 2 users, load average: 7.03, 7.08, 8.10
Tasks: 178 total, 3 running, 175 sleeping, 0 stopped, 0 zombie
Cpu(s): 80.5%us, 3.7%sy, 0.8%ni, 14.6%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 16507116k total, 16467736k used, 39380k free, 172624k buffers
Swap: 2031608k total, 176k used, 2031432k free, 9619044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30359 mysql 15 0 6052m 5.4g 4740 S 661 34.2 340838:26 mysqld



[root@boris ~]# ./tuning-primer.sh

-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -

MySQL Version 5.0.22-log x86_64

Uptime = 36 days 17 hrs 40 min 19 sec
Avg. qps = 863
Total Questions = 2741256566
Threads Connected = 27

Server has been running for over 48hrs.
It should be safe to follow these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.0/...variables.html

SLOW QUERIES
Current long_query_time = 10 sec.
You have 1014900 out of 2741256742 that take longer than 10 sec. to complete
The slow query log is NOT enabled.
Your long_query_time may be too high, I typically set this under 5 sec.

WORKER THREADS
Current thread_cache_size = 32
Current threads_cached = 6
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 100
Current threads_connected = 27
Historic max_used_connections = 38
The number of used connections is 38% of the configured maximum.
Your max_connections variable seems to be fine.

MEMORY USAGE
Max Memory Ever Allocated : 7 G
Configured Max Per-thread Buffers : 6 G
Configured Max Global Buffers : 5 G
Configured Max Memory Limit : 11 G
Total System Memory : 17 G
Max memory limit seem to be within acceptable norms

KEY BUFFER
Current MyISAM index space = 65 K
Current key_buffer_size = 7 M
Key cache miss rate is 1 : 0
Key buffer fill ratio = 2.00 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere

QUERY CACHE
Query cache is enabled
Current query_cache_size = 1.00 G
Current query_cache_used = 622 M
Current query_cach_limit = 1 M
Current Query cache fill ratio = 60.80 %
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = 64 M
Current record/read_rnd_buffer_size = 2 M
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 132.00 K
You have had 1 queries where a join could not use an index properly
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
If you are unable to optimize your queries you may want to increase your
join_buffer_size to accommodate larger joins in one pass.

Note! This script will still suggest raising the join_buffer_size when
ANY joins not using indexes are found.

OPEN FILES LIMIT
Current open_files_limit = 2510 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_cache value = 1200 tables
You have a total of 83 tables
You have 273 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current max_heap_table_size = 3.99 G
Current tmp_table_size = 1.00 G
Of 597440787 temp tables, 49% were created on disk
Perhaps you should increase your tmp_table_size and/or max_heap_table_size
to reduce the number of disk-based temporary tables
Note! BLOB and TEXT columns are not allow in memory tables.
If you are using these columns raising these values might not impact your
ratio of on disk temp tables.

TABLE SCANS
Current read_buffer_size = 128 K
Current table scan ratio = 82 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 133704927
Your table locking seems to be fine

[ahanson]

So when I upped the Poller count to 35, I'm now getting data just about every 30 seconds. For the first time I can see graphs all the way down to 1hr. I just migrated over all my zabbix agents checks to active checks, which is by far the bulk of my checking. Can't tell you how much easier this makes my life, and how much more I appreciate Zabbix!

[Alexei]

Be prepared to downgrade your quad CPUs and gigs of memory after upgrade to 1.6...

[ahanson]

Performance increased that much?? We are looking at doubling our infrastructure withing the next 6 months, so maybe I won't be, but that's certainly nice to know.

[Alexei]

Performance increased that much?? We are looking at doubling our infrastructure withing the next 6 months, so maybe I won't be, but that's certainly nice to know.

The development is not finished yet, so I have no exact numbers. I would expect ZABBIX 1.6 to be at least twice as fast comparing to 1.4.x.