Ad Widget

**xs-** · 20-06-2008, 12:31

Some (imho) good hints and practices:
(disclaimer, these are generic points, they don't have to apply to your situation)

Zabbix

Items
Be realistic about your items
- Poll interval. Default zabbix templates use poll intervals of 5s-60s. IMHO this is waay too often. Use 300s as a minimum, and larger numbers for items which do not change much (like disk space usage i.e. 15min).
- History save time. This is 'how long item data is stored using the poll interval'. If your poll interval is 1min and history save time is 1 year, you will have 60x24x365 records for that item alone. Is this really neccesairy?.
  Personally i use 3 days for everything.
- Trend save time. This works 'somewhat' like RRD. It takes history values and stores it with a higher (but fixed ) interval.
  Personally i use 0 days (none) for boolean values, 0 days (none) for log and snmp trap messages (we have a loghost for that) and 100 days for 'default' numeric-non-boolean values.
Triggers
Each trigger change generates an event record. This table can also grow significantly if you have a lot of flapping triggers or useless triggers.
Be realistic as to when you want a trigger fired.
IMHO its good practice to not just look at the triggers themselves but also look at the 'events'. This way you will be confronted which the amount of status changes per trigger. (only downside of this is that the event view on the webinterface is really really slow due to the lack of indexes i think)
Database housekeeping
Zabbix itself does housekeeping, but it does not sanitize the database.
This means that all orphaned data is left in the db.
- If you add/remove/add/remove distributed nodes, all data will be left in the database.
- If you use DM for syncing configuration (change slave node configs via the master node webinterface) data can be orphaned on child and master node!
- If you modify the database externally (i do via cmdb integration scripts) data can be orphaned.
There are several threads on the forum with hints and tips on cleaning this up. It helps, alot!
Clearing out old-node data is a different ball game. I've written some scripts for this, but i need to make them more generic before i post them on the forum/wiki, but this is necessary when you remove a DM node!

MySQL

Separation of Database and zabbix services
If you have a large environment, it pays off to run your database on a different server than your zabbix service and webinterface.
I have 2 servers for our DM master node. 1 for the zabbix_server + webinterface, 1 running mysql.
If your environment is large, do NOT run either on a VM!
Mysql tuning
Use mysql tuning scripts to optimize your database settings, like memory usage, buffer sizes, etc. For example:
http://rackerhacker.com/mysqltuner/ <-- I only use this one, its good.

404 Not Found

http://www.day32.com/MySQL/

https://hackmysql.com/archive/

http://hackmysql.com/mysqlreport

They will all bitch about query optimization, allas you need to ignore those
Seperate innodb file per table
This option in the my.cf is the only way you are able to use the mysqlcheck -o (optimize / shrink) functionality.
If you use the default InnoDB setting (1 file) you are not able to clear out empty record space like PostgreSQL does in its housekeeping. There is a thread on the forum about this.
Another option with this is that you could move/link the history/trends table file to a different set of disk spindles they don't interfere with each other when a full table scan is needed (often if it's done by the webinterface).
Dump/restore db
In relation to the previous point, if you are using the 'one-file-for-all-dbs-tables' default setting for InnoDB, you can only clear out all empty records by dumping all databases, remove the innodb data files and restore all databases in a fresh mysql data instance.
You could try this during some maintenance period and see how many gig's this will save, i think lots
Other database system?
You could try a different database.
In my personal experience PostgreSQL performs the same at best with a lot more maintenance requirement in the long term. I have switched mysql->posgresql->mysql and will stay with mysql5. I've had postgresql guru's look at the configs and systems, no go. The main issue is table indexes for the history, trends and events tables. The only switch i am willing to explore is Oracle with the help of a team of Oracle DBA's and custom database setups to tacke the index issue.

- This is what i could think of in 5 min, if i think of more options or tips will will try to add it to this post.
- Yes the post is probably full of typo's, deal with it

- This post is based on 1.4.x, but generally usable for all versions

**pierre-hoffmann** · 20-06-2008, 16:21

Hi,

Great thanks for this Post

: lots of good idea.

Related Posts:

Some Question:
-> Why you say no VMWare for mysql database ?
-> What type records do manualy clean in database ?? (i've seen sometimes message, in web interface, like 'error : host xxx doesn't exist')
-> I've try mySqlTunner ... Great : but it's not very easy to find good tunning of my.cnf

Code:

./mysqltuner.pl
 >>  MySQLTuner 0.9.0 - Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at [url]http://mysqltuner.com/[/url]
 >>  Run with '--help' for additional options and output filtering
[[COLOR="Red"]!![/COLOR]] Successfully authenticated with no password - SECURITY RISK!

-------- General Statistics --------------------------------------------------
[--] Unable to check for the latest MySQLTuner version
[[COLOR="SeaGreen"]OK[/COLOR]] Currently running supported MySQL version 5.0.26-log
[[COLOR="SeaGreen"]OK[/COLOR]] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 2K (Tables: 8)
[--] Data in InnoDB tables: 16G (Tables: 66)

-------- Performance Metrics -------------------------------------------------
[--] Up for: 6h 13m 32s (26M q [1K qps], 11K conn, TX: 407M, RX: 3B)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 18.2M per thread and 2.3G global
[COLOR="Red"][!!] Allocating > 2GB RAM on 32-bit systems can cause system instability
[!!] Maximum possible memory usage: 4.1G (206% of installed RAM)
[/COLOR][[COLOR="SeaGreen"]OK[/COLOR]] Slow queries: 0% (5/26M)
[[COLOR="SeaGreen"]OK[/COLOR]] Highest usage of available connections: 28% (29/100)
[[COLOR="SeaGreen"]OK[/COLOR]] Key buffer size / total MyISAM indexes: 500.0M/74.0K
[COLOR="Red"][!!] Key buffer hit rate: -1224.5%
[!!] Query cache efficiency: 15.9%[/COLOR]
[[COLOR="SeaGreen"]OK[/COLOR]] Query cache prunes per day: 0
[[COLOR="SeaGreen"]OK[/COLOR]] Sorts requiring temporary tables: 0%
[COLOR="Red"][!!] Joins performed without indexes: 3914
[!!] Temporary tables created on disk: 48%[/COLOR]
[[COLOR="SeaGreen"]OK[/COLOR]] Thread cache hit rate: 99%
[[COLOR="SeaGreen"]OK[/COLOR]] Table cache hit rate: 97%
[[COLOR="SeaGreen"]OK[/COLOR]] Open file limit used: 2%
[[COLOR="SeaGreen"]OK[/COLOR]] Table locks acquired immediately: 100%
[[COLOR="Red"]!![/COLOR]] InnoDB data size / buffer pool: 17.0G/1.5G

-------- Recommendations -----------------------------------------------------
General recommendations:
    MySQL started within last 24 hours - recommendations may be inaccurate
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
  *** MySQL's maximum memory usage exceeds your installed memory ***
  *** Add more RAM before increasing any MySQL buffer variables  ***
    query_cache_limit (> 16M, or use smaller result sets)
    join_buffer_size (> 2.0M, or always use indexes with joins)
    tmp_table_size (> 100M)
    max_heap_table_size (> 99M)
    innodb_buffer_pool_size (>= 16G)

-> I'm affraid by

Code:

[[COLOR="Red"]!![/COLOR]] Key buffer hit rate: -1224.5%

But don't know why ...
-> i'm increase query_cache_limit, join_buffer_size, tmp_table_size and max_heap_table_size but it wan't more and more memory ... i've only 2 Gb ....
-> And "innodb_buffer_pool_size (>= 16G)" seem to me very strange ... RAM = database size ??!!!

I'm cleaning my items retention time ...

Regards,
Pierre.

**xs-** · 23-06-2008, 13:54

Originally posted by pierre-hoffmann

-> link to another good post : http://www.zabbix.com/forum/showpost...1&postcount=27

I dont agree with the innodb_flush_log_at_trx_commit=0. While this does give a speed increase if you have a problem with disk io operations, it also corrupts your database on a crash/power outage.

Originally posted by pierre-hoffmann

-> Why you say no VMWare for mysql database ?

Migrate your mysql service to a physical server and find out

Mostly because of disk io operations. Besides that, processor time.
Try it, we've been able to run a zabbix/mysql server on 1 vm up until 400 hosts. At that time everything (including other VM's on the same server) started to grind to a halt.

Originally posted by pierre-hoffmann

-> What type records do manualy clean in database ?? (i've seen sometimes message, in web interface, like 'error : host xxx doesn't exist')

I dont know in which context to place your error, but an example of what i mean would be:

We’ll be back soon!

http://www.zabbix.com/forum/showthread.php?t=9781

You would effectively run the following (USE AT OWN RISK, it helped me, but i havent tested this on multiple scenarios to be sure this wont break stuff. Backup your database first!):

Originally posted by USE AT YOUR OWN RISK

DELETE history_str.* FROM history_str LEFT OUTER JOIN items ON history_str.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_uint.* FROM history_uint LEFT OUTER JOIN items ON history_uint.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_str_sync.* FROM history_str_sync LEFT OUTER JOIN items ON history_str_sync.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_sync.* FROM history_sync LEFT OUTER JOIN items ON history_sync.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_log.* FROM history_log LEFT OUTER JOIN items ON history_log.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history.* FROM history LEFT OUTER JOIN items ON history.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE trends.* FROM trends LEFT OUTER JOIN items ON trends.itemid = items.itemid WHERE items.itemid IS NULL;

But this is only part of it (specially with DM), because there is no use of foreign keys, a lot of data can be orphaned (which might be the case in your situation). I'm hoping the devs can provide a tool or method to check for this in a better way.

Originally posted by pierre-hoffmann

-> I've try mySqlTunner ... Great : but it's not very easy to find good tunning of my.cnf

You need to run mysql for at least 2 days straigt in order to get decent values from mysqltuner. (thus the -12k values).
Also, the inno_db_cache thiny: use 50%-75% of your total mem for this.

This is mostly trial and error.
Don't enlarge the values too fast, always document every step you take so you can roll back.

Below are the tuned values i'm currently using (as partly suggested by mysqltuner). This is for a 4G mem server.

innodb_file_per_table
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 8
max_connections = 300

table_cache = 128
tmp_table_size = 256M
max_heap_table_size = 256M
join_buffer_size = 5M

innodb_buffer_pool_size = 2G

# This is too high, need to lower next reboot
query_cache_limit = 256M
query_cache_size = 512M

Hope this helps

---- Edit ----
As JoelG indicates, decent knowledge about mysql and the settings you are changing is kinda mandatory. Dig into the docs, it pays off!

**JoelG** · 23-06-2008, 21:14

Caution

Make these changes one at time, as a standard practice. I made a change, mysql did not like that the log file info was different, and gave me a message that the .frm files were invalid. I recommend caution when changing anything dealing with a log file.

**pierre-hoffmann** · 25-06-2008, 09:48

Hi,

I'm working on Zabbix Mysql database:

-> Question about OPTIMIZE TABLE:

It exist any solution to avoid table fragmentation ??
(i've make an optimize table 1 week ago and i've lost 120 Mo on a 190 Mo table !!!!)

before:

Code:

root@lxsyst05 / >mysql zabbix
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 35522 to server version: 5.0.26-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> optimize table history_str;
+--------------------+----------+----------+----------+
| Table              | Op       | Msg_type | Msg_text |
+--------------------+----------+----------+----------+
| zabbix.history_str | optimize | status   | OK       |
+--------------------+----------+----------+----------+
1 row in set (1 min 56.80 sec)

mysql>

after:

-> Mysql memory parameters : not very better ...

I'm slowly change some paramters:

reduce innodb_buffer_pool_size to 1 Go (i've 2 Go of RAM)
increase query_cache_limit to 128 Mo
reduce key_buffer to 16 M (only for MyISAM tables ???) (no change on "Key buffer hit rate")

Any ideas ?? I need to disable "innodb_flush_log_at_trx_commit" ??

(http://www.mysqlperformanceblog.com/...-memory-usage/)

Code:

./mysqltuner.pl
 >>  MySQLTuner 0.9.0 - Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at [url]http://mysqltuner.com/[/url]
 >>  Run with '--help' for additional options and output filtering

-------- General Statistics --------------------------------------------------
[--] Unable to check for the latest MySQLTuner version
[[COLOR="SeaGreen"]OK[/COLOR]] Currently running supported MySQL version 5.0.26-log
[[COLOR="SeaGreen"]OK[/COLOR]] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 2K (Tables: 8)
[--] Data in InnoDB tables: 17G (Tables: 66)

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1d 0h 18m 4s (103M q [1K qps], 36K conn, TX: 1B, RX: 1B)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 18.1M per thread and 1.4G global
[[COLOR="Red"]!!] Allocating > 2GB RAM on 32-bit systems can cause system instability
[!!] Maximum possible memory usage: 3.2G (159% of installed RAM)[/COLOR]
[[COLOR="SeaGreen"]OK[/COLOR]] Slow queries: 0% (16/103M)
[[COLOR="SeaGreen"]OK[/COLOR]] Highest usage of available connections: 27% (27/100)
[[COLOR="SeaGreen"]OK[/COLOR]] Key buffer size / total MyISAM indexes: 16.0M/74.0K
[COLOR="Red"][!!] Key buffer hit rate: -1507.1%
[!!] Query cache efficiency: 16.0%[/COLOR]
[[COLOR="SeaGreen"]OK[/COLOR]] Query cache prunes per day: 0
[[COLOR="SeaGreen"]OK[/COLOR]] Sorts requiring temporary tables: 0%
[COLOR="Red"][!!] Joins performed without indexes: 13946
[!!] Temporary tables created on disk: 47%[/COLOR]
[[COLOR="SeaGreen"]OK[/COLOR]] Thread cache hit rate: 99%
[[COLOR="SeaGreen"]OK[/COLOR]] Table cache hit rate: 90%
[[COLOR="SeaGreen"]OK[/COLOR]] Open file limit used: 3%
[[COLOR="SeaGreen"]OK[/COLOR]] Table locks acquired immediately: 100%
[[COLOR="Red"]!![/COLOR]] InnoDB data size / buffer pool: 17.6G/1.0G

-------- Recommendations -----------------------------------------------------
General recommendations:
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
  *** MySQL's maximum memory usage exceeds your installed memory ***
  *** Add more RAM before increasing any MySQL buffer variables  ***
    query_cache_limit (> 128M, or use smaller result sets)
    join_buffer_size (> 2.0M, or always use indexes with joins)
    tmp_table_size (> 128M)
    max_heap_table_size (> 127M)
    innodb_buffer_pool_size (>= 17G)

My MySql Config:

Code:

key_buffer              = 16M    # 128Mo before
max_allowed_packet      = 16M    # 1Mo 06/06/2008
table_cache             = 512
sort_buffer_size        = 6M     # 2Mo before
read_buffer_size        = 2M
join_buffer_size        = 2M     # 128 Ko 20/06/2008
read_rnd_buffer_size    = 8M                        [URL="http://www.mysqlperformanceblog.com/2007/07/24/what-exactly-is-read_rnd_buffer_size/"][read_rnd_buffer_size][/URL] 
thread_cache_size       = 8
thread_stack            = 64K
query_cache_size        = 256M   # 32 Mo 06/06/2008
query_cache_limit       = 128M   # 16 Mo 24/06/2008
thread_concurrency      = 8      # 2 CPU
tmp_table_size          = 128M   # 34 Mo  [URL="http://www.mysqlperformanceblog.com/2007/01/19/tmp_table_size-and-max_heap_table_size/"][tmp_table_size][/URL]
max_heap_table_size     = 128M   # 16Mo 20/06/2008 

innodb_flush_log_at_trx_commit = 0 # Delayed flush on commits
innodb_buffer_pool_size = 1G     # 70-80% of memory.

# -----------------------------------------------------------------
# Configuration des log binaire = log de transaction ou archive-log
# -----------------------------------------------------------------
log-bin                 = /archiv/mysql/mysql-bin
log-bin-index           = /archiv/mysql/mysql-bin.index
max-binlog-size         = 100M
# -----------------------------------------------------------------
# Configuration des redo-logs
# -----------------------------------------------------------------
innodb_log_file_size            = 50M # 25% de buffer_pool_size
innodb_log_files_in_group       = 3
innodb_log_group_home_dir       = /dbfs/mysql/logfiles
# -----------------------------------------------------------------

Regards,
Pierre.

**xs-** · 25-06-2008, 10:06

The fragmentation you are talking about is actually dead tuples (as postgresql calls it, dunno what term mysql uses). Basically these are deleted records or updates records which takes up less space than the original one. These 'empty spaces' in a data file aren't automagically removed, that's what optimize tables is for in mysql (and only 'really' works if you have the innodb_file_per_table option).

*** MySQL's maximum memory usage exceeds your installed memory ***
*** Add more RAM before increasing any MySQL buffer variables ***

You might need more memory or clean up the contents of your database. You currently have more memory usage configured for mysql than actually available in your system.

**pierre-hoffmann** · 25-06-2008, 15:20

Originally posted by xs-

TYou might need more memory or clean up the contents of your database. You currently have more memory usage configured for mysql than actually available in your system.

Don't understand why it say "Maximum possible memory usage: 3.2G (159% of installed RAM)"
innodb_buffer_pool_size is 1Gb and other buffer was 256 Mo ou 128 Mo so where is this 3.2 Gb ???

I'm trying to increase "join_buffer_size" from 2M to 15M, "Maximum possible memory usage" increase to 4 Go and when i reduce it 10M "Maximum possible memory usage" decrease to 3.2 Go ????????

What's wrong ???

Regards,
Pierre.

**xs-** · 25-06-2008, 16:13

the innodb_buffer thingy is global for your database.
Most other buffers and cache sizes are per connection (or at least have a multiplier for something).

How is your swap usage?

**pierre-hoffmann** · 25-06-2008, 16:31

Originally posted by xs-

How is your swap usage?

Code:

Mem:   2076300k total,  2019768k used,    56532k free
Swap:  4200988k total,    [COLOR="Red"]16752k used[/COLOR],  4184236k free

Swap is unused .... :-)

So, all this parameters is per session/connexion ?

read_buffer_size
read_rnd_buffer_size
sort_buffer_size
thread_stack
join_buffer_size

Regards,
Pierre.

**pierre-hoffmann** · 26-06-2008, 10:32

Hi, I'v modify my "/etc/my.cnf" to be under 2 Go (size of my physical RAM)

Code:

[--] Total buffers: 11.2M per thread and 907.0M global
[!!] Maximum possible memory usage: 2.0G (100% of installed RAM)

Code:

max_allowed_packet      = 16M   # 1Mo 06/06/2008
table_cache             = 512
thread_cache_size       = 8
query_cache_size        = 100M  # 32 Mo 06/06/2008 / 256Mo 25/06/2008
query_cache_limit       = 512M  # 16Mo 24/06/2008 / 128 Mo 25/06/2008
thread_concurrency      = 8     # Try number of CPU's*2
max_heap_table_size     = 100M  # 16 Mo 20/06/2008
tmp_table_size          = 100M  # 34 Mo / 100 Mo

innodb_buffer_pool_size = 700M  # 1 Go 25/06/2008 / 70-80% of memory
innodb_log_buffer_size  = 2M    # 1M 25/06/2008

# === Per connection parameters ====================================
read_buffer_size        = 2M
read_rnd_buffer_size    = 2M    # 8M 25/06/2008
sort_buffer_size        = 2M    # 25/06/2008
thread_stack            = 256K  # 64K 25/06/2008 / 1Mo 26/06/2008
join_buffer_size        = 5M    # 2Mo 25/06/2008 / 128 Ko 20/06/2008
# -----------------------------------------------------------------

# === MyISAM parameters ===========================================
myisam_sort_buffer_size = 1M    # 8Mo 25/06/2008
key_buffer_size         = 4M    # 2Mo 25/06/2008
bulk_insert_buffer_size = 1M    # 8Mo 26/06/2008
# -----------------------------------------------------------------

But i'v allways this warnings:

Code:

[COLOR="Red"][!!] Key buffer hit rate: -1779.9%
[!!] Query cache efficiency: 14.6%
[!!] Joins performed without indexes: 756
[!!] Temporary tables created on disk: 49%
[...]
    query_cache_limit (> 512M, or use smaller result sets)
    join_buffer_size (> 5.0M, or always use indexes with joins)
    tmp_table_size (> 100M)
    max_heap_table_size (> 99M)
    innodb_buffer_pool_size (>= 17G)[/COLOR]

Regards,
Pierre.

**xs-** · 26-06-2008, 11:04

The output of the tool is not holy, its just an aid which indicated which configuration items could produce a performance gain.

**pierre-hoffmann** · 09-07-2008, 08:43

Hi,

Another related question :

How often "HouseKeeper" run,
is there a method to run it manualy ??

(i'm changing a lots of history and trends values / to reduce database size)

Regards,
Pierre..

**xs-** · 09-07-2008, 09:40

if you havent turned it off (it's on by default), every hour.

--------------------------8<-----------------------------
# How often ZABBIX will perform housekeeping procedure (in hours)
HousekeepingFrequency=1

# Uncomment this line to disable housekeeping procedure
#DisableHousekeeping=1
--------------------------8<-----------------------------

**pierre-hoffmann** · 09-07-2008, 09:59

Yes, you're right :-)

In my zabbix_server.conf: (so i use 1 hour default value)

Code:

# How often ZABBIX will perform housekeeping procedure
# (in hours)
# Default value is 1 hour
# Housekeeping is removing unnecessary information from
# tables history, alert, and alarms
# This parameter must be between 1 and 24

#HousekeepingFrequency=1

And in zabbix_server.log:

Code:

# cat zabbix_server.log | grep housekeeper
 14673:20080709:045803 Executing housekeeper
 14673:20080709:060121 Executing housekeeper
 14673:20080709:070430 Executing housekeeper
 14673:20080709:080808 Executing housekeeper

So i hope my dabase size decrease

For this moment it's better : Mysql tablespace doesn't grow anymore but Innodb Free size reduce ...

Regards,
Pierre.

Ad Widget

Mysql Database Grow | How Optimize parameters

Mysql Database Grow | How Optimize parameters

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment