ZABBIX Forums  
  #1  
Old 19-06-2008, 17:40
pierre-hoffmann pierre-hoffmann is offline
Senior Member
 
Join Date: Jan 2008
Location: France
Posts: 133
Angry Mysql Database Grow (36 Gb) | How Optimize parameters

Hi,

My Zabbix Mysql database become very very large : ibdata1 is about 36 Gb.
(Multiple extend of vg to avoid Zabbix down)

My problemes:
-> Database use defaut huge my.cnf (how to optimize ?)
-> I've two Large table with 150 (History) & 90 (Trends) Billions Records table, where a "select count(*)" take about 30 minutes
-> I've try to reduce History and trend on each object but small effect
-> I've try to make OPTIMIZE TABLE (for rebuild index and defragment my tables) but
-> It lock table and Zabbix hang (:-) Do you know Workarround like
on Oracle to delay transaction (like ALTER TABLESPACE BEGIN BACKUP)
-> After this Innodb space free was 10 Gb : Unable to reduce a Mysql "Tablespace" ???

In General: What the best Practices for this case ?
-> Use per table datafile is a good solution ??
-> Create multiple fixed size Innodb "tablespace" Better ?
-> I want to prepare 1.6 migration but without this big problem !

Zabbix report:
Code:
Number of hosts            346
Number of items           5570
Number of triggers        2338
Number of events	 265912
Number of alerts	  27905
Mysql info:
Code:
mysql> select sum(data_length), sum(index_length)
 from information_schema.tables;
+------------------+-------------------+
| sum(data_length) | sum(index_length) |
+------------------+-------------------+
|      18029174062 |        8558508032 |
+------------------+-------------------+
1 row in set (1.26 sec)
(18 Gb of data & 8 Go of index)

Biggest tables:




Regards Pierre.

Last edited by pierre-hoffmann; 25-06-2008 at 11:20.
Reply With Quote
  #2  
Old 20-06-2008, 13:31
xs- xs- is offline
Senior Member
Zabbix certified specialist
 
Join Date: Dec 2007
Location: Netherlands
Posts: 391
Send a message via MSN to xs-
Default

Some (imho) good hints and practices:
(disclaimer, these are generic points, they don't have to apply to your situation)

Zabbix
  1. Items
    Be realistic about your items
    • Poll interval. Default zabbix templates use poll intervals of 5s-60s. IMHO this is waay too often. Use 300s as a minimum, and larger numbers for items which do not change much (like disk space usage i.e. 15min).
    • History save time. This is 'how long item data is stored using the poll interval'. If your poll interval is 1min and history save time is 1 year, you will have 60x24x365 records for that item alone. Is this really neccesairy?.
      Personally i use 3 days for everything.
    • Trend save time. This works 'somewhat' like RRD. It takes history values and stores it with a higher (but fixed ) interval.
      Personally i use 0 days (none) for boolean values, 0 days (none) for log and snmp trap messages (we have a loghost for that) and 100 days for 'default' numeric-non-boolean values.
  2. Triggers
    Each trigger change generates an event record. This table can also grow significantly if you have a lot of flapping triggers or useless triggers.
    Be realistic as to when you want a trigger fired.
    IMHO its good practice to not just look at the triggers themselves but also look at the 'events'. This way you will be confronted which the amount of status changes per trigger. (only downside of this is that the event view on the webinterface is really really slow due to the lack of indexes i think)
  3. Database housekeeping
    Zabbix itself does housekeeping, but it does not sanitize the database.
    This means that all orphaned data is left in the db.
    • If you add/remove/add/remove distributed nodes, all data will be left in the database.
    • If you use DM for syncing configuration (change slave node configs via the master node webinterface) data can be orphaned on child and master node!
    • If you modify the database externally (i do via cmdb integration scripts) data can be orphaned.
    There are several threads on the forum with hints and tips on cleaning this up. It helps, alot!
    Clearing out old-node data is a different ball game. I've written some scripts for this, but i need to make them more generic before i post them on the forum/wiki, but this is necessary when you remove a DM node!

MySQL
  1. Separation of Database and zabbix services
    If you have a large environment, it pays off to run your database on a different server than your zabbix service and webinterface.
    I have 2 servers for our DM master node. 1 for the zabbix_server + webinterface, 1 running mysql.
    If your environment is large, do NOT run either on a VM!
  2. Mysql tuning
    Use mysql tuning scripts to optimize your database settings, like memory usage, buffer sizes, etc. For example:
    http://rackerhacker.com/mysqltuner/ <-- I only use this one, its good.
    http://www.day32.com/MySQL/
    http://hackmysql.com/mysqlreport
    They will all bitch about query optimization, allas you need to ignore those
  3. Seperate innodb file per table
    This option in the my.cf is the only way you are able to use the mysqlcheck -o (optimize / shrink) functionality.
    If you use the default InnoDB setting (1 file) you are not able to clear out empty record space like PostgreSQL does in its housekeeping. There is a thread on the forum about this.
    Another option with this is that you could move/link the history/trends table file to a different set of disk spindles they don't interfere with each other when a full table scan is needed (often if it's done by the webinterface).
  4. Dump/restore db
    In relation to the previous point, if you are using the 'one-file-for-all-dbs-tables' default setting for InnoDB, you can only clear out all empty records by dumping all databases, remove the innodb data files and restore all databases in a fresh mysql data instance.
    You could try this during some maintenance period and see how many gig's this will save, i think lots
  5. Other database system?
    You could try a different database.
    In my personal experience PostgreSQL performs the same at best with a lot more maintenance requirement in the long term. I have switched mysql->posgresql->mysql and will stay with mysql5. I've had postgresql guru's look at the configs and systems, no go. The main issue is table indexes for the history, trends and events tables. The only switch i am willing to explore is Oracle with the help of a team of Oracle DBA's and custom database setups to tacke the index issue.

- This is what i could think of in 5 min, if i think of more options or tips will will try to add it to this post.
- Yes the post is probably full of typo's, deal with it
- This post is based on 1.4.x, but generally usable for all versions

Last edited by xs-; 24-02-2009 at 17:23.
Reply With Quote
  #3  
Old 20-06-2008, 17:21
pierre-hoffmann pierre-hoffmann is offline
Senior Member
 
Join Date: Jan 2008
Location: France
Posts: 133
Wink

Hi,

Great thanks for this Post : lots of good idea.

Related Posts:
Some Question:
-> Why you say no VMWare for mysql database ?
-> What type records do manualy clean in database ?? (i've seen sometimes message, in web interface, like 'error : host xxx doesn't exist')
-> I've try mySqlTunner ... Great : but it's not very easy to find good tunning of my.cnf
Code:
./mysqltuner.pl
 >>  MySQLTuner 0.9.0 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering
[!!] Successfully authenticated with no password - SECURITY RISK!

-------- General Statistics --------------------------------------------------
[--] Unable to check for the latest MySQLTuner version
[OK] Currently running supported MySQL version 5.0.26-log
[OK] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 2K (Tables: 8)
[--] Data in InnoDB tables: 16G (Tables: 66)

-------- Performance Metrics -------------------------------------------------
[--] Up for: 6h 13m 32s (26M q [1K qps], 11K conn, TX: 407M, RX: 3B)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 18.2M per thread and 2.3G global
[!!] Allocating > 2GB RAM on 32-bit systems can cause system instability
[!!] Maximum possible memory usage: 4.1G (206% of installed RAM)
[OK] Slow queries: 0% (5/26M)
[OK] Highest usage of available connections: 28% (29/100)
[OK] Key buffer size / total MyISAM indexes: 500.0M/74.0K
[!!] Key buffer hit rate: -1224.5%
[!!] Query cache efficiency: 15.9%
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0%
[!!] Joins performed without indexes: 3914
[!!] Temporary tables created on disk: 48%
[OK] Thread cache hit rate: 99%
[OK] Table cache hit rate: 97%
[OK] Open file limit used: 2%
[OK] Table locks acquired immediately: 100%
[!!] InnoDB data size / buffer pool: 17.0G/1.5G

-------- Recommendations -----------------------------------------------------
General recommendations:
    MySQL started within last 24 hours - recommendations may be inaccurate
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
  *** MySQL's maximum memory usage exceeds your installed memory ***
  *** Add more RAM before increasing any MySQL buffer variables  ***
    query_cache_limit (> 16M, or use smaller result sets)
    join_buffer_size (> 2.0M, or always use indexes with joins)
    tmp_table_size (> 100M)
    max_heap_table_size (> 99M)
    innodb_buffer_pool_size (>= 16G)
-> I'm affraid by
Code:
[!!] Key buffer hit rate: -1224.5%
But don't know why ...
-> i'm increase query_cache_limit, join_buffer_size, tmp_table_size and max_heap_table_size but it wan't more and more memory ... i've only 2 Gb ....
-> And "innodb_buffer_pool_size (>= 16G)" seem to me very strange ... RAM = database size ??!!!


I'm cleaning my items retention time ...

Regards,
Pierre.
__________________
P.Hoffmann
System & Network Admin.
__________________________
Zabbix version 1.8.1
Hosts monitored 1300
OS Novell SLES 10 SP2
__________________________

Last edited by pierre-hoffmann; 25-06-2008 at 16:05.
Reply With Quote
  #4  
Old 23-06-2008, 14:54
xs- xs- is offline
Senior Member
Zabbix certified specialist
 
Join Date: Dec 2007
Location: Netherlands
Posts: 391
Send a message via MSN to xs-
Default

Quote:
Originally Posted by pierre-hoffmann
I dont agree with the innodb_flush_log_at_trx_commit=0. While this does give a speed increase if you have a problem with disk io operations, it also corrupts your database on a crash/power outage.

Quote:
Originally Posted by pierre-hoffmann
-> Why you say no VMWare for mysql database ?
Migrate your mysql service to a physical server and find out
Mostly because of disk io operations. Besides that, processor time.
Try it, we've been able to run a zabbix/mysql server on 1 vm up until 400 hosts. At that time everything (including other VM's on the same server) started to grind to a halt.

Quote:
Originally Posted by pierre-hoffmann
-> What type records do manualy clean in database ?? (i've seen sometimes message, in web interface, like 'error : host xxx doesn't exist')
I dont know in which context to place your error, but an example of what i mean would be:
http://www.zabbix.com/forum/showthread.php?t=9781
You would effectively run the following (USE AT OWN RISK, it helped me, but i havent tested this on multiple scenarios to be sure this wont break stuff. Backup your database first!):
Quote:
Originally Posted by USE AT YOUR OWN RISK
DELETE history_str.* FROM history_str LEFT OUTER JOIN items ON history_str.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_uint.* FROM history_uint LEFT OUTER JOIN items ON history_uint.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_str_sync.* FROM history_str_sync LEFT OUTER JOIN items ON history_str_sync.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_sync.* FROM history_sync LEFT OUTER JOIN items ON history_sync.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history_log.* FROM history_log LEFT OUTER JOIN items ON history_log.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE history.* FROM history LEFT OUTER JOIN items ON history.itemid = items.itemid WHERE items.itemid IS NULL;
DELETE trends.* FROM trends LEFT OUTER JOIN items ON trends.itemid = items.itemid WHERE items.itemid IS NULL;
But this is only part of it (specially with DM), because there is no use of foreign keys, a lot of data can be orphaned (which might be the case in your situation). I'm hoping the devs can provide a tool or method to check for this in a better way.

Quote:
Originally Posted by pierre-hoffmann
-> I've try mySqlTunner ... Great : but it's not very easy to find good tunning of my.cnf
You need to run mysql for at least 2 days straigt in order to get decent values from mysqltuner. (thus the -12k values).
Also, the inno_db_cache thiny: use 50%-75% of your total mem for this.

This is mostly trial and error.
Don't enlarge the values too fast, always document every step you take so you can roll back.

Below are the tuned values i'm currently using (as partly suggested by mysqltuner). This is for a 4G mem server.
Quote:
innodb_file_per_table
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 8
max_connections = 300

table_cache = 128
tmp_table_size = 256M
max_heap_table_size = 256M
join_buffer_size = 5M

innodb_buffer_pool_size = 2G

# This is too high, need to lower next reboot
query_cache_limit = 256M
query_cache_size = 512M
Hope this helps

---- Edit ----
As JoelG indicates, decent knowledge about mysql and the settings you are changing is kinda mandatory. Dig into the docs, it pays off!

Last edited by xs-; 24-06-2008 at 01:03.
Reply With Quote
  #5  
Old 23-06-2008, 22:14
JoelG JoelG is offline
Member
 
Join Date: Aug 2007
Location: McLean, VA
Posts: 32
Default Caution

Make these changes one at time, as a standard practice. I made a change, mysql did not like that the log file info was different, and gave me a message that the .frm files were invalid. I recommend caution when changing anything dealing with a log file.
Reply With Quote
  #6  
Old 25-06-2008, 10:48
pierre-hoffmann pierre-hoffmann is offline
Senior Member
 
Join Date: Jan 2008
Location: France
Posts: 133
Default

Hi,

I'm working on Zabbix Mysql database:

-> Question about OPTIMIZE TABLE:

It exist any solution to avoid table fragmentation ??
(i've make an optimize table 1 week ago and i've lost 120 Mo on a 190 Mo table !!!!)

before:

Code:
root@lxsyst05 / >mysql zabbix
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 35522 to server version: 5.0.26-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> optimize table history_str;
+--------------------+----------+----------+----------+
| Table              | Op       | Msg_type | Msg_text |
+--------------------+----------+----------+----------+
| zabbix.history_str | optimize | status   | OK       |
+--------------------+----------+----------+----------+
1 row in set (1 min 56.80 sec)

mysql>
after:


-> Mysql memory parameters : not very better ...

I'm slowly change some paramters:
  • reduce innodb_buffer_pool_size to 1 Go (i've 2 Go of RAM)
  • increase query_cache_limit to 128 Mo
  • reduce key_buffer to 16 M (only for MyISAM tables ???) (no change on "Key buffer hit rate")
Any ideas ?? I need to disable "innodb_flush_log_at_trx_commit" ??

(http://www.mysqlperformanceblog.com/...-memory-usage/)

Code:
./mysqltuner.pl
 >>  MySQLTuner 0.9.0 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

-------- General Statistics --------------------------------------------------
[--] Unable to check for the latest MySQLTuner version
[OK] Currently running supported MySQL version 5.0.26-log
[OK] Operating on 32-bit architecture with less than 2GB RAM

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 2K (Tables: 8)
[--] Data in InnoDB tables: 17G (Tables: 66)

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1d 0h 18m 4s (103M q [1K qps], 36K conn, TX: 1B, RX: 1B)
[--] Reads / Writes: 60% / 40%
[--] Total buffers: 18.1M per thread and 1.4G global
[!!] Allocating > 2GB RAM on 32-bit systems can cause system instability
[!!] Maximum possible memory usage: 3.2G (159% of installed RAM)
[OK] Slow queries: 0% (16/103M)
[OK] Highest usage of available connections: 27% (27/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/74.0K
[!!] Key buffer hit rate: -1507.1%
[!!] Query cache efficiency: 16.0%
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0%
[!!] Joins performed without indexes: 13946
[!!] Temporary tables created on disk: 47%
[OK] Thread cache hit rate: 99%
[OK] Table cache hit rate: 90%
[OK] Open file limit used: 3%
[OK] Table locks acquired immediately: 100%
[!!] InnoDB data size / buffer pool: 17.6G/1.0G

-------- Recommendations -----------------------------------------------------
General recommendations:
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
  *** MySQL's maximum memory usage exceeds your installed memory ***
  *** Add more RAM before increasing any MySQL buffer variables  ***
    query_cache_limit (> 128M, or use smaller result sets)
    join_buffer_size (> 2.0M, or always use indexes with joins)
    tmp_table_size (> 128M)
    max_heap_table_size (> 127M)
    innodb_buffer_pool_size (>= 17G)
My MySql Config:
Code:
key_buffer              = 16M    # 128Mo before
max_allowed_packet      = 16M    # 1Mo 06/06/2008
table_cache             = 512
sort_buffer_size        = 6M     # 2Mo before
read_buffer_size        = 2M
join_buffer_size        = 2M     # 128 Ko 20/06/2008
read_rnd_buffer_size    = 8M                        [read_rnd_buffer_size] 
thread_cache_size       = 8
thread_stack            = 64K
query_cache_size        = 256M   # 32 Mo 06/06/2008
query_cache_limit       = 128M   # 16 Mo 24/06/2008
thread_concurrency      = 8      # 2 CPU
tmp_table_size          = 128M   # 34 Mo  [tmp_table_size]
max_heap_table_size     = 128M   # 16Mo 20/06/2008 

innodb_flush_log_at_trx_commit = 0 # Delayed flush on commits
innodb_buffer_pool_size = 1G     # 70-80% of memory.

# -----------------------------------------------------------------
# Configuration des log binaire = log de transaction ou archive-log
# -----------------------------------------------------------------
log-bin                 = /archiv/mysql/mysql-bin
log-bin-index           = /archiv/mysql/mysql-bin.index
max-binlog-size         = 100M
# -----------------------------------------------------------------
# Configuration des redo-logs
# -----------------------------------------------------------------
innodb_log_file_size            = 50M # 25% de buffer_pool_size
innodb_log_files_in_group       = 3
innodb_log_group_home_dir       = /dbfs/mysql/logfiles
# -----------------------------------------------------------------
Regards,
Pierre.
__________________
P.Hoffmann
System & Network Admin.
__________________________
Zabbix version 1.8.1
Hosts monitored 1300
OS Novell SLES 10 SP2
__________________________

Last edited by pierre-hoffmann; 26-06-2008 at 09:27.
Reply With Quote
  #7  
Old 25-06-2008, 11:06
xs- xs- is offline
Senior Member
Zabbix certified specialist
 
Join Date: Dec 2007
Location: Netherlands
Posts: 391
Send a message via MSN to xs-
Default

The fragmentation you are talking about is actually dead tuples (as postgresql calls it, dunno what term mysql uses). Basically these are deleted records or updates records which takes up less space than the original one. These 'empty spaces' in a data file aren't automagically removed, that's what optimize tables is for in mysql (and only 'really' works if you have the innodb_file_per_table option).

Quote:
*** MySQL's maximum memory usage exceeds your installed memory ***
*** Add more RAM before increasing any MySQL buffer variables ***
You might need more memory or clean up the contents of your database. You currently have more memory usage configured for mysql than actually available in your system.
Reply With Quote
  #8  
Old 25-06-2008, 16:20
pierre-hoffmann pierre-hoffmann is offline
Senior Member
 
Join Date: Jan 2008
Location: France
Posts: 133
Default

Quote:
Originally Posted by xs- View Post
TYou might need more memory or clean up the contents of your database. You currently have more memory usage configured for mysql than actually available in your system.
Don't understand why it say "Maximum possible memory usage: 3.2G (159% of installed RAM)"
innodb_buffer_pool_size is 1Gb and other buffer was 256 Mo ou 128 Mo so where is this 3.2 Gb ???

I'm trying to increase "join_buffer_size" from 2M to 15M, "Maximum possible memory usage" increase to 4 Go and when i reduce it 10M "Maximum possible memory usage" decrease to 3.2 Go ????????

What's wrong ???

Regards,
Pierre.
__________________
P.Hoffmann
System & Network Admin.
__________________________
Zabbix version 1.8.1
Hosts monitored 1300
OS Novell SLES 10 SP2
__________________________

Last edited by pierre-hoffmann; 25-06-2008 at 17:08.
Reply With Quote
  #9  
Old 25-06-2008, 17:13
xs- xs- is offline
Senior Member
Zabbix certified specialist
 
Join Date: Dec 2007
Location: Netherlands
Posts: 391
Send a message via MSN to xs-
Default

the innodb_buffer thingy is global for your database.
Most other buffers and cache sizes are per connection (or at least have a multiplier for something).

How is your swap usage?
Reply With Quote
  #10  
Old 25-06-2008, 17:31
pierre-hoffmann pierre-hoffmann is offline
Senior Member
 
Join Date: Jan 2008
Location: France
Posts: 133
Default

Quote:
Originally Posted by xs- View Post
How is your swap usage?
Code:
Mem:   2076300k total,  2019768k used,    56532k free
Swap:  4200988k total,    16752k used,  4184236k free
Swap is unused .... :-)

So, all this parameters is per session/connexion ?
  • read_buffer_size
  • read_rnd_buffer_size
  • sort_buffer_size
  • thread_stack
  • join_buffer_size
Regards,
Pierre.
__________________
P.Hoffmann
System & Network Admin.
__________________________
Zabbix version 1.8.1
Hosts monitored 1300
OS Novell SLES 10 SP2
__________________________

Last edited by pierre-hoffmann; 26-06-2008 at 11:30.
Reply With Quote
Reply

Tags
fsfgsfgsgf

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 19:41.