Ad Widget

Collapse

LAMP Zabbix Server tunning

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • bashman
    Senior Member
    • Dec 2009
    • 432

    #1

    LAMP Zabbix Server tunning

    Hi,

    I'm experiencing low Zabbix Server 1.8.2 performance, because items are not updating and I can see them in Zabbix queue. I've tunned mysql and apache already, but I think the problem is zabbix_server tunning. The io host is high, however top host is ok.

    I need some help for zabbix_server tunning.

    This is my Zabbix Status:
    Code:
    Zabbix server is running 	Yes 	 -
    Number of hosts (monitored/not monitored/templates) 	546 	401 / 20 / 125
    Number of items (monitored/disabled/not supported) 	8152 	6507 / 1631 / 14
    Number of triggers (enabled/disabled)[true/unknown/false] 	4816 	3940 / 876  [43 / 401 / 3496]
    Number of users (online) 	26 	6
    Required server performance, new values per second 	53.1465 	-
    This is my zabbix_server.conf:

    Code:
    ############ ADVANCED PARAMETERS ################
    
    ### Option: StartPollers
    #       Number of pre-forked instances of pollers.
    #       You shouldn't run more than 30 pollers normally.
    #
    # Mandatory: no
    # Range: 0-255
    # Default:
    # StartPollers=5
    
    StartPollers=125
    
    ### Option: StartIPMIPollers
    #       Number of pre-forked instances of IPMI pollers.
    #
    # Mandatory: no
    # Range: 0-255
    # Default:
    # StartIPMIPollers=0
    
    StartIPMIPollers=5
    
    ### Option: StartPollersUnreachable
    #       Number of pre-forked instances of pollers for unreachable hosts.
    #
    # Mandatory: no
    # Range: 0-255
    # Default:
    # StartPollersUnreachable=1
    
    StartPollersUnreachable=15
    
    ### Option: StartTrappers
    #       Number of pre-forked instances of trappers
    #
    # Mandatory: no
    # Range: 0-255
    # Default:
    # StartTrappers=5
    
    StartTrappers=125
    
    ### Option: StartPingers
    #       Number of pre-forked instances of ICMP pingers.
    #
    # Mandatory: no
    # Range: 0-255
    # Default:
    # StartPingers=1
    
    StartPingers=80
    
    ### Option: StartDiscoverers
    #       Number of pre-forked instances of discoverers.
    #
    # Mandatory: no
    # Range: 0-255
    # Default:
    # StartDiscoverers=1
    
    StartDiscoverers=0
    
    ### Option: StartHTTPPollers
    #       Number of pre-forked instances of HTTP pollers.
    #
    # Mandatory: no
    # Range: 0-255
    # Default:
    # StartHTTPPollers=1
    
    StartHTTPPollers=25
    
    ### Option: CacheSize
    #       Size of configuration cache, in bytes.
    #       Shared memory size for storing hosts and items data.
    #
    # Mandatory: no
    # Range: 128K-1G
    # Default:
    # CacheSize=8M
    
    CacheSize=100M
    
    ### Option: CacheUpdateFrequency
    #       How often Zabbix will perform update of configuration cache, in seconds.
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # CacheUpdateFrequency=60
    
    #CacheUpdateFrequency=
    
    ### Option: HistoryCacheSize
    #       Size of history cache, in bytes.
    #       Shared memory size for storing history data.
    #
    # Mandatory: no
    # Range: 128K-1G
    # Default:
    # HistoryCacheSize=8M
    
    HistoryCacheSize=30M
    
    ### Option: TrendCacheSize
    #       Size of trend cache, in bytes.
    #       Shared memory size for storing trends data.
    #
    # Mandatory: no
    # Range: 128K-1G
    # Default:
    # TrendCacheSize=4M
    
    TrendCacheSize=10M
    
    ### Option: HistoryTextCacheSize
    #       Size of text history cache, in bytes.
    #       Shared memory size for storing character, text or log history data.
    #
    # Mandatory: no
    # Range: 128K-1G
    # Default:
    # HistoryTextCacheSize=16M
    
    HistoryTextCacheSize=20M
    
    ### Option: Timeout
    #       Specifies how long we wait for agent, SNMP device or external check (in seconds).
    #
    # Mandatory: no
    # Range: 1-30
    # Default:
    # Timeout=3
    
    Timeout=10
    
    ### Option: TrapperTimeout
    #       Specifies how many seconds trapper may spend processing new data.
    #
    # Mandatory: no
    # Range: 1-300
    # Default:
    # TrapperTimeout=300
    
    TrapperTimeout=300
    
    ### Option: UnreachablePeriod
    #       After how many seconds of unreachability treat a host as unavailable
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # UnreachablePeriod=45
    
    UnreachablePeriod=60
    
    ### Option: UnavailableDelay
    #       How often host is checked for availability during the unavailability period.
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # UnavailableDelay=60
    
    UnavailableDelay=120
    
    ### Option: UnreachableDelay
    #       How often host is checked for availability during the unreachability period
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # UnreachableDelay=15
    
    UnreachableDelay=60
    
    ### Option: LogSlowQueries
    #       If a query takes longer than this many milliseconds, the query is logged to the log file.
    #       0 - disable log slow queries
    #
    # Mandatory: no
    # Range: 0-3600000
    # Default:
    # LogSlowQueries=0
    
    LogSlowQueries=10000
    MySQL my.conf:

    Code:
    query_cache_limit       = 1M
    query_cache_size        = 16M
    [isamchk]
    key_buffer              = 16M
    [mysqld]
    lower_case_table_names  = 1
    bind-address            = 0.0.0.0
    #general_log             = 1
    #general_log_file        = /var/log/mysql/mysql.log
    #log-bin                 = /var/log/mysql/mysql-bin.log
    expire_logs_days        = 3
    max_binlog_size         = 100M
    log-short-format
    long_query_time         = 30
    log_slow_queries        = /var/log/mysql/mysql-slow.log
    # * Fine Tuning
    key_buffer              = 1024M
    thread_stack            = 1024K
    thread_cache_size       = 20
    read_rnd_buffer_size    = 524288
    read_buffer_size        = 256K
    sort_buffer_size        = 3145720
    max_allowed_packet      = 1024M
    max_connections         = 1000
    join_buffer_size        = 1M
    query_cache_size        = 128M
    query_cache_limit       = 2M
    tmp_table_size          = 256M
    max_heap_table_size     = 256M
    table_cache             = 512
    max_tmp_tables          = 64
    wait_timeout            = 28800
    interactive_timeout     = 28800
    # * InnoDB
    innodb_buffer_pool_size = 128M
    innodb_flush_method     = O_DIRECT
    innodb_flush_log_at_trx_commit = 2
    [mysqldump]
    max_allowed_packet      = 160M
    978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15
  • fjrial
    Senior Member
    • Feb 2010
    • 140

    #2
    man, what do you understand with "You shouldn't run more than 30 pollers normally"?? you have 125 pollers running... IMHO there are too much pollers configured.

    I don't know your hardware but, I'm monitoring pretty the same amount of items and host, my required server performance is about 80.7037 and I have a virtualized machine with both frontend and zabbix_server, 1 GB RAM.. the problem on performance is basically on I/O through hard disk device.. you need a fast hard disk drive..

    Edited:Attached my zabbix_server.conf and my.cnf:

    Zabbix_server: everything is on default options except for these ones:

    PHP Code:
    StartPollers=10
    StartTrappers
    =10
    CacheSize
    =128M
    Timeout
    =
    Mysql conf:
    PHP Code:
    [mysqld]
    datadir=/var/lib/mysql
    socket
    =/var/lib/mysql/mysql.sock
    user
    =mysql
    # Default to using old password format for compatibility with mysql 3.x
    # clients (those using the mysqlclient10 compatibility package).
    old_passwords=1

    #tunning
    skip-name-resolve
    skip
    -networking

    max_connection
    =60
    key_buffer_size
    =16M
    max_allowed_packet
    =8M
    table_cache
    =128
    sort_buffer_size
    =2M
    read_buffer_size
    =2M
    join_buffer_size
    =2M

    max_heap_table_size
    =64M
    tmp_table_size
    =64M

    myisam_sort_buffer_size
    =64M

    thread_cache
    =128
    query_cache_type
    =1
    query_cache_limit
    =1M
    query_cache_size
    =16M

    innodb_file_per_table
    innodb_buffer_pool_size
    =512M


    # Disabling symbolic-links is recommended to prevent assorted security risks
    symbolic-links=0

    # To enable the InnoDB Plugin, uncomment the 2 next lines
    ignore-builtin-innodb
    plugin
    -load=innodb=ha_innodb_plugin.so

    # To enable InnoDB-related INFORMATION_SCHEMA tables
    # Join the following options to above directive
      
    ;innodb_trx=ha_innodb_plugin.so
      
    ;innodb_locks=ha_innodb_plugin.so
      
    ;innodb_cmp=ha_innodb_plugin.so
      
    ;innodb_cmp_reset=ha_innodb_plugin.so
      
    ;innodb_cmpmem=ha_innodb_plugin.so
      
    ;innodb_cmpmem_reset=ha_innodb_plugin.so



    [mysqldump]
    quick
    max_allowed_packet
    =16M

    [mysql]
    no-auto-rehash

    [isamchk]
    key_buffer=64M
    sort_buffer_size
    =64M
    read_buffer
    =2M
    write_buffer
    =2M

    [myisamchk]
    key_buffer=64M
    sort_buffer_size
    =64M
    read_buffer
    =2M
    write_buffer
    =2M

    [mysqlhotcopy]
    interactive-timeout





    [mysqld_safe]
    log-error=/var/log/mysqld.log
    pid
    -file=/var/run/mysqld/mysqld.pid 
    Last edited by fjrial; 26-08-2010, 10:07. Reason: Attached my zabbix_server.conf and my.cnf:

    Comment

    • bashman
      Senior Member
      • Dec 2009
      • 432

      #3
      The reason to have 125 StartPollers is because it works better than with 30. With my old configuration:

      Code:
      StartPollers=30
      StartIPMIPollers=5
      StartPollersUnreachable=5
      StartTrappers=30
      StartPingers=30
      StartHTTPPollers=5
      Performance was worse and zabbix queue much higher.

      Is there any rule to calculate this values?.
      Last edited by bashman; 26-08-2010, 10:13.
      978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15

      Comment

      • fjrial
        Senior Member
        • Feb 2010
        • 140

        #4
        Sorry I was editing my last reply while you answered..

        I think that might be a problem with mysql configuration..
        In my previous message you can see my configurations (zabbix_server and mysql).

        Perhaps, I don't know if you already did it, you should take a look at the threads about mysql tunning/performance on this forums:
        [Edited: link provided] 5 star thread on the forum


        Cheers.
        Last edited by fjrial; 26-08-2010, 10:23. Reason: link provided

        Comment

        • bashman
          Senior Member
          • Dec 2009
          • 432

          #5
          Yes I did mysql tunning already, and everything was working fine until last Tuesday, when zabbix frontend users online increased from 4 to 7.

          I see that io read is much higher than io write.

          Does it have any relation?.

          Could it be apache performance?.

          Gracias por tu respuesta, un saludo.
          978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15

          Comment

          • fjrial
            Senior Member
            • Feb 2010
            • 140

            #6
            I don't know if this is related to what it's happening to you, but you are telling me that your fronted users increased...

            I have a few users using the frontend.
            I have a lot of screens for big routers (about 600 items (traffic in/out) in one screen).

            Everything was fine but someday one of the users opened about 10 tabs with one screen on each tab and an user refresh rate of 60 seconds....

            cpu load increased a lot an after a few hours, queue grew up a lot and zabbix_server finally died (image attached)..

            Cheers,
            Un saludo
            Attached Files

            Comment

            • fjrial
              Senior Member
              • Feb 2010
              • 140

              #7
              Originally posted by bashman
              Could it be apache performance?.
              For my web server, I'm using lighttpd with mod_fastcgi and apc, it helps me a lot to decrease the cpu load

              Hope it helps

              Comment

              • bashman
                Senior Member
                • Dec 2009
                • 432

                #8
                Thanks Fco. Javier,

                This sure will help me.

                I'll try to tune apache and current sessions and I'll tell you my results.

                Un saludo.
                978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15

                Comment

                • bashman
                  Senior Member
                  • Dec 2009
                  • 432

                  #9
                  Performance seems improving due to maintenance script launching every day and to apache, plus zabbix front-end adjustments.

                  Zabbix queue is much smaller.
                  978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15

                  Comment

                  • fjrial
                    Senior Member
                    • Feb 2010
                    • 140

                    #10
                    Great to hear that!

                    For what you are telling, it seems that your problem may be the frontend usage.. because of the many users that are using it and the screens/graphs monitoring section..

                    Just guessing:... maybe you can try isolating the frontend in another machine to see what it happens?

                    Cheers
                    Un saludo.

                    Comment

                    • bashman
                      Senior Member
                      • Dec 2009
                      • 432

                      #11
                      Hi Fco. Javier, thanks for your responses.

                      It seems that when Zabbix front-end users reaches up to 7 users on line, Zabbix queue and Zabbix hosts io read increase affecting Zabbix performance.

                      That means that items are not been updated when they should, so you don't get trigger alert, only those that are: agent.ping.nodata(600)=1.

                      I most say that I have Zabbix_server, MySQL, Apache frontend, everything on the same host, so a good practice would be to separate frontend or MySQL.

                      Muchas gracias, un saludo.
                      978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15

                      Comment

                      • bashman
                        Senior Member
                        • Dec 2009
                        • 432

                        #12
                        Performance problem solved upgrading to v1.8.3.
                        978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15

                        Comment

                        • fjrial
                          Senior Member
                          • Feb 2010
                          • 140

                          #13
                          It's good to hear that!!

                          I'm still in 1.8.2 because I've to update the patches against 1.8.3 source code.

                          I expect to get an improvement in frontend usage when I've updated our production server.

                          Cheers.

                          Comment

                          • bashman
                            Senior Member
                            • Dec 2009
                            • 432

                            #14
                            Yeah, it's great, I've reduced my StartPollers and StartTrappers from 125 to 30 and I have no performance problem, Zabbix queue is ok.

                            I recommend you to upgrade to 1.8.3, tell me when you do it.
                            978 Hosts / 16.901 Items / 8.703 Triggers / 44 usr / 90,59 nvps / v1.8.15

                            Comment

                            Working...