Ad Widget

Collapse

zabbix server working not good

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • dikaye
    Junior Member
    • Sep 2013
    • 11

    #1

    zabbix server working not good

    My zabbix server seems working not so good, can some body help me to check and give some suggestion?



    Server info:
    Physical machine with 32G RAM, 8 core 2.83GHz and 4T SCSI disk.

    Software:
    CentOS 5.8
    zabbix-2.0.8 with Percona-Server-server-56-5.6.13 (MySQL) in localhost

    Zabbix server configuration:

    Code:
    ############ GENERAL PARAMETERS #################
    
    ### Option: NodeID
    #       Unique NodeID in distributed setup.
    #       0 - standalone server
    #
    # Mandatory: no
    # Range: 0-999
    # Default:
    # NodeID=0
    
    ### Option: ListenPort
    #       Listen port for trapper.
    #
    # Mandatory: no
    # Range: 1024-32767
    # Default:
    # ListenPort=10051
    
    ### Option: SourceIP
    #       Source IP address for outgoing connections.
    #
    # Mandatory: no
    # Default:
    SourceIP=10.1xx.x.xx
    
    ### Option: LogFile
    #       Name of log file.
    #       If not set, syslog is used.
    #
    # Mandatory: no
    # Default:
    # LogFile=
    
    LogFile=/var/log/zabbix/zabbix_server.log
    
    ### Option: LogFileSize
    #       Maximum size of log file in MB.
    #       0 - disable automatic log rotation.
    #
    # Mandatory: no
    # Range: 0-1024
    # Default:
    # LogFileSize=1
    
    LogFileSize=100
    
    ### Option: DebugLevel
    #       Specifies debug level
    #       0 - no debug
    #       1 - critical information
    #       2 - error information
    #       3 - warnings
    #       4 - for debugging (produces lots of information)
    #
    # Mandatory: no
    # Range: 0-4
    # Default:
    # DebugLevel=3
    
    ### Option: PidFile
    #       Name of PID file.
    #
    # Mandatory: no
    # Default:
    # PidFile=/tmp/zabbix_server.pid
    
    PidFile=/var/run/zabbix/zabbix_server.pid
    
    ### Option: DBHost
    #       Database host name.
    #       If set to localhost, socket is used for MySQL.
    #       If set to empty string, socket is used for PostgreSQL.
    #
    # Mandatory: no
    # Default:
    DBHost=localhost
    
    
    ### Option: DBName
    #       Database name.
    #       For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored.
    #
    # Mandatory: yes
    # Default:
    # DBName=
    
    DBName=zabbix_db
    
    ### Option: DBSchema
    #       Schema name. Used for IBM DB2.
    #
    # Mandatory: no
    # Default:
    # DBSchema=
    
    ### Option: DBUser
    #       Database user. Ignored for SQLite.
    #
    # Mandatory: no
    # Default:
    # DBUser=
    
    DBUser=zabbix
    
    ### Option: DBPassword
    #       Database password. Ignored for SQLite.
    #       Comment this line if no password is used.
    #
    # Mandatory: no
    # Default:
    DBPassword=111111
    
    ### Option: DBSocket
    #       Path to MySQL socket.
    #
    # Mandatory: no
    # Default:
    # DBSocket=/tmp/mysql.sock
    
    DBSocket=/var/lib/mysql/mysql.sock
    
    ### Option: DBPort
    #       Database port when not using local socket. Ignored for SQLite.
    #
    # Mandatory: no
    # Range: 1024-65535
    # Default (for MySQL):
    # DBPort=3306
    
    ############ ADVANCED PARAMETERS ################
    
    ### Option: StartPollers
    #       Number of pre-forked instances of pollers.
    #
    # Mandatory: no
    # Range: 0-1000
    # Default:
    StartPollers=200
    
    ### Option: StartIPMIPollers
    #       Number of pre-forked instances of IPMI pollers.
    #
    # Mandatory: no
    # Range: 0-1000
    # Default:
    StartIPMIPollers=10
    
    ### Option: StartPollersUnreachable
    #       Number of pre-forked instances of pollers for unreachable hosts (including IPMI).
    #
    # Mandatory: no
    # Range: 0-1000
    # Default:
    StartPollersUnreachable=80
    
    ### Option: StartTrappers
    #       Number of pre-forked instances of trappers.
    #       Trappers accept incoming connections from Zabbix sender, active agents, active proxies and child nodes.
    #       At least one trapper process must be running to display server availability in the frontend.
    #
    # Mandatory: no
    # Range: 0-1000
    # Default:
    StartTrappers=64
    
    ### Option: StartPingers
    #       Number of pre-forked instances of ICMP pingers.
    #
    # Mandatory: no
    # Range: 0-1000
    # Default:
    StartPingers=12
    
    ### Option: StartDiscoverers
    #       Number of pre-forked instances of discoverers.
    #
    # Mandatory: no
    # Range: 0-250
    # Default:
    # StartDiscoverers=1
    
    ### Option: StartHTTPPollers
    #       Number of pre-forked instances of HTTP pollers.
    #
    # Mandatory: no
    # Range: 0-1000
    # Default:
    StartHTTPPollers=8
    
    ### Option: JavaGateway
    #       IP address (or hostname) of Zabbix Java gateway.
    #       Only required if Java pollers are started.
    #
    # Mandatory: no
    # Default:
    # JavaGateway=
    
    ### Option: JavaGatewayPort
    #       Port that Zabbix Java gateway listens on.
    #
    # Mandatory: no
    # Range: 1024-32767
    # Default:
    # JavaGatewayPort=10052
    
    ### Option: StartJavaPollers
    #       Number of pre-forked instances of Java pollers.
    #
    # Mandatory: no
    # Range: 0-1000
    # Default:
    # StartJavaPollers=0
    
    ### Option: SNMPTrapperFile
    #       Temporary file used for passing data from SNMP trap daemon to the server.
    #       Must be the same as in zabbix_trap_receiver.pl or SNMPTT configuration file.
    #
    # Mandatory: no
    # Default:
    # SNMPTrapperFile=/tmp/zabbix_traps.tmp
    
    SNMPTrapperFile=/var/log/snmptt/snmptt.log
    
    ### Option: StartSNMPTrapper
    #       If 1, SNMP trapper process is started.
    #
    # Mandatory: no
    # Range: 0-1
    # Default:
    # StartSNMPTrapper=0
    
    ### Option: ListenIP
    #       List of comma delimited IP addresses that the trapper should listen on.
    #       Trapper will listen on all network interfaces if this parameter is missing.
    #
    # Mandatory: no
    # Default:
    # ListenIP=0.0.0.0
    
    # ListenIP=127.0.0.1
    
    ### Option: HousekeepingFrequency
    #       How often Zabbix will perform housekeeping procedure (in hours).
    #       Housekeeping is removing unnecessary information from history, alert, and alarms tables.
    #
    # Mandatory: no
    # Range: 1-24
    # Default:
    HousekeepingFrequency=24
    
    ### Option: MaxHousekeeperDelete
    #       The table "housekeeper" contains "tasks" for housekeeping procedure in the format:
    #       [housekeeperid], [tablename], [field], [value].
    #       No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value])
    #       will be deleted per one task in one housekeeping cycle.
    #       SQLite3 does not use this parameter, deletes all corresponding rows without a limit.
    #       If set to 0 then no limit is used at all. In this case you must know what you are doing!
    #
    # Mandatory: no
    # Range: 0-1000000
    # Default:
    # MaxHousekeeperDelete=500
    
    ### Option: DisableHousekeeping
    #       If set to 1, disables housekeeping.
    #
    # Mandatory: no
    # Range: 0-1
    # Default:
    DisableHousekeeping=0
    
    ### Option: SenderFrequency
    #       How often Zabbix will try to send unsent alerts (in seconds).
    #
    # Mandatory: no
    # Range: 5-3600
    # Default:
    # SenderFrequency=30
    
    ### Option: CacheSize
    #       Size of configuration cache, in bytes.
    #       Shared memory size for storing host, item and trigger data.
    #
    # Mandatory: no
    # Range: 128K-2G
    # Default:
    CacheSize=768M
    
    ### Option: CacheUpdateFrequency
    #       How often Zabbix will perform update of configuration cache, in seconds.
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    CacheUpdateFrequency=300
    
    ### Option: StartDBSyncers
    #       Number of pre-forked instances of DB Syncers
    #
    # Mandatory: no
    # Range: 1-100
    # Default:
    StartDBSyncers=16
    
    ### Option: HistoryCacheSize
    #       Size of history cache, in bytes.
    #       Shared memory size for storing history data.
    #
    # Mandatory: no
    # Range: 128K-2G
    # Default:
    HistoryCacheSize=128M
    
    ### Option: TrendCacheSize
    #       Size of trend cache, in bytes.
    #       Shared memory size for storing trends data.
    #
    # Mandatory: no
    # Range: 128K-2G
    # Default:
    TrendCacheSize=384M
    
    ### Option: HistoryTextCacheSize
    #       Size of text history cache, in bytes.
    #       Shared memory size for storing character, text or log history data.
    #
    # Mandatory: no
    # Range: 128K-2G
    # Default:
    HistoryTextCacheSize=64M
    
    ### Option: NodeNoEvents
    #       If set to '1' local events won't be sent to master node.
    #       This won't impact ability of this node to propagate events from its child nodes.
    #
    # Mandatory: no
    # Range: 0-1
    # Default:
    # NodeNoEvents=0
    
    ### Option: NodeNoHistory
    #       If set to '1' local history won't be sent to master node.
    #       This won't impact ability of this node to propagate history from its child nodes.
    #
    # Mandatory: no
    # Range: 0-1
    # Default:
    # NodeNoHistory=0
    
    ### Option: Timeout
    #       Specifies how long we wait for agent, SNMP device or external check (in seconds).
    #
    # Mandatory: no
    # Range: 1-30
    # Default:
    Timeout=10
    
    ### Option: TrapperTimeout
    #       Specifies how many seconds trapper may spend processing new data.
    #
    # Mandatory: no
    # Range: 1-300
    # Default:
    # TrapperTimeout=300
    
    ### Option: UnreachablePeriod
    #       After how many seconds of unreachability treat a host as unavailable.
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # UnreachablePeriod=45
    
    ### Option: UnavailableDelay
    #       How often host is checked for availability during the unavailability period, in seconds.
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # UnavailableDelay=60
    
    ### Option: UnreachableDelay
    #       How often host is checked for availability during the unreachability period, in seconds.
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # UnreachableDelay=15
    
    ### Option: AlertScriptsPath
    #       Full path to location of custom alert scripts.
    #       Default depends on compilation options.
    #
    # Mandatory: no
    # Default:
    # AlertScriptsPath=${datadir}/zabbix/alertscripts
    
    AlertScriptsPath=/usr/lib/zabbix/alertscripts
    
    ### Option: ExternalScripts
    #       Full path to location of external scripts.
    #       Default depends on compilation options.
    #
    # Mandatory: no
    # Default:
    # ExternalScripts=${datadir}/zabbix/externalscripts
    
    ExternalScripts=/usr/lib/zabbix/externalscripts
    
    ### Option: FpingLocation
    #       Location of fping.
    #       Make sure that fping binary has root ownership and SUID flag set.
    #
    # Mandatory: no
    # Default:
    FpingLocation=/usr/sbin/fping
    
    ### Option: Fping6Location
    #       Location of fping6.
    #       Make sure that fping6 binary has root ownership and SUID flag set.
    #       Make empty if your fping utility is capable to process IPv6 addresses.
    #
    # Mandatory: no
    # Default:
    # Fping6Location=/usr/sbin/fping6
    
    ### Option: SSHKeyLocation
    #       Location of public and private keys for SSH checks and actions
    #
    # Mandatory: no
    # Default:
    # SSHKeyLocation=
    
    ### Option: LogSlowQueries
    #       How long a database query may take before being logged (in milliseconds).
    #       Only works if DebugLevel set to 3 or 4.
    #       0 - don't log slow queries.
    #
    # Mandatory: no
    # Range: 1-3600000
    # Default:
    LogSlowQueries=3000
    
    ### Option: TmpDir
    #       Temporary directory.
    #
    # Mandatory: no
    # Default:
    # TmpDir=/tmp
    
    ### Option: Include
    #       You may include individual files or all files in a directory in the configuration file.
    #       Installing Zabbix will create include directory in /usr/local/etc, unless modified during the compile time.
    #
    # Mandatory: no
    # Default:
    # Include=
    
    # Include=/usr/local/etc/zabbix_server.general.conf
    # Include=/usr/local/etc/zabbix_server.conf.d/
    
    ### Option: StartProxyPollers
    #       Number of pre-forked instances of pollers for passive proxies.
    #
    # Mandatory: no
    # Range: 0-250
    # Default:
    # StartProxyPollers=1
    
    ### Option: ProxyConfigFrequency
    #       How often Zabbix Server sends configuration data to a Zabbix Proxy in seconds.
    #       This parameter is used only for proxies in the passive mode.
    #
    # Mandatory: no
    # Range: 1-3600*24*7
    # Default:
    # ProxyConfigFrequency=3600
    
    ### Option: ProxyDataFrequency
    #       How often Zabbix Server requests history data from a Zabbix Proxy in seconds.
    #       This parameter is used only for proxies in the passive mode.
    #
    # Mandatory: no
    # Range: 1-3600
    # Default:
    # ProxyDataFrequency=1
    #DisableHousekeeper=1
    Mysql server configure:
    Code:
    [mysqld]
    #
    explicit_defaults_for_timestamp = true
    datadir=/data/db/zabbix/
    socket=/var/lib/mysql/mysql.sock
    user=mysql
    # Default to using old password format for compatibility with mysql 3.x
    # clients (those using the mysqlclient10 compatibility package).
    old_passwords=0
    bind-address=0.0.0.0
    
    connect_timeout         = 60
    wait_timeout            = 28800
    max_connections         = 2048
    max_allowed_packet      = 64M
    max_connect_errors      = 1000
    
    innodb_use_sys_malloc   = 1
    # limits
    tmp_table_size          = 512M
    max_heap_table_size     = 256M
    table_open_cache        = 512
    
    # logs
    log_error               = /var/log/mysql/mysql-error.log
    slow_query_log_file     = /var/log/mysql/mysql-slow.log
    slow_query_log          = 1
    #long_query_time         = 20
    
    # innodb
    innodb_data_home_dir    = /data/db/innodb
    innodb_data_file_path   = "ibdata1:128M;ibdata2:128M:autoextend:max:4096M"
    innodb_file_per_table   = 1
    innodb_status_file      = 1
    #innodb_additional_mem_pool_size        = 128M
    innodb_buffer_pool_size = 18G
    innodb_flush_method     = O_DIRECT
    innodb_io_capacity      = 2000
    innodb_flush_log_at_trx_commit  = 2
    innodb_support_xa               = 0
    innodb_log_file_size    = 512M
    innodb_log_buffer_size  = 128M
    
    # experimental
    #innodb_stats_updata_need_lock = 0
    
    # other stuff
    event_scheduler         = 1
    query_cache_type        = 0
    query_cache_limit       = 1M
    
    [mysqld_safe]
    log-error=/var/log/mysqld.log
    pid-file=/var/run/mysqld/mysqld.pid
    Attached Files
    Last edited by dikaye; 22-09-2013, 03:39.
  • tchjts1
    Senior Member
    • May 2008
    • 1605

    #2
    You've got housekeeper disabled. You using a partitioned DB?
    If not... how are you cleaning up old data?

    And a better way to understand what might be wrong would be to look at the graphs for the Zabbix internal items. Look at this post... near the bottom explains how to get this data. It would help pinpoint what is going on.




    .

    Comment

    • dikaye
      Junior Member
      • Sep 2013
      • 11

      #3
      Let me update some image information for your references.

      Thanks you very much.
      Attached Files

      Comment

      • dikaye
        Junior Member
        • Sep 2013
        • 11

        #4
        I don't use any partitioned DB.

        Does the zabbix 2.0.8 db support partitioned? thanks.

        Comment

        • tchjts1
          Senior Member
          • May 2008
          • 1605

          #5
          I think you will see some relief from your issue if you increase both your StartPollers and your HistoryCacheSize. I would also decrease the number of StartDBSyncers that you are using and put it back to the default value.

          Restart your Zabbix server process after you make any adjustments to zabbix_agentd.conf

          Also, either there is another graph for Zabbix server, or you can simply create one from the existing items, that show cache usage by %. This will help you tune those parameters a little better.

          Comment

          • Pada
            Senior Member
            • Apr 2012
            • 236

            #6
            Also, is it really necessary to monitor ~6000 items per host?
            I hope you're not using SNMP to monitor all those items. If you are, then you can make use of Zabbix proxies to split the load for SNMP hosts.

            Comment

            • dikaye
              Junior Member
              • Sep 2013
              • 11

              #7
              yes, all hosts are monitored by SNMP.

              Seems the MySQL is using high CPU.
              Attached Files

              Comment

              • dikaye
                Junior Member
                • Sep 2013
                • 11

                #8
                It seems there are some many items in queue from the proxy server, here is the proxy server monitoring graphs.


                Comment

                • dikaye
                  Junior Member
                  • Sep 2013
                  • 11

                  #9

                  Comment

                  • tchjts1
                    Senior Member
                    • May 2008
                    • 1605

                    #10
                    You also had housekeer disabled until just the other day, right?
                    How long have you been running your setup without having housekeeper enabled?

                    You also have it set to run every 24 hours. I tried that setting as well, and found poor performance out of it. I went back to the default of 1 hour. It runs 10 - 12 minutes then it's done.

                    Comment

                    • Pada
                      Senior Member
                      • Apr 2012
                      • 236

                      #11
                      I see you've disabled query cache in MySQL, by setting 'query_cache_type=0', like they've shown here: http://zabbixzone.com/zabbix/mysql-p...ps-for-zabbix/

                      Query Cache makes a huge difference in our Zabbix environment, and it is very much noticeable when viewing graphs over a day's span and even more so when the item had a very short interval (eg. 5s).
                      QC does not work with partitioned tables, which is the only reason why I'd think that someone would disable it.

                      Also seeing that you have a 32GB machine, you may want to increase your innodb_buffer_pool_size to like 24GB, up from 18GB.

                      I'd highly recommend that you remove the innodb_io_capacity setting or leave it on the default of 200. You should only increase it if you have really fast storage, such as SSD's in RAID.
                      Read more here:
                      http://dev.mysql.com/doc/refman/5.1/...db_io_capacity


                      So be very careful when using someone's "optimal" settings when you don't have a near identical setup!

                      Lastly, could you explain to me what "4T SCSI disk" entails exactly?
                      Typically for a DB server, you'd want at least 4+ disks in hardware RAID 10 with battery backup unit.

                      Comment

                      Working...