Ad Widget

Collapse

ZABBIX server daemon dont stores values

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • mbrand5371
    Member
    • May 2009
    • 55
    #1

    ZABBIX server daemon dont stores values

    Hi all.
    We'got a strange problem with one of out ZABBIX instances.

    The ZABBIX Serverdaemon will request values (different Items, different Templates, different OS) from ZABBIX Agent and get an response. The Debug Log will look like:

    Code:
      2007:20110304:120936.389 In substitute_simple_macros (data:'system.cpu.util[,user,avg1]')
  2007:20110304:120936.389 In get_value() key:'system.cpu.util[,user,avg1]'
  2007:20110304:120936.389 In get_value_agent() host:'slxvbs03.referenz.polizei.niedersachsen.de' addr:'10.100.217.18' key:'system.cpu.util[,user,avg1]'
  2007:20110304:120936.389 Sending [system.cpu.util[,user,avg1]
]
  2007:20110304:120936.389 Get value from agent result: '0.405748'
  2007:20110304:120936.389 End of get_value():SUCCEED
  2007:20110304:120936.389 In zbx_hashset_search()
  2007:20110304:120936.389 End of zbx_hashset_search()
  2007:20110304:120936.389 In zbx_hashset_search()
  2007:20110304:120936.389 End of zbx_hashset_search()
  2007:20110304:120936.389 In calculate_item_nextcheck (355026,150,"",1299236976)
  2007:20110304:120936.389 End calculate_item_nextcheck (nextcheck:1299237126 delay:150)
  2007:20110304:120936.389 In zbx_binary_heap_insert() key[355026]
  2007:20110304:120936.389 In zbx_hashmap_get() key[355026]
  2007:20110304:120936.389 End of zbx_hashmap_get() key[355026] value[-1]
  2007:20110304:120936.389 In zbx_hashmap_set() key[355026] value[1993]
  2007:20110304:120936.389 End of zbx_hashmap_set()
  2007:20110304:120936.389 End of zbx_binary_heap_insert()
  2007:20110304:120936.389 In DCflush_nextchecks()
  2007:20110304:120936.389 End of get_values()
  2007:20110304:120936.389 In DCconfig_get_poller_nextcheck() poller_type:0
  2007:20110304:120936.389 End of DCconfig_get_poller_nextcheck():1299236977
  2007:20110304:120936.389 Poller #3 spent 0.119082 seconds while updating  11 values. Sleeping for 1 seconds
    But the ZABBIX queue eg. for this item will be open.
    It seems the response is not noticed.

    So the ZABBIX queue grows and ZABBIX server will stop storing more values.

    Our Environment:
    ZABBIX and MySQL Server: RHEL 5.4
    ZABBIX Release: 1.8.3
    MySQL Release: mysql-5.0.77-4.el5_5.4

    Parameter Value Details
    Zabbix server is running Yes -
    Number of hosts (monitored/not monitored/templates) 316 247 / 1 / 68
    Number of items (monitored/disabled/not supported) 6234 5534 / 516 / 184
    Number of triggers (enabled/disabled)[problem/unknown/ok] 4587 4516 / 71 [74 / 452 / 3990]

    Config will be attached.
    Attached Files
    Tags: None
    • marcel
      Senior Member
      Zabbix Certified Specialist
      • Oct 2010
      • 112
      #2
      check your database processes, number of agents, number of pollers, etc

      zabbix queue is not a very good way to represent the status of zabbix application

      seems like your server is not being able to handle that many items

      marcel
      Zabbix Certified Specialist for Large Environments since 12/2010

        Comment

        • nelsonab
          Senior Member
          Zabbix Certified SpecialistZabbix Certified Professional
          • Sep 2006
          • 1233
          #3
          Also look a little closer at how you instantiate your innodb file. Also what is the current size of your innodb file. Most often innodb files are created in a manner where they do not grow, if you reach this max size it will not store data.
          RHCE, author of zbxapi
          Ansible, the missing piece (Zabconf 2017): https://www.youtube.com/watch?v=R5T9NidjjDE
          Zabbix and SNMP on Linux (Zabconf 2015): https://www.youtube.com/watch?v=98PEHpLFVHM

            Comment

            • mbrand5371
              Member
              • May 2009
              • 55
              #4
              OK!
              I've changed some "pollers" and "cache" settings.
              Current config:

              Code:
              ListenPort=10510
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=512
DebugLevel=4
DBName=zabbix1r
DBUser=zabbix
DBPassword=XXXXXXXXXXX
StartPollers=30
StartIPMIPollers=4
StartTrappers=10
StartPingers=3
StartDiscoverers=15
StartHTTPPollers=3
SenderFrequency=30
CacheSize=128M
CacheUpdateFrequency=600
StartDBSyncers=6
HistoryCacheSize=128M
TrendCacheSize=128M
HistoryTextCacheSize=128M
Timeout=10
              But no change. A few hours later, ZABBIX Queue will have hundert of items for more than 10 minutes and "latest data" of the items will report old values. The MySQL Data File is 16Gb big and not growing. Filesystem has 30Gb space left.

              The MySQL Config:

              Code:
              datadir=/service/mysql/var/lib/
thread_concurrency=2
key_buffer=16M
max_allowed_packet=1M
sort_buffer=512K
net_buffer_length=8K
socket=/var/lib/mysql/mysql.sock
user=mysql
old_passwords=1
tmp_table_size=64M
max_heap_table_size=32M
thread_cache_size=10
table_cache=128
innodb_lock_wait_timeout=180
innodb_file_io_threads=4
innodb_thread_concurrency=6
innodb_data_home_dir=/service/mysql/var/lib/
innodb_data_file_path=ibdata1:10M:autoextend
innodb_buffer_pool_size=128M
innodb_additional_mem_pool_size=20M
innodb_log_file_size=64M
innodb_log_buffer_size=32M
innodb_flush_log_at_trx_commit=1

[mysqld_safe]
log-error=/service/mysql/var/log/mysqld_error.log
pid-file=/var/run/mysqld/mysqld.pid

                Comment

                • nelsonab
                  Senior Member
                  Zabbix Certified SpecialistZabbix Certified Professional
                  • Sep 2006
                  • 1233
                  #5
                  Silly question, are your graphs actually working? IE do they update with new data?
                  RHCE, author of zbxapi
                  Ansible, the missing piece (Zabconf 2017): https://www.youtube.com/watch?v=R5T9NidjjDE
                  Zabbix and SNMP on Linux (Zabconf 2015): https://www.youtube.com/watch?v=98PEHpLFVHM

                    Comment

                    • mbrand5371
                      Member
                      • May 2009
                      • 55
                      #6
                      No! The graphs of ZABBIX "Items in queue" and "ZABBIX Processed Items" are stopped. The graphs and history of monitored and "outdated" items are stopped too.

                        Comment

                        • mbrand5371
                          Member
                          • May 2009
                          • 55
                          #7
                          Currently dont know why, but it's solved.

                          Stopped zabbix-server and set frontend in maintenance for 12h and let the database do his job standalone. Started zabbix-server and all will work fine again :-)

                            Comment

                            Previous template Next
                            Working...