Zabbix Queue

[jansonz]

Hello,

I have a strage problem. In my Zabbix many items are in "More than 10 minutes" queue. I tryed to increase poolers count, but no changes. What could be the problem? After I restart zabbix_server process, the queue drops, but after a while it starts to grow.

My Zabbixdatabase is Oracle and Zabbix status:

Parameter Value Details
Zabbix server is running Yes -
Number of hosts (monitored/not monitored/templates) 509 448 / 23 / 38
Number of items (monitored/disabled/not supported) 31814 24615 / 312 / 6887
Number of triggers (enabled/disabled)[true/unknown/false] 9937 6478 / 3459 [55 / 1233 / 5190]
Number of users (online) 28 2
Required server performance, new values per second 195.79908333333 -

[MrKen]

First I would click on the drop-down box in the corner which says "Overview" and select "Details". That might offer some clues.

Next I would check the zabbix_server.log for errors, timeouts, etc. Might need to set debug to 4 for a short while.

With so many different types of Items in the queue (rather than just snmp, or zabbix_agentd) I might start suspecting a mysql problem (backlog).

MrKen

[jansonz]

In Zabbix server log I see, that zabbix can't get many values.

For example SNMP:

......
Item [hostname:ifInErrors18] error: SNMP error [(noSuchName) There is no such variable name in this MIB.]]
Item [hostname:ifOutOctets43] error: SNMP error [(noSuchName) There is no such variable name in this MIB.]
......

Example from Zabbix agent:
.....
Item [hostname:mysql.qps] error: Not supported by Zabbix Agent
Item [hostname:apache_conn] error: Not supported by Zabbix Agent
Item [hostname:mysql.uptime] error: Not supported by Zabbix Agent
......


I think, that those errors are causing the queue to grow. I think the solution will be to disable all Items, which are Not Supported, and Items who are recieving errors.


It's a shame, that in Zabbix 1.8 it is not possible to disable Item, which is linked from template - this feature is not working

[untergeek]

I am suffering a similar problem and I don't know if you have found the cause.

http://www.zabbix.com/forum/showthread.php?p=56401

I, too, am using Oracle. I think that's the problem. I have nowhere near the number of items to monitor that you do, and yet with Zabbix 1.6 I didn't have any problems at all. With 1.8 I have the same problem with the queue backing up.

Somewhere I think that Oracle is simply not writing these items out fast enough to the database. Queries are working, but for some reason they're working more slowly. Could it be related to the new db caching engine and some incompatibility or bad interaction with Oracle or a version of Oracle?

[sersad]

I have a similar problem. The line is constantly growing.

Параметр 	Значение 	Детали
ZABBIX сервер запущен 	Да 	-
Количество узлов сети (контролируется/не контролируется/шаблоны/удалено) 	284 	149 / 83 / 52
Количество элементов данных (активных/неактивных/не поддерживается)[trapper] 	180405 	100540 / 38055 / 41810
Количество триггеров (активированных/деактивированных)[истина/неизвестно/ложь] 	76345 	76342 / 3  [628 / 2084 / 73630]
Количество пользователей 	6 	3
Требуемое быстродействие сервера, новые значения в секунду 	34 	-

In the different elements of the queue. The logs in debug 3 regularly appears similar. Change the number of poller from 15 to 120 to nothing lead.

top

top - 11:17:05 up 12 days, 21:38,  2 users,  load average: 0.73, 0.84, 0.85
Tasks: 391 total,   3 running, 388 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  5.5%us,  0.3%sy,  1.6%ni, 91.3%id,  0.6%wa,  0.0%hi,  0.6%si,  0.0%st
Cpu3  : 15.6%us,  2.0%sy,  0.7%ni, 80.7%id,  0.0%wa,  0.3%hi,  0.7%si,  0.0%st
Mem:  10247196k total,  9075588k used,  1171608k free,    76860k buffers
Swap: 16341404k total,   813636k used, 15527768k free,  7177204k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5163 ss        20   0 1772m 197m 2196 S   18  2.0   3594:28 tclmon
32753 postgres  20   0 5341m 1.9g 1.8g S    6 19.1  31:04.67 postgres
32744 zabbix    25   5 1739m 203m 201m S    2  2.0   9:20.04 zabbix_server
 2813 root      39  19     0    0    0 S    0  0.0  14:13.31 kipmi0
 5484 zabbix    20   0 19232 1556  988 R    0  0.0   0:00.08 top
11751 zabbix    25   5 18200  936  760 S    0  0.0   3:24.24 zabbix_agentd
11752 zabbix    25   5 18216  840  640 S    0  0.0   1:50.31 zabbix_agentd
28630 postgres  20   0 5322m 3.2g 3.2g S    0 32.9   1:44.97 postgres
28631 postgres  20   0 5322m 2756 1708 S    0  0.0   1:06.85 postgres
30065 mysql     20   0 2368m  10m 2044 S    0  0.1  19:19.52 mysqld
32480 zabbix    25   5 1739m 149m 148m S    0  1.5   0:09.02 zabbix_server
32601 postgres  20   0 5325m  38m  35m S    0  0.4   0:02.10 postgres
32606 zabbix    25   5 1739m 146m 144m S    0  1.5   0:08.11 zabbix_server
32641 zabbix    25   5 1739m 185m 184m S    0  1.9   0:11.34 zabbix_server
32644 zabbix    25   5 1739m 185m 183m S    0  1.9   0:11.13 zabbix_server
    1 root      20   0  4100  596  512 S    0  0.0   0:02.67 init
    2 root      15  -5     0    0    0 S    0  0.0   0:00.02 kthreadd
    3 root      RT  -5     0    0    0 S    0  0.0   0:02.61 migration/0
    4 root      15  -5     0    0    0 S    0  0.0   0:05.92 ksoftirqd/0

DB postgres

#------------------------------------------------------------------------------
# RESOURCE USAGE (except WAL)
#------------------------------------------------------------------------------

# - Memory -

shared_buffers = 5120MB                 # min 128kB or max_connections*16kB
                                        # (change requires restart)
temp_buffers = 256MB                    # min 800kB
#max_prepared_transactions = 5          # can be 0 or more
                                        # (change requires restart)
# Note:  Increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
work_mem = 1024MB                               # min 64kB
maintenance_work_mem = 512MB            # min 1MB
#max_stack_depth = 2MB                  # min 100kB

# - Free Space Map -

max_fsm_pages = 409600                  # min max_fsm_relations*16, 6 bytes each
                                        # (change requires restart)
max_fsm_relations = 1000                # min 100, ~70 bytes each
                                        # (change requires restart)

#------------------------------------------------------------------------------
# WRITE AHEAD LOG
#------------------------------------------------------------------------------

# - Settings -

fsync = on                              # turns forced synchronization on or off
synchronous_commit = off                # immediate fsync at commit
wal_sync_method = fsync         # the default is the first option
                                        # supported by the operating system:
                                        #   open_datasync
                                        #   fdatasync
                                        #   fsync
                                        #   fsync_writethrough
                                        #   open_sync
#full_page_writes = on                  # recover from partial page writes
wal_buffers = 1MB                       # min 32kB
                                        # (change requires restart)
#wal_writer_delay = 200ms               # 1-10000 milliseconds

commit_delay = 0                        # range 0-100000, in microseconds
commit_siblings = 20                    # range 1-1000

# - Checkpoints -

checkpoint_segments = 256               # in logfile segments, min 1, 16MB each
#checkpoint_timeout = 5min              # range 30s-1h
checkpoint_completion_target = 0.9      # checkpoint target duration, 0.0 - 1.0
#checkpoint_warning = 30s               # 0 is off


#------------------------------------------------------------------------------
# AUTOVACUUM PARAMETERS
#------------------------------------------------------------------------------

autovacuum = on                 # Enable autovacuum subprocess?  'on'
                                        # requires track_counts to also be on.
log_autovacuum_min_duration = -1        # -1 disables, 0 logs all actions and
                                        # their durations, > 0 logs only
                                        # actions running at least that time.
autovacuum_max_workers = 1              # max number of autovacuum subprocesses
autovacuum_naptime = 3min               # time between autovacuum runs
#autovacuum_vacuum_threshold = 50       # min number of row updates before
                                        # vacuum
#autovacuum_analyze_threshold = 50      # min number of row updates before
                                        # analyze
#autovacuum_vacuum_scale_factor = 0.2   # fraction of table size before vacuum
#autovacuum_analyze_scale_factor = 0.1  # fraction of table size before analyze
#autovacuum_freeze_max_age = 200000000  # maximum XID age before forced vacuum
                                        # (change requires restart)
#autovacuum_vacuum_cost_delay = 20      # default vacuum cost delay for
                                        # autovacuum, -1 means use
                                        # vacuum_cost_delay
#autovacuum_vacuum_cost_limit = -1      # default vacuum cost limit for
                                        # autovacuum, -1 means use
                                        # vacuum_cost_limit

zabbix_server.conf

StartPollers=120
StartPollersUnreachable=2
StartPingers=3
StartDiscoverers=1
HistoryCacheSize=256M
CacheSize=1024M
HistoryTextCacheSize=128M
Timeout=6

all other defaults

ОЧЕРЕДЬ ОЖИДАЮЩИХ ОБНОВЛЕНИЯ ЭЛЕМЕНТОВ ДАННЫХ 	
  	
Элементы данных 	5  	10 с 	30 c 	1 м 	5 м	Более 10 минут
ZABBIX агент 		3 	0 	1 	0 	0 	3
ZABBIX агент (ай) 	0 	0 	0 	0 	0 	0
SNMPv1 агент 		0 	0 	0 	0 	0 	0
SNMPv2 агент 		46 	38 	81 	639 	668 	13217
SNMPv3 агент 		0 	0 	0 	0 	0 	0
IPMI агент 	 	 	0 	0 	0 	0 	0 	0
SSH агент 		 	0 	0 	0 	0 	0 	0
TELNET агент 		0 	0 	0 	0 	0 	0
Простая проверка 	0 	0 	0 	0 	0 	1
ZABBIX internal 	 	0 	0 	0 	0 	0 	0
ZABBIX aggregate 	0 	0 	0 	0 	0 	0
Внешняя проверка 	0 	0 	0 	0 	0 	0

Any idea?

[MrKen]

Any idea?

Hi Sersad, I've got an idea!

I think that your problem is not the same as jansonz and untergeek. Their problem appears to be an Oracle issue, while you're using Postgresql. Also jansonz has many different types of Items in his queue, while yours is 99.9% snmpv2.

Do you use snmptraps? If Yes, perhaps try increasing the number of Trappers in zabbix_server.conf

Also, maybe you could try increasing your StartPollersUnreachable, 2 seems pretty low.

Just an idea!

MrKen

[sersad]

MrKen, thanks!
I don't use snmp traps.
I try increasing StartPollersUnreachable and see what will happen to the queue.

[sersad]

I increasing StartPollersUnreachable to 30.
The result is the same.

ZABBIX агент 	7 	14 	6 	1 	0 	2
ZABBIX (ак) 	0 	0 	0 	0 	0 	0
SNMPv1 агент 	0 	0 	0 	0 	0 	0
SNMPv2 агент 	54 	74 	123 	799 	844 	16996
SNMPv3 агент 	0 	0 	0 	0 	0 	0
IPMI агент 	 	0 	0 	0 	0 	0 	0
SSH агент 	 	0 	0 	0 	0 	0 	0
TELNET агент 	0 	0 	0 	0 	0 	0
Простая проверка 	0 	29 	15 	1 	0 	0
ZABBIX internal 	0 	1 	0 	0 	0 	0
ZABBIX aggregate 	0 	0 	0 	0 	0 	0
Внешняя проверка 	0 	0 	0 	0 	0 	0

Any idea?

[MrKen]

Hi sersad,

Just guessing, but I think that you recently upgraded to 1.8.

Perhaps upgrading net-snmp and the relevant dependencies might be worth trying. It shouldn't hurt anything .

Anything in the zabbix_server.log?

Счастливое Новый Год

MrKen

[sersad]

MrKen, no this is new install. I try this system with the alpha version.

in log (debug 3) are similar records. I simply copied the typically encountered record.

From the log shows that there is no access to some hosts, and some of an item does not exist (there is no card in the slot device). But in the review queue is and items such as irons Cisco 3560, 3400 and 2950 are connected gigabit links, and this is very strange.

insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (343449,1262268000,1,11,11,11);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (73449,1262268000,1,1015808,1015808,1015808);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (69849,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (51849,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (30249,1262268000,1,17,17,17);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44641,1262268000,1,11956224,11956224,11956224);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44642,1262268000,1,7372800,7372800,7372800);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44643,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44644,1262268000,1,8212480,8212480,8212480);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44645,1262268000,1,5005312,5005312,5005312);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44626,1262268000,1,14340096,14340096,14340096);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44646,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44619,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44647,1262268000,1,16642048,16642048,16642048);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44648,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (44649,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37441,1262268000,1,20,20,20);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37442,1262268000,1,25,25,25);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (393841,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37443,1262268000,1,17,17,17);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (393842,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37444,1262268000,1,31,31,31);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (390241,1262268000,1,14,14,14);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37445,1262268000,1,16,16,16);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (393843,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37426,1262268000,1,31,31,31);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (390242,1262268000,1,8,8,8);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37446,1262268000,1,18,18,18);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (393844,1262268000,1,8257536,8257536,8257536);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37419,1262268000,1,20,20,20);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (393845,1262268000,1,1146880,1146880,1146880);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37447,1262268000,1,14,14,14);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (390243,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (390249,1262268000,1,0,0,0);
]
  6754:20091231:182413.340 Item [110122060_batyr-p.bikshiki:4.13AdslAturCurrentRate] error: Timeout while connecting to [192.168.122.60:161]
  6754:20091231:182413.341 SNMP Host [110122060_batyr-p.bikshiki]: another network error, wait for 15 seconds
  6786:20091231:182413.490 Item [110122060_batyr-p.bikshiki:2.16AdslAtucSnr] error: Timeout while connecting to [192.168.122.60:161]
  6786:20091231:182413.491 SNMP Host [110122060_batyr-p.bikshiki]: another network error, wait for 15 seconds
  6694:20091231:182413.506 Item [110122115_kan-kvant:9.17ifDownAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.5.717] value has unknown type [0x81]
  6694:20091231:182413.629 Item [110120172_koms-urmaevo2:4.01AdslAtucMaxCurrentRate] error: SNMP error [Unknown Error]
  6694:20091231:182413.657 Item [110120056_urm-tegeshevo:2.10AdslAturSnr] error: SNMP error [Unknown Error]
  6694:20091231:182413.698 Item [110120041_kan_shibilgi:3.09AdslAtucSnr] error: SNMP error [Unknown Error]
  6694:20091231:182413.721 Item [110120034_kan-tugaevo:4.08AdslAtucAttenuation] error: SNMP error [Unknown Error]
  6868:20091231:182414.160 Item [110122060_batyr-p.bikshiki:4.03AdslAturAttenuation] error: Timeout while connecting to [192.168.122.60:161]
  6868:20091231:182414.161 SNMP Host [110122060_batyr-p.bikshiki]: another network error, wait for 15 seconds
  6750:20091231:182414.650 Item [110122060_batyr-p.bikshiki:1.16AdslAtucCurrentRate] error: Timeout while connecting to [192.168.122.60:161]
  6750:20091231:182414.651 SNMP Host [110122060_batyr-p.bikshiki]: another network error, wait for 15 seconds
  6814:20091231:182414.684 Item [110122176_koms-pochinok-bybyt:4.09AdslAtucSnr] error: SNMP error [Unknown Error]
  6862:20091231:182414.716 Item [110122058_batyr-suguty:4.09AdslAtucCurrentRate] error: SNMP error [Unknown Error]
  6691:20091231:182414.875 Item [110122176_koms-pochinok-bybyt:4.14AdslAtucSnr] error: SNMP error [Unknown Error]
  6730:20091231:182414.910 Item [110122060_batyr-p.bikshiki:1.13AdslAturCurrentRate] error: Timeout while connecting to [192.168.122.60:161]
  6730:20091231:182414.912 SNMP Host [110122060_batyr-p.bikshiki]: another network error, wait for 15 seconds
  6691:20091231:182414.935 Item [110120197_batyr-cats:9.23ifProfSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.8.1.4.723] value has unknown type [0x81]
  6691:20091231:182414.947 Item [110120197_batyr-cats:8.15ifUpOutputPowerADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.7.615] value has unknown type [0x81]
  6695:20091231:182415.017 Item [110120172_koms-urmaevo2:4.01AdslAtucSnr] error: SNMP error [Unknown Error]
  6695:20091231:182415.044 Item [110120114_jalch-polev-pinery:4.16AdslAtucAttenuation] error: SNMP error [Unknown Error]
  6691:20091231:182415.071 Item [110122056_shem-cats.1:9.18ifDownSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.11.1.2.718] value has unknown type [0x81]
  6695:20091231:182415.080 Item [110120041_kan_shibilgi:3.09AdslAturAttenuation] error: SNMP error [Unknown Error]
  6691:20091231:182415.083 Item [110122056_shem-cats.1:8.15ifUpSnrADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.4.615] value has unknown type [0x81]
  6683:20091231:182420.357 Item [110122176_koms-pochinok-bybyt:4.13AdslAturAttenuation] error: Timeout while connecting to [192.168.122.176:161]
  6683:20091231:182420.359 SNMP Host [110122176_koms-pochinok-bybyt]: another network error, wait for 15 seconds
  6696:20091231:182420.387 Item [110120197_batyr-cats:9.28ifProfSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.8.1.4.728] value has unknown type [0x81]
  6696:20091231:182420.399 Item [110120197_batyr-cats:8.18ifDownOutputPowerADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.7.618] value has unknown type [0x81]
  6683:20091231:182420.442 Item [110122113_urm-chelkasy:4.05AdslAturCurrentRate] error: SNMP error [Unknown Error]
  6696:20091231:182420.487 Item [110122056_shem-cats.1:9.01ifUpSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.8.701] value has unknown type [0x81]
  6696:20091231:182420.500 Item [110122056_shem-cats.1:8.18ifDownSnrADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.4.618] value has unknown type [0x81]
  6696:20091231:182420.566 Item [110122042_kan-vostoch:10.18ifDownAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.5.818] value has unknown type [0x81]
  6696:20091231:182420.625 Item [110122051_kan-shihazani:7.18ifDownAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.5.518] value has unknown type [0x81]
  6696:20091231:182420.636 Item [110122115_kan-kvant:9.18ifDownAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.5.718] value has unknown type [0x81]
  6750:20091231:182420.690 Item [KANASH-C2950:lmemFreeMem] error: Timeout while connecting to [192.168.123.93:161]
  6750:20091231:182420.692 SNMP Host [KANASH-C2950]: first network error, wait for 15 seconds

insert into history (itemid,clock,value) values (18443,1262273064,58382.454545);
insert into history (itemid,clock,value) values (18499,1262273065,97.675055);
insert into history (itemid,clock,value) values (18442,1262273065,1139077.612903);
insert into history (itemid,clock,value) values (18440,1262273065,23376.600000);
insert into history (itemid,clock,value) values (18469,1262273065,1.810000);
insert into history (itemid,clock,value) values (18474,1262273065,0.993804);
insert into history (itemid,clock,value) values (18445,1262273066,1122361.181818);
insert into history (itemid,clock,value) values (18505,1262273066,43.793950);
insert into history (itemid,clock,value) values (18529,1262273066,43.793950);
insert into history (itemid,clock,value) values (503665,1262273066,10000000000000.000000);
insert into history (itemid,clock,value) values (18506,1262273066,56.206059);
insert into history (itemid,clock,value) values (18473,1262273067,25.970894);
insert into history (itemid,clock,value) values (18524,1262273067,43.793941);
insert into history (itemid,clock,value) values (18444,1262273067,0.000000);
insert into history (itemid,clock,value) values (18467,1262273067,1.570000);
insert into history (itemid,clock,value) values (503652,1262273068,100000000000000.000000);
insert into history (itemid,clock,value) values (18478,1262273068,82.792054);
insert into history_uint (itemid,clock,value) values (404658,1262273063,12);
insert into history_uint (itemid,clock,value) values (383058,1262273063,18);
insert into history_uint (itemid,clock,value) values (404663,1262273063,19);
insert into history_uint (itemid,clock,value) values (379458,1262273063,0);
insert into history_uint (itemid,clock,value) values (375858,1262273063,15);
insert into history_uint (itemid,clock,value) values (386663,1262273063,5996544);
insert into history_uint (itemid,clock,value) values (451428,1262273063,0);
insert into history_uint (itemid,clock,value) values (383063,1262273063,0);
insert into history_uint (itemid,clock,value) values (365058,1262273063,0);
insert into history_uint (itemid,clock,value) values (357858,1262273063,0);
insert into history_uint (itemid,clock,value) values (379463,1262273063,0);
insert into history_uint (itemid,clock,value) values (343458,1262273063,13);

insert into history_str (itemid,clock,value) values (498268,1262273068,'Uplink to batyr-C3560');
]
  6984:20091231:182428.738 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR:  current transaction is aborted, commands ignored until end of transaction block
 [select distinct i.itemid,i.key_,h.host,h.port,i.delay,i.description,i.type,h.useip,h.ip,i.history,i.lastvalue,i.prevvalue,i.hostid,i.value_type,i.delta,i.prevorgvalue,i.last$
  6984:20091231:182428.739 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR:  current transaction is aborted, commands ignored until end of transaction block
 [select itemid,num,value_min,value_avg,value_max from trends_uint where clock=1262268000 and itemid in (30256,30257,30258,30259,30260,30262,30263,30264,30265,30266,30267,3745$
  6984:20091231:182428.741 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR:  current transaction is aborted, commands ignored until end of transaction block
 [insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (404658,1262268000,1,12,12,12);


insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (37467,1262268000,1,20,20,20);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (393867,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (390267,1262268000,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (30267,1262268000,1,43,43,43);
]
  6984:20091231:182428.742 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR:  current transaction is aborted, commands ignored until end of transaction block
 [select itemid,num,value_min,value_avg,value_max from trends_uint where clock=1262264400 and itemid in (422656,422659,422660,422661,422664,422665,422666,429649,437056,437059,$
  6984:20091231:182428.742 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR:  current transaction is aborted, commands ignored until end of transaction block
 [insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (451428,1262264400,1,7,7,7);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (451454,1262264400,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (443986,1262264400,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (422656,1262264400,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (437056,1262264400,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (458656,1262264400,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (443829,1262264400,1,0,0,0);


insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (437064,1262264400,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (422661,1262264400,1,0,0,0);
insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (473065,1262264400,1,637000,637000,637000);
]
  6984:20091231:182428.743 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR:  current transaction is aborted, commands ignored until end of transaction block
 [select itemid,num,value_min,value_avg,value_max from trends where clock=1262268000 and itemid in (500064)]
  6984:20091231:182428.743 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR:  current transaction is aborted, commands ignored until end of transaction block
 [insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (500064,1262268000,2,0.000000,0.000000,0.000000);
]
  6709:20091231:182428.744 Item [110120197_batyr-cats:9.03ifProfSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.8.1.4.703] value has unknown type [0x81]
  6709:20091231:182428.756 Item [110120197_batyr-cats:8.23ifUpOutputPowerADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.7.623] value has unknown type [0x81]
  6741:20091231:182428.780 Item [110122209_batyr-n.shigaly:2.08AdslAtucSnr] error: Timeout while connecting to [192.168.122.209:161]
  6741:20091231:182428.781 SNMP Host [110122209_batyr-n.shigaly]: another network error, wait for 15 seconds
  6693:20091231:182428.788 Item [110120197_batyr-cats:9.25ifProfSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.8.1.4.725] value has unknown type [0x81]
  6693:20091231:182428.800 Item [110120197_batyr-cats:8.16ifUpOutputPowerADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.7.616] value has unknown type [0x81]
  6709:20091231:182428.827 Item [110122056_shem-cats.1:9.26ifDownSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.11.1.2.726] value has unknown type [0x81]
  6709:20091231:182428.839 Item [110122056_shem-cats.1:8.23ifUpSnrADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.4.623] value has unknown type [0x81]
  6693:20091231:182428.867 Item [110122056_shem-cats.1:9.19ifDownSpeedADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.11.1.2.719] value has unknown type [0x81]
  6693:20091231:182428.879 Item [110122056_shem-cats.1:8.16ifUpSnrADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.10.1.4.616] value has unknown type [0x81]
  6709:20091231:182428.908 Item [110122042_kan-vostoch:10.23ifUpAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.5.823] value has unknown type [0x81]
  6693:20091231:182428.964 Item [110122042_kan-vostoch:10.16ifUpAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.5.816] value has unknown type [0x81]
  6709:20091231:182428.975 Item [110122051_kan-shihazani:7.23ifUpAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1.16.1.5.523] value has unknown type [0x81]
  6693:20091231:182429.022 Item [110122051_kan-shihazani:7.16ifUpAttenuationADSL] error: OID [1.3.6.1.4.1.231.7.1.2.2.1.6.1.1

It is not clear why some elements of the poll be postponed, even elements of zabbix agent located on the server and simple checks.
Perhaps bottleneck db postgres?
vacuum verbose analyze; does not give any hint

[sersad]

Happy new year!

[igor]

I have a similar problem. The line is constantly growing.

Параметр 	Значение 	Детали
ZABBIX сервер запущен 	Да 	-
Количество узлов сети (контролируется/не контролируется/шаблоны/удалено) 	284 	149 / 83 / 52
Количество элементов данных (активных/неактивных/не поддерживается)[trapper] 	180405 	100540 / 38055 / 41810
Количество триггеров (активированных/деактивированных)[истина/неизвестно/ложь] 	76345 	76342 / 3  [628 / 2084 / 73630]
Количество пользователей 	6 	3
Требуемое быстродействие сервера, новые значения в секунду 	34 	-

In the different elements of the queue. The logs in debug 3 regularly appears similar. Change the number of poller from 15 to 120 to nothing lead.

top

top - 11:17:05 up 12 days, 21:38,  2 users,  load average: 0.73, 0.84, 0.85
Tasks: 391 total,   3 running, 388 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  5.5%us,  0.3%sy,  1.6%ni, 91.3%id,  0.6%wa,  0.0%hi,  0.6%si,  0.0%st
Cpu3  : 15.6%us,  2.0%sy,  0.7%ni, 80.7%id,  0.0%wa,  0.3%hi,  0.7%si,  0.0%st
Mem:  10247196k total,  9075588k used,  1171608k free,    76860k buffers
Swap: 16341404k total,   813636k used, 15527768k free,  7177204k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5163 ss        20   0 1772m 197m 2196 S   18  2.0   3594:28 tclmon
32753 postgres  20   0 5341m 1.9g 1.8g S    6 19.1  31:04.67 postgres
32744 zabbix    25   5 1739m 203m 201m S    2  2.0   9:20.04 zabbix_server
 2813 root      39  19     0    0    0 S    0  0.0  14:13.31 kipmi0
 5484 zabbix    20   0 19232 1556  988 R    0  0.0   0:00.08 top
11751 zabbix    25   5 18200  936  760 S    0  0.0   3:24.24 zabbix_agentd
11752 zabbix    25   5 18216  840  640 S    0  0.0   1:50.31 zabbix_agentd
28630 postgres  20   0 5322m 3.2g 3.2g S    0 32.9   1:44.97 postgres
28631 postgres  20   0 5322m 2756 1708 S    0  0.0   1:06.85 postgres
30065 mysql     20   0 2368m  10m 2044 S    0  0.1  19:19.52 mysqld
32480 zabbix    25   5 1739m 149m 148m S    0  1.5   0:09.02 zabbix_server
32601 postgres  20   0 5325m  38m  35m S    0  0.4   0:02.10 postgres
32606 zabbix    25   5 1739m 146m 144m S    0  1.5   0:08.11 zabbix_server
32641 zabbix    25   5 1739m 185m 184m S    0  1.9   0:11.34 zabbix_server
32644 zabbix    25   5 1739m 185m 183m S    0  1.9   0:11.13 zabbix_server
    1 root      20   0  4100  596  512 S    0  0.0   0:02.67 init
    2 root      15  -5     0    0    0 S    0  0.0   0:00.02 kthreadd
    3 root      RT  -5     0    0    0 S    0  0.0   0:02.61 migration/0
    4 root      15  -5     0    0    0 S    0  0.0   0:05.92 ksoftirqd/0

DB postgres

#------------------------------------------------------------------------------
# RESOURCE USAGE (except WAL)
#------------------------------------------------------------------------------

# - Memory -

shared_buffers = 5120MB                 # min 128kB or max_connections*16kB
                                        # (change requires restart)
temp_buffers = 256MB                    # min 800kB
#max_prepared_transactions = 5          # can be 0 or more
                                        # (change requires restart)
# Note:  Increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
work_mem = 1024MB                               # min 64kB
maintenance_work_mem = 512MB            # min 1MB
#max_stack_depth = 2MB                  # min 100kB

# - Free Space Map -

max_fsm_pages = 409600                  # min max_fsm_relations*16, 6 bytes each
                                        # (change requires restart)
max_fsm_relations = 1000                # min 100, ~70 bytes each
                                        # (change requires restart)

#------------------------------------------------------------------------------
# WRITE AHEAD LOG
#------------------------------------------------------------------------------

# - Settings -

fsync = on                              # turns forced synchronization on or off
synchronous_commit = off                # immediate fsync at commit
wal_sync_method = fsync         # the default is the first option
                                        # supported by the operating system:
                                        #   open_datasync
                                        #   fdatasync
                                        #   fsync
                                        #   fsync_writethrough
                                        #   open_sync
#full_page_writes = on                  # recover from partial page writes
wal_buffers = 1MB                       # min 32kB
                                        # (change requires restart)
#wal_writer_delay = 200ms               # 1-10000 milliseconds

commit_delay = 0                        # range 0-100000, in microseconds
commit_siblings = 20                    # range 1-1000

# - Checkpoints -

checkpoint_segments = 256               # in logfile segments, min 1, 16MB each
#checkpoint_timeout = 5min              # range 30s-1h
checkpoint_completion_target = 0.9      # checkpoint target duration, 0.0 - 1.0
#checkpoint_warning = 30s               # 0 is off


#------------------------------------------------------------------------------
# AUTOVACUUM PARAMETERS
#------------------------------------------------------------------------------

autovacuum = on                 # Enable autovacuum subprocess?  'on'
                                        # requires track_counts to also be on.
log_autovacuum_min_duration = -1        # -1 disables, 0 logs all actions and
                                        # their durations, > 0 logs only
                                        # actions running at least that time.
autovacuum_max_workers = 1              # max number of autovacuum subprocesses
autovacuum_naptime = 3min               # time between autovacuum runs
#autovacuum_vacuum_threshold = 50       # min number of row updates before
                                        # vacuum
#autovacuum_analyze_threshold = 50      # min number of row updates before
                                        # analyze
#autovacuum_vacuum_scale_factor = 0.2   # fraction of table size before vacuum
#autovacuum_analyze_scale_factor = 0.1  # fraction of table size before analyze
#autovacuum_freeze_max_age = 200000000  # maximum XID age before forced vacuum
                                        # (change requires restart)
#autovacuum_vacuum_cost_delay = 20      # default vacuum cost delay for
                                        # autovacuum, -1 means use
                                        # vacuum_cost_delay
#autovacuum_vacuum_cost_limit = -1      # default vacuum cost limit for
                                        # autovacuum, -1 means use
                                        # vacuum_cost_limit

zabbix_server.conf

StartPollers=120
StartPollersUnreachable=2
StartPingers=3
StartDiscoverers=1
HistoryCacheSize=256M
CacheSize=1024M
HistoryTextCacheSize=128M
Timeout=6

all other defaults

ОЧЕРЕДЬ ОЖИДАЮЩИХ ОБНОВЛЕНИЯ ЭЛЕМЕНТОВ ДАННЫХ 	
  	
Элементы данных 	5  	10 с 	30 c 	1 м 	5 м	Более 10 минут
ZABBIX агент 		3 	0 	1 	0 	0 	3
ZABBIX агент (ай) 	0 	0 	0 	0 	0 	0
SNMPv1 агент 		0 	0 	0 	0 	0 	0
SNMPv2 агент 		46 	38 	81 	639 	668 	13217
SNMPv3 агент 		0 	0 	0 	0 	0 	0
IPMI агент 	 	 	0 	0 	0 	0 	0 	0
SSH агент 		 	0 	0 	0 	0 	0 	0
TELNET агент 		0 	0 	0 	0 	0 	0
Простая проверка 	0 	0 	0 	0 	0 	1
ZABBIX internal 	 	0 	0 	0 	0 	0 	0
ZABBIX aggregate 	0 	0 	0 	0 	0 	0
Внешняя проверка 	0 	0 	0 	0 	0 	0

Any idea?

Hi!
Seems that your problem appears because of this numeric field "overflow" error:

Line 207177: 3286:20100102:155134.118 [Z3005] Query failed: [0] PGRES_FATAL_ERROR:ERROR: numeric field overflow
DETAIL: A field with precision 16, scale 4 must round to an absolute value less than 10^12.
[insert into history (itemid,clock,value) values (503490,1262436691,100000000000000.000000);

Appropriate issue in JIRA: https://support.zabbix.com/browse/ZBX-1636

[sersad]

I increase history.value to 20,4 and increase trends.value_min trends.value_avg trends.value_max to 20,4.
After this problem with increasing the queue has disappeared

[bee]

Hi sersad,
How to change the value of history.value, trends.value_min, trends.value_avg and trends.value_max

Thanks

[sersad]

bee, I changed these values with phppgadmin.