zabbix_agentd hanging system since upgrade to 1.8.2 ?

[elvar]

Hello, I recently upgraded to 1.8.2 from 1.8.1 and since having done so I've twice witnessed what appears to be the zabbix agent hanging my system. I'm running Ubuntu Server 10.04 64bit. Here is what I see in the console when it's hung up...

[1134600.951394] INFO: task zabbix_agentd:910 blocked for more than 120 seconds.
[1134600.951403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.


Anyone else experiencing this or have any ideas?


Regards,

[Murz]

Got the same problem today (at first time, zabbix 1.8.2 installed from 2010-06-17).
Server quickly halts (reply to ping, but I can't connect via ssh, or other ports) and I can't remotely take access to it, solved by hardware restart.
In /var/log/kernel.log I see:

Jul 26 16:26:44 sea kernel: [2094840.504038] INFO: task zabbix_agentd:1625 blocked for more than 120 seconds.
Jul 26 16:26:44 sea kernel: [2094840.504121] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 26 16:26:44 sea kernel: [2094840.504203] zabbix_agentd D 0000000000000000     0  1625   1615 0x00000000
Jul 26 16:26:44 sea kernel: [2094840.504262]  ffff88011fa5b880 0000000000000082 0000000000000000 0000000281193c8a
Jul 26 16:26:44 sea kernel: [2094840.504352]  0000000000000000 000000000000f8a0 ffff88011c1bffd8 0000000000015640
Jul 26 16:26:44 sea kernel: [2094840.504442]  0000000000015640 ffff88011c26b170 ffff88011c26b468 00000001ffffffff
Jul 26 16:26:44 sea kernel: [2094840.504533] Call Trace:
Jul 26 16:26:44 sea kernel: [2094840.504578]  [<ffffffff81192342>] ? number+0x132/0x22e
Jul 26 16:26:44 sea kernel: [2094840.504628]  [<ffffffff812f9c82>] ? __mutex_lock_common+0x10b/0x17b
Jul 26 16:26:44 sea kernel: [2094840.504680]  [<ffffffff812f9daa>] ? mutex_lock+0x1a/0x31
Jul 26 16:26:44 sea kernel: [2094840.504730]  [<ffffffff810f5604>] ? do_lookup+0x80/0x15d
Jul 26 16:26:44 sea kernel: [2094840.504779]  [<ffffffff810f6084>] ? __link_path_walk+0x5a5/0x6f5
Jul 26 16:26:44 sea kernel: [2094840.504830]  [<ffffffff810f6402>] ? path_walk+0x66/0xc9
Jul 26 16:26:44 sea kernel: [2094840.504879]  [<ffffffff810f786c>] ? do_path_lookup+0x20/0x77
Jul 26 16:26:44 sea kernel: [2094840.504929]  [<ffffffff810f8d4e>] ? user_path_at+0x48/0x79
Jul 26 16:26:45 sea kernel: [2094840.504979]  [<ffffffff810f110b>] ? cp_new_stat+0xe9/0xfc
Jul 26 16:26:45 sea kernel: [2094840.505029]  [<ffffffff810f12d2>] ? vfs_fstatat+0x2c/0x57
Jul 26 16:26:45 sea kernel: [2094840.505078]  [<ffffffff810f13c5>] ? sys_newstat+0x11/0x30
Jul 26 16:26:48 sea kernel: [2094840.505128]  [<ffffffff810e4e35>] ? virt_to_head_page+0x9/0x2a
Jul 26 16:26:49 sea kernel: [2094840.505179]  [<ffffffff810ec4ce>] ? do_sys_open+0xec/0xfc
Jul 26 16:26:49 sea kernel: [2094840.505230]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Jul 26 16:26:49 sea kernel: [2094840.505305] INFO: task zabbix_server:3917 blocked for more than 120 seconds.
Jul 26 16:26:49 sea kernel: [2094840.505384] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 26 16:26:49 sea kernel: [2094840.505466] zabbix_server D 0000000000000002     0  3917   3894 0x00000000
Jul 26 16:26:49 sea kernel: [2094840.505524]  ffff88011faf0e20 0000000000000082 0000000000000000 00800000810b990e
Jul 26 16:26:49 sea kernel: [2094840.505614]  00000000036475f9 000000000000f8a0 ffff8800c05f5fd8 0000000000015640
Jul 26 16:26:49 sea kernel: [2094840.505704]  0000000000015640 ffff8800c047c6a0 ffff8800c047c998 00000000c05f5a98
Jul 26 16:26:49 sea kernel: [2094840.505794] Call Trace:
Jul 26 16:26:49 sea kernel: [2094840.505834]  [<ffffffff810170dd>] ? read_tsc+0xa/0x20
Jul 26 16:26:49 sea kernel: [2094840.505884]  [<ffffffff810b3885>] ? sync_page+0x0/0x46
Jul 26 16:26:49 sea kernel: [2094840.505932]  [<ffffffff812f9558>] ? io_schedule+0x73/0xb7
Jul 26 16:26:49 sea kernel: [2094840.505981]  [<ffffffff810b38c6>] ? sync_page+0x41/0x46
Jul 26 16:26:49 sea kernel: [2094840.506029]  [<ffffffff812f9a65>] ? __wait_on_bit+0x41/0x70
Jul 26 16:26:49 sea kernel: [2094840.506079]  [<ffffffff810b3a4a>] ? wait_on_page_bit+0x6b/0x71
Jul 26 16:26:49 sea kernel: [2094840.506130]  [<ffffffff81064b14>] ? wake_bit_function+0x0/0x23
Jul 26 16:26:49 sea kernel: [2094840.506182]  [<ffffffff810d7de3>] ? lookup_swap_cache+0x13/0x28
Jul 26 16:26:49 sea kernel: [2094840.506234]  [<ffffffff810c038b>] ? shmem_getpage+0x2b8/0x6c9
Jul 26 16:26:49 sea kernel: [2094840.506285]  [<ffffffff8100f6c4>] ? __switch_to+0x1ad/0x297
Jul 26 16:26:49 sea kernel: [2094840.506337]  [<ffffffff81047f49>] ? finish_task_switch+0x3a/0xa7
Jul 26 16:26:49 sea kernel: [2094840.506388]  [<ffffffff812f947e>] ? thread_return+0x79/0xe0
Jul 26 16:26:49 sea kernel: [2094840.506439]  [<ffffffff81144f2a>] ? ipc_lock+0x2a/0x42
Jul 26 16:26:49 sea kernel: [2094840.506487]  [<ffffffff81146fbd>] ? sys_semtimedop+0x451/0x619
Jul 26 16:26:49 sea kernel: [2094840.506538]  [<ffffffff810c07d6>] ? shmem_fault+0x3a/0x5f
Jul 26 16:26:49 sea kernel: [2094840.506589]  [<ffffffff810c9cca>] ? __do_fault+0x54/0x3c3
Jul 26 16:26:49 sea kernel: [2094840.506637]  [<ffffffff812f999f>] ? __wait_on_bit_lock+0x76/0x84
Jul 26 16:26:49 sea kernel: [2094840.506689]  [<ffffffff810cbf82>] ? handle_mm_fault+0x351/0x7a5
Jul 26 16:26:49 sea kernel: [2094840.506741]  [<ffffffff8103283b>] ? do_page_fault+0x266/0x282
Jul 26 16:26:49 sea kernel: [2094840.506792]  [<ffffffff812fab85>] ? page_fault+0x25/0x30
Jul 26 16:26:50 sea kernel: [2094840.506863] INFO: task mysqld:14020 blocked for more than 120 seconds.
Jul 26 16:26:50 sea kernel: [2094840.506915] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 26 16:26:50 sea kernel: [2094840.506997] mysqld        D 0000000000000002     0 14020  23599 0x00000000
Jul 26 16:26:50 sea kernel: [2094840.507054]  ffff88011faf0e20 0000000000000082 0000000000000000 ffffffff00000001
Jul 26 16:26:50 sea kernel: [2094840.507145]  ffff88000540fac0 000000000000f8a0 ffff88000cb1dfd8 0000000000015640
Jul 26 16:26:50 sea kernel: [2094840.507235]  0000000000015640 ffff88008dce69f0 ffff88008dce6ce8 000000001c3411a0
Jul 26 16:26:50 sea kernel: [2094840.507325] Call Trace:
Jul 26 16:26:50 sea kernel: [2094840.507364]  [<ffffffff810170dd>] ? read_tsc+0xa/0x20
Jul 26 16:26:50 sea kernel: [2094840.507412]  [<ffffffff810b3885>] ? sync_page+0x0/0x46
Jul 26 16:26:50 sea kernel: [2094840.507460]  [<ffffffff812f9558>] ? io_schedule+0x73/0xb7
Jul 26 16:26:50 sea kernel: [2094840.507509]  [<ffffffff810b38c6>] ? sync_page+0x41/0x46
Jul 26 16:26:50 sea kernel: [2094840.507558]  [<ffffffff812f9a65>] ? __wait_on_bit+0x41/0x70
Jul 26 16:26:50 sea kernel: [2094840.507608]  [<ffffffff810b3a4a>] ? wait_on_page_bit+0x6b/0x71
Jul 26 16:26:50 sea kernel: [2094840.507658]  [<ffffffff81064b14>] ? wake_bit_function+0x0/0x23
Jul 26 16:26:50 sea kernel: [2094840.507709]  [<ffffffff810bd9cf>] ? shrink_page_list+0x3fb/0x623
Jul 26 16:26:50 sea kernel: [2094840.515600]  [<ffffffff8106c015>] ? ktime_get_ts+0x68/0xb2
Jul 26 16:26:50 sea kernel: [2094840.515652]  [<ffffffff81099412>] ? delayacct_end+0x74/0x7f
Jul 26 16:26:50 sea kernel: [2094840.515703]  [<ffffffff81064ae6>] ? autoremove_wake_function+0x0/0x2e
Jul 26 16:26:50 sea kernel: [2094840.515757]  [<ffffffff810be3cd>] ? shrink_list+0x4fd/0x731
Jul 26 16:26:50 sea kernel: [2094840.515808]  [<ffffffff810414e4>] ? select_task_rq_fair+0x4fd/0x8ca
Jul 26 16:26:50 sea kernel: [2094840.515916]  [<ffffffff810be881>] ? shrink_zone+0x280/0x342
Jul 26 16:26:50 sea kernel: [2094840.515967]  [<ffffffff8103a4f7>] ? activate_task+0x20/0x26
Jul 26 16:26:50 sea kernel: [2094840.516022]  [<ffffffff810bf945>] ? try_to_free_pages+0x232/0x38e
Jul 26 16:26:50 sea kernel: [2094840.516075]  [<ffffffff810bc973>] ? isolate_pages_global+0x0/0x20f
Jul 26 16:26:50 sea kernel: [2094840.516127]  [<ffffffff810b9a1f>] ? __alloc_pages_nodemask+0x3bb/0x5d0
Jul 26 16:26:50 sea kernel: [2094840.516182]  [<ffffffff810e5bc9>] ? new_slab+0x42/0x1ca
Jul 26 16:26:50 sea kernel: [2094840.516230]  [<ffffffff810e5f41>] ? __slab_alloc+0x1f0/0x39b
Jul 26 16:26:50 sea kernel: [2094840.516282]  [<ffffffff8124214d>] ? sock_alloc_send_pskb+0xbd/0x2d8
Jul 26 16:26:50 sea kernel: [2094840.516335]  [<ffffffff812469c1>] ? __alloc_skb+0x3e/0x15a
Jul 26 16:26:50 sea kernel: [2094840.516385]  [<ffffffff810e6973>] ? __kmalloc_node_track_caller+0xbb/0x11b
Jul 26 16:26:50 sea kernel: [2094840.516439]  [<ffffffff8124214d>] ? sock_alloc_send_pskb+0xbd/0x2d8
Jul 26 16:26:50 sea kernel: [2094840.516491]  [<ffffffff812469ec>] ? __alloc_skb+0x69/0x15a
Jul 26 16:26:50 sea kernel: [2094840.516540]  [<ffffffff8124214d>] ? sock_alloc_send_pskb+0xbd/0x2d8
Jul 26 16:26:50 sea kernel: [2094840.516593]  [<ffffffff8103f9c3>] ? __wake_up_sync_key+0x3a/0x56
Jul 26 16:26:50 sea kernel: [2094840.516645]  [<ffffffff812b1a01>] ? unix_stream_sendmsg+0x133/0x2a1
Jul 26 16:26:50 sea kernel: [2094840.516698]  [<ffffffff8123ec4c>] ? sock_aio_write+0xb1/0xbc
Jul 26 16:26:50 sea kernel: [2094840.516748]  [<ffffffff8107185f>] ? futex_wake+0xd6/0xe7
Jul 26 16:26:50 sea kernel: [2094840.516798]  [<ffffffff810edc7e>] ? do_sync_write+0xce/0x113
Jul 26 16:26:50 sea kernel: [2094840.516849]  [<ffffffff81064ae6>] ? autoremove_wake_function+0x0/0x2e
Jul 26 16:26:50 sea kernel: [2094840.516902]  [<ffffffff810ee609>] ? vfs_write+0xbc/0x102
Jul 26 16:26:50 sea kernel: [2094840.516951]  [<ffffffff810ee70b>] ? sys_write+0x45/0x6e
Jul 26 16:26:50 sea kernel: [2094840.517001]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Jul 26 16:28:39 sea kernel: [2094960.516022] INFO: task zabbix_agentd:1625 blocked for more than 120 seconds.
Jul 26 16:28:39 sea kernel: [2094960.516104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 26 16:28:39 sea kernel: [2094960.516185] zabbix_agentd D 0000000000000000     0  1625   1615 0x00000000
Jul 26 16:28:39 sea kernel: [2094960.516243]  ffff88011fa5b880 0000000000000082 0000000000000000 0000000281193c8a
Jul 26 16:28:39 sea kernel: [2094960.516333]  0000000000000000 000000000000f8a0 ffff88011c1bffd8 0000000000015640
Jul 26 16:28:39 sea kernel: [2094960.516422]  0000000000015640 ffff88011c26b170 ffff88011c26b468 00000001ffffffff
Jul 26 16:28:39 sea kernel: [2094960.516512] Call Trace:
Jul 26 16:28:39 sea kernel: [2094960.516556]  [<ffffffff81192342>] ? number+0x132/0x22e
Jul 26 16:28:39 sea kernel: [2094960.516605]  [<ffffffff812f9c82>] ? __mutex_lock_common+0x10b/0x17b
Jul 26 16:28:39 sea kernel: [2094960.516657]  [<ffffffff812f9daa>] ? mutex_lock+0x1a/0x31
Jul 26 16:28:39 sea kernel: [2094960.516707]  [<ffffffff810f5604>] ? do_lookup+0x80/0x15d
Jul 26 16:28:39 sea kernel: [2094960.516755]  [<ffffffff810f6084>] ? __link_path_walk+0x5a5/0x6f5
Jul 26 16:28:39 sea kernel: [2094960.516806]  [<ffffffff810f6402>] ? path_walk+0x66/0xc9
Jul 26 16:28:39 sea kernel: [2094960.516855]  [<ffffffff810f786c>] ? do_path_lookup+0x20/0x77
Jul 26 16:28:39 sea kernel: [2094960.516905]  [<ffffffff810f8d4e>] ? user_path_at+0x48/0x79
Jul 26 16:28:39 sea kernel: [2094960.516955]  [<ffffffff810f110b>] ? cp_new_stat+0xe9/0xfc
Jul 26 16:28:39 sea kernel: [2094960.517004]  [<ffffffff810f12d2>] ? vfs_fstatat+0x2c/0x57
Jul 26 16:28:39 sea kernel: [2094960.517053]  [<ffffffff810f13c5>] ? sys_newstat+0x11/0x30
Jul 26 16:28:39 sea kernel: [2094960.517103]  [<ffffffff810e4e35>] ? virt_to_head_page+0x9/0x2a
Jul 26 16:28:39 sea kernel: [2094960.517153]  [<ffffffff810ec4ce>] ? do_sys_open+0xec/0xfc
Jul 26 16:28:39 sea kernel: [2094960.517204]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b
Jul 26 16:28:39 sea kernel: [2094960.517255] INFO: task zabbix_agentd:1628 blocked for more than 120 seconds.
Jul 26 16:28:39 sea kernel: [2094960.517334] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 26 16:28:39 sea kernel: [2094960.517416] zabbix_agentd D ffff880005495640     0  1628   1615 0x00000000
Jul 26 16:28:39 sea kernel: [2094960.517473]  ffff8800d8bb3170 0000000000000082 0000000000000000 ffffffff810b5270
Jul 26 16:28:39 sea kernel: [2094960.517562]  00000000000000c0 000000000000f8a0 ffff88011d0cdfd8 0000000000015640
Jul 26 16:28:39 sea kernel: [2094960.517652]  0000000000015640 ffff88011c26e2e0 ffff88011c26e5d8 000000011c3411a0
Jul 26 16:28:39 sea kernel: [2094960.517741] Call Trace:
Jul 26 16:28:39 sea kernel: [2094960.517781]  [<ffffffff810b5270>] ? mempool_alloc+0x55/0x106
Jul 26 16:28:39 sea kernel: [2094960.517832]  [<ffffffff810170dd>] ? read_tsc+0xa/0x20
Jul 26 16:28:39 sea kernel: [2094960.517880]  [<ffffffff810b3885>] ? sync_page+0x0/0x46
Jul 26 16:28:39 sea kernel: [2094960.517928]  [<ffffffff812f9558>] ? io_schedule+0x73/0xb7
Jul 26 16:28:39 sea kernel: [2094960.517977]  [<ffffffff810b38c6>] ? sync_page+0x41/0x46
Jul 26 16:28:39 sea kernel: [2094960.518025]  [<ffffffff812f9a65>] ? __wait_on_bit+0x41/0x70
Jul 26 16:28:39 sea kernel: [2094960.518075]  [<ffffffff810b3a4a>] ? wait_on_page_bit+0x6b/0x71
Jul 26 16:28:39 sea kernel: [2094960.518126]  [<ffffffff81064b14>] ? wake_bit_function+0x0/0x23
Jul 26 16:28:39 sea kernel: [2094960.518177]  [<ffffffff810bd9cf>] ? shrink_page_list+0x3fb/0x623
Jul 26 16:28:39 sea kernel: [2094960.518229]  [<ffffffff8105a545>] ? del_timer_sync+0xc/0x16
Jul 26 16:28:39 sea kernel: [2094960.518278]  [<ffffffff810170dd>] ? read_tsc+0xa/0x20
Jul 26 16:28:39 sea kernel: [2094960.518326]  [<ffffffff812f98ba>] ? schedule_timeout+0xad/0xdd
Jul 26 16:28:39 sea kernel: [2094960.518377]  [<ffffffff8106c015>] ? ktime_get_ts+0x68/0xb2
Jul 26 16:28:39 sea kernel: [2094960.518426]  [<ffffffff81099412>] ? delayacct_end+0x74/0x7f
Jul 26 16:28:39 sea kernel: [2094960.518476]  [<ffffffff81064ae6>] ? autoremove_wake_function+0x0/0x2e
Jul 26 16:28:39 sea kernel: [2094960.518528]  [<ffffffff810be3cd>] ? shrink_list+0x4fd/0x731
Jul 26 16:28:39 sea kernel: [2094960.518579]  [<ffffffff810ba558>] ? determine_dirtyable_memory+0xd/0x1d
Jul 26 16:28:39 sea kernel: [2094960.518632]  [<ffffffff810ba5d0>] ? get_dirty_limits+0x1d/0x259
Jul 26 16:28:39 sea kernel: [2094960.518683]  [<ffffffff810be881>] ? shrink_zone+0x280/0x342
Jul 26 16:28:39 sea kernel: [2094960.518733]  [<ffffffff810bf945>] ? try_to_free_pages+0x232/0x38e
Jul 26 16:28:39 sea kernel: [2094960.518785]  [<ffffffff810bc973>] ? isolate_pages_global+0x0/0x20f
Jul 26 16:28:39 sea kernel: [2094960.518837]  [<ffffffff810b9a1f>] ? __alloc_pages_nodemask+0x3bb/0x5d0
Jul 26 16:28:39 sea kernel: [2094960.518890]  [<ffffffff810e5bc9>] ? new_slab+0x42/0x1ca
Jul 26 16:28:39 sea kernel: [2094960.518938]  [<ffffffff810e5f41>] ? __slab_alloc+0x1f0/0x39b
Jul 26 16:28:39 sea kernel: [2094960.518989]  [<ffffffff8112f9d1>] ? proc_alloc_inode+0x19/0x89
Jul 26 16:28:39 sea kernel: [2094960.519040]  [<ffffffff8112f9d1>] ? proc_alloc_inode+0x19/0x89
Jul 26 16:28:39 sea kernel: [2094960.519090]  [<ffffffff810e641c>] ? kmem_cache_alloc+0x7f/0xf0
Jul 26 16:28:39 sea kernel: [2094960.519141]  [<ffffffff8112f9d1>] ? proc_alloc_inode+0x19/0x89
Jul 26 16:28:39 sea kernel: [2094960.519191]  [<ffffffff810ffd21>] ? alloc_inode+0x17/0x74
Jul 26 16:28:39 sea kernel: [2094960.519240]  [<ffffffff8110001c>] ? new_inode+0x18/0xa4
Jul 26 16:28:39 sea kernel: [2094960.519288]  [<ffffffff81130d2b>] ? proc_pid_make_inode+0xe/0x9b
Jul 26 16:28:39 sea kernel: [2094960.519340]  [<ffffffff81133018>] ? proc_pident_instantiate+0x1b/0x9a
Jul 26 16:28:39 sea kernel: [2094960.519392]  [<ffffffff811331be>] ? proc_pident_lookup+0x87/0xa2
Jul 26 16:28:39 sea kernel: [2094960.519443]  [<ffffffff810f5657>] ? do_lookup+0xd3/0x15d
Jul 26 16:28:39 sea kernel: [2094960.519491]  [<ffffffff810f6084>] ? __link_path_walk+0x5a5/0x6f5
Jul 26 16:28:39 sea kernel: [2094960.519542]  [<ffffffff810f6402>] ? path_walk+0x66/0xc9
Jul 26 16:28:39 sea kernel: [2094960.519590]  [<ffffffff810f786c>] ? do_path_lookup+0x20/0x77
Jul 26 16:28:39 sea kernel: [2094960.519640]  [<ffffffff810f8d4e>] ? user_path_at+0x48/0x79
Jul 26 16:28:39 sea kernel: [2094960.519690]  [<ffffffff810f110b>] ? cp_new_stat+0xe9/0xfc
Jul 26 16:28:39 sea kernel: [2094960.519738]  [<ffffffff810f12d2>] ? vfs_fstatat+0x2c/0x57
Jul 26 16:28:39 sea kernel: [2094960.519787]  [<ffffffff810f13c5>] ? sys_newstat+0x11/0x30
Jul 26 16:28:39 sea kernel: [2094960.519836]  [<ffffffff810e4e35>] ? virt_to_head_page+0x9/0x2a
Jul 26 16:28:39 sea kernel: [2094960.519887]  [<ffffffff810ec4ce>] ? do_sys_open+0xec/0xfc
Jul 26 16:28:39 sea kernel: [2094960.519936]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b

System is Debian 5.0 Lenny + Squeeze backports.

[elvar]

Glad this isn't only happening to me.

[walterheck]

That seems like a bug with pretty critical status, have you guys reported it?

[elvar]

That seems like a bug with pretty critical status, have you guys reported it?

I personally was waiting to see if other people were having the same issue before reporting it.

[Murz]

Maybe this bug is not zabbix problem, but the problem with other working daemons.
I got this problem only 2 times, but after that I clean the mysql zabbix database (it was 1GB size, I clear all history) and didn't got this problem.

And, I think, that the problem is in high system load or db load, or memory problems, and not in zabbix side.