Error "Out of memory: kill process #### (zabbix_server) score #### or a child"

[nima0102]

Hi
Sometimes zabbix are killed and I can not find any reason for this problem,the only this that I got from zabbix_log and I copied here :

[14144794.860059] Call Trace:
[14144794.860076] [<ffffffff802738b8>] oom_kill_process+0x57/0x1dc
[14144794.860080] [<ffffffff8023b4d1>] __capable+0x9/0x1c
[14144794.860082] [<ffffffff80273be3>] badness+0x188/0x1c7
[14144794.860085] [<ffffffff80273e17>] out_of_memory+0x1f5/0x28e
[14144794.860090] [<ffffffff80276b68>] __alloc_pages_internal+0x31d/0x3bf
[14144794.860096] [<ffffffff8027881e>] __do_page_cache_readahead+0x79/0x183
[14144794.860100] [<ffffffff802730cd>] filemap_fault+0x15d/0x33c
[14144794.860104] [<ffffffff8027e64c>] __do_fault+0x50/0x3e8
[14144794.860108] [<ffffffff8024fb2f>] futex_wake+0x74/0xfa
[14144794.860113] [<ffffffff80281a0f>] handle_mm_fault+0x452/0x8de
[14144794.860115] [<ffffffff80250724>] do_futex+0xa6/0x78a
[14144794.860119] [<ffffffff80229020>] hrtick_start_fair+0xfb/0x144
[14144794.860125] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
[14144794.860131] [<ffffffff8042a9c9>] error_exit+0x0/0x60
[14144794.860136]
[14144794.860137] Mem-info:
[14144794.860138] Node 0 DMA per-cpu:
[14144794.860140] CPU 0: hi: 0, btch: 1 usd: 0
[14144794.860142] CPU 1: hi: 0, btch: 1 usd: 0
[14144794.860143] CPU 2: hi: 0, btch: 1 usd: 0
[14144794.860145] CPU 3: hi: 0, btch: 1 usd: 0
[14144794.860146] CPU 4: hi: 0, btch: 1 usd: 0
[14144794.860147] CPU 5: hi: 0, btch: 1 usd: 0
[14144794.860148] CPU 6: hi: 0, btch: 1 usd: 0
[14144794.860150] CPU 7: hi: 0, btch: 1 usd: 0
[14144794.860151] CPU 8: hi: 0, btch: 1 usd: 0
[14144794.860152] CPU 9: hi: 0, btch: 1 usd: 0
[14144794.860154] CPU 10: hi: 0, btch: 1 usd: 0
[14144794.860155] CPU 11: hi: 0, btch: 1 usd: 0
[14144794.860157] CPU 12: hi: 0, btch: 1 usd: 0
[14144794.860158] CPU 13: hi: 0, btch: 1 usd: 0
[14144794.860159] CPU 14: hi: 0, btch: 1 usd: 0
[14144794.860161] CPU 15: hi: 0, btch: 1 usd: 0
[14144794.860162] Node 0 DMA32 per-cpu:
[14144794.860164] CPU 0: hi: 186, btch: 31 usd: 111
[14144794.860165] CPU 1: hi: 186, btch: 31 usd: 0
[14144794.860166] CPU 2: hi: 186, btch: 31 usd: 112
[14144794.860168] CPU 3: hi: 186, btch: 31 usd: 0
[14144794.860169] CPU 4: hi: 186, btch: 31 usd: 24
[14144794.860170] CPU 5: hi: 186, btch: 31 usd: 0
[14144794.860172] CPU 6: hi: 186, btch: 31 usd: 74
[14144794.860173] CPU 7: hi: 186, btch: 31 usd: 0
[14144794.860175] CPU 8: hi: 186, btch: 31 usd: 46
[14144794.860176] CPU 9: hi: 186, btch: 31 usd: 0
[14144794.860177] CPU 10: hi: 186, btch: 31 usd: 30
[14144794.860178] CPU 11: hi: 186, btch: 31 usd: 0
[14144794.860180] CPU 12: hi: 186, btch: 31 usd: 183
[14144794.860181] CPU 13: hi: 186, btch: 31 usd: 0
[14144794.860183] CPU 14: hi: 186, btch: 31 usd: 60
[14144794.860184] CPU 15: hi: 186, btch: 31 usd: 0
[14144794.860185] Node 0 Normal per-cpu:
[14144794.860186] CPU 0: hi: 186, btch: 31 usd: 174
[14144794.860188] CPU 1: hi: 186, btch: 31 usd: 123
[14144794.860189] CPU 2: hi: 186, btch: 31 usd: 131
[14144794.860190] CPU 3: hi: 186, btch: 31 usd: 161
[14144794.860192] CPU 4: hi: 186, btch: 31 usd: 152
[14144794.860193] CPU 5: hi: 186, btch: 31 usd: 83
[14144794.860194] CPU 6: hi: 186, btch: 31 usd: 174
[14144794.860195] CPU 7: hi: 186, btch: 31 usd: 38
[14144794.860197] CPU 8: hi: 186, btch: 31 usd: 175
[14144794.860198] CPU 9: hi: 186, btch: 31 usd: 180
[14144794.860199] CPU 10: hi: 186, btch: 31 usd: 172
[14144794.860200] CPU 11: hi: 186, btch: 31 usd: 184
[14144794.860202] CPU 12: hi: 186, btch: 31 usd: 165
[14144794.860203] CPU 13: hi: 186, btch: 31 usd: 45
[14144794.860204] CPU 14: hi: 186, btch: 31 usd: 160
[14144794.860206] CPU 15: hi: 186, btch: 31 usd: 53
[14144794.860207] Node 1 Normal per-cpu:
[14144794.860209] CPU 0: hi: 186, btch: 31 usd: 60
[14144794.860210] CPU 1: hi: 186, btch: 31 usd: 76
[14144794.860212] CPU 2: hi: 186, btch: 31 usd: 9
[14144794.860213] CPU 3: hi: 186, btch: 31 usd: 182
[14144794.860214] CPU 4: hi: 186, btch: 31 usd: 0
[14144794.860215] CPU 5: hi: 186, btch: 31 usd: 159
[14144794.860217] CPU 6: hi: 186, btch: 31 usd: 62
[14144794.860218] CPU 7: hi: 186, btch: 31 usd: 71
[14144794.860219] CPU 8: hi: 186, btch: 31 usd: 0
[14144794.860221] CPU 9: hi: 186, btch: 31 usd: 170
[14144794.860222] CPU 10: hi: 186, btch: 31 usd: 30
[14144794.860223] CPU 11: hi: 186, btch: 31 usd: 176
[14144794.860225] CPU 12: hi: 186, btch: 31 usd: 116
[14144794.860226] CPU 13: hi: 186, btch: 31 usd: 161
[14144794.860227] CPU 14: hi: 186, btch: 31 usd: 9
[14144794.860229] CPU 15: hi: 186, btch: 31 usd: 158
[14144794.860231] Active:6618885 inactive:2740 dirty:0 writeback:0 unstable:0
[14144794.860232] free:18666 slab:12063 mapped:34 pagetables:20469 bounce:0
[14144794.860233] Node 0 DMA free:11612kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:10692kB pages_scanned:0 all_unreclaimable? yes
[14144794.860236] lowmem_reserve[]: 0 3502 14107 14107
[14144794.860238] Node 0 DMA32 free:45160kB min:2764kB low:3452kB high:4144kB active:3299932kB inactive:6380kB present:3586464kB pages_scanned:9401336 all_unreclaimable? no
[14144794.860241] lowmem_reserve[]: 0 0 10605 10605
[14144794.860243] Node 0 Normal free:8504kB min:8376kB low:10468kB high:12564kB active:10890540kB inactive:1044kB present:10859520kB pages_scanned:893265 all_unreclaimable? no
[14144794.860247] lowmem_reserve[]: 0 0 0 0
[14144794.860249] Node 1 Normal free:9388kB min:9576kB low:11968kB high:14364kB active:12285068kB inactive:3536kB present:12410876kB pages_scanned:877361 all_unreclaimable? no
[14144794.860252] lowmem_reserve[]: 0 0 0 0
[14144794.860254] Node 0 DMA: 5*4kB 5*8kB 4*16kB 3*32kB 4*64kB 1*128kB 3*256kB 2*512kB 1*1024kB 0*2048kB 2*4096kB = 11612kB
[14144794.860259] Node 0 DMA32: 10266*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 2*2048kB 0*4096kB = 45160kB
[14144794.860264] Node 0 Normal: 288*4kB 16*8kB 1*16kB 1*32kB 0*64kB 2*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8752kB
[14144794.860269] Node 1 Normal: 254*4kB 20*8kB 3*16kB 3*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 9512kB
[14144794.860274] 3996 total pagecache pages
[14144794.860276] Swap cache: add 0, delete 0, find 0/0
[14144794.860277] Free swap = 0kB
[14144794.860278] Total swap = 0kB
[14144794.956192] 6815743 pages of RAM
[14144794.956195] 114302 reserved pages
[14144794.956195] 323833 pages shared
[14144794.956196] 0 pages swap cached
[14144794.956199] Out of memory: kill process 25243 (zabbix_server) score 317152 or a child
[14144794.956281] Killed process 25246 (zabbix_server)


Thanks for any help or guidance

[walterheck]

That's like asking: "There's a dent in my car, why is that?".

We need more information: what else is running on that server, what is your zabbix config, how much memory does the server have, etc.