Падает высоконагруженный Zabbix
Периодически крашится сервер с довольно высокой нагрузкой:
Number of hosts (monitored/not monitored/templates) 16511 16495 / 0 / 16
Number of items (monitored/disabled/not supported) 53855 40183 / 0 / 13672
При этом в логе видно следующее:
На сервере установлена Gentoo Hardened
Linux zabbix20.zet 2.6.32-hardened-r77-0 #1 SMP Tue Nov 29 02:35:45 MSK 2011 x86_64 Intel(R) Xeon(R) CPU E5450 @ 3.00GHz GenuineIntel GNU/Linux
Ну и 8 гигов памяти.
По граффикам нагрзуки на Postgre и "Zabbix internal process busy" мощных перегрузок непосредственно перед падением небыло.
Number of hosts (monitored/not monitored/templates) 16511 16495 / 0 / 16
Number of items (monitored/disabled/not supported) 53855 40183 / 0 / 13672
При этом в логе видно следующее:
Code:
27131:20120719:031026.195 SNMP item [ifOutOctets.10140] on host [cisco_cchel1] failed: another network error, wait for 15 seconds 27140:20120719:031026.225 SNMP item [ifInOctets.10142] on host [cisco_cchel3] failed: first network error, wait for 15 seconds 27149:20120719:031026.257 SNMP item [ifOutOctets.10135] on host [cisco_cchel3] failed: another network error, wait for 15 seconds 27130:20120719:031030.162 SNMP item [psu_status] on host [cisco_cchel6] failed: first network error, wait for 15 seconds 27172:20120719:031124.345 resuming SNMP checks on host [cisco_cdyb5]: connection restored 27172:20120719:031124.375 resuming SNMP checks on host [cisco_cdybe5]: connection restored 27172:20120719:031138.478 resuming SNMP checks on host [cisco_cmosk2]: connection restored 27172:20120719:031138.503 resuming SNMP checks on host [cisco_ckras2]: connection restored 27172:20120719:031201.670 resuming SNMP checks on host [cisco_cvnov1]: connection restored 27172:20120719:031201.724 resuming SNMP checks on host [cisco_cvnov4]: connection restored 27172:20120719:031201.796 resuming SNMP checks on host [cisco_cvnov3]: connection restored 27172:20120719:031201.839 resuming SNMP checks on host [cisco_ccher2]: connection restored 27172:20120719:031208.736 resuming SNMP checks on host [cisco_cyaro1]: connection restored 27172:20120719:031208.877 resuming SNMP checks on host [cisco_cchel1]: connection restored 27172:20120719:031208.995 resuming SNMP checks on host [cisco_cchel2]: connection restored 27172:20120719:031210.311 resuming SNMP checks on host [cisco_cchel3]: connection restored 27172:20120719:031210.572 resuming SNMP checks on host [cisco_cchel6]: connection restored *** glibc detected *** /usr/sbin/zabbix_server: double free or corruption (!prev): 0x0000002e2bf3c240 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x720be)[0x29a6ee450be] /lib64/libc.so.6(cfree+0x6c)[0x29a6ee49ad1] /usr/lib64/libnetsnmp.so.15(snmp_sess_close+0x8f)[0x29a6f8500f4] /usr/sbin/zabbix_server(get_value_snmp+0x53f)[0x2e2bbad488] /usr/sbin/zabbix_server(+0x29e47)[0x2e2bbb0e47] /usr/sbin/zabbix_server(main_poller_loop+0xa2)[0x2e2bbb115d] /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x632)[0x2e2bba894d] /usr/sbin/zabbix_server(daemon_start+0x243)[0x2e2bbdf929] /lib64/libc.so.6(__libc_start_main+0xec)[0x29a6edf5224] /usr/sbin/zabbix_server(+0x1d3c9)[0x2e2bba43c9] ======= Memory map: ======== 2e2bb87000-2e2bc52000 r-xp 00000000 09:03 1071699 /usr/sbin/zabbix_server 2e2be51000-2e2be90000 r--p 000ca000 09:03 1071699 /usr/sbin/zabbix_server 2e2be90000-2e2be93000 rw-p 00109000 09:03 1071699 /usr/sbin/zabbix_server 2e2be93000-2e2bfd8000 rw-p 00000000 00:00 0 [heap] 29a34000000-29a34021000 rw-p 00000000 00:00 0 29a34021000-29a38000000 ---p 00000000 00:00 0 29a39781000-29a39795000 r-xp 00000000 09:03 1309355 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.5.3/libgcc_s.so.1 29a39795000-29a39995000 ---p 00014000 09:03 1309355 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.5.3/libgcc_s.so.1 29a39995000-29a39996000 r--p 00014000 09:03 1309355 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.5.3/libgcc_s.so.1 29a39996000-29a39997000 rw-p 00015000 09:03 1309355 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.5.3/libgcc_s.so.1 29a39997000-29a399a3000 rw-s 00000000 00:04 24641549 /SYSV53030195 (deleted) 29a399a3000-29a3e4a3000 rw-s 00000000 00:04 24608780 /SYSV73030195 (deleted) 29a3e4a3000-29a58da3000 rw-s 00000000 00:04 24576011 /SYSV67030195 (deleted) 29a58da3000-29a5cda4000 rw-s 00000000 00:04 24543242 /SYSV74030195 (deleted) 29a5cda4000-29a64da5000 rw-s 00000000 00:04 24510473 /SYSV78030195 (deleted) 29a64da5000-29a6cdae000 rw-s 00000000 00:04 24477704 /SYSV68030195 (deleted) 29a6cdae000-29a6cdba000 r-xp 00000000 09:03 788405 /lib64/libnss_files-2.14.1.so 29a6cdba000-29a6cfb9000 ---p 0000c000 09:03 788405 /lib64/libnss_files-2.14.1.so 29a6cfb9000-29a6cfba000 r--p 0000b000 09:03 788405 /lib64/libnss_files-2.14.1.so 29a6cfba000-29a6cfbb000 rw-p 0000c000 09:03 788405 /lib64/libnss_files-2.14.1.so 29a6cfbb000-29a6cfc5000 r-xp 00000000 09:03 788397 /lib64/libnss_nis-2.14.1.so 29a6cfc5000-29a6d1c4000 ---p 0000a000 09:03 788397 /lib64/libnss_nis-2.14.1.so 29a6d1c4000-29a6d1c5000 r--p 00009000 09:03 788397 /lib64/libnss_nis-2.14.1.so 29a6d1c5000-29a6d1c6000 rw-p 0000a000 09:03 788397 /lib64/libnss_nis-2.14.1.so 29a6d1c6000-29a6d1da000 r-xp 00000000 09:03 788407 /lib64/libnsl-2.14.1.so 29a6d1da000-29a6d3da000 ---p 00014000 09:03 788407 /lib64/libnsl-2.14.1.so 29a6d3da000-29a6d3db000 r--p 00014000 09:03 788407 /lib64/libnsl-2.14.1.so 29a6d3db000-29a6d3dc000 rw-p 00015000 09:03 788407 /lib64/libnsl-2.14.1.so 29a6d3dc000-29a6d3de000 rw-p 00000000 00:00 0 29a6d3de000-29a6d3e5000 r-xp 00000000 09:03 787921 /lib64/libnss_compat-2.14.1.so 29a6d3e5000-29a6d5e4000 ---p 00007000 09:03 787921 /lib64/libnss_compat-2.14.1.so 29a6d5e4000-29a6d5e5000 r--p 00006000 09:03 787921 /lib64/libnss_compat-2.14.1.so 29a6d5e5000-29a6d5e6000 rw-p 00007000 09:03 787921 /lib64/libnss_compat-2.14.1.so 29a6d5e6000-29a6d5f9000 r-xp 00000000 09:03 1063306 /usr/lib64/libhogweed.so.2.1 29a6d5f9000-29a6d7f8000 ---p 00013000 09:03 1063306 /usr/lib64/libhogweed.so.2.1 29a6d7f8000-29a6d7f9000 r--p 00012000 09:03 1063306 /usr/lib64/libhogweed.so.2.1 29a6d7f9000-29a6d7fa000 rw-p 00013000 09:03 1063306 /usr/lib64/libhogweed.so.2.1 29a6d7fa000-29a6d863000 r-xp 00000000 09:03 1048214 /usr/lib64/libgmp.so.10.0.2 29a6d863000-29a6da62000 ---p 00069000 09:03 1048214 /usr/lib64/libgmp.so.10.0.2 29a6da62000-29a6da64000 r--p 00068000 09:03 1048214 /usr/lib64/libgmp.so.10.0.2 29a6da64000-29a6da6c000 rw-p 0006a000 09:03 1048214 /usr/lib64/libgmp.so.10.0.2 29a6da6c000-29a6da93000 r-xp 00000000 09:03 1063278 /usr/lib64/libnettle.so.4.3 29a6da93000-29a6dc93000 ---p 00027000 09:03 1063278 /usr/lib64/libnettle.so.4.3 29a6dc93000-29a6dc94000 r--p 00027000 09:03 1063278 /usr/lib64/libnettle.so.4.3 29a6dc94000-29a6dc95000 rw-p 00028000 09:03 1063278 /usr/lib64/libnettle.so.4.3 29a6dc95000-29a6dca5000 r-xp 00000000 09:03 1049482 /usr/lib64/libtasn1.so.3.1.15 29a6dca5000-29a6dea4000 ---p 00010000 09:03 1049482 /usr/lib64/libtasn1.so.3.1.15 29a6dea4000-29a6dea5000 r--p 0000f000 09:03 1049482 /usr/lib64/libtasn1.so.3.1.15 29a6dea5000-29a6dea6000 rw-p 00010000 09:03 1049482 /usr/lib64/libtasn1.so.3.1.15 29a6dea6000-29a6dea8000 r-xp 00000000 09:03 786929 /lib64/libdl-2.14.1.so 29a6dea8000-29a6e0a8000 ---p 00002000 09:03 786929 /lib64/libdl-2.14.1.so 29a6e0a8000-29a6e0a9000 r--p 00002000 09:03 786929 /lib64/libdl-2.14.1.so 29a6e0a9000-29a6e0aa000 rw-p 00003000 09:03 786929 /lib64/libdl-2.14.1.so 29a6e0aa000-29a6e0c1000 r-xp 00000000 09:03 788388 /lib64/libpthread-2.14.1.so 29a6e0c1000-29a6e2c0000 ---p 00017000 09:03 788388 /lib64/libpthread-2.14.1.so 29a6e2c0000-29a6e2c1000 r--p 00016000 09:03 788388 /lib64/libpthread-2.14.1.so 29a6e2c1000-29a6e2c2000 rw-p 00017000 09:03 788388 /lib64/libpthread-2.14.1.so 29a6e2c2000-29a6e2c6000 rw-p 00000000 00:00 0 29a6e2c6000-29a6e2db000 r-xp 00000000 09:03 784904 /lib64/libz.so.1.2.5.1 29a6e2db000-29a6e4da000 ---p 00015000 09:03 784904 /lib64/libz.so.1.2.5.1 29a6e4da000-29a6e4db000 r--p 00014000 09:03 784904 /lib64/libz.so.1.2.5.1 29a6e4db000-29a6e4dc000 rw-p 00015000 09:03 784904 /lib64/libz.so.1.2.5.1 29a6e4dc000-29a6e596000 r-xp 00000000 09:03 1070211 /usr/lib64/libgnutls.so.26.22.1 29a6e596000-29a6e796000 ---p 000ba000 09:03 1070211 /usr/lib64/libgnutls.so.26.22.1 29a6e796000-29a6e79d000 r--p 000ba000 09:03 1070211 /usr/lib64/libgnutls.so.26.22.1 29a6e79d000-29a6e79e000 rw-p 000c1000 09:03 1070211 /usr/lib64/libgnutls.so.26.22.1 29a6e79e000-29a6e94c000 r-xp 00000000 09:03 1049662 /usr/lib64/libcrypto.so.1.0.0 29a6e94c000-29a6eb4b000 ---p 001ae000 09:03 1049662 /usr/lib64/libcrypto.so.1.0.0 29a6eb4b000-29a6eb65000 r--p 001ad000 09:03 1049662 /usr/lib64/libcrypto.so.1.0.0 29a6eb65000-29a6eb6f000 rw-p 001c7000 09:03 1049662 /usr/lib64/libcrypto.so.1.0.0 29a6eb6f000-29a6eb72000 rw-p 00000000 00:00 0 29a6eb72000-29a6ebcb000 r-xp 00000000 09:03 1050074 /usr/lib64/libssl.so.1.0.0 29a6ebcb000-29a6edca000 ---p 00059000 09:03 1050074 /usr/lib64/libssl.so.1.0.0 29a6edca000-29a6edce000 r--p 00058000 09:03 1050074 /usr/lib64/libssl.so.1.0.0 29a6edce000-29a6edd3000 rw-p 0005c000 09:03 1050074 /usr/lib64/libssl.so.1.0.0 29a6edd3000-29a6ef4e000 r-xp 00000000 09:03 788399 /lib64/libc-2.14.1.so 29a6ef4e000-29a6f14d000 ---p 0017b000 09:03 788399 /lib64/libc-2.14.1.so 29a6f14d000-29a6f151000 r--p 0017a000 09:03 788399 /lib64/libc-2.14.1.so 29a6f151000-29a6f152000 rw-p 0017e000 09:03 788399 /lib64/libc-2.14.1.so 29a6f152000-29a6f157000 rw-p 00000000 00:00 0 29a6f157000-29a6f169000 r-xp 00000000 09:03 788402 /lib64/libresolv-2.14.1.so 29a6f169000-29a6f369000 ---p 00012000 09:03 788402 /lib64/libresolv-2.14.1.so 29a6f369000-29a6f36a000 r--p 00012000 09:03 788402 /lib64/libresolv-2.14.1.so 29a6f36a000-29a6f36b000 rw-p 00013000 09:03 788402 /lib64/libresolv-2.14.1.so 29a6f36b000-29a6f36d000 rw-p 00000000 00:00 0 29a6f36d000-29a6f3ed000 r-xp 00000000 09:03 788404 /lib64/libm-2.14.1.so 29a6f3ed000-29a6f5ec000 ---p 00080000 09:03 788404 /lib64/libm-2.14.1.so 29a6f5ec000-29a6f5ed000 r--p 0007f000 09:03 788404 /lib64/libm-2.14.1.so 29a6f5ed000-29a6f5ee000 rw-p 00080000 09:03 788404 /lib64/libm-2.14.1.so 29a6f5ee000-29a6f617000 r-xp 00000000 09:03 1057030 /usr/lib64/libssh2.so.1.0.1 29a6f617000-29a6f816000 ---p 00029000 09:03 1057030 /usr/lib64/libssh2.so.1.0.1 29a6f816000-29a6f817000 r--p 00028000 09:03 1057030 /usr/lib64/libssh2.so.1.0.1 29a6f817000-29a6f818000 rw-p 00029000 09:03 1057030 /usr/lib64/libssh2.so.1.0.1 29a6f818000-29a6f8b2000 r-xp 00000000 09:03 1051626 /usr/lib64/libnetsnmp.so.15.1.2 29a6f8b2000-29a6fab2000 ---p 0009a000 09:03 1051626 /usr/lib64/libnetsnmp.so.15.1.2 29a6fab2000-29a6fab4000 r--p 0009a000 09:03 1051626 /usr/lib64/libnetsnmp.so.15.1.2 29a6fab4000-29a6fab6000 rw-p 0009c000 09:03 1051626 /usr/lib64/libnetsnmp.so.15.1.2 29a6fab6000-29a6faea000 rw-p 00000000 00:00 0 29a6faea000-29a6faf1000 r-xp 00000000 09:03 788396 /lib64/librt-2.14.1.so 29a6faf1000-29a6fcf1000 ---p 00007000 09:03 788396 /lib64/librt-2.14.1.so 29a6fcf1000-29a6fcf2000 r--p 00007000 09:03 788396 /lib64/librt-2.14.1.so 29a6fcf2000-29a6fcf3000 rw-p 00008000 09:03 788396 /lib64/librt-2.14.1.so 29a6fcf3000-29a6fd48000 r-xp 00000000 09:03 1068215 /usr/lib64/libcurl.so.4.2.0 29a6fd48000-29a6ff48000 ---p 00055000 09:03 1068215 /usr/lib64/libcurl.so.4.2.0 29a6ff48000-29a6ff4a000 r--p 00055000 09:03 1068215 /usr/lib64/libcurl.so.4.2.0 29a6ff4a000-29a6ff4b000 rw-p 00057000 09:03 1068215 /usr/lib64/libcurl.so.4.2.0 29a6ff4b000-29a6ff5a000 r-xp 00000000 09:03 1074734 /usr/lib64/libiksemel.so.3.1.1 29a6ff5a000-29a70159000 ---p 0000f000 09:03 1074734 /usr/lib64/libiksemel.so.3.1.1 29a70159000-29a7015a000 r--p 0000e000 09:03 1074734 /usr/lib64/libiksemel.so.3.1.1 29a7015a000-29a7015b000 rw-p 0000f000 09:03 1074734 /usr/lib64/libiksemel.so.3.1.1 29a7015b000-29a70184000 r-xp 00000000 09:03 1320339 /usr/lib64/postgresql-9.1/lib64/libpq.so.5.4 29a70184000-29a70383000 ---p 00029000 09:03 1320339 /usr/lib64/postgresql-9.1/lib64/libpq.so.5.4 29a70383000-29a70385000 r--p 00028000 09:03 1320339 /usr/lib64/postgresql-9.1/lib64/libpq.so.5.4 29a70385000-29a70387000 rw-p 0002a000 09:03 1320339 /usr/lib64/postgresql-9.1/lib64/libpq.so.5.4 29a70387000-29a703a7000 r-xp 00000000 09:03 788391 /lib64/ld-2.14.1.so 29a7058b000-29a70595000 rw-p 00000000 00:00 0 29a705a2000-29a705a5000 rw-p 00000000 00:00 0 29a705a5000-29a705a6000 r-xp 00000000 00:00 0 [vdso] 29a705a6000-29a705a7000 r--p 0001f000 09:03 788391 /lib64/ld-2.14.1.so 29a705a7000-29a705a8000 rw-p 00020000 09:03 788391 /lib64/ld-2.14.1.so 29a705a8000-29a705a9000 rw-p 00000000 00:00 0 3de3d4e3000-3de3d512000 rw-p 00000000 00:00 0 [stack] ffffffffff600000-ffffffffff601000 r--p 00000000 00:00 0 [vsyscall] 27098:20120719:031838.500 One child process died (PID:27151,exitcode/signal:6). Exiting ... 27098:20120719:031843.306 syncing history data... 27098:20120719:031846.645 syncing history data done 27098:20120719:031846.645 syncing trends data... 27098:20120719:031909.790 syncing trends data done 27098:20120719:031909.790 Zabbix Server stopped. Zabbix 2.0.1 (revision 28455).
Linux zabbix20.zet 2.6.32-hardened-r77-0 #1 SMP Tue Nov 29 02:35:45 MSK 2011 x86_64 Intel(R) Xeon(R) CPU E5450 @ 3.00GHz GenuineIntel GNU/Linux
Ну и 8 гигов памяти.
По граффикам нагрзуки на Postgre и "Zabbix internal process busy" мощных перегрузок непосредственно перед падением небыло.
Comment