Zabbix proxy - realloc(): invalid next size

[nail]

 23834:20130507:223120.484 proxy #9 started [unreachable poller #1]
 23836:20130507:223120.485 proxy #11 started [trapper #2]
 23826:20130507:223120.485 proxy #1 started [configuration syncer #1]
 23837:20130507:223120.486 proxy #12 started [trapper #3]
 23843:20130507:223120.490 proxy #15 started [icmp pinger #1]
 23844:20130507:223120.490 proxy #16 started [housekeeper #1]
 23842:20130507:223120.491 proxy #14 started [trapper #5]
 23841:20130507:223120.491 proxy #13 started [trapper #4]
 23824:20130507:223120.493 proxy #0 started [main process]
 23849:20130507:223120.496 proxy #18 started [discoverer #1]
 23851:20130507:223120.498 proxy #20 started [history syncer #2]
 23852:20130507:223120.498 proxy #21 started [history syncer #3]
 23848:20130507:223120.499 proxy #17 started [http poller #1]
 23850:20130507:223120.499 proxy #19 started [history syncer #1]
 23853:20130507:223120.499 proxy #22 started [history syncer #4]
*** Error in `/usr/local/sbin/zabbix_proxy': realloc(): invalid next size: 0x0000000000fb3200 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7aac6)[0x7f8105e2dac6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7d736)[0x7f8105e30736]
/lib/x86_64-linux-gnu/libc.so.6(realloc+0xed)[0x7f8105e3166d]
/usr/local/sbin/zabbix_proxy(zbx_realloc2+0x3a)[0x43b72a]
/usr/local/sbin/zabbix_proxy(proxy_get_hist_data+0x21f)[0x45f0ef]
/usr/local/sbin/zabbix_proxy[0x421c65]
/usr/local/sbin/zabbix_proxy(main_datasender_loop+0x152)[0x421e82]
/usr/local/sbin/zabbix_proxy(MAIN_ZABBIX_ENTRY+0x555)[0x40ee25]
/usr/local/sbin/zabbix_proxy(daemon_start+0x199)[0x439a59]
/usr/local/sbin/zabbix_proxy(main+0x21b)[0x40dfab]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f8105dd4a55]
/usr/local/sbin/zabbix_proxy[0x40e2c1]
======= Memory map: ========
00400000-004fa000 r-xp 00000000 08:11 405802                             /usr/local/sbin/zabbix_proxy
006f9000-006fc000 rw-p 000f9000 08:11 405802                             /usr/local/sbin/zabbix_proxy
006fc000-00702000 rw-p 00000000 00:00 0
00f78000-00f99000 rw-p 00000000 00:00 0                                  [heap]
00f99000-00fc5000 rw-p 00000000 00:00 0                                  [heap]
00fc5000-01031000 rw-p 00000000 00:00 0                                  [heap]
7f80fc000000-7f80fc021000 rw-p 00000000 00:00 0
7f80fc021000-7f8100000000 ---p 00000000 00:00 0
7f8101ac4000-7f8101ac9000 r-xp 00000000 08:11 262152                     /lib/x86_64-linux-gnu/libnss_dns-2.17.so
7f8101ac9000-7f8101cc8000 ---p 00005000 08:11 262152                     /lib/x86_64-linux-gnu/libnss_dns-2.17.so
7f8101cc8000-7f8101cc9000 r--p 00004000 08:11 262152                     /lib/x86_64-linux-gnu/libnss_dns-2.17.so
7f8101cc9000-7f8101cca000 rw-p 00005000 08:11 262152                     /lib/x86_64-linux-gnu/libnss_dns-2.17.so
7f8101cca000-7f8102397000 rw-s 00000000 00:04 2654211                    /SYSV67113121 (deleted)
7f8102397000-7f8103397000 rw-s 00000000 00:04 2621442                    /SYSV78113121 (deleted)
7f8103397000-7f8103b97000 rw-s 00000000 00:04 2588673                    /SYSV68113121 (deleted)
7f8103b97000-7f8103b98000 ---p 00000000 00:00 0
7f8103b98000-7f8104398000 rw-p 00000000 00:00 0
7f8104398000-7f81043a4000 r-xp 00000000 08:11 262159                     /lib/x86_64-linux-gnu/libnss_files-2.17.so
7f81043a4000-7f81045a3000 ---p 0000c000 08:11 262159                     /lib/x86_64-linux-gnu/libnss_files-2.17.so
7f81045a3000-7f81045a4000 r--p 0000b000 08:11 262159                     /lib/x86_64-linux-gnu/libnss_files-2.17.so
7f81045a4000-7f81045a5000 rw-p 0000c000 08:11 262159                     /lib/x86_64-linux-gnu/libnss_files-2.17.so
7f81045a5000-7f81045af000 r-xp 00000000 08:11 262168                     /lib/x86_64-linux-gnu/libnss_nis-2.17.so
7f81045af000-7f81047ae000 ---p 0000a000 08:11 262168                     /lib/x86_64-linux-gnu/libnss_nis-2.17.so
7f81047ae000-7f81047af000 r--p 00009000 08:11 262168                     /lib/x86_64-linux-gnu/libnss_nis-2.17.so
7f81047af000-7f81047b0000 rw-p 0000a000 08:11 262168                     /lib/x86_64-linux-gnu/libnss_nis-2.17.so
7f81047b0000-7f81047c5000 r-xp 00000000 08:11 262147                     /lib/x86_64-linux-gnu/libnsl-2.17.so
7f81047c5000-7f81049c4000 ---p 00015000 08:11 262147                     /lib/x86_64-linux-gnu/libnsl-2.17.so
7f81049c4000-7f81049c5000 r--p 00014000 08:11 262147                     /lib/x86_64-linux-gnu/libnsl-2.17.so
7f81049c5000-7f81049c6000 rw-p 00015000 08:11 262147                     /lib/x86_64-linux-gnu/libnsl-2.17.so
7f81049c6000-7f81049c8000 rw-p 00000000 00:00 0
7f81049c8000-7f81049cf000 r-xp 00000000 08:11 262158                     /lib/x86_64-linux-gnu/libnss_compat-2.17.so
7f81049cf000-7f8104bce000 ---p 00007000 08:11 262158                     /lib/x86_64-linux-gnu/libnss_compat-2.17.so
7f8104bce000-7f8104bcf000 r--p 00006000 08:11 262158                     /lib/x86_64-linux-gnu/libnss_compat-2.17.so
7f8104bcf000-7f8104bd0000 rw-p 00007000 08:11 262158                     /lib/x86_64-linux-gnu/libnss_compat-2.17.so
7f8104bd0000-7f8104bd3000 r-xp 00000000 08:11 261788                     /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7f8104bd3000-7f8104dd2000 ---p 00003000 08:11 261788                     /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7f8104dd2000-7f8104dd3000 rw-p 00002000 08:11 261788                     /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
7f8104dd3000-7f8104e4d000 r-xp 00000000 08:11 261790                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7f8104e4d000-7f810504d000 ---p 0007a000 08:11 261790                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7f810504d000-7f810504e000 r--p 0007a000 08:11 261790                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7f810504e000-7f8105051000 rw-p 0007b000 08:11 261790                     /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
7f8105051000-7f810505a000 r-xp 00000000 08:11 397140                     /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7f810505a000-7f8105259000 ---p 00009000 08:11 397140                     /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7f8105259000-7f810525a000 rw-p 00008000 08:11 397140                     /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7f810525a000-7f810526f000 r-xp 00000000 08:11 261637                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f810526f000-7f810546f000 ---p 00015000 08:11 261637                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f810546f000-7f8105470000 rw-p 00015000 08:11 261637                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8105470000-7f810555a000 r-xp 00000000 08:11 394397                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.18
7f810555a000-7f8105759000 ---p 000ea000 08:11 394397                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.18
7f8105759000-7f8105761000 r--p 000e9000 08:11 394397                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.18
7f8105761000-7f8105763000 rw-p 000f1000 08:11 394397                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.18
7f8105763000-7f8105778000 rw-p 00000000 00:00 0
7f8105778000-7f810577f000 r-xp 00000000 08:11 262151                     /lib/x86_64-linux-gnu/librt-2.17.so
7f810577f000-7f810597e000 ---p 00007000 08:11 262151                     /lib/x86_64-linux-gnu/librt-2.17.so
7f810597e000-7f810597f000 r--p 00006000 08:11 262151                     /lib/x86_64-linux-gnu/librt-2.17.so
7f810597f000-7f8105980000 rw-p 00007000 08:11 262151                     /lib/x86_64-linux-gnu/librt-2.17.so
7f8105980000-7f8105996000 r-xp 00000000 08:11 261725                     /lib/x86_64-linux-gnu/libz.so.1.2.7
7f8105996000-7f8105b95000 ---p 00016000 08:11 261725                     /lib/x86_64-linux-gnu/libz.so.1.2.7
7f8105b95000-7f8105b96000 r--p 00015000 08:11 261725                     /lib/x86_64-linux-gnu/libz.so.1.2.7
7f8105b96000-7f8105b97000 rw-p 00016000 08:11 261725                     /lib/x86_64-linux-gnu/libz.so.1.2.7
7f8105b97000-7f8105bae000 r-xp 00000000 08:11 262163                     /lib/x86_64-linux-gnu/libpthread-2.17.so
7f8105bae000-7f8105dad000 ---p 00017000 08:11 262163                     /lib/x86_64-linux-gnu/libpthread-2.17.so
7f8105dad000-7f8105dae000 r--p 00016000 08:11 262163                     /lib/x86_64-linux-gnu/libpthread-2.17.so
7f8105dae000-7f8105daf000 rw-p 00017000 08:11 262163                     /lib/x86_64-linux-gnu/libpthread-2.17.so
7f8105daf000-7f8105db3000 rw-p 00000000 00:00 0
7f8105db3000-7f8105f57000 r-xp 00000000 08:11 262162                     /lib/x86_64-linux-gnu/libc-2.17.so
7f8105f57000-7f8106156000 ---p 001a4000 08:11 262162                     /lib/x86_64-linux-gnu/libc-2.17.so
7f8106156000-7f810615a000 r--p 001a3000 08:11 262162                     /lib/x86_64-linux-gnu/libc-2.17.so
7f810615a000-7f810615c000 rw-p 001a7000 08:11 262162                     /lib/x86_64-linux-gnu/libc-2.17.so
7f810615c000-7f8106160000 rw-p 00000000 00:00 0
7f8106160000-7f8106173000 r-xp 00000000 08:11 262161                     /lib/x86_64-linux-gnu/libresolv-2.17.so
7f8106173000-7f8106373000 ---p 00013000 08:11 262161                     /lib/x86_64-linux-gnu/libresolv-2.17.so
7f8106373000-7f8106374000 r--p 00013000 08:11 262161                     /lib/x86_64-linux-gnu/libresolv-2.17.so
7f8106374000-7f8106375000 rw-p 00014000 08:11 262161                     /lib/x86_64-linux-gnu/libresolv-2.17.so
7f8106375000-7f8106377000 rw-p 00000000 00:00 0
7f8106377000-7f810637a000 r-xp 00000000 08:11 262167                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f810637a000-7f8106579000 ---p 00003000 08:11 262167                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f8106579000-7f810657a000 r--p 00002000 08:11 262167                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f810657a000-7f810657b000 rw-p 00003000 08:11 262167                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f810657b000-7f8106678000 r-xp 00000000 08:11 262153                     /lib/x86_64-linux-gnu/libm-2.17.so
7f8106678000-7f8106877000 ---p 000fd000 08:11 262153                     /lib/x86_64-linux-gnu/libm-2.17.so
7f8106877000-7f8106878000 r--p 000fc000 08:11 262153                     /lib/x86_64-linux-gnu/libm-2.17.so
7f8106878000-7f8106879000 rw-p 000fd000 08:11 262153                     /lib/x86_64-linux-gnu/libm-2.17.so
7f8106879000-7f81068a0000 r-xp 00000000 08:11 407558                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f81068a0000-7f8106aa0000 ---p 00027000 08:11 407558                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f8106aa0000-7f8106aa1000 r--p 00027000 08:11 407558                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f8106aa1000-7f8106aa2000 rw-p 00028000 08:11 407558                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f8106aa2000-7f8106b02000 r-xp 00000000 08:11 404164                     /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
7f8106b02000-7f8106d01000 ---p 00060000 08:11 404164                     /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
7f8106d01000-7f8106d09000 rw-p 0005f000 08:11 404164                     /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
7f8106d09000-7f8106d0a000 rw-p 00000000 00:00 0
7f8106d0a000-7f8106fc9000 r-xp 00000000 08:11 397213                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f8106fc9000-7f81071c8000 ---p 002bf000 08:11 397213                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f81071c8000-7f81071ce000 r--p 002be000 08:11 397213                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f81071ce000-7f810724c000 rw-p 002c4000 08:11 397213                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f810724c000-7f8107251000 rw-p 00000000 00:00 0
7f8107251000-7f8107272000 r-xp 00000000 08:11 262166                     /lib/x86_64-linux-gnu/ld-2.17.so
7f810732d000-7f8107461000 rw-s 00000000 00:04 2686980                    /SYSV73113121 (deleted)
7f8107461000-7f810746b000 rw-p 00000000 00:00 0
7f810746c000-7f810746d000 rw-p 00000000 00:00 0
7f810746d000-7f810746f000 rw-s 00000000 00:04 2719750                    /SYSV53113121 (deleted)
7f810746f000-7f8107470000 rw-p 00000000 00:00 0
7f8107470000-7f8107472000 rw-p 00000000 00:00 0
7f8107472000-7f8107473000 r--p 00021000 08:11 262166                     /lib/x86_64-linux-gnu/ld-2.17.so
7f8107473000-7f8107474000 rw-p 00022000 08:11 262166                     /lib/x86_64-linux-gnu/ld-2.17.so
7f8107474000-7f8107475000 rw-p 00000000 00:00 0
7fffcb1e1000-7fffcb202000 rw-p 00000000 00:00 0                          [stack]
7fffcb33d000-7fffcb33f000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 23824:20130507:224038.867 One child process died (PID:23828,exitcode/signal:6). Exiting ...
 23824:20130507:224040.875 Zabbix Proxy stopped. Zabbix 2.1.0 (revision 35380).

any ideas how to fix this ?

ps. got another debian host with this version of zabbix proxy running successfully.

[nail]

found that libc6 versions differ on servers, will try to use the same version.

[nail]

after series of test I can say positively that Zabbix Proxy crashes because of libc6_2.17-1.
works fine with libc6 2.13-38

upd. heavy loaded proxies crash with 2.17-1, proxy with 1 host works fine with it.

[zalex_ua]

Nice to see trunk users here

Are you sure that zabbix server is the same revision ?

[zalex_ua]

Rich shared with probably interesting link http://stackoverflow.com/questions/2...alid-next-size

nail, this case worth to be posted as a bug report https://support.zabbix.com/browse/ZBX
Could you please do it ?

[nail]

I'll test latest version today with latest debian updates, and then'll write bug report if it still persists.

[nail]

Zabbix proxy v2.1.0 35609 + latest debian updates crashed after 10-15 minutes of working:

https://support.zabbix.com/browse/ZBX-6601

[asdftyjkl]

--- src/libs/zbxdbhigh/proxy.c.orig 2013-05-20 17:26:23.373954446 +0800
+++ src/libs/zbxdbhigh/proxy.c 2013-05-20 17:27:31.706994466 +0800
@@ -1376,7 +1376,7 @@ static int proxy_get_history_data(struct
len1 = strlen(row[5]) + 1;
len2 = strlen(row[7]) + 1;

- if (string_buffer_alloc < string_buffer_offset + len1 + len2)
+ while (string_buffer_alloc < string_buffer_offset + len1 + len2)
{
string_buffer_alloc += ZBX_KIBIBYTE;
string_buffer = zbx_realloc(string_buffer, string_buffer_alloc);

[nail]

fix works for now, thx
using Zabbix proxy v2.1.0 (revision 35860)
(currently 1 day uptime, no problems so far)

[dimir]

This is a regression caused by

https://support.zabbix.com/browse/ZBX-3817

Fixed in trunk r36011. Thank you for the patch.