I send application performance data to zabbix via sender and the proxy. About half the time I stop my script one of the proxy trapper child processes die. I have done enough debugging to convince myself it is crashing in zbx_tcp_send_ext in the write system calls (not always the same one, there are 3). This is a Sparc Solaris 10 system. Upgrading to 1.8.3 did not fix the problem.
10318:20100629:084210.117 Timeout while answering request
10318:20100629:084210.117 Got signal [signal:11(SIGSEGV),reason:1,refaddr:c]. Crashing ...
10318:20100629:084210.117 ====== Fatal information: ======
10318:20100629:084210.117 program counter not available for this architecture
10318:20100629:084210.117 === Registers: ===
10318:20100629:084210.117 register dump not available for this architecture
10318:20100629:084210.117 === Backtrace: ===
10318:20100629:084210.117 backtrace not available for this platform
10318:20100629:084210.118 === Memory map: ===
10318:20100629:084210.118 memory map not available for this platform
10318:20100629:084210.118 ================================
10291:20100629:084210.126 One child process died (PID:10318,exitcode/signal:-1). Exiting ...
10291:20100629:084210.126 zbx_on_exit() called
Output from runme_on_app_crash
Program: zabbix_proxy
Process ID: 10318
Received signal: 11
Application Debugging Data
--------------------------
> /bin/pstack 10318
10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
0003ab68 child_signal_handler (d, 622a0, ffbeb8f0, 0, 0, 0) + 100
fec44b4c __sighndlr (d, 0, ffbeb8f0, 3aa68, 0, 1) + c
fec39b24 call_user_handler (d, 0, 8, 0, feed2a00, ffbeb8f0) + 3b8
fec45f64 _write (6, ffbebc68, 8, 0, 0, 0) + c
000448e4 zbx_tcp_send_ext (ffbfef40, ffbebce4, 0, 0, ffbebc70, a39b0) + 9c
00045668 zbx_send_response (ffbfef40, 0, ffffefe8, 3, fa, 73800) + cc
00022750 process_trapper_child (ffbfef40, 5c8b8, 0, e, 10b4, fec73a80) + 534
00022cf4 child_trapper_main (5c800, ffbfef40, 9f000, 9e800, ffbfeed0, a39b0) + b4
00018d2c MAIN_ZABBIX_ENTRY (1a, fec39cac, 85c00, 85c00, 85c00, 85c00) + 570
0003af04 daemon_start (ffbffbd0, ffbffc6c, ffbffc7c, a3cfc, feed0140, feed0180) + 2fc
00017edc _start (0, 0, 0, 0, 0, 0) + 5c
> /bin/pmap -x 10318
10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
Address Kbytes RSS Anon Locked Mode Mapped File
00010000 408 352 - - r-x-- zabbix_proxy
00084000 104 24 8 - rwx-- zabbix_proxy
0009E000 16 8 8 - rwx-- zabbix_proxy
000A2000 744 680 648 - rwx-- [ heap ]
FA000000 13928 96 - - rwxs- [ shmid=0x1000001 ]
FB000000 16392 16 - - rwxs- [ shmid=0x1000000 ]
FC400000 16392 16 - - rwxs- [ shmid=0x7f ]
FD580000 2464 16 - - rwxs- [ shmid=0x1000002 ]
FD800000 16416 24 - - rwxs- [ shmid=0x7e ]
FEA40000 32 32 - - r-x-- nss_files.so.1
FEA58000 8 8 - - rwx-- nss_files.so.1
FEA5C000 8 8 - - rwxs- [ anon ]
FEA70000 64 64 8 - rwx-- [ anon ]
FEA90000 64 32 - - rwx-- [ anon ]
FEAB0000 8 8 - - rwx-- [ anon ]
FEAC0000 128 88 - - r-x-- libelf.so.1
FEAE0000 8 8 - - rwx-- libelf.so.1
FEAF0000 80 16 - - r-x-- libmd.so.1
FEB14000 8 8 - - rwx-- libmd.so.1
FEB20000 32 16 - - r-x-- libaio.so.1
FEB38000 8 8 8 - rwx-- libaio.so.1
FEB40000 40 24 - - r-x-- libintl.so.3.4.3
FEB58000 8 8 - - rwx-- libintl.so.3.4.3
FEB60000 40 32 - - r-x-- libgcc_s.so.1
FEB78000 8 8 - - rwx-- libgcc_s.so.1
FEB80000 888 832 - - r-x-- libc.so.1
FEC6E000 32 32 24 - rwx-- libc.so.1
FEC76000 8 8 8 - rwx-- libc.so.1
FEC80000 920 32 - - r-x-- libiconv.so.2.4.0
FED74000 16 16 - - rwx-- libiconv.so.2.4.0
FED80000 248 208 - - r-x-- libresolv.so.2
FEDCE000 16 16 - - rwx-- libresolv.so.2
FEDE0000 8 8 - - r-x-- libkstat.so.1
FEDF2000 8 8 - - rwx-- libkstat.so.1
FEE00000 680 144 - - r-x-- libm.so.2
FEEB8000 32 24 - - rwx-- libm.so.2
FEED0000 24 16 8 - rwx-- [ anon ]
FEEE0000 16 8 - - r-x-- libkvm.so.1
FEEF4000 8 8 - - rwx-- libkvm.so.1
FEF00000 584 304 - - r-x-- libnsl.so.1
FEFA2000 40 40 8 - rwx-- libnsl.so.1
FEFAC000 24 16 - - rwx-- libnsl.so.1
FEFC0000 8 8 - - rwx-- [ anon ]
FEFD0000 8 8 - - rwx-- [ anon ]
FEFE0000 72 24 - - r-x-- libz.so.1.0.2
FF000000 8 8 - - rwx-- libz.so.1.0.2
FF010000 8 8 - - r-x-- libdl.so.1
FF022000 8 8 - - rwx-- libdl.so.1
FF030000 48 40 - - r-x-- libsocket.so.1
FF04C000 8 8 8 - rwx-- libsocket.so.1
FF060000 24 24 - - r-x-- librt.so.1
FF076000 8 8 - - rwx-- librt.so.1
FF080000 1168 744 - - r-x-- libcrypto.so.0.9.8
FF1B2000 96 96 - - rwx-- libcrypto.so.0.9.8
FF1CA000 8 - - - rwx-- libcrypto.so.0.9.8
FF1E0000 256 256 - - r-x-- libssl.so.0.9.8
FF22E000 24 24 - - rwx-- libssl.so.0.9.8
FF240000 192 32 - - r-x-- libidn.so.11.6.1
FF27E000 16 16 - - rwx-- libidn.so.11.6.1
FF290000 8 8 - - rwx-- [ anon ]
FF2A0000 296 64 - - r-x-- libcurl.so.4.2.0
FF2F8000 16 16 - - rwx-- libcurl.so.4.2.0
FF300000 488 456 - - r-x-- libsqlite3.so.0.8.6
FF388000 16 16 16 - rwx-- libsqlite3.so.0.8.6
FF3A0000 16 16 - - r-x-- libc_psr.so.1
FF3B0000 208 208 - - r-x-- ld.so.1
FF3F4000 8 8 8 - rwx-- ld.so.1
FF3F6000 8 8 8 - rwx-- ld.so.1
FFBC8000 224 224 176 - rwx-- [ stack ]
-------- ------- ------- ------- -------
total Kb 74208 5656 944 -
> /bin/pfiles 10318
10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
Current rlimit: 256 file descriptors
0: S_IFCHR mode:0666 dev:311,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDONLY
/devices/pseudo/mm@0:null
1: S_IFREG mode:0664 dev:136,8 ino:552348 uid:125 gid:1 size:5330396
O_WRONLY|O_APPEND|O_CREAT
/opt/thunder/var/log/zabbix_proxy.log
2: S_IFREG mode:0664 dev:136,8 ino:552348 uid:125 gid:1 size:5330396
O_WRONLY|O_APPEND|O_CREAT
/opt/thunder/var/log/zabbix_proxy.log
3: S_IFREG mode:0664 dev:136,8 ino:734536 uid:125 gid:1 size:5
O_WRONLY|O_CREAT|O_TRUNC
advisory write lock set by process 10291
/opt/thunder/var/run/zabbix_proxy.pid
4: S_IFSOCK mode:0666 dev:317,0 ino:57069 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_ NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 10052
5: S_IFREG mode:0644 dev:136,8 ino:552349 uid:125 gid:1 size:336896
O_RDWR|O_LARGEFILE FD_CLOEXEC
/opt/thunder/var/data/zabbix_proxy.db
6: S_IFSOCK mode:0666 dev:317,0 ino:21831 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_ NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 0
10318:20100629:084210.117 Timeout while answering request
10318:20100629:084210.117 Got signal [signal:11(SIGSEGV),reason:1,refaddr:c]. Crashing ...
10318:20100629:084210.117 ====== Fatal information: ======
10318:20100629:084210.117 program counter not available for this architecture
10318:20100629:084210.117 === Registers: ===
10318:20100629:084210.117 register dump not available for this architecture
10318:20100629:084210.117 === Backtrace: ===
10318:20100629:084210.117 backtrace not available for this platform
10318:20100629:084210.118 === Memory map: ===
10318:20100629:084210.118 memory map not available for this platform
10318:20100629:084210.118 ================================
10291:20100629:084210.126 One child process died (PID:10318,exitcode/signal:-1). Exiting ...
10291:20100629:084210.126 zbx_on_exit() called
Output from runme_on_app_crash
Program: zabbix_proxy
Process ID: 10318
Received signal: 11
Application Debugging Data
--------------------------
> /bin/pstack 10318
10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
0003ab68 child_signal_handler (d, 622a0, ffbeb8f0, 0, 0, 0) + 100
fec44b4c __sighndlr (d, 0, ffbeb8f0, 3aa68, 0, 1) + c
fec39b24 call_user_handler (d, 0, 8, 0, feed2a00, ffbeb8f0) + 3b8
fec45f64 _write (6, ffbebc68, 8, 0, 0, 0) + c
000448e4 zbx_tcp_send_ext (ffbfef40, ffbebce4, 0, 0, ffbebc70, a39b0) + 9c
00045668 zbx_send_response (ffbfef40, 0, ffffefe8, 3, fa, 73800) + cc
00022750 process_trapper_child (ffbfef40, 5c8b8, 0, e, 10b4, fec73a80) + 534
00022cf4 child_trapper_main (5c800, ffbfef40, 9f000, 9e800, ffbfeed0, a39b0) + b4
00018d2c MAIN_ZABBIX_ENTRY (1a, fec39cac, 85c00, 85c00, 85c00, 85c00) + 570
0003af04 daemon_start (ffbffbd0, ffbffc6c, ffbffc7c, a3cfc, feed0140, feed0180) + 2fc
00017edc _start (0, 0, 0, 0, 0, 0) + 5c
> /bin/pmap -x 10318
10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
Address Kbytes RSS Anon Locked Mode Mapped File
00010000 408 352 - - r-x-- zabbix_proxy
00084000 104 24 8 - rwx-- zabbix_proxy
0009E000 16 8 8 - rwx-- zabbix_proxy
000A2000 744 680 648 - rwx-- [ heap ]
FA000000 13928 96 - - rwxs- [ shmid=0x1000001 ]
FB000000 16392 16 - - rwxs- [ shmid=0x1000000 ]
FC400000 16392 16 - - rwxs- [ shmid=0x7f ]
FD580000 2464 16 - - rwxs- [ shmid=0x1000002 ]
FD800000 16416 24 - - rwxs- [ shmid=0x7e ]
FEA40000 32 32 - - r-x-- nss_files.so.1
FEA58000 8 8 - - rwx-- nss_files.so.1
FEA5C000 8 8 - - rwxs- [ anon ]
FEA70000 64 64 8 - rwx-- [ anon ]
FEA90000 64 32 - - rwx-- [ anon ]
FEAB0000 8 8 - - rwx-- [ anon ]
FEAC0000 128 88 - - r-x-- libelf.so.1
FEAE0000 8 8 - - rwx-- libelf.so.1
FEAF0000 80 16 - - r-x-- libmd.so.1
FEB14000 8 8 - - rwx-- libmd.so.1
FEB20000 32 16 - - r-x-- libaio.so.1
FEB38000 8 8 8 - rwx-- libaio.so.1
FEB40000 40 24 - - r-x-- libintl.so.3.4.3
FEB58000 8 8 - - rwx-- libintl.so.3.4.3
FEB60000 40 32 - - r-x-- libgcc_s.so.1
FEB78000 8 8 - - rwx-- libgcc_s.so.1
FEB80000 888 832 - - r-x-- libc.so.1
FEC6E000 32 32 24 - rwx-- libc.so.1
FEC76000 8 8 8 - rwx-- libc.so.1
FEC80000 920 32 - - r-x-- libiconv.so.2.4.0
FED74000 16 16 - - rwx-- libiconv.so.2.4.0
FED80000 248 208 - - r-x-- libresolv.so.2
FEDCE000 16 16 - - rwx-- libresolv.so.2
FEDE0000 8 8 - - r-x-- libkstat.so.1
FEDF2000 8 8 - - rwx-- libkstat.so.1
FEE00000 680 144 - - r-x-- libm.so.2
FEEB8000 32 24 - - rwx-- libm.so.2
FEED0000 24 16 8 - rwx-- [ anon ]
FEEE0000 16 8 - - r-x-- libkvm.so.1
FEEF4000 8 8 - - rwx-- libkvm.so.1
FEF00000 584 304 - - r-x-- libnsl.so.1
FEFA2000 40 40 8 - rwx-- libnsl.so.1
FEFAC000 24 16 - - rwx-- libnsl.so.1
FEFC0000 8 8 - - rwx-- [ anon ]
FEFD0000 8 8 - - rwx-- [ anon ]
FEFE0000 72 24 - - r-x-- libz.so.1.0.2
FF000000 8 8 - - rwx-- libz.so.1.0.2
FF010000 8 8 - - r-x-- libdl.so.1
FF022000 8 8 - - rwx-- libdl.so.1
FF030000 48 40 - - r-x-- libsocket.so.1
FF04C000 8 8 8 - rwx-- libsocket.so.1
FF060000 24 24 - - r-x-- librt.so.1
FF076000 8 8 - - rwx-- librt.so.1
FF080000 1168 744 - - r-x-- libcrypto.so.0.9.8
FF1B2000 96 96 - - rwx-- libcrypto.so.0.9.8
FF1CA000 8 - - - rwx-- libcrypto.so.0.9.8
FF1E0000 256 256 - - r-x-- libssl.so.0.9.8
FF22E000 24 24 - - rwx-- libssl.so.0.9.8
FF240000 192 32 - - r-x-- libidn.so.11.6.1
FF27E000 16 16 - - rwx-- libidn.so.11.6.1
FF290000 8 8 - - rwx-- [ anon ]
FF2A0000 296 64 - - r-x-- libcurl.so.4.2.0
FF2F8000 16 16 - - rwx-- libcurl.so.4.2.0
FF300000 488 456 - - r-x-- libsqlite3.so.0.8.6
FF388000 16 16 16 - rwx-- libsqlite3.so.0.8.6
FF3A0000 16 16 - - r-x-- libc_psr.so.1
FF3B0000 208 208 - - r-x-- ld.so.1
FF3F4000 8 8 8 - rwx-- ld.so.1
FF3F6000 8 8 8 - rwx-- ld.so.1
FFBC8000 224 224 176 - rwx-- [ stack ]
-------- ------- ------- ------- -------
total Kb 74208 5656 944 -
> /bin/pfiles 10318
10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
Current rlimit: 256 file descriptors
0: S_IFCHR mode:0666 dev:311,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDONLY
/devices/pseudo/mm@0:null
1: S_IFREG mode:0664 dev:136,8 ino:552348 uid:125 gid:1 size:5330396
O_WRONLY|O_APPEND|O_CREAT
/opt/thunder/var/log/zabbix_proxy.log
2: S_IFREG mode:0664 dev:136,8 ino:552348 uid:125 gid:1 size:5330396
O_WRONLY|O_APPEND|O_CREAT
/opt/thunder/var/log/zabbix_proxy.log
3: S_IFREG mode:0664 dev:136,8 ino:734536 uid:125 gid:1 size:5
O_WRONLY|O_CREAT|O_TRUNC
advisory write lock set by process 10291
/opt/thunder/var/run/zabbix_proxy.pid
4: S_IFSOCK mode:0666 dev:317,0 ino:57069 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_ NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 10052
5: S_IFREG mode:0644 dev:136,8 ino:552349 uid:125 gid:1 size:336896
O_RDWR|O_LARGEFILE FD_CLOEXEC
/opt/thunder/var/data/zabbix_proxy.db
6: S_IFSOCK mode:0666 dev:317,0 ino:21831 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_ NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 0
Comment