Ad Widget

Collapse

Падает zabbix server в defunct.

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Oracle
    Junior Member
    • Mar 2010
    • 6

    #1

    Падает zabbix server в defunct.

    Hi all, наблюдаю следующую проблему :
    процессы zabbix_server падают в defunct после того как в интерфейсе меняю статус сервера Not monitoring на monitoring.

    Конфигурация сервера:
    Solaris10, apache2.2.13, php5.3.1, oracle_client10g 32bit, Oracle Server 10.2.0.4 (проинсталлирован на другом сервере), zabbix 1.8.1.

    Дополнительные пакеты:
    -bash-3.00$ pkginfo | sort -k 2 |egrep 'SMC'
    application SMCatk atk
    application SMCautoc autoconf
    application SMCautom automake
    application SMCbzip2 bzip2
    application SMCcairo cairo
    application SMCccrypt ccrypt
    application SMCcgrep cgrep
    application SMCcoreu coreutils
    application SMCcpio cpio
    application SMCcurl curl
    application SMCdb db
    application SMCdbus dbus
    application SMCdiffu diffutils
    application SMCdiskt disktool
    application SMCevtlog eventlog
    application SMCexpat expat
    application SMCfindu findutils
    application SMCfontc fontconfig
    application SMCfping fping
    application SMCftype freetype
    application SMCgawk gawk
    application SMCgcc gcc
    application SMCgcrypt libgcrypt
    application SMCgd gd
    application SMCgdb gdb
    application SMCgdbm gdbm
    application SMCgettxt gettext
    application SMCglib glib
    application SMCgmp gmp
    application SMCgnuplot gnuplot
    application SMCgrsync grsync
    application SMCgtk gtk+
    application SMCgtk1 gtk+
    application SMCgtksp gtkspell
    application SMCintlt intltool
    application SMCjpeg jpeg
    application SMCless less
    application SMClgcc346 libgcc
    application SMClgtkh libgtkhtml
    application SMClibart libart_lgpl
    application SMClibgcc libgcc
    application SMClibidn libidn
    application SMClibpng libpng
    application SMCliconv libiconv
    application SMClintl libintl
    application SMClogr logrotate
    application SMClpcap lpcap
    application SMClsgsgv libsigsegv
    application SMClsof lsof
    application SMClssh2 libssh2
    application SMClungif libungif
    application SMClxml2 libxml2
    application SMCm4 m4
    application SMCmake make
    application SMCmc mc
    application SMCmd5 md5
    application SMCmtail multitail
    application SMCnc nc
    application SMCncurs ncurses
    application SMCnetsnmp netsnmp
    application SMCngrep ngrep
    application SMCnload nload
    application SMCorbt2 orbit2
    application SMCossl openssl
    application SMCpango pango
    application SMCpconf pkgconfig
    application SMCpcre pcre
    application SMCpopt popt
    application SMCprngd prngd
    application SMCputty putty
    application SMCpython python
    application SMCqt qt
    application SMCreadl readline
    application SMCrender render
    application SMCrenpro renderproto
    application SMCrrdt rrdtool
    application SMCrssh rssh
    application SMCrsync rsync
    application SMCsasl sasl
    application SMCscrn screen
    application SMCsed sed
    application SMCslang slang
    application SMCsudo sudo
    application SMCsudosh2 sudosh2
    application SMCsyslng syslogng
    application SMCtar tar
    application SMCtcl tcl
    application SMCtcpd tcpdump
    application SMCtcpdwr tcp_wrappers
    application SMCtiff tiff
    application SMCtk tk
    application SMCtop top
    application SMCunrar unrar
    application SMCunzip unzip
    application SMCutf8p utf8proc
    application SMCvim vim
    application SMCwxgtk wxgtk
    application SMCx11vnc x11vnc
    application SMCxcpus xcpustate
    application SMCxft xft
    application SMCxmlpsr xmlparser
    application SMCxpm xpm
    application SMCxrend xrender
    application SMCxtail xtail
    application SMCxterm xterm
    application SMCzip zip
    application SMCzlib zlib


    Компиляция apache выполнена с опциями:
    ------------------------------------------
    cd /app01/zabbix/apache/httpd-2.2.13
    ./configure --prefix=/app01/zabbix/apache --enable-so --enable-cgi --enable-info --enable-rewrite --enable-speling \
    --enable-usertrack --enable-deflate --enable-mime-magic --with-mpm=worker --enable-cache --with-port=4444

    Компиляция php выполнена с опциями:
    ------------------------------------------
    cd /app01/zabbix/php5/5.3.1/
    ./configure --prefix=/app01/zabbix/php5 \
    --with-apxs2=/app01/zabbix/apache/bin/apxs \
    --with-config-file-path=/app01/zabbix/php5 \
    --enable-cgi \
    --with-zlib \
    --with-gdbm --with-libxml-dir --with-openssl --with-pcre-regex --without-sqlite3 \
    --with-bz2 --with-gettext --enable-dba --with-mhash --without-sqlite --without-pdo-sqlite \
    --enable-bcmath \
    --with-curl \
    --with-gd \
    --enable-mbstring \
    --enable-gd-jis-conv \
    --enable-gd-native-ttf \
    --enable-sockets \
    --with-oci8=/app01/zabbix/oracle/client10g \
    --with-pdo-oci=/app01/zabbix/oracle/client10g \
    --with-snmp

    Установка репозитария oracle:
    ------------------------------------------
    Изменен размер varchar2(2048) на varchar2(2000)

    Компиляция zabbix выполнена с опциями:
    ------------------------------------------
    ./configure --prefix=/app01/zabbix/zabbix-1.8 --enable-server --with-oracle=/app01/zabbix/oracle/client10g --with-net-snmp --with-libcurl --with-ssh2 --with-PACKAGE=SMClssh2


    Диагностика ошибок:
    -------------------------------------------
    В логе после падения - ничего не появляется
    последние строки такие ...

    23588:20100302:125355.580 In process_escalations()
    23588:20100302:125355.580 Query [txnlev:0] [select escalationid,actionid,triggerid,eventid,r_eventid, esc_step,status from escalations where status in (0,1) and nextcheck<=1267523635 and escalationid between 000000000000000 and 099999999999999]
    23588:20100302:125355.586 Escalator spent 0.005911 seconds while processing escalation items. Nextcheck after 3 sec.
    23568:20100302:125356.450 Syncing ...
    23568:20100302:125356.450 In DCsync_confguration()
    23568:20100302:125356.450 In DCsync_hosts()
    23568:20100302:125356.450 Query [txnlev:0] [select hostid,proxy_hostid,host,useip,ip,dns,port,status, useipmi,ipmi_ip,ipmi_port,ipmi_authtype,ipmi_privi lege,ipmi_username,ipmi_password,maintenance_statu s,maintenance_type,maintenance_from,errors_from,av ailable,disable_until,snmp_errors_from,snmp_availa ble,snmp_disable_until,ipmi_errors_from,ipmi_avail able,ipmi_disable_until from hosts where status in (0) and hostid between 000000000000000 and 099999999999999 order by hostid]
    23584:20100302:125356.510 Starting sync with nodes
    23584:20100302:125356.510 Connect to the database
    23584:20100302:125356.620 In main_historysender()
    23584:20100302:125356.623 Sleeping 10 seconds
    23556:20100302:125356.634 One child process died (PID:23568). Exiting ...
    23556:20100302:125358.640 Connect to the database
    23556:20100302:125358.736 In free_database_cache()
    23556:20100302:125358.736 In DCsync_all()
    23556:20100302:125358.736 In DCsync_history(history_first:0 history_num:0)
    23556:20100302:125358.737 Syncing history data...
    23556:20100302:125358.737 In DCsync_trends() trends_num:0
    23556:20100302:125358.737 Syncing trends data...
    23556:20100302:125358.737 Syncing trends data...done.
    23556:20100302:125358.737 End of DCsync_trends()
    23556:20100302:125358.737 End of DCsync_all()
    23556:20100302:125358.737 End of free_database_cache()
    23556:20100302:125358.737 In free_configuration_cache()




    Было

    root@nms2:bin #/usr/ucb/ps auxwww | grep zabbix_ser
    zabbix 23584 0.1 0.98541636192 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23585 0.1 0.98541636120 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23588 0.1 0.98541636192 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    root 1671 0.0 0.1 1616 1032 pts/11 S 12:53:50 0:00 grep zabbix_ser
    zabbix 28732 0.0 0.1 3936 1648 pts/6 S 12:52:34 0:00 tail -f /tmp/zabbix_server.log
    zabbix 23569 0.0 1.18596841696 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23570 0.0 1.18596841712 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23571 0.0 1.18596841712 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23572 0.0 1.18596841712 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23573 0.0 1.18596841720 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23579 0.0 1.08604040792 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23583 0.0 1.18596841704 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23556 0.0 1.18541645496 ? S 12:39:55 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23568 0.0 1.08541640264 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23574 0.0 0.98541636136 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23575 0.0 0.98541636128 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23576 0.0 0.98541636120 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23577 0.0 0.98541636128 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23578 0.0 0.98541636144 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23580 0.0 0.98541636096 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23581 0.0 0.98541636128 ? S 12:39:56 0:11 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23582 0.0 0.98541636144 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23586 0.0 0.98565637336 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 23587 0.0 1.08541640304 ? S 12:39:56 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf


    стало
    #/usr/ucb/ps auxwww | grep zabbix_ser
    root 2311 0.0 0.1 1616 1032 pts/11 S 12:54:06 0:00 grep zabbix_ser
    zabbix 23556 0.0 1.18541645504 ? S 12:39:55 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 28732 0.0 0.1 3936 1648 pts/6 S 12:52:34 0:00 tail -f /tmp/zabbix_server.log
    root@nms2:bin #/usr/ucb/ps auxwww | grep zabbix
    zabbix 21906 0.1 1.57928058576 ? S 12:22:43 0:03 /app01/zabbix/apache/bin/httpd -k start
    zabbix 21907 0.1 1.57856857912 ? S 12:22:43 0:01 /app01/zabbix/apache/bin/httpd -k start
    root 3871 0.0 0.1 1616 1032 pts/11 S 12:54:20 0:00 grep zabbix
    zabbix 23556 0.0 1.18541645504 ? S 12:39:55 0:00 /app01/zabbix/zabbix-1.8/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
    zabbix 21827 0.0 1.26415246624 ? S 12:22:41 0:00 /app01/zabbix/apache/bin/httpd -k start
    zabbix 28732 0.0 0.1 3936 1648 pts/6 S 12:52:34 0:00 tail -f /tmp/zabbix_server.log
    zabbix 19217 0.0 0.1 3504 2384 pts/6 S 11:20:03 0:00 -bash
    zabbix 21828 0.0 0.159400 1992 ? S 12:22:41 0:00 /app01/zabbix/apache/bin/httpd -k start
    zabbix 21896 0.0 0.322632 9776 ? S Feb 03 0:57 Xvnc :3 -desktop nms2:3 () -auth /export/home/zabbix/.Xauthority -geometry 1024x768 -depth 16 -rfbwait 30000 -rfbauth /export/home/zabbix/.vnc/passwd -rfbport 5903 -pn
    zabbix 21905 0.0 1.47851257104 ? S 12:22:43 0:01 /app01/zabbix/apache/bin/httpd -k start
    zabbix 22032 0.0 1.47928057136 ? S 12:22:46 0:02 /app01/zabbix/apache/bin/httpd -k start
    zabbix 22223 0.0 0.1 3992 736 ? S Feb 03 0:00 vncconfig -iconic
    zabbix 22225 0.0 0.1 4400 720 ? S Feb 03 0:00 twm
    zabbix 22283 0.0 0.1 3416 944 pts/7 S Feb 03 0:00 -bash
    zabbix 23568 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23569 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23570 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23571 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23572 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23573 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23574 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23575 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23576 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23577 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23578 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23579 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23580 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23581 0.0 0.0 0 0 Z 0:11 <defunct>
    zabbix 23582 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23583 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23584 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23585 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23586 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23587 0.0 0.0 0 0 Z 0:00 <defunct>
    zabbix 23588 0.0 0.0 0 0 Z 0:00 <defunct>
    #

    Need help.
    --
    С уважением, Александр.
  • Oracle
    Junior Member
    • Mar 2010
    • 6

    #2
    Кусочек, трассировки головного процесса zabbix_server pid=25932


    25932: 43.8374 0.0004 0.0001 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 8
    25932: 43.8376 0.0002 0.0000 getpid() = 25932 [1]
    25932: 43.8377 0.0001 0.0000 fstat64(8, 0xFFBECE28) = 0
    25932: 43.8378 0.0001 0.0000 fstat64(8, 0xFFBECCD0) = 0
    25932: 43.8378 0.0000 0.0000 ioctl(8, TCGETA, 0xFFBECDB4) Err#25 ENOTTY
    25932: 43.8381 0.0003 0.0000 write(8, " 2 5 9 3 2 : 2 0 1 0 0".., 52) = 52
    25932: 43.8382 0.0001 0.0000 close(8) = 0
    25932: 43.8383 0.0001 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEE428) = 0
    25932: nanosleep(0xFFBFE4B8, 0xFFBFE4B0) (sleeping...)
    25932: 101.8263 57.9880 0.0000 Received signal #18, SIGCLD, in nanosleep() [caught]
    25932: siginfo: SIGCLD CLD_EXITED pid=25933 status=0x0001
    25932: 101.8266 0.0003 0.0001 nanosleep(0xFFBFE4B8, 0xFFBFE4B0) Err#4 EINTR
    25932: 101.8267 0.0001 0.0000 lwp_sigmask(SIG_SETMASK, 0x00020000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
    25932: 101.8268 0.0001 0.0000 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 8
    25932: 101.8270 0.0002 0.0000 getpid() = 25932 [1]
    25932: 101.8271 0.0001 0.0000 fstat64(8, 0xFFBEC850) = 0
    25932: 101.8272 0.0001 0.0000 fstat64(8, 0xFFBEC6F8) = 0
    25932: 101.8272 0.0000 0.0000 ioctl(8, TCGETA, 0xFFBEC7DC) Err#25 ENOTTY
    25932: 101.8273 0.0001 0.0000 write(8, " 2 5 9 3 2 : 2 0 1 0 0".., 75) = 75
    25932: 101.8274 0.0001 0.0000 close(8) = 0
    25932: 101.8275 0.0001 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEDE50) = 0
    25932: 101.8277 0.0002 0.0000 kill(25933, SIGTERM) = 0
    25932: 101.8278 0.0001 0.0000 kill(25934, SIGTERM) = 0
    25932: 101.8280 0.0002 0.0000 kill(25935, SIGTERM) = 0
    25932: 101.8281 0.0001 0.0000 kill(25936, SIGTERM) = 0
    25932: 101.8282 0.0001 0.0000 kill(25937, SIGTERM) = 0
    25932: 101.8282 0.0000 0.0000 kill(25938, SIGTERM) = 0
    25932: 101.8283 0.0001 0.0000 kill(25939, SIGTERM) = 0
    25932: 101.8285 0.0002 0.0000 kill(25940, SIGTERM) = 0
    25932: 101.8285 0.0000 0.0000 kill(25941, SIGTERM) = 0
    25932: 101.8286 0.0001 0.0000 kill(25942, SIGTERM) = 0
    25932: 101.8287 0.0001 0.0000 kill(25943, SIGTERM) = 0
    25932: 101.8288 0.0001 0.0000 kill(25944, SIGTERM) = 0
    25932: 101.8288 0.0000 0.0000 kill(25945, SIGTERM) = 0
    25932: 101.8289 0.0001 0.0000 kill(25946, SIGTERM) = 0
    25932: 101.8290 0.0001 0.0000 kill(25947, SIGTERM) = 0
    25932: 101.8291 0.0001 0.0000 kill(25948, SIGTERM) = 0
    25932: 101.8291 0.0000 0.0000 kill(25949, SIGTERM) = 0
    25932: 101.8292 0.0001 0.0000 kill(25950, SIGTERM) = 0
    25932: 101.8293 0.0001 0.0000 kill(25951, SIGTERM) = 0
    25932: 101.8294 0.0001 0.0000 kill(25952, SIGTERM) = 0
    25932: 101.8294 0.0000 0.0000 kill(25953, SIGTERM) = 0
    25932: 101.8295 0.0001 0.0000 getpid() = 25932 [1]
    25932: 101.8296 0.0001 0.0000 fcntl(3, F_SETLK, 0xFFBFDE40) = 0
    25932: 101.8297 0.0001 0.0000 close(3) = 0
    25932: 101.8298 0.0001 0.0000 unlink("/tmp/zabbix_server.pid") = 0
    25932: nanosleep(0xFFBFDE70, 0xFFBFDE68) (sleeping...)
    25932: 103.8322 2.0024 0.0001 nanosleep(0xFFBFDE70, 0xFFBFDE68) = 0
    25932: 103.8326 0.0004 0.0001 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 3
    25932: 103.8329 0.0003 0.0000 getpid() = 25932 [1]
    25932: 103.8330 0.0001 0.0000 fstat64(3, 0xFFBEC770) = 0
    25932: 103.8331 0.0001 0.0000 fstat64(3, 0xFFBEC618) = 0
    25932: 103.8332 0.0001 0.0000 ioctl(3, TCGETA, 0xFFBEC6FC) Err#25 ENOTTY
    25932: 103.8334 0.0002 0.0000 write(3, " 2 5 9 3 2 : 2 0 1 0 0".., 51) = 51
    25932: 103.8336 0.0002 0.0000 close(3) = 0
    25932: 103.8337 0.0001 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEDD70) = 0
    ..............

    Comment

    • Oracle
      Junior Member
      • Mar 2010
      • 6

      #3
      Кусочек, трассировки дочернего процесса zabbix_server pid=25933

      из /tmp/zabbix_server.log
      25932:20100302:164305.594 One child process died (PID:25933). Exiting ...


      .....
      25933: 40.6703 0.0000 0.0000 ioctl(9, TCGETA, 0xFFBED464) Err#25 ENOTTY
      25933: 40.6704 0.0001 0.0000 write(9, " 2 5 9 3 3 : 2 0 1 0 0".., 49) = 49
      25933: 40.6705 0.0001 0.0000 close(9) = 0
      25933: 40.6706 0.0001 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEEAD8) = 0
      25933: 40.6707 0.0001 0.0000 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 9
      25933: 40.6708 0.0001 0.0000 getpid() = 25933 [25932]
      25933: 40.6709 0.0001 0.0000 fstat64(9, 0xFFBED580) = 0
      25933: 40.6710 0.0001 0.0000 fstat64(9, 0xFFBED428) = 0
      25933: 40.6710 0.0000 0.0000 ioctl(9, TCGETA, 0xFFBED50C) Err#25 ENOTTY
      25933: 40.6711 0.0001 0.0000 write(9, " 2 5 9 3 3 : 2 0 1 0 0".., 91) = 91
      25933: 40.7630 0.0919 0.0000 close(9) = 0
      25933: 40.7632 0.0002 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEEB80) = 0
      25933: 40.7634 0.0002 0.0000 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 9
      25933: 40.7635 0.0001 0.0000 getpid() = 25933 [25932]
      25933: 40.7636 0.0001 0.0000 fstat64(9, 0xFFBED620) = 0
      25933: 40.7637 0.0001 0.0000 fstat64(9, 0xFFBED4C8) = 0
      25933: 40.7638 0.0001 0.0000 ioctl(9, TCGETA, 0xFFBED5AC) Err#25 ENOTTY
      25933: 40.7639 0.0001 0.0000 write(9, " 2 5 9 3 3 : 2 0 1 0 0".., 119) = 119
      25933: 40.7640 0.0001 0.0000 close(9) = 0
      25933: 40.7640 0.0000 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEEC20) = 0
      25933: nanosleep(0xFFBFECB0, 0xFFBFECA8) (sleeping...)
      25933: 100.7713 60.0073 0.0000 nanosleep(0xFFBFECB0, 0xFFBFECA8) = 0
      25933: 100.7725 0.0012 0.0000 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 9
      25933: 100.7727 0.0002 0.0000 getpid() = 25933 [25932]
      25933: 100.7728 0.0001 0.0000 fstat64(9, 0xFFBED620) = 0
      25933: 100.7728 0.0000 0.0000 fstat64(9, 0xFFBED4C8) = 0
      25933: 100.7729 0.0001 0.0000 ioctl(9, TCGETA, 0xFFBED5AC) Err#25 ENOTTY
      25933: 100.7730 0.0001 0.0000 write(9, " 2 5 9 3 3 : 2 0 1 0 0".., 39) = 39
      25933: 100.7731 0.0001 0.0000 close(9) = 0
      25933: 100.7732 0.0001 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEEC20) = 0
      25933: 100.7733 0.0001 0.0000 close(0) = 0
      25933: 100.7734 0.0001 0.0001 open("/dev/null", O_RDONLY) = 0
      25933: 100.7735 0.0001 0.0000 open("/tmp/zabbix_server.log", O_WRONLY|O_APPEND|O_CREAT, 0666) = 9
      25933: 100.7737 0.0002 0.0000 fcntl(9, F_DUP2FD, 0x00000002) = 2
      25933: 100.7737 0.0000 0.0000 fcntl(9, F_DUP2FD, 0x00000001) = 1
      25933: 100.7738 0.0001 0.0000 close(9) = 0
      25933: 100.7739 0.0001 0.0000 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 9
      25933: 100.7740 0.0001 0.0000 getpid() = 25933 [25932]
      25933: 100.7741 0.0001 0.0000 fstat64(9, 0xFFBED580) = 0
      25933: 100.7742 0.0001 0.0000 fstat64(9, 0xFFBED428) = 0
      25933: 100.7742 0.0000 0.0000 ioctl(9, TCGETA, 0xFFBED50C) Err#25 ENOTTY
      25933: 100.7743 0.0001 0.0000 write(9, " 2 5 9 3 3 : 2 0 1 0 0".., 52) = 52
      25933: 100.7744 0.0001 0.0000 close(9) = 0
      25933: 100.7745 0.0001 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEEB80) = 0
      25933: 100.7746 0.0001 0.0000 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 9
      25933: 100.7747 0.0001 0.0000 getpid() = 25933 [25932]
      25933: 100.7748 0.0001 0.0000 fstat64(9, 0xFFBED580) = 0
      25933: 100.7749 0.0001 0.0000 fstat64(9, 0xFFBED428) = 0
      25933: 100.7749 0.0000 0.0000 ioctl(9, TCGETA, 0xFFBED50C) Err#25 ENOTTY
      25933: 100.7750 0.0001 0.0000 write(9, " 2 5 9 3 3 : 2 0 1 0 0".., 45) = 45
      25933: 100.7752 0.0002 0.0000 close(9) = 0
      25933: 100.7752 0.0000 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEEB80) = 0
      25933: 100.7754 0.0002 0.0000 open("/tmp/zabbix_server.log", O_RDWR|O_APPEND|O_CREAT, 0666) = 9
      25933: 100.7755 0.0001 0.0000 getpid() = 25933 [25932]
      25933: 100.7755 0.0000 0.0000 fstat64(9, 0xFFBED478) = 0
      25933: 100.7756 0.0001 0.0000 fstat64(9, 0xFFBED320) = 0
      25933: 100.7757 0.0001 0.0000 ioctl(9, TCGETA, 0xFFBED404) Err#25 ENOTTY
      25933: 100.7758 0.0001 0.0000 write(9, " 2 5 9 3 3 : 2 0 1 0 0".., 479) = 479
      25933: 100.7759 0.0001 0.0000 close(9) = 0
      25933: 100.7760 0.0001 0.0000 stat("/tmp/zabbix_server.log", 0xFFBEEA78) = 0
      25933: 100.7762 0.0002 0.0000 write(8, "02 6\0\006\0\0\0\0\011 i".., 566) = 566
      25933: 100.7778 0.0016 0.0000 read(8, "07 -\0\006\0\0\0\0\01017".., 8208) = 1837
      25933: 100.7784 0.0006 0.0000 semop(64, 0xFFBFEC00, 1) = 0
      25933: 100.7788 0.0004 0.0001 Incurred fault #5, FLTACCESS %pc = 0x0004087C
      25933: siginfo: SIGBUS BUS_ADRALN addr=0xF98000C4
      25933: 100.7790 0.0002 0.0001 Received signal #10, SIGBUS [caught]
      25933: siginfo: SIGBUS BUS_ADRALN addr=0xF98000C4
      25933: 100.7791 0.0001 0.0000 lwp_sigmask(SIG_SETMASK, 0x00000200, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
      25933: 100.7793 0.0002 0.0000 lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
      25933: 100.7804 0.0011 0.0000 sysinfo(SI_HOSTNAME, "nms2", 241) = 11
      25933: 100.7805 0.0001 0.0000 getpid() = 25933 [25932]
      25933: 100.7808 0.0003 0.0000 lwp_sigmask(SIG_SETMASK, 0xFFBFF8FF, 0x0000FFF7) = 0xFFBFFEFF [0x0000FFFF]
      25933: 100.7810 0.0002 0.0000 sigaction(SIGBUS, 0x00000000, 0xFEE47AA0) = 0
      25933: 100.7811 0.0001 0.0000 sigaction(SIGBUS, 0xFFBF5E78, 0x00000000) = 0
      25933: 100.7812 0.0001 0.0000 sigaction(SIGSEGV, 0x00000000, 0xFEE47A80) = 0
      25933: 100.7813 0.0001 0.0000 sigaction(SIGSEGV, 0xFFBF5E78, 0x00000000) = 0
      25933: 100.7814 0.0001 0.0000 sigaction(SIGBUS, 0xFFBF5E78, 0x00000000) = 0
      25933: 100.7815 0.0001 0.0000 sigaction(SIGSEGV, 0xFFBF5E78, 0x00000000) = 0
      25933: 100.7817 0.0002 0.0000 lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
      25933: 100.7817 0.0000 0.0000 lwp_sigmask(SIG_SETMASK, 0xFFBFF8FF, 0x0000FFF7) = 0xFFBFFEFF [0x0000FFFF]
      .........

      Comment

      • Oracle
        Junior Member
        • Mar 2010
        • 6

        #4
        продолжение
        ...
        25933: 101.3354 0.0001 0.0000 sigaction(SIGBUS, 0x00000000, 0xFEE47AA0) = 0
        25933: 101.3355 0.0001 0.0000 sigaction(SIGBUS, 0xFFBF5E78, 0x00000000) = 0
        25933: 101.3355 0.0000 0.0000 sigaction(SIGSEGV, 0x00000000, 0xFEE47A80) = 0
        25933: 101.3356 0.0001 0.0000 sigaction(SIGSEGV, 0xFFBF5E78, 0x00000000) = 0
        25933: 101.3357 0.0001 0.0000 Incurred fault #6, FLTBOUNDS %pc = 0xFDF14934
        25933: siginfo: SIGSEGV SEGV_MAPERR addr=0x00000008
        25933: 101.3358 0.0001 0.0001 Received signal #11, SIGSEGV [caught]
        25933: siginfo: SIGSEGV SEGV_MAPERR addr=0x00000008
        25933: 101.3359 0.0001 0.0000 lwp_sigmask(SIG_SETMASK, 0xFFBFFCFF, 0x0000FFF7) = 0xFFBFFEFF [0x0000FFFF]
        25933: 101.3361 0.0002 0.0000 setcontext(0xFFBF55E0)
        25933: 101.3361 0.0000 0.0000 sigaction(SIGBUS, 0xFFBF5E78, 0x00000000) = 0
        25933: 101.3362 0.0001 0.0000 sigaction(SIGSEGV, 0xFFBF5E78, 0x00000000) = 0
        25933: 101.3363 0.0001 0.0000 lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
        25933: 101.3370 0.0007 0.0003 open64("/export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/trace
        /ora_25932_1.trc", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0660) = 11
        25933: 101.3372 0.0002 0.0000 llseek(11, 0, SEEK_END) = 0
        25933: 101.3374 0.0002 0.0001 write(11, " T r a c e f i l e /".., 112) = 112
        25933: 101.3377 0.0003 0.0001 open64("/export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/trace
        /ora_25932_1.trm", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0660) = 12
        25933: 101.3378 0.0001 0.0000 llseek(12, 0, SEEK_END) = 0
        25933: 101.3383 0.0005 0.0001 write(12, " @ 2 | 2 | x 4 u N G 1 z".., 43) = 43
        25933: 101.3384 0.0001 0.0000 write(11, " s s k g d s _ g e t c a".., 62) = 62
        25933: 101.3386 0.0002 0.0000 llseek(11, 0, SEEK_CUR) = 174
        25933: 101.3387 0.0001 0.0000 write(11, " s s k g d s _ g e t c a".., 62) = 62
        25933: 101.3389 0.0002 0.0000 write(12, " 6 / x 4 u N G 1 z 8 1 ~".., 16) = 16
        25933: 101.3390 0.0001 0.0000 write(11, " * * * W A R N I N G :".., 43) = 43
        25933: 101.3392 0.0002 0.0001 munmap(0xFC448000, 204800) = 0
        25933: 101.3393 0.0001 0.0000 munmap(0xFCBAE000, 16384) = 0
        25933: 101.3394 0.0001 0.0000 close(10) = 0
        25933: 101.3396 0.0002 0.0001 munmap(0xFC22E000, 1318912) = 0
        25933: 101.3397 0.0001 0.0000 munmap(0xFCBEC000, 8192) = 0
        25933: 101.3398 0.0001 0.0000 close(9) = 0
        25933: 101.3402 0.0004 0.0000 lstat64("/export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/aler
        t/log.xml", 0xFFBF28C0) = 0
        25933: 101.3403 0.0001 0.0000 stat64("/export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/alert
        /log.xml", 0xFFBF2958) = 0
        25933: 101.3407 0.0004 0.0000 open64("/export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/alert
        /log.xml", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0660) = 9
        25933: 101.3409 0.0002 0.0001 write(9, " < m s g t i m e = ' 2".., 324) = 324
        25933: 101.3410 0.0001 0.0000 close(9) = 0
        ...

        А теперь посмотрим вот эти файлики из truss
        /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/trace/ora_25932_1.trc
        /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/trace/ora_25932_1.trm
        /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/alert/log.xml

        Comment

        • Oracle
          Junior Member
          • Mar 2010
          • 6

          #5
          root@nms2:tmp #cat /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/trace/ora_25932_1.trc
          Trace file /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/trace/ora_25932_1.trc
          sskgds_getcall: WARNING! cannot backtrace from signal handler
          sskgds_getcall: WARNING! premature termination of stack trace
          *** WARNING: Unable to skip initial frames
          DDE: Flood control is not active
          Incident 401 created, dump file: /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/incident/incdir_401/ora_25932_1_i401.trc
          oci-24550 [10] [[si_signo=10] [si_errno=0] [si_code=1] [si_addr=f98000c4]] [] [] [] [] [] [] [] [] [] []

          root@nms2:tmp #cat /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/trace/ora_25932_1.trm
          @2|2|x4uNG1z81"25932_1|host_2485964088_76|
          6/x4uNG1z81~k2-
          02s61$dbgexProcessError*diag_dde*dbgex.c*DGhX
          9Ls1c$a1incident_id`39:dbgrimbid_begin_incdump*dbg rim.c*incident_file*14D1X0dm"
          C4001$2u1H6"
          TK$010y4"
          EK006$Error Stack*13063"
          C4001$3f1M3"
          UK$030C5"
          root@nms2:tmp #
          root@nms2:tmp #cat /export/home/zabbix/oradiag_zabbix/diag/clients/user_zabbix/host_2485964088_76/alert/log.xml | less
          <msg time='2010-02-03T18:51:11.416+03:00' org_id='oracle' comp_id='clients'
          msg_id='dbgrmmdcrf_create_relation_full:1943:15034 27940' type='NOTIFICATION' group='create'
          level='16' host_id='nms2' host_addr='10.254.0.20'
          version='1'>
          <txt>Create Relation ADR_CONTROL
          </txt>
          </msg>
          <msg time='2010-02-03T18:51:11.766+03:00' org_id='oracle' comp_id='clients'
          msg_id='dbgrmmdcrf_create_relation_full:1943:15034 27940' type='NOTIFICATION' group='create'
          level='16' host_id='nms2' host_addr='10.254.0.20'>
          <txt>Create Relation ADR_INVALIDATION
          </txt>
          </msg>
          <msg time='2010-02-03T18:51:12.192+03:00' org_id='oracle' comp_id='clients'
          msg_id='dbgrmmdcrf_create_relation_full:1943:15034 27940' type='NOTIFICATION' group='create'
          level='16' host_id='nms2' host_addr='10.254.0.20'>
          <txt>Create Relation INC_METER_IMPT_DEF
          </txt>
          </msg>
          <msg time='2010-02-03T18:51:13.009+03:00' org_id='oracle' comp_id='clients'
          msg_id='dbgrmmdcrf_create_relation_full:1943:15034 27940' type='NOTIFICATION' group='create'
          level='16' host_id='nms2' host_addr='10.254.0.20'>
          <txt>Create Relation INC_METER_PK_IMPTS
          </txt>
          </msg>
          <msg time='2010-02-03T18:51:13.339+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt>Unable to find environment variable: ORACLE_HOME
          </txt>
          </msg>
          <msg time='2010-02-03T18:51:14.723+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt>Unable to find environment variable: ORACLE_HOME
          </txt>
          </msg>
          ............

          И так много много раз про ORACLE_HOME
          org_id='oracle' - Откуда на этом сервере unix-пользователь oracle ?
          тут есть пользователь zabbix и оракловый клиент на нем ?
          Или я не прав ?
          Естественно, т.к. нет пользователя oracle не будет и ORACLE_HOME !
          .............
          <msg time='2010-02-05T15:04:30.366+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt>Directory does not exist for read/write [/app01/zabbix/oracle/client10g/log] []
          </txt>
          </msg>

          ............
          С чего бы это вдруг у oracle клиента должна появить директория log
          как у сервера в $ORACLE_HOME ? На этом хосте где стоит zabbix - ораклового сервера нет !!
          ............

          Ни и далее естественно ошибки соединения Oracle Net

          ************************************************** *********************
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.510+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt>
          Fatal NI connect error 12514, connecting to:
          (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=alfa)(CID =(PROGRAM=httpd)(HOST=nms2)(USER=zabbix)))(ADDRESS =(PROTOCOL=TCP)(HOST=10.2
          54.0.10)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=1 0.254.0.10)(PORT=1521)))
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.510+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt>
          VERSION INFORMATION:
          TNS for Solaris: Version 11.2.0.1.0 - Production
          TCP/IP NT Protocol Adapter for Solaris: Version 11.2.0.1.0 - Production
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.510+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt> Time: 05-FEB-2010 15:06:16
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.511+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt> Tracing not turned on.
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.511+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt> Tns error struct:
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.511+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt> ns main err code: 12564
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.511+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt>
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.634+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt>TNS-12564: TNS:connection refused
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.635+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt> ns secondary err code: 0
          </txt>
          </msg>
          <msg time='2010-02-05T15:06:16.635+03:00' org_id='oracle' comp_id='clients'
          type='UNKNOWN' level='16' host_id='nms2'
          host_addr='10.254.0.20'>
          <txt> nt main err code: 0
          </txt>
          </msg>
          Last edited by Oracle; 02-03-2010, 14:38.

          Comment

          • Oracle
            Junior Member
            • Mar 2010
            • 6

            #6
            Уважаемые, кто-нибудь делал распределенную структуру
            (а именно, с выносом базы репозитария на др. хост) ?

            Comment

            • ZhidkovAN
              Junior Member
              • Dec 2006
              • 16

              #7
              Originally posted by Oracle
              Уважаемые, кто-нибудь делал распределенную структуру
              (а именно, с выносом базы репозитария на др. хост) ?
              А в чем проблема?
              У нас zabbix_server на одном хосте (linux), а БД (oracle, sparc) на другом. Все прекрасно работает... после напильника

              Comment

              Working...