Zabbix Proxy 1.6.4 consistently dying on Solaris 9 (One child process died)

[douglaz]

I'm using sqlite 3.6.14.1 as backend. Agent and server are 1.6.4. It happens on two different Solaris 9 machines. A similar configuration on a linux machine works without problems.

I've tested with zabbix proxy version 1.6.2 and have got the same issues.

Log example:

The problem happens only after the proxy needs to collect data from the agent. (If it's monitoring a agent without items there is no problems).


The first run after adding just one item to the agent (system.uptime):

3348:20090526:155445 Starting zabbix_proxy. ZABBIX 1.6.4.
  3348:20090526:155445 **** Enabled features ****
  3348:20090526:155445 SNMP monitoring:        NO
  3348:20090526:155445 WEB monitoring:         NO
  3348:20090526:155445 ODBC:                   NO
  3348:20090526:155445 IPv6 support:           NO
  3348:20090526:155445 **************************
  3349:20090526:155445 server #1 started [Configuration syncer]
  3351:20090526:155445 server #3 started [Poller. SNMP: NO]
  3352:20090526:155445 server #4 started [Poller. SNMP: NO]
  3353:20090526:155445 server #5 started [Poller. SNMP: NO]
  3350:20090526:155445 server #2 started [Datasender]
  3355:20090526:155445 server #7 started [Poller. SNMP: NO]
  3356:20090526:155445 server #8 started [Trapper]
  3358:20090526:155445 server #10 started [Trapper]
  3360:20090526:155445 server #12 started [Trapper]
  3362:20090526:155445 server #14 started [Housekeeper]
  3362:20090526:155445 Executing housekeeper
  3354:20090526:155445 server #6 started [Poller. SNMP: NO]
  3357:20090526:155445 server #9 started [Trapper]
  3359:20090526:155445 server #11 started [Trapper]
  3364:20090526:155445 server #16 started [Discoverer. SNMP: NO]
  3348:20090526:155445 server #0 started [Heartbeat sender]
  3361:20090526:155445 server #13 started [ICMP pinger]
  3363:20090526:155445 server #15 started [Poller for unreachable hosts. SNMP: NO]
  3362:20090526:155445 Deleted 0 records from history [0.000483 seconds]
  3351:20090526:155450 Enabling host [vivo_apptest]
  3348:20090526:155501 One child process died. Exiting ...
  3348:20090526:155501 zbx_on_exit()
  3348:20090526:155503 ZABBIX Proxy stopped. ZABBIX 1.6.4.

Running it again gives a similar. See the attachment with verbose=4.

Any ideas?

[richlv]

a generic suggestion - compile proxy with debugging information, run it inside gdb, make it crash. if you get a meaningful backtrace, report it on the tracker (though the crash might be in sqlite, in which case it would be of a little use to report it against zabbix).

[douglaz]

a generic suggestion - compile proxy with debugging information, run it inside gdb, make it crash. if you get a meaningful backtrace, report it on the tracker (though the crash might be in sqlite, in which case it would be of a little use to report it against zabbix).

I've tried that (compiled with CFLAGS="-g -DDEBUG"). But I can't get a backtrace because the proxy goes in daemon mode (I think) before dying.

Any tips on debugging this?

[richlv]

i don't think 'daemon mode' should impact gdb (though i'm wondering here in an area i don't quite understand).
you can also try passing something like 'export CFLAGS="-ggdb"' before configure/compile.

even if main process forks subprocesses, you can start it, then attach to all children (something like 'gdb $(which zabbix_proxy) -p <pid of the process>' for each proxy process).

[douglaz]

The problem was in sqlite. A workaround is to recompile sqlite without optimizations (CFLAGS="-O0").

Thanks.