One server process died. Shutting down...

[gadget]

If anybody else resolves this issue, please let us know. I like to thank every one of you for trying to help.

Have had this issue a few months and versions ago - as far as I remember in my case the server stopped dying after removing an item of type "logfile". Didn't really used it - I was just playing around with various items and wondered what these could be useful for... :-) Anyway - I'm running with snmp doing polls an catching traps and never have seen a server process dying since...

[nick5000]

zabbix dies with --with-net-snmp

Hi everyone,

I'm having a similar problem as the ones described above. I'm running FC5 and Zabbix 1.1. When I compile with the --with-net-snmp, zabbix_server dies right after I start it. Looking in the log, I get "One server process died. Shutting down..." However, when I do not compile with --with-net-snmp, Zabbix works fine. I need SNMP support. I installed net-snmp, net-snmp-libs, net-snmp-devel, and net-snmp-utils RPMs via yum. Does anyone have any ideas of what may be causing this?

Thanks in advance for your help.

Nick

[nick5000]

zabbix (still) dies

I've also tried compiling net-snmp from source, and moving back to 5.2.2 (I was originally at 5.3.0.1). However, I still get the same problem. Any ideas?

Thanks,
Nick

[nick5000]

From zabbix_server.log:

026901:20060619:140209 Starting zabbix_server. ZABBIX 1.1.
026901:20060619:140210 Executing query:select refresh_unsupported from config
026901:20060619:140210 In DBupdate_triggers_after_restart()
026901:20060619:140210 SQL [select distinct t.triggerid,t.value from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck+i.delay<1150740130 and i.key_<>'status' and h.status not in (4,3)]
026901:20060619:140210 Executing query:select distinct t.triggerid,t.value from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck+i.delay<1150740130 and i.key_<>'status' and h.status not in (4,3)
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12233 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12233 and i.type<>2
026901:20060619:140211 In update_trigger_value[12233,2,1150738620,ZABBIX was down.]
026901:20060619:140211 In DBget_prev_trigger_value[12233]
026901:20060619:140211 SQL [select max(clock) from alarms where triggerid=12233]
026901:20060619:140211 Executing query:select max(clock) from alarms where triggerid=12233
026901:20060619:140211 SQL [select max(clock) from alarms where triggerid=12233 and clock<1150738582]
026901:20060619:140211 Executing query:select max(clock) from alarms where triggerid=12233 and clock<1150738582
026901:20060619:140211 SQL [select value from alarms where triggerid=12233 and clock=1150734780]
026901:20060619:140211 Executing query:select value from alarms where triggerid=12233 and clock=1150734780
026901:20060619:140211 In add_alarm(12233,2,0)
026901:20060619:140211 In latest_alarm()
026901:20060619:140211 SQL [select value from alarms where triggerid=12233 order by clock desc]
026901:20060619:140211 Executing query:select value from alarms where triggerid=12233 order by clock desc limit 1
026901:20060619:140211 Executing query:insert into alarms(triggerid,clock,value) values(12233,1150738620,2)
026901:20060619:140211 In DBinsert_id()
026901:20060619:140211 End of add_alarm()
026901:20060619:140211 Executing query:update triggers set value=2,lastchange=1150738620,error='ZABBIX was down.' where triggerid=12233
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12230 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12230 and i.type<>2
026901:20060619:140211 In update_trigger_value[12230,2,1150733160,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12234 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12234 and i.type<>2
026901:20060619:140211 In update_trigger_value[12234,2,1150736400,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12235 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12235 and i.type<>2
026901:20060619:140211 In update_trigger_value[12235,2,1150733160,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12236 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12236 and i.type<>2
026901:20060619:140211 In update_trigger_value[12236,2,1150733460,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12238 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12238 and i.type<>2
026901:20060619:140211 In update_trigger_value[12238,2,1150733130,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12239 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12239 and i.type<>2
026901:20060619:140211 In update_trigger_value[12239,2,1150733160,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12240 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12240 and i.type<>2
026901:20060619:140211 In update_trigger_value[12240,2,1150736400,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12241 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12241 and i.type<>2
026901:20060619:140211 In update_trigger_value[12241,2,1150733700,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12242 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12242 and i.type<>2
026901:20060619:140211 In update_trigger_value[12242,2,1150733430,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12243 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12243 and i.type<>2
026901:20060619:140211 In update_trigger_value[12243,2,1150733160,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12244 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and

[nick5000]

f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12244 and i.type<>2
026901:20060619:140211 In update_trigger_value[12244,2,1150733160,ZABBIX was down.]
026901:20060619:140211 SQL [select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12245 and i.type<>2]
026901:20060619:140211 Executing query:select min(i.nextcheck+i.delay) from hosts h,items i,triggers t,functions f where f.triggerid=t.triggerid and f.itemid=i.itemid and h.hostid=i.hostid and i.nextcheck<>0 and t.triggerid=12245 and i.type<>2
026901:20060619:140211 In update_trigger_value[12245,2,1150733160,ZABBIX was down.]
026901:20060619:140211 End of DBupdate_triggers_after_restart()
026903:20060619:140211 server #1 started [Alerter]
026904:20060619:140211 server #2 started [Timer]
026905:20060619:140211 server #3 started [ICMP pinger]
026908:20060619:140211 In child_main()
026908:20060619:140211 server #6 started [Trapper]
026908:20060619:140211 Before DBconnect()
026909:20060619:140211 In child_main()
026909:20060619:140211 server #7 started [Trapper]
026909:20060619:140211 Before DBconnect()
026910:20060619:140211 In child_main()
026910:20060619:140211 server #8 started [Trapper]
026910:20060619:140211 Before DBconnect()
026911:20060619:140211 In child_main()
026911:20060619:140211 server #9 started [Trapper]
026911:20060619:140211 Before DBconnect()
026912:20060619:140211 In child_main()
026912:20060619:140211 server #10 started [Trapper]
026912:20060619:140211 Before DBconnect()
026901:20060619:140211 server #0 started [Housekeeper]
026901:20060619:140211 0. PID=[26903]
026901:20060619:140211 1. PID=[26904]
026901:20060619:140211 2. PID=[26905]
026901:20060619:140211 3. PID=[26906]
026901:20060619:140211 4. PID=[26907]
026901:20060619:140211 5. PID=[26908]
026901:20060619:140211 6. PID=[26909]
026901:20060619:140211 7. PID=[26910]
026901:20060619:140211 8. PID=[26911]
026901:20060619:140211 9. PID=[26912]
026901:20060619:140211 ZABBIX server is up.
026905:20060619:140211 In create_host_file()
026905:20060619:140211 Executing query:select distinct h.ip from hosts h,items i where i.hostid=h.hostid and (h.status=0 or (h.status=0 and h.available=2 and h.disable_until<=1150740131)) and (i.key_='icmpping' or i.key_='icmppingsec') and i.type=3 and i.status=0 and h.useip=1
026904:20060619:140211 Executing query:select distinct i.itemid,i.key_,h.host,h.port,i.delay,i.descriptio n,i.nextcheck,i.type,i.snmp_community,i.snmp_oid,h .useip,h.ip,i.history,i.lastvalue,i.prevvalue,i.ho stid,h.status,i.value_type,h.errors_from,i.snmp_po rt,i.delta,i.prevorgvalue,i.lastclock,i.units,i.mu ltiplier,i.snmpv3_securityname,i.snmpv3_securityle vel,i.snmpv3_authpassphrase,i.snmpv3_privpassphras e,i.formula,h.available,i.status,i.trapper_hosts,i .logtimefmt,i.valuemapid from hosts h, items i, functions f where h.hostid=i.hostid and h.status=0 and i.status=0 and f.function in ('nodata','date','dayofweek','time','now') and i.itemid=f.itemid
026908:20060619:140212 After DBconnect()
026908:20060619:140212 Before accept()
026909:20060619:140212 After DBconnect()
026909:20060619:140212 Before accept()
026910:20060619:140212 After DBconnect()
026910:20060619:140212 Before accept()
026903:20060619:140212 Executing query:select a.alertid,a.mediatypeid,a.sendto,a.subject,a.messa ge,a.status,a.retries,mt.mediatypeid,mt.type,mt.de scription,mt.smtp_server,mt.smtp_helo,mt.smtp_emai l,mt.exec_path,a.delay,mt.gsm_modem from alerts a,media_type mt where a.status=0 and a.retries<3 and (a.repeats<a.maxrepeats or a.maxrepeats=0) and a.nextcheck<=1150740132 and a.mediatypeid=mt.mediatypeid order by a.clock
026905:20060619:140212 IP [10.9.1.19]
026905:20060619:140212 Executing query:select distinct h.host from hosts h,items i where i.hostid=h.hostid and (h.status=0 or (h.status=0 and h.available=2 and h.disable_until<=1150740131)) and (i.key_='icmpping' or i.key_='icmppingsec') and i.type=3 and i.status=0 and h.useip=0
026912:20060619:140212 After DBconnect()
026912:20060619:140212 Before accept()
026911:20060619:140212 After DBconnect()
026911:20060619:140212 Before accept()
026905:20060619:140212 In do_ping()
026901:20060619:140212 In housekeeping_process_log()
026901:20060619:140212 Executing query:select housekeeperid, tablename, field, value from housekeeper order by tablename
026905:20060619:140212 Update IP [10.9.1.19 is alive (2.30 ms)]
026905:20060619:140212 Mseconds [2.300000]
026905:20060619:140212 IP [10.9.1.19] alive [1]
026905:20060619:140212 In process_value([email protected])
026905:20060619:140212 In process_ip([10.9.1.19])
026905:20060619:140212 End of process_ip([0])
026905:20060619:140212 SQL [select i.itemid,i.key_,h.host,h.port,i.delay,i.descriptio n,i.nextcheck,i.type,i.snmp_community,i.snmp_oid,h .useip,h.ip,i.history,i.lastvalue,i.prevvalue,i.va lue_type,i.trapper_hosts,i.delta,i.units,i.multipl ier,i.formula from items i,hosts h where h.status=0 and h.hostid=i.hostid and h.ip='10.9.1.19' and i.key_='icmpping' and i.status=0 and i.type=3]
026905:20060619:140212 Executing query:select i.itemid,i.key_,h.host,h.port,i.delay,i.descriptio n,i.nextcheck,i.type,i.snmp_community,i.snmp_oid,h .useip,h.ip,i.history,i.lastvalue,i.prevvalue,i.va lue_type,i.trapper_hosts,i.delta,i.units,i.multipl ier,i.formula from items i,hosts h where h.status=0 and h.hostid=i.hostid and h.ip='10.9.1.19' and i.key_='icmpping' and i.status=0 and i.type=3
026905:20060619:140212 In process_new_value()
026905:20060619:140212 In add_history(icmpping,,3,1)
026905:20060619:140212 In add_history(17249,UINT64:1)
026905:20060619:140212 In add_history_uint()
026905:20060619:140212 Executing query:insert into history_uint (clock,itemid,value) values (1150740132,17249,1)
026905:20060619:140213 In add_trend()
026905:20060619:140213 SQL [select num,value_min,value_avg,value_max from trends where itemid=17249 and clock=1150740000]
026905:20060619:140213 Executing query:select num,value_min,value_avg,value_max from trends where itemid=17249 and clock=1150740000
026901:20060619:140213 Executing query:delete from history where itemid=17223 limit 500
026901:20060619:140213 Executing query:delete from housekeeper where housekeeperid=130
026905:20060619:140213 Executing query:insert into trends (clock,itemid,num,value_min,value_avg,value_max) values (1150740000,17249,1,1.000000,1.000000,1.000000)
026901:20060619:140213 In housekeeping_alarms(1150740131)
026901:20060619:140213 Executing query:select alarm_history from config
026901:20060619:140213 Executing query:select alarmid from alarms where clock<1119204131
026901:20060619:140213 In housekeeping_alerts(1150740131)
026901:20060619:140213 Executing query:select alert_history from config
026901:20060619:140213 Executing query:delete from alerts where clock<1119204131
026901:20060619:140213 Deleted [0] records from table [alerts]
026901:20060619:140213 In housekeeping_sessions(1150740131)
026901:20060619:140213 Executing query:delete from sessions where lastaccess<1150653731
026901:20060619:140213 Deleted [0] records from table [sessions]
026901:20060619:140213 Sleeping for 1 hours
026905:20060619:140214 End of add_history
026905:20060619:140214 In update_item()
026905:20060619:140214 Executing query:update items set nextcheck=1150740150,lastclock=1150740132 where itemid=17249
026901:20060619:140214 One server process died. Shutting down...
026901:20060619:140214 0. Killing PID=[26903]
026901:20060619:140214 1. Killing PID=[26904]
026901:20060619:140214 2. Killing PID=[26905]
026901:20060619:140214 3. Killing PID=[26906]
026901:20060619:140214 4. Killing PID=[26907]
026901:20060619:140214 5. Killing PID=[26908]
026901:20060619:140214 6. Killing PID=[26909]
026901:20060619:140214 7. Killing PID=[26910]
026901:20060619:140214 8. Killing PID=[26911]
026901:20060619:140214 9. Killing PID=[26912]
026901:20060619:140214 ZABBIX server is down.
026905:20060619:140214 Server [3]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026903:20060619:140214 Server [1]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026904:20060619:140214 Server [2]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026908:20060619:140214 Server [6]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026909:20060619:140214 Server [7]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026910:20060619:140214 Server [8]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026911:20060619:140214 Server [9]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026912:20060619:140214 Server [10]. Got QUIT or INT or TERM or PIPE signal. Exiting...
026906:20060619:140214 Server [4]. Got QUIT or INT or TERM or PIPE signal. Exiting...

[nick5000]

Bump... Is this a bug? Lots of people seem to be getting this...

[nick5000]

Switched to Ubuntu Dapper. Works now.

[psigurdson]

Proximo...thanks. Running it without --net-snmp seems to have fixed it. I don't know what functionalities I would lose by not having --net-snmp, but I could work with what I have for right now.

If anybody else resolves this issue, please let us know. I like to thank every one of you for trying to help.

Oh happy day!

Ambex

Anybody resolve this problem without disabling snmp? I need to monitor using snmp, but the server dies right after it starts (no errors, no core file). I'm using redhat el3 (Kernel 2.4.21-27.EL on an i686).

[Alexei]

Anybody resolve this problem without disabling snmp? I need to monitor using snmp, but the server dies right after it starts (no errors, no core file). I'm using redhat el3 (Kernel 2.4.21-27.EL on an i686).

I would be very interested to resolve this issue. Unfortunately it is nearly impossible as the problem seems to be platform specific. Let me (a l e x @ z a b b i x . c o m) know if you're ready to give us remote access to your system, so we could investigate this problem.

[ShivaS]

I also have same problem and need snmp;-(
unfortunately i cannot give access to my system...
i am using centos 4.3 (this also happened on rhel 4.2)

thanks

[psigurdson]

I also have same problem and need snmp;-(

According to the installation guide, the OpenSSL library and headers are required for SNMP. I just noticed that my OpenSSL version is dated 2003. I wonder if the "server dying without error when using snmp" might be related to using a too old Open SSL release.

Thoughts?

-- Paul

[ShivaS]

i am using centos 4.3 and have latest openssl and openssl-dev (used for apache)

[Alexei]

We spent whole day trying to debug this. It seems to be related either to GCC bug or to MySQL which seems to have statically linked OpenSSL symbols, so any application which used both MySQL libraries and OpenSSL shared libs won't compile and (or) run correctly because of duplicate symbol.

The simple program crashes on RedHat ES4:

#include <stdio.h>
#include <net-snmp/net-snmp-config.h>
#include <net-snmp/net-snmp-includes.h>

#include "mysql.h"
#include "errmsg.h"
#include "mysqld_error.h"

int main()
{
printf("Before init_snmp()\n");
init_snmp("zabbix_server");
printf("After init_snmp()\n");
exit(0);
}

Compile: gcc a.c -lnetsnmp -lmysqlclient -lcrypto -I/usr/include/mysql -L/usr/lib/mysql -L/usr/lib

Try to compile and run this program to see if your distribution affected. Let me know if know any GOOD solution.

[ShivaS]

dunno if this helps..i succeeded to compile small prog with static netsnmp library and run it

g++ -lcrypto -I/usr/include/mysql a.c /usr/local/lib/libnetsnmp.a

btw instead of getting access to someone's server
download centos 4.3 (i just did another clean server install for zabbix)
install with no packages at all (minimum), then add netsnmp,gcc,mysql5 rpm and other essential packages
in my quick install i didn't compile anything but used rpm...(no apache/php used at all)
yum upgrade -y (just in case...)
and then try compile zabbix ..

[ShivaS]

Alexei
attached a file with valgrind output
i think it might help a lot

plz update us ;-)

thanks