Zabbix server stopped running

[Kerrygeek]

Over the weekend, I got an alert from my Zabbix server that the database was down. When I looked at the server, mysql was still running fine but I found this in my zabbix_server.log file:


6952:20090411:073148 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending password information', system error: 32
6801:20090411:073142 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending authentication information', system error: 32
6801:20090411:073227 Watchdog: Database is down
6952:20090411:073511 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending authentication information', system error: 32
6942:20090411:073732 Executing housekeeper
6952:20090411:074430 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading final connect information', system error: 104
6801:20090411:074712 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending authentication information', system error: 32
6801:20090411:075119 Watchdog: Database is down
6952:20090411:075914 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading authorization packet', system error: 104
6801:20090411:080733 One child process died. Exiting ...
6801:20090411:080743 ZABBIX Server stopped. ZABBIX 1.6.2.


The last entry in the log before this was that one of the parameters looking at a router didnt' work but I've got a LOT of these in the log:


6887:20090411:072610 Expression [{12944}>150000] for item [24301][xxxx xxx router:ifInOctets1] cannot be evaluated: unable to get function value: lastvalue IS NULL for function [12944][xxxx xxxx router:ifInOctets1.delta(0)]


I believe these are due to applying templates to a router that isn't using that particular interface (I put in X's instead of the router name) so I'm going through the log and disabling those tests in Zabbix so it will stop complaining about them. However, I don't have a clue if this is related to the problem of the Zabbix server shutting down.

When I came in this morning, mysql was running so I started the zabbix_server and everything seems to be fine now. So, the database wasn't down, according to the log maybe Zabbix lost contact with the database then shut down. I'm not sure what would cause the authentication problem to the database, I didn't change anything when I restarted it and everything started just fine.

I'm only using snmp and ping, no agents anywhere.

Any ideas why this would happen? I hate to resort to it but do I need to have a cron job to restart Zabbix periodically?

Thanks,
Kerry
PS Running Zabbix 1.6.2 on CentOS 5.2 on a dual Xeon with 4 gigs of ram.
Required server performance, new values per second 17.1938
load average: 0.64, 0.61, 0.51

[3vK]

I've same problem with my zabbix: every weekend zabbix_server crash and this is the log:

3011:20090423:022140 Expression [{26237}>150000] cannot be evaluated [Unable to get function value: function.lastvalue IS NULL for functionid [26237]]
3030:20090423:022140 Timeout while answering request
3030:20090423:022141 Timeout while connecting to [172.16.35.2:161]
3014:20090423:022141 OID [1.3.6.1.2.1.2.2.1.16.9] value #2 has unknow type [81]
3012:20090423:022141 Expression [{20927}>150000] cannot be evaluated [Unable to get function value: function.lastvalue IS NULL for functionid [20927]]
3014:20090423:022209 Timeout while answering request
3014:20090423:022216 OID [1.3.6.1.2.1.2.2.1.10.8] value #2 has unknow type [81]
3032:20090423:022327 Failed to connect to database: Error: Lost connection to MySQL server at 'sending authentication information', system error: 32 [2013]
3010:20090423:022538 Timeout while answering request

Any ideas?

[Cray]

Hello all,

I had this very similar problem a few days ago :

6952:20090411:075914 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading authorization packet', system error: 104

Even if the database was reachable...

I did a few researches on this problem, some clues lead to a bug between different versions of mysql client and mysql server...but it has to be confirmed as I haven't been able to reproduce this problem yet (it may also be a zabbix's related problem )

Any clues appreciated

[Kerrygeek]

For what it's worth, I haven't seen that error again on mine. I upgraded from 1.6.2 to 1.6.4 over the weekend and haven't had any repeats of this problem or of running out of memory. I set up a cron job to restart my zabbix_server process every morning at 4am but I have since disabled it to make sure 1.6.4 is stable.

I do still have a lot of messages in the log like this:

25495:20090423:081602 Expression [{12944}>150000] for item [24301][xxxx xxx router:ifInOctets1] cannot be evaluated: unable to get function value: lastvalue IS NULL for function [12944][xxx xxx router:ifInOctets1.delta(0)]

but things seem to be stable so I don't know if it's really a problem.

If you haven't upgraded to 1.6.4, give it a try, if nothing else it seems to take care of the memory leak problem. If anybody knows if this message in the log is a problem, let me know. It looks like I get this anytime the value doesn't change between samples.

Kerry

[IvanG]

Same issue on 1.8.3

Hi All!

Someone has the same issue on Zabbix 1.8.3? I am getting this error, on a random basis.

Thanks!

[Eric]

Same problem - Can someone reply

I have the same problem on zabbix 1.6.2. Has anyone found the root cause?

[MrKen]

Hi All!

Someone has the same issue on Zabbix 1.8.3? I am getting this error, on a random basis.

Thanks!

Which error would that be? 'Connection to database', or 'Unable to get function value'?

@Eric Which problem?

MrKen

[Eric]

Same problem

MrKen,

Sorry for not being specific. The problem was:

Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending password information', system error: 32

The question is, what is the root cause zabbix or mysql?

Just check the zabbix server is 1.6.4.

Regards,

Eric

[hernan]

Over the weekend, I got an alert from my Zabbix server that the database was down. When I looked at the server, mysql was still running fine but I found this in my zabbix_server.log file:


6952:20090411:073148 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending password information', system error: 32
6801:20090411:073142 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending authentication information', system error: 32
6801:20090411:073227 Watchdog: Database is down
6952:20090411:073511 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending authentication information', system error: 32
6942:20090411:073732 Executing housekeeper
6952:20090411:074430 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading final connect information', system error: 104
6801:20090411:074712 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'sending authentication information', system error: 32
6801:20090411:075119 Watchdog: Database is down
6952:20090411:075914 [Z3001] Connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading authorization packet', system error: 104
6801:20090411:080733 One child process died. Exiting ...
6801:20090411:080743 ZABBIX Server stopped. ZABBIX 1.6.2.


The last entry in the log before this was that one of the parameters looking at a router didnt' work but I've got a LOT of these in the log:


6887:20090411:072610 Expression [{12944}>150000] for item [24301][xxxx xxx router:ifInOctets1] cannot be evaluated: unable to get.

function value: lastvalue IS NULL for function [12944][xxxx xxxx router:ifInOctets1.delta(0)]


I believe these are due to applying templates to a router that isn't using that particular interface (I put in X's instead of the router name) so I'm going through the log and disabling those tests in Zabbix so it will stop complaining about them. However, I don't have a clue if this is related to the problem of the Zabbix server shutting down.

When I came in this morning, mysql was running so I started the zabbix_server and everything seems to be fine now. So, the database wasn't down, according to the log maybe Zabbix lost contact with the database then shut down. I'm not sure what would cause the authentication problem to the database, I didn't change anything when I restarted it and everything started just fine.

I'm only using snmp and ping, no agents anywhere.

Any ideas why this would happen? I hate to resort to it but do I need to have a cron job to restart Zabbix periodically?

Thanks,
Kerry
PS Running Zabbix 1.6.2 on CentOS 5.2 on a dual Xeon with 4 gigs of ram.
Required server performance, new values per second 17.1938
load average: 0.64, 0.61, 0.51

Hello Guys, I have the same problem with Zabbix 4.4.0.

Could you please help me to solve this issue?

Thanks in advance.

Regards.

[hernan]

Hello Guys, I have the same problem with Zabbix 4.4.0.

Could you please help me to solve this issue?

Thanks in advance.

Regards.

Hello Guys, Could you plase help me with this issue?

Thanks in advance.

Regards

[hernan]

Hello Guys, Could you plase help me with this issue?

Thanks in advance.

Regards

Hello team, Can anybody help me with this error?

Thanks in advance.

Regards.

[hernan]

Hello team, Can anybody help me with this error?

Thanks in advance.

Regards.

Hello Guys, I have the same problem with Zabbix 4.4.0.

Could you please help me to solve this issue?

Thanks in advance.

Regards.