Ad Widget

**TheReaper13** · 13-07-2021, 09:43

Hello.

We are having THE SAME EXACT problem. The same exact crash & crash dump, "One child process died", so we will not put the dump by now. It's driving us nuts. We thought we had it, but today we found another crash and burn in our job network... Beware: this might be a little long but we have included some technical info.

We are handling a >300 server network and we are pretty tired. We updated also from v3 to 5.2.4 a few weeks ago (following the guidelines provided and with help from the DB department) and started to feel this pain. Updated to 5.2.6 last week, hoping some light would be shed, and nothing. It has been constantly crashing, often more than once a day, with a message flood of remote server processes or services unavailable (false, they are up and running) when server goes down, as if they were shown from the start and immediately checked (instead of starting a new check from scratch). It is totally annoying to feel the screen filled with false messages which clean up after a while.

We tuned the server with these values and it helped us quite a lot but did not resolve our dramas. This is the server conf file and the process typical system limits:

Code:

# egrep -v "^#|^$" zabbix_server.conf
LogFile=/var/log/zabbix/zabbix-server.log
LogFileSize=200
PidFile=/run/zabbix/zabbix_server.pid
DBHost=mysql01.ournet
DBName=zabbix
DBUser=zabbix
DBPassword=we_will_not_tell
DBPort=3399
StartPollers=200
StartPollersUnreachable=20
StartTrappers=40
StartPingers=15
StartDiscoverers=5
StartHTTPPollers=20
StartTimers=1
StartEscalators=1
JavaGateway=localhost
JavaGatewayPort=10052
StartJavaPollers=5
SNMPTrapperFile=/tmp/zabbix_traps.tmp
StartSNMPTrapper=1
CacheSize=512M
CacheUpdateFrequency=300
StartDBSyncers=5
HistoryCacheSize=256M
HistoryIndexCacheSize=64M
TrendCacheSize=256M
ValueCacheSize=512M
Timeout=30
AlertScriptsPath=/etc/zabbix/alerts.d
ExternalScripts=/etc/zabbix/external.d
LogSlowQueries=3000
LoadModulePath=/usr/lib/zabbix/modules

# cat /proc/PID/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 16384 16384 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 65535 65535 signals
Max msgqueue size 1638400 1638400 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

This server is handled under a cluster package, based on Pacemaker/Corosync. Basically, it works normally until something brings down dramatically the server. Only if we put it under debug mode we can notice what machine could cause it, but it has, being kind, zero logic for programming this way. Whenever a poller gets a handled, unhandled, expected or unexpected message, it crashes with a SIGSEGV and brings the whole zabbix server instance down with it. It does not die alone and respawn or start another poller, it just nukes it all.

If we translate it to human behaviour, it could be like this:

A family lives well and having it fine
One of the children speaks with a long distance relative and suffers communication problems that shuts the line down, himself dying in the process.
Because of this the father slaughters all children, wife, the dog and all cats from the backdoor, and then kills himself.
"God" (our cluster) starts/respawn again this family
Goto 1

It is pretty annoying since, of all the traces we have started, the behaviour is the same: we have reached the problematic nodes, and we have found out they suffer something at the same time the server crashes; we have seen oom-killers (java, stressing the server), typical process upgrades like the agent's packages, reboots.... There is nothing special, servers have these moments and they get nervous/unstable/hot/blocked/restarted/whatever; normally, the monitoring process informs the node has some kind of problem, waiting for it, marking it unavailable, etc. I insist, nothing apart the typical behaviour of saying 'node N has problems, I will try to reach it again or mark it unavailable'. Even zabbix seems to do it often, but when something strange happens, the house gets burnt down. It used to happen too while on our old v3, the main difference is that there was no message flood, just a "zabbix server restarted" message. Now this one is displayed along with all the false messages of unavailability.

What does it make blow up? What can we do to mitigate it, to isolate it, to help zabbix wait for the remote node to become alive or not, and mark it accordingly, instead of dying the most radical way? If it is too advanced, how can we mitigate the message flood?

This crash has been happening FOR YEARS with the SAME LOGIC. We have been searching and QUITE A LOT of people reports the "one child process died" event and the slaughter. Even from 2008, some 13 nice years ago. How come this is still not corrected? Is it intentional? Is there no will to implement some kind of error control on the functions, in case a certain problem arises make sure the problem gets isolated and treated apart? We have not advanced C development skills for applying ourselves the patch, we even wonder to improve them...

Thank you all.

**TheReaper13** · 15-07-2021, 09:07

More misery today. A cluster has had problems, being noticed by zabbix server, until a point when it crashed while trying to obtain data from one of the troubled nodes.

Code:

18318:20210714:194005.265 In get_value_agent() host:'troublesome2.ournet' addr:'192.168.1.12' key:'vm.memory.size[available]' conn:'unencrypted'
18318:20210714:194005.269 Sending [vm.memory.size[available]]
18318:20210714:194008.308 get value from agent result: ''
18318:20210714:194008.359 End of get_value_agent():NETWORK_ERROR
18318:20210714:194008.360 Item [troublesome2.ournet:vm.memory.size[available]] error: Received empty response from Zabbix Agent at [192.168.1.12]. Assuming that agent dropped connection because of access permissions.
18318:20210714:194008.360 End of get_value():NETWORK_ERROR
18318:20210714:194008.392 In zbx_deactivate_item_host() hostid:13066 itemid:261505 type:0
18318:20210714:194008.403 query [txnlev:1] [begin;]
18318:20210714:194008.469 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...

Up until now the same PID has been working fine, even connecting to that troublesome node. However, there is certain point when it crashes. 'NETWORK_ERROR' message is false: one could connect there to any port, although not easily, but it was possible, and there was no access permission trouble as it would have existed before and now. I wonder why it crashed then, and not before, and above all, why does it happen, instead of just reporting and not crashing. If a prison watchman died any time two prisoners have a heart attack, what a prison it would be!

We even had our regular message storm. That crash must be avoided, but at least we need not to suffer any false warning flood. We would like to know, by now, how to avoid that flooding, and then look inside of those unexpected crashes.

**vso** · 15-07-2021, 13:54

Which MySQL version is Zabbix compiled against, it seems to be crashing inside of MySQL library on simple "begin;" query

Code:

3113:20210623:140132.039 27: /usr/sbin/zabbix_server: poller #5 [got 1 values in 0.298354 sec, getting values]() [0x54bb76]
3113:20210623:140132.039 26: /lib64/libpthread.so.0(+0x10c70) [0x7f7e6f76fc70]
3113:20210623:140132.039 25: /usr/lib64/libmysqlclient.so.18(+0x590ab) [0x7f7e6f9d50ab]
3113:20210623:140132.039 24: /usr/lib64/libmysqlclient.so.18(+0x4fdfc) [0x7f7e6f9cbdfc]
3113:20210623:140132.039 23: /usr/lib64/libmysqlclient.so.18(+0x5026b) [0x7f7e6f9cc26b]
3113:20210623:140132.039 22: /lib64/libpthread.so.0(+0x6c09) [0x7f7e6f765c09]
3113:20210623:140132.039 21: /usr/lib64/libmysqlclient.so.18(+0x5090e) [0x7f7e6f9cc90e]
3113:20210623:140132.039 20: /usr/lib64/libmysqlclient.so.18(get_charset_by_csname+0x42) [0x7f7e6f9cc9e2]
3113:20210623:140132.039 19: /usr/lib64/libmysqlclient.so.18(+0x2a807) [0x7f7e6f9a6807]
3113:20210623:140132.039 18: /usr/lib64/libmysqlclient.so.18(mysql_real_connect+0x8ba) [0x7f7e6f9a973a]
3113:20210623:140132.039 17: /usr/lib64/libmysqlclient.so.18(+0x2e147) [0x7f7e6f9aa147]
3113:20210623:140132.039 16: /usr/lib64/libmysqlclient.so.18(+0x2e43f) [0x7f7e6f9aa43f]
3113:20210623:140132.039 15: /usr/lib64/libmysqlclient.so.18(mysql_send_query+0x46) [0x7f7e6f9a7d16]
3113:20210623:140132.039 14: /usr/lib64/libmysqlclient.so.18(mysql_real_query+0x9) [0x7f7e6f9a7d69]
3113:20210623:140132.039 13: /usr/sbin/zabbix_server: poller #5 [got 1 values in 0.298354 sec, getting values](zbx_db_vexecute+0xdb) [0x5c5beb]
3113:20210623:140132.039 12: /usr/sbin/zabbix_server: poller #5 [got 1 values in 0.298354 sec, getting values]() [0x5c5e3c]
3113:20210623:140132.039 11: /usr/sbin/zabbix_server: poller #5 [got 1 values in 0.298354 sec, getting values](zbx_db_begin+0x23) [0x5c5e73]
3113:20210623:140132.039 10: /usr/sbin/zabbix_server: poller #5 [got 1 values in 0.298354 sec, getting values](DBbegin+0x9) [0x5a0239]

**TheReaper13** · 16-07-2021, 07:34

Hi.

The calls are referred to a library inside a package, "libmysqlclient18" (Shared Libraries for MariaDB) version 10.0.40.1-2.9.1 against MySQL 5.7.30-enterprise-commercial-advanced. It's running on SLES 12 SP4. Our tests environment has a slighter version, 10.0.40.4-2.20.1, and also suffers the same, so we discarded upgrading the package could correct it; also, if the library was the problem, we could not have compiled it (this version is compiled by us under OBS, but we tried the official packages from Zabbix too) or we could see messages on startup or while running. Or even it would not start, it could never write properly to DB, it could corrupt the DB, it would crash even with OK responses, or whatever...

We have no higher versions available. We also have not found out any minimal version/release of this library on the requisites and we cannot find a higher version for download and compile. So we have to stick with it. However, if this version was the culprit, what is the minimum required?

Btw, another server crash yesterday, a remote agent shut down (SIGTERM)... It always happens when functions get_value() and get_value_agent() receive NETWORK_ERROR (like yesterday) or another like TIMEOUT.

Thanks.

**vso** · 16-07-2021, 16:40

What does it make blow up?

Can almost guarantee that it's UnixODBC that is loading it's own library on top of the existing MySQL library used by Zabbix server, if libraries mismatch or especially if MariaDB and MySQL is mixed then allot can go wrong.
Please see notes on UnixODBC in https://www.zabbix.com/documentation...n/known_issues if recommendations are followed and it is ensured that UnixODBC does not load anything that conflicts with already loaded libraries then issue should never occur again.
Also please have a look at https://support.zabbix.com/browse/ZBXNEXT-782 it eliminates database connections in pollers so the crash that you mentioned cannot occur but still perfect scenario would be that recommendations are followed.

**TheReaper13** · 19-07-2021, 13:29

Hello.

We have taken note of both ways; they seem to be compatible one and the other. For the one regarding libraries we are kind of limited, as we can only rely of what our repos/providers bring us, and what can we find elsewhere; this will take longer for us to find an acceptable solution for our standards.

As for the other, if we have read it correct, from vpre-5.4.0alpha1 onwards pollers do not communicate with database so it seems we could not fall in this trouble again. Does it mean the newer 5.4.x packages, such as 5.4.2, have the workaround applied? Could we try this latest version?

This last one will not stop looking for solving the other, but we can gain stability at least for quite a while which is something we are really needed of.

Thank you very much.

**vso** · 19-07-2021, 13:54

Yes, in 5.4.2 pollers never connect to database so crash in mentioned place cannot occur, however still recommend that UnixODBC MySQL library match MySQL C connector Zabbix server is compiled against.
Safest would be to isolate each UnixODBC driver as much as possible by having proxy for each UnixODBC library and to making sure that proxy database does not match the one used by UnixODBC, unlikely but if problem should occur then only one proxy would have problems instead of whole server, but I understand that this might require more setup and configuration which is not always convenient so it is best to try upgrading to 5.4 first.

**TheReaper13** · 21-07-2021, 12:50

Hi.

By now we have it working on our tests environment. It usually did not crash that often but when a lot of machines were attached it had its problems. Thus, we have overloaded it now and we have started to make the same movements that made the 5.2 version to crash. So far so good, but fingers crossed...

Thanks.

**TheReaper13** · 23-07-2021, 08:06

Hi again.

We have upgraded our production environment and, by now, so far so very good. The typical load and procedures that brought wildly Zabbix down could not this time! Even after one day of doing so. Of all the things we suspected could make it crash as before, now they are not capable. It is such a relief.

However, the matter of different libraries is something we are going to face as other departments have told so regarding other products like DataStage. Almost the same: until they found out a certain library, they suffered from similar troubles. In our own case, we simply could not find anything apart from our sources due to policies. Now it seems these policies are going to be stretched or pushed beyond.

Thanks.

**vso** · 23-07-2021, 08:42

That's good news thank you for your feedback, hope it will be stable now or at least will be clear what causes it and how to work your way around it.

**TheReaper13** · 09-08-2021, 15:30

Hello.

Back from holidays, and I found out the server is up and running since last post, no stops or crashes. Neither of my job mates have said anything, which means it works as it should. Now it is just time to toy around with configuration of the thing, and to look into those known issues, but as for now, Zabbix chapter is closed.

Thank you very much.

Ad Widget

Continuous Crashing in zabbix-server-mysql-5.2.6

Continuous Crashing in zabbix-server-mysql-5.2.6

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment