Ad Widget

**tim.mooney** · 22-08-2022, 07:43

Originally posted by johannesraff

This Update path has updated the Database and worked well. Now I upgraded to 6.0.1 and then the server doesn't start anymore and crashes:

9744:20220821:063155.769 Got signal [signal:11(SIGSEGV),reason:1,refaddr

nil)]. Crashing ...

Segmentation Violation (SEGV) is a strong indicator of a software bug.

6.0.7 has been released, it's possible 6.0.8 is out by this point. It's completely safe to update from 6.0.0 or 6.0.1 to 6.0.7 or 6.0.8. I suggest you try that. Many, many bugs have been fixed since 6.0.1 was released.

**johannesraff** · 23-08-2022, 07:07

Many thanks for your feedback.

I had now tried to execute the Add ForeignKey guideline on the Database when it was running under 6.0.0. then I updated again to 6.0.1 up to 6.0.7.

Unfortunately, it's still not starting. Is there anything further I can do to get behind the issue?

Thanks

**tim.mooney** · 23-08-2022, 22:34

Is 6.0.7 still crashing with a SIGSEGV in the server log, or have the errors in the zabbix_server.log changed to something else?

You mentioned that you have "4 CPUs" for this server: is it a VM, or is it a physical server? The reason I ask about virtual vs. physical is that sometimes (rarely) a SEGV can be triggered when you're using binaries that are compiled for a newer architecture variant than what your actual CPUs provide. That's rare, and typically I would expect to see a different signal, but it can sometimes be SEGV.

**johannesraff** · 08-09-2022, 10:32

Hi Tim,

many thanks for the feedback. It remained to be the SIGSEGV error. I have also gone back (with a Snapshot) to version 6.0.0 and then upgraded to 6.0.8 and also to 6.2.1, both behave the same.

The machine is a VM, in this case, I consolidated the Server, Web and Database back to one system, but it behaves as the production setup, where the 3 roles are separated.

Regarding your hint with the CPU, the issue comes up only after 6.0.0 which is strange. But it is a VM and Intel Xeoan CPUs are probably 5 years old. I can try to transfer the VM to another ESX server, if you think that's a possible issue.
Also after 6.0.0 I read, that FK could be introduced to the Datasbes History tables. Could that be an issue?

Thanks

**tim.mooney** · 08-09-2022, 22:01

I suppose it's possible that the database changes related to indexing on some of the history tables could somehow be related to the SIGSEGV. It wouldn't be my first guess, but I don't have any better idea where the problem is.

Have you ever used a debugger on a program that is crashing on Linux? If you have, and you're comfortable with using a debugger like 'gdb', then it's likely possible to isolate what the problem is, since it happens very shortly after the server starts up.

If you've never used a debugger, though, getting to the point where you can look through the stack and identify where the problem originated is going to be a very complicated process.

Have you compared the zabbix_serverd.conf file from your (working) lab upgrade with the zabbix_serverd.conf from the failed production upgrade? Obviously there are going to be some differences for things like database connection parameters and possibly # of collectors for various subsystems, but looking carefully for differences between those two configs might be worth a try.

**johannesraff** · 18-09-2022, 10:45

I followed up with the history and trend data, reduced the storage amount to 60 days and tested the upgrade again, but with no luck. Also, I had all hosts disabled before the upgrade.

I haven't used a debugger yet, but I will go through the topic and see what I can do. Any further hint?

The Zabbix Server config I have validated quite well, I checked every value 1 by 1, also used the old config completely and built a new one, I don't think I overlooked anything here.

**tim.mooney** · 19-09-2022, 06:45

The additional testing you've done probably eliminates a config issue as the source of the problem. If I were in your situation, the next tool I would reach for is a debugger, to see if I could figure out where in the code the crash is happening.

If you're not used to using a debugger, it's going to be a lot to learn. The packages the Zabbix developers provide have probably been "stripped", meaning any even minimal debugging symbols have been removed, to reduce the on-disk size of the executables. Without debug symbols, the debugger is missing some of the instrumentation the compiler provided, to help it understand what the compiler did when it converted the C-language source code into binary object files, as well as what the linker did when it joined those object files into an executable.

Modern Linux distros have the ability to preserve the "debuginfo" symbols in separate packages that can optionally be loaded to help with debugging, but unlike the distro itself, the Zabbix developers unfortunately don't provide debuginfo packages.

Without debugging symbols, the debugger will have a very limited ability to map between the source code and the final executable, but it should get you more info than you currently have.

You will need to have an extracted copy of the source code for the exact version of Zabbix that you're trying to debug. For example, if you're running Zabbix 6.0.7, you'll need the source code for 6.0.7. You can extract it into /tmp or /var/tmp/ or your home directory or whatever, but it needs to be on the Zabbix server where you want to do the debugging.

Assuming zabbix_server is still crashing basically right after startup, you'll want to do something like:

sudo -u zabbix gdb /usr/sbin/zabbix_server

(sudo -u zabbix is to run both "gdb" and the /usr/sbin/zabbix_server as the "zabbix" user).

Once gdb starts up and reads the executable, you'll need to use several statements within the gdb environment to tell it where to find all of the directories that contain the Zabbix source code. The source code is split into multiple subdirectories, and gdb needs to know about every one of them. I believe the gdb subcommand to specify a directory with source code is "use /path/to/dir", but you'll have to verify.

Once you've told it where to find all the source code, tell it to run the executable, and wait for the crash. From that point, you would use commands like "where" and "backtrace -full" to get a stack trace. The place where the crash happens is going to be the top of the stack, but the actual source of the error could be farther down the stack. Since I don't know how much gdb (or whatever debugger you use) is going to be able to tell you without any debuginfo symbols, it's hard to give any more suggestions. If you get a stack trace, you could spend some time looking through the bug tracker at support.zabbix.com, to see if you can find a trace that looks similar to yours. It's possible the issue you're running into has been reported by someone else, and there may be comments or ideas for further debugging in the ticket. Keep in mind that's the "free" (not paid support) bug tracker.

**johannesraff** · 24-09-2022, 14:29

Hello,

thank you very much for this explanation, I will definitely invest more time in this, but it seams Zabbix isn't making this easier, than it could be.

In the meanwhile, I think I have found the issue and I am able to upgrade now. I had gone down the road of deleting a bunch of hosts (I am on a clone anyway) to see if one of the hosts is somewhat corrupt. In this process, I had also to delete the still-existing maintenance windows, that had been created. Some where expired already, since they are not cleaning up automatically.

While doing this, I figured, that I only had to delete all Maintenance windows and no host and the Upgrade was successful.

I now did it again, back from 5.2.6 directly to 6.0.9 without the maintenance windows, and it worked like a charm.

Is that something, someone likes to look at for references or to tune the upgrade process, to throw an error? I will keep the setup for some time and plan the production upgrade now again.

Cheers
Johannes

**johannesraff** · 26-09-2022, 07:11

I have now run through the upgrade process, as I had tested, upgraded Prod to version 6.0.9 and also run through the FK migration of the DB tables successfully

The server runs stable and gets its values for items.

Now, if I recreate the maintenance windows, I need, the server starts to crash again. Especially if the Maintenance contains a Host Group or a period of 1 month.

Here is the log:

52809:20220925:080209.745 === Backtrace: ===
52809:20220925:080209.746 15: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](zbx_backtrace+0x3f) [0x55cc59079a19]
52809:20220925:080209.746 14: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](zbx_log_fatal_info+0x141) [0x55cc59079c76]
52809:20220925:080209.746 13: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](+0x24c462) [0x55cc5907a462]
52809:20220925:080209.746 12: /lib64/libpthread.so.0(+0x12cf0) [0x7f783cf78cf0]
52809:20220925:080209.746 11: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](zbx_deserialize_uint31_compact+0x10) [0x55cc59088ffd]
52809:20220925:080209.746 10: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](zbx_get_serialized_expression_functionids+0x27) [0x55cc5906a824]
52809:20220925:080209.746 9: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](zbx_dc_get_event_maintenances+0x240) [0x55cc59028573]
52809:20220925:080209.746 8: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](+0x9f4cd) [0x55cc58ecd4cd]
52809:20220925:080209.747 7: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](timer_thread+0x2ca) [0x55cc58ecde31]
52809:20220925:080209.747 6: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](zbx_thread_start+0x37) [0x55cc59089b0c]
52809:20220925:080209.747 5: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](+0x66bcf) [0x55cc58e94bcf]
52809:20220925:080209.747 4: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](MAIN_ZABBIX_ENTRY+0x7b8) [0x55cc58e95afa]
52809:20220925:080209.747 3: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](daemon_start+0x384) [0x55cc59079638]
52809:20220925:080209.747 2: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](main+0x33a) [0x55cc58e9420e]
52809:20220925:080209.747 1: /lib64/libc.so.6(__libc_start_main+0xe5) [0x7f783b0f5d85]
52809:20220925:080209.747 0: /usr/sbin/zabbix_server: timer #3 [started, processing maintenances](_start+0x2e) [0x55cc58e92f5e]
52809:20220925:080209.747 === Memory map: ===

Ad Widget

Server Crash after Upgrade from 5.2 -> 5.4 -> 6.0.0 -> 6.0.1

Server Crash after Upgrade from 5.2 -> 5.4 -> 6.0.0 -> 6.0.1

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment