Zabbix Proxy 3.4 Debian Sparc Dies

[laserted007]

Greets - I've been using Zabbix on intel boxes (typically Debian distros) for a number of years quite happily, and my fleet has grown, as one hopes. One particular instance has now grown to the point that I'd like to make more use of proxies at remote sites that are typically linked by VPN, since of course VPNs go down occasionally.
I already tested my setup both in lab and field with ONLY intel-based boxes and it is functional. For those I have been using the Debian package of 3.2.6 (3.4 will be an inline upgrade once I get my "problem" below fixed).....

I have a bunch of Sun V100's sitting around gathering dust, that I would prefer to get up and going again. I've used DebianSparc32 on these machines before, as well as Solaris9, both ok. I prefer Debian. This is not a request for help with Sparc or Debian - please note that I have fully working Debian environments with PostgreSql 9.1 and networking, and libsnmp-dev running on these units. (In case someone is asking, the Sparc64-debian buildbot is apparently currently broken and wouldn't complete installation, so 32 bit is as functional as it gets at the moment.)

Unfortunately, the Deb-Sparc binaries no longer install for Zabbix proxy on this platform so I compiled from sources (zabbix-3.4.7.tar.gz to be precise). I believe that the make process worked well - there were no errors indicated (there were a lot of "Entering directory, nothing to do, exiting directory" instances, but given how many temp folders it was making in the process, that's not a big surprise. After make install (my configure was: ./configure --prefix=/usr --enable-proxy --with-net-snmp --with-postgresql --enable-agent) I ran the dbase setup and the schema was created as expected.

I make a couple quick changes to the agent conf file, and upon attempting the traditional 'service zabbix_agentd start' was informed that it was an unrecognized service....hmm.... inet.d also showed no entry.....also hmm... I then located the binaries (/usr/sbin) and ran zabbix_agentd from there.

And success - the agent would communicate with my master server no issues. Thus, step 1 is successful.

And now for the proxy.
A couple quick changes to the proxy config just to get the baseline, but unfortunately, the daemon gets terminated (per logs).
A snippet from the first fatal entry:

17390:20180327:062551.063 proxy #1 started [housekeeper #1]
17390:20180327:062551.064 __zbx_zbx_setproctitle() title:'housekeeper [startup idle for 30 minutes]'
17390:20180327:062551.064 Got signal [signal:10(SIGBUS),reason:1,refaddr:0xf52fd664]. Crashing ...
17390:20180327:062551.064 ====== Fatal information: ======
17390:20180327:062551.064 program counter not available for this architecture

There's a lot more in the attached logs, but it gets a little cryptic when the .so modules lists start whizzing by. By my read, it appears to access the dbase ok (the dbase has a full schema, but the only data populated is the dbase version table) - not unexpectedly since the process dies before it queries the master server for data.

My belief is that there must be a dependency missing or an incompatibility somewhere. Although I would love for it to be a line in the config that is causing it.

My agent, proxy conf's and the proxy log are in the attached zip file (sorry for the zip file, a 70kb log doesn't fit in a 20kb max upload allowance).

Any thoughts?

[bbrendon]

My first thought is ...ummm.... why are you torturing yourself? What the real reason because we all know this is just the _surface_level_ reason....

"I have a bunch of Sun V100's sitting around gathering dust, that I would prefer to get up and going again."

[laserted007]

I don't see any torture in it at all - I have a bunch of raspi3's on my lab bench too, and they run debian, and they run Zabbix - if you compare strictly cost or brain-dead-install steps, or even power consumption perhaps, then the raspi will win. If you happen to want industrial stability then the raspi is not even a contender in my books (tried and field tested, not just baseless opinion).

It's not a hardship to have another intel machine be the proxy unit, but for each rack that I do this on, it would take an intel machine out of the running for doing other "more value added" tasks. The V100's were absolutely rock-solid web servers in their day, with their little 512mb ram and 60gb hard disks. Which unfortunately now makes them not very useful webservers today. But to monitor 3 or 4 other blades, plus a couple dozen other clients (like phones, tablets, pcs, printers and switchgear) - would keep them out of the landfill and make the accountant happy.

But all the above is beside the point - let's for a moment take the Sparc out of the picture - if I had this same build problem on an x86 architecture, would you have some advice to share in resolving the issue as described?

Thanks,
Ted.

[laserted007]

bbrendon (2) -- however that does raise the question to be open to re-evaluate the entire belief structure from a different angle - if there isn't an obvious solution to get the proxy to work on the sun, I can use the sun's strengths to bolster another embedded sbc's (like a raspi) weaknesses - the biggest problem with a raspi is sd card writes (the native storage is the primary reason a raspi is not a contender for field data control per my above opinion). However the sun, as-is right now is very efficient at db. I can host the db and python needs on the sun, and just run the proxy on the raspi (which has more ram anyway, handy for proxy), then just stuff the raspi in the sun case; still 1U tall done.....and no, I don't have a particular affinity to raspi, but having surveyed a LOT of sbc's over the years even in the PC104 form factor, LONG before arm architecture), it and Intel's NUCs are really the only good contenders - and although a NUC is a full-fledged x86, they can't keep a standardized platform (power, for example) for any longer than a year before dumping it. :-( Even out of the case, NUCs are also taller than 1U (excluding the 5i5).

That does turn into double the network traffic (server <-> proxy <-> agent <-> proxy <-> db <-> proxy <-> server).....but I can vlan out the proxy <-> db traffic or add a 2nd ethernet interface to a raspi (as the sun has two already).

If I had $250,000 for each rack, I would just blade it up and be rolling in 32 core luxury. I *do* have closer to $25.00, so creative solutions win out to large purchase orders....

Ted.

[vso]

Can you please increase log level in configuration file to maximum, will you be able to apply a patch with additional debug information if I will provide one ?

[laserted007]

V - absolutely. I should be able to get that to you inside of a day. This particular box is a test unit currently, but is serviced by a production backend (server, etc) so can have real data flowing to test, but not cause any undue service interruptions if a patch doesn't work as planned....

Many thanks!
TH.

[laserted007]

Was able to get it quicker than expected (@ debug level 5) - thanks for the review!
TH.

[vso]

Please try attached patch, also please after it crash again, try setting HousekeepingFrequency to 0

[laserted007]

Ok - I applied the patch from within my sources folder (eg. # patch -p0 < sparc_dies.txt) of which the output was:
patching file src/libs/zbxself/selfmon.c
patching file src/zabbix_proxy/housekeeper/housekeeper.c

so far so good.
A .configure and make install (same as in top post) completed successfully...

There appears to still be fatals (including after setting HousekeepingFrequency=0) on housekeeper#1 waiting for user command (per att log).
Thank you,
Ted.

[vso]

You can revert previous patch and try this one, It is just additional debug to narrow down cause of crash.
It would also help to try 3.0 version of Zabbix proxy and see if it had similar issue.

[laserted007]

Sorry for the delay in getting back (a vacation from the IT world for a bit) - I removed 3.4 and tried to build 3.0.17 in place, but it failed compilation at:

checking size of void *... 4
./configure: line 9035: syntax error near unexpected token `IKSEMEL,iksemel,'
./configure: line 9035: ` PKG_CHECK_MODULES(IKSEMEL,iksemel,'

So I cleaned everything up and put 3.4.7 back into active build state (reverting to original with no patches, but patching in selfmon_counter.txt per your request above.

There seems to be a lot more output into the log, and a lot of successful activities; however the log still gets that "not for this architecture" signal, then dies. If you think there's no real help here, that's ok, I can put the proxy on a raspi.

Many thanks,
Ted.

[vso]

I think you haven't recompiled after applying new patch, why I ask is that there could be a bug, and would be nice to have exact place of crash to investigate.

[laserted007]

Hi - my normal process for applying patches is as sudo:

enter dir for sources (in my case it's /home/{user}/zabbix-3.4.7
patch -p0 {patch source file} ---- I try to start in the closer root to the sources, hence the -p0, the patch source file is placed at this level in the tree.....
given the single file touches in these patches, it most typically responds with something like: patching file src/libs/zbxself/selfmon.c

make clean
make
make install

I then pop over to /usr/sbin and execute zabbix_proxy

Am I missing a step or procedure that deviates from this, or is unique to zabbix/this distro?

Thanks,
Ted.

[laserted007]

To ensure that perhaps there wasn't something straggling, (and I had an SSD I wanted to start migrating this to anyway) I did a fresh install, and patched prior to the makes, but ONLY patched from the patch file you sent in post #10. The log file was longer this time, perhaps co-incidental, perhaps not. Hope this can assist.
Thanks,
Ted.