Ad Widget

**Sven** · 18-02-2020, 15:32

I have now just increased the "StartLLDProcessors" option up to 20.

This Oracle template (and others I've created) include lots of auto discovery, and I'm building more to cope with some of the dynamic environments.

I'm not sure this will help, but I will let you know if it happens again.

**Sven** · 06-04-2020, 12:34

Well, it happened again.

One of the databases was brought down to install a new version of Oracle. This started at around 10:53 on Saturday and an event triggered and I was rightfully alerted.

At around 12:14 I had more alerts stating that 3 other databases had gone offline, even though they hadn't.

It almost looks like the the system locks up from being able to query other ODBC sources and Zabbix therefore states that it hasn't received any data from the databases as no new values have come in.

Has *anyone* any suggestions?

**tim.mooney** · 07-04-2020, 07:27

We're not monitoring Oracle using db.* items so I have limited experience here.

My guess is that the problem relates to these two notes from the ODBC item page:

Important notes

Zabbix does not limit the query execution time. It is up to the user to choose queries that can be executed in a reasonable amount of time.
The Timeout parameter value from Zabbix server is used as the ODBC login timeout (note that depending on ODBC drivers the login timeout setting might be ignored).

In your case, I'm guessing that the *login* process still works, since the listener remains up, but the actual queries fail. That seems like it must be causing a poller to hang? If that's the case, then it makes sense that once all the pollers have tried an ODBC query that fails, they're all effectively hung, so then other systems can't be checked.

Your logs clearly show that Oracle returns an ODBC error, though. I would think that would cause Zabbix to at least close the connection and let the poller move on to its next tasks from the queue. It seems like there's probably a bug there, but without being able to test it myself it's hard to say what the problem is.

If I were in your situation, I would probably set up a test where I

put in a maintenance period for nearly everything, so I don't cause an alert storm
identify an Oracle database that can be taken offline without causing an actual outage for your users
right before you actually take down the Oracle DB you've idenitified, I would use 'strace' to attach to every one of your Zabbix pollers on your Zabbix server, and use the options to write the output to separate files.
take the Oracle DB offline, to trigger the eventual hang.

If all the pollers are eventually getting tied up because of hung queries, you should be able to eventually see every one of them stuck in roughly the same spot in each of the output files. In other words, they should all be doing the same thing.

If you make progress on this issue, please do update this thread.

**Sven** · 24-05-2021, 17:24

Well - I've found the issue.

I had some more alerts and happen to be checking the logs around that time. There was a kernel message regarding the netpf10 module. Turns out that there was a new scan from the security teams that was causing these odd things to happen.

Oddly, it is a combination of IPv6 being disabled on the server and SELinux, even though SELinux was set to permissive.

I'm not an selinux guy, so I cobbled a list of commands on how to solve this. There may be a cleaner way:

Code:

cd /root
grep zabbix_server /var/log/audit/audit.log | audit2allow -m zabbix_netpf10 > zabbix-netpf10.te
checkmodule -M -m -o zabbix-netpf10.mod zabbix-netpf10.te
semodule_package -m zabbix-netpf10.mod -o zabbix-netpf10.pp
semodule -i zabbix-netpf10.pp

Hope that helps everyone!

**Sven** · 06-09-2021, 13:27

Well - I guess I was wrong.

One of the engineers brought down one of the ODAs for patching on Friday and hadn't put it into maintenance mode. This resulted in another storm alert as the ODBC queries all thought that there was no more data coming in.

Guess it must be something else.

Anyone have any ideas ?

**Sven** · 09-10-2021, 15:45

I’m now thinking that moving to zabbix_agent2 be the way to go instead of the main server polling, offload the querying to the agents installed on the ODAs/Solaris.

I am going to have an issue with this route in the next few months as we’re planning on having 2 Exadata systems installed and I’m not sure what we can do about agents being installed on there.

does anyone have any thoughts on that?

Ad Widget

ODBC Queries on an offline database causes massive freeze and cascading alerts

ODBC Queries on an offline database causes massive freeze and cascading alerts

Comment

Comment

Comment

Comment

Comment

Comment