I'm trying to set up monitoring for Palo Alto Firewalls throughout our company and I'm running into so very strange issues. After about a week of digging deeper than I ever thought i would into SNMP and tcpdumps, we have discovered that ,at least it appears, Zabbix is switching the Privacy protocol from AES to DES on some polling requests, thus causing the host to timeout. I have double checked that all of the items and item prototypes are set to SHA/AES and that the credentials are all correct. We have found a few things on the internet that come close to describing this, but not quite. I've spent almost two weeks digging into this so any help would be very much appreciated.
-
-
Hello,
In my point of view, snmp v3 zabbix polling is not really usable even in 3.4.12
Regard
sfl
Envoyé de mon SM-G950F en utilisant Tapatalk
-
I'm going to be brutally honest.... i'm not sure how that comment was even close to being constructive. I have literally THOUSANDS of devices being monitored by Zabbix using SNMPv3 with ZERO issues. The seems to only relate to Palo Alto Firewall devices. I have read on a few forums that Palo Alto seems to handle SNMPv3 a bit differently, but no one really seems to understand how or what the solution would/should be.Comment
-
Food for thought on this.... one most of our proxies, we have several discovery rules, the vast majority of them using SHA/AES as the protocols. There is however, one discovery rule on each that is set up using MD5/DES specifically. Is there any way that this could be getting pulled into what ever mechanism that Zabbix uses when it builds a polling request to send to the host? There are no snmpd logs to go off of. I have bumped the logging on the proxies up as far as they will go and I get nothing. I really don't know where else to go with this. We have narrowed down the issue as being Zabbix, but thats as far as we can get. The templates are all set to SHA/AES, though I did find, in the XML a couple of items that even though were using SNMPv3, still had the community set, so I fixed that. This makes absolutely no sense. Why would SNMPv3 monitoring on all other devices be working 100%, yet when it comes to these PAN devices, the polling requests coming from Zabbix are somehow switching the decryption protocol from what is set (AES) to something else (DES)?Comment
-
Another follow up with more info.... We did packet traces for almost an hour on three different proxy servers not filtering on any device so we got all packets, incoming and outgoing, for snmp. ONLY the Palo Alto devices are experiencing this issue. Even though items and item prototypes are configured as SHA/AES for the protocols, somehow, Zabbix is sending the request using SHA/DES for some items.... and not even consistently. One item might timeout and not be able to be decrypted by the host, but the next time Zabbix tries to poll the device it will send a properly encrypted request using SHA/AES. We had hopes that perhaps it might have been the case of specific items or item prototypes were causing this issue, but it wasn't the case. Still looking for some insight into this. According to three if our network engineers looking into this, it is not the host as the improperly encrypted request is originating from Zabbix.Comment
-
Could that be specifics of the snmp library installed?Comment
-
In what way? Are you referring to the net-snmp libraries them selves or the MIB files? If the MIB files, they don't have anything to do with the encryption/decryption of the messaging. If you are referring to the net-snmp packages installed, we are running the most current version for CentOS 7. I tried to download and install the latest version and install from source but its not compatible with CentOS so I am forced to use the latest packages.
-
-
I was referring to the net-snmp library installed on a Zabbix host. Not sure, I was thinking maybe net-snmp has this ability to switch encryption/hash algorithms in some specific cases, but I doubt it. Does this ever happen when you use snmpwalk/snmpget?Comment
-
When an snmpwalk or snmpget is issued its fine. I've even tried switching the interface to/from bulk. What's really confusing here is, we had thought maybe it was one or a handful of specific items that might have been the issue, but after doing a fairly long tcpdump, it was very random. What's more, this doesn't happen on any other vendor device, just Palo Alto (aka PAN). But from PAN devices, they are simply receiving a request from Zabbix using incorrect Privacy encryption. But according to the template, its set up correctly. Initially, it was very easy to say that this was a PAN issue, but we've done such exhaustive digging into this issue, we are pretty certain its not PAN, but we have nothing to show exactly WHAT is causing the issue on the Zabbix end. Oh and here's one more thing. We have an instance of Zabbix 2.4 running that this is happening on as well... so I don't think its Zabbix version specific either. My next steps are going to be creating a stripped down template starting with one single item then expanding it.Comment
-
That is really weird. Do you know if it's the initial request coming with incorrect encryption algo or perhaps there's something like a fallback to default (could be DES) if AES fails? In any case, good luck and hopefully you are able to resolve it.Comment
-
I don't know of anyway to find out the specific request coming out of zabbix itself. Just what the tcpdump shows us and it shows that on some items its coming out as DES. I just set up a template with one single item, in my case looking at Total Active Sessions, which is a very well documented OID (btw... i am using the correct version of MIB's even though i'm skipping past the mibs and just using OID's) and even that one single item times out. I'll look around and see if there is a way to find a setting in net-snmp for the default encryption. Thanks.Comment
-
I am seeing this issue in version 3 and version 5 of zabbix. Anyone find a workaround or fix for this issue.Comment
Comment