Zabbix 2.2 causing High Session counts on Firewall

[SpiraL]

I've done some looking around on this forum and I can't seem to find anyone else having the same problem as me.

I recently installed Zabbix 2.2 server and its agents on about 80 servers. After adding the agents over a the course of a week or so, I was notified by our security team that the firewall between the zabbix server and its agents had a session count over 100,000 and this was causing impacts and degradation for everything passing through that firewall (this is up from the normal 25,000 sessions on it). After they blocked zabbix and I took it down, the firewall was fine back to the normal session count.

One of the firewall administrators suggested connections were not getting closed by Zabbix and maybe no FIN was being sent. I brought Zabbix back up for a short amount of time and watched the session count grow on the firewall by 30k more in only about 25 minutes time. During that time I observed both the Zabbix server and agent servers had a high number of TIME_WAIT connections existing. After reading up a bit more about TIME_WAIT, supposedly the connection is technically already closed at that point. So it appears to me Zabbix is closing the connections.

Does anybody have any clue why the firewall might be keeping the sessions open instead of closing them? It seems like they are only getting closed on the firewall when they hit a timeout 30 minute value. Even after I stopped Zabbix server and observed the TIME_WAIT connections disappear pretty quickly, the firewall did not drop the Zabbix sessions for about 30 minutes.

Really need some help for this one. Maybe there's some sort of firewall setting that needs to be adjusted that security guru on here might know? I put a lot of work into getting Zabbix 2.2.1 implemented and configured for our systems and now I'm getting lots of flak about it bringing our firewall to its knees and will have to ditch it if no resolution can be found.

[aib]

Did you check how your Firewall works when you change settings from "Zabbix agent" to "Zabbix agent(active)"?
In this case the agents will initiate and close the sessions to Zabbix server.
Maybe it will help to Firewall.

Also any Firewall can be killed if you setup the rule "Login ALL traffic from Zabbix server to any". It will make a big overhead because of file operations for any block of traffic. Did you ask your security team to make the rule as simple as possible? Just "Allow traffic from {Zabbix_Server_IP} to {Zabbix_Agents_subnet} where port={10050} or {10051} and (no log)"

[eskytthe]

Try to look at post #4 here:

https://www.zabbix.com/forum/showthread.php?t=42343

Also look here:
http://www.zabbix.com/img/zabconf201...nce_Tuning.pdf

So in general: use active items, proxys and even zabbix_sender to reduce the sessions.
But also carefully look at your items - how often are you polling / getting data etc.
Check for network time outs in zabbix server logfile, and agents/hosts not responding fast.
Check the agent config file for tuning of sesshins (amount of current running zabbix_agent process etc.)

Normally your firewalls will have some settings for how long it should keep sessions, you will need to take a dialogue with your firewall people about this and maybe any tuning.
Normal firewalls today can take way over 100000 current sessions with no problems.
(I have a Internet focused environment - some of out hosts peek with more then 100000 current sessions).

Btw: your network / firewall colleges should be able to help point out if some zabbix agent hosts have extra high no. of sessions....
Hope this help.
BR
Erik

[SpiraL]

Appreciate yall's time and help!

Thanks for the quick replies guys!

eskytthe- Right now I am doing all passive checks with no agents. So I can infer that if I implement some proxies and switch to all active checks, I should see less connections. I'm hoping significantly less. What would you say is typical normal amount of sessions a firewall should be able to handle today if 100k should not be considered high?

As for my items configured, I am only using the default Linux OS & Agent template and 1 custom template. The custom template just has like 10 items doing pings, port checks, and web monitoring. I had been tweaking the sample rate between either 30 or 60 seconds for that custom template.

Aib - To your comment, you are correct that the firewall team has setup an "any in this group to any in this group" rule that is being applied to the Zabbix server. I basically setup Zabbix in a vlan that is used for deployments to various other servers in other vlans/environments, thus I thought that would be the best place for zabbix as no additional FW rules would be needed..... So now you are saying the "file of operations block" that is checked when the firewall is setting up a connection would be very large because of the aforementioned rule, and thus cause a big slow down if a lot of connections are applied to that rule?

The only curiosity I see with that is the fact I know is this type of rule is setup in a lot of places, particularly with some recent environments that have been setup and this type of impact has not been seen at all. That would mean you would have to conclude that Zabbix is by far the heaviest hitter on the network than all the applications combined that it is monitoring.... In which case I don't think Zabbix will want to be used.

eskytthe/Aib - Also is the fact that the sessions just simply did not die on the firewall until 30 minutes after I shutdown Zabbix. This is likely the biggest cause of the high session on the firewall --- Does anybody have any idea why the FW would not be closing these connections?