Zabbix creates lots of connects to each agent

[yngmike]

Hello all,
Thanks for taking the time to read my post.

Basically Zabbix is creating tons of connections (about 140/min) to each agent that is set up (discovered hosts). We currently have about 60 but will be increasing to about 400+ in the next 3-6 months. I have since turned off discovery and it doesn't seem to make a difference.

Is this normal? Is there a way to reduce this amount to something less invasive? The reason I ask is, I have several network monitoring devices that have a max connections/sec limitation and Zabbix is like 80% of all connections/sec when running.

Any thoughts?


We have a recent vanilla installation of Zabbix on CentOS 6.3 (2.6.32-279.5.2.el6.centos.plus.x86_64)

Installed RPM's and versions.
zabbix-web-mysql-1.8.14-1.el6.noarch
zabbix-server-1.8.14-1.el6.x86_64
zabbix-agent-1.8.14-1.el6.x86_64
zabbix-1.8.14-1.el6.x86_64
zabbix-web-1.8.14-1.el6.noarch
zabbix-server-mysql-1.8.14-1.el6.x86_64


Thanks!

-Mike

[yarisken]

I had the same problem. I used the dns name of zabbix in the .server-conf file instead of the ip. After i changed this to the ip of my zabbix server the load was reduced a lot.

[Yello]

Hi,
How are you defining/measuring a "connection"? Also, are your items configured as active or passive? If you want less network traffic look into active mode collection. The downside of that is that it puts relatively more load on the monitored host...


Regards,
David

[yngmike]

Update

Yello,

Thanks for the reply.

I was measuring connections/s based on metrics from "ip_conntrack" and "conntrack tools" piped through "pv" with a filter based on new connections only to/from the zabbix server. This yielded results of about 200 new connections per minute. This was with the default Linux template trimmed only slightly.

I spent a significant amount of time yesterday reading and learning a bit more about active vs. passive connections.

I cloned the "Linux Template" and converted all checks that could be to "Zabbix Agent Active", I then trimmed the total list to about 65 monitored items. Critical items such as ICMP, SSH, HTTP, MySQL, Oracle etc.. were created as simple/passive checks.

This yielded more palatable result of about 21 connections per minute per agent.

I plan on implementing around 300 hosts initially, and likely not growing to more than 750-1k during this servers life cycle.

I still likely need to spend some time reviewing the item intervals to make sure they make sense for my environment. One item that comes to mind is checking the free space on "/" every 30 seconds. This is more than a little excessive.

Any other suggestions on getting the connections/sec down and increasing perfomance are greatly welcomed.

Thanks!!

-Mike

[Yello]

Increase the sampling interval for your items. That is the critical driver for the connection rate you're seeing. Do you regard this rate of connection as beign a problem?

Regards,
David

[yngmike]

Zabbix connections per second

The connection count in itself is not a functional problem.

The base problem here is that I am in a highly security oriented organization.

Essentially, I have intrustion prevention, detection and threat assement systems that black holed my Zabbix server shortly after turning it on (thinking it was a threat). Also the security guys griped a bunch about all the logs it created, and that they have to review and sign off on them.

Thanks,

-Mike

[mbsit]

Hi

Remember that templates included in Zabbix are for TEST PURPOSE - not for production. Its only example.

You shouldn't use any of them. You should create (clone) and change intervals, (at least) to acceptable value.
Also google to find "zabbix performances" subjects.

And use the active agent checking, this is the correct way to implement monitoring unlimited hosts.
Read about security - checking behind the firewall is done only via Active Checking.

[Yello]

The connection count in itself is not a functional problem.

The base problem here is that I am in a highly security oriented organization.

Essentially, I have intrustion prevention, detection and threat assement systems that black holed my Zabbix server shortly after turning it on (thinking it was a threat). Also the security guys griped a bunch about all the logs it created, and that they have to review and sign off on them.

Thanks,

-Mike

Hi,
Do you have a separate network for network management traffic? And now that your security guys know what zabbix is and can identify it, they're cool with it right?

And yes, tailoring your collection to exactly what you need and no more is always the sensible way to go.

Regards,
David