2013 and agent traffic STILL not encrypted?

[BrentN]

Maybe I missed an announcement, but is agent traffic, specifically active agent traffic, still sent in clear text? How is this not a bigger problem for people? This really needs to be native to Zabbix, not a 3rd party project with limited support.

I want to recommend to my company that we start using Zabbix to monitor some 300 customer servers (active checks) but certainly not when all of that traffic can be very easily read and even exploited.

I'm just a bit baffled at why this is not made it into Zabbix yet. It is a huge problem for an otherwise superb system!

[richlv]

it's been on the list of features that could be financed for quite some time, but apparently the interest in it is not that great...

http://www.zabbix.com/development_se...ctive_projects

also, would you be interested in psk, ssl or krb ? companies of various sizes have various requirements, so interest is also diluted between (at least) these 3 possible implementations

[BrentN]

Hi Rich,

I do appreciate the fact that such a change is not as simple as "add encryption". The fact that so many people use Zabbix in so many ways makes it a daunting task. Still, I am really surprised that agent security is such a low priority for people. If I had any money I'd fund that right away! :-)

Here's how I want to use Zabbix to replace our current solution: We are an IT consulting company and as part of our service offering we monitor some simple metrics on customer servers. As such, we want to use active checks for everything. We also want to be able to issue remote commands to servers. It isn't feasible for us to create tunnels to all of these customer networks and almost all are Windows so using ssh tunneling isn't exactly simple.

I'm going to assume there maybe aren't many people using Zabbix like this - that is, monitoring systems with active checks from many different customers. If a lot of people did this, the need for some simple security on the data transmitted would be higher, I think.

At the very least, it would make sense to me to transmit over SSL (self signed even) and a simple password authentication for the agent. I understand there is an add-on to kind of support this, but it isn't integrated and it doesn't support the zsender.

Since it looks like is just isn't on the radar for many people I don't think we'll be seeing it soon, which is too bad. It really surprises me too because in my general experience as a Windows admin, Linux admins are far more concerned with how secure their systems are and hence Linux based solutions generally start out with all the holes plugged. You know? Even more disappointing is that I really want to use Zabbix and probably can't.

Additional comment: I totally support how Zabbix is funded and that the developers have to get paid somehow, especially since the core product is free. However, I'm not really sure this particular issue should be up for debate. The product is inherently insecure which is honestly more of a root design choice/problem than a missing feature. Perhaps the team should consider implementing some basic agent security (especially in the case of active agents!) and THEN request customer funding for further customizations and or enhancements to security.

[Jason]

With active communication you've got the agent contacting the server and not the other way round. Also as it's only monitoring information being passed then what good would interception do someone other than give them a bit of your monitoring data?

If you enable remote commands then you need to be 100% sure your server is secure i.e. passwords are tight and access is properly controlled otherwise a successful exploit there and all of your servers would also be compromised using the remote commands.

It's the reason we decided not to use remote commands as our zabbix server is accessible externally and the risk of a compromise there was too great.

Saying all this encryption of traffic is still a very good idea. I don't see how it's got such a big price tag attached to the development of it as all the libraries are out there and it seems standard on a lot of projects and I can't see it would be that big a modification to the code.

PSK would be the simplest to implement surely? Just a value in the config file and it must match up with the host definition on the server side. For a key of a decent length this should be secure enough for most environments

[Colttt]

BrentN: i know what you mean..

take a look at this:

https://support.zabbix.com/browse/ZBXNEXT#selectedTab=com.atlassian.jira.plugin.system.project%3Apopularissues-panel

https://support.zabbix.com/browse/ZBXNEXT-17

psk is the most wanted feature, and zabbix sia said: pay for it!

[BrentN]

Also as it's only monitoring information being passed then what good would interception do someone other than give them a bit of your monitoring data?

You're right, if somebody found out how much drive space a server was using, big deal. However, giving somebody the host name / AD domain name / private IP info is more than enough of a risk IMO. The more info somebody can collect about a network the easier it is to break into it.

I have a really hard time accepting any argument for not securing this traffic.

[mushero]

As one of the largest MSPs using Zabbix across hundreds of customers all over the world, including some of the world's largest Internet companies, games, and e-commerce systems, we just don't worry about this. It's far down our security priority list (user auth being 1000X more of a problem, with no good solutions at all, especially LDAP).

Should we care ? Sure, but we have lots of other things to worry about, and we:

- Run in passive mode so only Zabbix reaches out to the agent (to protect Zabbix servers)
- Lockdown every host via iptables to the zabbix server IPs
- Often/usually go through a firewall NAT (except on AWS, clouds), with ACL
- NEVER use remote commands (this may change)

We are also very concerned about performance, and doing 1000+ SSL connections per second or more would seem a challenge on load on Zabbix, and also on smaller hosts where we average one connection per second (not huge, I know).

So two issues:

- Intercepting data via sniffing, etc.
- Faking the server/agent, mostly for commands

We don't really pass any valuable data that I know of - no IP addresses, really just performance and status data. Perhaps some domain info in HAProxy, but hardly useful in context. Hostname is potentially useful.

Would we like a good solution ? Sure, though if SSL or cert based, we need a very good way to manage and distribute those, maybe one per Zabbix server, as per host would be a management disaster.

[timbo]

My thoughts

Note: I may have gotten carried away with my post (it's quite a lot longer than I anticipated). I was testing as I wrote it and figured I could document my progress...


This thread inspired me to finally create a profile, so hello everyone! I've been working with Zabbix for a couple of months and I absolutely love it!


The issue:

I was quite concerned about the lack of support for encryption, but then I thought about the usual implementations for Zabbix (like in enterprises) where it is quite trivial to implement IPsec on specific traffic and Kerberos is most likely already in place. This (I imagine) would have MUCH LESS administrate overhead then implementing an encryption scheme/policy within Zabbix itself, as IPsec can be managed and pushed out through policy. Then the smaller environments that host everything locally (limiting risk to insiders) probably have little use for encryption.

So I can kind of see why demand for this is a little low - Enterprise have many solutions already, SMB can't afford to donate/are overwhelmed with the donation target/don't need (all hosted locally)/don't care (no active hosts/trapper)/are ignorant (is bliss)/have accepted the risk (mushero).

Anyway, that leaves people (such as myself, the SMB enthusiast who wish to host Zabbix externally/want to monitor distributed/external systems/utilize active hosts or trappers) without an appropriate option for confidentiality/integrity.

Anyway, this is what I have done to mitigate the risks involved:
Zabbix hosted on an Amazon EC2:

EC2 offers in/outbound firewalls - Block all, white list appropriate IPs and ports
Use Apache .htaccess files to block all http access to the whole web/Zabbix directory, whitelist appropriate IPs and ports
Change default passwords (duh), change passwords (and SSH Auth keys) often, Patch, tripwire (hash and scan important files), virus-scan (it can't hurt), make system images often (backup), log everything (e.g. access attempts, logins, etc.).

Monitoring external systems:

Well, this is our problem...

My main issue is integrity (if I had to choose), I don't want people spoofing IP's and injecting their own values (messing up my data and activating triggers).

Confidentiality concerns me, obviously because it's unlikely someone will be able to successfully spoof one of my Hosts unless it captures the communication over the wire. Then there is the obvious issue of an advanced hacker enumerating the internal network/s. Anyway, the best I can do is rely on strong security practices elsewhere that may mitigate the usefulness of any information leaked.

'mushero' has some good suggestions to mitigate your exposure (Run in passive mode, lockdown hosts & servers, disable remote commands). This theoretically will protect the integrity of the values received, doesn't address confidentiality, and does nothing for people with Active Hosts or Zabbix Trappers.

Active hosts and/or trappers:

I do a lot of automating and scripting and I love the fact I can have Zabbix keep an eye on them all, ensure they've run successfully, how long they ran for etc. I use Zabbix Trappers for this.

The major concern is I had no problems spoofing my Zabbix Trapper Items... Obviously I have firewalls to block unauthorised IPs, but it's not too difficult to spoof one of my whitelisted IPs if they can capture the clear text traffic.

I had a quick attempt at spoofing one of my Passive Items, but that didn't work, so I can only assume the Zabbix server only actions values that come back from its request to the host?

Zabbix API:

Firstly, I use the Zabbix API to build Items and Triggers. The thing about the Zabbix API is that it requires you to login with a valid username and password (rightly so). This was the first major security concern for me, "I'm not sending login details (capable of deleting hosts, groups, etc.) in clear text!" Anyway, that was easily resolved, Self-Signed SSL...

The API is only designed to create/manage Zabbix objects (Hosts, Items, Groups, etc.) in the Zabbix server and doesn't accept/process values for Host Items, fair enough too. During my testing the performance hit between clear text and SSL was noticeable (a dozen or so JSON transactions - an extra couple of seconds, maybe?), but that delay was acceptable for me. I'm running on the weakest of all freebie Amazon EC2 instances though, a real server could probably absorb this a little more readily (not to mention the Core 2 Duo workstation I'm on). So I can only assume that if I ran EVERYTHING through SSL, it would completely obliterate my baby EC2 instance. So, I decided to concentrate on protecting only my Zabbix Trappers for the time being.

Emulating the Zabbix Sender:

Firstly, I have never used the Zabbix Sender. OMG! When scripting or automating I try to avoid using 3rd party exe's unless completely unavoidable. Turns out you simply need to establish a TCP connection to 10051, and shoot a JSON string off to it (e.g. {"request":"agent data","data":[{"host":"MyHost","key":"MyKey","value" :"0"}]}).

Disclaimer: Using the Zabbix Sender is a MUCH better idea, the Zabbix Team will ensure it's always working properly, whereas scripting it as I have done may eventually stop working if the Zabbix Team make necessary (but script breaking) changes. I'd also have to manually update all those scripts rather than just updating the Zabbix Sender on each machine.

So, I have a script that can push JSON requests over an SSL connection, maybe I could create a PHP page to accept my Zabbix Trapper JSON requests...? Needless to say it worked, but you're not interested in that, does that method work with the Zabbix Sender!? Well, it seems to...

Zabbix Sender:

Just while I was typing this post I tested redirecting Zabbix Sender to the localhost (127.0.0.1 didn't work, so I used my local IP) and set my script to listen to Port 10051. I then executed the Zabbix Sender command as follows:
zabbix_sender.exe -z 192.168.XXX.XXX -s Monitored Host -k mysql.queries -o 342.45

As I suspected, the script received the command (in JSON form). Now all I need to do with it is push it to the PHP page I created over SSL. This encrypts 100% of my Zabbix Sender traffic and potentially crippled my micro Amazon EC2 instance...

But... This only solves the unencrypted Zabbix Sender/Trappers issue, the requests sent to/from the Zabbix Server for Passive Hosts will still be in clear text. Maybe we should change to Active Hosts...

Active Hosts:

In theory, if you removed all Passive Hosts and set all your Hosts as Active, then they'd make the request for Items over SSL (redirected through the script), the Zabbix server would generate the appropriate response (displayed on the PHP page and passed back over the same SSL connection). The Active Host would then collect the values from the system, and then submit the values back over a new SSL connection to the server. Encrypting 100% of the Active Host traffic (IN THEORY!).

A few other suggestions/solutions:

Reduce exposure by ensuring your Hosts ISP's connects straight to your Zabbix Server's Hosting Provider. For example my ISP connects straight to Amazons data center (you can check via a tracert from your host to your Zabbix Server). So the only people that can intercept my data are people on my network (though the network switch keeps that risk down), my ISP and Amazon (which I believe may look at the data, but won't change it or use it to spoof me). And possibly the NSA...

For the medium enterprises, theoretically you could use Zabbix Proxies at each site, and then establish a software VPN tunnel from each proxy to the Zabbix Server.

Finally, I've considered using my firewalls to establish persistent VPNs to the Zabbix Server (or its associated firewall).

Signing off:

Well, I'd love to hear your thoughts. Sorry this post turned into an essay, I just got on a roll I guess...

-Timbo

[tchjts1]

Note: I may have gotten carried away with my post...

Well thought out and well written. The input is appreciated.

[eskytthe]

Maybe I was sleeping at the zabbix conference, but as I remember,
Alexei was saying that agent encryption will be implemented from zabbix ver. 2.4 ...
BR
Erik

[BDiE8VNy]

[...] environments that host everything locally (limiting risk to insiders) probably have little use for encryption.

Not necessarily. Security standards like PCI-DSS require encryption particularly to limit risk from inside.

There are others (e.g. 3-D Secure) but PCI-DSS is a nice example.
PCI-DSS enforces (mostly) common good procedures and practices that everyone is aware of but many tend to neglect because the added effort/resources won't be paid.

Bad guys are not always coming from outside.
The ones who do might find a gap - possibly via social vulnerabilities that can't be technically controlled

[timbo]

Not in general. Security standards like PCI-DSS require encryption particularly to limit risk from inside.

I'd agree entirely, but I'm referring to Zabbix Traffic. So PCI-DSS is a little off topic (unless you argue that some of the data sent to/from the Zabbix Server would fall under the scope of PCI-DSS).

Anyway, my statement was hardly definitive "probably have little use for encryption" rather than "definitely have no use for encryption".

And I'd argue that "in general" "smaller environments that host everything locally" do not need network encryption (as the majority of smaller businesses (being the "general") have nothing to do with accepting payments online thus nullifying the PCI-DSS argument). If they do accept payments online, it may often be through Paypal (thus avoiding PCI-DSS obligations). This is without going into the technical limitations that small businesses face (qualified/competent IT staff, and appropriate infrastructure).

Anyway, \Rant.

Sorry, I didn't mean to rant, but I thought the comment was well outside the realms of this discussion on "2013 and agent traffic STILL not encrypted?".

Hmmm... I just reread your post - perhaps you were merely stating "do not underestimate the threat of insiders", which is good advice and would mean my rant was misguided. - http://xkcd.com/386/

-Timbo

[Pada]

I see that SNMP v3 with AES/SHA is now supported in the alpha versions (2.1.x). At least this is a step in the right direction!

If you really need to have your monitoring data sent securely right now, then use something like stunnel (both client + server side) to encrypt & decrypt it using SSL.

[Colttt]

my opinion is zabbix must think like "dont reinvent the wheel" .. zabbix can use some exist libaries for agent encryption, like ssh, openvpn, stunnel etc..

[richlv]

also keep in mind that if you don't test the development branch of https://support.zabbix.com/browse/ZBXNEXT-1263 , you get less reason to complain if it does not work in your environment in the final release