Ad Widget

**nelsonab** · 29-12-2010, 09:41

I'm interested. I also looked at implementing this at one point in time, but alas I am but one person with lots of Zabbix interests... :-)

I think no matter what the solution it should leverage common standards. Two of which come to mind Kerberos and SSL. Kerberos is very nice because it has a forced authentication design where every host must know the pre shared key before they can even begin to authenticate themselves.

I think before choosing which technology a bigger question needs to be asked. Who is the end market? If the end market is the larger enterprise space then Kerberos is a better idea as it would allow Zabbix to be integrated with an in house authentication and auditing system. If the end market is not the enterprise then SSL is a better idea. It does not require as much effort up front to set up, however understanding SSL can be a bit trickier with regards to using pre-shared keys and keyed authentication schemes. SSL does not easily allow for centralized auditing and control either. When I talk about centralized I am referring to something like the Auth Token received i the first phase of a Kerberos authentication.

Personally I would rather something like this be a community effort, I can appreciate the desire to pay someone to do this and get it done faster but I think the only way we are going to gain higher community involvement is if people such as yourself champion an idea for others to help with. In addition I think this may dove tail very nicely with some community rumors I have been hearing.

**nelsonab** · 29-12-2010, 10:58

I've done some more research and I'm leaning more towards Kerberos than before.

Here are some links
Overview of how Kerberos works:

Page not found - ITPRC

http://learn-networking.com/network-security/how-kerberos-authentication-works

Some conversations of SSL vs Kerberos
http://stackoverflow.com/questions/1...authentication

Kerberos FAQ, v2.0 (last modified 8/18/2000)Section - 1.29. What are the advantages/disadvantages of Kerberos vs. SSL?

http://www.faqs.org/faqs/kerberos-faq/general/section-31.html

Kerberos FAQ, v2.0 (last modified 8/18/2000)Section - 1.29. What are the advantages/disadvantages of Kerberos vs. SSL?

The biggest advantage of Kerberos is that is can very strongly establish a trust relationship between the client and the server, both parties can know that each is who they say they are and that they are allowed to talk to each other. To do the same with SSL would require a bit more effort. I have not yet found any data on the load and speed differences between the two but I will keep digging.

**walterheck** · 29-12-2010, 13:23

Originally posted by nelsonab

I think no matter what the solution it should leverage common standards.

I couldn't agree more. No need to reinvent the wheel.

Originally posted by nelsonab

I think before choosing which technology a bigger question needs to be asked. Who is the end market? If the end market is the larger enterprise space then Kerberos is a better idea as it would allow Zabbix to be integrated with an in house authentication and auditing system. If the end market is not the enterprise then SSL is a better idea. It does not require as much effort up front to set up, however understanding SSL can be a bit trickier with regards to using pre-shared keys and keyed authentication schemes. SSL does not easily allow for centralized auditing and control either. When I talk about centralized I am referring to something like the Auth Token received i the first phase of a Kerberos authentication.

SSL would probably be easier to set up since it doesn't need a separate daemon, but I lean towards kerberos as it can be easily used for other purposes as well and it allows scaling. Smaller setups might have a simple kerberos daemon on the zabbix server (or none at all if they don't care about authentication, for instance inside a private network), bigger setups can go all crazy with external redundant Kerberos setups.

Originally posted by nelsonab

Personally I would rather something like this be a community effort, I can appreciate the desire to pay someone to do this and get it done faster but I think the only way we are going to gain higher community involvement is if people such as yourself champion an idea for others to help with.

I think the best division of time is for me to not be programming C

I don't mind rallying support for implementing this, and being involved in any stage of it, but the actual programming is better not to be done by a C-noob like myself.
That said, i'm all for community involvement here

Originally posted by nelsonab

In addition I think this may dove tail very nicely with some community rumors I have been hearing.

Ooh gossip, Curious!

**nelsonab** · 29-12-2010, 17:54

Originally posted by walterheck

SSL would probably be easier to set up since it doesn't need a separate daemon, but I lean towards kerberos as it can be easily used for other purposes as well and it allows scaling. Smaller setups might have a simple kerberos daemon on the zabbix server (or none at all if they don't care about authentication, for instance inside a private network), bigger setups can go all crazy with external redundant Kerberos setups.

True, but from our conversation I think Kerberos is the best choice for what you are looking for, thus I think implementing Kerberos first and then SSL would be the best. I do however think both should be implemented at the same time, no use in figuring out how to implementing one, hacking it in and then doing it all over again to hack the other and break the first. In addition it is very easy for people to not implement SSL correctly and leave it just as weak as unencrypted.

Originally posted by walterheck

I think the best division of time is for me to not be programming C

I don't mind rallying support for implementing this, and being involved in any stage of it, but the actual programming is better not to be done by a C-noob like myself.
That said, i'm all for community involvement here

I'm willing to work with your programmer to help make sure what the community get's is a solid effort worthy of inclusion. In addition I'll be sure the comments in the source code help narrate what is going on... ;-)

Originally posted by walterheck

Ooh gossip, Curious!

Hang out in IRC, you get to hear lots of gossip... ;-)

**qix** · 04-01-2011, 13:52

I would like to opt for SSL/TLS.

Not only would this enable authentication via certificates, it would also encrypt the traffic which is somewhat of a weak point for zabbix as well.

Two birds, one stone and all that

I think the stunnel project has a nice codebase for doing authentication via certs. If the license is compatible, it might be a good starting point.

I don't know how the Bacula PSK scheme works that was mentioned earlier, but that might also work quite nicely if you can use PSK pairs per agent.

**nelsonab** · 04-01-2011, 17:09

Kerberos also gives you encryption, and it gives you something SSL/TLS does not, cenralized management for authentication. Though on the negative side Kerberos is kinda herd to understand the first time you look at it.

SSL/TLS however is a little easier to impliment but has some serious pitfalls if you are not careful. The biggest two of which are not managing your randomization correctly and not implementing the key exchange correctly. If you use the standard OpenSSL libraries hopefully the trickier pieces of the key exchange are managed for you, however randomization is something to be aware of.

Slightly less of a problem, but one of concern for Zabbix, is the overhead of communication setup/teardown for SSL. If I remember correctly the initial key exchange takes a few packets and if you have to do this with every Zabbix you've probably increased your average packet size by several fold and the number of packets again several fold. Not good. There is a facility in the OpenSSL library to continue a pre-existing SSL session. If you can use that then you can save a lot of overhead.

Overhead I think is one area where Kerberos has an advantage over SSL. The initial session setup is a little more complex but once you get past that Kerberos is very straight forward.

**qix** · 05-01-2011, 11:10

Hi Nelsonab,

Kerberos also gives you encryption, and it gives you something SSL/TLS does not, cenralized management for authentication. Though on the negative side Kerberos is kinda herd to understand the first time you look at it.

I believe the encryption Kerberos uses is purely for the exchange of the authentication tokens (tickets). I can not remember that Kerberos will do encryption of the data flow too, I think that is up to the specific application that relies on Kerberos.

IMHO I believe SSL/TLS certificates also provide central mangement of authentication, if you run your own CA you can generate and revoke certificates as you see fit. Better yet, you can rely on the already existent PKI. With Kerberos you will need to setup your own KDC, etc.

However, I don't actually have any working experience with Kerberos, so I could be very wrong

Slightly less of a problem, but one of concern for Zabbix, is the overhead of communication setup/teardown for SSL. If I remember correctly the initial key exchange takes a few packets and if you have to do this with every Zabbix you've probably increased your average packet size by several fold and the number of packets again several fold. Not good. There is a facility in the OpenSSL library to continue a pre-existing SSL session. If you can use that then you can save a lot of overhead.

I agree, this is something to be very careful with indeed. It could eat up resources very quickly.
I was under the impression that SSL negotiation will only be used during a setup of a TCP connection, and that is will only add a little overhead per packet so long as the TCP connection is still active. If I'm correct, there shouldn't be much of a problem for Zabbix, since it uses 1 TCP connection to send it's values since version 1.6, or did I misunderstand?

Overhead I think is one area where Kerberos has an advantage over SSL. The initial session setup is a little more complex but once you get past that Kerberos is very straight forward.

If you mean network overhead, I completely agree. Kerberos seems to be very efficient. For management overhead, I'm not sure.
If you already have an existing Kerberos domain like a M$ AD, this might be true. However, most customers I know don't really have a very elaborate Kerberos setup, so this would mean investing in a (redundant) Kerberos infrastructure. PKI is already in place, although it will cost you a few bucks for the certs. If you have a large setup, you can run your own CA (even with self signed certs) and do it more cost effective.

Just some thoughts from my point of view. I really would like this feature, so I hope this discussion will attract some other bright minds to share their ideas on this.

**nelsonab** · 05-01-2011, 19:25

Originally posted by qix

I believe the encryption Kerberos uses is purely for the exchange of the authentication tokens (tickets). I can not remember that Kerberos will do encryption of the data flow too, I think that is up to the specific application that relies on Kerberos.

It depends on what you read. Some say it does not and only affects the Authentication phase, where others talk about the whole thing being encrypted. I think strictly speaking the first is correct, however GSSAPI seems to often be used with Kerberos which allows for encryption of the payload.

[QUOTE=qix;78123]IMHO I believe SSL/TLS certificates also provide central mangement of authentication, if you run your own CA you can generate and revoke certificates as you see fit. Better yet, you can rely on the already existent PKI. With Kerberos you will need to setup your own KDC, etc.
[quote]

Yes and no. The problem with key revocation is the requirement to keep the revocation chain, with kerberos you just disallow access the next time the TGT (Ticket Granting Ticket) is requested from the KDC. This brings up an interesting question though. If access is revoked how long does it take to propagate?

Also one interesting thing I learned when I was researching this before was that Kerberos gives a very high degree of assurance that each side of the conversation is who they say they are. PKI is often one sided. If you were to give assurance to each side you would have to issue keys to each party, and with the revocation chains that can become a management nightmare.

I would also say that due to the nature of how Zabbix is designed this is key. The most important thing is that the server is assured that they are talking to the client they should be talking to. In the use case brought up by Walterheck this is critical.

.

Originally posted by qix

I was under the impression that SSL negotiation will only be used during a setup of a TCP connection, and that is will only add a little overhead per packet so long as the TCP connection is still active. If I'm correct, there shouldn't be much of a problem for Zabbix, since it uses 1 TCP connection to send it's values since version 1.6, or did I misunderstand?

I will need to do some tcpdumps to be certain here but as I understand it passive checks still do one item per TCP connection. Active checks spool data and can send a large dump of data per TCP connection. What is not clear in the documentation is the TCP tear down. Another unknown right now is can SSL handle multiple TCP connections from one negotiation or does each TCP connection require it's own negotiation. If so then SSL resume will be a major challenge. AFAIK this is not as much of an issue with Kerberos.

[QUOTE=qix;78123]If you already have an existing Kerberos domain like a M$ AD, this might be true. However, most customers I know don't really have a very elaborate Kerberos setup, so this would mean investing in a (redundant) Kerberos infrastructure. PKI is already in place, although it will cost you a few bucks for the certs. If you have a large setup, you can run your own CA (even with self signed certs) and do it more cost effective.

I would argue that in the long run PKI would require much more work than Kerberos especially for larger environments (In my mind that starts at around 20 systems without any scripting). When you get to the hundreds or thousands of systems I would think that PKI has a greater potential to become unruly without good management and scripting.

Originally posted by qix

Just some thoughts from my point of view. I really would like this feature, so I hope this discussion will attract some other bright minds to share their ideas on this.

Agreed! I'm not an expert either and I think the more we debate and discuss the greater chance we have of discussing potential pitfalls of either solution and the greater chance we have at a good implementation. Though in the long run I think both will need to be added for reasons we have both been talking about. :-)

**mushero** · 04-02-2011, 11:17

Kerberos Painful for customer servers

Kerberos would seem good for big setups and in theory, though in my experience it often doesn't get done due to real or perceived complexity (we haven't done it for this reason). More importantly, we monitor customers' servers globally and the less we have to install on the server, the better, fewer firewall ports, port forwards, daemons, etc. is far, far better for us. It's not realistic for us to install Kerberos on thousands of hosts in hundreds of customer sites/data centers, especially on existing servers that are not (until we take over), well-managed / stable.

Thus I'd vote for SSL first as it's easy to use, simple even for simple setups such as a few remote offices, and everyone who is scared of or too small for Kerberos. This would be the majority of Zabbix shops, I think.

I agree SSL has issues as it scales, but with SSL, we'd use it today, easily; Kerberos is a low priority in our infrastructure and off in the future, thus SSL useful now, to everyone, with things like Kerberos of course helpful to some.

I'd also think SSL easier to implement using all the already installed and well-understood tools, libraries, etc. and thus we could all use it and learn how to best use it before plunging into Kerberos.

**JBo** · 09-02-2011, 00:18

Hi all,

Sorry for joining this thread so late but after reading it I'd like to share my opinion.

I agree with mushero that Kerberos is too complex. I need to monitor customer servers on remote sites where Kerberos deployment would just be impossible.

About SSL negotiation overhead, I have developed some time ago an https proxy that allowed to collect data from agents at remote locations where the only open port was 443 on a web server. This was not the most efficient way to collect data from Zabbix agents but it worked.

I have started thinking some time ago about using stunnel to encapsulate traffic between zabbix agents and zabbix server but I haven't found any simple solution yet.

Stunnel seems a good base because:

communication is encrypted;
it supports server and client authentication with X509 certificates.
it is available on a lot of systems including various Unix, Linux and Windows
it is licenced under GPL as Zabbix

The architecture I have imagined looks like:

Code:

[FONT=Courier New][SIZE=1]+------+    +-------+   +-------+                      +-------+   +------+
|Zabbix|    |stunnel|   |       |                      |       |   |zabbix|
|server|<-->| proxy |<->|stunnel|<--(encrypted data)-->|stunnel|<->|agent |
+------+    +-------+   +-------+                      +-------+   +------+
\------------ local ------------/                      \----- remote -----/
[/SIZE][/FONT]

stunnel proxy would be a modified zabbix proxy that would talk to Zabbix server with standard Zabbix proxy protocol and rely on stunnel to encrypt communication to remote device.

On remote device, zabbix agent would be bound to 127.0.0.1 only and stunnel would proxy all communication with outside.
There are still a lot of problems to solve (do we support active mode, passive mode, both ?,...).

Any feedback is welcome.

Regards,
JBo

**nelsonab** · 09-02-2011, 06:30

I've been digging into the code and various auth mechanisms, and I think I agree with others that the easiest thing would be to add SSL first and then add auth via SASL or some similar mechanism should it be necessary.

There are a few challenges in implementing SSL along with a few questions of implementation steps. Right now the biggest challenge I see is the volume of connections to manage. Let's assume a small configuration of 10 hosts with a volume of 10 items per second, let's also assume there are no active connections. Also let's assume there are only 5 poller threads. Assuming an even distribution each thread is making two connections per second. They could all be to the same host or to many hosts. If they are all to the same host then there are 5 different TCP connections to the agent. Unless an effort is made to reuse SSL session ID's then each connection needs to go through a full reauth process. It would be nice to use session reuse, but that will likely require creating a session/key cache for all of the threads to use, in addition there are some potential race conditions to be aware of, what happens when two threads try to initiate a connection to the same host which does not have a pre-existing session id?

I'll keep digging around on this and may write a few test programs to test some ideas but this is definitely going to be an interesting challenge. Walter, how are things coming on your end?

JBo, you are right using IPSEC or stun or a similar solution is the easiest way for now to encrypt the traffic, but encrypted traffic is not the same as authenticated traffic and vice versa.

**walterheck** · 15-02-2011, 01:39

Originally posted by nelsonab

Walter, how are things coming on your end?

I've been distracted a bit, but I just revived email conversation with Rihard, seeing which agreement we can come to. I don't really want to start work without a proper agreed solution, as that can easily lead to waste of time and money (even though it can be fun as well

)

Richard suggested going even simpler then SSL, using symmetric pre-shared keys. This would simplify things even more, and it would be more then sufficient for my requirements.

Thoughts?

**richlv** · 15-02-2011, 10:27

Originally posted by walterheck

...even simpler then SSL, using symmetric pre-shared keys. This would simplify things even more, and it would be more then sufficient for my requirements.

it would be more simple, but i'm not sure by how much - technical foundation would be somewhat similar, i believe.

if you haven't already, it might be worth looking at how other projects have implemented this - one example brought up before already is bacula.

http://www.bacula.org/manuals/en/ins...ty_Issues.html is quite short, just saying "The Bacula daemons are password protected using CRAM-MD5" - which probably is the simple approach i mentioned

then, http://www.bacula.org/2.4.x-manuals/...opers/TLS.html talks a bit about tls implementation in bacula.

while fairly old, that information might provide more background information.

**walterheck** · 14-06-2011, 11:36

Update: Ok, here's what my developer is working on and a preliminary schedule:

Development schedule
6 June: Completion of design (2 weeks)
27 June: Finalization of design (3 weeks) (will involve discussions and some development for prototyping + laying the ground work stuff)
11 July: Code freeze (2 weeks) (the module should be ready and usable by then)
18 July: Writing documentation + bug fixes (1 week)

Authentication module
Authentication protocol
Challenge-handshake authentication protocol will be used in this scenario. This is similar to the authentication protocol used by Bacula. Either MD5/SHA-1 hashing will be used.

Consideration must be taken to add encryption after the handshake succeeded.

The authentication will be checked against the hostname of the agent/proxy as configured on the server and the password.

Enforcing authentication
Authentication will be enforced for the following clients:

Active checks from agents (needs to be enforced regardless it is connected to a server or proxy)
Active proxy to server
zabbix_sender
Authentication will not be enforced for the following clients:

Node communication
Passive checks to agent
Server to passive proxy
zabbix_get
NOTE Needs peer review

Authentication sessions
Once a client successfully authenticated itself to the server, the server should maintain the authenticated session without requiring the client to resubmit the password to reauthenticate themselves.

The client should send a request to logout their current session before the client program is terminated by the user.

The server must also terminate client session if no communication is exchanged between the server and the agent/proxy over a configured timeout period (e.g. due to a network outage). The client programs must be programmed to reauthenticate themselves when the server requests for it.

Configuration
Host configurations At the Zabbix server, each host must contain two new configuration parameters:
Enable/Disable authentication (Defaults to Enabled)
Authentication password (must be set if authentication is enabled)
The connecting clients (zabbix_agent, zabbix_proxy and zabbix_sender) must have an additional parameter to keep its authentication password. The parameter must only be mandatory for active agents/proxies.

QUESTION Should we have a system-wide default password for hosts?

System-wide configuration
TODO Fill this in, probably the only one we need is a configurable timeout/retry attempts to keep authenticated session alive

Backward compatibility
The server must enforce the authentication rule regardless of the client's version.

Ad Widget

Request for feedback (and help) implementing authentication for zabbix agents

Request for feedback (and help) implementing authentication for zabbix agents

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment