Request for feedback (and help) implementing authentication for zabbix agents

[Alexei]

Once a client successfully authenticated itself to the server, the server should maintain the authenticated session without requiring the client to resubmit the password to reauthenticate themselves.

The client should send a request to logout their current session before the client program is terminated by the user.

How would it affect TCP layer? Does it imply that a TCP connection should be established and maintained for each authenticated session?

[nelsonab]

Development schedule
6 June: Completion of design (2 weeks)
27 June: Finalization of design (3 weeks) (will involve discussions and some development for prototyping + laying the ground work stuff)
11 July: Code freeze (2 weeks) (the module should be ready and usable by then)
18 July: Writing documentation + bug fixes (1 week)

I wonder if this schedule is a little aggressive, maybe increasing your design/documenting/prototyping may be helpful. Is your design phase documentation only or are you coding in this phase? If you are coding where is your documentation phase? Taking the time to document up front sucks but can make life a lot easier down the road. It makes it easier to write complex code as most of the complexity has been thought out and if there are bugs later it can be easier to determine if the bug is a logic or implementation bug.

Are you looking to modify only the client/server code or also the web interface? If so will you also be making api hooks available? Are you also going to work on moving to some of the new coding styles for Zabbix?

Authentication module
Authentication protocol
Challenge-handshake authentication protocol will be used in this scenario. This is similar to the authentication protocol used by Bacula. Either MD5/SHA-1 hashing will be used.

Are you going to use a standard library from somewhere or are you writing your own code for this? Copying someone else's library is good but it brings with it maintenance problems, using someone else's library is better. That way if they fix their libraries or you find a bug both benefit easily. Also most authentication/crypto errors are caused when people implement their own algorithms. Also for some environments MD5/SHA-1 may be considered unsuitable and too weak. Perhaps SHA-2 or other 256+ bit hashing methods.

The authentication will be checked against the hostname of the agent/proxy as configured on the server and the password.

Will this also include the ability to have pre-shared keys which are done on a per-domain or per-host basis? Are you also going to leave hooks for external authentication methods such as Kerberos or Radius should someone else wish to pick up and add that later?

Authentication sessions
Once a client successfully authenticated itself to the server, the server should maintain the authenticated session without requiring the client to resubmit the password to reauthenticate themselves.

The client should send a request to logout their current session before the client program is terminated by the user.

The server must also terminate client session if no communication is exchanged between the server and the agent/proxy over a configured timeout period (e.g. due to a network outage). The client programs must be programmed to reauthenticate themselves when the server requests for it.

Two questions, are you going to have this run under a different port than the standard Zabbix port? If you are going to use the same port how are you going to integrate this to be compatible with the existing protocol? Active and node (including proxy) will be easier since they have a structure, but passive checks have no real structure except in the data returned. Has much research already been made into how this aspect of Zabbix works?

Why is there a logout requirement? Zabbix is long running and as such there really isn't a way to enforce logouts. How does NTP handle authentication? Does authenticated NTP require logouts? Perhaps tying the timeout to twice the average check interval for all items on a host?

Also will your authentication hash be renegotiated from time to time? How will you handle this? How are you intending to guard against replay attacks?

QUESTION Should we have a system-wide default password for hosts?

Use pre-shared keys as mentioned above which can either be on per-host or per domain(system/node) basis. Perhaps the pre-shared key can be editable in the web gui and api.

[pierra]

Up

Hello,

Any news of this very interesting project ?

I need to monitor clients on remotes locations and all my agents will be in active mode because I can't configure the firewall of those sites.

I think there is a risk due to the unencrypted informations that'll arrive to the server.
Anybody who intercept packet will be able to draw a map of my network and have access to sensible data about configurations on servers I monitor.

Do I think wrong? Is there no risk ?

[walterheck]

Any news of this very interesting project ?

My developer is hard at work. Development is scheduled to be done in the next week or so, final feature released by the end of July. I'll keep you guys posted. We chose to base off of 1.9.X though, so it won't be in 1.8.x

cheers,

Walter

[walterheck]

Alright, it took a bit longer then expected, but we have an alpha version ready. Here's my developers comments:

---

Hey guys,

here's the initial implementation of authentication for hosts. Could one or more of you please have a look over this and tell us if you guys agree with it all? If so, we can continue with the improvements as mentioned

The current code can be found here, based on 1.9:

https://github.com/felixleong/zabbix/tree/auth

Looking forward to your comments!

Walter

---------- Forwarded message ----------
From: xx
Date: Sun, Aug 7, 2011 at 15:56
Subject: Re: checking in


Hmm… My current thought is that I personally don't mind having the zabbix folks looking at this first and get their input at this stage, since I believe this is the least amount of code required to get authentication running and it's probably better to have my naive assumptions critiqued for now.

I'd probably work on the other aspects of the authentication module as we go along (zabbix_send, authentication from agent -> proxy and proxy -> server, as well as getting rid of plaintext passwords are a few of the things that I have yet to implement yet for a more acceptable solution)

--

I have just checked in some changes so that the server will ignore updates from active agents that are not authenticated when the agent request the active checks. Together with some bug fixes so that the server won't go crashing down.

Just to mention a few things:
This is meant to be a proof of concept, it's best to check in with the Zabbix team to make sure they are comfortable with me injecting new fields in their JSON packets
… as a corollary to the above, the latest changes in the server is NOT BACKWARD COMPATIBLE with 1.8 agents due to one additional field I added to the "agent data" JSON packet (it's the hostname field – which I use to check for authentication status)
I've only tested it on a logical level and some simple cursory tests to see whether it works, please help me to play around with it and spot some bugs. Set the logging level to DEBUG (4) in the configs and send me the log files in case you found bugs
I've yet to propogate the changes to support authentication for proxies, I'm pretty sure that something I need to get done soonish.
I think I may have broke zabbix_send due to authentication requirement, let me check on that first
API is definitely untested, help me test that portion out and let me know the outcome
On the authentication, here's the gist of the checking logic (and hence the possible permutations):
The host can be configured with authentication switched on or off
If the host have the authentication set to off:
the processing will be similar to the 1.8 release
the password field in the "active check" packet will not be used and ignored
If the host have the authentication set to on:
upon initialization, the active agent must send the password in the "active checks" request
the server will check the password against what's stored on the database and reject any invalid passwords
when rejected, the active agent will receive a "auth_required: 1" in the rejection packet
when successful, the server will consider the login session valid for 5 minutes after the "active checks" is first received
the active agent is not expected to send its password again unless otherwise required (i.e. a rejection received with auth_required: 1)
the 1.9 active agent will only send agent data back to the server if it's authenticated (in this case, receiving a success "active checks" response would do)
if the server receives active agent data from unauthenticated agents, the agent data will be discarded and ignored

[nelsonab]

I'm starting to dig into it, but I have one important question, where is the design document? That way we can understand what the code *should* be doing, and if it isn't then perhaps more quickly understand why, and most importantly so you can know when to declare success.

Without a design document all we can do is just dive into the code and *hope* we've understood a dynamic program.

[walterheck]

Design doc attached. It's not much, but it's all we got/figured we'd need. Hope it helps

[nelsonab]

Cool! Thanks for posting the design doc. Something is a lot better than nothing I think I found the first bug according to it.

From what I can tell you are not following CHAP as delineated in the design doc. Rather from what I can tell the password is sent in plaintext and as such is vulnerable to a replay attack. What I don't see/can't find is the password being hashed before being sent.

I think some of the functions could use some logic cleanup and additional comments. The function authenticate in the server/trapper/active.c is one example. The parameter valid, what is it's intended purpose? A comment about that would be nice. The comment about the return value also says nothing about what happens when the password is invalid, as you return SUCCEED even when there's a password fail, that's set in the valid parameter, which at first read left me saying WTF?

I'll dig into some others, but along the lines above, consider extending the standard Zabbix error messages rather than bolt on a new one.

[walterheck]

Cool! Thanks for posting the design doc. Something is a lot better than nothing I think I found the first bug according to it.

Yeah, something is better then nothing indeed.

From what I can tell you are not following CHAP as delineated in the design doc. Rather from what I can tell the password is sent in plaintext and as such is vulnerable to a replay attack. What I don't see/can't find is the password being hashed before being sent.

That is in the message I quoted above somewhere, not yet implemented. This is proof of concept/alpha status after all

I think some of the functions could use some logic cleanup and additional comments. The function authenticate in the server/trapper/active.c is one example. The parameter valid, what is it's intended purpose? A comment about that would be nice. The comment about the return value also says nothing about what happens when the password is invalid, as you return SUCCEED even when there's a password fail, that's set in the valid parameter, which at first read left me saying WTF?

I'll dig into some others, but along the lines above, consider extending the standard Zabbix error messages rather than bolt on a new one.

I'll pass those on to Felix, thanks!

Walter

[nelsonab]

I was thinking more on this. Rather than having a return status and an overall status, merge the two and have the password return SUCCEED, FAIL, PASSWORD_ERROR etc.

I think the bigger challenge will be adding the four way handshake to the Zabbix communication. Right now there really isn't anything like that, to make it happen right and to be compatible with future needs each end needs to have a state engine of sorts, which can be used to gate what commands/options are available at each stage. Also to make it easier to debug/test all of this should work via text.

[walterheck]

Another update, I moved our internal docs to zabbix.org so it is easier for all interested parties to stay up to date.

http://zabbix.org/wiki/Active_agent_authentication

[nelsonab]

Sweet! Community FTW!

[ddf]

Since I have some agent outside (sending data actively), the whole authentication mechanism is of my interest.
I gave a deeper look at the code contained on the branch -auth- and I want to ask you an information: what is the difference between providing and unexploitable hostname (random md5, and so on, since there is a displayed-hostname field which beautify it) and having an extremely simple CHAP mechanism (like the one developed so far).

[walterheck]

I have great news: Felix (my dev) has
finished the first version of the libgsasl-enabled zabbix. It's
implemented on top of the tip of zabbix SVN, so it will only work for
the 1.9.X (soon to be 2.0) versions of zabbix.

I attached a build script that Felix gave me, which should work for
ubuntu 11.10.

Now, what I need from you guys (since you are much smarter then me :P
) is: TEST PLEASE! The more of us can test this, the more solid we
can get it. Please test across platforms, with different config
params, etc.

If the boys at zabbix could do a code review, that would be awesome,
it should be relatively easy. Code can be found here:

https://github.com/Tribily/zabbix/tree/sasl-auth

Please, please, please free up some time and resources to test this.
We will all benefit from it, my company is already carrying all the
cost for the coding and since a lot of people on this topic are
professionally interested in it, it is important that you contribute
as well.

kind regards,

[dimaqq]

This seems to go pretty slow, meanwhile I'm planning to fork https://github.com/nikicat/zabbix-agent-ng and wrap sending of the data in https post, authentication being done with a permanent cookie.
Simple php on the server side will pass post data to zabbix localhost socket.
Shouldn't take me 3 years like this project ;-)