Packet Monitoring Agent

[nelsonab]

I don't quite know where to put this as it definitely is not a patch. :-)

So here goes! I have been working on a specialized agent which interfaces directly with LibPcap. The agent will allow you to monitor two types of packets on the wire, TCP/UDP traffic occurring on a particular port or traffic bound to an Ethernet MAC address.

Before using the agent you will need to download and install the Ruby LibPcap libraries from http://www.goto.info.waseda.ac.jp/~fukusima/ruby/pcap-e.html
Inside the tarball will be instructions for compiling and installing the libraries. Before running the agent you will need to edit line 293 "PacketAgent.new". The first argument is the IP address you want the agent to bind to, the second is the port, and the third is the arguments you wish to pass to the LibPcap libraries, this includes the Ethernet adapter you want the agent to listen on. Also the agent will need to run as root. It may run as another privileged user but I have not tested this.

How it works:
All information regarding what packets are to be captured is stored on the Zabbix server in the item's key. If the agent detects a new filter it will be added dynamically.

*KEY POINT*
In every key there is a field for name, this must be unique for each host you wish to monitor! This is how the agent knows which filter you are looking for. This will likely change in the future, but it made for a faster path to testing. Each filter stores byte and packet information.

There are three key types:
monitor.tcpudp[name,ip,port,p/b]
name: This is a unique name for this filter
ip: The IP address you wish to monitor
port: The port you wish to monitor
p/b: Do you want to store packets or bytes? Only p or b is allowed.
Edit: The filter does not distinguish between TCP or UDP. It matches IP and Port number only, not protocol.

monitor.macaddr[name,macaddr,p/b]
name: The unique name for this filter
macaddr: The mac address for this host
p/b: packets or bytes to be returned

zabbix.agent
This returns some version information about the agent, this will likely change to conform to the standard agent string if it does not already. (I was too lazy to check)

Some examples:
Name: SomeHost Web bytes
Key: monitor.tcpudp[SH_HTTP,1.2.3.4,80,b]
Name: SomeHost Web packets
Key: monitor.tcpudp[SH_HTTP,1.2.3.4,80,p]
Name: SomeHost SSH bytes
Key: monitor.tcpudp[SH_SSH,1.2.3.4,22,b]
Name: SomeHost SSH packets
Key: monitor.tcp.udp[SH_SSH,1.2.3.4,22,p]
*Key: AnotherHost Web bytes
*Key: monitor.tcpudp[SH_HTTP,5.6.7.8,88,b]
Key: Router Mac bytes
Key: monitor.macaddr[RTR_MAC,00:01:02:03:04:05:06,b]

One of the entries is marked with a * for a reason to show a side effect of the name requirement. In the above example "AnotherHost Web bytes" would actually be storing the byte information for "SomeHost." The IP and port information are different but right now the agent keys on the name field.

You will likely want to import the data as a delta over time.

If you have any questions/comments please feel free to ask/say. I will be updating my blog on here as I make progress. Some day I'll probably port this to C for performance reasons.

Attached is a screenshot of some of the graphs, along with some of the items and the ruby agent gzipped.

[Tenzer]

It sounds really neat!
I would recommend you to add it to the wiki: http://www.zabbix.com/wiki/doku.php

[bbrendon]

Very cool.

Though how come you didn't use iptables to do this?

[nelsonab]

I wanted to use IPTables originally but I just couldn't find documentation that explained what I needed well enough. On top of that if you were to use IPTables all traffic would have to flow through the box, using LibPcap you can sit on a monitor port on a switch and sniff the traffic as it floats on by so to speak. If the server running the agent dies the network is not affected.

       ----------
Host  -| switch | - Host
       ----------
        |      |  (mirrored ports)
     Zabbix   Router
      Agent

In the testing environment I'm running a Dual Core Intel with two VMWare instances. One is the server, the other is the agent. The Box itself has three NICs, one with an IP address to the network and the other two are plugged into mirror ports on two switches both without IP address.


My next goal is to add a few more filter types, remove the name requirement and add support for multiple NICs.

[bbrendon]

I see. good point. I'm not sure how you could do iptables+promiscuous mode either. This guy suggests its not possible:

http://www.linuxarkivet.se/mlists/netfilter-devel/0211/msg00062.html

[kilativv]

Very awesome - I'm surprised this thread is so relatively obscure. I've been using ntop for packet monitoring, but have been looking for something that integrates into zabbix nicely.

However, I wasn't able to get this to work. I installed the script on my sensor box. It has two NICs - one is connected to a mirrored port on a switch and another one has IP and the scripts binds to it. Ntop works just fine on this box, so I can conclude that libpcap is functioning.

My zabbix server connects to the agent and I can see that agent is creating new filters. But in zabbix I see that all values equal to zero. The filters are:
1) monitor.tcpudp[SH_HTTP,10.10.15.251,80,b]
2) monitor.tcpudp[SH_BThomPr,10.10.15.223,10000,b]
3) monitor.tcpudp[SH_ThomPr,10.10.15.254,4900,b]

There's definetly a lot of traffic in those ports(I can see it in ntop)

[nelsonab]

That's strange. How are you running the agent? As root, within nohup?

I found that when I ran it as root inside nohup the total packet items would not return, yet everything else did.

I have attached the latest version of the agent I am currently using. Let me know if that helps.

Also when I am having issues I'll telnet to the port the agent is running on and manually request the key. In your case you would do something similar to the following:

telnet host 10060
monitor.tcpudp[SH_HTTP,10.10.15.251,80,b]
-> agent returns something and closes connection

One other thing to look at. Which ethernet adapter is associated with packet capture? The agent is hard coded to eth1 for monitoring. If you wish to change this change eth1 on line 386 (in the version attached) to the appropriate adapter. I'll work on setting up the agent to understand command line args and config files at some point.

[kilativv]

Thank you very much for your answer. Actually, the problem was due to my stupidity. I need to monitor the overall traffic that enters our office - thus I don't need to match hosts at all. I made simple alteration to the script so I can calculate the traffic on the interface.

[nelsonab]

Cool!

If you would like me to add a feature let me know. Did you find modifying the script to be relatively straight forward?

[odium4u]

I am sorry, but this deserves a bump.

Nice work, works well.