Sniffer Appliance (1)

One of the interesting projects that I worked on recently is - to capture and store packets going across the network on a really large storage device and perform analysis later on. I started out building a home-grown PC with a few hard drives in it and run wireshark to capture data. I couldn't use tcpdump because it does not have an option to wrap the number of files it writes.

(tshark -b filesize:50000 -b files:9999 -w capture-files) was the command that I used.
-b filesize:50000 - save each file at 50MB
-b files:9999 - do not create more than 9,999 files
So it nicely fits into 750GB hard-drive.

I did not have much problem with capturing and storing files. Captured packets are saved in native libpcap format which can be read with almost all the sniffer program.

Problems arose when I tried to extract data from it.
1. Lets say I want to see some packets from certain data and time. When there are lots of traffic on the network, the files are created very quickly. Sometimes, 10 files in 1 minute which equates to 500MB of packet data. Now I have to run query against the file using the time stamp of the file (remember there are about 10 files in that approximate time range) and hope I get the right one.
2. Statistical analysis tools and graphing tools are not very good with wireshark client software, especially when working with large amount of data. It is slow and painful to merge those files. It is almost impossible to generate daily traffic report from the captured files.
3. libpcap file format is not really optimized for database task and hence I cannot create index nor use SQL-ish query against it.

Wireshark is a great program for ad-hoc sniffing. It is just not designed to handle the way I want to use it, capturing and analyzing large amount of packets.

So, I gave up the idea of home-grown solution and started looking for a vendor who will solve the problems that I have. I found following three vendors/products and they all sound promising.
1. Network General's Infinistream
2. Network Instrument's GigaStor
3. Niksun's NetVCR

I contacted them and asked for an evaluation unit. I put their units in our lab and started testing out. I will write my experience with my lab in the next entry.

I am just wondering if
1. it there any open-source solution
2. how difficult would it be to write an application which store packets into a SQL or similar database