Skip content, jump to navigation.

High Performance Packet Capture

Outline

Capturing packets from a highly utilized link can easily miss a significant fraction of the packets. To avoid this we on this site present some possibilities to increase the capturing performance of a system. Furthermore we recommend a combination of operating system and processor architecture which performs best regarding our investigations.

For those of you who can afford special hardware for the purpose of packet capture we recommend Endace Network Monitoring Cards which offloads the Interrupt handling, exact timestamping, filtering and extracting the relevant data out of the packet from the CPU.

up top

Results

Most of the results in this section are based on our investigations. Please see [Sch04de-sep, Sch05en-da, SW05-conext05].

Tuning FreeBSD

Packet capturing utilizes the BPF device, in general in combination with the libpcap. This device executes a filter on each packet and store the packet afterwards in a double-buffer (see [Sch04de-sep, Sch05en-da] for details). The size of this double-buffer can be adjusted via a sysctl. Setting it to 2×10 Mbytes has proven to be a good choice.

Since FreeBSD 6 you can do this by setting:

sysctl -w net.bpf.bufsize=10485760
sysctl -w net.bpf.maxbufsize=10485760

For older FreeBSD's just use:

sysctl -w debug.bpf_bufsize=10485760
sysctl -w debug.maxbpf_bufsize=10485760

Tuning Linux

Like in FreeBSD Linux offer some possibilities to tune it capturing performance as well. Due to the different capturing stack witch does not have a buffer but a queue of pointers behind the filter, one has to increase the receive buffer for all incomming packets and the queue length. This can be done via the /proc filesystem (with Linux we found that 32 Mbytes is a good amount of memory):

echo 33554432 > /proc/sys/net/core/rmem_default
echo 33554432 > /proc/sys/net/core/rmem_max
echo 10000 > /proc/sys/net/core/netdev_max_backlog

An additional possibility to increase the performance is to use the mmap patch of Phil Wood. Please refer to his site for details. But note that this patch does not support non-blocking reads (select) on the packet socket. This is needed by some network analyser software. Bro for example is one of those.

FreeBSD performs better than Linux

As our investigations have shown FreeBSD outperforms Linux in terms of packet capturing. Thus we suggest to use FreeBSD rather than Linux to the purpose of packet capturing.

AMD Opterons better than Intel Xeons

In our comparison the AMD Opteron processors perform better than Intel Xeon processors. This is not the case when compression is involed in the capturing process. Furthermore the new Intel I/O Acceleration Technology was not yet subject to our investigations.

up top

Related Software / Downloads

Linux-Kernel Config

This Kernel configuration was used for the newest measurements (wrote 600~Mbit/s full packet capture to disk without loosing packets).

Capture Group Patch for Linux

This patch allows to define a UNIX group number which is allowed to open capturing sockets. This is useful to grant the permission to capture packet to user which should not be granted the other rights of the root user. Please see Robin Sommers page for details.

Monitoring the CPU usage

The small utility cpusage write the statistics like in the CPU: line from top twice per second to a file.

LKPG enhancement

We enhance the Linux Kernel Packet Generator to produce packets following a given size distribution, instead of produce packet of only one size. We used this tool to generate the traffic for our measurements.

up top

Contributors

Contributors:

I you want to contribute to this web page, have a hint or recent expierieces with capturing in Gigabit++ enviroments, feel free to write an e-mail to Fabian Schneider. We will then put it on the page.

up top

Publications on the topic of High Performance Packet Capture

Conferences and Workshops

  1. Fabian Schneider, Jörg Wallerich, Anja Feldmann. Packet Capture in 10-Gigabit Ethernet Environments Using Contemporary Commodity Hardware. In Proceedings of the 8th International Conference on Passive and Active Network Measurement, (Location: Louvain-la-Neuve, Belgium), Lecture Notes in Computer Science, Volume 4427, Pages 207-217, Springer-Verlag Berlin Heidelberg, New York, NY, USA, April 2007. details
  2. Fabian Schneider, Jörg Wallerich. Performance evaluation of packet capturing systems for high-speed networks. In CoNEXT'05 Student Workshop: Proceedings of the 2005 ACM conference on Emerging network experiment and technology, (Location: Toulouse, France), Pages 284-285, ACM Press, New York, NY, USA, October 2005. details

Misc (Posters, Talks, etc.)

  1. Fabian Schneider, Jörg Wallerich, Anja Feldmann. Packet Capturing in 10-GigE Environments using Contemporary Commodity Hardware. Poster at Deutsche Telekom Laboratories, Annual Review Meeting, February 2007. details

Bachelor/Master Thesis and Projects

  1. Fabian Schneider. Performance evaluation of packet capturing systems for high-speed networks. Diplomarbeit Technische Universität München, Munich, Germany, November 2005. details
  2. Fabian Schneider. Analyse der Leistung von BPF und libpcap in Gigabit-Ethernet Umgebungen. Systementwicklungsprojekt Technische Universität München, Munich, Germany, October 2004. details

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

It has been automatically generated using the bib2html program.