High Performance Packet Capture

Outline

Capturing packets from a highly utilized link can easily miss a significant fraction of the packets. To avoid this we on this site present some possibilities to increase the capturing performance of a system. Furthermore we recommend a combination of operating system and processor architecture which performs best regarding our investigations.

For those of you who can afford special hardware for the purpose of packet capture we recommend Endace Network Monitoring Cards which offloads the Interrupt handling, exact timestamping, filtering and extracting the relevant data out of the packet from the CPU.

There is some new work presented in IMC'10: "Comparing and Improving Current Packet Capturing Solutions based on Commodity Hardware" (by L. Braun, A. Didebulidze, N. Kammenhuber, G. Carle (TU M√ľnchen) and "High Speed Network Traffic Analysis with Commodity Multi-core Systems" (by F. Fusco (IBM Research), Luca Deri (ntop.org)). There is also Alexandre Fiveg working on improving packet capture for FreeBSD (see: Short Summary (german, pdf), Slides (pdf), Slides from FreeBSD Developers Summit 2010 (pdf)).

Results

Most of the results in this section are based on our investigations. Please see [Sch04de-sep, Sch05en-da, SW05-conext05].

Tuning FreeBSD

Packet capturing utilizes the BPF device, in general in combination with the libpcap. This device executes a filter on each packet and store the packet afterwards in a double-buffer (see [Sch04de-sep, Sch05en-da] for details). The size of this double-buffer can be adjusted via a sysctl. Setting it to 2×10 Mbytes has proven to be a good choice.

Since FreeBSD 6 you can do this by setting:

sysctl -w net.bpf.bufsize=10485760
sysctl -w net.bpf.maxbufsize=10485760

For older FreeBSD's just use:

sysctl -w debug.bpf_bufsize=10485760
sysctl -w debug.maxbpf_bufsize=10485760

Tuning Linux

Like in FreeBSD Linux offer some possibilities to tune it capturing performance as well. Due to the different capturing stack witch does not have a buffer but a queue of pointers behind the filter, one has to increase the receive buffer for all incomming packets and the queue length. This can be done via the /proc filesystem (with Linux we found that 32 Mbytes is a good amount of memory):

echo 33554432 > /proc/sys/net/core/rmem_default
echo 33554432 > /proc/sys/net/core/rmem_max
echo 10000 > /proc/sys/net/core/netdev_max_backlog

An additional possibility to increase the performance is to use the mmap patch of Phil Wood. Please refer to his site for details. But note that this patch does not support non-blocking reads (select) on the packet socket. This is needed by some network analyser software. Bro for example is one of those.

FreeBSD performs better than Linux

As our investigations have shown FreeBSD outperforms Linux in terms of packet capturing. Thus we suggest to use FreeBSD rather than Linux to the purpose of packet capturing.

AMD Opterons better than Intel Xeons

In our comparison the AMD Opteron processors perform better than Intel Xeon processors. This is not the case when compression is involed in the capturing process. Furthermore the new Intel I/O Acceleration Technology was not yet subject to our investigations.

Related Software / Downloads

Linux-Kernel Config

This Kernel configuration was used for the newest measurements (wrote 600~Mbit/s full packet capture to disk without loosing packets).

Capture Group Patch for Linux

This patch allows to define a UNIX group number which is allowed to open capturing sockets. This is useful to grant the permission to capture packet to user which should not be granted the other rights of the root user. Please see Robin Sommers page for details.

Monitoring the CPU usage

The small utility cpusage write the statistics like in the CPU: line from top twice per second to a file.

LKPG enhancement

We enhance the Linux Kernel Packet Generator to produce packets following a given size distribution, instead of produce packet of only one size. We used this tool to generate the traffic for our measurements.

Ringmap FreeBSD kernel patches

We developed a patch to the FreeBSD kernel that improves capturing performance, see Ringmap Project page for details and download. The basic approach is to:

  • Use ring buffers for eliminating memory allocations and reusing packet buffers to save new allocations.
  • Use shared memory buffers (memory mapping) for eliminating packet copy operations by mapping DMA buffers into the user-space.
Thus the project name: ring + mapping = ringmap. Make sure to look at: Short Summary (german, pdf), Slides (pdf), Slides from FreeBSD Developers Summit 2010 (pdf).

Contributors

Contributors:

I you want to contribute to this web page, have a hint or recent expierieces with capturing in Gigabit++ enviroments, feel free to write an e-mail to Fabian Schneider. We will then put it on the page.

Publications on the topic of High Performance Packet Capture