Capturing packets from a highly utilized link can easily miss a significant fraction of the packets. To avoid this we on this site present some possibilities to increase the capturing performance of a system. Furthermore we recommend a combination of operating system and processor architecture which performs best regarding our investigations.
For those of you who can afford special hardware for the purpose of packet capture we recommend Endace Network Monitoring Cards which offloads the Interrupt handling, exact timestamping, filtering and extracting the relevant data out of the packet from the CPU.
There is some new work presented in IMC'10: "Comparing and Improving Current Packet Capturing Solutions based on Commodity Hardware" (by L. Braun, A. Didebulidze, N. Kammenhuber, G. Carle (TU München) and "High Speed Network Traffic Analysis with Commodity Multi-core Systems" (by F. Fusco (IBM Research), Luca Deri (ntop.org)). There is also Alexandre Fiveg working on improving packet capture for FreeBSD (see: Short Summary (german, pdf), Slides (pdf), Slides from FreeBSD Developers Summit 2010 (pdf)).
Packet capturing utilizes the BPF device, in
general in combination with the libpcap. This device executes
a filter on each packet and store the packet afterwards in a
double-buffer (see [Sch04de-sep, Sch05en-da] for details). The size of this
double-buffer can be adjusted via a
it to 2×10 Mbytes has proven to be a good choice.
Since FreeBSD 6 you can do this by setting:
sysctl -w net.bpf.bufsize=10485760 sysctl -w net.bpf.maxbufsize=10485760
For older FreeBSD's just use:
sysctl -w debug.bpf_bufsize=10485760 sysctl -w debug.maxbpf_bufsize=10485760
Like in FreeBSD Linux offer some possibilities
to tune it capturing performance as well. Due to the different
capturing stack witch does not have a buffer but a queue of
pointers behind the filter, one has to increase the receive buffer
for all incomming packets and the queue length. This can be done
/proc filesystem (with Linux we found that
32 Mbytes is a good amount of memory):
echo 33554432 > /proc/sys/net/core/rmem_default echo 33554432 > /proc/sys/net/core/rmem_max echo 10000 > /proc/sys/net/core/netdev_max_backlog
An additional possibility to increase the performance is to use the mmap patch of Phil Wood. Please refer to his site for details. But note that this patch does not support non-blocking reads (select) on the packet socket. This is needed by some network analyser software. Bro for example is one of those.
FreeBSD performs better than Linux
As our investigations have shown FreeBSD outperforms Linux in terms of packet capturing. Thus we suggest to use FreeBSD rather than Linux to the purpose of packet capturing.
AMD Opterons better than Intel Xeons
In our comparison the AMD Opteron processors perform better than Intel Xeon processors. This is not the case when compression is involed in the capturing process. Furthermore the new Intel I/O Acceleration Technology was not yet subject to our investigations.
- Linux-Kernel Config
This Kernel configuration was used for the newest measurements (wrote 600~Mbit/s full packet capture to disk without loosing packets).
- Capture Group Patch for Linux
This patch allows to define a UNIX group number which is allowed to open capturing sockets. This is useful to grant the permission to capture packet to user which should not be granted the other rights of the root user. Please see Robin Sommers page for details.
- Monitoring the CPU usage
The small utility cpusage write the statistics like in the CPU: line from top twice per second to a file.
- LKPG enhancement
We enhance the Linux Kernel Packet Generator to produce packets following a given size distribution, instead of produce packet of only one size. We used this tool to generate the traffic for our measurements.
- Ringmap FreeBSD kernel patches
We developed a patch to the FreeBSD kernel that improves capturing performance, see Ringmap Project page for details and download. The basic approach is to:
- Use ring buffers for eliminating memory allocations and reusing packet buffers to save new allocations.
- Use shared memory buffers (memory mapping) for eliminating packet copy operations by mapping DMA buffers into the user-space.
TU Berlin): FreeBSD tuning, ringmap Kernel patches
Jan Böttger (TU Berlin/DT Laboratories)
Fabian Schneider (TU Berlin/DT Laboratories): Linux tuning, Kernel Patches
Jörg Wallerich (TU Berlin/DT Laboratories): FreeBSD tuning
Anja Feldmann (TU Berlin/DT Laboratories)
Robin Sommer (ICSI): Capture Group Patch
I you want to contribute to this web page, have a hint or recent expierieces with capturing in Gigabit++ enviroments, feel free to write an e-mail to Fabian Schneider. We will then put it on the page.