PaperlayerView exact revision on arXiv

Source-linked AI summary

MoonGen: A Scriptable High-Speed Packet Generator

Paul Emmerich, Sebastian Gallenmüller, Daniel Raumer, Florian Wohlfart, Georg Carle

arXiv:1410.3322v4cs.NI

TL;DR

MoonGen addresses the cost, flexibility, performance, and precision gaps between hardware and software packet generators. It combines user-controlled Lua scripts, DPDK, and commodity-NIC hardware features to generate and measure traffic. The system provides sub-microsecond latency measurement and tested rates up to 178.5 Mpps at 120 Gbit/s, while its timing approach has defined precision and hardware-scope limitations.

  • Problem

    Existing packet generators often lack the performance or flexibility of software tools or the precision and affordability needed for diverse network evaluations.

  • Method

    MoonGen combines user-defined LuaJIT packet-generation scripts, DPDK packet processing, commodity-NIC hardware timestamping and rate control, and a software inter-packet-gap mechanism.

  • Results

    Sub-microsecond precision and accuracy are achieved for round-trip latency measurement, with tested throughput up to 178.5 Mpps at 120 Gbit/s.

  • Takeaways & Limitations

    MoonGen provides packet-generation functionality previously available only through expensive special-purpose hardware using commodity NIC features and software approaches.

  • Takeaways & Limitations

    Gaps from 1 to 75 bytes cannot be precisely represented, and the approach is optimized for software-based systems rather than hardware appliances.

Abstract

from arXiv · show

We present MoonGen, a flexible high-speed packet generator. It can saturate 10 GbE links with minimum sized packets using only a single CPU core by running on top of the packet processing framework DPDK. Linear multi-core scaling allows for even higher rates: We have tested MoonGen with up to 178.5 Mpps at 120 Gbit/s. We move the whole packet generation logic into user-controlled Lua scripts to achieve the highest possible flexibility. In addition, we utilize hardware features of Intel NICs that have not been used for packet generators previously. A key feature is the measurement of latency with sub-microsecond precision and accuracy by using hardware timestamping capabilities of modern commodity NICs. We address timing issues with software-based packet generators and apply methods to mitigate them with both hardware support on commodity NICs and with a novel method to control the inter-packet gap in software. Features that were previously only possible with hardware-based solutions are now provided by MoonGen on commodity hardware. MoonGen is available as free software under the MIT license at https://github.com/emmericp/MoonGen

1. INTRODUCTION

MoonGen addresses the performance, flexibility, and precision limitations of existing packet generators with a scriptable software design built on DPDK. It reaches line rate on 10 GbE from one core, scales across cores, measures latency precisely, and supports hardware-assisted and software rate control.

  • Existing packet generators trade off cost, flexibility, performance, or precision, motivating a more capable software solution.
  • MoonGen moves packet-generation logic into user-controlled Lua scripts while using LuaJIT and DPDK.
  • 14.88 Mpps saturates 10 GbE with minimum-sized packets from a single CPU core while executing script code per packet.
  • 178.5 Mpps at 120 Gbit/s was tested using twelve 2 GHz CPU cores, demonstrating multi-core scaling.
  • MoonGen measures round-trip latency with sub-microsecond precision and accuracy using hardware features in Intel commodity NICs.
  • Its rate-control work combines software and hardware mechanisms to generate complex traffic patterns without additional hardware support.

2. STATE OF THE ART

Prior packet generators divide between fast but limited systems and flexible but slower or less precise software tools. MoonGen is motivated as a hybrid intended to combine software flexibility with hardware-like precision.

  • Packet generators face a tradeoff between limited high-speed operation and feature-rich functionality that cannot sustain high data rates.
  • Hardware packet generators provide high packet rates, accurate sending rates, and precise timestamping, but are tailored to special use cases.
  • NetFPGA-based generators offer highly accurate inter-packet delays and nanosecond latency measurement, but remain more expensive than commodity hardware.
  • Software generators on commodity hardware commonly struggle with minimum-sized 10 GbE traffic, with one cited system reaching 12 Mpps using multiple cores.
  • Ostinato supports Python scripting and high-speed packet I/O, but its scripts configure predefined settings and lack per-packet execution, precise timestamping, and rate control.
  • Existing options force a choice between software flexibility and hardware precision, motivating a hybrid usable across measurement scenarios.

3. IMPLEMENTATION

MoonGen combines user-defined LuaJIT scripts with DPDK and commodity-NIC hardware features to provide flexible, high-speed packet processing and precise control. Its design also accounts for runtime pauses, hardware scope, and timing limitations.

  • MoonGen’s requirements are commodity software operation, multi-link 10 GbE saturation, maximum flexibility, and precise timestamping and rate control.
  • DPDK was chosen for broad multi-vendor NIC support, documentation, speed, licensing, and direct NIC-register access needed for hardware timestamping and rate control.
  • The packet-generation logic resides in user-defined LuaJIT scripts, integrated with DPDK through LuaJIT’s foreign function interface.
  • LuaJIT can introduce microsecond-scale pauses from compilation and garbage collection, although DPDK buffers remain invisible to the collector and collection can often be disabled.
  • NIC buffers conceal short pauses; the X540’s 160 kB transmit buffer stores 128 µs of data at 10 GbE, sufficient for the reported tests.
  • MoonGen uses NIC queues and hardware features for multi-core scaling, timestamping, and rate control, but hardware-assisted features are currently limited to selected Intel chips.
  • MoonGen’s core is a Lua wrapper for DPDK, with about 80% of the codebase in Lua and support for arbitrary packet-processing tasks.
  • The architecture runs a user-provided script whose master task configures NICs and launches independent LuaJIT slave tasks pinned to CPU cores.

4. SCRIPTING API

MoonGen’s scripting API exposes packet transmission, reception, counters, queue configuration, and hardware rate control through Lua tasks. Example scripts demonstrate batched packet generation, per-flow statistics, and adaptable measurement setups.

  • Example scripts configure devices and queues, set foreground and background rates, launch transmission tasks, and start a packet-counter task.
  • The master function receives devices and rates as arguments, configures transmit and receive devices, waits for links, enables hardware rate control, and starts slave tasks.
  • Transmission tasks allocate pre-filled packet buffers, modify changing fields such as randomized source IP addresses, enable checksum offloading, and transmit batches.
  • Counters can report average packet and byte rates with standard deviations and support plain, CSV, or file-directed output.
  • Batch processing uses a bufArray rather than a normal Lua array to process multiple packets per DPDK call for performance.
  • Because transmission is asynchronous, packet buffers cannot be reused until DPDK recycles them after the NIC has sent the packets.
  • Packet reception extracts UDP destination ports and maintains counters to measure throughput for different flows.
  • The full repository example adds a timestamping task for latency statistics and can be adapted by changing addresses, ports, and other constants.

5. PERFORMANCE

MoonGen’s LuaJIT-based scripting approach supports efficient packet generation and scales across CPU cores and NIC queues under demanding workloads. In a comparison, it reaches 10 GbE line rate at a lower CPU frequency than Pktgen-DPDK.

  • Implementation considerations: LuaJIT’s direct access to C structs and arrays avoids bounds checks and pointer validation, improving packet-crafting efficiency.The trade-off is that unchecked accesses can cause memory corruption; typical user scripts can corrupt memory by exceeding packet-buffer boundaries or using null buffers.
  • Comparison with Pktgen-DPDK: 1.5 GHz lets MoonGen reach 14.88 Mpps at 10 GbE line rate, while Pktgen-DPDK requires 1.7 GHz.At 1.5 GHz, Pktgen-DPDK achieves 14.12 Mpps in the same minimum-sized UDP-packet scenario.
  • Comparison with Pktgen-DPDK: MoonGen’s tight user script performs only the required packet-buffer allocation, IP modification, and checksum-offloaded transmission.This avoids the broader configuration-handling loop required by Pktgen-DPDK for the tested pattern.
  • Multi-core scaling: Under heavy load, each core generates minimum-sized packets with random payloads, addresses, and ports while producing eight random numbers per packet.Each core sends through two different 10 GbE interfaces simultaneously.
  • Multi-core scaling: Linear scaling is observed as CPU cores increase, until the network line-rate limit is reached.The test uses two transmission interfaces per core and randomizes packet fields without use-case-specific code optimizations.
  • Multi-core scaling: Sending through multiple queues on one NIC scales linearly, supporting the architecture’s multi-core design.The scalability test reduced the CPU clock to 1.2 GHz and used two NICs because the script was initially too fast for meaningful measurements.

5.4 Scaling to 40 Gigabit Ethernet

MoonGen’s evaluation on an Intel XL710 shows packet-size and aggregate-bandwidth limits imposed by first-generation 40 GbE hardware. Synthetic operation benchmarks also quantify packet-generation costs and predict measured throughput closely.

  • Scaling to 40 Gigabit Ethernet: Packets of 128 bytes or less cannot be generated at line rate on the XL710, and more than two CPU cores provide no additional speed.The result identifies the limit as a hardware bottleneck rather than a lack of CPU capacity.
  • Scaling to 40 Gigabit Ethernet: The XL710 reaches 50 Gbit/s with large packets on both ports and 42 Mpps with 64-byte frames.Its two-port aggregate bandwidth is constrained by the chip’s 40 Gbit/s MAC limit, despite a 63 Gbit/s PCIe 3.0 x8 connection.
  • Per-packet costs: The benchmark baseline for packet I/O allocates a packet batch and sends it without modifying contents, revealing substantial DPDK-related per-packet cost.Table 1 reports average per-packet costs for basic I/O and memory-access operations.
  • Per-packet costs: Writing additional data within a 64-byte cache line adds no measurable cost, whereas accessing another cache line in a larger packet is noticeable.Modification operations require the CPU to load constants into the layer 1 cache.
  • Randomizing packets: Random fields cost 17 cycles/pkt each, while wrapping counters cost 1 cycle/pkt.The reported baseline for writing a constant and sending it is 85.1 cycles/pkt, so counters are preferred when the traffic scenario permits them.
  • Performance prediction: 229.2 ± 3.9 cycles/pkt predicts 10.47 ± 0.18 Mpps on one 2.4 GHz core, matching the measured 10.3 Mpps.The calculation combines packet I/O, modification, random-number generation, and IP checksum offloading, supporting hardware-requirement estimates.

5.7 Effects of Packet Sizes

For 64–128-byte packets, MoonGen observes no measurable transmission or reception cost difference on the tested hardware. The stated exception is the XL710, where some packet sizes encounter hardware bottlenecks.

  • Effects of Packet Sizes: Larger packets reduce the number of packets needed at line rate, lowering the relative impact of fixed packet-I/O costs.On 1 and 10 GbE NICs, allocating and sending larger unmodified packets adds no additional MoonGen cost; per-packet modification remains comparatively inexpensive.
  • Effects of Packet Sizes: 64–128-byte packets require the same CPU cycles per packet for transmission, with no measurable packet-size impact on reception.The result concerns the tested MoonGen hardware and packet-size range.
  • Effects of Packet Sizes: The XL710 is an exception: some packet sizes can encounter hardware bottlenecks.This qualification limits the broader packet-size finding to the hardware tested outside the XL710 case.
  • Comparison with prior work: Unlike an earlier netmap evaluation, MoonGen finds no measurable transmission or reception impact from packet sizes near one cache line.The authors attribute the discrepancy likely to differences between current 2014 server hardware and the older 2012 system used for netmap.

6. HARDWARE TIMESTAMPING

MoonGen uses commodity Intel NIC hardware to provide precise latency timestamping across multiple ports, while documenting clock, protocol, packet-size, and throughput limitations. The measurements support high precision and accuracy without special-purpose hardware.

  • Hardware support: Intel 82580, 82599, and X540 NICs can timestamp selected PTP Ethernet or UDP packets, allowing latency measurement for almost any packet type.Only the first payload byte and PTP version byte must match the configured criteria; other PTP fields may be arbitrary.
  • Precision and accuracy: 6.4 ns precision applies to 82599 and X540 NICs at 10 GbE, while the 82580 provides 64 ns precision.The 82599 timestamp timer has 12.8 ns granularity despite 6.4 ns timestamping; the X540 timer increments every 6.4 ns.
  • Precision and accuracy: More than 99.5% of X540 latency measurements fell within ± 6.4 ns of the median, although layer-1 block coding introduced a 64 ns minimum-to-maximum range.On the 82599, an 8.5 m cable produced 345.6 ns in 50.2% of measurements and 358.4 ns in 49.8%.
  • Clock synchronization: Clock synchronization between ports achieves ±1 cycle error, limiting multi-interface accuracy to 19.2 ns on 10 GbE NICs.MoonGen reads both clocks in both orders and repeats the measurement seven times to reject outliers.
  • Clock drift: Clock drift can reach 35 µs per second between a mainboard NIC and a discrete NIC, so MoonGen resynchronizes before timestamped packets.This reduces the relative error from drift to 0.0035% for latency measurements.
  • Limitations: Latency measurement is limited to Ethernet frames with the PTP EtherType or UDP packets, and UDP PTP packets below 80 bytes are not timestamped.Inter-arrival-time measurements are restricted to GbE because 10 GbE NICs lack hardware support for timestamping at line rate.

7. RATE CONTROL

MoonGen combines NIC hardware rate control with software mechanisms to generate controlled timing patterns. Hardware control improves precision and batching for constant-rate traffic, but software control remains necessary for more complex patterns and exhibits limitations at higher rates.

  • Software rate control: Software rate control is error-prone because asynchronous NIC queues prevent software from controlling exactly when packets are retrieved and transmitted.Restricting queues to one packet avoids back-to-back transmission but undermines batch processing at high rates.
  • Hardware rate control: Intel 10 GbE NICs can generate constant bit-rate traffic by controlling each transmit queue’s rate in hardware.Software can keep queues filled while the NIC determines packet timing, resolving the precision and batching problems of software-only control.
  • Evaluation: Hardware-controlled traffic oscillated around the target inter-arrival time by up to 256 ns and almost completely avoided bursts.The comparison used at least 1,000,000 packets and 64 ns histogram bins based on 82580 timestamp precision.
  • Evaluation: Micro-bursts from software rate control produced a low interrupt rate in an Open vSwitch forwarder, likely because driver interrupt moderation triggered earlier than expected.The experiment used increasing loads generated by MoonGen and zsend.
  • Evaluation: Above ∼9 Mpps, corresponding to ∼6 Gbit/s with 64-byte packets, Intel X520 and X540 NICs showed unpredictable non-linear behavior.Using two queues as a workaround does not equal proper single-queue rate control because the queues regulate transmission independently.
  • Limitations: Hardware rate control is restricted to constant-bit-rate traffic, so MoonGen still requires software rate control for other traffic patterns.MoonGen’s software mechanism targets realistic patterns such as those based on a Poisson process.

8. CONTROLLING INTER-PACKET GAPS IN SOFTWARE

MoonGen controls inter-packet gaps in software by transmitting invalid packets whose lengths determine precise timing, enabling arbitrary traffic patterns without additional hardware. The approach affects latency experiments and has boundaries imposed by NIC behavior, link characteristics, and device-under-test assumptions.

  • 8.1 Sending Gaps on the Wire: MoonGen creates arbitrary traffic patterns by replacing idle gaps with invalid packets whose lengths determine the time between valid packets.This keeps NIC queues filled and avoids timing dependent on DMA accesses.
  • 8.1 Sending Gaps on the Wire: The device under test must ignore invalid packets in hardware; MoonGen uses incorrect CRC checksums and, when needed, illegal lengths for short gaps.Tested NICs drop these packets before receive-queue assignment and only increment an error counter.
  • 8.1 Sending Gaps on the Wire: 15.6 Mpps is the maximum packet rate found with short frames on tested Intel NICs, while gaps below 33 wire-bytes cannot be generated.The tested NICs rejected wire lengths below 33 bytes, excluding 1–32-byte gaps, or 0.8–25.6 ns.
  • 8.2 Evaluation: 0.1–1.9 Mpps was the evaluation range for Open vSwitch because higher rates overloaded the device under test and made latency depend on buffer size.Across repeated measurements, hardware and software CBR rate control showed nearly identical forwarding-latency percentiles.
  • 8.3 Example: Poisson Traffic: CBR and Poisson traffic produced the same overall throughput, but traffic patterns measurably affected Open vSwitch latency before overload.At about 1.9 Mpps, buffers filled, causing packet drops and approximately 2 ms latency.
  • 8.4 Limitations of our Approach: The software method works best on 10 GbE, where transmission-time granularity is 0.8 ns, but 1–75-byte gaps remain imprecise under the enforced 76-byte minimum.Unrepresentable gaps are approximated by skipping some invalid packets and lengthening others, preserving average-rate accuracy but reducing precision.
  • 8.4 Limitations of our Approach: The method is optimized for software packet-processing devices; hardware appliances may be affected by invalid packets and may require a store-and-forward switch.The switch must drop invalid CRC frames, and its effect on inter-arrival times requires evaluation.

9. REPRODUCIBLE RESEARCH

MoonGen’s experiments are accompanied by scripts, hardware details, software versions, and test configurations intended to support reproduction of the reported results.

  • 9. Reproducible Research: All experiments can be reproduced with included example scripts and NICs based on Intel 82599, X540, 82580, and XL710 chips.The paper specifically encourages readers to install MoonGen and reproduce its results.
  • 9. Reproducible Research: The evaluations use named scripts for performance, timestamping, clock drift, inter-arrival times, traffic generation, rate control, and latency measurements.The performance tests use an Intel Xeon E5-2620 v3 and Intel X540 NICs, while later tests specify their own scripts and setups.
  • 9. Reproducible Research: Measurements used multiple Intel NIC models for different experiments, and the paper reports the development versions and repository commits used.The 10 GbE and 40 GbE performance evaluations used commits 492c0e4 and a70ca21, respectively.

10. CONCLUSIONS AND FUTURE WORK

MoonGen combines commodity-NIC hardware features with software mechanisms to provide flexible, precise packet generation and measurement. Its scripting architecture supports real-time packet crafting and reception, while future work targets broader NIC support and additional examples.

  • 10. Conclusions: MoonGen is a general-purpose hybrid packet generator combining commodity-NIC hardware features with novel software approaches.This implements functionality previously available only through expensive special-purpose hardware.
  • 10. Conclusions: MoonGen measures latency with sub-microsecond accuracy and precision while controlling packet rates through hardware support and invalid-packet gap filling.The two mechanisms address timestamping and rate-control requirements within the same system.
  • 10. Conclusions: Lua scripts can craft every packet in real time without sacrificing speed, making MoonGen flexible, extensible, and capable of responding to incoming traffic.MoonGen also supports packet reception and analysis for real-time tests.
  • 10. Future Work: MoonGen’s future work includes additional example scripts and support for hardware features of more NICs.Existing examples cover IPv4, IPv6, UDP, TCP, ICMP, IPsec, and ARP traffic.
  • 10. Future Work: The architecture supports further applications including line-rate traffic analysis on 10 GbE networks and Internet-wide scans from 10 GbE uplinks.The latest version is maintained in the public Git repository.
Loading 1410.3322v4…