Source-linked AI summary
SlimTCP: It's fast, but not because it's slim
Mihai Drosi Caju, Costin Raiciu
TL;DR
The paper asks whether simplifying TCP/IP stacks improves performance in data-center networks. It develops SlimTCP by removing mechanisms unnecessary over reliable, ordered channels and evaluates it against other stacks. SlimTCP achieves the strongest tested throughput in several scenarios, but the paper concludes that simplicity alone does not clearly explain the gains and that the resulting stack has limited practical use.
Problem
The paper examines whether simplifying a TCP/IP stack improves performance for data-center networks.
Method
SlimTCP removes TCP features associated with packet loss or reordering and evaluates the resulting user-space stack using controlled data-center-oriented benchmarks.
Results
SlimTCP paired with SlimTCP reaches 23.7G goodput for one connection and 23.6G across 4096 connections, while out-competing f-stack and mTCP in most tested scenarios.
Takeaways & Limitations
The paper concludes that the performance benefit of a simpler TCP/IP stack remains ambiguous and reinforces that quality work contributes substantially to software performance.
Takeaways & Limitations
SlimTCP has limited practical use because it lacks feature parity and code maturity and is difficult to implement.
Abstract
from arXiv · showhide
In this paper, the authors explore the possibility of improving the performance of TCP/IP stacks in the context of data-center networks. This paper will focus particularly on the claim that simplifying the code-base of the stack increases its performance.
I. INTRODUCTION
SlimTCP targets data-center TCP/IP performance when the underlying channel already provides reliable, in-order delivery. It removes fast-path features made unnecessary by that deployment scenario.
- SlimTCP is a TCP/IP stack geared toward data-center use, implementing the bare minimum required for TCP and UDP connections.
- The paper focuses on Ultra Ethernet implemented inside a NIC, where Reliable Ordered Delivery provides a reliable-in-order Ethernet channel.
- The proposed optimization strips features from the stack’s fast path that are no longer needed under the assumed channel conditions.
II. DESIGN
SlimTCP removes TCP mechanisms that assume packet loss or reordering, retaining only selected extensions needed for its target environments and experiments.
- SlimTCP removes extensions that assume packet loss or reordering, including SACK, PAWS, timestamps, congestion control, fast retransmit, and reorder buffers.
- RTO is retained because SlimTCP was originally developed to run on EQDS, which offered imperfect packet delivery.
- Window scaling remains to improve performance at high BDPs, while MSS remains to test performance across segment sizes.
III. API
The API preserves a POSIX-like control interface while adding an opt-in zero-copy data path for applications that need direct packet-buffer access.
- The control path follows the POSIX socket API with equivalent functions for socket lifecycle operations.
- SlimTCP is internally called ndpip, and applications must voluntarily use its prefixed functions rather than transparently replacing POSIX calls.
- The alternative zero-copy API lets applications write directly to NIC transmit buffers and receive buffers directly from the NIC receive ring.
- Using the zero-copy transmit path, applications allocate and prepare packet buffers before send either transfers ownership to the NIC or returns buffers to the application.
- The API also provides receive, buffer-management, and epoll-equivalent operations for packet consumption and asynchronous socket waiting.
IV. ARCHITECTURE
SlimTCP runs entirely in user space over DPDK and divides application-facing work from packet-processing work across two cores. Its architecture avoids several hardware and kernel-assisted mechanisms to isolate the stack’s performance.
- The stack uses DPDK as its networking driver and runs entirely in user space to avoid the complexity of writing kernel code.
- One core handles application data, socket lifecycle, and NIC enqueueing, while a second worker core processes incoming packets, timers, and protocol replies.
- TSO, LRO, and RSS are disabled so packet transmission and reception remain exclusively single-core with one processed buffer per on-wire data unit.
- Lock-less single-producer single-consumer ring buffers pass buffers between the stack and application and support TCP retransmission and receive processing.
- The epoll implementation uses busy waiting to simplify synchronization and reduce kernel invocation on the receive path.
V. OPTIMIZATIONS
SlimTCP optimizes processing by batching acknowledgments and received packets, reducing locking and memory-access overhead. These choices target efficient handling of bursts and improved cache use.
- ACK handling: Pure ACKs are sent once per receive burst per socket, with the same delay used to free the retransmission ring.This batching has a similar effect to LRO.
- Locking: SlimTCP buffers received packets per socket before delivery, allowing one lock acquisition per burst and avoiding unnecessary bounds checks.The fixed maximum burst size makes bounds checking unnecessary; this optimization applies to TCP because UDP lacks equivalent receive-side state.
- Memory access: Pointer dereferencing was minimized to improve cache use because memory-intensive instructions were especially costly on cache misses.
VI. EVALUATION
The evaluation compares SlimTCP, F-Stack, and mTCP on a controlled two-host bare-metal testbed. It measures packet rate and goodput across segment sizes and connection counts using matched benchmark conditions.
- Testbed: The testbed used two 10-core, 32GiB hosts with Broadcom StingRay PS225 NICs connected through a switch, with one port limited by CPU packet-processing capacity.The NICs supported up to 25Gb/s and 68Mpps per port.
- Metrics and workloads: Evaluation measured achieved packet rate and goodput while varying transmitted segment sizes and the number of connections.
- Compared stacks: The study benchmarked SlimTCP, F-Stack, and mTCP after patching receive checksum verification and features including LRO, TSO, RSS, and timestamps for consistency.mTCP was also adjusted for matching core affinity and configurable MSS.
- Benchmark implementation: Three stack-specific iperf-like programs generated TCP traffic, while servers measured goodput and, in the second experiment, polled events across multiple sockets.
- Reproducibility: All stacks used GCC 9 with -g -O3 -march=sandybridge, were deployed through Nix, and retained separate DPDK versions because a common version was infeasible.
- Benchmark design: The experiments varied segment size with one connection and varied connection count at a fixed 1400 MSS.
- Benchmark design: Each transmitter-receiver combination was tested, producing nine stack-and-protocol tuples.
VII. RESULTS
SlimTCP paired with SlimTCP delivered the strongest results for larger segments and was the most scalable pairing as connection counts increased. For smaller segments, mTCP sending to SlimTCP was faster, while some mixed pairings triggered retransmission timeouts.
- Segment-size benchmark: 23.6Gb/s goodput was achieved by SlimTCP sending to and receiving from SlimTCP for MSS values above or equal to 1024.This was the only tested tuple to saturate the link.
- Segment-size benchmark: mTCP sending to SlimTCP was faster than the other tuples for smaller MSS values.
- Connection-scaling benchmark: SlimTCP paired with SlimTCP was the most scalable tuple in the benchmark varying the number of connections.
- Connection-scaling benchmark: SlimTCP sending to other stacks sometimes triggered RTO, possibly because the sender was too fast and overflowed the NIC receive rings.
VIII. DISCUSSION
SlimTCP outperforms the comparison stacks in several tested conditions, but the evaluation does not establish that reduced code complexity caused the gains. The discussion identifies implementation limitations, deployment constraints, and unresolved fairness concerns.
- Limitations: SlimTCP currently requires applications to consume packet buffers quickly enough to avoid full receive rings or depleted memory pools because receive-window scaling is absent.This constrains operation under the stated buffer-management conditions.
- Interpretation: The hypothesis that simpler code improves performance remains unvalidated because bugs, optimizations, and non-conformity may have influenced performance and scalability.The authors also state that performance cannot be clearly correlated with code complexity when implementation quality and other factors vary.
- Implementation effects: SlimTCP's sub-1024-segment sender performance drops because application and worker threads contend over the socket-structure lock during frequent ACK handling.The discussion suggests moving the sender to the worker thread, while noting uncertain real-world effects.
- Evaluation outcomes: 23.7G goodput was achieved by SlimTCP paired with SlimTCP at 1460B MSS, while the same pair sustained 23.7G across one to 2048 connections.The segment-size and connection-scaling tables report these results separately.
- Limitations: SlimTCP's adoption is close to impossible with monolithic modern operating-system kernels, despite competing with or outperforming F-Stack and mTCP in some conditions.The stack also lacks Fast Retransmit, SACK, and congestion control, making a proxying middle-box a proposed alternative for Internet connections.
IX. CONCLUSION
The claim that simplifying TCP/IP stacks improves performance remains ambiguous under sane software-engineering premises. Although SlimTCP is faster in laboratory and possible future data-center conditions, limited feature parity, maturity, and implementation difficulty constrain its usefulness, while the paper reinforces software quality’s importance.
- The performance benefit of a simpler protocol or TCP/IP stack remains ambiguous under sane software-engineering premises.
- SlimTCP is faster under laboratory and possibly future data-center conditions, but its limited feature parity, code maturity, and implementation difficulty make it of limited use.
- The paper reinforces the common wisdom that quality work is a major contributor to software performance.
- Source code for the performance benchmark and SlimTCP is available in separate GitHub repositories.
X. FUTURE WORK
Future work should validate the stack through real-world applications and examine thread-placement choices under bidirectional traffic. The authors also identify porting SlimTCP to additional TCP applications and rewriting it in Rust as possible directions.
- Benchmarking real-world applications on top of SlimTCP would provide better validation of the stack’s performance.
- Application-level validation would include bidirectional traffic and enable investigation of moving sender functions to the receiver thread.
- Future directions include porting SlimTCP to in-hypervisor TCP accelerators or unikernel TCP stacks and undertaking a Rust rewrite.