PaperlayerView exact revision on arXiv

Source-linked AI summary

Datacenter RPCs can be General and Fast

Anuj Kalia, Michael Kaminsky, David G. Andersen

arXiv:1806.00680v2cs.OS

TL;DR

Datacenter networking has often traded generality for performance through specialized hardware and software. This paper presents eRPC, a general-purpose RPC system that supports lossy networks and broad functionality while achieving state-of-the-art performance comparable to specialized systems.

  • Problem

    Datacenter networking software often sacrifices performance or generality, motivating specialized systems built around RDMA, lossless networks, FPGAs, and programmable switches.

  • Method

    eRPC implements a general-purpose RPC API over unreliable packet I/O using end-host mechanisms, a userspace NIC driver, and a network transport with congestion control.

  • Results

    eRPC provides high small-message rates, high large-message bandwidth, and scalability across nodes and CPU cores while handling packet loss, failures, congestion, and long-running requests.

  • Takeaways & Limitations

    General-purpose RPCs can achieve state-of-the-art datacenter networking performance without lossless fabrics or additional network support.

  • Takeaways & Limitations

    Existing fully general networking stacks can leave substantial performance on the table, especially for small messages.

Abstract

from arXiv · show

It is commonly believed that datacenter networking software must sacrifice generality to attain high performance. The popularity of specialized distributed systems designed specifically for niche technologies such as RDMA, lossless networks, FPGAs, and programmable switches testifies to this belief. In this paper, we show that such specialization is not necessary. eRPC is a new general-purpose remote procedure call (RPC) library that offers performance comparable to specialized systems, while running on commodity CPUs in traditional datacenter networks based on either lossy Ethernet or lossless fabrics. eRPC performs well in three key metrics: message rate for small messages; bandwidth for large messages; and scalability to a large number of nodes and CPU cores. It handles packet loss, congestion, and background request execution. In microbenchmarks, one CPU core can handle up to 10 million small RPCs per second, or send large messages at 75 Gbps. We port a production-grade implementation of Raft state machine replication to eRPC without modifying the core Raft source code. We achieve 5.5 microseconds of replication latency on lossy Ethernet, which is faster than or comparable to specialized replication systems that use programmable switches, FPGAs, or RDMA.

1 Introduction

eRPC challenges the trade-off between generality and performance in datacenter networking by providing a general-purpose RPC library that operates efficiently on commodity networks. It supports lossy Ethernet, large-scale deployments, and integration with existing software.

  • Conclusion: eRPC is presented as a general-purpose alternative to specialized networking systems and additional in-network functionality.The paper reports that it can achieve state-of-the-art performance without additional network support.
  • Capabilities: eRPC provides high small-message rate, high large-message bandwidth, and scalability across nodes and CPU cores on lossy Ethernet or lossless fabrics.It handles packet loss, node failures, congestion control, and background requests while supporting unmodified systems.
  • Motivation: Existing datacenter networking options force a choice between fast but limited interfaces and general but slower networking stacks.DPDK provides unreliable packet I/O, while mTCP and IX leave performance on the table, especially for small messages.
  • Approach: eRPC targets common-case RPC workloads while handling large messages, congestion, and long-running handlers through more expensive paths.Its API, message format, and wire protocol prioritize small messages, congestion-free networks, and short handlers.
  • Results: 2.3 µs median RPC latency and 10 million RPCs per second with one core demonstrate eRPC’s datacenter performance.The same evaluation reports 75 Gbps large-message transfer with one core and 20,000 connections per node.

2 Background and motivation

The paper motivates eRPC by the performance and deployment costs of specialized networking, while arguing that software can provide comparable performance on commodity networks. Specialization can complicate reuse, implementation, and deployment without always improving performance.

  • Lossy networking: eRPC restricts each flow to one BDP of outstanding data to prevent most packet drops on lossy networks.The paper notes that datacenter switch buffers are substantially larger than the BDP, supporting this design choice.
  • Lossless networking: PFC prevents congestion-based packet drops but introduces head-of-line blocking, deadlocks, and complex switch configuration.These issues make operators reluctant to deploy PFC, while RDMA systems may be unusable without it.
  • Existing software: Fully general stacks leave performance on the table: IX handles around 1.5 million 64 B RPC requests per second per core, versus 10 million with eRPC.The comparison specifically concerns small-message RPC throughput.
  • Existing software: FaSST requires a lossless fabric and omits packet-loss, congestion, large-message, long-handler, and node-failure handling.The paper claims that eRPC supports these features while matching FaSST’s performance on lossy networks.
  • Specialization: Specialized systems use RDMA, lossless fabrics, FPGAs, or programmable switches to pursue high performance, but they increase design and deployment complexity.They can break abstraction boundaries, prevent reuse, and require custom hardware or operator control of shared switches.
  • Performance trade-offs: Specialization does not always improve performance because one RPC can replace multiple remote memory accesses required by RDMA-based designs.The paper also reports comparable RPC performance with switch- and FPGA-based replication systems.

3 eRPC overview

eRPC builds RPCs over unreliable packet I/O with userspace event-driven execution, combining dispatch-thread and worker-thread handlers. Its evaluation spans lossy Ethernet, lossless InfiniBand, and multiple NIC generations.

  • Transport: eRPC implements RPCs over unreliable packet I/O such as UDP or InfiniBand Unreliable Datagram transport.A userspace NIC driver supports its performance, while the library supplies end-host mechanisms and transport features.
  • RPC API: RPCs execute at most once and asynchronously, with event loops providing intra-thread concurrency and continuations reporting completion.Messages reside in DMA-capable opaque buffers called msgbufs.
  • RPC API: Each sending or receiving user thread owns an exclusive Rpc endpoint containing packet queues, an event loop, and sessions.A session is a one-to-one connection between two Rpc endpoints, and an endpoint can participate in multiple sessions.
  • Threading: Dispatch-thread event loops perform packet I/O, congestion control, management, and request-handler or continuation execution.Long-running handlers can be dispatched to worker threads.
  • Threading: eRPC lets programmers assign short handlers to dispatch threads and longer handlers to worker threads.This avoids the inter-thread communication cost of sending every request from a network-I/O thread to a worker.
  • Evaluation: The evaluation covers lossy Ethernet and lossless InfiniBand across three NIC generations, using traditional UDP on Ethernet.The primary CX4 evaluation uses 100 nodes in a shared CloudLab cluster.

4 eRPC design

eRPC’s design uses userspace packet I/O, CPU-managed state, zero-copy buffers, and NIC features to scale across nodes while preserving high performance.

  • Packet I/O and scalability: eRPC uses userspace packet I/O instead of RDMA writes, avoiding NIC connection-state scalability limits.RDMA connection state consumes limited NIC SRAM, while packet I/O enables constant-time completion-queue checks.
  • Packet I/O and scalability: ConnectX-5 NICs avoid receive-queue thrashing with 28 64K-entry queues, unlike Connect-IB’s limit of 14 2K-entry queues.The improvement comes from more intelligent prefetching and caching rather than a proportional increase in NIC cache.
  • Packet I/O and scalability: Constant NIC memory per CPU core is independent of cluster node count through novel use of multi-packet receive descriptors.This design can simplify future NICs, although current NICs already support very large receive queues.
  • Packet I/O and scalability: 20,000 connections preserve peak eRPC throughput, whereas RDMA loses approximately 50% throughput with 5,000 connections.The RDMA decline results from NIC cache misses and expensive DMA reads.
  • Zero-copy transmission: eRPC’s zero-copy msgbuf layout keeps application data contiguous while placing the first packet’s header and data together for one DMA read.Subsequent packets use separate header and data DMAs to preserve contiguous application data.
  • Zero-copy transmission: Unsignaled transmission avoids completion-queue overhead, which would reduce message rates by up to 25%.eRPC must also preserve msgbuf ownership across retransmissions and failures before applications reuse buffers.

4.3 Sessions

eRPC sessions keep multiple requests in flight, regulate packets with credits, and balance scalability against per-session throughput.

  • Outstanding requests: Eight concurrent requests per session are supported by default, with additional requests transparently queued.Out-of-order completion prevents long-running worker-mode RPCs from blocking dispatch-mode RPCs.
  • Session optimization: Preallocated MTU-size response buffers improve eRPC’s message rate by up to 13%.eRPC selects them automatically according to the handler’s desired response size, avoiding dynamic allocation overhead.
  • Credit-based flow control: A session’s credit quota limits unacknowledged packets and allows receive-queue descriptors to be replenished.Credits also provide end-to-end flow control and can reduce switch queueing.
  • Session scalability: An RPC can participate in up to |RQ|/C sessions, counting both server-mode and client-mode sessions.Session creation fails after this limit is reached.
  • Credit-based flow control: Lower credit counts improve scalability but reduce session throughput by restricting the packet window.C = 1 suits low-latency, small-message or many-session workloads, while BDP/MTU credits suit individually high-throughput sessions.
  • Session scalability: Up to 20,000 sessions with 32 credits per session are supported on ConnectX-4 NICs.Large receive queues enable this scale, but each receive-queue entry consumes host-memory packet-buffer space.

5 Wire protocol

eRPC’s client-driven protocol minimizes coordination while supporting credits, retransmissions, and software congestion control on lossy networks.

  • Wire protocol: A client-driven protocol keeps rollback state only at clients, reducing retransmission coordination and shifting rate-limiting overhead away from servers.Server packets are sent in response to client packets.
  • Wire protocol: With 2 credits, single-packet RPCs use the fewest possible packets, while multi-packet exchanges use credit returns and request-for-response packets.The first response implicitly returns the credit consumed by the final request packet.
  • Wire protocol: Multi-packet responses incur up to one additional RTT because clients trigger subsequent response packets.In practice, the added latency is less than 20% for responses with four or more packets.
  • Congestion control: eRPC implements Timely entirely at client session endpoints because its test clusters lack ECN marking for DCQCN.Server-only RPC endpoints incur no congestion-control overhead.
  • Congestion control: Three common-case optimizations reduce congestion-control overhead from 20% to 9% without reducing effectiveness.They bypass unnecessary updates and rate limiting for uncongested sessions and batch timestamp sampling.
  • Loss handling: eRPC omits selective acknowledgments because packet losses are rare with dynamic switch buffers, making the transport dependent on that buffering assumption.Implementing SACK could reduce this dependence.
  • Loss handling: eRPC uses a conservative 5 ms retransmission timeout because dynamic buffers can add up to 3.8 ms of queueing delay.The smaller-buffer IRN setting permits an approximately 300 µs timeout.
  • Loss handling: One-BDP flow control prevents most packet drops on lossy networks, while eRPC uses go-back-N recovery and at-most-once request execution.Reordered packets are treated as losses and dropped.

6 Microbenchmarks

eRPC delivers high message rates, bandwidth, scalability, and low latency across commodity datacenter networks, including lossy Ethernet. Its performance remains close to specialized systems while supporting general networking features such as congestion control and packet-loss handling.

  • Small RPC rate: 5% lower than FaSST: eRPC’s per-thread request issue rate is only 5% below the specialized system at B = 3.Across all batch sizes, eRPC is at most 18% lower than FaSST.
  • Small RPC rate: 10 million RPCs per second: each CX4 thread processes this rate at B = 3 while issuing 5 million requests per second.The symmetric experiment means each thread also handles 5 million incoming requests per second.
  • Small RPC rate: 40% lower throughput: disabling preallocated responses and zero-copy request processing reduces eRPC’s rate to 3 Mrps from its peak.Disabling all three congestion-control optimizations alone increases congestion-control overhead from 9% to 20%.
  • Session scalability: 12.3 Mrps per node: with 10 threads per node across 100 CX4 nodes, eRPC sends and receives 18.1 Gbps.The achieved bandwidth is close to the link’s 23 Gbps achievable bandwidth out of 25 Gbps.
  • Large RPC bandwidth: 75 Gbps: eRPC reaches this large-transfer throughput with one core and at least 70% of RDMA write throughput for requests of 32 kB or larger.The large-transfer experiment uses a 100 Gbps InfiniBand switch and ConnectX-5 NICs.
  • Congestion control: Over 5x lower median queuing: congestion control handles target workloads up to 50-way incasts, while 99.99th-percentile latency remains below 700 µs with 10 threads per node.For 100-way incasts, median queueing improves 3x, but 99th-percentile queueing is not substantially reduced.

7 Full-system benchmarks

The full-system benchmarks test eRPC with unmodified Raft and Masstree software, showing low-latency replication and database queries while supporting background scans.

  • Raft state machine replication: eRPC ports LibRaft without modifying its source code by implementing its RPC callbacks.LibRaft is an existing, well-tested Raft implementation used as-is.
  • Raft state machine replication: 850 ns of eRPC software-networking latency is around 2x the 400 ns latency added by current programmable switches.This challenges NetChain’s assumption that software networking adds one to two orders of magnitude more latency than switches.
  • Raft state machine replication: 5.5 µs Raft replication latency is lower than NetChain’s 9.7 µs.The comparison has caveats because the systems use different NICs and NetChain has several limitations.
  • Raft state machine replication: eRPC’s median leader commit latency is only 3% worse than ZabFPGA’s direct-connect comparison.The comparison uses ZabFPGA’s leader commit latency, including a mode where FPGAs communicate over point-to-point links.
  • Masstree database index: 14.3 million GETs/s with 12 µs 99th-percentile latency is achieved on CX3 while Masstree runs scans in worker threads.Without worker threads, 99th-percentile GET latency rises to 26 µs; median low-load latency is 2.7 µs.

8 Related work

Related work contrasts eRPC with earlier RPC designs and specialized network co-designs, emphasizing eRPC’s combination of performance and generality.

  • RPCs: Earlier RPC systems optimized wire protocols for small messages, while others used RDMA reads or writes for bulk data transfer.eRPC instead transfers all data over datagram messaging to avoid scalability problems.
  • Co-design: Network co-design projects span key-value stores, databases, transaction processing, state machine replication, and graph processing.The paper positions eRPC as a way to study performance without sacrificing the networking abstraction.
  • Co-design: Unlike RAMCloud and FaSST, which lack either performance or generality, eRPC provides both.The comparison frames eRPC as an RPC alternative to co-designed systems.

9 Conclusion

The conclusion presents eRPC as a fast, general-purpose alternative to specialized network hardware, achieving near-hardware performance for unmodified applications.

  • Conclusion: eRPC’s speed comes from common-case optimization, combined optimizations, and exploiting switch buffers larger than datacenter BDP.These design choices support performance on commodity networks without additional network support.
  • Conclusion: eRPC delivers performance previously associated with lossless RDMA fabrics or specialized network hardware.It is presented as an alternative to putting more functions in network hardware.
  • Conclusion: Ported LibRaft and Masstree versions are reported as the fastest replicated key-value store and networked database index in the academic literature.They operate end-to-end without additional network support.

Appendix A. eRPC’s NIC memory footprint

eRPC keeps its NIC memory footprint scalable through fixed-size queues, compact receive descriptors, and an overrunning receive completion queue.

  • NIC memory footprint: 64-entry transmit and transmit-completion queues keep their NIC footprint independent of cluster size.The queues provide sufficient pipelining to hide PCIe latency in the reported configuration.
  • NIC memory footprint: 512-way multi-packet receive descriptors reduce receive-queue size by 512x.Traditional receive-queue footprint grows with connected sessions, whereas these descriptors specify multiple contiguous packet buffers.
  • NIC memory footprint: eRPC shrinks the receive completion queue by allowing NIC entries to be overwritten in round-robin fashion.The implementation needs only the number of newly received packets, not the information DMA-written into each entry.

Appendix B. Handling node failures

eRPC detects suspected remote-node failures through session-management timeouts and coordinates cleanup across dispatch threads before continuing session-specific handling.

  • Session management uses sockets-based messaging to create and destroy sessions.
  • Timeouts allow the management thread to detect suspected failures of remote nodes.
  • After a suspected failure, dispatch threads flush NIC transmit DMA queues to release references held by the NIC.

Appendix C. Rate limiting with zero-copy

eRPC handles retransmission references in its rate limiter by avoiding complex deletions, while NIC DMA-queue references require flushing because unsignaled transmission obscures queue state.

  • Deleting retransmitted-request references from the rate limiter is complex because Carousel and shared Timely state complicate rollback.Carousel requires bounded scheduling-time differences, while each Timely instance is shared by all slots in a session.
  • eRPC drops responses received while a retransmitted request remains in the rate limiter.These responses signal false positives in retransmission detection and are therefore rare.
  • NIC DMA-queue references cannot generally be removed by software inspection because eRPC uses unsignaled transmission, so the queue must be flushed.
Loading 1806.00680v2…