Source-linked AI summary
The End of Slow Networks: It's Time for a Redesign
Carsten Binnig, Andrew Crotty, Alex Galakatos, Tim Kraska, Erfan Zamanian
TL;DR
Modern distributed DBMSs were designed to avoid communication because networks were assumed to be bottlenecks, but RDMA-capable networks now offer memory-scale bandwidth and improved latency. The paper proposes architectural redesigns and workload-specific algorithms for OLTP and OLAP that use RDMA and Network-Attached Memory. The paper reports that distributed transactions can scale under RDMA-enabled networks and introduces RDMA-aware join processing for the NAM architecture.
Problem
Distributed DBMSs assume the network is the main bottleneck and therefore use communication-avoiding designs that may not fit next-generation RDMA-capable networks.
Method
The paper benchmarks InfiniBand and RDMA, proposes Network-Attached Memory and redesigned distributed DBMS architectures, and develops RDMA-based transaction and join techniques.
Results
Distributed transactions can scale under RDMA-enabled networks, and the RRJ algorithm directly leverages RDMA while enabling scale-out by adding compute servers.
Takeaways & Limitations
Fast RDMA-capable networks motivate replacing communication avoidance as a universal design principle with architectures and algorithms that exploit remote memory access.
Takeaways & Limitations
IPoIB can benefit large-message data-flow operations but may decrease performance for control-flow operations involving many small messages.
Abstract
from arXiv · showhide
Next generation high-performance RDMA-capable networks will require a fundamental rethinking of the design and architecture of modern distributed DBMSs. These systems are commonly designed and optimized under the assumption that the network is the bottleneck: the network is slow and "thin", and thus needs to be avoided as much as possible. Yet this assumption no longer holds true. With InfiniBand FDR 4x, the bandwidth available to transfer data across network is in the same ballpark as the bandwidth of one memory channel, and it increases even further with the most recent EDR standard. Moreover, with the increasing advances of RDMA, the latency improves similarly fast. In this paper, we first argue that the "old" distributed database design is not capable of taking full advantage of the network. Second, we propose architectural redesigns for OLTP, OLAP and advanced analytical frameworks to take better advantage of the improved bandwidth, latency and RDMA capabilities. Finally, for each of the workload categories, we show that remarkable performance improvements can be achieved.
1. INTRODUCTION
Modern distributed DBMSs assume networks are bottlenecks and therefore minimize communication, but RDMA-capable networks can provide memory-scale bandwidth and rapidly improving latency. The paper argues that distributed database architectures should be redesigned to exploit these capabilities.
- Motivation: Current distributed DBMSs minimize machine-to-machine communication because they assume the network is the main bottleneck.Examples include locality-aware partitioning, semi-reductions for joins, and complicated preprocessing.
- Network capabilities: InfiniBand FDR 4× provides network bandwidth in the same ballpark as one memory channel, while future HDR and NDR standards promise substantially more.DDR3 bandwidth ranges from 6.25 GB/s to 16.6 GB/s per channel, whereas InfiniBand reaches 37.5GB/s per NIC port for EDR 12×.
- Network capabilities: A dual-socket DDR3-1600 system with four dual-port FDR 4× NICs has roughly equal aggregate memory and network bandwidth, with full-duplex networking further narrowing the gap.The cited configuration provides 51.2GB/s aggregate memory bandwidth and roughly the same network bandwidth; two NICs could saturate memory bandwidth for read/write workloads.
- Network capabilities: Network transfer in the tested dual-socket system is limited by total memory bandwidth rather than network bandwidth.The comparison uses DDR3-1600 memory and two FDR 4× NICs per socket.
- Network capabilities: RDMA transfers 1KB in approximately 1µs versus approximately 0.08µs for a CPU memory read, and access-time differences nearly disappear at 256KB as bandwidth dominates.The paper does not claim network latency will match memory latency; it expects cache and memory locality to matter more for small requests.
- Architectural redesign: The proposed redesign extends across distributed DBMS architecture, including transactions, storage, execution algorithms, and cache-aware processing.The paper introduces Network-Attached Memory and argues that fast networks make communication avoidance an inadequate universal design principle.
2. BACKGROUND
InfiniBand and RDMA provide communication models that can move data directly between machines with low latency and limited remote CPU involvement. Microbenchmarks show that RDMA outperforms IP-based stacks in throughput, latency, and CPU-overhead behavior, motivating database redesigns around these capabilities.
- InfiniBand and RDMA: RDMA verbs use RNIC processing, with most work executed without operating-system involvement to achieve low latency.The verbs API transfers data through the processing capabilities of the RDMA NIC.
- InfiniBand and RDMA: RDMA one-sided verbs read, write, or atomically modify remote memory without involving the remote CPU.Two-sided SEND and RECEIVE verbs instead support RPC-based communication.
- Micro-benchmarks: Microbenchmarks vary message sizes from 32B to 32MB across IPoEth, IPoIB, and RDMA communication modes to represent OLTP and OLAP workloads.The experiments measure throughput, latency, and RDMA atomic operations in single-threaded tests.
- Micro-benchmarks: 6.8GB/s is reached by all RDMA verbs for messages larger than 2KB, compared with 3.5GB/s maximum for IPoIB.IPoIB uses the same InfiniBand hardware but achieves lower throughput and higher latency than RDMA.
- Micro-benchmarks: 1µs latency is measured for an RDMA WRITE at 8B, compared with 20µs for IPoIB and 30µs for IPoEth.For messages below 256B, RDMA WRITE and SEND take 1µs, while RDMA READ takes 2µs.
- Micro-benchmarks: RDMA client overhead is approximately 450 cycles and independent of message size, while one-sided verbs impose no server-side CPU overhead in the reported measurements.The RNIC handles the actual data transfer as a coprocessor after registering a work queue element.
3. RETHINKING THE ARCHITECTURE
Traditional distributed in-memory DBMSs minimize communication because they assume slow networks, but RDMA-capable networks motivate architectures that directly exploit remote memory access. The proposed network-attached-memory design separates compute from storage and requires redesigns across database-system layers.
- 3.1.1 The Traditional Shared-Nothing Architecture: Traditional shared-nothing systems partition database state across nodes’ local RAM and use socket-based communication for distributed control- and data-flow.
- 3.1.1 The Traditional Shared-Nothing Architecture: Slow-network designs prioritize locality and communication avoidance, yet unavoidable communication imposes performance penalties and network bottlenecks worsen as machines are added.
- 3.1.2 The Shared-Nothing Architecture for IPoIB: IPoIB offers an incremental migration path that benefits large-message data flow, but its CPU overhead for small messages can reduce distributed transaction performance.
- 3.1.3 The Distributed Shared-Memory Architecture: RDMA improves bandwidth utilization, latency, and CPU overhead, but remote access lacks cache coherence and creates independent data copies with implications for memory and consistency management.
- 3.1.4 The Network-Attached-Memory Architecture: Network-attached memory logically decouples compute and storage while exposing a shared distributed memory pool accessed by compute nodes.
- 3.1.4 The Network-Attached-Memory Architecture: NAM enables independent scaling of storage and compute and lets any node access remote partitions, improving handling of data imbalance without prior redistribution.
- 3.2 Challenges and Opportunities: NAM requires redesign across storage, query processing, transaction management, compilation, metadata, optimization, and load balancing to exploit RDMA effectively.
4. THE CASE FOR OLTP
For OLTP, the paper argues that 2PC scalability problems arise mainly from network-message CPU and bandwidth overheads, then presents RSI over NAM to exploit RDMA and improve throughput.
- 4.1.2 Increased Contention Likelihood: 2PC increases transaction latency and contention because network delays dominate short transactions and amplify the bottleneck.Small Ethernet messages average roughly 35µs, while transaction work without disk or network takes 10–60µs.
- 4.1.2 Increased Contention Likelihood: Under the paper’s model, conflict likelihood rises with transaction service time and increases across transactions accessing multiple records.For n records, the modeled likelihood is 1 − (1 − 6λt)^n under similar, independent access rates.
- 4.1.3 CPU Overhead: Message batching and IPoIB improve communication characteristics only partially because processing time and CPU overhead remain bottlenecks without protocol and data-structure redesign.IPoIB increases bandwidth and reduces latency relative to IPoEth but does not reduce CPU overhead and can sometimes worsen it.
- 4.1.3 CPU Overhead: The dominant scalability constraint is message-processing CPU load and network bandwidth rather than the intrinsic complexity of 2PC.With 10Gb Ethernet, three servers, 1KB records, and three updated records per transaction, throughput is limited to approximately 218,500 transactions per second.
- 4.2 RSI: An RDMA-based SI Protocol: RSI redesigns snapshot isolation for NAM by moving transaction logic to compute nodes and using one-sided RDMA while storage nodes provide shared memory.The storage nodes remain unaware of database-specific operations, and clients act as transaction managers accessing data on any storage node.
- 4.3 Experimental Evaluation: 1.8 million distributed transactions per second: RSI substantially outperformed traditional SI over IPoIB and IPoEth and scaled nearly linearly through 60 clients.Traditional SI achieved approximately 22,000 transactions per second over IPoIB and 32,000 over IPoEth; the two-sided RDMA variant reached 1.1 million before declining at higher client counts.
- 4.3 Experimental Evaluation: The paper therefore speculates that distributed transactions may cease to be a scalability limit when network bandwidth matches memory bandwidth, potentially reducing reliance on complex partitioning.Such partitioning can still reduce latency or help manage hot items.
5. THE CASE FOR OLAP
Fast RDMA-capable networks expose limits in shared-nothing OLAP operators designed to minimize communication. The paper therefore redesigns joins and aggregation for the NAM architecture, using RDMA and fine-grained parallelism to improve performance, scale-out, and robustness.
- Distributed joins: Distributed joins dominate network-intensive OLAP workloads, but shared-nothing algorithms were designed around slow networks and communication avoidance.These algorithms combine local joins, partitioning, and optional reduction techniques; the paper argues they are often suboptimal for in-memory databases over fast RDMA networks.
- Distributed joins: Network traffic can comprise up to 90% of join runtime, motivating algorithms that reduce communication through extra computation or communication round-trips.The paper argues that this trade-off changes on fast networks because memory accesses and network transfers have different relative costs.
- RDMA-optimized joins: RDMA GHJ writes partitions directly into remote storage memory, overlaps transfers with computation through selective signaling, and enables data-parallel join execution.Its partitioning cost reduces to Tpart = Tmem(R) + Tmem(S) because RNICs execute remote writes in the background.
- RDMA-optimized joins: RRJ extends an in-memory radix join for RDMA and NAM, enabling efficient scale-out by adding compute servers rather than adapting a shared-nothing join.Experiments show RRJ significantly outperforms the other evaluated join algorithms across different semi-join selectivities.
- RDMA-optimized joins: Semi-join reduction pays off only in corner cases with RDMA GHJ and RRJ, typically at very low join selectivities.For IPoIB, it pays off only when Bloom-filter selectivity is below sel < 0.8, accounting for the stated false-positive behavior.
- RDMA aggregation: The RDMA aggregation operator pre-aggregates in cache-sized hash tables, copies overflow partitions remotely in the background, and post-aggregates partitions in parallel.This fine-grained design is more robust to data skew and varying numbers of distinct group-by keys, with major gains for high key counts.
6. RELATED WORK
Prior systems often add RDMA to existing designs or limit its role to storage and coordination, while newer work develops RDMA-aware joins and transaction processing. The paper’s approach instead treats RDMA as a first-class architectural concern for distributed DBMSs.
- HPC techniques for fast networks provide relevant experience for next-generation distributed DBMS development, despite often targeting specific applications.
- Many existing systems add RDMA after the fact rather than designing transaction processing and query execution around the network.Examples include Oracle RAC, RDMA-aware systems built on key/value stores, and systems separating storage from compute.
- IBM pureScale uses RDMA for active-active DB2 scaleout but relies on a centralized manager for distributed-transaction coordination.
- The proposed RDMA ideas build on established distributed-transaction and distributed-join research, while the paper notes limited prior work on RDMA-enabled transaction protocols.
- Earlier RDMA join work assumed limited bandwidth and streamed one relation across nodes, whereas RRJ extends in-memory join algorithms for RDMA.The paper states that RRJ is significantly more efficient than the cited earlier RDMA join.
7. CONCLUSION
The paper argues that emerging fast networks require distributed DBMSs to be fundamentally rethought. Initial OLTP and OLAP experiments indicate the potential of fully leveraging the network, while important design challenges remain.
- Emerging fast network technologies necessitate a fundamental rethinking of distributed DBMS construction.
- Initial OLTP and OLAP experiments indicate the potential of fully leveraging the network.
- Open challenges include balancing local versus remote processing and creating abstractions that hide RDMA verbs’ complexity.