Source-linked AI summary
The End of a Myth: Distributed Transactions Can Scale
Erfan Zamanian, Carsten Binnig, Tim Kraska, Tim Harris
TL;DR
Distributed transactions are commonly considered non-scalable because of network and architecture constraints, motivating a redesign that makes locality less central. NAM-DB combines RDMA-enabled networking with scalable Snapshot Isolation mechanisms and reports linear scale-out to 6.5 million TPC-C transactions per second on 56 machines.
Problem
Distributed transactions are widely considered unscalable, and existing approaches often require application developers to design around data locality and co-partitioning.
Method
NAM-DB redesigns distributed transaction processing around RDMA, distributed Snapshot Isolation algorithms, and a scalable global counter, treating locality as an optimization.
Results
6.5 million total TPC-C transactions per second were achieved with nearly perfect linear scale-out to 56 machines.
Takeaways & Limitations
Distributed transactions can scale when the database architecture and transaction protocols are redesigned for RDMA, assuming the workload itself is scalable.
Takeaways & Limitations
Workloads with a single serialization point remain inherently unscalable, and NAM-DB does not currently handle network-partition failures.
Abstract
from arXiv · showhide
The common wisdom is that distributed transactions do not scale. But what if distributed transactions could be made scalable using the next generation of networks and a redesign of distributed databases? There would be no need for developers anymore to worry about co-partitioning schemes to achieve decent performance. Application development would become easier as data placement would no longer determine how scalable an application is. Hardware provisioning would be simplified as the system administrator can expect a linear scale-out when adding more machines rather than some complex sub-linear function, which is highly application specific. In this paper, we present the design of our novel scalable database system NAM-DB and show that distributed transactions with the very common Snapshot Isolation guarantee can indeed scale using the next generation of RDMA-enabled network technology without any inherent bottlenecks. Our experiments with the TPC-C benchmark show that our system scales linearly to over 6.5 million new-order (14.5 million total) distributed transactions per second on 56 machines.
1 Introduction
NAM-DB challenges the view that distributed transactions cannot scale by combining RDMA-enabled networking with a redesigned database architecture. The paper targets predictable linear scale-out while making co-partitioning an optimization rather than a prerequisite.
- Motivation: Distributed transactions are widely viewed as unscalable, leading systems to adopt locality-aware partitioning, speculative execution, weaker consistency, or relaxed durability.These techniques often require developers to understand their implications and design applications around them.
- Motivation: Under the proposed design, co-partitioning becomes a secondary optimization for selected queries rather than a necessity for scalable execution.The paper compares this role to indexing for a selected class of queries.
- System Redesign: RDMA can remove dominant network and CPU limits, but achieving scalability also requires redesigning database components, data structures, and transaction protocols.Keeping a traditional architecture while moving to a high-bandwidth network can even reduce performance.
- Why Distributed Transactions Are Considered Not Scalable: CPU overhead from TCP/IP processing and limited network bandwidth are identified as major scalability constraints, while contention is often only a side effect.High latency can produce timeouts and increased abort rates, further reducing throughput.
- Contributions: NAM-DB presents scalable Snapshot Isolation algorithms, a scalable global counter, and a full TPC-C evaluation using RDMA-based distributed transactions.The contribution includes support for varied distribution and contention conditions.
2 System Overview
NAM uses RDMA to separate transaction execution from database state, enabling compute servers to access shared distributed memory directly. Its design emphasizes one-sided operations, data-location independence, scalable data structures, and scale-out of memory bandwidth.
- RDMA-Based Execution: RDMA one-sided operations reduce CPU involvement in remote data access compared with traditional message-based communication.The design uses byte-level memory access so transaction computation can be distributed across compute servers.
- Design Principles: NAM-DB builds on the NAM architecture to provide a scalable transactional system without an inherent bottleneck beyond the workload itself.The system overview frames RDMA-aware redesign as necessary for this goal.
- NAM Architecture: The NAM architecture logically separates compute and memory servers, with RDMA connecting compute servers to a shared distributed memory pool.Transactions execute on compute servers while database state resides on memory servers.
- Scalability: Scaling memory servers can increase aggregate memory bandwidth when memory bandwidth is the dominant bottleneck.The architecture is designed so the network switch does not inherently limit simultaneous full-duplex transfers.
- Data Location Independence: NAM treats locality as an optimization because every compute server can access data regardless of its storage location.This also supports moving data and work stealing for distributed load balancing.
- Partitionable Data Structures: Every externalized data structure must be partitionable to prevent shared regions such as global timestamps from becoming bottlenecks.This principle motivates the paper’s decentralized partitionable timestamp design.
3 The Basic SI-Protocol
The basic Snapshot Isolation protocol uses RDMA to obtain snapshots, execute transactions, validate and lock write sets, install updates, and record outcomes. Its naïve global timestamp design introduces scalability, abort-rate, and fault-tolerance problems.
- Snapshot Isolation reads a committed database snapshot and aborts only when an updated item was written since the transaction began.
- The protocol fetches a read timestamp, executes the transaction remotely, builds read and write sets, then enters the commit phase.
- A compute server obtains a unique commit timestamp with RDMA fetch-and-add, then validates and locks every record in its write set using compare-and-swap.
- Successful transactions install their write sets with RDMA writes, while failed transactions reset their locks.
- 3.2 Open Problems and Challenges: The naïve protocol scales poorly because concurrent fetch-and-add operations target one memory location and timestamp management becomes costly at hundreds of thousands of transactions per second.
- 3.2 Open Problems and Challenges: Stale snapshots, slow workers, compute-server failures, and simplifying assumptions about durability and recovery create additional abort, progress, and correctness challenges.
4 Timestamp Oracle
The scalable timestamp oracle replaces a single global counter with a timestamp vector whose components are owned by transaction execution threads. This removes several synchronization bottlenecks, while partitioning the vector trades bandwidth improvements for strict monotonicity.
- The timestamp oracle’s main bottlenecks are its management thread, slow workers blocking read-timestamp advancement, and synchronization on one shared commit timestamp.
- NAM-DB represents the read timestamp with a timestamp vector similar to a vector clock.
- Each vector component is a unique counter assigned to a globally identified transaction execution thread, while records store only the latest updating thread’s timestamp.
- A thread can create commit timestamps without communication by incrementing its own latest counter, and it publishes visibility with an RDMA write rather than an atomic operation.
- Long-running transactions, stragglers, and crashed machines no longer prevent read-timestamp advancement when transaction threads operate independently.
- 4.2 Further Optimizations: Prefetching, compressing, and partitioning the timestamp vector reduce network load or improve bandwidth, but partitioning may remove strict global monotonicity.
5 Memory Servers
NAM-DB memory servers use multi-version records, RDMA-accessible table and index structures, and memory-management mechanisms designed for remote access. The design prioritizes scalable distributed transactions over locality.
- NAM-DB stores the current record version in a dedicated region and moves prior versions into an old-version buffer when updates install new current versions.
- Record Layout: Each record has a metadata header and a payload data section, with fixed-length payloads supported in the current implementation.
- Record Layout: The eight-byte header encodes the installing thread identifier and commit timestamp, enabling version identification and remote compare-and-swap updates.
- Version Management: Separate circular buffers for headers and data let transactions identify versions by fetching headers before retrieving the matching payload.
- Version Management: Installing a version combines validation and locking in one RDMA compare-and-swap, then copies the current header and data into the old-version buffers.
- NAM-DB provides partitioned hash tables, secondary hash and B+-tree indexes, and remote memory allocation through RDMA operations.
6 Compute Servers
Compute servers execute transactions through sequential transaction execution threads using timestamp vectors, while catalog access and fault tolerance are implemented with RDMA-based mechanisms. Recovery handles memory-server failures through replicated logs and checkpoints and compute-server failures through monitoring.
- Transaction execution threads run sequentially and use the complete timestamp vector as the read timestamp and their own thread timestamp to tag new versions.
- The catalog is hash-partitioned across memory servers and accessed with two-sided RDMA operations because catalog activity is lower than transaction execution load.
- 6.2 Failures and Recovery: NAM-DB tolerates compute- and memory-server failures, but does not currently handle network partitioning and does not target high availability.
- 6.2 Failures and Recovery: Each transaction execution thread writes its private journal to more than one memory server to avoid losing the log after a memory-server failure.
- 6.2 Failures and Recovery: After a memory-server failure, the system halts and reconstructs a consistent state from checkpoints and a read-timestamp-ordered merged log.
- 6.2 Failures and Recovery: Monitoring compute servers detect failed compute servers and unlock abandoned locks using their transaction execution logs.
7 Evaluation
NAM-DB’s evaluation on TPC-C shows near-linear scalability for distributed transactions, with locality providing a moderate performance benefit rather than being essential. The experiments also identify scalable timestamp-oracle behavior and workload contention as key boundaries.
- System scalability: The traditional Snapshot Isolation implementation did not scale and degraded beyond 10 machines because of high message-handling CPU costs.Its increasing latency reflects the growing amount of message processing required as the cluster expands.
- Latency: NAM-DB latency stayed almost constant as machines were added, whereas the classic Snapshot Isolation implementation’s latency increased.NAM-DB’s work per machine remains constant; its slight latency increase without locality mainly came from installing new versions.
- Timestamp oracle: The new timestamp oracle sustained the full TPC-C load, whereas the original oracle reached only 2 million timestamp transactions per second and degraded with more than 20 clients.The basic new oracle achieved 20 million t-trxs/sec, with network bandwidth becoming the bottleneck as the timestamp vector grew.
- Locality: Locality improved throughput and latency by roughly 30%, while NAM-DB still achieved more than 1.5 million transactions with 100% distributed transactions.The comparison indicates that locality is beneficial but not required for high distributed-transaction throughput.
- Contention: Increasing workload skew raised abort rates through contention on a single machine, although throughput remained stable for uniform and low-skew distributions.The evaluation supports a boundary: RDMA cannot make workloads with inherently concentrated contention scalable.
8 Related Work
Related work spans distributed transaction processing, RDMA-enabled data management, and high-performance OLTP, while NAM-DB differs by redesigning the database architecture around one-sided RDMA operations and scale-out.
- NAM-DB argues that distributed transactions can scale with recent RDMA-enabled networking technology, unlike approaches centered on locality.
- FaRM uses message-based communication and focuses on serializability, whereas NAM-DB implements snapshot isolation and treats locality as an optimization.
- A related design separates storage from compute but uses a centralized commit manager likely to bottleneck at larger cluster sizes.
- Existing industrial systems use RDMA for support functions, but do not directly redesign distributed transaction processing around the network.Oracle RAC uses RDMA support without directly exploiting it for transaction processing; IBM pureScale uses centralized coordination, while SQLServer extends a single-node buffer pool.
- Academic RDMA projects target joins or traditional shared-nothing architectures, whereas NAM-DB redesigns the full database stack for scale-out.
- Distributed transaction research over slower networks reduces overhead through locality, speculation, weaker consistency, or relaxed durability guarantees.
9 Conclusions
The paper presents NAM-DB as a scalable distributed database that uses distributed transactions by default and treats locality as an optimization. Its evaluation reports nearly linear scale-out through 56 machines and 6.5 million TPC-C transactions per second.
- NAM-DB uses distributed transactions by default and considers locality an optimization.
- 6.5 million total TPC-C transactions per second were achieved with nearly perfect linear scale-out to 56 machines.
- The paper concludes that NAM-DB is at most limited by the workload itself.