Source-linked AI summary
Clio: A Hardware-Software Co-Designed Disaggregated Memory System
Zhiyuan Guo, Yizhou Shan, Xuhao Luo, Yutong Huang, Yiying Zhang
TL;DR
Memory disaggregation must avoid the cost and limitations of server-based nodes while providing more functionality than raw memory devices. Clio addresses this gap with a hardware-based, co-designed system that virtualizes and manages remote memory at memory nodes, achieving strong throughput, latency, scalability, and energy results in an FPGA prototype.
Problem
Existing memory disaggregation uses either costly server-based nodes with latency and scalability limitations or raw devices with performance, security, and management problems.
Method
Clio co-designs hardware, software, compute nodes, memory nodes, networking, and virtual memory around hardware-based memory nodes with separated data and control paths.
Results
Clio achieves similar throughput and median latency to RDMA, orders-of-magnitude lower RDMA tail latency, 1.1× to 3.4× energy savings, and 2.7× higher speed than SmartNIC solutions.
Takeaways & Limitations
Clio demonstrates that hardware-based memory nodes can provide high performance and scalability while using limited processing resources and supporting remote-memory virtualization and management.
Takeaways & Limitations
The current implementation shares its TLB across application processes and lacks per-connection network bandwidth limits, leaving stronger isolation for future work.
Abstract
from arXiv · showhide
Memory disaggregation has attracted great attention recently because of its benefits in efficient memory utilization and ease of management. So far, memory disaggregation research has all taken one of two approaches: building/emulating memory nodes using regular servers or building them using raw memory devices with no processing power. The former incurs higher monetary cost and faces tail latency and scalability limitations, while the latter introduces performance, security, and management problems. Server-based memory nodes and memory nodes with no processing power are two extreme approaches. We seek a sweet spot in the middle by proposing a hardware-based memory disaggregation solution that has the right amount of processing power at memory nodes. Furthermore, we take a clean-slate approach by starting from the requirements of memory disaggregation and designing a memory-disaggregation-native system. We built Clio, a disaggregated memory system that virtualizes, protects, and manages disaggregated memory at hardware-based memory nodes. The Clio hardware includes a new virtual memory system, a customized network system, and a framework for computation offloading. In building Clio, we not only co-design OS functionalities, hardware architecture, and the network system, but also co-design compute nodes and memory nodes. Our FPGA prototype of Clio demonstrates that each memory node can achieve 100 Gbps throughput and an end-to-end latency of 2.5 us at median and 3.2us at the 99th percentile. Clio also scales much better and has orders of magnitude lower tail latency than RDMA. It has 1.1x to 3.4x energy saving compared to CPU-based and SmartNIC-based disaggregated memory systems and is 2.7x faster than software-based SmartNIC solutions.
1 INTRODUCTION
Clio targets the cost, scalability, and latency limitations of server-based and raw-device memory disaggregation with a hardware-based system that places limited processing at memory nodes. Its co-designed architecture separates control from data processing and reports strong prototype performance and energy savings.
- Motivation: Memory disaggregation addresses growing memory demands and resource-management needs by placing memory in independently scalable network-attached pools.The motivation includes capacity limits from server pins, space, and power.
- Motivation: Server-based memory nodes incur monetary and energy costs plus performance and scalability limitations, while raw devices create performance, security, and management problems.These approaches represent the two extremes that Clio seeks to bridge.
- System: Clio provides process-specific remote virtual address spaces, cross-node sharing, byte-granularity accesses, and synchronization primitives through its hardware-based memory system.Multiple processes can allocate from the same CBoard, and one remote address space can span multiple CBoards.
- Design: Clio separates metadata and control operations onto a low-power ARM SoC while handling latency-sensitive memory accesses in hardware.The design assigns rarer metadata operations to software and frequent data operations to hardware.
- Design: Clio moves request-related transport state toward compute nodes and eliminates as much memory-node hardware state as correctness and dataplane performance permit.The design uses a compute-node-request, memory-node-response model and retains only necessary state.
- Design: Clio implements bounded-latency hash-based address translation and hardware page-fault handling to keep frequent data-plane operations scalable and deterministic.Its page-table lookups take at most one DRAM access time in the implementation, while page-fault handling completes within bounded hardware cycles.
- Evaluation: Clio achieves similar throughput and median latency to RDMA, orders-of-magnitude lower RDMA tail latency, 1.1× to 3.4× energy savings, and 2.7× higher speed than SmartNIC solutions.These results come from FPGA prototypes evaluated with three applications and multiple comparison systems.
2 GOALS AND RELATED WORKS
The paper frames memory disaggregation around high utilization, scalability, throughput, latency, and cost goals, then contrasts server-based, raw-device, and intermediate hardware approaches. Existing designs face host, connection, translation, management, or coherence limitations that motivate Clio.
- Goals: Resource disaggregation places resources in independently managed and scalable pools, enabling applications to allocate from any node and improve resource packing.The passage notes that datacenters have widely adopted disaggregation, especially for storage.
- Goals: MemDisagg aims to host hundreds of gigabytes to a few terabytes per memory node with high utilization, while supporting 100 Gbps throughput and sub-2 μs median latency.Low tail latency is also identified as important for datacenter service-level objectives.
- Server-Based Memory: Server-based designs rely on host servers and incur CPU energy costs, while RDMA additionally faces address-translation and protection-domain scalability issues.RDMA performance degrades when its page-table-entry cache is exceeded, and preallocated memory regions waste space.
- Server-Based Memory: RDMA also suffers connection queue-pair scalability problems, while LegoOS avoids hardware complexity only by using software address translation that becomes a performance bottleneck.These limitations affect scalability and the target dataplane performance.
- Physical Disaggregated Memory: Physical disaggregated memory requires client-side coordination and multiple network round trips for complex operations, including pointer chasing and synchronization.The model also makes allocation and other management tasks harder to perform at the memory node.
- Physical Disaggregated Memory: Coherent-interconnect proposals require new endpoint hardware and switches, and automatic cache coherence can impose performance overhead that is not always necessary.The cited proposals and Memory-Driven Computing project had not been fully built in the discussed work.
- Intermediate Designs: soNUMA inserts hardware processing between memory and computation but remains a bundled, single-server solution rather than physically separated memory nodes.Clio instead targets generic datacenter networks with separate compute and memory nodes.
3 CLIO OVERVIEW
Clio combines a co-designed interface and architecture in which compute nodes use a library to access remote virtual memory, while memory nodes execute the data path in hardware. The design supports explicit byte-granularity operations, application-managed caching, and separate control and offload components.
- Architecture: Clio co-designs software and hardware, compute and memory nodes, and the network stack and virtual memory system to keep the memory-node datapath fast and resource-efficient.The entire data path is handled in hardware at the memory node.
- Interface: Clio exposes a non-transparent interface in which applications explicitly allocate and access disaggregated memory through API calls.The APIs can also be used by runtimes or node-side kernel and hardware components for transparent interfaces.
- Interface: Each process has a separate Remote virtual memory Address Space, with a unique global PID across compute nodes.Remote address-space programming resembles multithreaded programming except that remote reads and writes are explicit.
- Interface: Processes on different compute nodes can share one remote address space, and Figure 1 illustrates this usage.The shared address space supports distributed application processes.
- Interface: Remote allocation, freeing, reading, writing, and synchronization use byte-granularity operations with synchronous and asynchronous API options.The interface includes ralloc, rfree, rread, rwrite, and primitives such as rlock, runlock, and rfence.
- Interface: Applications may cache remote reads locally, but Clio does not automatically maintain coherence among cached copies and leaves coherence protocols to applications.This avoids the overhead of automatic coherence on every read and write.
- Architecture: A CBoard combines ASIC datapath logic, an ARM processor for metadata and control, and an FPGA for application computation offloading.Compute nodes use regular servers and Ethernet NICs, while applications run through the CLib user-space library.
4 CLIO DESIGN
Clio addresses hardware-based memory disaggregation by minimizing memory-node state while redesigning virtual memory, networking, consistency, and offloading around hardware constraints. Its design separates fast data handling from slower metadata and control operations.
- Design Challenges and Principles: Hardware-based memory disaggregation must avoid complex data structures and unbounded buffers because memory nodes have limited on-chip resources.Traditional software structures and reliable transports can require metadata and buffers that grow with scale.
- Design Challenges and Principles: Clio uses a clean-slate design that aims to eliminate unnecessary hardware state and bound performance and space overhead.The design principles include avoiding state where possible and making each data unit independently processable.
- Scalable, Fast Address Translation: Clio’s virtual memory system uses hash-based page tables to provide bounded page-table lookup latency while avoiding page-table growth with client-process count.The design addresses frequent TLB misses and the cost of accessing page tables across the network.
- Low-Tail-Latency Page Fault Handling: Clio handles page faults in hardware with a constant three-cycle latency in the CBoard implementation.Physical-address allocation remains on the slow path, while the fast-path handler avoids waiting for allocation on every fault.
- Asymmetric Network Tailored for MemDisagg: Clio keeps transport logic, state, and data buffers at compute nodes, while memory nodes respond to compute-node requests using an asymmetric network design.Consistency ordering is enforced at compute nodes, and Clio also supports application computation offloading at memory nodes.
5 CLIO IMPLEMENTATION
Clio is implemented on FPGA boards combining FPGA logic, an ARM processor, on-board memory, and Ethernet connectivity. The implementation uses shadow metadata, ARM/FPGA polling, and a thin network layer to address prototype constraints and software overhead.
- CLIO IMPLEMENTATION: The prototype uses a small FPGA, a quad-core ARM Cortex-A53, two 10 Gbps SFP+ ports, and 2 GB of off-chip memory.The board also provides 504K logic cells and 4.75 MB of FPGA memory.
- CLIO IMPLEMENTATION: The prototype’s network bandwidth, memory capacity, and clock frequency are below the anticipated real CBoard targets.No available board combines the small FPGA/ARM resources needed for low cost with large memory and high-speed network ports.
- CLIO IMPLEMENTATION: ARM/FPGA communication is optimized with shadow copies of metadata and a dedicated ARM core that polls the interconnect receive ring.Shadow page-table copies let the control path inspect page-table content faster despite slower ARM access to on-board DRAM.
- CLIO IMPLEMENTATION: CBoard’s network stack adds checksum verification and acknowledgment generation over vendor Ethernet physical and link-layer IPs.The implementation uses lossless Ethernet with Priority Flow Control while relying on congestion and incast control to avoid triggering it.
- CLIO IMPLEMENTATION: Clio includes a software CBoard simulator so developers can test CLib applications without running an actual CBoard.CLib is designed for low-latency end-to-end performance because host software can substantially affect application performance.
6 BUILDING APPLICATIONS ON CLIO
The authors build applications that exercise Clio’s basic APIs, extended APIs, and computation offloading. These applications demonstrate support for protected data, distributed key-value storage, consistency guarantees, and split compute-node/memory-node processing.
- Image compression: Clio’s image-compression utility runs at compute nodes and stores each client’s photos in separate memory-node arrays.A separate process per client protects clients’ photos from one another.
- Key-value store: Clio-KV creates key-value pairs through remote allocation and writes, then indexes them using hash buckets, fingerprints, and stored virtual addresses.Reads search bucket slots and compare fingerprints before checking the full key.
- Key-value store: Clio-KV uses a compute-node load balancer to partition key-value pairs across memory nodes and guarantees atomic-write, read-committed consistency.Requests for the same partition go to the same memory node, where Clio APIs are properly ordered.
- Key-value store: Clio-KV’s physical-memory implementation improves latency by 4%–12% and throughput by 1%–5% over the virtual-memory-based Clio-KV.The physical-memory implementation also takes more time to develop.
- Multi-version object store: Clio-MV supports concurrent compute-node accesses and guarantees sequential consistency for each object.Its compound requests remain ordered because Clio’s fast and slow paths provide sequential delivery of operations.
7 EVALUATION
Clio’s evaluation shows strong scalability, throughput, latency, application performance, and energy efficiency across microbenchmarks and workloads. Its benefits are especially pronounced for scalability, tail latency, and selective computation offloading.
- Scalability: Clio scales perfectly with client processes, while RDMA scales poorly because each process uses its own queue pair.Clio’s connectionless design avoids the queue-pair scalability problem observed in RDMA.
- Scalability: RDMA degrades beyond 28 or 212 memory regions depending on the testbed and fails beyond 2^18 memory regions, while Clio remains stable up to 4 TB.Clio exhibits stable TLB-hit and TLB-miss latency levels as memory-region and page-table scale increase.
- Latency: 16.8 ms RDMA page-fault handling causes severe latency degradation, whereas Clio incurs only a small TLB-miss cost and no additional page-fault cost.The comparison covers 16-byte reads and writes under TLB hits, TLB misses, first-access page faults, and RDMA memory-region misses.
- Throughput: Clio reaches 9.4 Gbps end-to-end throughput on the testbed and exceeds 110 Gbps for large on-board FPGA read and write requests.The on-board test removes the board’s 10 Gbps port bottleneck; smaller requests have lower read than write throughput.
- Latency: Clio has much less latency variation and a shorter tail than RDMA, while its projected ASIC read latency is better than RDMA and projected write latency is worse.The ASIC projection scales FPGA components to 2 GHz and replaces board-memory access time with server DDR access time.
- Application Performance: For data analytics, Clio outperforms RDMA when the select ratio is low because computation offloading transfers less data, but performs worse when the select ratio is high.Clio executes selection and averaging at memory nodes, while RDMA transfers rows to compute nodes for all operations.
8 DISCUSSION AND CONCLUSION
Clio is presented as a hardware-based disaggregated memory system whose protection checks occur at memory nodes. The discussion identifies security, isolation, and failure handling as areas requiring future work.
- Clio performs permission checks at memory nodes, restricting each process to its remote memory address space using the global PID.
- Security depends on authentic process IDs attached to Clio requests by trusted compute-node software or hardware.
- Clio currently lacks built-in failure handling, leaving users to construct services such as replicated writes.
- Stronger protection against side-channel and denial-of-service attacks could increase CBoard cost or reduce performance.