PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices

Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, Nam Sung Kim

arXiv:2303.15375v4cs.PFcs.AR

TL;DR

Growing memory-capacity and bandwidth demands motivate CXL, but prior work commonly used remote NUMA memory because true CXL hardware was unavailable. This paper evaluates genuine CXL systems and devices, then uses the findings to design Caption, which improves memory-bandwidth-intensive application performance by up to 24% over traditional NUMA page allocation.

  • Problem

    True CXL memory was insufficiently characterized because commercially available hardware was lacking, leading recent research to rely on remote-NUMA emulation despite fundamental differences.

  • Method

    The paper evaluates a 4th-generation Intel Xeon CXL-ready system with three manufacturers’ devices, compares true and emulated CXL memory, analyzes CPU–memory interactions, and develops Caption.

  • Results

    Caption improves memory-bandwidth-intensive application performance by up to 24% compared with the default page allocation policy for traditional NUMA systems.

  • Takeaways & Limitations

    True CXL memory has device- and controller-dependent behavior, and CXL-aware page allocation can use it as a memory-bandwidth expander.

  • Takeaways & Limitations

    Caption’s estimation model uses weight values fitted exclusively from DLRM counter values, though broader application fitting could improve it.

Abstract

from arXiv · show

The ever-growing demands for memory with larger capacity and higher bandwidth have driven recent innovations on memory expansion and disaggregation technologies based on Compute eXpress Link (CXL). Especially, CXL-based memory expansion technology has recently gained notable attention for its ability not only to economically expand memory capacity and bandwidth but also to decouple memory technologies from a specific memory interface of the CPU. However, since CXL memory devices have not been widely available, they have been emulated using DDR memory in a remote NUMA node. In this paper, for the first time, we comprehensively evaluate a true CXL-ready system based on the latest 4th-generation Intel Xeon CPU with three CXL memory devices from different manufacturers. Specifically, we run a set of microbenchmarks not only to compare the performance of true CXL memory with that of emulated CXL memory but also to analyze the complex interplay between the CPU and CXL memory in depth. This reveals important differences between emulated CXL memory and true CXL memory, some of which will compel researchers to revisit the analyses and proposals from recent work. Next, we identify opportunities for memory-bandwidth-intensive applications to benefit from the use of CXL memory. Lastly, we propose a CXL-memory-aware dynamic page allocation policy, Caption to more efficiently use CXL memory as a bandwidth expander. We demonstrate that Caption can automatically converge to an empirically favorable percentage of pages allocated to CXL memory, which improves the performance of memory-bandwidth-intensive applications by up to 24% when compared to the default page allocation policy designed for traditional NUMA systems.

1 INTRODUCTION

CXL addresses growing memory-capacity and bandwidth demands, but prior studies often emulated it with remote NUMA memory despite fundamental differences from true CXL systems. This paper evaluates genuine CXL hardware, characterizes those differences, and proposes Caption for dynamic page allocation.

  • Motivation: CXL uses a PCIe-based serial interface to expand memory capacity and bandwidth with fewer pins while decoupling memory technologies from the CPU interface.CXL devices use about 3× fewer pins than DDR5, while PCIe-based links offer higher transfer rates and lower energy per bit at the cost of longer latency.
  • Research gap: Remote NUMA emulation can mischaracterize CXL memory because true CXL memory differs fundamentally in latency, bandwidth, and cache-coherence behavior.The paper argues that such differences may change prior performance conclusions and system-level design decisions.
  • Approach: The study evaluates a 4th-generation Intel Xeon CXL-ready system with three memory devices from different manufacturers and compares true CXL memory with emulation.It also analyzes the CPU–CXL-memory interplay using microbenchmarks.
  • Key findings: True CXL memory achieves up to 26% lower latency and 3–66% higher bandwidth efficiency than emulated CXL memory, depending on access instructions and devices.Controller designs and memory technologies produce a wide range of true-device latency and bandwidth values.
  • Application impact: Naïve CXL placement can increase tail latency for latency-sensitive applications by 10–82% relative to local DDR memory.The impact is smaller for complex millisecond-scale applications, whose end-to-end latency is only marginally affected by CXL access latency.
  • Policy: Caption dynamically adjusts the fraction of newly allocated pages placed in CXL memory using runtime counters and memory-subsystem history to improve throughput.The policy targets CXL’s bandwidth-expansion capability rather than treating CXL as an interchangeable NUMA tier.

2 BACKGROUND

CXL extends PCIe with protocols that support coherent communication between CPUs, devices, and attached memory. The CXL.mem protocol lets CPUs access device memory using ordinary load and store instructions while exposing it as remote NUMA memory.

  • CXL protocols: CXL defines CXL.io, CXL.cache, and CXL.mem protocols over PCIe’s physical layer for initialization, device access to CPU memory, and CPU access to device memory.CXL.io uses PCIe transaction- and data-link-layer features, while CXL.cache and CXL.mem support the respective coherent memory directions.
  • CXL.mem: The CXL.mem protocol is handled by the CPU Home Agent and CXL controller, exposing device memory transparently as a remote NUMA node.CPUs access CXL memory with load and store instructions, unlike technologies such as RDMA that use a device DMA engine.
  • Hardware support: CXL requires support from both CPU and device hardware, with controllers integrating PCIe, CXL link and transaction layers, and memory controllers.4th-generation Intel Xeon and AMD EPYC processors are among early commodity server CPUs supporting CXL 1.1.

3 EVALUATION SETUP

The evaluation uses a two-socket Intel Sapphire Rapids server, three heterogeneous CXL memory devices, and microbenchmarks and applications spanning latency-sensitive and throughput-oriented workloads.

  • System configuration: The server has two Intel Sapphire Rapids sockets, with eight DDR5-4800 DIMMs on one socket and one DDR5-4800 DIMM on the other for remote-NUMA CXL emulation.The populated socket provides 128 GB across eight memory channels.
  • CXL devices: Three CXL devices use different CXL implementations and DRAM technologies: ASIC or FPGA CXL IP with DDR5-4800, DDR4-2400, or DDR4-3200.Because CXL does not prescribe the underlying memory technology, devices can exhibit different latency and bandwidth characteristics.
  • Microbenchmarks: Intel MLC measures memory latency and bandwidth, while the custom memo benchmark provides control over access instructions and characterization scenarios.Memo can measure latency and bandwidth for specific instructions such as AVX-512 non-temporal loads and stores.
  • Applications: The application evaluation includes Redis with YCSB and DeathStarBench microservices for latency-sensitive behavior, plus DLRM and SPECrate CPU2017 for throughput.These workloads cover differing memory-access characteristics and performance metrics.

4 MEMORY LATENCY AND BANDWIDTH CHARACTERISTICS

The evaluation compares emulated remote-NUMA memory with three true CXL devices across latency, bandwidth, instruction types, and cache interactions. True CXL performance varies substantially by controller and device, while cache-coherence and interface behavior explain several differences.

  • Evaluation setup: The study evaluates DDR5-R emulated CXL memory and three true CXL devices using different memory-access instructions and cache-hierarchy analyses.The devices are CXL-A, CXL-B, and CXL-C.
  • Latency: 76% lower ld latency for emulated CXL memory under memo than Intel MLC shows that parallel accesses exploit full-duplex UPI communication.Serialized pointer-chasing in Intel MLC cannot exploit this capability as effectively as memo’s random parallel accesses.
  • Latency: CXL-A has 35% longer ld latency than DDR5-R, whereas CXL-B and CXL-C have almost 2× and 3× longer latency, respectively.Even with the same DDR4 technology, CXL-C’s ld latency is 67% longer than CXL-B’s.
  • Latency: True CXL can reduce latency and coherence overhead relative to emulated CXL because it lacks remote CPU caches and cores requiring coherence checks.For nt-st, CXL-A has 25% lower latency than DDR5-R; stores also incur implicit reads and doubled interface traversal.
  • Bandwidth: DDR5-R, CXL-A, CXL-B, and CXL-C deliver 70%, 46%, 47%, and 20% of theoretical maximum bandwidth for All Read.Bandwidth efficiency depends strongly on controller design, instruction type, and DRAM technology; CXL-A is 23 percentage points below DDR5-R despite using the same DRAM technology.
  • Bandwidth: CXL-A provides 23 percentage points higher bandwidth efficiency than DDR5-R for the 2:1-RW workload, while CXL-B exceeds CXL-A for ld, nt-ld, and nt-st.These patterns suggest controller-specific handling of interleaved reads and writes and differences between newer DDR5 and mature DDR4 controllers.
  • Interaction with Cache Hierarchy: CXL memory exposes CPU cores to a larger effective LLC capacity in SNC mode, changing LLC hit, miss, and interference behavior relative to local DDR memory.This cache-hierarchy difference affects application performance and must be considered when evaluating CXL memory.

5 IMPACT OF USING CXL MEMORY ON APPLICATION PERFORMANCE

Application performance depends on whether workloads are latency-sensitive or bandwidth-intensive and on how pages are placed across DDR and CXL memory. CXL can hurt tail latency for sensitive workloads but improve throughput when it supplies additional memory bandwidth.

  • Experimental setup: Changing page placement across DDR and CXL exposes workload-dependent performance effects, using application-specific metrics and an SNC configuration with two DDR channels and one CXL channel.The study uses CXL-A and varies the percentage of pages allocated to CXL memory.
  • Latency: 105% higher p99 latency occurs for Redis with 100% of pages in CXL memory than DDR memory at 85 K target QPS.The corresponding increases are 10% at 25 K and 73% at 45 K target QPS.
  • Latency: 174% higher p99 latency results from TPP compared with statically allocating 25% of random pages to CXL memory.TPP’s page copying and page-table updates create migration overhead, and continued migration further increases latency.
  • Latency: DSB’s compose-posts, read-user-timelines, and mixed workloads show little p99-latency difference between CXL 100% and DDR 100%.Front-end and logic components contribute most of the p99 latency, amortizing CXL access latency.
  • Latency: CXL 100% gives mixed workloads lower p99 latency than DDR 100% between 5 K and 11 K QPS because mixed workloads consume more memory bandwidth.As QPS approaches 11 K, CPU compute becomes the dominant bottleneck and narrows the latency gap.
  • Throughput: 63% of DLRM pages in CXL memory maximizes 32-thread throughput, providing 88% higher throughput than DDR 100%.CXL supplements DDR bandwidth, and devices with lower maximum bandwidth require a lower CXL-page percentage for peak throughput.
  • Throughput: Redis throughput decreases as more pages move to CXL memory, reaching 30% lower throughput at 100% CXL allocation for YCSB-A.Redis is latency-bounded because it does not fully utilize memory bandwidth.

6 CXL-MEMORY-AWARE DYNAMIC PAGE ALLOCATION POLICY

Caption dynamically tunes the fraction of newly allocated pages placed in CXL memory using runtime memory-subsystem signals, targeting higher application throughput. Its design uses monitored CPU counters, a performance-estimation model, and feedback from prior allocation decisions.

  • Policy Overview: Caption tunes new-page placement between DDR and CXL memory using CXL bandwidth, application memory intensity, and average memory access latency.The policy is intended to use CXL as an auxiliary bandwidth resource while accounting for its latency and device-specific capabilities.
  • Runtime Modules: Caption periodically samples CPU counters and estimates memory-subsystem performance before using that estimate to guide page-allocation decisions.The monitored signals include metrics such as L1 miss latency, DDR read latency, and IPC.
  • Performance Signals: DLRM throughput rises with consumed memory bandwidth until contention increases memory-access latency and reduces throughput.The observed turning point motivates using memory-subsystem measurements rather than simply maximizing CXL placement.
  • Performance Signals: More CXL placement initially lowers DDR-controller pressure and L1 miss latency, but excessive placement makes CXL latency dominate and decreases throughput.Thus, the favorable allocation ratio depends on the balance between DDR contention and CXL access latency.
  • Performance Estimation: Caption fits a linear-regression model that maps sampled CPU-counter values to estimated memory-subsystem performance at runtime.The current implementation uses a simple model to keep OS overhead low; Y is estimated performance, X_n a counter value, and β_n its learned weight.
  • Allocation Tuning: When performance changes, Caption adjusts the percentage of new pages allocated to CXL memory based on the previous decision and observed performance history.The implementation also applies a minimum allocation limit and mechanisms for very small or sudden performance changes.
  • Scope: The policy was developed using DLRM statistics but is expected to generalize because its monitored metrics are fundamental memory-subsystem indicators for bandwidth-intensive workloads.The paper identifies broader counter coverage and fitting across more applications as possible extensions.

(b) SPEC-Mix

Caption is evaluated on SPEC-Mix and other workload combinations by tracking estimated and measured performance while automatically tuning the CXL-page percentage. Across the reported mixes, it generally improves throughput over static policies, though gains vary by workload.

  • Model Scope: The estimation model is adequate for tuning DLRM and SPEC-Mix, but its weights were fitted using DLRM counter values exclusively.The authors identify fitting the model on a more diverse application set as a potential improvement.
  • Evaluation Scope: Caption evaluates 16 individual SPEC benchmarks, two SPEC-Mix workloads, and a Redis-DLRM mix against static DDR/CXL allocation policies.The comparison uses throughput normalized to the default static policy allocating 50% of pages to CXL memory.
  • SPEC-Mix Results: 24% higher throughput is achieved for the cactuBSSN-roms mix than the best static allocation policy, with 33%–41% of pages allocated to CXL memory.The other reported mixes improve by 1% and 4%.
  • SPEC Results: Caption provides 19%, 18%, 8%, and 20% higher throughput for fotonik3d, mcf, roms, and cactuBSSN, respectively, than the best static policy.These results use steady-state CXL-page allocations ranging from 29% to 41%.
  • Individual Workloads: For DLRM, Caption delivers 80% higher throughput than 100% DDR allocation and 4% lower throughput than 50% DDR allocation.The paper does not compare Caption with the static policy for DLRM and Redis individually in Figure 13.
  • Individual Workloads: For Redis, Caption achieves 3.2% higher throughput than 50% DDR allocation but 8.6% lower throughput than 100% DDR allocation.Caption therefore identifies the benefit of favoring low-latency DDR for Redis without matching the best static allocation.

7 RELATED WORK

Prior CXL-memory research has explored tiering, swapping, disaggregation, simulation, prototype systems, and application-specific studies, often using NUMA servers to emulate CXL behavior. This paper distinguishes itself by evaluating true CXL memory on commercial hardware against such emulations.

  • Heterogeneous Memory: Heterogeneous-memory research covers persistent memory, remote or disaggregated memory, and byte-addressable SSDs, each with distinct device characteristics and trade-offs.These technologies differ from standard DDR-based DRAM devices.
  • CXL Research: CXL research has investigated memory tiering, swapping, disaggregation, software simulators, and FPGA-based prototype systems.Examples include efforts by Meta, Microsoft, and Gouk et al.
  • CXL Research: Most prior CXL-memory studies used NUMA servers to emulate CXL behavior, while other work used software simulators or FPGA-based prototypes.The prior literature also includes studies focused on particular applications.
  • This Paper: Unlike prior studies, this paper comprehensively compares true and emulated CXL memory using a commercial high-performance CPU, commercial CXL devices, microbenchmarks, and widely used applications.The comparison is intended to inform design-space exploration for CXL-memory software systems and simulators.

8 CONCLUSION

The paper analyzes device-specific true CXL-memory behavior against NUMA-based emulation and identifies performance implications. It also develops and evaluates dynamic page allocation that uses CXL memory as a bandwidth expander for bandwidth-intensive applications.

  • Conclusion: The study analyzes device-specific characteristics of true CXL memory and compares them with NUMA-based emulations commonly used in CXL research.The analysis reports important performance implications of differences between the two settings.
  • Conclusion: The study identifies opportunities to use CXL memory as a bandwidth expander and develops a CXL-memory-aware dynamic page allocation policy.The policy’s efficacy is demonstrated in the paper’s evaluation.
Loading 2303.15375v4…