PaperlayerView exact revision on arXiv

Source-linked AI summary

An Introduction to the Compute Express Link (CXL) Interconnect

Debendra Das Sharma, Robert Blankenship, Daniel S. Berger

arXiv:2306.11227v3cs.ARcs.OS

TL;DR

Existing PCIe and DDR interfaces have limitations, including missing coherency for device-attached memory. This tutorial surveys CXL’s successive standards and implementations, showing how they address coherency, memory scaling, resource pooling, and distributed sharing.

  • Problem

    PCIe does not support coherency or mapping device-attached memory into coherent memory space, limiting its replacement of DDR.

  • Method

    The tutorial provides accessible technical details on CXL 1.0, 2.0, and 3.0, implementations, performance, broader impacts, and future directions.

  • Results

    CXL 1.0 addresses coherency and memory scaling, CXL 2.0 enables multi-host resource pooling, and CXL 3.0 enables composable systems and fine-grained memory sharing.

  • Takeaways & Limitations

    CXL has gained broad commercial traction and offers a path toward addressing key industry challenges through broad deployment.

  • Takeaways & Limitations

    CXL.io/PCIe ordering cannot enforce some synchronization patterns with pipelined accesses, requiring smart NICs to serialize writes after reads when ordering matters.

Abstract

from arXiv · show

The Compute Express Link (CXL) is an open industry-standard interconnect between processors and devices such as accelerators, memory buffers, smart network interfaces, persistent memory, and solid-state drives. CXL offers coherency and memory semantics with bandwidth that scales with PCIe bandwidth while achieving significantly lower latency than PCIe. All major CPU vendors, device vendors, and datacenter operators have adopted CXL as a common standard. This enables an inter-operable ecosystem that supports key computing use cases including highly efficient accelerators, server memory bandwidth and capacity expansion, multi-server resource pooling and sharing, and efficient peer-to-peer communication. This survey provides an introduction to CXL covering the standards CXL 1.0, CXL 2.0, and CXL 3.0. We further survey CXL implementations, discuss CXL's impact on the datacenter landscape, and future directions.

1 INTRODUCTION

CXL is an open interconnect standard designed to address limitations in coherent access, memory scalability, resource utilization, and distributed data sharing. Across CXL 1.0, 2.0, and 3.0, it adds coherency and memory semantics, resource pooling, dynamically composable systems, and fine-grained sharing while remaining backward compatible.

  • CXL’s scope: CXL is an open industry standard defining interconnect protocols between CPUs and a broad range of devices.Its scope spans the compute stack, including accelerators, storage, and memory-related devices.
  • Motivating challenges: CXL addresses four major challenges: non-coherent device access, insufficient memory scalability, stranded resources, and fine-grained distributed data sharing.These challenges arise from limitations of PCIe, DDR, and tightly coupled server resources.
  • CXL generations: CXL 1.0 multiplexes coherency and memory semantics on top of the PCIe physical layer.This addresses PCIe’s lack of coherency and its inability to map device-attached memory into the coherent memory space.
  • CXL generations: CXL 2.0 enables resource pooling across multiple hosts, allowing resources such as memory to be reassigned without rebooting hosts.CXL switches provide the network structure needed for pooling and help overcome resource stranding and fragmentation.
  • CXL generations: CXL 3.0 supports rack- or pod-scale dynamically composable systems and fine-grained memory sharing across host boundaries.Multiple levels of CXL switching address larger-scale resource composition and distributed data sharing.
  • Adoption and scope: CXL has gained widespread traction through commercial adoption by virtually all silicon vendors and requires support from both CPUs and devices.The tutorial reviews CXL generations, implementations, performance, broader impacts, and future directions.

2 CXL BACKGROUND AND DESIGN CHOICES

CXL adopts an asymmetric, host-orchestrated coherence design with a simple MESI-based device agent and multiple protocol options. It builds on PCIe for backward-compatible interoperability and ecosystem deployment while evolving from earlier efforts to add coherency and memory semantics to PCIe.

  • Prior interconnect approaches: Earlier Intel efforts added transaction hints and atomic semantics to PCIe for accelerator access while keeping PCIe non-coherent.Intel also pursued memory semantics on PCIe 3.0 to enable pooling with a shared memory controller.
  • CXL design choices: CXL decouples coherence from host-specific protocols and assigns the host processor responsibility for orchestrating device coherency.Devices use a simple MESI coherency protocol with a small command set.
  • CXL design choices: CXL supports multiple use cases through protocols with differing complexity, allowing devices to implement only a subset.This protocol flexibility is presented as a way to accommodate diverse device requirements.
  • CXL design choices: CXL uses the PCIe physical layer and slots, enabling interoperability with prior-generation CXL devices and PCIe devices.Building on PCIe also permits reuse of IP blocks, channels, and software infrastructure.
  • CXL ecosystem: CXL emerged in 2019 amid competing standards including OpenCAPI, GenZ, and CCIX, and its membership later grew to about 250 companies.The consortium published CXL 1.1 in September 2019, followed by CXL 2.0 and CXL 3.0 in November 2020 and August respectively.

3 CXL 1.1 PROTOCOL

CXL 1.1 connects host processors and devices with coherency and memory semantics, enabling shared-data processing and memory bandwidth or capacity expansion. Its asymmetric, PCIe-based architecture multiplexes CXL.io, CXL.cache, and CXL.mem protocols, with Flit-based transport and low-latency coherent accesses.

  • 3 CXL 1.1 PROTOCOL: CXL 1.1 introduces coherency and memory semantics for devices directly attached to a host, supporting shared-data processing and memory bandwidth or capacity expansion.These capabilities target heterogeneous CPU–accelerator processing and cost-effective scale-up.
  • 3 CXL 1.1 PROTOCOL: CXL uses an asymmetric Root Complex–End Point architecture in which the host orchestrates cache coherency and software configures devices through host-generated transactions.Native link widths are x16, x8, and x4, while x2 and x1 operate in degraded mode.
  • 3 CXL 1.1 PROTOCOL: CXL.io, CXL.cache, and CXL.memory are dynamically multiplexed on the PCIe physical layer, supporting discovery, DMA, cache coherence, and memory access.CXL.io reuses PCIe-based, non-coherent load-store semantics, while the other protocols provide coherent and memory-oriented functions.
  • 3 CXL 1.1 PROTOCOL: CXL 1.0, 1.1, and 2.0 use 68-byte Flits comprising a 2-byte protocol-ID, 64-byte payload, and 2-byte CRC.CXL.cache and CXL.mem share Flit-based transport, whereas CXL.io carries PCIe TLP/DLLP packets within the Flit payload.
  • 3 CXL 1.1 PROTOCOL: CXL.cache and CXL.mem provide low-latency accesses comparable to a native CPU-to-CPU symmetric coherency link and similar to remote-socket DDR access.Remote-socket access remains slower than local-socket DDR, but is acceptable in two-socket systems with NUMA optimization and higher bandwidth.
  • 3 CXL 1.1 PROTOCOL: CXL.cache overcomes a CXL.io ordering limitation that can require smart NICs to serialize writes after reads when synchronization matters.CXL.io writes can bypass prior reads in pipelined accesses, whereas CXL.cache allows out-of-order prefetching and low-latency local-cache completion.

A. HDM-H Read B. HDM-DB Read

CXL’s HDM-D coherence model lets a device change host cache state through CXL.cache, using a Bias Flip flow. CXL also supports varied CPU coherence implementations and uses protocol dependence graphs to reason about channel blocking and deadlock freedom.

  • A. HDM-H Read: HDM-D devices can change host cache state through CXL.cache requests in the Bias Flip flow.The device sends ‘RdOwnNoData X’ to the host, which recognizes X as an HDM-D address owned by that device.
  • A. HDM-H Read: CXL.cache is the natural mechanism for HDM-D because HDM-D is defined only for Type-2 devices.
  • B. HDM-DB Read: CXL accommodates multiple CPU coherence implementations, including CXL.cache beneath a host L3 cache shared with CPU cores.In the example, CXL.io/PCIe access to host memory passes through the same cache using destination coherent semantics.
  • B. HDM-DB Read: The CXL protocol dependence graph identifies channel dependencies and helps establish deadlock freedom when no circular dependence exists.For example, an L1 Req may depend on the L1-Snp channel completing before another L1 Req is processed.

4 CXL 2.0 PROTOCOL

CXL 2.0 introduces hot-plug, switching, pooling, QoS, and persistent-flush capabilities that enable dynamic resource assignment across hosts. Its protocol combines virtual hierarchies, logical devices, and Fabric Manager commands to manage pooled resources at run time.

  • Resource Pooling: CXL 2.0 enables resource pooling, allowing operators to reassign device memory among hosts at run time and thereby reduce resource stranding.A compute-intensive host’s unused pooled memory can be reassigned to a memory-intensive host.
  • CXL 2.0 Additions: CXL 2.0 adds hot-plug, single-level switching, QoS for memory, memory pooling, device pooling, and Global Persistent Flush.Standard PCIe hot-plug mechanisms support both physical hot-plug and dynamic resource pooling.
  • Switching and Topology: Single-level switching standardizes hierarchical address decoding, while virtual hierarchies represent multi-host topologies and enable device pooling.HDM decoders allow address decoding at each switch rather than requiring complete hierarchy decoding by the host or switch.
  • Device Pooling: Multi-Logical-Devices divide one CXL.mem device into up to 16 logical devices that can be assigned to different hosts simultaneously.Each logical device is identified by an LD-ID; CXL 3.0 later adds Dynamic Capacity Device extensions.
  • Pool Management: The CXL Fabric Manager assigns and removes devices at run time through CCI commands, including bind, Set-LD, and unbind operations.The FM may run on a host, in BMC firmware, within a CXL switch, or as a dedicated device; host software enumerates resources attached or detached at run time.

5 CXL 3.0 PROTOCOL

CXL 3.0 extends resource pooling to large, multi-level fabrics supporting up to 4096 endpoints while preserving ordering across multiple paths. Its Back-Invalidate flow and routing enhancements enable peer-to-peer access, coherent shared memory, and scalable memory pooling.

  • 5 CXL 3.0 PROTOCOL: CXL 3.0 expands resource pooling through multi-level switching and supports up to 4096 end devices, including hosts, memory, accelerators, and other I/O devices.The design targets dynamically composable distributed systems with low latency and sufficient bandwidth.
  • 5 CXL 3.0 PROTOCOL: CXL 3.0 supports fabric topologies with multiple paths between source-destination pairs, enabling lower latency, higher bisection bandwidth, and fail-over capability.This removes tree-topology restrictions that limit scaling to thousands of devices.
  • 5 CXL 3.0 PROTOCOL: Direct peer-to-peer access lets PCIe/CXL devices reach coherent HDM memory without involving the host processor when no conflict arises.In the cited example, NIC-to-memory access is 8 hops using direct P2P versus 16 hops round-trip through a CPU.
  • 5 CXL 3.0 PROTOCOL: The Back-Invalidate flow introduces HDM-DB memory and supports peer-to-peer communication, Type-2 snoop filters, and hardware-enforced coherent shared memory across independent hosts.UIO combined with Back-Invalidate reduces snooping overhead and latency from repeatedly accessing the host processor.
  • 5 CXL 3.0 PROTOCOL: Port Based Routing provides an optional, simpler routing format for scaling to 4096 endpoints and non-tree, multi-path topologies.Its edge-port design lets hosts and devices use PBR without directly being aware of it.
  • 5 CXL 3.0 PROTOCOL: A Global Fabric-Attached-Memory Device can provide high-scalability memory shared or pooled across all 4095 other agents in the CXL fabric.CXL 2.0 Multi-Logical Devices can be shared by a maximum of 16 hosts.

6 SURVEY OF CXL IMPLEMENTATIONS

CXL implementations span CPU, FPGA, memory, near-memory-computing, and interconnect IP products, with CXL 1.1 and 2.0 already deployed while CXL 3.0 products remain forthcoming. The survey characterizes implementation structure, latency, link efficiency, bandwidth, and emerging direct device-to-device flows.

  • Implementations: CXL support spans Intel Sapphire Rapids CPUs and Agilex7 FPGAs, AMD Genoa and Bergamo CPUs, and announced ARM V2, N2, and E2 CPUs.Intel supports all three CXL protocols, while ARM has announced CXL 2.0 support.
  • Implementations: Device implementations include interoperable CXL IP, Samsung’s CXL 1.1 memory expansion device, announced Type-3 memory devices, and a prototyped CXL 1.1 near-memory-computing device.IP vendors demonstrated interoperability with Sapphire Rapids CPUs, while multiple vendors announced CXL Type-3 memory products.
  • Implementation architecture: CXL IP combines standard interfaces with CXL.io and CXL.cache+mem stacks, including PHY, arbitration/multiplexing, link, and transaction layers.The link layer handles CRC, credits, and retry; the transaction layer packs Flits, queues transactions, and processes them.
  • Latency: 21 ns common-clock and 25 ns independent-clock round-trip latency is measured from the SERDES pin to the internal application layer and back.Implementation variation depends on placement, process technology, and PHY design.
  • Latency: 57 ns is the estimated end-to-end latency adder for a memory access across a CXL link, including two port traversals and 15 ns of retimer flight time.The estimate combines 21–25 ns round-trip latency on both CPU and device sides with conservative retimer flight time.
  • Bandwidth and efficiency: CXL link efficiency ranges from 0.906 to 0.92 for CXL.io 68-Byte Flits and reaches 0.924 or 0.939 for CXL.cache+mem with sync headers on or off.A x16 CXL device achieves 56.6 GB/s for processor reads and 40 GB/s per direction for the specified 68-Byte-Flit write flow.

7 DISCUSSION

CXL reshapes the compute landscape through device adoption, scalable and efficient memory, rack- or cluster-level interconnects, and composable systems. Its open, simple, backward-compatible design also motivates broad research directions in architecture, systems, and engineering.

  • Implications: CXL’s lower latency, pooling, and hardware coherency expand usage models beyond PCIe, encouraging accelerators and memory/storage devices to adopt CXL.CXL.cache and CXL.mem can provide substantial performance uplift, while CXL pooling can reduce total cost of ownership.
  • Implications: CXL provides 8x higher bandwidth per pin and higher capacity than locally attached DDR, while pooling and heterogeneous media improve cost efficiency.CXL also enables longer, more flexible motherboard routing, helping address thermal constraints.
  • Implications: CXL could become a rack- or cluster-level interconnect because it lowers latency by an order of magnitude versus Ethernet and InfiniBand and supports coherent memory sharing.Dedicated cabling with stringent requirements remains a constraint.
  • Implications: CXL enables composable systems in which pooled memory and I/O devices dynamically assign capacity fractions to hosts, improving resource usage.Expected multi-host capabilities include NICs, accelerators, storage, and memory devices.
  • Future directions: CXL opens research directions including independently evolving external memory controllers, adaptive DRAM refresh, reliability improvements, and reduced memory waste.External controllers can lower cost and facilitate faster iteration and customization.
  • Future directions: Systems and engineering research must address memory-pressure guarantees, fabric congestion, failures, latency reduction, packing, and CXL’s increased error-management blast radius.CXL.mem QoS currently does not address fabric congestion, motivating additional systems-level mechanisms.
  • Conclusion: CXL’s openness, simplicity, and backward compatibility have enabled wide industry traction and established it as a growing academic research area.The paper presents the tutorial as an introduction and starting point for research ideas.
Loading 2306.11227v3…