PaperlayerView exact revision on arXiv

Source-linked AI summary

Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving

Hongjian Fan, Kevin Zhang, David Habinsky, Sean Dykstra

arXiv:2609.10790v1cs.DCcs.LG

TL;DR

Composable CXL memory addresses the lack of a large, shared KV-cache tier that can be scheduled by Kubernetes. The paper introduces a DRA driver and shared-memory KV tier, achieving 5.5–36.6× TTFT reductions for cross-node prefix reuse on a two-node testbed. The study is limited to memory disaggregation and isolated TTFT measurements rather than prefill/decode disaggregation or loaded serving.

  • Problem

    Kubernetes lacks a schedulable resource type for composable CXL memory shared across multiple hosts, limiting shared-memory multiplexing and reclamation.

  • Method

    The paper introduces a Kubernetes DRA driver that composes CXL regions on demand and a vLLM/llm-d KV tier with an in-region directory for cross-node reuse.

  • Results

    5.5–36.6× lower TTFT is achieved for cross-node prefix reuse on a two-node cluster with a 512 GiB CXL appliance, while VRAM caching remains up to 1.7× faster for local reuse.

  • Takeaways & Limitations

    The system demonstrates that dynamically composed multi-host CXL memory can serve as a shared KV-cache tier exposed as a first-class Kubernetes resource.

  • Takeaways & Limitations

    The study uses full engines and isolated one-session TTFT measurements, so it does not evaluate prefill/decode disaggregation, goodput, tail latency under load, or contention.

Abstract

from arXiv · show

We present a Kubernetes Dynamic Resource Allocation (DRA) driver that makes composable CXL memory a schedulable cluster resource, and evaluate the resulting shared-memory tier for cross-node KV-cache reuse in LLM serving. The driver composes CXL regions on demand, materializes them as DAX devices on each participating host, and injects them into pods under a single Container Device Interface (CDI) name so that pods on different nodes access the same physical region. A shared-memory connector for vLLM/llm-d uses that region as a KV-cache tier with a slot directory embedded inside the shared medium, which eliminates the need for an external metadata service. On a two-node cluster with a 512\,GiB CXL appliance and Qwen2.5-7B-Instruct, cross-node prefix reuse reduces TTFT by 5.5$\times$--36.6$\times$ at an external hit rate of 95.4--99.5\,\%, while node-local tiers (GPU prefix caching, CPU-DRAM offload) fall back to full recompute. The sharing gap, defined as the latency ratio between cross-node and same-node reuse, is 1--4\%, indicating that cross-node reuse incurs little additional latency relative to same-node reuse on our testbed. Both replicas run full engines; the study demonstrates memory disaggregation rather than prefill/decode disaggregation. We report this as a feasibility study rather than a performance evaluation.

1 Introduction

LLM KV caches make long-prefix reuse valuable, but existing tiers trade off capacity, sharing, and access cost. This paper makes composable CXL memory schedulable in Kubernetes and evaluates it as a shared KV-cache tier for cross-node reuse.

  • Motivation: A 32 K-token Qwen2.5-7B-Instruct prompt produces 1.75 GiB of KV tensors, making large shared cache capacity important for repeated prefixes.Each attended token leaves 56 KiB of key and value tensors.
  • Motivation: Existing tiers separate the desired properties: GPU VRAM is fast but private, host DRAM is larger but node-local, and RDMA or object tiers are shared through transfer protocols.
  • Motivation: CXL provides byte-addressable pooled memory across hosts; on the testbed it reads at 27–28 GB/s and avoids a transport layer on the reuse path.The KV block is loaded with memcpy from a shared physical address.
  • Contribution: The paper addresses Kubernetes’s missing topology-aware resource type by composing CXL regions on demand and exposing them as schedulable shared resources.The driver advertises blade capacity, materializes regions on participating hosts, and injects them under one CDI name.
  • Scope: The study is a feasibility report focused on isolated, closed-loop TTFT with full engines, not prefill/decode disaggregation, goodput, tail latency, or contention.It explicitly excludes zero-copy GPU↔CXL DMA, pooled RDMA, and multi-tenant scheduling experiments.

2 Scheduling Composable CXL Memory

The scheduling design extends Kubernetes DRA to composable CXL regions whose topology spans connected hosts. Controllers compose and release regions, while host plugins materialize isolated DAX devices and expose one shared allocation through CDI.

  • 2.1 Resource model: A ResourceSlice spans every host cabled to a blade, allowing the scheduler to place consumers on any connected host.The driver sets a NodeSelector over the blade’s connected hosts rather than a single NodeName.
  • 2.1 Resource model: Region size, share count, and QoS are claim annotations, so the scheduler selects a blade but does not perform byte-level capacity accounting.Capacity is refreshed on a 30-second publish cycle, allowing close-together claims to overcommit and causing a later compose failure.
  • 2.3 Compose-on-demand: One elected DaemonSet controller waits for a consumer, adds a finalizer, composes the region, and publishes CDI device names for kubelet preparation.PrepareResourceClaims polls because composition occurs out of band from kubelet’s request.
  • 2.4 Single CDI name across nodes: The same allocation is materialized independently on each host, potentially at different DAX indices, while its allocation UUID remains the single CDI device name.A pod referencing one claim therefore resolves to a node-local character device backed by the same composed region.
  • 2.5 Host-side materialization and injection: DAX sub-devices provide per-allocation isolation within the physically shared pool, and deletion releases memory through controller-side teardown.Teardown metadata is persisted in memory, annotations, and device status to survive plugin restarts.
  • 2.5 Host-side materialization and injection: Materialization failure on one host is non-fatal, and power-cycle recovery requires operator resynchronization because composed-region metadata is not persisted across power cycles.

3 A Shared Memory Backend for LLM Serving

This section presents a shared-memory KV backend that embeds its directory in a composable CXL region and integrates with vLLM’s OffloadingConnector. It uses DAX mappings, cross-process lookup synchronization, host-staged transfers, and explicit integrity and coherence safeguards.

  • The connector exposes a DAX-backed CXL region through vLLM’s OffloadingConnector, with device path and size supplied by CDI so the same deployment works on any node.
  • The region embeds its own directory using an occupancy bitmap, parallel slot-key array, CRC32 array, and fixed-size data slots.The layout includes headers and writer configuration for attachment and compatibility validation.
  • Lookups derive a 64-bit key from the block-hash filename, use modulo indexing with linear probing, and require both an occupied bit and matching stored key.Acquire-release ordering makes publication and lookup safe across processes.
  • The scheduler reads the in-region directory directly through mmap, eliminating a separately deployed metadata service.
  • The design lacks a model discriminator, can theoretically alias after release-and-reacquire, has no eviction, and treats DAX coherence as a threat to validity.The experiments report tens of thousands of cross-node hits, zero false positives, a 1–4% same-versus-cross latency difference, and zero CRC32 mismatches over 1.2 TB of writes.
  • Stores and loads use host staging buffers rather than zero-copy GPU-to-CXL DMA, achieving 5.8 GB/s against a 27 GB/s fabric.The design therefore leaves 3–5× of the fabric headroom unrealized.

4 Evaluating Shared Memory for Cross-Node KV Reuse

The evaluation uses a two-node Kubernetes testbed to compare local and shared KV-reuse tiers under controlled cross-node requests. It measures closed-loop TTFT across calibrated prefix lengths with leakage, false-hit, and fresh-region controls.

  • The testbed has two AMD EPYC worker nodes, one NVIDIA L4 per node, a 512 GiB two-host CXL region, and Qwen2.5-7B-Instruct served by one vLLM replica per node.
  • The study compares four KV-reuse configurations, with only T4 placing KV blocks where a replica on the other node can reach them.
  • The CXL appliance measures 461–507 ns and 27.1–28.5 GB/s, while the RoCEv2 link measures 1.13 µs and 12.25 GB/s.On this hardware, CXL is 2.4× lower latency and 2.2× higher bandwidth than the RDMA fabric.
  • Cross-node requests place the reuse request on the other replica, where node-local tiers cannot serve the prefix and the shared-memory effect is isolated.
  • The experiment runs 1,920 request pairs across four tiers, two placement arms, four prefix lengths, twenty sessions, and three repetitions.Tier order varies across repetitions, and the CXL region is wiped between repetitions.
  • Validation found at most 0.2% prompt leakage on every tier, zero hits for 12,298 never-stored keys, and the required fresh-region initialization pattern.

4.4 Only a shared tier delivers cross-node reuse

Cross-node reuse succeeds only with the shared CXL tier: local GPU and CPU-DRAM tiers collapse to recomputation, while T4’s advantage increases with context length. The measured sharing gap remains small relative to same-node reuse.

  • 5.5–36.6× median TTFT speedups over recompute occur for cross-node reuse with the shared region, while T1 and T2 are indistinguishable from no cache.The result uses 60 measurements per cell.
  • The sharing gap is 1–4%, meaning cross-node and same-node reuse have closely matched latency on the testbed.
  • The speedup grows with prefix length because recomputation scales superlinearly with context while the CXL read path is dominated by data movement.The effective prefill rate falls from 3,657 to 2,767 tok/s between 2 K and 32 K.
  • Extrapolation places the reuse break-even prefix length in the low hundreds of tokens.

4.5 The sharing gap

The sharing gap measures the latency cost of moving reuse across nodes: T4 remains near its same-node latency, while node-local tiers lose their cache hit. The cross-node path is dominated by fixed engine overhead and host-staged data movement rather than the CXL fabric ceiling.

  • The sharing gap: T4 remains within 1–4% of its same-node TTFT, whereas node-local tiers incur a 6–63× cross-node penalty as context grows.A sharing-gap ratio near 1 indicates nearly perfect sharing; node-local tiers lose their hit when requests move.
  • The sharing gap: 95.4%, 98.0%, 99.0%, and 99.5% of cross-node requests hit at 2 K, 8 K, 16 K, and 32 K tokens, respectively.The 256-token granularity leaves only a trailing partial block to recompute, and measured hits track the alignment bound.
  • Where the hits and the time go: 6.90 µs/token yields 8.3 GB/s on the data path, or 31% of the 27 GB/s fabric limit.The cross-arm decomposition also includes an approximately 104 ms fixed component, mostly engine overhead rather than CXL.
  • Where the hits and the time go: The host-staged CXL → CPU → GPU path limits measured throughput, while a zero-copy path offers 3.3× marginal-rate headroom.The current implementation uses two memcpy hops instead of DMA.
  • Where the hits and the time go: At 32 K tokens, effective read rate reaches 5.81 GB/s, or 21% of the 27.1 GB/s fabric ceiling after fixed overhead is included.The fixed cost amortizes with prefix length; at 2 K tokens, the effective rate is only 1.16 GB/s.

4.7 TTFT distribution and first-touch warm-up

TTFT distributions are generally tight, with T4 approaching same-tier stability at longer prefixes. Its visible tail is a one-time first-touch cost, while the shared tier remains useful when local VRAM caching is unavailable or reuse moves across replicas.

  • TTFT distribution: At prefixes of at least 8 K tokens, T4’s p99/p50 is 1.11–1.18, while its absolute TTFT remains an order of magnitude below T0.T0’s corresponding p99/p50 is 1.01–1.03.
  • First-touch warm-up: T4’s only visible tail occurs at 2 K tokens, where one first read adds approximately 420 ms, raising TTFT from 101 ms steady state to 521 ms.The DAX page-fault and staging-buffer cost is paid once per process; later prefix lengths show no spike.
  • First-touch warm-up: Three of 240 cross-arm requests exceed 450 ms, one first read per repetition.This confines the observed tail to the one-time warm-up event in the closed-loop experiment.
  • Cross-node availability: Local VRAM prefix caching is 1.15×–1.70× faster when reuse stays local, but CXL avoids a 6–63× penalty when reuse moves across nodes.The shared region persists independently of per-replica VRAM pressure and serves replicas after local eviction or load balancing.
  • Correctness: Only 6 of 10 replayed prompts produce byte-identical continuations across T0 and T4, with four divergences attributed to bf16 rounding.The reported divergences are at most 1 ULP in the mantissa, and CRC32 protects against corruption rather than legitimate numerical divergence.

5 Related Work

Prior work develops KV caching, pooled remote storage, CXL KV systems, and Kubernetes resource management largely as separate concerns. This paper combines dynamically composed, multi-host CXL memory with Kubernetes scheduling and contrasts byte-addressable access with transport-based pooling.

  • KV-cache systems: vLLM and SGLang provide single-GPU prefix caching, while LMCache and CacheBlend extend KV storage to host DRAM and remote object storage.Mooncake pools KV across nodes using RDMA.
  • CXL KV systems: TraCT, SAC, and HyMCache target CXL KV storage but assume the CXL region already exists, whereas this approach makes the region schedulable by Kubernetes.The comparison also notes that these systems do not integrate with Kubernetes scheduling beyond static node assignment.
  • Prefill/decode disaggregation: Prefill/decode disaggregation improves accelerator utilization by separating phases, but it does not address KV sharing across replicas.The paper presents memory disaggregation as complementary and potentially useful beneath future P/D handoff.
  • Memory disaggregation: RDMA-based memory-disaggregation systems use explicit transport protocols, while composable CXL exposes shared memory through ordinary load/store semantics.The paper positions its CXL tier as byte-addressable rather than transfer-protocol based.

6 Limitations and Threats to Validity

The study is a two-node feasibility report with full-engine replicas, not prefill/decode disaggregation. Its isolated-prefill setup and narrow hardware/workload scope limit claims about goodput, tail latency, contention, and broader deployment.

  • Both replicas are full kv_both engines, so the study demonstrates memory disaggregation rather than prefill/decode disaggregation.
  • The two-node, single-GPU harness measures isolated-prefill TTFT and cannot characterize background traffic, queueing, goodput, tail latency, or contention.Each node uses one 24 GB NVIDIA L4, and the workload exceeds VRAM capacity with active decode slots.
  • The evaluation covers one model and size with constructed 100% prefix reuse, no reuse-ratio sweep, no capacity pressure, and no multi-tenant scheduling experiment.Evaluating more than two consumers requires a production appliance with additional ports.
  • The study omits a pooled RDMA baseline and leaves prefill/decode handoff, multi-tenant scheduling, and eviction or key re-verification for future work.

7 Conclusion and Future Work

The paper demonstrates dynamically composed multi-host CXL memory as a Kubernetes resource and shared KV-cache tier. On the two-node testbed, cross-node reuse preserved near-local latency, while future work targets disaggregation, stronger baselines, scheduling, and cache consistency.

  • 5.5–36.6× lower TTFT was achieved for cross-node prefix reuse, while node-local tiers fell back to full recompute.The reported sharing gap was 1.01–1.04× for shared CXL versus 6.5–62.6× for node-local tiers when reuse moved nodes.
  • The system exposes dynamically composed, multi-host CXL memory as a first-class Kubernetes DRA resource and uses it as a shared KV-cache tier.The KV directory is embedded in the shared region, removing dependence on an external metadata service.
  • VRAM prefix caching remains up to 1.7× faster for local reuse, whereas the CXL tier remains available when requests land on another node.
  • Future work includes shared-region prefill/decode handoff, a pooled RDMA baseline, multi-tenant scheduling with capacity accounting, and eviction with post-copy key re-verification.

Artifact Availability

The authors plan to release the implementation, evaluation logs, and reproduction scripts.

  • The DRA driver, CXL KV connector, benchmark harness, raw logs, and table and figure generation scripts will be released on GitHub.The evaluation can be regenerated from raw logs in three commands.
Loading 2609.10790v1…