PaperlayerView exact revision on arXiv

Source-linked AI summary

xTier: Intelligent Tiering for CXL-Enabled Memory

Sriranga Ramaswamy, Yueqi Chen

arXiv:2609.27266v1cs.OS

TL;DR

CXL expands memory capacity but requires timely choices about which pages belong in DRAM rather than slower CXL memory. xTier uses kernel-resident learned ranking with admission control; across evaluated workloads and ratios, it matches or beats existing policies while moving far fewer pages.

  • Problem

    CXL memory creates a placement problem: the OS must decide which pages merit scarce DRAM and which can remain in slower CXL memory.

    Source: §1 · p. 1

  • Method

    xTier uses eBPF and PEBS to score promotion candidates inside the kernel, with a compact learned model and runtime gates controlling placement.

    Source: §1 · p. 2, §9 · p. 12

  • Results

    Across six workloads and five DRAM:CXL ratios, xTier matches or beats existing policies while moving far fewer pages.

    Source: §9 · p. 12

  • Takeaways & Limitations

    Effective CXL tiering depends on deciding which moves are worthwhile and bounding their cost, rather than migrating more aggressively.

    Source: §9 · p. 12

    NUMA emulation does not reproduce physical CXL device contention, topology, or bandwidth behavior, so evaluation on multiple CXL hardware generations is needed.

    Source: §7 · p. 11

Abstract

from arXiv · show

CXL-enabled memory expands server memory capacity, but introduces a page-placement problem: the operating system must decide which pages should reside in DRAM and which should reside on slower CXL memory. Existing systems make this tradeoff in one of two ways. Userspace controllers support flexible policies, but expose placement decisions to scheduler jitter and kernel-userspace crossing overhead. Kernel-space systems avoid this latency, but rely on fixed heuristics that must generalize across workloads. We present xTier, a kernel-resident learned memory-tiering system. xTier attaches eBPF programs to PEBS events and uses a compact quantized MLP to score sampled pages inside the kernel at microsecond-scale latency. Rather than reacting to every candidate, xTier converges to a low-churn placement for the current workload phase, reduces sampling cost after convergence, and returns to a higher sampling cadence when the workload shifts. We evaluate xTier on six memory-bound workloads at DRAM:CXL ratios from 1:5 to 1:25. The advantage grows as the DRAM budget tightens. At 1:15 and beyond, xTier is the fastest system in 14 of 18 configurations. Where it is not fastest, it trails the best baseline by 3.9% on average. It reaches this performance while moving 13% fewer pages in geometric mean, and 22% fewer at the tighter ratios. When a workload changes phase, xTier rebuilds its hot set in DRAM faster and more completely than any baseline.

1 Introduction

xTier addresses CXL page placement by combining kernel-fast decisions with workload-specific learned ranking and bounded migration. Across six memory-bound workloads, it performs best in most tighter-memory configurations while moving fewer pages.

1 Introduction

  • CXL expands memory beyond local DIMM capacity but requires the OS to place pages between DRAM and slower CXL memory as workloads run.CXL memory appears as a CPU-less NUMA node, with load latency roughly 1.5–2× local DRAM.Source: §1 · p. 1
  • Existing userspace controllers offer flexible policies but face scheduler delay, while kernel-resident systems reduce latency but typically rely on fixed rules that must generalize across workloads.The design target is kernel-local, learned ranking with migration bounded by pages whose moves are worth their cost.Source: §1 · p. 1
  • xTier attaches eBPF to PEBS events and uses an offline-trained, quantized MLP to score promotion candidates inside the kernel at microsecond-scale latency.A per-workload model is installed into BPF maps at startup without changing the eBPF object or rebuilding the kernel.Source: §1 · pp. 1–2
  • At DRAM:CXL ratios of 1:15 and beyond, xTier is fastest in 14 of 18 configurations and trails the best baseline by 3.9% on average where it is not.It moves 13× fewer pages in geometric mean and 22× fewer at tighter ratios; after phase changes, it rebuilds the DRAM hot set faster and more completely than any baseline.Source: §1 · p. 2
  • xTier separates candidate ranking from migration admission, using top-K selection, per-page cooldowns, Accessed-bit demotion, and adaptive sampling to limit migration traffic.The sampling controller slows after placement converges; per-epoch admission gates and cooldowns further throttle movement.Source: §1 · p. 2

2 Background

The background describes access signals suited to different placement decisions, constraints on in-kernel policy execution, and the CXL latency gap that motivates tiering.

2.1 Access tracking for page placement

  • PEBS provides sampled evidence of page activity for promotion, while persistently clear Accessed bits provide inexpensive evidence for demotion.PEBS records sampled events but incurs overhead that rises with sampling rate; Accessed-bit scans reveal whether a page was touched since the prior observation, not how often or recently.Source: §2.1 · p. 2

2.2 eBPF for in-kernel policy

  • eBPF runs user-supplied programs at kernel event hooks, but verifier limits on program size and control flow and the lack of native floating-point arithmetic complicate conventional ML inference.The verifier checks memory safety, bounded control flow, and stack use before programs are JIT-compiled.Source: §2.2 · p. 2

2.3 CXL memory tiering

  • CXL exposes byte-addressable memory as an additional NUMA node, with production CXL 1.1 latency around 2× DRAM at idle and 2.9× under bandwidth pressure.Tiering keeps frequently accessed pages in DRAM and uses CXL for remaining capacity, promoting and demoting pages as access patterns change.Source: §2.3 · p. 3

3 Motivation

The motivation argues that effective tiering must be timely and selective: access patterns shift quickly, userspace control paths can miss the decision window under contention, and ranking should reflect workload-specific future benefit.

3 Motivation

  • The section frames effective tiering as a combination of timely in-kernel decisions, selective migration, and workload-specific ranking rather than a single retrospective hotness rule.Profiling and coarse coordination can remain in userspace, where flexibility matters more than latency.Source: §3 · p. 3, §3.2 · p. 3

3.1 Tiering must be both timely and selective

  • Page activity often lasts only single-digit milliseconds, so 100 ms scanner intervals can act after a burst ends; migrating every recently active page can cost more than placement saves.The section identifies a 10 ms decision window and calls for migrating only pages whose expected future accesses can amortize migration cost.Source: §3.1 · p. 3

3.2 Latency-critical operations must be in the kernel

  • Under CPU contention, userspace per-epoch p99.9 latency reaches 34.7 ms, while the in-kernel path remains below the 10 ms target at 0.30 ms.The gap persists when policy computation is replaced with an O(1) operation; userspace transfers observations and decisions across the kernel boundary and depends on control-thread scheduling.Source: §3.2 · p. 3

3.3 Page ranking should be workload-specific

  • Fixed access-count or recency rules describe past activity, but the same statistic can indicate sustained reuse in one workload and a short-lived burst in another.The proposed direction is workload-specific ranking that predicts future migration benefit from current runtime observations; recurring jobs can amortize profiling and training across later runs.Source: §3.3 · p. 3

4 Technical Design

xTier combines event-driven page telemetry and workload-trained ranking with kernel-side admission and migration controls. Its design bounds page movement, adapts observation frequency after placement stabilizes, and uses a conservative signal for demotion.

4.1 Event-driven telemetry and per-page state

  • At each millisecond-scale epoch, PEBS samples update per-page access, rate-change, recency, and migration state before eligible candidates are scored.Eligibility checks exclude pages already in DRAM, queued pages, and regions governed by static rules; collection overlaps inference on the preceding epoch’s buffer.Source: §4 · p. 3, §4.1 · p. 4

4.2 Ranking promotion candidates

  • The quantized MLP ranks epoch candidates using 32 inputs, hidden layers of 192 and 96 units, and one scalar output.The training target is a percentile rank among reused pages, rather than raw benefit; runtime admission uses score order, not a calibrated probability.Source: §4.3 · pp. 4–5, §4.2 · p. 4

4.4 Why learn instead of replay?

  • xTier profiles each workload and ranks runtime pages by behavioral features rather than replaying profiled addresses, making placement responsive to changes in layout and phase.The model estimates future access benefit from recent page history; a recorded address-to-tier map can become stale when page contents or workload phase change.Source: §4.4 · p. 5

4.5 Bounded promotion through admission control

  • Three admission controls bound movement: page cooldowns prevent rapid repeat migration, histogram thresholds approximate top-K selection, and occupancy targets stop promotions when DRAM is full.These controls separate candidate priority from actual migration and limit the effect of inaccurate or stale scores primarily to placement quality.Source: §4.5 · p. 5

4.6 Adaptive sampling decision cadence

  • After candidate rate, admitted promotions, and cross-tier movements stay below thresholds for N stable epochs, xTier lengthens epochs and raises PEBS sampling intervals; renewed activity returns it to Active mode.The ranking model and admission policy remain unchanged across modes.Source: §4.6 · p. 5

4.7 Conservative A-bit demotion

  • Demotion waits for a DRAM page’s Accessed bit to remain clear across a scan period, so a single observation interval without access—not lack of PEBS samples—triggers demotion.The scanner processes 50,000 page-table entries about once per second, clearing set bits and resuming each traversal from its prior stopping point.Source: §4.7 · pp. 5–6

4.8 Batched migration execution

  • A shared executor batches promotion and demotion intents from a bounded lock-free queue, merges adjacent pages into ranges, and updates tier state only after migration succeeds.Per-cycle migration budgets limit executor work; capacity tracking starts demotion at 70% of the fast-tier target and permits overshoot of at most one cycle’s budget.Source: §4.8 · p. 6

5 Implementation

The implementation splits telemetry and inference across eBPF programs and uses a kernel module for demotion and migration. A userspace loader handles setup, training deployment, and threshold updates outside the per-decision path.

Verifier-compatible inference

  • The full MLP forward pass is partitioned into tail-called eBPF stages, with intermediate activations in a per-CPU scratch map and quantized weights in BPF map state.Candidate buffers and score histograms are double-buffered so PEBS telemetry can continue during inference and threshold computation.Source: §5 · p. 6

Kernel migration support

  • A Linux 6.11 kernel wrapper exposes the migration operation required by the module, while the rest of the base kernel remains unmodified.The wrapper accepts an address-space structure, virtual-address range, and target NUMA node, then invokes existing migration machinery.Source: §5 · p. 6
  • The migration module drains at most 64 intents per cycle from a 32,768-entry lock-free ring, merges contiguous pages, and drops duplicate or excess intents rather than blocking PEBS handling.Tier state changes only when a migration call succeeds.Source: §5 · p. 6

Userspace control and training

  • The loader installs the model and PEBS configuration, while userspace computes and publishes each next-epoch admission threshold outside the per-decision path.Profiling takes 15 minutes and training takes one to two minutes on a single CPU; retraining replaces the weight header without changing the eBPF object.Source: §5 · p. 6

5 Implementation

  • Figure 4 reports normalized speedup across workloads and DRAM:CXL ratios, with higher values indicating better performance.Each workload’s speedup is normalized to its slowest system–ratio configuration.Source: Figure 4 · p. 7

6 Evaluation

Across the evaluated workloads, xTier’s strongest end-to-end results appear when fast-tier capacity is constrained. The evaluation attributes its gains to selective migration and learned ranking, and examines convergence, phase adaptation, generalization, and deployment costs.

6.1 Methodology

  • Tests cover tree-training and graph-analytics workloads, compare four tiering systems, and use a tier-latency gap smaller than real CXL devices impose.The evaluation reports runtime, fast-tier occupancy, migration volume, and decision-path overhead.Source: §6.1 · p. 7

6.2 RQ1: How does xTier compare to existing tiered-memory systems?

  • xTier’s end-to-end advantage grows as DRAM capacity shrinks, while differences are generally smaller at larger DRAM fractions.It remains close to the best baseline when it is not the top performer.Source: §6.2 · p. 7

6.3 RQ2: Where do xTier’s gains come from?

  • Filters bound decision and migration work, while learned ranking improves page selection; without filters, neither ranker completes within the fixed cutoff.Scoring costs 34 µs per candidate, and an admitted promotion adds 19 µs for migration.Source: §6.3 · pp. 8–9
  • The INT8 ranker reaches AUCs of 0.767 on LightGBM, 0.791 on XGBoost, and 0.848 on PageRank, while quantization preserves score ordering with Spearman correlation above 0.98.Recent access deltas and multi-epoch access-count history are the two most influential feature groups.Source: §6.3 · pp. 8–9

6.4 RQ3: Does xTier’s learned policy generalize and adapt over time?

  • xTier performs most migrations during initial placement discovery, then approaches a low steady-state rate as useful opportunities are exhausted.For XGBoost, several baselines continue adding migration work or show later bursts.Source: §6.4 · p. 9, Figure 6 · p. 9
  • After a forced phase change, xTier’s DRAM coverage rises to 94%, while Memtis reaches 55%, TPP 74%, and FreqTier 30% after respective plateaus.xTier begins recovering within a few seconds; the figure marks the first 60 seconds after the change.Source: Figure 7 · p. 9, §6.4 · p. 10
  • Swapping trained models changes admission-rate precision by 0.4% for LightGBM, 0.1% for PageRank, and 0.3% for XGBoost across tested inputs.The results support amortizing profiling and training across recurring runs of the same workload.Source: §6.4 · p. 10

6.5 RQ4: What are the costs of deploying xTier?

  • Steady-state CPU use remains below 4% of one core; xTier’s 218 MB footprint is 0.27% of an 80 GB NUMA node, and training takes 1–2 minutes.The deployed INT8 model occupies 114 KB; steady-state memory is dominated by BPF map state.Source: §6.5 · p. 10

7 Discussion and Future Work

xTier uses an offline, workload-specific ranking model that remains fixed during execution, with online adaptation proposed as future work. The discussion identifies boundaries involving workload multiplicity, generalization, hardware signals, and CXL emulation.

Online learning

  • xTier trains its workload-specific ranking model offline and holds it fixed during execution.The proposed extension keeps inference in the kernel while training and lightweight adaptation run asynchronously in userspace, with updated weights validated before deployment.Source: §7 · pp. 10–11

Multiple workloads

  • xTier currently assumes one managed workload; shared-tier arbitration requires a cross-workload policy for fairness, priority, or QoS.Scores from separate models are not comparable because each is meaningful only against its own workload’s distribution; policy design and evaluation under contention are left to future work.Source: §7 · p. 11

Generalization

  • Offline specialization suits repeated, managed workloads, but its benefit may diminish for one-off workloads or when access behavior changes after profiling.Inaccurate rankings reduce placement quality; the proposed future direction is retraining from differences between predicted and observed reuse for promoted pages.Source: §7 · p. 11

Hardware dependence

  • The prototype uses Intel PEBS for precise evidence of active pages and page-table Accessed bits as inexpensive evidence of inactivity.Porting to other architectures requires mapping these signals to available sampling and page-reference mechanisms.Source: §7 · p. 11

CXL emulation

  • The evaluation uses remote NUMA memory, whose measured 1.25× tier-latency ratio is below reported CXL 1.1 ratios of approximately 2× idle and 2.9× loaded.NUMA emulation does not reproduce physical CXL device-side contention, topology, or bandwidth behavior; evaluation across CXL hardware generations is needed to establish their effect on performance.Source: §7 · p. 11

8 Related Work

Related work addresses tiered-memory placement, kernel machine learning, and BPF customization. xTier differs by learning per-workload page rankings and keeping inference and placement decisions on the in-kernel path.

Related work

  • xTier builds on prior work in tiered-memory placement, in-kernel machine learning, and BPF kernel customization.The passage frames these as three areas of related work.Source: §8 · p. 11

Tiered-memory placement

  • Prior tiered-memory systems use hand-designed signals, while xTier learns a workload-specific ranking over access history, recency, and structural context.xTier also adapts sampling and decision cadence as placement converges.Source: §8 · p. 11
  • Userspace systems support learned policies, but xTier avoids their kernel-boundary and scheduling overheads by placing inference on the in-kernel decision path.The passage names HeMem, Kleio, ArtMem, and FarSight as userspace approaches.Source: §8 · p. 11

Machine learning inside the kernel

  • Prior kernel machine-learning systems use custom frameworks, remote GPU invocation, or eBPF decision trees; xTier runs a quantized neural network entirely within eBPF for page placement.The passage also situates xTier among broader uses of learning in cache prediction, prefetching, and replacement.Source: §8 · p. 11

BPF kernel customization

  • Prior BPF customization systems expose hand-authored policies or userspace-profile-driven policies; xTier applies learned tiered-memory placement with in-kernel neural inference.The passage names FetchBPF, cache_ext, PageFlex, and eBPF-mm as examples of prior policy extensions.Source: §8 · p. 11

Complementary axes

  • Other tiered-memory systems improve observation signals or adapt profiling and migration cadence, whereas xTier learns the page-ranking policy itself.Latency- or stall-based signals are described as orthogonal and potentially incorporable as model features; TierBPF exposes hooks for hand-written admission rules.Source: §8 · p. 11

9 Conclusion

CXL memory expansion makes page placement a central operating-system problem, while existing systems trade flexible but delayed userspace decisions against fixed kernel heuristics. xTier keeps learned scoring and admission-controlled placement in the kernel, matching or beating existing policies while moving fewer pages.

9 Conclusion

  • Existing systems put flexible policy in userspace, where scheduling and boundary crossings delay decisions, or use fixed kernel heuristics that cannot adapt to a particular workload.xTier is presented as a kernel-resident learned tiering controller that uses eBPF and PEBS to score promotion candidates inside the kernel.Source: §9 · pp. 11–12
  • Across six workloads and five DRAM:CXL ratios, xTier matches or beats existing policies while moving far fewer pages.DRAM acts as an admission-controlled tier: a page moves only when the learned model and runtime gates agree that the move will repay its cost.Source: §9 · p. 12
  • xTier’s conclusion is that effective CXL tiering depends on deciding well and bounding migration costs, not migrating more aggressively.The passage describes xTier as combining a compact learned model with runtime gates for page movement.Source: §9 · p. 12

Algorithm 1: Hot page promotion

The supplied algorithm excerpts identify PEBS samples as promotion input and batched promotion intents as output; separate excerpts describe a DRAM-occupancy check and batched demotion intents.

Algorithm 1: Hot page promotion

  • PEBS samples are the input to the promotion algorithm, which outputs batched promotion intents.The excerpt also says to submit the promotion batch to the migration executor.Source: Algorithm 1: Hot page promotion · p. 14

Algorithm 2: Cold page demotion

Loading 2609.27266v1…