Source-linked AI summary
Pond: CXL-Based Memory Pooling Systems for Cloud Platforms
Huaicheng Li, Daniel S. Berger, Stanko Novakovic, Lisa Hsu, Dan Ernst, Pantea Zardoshti, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, Ricardo Bianchini
TL;DR
Public clouds must provide excellent performance for opaque VMs while controlling hardware costs, but existing memory-saving and pooling techniques can add microsecond-scale latency or require page faults and VM changes. Pond uses CXL load/store access, small 8-16-socket pools, and prediction models to allocate local and pooled memory; it reduces needed DRAM by 7% while targeting configurable performance slowdowns of 1-5%.
Problem
Public cloud providers need excellent performance for opaque VMs at competitive hardware cost, while existing memory-saving and pooling techniques can incur microsecond latency, page faults, or VM changes.
Method
Pond combines CXL cacheable load/store access, a small 8-16-socket memory pool, and prediction models that allocate local and pool memory to VMs.
Results
7% less needed DRAM with a 16-socket pool, corresponding to a 3.5% reduction in overall cloud server cost.
Takeaways & Limitations
Pond provides a full-stack CXL memory pool that satisfies cloud-provider requirements while achieving substantial DRAM savings.
Takeaways & Limitations
Existing pooling approaches can require page faults or guest-VM changes, while process-level compression adds microseconds of latency.
Abstract
from arXiv · showhide
Public cloud providers seek to meet stringent performance requirements and low hardware cost. A key driver of performance and cost is main memory. Memory pooling promises to improve DRAM utilization and thereby reduce costs. However, pooling is challenging under cloud performance requirements. This paper proposes Pond, the first memory pooling system that both meets cloud performance goals and significantly reduces DRAM cost. Pond builds on the Compute Express Link (CXL) standard for load/store access to pool memory and two key insights. First, our analysis of cloud production traces shows that pooling across 8-16 sockets is enough to achieve most of the benefits. This enables a small-pool design with low access latency. Second, it is possible to create machine learning models that can accurately predict how much local and pool memory to allocate to a virtual machine (VM) to resemble same-NUMA-node memory performance. Our evaluation with 158 workloads shows that Pond reduces DRAM costs by 7% with performance within 1-5% of same-NUMA-node VM allocations.
1. Introduction
Pond addresses the cost and performance tension of public-cloud memory by pooling stranded DRAM through CXL while preserving near-local VM performance. Its small-pool architecture, ML-guided allocation, and monitoring pipeline target practical cloud deployment.
- Motivation: Public-cloud providers must deliver dedicated-cloud-like VM performance without inspecting VM workloads or incurring equivalent hardware cost.Main memory is a major driver of both performance and cost.
- Motivation: Up to 25% of DRAM becomes stranded when all server cores are rented but memory capacity remains unallocated.Pooling can return this stranded capacity to a shared resource for use by other servers.
- Limitations of the state of the art: Existing compression and pooling approaches can introduce microsecond latency, page faults, or guest-VM changes that conflict with cloud requirements.These limitations make reducing DRAM usage difficult while preserving statically preallocated, low-latency memory access.
- Pond: Pond uses CXL cacheable load/store accesses so memory can remain statically preallocated while physically residing in a shared pool.CXL supports pooled-memory access at nanosecond-scale latency, although it remains slower than same-NUMA-node memory.
- Pond: Production-trace analysis finds that pooling across 8-16 sockets captures most benefits while avoiding the higher latency estimated for rack-scale pools.The analysis estimates 70-90ns added latency for 8-16-socket pools and more than 180ns for rack-scale pooling.
- Pond: Pond predicts VM latency sensitivity and untouched memory to allocate local and pool memory, then monitors performance and migrates poorly performing VMs to local memory.The model inputs come from existing hardware telemetry without measurable overhead.
2. Background
Cloud memory management must balance multi-resource VM placement with low-latency access. Memory pooling addresses stranded DRAM, while CXL provides load/store access without page faults or DMA.
- Memory stranding: VM scheduling is a multi-dimensional bin-packing problem, making resource mismatches and stranded memory difficult to avoid.Memory is typically stranded when all cores are rented but usable DRAM remains unavailable because another resource is exhausted.
- Memory pooling: Memory disaggregation dynamically reassigns pooled memory across hosts instead of provisioning every server for worst-case resource ratios.Pooling targets deviations from average DRAM-to-core ratios by sharing memory among multiple hosts.
- Pooling via CXL: CXL.mem maps pooled device memory into the system address space, allowing LLC misses and write-backs to access it through CXL transactions.These accesses use load/store memory semantics and involve neither page faults nor DMAs.
- Pooling via CXL: CXL access latency depends largely on the CXL port and adds 70ns to CXL reads relative to NUMA-local DRAM reads.With PCIe 5.0, a bidirectional ×8 CXL port can match a DDR5-4800 channel at a typical 2:1 read:write ratio.
3. Memory Stranding & Workload Sensitivity to Memory Latency
Azure traces show substantial memory stranding and strong workload variability in sensitivity to remote memory latency. Pooling can reduce DRAM demand, but practical designs must preserve local memory for latency-sensitive workloads.
- Memory Stranding: At the 95th percentile, stranded memory reaches 25% during high-utilization periods, with individual outliers reaching 30%.Stranding rises as more CPU cores are allocated to VMs.
- Memory Stranding: Pooling across 16-32 sockets can reduce cluster memory demand by 10%.The benefit assumes that a high percentage of DRAM can be allocated on memory pools.
- Workload Sensitivity: Under a 182% latency increase, 26% of workloads slow down by less than 1%, while 21% slow down by more than 25%.The evaluation covers 158 workloads relative to NUMA-local memory.
- Workload Sensitivity: Workload sensitivity varies substantially within classes, with every class except SPLASH2x containing workloads below 5% and above 25% slowdown.Even the same graph kernel can respond differently depending on its graph dataset.
- Workload Sensitivity: Under a 222% latency increase, 23% of workloads slow down by less than 1%, while more than 37% slow down by more than 25%.Higher latency magnifies the effects observed under the lower-latency scenario.
4. Pond Design
Pond combines a small CXL memory pool, zNUMA exposure, prediction models, and memory-management mechanisms to target near-NUMA-local performance while reducing pooling overheads.
- Design goals: Pond defines configurable performance degradation margins relative to NUMA-local DRAM and uses small CXL pools to meet them.The design also targets accelerator compatibility, opaque VMs, unchanged guests, and low host overhead.
- Hardware layer: Pond reduces pool latency by 1/3 versus switch-only designs, while 8- and 16-socket pools add 70-90ns over NUMA-local DRAM.The design favors small pools because larger pools incur higher latency and cost with diminishing returns.
- Pool management: Pond assigns pool memory in 1GB slices and uses ownership management, fragmentation controls, and asynchronous release to reallocate capacity across hosts.Offlining takes 10-100 milliseconds/GB, while onlining is near instantaneous with microseconds/GB.
- Control plane: The control plane predicts latency sensitivity and untouched memory, then monitors QoS and can reconfigure VMs when predictions fail.The latency-insensitivity model is parameterized by a target false-positive rate, trading the fraction labeled insensitive against prediction risk.
- VM memory exposure: Pool memory is exposed to VMs as a zero-core virtual NUMA node, allowing the guest OS to prefer local memory before using the pool.Pond sizes zNUMA using predicted untouched memory so that this capacity can remain available without affecting active working sets.
5. Implementation
Pond is implemented across the server, hypervisor, telemetry, and distributed control plane to emulate and manage CXL-pool behavior in production-oriented environments.
- Prototype platform: Pond emulates a CXL pool on a two-socket server by disabling one socket’s cores while retaining access to its memory.The retained memory serves as the pool for prototype experiments.
- Hypervisor: The hypervisor supports arbitrary zNUMA topologies and receives desired zNUMA configurations from the control plane.This allows VM placement and memory allocation decisions to be expressed through the VM interface.
- Telemetry: Pond gathers per-VM PMU telemetry once per second and scans hypervisor page-table access bits every 30 minutes.The reported sampling and scanning costs are 1ms and 10s, respectively.
- Distributed control plane: The control plane trains RandomForest and gradient-boosted regression models from aggregated telemetry and adds zNUMA predictions to VM requests.The scheduler incorporates zNUMA requests and pool memory into bin packing.
6. Evaluation
Pond is evaluated with diverse cloud workloads, production-oriented experiments, and trace-driven simulations covering zNUMA performance, model accuracy, and DRAM savings.
- Evaluation scope: 158 workloads span databases, KV stores, analytics, graph processing, HPC, CPU benchmarks, shared-memory benchmarks, and Azure internal workloads.The evaluation quantifies DRAM savings through simulations.
- Metrics: The study uses NUMA-local DRAM as its baseline and reports workload-specific normalized slowdowns.Metrics include job runtime, throughput, and tail latency.
- Experimental design: The evaluation includes more than 3,500 experiments and 10,000 machine hours across seven zNUMA sizes, with outlier checks on production servers.Most experiments use lab servers configured to resemble production systems.
- Simulation: Production traces cover 100 randomly selected clusters across 34 global datacenters over 75 days for trace-driven simulation.The simulator tracks server and pool capacity at second accuracy while applying alternative memory-allocation policies.
- zNUMA effectiveness: zNUMA contains most memory accesses locally: the video workload sends fewer than 0.25% of accesses to zNUMA, while four workloads range from 0.06-0.38%.These production-node measurements support negligible impact when untouched-memory predictions are correct.
6.3. zNUMA VMs in the Lab
Lab experiments show that correctly sized zNUMA preserves all-local performance, whereas undersized local nodes cause slowdowns that grow with memory spilled into zNUMA.
- Correct sizing: Correctly predicted untouched memory yields workload slowdowns with a distribution similar to all-local memory.This result is consistent with zNUMA receiving few accesses under correct sizing.
- Misprediction: Overpredicting untouched memory spills active working-set pages into zNUMA and can produce slowdowns as high as 30-35%.These slowdowns occur with 20-75% of workload memory spilled.
- Misprediction: Slowdowns increase further as more workload memory spills into zNUMA, reaching up to 50% when the workload is entirely allocated on pool memory.Access-bit scans verify that affected workloads actively access their entire working sets.
- Design implication: Because fixed pool allocations can cause significant slowdowns, Pond identifies slowdown-prone workloads and allocates untouched memory to the pool.The design uses both strategies to limit the impact of spilled active memory.
6.4. Performance of Prediction Models
Pond combines prediction models to allocate pooled memory while controlling performance risk. The models identify latency-insensitive workloads and predict untouched memory, enabling substantial pool placement with low misprediction rates.
- Predicting Latency Sensitivity: Pond evaluates latency-insensitivity prediction across performance degradation margins (PDM) from 1–10%, reporting detailed results for 5% PDM and 182% latency increase.The 222% latency-increase model is 16% less effective at the same false-positive-rate target.
- Predicting Latency Sensitivity: Multiple workloads exceed 20% slowdown with just two percent of DRAM boundedness, making latency-sensitivity prediction intrinsically difficult.The paper concludes that both heuristics and predictors will make statistical errors.
- Predicting Latency Sensitivity: RandomForest places 30% of workloads on the pool with only 2% false positives, while DRAM bound significantly outperforms Memory bound.RandomForest performs slightly better than DRAM bound.
- Predicting Untouched Memory: GBM overpredicts only 2.5% of VMs when labeling 20% of memory as untouched, versus 12% for the static policy.This is reported as 5× greater accuracy than the static policy.
- Predicting Untouched Memory: The production untouched-memory model identifies 25% of untouched memory while overpredicting 4% of VMs, with performance similar to simulation despite distributional shifts.The model is trained nightly using metadata and resource-usage logs from 100 clusters over 75 days.
- Combined Model: With a 2% scheduling-misprediction target, Pond schedules 44% and 35% of DRAM on pools under 182% and 222% memory-latency increases, respectively.The combined model outperforms its individual models by finding their optimal combination; scheduling mispredictions incorporate untouched-memory overprediction and PDM violations.
6.5. End-to-end Reduction in Stranding
Pond reduces aggregate DRAM requirements in end-to-end simulations while maintaining bounded scheduling mispredictions and practical pool-memory offlining rates. Its savings exceed those of static allocation across the evaluated latency settings.
- Evaluation setup: The end-to-end evaluation uses PDM =5% and TP =98%, with a strawman static policy assigning each VM 15% of pool DRAM.About 2.5% of strawman scheduling decisions are mispredictions.
- End-to-end reduction: At a pool size of 16 sockets, Pond reduces overall DRAM requirements by 9% under 182% latency increases and 7% under 222% latency increases.Static allocation reduces DRAM by 3%; relative savings remain qualitatively similar across PDM values from 1–10% and TP values from 90–99.9%.
- End-to-end reduction: Static allocation reduces DRAM by 3% at the 16-socket pool size, compared with Pond’s 9% and 7% reductions under the two latency settings.The comparison is reported for the same end-to-end evaluation.
- End-to-end reduction: Pond can safely reduce cost, and a QoS monitor mitigating more than 1% of mispredictions can support more aggressive performance targets.The monitor is part of Pond’s mechanism for handling scheduling mispredictions.
- Practicality: Pond’s pool-memory offlining speed stays below 1GB/s for 99.99% of VM starts and below 10GB/s for 99.999% of VM starts.These rates are reported throughout the simulations.
7. Discussion
Pond’s discussion addresses robustness and deployment assumptions for ML-driven pooled-memory allocation. It also situates the design relative to static allocation, hardware availability, and customer behavior.
- Design implications: Pond’s combined model and QoS monitor support cost reduction while retaining the paper’s performance-targeting framework.The discussion connects practical deployment assumptions with the system’s resource-management approach.
- Robustness of ML: Oversubscribed-resource multiplexing can limit the impact of any individual customer’s resource overuse when serving millions of customers.Providers may also offer small discounts when resources are not fully utilized.
- Alternatives to static memory preallocation: Pond retains static memory compatibility because ATS/PRI devices are not broadly available and virtual IOMMUs require guest-OS changes and introduce overhead.ATS/PRI would enable compatibility with page faults, while virtual IOMMUs allow fine-grained pinning.
8. Related Work
Related work spans hardware, operating-system, runtime, tiering, ML, and NUMA approaches to disaggregated or optimized memory. Pond differs by using platform-level ML and a pooled-memory design that avoids several compatibility or deployment costs.
- Hardware-level disaggregation: Hardware-level disaggregation designs often rely on noncommodity hardware, while Pond targets a more deployable platform-level design.ThymesisFlow and Clio use FPGA-based rack-scale designs over OpenCAPI and RDMA.
- Hypervisor/OS-level disaggregation: Hypervisor/OS approaches rely on page faults and access monitoring, bringing overhead and jitter and conflicting with virtualization acceleration such as DDA.These approaches maintain working sets in local DRAM.
- Runtime/application-level disaggregation: Runtime-based designs can be effective but require developers to use customized application-level mechanisms for remote memory access.This differs from Pond’s platform-level approach.
- Memory tiering: Google achieves 6µs latency through proactive hot/cold page detection and compression, whereas Pond uses an ML-based platform-level approach to memory pooling.Pond is described as orthogonal to prior memory-tiering work.
- ML for systems: Pond uniquely applies ML to untouched-memory prediction for pooled-memory provisioning to VMs without jeopardizing QoS.The paper places this application alongside other ML-for-systems uses in cloud efficiency and memory optimization.
- Coherent memory and NUMA optimizations: Pond’s ownership model avoids coherence across the memory pool, while zNUMA’s zero-core design requires rethinking existing NUMA optimizations.Traditional coherent NUMA systems use specialized interconnects for a shared address space.
9. Conclusion
Pond is a full-stack memory pool motivated by stranded and untouched memory in production cloud clusters. With a 16-socket pool and a 222% CXL latency increase, it reduces required DRAM and overall cloud server cost.
- 7% less DRAM is needed with a 16-socket pool, assuming CXL increases latency by 222%.
- 3.5% lower overall cloud server cost results from Pond's reduced DRAM requirement.
- Pond is a full-stack memory pool spanning hardware, system software, and distributed system layers.