Ember: Making Unified Virtual Memory Practical for Discrete-Memory GPUs

1 Introduction

• Oversubscription is needed when working sets exceed GPU memory, since otherwise applications cannot run; larger GPUs or multi-GPU scaling raise cost without removing the capacity ceiling.The largest models grew by about 1000× while GPU memory grew by about 5× over the cited period.Source: §1 · p. 1
• UVM’s runtime rises from 4.1 s to 1150 s at 1.5× oversubscription in the GPU-sort example, illustrating its collapse beyond device capacity.The figure uses an H200 capped to 16 GiB and sorts scattered data.Source: §1 · p. 1, Figure 1 · p. 1
• UVM’s fault-driven placement causes reclamation stalls under pressure, while mapped access avoids migration but remains interconnect-bound and prefetching can amplify reclamation traffic by up to about 1.5×.Placement hints are allocation-wide and set before launch, so shifting access patterns can leave placement stale.Source: §1 · p. 2
• Ember starts allocations mapped into host memory, promotes hot regions using developer advice or hardware-counter reports, and re-evaluates residency at runtime by demoting cold regions and promoting newly hot ones.Batched reclamation of cold regions is intended to avoid computation stalls.Source: §1 · p. 2
• Ember provides a kernel-module variant for transparent support of unmodified applications and a stock-CUDA userspace variant that fixes the data split at allocation time.The adaptive kernel variant decides the split online; the static userspace variant does not.Source: §1 · p. 2
• At 1.5× oversubscription, Ember achieves a 6.8–10.6× geometric-mean speedup over naive UVM across six applications, sustains LLM generation at 4.7–16× UVM throughput for models exceeding GPU memory capacity, and outperforms profile-guided placement by 2.1× without a profiling run.The evaluation covers four GPUs, three deployment settings, and three workload classes.Source: §1 · p. 2

2.1 Unified Virtual Memory (UVM)

• UVM tracks residency within fixed-size 2 MiB virtual-address blocks, called regions, and a nonresident-page access faults into the driver and stalls until migration completes.Physical GPU memory is allocated and reclaimed in chunks of up to 2 MiB.Source: §2.1 · p. 2

2.2 Data Placement

• Migration faults can trigger tree-based neighborhood prefetching, while explicit prefetch moves ranges ahead of use and mapping keeps ranges in host memory for direct GPU access; pinned explicit copies do not support oversubscription.Migration and prefetching require reclamation once GPU memory is full; mapped access fetches requested cache lines over the interconnect.Source: §2.2 · pp. 2–3

2.3 GPU Hardware Access Counters

• Roughly 256 hardware access counters report accesses to host-resident memory through buffered notifications without stalling, but are too few for large datasets and are unused by default on most systems.On x86 Linux with 4 KiB pages, the driver discards notifications because the corresponding migration path is unsupported at that size.Source: §2.3 · p. 3

2.4 Coherent Interconnects

• Cache-coherent interconnects such as NVLink-C2C let the GPU access host memory through a shared page table and narrow the bandwidth gap, but GPU memory remains finite and a resident set must still be chosen.Physically unified memory is described as a possible way to remove the host-device distinction altogether.Source: §2.4 · p. 3

C1: Reclamation-induced execution stalls

• When GPU memory fills, UVM interleaves eviction with execution, while Ember maps allocations by default to avoid reclamation traffic and page faults.UVM eviction stalls warps and can thrash; Ember’s first key idea is mapped-by-default allocation.Source: §3 · p. 3

C2: Interconnect-bound mapped access

• Host-only mapping avoids reclamation but leaves accesses interconnect-bound, so Ember keeps selected hot regions in device memory.The second key idea is a hybrid address space with mapped and device-resident regions in one allocation.Source: §3 · p. 4

C3: Prefetch-amplified reclamation traffic

• Prefetching can evict up to ∼1.5× the data of stock UVM on random accesses, while Ember promotes counter-identified hot regions and stops before memory fills.Prefetching replaces faults with bulk transfers but increases eviction volume and reclamation traffic.Source: §3 · p. 4

C4: Stale, allocation-granular placement

• At an A-to-B phase shift, a fixed split drops throughput from 750 to 80 GiB/s, while online re-derivation recovers throughput.Ember uses decay-based eager reclamation and online region replacement to keep placement current as the hot set moves.Source: Figure 7 · p. 4, §3 · p. 4

4 Overview

• Ember aims to use GPU memory performance when the working set fits, avoid oversubscription collapse, and require no hardware, kernel, or application changes.It provides Ember-static for fixed placement and Ember-adaptive for runtime placement.Source: §4 · p. 4, §4.1 · p. 4

4.1 System Architecture

• The migration engine, access monitor, and placement policy engine manage placement, translate hardware-counter notifications into demand, and choose region moves.Ember-adaptive makes runtime decisions from GPU hardware access counters; Ember-static fixes the split before launch.Source: §4.1 · pp. 4–5

4.2 Life of an Allocation

• Allocations start mapped in host memory; hardware-counter thresholds trigger hot-region promotion, and cold resident regions are reclaimed in batches before GPU memory fills.A first GPU touch with no host copy creates the page in device memory without a transfer.Source: §4.2 · p. 5

4.3 System Model

• Ember preserves UVM semantics and supports unmodified applications through Ember-adaptive, but allocations outside its managed path remain outside its control.Ember-static uses placement hints and fixes the split before launch; Ember-adaptive requires a kernel module and hardware access counters.Source: §4.3 · p. 5

5 Design

• The design separates migration, access monitoring, and placement policy into components that move data, generate placement hints, and decide region placement.The migration engine provides the hybrid address space; the access monitor and policy engine supply and act on placement signals.Source: §5 · p. 5

5.1 Migration Engine

• Mapped and device-resident regions coexist within an allocation, and each region is one VA block that remains mapped until it earns promotion.Promotions move a block’s host-resident pages in one operation; demotions write physical chunks back to host in bounded batches off the critical path.Source: §5.1 · p. 6

5.2 Access Monitor

• Hardware access counters report accesses to host-resident ranges without costing GPU time, but cannot identify cold GPU-resident regions or cover every VA block.A notification can trigger promotion across a configurable span of consecutive blocks, and the tunable threshold represents a break-even point for migration cost.Source: §5.2 · pp. 6–7

5.3 Placement Policy Engine

• Threshold promotion moves mapped regions once demand crosses a tunable threshold, while decay-based eager reclamation frees space before demand arrives.Decayed demand estimates coldness for eviction; the occupancy gap leaves room for promotions, and migration-aware protection delays eviction of regions that return hot.Source: §5.3 · p. 7
• Conservative capacity accounting maintains headroom so the stock fault-and-evict path does not run, including when GPU memory use is invisible to Ember.Examples of invisible use include allocations outside Ember’s managed path and a free-memory figure that overstates usable capacity.Source: §5.3 · p. 7

6 Implementation

• Ember-static runs on stock CUDA with the unmodified UVM driver, while Ember-adaptive patches NVIDIA’s open-source UVM kernel module.The adaptive prototype is built on CUDA 12.4 and Linux 6.17.Source: §6 · p. 7

6.1 Ember-adaptive

• Ember-adaptive sets managed allocations to prefer CPU memory without moving data at allocation, then promotes counter-notified blocks and performs periodic reclamation.Its worker halves chunk demand every 25 ms, reclaims above 90% occupancy down to 85%, and evicts at most 16 chunks per pass.Source: §6.1 · p. 8

6.2 Ember-static

• Ember-static uses placement hints and prefetches a leading allocation prefix to GPU memory, fixing the split before launch without reclamation or access counters.It suits applications with contiguous hot data whose split need not be revised at runtime.Source: §6.2 · p. 8

7.2 Hybrid Address Space

• On GH200, a sliding 14 GiB read window shows Ember-static matching prefetch for a fixed hot set and outperforming the baselines at full drift.Ember-static’s advice and prefetch add a one-time 1.33 s, amortized over repeated reads.Source: §7.2 · p. 9

7.3 Use Case #1: Data-Intensive Applications

• At 1.5× oversubscription, Ember-static achieves mean speedups over stock UVM of 9.12×, 6.83×, and 10.62× on the RTX 5080, A40, and H200, respectively.Ember-adaptive achieves 8.83×, 5.52×, and 8.01× on those GPUs; across access patterns, Ember’s gains range from 1.8–10× on sequential applications to 11–225× on random ones.Source: §7.3 · pp. 9–10

7.5 Use Case #3: Database Analytics

• Ember-adaptive is fastest in 51 of 60 measured query–GPU–size combinations and all 22 on H200, with mean speedups over UVM of 1.50–2.36× across GPUs.Figure 12 reports SSB query runtimes; q11 and q12 emitted no runtime summary.Source: Figure 12 · p. 10, §7.5 · p. 10

7.4 Use Case #2: LLM Inference

• At 1.41× oversubscription, Ember-static generates tokens 4.7× faster on the RTX 5080 and 8–13× faster on the A40 and H200 than UVM.Ember-adaptive reaches 3–10×; stock UVM and prefetching fall to 0.2–0.5 tokens/s, while no strategy exceeds 3 tokens/s.Source: Figure 13 · p. 10, §7.4 · p. 10

7.6 GPU Classes & Coherent Interconnects

• On GH200, Ember-static’s geometric-mean gain over UVM remains near 3.5× at both tested footprints, while UVM-mapped is faster for workloads that read data once.For reused data, Ember runs k-means in 3.1 s versus 14.2 s for UVM-mapped, and generates 128 LLM tokens in 62 s versus 211 s.Source: Figure 14 · p. 11, §7.6 · p. 11

7.7 Comparison with Profiled Placement

• Against HELM, Ember-adaptive is faster across five benchmarks by a mean of 7.9× including profiling, and has a mean kernel-time advantage of 2.1×.HELM profiling and selection take 39–1449 s; Ember-adaptive is slower on bfs and gramschm when HELM’s placement choice fits the workload.Source: Figure 15 · p. 11, §7.7 · p. 11

7.8 Ember-adaptive vs Ember-static

• Ember-adaptive is 1.3–4× faster at low reuse and up to 3.2× slower than Ember-static when pages are reused at least 128 times.Ember-adaptive moves 6.8–24.4 GB within a pass and 17.7–65.2 GB across passes, while Ember-static migrates its 16 GB prefix once. When data fits in GPU memory, Ember-adaptive runs 1.2–1.8× slower; the recommended variant depends on reuse and hot-set behavior.Source: Figure 16 · p. 11, §7.8 · pp. 11–12

8 Limitations and Future Work

• Ember can place only `cudaMallocManaged` allocations; library-internal buffers consume device memory but remain invisible to both variants.Ember-static can also have its precomputed prefix invalidated by later library allocations, whereas Ember-adaptive stops promoting when a library takes memory.Source: §8 · p. 12
• Because Ember infers coldness from demand before promotion, it can mistakenly reclaim a still-hot region and move it twice.The driver’s access-bit buffer could replace inference with measurement for some ranges, but is kernel-only, limited to eight ranges, and disabled by default on Blackwell.Source: §8 · p. 12
• Ember-adaptive requires bare-metal access to load a kernel module, and the module must be rebased for each open-source driver release.A release that changes internal interfaces can make a mechanism harder to implement or unavailable; the GH200 cloud VM therefore reports Ember-static alone.Source: §8 · p. 12
• The hybrid address space is not CUDA-specific, but Ember-adaptive additionally requires a non-faulting signal of remote access frequency.The passage names AMD ROCm as another stack with driver-controlled migration and host mapping.Source: §8 · p. 12
• Ember currently places data between one GPU and host memory; multi-GPU use would add peer memory as a third tier and require GPU-specific notification attribution.The authors leave multi-GPU placement for future work.Source: §8 · p. 12

9 Related Work

• UVM optimizations use hardware changes, eviction policies, adaptive runtimes, or migration heuristics, but retain reclamation traffic on the migration path.Ember leaves the cold set mapped in host memory while adjusting the GPU-resident set.Source: §9 · p. 12
• Prior placement systems use offline analysis, profiling, access thresholds, instrumentation, concurrent faults, or sampled pages; Ember defaults to mapping and promotes regions when GPU counters indicate demand.Ember requires no offline profiling, additional faults, or sampling.Source: §9 · p. 12
• Application-level placement systems target settings such as graph processing, DNN training, and LLM inference with known dataflow, whereas Ember keeps applications intact and needs no such knowledge [37] [49] [53].Source: §9 · p. 12
• CPU tiering ideas cannot directly transfer to GPU tiering because GPU counters observe only host memory and the slower tier sits 15–150× below GPU memory.Ember instead aggregates counter notifications into decaying per-region demand to infer when resident data turns cold; multi-GPU placement is described as orthogonal.Source: §9 · p. 13

10 Conclusion

• Ember splits allocations into regions, initially maps them into host memory, and migrates a region only when GPU counters report it as hot.It adjusts placement at runtime and proactively reclaims memory to keep eviction off the critical path.Source: §10 · p. 13
• Across workload classes, GPUs, and interconnects, Ember outperforms existing UVM placement strategies and optimized profile-guided frameworks without requiring a profiling run.Source: §10 · p. 13
• Ember preserves the UVM memory and programming model, and the artifact will include detailed experimental results and integration costs.Source: §10 · p. 13

A Detailed Experimental Results

• Table 4 reports application runtimes in seconds on three PCIe GPUs, comparing UVM, prefetch, UVM-mapped, Ember-static, and Ember-adaptive at 0.5× and approximately 1.5× device-memory input sizes.Inputs at approximately 1.5× are 22 GiB on RTX 5080, 24 GiB on A40, and 16 GiB on H200; >1800 marks a capped run.Source: Table 4 · p. 16, A Detailed Experimental Results · p. 16
• Table 6 gives LLM generation in tokens per second for 128 tokens after a 512-token prompt, with each GPU capped at 16 GB.Table 5 separately reports GH200 runtimes for applications at 48 and 142 GiB and models at 0.41× and 1.37× of 96 GiB.Source: Table 6 · p. 16, Table 5 · p. 16, A Detailed Experimental Results · p. 16
• Tables 7 and 8 report SSB query runtimes on Crystal at 0.5× and 1.5× device memory, respectively, with each GPU capped at 16 GiB.Both compare UVM, UVM-mapped, Ember-static, and Ember-adaptive; q41–q43 exceed the RTX 5080 host’s memory.Source: A Detailed Experimental Results · p. 16, Table 7 · p. 17, Table 8 · p. 17
• Table 9 compares HELM with Ember-adaptive on RTX 5080, reporting policy kernel time and total time including profiling, metrics, and selection.Table 10 reports GH200 moving-hot-set kernel time over eight phases and one-time placement cost as the share of a 14 GiB window replaced per phase varies.Source: A Detailed Experimental Results · p. 16, Table 9 · p. 17, Table 10 · p. 17

• Table 11 lists the application-specific lines added to run each application under Ember-static.Source: B Integration Cost · p. 17, Table 11 · p. 17