PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoUVM: Automated Prefetching Framework for LLMs under UVM Oversubscription

Mao Lin, Hui Feng, Xianzhong Ding, Guilherme Cox, Qian Wang, Hyeran Jeon

arXiv:2609.06172v1cs.OScs.PF

TL;DR

LLM workloads can exceed commodity GPU memory, while UVM’s coarse object-level view misses tensor behavior inside deep-learning frameworks. AUTOUVM profiles kernel–tensor access and adapts policy-driven, fine-grained prefetching; across ten LLMs, it reports speedups over baseline UVM and prior prefetchers.

  • Problem

    Framework allocators hide heterogeneous tensor access within large managed objects, limiting existing UVM prefetchers’ ability to match migrations to individual kernels.

    Source: §I · p. 1, B. Limitations of Existing UVM… · p. 3

  • Method

    AUTOUVM profiles kernel–tensor access offline and uses a roofline-inspired policy with an adaptive online executor to issue fine-grained prefetches without model-code changes.

    Source: §I · pp. 1–2, 1) Prefetch Engine · p. 5

  • Results

    Across 10 LLMs, AUTOUVM achieves an average 3.1× speedup over baseline UVM, outperforms DeepUM by 1.9×, and improves over object-level prefetchers by up to 4.7×.

    Source: §VII · p. 8

  • Takeaways & Limitations

    AUTOUVM offers a practical, programmer-agnostic path for memory-intensive LLM workloads on commodity GPUs.

    Source: §VII · p. 8

    Existing object-level prefetchers assume similar access behavior within managed allocations, but LLM allocations can contain tensors with heterogeneous access patterns and lifetimes.

    Source: B. Limitations of Existing UVM… · p. 3

Abstract

from arXiv · show

Large language models (LLMs) increasingly exceed the memory capacity of commodity GPUs, making memory oversubscription common in practical deployments. NVIDIA Unified Virtual Memory (UVM) provides transparent access to host memory, but its page-fault-driven migrations introduce severe performance overhead. While UVM exposes primitives (e.g., prefetching and placement hints) to mitigate these costs, they require low-level CUDA modifications, limiting their applicability for most LLM users. Meanwhile, existing UVM optimizations operate at coarse managed-object granularity and fail to capture deep learning frameworks' internal tensor-level memory behavior, leading to excessive data movement and CPU-GPU interconnect bottlenecks. We propose AutoUVM, an automated, framework-aware UVM prefetching system for efficient LLM execution under memory oversubscription. AutoUVM bridges the semantic gap between deep learning frameworks and UVM by exposing tensor-level access information and enabling policy-driven prefetching at fine granularity. Implemented as a transparent extension, AutoUVM requires no changes to model code and dynamically adapts to runtime memory pressure. We instantiate AutoUVM with a roofline-inspired policy to identify performance-critical data transfers. Across ten LLMs, AutoUVM achieves an average 3.1x speedup over baseline UVM and consistently surpasses the best-performing prior UVM prefetcher by 1.9x, with improvements of up to 4.7x over object-level prefetchers, while significantly reducing page faults.

I. INTRODUCTION

AUTOUVM addresses the mismatch between framework-managed tensor memory and UVM’s coarse object-level view. It profiles kernel–tensor access and uses policy-driven, adaptive tensor prefetching; the paper reports speedups over baseline and prior UVM prefetchers.

I. INTRODUCTION

  • Framework allocators hide tensor-level accesses inside large managed pools, preventing UVM drivers from accurately prefetching tensors and limiting other UVM optimizations.The mismatch also affects page placement, proactive eviction, and offloading.Source: §I · p. 1
  • AUTOUVM uses a single offline profiling pass to recover kernel–tensor access relationships, then applies policy-driven, fine-grained tensor prefetching without model-code changes.A lightweight online executor adapts decisions as memory pressure and eviction behavior change.Source: §I · pp. 1–2
  • Tensor-aware profiling and roofline-inspired analysis are designed to reduce unnecessary migrations, mitigate CPU–GPU interconnect bottlenecks, and reduce page faults.The paper instantiates the policy to distinguish interconnect-, GPU-memory-, and compute-bound kernels.Source: §I · p. 2
  • Across ten LLMs, AUTOUVM achieves an average 3.1× speedup over baseline UVM, outperforms DeepUM by 1.9×, and improves over object-level prefetchers by up to 4.7×.The authors also report significantly reduced page faults.Source: §I · p. 2

II. BACKGROUND

UVM supports shared CPU–GPU addressing and oversubscription, with page migration and eviction when data is not resident on the GPU. PyTorch’s caching allocator pools memory and subdivides it among tensors, leaving low-level systems with limited visibility into framework-managed reuse.

A. Unified Virtual Memory

  • UVM migrates pages on demand when the GPU first accesses them, while page-fault handling introduces substantial latency; prefetch and placement-advice APIs can enable proactive migration.The driver evicts least recently used pages to host memory when GPU memory is full.Source: A. Unified Virtual Memory · p. 2
  • The oversubscription factor is managed UVM memory divided by GPU capacity; values above 1.0 require frequent evictions.A factor of 1.5 on an 80 GB GPU corresponds to 120 GB of managed memory.Source: A. Unified Virtual Memory · p. 2

B. Memory Management of Modern DL Frameworks

  • PyTorch’s caching allocator obtains large blocks and subdivides them for tensors, retaining unused segments for reuse rather than freeing them individually.After warm-up, most tensor allocations are served from cached blocks.Source: B. Memory Management of Modern DL… · p. 2
  • Framework metadata tracks tensor boundaries and reuse inside pools, but the GPU runtime sees only a small number of large allocations.The pools contain separate large and small allocations, with multiple preallocated blocks.Source: B. Memory Management of Modern DL… · p. 2, Figure 1 · p. 2

III. MOTIVATION

UVM lets memory-intensive workloads run beyond GPU capacity, but oversubscription can make page-fault-driven migration costly. Existing object-level prefetching misses heterogeneous tensor access patterns, resulting in substantial unused transfers.

A. High Memory Demand Calls for UVM

  • On an RTX 3060, UVM lets Qwen1.5 inference run at sequence lengths beyond 10K, where PyTorch without UVM fails with an out-of-memory error.With UVM, memory usage can exceed the GPU’s 12 GB capacity.Source: A. High Memory Demand Calls for UVM · p. 3
  • UVM performance can degrade sharply under oversubscription because page-fault-driven, on-demand migrations are costly.The paper identifies effective prefetching as essential to unlocking UVM’s potential.Source: A. High Memory Demand Calls for UVM · p. 3

B. Limitations of Existing UVM Prefetching

  • Existing prefetchers operate on managed objects and assume their contents share access behavior, although one allocation can contain tensors with heterogeneous patterns and lifetimes.Without tensor-level visibility, they cannot capture individual kernels’ fine-grained memory needs.Source: B. Limitations of Existing UVM… · p. 3
  • Useless object-level prefetch is 6× more than useful prefetch on average and up to 23× more, wasting interconnect bandwidth and scarce GPU memory.Figure 3 presents used and unused prefetch sizes across LLM models.Source: B. Limitations of Existing UVM… · p. 3, Figure 3 · p. 3

C. Heterogeneous Migration Sensitivity Across Kernels

Kernels vary in sensitivity to CPU–GPU migration latency, and some operators run near or below the interconnect roofline. AUTOUVM’s design goals therefore include selective prefetching alongside tensor awareness, runtime adaptivity, and framework transparency.

C. Heterogeneous Migration Sensitivity Across Kernels

  • On an NVIDIA RTX 3060, Qwen1.5 elementwise kernels are memory-bound, GEMV/GEMM approach the compute roof, and several operators fall near or below the lower CPU–GPU interconnect roofline, indicating substantial throughput degradation during UVM page migration.Source: C. Heterogeneous Migration Sensitivity… · p. 3
  • Kernels differ in sensitivity to migration latency, so prefetching should prioritize those most affected by interconnect bottlenecks rather than blindly prefetching all tensors.LLM workloads contain memory- and compute-bound kernels that may be invoked thousands of times during inference.Source: C. Heterogeneous Migration Sensitivity… · p. 3

A. Design Goals

  • AUTOUVM’s design goals are tensor-granularity prefetching, selectivity for migration-sensitive kernels, adaptation to memory pressure and eviction, and transparency to model code.The framework is intended to integrate with systems such as PyTorch without model-code modifications.Source: A. Design Goals · p. 4

B. Key Ideas

AUTOUVM combines tensor-level visibility with selective prefetching, using offline kernel–tensor profiles and runtime adaptation to changing memory pressure.

B. Key Ideas

  • AUTOUVM’s roofline-inspired policy prioritizes CGI-bound kernels for prefetching while avoiding unnecessary migration for compute-bound and GPU-memory-bound kernels.It also identifies tensors reused across consecutive kernels and marks high-reuse tensors for GPU residency.Source: B. Key Ideas · p. 4, 2) UVM Analyzer · p. 4, 1) Execution Tracer · p. 4
  • Because framework allocators pack unrelated tensors into shared regions, AUTOUVM identifies and migrates only tensors required by each kernel.Stable tensor identities use creation-site call stacks and a global allocation counter, remaining robust to address randomization.Source: B. Key Ideas · p. 4

C. System Overview

  • The Offline Profiler records kernel–tensor access patterns and runtime statistics, then stores profiles for the Online Executor to use during inference.The executor monitors runtime behavior and adjusts prefetching as oversubscription pressure changes.Source: C. System Overview · p. 4

1) Execution Tracer:

  • Kernel–tensor access tracing avoids relying on computation graphs, whose operator sequences can differ across runs, and identifies tensors needed by the next kernel.The approach addresses both dynamic computation graphs and address randomization across runs.Source: 1) Execution Tracer · p. 4
  • AUTOUVM tracks tensors across runs with creation-site call stacks and a monotonic allocation counter; on a runtime mismatch, it disables prefetch for affected tensors.The caching allocator records the identifiers through a lightweight hook.Source: 1) Execution Tracer · p. 4

2) UVM Analyzer:

Execution profiles consolidate kernel-to-tensor mappings and metadata, including tensor-affinity hints used to guide later prefetching and pinning.

2) UVM Analyzer:

  • Execution profiles map kernels to the tensors they access and store metadata for later execution.Kernel identities use launch-site call-stack hashes and launch counters; tensor identities use creation-site call-stack hashes and allocation counters.Source: 3) Execution Profiles · p. 4
  • The profiles also store runtime metadata and policy information for use during real execution.These fields accompany the kernel-to-tensor mappings.Source: 3) Execution Profiles · pp. 4–5
  • Tensor-affinity hints in the profiles guide subsequent prefetching and pinning decisions.The passage identifies these hints as part of the execution profiles.Source: 3) Execution Profiles · p. 5

1) Prefetch Engine:

The Prefetch Engine selectively migrates tensors based on kernel behavior, while the Runtime Monitor updates decisions as runtime memory pressure shifts.

1) Prefetch Engine:

  • The Prefetch Engine prefetches only tensors needed by upcoming kernels, targeting CGI-bound kernels while avoiding unnecessary migration for compute- and GPU-memory-bound kernels.It uses cudaMemAdvise to keep tensors reused across consecutive kernels resident on the GPU during their reuse window.Source: 1) Prefetch Engine · p. 5

2) Runtime Monitor:

  • The Runtime Monitor tracks kernel performance and revises kernel classifications and migration decisions when evictions or memory pressure change behavior.Policy updates run in parallel with subsequent kernels, with lightweight timestamping overhead.Source: 2) Runtime Monitor · p. 5

F. Workflow

  • AUTOUVM profiles kernel–tensor relationships offline, then loads the profiles to guide online prefetching and dynamically adjusts decisions using monitored runtime behavior.The workflow stores generated profiles in a database for reuse across executions.Source: F. Workflow · p. 5

G. System Implementation

  • AUTOUVM is integrated into PyTorch as a transparent extension requiring no model-code changes, with allocator hooks recording tensor metadata and APIs injecting prefetch and affinity primitives.The implementation uses NVBit for instruction counts and memory loads, and changes the caching allocator to use cudaMallocManaged.Source: G. System Implementation · p. 5

V. EVALUATION

The evaluation compares AUTOUVM with baseline UVM and three prefetchers across LLMs under controlled oversubscription; AUTOUVM achieves the strongest reported average speedups.

A. Methodology

  • The evaluation covers 10 LLMs on three GPU platforms and controls oversubscription by limiting effective GPU memory with a separate helper process.Page faults are counted using a counter added to the NVIDIA UVM driver module.Source: A. Methodology · p. 5

1) Compared solutions:

  • The comparisons include baseline UVM, object-level SUV and Forest, and UM-block-level DeepUM, which predicts future page accesses from historical fault patterns.Forest is emulated with offline access-pattern analysis, avoiding its online detection overhead.Source: 1) Compared solutions · pp. 5–6

B. Overall Performance

  • 3.1×, 4.7×, 4.1×, and 1.9× average speedups over Baseline, SUV, Forest, and DeepUM, respectively, at 2.0 oversubscription.Figure 7 reports execution time normalized to Baseline, with lower values better.Source: Figure 7 · p. 6, B. Overall Performance · p. 6

C. Page Fault

AUTOUVM issues the fewest page faults among evaluated solutions, and its speedups persist as oversubscription increases and across GPU platforms. The reported gains are greatest under the most constrained memory setting.

C. Page Fault

  • AUTOUVM issues the fewest page faults among evaluated solutions, and page-fault counts strongly correlate with execution time across models and methods.The authors attribute the low fault count to policy-driven tensor prefetching that reserves limited interconnect bandwidth and GPU memory for tensors that need them.Source: C. Page Fault · p. 6

1) Sensitivity of Oversubscription Levels:

2) Sensitivity to GPU Platforms:

  • AUTOUVM achieves speedups of 3.1× on RTX 3060, 2.9× on A100, and 2.7× on H100.The gains decrease slightly on newer GPUs with higher interconnect bandwidth and larger memory capacity, but remain substantial across the evaluated architectures.Source: 2) Sensitivity to GPU Platforms · p. 6

E. Ablation Study

The ablation results favor the full policy-driven design; AUTOUVM also reduces energy, incurs a one-time profiling cost, and remains effective with fused kernels.

E. Ablation Study

  • 3.1× average speedup for T+A+P exceeds T+A at 2.5× and T at 2.0× against the baseline.The variants add look-ahead tensor affinity and then policy-driven selectivity; Figure 11 identifies the full design as the highest-performing variant.Source: E. Ablation Study · pp. 6–7, Figure 11 · p. 7

F. Energy and Power

  • 31% lower total energy on average than baseline UVM and up to 74% lower than object-level prefetchers, despite a 3–4 W average GPU-power increase.The authors attribute the energy reductions to shorter execution time from fewer page faults and unnecessary transfers outweighing the small power increase.Source: F. Energy and Power · p. 7, Figure 12 · p. 7

G. Profiling Overhead

  • 41.5× average profiling cost per inference iteration is a one-time offline expense, reused across subsequent inference runs.It is amortized and does not affect steady-state performance.Source: G. Profiling Overhead · p. 7, Figure 13 · p. 7

H. Generalization to Fused Kernels

VI. RELATED WORK

AUTOUVM differs from prior UVM prefetchers by incorporating deep-learning framework semantics; the conclusion reports speedups across ten LLMs and reduced page faults and energy.

VI. RELATED WORK

  • Prior UVM prefetchers lack modern deep-learning framework tensor semantics, whereas AUTOUVM enables programmer-agnostic software prefetching using those semantics.Capuchin and TERAIO do not target UVM prefetching; SUV and Forest operate at managed-object granularity, and DeepUM predicts future pages from historical fault patterns.Source: §VI · p. 7

VII. CONCLUSION

  • Across 10 LLMs, AUTOUVM averages 3.1× speedup over baseline UVM, beats DeepUM by 1.9×, and improves over object-level prefetchers by up to 4.7×.The conclusion also reports reduced page faults and energy, and describes AUTOUVM as a programmer-agnostic path for memory-intensive LLM workloads on commodity GPUs.Source: §VII · p. 8

Related papers

Loading 2609.06172v1…