PaperlayerView exact revision on arXiv

Source-linked AI summary

EMA: Elastic and Performance Transparent Memory Across GPUs

Yi Xu, Tian Xia, Ion Stoica

arXiv:2609.27040v1cs.DCcs.AIcs.LG

TL;DR

Dynamic memory demands can leave one GPU short while capacity on another remains underused, despite the high-bandwidth links in multi-GPU servers. EMA lets GPUs borrow and reclaim memory through prefetched memory slices, preserving performance transparency for borrowers and lenders. It improves throughput over static partitioning and approaches the throughput of a system with twice the KV-cache capacity, with similar per-token latency.

  • Problem

    Dynamic memory demand can leave one GPU short while other GPUs remain partly idle under fixed assignments, stranding capacity.

    Source: §1 · p. 1

  • Method

    EMA shares memory across GPUs through slices and prefetches data so remote memory performs like local memory while lenders can reclaim borrowed capacity.

    Source: §1 · pp. 1–2

  • Results

    EMA improves throughput by up to 52% over static partitioning, achieves 96% of throughput with 2× KV-cache capacity, and maintains similar per-token latency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    EMA demonstrates elastic, transparent memory sharing for LLM inference, and the paper frames on-demand resource pooling as applicable beyond memory.

    Source: §1 · p. 2

    Slab sizing must balance interconnect utilization against internal fragmentation and sharing efficiency.

    Source: §4.2.1 · p. 7

Abstract

from arXiv · show

Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects. At the same time, workloads such as LLM inference exhibit highly dynamic memory demands, which can cause one GPU to exhaust its local memory while others remain underutilized. This mismatch motivates a model of elastic resource sharing across GPUs. We present EMA, a memory sharing system that allows GPUs within a server to borrow and reclaim memory from each other, forming an elastic pool of capacity. EMA ensures performance transparency for both borrowers and lenders. For borrowers, prefetching hides remote access costs so that applications experience remote and local memory as indistinguishable in performance. For lenders, borrowed resources remain reclaimable on demand, guaranteeing that performance never falls below that of static partitioning. While our design focuses on memory, the same principle naturally extends to other GPU resources. Our evaluation shows that EMA improves individual user throughput by up to 52%, achieves 96% of the throughput of a system provisioned with 2X capacity, and maintains latency similar to the static local baseline.

1 Introduction

EMA addresses stranded capacity under dynamic GPU memory demand by enabling elastic sharing across GPUs while preserving performance transparency for borrowers and lenders. Its memory-slice design uses prefetching to make remote and local memory perform alike, and evaluation reports throughput gains over static partitioning.

1 Introduction

  • Dynamic memory demand can leave one GPU short while other GPUs remain partly idle under fixed assignments, stranding capacity and increasing operational cost.In LLM inference, KV-cache size varies with concurrent queries and their lengths.Source: §1 · p. 1
  • EMA organizes shareable memory into slices and uses application behavior to prefetch data before access, making local and remote memory indistinguishable in application performance.Slices are dedicated to pairs of directly linked GPUs, and EMA sequences transfers to make their duration predictable.Source: §1 · p. 2
  • Up to 52% higher throughput than static partitioning, 96% of throughput with 2× KV-cache capacity, and similar per-token latency.EMA’s reported real-workload results.Source: §1 · p. 2
  • The paper frames on-demand pooling across a multi-GPU node as a principle that may extend beyond memory to other GPU resources.The prototype and evaluation focus on memory for LLM inference.Source: §1 · p. 2
  • EMA introduces performance-transparent memory sharing across GPUs, with a memory-slice abstraction intended to make remote memory behave like local memory for borrowers and preserve lenders’ ability to reclaim their allocation.The paper describes these guarantees as Local–Remote Symmetry and Static–Dynamic Symmetry.Source: §1 · pp. 1–2

2 Background

GPU workloads vary over time, while static partitions can leave capacity idle even as another user faces shortages. The section describes this mismatch in LLM inference, its effects under static allocation, and the challenges of swapping beyond local HBM.

2.1 Fluctuation of GPU Workloads

  • LLM inference memory fluctuates as KV caches grow with concurrent requests and output length, then are released when requests finish.The resulting memory usage is described as volatile and unpredictable.Source: §2.1 · p. 2

2 Background

  • Static provisioning fixes user partitions during extended sharing periods, despite workload variation that can leave capacity unused or insufficient.The section identifies this mismatch as a source of missed utilization and workload-accommodation opportunities.Source: §2 · p. 2, §2.2 · p. 3

2.2 Inefficiency of Static Partition

  • 49.5% of the time, statically allocated memory experiences shortages, while aggregate memory usage stays within total capacity 27.5% of the time.The comparison illustrates how individual static allocations can be insufficient while pooled demand fits within combined capacity.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Performance Transparent Swapping

  • Traditional swapping beyond local HBM incurs high overhead, while Pie uses the layer access window to swap KV-cache data between GPU HBM and CPU DRAM before reuse.The window is available because inference accesses layers sequentially and cyclically.Source: §2.3 · pp. 3–4

3 A Path to Cross-User Memory Sharing

Cross-user memory sharing is framed around two performance guarantees: users retain their static-allocation performance floor, and remote resources perform like local ones. The section identifies preemption and limited workload information exchange as principles for pursuing those guarantees.

3 A Path to Cross-User Memory Sharing

  • Static–Dynamic Symmetry requires each user to perform at least as well as under static allocation, while Local–Remote Symmetry makes remote and local resource access indistinguishable in performance.Together, these objectives define the paper’s double-symmetry goal for sharing.Source: §3 · p. 4
  • Preemptible sharing lets a user reclaim borrowed resources on demand, preserving its original static allocation as a guaranteed lower bound.Applications need safe preemption points; more frequent points allow smoother sharing.Source: §3 · p. 4
  • Sharing also requires applications to expose resource needs and operational characteristics, because access patterns affect transparency and users can benefit unequally.EMA uses latency windows and changes in dynamic memory requirements for coordination, rather than broadly exposing workload information across GPUs.Source: §3 · p. 4
  • EMA divides user memory into private memory dedicated to one user and memory slices contributed by users for sharing.The illustration is described as including both regions in the user memory space.Source: §3 · p. 4

4 Memory Slice: Sharing Between Two GPUs

EMA’s memory slice shares capacity between two directly connected GPUs while aiming to preserve local-like access for borrowers and reclaimability for lenders. Its allocation and swapping designs address dynamic ownership, transfer efficiency, and the costs of maintaining contiguity.

4 Memory Slice: Sharing Between Two GPUs

  • A memory slice combines memory contributed by two directly connected GPUs, giving both users access to shared capacity with performance comparable to local HBM.Segments are assigned to one user on demand, and virtual-to-physical mapping restricts access to the assigned user.Source: §4 · p. 5

4.1 Static-Dynamic Symmetry

  • Static–dynamic symmetry preserves each GPU’s contributed capacity by reclaiming borrowed memory when needed, while dependent local data may also need release.For expansion ratio r and reclaimed remote capacity k, the system treats (r − 1)·k GB of corresponding local data as dependent.Source: §4.1 · pp. 5–6

4.2 Local-Remote Symmetry

  • To sustain transparent transfers across interleaved ownership, EMA uses slabs and groups temporally adjacent layers so data can move in larger chunks.Small transfers can reduce effective bandwidth as copy overhead dominates; EMA’s grouping supports relatively large transfers with high bandwidth utilization.Source: §4.2 · pp. 6–7

4.2.1 Layer-major Allocation

  • Layer-major allocation keeps each GPU’s data relatively contiguous, but dynamic workloads can force single-slab swaps and require a balance between transfer efficiency and fragmentation.Slabs must be large enough to use interconnect bandwidth, yet overly large slabs can transfer unused data and delay reuse of free space.Source: §4.2.1 · p. 7

4.2.2 Slab-major Allocation

  • Slab-major allocation makes logically consecutive layers physically contiguous, enabling bulk transfers, but larger layer groups add buffer overhead and narrow the latency window.EMA checks for swaps at layer-group boundaries; computation and swapping retain the same latency ratio, preserving the original design’s performance transparency.Source: §4.2.2 · p. 7

4.2.3 Layer-group-major Allocation

  • Layer-group-major allocation exploits contiguity across slabs as well as within groups, allowing adjacent slab transfers to be combined, though groups are not swapped together.Unlike slab-major allocation’s fixed kS swap size, this layout’s effective swap size ranges from kS to mkS.Source: §4.2.3 · pp. 7–8

5 Multi-GPU: Aggregating Memory Slices

EMA aggregates peer memory slices into an elastic pool while preserving guaranteed local capacity. It coordinates slice assignment to limit preemption and caps allocations after preemption to reduce repeated disruption under bursty workloads.

5 Multi-GPU: Aggregating Memory Slices

  • Multiple memory slices can coexist independently on a GPU, and each participating peer has a memory-slice process; a process spawns one such process for each directly connected sharing peer.A slice process translates addresses and prepares allocations through prediction and prefetching before the compute process accesses them.Source: §5 · p. 8, §5.1 · p. 8

5.1 Process Address Space

  • Each compute process receives an Elastic Address Space that includes reclaimable peer slices beyond its guaranteed Local Address Space, enabling allocations larger than local capacity while retaining a guaranteed local fallback.The amount of elastic memory available depends on system load and sharing state.Source: §5.1 · p. 8

5.2 Scheduling

  • A local greedy scheduler assigns each request to the slice with the smallest combined utilization of its two peers, aiming to avoid the 100% utilization threshold that triggers preemption.The scheduling problem is described as edge-constrained, unknown-length load balancing; the algorithm has provably good performance bounds.Source: §5.2 · p. 9

5.3 Bursty Workload Preemption Control

  • Static–dynamic symmetry can trigger repeated preemptions when one user’s demand rises and falls as another user reclaims and releases memory.The resulting back-and-forth can repeat indefinitely without a control mechanism.Source: §5.3 · p. 9
  • EMA caps a user’s future allocations at its preempted level for a sliding window, resetting the cap after another lower-level preemption or lifting it when the window expires.The mechanism is intended to reduce disruption under dynamic workloads while allowing aggressive sharing when preemptions are unlikely.Source: §5.3 · p. 9

6 Evaluation

Across the evaluated workloads, EMA improves throughput over static partitioning, with gains depending on workload and memory pressure. The evaluation also examines capacity expansion, latency, symmetry guarantees, and memory-slice configuration.

6.1 Experiment Setup

  • Experiments use 4×A100-40GB and 8×A100-80GB nodes with NVLink, and evaluate ShareGPT and Alpaca on LLaMA and OPT under Poisson-distributed request arrivals.Each GPU runs one model independently; the tested systems include EMA, vLLM, and a modified Pie configuration.Source: §6.1 · p. 9

6.2 Throughput Gains

  • EMA’s throughput gains exceed 50% for selected Alpaca loads, and it achieves 96% of Pie’s performance across settings.Pie represents vLLM with a 2× larger KV cache, while EMA has no guarantee of remote-memory access.Source: §6.2 · p. 10, Figure 7 · p. 10

6.3 Memory Utilization and Performance Breakdown

  • GPU 0, GPU 1, and GPU 2 reach 1.3×, 1.3×, and 1.2× static-version performance, while GPU 3 reaches 0.95×; utilization reaches 200% of physical HBM capacity.The reported 5% overhead for GPU 3 mostly comes from memory allocation.Source: §6.3 · p. 11, Figure 9 · p. 11

6.4 Latency Comparison

  • EMA sustains nearly the same compute latency as memory expansion varies, with a 10% latency increase at large expansions such as 20 GB on A100-80GB.The increase is attributed primarily to larger enabled batch sizes and DMA synchronization and coordination overhead; added capacity still improves overall throughput.Source: §6.4 · p. 11, Figure 10 · p. 11

6.5 Local–Remote Symmetry

  • Local–remote symmetry matters for throughput and latency because computation stalls when accessed data resides remotely and must be swapped in on the critical path.The evaluated variant forgoes prefetching and swaps an entire KV-cache layer on demand.Source: §6.5 · p. 11, Figure 11 · p. 12

6.6 Static–Dynamic Symmetry

  • With an effective reclamation strategy, EMA with static–dynamic symmetry achieves aggregate throughput within −5% to +10% of the version without the guarantee.The comparison also reports individual GPU throughput and preemptions due to out-of-memory conditions and reclamation.Source: §6.6 · p. 12, Figure 12 · p. 12

6.7 Maximizing Performance per Slice

  • Small slabs and layer groups can increase stalls or waiting, while large ones reduce sharing opportunities or consume buffer memory; group size 4 and slab size 10 usually perform consistently well.The study varies slab size from 1 to 100 and layer-group size from 1 to 4 on OPT models evaluated with ShareGPT.Source: §6.7 · p. 12, Figure 13 · p. 12

7 Related Work

Prior systems improve inference efficiency or expand memory, but the cited approaches generally treat memory as fixed or retrieve remote data reactively. EMA instead presents peer memory as a potentially performance-transparent extension with lender reclamation.

General Model Serving Systems

  • General serving systems optimize kernels, scheduling, and batching while treating GPU memory as fixed; capacity pressure can still force queuing, preemption, or model downsizing.Examples include FasterTransformer, LightSeq, TurboTransformers, DeepSpeed-Inference, and Accelerate [14] [57].Source: §7 · p. 12

Recomputation and Offloading/Swapping to CPU

  • Recomputation and CPU or NVMe swapping add memory headroom but retrieve data reactively, leaving a performance difference between local HBM and slower tiers.The cited approaches include recomputation and checkpointing [6] [12] [13] [19] [23] [28] and swapping pipelines [29] [46] [47] [51] [56] [59].Source: §7 · p. 12

Swapping/Sharing Between GPUs

  • Peer-GPU sharing systems such as HUVM, AQUA [55], and Harvest [17] retrieve data reactively, whereas EMA prefetches ahead of use and supports on-demand lender reclamation.The passage characterizes EMA’s goal as making remote HBM performance-transparent during compute.Source: §7 · p. 13

8 Conclusion

The conclusion presents EMA as a system for transparent peer-memory access in dynamic, unbalanced environments. It reports higher throughput without sacrificing latency and frames elastic sharing as a broader direction for multi-GPU resources.

8 Conclusion

  • EMA uses predictable memory access patterns, high-bandwidth interconnects, and real-time information to optimize memory utilization in unbalanced, dynamic environments.Source: §8 · p. 13
  • EMA lets users access peer memory transparently, with performance described as indistinguishable from local memory, and achieves higher throughput without sacrificing latency.Source: §8 · p. 13
  • The conclusion presents objectives and principles for cross-GPU memory sharing and introduces EMA.Source: §8 · p. 13

A Proof of Capacity, Latency and Bandwidth Relationships

The proof frames the capacity requirement for local–remote symmetry through transmission and reception constraints, then derives conditions for those inequalities to hold.

A Proof of Capacity, Latency and Bandwidth Relationships

  • The proof begins from four inequalities governing the capacity requirement for local–remote symmetry in memory sharing.It states that each GPU must transmit and receive the same amount of data in both directions, then introduces bandwidth decomposition equations and substitutes them into the inequalities.Source: A Proof of Capacity, Latency and… · p. 16
  • The substituted inequalities yield constraints, followed by a necessary and sufficient condition for all four inequalities to hold.The passage identifies equality in the condition as the tightest case.Source: A Proof of Capacity, Latency and… · p. 16

B Proof of Per GPU Contribution to Memory Slice

The proof bounds the total per-GPU contribution to a memory slice using the smaller reported latency window, then maximizes that bound over admissible rates.

B Proof of Per GPU Contribution to Memory Slice

C Proof of Scheduling Algorithm

The section models memory-slice allocation as online scheduling and presents Greedy–Least-Loaded as a low-overhead local algorithm with a Θ(m) worst-case gap from an omniscient allocator.

C Proof of Scheduling Algorithm

  • The model gives each GPU pair a shared slice of normalized capacity 1 and treats requests as online jobs with time-varying demand and releases.Positive demand increments are irrevocably assigned to a slice incident to the owning GPU without knowledge of future demands.Source: C Proof of Scheduling Algorithm · p. 17
  • GLL assigns each arriving unit to the least-utilized incident slice, using adjacent per-slice counters and O(m) work per decision.Each slice has normalized capacity 1.Source: C Proof of Scheduling Algorithm · p. 17
  • Greedy–Least-Loaded may saturate up to Θ(m) more slices than an omniscient allocator, while using only local state and O(m) work per decision.The worst case comes from adversarial demand sequences that make early placements coalesce on the same slices.Source: C Proof of Scheduling Algorithm · p. 17

Related papers

Loading 2609.27040v1…