Characterizing CPU-GPU Memory Sharing over Unified Memory Architecture

Motivation

• Applications often incur unnecessary copy overhead or use UMA features incorrectly, causing data inconsistency or crashes.Examples include redundant weight-loading and I/O copies in PyTorch, OpenVINO, vLLM, and llama.cpp, and incorrect CPU–GPU access patterns in MoltenVK.Source: §1 · p. 1
• UMA complexity spans hardware differences, opaque software settings, and opposing data-plane bandwidth versus control-plane synchronization needs, so no single sharing recipe is universally optimal.Platforms vary in interconnect, memory-side cache, and GPU-to-CPU coherence; page size, cache policy, and coherence settings may be hidden in drivers and runtimes.Source: §1 · p. 1

Study design

• The study spans application semantics, memory-sharing implementations, and platform properties, evaluating four x86/ARM platforms with Intel/NVIDIA GPUs.Sharing operations are decomposed into configurable sending and receiving stages, with page size, cache policy, and coherence choices used to identify correct and fast implementations.Source: §1 · pp. 1–2
• The analysis covers representative applications and platforms, yielding correct implementations for each platform and four performance findings.Its application taxonomy distinguishes data-plane transmission from control-plane coordination.Source: §1 · pp. 1–2

Findings

• For large regions, 4 KB pages inflate setup time by nearly 100× over 2 MB pages and degrade bandwidth through TLB pressure; for small regions, 2 MB allocations waste memory and are up to 5× slower on Lunar and Orin.The authors recommend enabling Transparent Huge Pages and prefaulting the region at setup.Source: §1 · p. 2
• On Orin, uncached memory makes direct access correct without code changes but inflates single-threaded CPU latency by up to 300× and reduces L2-optimized GEMM performance by 16%.The authors recommend reserving uncached memory for cases where neither code modification nor coherence support is feasible.Source: §1 · p. 2

Findings

• Under high concurrency, control-plane coordination is bottlenecked by contention rather than access latency; block-level barriers handle up to 1,000 threads, while grid-level barriers reduce contention beyond that threshold.Grid-level barriers add deadlock-avoidance complexity, and a single-thread kernel remains a safe fallback.Source: §3 · p. 2
• GPU-to-CPU cache coherence enables correct CPU access to write-back memory without application-level changes, with negligible overhead on MSC-enabled platforms but GPU write time 2.07× higher on Arrow without an MSC.The passage also reports that uncached memory has zero cost on MSC-enabled platforms.Source: §3 · p. 2

Library and extensibility

• libumsh selects correct, efficient implementations through high-level sender-receiver operations based on platform and application requirements, and an AI-assisted workflow ports it to NVIDIA DGX Spark with two-way coherence.The paper presents the library as its high-level API contribution alongside a systematic methodology and characterization of four UMA platforms.Source: §3 · p. 2

Case study

• Across nine real-world systems, libumsh reduces data-plane transmission time by 16–75% and control-plane coordination overhead by 7–90%.The data-plane cases include model loading, image preprocessing, output sampling, and point-cloud processing; control-plane cases include hybrid inference, model offloading, GPU filesystem operations, and file locking.Source: §3 · p. 2

2.1 Unified Memory Architecture

• UMA gives CPU and GPU direct access to shared physical memory, whereas discrete memory uses separate spaces and requires explicit copies for communication.Direct access avoids data movement, but correctness and performance require understanding the hardware and coordinating the processors.Source: §2.1 · p. 3
• UMA platforms use Ring or NoC interconnects, with differing cache and coherence arrangements; the four studied platforms provide only one-way coherence, from GPU-initiated accesses to CPU caches.NoC platforms may include a memory-side cache, while GPU-to-CPU coherence overhead depends on its implementation and cache organization.Source: §2.1 · p. 3

2.2 CPU-GPU Communication

• Data-plane transmission moves producer-consumer data between CPU and GPU at task boundaries, while control-plane coordination exchanges small, frequent state during GPU tasks.Control patterns include asymmetric waits for task dependencies and symmetric barriers or locks for mutual coordination; the communication microbenchmarks do not model application-level caching or reuse.Source: §2.2 · pp. 3–4

2.3 Memory Sharing

• Data copying transfers data into a separate receiver region, adding copy overhead and duplicate memory; direct access uses one shared region and avoids both costs.Direct access requires a physically shared region, which UMA provides.Source: §2.3 · p. 4
• Many inference frameworks still copy data on UMA, while direct access can cause inconsistency or crashes when cache flushes or memory barriers are missing.These problems reflect systems designed for discrete memory and variation across UMA hardware, software interfaces, and performance, which complicate correct and efficient sharing.Source: §2.3 · p. 4

3 Target Platforms

• The four-platform study covers one-way coherence, with Intel systems spanning Ring, NoC without MSC, and NoC with MSC, plus NVIDIA Orin’s ARM-based NoC-with-MSC design.The Intel subset shares a driver/runtime ecosystem, while Orin broadens the study beyond x86.Source: §3 · p. 4

3.1 Ring Interconnect: Raptor

• Raptor combines a Ring interconnect, shared 24 MB LLC, and one-way coherence; its GPU private caches are not coherent with the CPU.The GPU and memory controller are agents on the ring, enabling coherent access to the shared LLC and memory.Source: §3.1 · pp. 4–5

3.2 NoC Interconnect without MSC: Arrow

• Arrow uses a NoC without an MSC: GPU-initiated accesses can maintain consistency with CPU caches, but CPU-initiated accesses cannot do so in reverse.Its CPU LLC is not directly accessible to the GPU as a shared cache tier.Source: §3.2 · p. 5

3.3 NoC Interconnect with MSC: Lunar and Orin

• Lunar and Orin use NoC interconnects with MSCs and one-way coherence; Lunar has an 8 MB MSC, while Orin has a 4 MB MSC.Orin adds an ARM-based platform to the analysis; its MSC is intended to improve memory performance.Source: §3.3 · p. 5

4.1 Correctness of Memory Sharing

• Correctness tests model sending as Bind–Write–Unbind and receiving as Bind–Read–Unbind, testing candidate combinations in both directions with a persistent GPU kernel.The tests assess visibility for a single sender and receiver, not write-write contention, and provide empirical evidence only for tested platforms, configurations, and sharing models—not formal guarantees for general CPU–GPU shared memory.Source: §4.1 · p. 5

4.1.1 Bind and Unbind

• Page size affects performance, but cache policy and data coherence determine correctness by governing which write and read implementations are needed.Bind sets page tables for both processors; CPU and GPU page sizes must satisfy alignment requirements, and unbinding reverses Bind and releases the region.Source: §4.1.1 · pp. 5–6

4.1.2 Write

• Write-Alone works with uncached memory or with write-back memory when receiver-side CC is enabled; Write-then-Flush makes writes visible through cache flushing, while Write-Bypass avoids private caches.GPU cache-bypass writes are supported on Arrow and Lunar, but not Raptor or Orin; GPU flush support and CPU-issued external flush behavior also differ across platforms.Source: §4.1.2 · pp. 6–7

4.1.3 Read

• Read-Alone works with uncached memory or with write-back memory when sender-side CC is enabled; Read-after-Invalidation and Read-Bypass provide alternatives for cached memory.Read-after-Invalidation can use invalidation instructions or flush mechanisms; the ARM CPU supports a dedicated invalidation instruction, dc ivac.Source: §4.1.3 · p. 7

4.1.4 Acquire and Release

• UMA platforms can implement Acquire and Release with atomic operations directly on shared memory, avoiding CPU-mediated delegation for supported CPU–GPU atomics.The paper reports that its lock implementations support these operations with CPU mappings configured as write-back; system-scope atomicity remains platform-dependent.Source: §4.1.4 · p. 7

4.2.1 Memory Binding

• For large regions exceeding 32 KB, 2 MB pages avoid the substantial binding-time overhead of default 4 KB pages; for small regions, 4 KB pages are faster on Lunar and Orin, by about 5× on Orin.Transparent Huge Pages (THP) provide optimal performance in most scenarios; the paper recommends THP and initializing the full region with memset, while noting compaction latency spikes.Source: §4.2.1 · pp. 7–8

4.2.2 From CPU to GPU (CPU⇒GPU)

• UC memory (WA/UC) raises CPU write time by up to 300×; DMA-based WB reduces SSD-transfer time versus mmap-and-copy by 70% on average and up to 98% for 512-byte transfers on Arrow.WB (WA/WB) uses the CPU private cache for write combining and achieves the lowest write time across all four platforms.Source: §4.2.2 · p. 8

4.2.3 From GPU to CPU (GPU⇒CPU)

• On Arrow, GPU write time with CC is 2.07× that without CC, whereas Lunar’s memory-side cache (MSC) reduces the coherence-related DRAM traffic overhead.On Orin, CPU invalidation uses a flush instruction because its invalidation instruction is much slower, despite skipping writeback.Source: §4.2.3 · pp. 9–10
• For GPU-to-CPU communication, Lunar retains CC with CPU Read-Alone, Arrow disables CC and uses CPU Read-after-Inval, and Raptor lacks intra-kernel communication.On Orin, the recommended pairing is GPU Write-Alone on UC memory and CPU Read-Alone on WB memory.Source: §4.2.3 · p. 10

4.2.4 High Contention

• Grid-level barriers outperform block-level barriers above 1,000 threads, while a separate one-thread kernel performs worse than grid-level barriers but avoids deadlock risks and special instructions.Block-level barriers scale well up to 1,000 threads; grid-level barriers have fixed overhead and require Cooperative Groups or deadlock-avoiding atomic operations.Source: §4.2.4 · p. 10
• Block-level barriers mitigate contention below 1,000 threads, while grid-level barriers are preferred at higher concurrency despite greater implementation complexity.High GPU concurrency sharply degrades control-plane performance; use the kernel-level approach only when source modification is infeasible.Source: §4.2.4 · p. 10

5 libumsh and Platform Extension

• libumsh builds high-level wait/set helpers for asymmetric synchronization and lock/unlock helpers for symmetric synchronization from six core operations.The core operations are bind, unbind, write, read, acquire, and release.Source: §5 · p. 10
• Applications provide transfer direction, I/O origin, and kernel boundary; libumsh selects default Read/Write implementations and cache/coherence settings for the compile-time platform.Its backend handles platform-specific binding and cache maintenance.Source: §5 · p. 10

5.1 Porting Workflow

• The porting workflow locates platform controls, validates them with adapted correctness tests and benchmarks, then maps them into the backend, patching a driver only when needed.The workflow uses AI-assisted inspection to find relevant APIs and internal behaviors.Source: §5.1 · pp. 10–11

5.2 DGX Spark Extension

• DGX Spark extends the abstractions to two-way coherence; its backend maps validated controls to libumsh, with about 100 lines of backend code and no Spark-specific application changes.The extension identified controls through driver/runtime inspection and exposed controls lacking public interfaces with a roughly 200-line kernel patch.Source: §5.2 · p. 11
• On Spark, the recommendations retain THP for binding, favor UC for large streaming GPU writes, and use coherent WB for GPU reads and small control flags.The reported performance comparisons and polling options are described separately in the throughput result.Source: §5.2 · p. 11
• GPU Read-Alone and Write-Alone on WB memory with two-way coherence lose about 8% and up to 27% throughput, respectively, versus their UC counterparts.For high polling contention, kernel-level throttling has lower fixed overhead than on Orin; block-level barriers remain an alternative when polling stays in the original kernel.Source: §5.2 · p. 11

6.1.1 CPU-to-GPU Transfer

• Model loading cut time 75% and memory use in half; image preprocessing reduced total time 16% using WB with GPU-side CC.THP reduced model-loading time by an additional 5%; naive UC increased image-preprocessing write time 20×.Source: §6.1.1 · pp. 11–12

6.1.2 GPU-to-CPU Transfer

• 18% lower end-to-end time for LLM output sampling and 34% lower transfer latency for sparse point-cloud processing versus copy baselines.The point-cloud configuration disables GPU-side CC and invalidates the CPU cache before reading.Source: Figure 10 · p. 12, §6.1.2 · p. 12

6.2.1 Asymmetric Synchronization

• Wait/set signaling reduced hybrid-inference operator time by 4.9% on Lunar and 45.4% on Orin versus EdgeNN.Native asynchronous runtime events increased latency to as much as 1.1× baseline on Lunar.Source: Figure 11 · p. 12, §6.2.1 · p. 12
• Accelerate’s 10-token prefill TTFT fell 11%; Demand Layering’s grid barriers cut execution time 14%, while block barriers raised it to 1.8× baseline.Accelerate’s kernel-level coordination supports a black-box integration; Demand Layering requires deadlock avoidance for grid-level barriers.Source: Figure 12 · p. 12, §6.2.1 · p. 13
• Grid-level barriers reduced GPUfs synchronization time by about 7% at high block counts on Orin.The comparison addresses contention in kernel-wide file operations.Source: Figure 13 · p. 13, §6.2.1 · p. 13

6.2.2 Symmetric Synchronization

• A direct-access CAS lock reduced CPU-heavy file-lock overhead by about 90% versus delegation under low contention on Orin.The comparison measures time per lock operation minus critical-section time.Source: Figure 13 · p. 13, §6.2.2 · p. 13

6.3 End-to-End Evaluation

• 9.0–15.9% lower estimated per-request latency across four platforms, with model loading accounting for most savings.On Arrow, latency fell 14.0%, from 43.78 s to 37.65 s.Source: Figure 14 · p. 13, §6.3 · p. 13

• OS abstractions hide page size, device CC status, and cache policy; THP may silently fall back to 4 KB pages under fragmentation, causing unpredictable performance.libumsh mitigates worst cases by prefaulting THP regions and patching the driver to expose hidden parameters, but lack of visibility remains a systemic obstacle.Source: §7 · pp. 13–14
• Hardware coherence reduces software burden, but platform-specific analysis remains necessary: AMD omits GPU L0/L1 caches, and NVIDIA recommends benchmarking for potential overheads.The DGX Spark extension illustrates the need to validate cache coverage and performance even with a wider coherence domain.Source: §7 · p. 14
• Applying the methodology to NPUs and ASICs still requires access to their hardware controls and validation of cache visibility and synchronization behavior.Most ISAs and runtimes remain closed and undocumented; the AI-assisted porting workflow narrows the search for controls.Source: §7 · p. 14

• Prior UMA studies often isolate architectural features or target specific platforms; cross-platform analyses may omit correctness or cover narrow application scenarios.The paper positions its methodology as spanning correctness and performance across diverse platforms and use cases.Source: §8 · p. 14
• Prior CPU-GPU communication work covers unified-memory transfers, scenario-specific transmission, hybrid computing, scheduling, and memory-space abstractions, but these efforts are largely conducted in isolation.Examples include Kokkos abstractions and CUDA vendor-specific allocation and registration APIs.Source: §8 · p. 14
• The paper analyzes application-level CPU-GPU communication requirements and provides a library that automatically selects implementations across platforms.This addresses the isolation of prior efforts without claiming that the cited tools lack their own use cases.Source: §8 · p. 14

9 Conclusion

• Correctness and performance depend on how page size, cache policy, coherence, MSC, and contention interact across four representative UMA platforms.The methodology decomposes sharing into memory binding and paired write and read operations; direct access alone does not ensure correct and efficient sharing.Source: §9 · p. 14
• libumsh selects platform-specific sender-receiver implementations according to application requirements; applications to nine real-world systems show benefits in both data-plane transmission and control-plane coordination.The DGX Spark extension applies the same methodology and high-level APIs to a platform with two-way coherence.Source: §9 · p. 14

Appendix A Artifact Appendix

• The artifact includes libumsh source code, correctness tests, microbenchmarks, reference measurements, plotting scripts, and build and evaluation instructions.The repository provides instructions for evaluation on supported Intel and NVIDIA UMA platforms.Source: Appendix A · p. 20

• The appendix measurements and case studies exercise the libumsh backend added for NVIDIA DGX Spark; platform details and main conclusions are in §5.2.Source: Appendix B · p. 20

Bind

• THP is a practical default for shared regions: it avoids explicit 2 MB mappings’ fixed setup cost for small regions without penalizing large ones.Small regions favor 4 KB pages or THP, while large streaming regions amortize the larger mapping setup.Source: Appendix B.1 · p. 20, Figure B.1 · p. 20

Write and Read

• Up to 27% GPU Write throughput is lost with WA/WB+CC relative to WA/UC; RA/WB+CC loses about 8% GPU Read throughput relative to RA/UC on the largest transfers.Source: Appendix B.1 · p. 20, Figure B.2 · p. 20
• WB keeps CPU Write efficient for CPU-generated data; explicit fine-grained flushing or invalidation is needed only when software must maintain visibility without coherence.Spark exposes WA/WB, WF/Fine, RA/WB, and RI/Fine CPU paths.Source: Appendix B.1 · p. 20
• For SSD-backed CPU-to-GPU transfers, Write-Bypass outperforms mmap-and-copy by avoiding page-cache staging and an extra memory copy.Source: Appendix B.1 · p. 20, Figure B.3 · p. 21

Implications

• For MB-scale streaming, GPU writes should use non-coherent UC; coherent WB remains practical for GPU reads and small GPU-generated flags.Inter-kernel streaming transfers favor WUC, while intra-kernel flag passing favors the coherent WB path because two-way coherence has negligible cost at that scale.Source: Appendix B.1 · pp. 20–21, Table B.1 · p. 21

Two-way coherence

• With WB, two-way CC becomes noticeably slower than one-way CC above 32 MB, while the policies are indistinguishable under UC.The authors hypothesize the WB penalty reflects GPU L2 cache-line registration, ownership, and eviction traffic under the AMBA CHI protocol; UC bypasses L2.Source: Appendix B.1 · p. 21, Figure B.4 · p. 21

GPU contention

• Under GPU polling contention, grid-level barriers nearly eliminate concurrent polling but require stronger synchronization; block-level barriers have low fixed overhead.On Spark, kernel-level barriers are preferred for high-contention polling, while block-level barriers suit cases where a separate polling kernel is undesirable.Source: Figure B.5 · p. 21, Appendix B.1 · p. 22

Portability

• The case studies run through platform-independent libumsh interfaces without Spark-specific application code once the backend exposes the required operations.Source: Appendix B.2 · p. 22

Hybrid inference

• 14.7% lower normalized operator time on average results from replacing EdgeNN’s runtime coordination with libumsh direct-access wait/set signaling.The gain is smaller than on Orin because Spark’s lower kernel launch overhead leaves less control-plane overhead for the optimization to remove.Source: Appendix B.2 · p. 22

GPUfs and file locking

• Up to 5.1% lower synchronization time comes from grid-level barriers at high block counts; direct-access CAS locking cuts overhead by about 85%, 37%, and 26% in CPU-heavy, GPU-heavy, and balanced configurations.These Spark results carry over Orin’s GPUfs throttling and file-locking optimizations.Source: Appendix B.2 · p. 22