PaperlayerView exact revision on arXiv

Source-linked AI summary

Echo: Merging Host Device Buffers to Avoid Redundant Data Movement on Unified Memory SoCs

Yuheng Zhu, Yanbo Zhao, Jiajia Li, Man-Ki Yoon

arXiv:2609.05635v2cs.DCcs.OS

TL;DR

On UMA platforms, dGPU-style applications copy between separate host and device buffers even though both use the same physical DRAM, and safely eliminating those copies is nontrivial. Echo provides source-level and profile-guided binary paths for unifying buffers while preserving program behavior. Across seven benchmarks on three Jetson platforms, its benefits grow with baseline copy share, reaching up to 7.05× speedup for copy-dominated workloads.

  • Problem

    UMA software often retains dGPU-style separate buffers and copies despite shared physical DRAM, and merging them can alter pointer-dependent behavior or access ordering.

    Source: §1 · p. 1

  • Method

    Echo-SR checks source-level safety conditions before unifying buffers, while Echo-DR uses profile-guided validation to optimize matching patterns in binaries.

    Source: §1 · p. 2

  • Results

    Across seven benchmarks on three Jetson platforms, Echo's benefit grows with copy fraction; copy-dominated Orin workloads reach up to 7.05× speedup.

    Source: §1 · p. 2

  • Takeaways & Limitations

    On five source-available Orin benchmarks, both automatic paths recover at least 99% of the gain from manual optimization, and binary-only applications benefit without recompilation.

    Source: §1 · p. 2

    Echo-DR provides profile-conditional correctness, so later inputs may trigger host accesses or allocation, copy-order, or stream patterns not observed during profiling.

    Source: §5 · p. 11

Abstract

from arXiv · show

GPU applications on unified-memory (UMA) edge platforms often inherit a discrete-GPU memory abstraction in which they allocate one buffer for the CPU, another for the GPU, and copy data between them before and after GPU execution. On UMA hardware these buffers reside in the same physical DRAM pool, so the copies consume bandwidth, time, and energy. Despite the growing adoption of UMA platforms, this pattern remains common because production software stacks, libraries, and samples were written for portability across discrete GPUs. However, removing these copies is not as simple as merging the two buffers, because the original program may rely on the two buffers being distinct, or on the copy itself ordering CPU and GPU accesses. Echo removes these copies only when it preserves the data values and access ordering on which the program depends. It does so along two complementary paths: source-level rewriting and binary deployment. Echo-SR is a compile-time LLVM transformation that proves safety and rewrites accepted pairs in place. Echo-DR is a profile-guided binary optimizer for existing applications that profiles and validates stable allocation/copy patterns and, at runtime, intercepts the matching calls to unify profile-matched pairs while preserving the ordering effects of removed copies. Across seven benchmarks on three NVIDIA Jetson platforms, Echo's benefit grows with the fraction of baseline time spent on copies. Copy-dominated workloads speed up by up to 7.05x, closed-source end-to-end applications speed up by up to 1.40x. On the five source-available Orin benchmarks, both automatic paths recover >=99% of the gain of a manually optimized reference.

1 Introduction

On UMA platforms, dGPU-style software still copies between separate host and device buffers backed by the same DRAM, wasting resources. Echo removes such copies only when it can preserve program behavior, using source-level analysis or profile-guided binary optimization.

1 Introduction

  • On UMA, separate host and device buffers occupy the same physical DRAM, yet legacy copies still consume bandwidth, cycles, and energy.The pattern persists because libraries, samples, and portability-oriented workflows retain the discrete-GPU programming model.Source: §1 · p. 1
  • A survey found the target pattern in 19 of 34 Jetson repositories (56%) and 10 of 11 inference-specific projects (91%).The surveyed software ecosystem includes widely used libraries, manufacturer samples, and developer workflows that retain separate host and device buffers.Source: §1 · p. 1
  • On FCN-ResNet50 on AGX Orin, avoidable copies consume over 30% of per-iteration wall-clock time.The figure also relates Orin speedup to the baseline fraction of runtime spent on copies.Source: §1 · p. 2, Figure 1 · p. 2
  • Removing copies is safe only if programs do not depend on distinct pointers or on copies to order host and GPU accesses.Such dependencies make automatic detection and transformation difficult, especially when copy paths are inside precompiled libraries.Source: §1 · p. 2
  • Echo-SR applies an LLVM transformation after checking safety conditions, while Echo-DR uses profile-guided specialization for binary deployments.The two paths combine source-level analysis and binary deployment to preserve data values and access ordering.Source: §1 · p. 2
  • On AGX Orin, copy-dominated micro-benchmarks achieve up to 7.05× speedup, while closed-source applications improve by 1.17–1.40×; across five source-available Orin benchmarks, both automatic paths closely match manual optimization.Source: §1 · p. 2

2 Background and Motivation

UMA places CPU and GPU allocations in shared physical DRAM, but dGPU-style code can still duplicate buffers and copy between them. The section measures the performance opportunity and surveys how often the target allocation-and-copy pattern appears in Jetson repositories.

2.1 Unified Memory Architecture

  • On UMA, host and device allocations draw from the same physical DRAM, so explicit copies move data between virtual-address regions without crossing a bus.The transfers still consume DMA bandwidth and CPU and GPU cycles; duplicate buffers can also reduce shared-memory capacity available to GPU workloads.Source: §2.1 · p. 3

2.2 CUDA Unified Virtual Memory

  • The targeted UMA overhead comes from duplicated virtual-address regions and explicit copies, unlike discrete-GPU UVM overhead from physical page migration.The cited 34.2% average kernel slowdown under UVM applies to a discrete-GPU benchmark suite, not the copy overhead Echo targets on UMA.Source: §2.2 · p. 3

2.3 Measured Opportunity

  • On FCN-ResNet50 on AGX Orin, Echo-DR reduces iteration time by 31.8% (1.47× speedup), nearly matching a hand-rewritten version.Copies account for 32% of baseline iteration time; the manual version is only 0.029 ms faster than Echo.Source: §2.3 · p. 3, Table 1 · p. 3

2.4 Prevalence of the Target Pattern

  • The candidate pattern appears in 19 of 34 surveyed Jetson repositories (56%), rising to 10 of 11 inference-specific projects (91%).The static scan finds syntactic co-occurrence, not semantic optimizability; each pair still requires safety analysis or guarded runtime profiling.Source: Table 2 · p. 3, §2.4 · p. 4

3 Design of Echo

Echo unifies host and device buffers on UMA only when it can preserve program behavior, using compile-time safety analysis for source and profile-guided optimization for binaries. SR checks sufficient safety conditions; DR validates profiled host-access windows and applies only calibrated plans in matching deployment contexts.

3.2 SR: Source-Level Provably Safe Unification

  • SR identifies a host staging buffer and device buffer linked by a full-buffer H2D copy, then replaces the pair with one managed buffer and removes linking-copy data movement while retaining its ordering effects.The rewrite renames both pointers to the shared buffer and places its deallocation after both original frees and all accessing GPU work.Source: §3.2.1 · p. 4
  • SR permits unification only when S1–S4 preserve values, cross-buffer access ordering, pointer observability, and the shared buffer's lifetime.The four motivating cases include a still-live device value, a synchronous D2H completion wait, pointer equality, and a buffer freed before GPU use.Source: §3.2.2 · p. 5, Figure 2 · p. 5, §3.2.1 · p. 5

3.2.3 Soundness.

  • Under race freedom, successful allocations, and no reads of uninitialized bytes except by linking copies, safe unification preserves externally visible behavior.The theorem applies when the safety conditions hold.Source: §3.2.3 · p. 5

3.2 SR: Source-Level Provably Safe Unification

  • SR checks S1–S4 on accesses attributed to each candidate in LLVM IR, declining pairs when a condition is undecidable or the pointers cross compilation units.It also skips pairs below a copy-cost threshold.Source: §3.2.4 · p. 6

3.3 DR: Binary-Level Pattern-Guided Optimization

  • DR checks profiled host-access windows with page protection and preserves copy ordering with synchronization when needed, but assumes other safety conditions and relies on later runs behaving like profiled runs.Calibration separately tests profitability because a unified buffer and added synchronization can cost more than the removed copies.Source: §3.3.1 · p. 6, §3.3.2 · pp. 6–7
  • DR profiles allocation and copy events, validates candidate pairs in a separate run, and enables a plan only when calibration shows a positive median reduction in complete-loop latency.At deployment, it checks the calibration context before applying the plan; otherwise the program runs unchanged.Source: §3.3.2 · pp. 6–7, Figure 3 · p. 7

4 Evaluation

The evaluation measures Echo across seven benchmarks and three Jetson platforms, emphasizing how removable-copy share shapes speedups and also reporting throughput, memory, profiling-cost, and energy outcomes.

4.1 Experimental Setup

  • The suite covers library kernels, TensorRT pipelines, application binaries, and a managed-memory negative control across three Jetson platforms.Each process ran 100 warmups and 100 timed iterations; checksum checks used relative tolerance 10−5 and were not element-wise proof.Source: §4.1.1 · p. 7, §4.1.3 · p. 8, §4.1.4 · p. 8

4.2 Per-Benchmark Results

  • On Orin, primary-region speedups were 7.05× (BM2), 3.37× (BM3), 1.69× (BM1), 1.47× (C1), 1.28× (C2), 1.40× (E1), and 1.17× (E2).Source: §4.2 · p. 9
  • DR and SR recover at least 99% of manual gains on the micro-benchmarks, while the canonical-pipeline latency gap from manual optimization is below 0.5%.Source: §4.2 · p. 9
  • DR removes GPU copy-only intervals while leaving kernel time nearly unchanged; kernel work limits C1/C2 gains, Other time limits E1/E2 gains, and Other time also shrinks BM2’s complete-iteration gain.Source: Figure 5 · p. 9, §4.2.1 · pp. 9–10
  • Across 21 platform-benchmark observations, speedup fit copy fraction with R² = 0.989, while hardware and software differences changed each workload’s affected share.Source: Figure 7 · p. 10, §4.2.2 · p. 10

4.4 Practical Costs and Energy Consumption

  • DR profiling adds less than 1% per-iteration overhead; for C1, 50 profiling iterations take about 1.67 s, recovered within 156 optimized iterations, or about 5.2 s.Source: §4.4 · p. 10

4.3 Throughput and Memory Savings

  • On Orin, DR increases throughput on every workload, halves selected buffer capacity, and skips 12.25–51.43 MiB of copy payload per complete iteration.Source: §4.3 · p. 10, Table 5 · p. 11

4.4 Practical Costs and Energy Consumption

  • On Orin, estimated energy falls 3.5–83.2%; BM2’s estimate falls 83.2% although mean rail power rises 18.8%, as latency drops from 4.27 to 0.61 ms.Thor BM3 and E2 use baseline fallback, so plotted differences reflect variation between baseline runs.Source: §4.4 · pp. 10–11, Figure 8 · p. 11

5 Limitations and Future Work

Echo-SR proves source-available rewrites, while Echo-DR offers a guarded contract for stable binary deployments. The paths remain bounded by profile coverage, supported APIs and execution patterns, compile-time visibility, and CUDA-specific implementation.

Binary deployment contract

  • Echo-DR is profile-conditional rather than input-universal: unseen inputs may introduce unobserved host accesses, pairings, copy orderings, or stream patterns.Page protection checks candidate buffers during validation but is not active in later optimized runs, and startup checks and API matching do not detect such changes.Source: §5 · p. 11

API coverage

  • Echo-DR intercepts CUDA runtime calls, but does not cover CUDA driver API applications or two-dimensional and pitched copies.Adding those cases would require new interception hooks, not changes to the core analysis.Source: §5 · p. 11

Compile-time visibility

  • Echo-SR conservatively declines pairs with dynamic sizes, cross-compilation-unit pointer flow, or complex control flow; separately compiled shared-library copies remain invisible at compile time.Link-time optimization could widen whole-program scope, but does not resolve the dynamic-library boundary.Source: §5 · p. 11
  • Echo-DR can observe runtime CUDA API calls across library boundaries that SR cannot reach, but its guarantee is profile-based rather than a compile-time proof.This complements SR’s source-level reach while retaining the binary path’s profile-coverage boundary.Source: §5 · pp. 11–12

Multi-stream workloads

  • Pairs involving multiple streams, events, or overlapped copy and compute are ineligible and run through the original CUDA runtime path.Optimizing these cases would require stream-aware dependency analysis and additional synchronization repair.Source: §5 · p. 12

Platform scope

  • Echo’s transformation principle applies to UMA platforms with duplicated host/device objects, but the implementation is specific to CUDA.Adapting it to Metal, Vulkan on integrated GPUs, or AMD APU OpenCL is future work subject to preserving the same safety guarantees.Source: §5 · p. 12

6 Related Work

Prior work documents redundant transfers and platform-level zero-copy or unified-memory alternatives, while compiler and source-refactoring approaches address different scopes or lack Echo-SR’s safety analysis. Echo combines a source-level safety path with guarded binary optimization for stable legacy CUDA patterns.

Redundant Copies on Other UMA Platforms

  • UMA studies report persistent explicit-copy patterns on integrated systems, Snapdragon, and ARM Mali, despite available zero-copy or map/unmap alternatives.Dashti and Fedorova, Wang et al., and Grasso et al. describe these settings.Source: §6 · p. 12
  • Unified-memory and zero-copy approaches on AMD MI300A and Apple Silicon reduce memory costs or manual transfers, but benefit applications written for or ported to their APIs.Legacy dGPU-style patterns persist in Jetson and existing CUDA codebases, including closed-source deployments.Source: §6 · p. 12

Compiler and Runtime Copy Optimization

  • Standard compiler and memcpy optimizations do not unify host and device allocations or repair ordering from removed copies; OpenMP transfer analysis targets implicit, not native CUDA explicit-copy, management.The Grace Hopper coherent-memory feature also leaves explicit copies between separate allocations as a software choice.Source: §6 · p. 12

Source-Level Refactoring

  • A Clang refactoring tool replaces malloc/cudaMalloc syntactically without analyzing control flow, dependencies, or aliasing, whereas Echo-SR uses LLVM IR and sufficient safety conditions.Echo-DR additionally reaches stable binary-only patterns hidden in closed-source runtimes; to the authors’ knowledge, Echo is the first to combine a safety-proving source path with a guarded binary path.Source: §6 · pp. 12–13

7 Conclusion

Echo targets redundant copies caused by separate host and device buffers on UMA systems. Its static and dynamic approaches unify those allocations and remove copies only when safety conditions preserve program semantics.

7 Conclusion

  • Echo targets dGPU-style code that copies between separate buffers residing in the same physical DRAM on UMA systems.The conclusion identifies wasted bandwidth as the motivation for eliminating those redundant copies.Source: §7 · p. 13
  • Echo’s static and dynamic approaches verify safety conditions before unifying separate host and device allocations and removing their associated copies.The conclusion frames this as preserving program semantics.Source: §7 · p. 13
  • The transformation principle applies broadly to UMA platforms with separate CPU and GPU memory abstractions, while CUDA on Jetson is the evaluated case study.The conclusion describes this case study as demonstrating potential performance gains with minimal engineering effort.Source: §7 · p. 13

A Proof of SR Soundness

The proof argues that the transformed program preserves the values returned by retained reads. It constructs a matching execution order, maintains a value invariant, and relies on the soundness conditions to preserve race freedom, lifetimes, and address-only uses.

Proof of Theorem 1

  • The proof constructs a matching execution of the original program and shows every retained read returns the same value, yielding inclusion of transformed behaviors in the original program’s behaviors.Branches and external calls depend only on read values, so matching reads also preserve those choices.Source: A Proof of SR Soundness · p. 14
  • The transformed execution is race-free because conflicting accesses are ordered either by original program order or condition S2, with synchronization events accessing no memory.The proof then chooses a total order consistent with the transformed execution’s ordering.Source: A Proof of SR Soundness · p. 14
  • The proof forms a common order by reinserting removed copy and allocation events after their last predecessors; condition S1 prevents stale counterpart reads after retained writes.This order supports the subsequent value-preservation invariant.Source: A Proof of SR Soundness · p. 14
  • A byte-value invariant links initialized host or device bytes that will be read with the corresponding shared-buffer byte, while bytes outside those regions remain equal.Retained writes preserve the invariant because S1 ensures the overwritten counterpart is not read before being rewritten; allocation and free ordering follows S4.Source: A Proof of SR Soundness · p. 14
  • Condition S3 ensures replacing host and device pointers with the shared pointer changes neither computed values nor touched bytes when they appear only as addresses.Synchronization events access no memory, and the proof extends the argument to several disjoint pairs.Source: A Proof of SR Soundness · p. 14

B Cross-Platform Results

Across Orin, Nano, and Thor, results vary with platform and workload: Nano gains are smaller than Orin’s, while Thor’s profitability guard rejects some transformed plans. SR and DR generally track one another, but SR lacks runtime profitability calibration.

Per-Platform Results (Orin, Nano, and Thor)

  • Nano follows Orin’s qualitative trend but has smaller gains, as its fewer CUDA cores make computation a larger share of runtime and lower 𝑓copy.Table 7 reports recovered fractions of 97.1%–103.9% on benchmarks with a manual reference; 𝐵E2 has no independent manual reference.Source: B Cross-Platform Results · p. 14, Table 7 · p. 15
  • Thor’s guard enables five net-positive workloads: 𝐵M1, 𝐵M2, 𝐵C1, 𝐵C2, and 𝐵E1 achieve primary-region speedups of 1.09×, 1.59×, 1.16×, 1.11×, and 1.04×, respectively.It falls back for 𝐵M3 and 𝐵E2 after negative profitability calibration; the near-1× values are baseline-process variation, not transformed execution.Source: B Cross-Platform Results · pp. 14–15, Table 8 · p. 15
  • 𝐵E2 is a capped Thor feature-extraction job: its engine supports at most 16 crops per frame versus 23–32 detections measured, so absolute throughput is not compared with Orin/Nano.The Orin/Nano engine supports 128 crops.Source: B Cross-Platform Results · p. 15

SR Results on Nano and Thor

  • SR makes the same analysis decisions on every platform; on Nano its speedups differ from DR by at most 0.02×.On Thor, SR yields 1.71× on 𝐵M2 versus DR’s 1.59×, while 𝐵M3 regresses to 0.91×; DR falls back there because SR has no runtime profitability calibration.Source: B Cross-Platform Results · p. 15, Table 9 · p. 16

𝑓copy Across Platforms

  • Across 21 platform/benchmark points, primary-region measurements follow the same Amdahl trend (R2 = 0.989); excluding Thor 𝐵M2 and 𝐵M3 gives R2 = 0.994.Those two Thor points deviate because managed-memory kernel slowdown offsets part or all of the copy-elimination benefit.Source: B Cross-Platform Results · p. 16

Power and Energy on Nano

  • 4–76% energy savings are reported on Nano; across enabled benchmarks where both apply, DR and SR agree within 3% in primary-region energy.Mean measured-rail power rises approximately 3–60% under Echo, but primary-region energy falls because the optimized region finishes sooner.Source: Table 10 · p. 16, B Cross-Platform Results · p. 16

C Complete-Loop Throughput and Buffer Capacity

The supplementary tables report complete-loop throughput and selected buffer capacity for Nano and Thor, alongside primary-region latency results across platforms.

C Complete-Loop Throughput and Buffer Capacity

  • Tables 12–13 report complete-loop throughput and selected buffer capacity on Nano and Thor; Tables 6–8 report primary-region latency.Throughput uses the complete timed loop and each workload’s work unit; selected capacity counts live allocations chosen for unification, not process RSS.Source: C Complete-Loop Throughput and Buffer… · p. 16

D Managed-Memory Slowdown on Thor

On Thor, managed allocations slow the tested 𝐵M2 and 𝐵M3 computation, with the impact depending on access pattern. The copy savings outweigh that cost for 𝐵M2 but not 𝐵M3, so DR retains the baseline for the latter.

The M2/M3 contrast

  • Both Thor pairs pass observed host-access validation, but semantic eligibility does not ensure net benefit: saved copies outweigh the managed-computation cost for 𝐵M2, not 𝐵M3.DR therefore enables 𝐵M2 and retains baseline for 𝐵M3; complete-loop calibration matters because copy fraction alone cannot determine the plan.Source: D Managed-Memory Slowdown on Thor · pp. 16, 18

Access-pattern characterization

  • The probes characterize the measured CUDA 13/R38.2.2 Thor stack; they do not establish a vendor-confirmed cache mechanism.The controlled probes vary access type, buffer size, computation, stride, and repetition.Source: D Managed-Memory Slowdown on Thor · pp. 17–18

The M2/M3 contrast

  • For 𝐵M2, compute-phase time rises 65% (0.156 to 0.257 ms), yet complete-loop latency falls a median 6.3%; for 𝐵M3, compute time rises 58% (0.745 to 1.174 ms) and loop latency rises 16.7%.DR enables the 𝐵M2 plan and retains baseline for 𝐵M3; SR has no runtime profitability gate and measures a 0.91× primary-region speedup on 𝐵M3.Source: D Managed-Memory Slowdown on Thor · p. 17, Table 14 · p. 17

Access-pattern characterization

  • At 16 MB, Thor’s managed-allocation read probe slows 16.8× versus 1.3× for writes; penalties depend on access pattern and persist over 100 launches.The penalty disappears at larger buffer sizes or with sufficient computation, but remains substantial for large strides; matching Orin probes remain near 1.0×.Source: D Managed-Memory Slowdown on Thor · p. 17, Table 15 · p. 17
Loading 2609.05635v2…