PaperlayerView exact revision on arXiv

Source-linked AI summary

Atlas: Algorithm-Hardware Co-Design for On-Device City-Scale 3D Gaussian Splatting in VR

He Zhu, Zheng Liu, Xingyang Li, Anbang Wu, Zihan Liu, Ruyang Li, Hui Wei, Yaqian Zhao, Jingwen Leng, Minyi Guo, Yu Feng

arXiv:2609.02352v1cs.AR

TL;DR

City-scale 3DGS on mobile VR is constrained by device memory and dependence on host devices or network availability. Atlas addresses this with hierarchical memory offloading and pipeline optimizations, achieving substantial speedups and energy savings over mobile GPU and accelerator baselines.

  • Problem

    City-scale 3DGS is difficult on mobile VR because large-scale scenes exceed mobile GPU memory, while host-device dependence is less reliable when network availability is limited.

  • Method

    Atlas uses hierarchical memory offloading for on-device city-scale 3DGS rendering, with lightweight hardware pipelining and methods that reduce repeated Gaussian loading and rendering computation.

  • Results

    18.5× motion-to-photon speedup and 92.4% lower overall energy versus a mobile GPU; integrated with state-of-the-art 3DGS accelerators, Atlas achieves 3.9× speedup.

  • Takeaways & Limitations

    Offloading most Gaussians to disk supports real-time, infinite-scale 3DGS splatting in VR while fetching only newly required attributes.

  • Takeaways & Limitations

    Stereo rasterization must account for view-dependent 3DGS characteristics because transferring from the left eye to the right eye can produce less reliable results.

Abstract

from arXiv · show

3D Gaussian splatting (3DGS) has drawn significant attention in the architectural community recently. However, enabling city scale 3DGS on mobile VR devices remains challenging, as the memory requirement of large scale scenes far exceeds the memory capacity of today's mobile GPUs. This paper presents Atlas, an on device city scale 3DGS rendering framework that enables scalable rendering without runtime Internet access. The key insight is that although the full 3DGS model is massive, each frame only requires a small subset of Gaussians under the current pose and level of detail requirement. Based on this insight, Atlas introduces a hierarchical memory offloading mechanism that dynamically loads only necessary Gaussian data into device memory. To further improve performance, Atlas proposes temporal aware LoD search and stereo rasterization to avoid redundant computation in VR. We further show that our technique can be integrated with existing 3DGS accelerators with negligible hardware overhead. Overall, Atlas achieves 18.5x speedup over the GPU baseline and 3.9x speedup over the state of the art 3DGS accelerators, with 92.4% energy savings.

1 Introduction

Atlas targets city-scale 3DGS rendering on mobile VR by reducing on-device memory demands and avoiding runtime Internet access. It combines hierarchical Gaussian management with temporal-aware LoD search, stereo rasterization, and lightweight hardware support.

  • Motivation: City-scale 3DGS scenes can require up to 66 GB, exceeding the less-than-12-GB memory capacity of typical mobile VR devices.Supporting level-of-detail representations further increases memory pressure.
  • Framework: Atlas addresses this gap with a hierarchical memory offloading framework that retains only necessary Gaussians in device memory.Its key insight is that each frame needs only a small subset of Gaussians under the current pose and level-of-detail requirement.
  • Memory Management: Runtime management partitions scenes into spatial blocks, loads viewable blocks on demand, and overlaps data movement with rendering to maintain a compact memory footprint.Remaining blocks stay in lower-tier storage, while the active Gaussian working set determines on-demand loading.
  • Algorithmic Optimizations: Temporal-aware LoD search exploits cross-frame coherence, while stereo rasterization shares computations between tightly coupled left- and right-eye views.Stereo processing reuses shared results and skips repetitive preprocessing and sorting for the second view.
  • Evaluation: 18.5× motion-to-photon speedup and 92.4% energy reduction are achieved versus a mobile GPU, while integration with state-of-the-art accelerators provides 3.9× speedup and 4.7× energy savings.The integrated design incurs only minimal hardware overhead, including a lightweight line buffer for fine-grained stereo pipelining.

2 Large-Scale 3DGS pipeline

Large-scale 3DGS organizes Gaussians hierarchically and renders a pose-dependent level-of-detail selection through four stages: LoD search, preprocessing, sorting, and rasterization.

  • Hierarchical Representations: An LoD tree hierarchically organizes Gaussians, with lower levels generally capturing finer details.The paper describes irregular trees as the general representation; octrees and other tree-like structures are special cases.
  • LoD Search: LoD search traverses the tree to select Gaussians whose projected sizes satisfy a target pixel-space threshold.Selected Gaussians form a cut separating the upper and lower tree levels.
  • Pipeline: Preprocessing projects selected Gaussians onto the canvas and filters those outside the view frustum.The remaining Gaussians proceed to depth sorting and tile-based rasterization.
  • Pipeline: Sorting orders remaining Gaussians from nearest to farthest, ensuring visually consistent blending before rasterization.Rasterization processes tiles, builds local Gaussian lists, and blends contributing Gaussians into pixels.
  • Rasterization: Each pixel’s final color is computed by weighted blending of Gaussian colors using transparency and accumulated transmittance.Gaussians below the transparency threshold are skipped during blending.

3 Challenges and Opportunities

City-scale 3DGS exceeds mobile-VR memory capacity and shifts computation toward LoD search, but stage-dependent memory, temporal reuse, and stereo overlap provide optimization opportunities.

  • Memory Pressure: Large-scale scenes rapidly exceed mobile-VR memory capacity, with mainstream devices often offering less than 12 GB and one HierGS scene exceeding 66 GB.This creates a central scalability challenge for local city-scale rendering.
  • Bottleneck Shift: 47% of end-to-end latency can come from LoD search on large-scale scenes, whose cost grows rapidly with scene complexity.Rasterization time does not grow similarly because LoD search limits the number of Gaussians contributing to the frame.
  • Memory Demand: The pipeline’s initial LoD search stage requires the most memory, while the active Gaussian set drops rapidly afterward to a mobile-accommodable scale.This suggests separating high-memory LoD search from lower-memory rendering and managing the former carefully.
  • Stereo Similarity: Fewer than 1% of pixels are non-overlapping between left- and right-eye views, motivating shared computation across VR stereo pipelines.Direct pixel warping remains unsuitable because view-dependent rendering can introduce artifacts.
  • Summary: The paper identifies three opportunities: stage-aware memory management, temporal reuse of LoD-search results, and computation sharing between stereo views.These opportunities target the memory and redundant-computation constraints of on-device city-scale rendering.

4 Atlas Overview

Atlas enables fully on-device city-scale 3DGS rendering by keeping only necessary data in GPU memory and asynchronously managing the rest across a local memory hierarchy. Temporal-aware LoD search and stereo rendering reduce redundant work while preserving required rendering behavior.

  • Workflow: Atlas loads selected Gaussians’ rendering attributes on demand, reuses unchanged data across frames, and evicts no-longer-needed data.This combines spatial locality across blocks with temporal similarity across frames to keep the GPU footprint compact.
  • Workflow: Atlas partitions city-scale scenes into spatial blocks and keeps nearby blocks’ lightweight LoD metadata in GPU memory.Inactive blocks and storage-intensive rendering attributes remain in CPU memory or disk and are fetched on demand.
  • Workflow: Atlas uses temporal-aware LoD search for subsequent frames and fully streaming traversal for initial or newly activated regions.The fully streaming traversal regularizes memory accesses, while temporal search updates cuts within relevant subtrees.
  • Timing: Atlas performs missing-data transfers asynchronously while rendering continues, leaving only rendering operations on the critical path.The execution flow hides much of the memory traffic behind rendering before displaying the next image at VSync.
  • Stereo rendering: Atlas shares computation between stereo views by mapping active Gaussians geometrically while still producing bit-accurate images.The stereo rasterization pipeline uses the geometric relationship between Gaussians and the stereo camera rather than rendering two independent frames.

5 Temporal-Aware LoD Search

Atlas accelerates LoD search by reorganizing irregular trees for GPU-friendly streaming traversal and exploiting temporal similarity between adjacent frames. The approach avoids unnecessary node visits while preserving bit-accurate cuts.

  • Subtree partitioning: Atlas partitions each LoD tree with breadth-first search into subtrees constrained by a predefined size limit.Small subtrees sharing a parent are greedily merged when possible to reduce size variation and improve workload balance.
  • Fully-Streaming LoD Tree Traversal: The fully streaming traversal assigns balanced node blocks across GPU warps and uses shared memory to avoid irregular DRAM accesses.Traversal stops at a clean cut, visiting necessary nodes while skipping irrelevant nodes.
  • Fully-Streaming LoD Tree Traversal: Atlas loads only lightweight LoD metadata during traversal and fetches storage-intensive rendering attributes later for cut-selected Gaussians.This separation allows local LoD search without loading the full rendering data into GPU memory.
  • Temporal-Aware LoD Search: Temporal-aware LoD search examines only subtrees containing the previous frame’s cut instead of traversing all active-block subtrees.Newly activated blocks fall back to fully streaming traversal, while incomplete local cuts expand to parent or neighboring subtrees.
  • Results: Up to 8.1× speedup is achieved for tree traversal, while temporal-aware search remains bit-accurate relative to full-tree traversal.The method also maintains approximately equal subtree sizes to balance GPU warp workloads.

6 Runtime Gaussian Management

Atlas manages city-scale Gaussian data through block-level activation and runtime attribute residency, retaining only current and near-future working sets in GPU memory. On-demand loading and reuse-based eviction support fully on-device rendering under tight memory limits.

  • Block-Level Management: Atlas partitions the scene into spatial blocks and activates blocks near the current camera pose while keeping inactive blocks in lower-tier storage.Neighboring blocks are proactively loaded near active-block boundaries, and unused blocks are evicted after a frame-based threshold.
  • Gaussian Attribute Management: Only lightweight LoD attributes remain resident for local search, while rendering attributes such as SH coefficients are loaded after selection.This separates search-critical metadata from storage-intensive rendering data.
  • Gaussian Attribute Management: Atlas asynchronously loads missing rendering attributes into GPU memory after each LoD cut identifies the Gaussians required for rendering.Loaded attributes are gathered into request groups from CPU memory or disk storage.
  • Gaussian Attribute Management: Atlas removes obsolete Gaussian attributes using a reuse-window threshold, treating data unlikely to be reused soon as eviction candidates.The policy is specialized for 3DGS and resembles garbage collection.
  • Local Rendering Queue Generation: Atlas keeps GPU memory usage low while enabling fully on-device rendering of city-scale 3DGS scenes.The active Gaussian set and rendering queue include only data available for the current rendering operation.

7 Stereo Rasterization

Atlas reduces redundant stereo-rendering computation by deriving right-eye Gaussian contributions from left-eye processing and stereo geometry. The resulting pipeline preserves bit-accurate rendering while improving mobile-GPU performance.

  • Motivation: Existing stereo methods are limited by reliance on high-fidelity depth maps, which can be unreliable for 3DGS.Directly warping left-eye pixels also compromises 3DGS view dependence and can reduce photorealism.
  • Intuition: Atlas uses triangulation and stereo-camera geometry to compute right-eye pixel contributions without repeating preprocessing and sorting.Disparity is computed from the stereo baseline, focal length, and Gaussian depth.
  • Preprocessing&Sorting: Preprocessing and sorting are shared across eyes by using a wider field of view that covers both eye views.Right-eye tile lists are formed by mapping left-eye contributing Gaussians according to their computed disparity.
  • Stereo Rasterization: The right-eye pipeline merges four already sorted candidate lists and removes duplicates instead of re-sorting them.The merge process is analogous to merging four pre-sorted lists in merge sort.
  • Results: 1.4× speedup is achieved on mobile GPUs, and the stereo rasterization produces bit-accurate results.Only Gaussians passing the left-eye α-check are transformed and inserted into corresponding right-eye tile lists.

8 Architectural Support

Atlas combines GPU-based LoD search and runtime Gaussian management with an accelerator augmented for stereo rasterization. Its stereo design shares computation and reduces redundant memory accesses across VR views.

  • Hardware Support: Supporting stereo rasterization on existing 3DGS accelerators requires only minimal hardware augmentation.The paper identifies shared computation and reduced redundant memory accesses as benefits of the stereo design.
  • Architecture: The GPU performs LoD search, while runtime Gaussian management fetches visible rendering attributes and streams them to the accelerator.The accelerator pipelines projection, sorting, and volume rendering for different image tiles within a frame.
  • Stereo Rasterization: Atlas augments the volume rendering core with stereo rasterization so left- and right-eye views can share computation.The stereo reprojection unit forwards contributing Gaussians into right-eye stereo-buffer lists based on disparity.
  • Stereo Rasterization: The stereo buffer organizes Gaussians by disparity category, while a merge unit sorts the current lists for each right-eye tile.Circular buffers are used to maximize utilization and avoid bank conflicts.
  • Pipeline: The architecture pipelines preprocessing, sorting, and rasterization, rendering corresponding left- and right-eye tiles sequentially.Right-eye rendering begins after left-eye rendering, with the first three right-eye tiles rendered independently.

9 Experimental Setup

Atlas is evaluated with RTL and cycle-accurate simulation against mobile GPU and dedicated accelerator baselines. Experiments cover large- and small-scale datasets, rendering quality, memory, speed, and energy.

  • Hardware and Simulation: The cycle-accurate simulator parameterizes GPU latency and power from measurements and accelerator latency and power from post-synthesis results.System energy sums GPU, VRC, and DRAM energy.
  • Hardware Configuration: Atlas adds around 14% area overhead, primarily from an additional 16 KB SRAM per VRC.The reported overhead is 0.25 mm2 compared with 1.78 mm2 for GSCore in 16nm.
  • Datasets and Algorithms: Experiments use Urban, Mega, and HierGS large-scale datasets alongside T&T, DB, and M360 small-scale datasets.Large-scale algorithmic comparisons include HierGS, CityGS, and OctreeGS, whose main difference is LoD search.
  • Baselines: Hardware efficiency is compared against a mobile Ampere GPU, GSCore, and GBU, with GBU configured with 128 Row PEs to align with GSCore.GBU accelerates rasterization while remaining operations execute on the mobile GPU.

10 Evaluation

Atlas improves stereo quality, memory use, GPU-side performance, accelerator performance, and energy efficiency across the evaluated settings. Its main trade-offs involve hardware scaling, frame-interval choices, tile size, and small quality loss from delayed Gaussian availability.

  • Stereo Rendering: Atlas nearly matches Base stereo quality, with only a 0.1 dB PSNR loss and no SSIM quality loss.Warp and Cicero introduce noticeable accuracy loss against Base, whereas Atlas’s stereo rasterization is bit-accurate.
  • GPU Speedup: 2.1× GPU-side speedup is achieved by Atlas over Base when applying all algorithmic optimizations.The evaluated variants add runtime Gaussian management and temporal-aware LoD search before the full Atlas configuration.
  • Memory Footprint: Up to 7.0× lower GPU memory footprint keeps all evaluated large-scale scenes below 12 GB compared with Base.The reduction is most pronounced on HierGS, where street-level viewpoints activate fewer spatial blocks.
  • Overall Performance: 18.5× average speedup over GPU and 70.9 FPS are reported for the overall system.Atlas can reach 90 FPS by scaling up the VRC, although the default configuration does not meet that VR requirement.
  • Energy: 13.1× lower energy than GPU and 1.5× lower energy than GSCore are achieved on average.The comparison excludes constant display power and normalizes energy numbers to GPU.
  • LoD Search: 8.1× LoD-search speedup is achieved by exploiting temporal similarity to eliminate redundant node accesses.The comparison uses OctreeGS as the baseline and compares CityGau and HierGS with Atlas.

11 Related Work

Prior work addresses rasterization, warping, caching, collaborative rendering, and video streaming, but existing approaches do not directly resolve 3DGS-specific LoD-search and on-device constraints. Atlas targets these gaps while retaining bit-accurate rendering and compatibility with existing accelerators.

  • Collaborative Rendering: Existing collaborative-rendering methods target mesh-based rasterization rather than 3DGS.Cicero and CollabVR offload rendering to the cloud and perform lightweight client-side warping.
  • Streaming Limitations: Prior streaming methods face bandwidth limitations at higher FPS or resolution and partition work at the pixel level.Pixel-level partitioning remains incompatible with 3DGS because clients still perform compute-intensive LoD search.
  • Atlas Positioning: Atlas is designed for 3DGS to reduce on-device workload while minimizing network traffic.The related-work discussion contrasts this focus with prior cloud-client approaches.
  • Relationship to Prior Work: Atlas leverages spatial and temporal similarities like prior warping work but applies them differently for 3DGS.The paper states that Atlas is orthogonal to prior 3DGS acceleration work and can integrate with existing frameworks.
  • Rendering Quality: Atlas achieves bit-accurate rendering while delivering speedup.This distinguishes its stereo-rasterization result from warping approaches that can introduce visual accuracy loss.

12 Conclusion

Atlas targets real-time, infinite-scale 3DGS splatting in VR by offloading most Gaussian data to disk. Its rendering pipeline alleviates communication bottlenecks and reduces redundant stereo computation.

  • Atlas targets real-time, infinite-scale 3DGS splatting in VR.
  • Offloading the majority of Gaussians to disk storage supports scalable rendering beyond device-memory limits.
  • Atlas alleviates communication bottlenecks through its rendering framework.
  • Its stereo rasterization pipeline eliminates redundancy in VR stereo rendering.
Loading 2609.02352v1…