PaperlayerView exact revision on arXiv

Source-linked AI summary

TideGS: Scalable Training of Over One Billion 3D Gaussian Splatting Primitives via Out-of-Core Optimization

Chonghao Zhong, Linfeng Shi, Hua Chen, Tiecheng Sun, Hao Zhao, Binhang Yuan, Chaojian Li

arXiv:2605.20150v2cs.CVcs.PF

TL;DR

Billion-scale 3D Gaussian Splatting training is constrained by GPU memory. TideGS uses out-of-core working-set management across SSD, CPU, and GPU, training a 1.1B-Gaussian scene on one 24 GB GPU while preserving in-memory quality and improving city-scale reconstruction fidelity.

  • Problem

    3DGS training remains memory-constrained because parameter, gradient, and optimizer-state copies exceed typical 24 GB GPU budgets at large scene scales.

  • Method

    TideGS virtualizes Gaussian parameters across SSD, CPU, and GPU using block-virtualized geometry, asynchronous execution, and trajectory-adaptive differential streaming.

  • Results

    1.1B-Gaussian MatrixCity scene training on a single 24 GB GPU preserves Native 3DGS quality in-memory and improves reconstruction fidelity at city scale.

  • Takeaways & Limitations

    TideGS lowers the hardware floor for billion-scale 3DGS training on a single GPU.

  • Takeaways & Limitations

    TideGS is most effective with spatiotemporally local camera trajectories and sufficiently fast NVMe storage; weaker locality or slower storage increases traffic and latency.

Abstract

from arXiv · show

Training 3D Gaussian Splatting (3DGS) at billion-primitive scale is fundamentally memory-bound: each Gaussian primitive carries a large attribute vector, and the aggregate parameter table quickly exceeds GPU capacity, limiting prior systems to tens of millions of Gaussians on commodity single-GPU hardware. We observe that 3DGS training is inherently sparse and trajectory-conditioned: each iteration activates only the Gaussians visible from the current camera batch, so GPU memory can serve as a working-set cache rather than a persistent parameter store. Building on this insight, we introduce TideGS, an out-of-core training framework that manages parameters across an SSD-CPU-GPU hierarchy via three synergistic techniques: block-virtualized geometry for SSD-aligned spatial locality, a hierarchical asynchronous pipeline to overlap I/O with computation, and trajectory-adaptive differential streaming that transfers only incremental working-set deltas between iterations. Experiments show that TideGS enables training with over one billion Gaussians on a single 24 GB GPU while achieving the best reconstruction quality among evaluated single-GPU baselines on large-scale scenes, scaling beyond prior out-of-core baselines (e.g., approximately 100M Gaussians) and standard in-memory training (e.g., approximately 11M Gaussians).

1. Introduction

TideGS addresses the memory barrier to large-scale 3DGS by treating GPU memory as a working-set cache and coordinating parameter storage across SSD, CPU, and GPU. Its out-of-core design combines spatial virtualization, asynchronous overlap, and differential streaming to train over one billion Gaussians on a single 24 GB GPU.

  • Memory Bottleneck: 100 million Gaussians require nearly 90 GB of memory, exceeding a typical 24 GB single-GPU budget because parameters, gradients, and optimizer states require multiple copies.Each Gaussian is parameterized by 59 floating-point values.
  • Memory Bottleneck: On a 24 GB GPU, vanilla 3DGS reaches approximately 11M Gaussians, while optimized host-offloading pipelines remain around approximately 100M Gaussians.Increasing Gaussian count can improve rendering fidelity, especially in large-scale environments.
  • Core Insight: Only visible Gaussians participate in rasterization and receive non-zero gradients for each camera batch, motivating VRAM as a high-bandwidth working-set cache rather than a persistent parameter store.Most primitives remain inactive during each optimization step.
  • TideGS Framework: TideGS manages parameters across an SSD–CPU–GPU hierarchy using block-virtualized geometry, hierarchical asynchronous pipelining, and trajectory-adaptive differential streaming.The techniques respectively provide SSD-aligned spatial locality, overlap data movement with rendering and backpropagation, and transfer incremental working-set deltas.
  • Results: Over one billion Gaussians can be trained on a single 24 GB GPU, while TideGS preserves Native 3DGS quality at feasible scales with less than 15% overhead over GPU-resident training.In the out-of-core regime, TideGS remains throughput-competitive and scales an order of magnitude beyond prior single-GPU methods.

2. Preliminaries

Standard 3DGS stores a dense, state-heavy parameter table, but each iteration updates only a sparse visibility-induced subset of Gaussians. TideGS therefore frames training around block-level working sets and out-of-core movement across SSD, CPU, and GPU tiers.

  • Per-Gaussian parameter table: Each Gaussian carries a D=59 parameter vector under the standard degree-3 spherical-harmonics parameterization.The vector encodes geometry and appearance.
  • Per-Gaussian parameter table: The full training state grows linearly with N while including gradients and optimizer states with a large constant factor.Adam moments are among the maintained optimizer states.
  • Visibility-induced sparse updates: Each iteration touches only visible Gaussians receiving non-zero gradients, with |It| ≪ N in large scenes.It denotes the union of Gaussian indices visible after rasterization for the current camera batch.
  • Block-level working sets: TideGS uses blocks as transfer and cache units, with block-level working sets conservatively covering the Gaussian-level active sets.This separates block-level staging and caching from Gaussian-level activity.
  • Out-of-core training: Out-of-core training keeps most state on CPU DRAM or SSD and materializes the current working set on GPU.Throughput depends on small sparse working sets and overlapping data movement with rendering and backpropagation.

3. Method

TideGS enables billion-scale 3DGS training on a single 24 GB GPU by treating VRAM as a working-set cache over SSD and CPU memory. It combines spatially localized block virtualization, conservative CPU visibility filtering with exact GPU rendering, and asynchronous differential data movement.

  • Out-of-core hierarchy: TideGS stores the full parameter table on SSD while materializing only camera-batch-specific blocks in VRAM through a coordinated SSD–CPU–GPU hierarchy.GPU VRAM is managed as a high-bandwidth working-set cache, while CPU memory and SSD provide lower tiers.
  • Differential streaming: TideGS stages only incoming working-set differences, maintains a capacity-bounded resident set, and asynchronously writes back evicted dirty blocks.This design exploits overlapping block working sets under smooth camera motion and uses CPU memory as a warm cache.
  • Block virtualization: Morton-sorted Gaussians are partitioned into SSD-aligned spatial blocks, whose bounding spheres support coarse CPU frustum culling before GPU transfers.The default block size is B=4096; each full fp32 block with D=59 carries about 944 KiB.
  • Asynchronous storage pipeline: SSD, PCIe, and cache operations overlap GPU rendering and backpropagation, while log-structured append-only patch segments avoid random SSD writes.Updated blocks are written sequentially rather than overwriting existing block locations in place.
  • Two-stage visibility: CPU-side block filtering is conservative, while GPU-side Gaussian filtering and standard 3DGS rasterization preserve exact rendering and per-Gaussian update semantics.Only Gaussians in the final contributing set receive non-zero gradients; the coarse stage may admit extra blocks.

4. Experiments

Experiments show that TideGS scales from in-memory benchmarks to 1.1B Gaussians on a single 24 GB GPU while preserving reconstruction quality and reducing out-of-core overhead. Ablations confirm that differential streaming, asynchronous overlap, and spatial locality are central to its efficiency.

  • Scalability: TideGS trains ∼1.1B Gaussians on a single 24 GB GPU, whereas Native 3DGS fails at ∼11.5M and CLM fails at ∼1.1B.Native 3DGS is limited by full training-state residency, while CLM requires GPU-resident geometry estimated at ∼45 GB at billion scale.
  • Overhead and efficiency: 3.50 img/s: TideGS with Trajectory outperforms CLM by 6.4% in the in-memory regime.With Shuffle, TideGS achieves 3.24 img/s, within 1.2% of Naive Offload’s 3.28 img/s.
  • Efficiency: 0.10 GB/iter: TideGS at ∼102M reduces PCIe traffic 4× versus CLM’s 0.41 GB/iter and reaches 90.7 ms/iter.Naive Offload runs out of memory at ∼102M Gaussians.
  • Quality: 26.1 dB PSNR: TideGS at ∼1.1B Gaussians improves over CLM’s 25.0 dB PSNR at ∼102M.CLM fails with OOM at ∼1.1B, while TideGS remains feasible at the billion scale.
  • Ablations: 8.5× higher PCIe traffic: disabling differential streaming increases transfers from 0.10 GB/iter to 0.85 GB/iter and iteration latency to 145.3 ms.The ablation retransmits all visible blocks regardless of VRAM residency.
  • Ablations: 95.2% to 42.1%: removing Morton ordering drops CPU cache hit rate and raises PCIe traffic to 0.45 GB/iter.Random block arrangement increases working-set churn, evictions, and re-fetches.

5. Related Work

Prior 3DGS scalability work distributes or partitions scenes, reduces memory or compute through compression and kernel optimization, or offloads state to host memory. TideGS is most closely related to hierarchical training systems, but those systems remain bounded by GPU-resident data at large N.

  • Distributed and scene-partitioned scaling: Distributed and scene-partitioned methods increase capacity across multiple GPUs or independently trained regions, but rely on additional GPU memory and interconnects.These approaches scale scene size by spreading parameters across GPUs or decomposing scenes into regions.
  • Compression, pruning, and kernel optimization: Compression, pruning, and kernel optimizations reduce 3DGS memory or compute costs, though many primarily benefit inference or per-iteration throughput.The techniques prune or compress Gaussians and improve rasterization or scheduling efficiency, remaining largely orthogonal to TideGS.
  • Host-offloading and hierarchical training: Host-offloading systems such as GS-Scale and CLM move parameters or optimizer states to CPU memory but retain key geometry or rasterization-dependent state in VRAM.Their scalability remains bounded by GPU-resident data at large N.
  • Host-offloading and hierarchical training: ZeRO-style partitioning and embedding-table caching provide more general hierarchical-system precedents closely related to TideGS.The passage identifies these systems as more general alternatives after discussing host-offloading methods.

6. Limitations

TideGS depends on trajectory locality and fast NVMe storage, while evicting optimizer moments can cause cold starts under churn. It lowers the hardware floor for billion-scale training on a single GPU but complements distributed multi-GPU systems.

  • Trajectory and storage dependence: Weak trajectory continuity reduces differential-streaming reuse, increasing block churn and cross-tier traffic for unstructured image collections.TideGS is most effective when consecutive camera views share substantial block working sets.
  • Trajectory and storage dependence: 3.3 GB/s internal NVMe SSD is the measured operating point, making slower SATA-class storage a potential performance limitation.TideGS performance depends on sufficiently fast NVMe storage.
  • Optimizer-state persistence: Discarding Adam moments for evicted blocks trades optimizer-state persistence for capacity and can increase cold starts under higher block churn.Larger CPU caches or selective optimizer-state persistence may benefit such workloads.
  • Relationship to multi-GPU systems: TideGS lowers the hardware floor for billion-scale training on a single GPU.The framework is complementary to distributed in-memory multi-GPU systems, which can improve wall-clock throughput with sufficient hardware and interconnects.

7. Conclusion

TideGS is an out-of-core 3D Gaussian Splatting training framework that virtualizes the full Gaussian parameter table across SSD, CPU, and GPU while materializing only each iteration’s working set on the GPU. It combines block-virtualized geometry, asynchronous cross-tier execution, and trajectory-adaptive differential streaming to replace persistent VRAM residency with locality-aware working-set management.

  • Core framework: TideGS virtualizes the full Gaussian parameter table across an SSD–CPU–GPU hierarchy and materializes only the per-iteration working set on GPU.This makes GPU memory a working-set space rather than a persistent parameter store.
  • Core framework: The framework combines block-virtualized geometry, asynchronous cross-tier execution, and trajectory-adaptive differential streaming.These techniques jointly organize storage locality, overlap execution across tiers, and transfer trajectory-dependent parameter differences.
  • Core framework: TideGS shifts the single-GPU bottleneck from persistent VRAM residency to locality-aware working-set management.The shift follows from virtualizing parameters across storage tiers and retaining only active per-iteration data on GPU.

A. Appendix · A.1. Ordering and Convergence Discussion

TideGS replaces standard randomized view ordering with trajectory-aware processing and offers intuition rather than a new convergence theorem. This discussion grounds the approach in visibility-induced sparsity and trajectory continuity, consistent with ordering-aware SGD analyses.

  • A.1. Ordering and Convergence Discussion: TideGS departs from the standard randomized view order used in 3DGS.It processes views in a trajectory-aware order.
  • A.1. Ordering and Convergence Discussion: TideGS processes views in a trajectory-aware order.This ordering is presented as a deliberate departure from standard randomized view ordering.
  • A.1. Ordering and Convergence Discussion: The discussion does not aim to prove a new convergence theorem for masked adaptive optimizers.The authors explicitly frame the section as intuition rather than a new theorem.
  • A.1. Ordering and Convergence Discussion: The provided intuition is consistent with ordering-aware SGD analyses.The passage cites Mohtashami et al., 2022 in this context.
  • A.1. Ordering and Convergence Discussion: TideGS exploits visibility-induced sparsity as an empirical property.Each iteration updates only a small subset of Gaussians.
  • A.1. Ordering and Convergence Discussion: TideGS also exploits trajectory continuity as an empirical property.The passage identifies trajectory continuity alongside visibility-induced sparsity as a basis for its intuition.

A.1.1. PROBLEM SETUP AND NOTATION

The setup represents all Gaussian parameters as a single vector and defines view-wise photometric training over an ordered sequence. TideGS further formalizes spatial blocks, trajectory-dependent working sets, and visibility-induced masked updates, capturing the sparsity that most parameters remain untouched per step.

  • Notation: All Gaussian primitive parameters are concatenated into a vector θ ∈ R^d, with training defined over M views and view-specific photometric losses.The objective is described over the training views, with f_i(θ) denoting the rendering loss for view i.
  • Spatial blocks for storage and streaming: Gaussians are partitioned by Morton-code ordering into fixed-size spatial blocks that form the SSD–CPU–GPU streaming units.With K blocks, K_t denotes the coarse block working set selected by block-wise frustum culling at iteration t.
  • Trajectory ordering over views: TideGS orders training views using clustered TSP camera-pose ordering instead of random shuffling while visiting each view once per training pass.The resulting permutation is π = (π_1, . . . , π_M), and it orders views rather than spatial blocks.
  • Visibility-induced sparse updates: Most parameters are untouched at each optimization step, motivating TideGS’s sparse, working-set-oriented update formulation.The update direction can come from an optimizer such as Adam with first- and second-moment estimates.

A.1.2. EMPIRICAL INTUITIONS

TideGS’s stable trajectory ordering is explained by two empirical properties: visibility sparsity localizes updates, while trajectory continuity reduces gradient variation. Nearby views activate overlapping spatial blocks and produce more similar gradients on relevant coordinates than random ordering.

  • A.1.2. EMPIRICAL INTUITIONS: Together, localized updates and reduced gradient variation explain why trajectory ordering is stable in practice.These two properties are TideGS-specific empirical intuitions for the observed stability.
  • A.1.2. EMPIRICAL INTUITIONS: Visibility sparsity localizes active updates: nearby views activate nearby spatial blocks, while distant regions remain largely inactive.This limits unrelated cross-region interference and makes adjacent views induce overlapping active sets.
  • A.1.2. EMPIRICAL INTUITIONS: Trajectory continuity makes consecutive views geometrically close, yielding more similar per-view gradients on relevant active coordinates than random ordering.For the trajectory-ordered permutation πTSP, this is summarized by a smaller gradient-variation surrogate than for random permutations.

A.1.3. TWO INTUITIONS: SPARSITY LIMITS STALENESS; ORDERING LIMITS VARIATION … A.5. Single-GPU vs. Distributed In-Memory Operating Points

TideGS combines visibility-sparse updates with trajectory-aware ordering to stabilize out-of-core optimization, while experiments show comparable reconstruction quality, broader hardware feasibility, and a distinct single-GPU operating point relative to distributed in-memory systems.

  • A.1.3. TWO INTUITIONS: SPARSITY LIMITS STALENESS; ORDERING LIMITS VARIATION: Visibility-sparse masked updates restrict optimizer-state changes to active Gaussian coordinates, limiting cross-region interference and stale-state effects on inactive parameters.Inactive parameters are not repeatedly perturbed by unrelated views, so their update history is dominated by iterations in which they are visible.
  • A.1.4. TAKEAWAY: WHY TRAJECTORY ORDERING IS STABLE IN TIDEGS: Trajectory-aware ordering makes consecutive views similar, reducing the ordering-dependent gradient variation captured by Eq. (8) and supporting stable training despite non-randomized ordering.The variation surrogate is most relevant on active coordinates, whereas random permutations typically produce unrelated consecutive views.
  • A.2. Ordering Ablation: Shuffle vs. Trajectory: Trajectory ordering improves iteration time while maintaining similar final reconstruction quality relative to randomized view shuffling on Mip-NeRF 360 bicycle.Both variants use the same training views, loss, and optimization recipe; only view presentation order differs.
  • A.3. Dense Initialization Without Training-Time Densification: Dense-initialized fixed-size training reaches comparable final quality to standard training-time densification on both bicycle and bonsai scenes.The comparison isolates out-of-core memory management by disabling densification and pruning in the large-scale MatrixCity experiments.
  • A.4. Additional System Measurements: At the ∼102M-Gaussian scale, TideGS reduces PCIe traffic and iteration time on an RTX 3090 and an A800, matching its qualitative A5000 behavior.These measurements extend the MatrixCity scalability evaluation beyond the A5000 setting.
  • A.4. Additional System Measurements: At the ∼1.1B-Gaussian scale, TideGS remains feasible on the validated hardware while CLM runs out of memory.The additional hardware measurements report the same qualitative scaling behavior on an RTX 3090 and an A800.
  • A.4. Additional System Measurements: Residency and cold-restart measurements quantify block eviction, re-admission, and optimizer-state re-initialization under the evaluated out-of-core settings.These statistics support the optimizer-state placement design described in Sec. 3.5.
  • A.5. Single-GPU vs. Distributed In-Memory Operating Points: TideGS and distributed in-memory systems target different operating points: distributed training uses multiple GPUs and interconnect bandwidth, while TideGS lowers the hardware floor through storage-hierarchy management.The paper frames Table 10 as a capacity-per-dollar comparison and the convergence figures as contextualizations of wall-clock and iteration-wise trade-offs.
Loading 2605.20150v2…