PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets

Bernhard Kerbl, Andréas Meuleman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, George Drettakis

arXiv:2406.12080v1cs.CVcs.GR

TL;DR

Very large scenes exceed the resources available to conventional radiance-field and 3DGS training and rendering. The paper addresses this with chunk-based training and an optimizable Gaussian hierarchy supporting dynamic LOD, demonstrating real-time rendering on kilometer-scale captures with tens of thousands of images. Its results remain constrained by sparse or imperfect capture data, which produces artifacts and limits extrapolation.

  • Problem

    Resource requirements limit the scene size that can be represented and rendered with good quality, while prior large-scene methods remain expensive.

  • Method

    The method trains scene chunks independently, consolidates them into an optimizable 3D Gaussian hierarchy, and uses dynamic LOD with efficient selection and smooth transitions.

  • Results

    The system handles street-level scenes spanning several kilometers and tens of thousands of input images with real-time rendering.

  • Takeaways & Limitations

    The approach enables parallel processing and resource-adaptive real-time rendering of very large captured scenes.

  • Takeaways & Limitations

    Visual artifacts largely arise from bad view-space coverage, calibration errors, and moving distractors, while limited capture restricts radiance-field extrapolation.

Abstract

from arXiv · show

Novel view synthesis has seen major advances in recent years, with 3D Gaussian splatting offering an excellent level of visual quality, fast training and real-time rendering. However, the resources needed for training and rendering inevitably limit the size of the captured scenes that can be represented with good visual quality. We introduce a hierarchy of 3D Gaussians that preserves visual quality for very large scenes, while offering an efficient Level-of-Detail (LOD) solution for efficient rendering of distant content with effective level selection and smooth transitions between levels.We introduce a divide-and-conquer approach that allows us to train very large scenes in independent chunks. We consolidate the chunks into a hierarchy that can be optimized to further improve visual quality of Gaussians merged into intermediate nodes. Very large captures typically have sparse coverage of the scene, presenting many challenges to the original 3D Gaussian splatting training method; we adapt and regularize training to account for these issues. We present a complete solution, that enables real-time rendering of very large scenes and can adapt to available resources thanks to our LOD method. We show results for captured scenes with up to tens of thousands of images with a simple and affordable rig, covering trajectories of up to several kilometers and lasting up to one hour. Project Page: https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/

1 INTRODUCTION

The paper targets resource limits that prevent high-quality real-time rendering of very large scenes. It introduces chunk-based training and a hierarchical 3DGS representation with dynamic LOD.

  • 3DGS combines high visual quality, fast training, and real-time rendering, but resource limits constrain scene size.
  • Existing large-scene radiance-field methods remain expensive to train and render, without demonstrated real-time rendering.
  • The method trains independent scene chunks with manageable resources and supports parallel processing.
  • Its hierarchy supports efficient distant-content rendering through level selection, smooth interpolation, and optimized intermediate nodes.
  • Datasets contain 5 800 to 28 000 images over 450m to several kilometers, enabling real-time 3D navigation.

2 RELATED WORK

Related work spans mesh-based, neural-field, divide-and-conquer, Gaussian, and LOD approaches to scaling scene reconstruction. The paper positions its method as addressing resource saturation for very large ground-level scenes.

  • Mesh-based methods scale across scene sizes but depend heavily on accurate meshes.
  • NeRF-based methods reach apartment-scale indoor scenes, while the paper’s city-scale datasets are an order of magnitude larger in extent and image count.
  • 3DGS supports arbitrary camera paths and dynamic capacity allocation, but arbitrarily large scenes eventually saturate resources.
  • LOD methods regulate displayed detail using heuristics or resource budgets and provide an established real-time rendering optimization.
  • Few methods handle city-scale ground-level scenes; the cited alternatives require specialized data or target substantially smaller extents.
  • Divide-and-conquer methods partition scenes into independent blocks, with BlockNeRF using overlapping parts for large-scale reconstruction.

3 OVERVIEW AND BACKGROUND

The system subdivides large captures into chunks, builds a coarse scaffold, and represents scenes with volumetric Gaussian primitives rendered by projection and alpha blending.

  • Large-scene optimization begins by subdividing the scene into chunks and creating a coarse scaffold with a small fixed number of Gaussians.
  • The overview proceeds from 3DGS background to hierarchy generation, interior-node optimization, and chunk-based large-scene training.
  • Each 3DGS primitive stores position, covariance, opacity, and spherical-harmonics coefficients for view-dependent appearance.
  • Primitives are projected into screen space and rasterized using alpha blending.
  • Projected means and covariances, together with view-dependent colors, combine through alpha blending to reproduce captured appearance.

4 HIERARCHICAL LOD FOR 3D GAUSSIAN SPLATTING

The method builds a hierarchical 3D Gaussian representation by merging spatially compact primitives, selecting view-dependent cuts, and smoothly interpolating between hierarchy levels. It preserves blended appearance through weighted Gaussian attributes and specialized falloff handling while addressing rotation ambiguity and non-isotropic Gaussians.

  • Hierarchy Generation: The hierarchy merges 3DGS primitives into intermediate Gaussian nodes that retain the same rasterization attributes as leaf nodes.Each node has a mean, covariance, spherical-harmonic coefficients, and opacity or falloff.
  • Hierarchy Generation: Weighted child contributions determine merged means, covariances, spherical harmonics, and appearance, under assumptions of near-isotropy, limited overlap, and low perspective distortion.The weights are derived in screen space so the parent contribution matches the combined child contribution.
  • Hierarchy Generation: Intermediate-node falloff replaces opacity when blended children produce slower-than-Gaussian cumulative falloff, with values above 1 clamped during rendering.This preserves the slower falloff of overlapping child Gaussians.
  • Hierarchy Generation: A top-down AABB BVH with recursive median splits creates spatially compact children before recursively merging their Gaussian representations.The split uses projected Gaussian means along the longest bounding-box axis.
  • Hierarchy Cut Selection and Level Switching: View-dependent cuts select nodes using projected screen granularity, choosing nodes below a target threshold such as 1 pixel to balance rendering performance and visual quality.Granularity is computed from the projected bounding box over the leaf Gaussians contained in each node.
  • Hierarchy Cut Selection and Level Switching: Level switching interpolates Gaussian attributes between parent and children; orientation matching reduces undesired rotations, while scale and rotation are interpolated separately for covariance.The interpolation produces smooth transitions between hierarchy levels.

5 OPTIMIZING AND COMPACTING THE HIERARCHY

The method optimizes hierarchy intermediate nodes across multiple cuts while preserving full-resolution detail, then sparsifies the tree to avoid ineffective parent nodes and supports smooth level transitions.

  • Hierarchy optimization: Intermediate Gaussian nodes can be optimized after hierarchy construction to improve the visual quality they represent.Gradients propagate through intermediate nodes, enabling appearance optimization beyond geometric merging.
  • Level selection: Distance-based LOD descends toward child nodes nearby and selects higher-level nodes farther away.Corresponding hierarchy cuts determine which nodes are rendered at each viewpoint.
  • Level selection: Target granularity is selected from projected screen area rather than forcing hierarchy cuts to match image resolution.This preserves the ability of large, anisotropic Gaussians to model higher-frequency details.
  • Hierarchy optimization: Full-resolution optimization with random target granularities samples multiple cuts while preserving visual details.Random views and target granularities jointly define the cuts used during optimization.
  • Hierarchy optimization: Leaf nodes remain unchanged during multi-level optimization to prevent higher-level updates from degrading leaf quality.Gradients are propagated through interpolation weights and merged-node opacity expressions while child and parent nodes are optimized together.
  • Compacting the hierarchy: The generated hierarchy is sparsified so parent nodes are not only marginally larger than their children and therefore rarely selected.The process retains leaf nodes and identifies the highest-detail nodes required by training-view cuts at τ_min = 3 pixels.

6 LARGE SCENE TRAINING

Large scenes are trained by dividing them into independently processed chunks, then consolidating those chunks into a global hierarchy while adapting optimization to sparse captures and scene-scale artifacts.

  • Chunk-based training: Efficient Gaussian hierarchies render distant scene regions at coarser levels and support divide-and-conquer training through scene chunks.This follows large-data rendering practice and makes very large scenes more tractable.
  • Chunk-based training: Chunk sizes range from 50×50 m for walking captures to 100×100 m for vehicle captures.These chunks remain larger than those used by the original 3DGS approach and must handle sparser capture density.
  • Chunk consolidation: Chunks are processed independently and then consolidated to resolve inconsistencies between neighboring chunks.Consolidation also creates a global hierarchy spanning the scene.
  • Chunk-based training: A coarse full-scene optimization provides a shared scaffold and skybox before individual chunk training.Camera calibration precedes this initialization, while densification and primitive-position optimization are disabled in the coarse pass.
  • Chunk-scale training: Each chunk combines refined SfM points, neighboring scaffold Gaussians, hierarchy construction, optimization, and consolidation.Large chunks differ from conventional radiance-field settings in extent, capture density, and the presence of exposure changes and moving objects.
  • Chunk-scale training: A skybox of 100 000 primitives on a sphere 10× the scene diameter models the sky and prevents inconsistent sky content across chunks.The coarse environment outside each chunk is loaded during chunk training, with only opacity and spherical-harmonic coefficients temporarily optimized.
  • Sparse-capture adaptation: Sparse unbounded-scene captures require adapted densification because screen-space gradient statistics can encourage inappropriate growth.The method also uses monocular depth, scaled and shifted using SfM points, to regularize reconstruction and improve road quality.

7 IMPLEMENTATION, CAPTURE AND PREPROCESSING

The system combines a modified 3DGS implementation with a multi-camera capture workflow, pose estimation, preprocessing, and rendering adaptations for hierarchical Gaussians.

  • Implementation: The method is implemented in C++ and Python/PyTorch on top of 3DGS, with a modified SIBR viewer for fast rendering.PyTorch autodifferentiation is used for hierarchy optimization, except gradients of Eq. 12 are derived manually.
  • Datasets: The captured-scene evaluation includes SmallCity, BigCity, Campus, and a Wayve dataset.The paper reports three self-captured outdoor scenes plus one externally provided scene.
  • Capture: A helmet-mounted rig uses six GoPro HERO6 cameras, or five for Campus, recording 1444×1080 timelapse images every 0.5 seconds.SmallCity and BigCity were captured by bicycle at roughly 6–7 km/h, while Campus was captured on foot.
  • Preprocessing: COLMAP, hierarchical mapping, and per-chunk bundle adjustment address pose estimation for datasets containing 5 800 to 40 000 photographs.Customized parameters are used to keep processing times reasonable.
  • Preprocessing: Preprocessing corrects per-image exposure and removes moving objects, humans, and license plates using CNN-based segmentation and SfM-point checks.Cars and bicycles are removed when their segmented regions have corresponding SfM points.
  • Hierarchy rendering: Merged hierarchy nodes can produce α values above 1, requiring absolute-value opacity activation after optimization and gradient zeroing when rasterization clamps opacity.The rasterizer internally clamps α-blending values to 0.99.
  • Evaluation: A qualitative comparison evaluates the method against prior solutions using a reduced level of detail with τ_ε = 6 pixels.The comparison is shown on a single chunk.

8 RESULTS AND EVALUATION

The evaluation shows that the hierarchical representation supports large-scene rendering with controllable quality, efficient resource use, and real-time performance. Ablations and comparisons attribute gains to chunk training, depth supervision, modified densification, hierarchy optimization, and smooth interpolation.

  • The hierarchical scenes require 6–88 GB of disk space and 11–16 GB peak memory for hierarchy optimization, while each chunk contains 2–8M leaf Gaussians.The hierarchical files are approximately 68% larger than non-hierarchical 3DGS files.
  • Single-chunk evaluation shows the method outperforming previous methods on the large, sparse-capture chunks used in the comparison.The comparison uses identical chunk images for all methods and disables exposure optimization because competing methods do not handle exposure change by default.
  • Optimizing hierarchy interior nodes improves quality at coarser cuts, while smooth interpolation improves PSNR across multiple hierarchy levels.Without interpolation, Wayve PSNR is 25.21 dB, 24.73 dB, and 23.44 dB for τ1, τ2, and τ3.
  • 30+ FPS is achieved on average at high quality and approximately 60 FPS at medium quality across the full-scene camera paths.The original 3DGS renderer exceeds the test system's capacities for the largest scenes.
  • The LOD mechanism reduces rendered load and memory consumption, with larger scenes receiving greater reductions relative to the Gaussian count required by 3DGS.The hierarchy uses cut selection and transfers only the required Gaussians for each frame.
  • Ablations show that consolidation, bundle-adjusted poses, depth supervision, exposure compensation, and hierarchy optimization each address distinct visual-quality issues.Depth supervision particularly improves road appearance, while refined poses reduce blurriness and increase detail.

9 LIMITATIONS, DISCUSSION AND FUTURE WORK

The method remains constrained by capture quality and by the prototype's current fixed-granularity design. The authors identify broader radiance-field uses and dynamic resource-aware LOD as future directions.

  • Visual artifacts primarily arise from poor view-space coverage, calibration errors, and moving distractors in the captured data.The authors treat solving these input-data problems as orthogonal to their method.
  • Free-viewpoint extrapolation is limited across most of the shown environments because capture coverage is limited.The authors suggest that future high-quality priors could improve navigation capability.
  • The hierarchy could support scene-graph representations for radiance fields and applications such as animation and collision detection.
  • The current prototype uses fixed granularity, while dynamic LOD selection and visibility- or distance-based cutoffs are left for future work.These additions are intended to improve quality under a given resource budget and further increase efficiency.

10 CONCLUSION

The paper presents a complete system for real-time novel-view synthesis across street-level scenes spanning several kilometers and tens of thousands of images. Its hierarchy, chunk-based optimization, and level-of-detail rendering make processing and navigating very large environments practical.

  • The method handles street-level scenes spanning several kilometers and tens of thousands of input images with real-time rendering.
  • An efficient 3D Gaussian hierarchy provides smooth level-of-detail rendering for massive scenes.
  • Hierarchy optimization improves the quality/speed tradeoff by refining Gaussians merged into intermediate nodes.
  • Chunk-based optimization enables parallel processing of independent chunks and full processing of tens of thousands of images in a few hours on a compute cluster.
  • The system makes capturing and navigating very large environments accessible through real-time rendering.

A DATA CLEANUP

The capture pipeline cleans long multi-camera sequences by removing blurry frames, masking people and animals, detecting moving vehicles, and blurring license plates. These steps target reconstruction quality and privacy in scenes containing dynamic content.

  • Frames more than 1.5 standard deviations below the mean sharpness are discarded to improve reconstruction quality.Sharpness is measured using the variance of Laplacians, though the detector is not foolproof.
  • People and animals are masked, while vehicle motion is detected using Mask R-CNN masks and corresponding SfM-point errors.Static objects have dense SfM points, whereas moving objects have sparse or inconsistent support.
  • EgoBlur removes vehicle license plates, with Mask R-CNN constraining license-plate masks within segmented vehicle masks.

B POSE ESTIMATION

The pose-estimation pipeline adapts COLMAP for tens of thousands of multi-camera images, then uses hierarchical mapping and chunk-level refinement. A final Procrustes realignment compensates for drift and improves results, while sparse viewing angles motivate additional regularization.

  • A custom sequential-style matcher links each rig camera’s images across captures separated by 2^k, with k∈⟦0, 10⟧.This avoids the prohibitive cost and failures of exhaustive matching on scenes with more than a few thousand images.
  • COLMAP’s hierarchical mapper estimates camera parameters, poses, and SfM points for large datasets in reasonable time.For 40K images, the reported runtime is 110 minutes on two Intel Xeon Gold 6240 CPUs.
  • 30 minutes to 5 hours of Procrustes realignment on an NVIDIA V100 16GB compensates for drift and greatly improves result quality.Runtime depends on chunk complexity, including camera and SfM-point counts.
  • Multi-camera vehicle rigs provide much sparser viewing-angle coverage than typical radiance-field captures, motivating additional regularization.

C.1 Depth Regularization

Sparse vehicle-based captures leave regions such as road surfaces underdetermined, so the method combines monocular depth with SfM scale information and decays depth supervision during chunk optimization. It also compensates per-image exposure changes with affine color transformations.

  • Depth Regularization: Sparse coverage can leave the road underdetermined, allowing a central mound to explain all training views while producing an incorrect scene structure.
  • Depth Regularization: Monocular DPT depth is scaled using per-frame inverse depth from COLMAP SfM points instead of expensive MVS depth.
  • Depth Regularization: Rendered Gaussian depth is compared with the scaled depth target, and gradients reach Gaussian depth and rendering α, affecting opacity and screen-space position.
  • Depth Regularization: Depth-loss weighting decays exponentially from 1 to 0.01 during per-chunk optimization.
  • Exposure Compensation: Per-image exposure changes are modeled with a 3 × 4 affine transformation applied to rendered color and optimized per camera.Exposure optimization uses an identity initialization and Adam, with delayed learning-rate scheduling to establish model coherence first.
Loading 2406.12080v1…