PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos

Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wetzstein, Leonidas Guibas

arXiv:2406.18717v2cs.CV

TL;DR

Casual monocular videos make dynamic-scene reconstruction difficult because geometry, motion, and radiance must be recovered from pixels alone, while current dynamic Gaussian methods are underconstrained without multi-view information. Dynamic Gaussian Marbles combines isotropic Gaussians, divide-and-conquer optimization, and image- and geometry-level priors. It outperforms other dynamic Gaussian methods and is on-par with NeRF-based methods while retaining Gaussian advantages.

  • Problem

    Casual monocular videos require recovering 3D geometry, motion, and radiance from pixels alone, and dynamic Gaussian methods are underconstrained without multi-view information.

    Source: §1 · pp. 1–2

  • Method

    Dynamic Gaussian Marbles uses isotropic marbles, divide-and-conquer learning, and image-space and 3D priors to guide optimization.

    Source: §1 · p. 2

  • Results

    Gaussian Marbles greatly outperforms other dynamic Gaussian methods and is on-par with NeRF-based methods in the casual monocular setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method retains efficient rendering, tracking, and editability while improving dynamic Gaussian reconstruction and tracking accuracy.

    Source: §1 · p. 2, §6 · p. 8

    Gaussian Marbles does not comprehensively solve the challenging problem of open-world dynamic monocular novel-view synthesis.

    Source: §6 · p. 8

Abstract

from arXiv · show

Gaussian splatting has become a popular representation for novel-view synthesis, exhibiting clear strengths in efficiency, photometric quality, and compositional edibility. Following its success, many works have extended Gaussians to 4D, showing that dynamic Gaussians maintain these benefits while also tracking scene geometry far better than alternative representations. Yet, these methods assume dense multi-view videos as supervision. In this work, we are interested in extending the capability of Gaussian scene representations to casually captured monocular videos. We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. Building off this finding, we propose a method we call Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. First, we use isotropic Gaussian "marbles'', reducing the degrees of freedom of each Gaussian. Second, we employ a hierarchical divide and-conquer learning strategy to efficiently guide the optimization towards solutions with globally coherent motion. Finally, we add image-level and geometry-level priors into the optimization, including a tracking loss that takes advantage of recent progress in point tracking. By constraining the optimization, Dynamic Gaussian Marbles learns Gaussian trajectories that enable novel-view rendering and accurately capture the 3D motion of the scene elements. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality, and is on-par with non-Gaussian representations, all while maintaining the efficiency, compositionality, editability, and tracking benefits of Gaussians. Our project page can be found here https://geometry.stanford.edu/projects/dynamic-gaussian-marbles.github.io/.

1 INTRODUCTION

Casual monocular videos make dynamic-scene reconstruction from pixels alone challenging, while existing dynamic Gaussian methods are designed for simultaneous multi-view capture. Dynamic Gaussian Marbles targets this setting with constrained Gaussians, hierarchical optimization, and image- and geometry-level priors, outperforming other dynamic Gaussian methods and comparing favorably with NeRF-based methods.

1 INTRODUCTION

  • Casual monocular novel-view reconstruction must recover 3D geometry, motion, and radiance from pixels alone.Robust solutions could extend capabilities in video production, 3D content creation, virtual reality, and synthetic data generation.Source: §1 · p. 1
  • Existing dynamic Gaussian methods are underconstrained without multi-view information, so Gaussian Marbles combines an optimization strategy, depth and motion estimates, and geometry regularization.The method targets casual videos captured by a single camera moving smoothly around a dynamic scene.Source: §1 · p. 2
  • Gaussian Marbles uses isotropic marbles, divides videos into independently optimized subsequences that are merged, and guides them with image-space and 3D priors.Image priors include segmentation, point tracking, and depth; 3D priors include rigidity and Chamfer regularization.Source: §1 · p. 2
  • Gaussian Marbles greatly outperforms other dynamic Gaussian methods in the casual monocular setting and is on-par with NeRF-based methods.It retains efficient rendering, tracking, and editability.Source: §1 · p. 2

2 RELATED WORK

Prior work develops Gaussian scene representations for efficient rendering and editing, then extends them to dynamic scenes through trajectories, deformation networks, and other 4D models. Other neural representations also address dynamic scenes, while dynamic Gaussians offer tracking, rendering-speed, and editability advantages.

2.1 Gaussian Splatting

  • 3D Gaussian Splatting represents scenes with anisotropic Gaussians that support efficient color rendering and compositional editing.Prior work also explores its speed, quality, and applications across several downstream tasks.Source: §2.1 · p. 2

2.2 Gaussians for Dynamic Scenes

  • Dynamic Gaussian methods model motion with per-Gaussian trajectories, sparse control points or motion bases, deformation networks, or 4D Gaussians.Trajectory-based methods often track scene geometry well; deformation networks warp canonical Gaussians but can be difficult to learn and prone to local minima.Source: §2.2 · p. 2

2.3 Other Neural Scene Representations for Dynamic Scenes

  • Earlier dynamic-scene work extends NeRFs into 4D or combines canonical NeRFs with time-conditioned deformation fields, alongside methods for casual monocular capture.The deformation field can help disentangle motion and geometry, resulting in a more constrained and better-behaved scene.Source: §2.3 · pp. 2–3
  • Dynamic Gaussians offer tracking, fast rendering, and compositional editability; Gaussian Marbles builds trajectories by merging per-frame sets and optimizing rendering, tracking, and geometry losses.The overview describes a bottom-up divide-and-conquer process that produces multi-frame Gaussian trajectories for rendering at a timestep.Source: §2.3 · p. 3, Figure 2 · p. 3

3 PRELIMINARIES

3D Gaussian Splatting represents scenes as parameterized Gaussians, transforms their covariance into camera coordinates, and rasterizes them efficiently. A monocular RGBD example illustrates that isotropic marbles can generalize to novel views where anisotropic Gaussians produce artifacts.

3.1 3D Gaussian Splatting

  • A 3D Gaussian is parameterized by its mean, rotation, scale, color, and opacity, with rotation and scale forming its covariance matrix.In camera coordinates, the covariance is computed using the camera viewing transformation and the Jacobian of the approximately affine projective transformation.Source: §3.1 · p. 3
  • After 100K iterations on one monocular RGBD image, anisotropic Gaussians produce novel-view artifacts while Gaussian marbles generalize well.Both reconstruct the training view perfectly.Source: Figure 3 · p. 3
  • Tile-based rasterization sorts influencing Gaussians by depth and alpha-composites them, making Gaussian Splatting highly efficient.The image is divided into 16 × 16 tiles; the passage reports rendering over 100 times faster than volumetric rendering approaches.Source: §3.1 · p. 3

4 METHOD

Gaussian Marbles represents dynamic scenes with isotropic Gaussian marbles and builds longer trajectories through iterative merging of adjacent sets. Rendering, tracking, and geometry losses guide the optimization.

4.2.1 Overview.

  • The divide-and-conquer strategy joins temporally adjacent marble sets through motion estimation, merging, and global adjustment, doubling trajectory length at each iteration.The method initializes per-frame marble sets with length-one trajectories and iteratively merges them into longer trajectories.Source: §4 · p. 3, §4.2.1 · p. 4, Figure 4 · p. 4

4.1.2 Why Isotropic Marbles?

  • Isotropic Gaussian marbles replace anisotropic Gaussians, whose extra degrees of freedom fit a monocular training image but can produce artifacts in novel views.In the toy experiment, simpler marbles generalized to novel views.Source: §4.1.2 · p. 4, §4.1.1 · p. 4

4.2.3 Motion Estimation Phase.

  • For motion estimation, adjacent sets are optimized in both directions to extend trajectories across the partner set’s frames, using constant velocity to initialize missing-frame positions.The initial per-frame marbles use depth maps and temporally consistent segmentations to initialize positions and semantic instances.Source: §4.2.3 · p. 4, §4.2.2 · p. 4

4.2.4 Merging.

  • After motion estimation, overlapping sets are merged by taking their union; global adjustment then jointly updates colors, scales, opacities, and trajectory translations.Low-opacity or small-scale marbles are dropped and random downsampling keeps set size constant.Source: §4.2.4 · p. 4, §4.2.5 · pp. 4–5

4.3 Losses

  • The objective combines point-tracking constraints on Gaussian trajectories with image, disparity, and segmentation rendering losses and an LPIPS image loss.Tracking uses off-the-shelf 2D point tracks to regularize trajectory motion.Source: §4.3.1 · p. 5, §4.3.2 · p. 5

4.3.3 Geometry Losses.

  • Geometry losses encourage locally rigid and instance-level isometric motion, while randomized two-way Chamfer comparisons align merged sets in 3D without overfitting differences in observed scene content.The Chamfer loss is motivated by cloudy 3D merges despite sharp training-view projections, especially when depth estimates vary across time.Source: §4.3.3 · p. 5

5 EXPERIMENTS

Across the evaluated settings, Gaussian Marbles outperforms Gaussian baselines and is on-par with NeRF baselines on DyCheck iPhone; on Total-Recon it outperforms general-purpose dynamic NeRFs but remains behind the domain-specific method. Experiments also assess tracking, editing, runtime, memory, and component ablations.

5.1 Datasets

  • Evaluation covers Nvidia single-camera training with other cameras for evaluation, DyCheck's official and camera-pose-withheld settings, and four Total-Recon scenes.Nvidia uses camera 4 for training and cameras 3, 5, and 6 for evaluation; the camera-pose-withheld DyCheck setting transfers camera motion into scene dynamics.Source: §5.1 · p. 5, §5.1.1 · p. 6, §5.1.2 · p. 6

5.2.2 Runtime and Memory

  • Training takes 3.5–8 hours on Nvidia and 5–9 hours on DyCheck, with inference above 200Hz; runtime and memory grow with video length.Reported training peaks range from 6–10GB on Nvidia and 13–22GB on DyCheck; implementation details also specify dataset-dependent initialization and curriculum lengths.Source: §5.2.1 · pp. 6–7, §5.2.2 · p. 7

5.3 Dynamic Novel View Synthesis with Gaussians

  • Gaussian Marbles significantly outperforms Gaussian baselines across the DyCheck iPhone and Nvidia Dynamic Scenes settings, including evaluations with less multi-view information.Table 1 reports PSNR and LPIPs for iPhone with and without camera pose and Nvidia Dynamic Scenes.Source: Table 1 · p. 6, §5.3 · p. 7
  • Without camera poses on DyCheck iPhone, 4D Gaussians average static information across frames while Dynamic Gaussians diverge from scene geometry and overfit the training view.On a real-world tiger video, Gaussian Marbles achieves high-quality reconstruction and novel-view synthesis while 4D Gaussians struggles to reconstruct the training sequence.Source: §5.3 · p. 7

5.4 Tracking and Editing with Gaussians

  • On DyCheck iPhone with camera poses, Gaussian Marbles significantly outperforms other NeRF- and Gaussian-based methods in tracking and slightly improves on CoTracker predictions.The evaluation reports PCK-T at a 5% interval.Source: Table 2 · p. 6, §5.4 · p. 7
  • Gaussian Marbles tracks foreground and background geometry in the camera-pose-withheld setting, and its tracks support temporally consistent video edits.The paper demonstrates propagating a color edit across a tiger video.Source: §5.4 · p. 7

5.5 Dynamic Novel View Synthesis with NeRF

  • On Total-Recon, Gaussian Marbles significantly outperforms HyperNeRF and D2NeRF but remains behind domain-specific Total-Recon, which uses human and quadruped priors.On DyCheck iPhone and Nvidia, its novel-view synthesis is on-par with NeRF baselines.Source: Table 3 · p. 6, §5.5 · p. 7, Table 4 · p. 7

5.6 Ablations

  • Ablations on three Nvidia scenes indicate that each design choice helps novel-view synthesis, with motion estimation and global adjustment both crucial to reconstruction quality.Table 5 reports the component ablations.Source: §5.6 · pp. 7–8, Table 5 · p. 8

6 LIMITATIONS AND CONCLUSION

Gaussian Marbles improves monocular dynamic novel-view synthesis, but does not comprehensively solve open-world cases; its results can be constrained by errors in 2D predictions and rapid non-rigid motion.

6 LIMITATIONS AND CONCLUSION

  • Errors in depth, segmentation, or tracking predictions can lead the optimization to suboptimal results.The method relies on 2D image priors.Source: §6 · p. 8
  • Geometric priors may guide optimization incorrectly in scenes with rapid, non-rigid motion.The authors identify progress in 3D priors and visual tracking as vital for this setting.Source: §6 · p. 8
  • Gaussian Marbles significantly outperforms previous Gaussian methods in novel-view synthesis and reconstruction methods in tracking accuracy, while remaining suited to tracking and editing.The authors state that it does not comprehensively solve open-world dynamic monocular novel-view synthesis.Source: §6 · p. 8

A.1 Initializing Gaussian Marbles

Gaussian Marbles initializes per-frame Gaussians from depth-derived point clouds, with depth sources varying by dataset and capture type.

A.1 Initializing Gaussian Marbles

  • Each input frame is initialized with Gaussians from a point cloud obtained by unprojecting its depth map.Depth maps come from device LiDAR for DyCheck iPhone and Total-Recon, and DepthAnything for Nvidia Dynamic Scenes.Source: §A.1 · p. 12
  • For Nvidia Dynamic Scenes, per-frame depth shift and scale are solved using a COLMAP point cloud from all 12 calibrated cameras.This adjustment ensures monocular depth estimates have the correct shift and scale for novel-view evaluation.Source: §A.1 · p. 12
  • Qualitative evaluations on DAVIS, YouTube-VOS, and internet videos use metric depth estimated with DepthAnythingV2.After unprojection, the point cloud is filtered with Open3D statistical outlier removal and downsampled by sampling nearer points more often.Source: §A.1 · p. 12

A.2 Motion Estimation Phase

During motion estimation, rendering between Gaussian sets can include samples from the target set without updating that set's gradients.

A.2 Motion Estimation Phase

  • Motion estimation renders Gaussians from G_b with probability 0.5 when rendering G_a into a frame covered by G_b.Gradient updates are not propagated to G_b in this setting.Source: §A.2 · p. 12
  • The learned trajectory translations ΔX use a learning rate of 0.002 during motion estimation.Source: §A.2 · p. 12
  • The authors find motion-estimation optimization relatively robust to the trajectory-translation learning-rate parameter.Source: §A.2 · p. 12

A.3 Merging

The merging stage prunes and downsamples merged Gaussian sets to keep their number constant, with an optional qualitative scale adjustment.

A.3 Merging

  • Each merge prunes Gaussians with opacity below 0.02 or scale below 0.002, then randomly downsamples to maintain a constant set size.Source: §A.3 · p. 12
  • After merging, reducing each Gaussian's scale to 85% of its original size is slightly favorable qualitatively.The authors report that this adjustment has a marginal effect.Source: §A.3 · p. 12

A.4 Global Adjustment Phase

Global adjustment uses random Gaussian dropout and specified learning rates for color, opacity, scaling, and trajectory translation.

  • Global adjustment randomly drops 50% of Gaussians with probability 0.5.Source: §A.4 · p. 12
  • The color, opacity, and scaling learning rates are 0.0025, 0.05, and 0.002, respectively.Source: §A.4 · p. 12
  • Trajectory translation ΔX uses a 0.0004 learning rate, five times smaller than during motion estimation.Source: §A.4 · p. 12

A.5 Rendering a Sliding Window at Inference

Sliding-window rendering smooths transitions between trajectory subsequences, while trajectory stitching supports tracking across them. The appendix also specifies dataset settings and guidance for real-world videos.

A.5 Rendering a Sliding Window at Inference

  • An extra motion-estimation phase expands trajectories so adjacent sets cover each frame, and sliding-window sampling improves temporal coherence at subsequence boundaries.This slightly harms single-frame rendering quality; the reported metrics use this approach.Source: §A.5 · p. 12, Figure 8 · p. 12

A.6 Tracking at Test Time

  • Cross-subsequence tracking forecasts trajectories with constant velocity, then matches neighbors within the same segmentation class using 3D and projected 2D distances.Source: §A.6 · p. 12

A.6.1 Additional Details on DyCheck Evaluation.

  • DyCheck evaluation uses 360×480 images and omits the 3D alignment Chamfer loss because its LiDAR depth maps are metrically consistent.Source: §A.6.1 · p. 13

A.6.2 Additional Details on Nvidia Evaluation.

  • Nvidia evaluation resizes images to 248 pixels high and omits the 3D alignment Chamfer loss because COLMAP-scaled depth maps are sufficiently consistent.Source: §A.6.2 · p. 13

A.6.3 Additional Details on Total-Recon Evaluation.

  • Total-Recon evaluation uses 360×480 images and omits the 3D alignment Chamfer loss because its LiDAR depth maps are metrically consistent.Source: §A.6.3 · p. 13

A.6.4 Additional Details on Real World Videos.

  • For real-world videos, the curriculum stops at 32-frame subsequences, while extending to 128 frames causes an observable but not excessive quality drop.The authors suggest continuing to 128-frame subsequences for applications requiring longer-term tracking, such as edit propagation.Source: §A.6.4 · p. 13

Related papers

Loading 2406.18717v2…