MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors

1 INTRODUCTION

• Slow or static cameras weaken multiview constraints, and existing dynamic view synthesis methods consequently struggle to render high-quality novel views from casual monocular videos.Static-scene novel view synthesis has improved with neural representations and Gaussian splatting, but dynamic scenes from one monocular video remain challenging.Source: §1 · p. 1
• MoDGS adopts monocular depth estimates to help reconstruct dynamic scenes, but framewise depth alone neither associates 3D points across time nor ensures consistent depth values across frames.The paper therefore identifies both deformation-field learning and cross-frame depth inconsistency as challenges.Source: §1 · pp. 1–2
• A 3D-aware initialization scheme uses 3D-space initialization to provide a stronger basis for learning the deformation field than relying solely on 2D flow.The paper reports that this improves rendering quality.Source: §1 · p. 2
• Because estimated depth values can remain inconsistent even after scale normalization, MoDGS supervises stable pixel-depth orderings with an ordinal depth loss.The paper contrasts this with prior scale-invariant and Pearson-correlation losses and says the ordinal loss enables fuller use of estimated depth maps.Source: §1 · p. 2
• Experiments on three datasets and a self-collected in-the-wild dataset report that MoDGS outperforms previous dynamic view synthesis methods by a large margin.The evaluation uses one camera's video as input and another camera's video for evaluation.Source: §1 · p. 2

Background

• Static-scene novel view synthesis has advanced with NeRF, Instant-NGP, and Gaussian Splatting, but dynamic scenes from a single monocular video remain challenging.The passage frames dynamic view synthesis as the focus of this work.Source: §2 · pp. 2–3

Dynamic NeRF

• Dynamic NeRF methods use time-varying radiance fields or canonical-space representations with deformation fields, while grid-based and hybrid variants target faster optimization.Examples include NSFF's forward and backward 3D flow fields and Nerfies and HyperNeRF's deformation to canonical space.Source: §2 · p. 3

Dynamic Gaussian Splatting

• Dynamic Gaussian Splatting work models motion through tracked or deformable Gaussians, 4D primitives, per-Gaussian features, or clustered motion representations.Some methods use deformation fields, including Hexplane-based representations.Source: §2 · p. 3

DVS from Casual Monocular Videos

• Many monocular dynamic-view-synthesis benchmarks use substantial camera motion, unlike casual video, while casual-video methods trade among training cost, extra masks, computational demand, and motion priors.The cited methods include image-based rendering, hybrid static-dynamic representations, joint NeRF-camera estimation, diffusion supervision, and explicit motion representations.Source: §2 · pp. 3–4

Ordinal Relation in Depth Maps

• Prior ordinal-depth and pose methods train on limited depth-order pairs, whereas MoDGS applies ordinal supervision to dense rendered metric-depth maps.The passage notes that direct use of ranking loss as depth supervision had not been explored in those earlier works.Source: §2 · p. 4

3.1 GAUSSIANS AND DEFORMATION FIELDS

• MoDGS deforms canonical-space Gaussians to each timestamp and uses splatting to render images from arbitrary viewpoints.The canonical space is virtual rather than tied to a particular timestamp, and the deformation field is an invertible MLP shared across timestamps.Source: §3 · p. 4, §3.1 · p. 5

3.2 3D-AWARE INITIALIZATION

• Per-frame depth scales are aligned using static regions, then depth maps and 2D optical flow provide 3D correspondences for deformation-field initialization.Depth points from all timestamps are transformed to canonical space, voxel-downsampled, and used to initialize the Gaussians.Source: §3.2 · p. 5

3.3 ORDINAL DEPTH LOSS

• Ordinal depth supervision matches the rendered depth ordering to estimated depth ordering, using pixel pairs rather than requiring depth values to align exactly.The estimated maps may not align after scale normalization, although the order of corresponding pixels can remain stable; the implementation samples 100k pixel pairs.Source: Figure 4 · p. 6, §3.3 · p. 6

3.4 TRAINING OF MODGS

• Training combines a rendering loss with the ordinal depth loss after initializing the Gaussians and deformation field.The paper also describes Pearson correlation as a prior approach for reducing the effects of depth-scale ambiguity.Source: §3.3 · p. 6, §3.4 · p. 7

4.2 COMPARISON WITH BASELINES

• MoDGS achieves the best performance across all reported metrics on both DyNeRF and Nvidia.The comparison is against SC-GS, Deformable GS, and HexPlane.Source: Table 1 · p. 8, §4.2 · p. 9

4.1 EVALUATION PROTOCOLS

• The evaluation trains on one camera and renders from another, avoiding the teleporting-camera training videos used by previous methods.DyNeRF and Nvidia provide synchronized multiview captures; the MCV in-the-wild dataset supports qualitative evaluation only.Source: §4.1 · pp. 7–8
• Rendering quality is evaluated with SSIM, LPIPS, and PSNR after aligning each method’s rendered depth scale to labeled correspondences.This scale alignment addresses ambiguity when the input video is nearly static.Source: §4.1 · p. 8
• The baselines include NeRF-based HexPlane and RoDynRF and Gaussian-splatting-based Deformable-GS and SC-GS.HexPlane receives an additional different-view video because it did not produce reasonable results from a single-camera video alone.Source: §4.1 · p. 8

4.3.1 3D-AWARE INITIALIZATION

• Figure 7 compares rendering results with and without 3D-aware initialization.The ablation studies evaluate initialization and depth loss on DyNeRF.Source: Figure 7 · p. 9, §4.3 · p. 9

4.3.2 ORDINAL DEPTH LOSS

• Ordinal depth loss produces smoother depth interiors while retaining sharp boundaries, whereas no depth loss yields noisy geometry and Pearson loss leaves noisy regions.The qualitative comparison is shown in Figure 8.Source: §4.3.2 · pp. 9–10, Figure 8 · p. 10, Table 2 · p. 10

4.4 LIMITATIONS

• The method reconstructs only visible 3D parts, and novel views of unseen parts can contain artifacts.Other limitations include several-hour training, reliance on estimated depth for static cameras, a consistent-depth-order assumption, inaccurate poses in rapid motion, and difficulty with heavy reflections or low light.Source: §4.4 · p. 10

• MoDGS is designed to render novel-view images from casually captured monocular video rather than relying on teleporting camera motions.Source: §5 · p. 10
• A 3D-aware initialization directly initializes the deformation field to provide a reasonable starting point for optimization.Source: §5 · p. 10
• Ordinal depth loss supervises scene-geometry learning, and experiments on three datasets show superior performance over baseline methods on in-the-wild monocular videos.Source: §5 · p. 10

• The deformation field is initialized for 20k steps, then jointly trained with the 3D Gaussians for another 20k steps.The second stage uses rendering loss and ordinal depth loss.Source: §A.1 · p. 16
• The full training takes about 3.5 hours on an NVIDIA RTX A6000 GPU and uses about 14G memory.The initialization takes 2 hours and subsequent optimization 1.5 hours.Source: §A.1 · p. 16
• MoDGS renders at about 75 FPS.Source: §A.1 · p. 16
• Ordinal depth loss uses weight 0.1 and excludes depth pairs whose normalized difference is 0.02 or less.The depth range is first normalized to [0,1].Source: §A.1 · p. 16

• GeoWizard depth values in [0,1] are converted to real depth using an optimized scale and offset based on normal-map alignment.The optimization takes several seconds, but the resulting depth maps can still differ in scale across timestamps.Source: §A.2 · p. 16
• In the iPhone example, epipolar lines computed from the provided camera poses deviate from matched correspondences, indicating inaccurate poses.Source: Figure 9 · p. 16

A.3 POSE ERRORS IN THE IPHONE DATASET

• The authors do not use DyCheck’s iPhone dataset because its provided camera poses are not accurate enough.They tried rerunning COLMAP but could not obtain reasonable results.Source: §A.3 · pp. 16–17
• The provided epipolar lines do not pass through corresponding points, indicating inaccurate poses.Source: §A.3 · p. 17

A.4 DIFFERENCE FROM DEPTH MAP WARPING

• MoDGS produces more complete reconstructions than warping a training view with estimated single-view depth.Its canonical-space Gaussians and deformation field accumulate information across timestamps, reconstructing content not visible at the current timestamp and correcting single-view depth maps.Source: §A.4 · p. 17

A.5 PEARSON DEPTH LOSS AND SCALE-SHIFT INVARIANT DEPTH LOSS

• Minimizing the normalized-depth L2 distance is equivalent to maximizing the Pearson correlation coefficient.The first term is a simple transformation of the coefficient.Source: §A.5 · pp. 17–18
• The second and third terms in the expression are constant.The section defines the predicted depth prior’s mean and standard deviation; the second term is constant because it is based on the input.Source: §A.5 · p. 18

A.6.1 PREPROCESSING PROCEDURE; A.6.2 COMPARISON WITH SHAPE-OF-MOTION

• MoDGS produces results comparable to Shape-of-Motion on Davis, with more detail in some cases; its Davis camera poses use UniDepth depth maps with Driod-SLAM.Table 3 provides the quantitative Shape-of-Motion comparison on Nvidia, while Figure 11 shows the Davis visual comparison.Source: §A.6.2 · p. 18, Table 3 · p. 18, §A.6.1 · p. 18

A.7 DISCUSSION ABOUT ALPHA VALUE IN ORDINAL DEPTH LOSS; A.8 DISCUSSION ABOUT THE ROBUSTNESS TO THE DEPTH NOISES

• With alpha set empirically to 100, decreasing it worsens results, while increasing it yields comparable results; MoDGS is also reported robust to small added depth noise.The authors attribute the alpha behavior to larger values making the loss focus more on ordinality than depth-value differences.Source: §A.7 · p. 18, §A.8 · pp. 18–19, Table 4 · p. 19

A.10 COMPARISON WITH DEPTH RANKING LOSS

• The ordinal depth loss significantly outperforms the cited depth-ranking losses across all scenes, producing smoother, less noisy depth maps than the ranking-loss alternative.The comparison replaces the loss while holding other configurations fixed; Figure 12 visualizes rendered depth maps.Source: §A.10 · p. 20, Table 8 · p. 20, Figure 12 · p. 21

A.11 ABLATION USING DEPTH FROM VIDEO-DEPTH ESTIMATION METHODS; A.12 RESULTS OF NERF-BASED METHODS USING DEPTH SUPERVISION

• Video-estimated depth improves results, and ordinal loss consistently improves them further; depth supervision only slightly improves HexPlane, which MoDGS still outperforms by a large margin.The video-depth ablation is reported in Table 9, and the HexPlane depth-supervision results in Table 10.Source: §A.11 · p. 21, §A.12 · p. 21, Table 9 · p. 21

A.13 DIFFERENT RANDOM INITIALIZATION METHODS; A.14 LENGTH OF VIDEOS THAT MODGS CAN HANDLE

• MoDGS currently handles videos up to 10 seconds; scaling to roughly 30 seconds is described as possible with a larger deformation MLP and more training iterations.The appendix identifies Kaiming and zero-deformation initialization variants but supplies no comparative outcome in the provided text.Source: §A.14 · p. 22, §A.13 · p. 22, Table 11 · p. 22

A.15 DISCUSSION ABOUT DEPTH CONSISTENCY METHODS; A.16 COMPARISON WITH PERCEPTUAL DEPTH LOSS

• MoDGS depth maps are reported more detailed and temporally stable than those from a video-depth stabilization method; ordinal loss also outperforms LPIPS depth loss quantitatively and visually.Figure 16 compares MoDGS with the stabilization method, while Figure 13 shows smoother, less noisy maps than LPIPS.Source: §A.15 · p. 22, §A.16 · p. 22, Table 12 · p. 22

A.17 RESULTS ON SCENES WITH HIGHLY COMPLEX MOTIONS; A.18 RESULTS ON SCENES WITH RAPID MOTION

• MoDGS reconstructs both moving objects reasonably in the playground and balloon2−2 scenes and produces reasonable results in blurred regions of the rapid-motion rollerblade scene.The authors note that more complex motion may fail because robust depth maps are difficult to predict; rapid motion can also make camera-pose estimates inaccurate.Source: §A.17 · p. 23, §A.18 · p. 23, Figure 14 · p. 24

A.19 DISCUSSION ON HOW TO HANDLE SCENARIOS WITH HEAVY OCCLUSIONS OR SPECULAR.; A.20 DISCUSSION ABOUT THE ROBUSTNESS TO DIFFERENT DEPTH ESTIMATORS

• MoDGS cannot generate new content for occluded regions and may show artifacts on specular objects, while its rendered depth remains stable across varying depth estimators despite input flicker and inconsistencies.The authors identify generative priors and inverse rendering as possible avenues for the occlusion and specular challenges, respectively.Source: §A.19 · p. 23, §A.20 · p. 24, Table 13 · p. 24