PaperlayerView exact revision on arXiv

Source-linked AI summary

DynOMo: Online Point Tracking by Dynamic Online Monocular Gaussian Reconstruction

Jenny Seidenschwarz, Qunjie Zhou, Bardienus Duisterhof, Deva Ramanan, Laura Leal-Taixé

arXiv:2409.02104v2cs.CV

TL;DR

Existing 3D point-tracking methods may require offline processing or multi-view cameras, limiting online tracking from unposed monocular video. DynOMo reconstructs dynamic scenes with 3D Gaussians while estimating camera motion, using feature reconstruction and similarity-guided regularization. The authors report that it outperforms online monocular baselines and is competitive with state-of-the-art offline 2D trackers.

  • Problem

    Existing approaches to 3D point tracking require offline processing or multi-view camera setups, unlike the online unposed monocular setting targeted here.

    Source: §1 · p. 2

  • Method

    DynOMo reconstructs dynamic scenes online with 3D Gaussians, estimates camera motion, and uses image features and similarity-enhanced regularization without correspondence-level supervision.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    DynOMo significantly outperforms baseline methods in the same online monocular setting and is competitive with state-of-the-art offline 2D trackers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work provides an initial baseline for online point tracking and reconstruction with unposed monocular cameras.

    Source: §1 · p. 2

    Temporally inconsistent depth maps can initialize Gaussians at highly incorrect 3D locations that are difficult to recover from.

    Source: §10 · p. 17

Abstract

from arXiv · show

Reconstructing scenes and tracking motion are two sides of the same coin. Tracking points allow for geometric reconstruction [14], while geometric reconstruction of (dynamic) scenes allows for 3D tracking of points over time [24, 39]. The latter was recently also exploited for 2D point tracking to overcome occlusion ambiguities by lifting tracking directly into 3D [38]. However, above approaches either require offline processing or multi-view camera setups both unrealistic for real-world applications like robot navigation or mixed reality. We target the challenge of online 2D and 3D point tracking from unposed monocular camera input introducing Dynamic Online Monocular Reconstruction (DynOMo). We leverage 3D Gaussian splatting to reconstruct dynamic scenes in an online fashion. Our approach extends 3D Gaussians to capture new content and object motions while estimating camera movements from a single RGB frame. DynOMo stands out by enabling emergence of point trajectories through robust image feature reconstruction and a novel similarity-enhanced regularization term, without requiring any correspondence-level supervision. It sets the first baseline for online point tracking with monocular unposed cameras, achieving performance on par with existing methods. We aim to inspire the community to advance online point tracking and reconstruction, expanding the applicability to diverse real-world scenarios.

1. Introduction

Existing point-tracking approaches face a trade-off between 2D ambiguity and methods that require offline processing or multi-view cameras. DynOMo targets online tracking from unposed monocular video, and the authors report strong results against online monocular baselines.

1. Introduction

  • Existing 2D correspondence methods require correspondence-level training annotations, while 2D motion is more ambiguous because depth information is discarded [4] [5] [11] [39].Source: §1 · p. 1
  • Prior 3D tracking approaches either optimize offline with 2D supervision or require 27 static cameras and an initial 3D point cloud [24] [38] [39].Source: §1 · p. 2
  • DynOMo combines online dynamic-scene modeling and camera localization with pretrained depth, semantic-label, and feature encoders, without temporal supervision or correspondence-level supervision.It groups Gaussians for regularization by descriptor similarity rather than proximity and uses densification to explore newly emerging scene content.Source: §1 · p. 2
  • The authors present DynOMo as a baseline for online 2D and 3D tracking with unposed monocular cameras, where temporal correspondence can emerge without correspondence-level optical-flow supervision or known poses.Source: §1 · p. 2
  • DynOMo significantly outperforms baseline methods in the same online monocular setting and is competitive with state-of-the-art offline 2D trackers.Source: §1 · p. 2

2. Related Work

Related work spans dynamic scene reconstruction with implicit and explicit representations, and point tracking in 2D or 3D. DynOMo builds on Gaussian-based reconstruction to address online tracking from unposed monocular video.

2.1. Dynamic Scene Reconstruction

  • 3D Gaussian Splatting represents scenes with explicit primitives and efficient differentiable rasterization, whereas NeRF uses implicit representations and time-consuming volumetric rendering [15] [27].Fast per-sequence optimization motivates adopting 3DGS for dynamic-scene reconstruction in a monocular setting.Source: §2.1 · p. 2

2.2. Point Tracking

  • Point-tracking work ranges from 2D particle and learned trackers [4] [5] [11] [13] [47] to 3D methods that lift points or optimize scene representations [24] [38] [39] [42].Earlier methods include offline reconstruction or posed multi-camera inputs; DynOMo instead targets online tracking with unposed monocular video.Source: §2.2 · pp. 2–3

3. Method

DynOMo performs online point tracking by jointly reconstructing a dynamic Gaussian scene and localizing the camera from unposed monocular video. It combines enhanced image reconstruction signals, semantic separation, and feature-similarity-guided 3D regularization to estimate trajectories.

3. Method

  • DynOMo adapts dynamic 3D Gaussian reconstruction for online tracking through stronger image features and depth supervision, semantic foreground-background separation, and feature-similarity-guided motion regularization.Source: §3 · p. 3

3.1. Dynamic Gaussian Scene Representation

  • Each monocular dynamic Gaussian adds a constant semantic instance label and visual feature to the Gaussian representation, providing feature-space reconstruction constraints and guidance for regularization.Instance labels separate foreground from background and support motion consistency for moving particles.Source: §3.1 · p. 4

3.2. Monocular Online Tracking Pipeline

  • The online pipeline initializes Gaussians from depth, optimizes camera pose, densifies under-represented regions, optimizes Gaussians, then propagates Gaussian and camera states forward.Camera optimization uses previously observed background pixels; Gaussian motion propagation uses a k-nearest-neighbor constant-velocity assumption weighted by feature similarity.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3. Reconstruction Supervision

  • Reconstruction supervision combines rendered RGB, depth, visual-feature, and semantic information to constrain optimization from multiple image perspectives.The feature loss uses L2 distance; RGB, depth, and background-mask reconstruction use L1 distance.Source: §3.3 · p. 5

3.4. 3D Regularization

  • 3D regularization combines local rigidity, rotation similarity, local isometry, and temporal smoothness losses, with feature similarity weighting Gaussian pairs instead of 3D proximity.Instance IDs restrict neighborhoods to the same object; temporal smoothness also constrains background Gaussians to remain static.Source: §3.4 · p. 5

3.5. Trajectory Estimation

  • A 2D pixel may correspond to multiple Gaussians, so the method selects visible Gaussians for a query pixel and follows them over time to obtain 3D trajectories and projected 2D trajectories.Visibility is estimated by accumulating Gaussian opacity over intersecting pixels.Source: §3.5 · pp. 5–6

4. Experiments

The experiments evaluate online monocular point tracking on PanopticSports and TAPVid-DAVIS, then ablate reconstruction losses, 3D regularizers, and Gaussian attribute regularization.

4.1. Evaluation on Panoptic Sports

  • On PanopticSports, DynOMo outperforms D-3DGS for 2D tracking despite a harder camera setting and beats D-3DGS-Mono considerably for 3D.D-3DGS-Mono has 121.9% and 308.8% higher 2D and 3D median translation errors, respectively, than multi-view D-3DGS.Source: §4.1 · p. 6, Table 1 · p. 7

4.2. Evaluation on TAPVid-DAVIS

  • On TAPVid-DAVIS, DynOMo matches OmniMotion and beats Pips/TAPIR offline, significantly outperforms online SplaTAM, but trails SpaTracker slightly.The comparison includes offline trackers and an online SpaTracker variant with a two-frame window.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. The Magic Sauce

  • Visual-feature and depth reconstruction losses matter most; features constrain Gaussian motion to observations, while depth supplies geometric signal without multi-view images.RGB and background masks have lower impact but still help when other predictions are noisy.Source: §4.3 · p. 8
  • Local rigidity is the most influential physical regularizer, while feature-guided weighting and instance-guided kNN help reject noisy Gaussian pairs.Rigidity limits random 3D motion used only to fit 2D observations; instance guidance improves kNN selection when visual features are imperfect.Source: §4.3 · p. 8
  • kNN forward propagation filters outlier motions; temporal smoothness stabilizes Gaussian color and features while allowing viewpoint-driven updates.The authors fix scale and opacity together with instance ID because updating them may cause Gaussians to disappear.Source: §4.3 · p. 8

5. Conclusion

DynOMo performs online point tracking from pose-free monocular videos by reconstructing dynamic 3D scenes and localizing the camera. The authors report emergent trajectories without explicit correspondence supervision, but the method is not yet real-time.

5. Conclusion

  • DynOMo combines dynamic 3D Gaussian reconstruction with simultaneous camera localization for online point tracking from pose-free monocular video.Depth maps, instance IDs, visual features, and 3D regularizers guide Gaussian optimization.Source: §5 · p. 8
  • Accurate point trajectories emerge without explicit 2D/3D correspondence supervision, and the authors position the method as a strong baseline for monocular online tracking.The conclusion names robotic navigation and mixed reality as intended application areas.Source: §5 · p. 8
  • The method has yet to achieve real-time performance; the authors anticipate future work will improve runtime.Source: §5 · p. 8

Reconstruction Supplementary Material

The supplementary material adds implementation and method details, dataset results, ablations, failure cases, and trajectory visualizations.

Reconstruction Supplementary Material

6. Method Details

DynOMo regularizes neighboring Gaussians with physically based rigidity, rotation, and isometry priors, plus temporal smoothness on selected attributes. Semantic weighting and attribute-specific constraints shape how these priors act over time.

Densification Mask

  • New Gaussians are added only in unobserved regions identified by thresholding pixel densities in a densification mask.This mask process is adopted from [14].Source: §6 · p. 11

Physically-Based 3D Regularization

  • Physically based rigidity, isometry, and rotation losses apply to every Gaussian over its k nearest neighbors, with semantic weights for Gaussian pairs.Unlike the 3D-distance weighting in [24], DynOMo uses semantic-based weights.Source: §6 · p. 11
  • The local rigidity loss makes nearby Gaussians undergo similar rigid transformations, an assumption intended to hold within local regions of the same object, including non-rigid objects.The relative rotation ΔR_i is defined from Gaussian i’s rotations across consecutive time steps.Source: §6 · p. 11
  • A short-term rotation loss encourages nearby Gaussians to undergo similar rotation changes; adding it slightly improves performance despite overlap with rigidity regularization.The loss compares normalized quaternions across consecutive time steps.Source: §6 · p. 11
  • Long-term isometry preserves each Gaussian’s initial distance to its k nearest neighbors across time.The loss uses a semantic-based pairwise weight rather than the 3D-distance weight defined in [24].Source: §6 · p. 11

Temporal Smoothness Regularization

  • Temporal smoothness regularizes Gaussian features and colors, as well as the means of background Gaussians.The combined smoothness loss is the sum of feature, color, and background-mean terms.Source: §6 · p. 11

7. Implementation Details

Implementation initializes 3D Gaussians by lifting image pixels using depth and camera projection, then selects feature-similar neighbors within an instance. Optimization settings vary by dataset, and AJ uses threshold-based point and visibility counts.

Gaussian Initialization

  • Each second image pixel is lifted using depth and camera projection to initialize Gaussian means, with RGB, feature, and instance values unprojected alongside it.Rotations start as unit quaternions, scales depend on depth and focal lengths, and opacity is initialized with a visibility bias.Source: §7 · p. 11

Nearest Neighbourhood Selection

  • For each Gaussian, the k=20 neighbors are selected from 2k nearest same-instance Gaussians by feature cosine distance after spatial-neighbor search.The selected neighborhood is used for Gaussian forward propagation and physically based priors.Source: §7 · p. 12

Gaussian Optimization

  • Optimization uses Adam with specified per-parameter learning rates and loss weights, running 500 iterations per time step on Panoptic Sports and 200 on other experiments.Camera optimization also runs for 200 iterations on the other experiments.Source: §7 · p. 12

Average Jaccard Metrics

  • AJ uses threshold-based counts: TPh comprises points within pixel threshold h with correctly computed visibility, while FPh counts other predicted points.The metric’s denominator also includes the number of visible ground-truth evaluation points.Source: §7 · p. 12

8. Comparison on iPhone Dataset

On the iPhone benchmark, DynOMo is evaluated for 2D and 3D tracking in moving-camera real-world captures. The reported comparisons emphasize emergent trajectories without correspondence-level supervision and competitive results despite optimized camera poses.

Benchmark Comparison

  • DynOMo achieves on-par 2D and 3D performance with many existing approaches on iPhone despite online tracking without correspondence-level supervision.The table compares against prior approaches that are offline and mostly use correspondence-level motion supervision.Source: §8 · p. 12, Table 4 · p. 13

Optimized Poses

  • With optimized camera poses, DynOMo’s EPE is highly competitive against approaches using ground-truth or refined poses, but pose inaccuracies reduce high-precision 3D performance.The optimized poses have an ATE RMSE of 10.49 cm.Source: §8 · p. 12, Table 4 · p. 13

Additional Ablations

  • Table 5 covers additional single-part design choices described as having less impact on final performance than the main-paper ablations.Source: Table 5 · p. 13

Depth and Pose Robustness

  • Using the dataset’s original LiDAR signal and original poses causes only a slight performance decrease in both 2D and 3D tracking.Source: §8 · p. 13

9. Additional Ablation Studies

Additional ablations examine temporal regularization, camera-motion modeling, depth priors, and Gaussian selection for trajectories. They also identify failure settings involving occlusion, motion, and limited background observations.

Temporal Smoothness Regularization

  • Fixing features, colors, and background means prevents adaptation to changing predictions or viewpoints, while smoothing opacity, scale, and instance IDs can make Gaussians disappear.Removing all regularization lets Gaussian attributes change freely to fit each frame’s supervision; the tested alternatives significantly degrade performance.Source: §9 · p. 13

Choice of Gaussians in Trajectory Estimation

  • For 2D tracking, the proposed closest-2D-projection Gaussian selection performs best; α-composition performs worse, while selecting by 3D means improves 3D but worsens 2D tracking.The 3D-mean option assumes ground-truth 3D trajectories, which are not generally available.Source: Table 7 · p. 14, §9 · p. 14

Disentangled Camera Motion

  • Fixing camera pose causes a significant performance drop for DynOMo, and explicit camera-motion modeling is described as important for real-world interaction.Source: §9 · p. 14

Depth Priors and Camera Poses

  • On Panoptic Sports, using rendered depth predictions rather than DepthAnything causes a severe drop across 3D tracking metrics, while overall 2D performance changes little and high-precision 2D metrics show a more pronounced drop.Source: §9 · p. 14, Table 6 · p. 14

Failure Cases

  • DynOMo struggles with extreme occlusions, extreme acceleration, and combinations of extreme camera or object motion with little background texture or visibility.Source: Figure 4 · p. 15

10. Failure Cases

DynOMo has difficulty reconstructing motion or camera pose under several demanding conditions, including extreme camera or object motion, severe occlusions, and unreliable depth. The section also identifies challenges when new scene content or newly visible object surfaces appear.

10. Failure Cases

  • For extreme camera motion and backgrounds with little texture, DynOMo struggles to reconstruct motion appropriately, a failure also common in SLAM methods.Source: §10 · p. 15
  • Extreme occlusions can disrupt camera-pose reconstruction, while long-term occlusions make point tracking difficult in DynOMo’s online monocular setting.Source: §10 · p. 15
  • Visualizations show emergent trajectories on scenes with non-rigid motion, occlusions, appearing objects, and fast motion, plus 2D tracking during challenging spinning motions.Source: Figure 5 · p. 16, Figure 6 · p. 17
  • Extreme object acceleration breaks the constant-velocity assumption used to forward-propagate points, making sudden motion changes difficult to model.Source: §10 · p. 17
  • New objects appearing in front of previously observed regions may receive no new Gaussians, while newly visible sides of spinning objects can be difficult to populate.Source: §10 · p. 17
  • Noisy depth maps can degrade supervision, and depth maps inconsistent over time may place added Gaussians incorrectly, making recovery highly challenging.Source: §10 · p. 17

11. Additional Visualizations

Additional visualizations show DynOMo’s emergent trajectories on challenging TAPVID-Davis scenes and robust 2D point tracking on the iPhone dataset.

11. Additional Visualizations

  • TAPVID-Davis visualizations include occlusions, appearing objects, highly non-rigid motion, changing lighting, and very fast object motion; iPhone examples show emergent trajectories and robust 2D point tracking.Source: §11 · p. 17

Related papers

Loading 2409.02104v2…