PaperlayerView exact revision on arXiv

Source-linked AI summary

St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World

Haiwen Feng, Junyi Zhang, Qianqian Wang, Yufei Ye, Pengcheng Yu, Michael J. Black, Trevor Darrell, Angjoo Kanazawa

arXiv:2504.13152v1cs.CV

TL;DR

Dynamic videos require reconstruction and correspondence to be handled together, but existing approaches commonly separate these tasks and world-frame tracking remains underrepresented. St4RTrack uses shared-world pointmaps and reprojection-based adaptation to jointly reconstruct and track dynamic content, achieving state-of-the-art performance across synthetic and real-world datasets.

  • Problem

    Dynamic 3D reconstruction and point tracking are typically treated as separate tasks, while prior benchmarks largely evaluate 3D tracking only in camera coordinates.

  • Method

    St4RTrack predicts two pointmaps at a shared timestamp and world coordinate, then uses reprojection losses for adaptation without 4D labels.

  • Results

    St4RTrack achieves state-of-the-art performance on synthetic and real-world datasets for world-frame 3D tracking and dynamic reconstruction.

  • Takeaways & Limitations

    The unified representation supports dense world-frame tracking and dynamic reconstruction within a single inference, with further improvement through test-time adaptation.

  • Takeaways & Limitations

    The per-frame setting does not fully resolve scale misalignment, large camera movements, or occlusions, and highly complex motions remain difficult.

Abstract

from arXiv · show

Dynamic 3D reconstruction and point tracking in videos are typically treated as separate tasks, despite their deep connection. We propose St4RTrack, a feed-forward framework that simultaneously reconstructs and tracks dynamic video content in a world coordinate frame from RGB inputs. This is achieved by predicting two appropriately defined pointmaps for a pair of frames captured at different moments. Specifically, we predict both pointmaps at the same moment, in the same world, capturing both static and dynamic scene geometry while maintaining 3D correspondences. Chaining these predictions through the video sequence with respect to a reference frame naturally computes long-range correspondences, effectively combining 3D reconstruction with 3D tracking. Unlike prior methods that rely heavily on 4D ground truth supervision, we employ a novel adaptation scheme based on a reprojection loss. We establish a new extensive benchmark for world-frame reconstruction and tracking, demonstrating the effectiveness and efficiency of our unified, data-driven framework. Our code, model, and benchmark will be released.

1. Introduction

St4RTrack unifies dynamic 3D reconstruction and point tracking by predicting world-consistent geometry and motion directly from RGB frame pairs. Its reprojection-based adaptation enables training and test-time refinement without requiring 4D supervision, while WorldTrack evaluates the resulting world-frame capability.

  • Motivation: Dynamic reconstruction and correspondence are usually treated separately, although 3D geometry and correspondence are linked through scene evolution and 3D motion.Tracking scene dynamics restores the interplay between reconstruction and correspondence that exists in rigid multi-view geometry.
  • Contribution: St4RTrack jointly reconstructs static and dynamic content and tracks it in a consistent world coordinate frame directly from RGB video.World-frame tracking separates scene motion from camera motion and avoids an additional alignment stage.
  • Representation: The method redefines pointmaps so two predictions from frames at different timestamps share one timestamp and world, preserving geometry and temporal correspondence.For images (Ii, Ij), both pointmaps are predicted in Ii’s coordinate frame at the time specified by Ij.
  • Training and adaptation: Reprojection loss enables 4D reconstruction training on in-the-wild videos without 4D supervision by jointly establishing cameras, geometry, and motion.Camera parameters for the second frame are differentiably computed via PnP, then used to project the first pointmap for reprojection-based training.
  • Evaluation: On the WorldTrack benchmark, the unified method outperforms strong multi-component baselines, and feed-forward predictions improve through test-time adaptation.The framework is evaluated for tracking and reconstruction in world coordinates.

2. Related Works

Prior work addresses camera estimation, dynamic reconstruction, and point tracking through largely separate pipelines. St4RTrack instead combines reconstruction and tracking in a feed-forward framework that also supports adaptation on unlabeled videos.

  • Camera Estimation and Scene Reconstruction: Classical SfM and SLAM jointly estimate camera motion and geometry but are primarily designed for static scenes.Learning-based monocular and video depth methods have expanded opportunities for dynamic-scene reconstruction.
  • 2D/3D Point Tracking: Optical and scene-flow methods provide dense motion vectors but are inherently short-ranged and struggle with large displacements and occlusions.Supervised long-range tracking improves tracking over extended periods, but these methods produce only 2D pixel trajectories.
  • Joint Dynamic Reconstruction and Tracking: Dynamic reconstruction methods such as MonST3R reconstruct pointmaps in their own timestamps, which provides geometry but does not establish temporal correspondence.St4RTrack changes the pointmap interpretation to predict motion-linked geometry across frames.
  • 2D/3D Point Tracking: Recent 3D trackers lift points using monocular depth priors but operate in camera coordinates, without explicitly separating scene motion from camera motion.St4RTrack addresses this distinction through joint camera and scene-motion estimation for world-space tracking.
  • Joint Dynamic Reconstruction and Tracking: Unlike optimization-heavy or supervised pipelines, St4RTrack performs reconstruction and tracking feed-forward and supports test-time adaptation on unlabeled videos.The same architecture is intended to approach the quality of optimization-based methods while retaining feed-forward inference.

3. Simultaneous Reconstruction and Tracking

St4RTrack unifies dynamic 3D reconstruction and tracking by predicting time-dependent pointmaps in a shared world coordinate frame. The representation preserves geometry and 3D correspondence across frames, supports long-range tracking through an anchor frame, and enables reprojection-based adaptation without explicit 4D supervision.

  • Unified 4D representation: Two appropriately defined pointmaps encode dynamic scene geometry and temporal 3D correspondence in a shared world coordinate system.The representation describes scene points at specific timestamps while preserving a common reference across frames.
  • Unified 4D representation: The tracking branch predicts the first frame’s content at a later timestamp, while the reconstruction branch predicts the later frame’s content at its own timestamp.Both outputs are expressed relative to the world coordinate system established by the first frame.
  • Video chaining: Pairing every video frame with the first frame aligns all predictions to one reference, naturally producing world-frame long-range correspondences and per-frame reconstruction.The first frame serves as the world coordinate frame for the sequence.
  • Relation to prior work: Unlike MonST3R’s timestamp-specific geometry, St4RTrack predicts the first image’s 3D content at future timestamps to preserve temporal correspondence in dynamic scenes.In static scenes, the formulation reduces to the reconstruction-correspondence relationship used by DUSt3R.
  • Model implementation: A siamese transformer processes paired frames concurrently and generates two pointmaps in the common reference frame established by the first view.The decoder combines within-frame self-attention, cross-frame attention, and an MLP before producing the outputs.
  • Training and adaptation: Differentiable PnP and reprojection losses align projected 3D structure with 2D trajectories, monocular depth, and cross-branch 3D consistency.This supports target-specific test-time refinement without explicit 4D labels, while instance-level adaptation freezes the reconstruction branch.

4. Experiments

St4RTrack is evaluated for unified world-coordinate tracking and reconstruction on synthetic and real-world data, including a new WorldTrack benchmark. Results show strong tracking, reconstruction, qualitative consistency, and benefits from test-time adaptation.

  • Experimental setup: The benchmark reports APD3D after global median alignment for both all points and dynamic points.Predicted trajectories are compared over the first 64 frames at thresholds of 0.1m, 0.3m, 0.5m, and 1.0m.
  • Experimental setup: WorldTrack evaluates dense 3D tracking in world coordinates, addressing prior benchmarks’ focus on camera-coordinate systems.The benchmark contains real-world and synthetic datasets, with additional synthetic sets providing both camera and scene motion.
  • Dynamic 3D reconstruction: St4RTrack also achieves state-of-the-art world-coordinate 3D reconstruction and outperforms MonST3R even with global alignment.Because the reconstruction head is frozen during adaptation, reconstruction results remain similar with test-time adaptation.
  • Joint tracking and reconstruction: Qualitative results show aligned pair outputs, stable accumulated reconstruction in static regions, and accurate, smooth accumulated tracking over time.The visualization presents input video, overlaid pointmaps, accumulated reconstruction, and accumulated tracking.
  • Ablation study: Ablations show that synthetic pretraining aligns the two pointmap branches, while test-time adaptation corrects drifting points and improves dynamic-region geometry.The adapted model produces fewer drifting points and sharper renderings, especially near moving-object boundaries.

5. Discussion

St4RTrack offers a minimalist unified approach to dynamic scene geometry and motion, but its per-frame setting leaves several challenges unresolved. Limited pretraining diversity and realism also necessitate test-time adaptation, while highly complex motions remain difficult.

  • Per-frame processing does not fully resolve scale misalignment, large camera movements, or occlusions.
  • Limited diversity and realism in pretraining geometry and motion require test-time adaptation for improved robustness out of distribution.
  • St4RTrack still struggles with highly complex motions, motivating larger training sets and potentially large-scale pretraining.

6. Conclusion

St4RTrack unifies world-coordinate 3D point tracking and dynamic reconstruction from monocular videos, while WorldTrack evaluates both tasks systematically. The method achieves state-of-the-art performance on synthetic and real-world datasets and supports test-time adaptation beyond fully supervised training.

  • St4RTrack simultaneously performs 3D point tracking and dynamic reconstruction in world coordinates from monocular videos using a unified representation.
  • WorldTrack provides a benchmark for evaluating dynamic 3D scene geometry and motion estimation in a global reference frame.
  • The method achieves state-of-the-art performance on synthetic and real-world datasets while enabling test-time adaptation beyond fully supervised paradigms.

Supplementary Material

The differentiable pose-estimation procedure replaces non-differentiable RANSAC updates with a Gauss–Newton refinement, allowing reprojection loss gradients to refine the predicted 3D pointmaps.

  • RANSAC-PnP first estimates camera pose by matching per-pixel 2D–3D correspondences in the reconstruction pointmap.
  • Because RANSAC is non-differentiable, the method refines its detached pose estimate with one derivative-based Gauss–Newton step.
  • F(P*) represents the flattened reprojection error across all N points, while J is its Jacobian evaluated at P*.
  • JᵀJ approximates the negative-log-likelihood Hessian and JᵀF(P*) its pose gradient, directing the increment toward lower reprojection error.
  • With P* detached, only the Gauss–Newton increment remains differentiable, enabling reprojection loss backpropagation through pose into the 3D pointmaps.

B.1. Datasets

The benchmark combines transformed world-coordinate annotations from real datasets with synthetic test sets, then filters samples for depth quality, realism, and camera consistency.

  • Dataset Preparation: Real-world annotations from ADT and Panoptic Studio are transformed from camera coordinates into world coordinates using paired camera parameters.
  • Dataset Preparation: Synthetic evaluation uses test sets from Point Odyssey and Dynamic Replica Dataset, with approximately 1,000 query points sampled per sequence.
  • Filtering Criteria: Filtering removes unreliable long-range TUM depth, unrealistic Kubric-style Point Odyssey sequences, ambiguous-depth scenes, and frames with dynamic intrinsics.

B.2. Additional Quantitative Evaluation

Additional evaluations use world-coordinate tracking and reconstruction metrics under global median or SIM(3) alignment. St4RTrack achieves state-of-the-art performance across these evaluations, including EPE on all tracking sub-test sets.

  • APD3D measures world-coordinate trajectory accuracy, while EPE measures complementary endpoint localization accuracy.Both metrics are reported after global median alignment for tracking and under SIM(3) alignment in additional evaluations.
  • St4RTrack attains state-of-the-art EPE on all WorldTrack sub-test sets.
  • Comprehensive SIM(3)-aligned evaluations show state-of-the-art performance in most tracking and reconstruction scenarios.SIM(3) combines SE(3) alignment with a global scale factor.

B.3. Qualitative Evaluation

The paper presents qualitative reconstruction and tracking results from its fully feed-forward approach on the WorldTrack benchmark, alongside specified test-time adaptation settings.

  • Qualitative reconstruction results are shown for TUM-Dynamics and Point Odyssey, while tracking results cover four WorldTrack datasets.The tracking datasets are Point Odyssey, Dynamic Replica, Arial Digital Twin, and Pnapotic Studio.
  • WorldTrack evaluation uses sequence-level instance adaptation and dataset-level domain adaptation as two test-time adaptation settings.Instance adaptation fine-tunes a separate model for each of 50 sequences; domain adaptation fine-tunes one model on the entire dataset.
  • The loss weights are λtraj = 1, λdepth = 10, and λalign = 5.

C.2. Ablation Studies

The ablation study evaluates feed-forward inference, two adaptation variants, and the individual contributions of the test-time adaptation components.

  • Table 6 reports median-scale world-coordinate 3D tracking using end-point error and APT3D for Dynamic Replica and Pnapotic Studio.
  • Both test-time adaptation variants substantially improve performance over the feed-forward mode.
  • Instance-level adaptation achieves the highest accuracy because it specializes to each test sequence.
  • Removing trajectory, depth, alignment, or synthetic-pretraining components reduces performance across all evaluated scenarios.

D. Additional Results

Additional results visualize feed-forward and test-time-adapted reconstruction and tracking, including pairwise outputs and accumulated pointmaps anchored to a reference frame.

  • Reconstruction examples on Point Odyssey compare sampled input frames, ground-truth pointmaps, predicted pointmaps, and median-scale aligned results.
  • WorldTrack tracking visualizations align predicted and ground-truth tracks in both 2D and 3D across four datasets.The datasets are Point Odyssey, Dynamic Replica, Arial Digital Twin, and Pnapotic Studio.
  • Feed-forward inference results show input videos, pairwise tracking and reconstruction outputs, accumulated reconstruction pointmaps, and accumulated tracking pointmaps.
  • The middle frame is used as the reference frame for point tracking in the fully feed-forward visualization.
  • Test-time adaptation results use the same four-part visualization, including pairwise outputs and accumulated reconstruction and tracking pointmaps.
Loading 2504.13152v1…