Source-linked AI summary
SpatialTrackerV2: 3D Point Tracking Made Easy
Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, Xiaowei Zhou
TL;DR
Monocular 3D point tracking needs scalable alternatives to per-scene optimization and better joint reasoning over geometry, camera motion, and object motion. SpatialTrackerV2 unifies these components in a differentiable feed-forward system trained across heterogeneous data, achieving state-of-the-art tracking and strong depth, pose, and reconstruction results with substantially faster inference.
Problem
Existing 3D tracking relies on computationally expensive per-scene optimization or ground-truth 3D trajectories and lacks joint reasoning over geometry, camera motion, and object motion.
Method
SpatialTrackerV2 decomposes world-space motion into video depth, camera ego-motion, and object motion within a fully differentiable front-end and back-end pipeline trained across heterogeneous datasets.
Results
21.2 AJ and 31.0 APD3D establish state-of-the-art TAPVid-3D performance, surpassing DELTA by 61.8% and 50.5%, respectively.
Takeaways & Limitations
The unified pipeline supports scalable 3D tracking and matches leading dynamic reconstruction accuracy while running 50 × faster than MegaSAM.
Abstract
from arXiv · showhide
We present SpatialTrackerV2, a feed-forward 3D point tracking method for monocular videos. Going beyond modular pipelines built on off-the-shelf components for 3D tracking, our approach unifies the intrinsic connections between point tracking, monocular depth, and camera pose estimation into a high-performing and feedforward 3D point tracker. It decomposes world-space 3D motion into scene geometry, camera ego-motion, and pixel-wise object motion, with a fully differentiable and end-to-end architecture, allowing scalable training across a wide range of datasets, including synthetic sequences, posed RGB-D videos, and unlabeled in-the-wild footage. By learning geometry and motion jointly from such heterogeneous data, SpatialTrackerV2 outperforms existing 3D tracking methods by 30%, and matches the accuracy of leading dynamic 3D reconstruction approaches while running 50$\times$ faster.
1. Introduction
SpatialTrackerV2 addresses scalability and compounded-error limitations in monocular 3D point tracking by jointly modeling scene geometry, camera motion, and object motion in a differentiable feed-forward pipeline. It scales training across heterogeneous datasets and achieves state-of-the-art tracking and strong reconstruction-related results.
- Existing optimization-based 3D tracking methods achieve promising results but are computationally expensive because they optimize separately for each scene.
- Ground-truth 3D trajectory dependence limits the scalability of existing feed-forward models across diverse and weakly supervised data sources.
- SpatialTrackerV2 decomposes 3D point tracking into video depth, camera ego-motion, and object motion within a fully differentiable pipeline.
- Its front end estimates video depth and initializes camera poses, then applies scale-shift estimation to make depth and motion predictions consistent.
- The Joint Motion Optimization Module iteratively estimates 2D and 3D trajectories, trajectory-wise dynamics, and visibility while optimizing camera poses through bundle adjustment.
- Training uses depth and pose consistency for RGB-D data, and pose, 2D tracking, and 3D tracking consistency for videos without depth annotations.
- 21.2 AJ and 31.0 APD3D surpass DELTA by relative improvements of 61.8% and 50.5%, while inference runs 50 × faster than MegaSAM.
2. Related work
Prior work established 2D tracking, monocular depth, and camera-pose estimation as useful components, but 3D point tracking remains comparatively new. SpatialTrackerV2 combines scalable mixed-data training, explicit camera-motion modeling, and video depth with scale alignment in a unified framework.
- Point tracking: 2D point tracking progressed from PIPs and TAP-Net to TAPIR, Co-Tracker, TAPTR, LocoTrack, BootsTAPIR, and CoTracker3, including approaches using unlabeled data.
- Point tracking: 3D point tracking began with test-time optimization in OmniMotion, followed by feed-forward methods using 2D tracking and monocular-depth priors.
- Point tracking: SpatialTrackerV2 differs by training on real and synthetic datasets while explicitly modeling camera motion for egocentric videos.
- Depth estimation: Monocular-depth research evolved from CNN-based single-view estimation toward mixed-dataset, metric-depth, and jointly estimated intrinsic-and-depth models.
- Depth estimation: Recent diffusion- and DINO-based models advanced zero-shot depth, with extensions to video estimation.
- Depth estimation: SpatialTrackerV2 extends DepthAnythingV2 to video while jointly predicting consistent depth, camera poses, and tracks with scale alignment.
- Camera pose estimation: Traditional camera-pose estimation uses image correspondences, geometric algorithms, and bundle adjustment, while neural approaches balance accuracy and inference cost through iterative refinement.
3. Method
SpatialTrackerV2 combines front-end geometry and camera estimation with iterative joint optimization of 2D/3D tracks and camera poses. Its design separates motion representations while coupling them through differentiable consistency constraints and diverse training supervision.
- Motion representation: The method decomposes tracking into 2D image-space and 3D camera-coordinate trajectories, while modeling camera ego-motion separately.This separates UV-space updates from 3D updates in the camera coordinate system.
- Front-end: The front end estimates video depth and camera poses, aligns their scale and shift, and constructs initial static 3D tracks.The aligned depth and camera trajectories provide the initialization for subsequent motion refinement.
- Joint Motion Optimization: SyncFormer iteratively updates trajectories, visibility, dynamic probabilities, and camera poses using separate 2D and 3D branches linked by cross-attention.Proxy tokens compact the correlation embeddings before information exchange between branches.
- Camera Motion Optimization: The iterative loop uses trajectory updates to form self-consistency constraints, applies weighted Procrustes alignment and bundle adjustment, and feeds revised poses into later iterations.Dynamic scores provide the alignment weights, while camera-pose updates influence subsequent iterations through global position embeddings.
- Training: The differentiable pipeline trains across 17 datasets with combinations of tracking, depth, camera-pose, dynamic-segmentation, and unlabeled supervision.Posed RGB-D data, pose-only data, and unlabeled data contribute different supervision signals to the unified training framework.
4. Experiments
Experiments evaluate SpatialTrackerV2 on 3D tracking, video depth, camera pose estimation, Internet videos, and ablations. The method achieves strong benchmark performance, benefits from camera-motion decomposition and joint training, and runs substantially faster than optimization-based reconstruction.
- The evaluation covers 3D tracking, dynamic 3D reconstruction, and ablations of unified modeling and training choices.
- 3D Point Tracking: TAPVid-3D evaluates Aria, DriveTrack, and PStudio using AJ, APD3D, and OA across 4,569 videos.AJ measures position and occlusion accuracy, APD3D measures errors across thresholds, and OA measures occlusion prediction precision.
- 3D Point Tracking: 18.7 vs. 17.3 in AJ and 27.4 vs. 25.9 in APD3D show Ours-offl− outperforming CoTracker3 with MegaSAM despite using UniDepthV2.Back-projection-based Type I methods are more sensitive to depth inconsistencies, whereas the method directly predicts 3D trajectories.
- 3D Point Tracking: 24.7 vs. 22.3 in AJ and 35.2 vs. 32.2 in APD3D show improved Aria tracking when camera pose estimation is incorporated.The gain is linked to Aria’s many background points, which primarily reflect camera motion and challenge camera-space tracking.
- Video Depth Evaluation: 0.081 vs. 0.104 AbsRel and 0.910 vs. 0.881 δ1.25 outperform VGGT, while 5–10 seconds versus 5–10 minutes for 100 frames is nearly 50 × faster than MegaSAM.The video-depth evaluation spans KITTI, Sintel, Bonn, and TUM Dynamics, covering indoor and outdoor sequences.
- Camera Pose Estimation: Camera pose estimation outperforms regression-based methods, is on par with MegaSAM, and becomes nearly twice accurate after joint motion optimization.Evaluation uses Sintel, TUM Dynamics, and Lightspeed with ATE, rotational RPE, and translational RTE after Sim(3) alignment.
- Ablation Analysis: Joint training on VKITTI, Kubric, and PointOdyssey improves DriveTrack from 7.4 to 14.7 AJ and from 13.3 to 21.9 APD3D.The ablation attributes the qualitative benefit to minimizing 3D tracking drift on new data patterns.
5. Conclusion
SpatialTrackerV2 is a feedforward, scalable, state-of-the-art approach that unifies scene geometry, camera motion, and pixel-wise 3D motion in an end-to-end pipeline. It reconstructs 3D trajectories from monocular videos and reports strong benchmark and Internet-video performance.
- SpatialTrackerV2 unifies consistent scene geometry, camera motion, and pixel-wise 3D motion in a fully differentiable end-to-end pipeline.
- The method reconstructs accurate 3D trajectories from monocular videos.
- SpatialTrackerV2 achieves strong quantitative results on public benchmarks and robust performance on casually captured Internet videos.