Source-linked AI summary
SpatialTracker: Tracking Any 2D Pixels in 3D Space
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, Xiaowei Zhou
TL;DR
Dense, long-range video tracking remains difficult because 2D image-space tracking struggles with complex deformation and self-occlusion while overlooking 3D motion structure. SpatialTracker lifts pixels into 3D using monocular depth, represents frames with triplanes, and tracks with 3D motion priors and learned rigidity; it achieves state-of-the-art results on multiple tracking benchmarks and challenging videos.
Problem
Current tracking methods struggle with complex deformation and frequent self-occlusions, partly because they track only in 2D image space and disregard the 3D nature of motion.
Method
SpatialTracker lifts 2D pixels into 3D using monocular depth, represents 3D frame content with triplanes, and uses 3D tracking with ARAP constraints and learned rigidity embeddings.
Results
SpatialTracker achieves state-of-the-art performance on public benchmarks including TAP-Vid, BADJA, and PointOdyssey, while handling fast complex motion and extended occlusion in challenging videos.
Takeaways & Limitations
Tracking in 3D better leverages motion regularity, including ARAP constraints, and learned rigid groups can help infer motion for ambiguous or occluded pixels.
Takeaways & Limitations
The model relies on off-the-shelf monocular depth estimators, whose accuracy may affect final tracking performance.
Abstract
from arXiv · showhide
Recovering dense and long-range pixel motion in videos is a challenging problem. Part of the difficulty arises from the 3D-to-2D projection process, leading to occlusions and discontinuities in the 2D motion domain. While 2D motion can be intricate, we posit that the underlying 3D motion can often be simple and low-dimensional. In this work, we propose to estimate point trajectories in 3D space to mitigate the issues caused by image projection. Our method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth estimators, represents the 3D content of each frame efficiently using a triplane representation, and performs iterative updates using a transformer to estimate 3D trajectories. Tracking in 3D allows us to leverage as-rigid-as-possible (ARAP) constraints while simultaneously learning a rigidity embedding that clusters pixels into different rigid parts. Extensive evaluation shows that our approach achieves state-of-the-art tracking performance both qualitatively and quantitatively, particularly in challenging scenarios such as out-of-plane rotation.
1. Introduction
SpatialTracker addresses dense, long-range pixel tracking by lifting image pixels into 3D, where motion structure and rigidity can be modeled more naturally. It combines triplane representations, transformer-based trajectory updates, and ARAP regularization with learned rigidity embeddings.
- Existing optical-flow and feature-tracking methods do not fully provide dense, long-range pixel trajectories.Optical flow is limited to adjacent frames, while feature tracking typically handles sparse pixels.
- 2D-only tracking struggles with complex deformation and self-occlusion because projection obscures 3D motion structure.The authors note that rotation and occlusion are more naturally represented in 3D than in image space.
- SpatialTracker lifts 2D pixels into 3D using monocular depth and performs feature correlation in 3D to provide richer motion context.The 3D formulation also supports motion priors that help infer ambiguous or occluded pixels from visible neighboring regions.
- The method encodes each frame with compact triplane feature maps and uses a transformer to iteratively estimate 3D trajectories.Triplanes provide dense access to features at arbitrary 3D points while avoiding the cost of dense 3D volumes.
- Learned rigidity embeddings softly group pixels with similar rigid motion and support ARAP regularization for each rigid cluster.The embeddings can also produce reasonable segmentation of different rigid parts.
- SpatialTracker achieves state-of-the-art results on TAP-Vid, BADJA, and PointOdyssey, including strong qualitative performance under fast motion and extended occlusion.These results cover both public benchmarks and challenging Internet videos.
2. Related Work
Related work spans optical flow, long-range point tracking, and scene flow, but SpatialTracker differs by performing efficient feedforward long-range tracking directly in 3D. Its design combines triplane-based processing with rigidity-aware motion regularization.
- Optical flow: Pairwise optical flow estimates dense motion but is unsuitable for long-term tracking, while multi-frame variants usually cover only short contexts.The cited multi-frame methods typically use three to five frames, which remains insufficient for long occlusions over tens or hundreds of frames.
- Tracking any point: Particle Video introduced long-range particles, but it produces only semi-dense tracks and cannot recover from occlusions.Later tracking-any-point methods revisit long-range tracking with feedforward learning frameworks.
- SpatialTracker: SpatialTracker’s pipeline encodes frames into triplanes, iteratively updates 3D trajectories, and applies ARAP regularization with learned rigidity embeddings.The ARAP constraint preserves distances between points with similar rigidity embeddings over time.
- SpatialTracker: Unlike prior methods that correlate features in 2D image space, SpatialTracker performs long-range 3D tracking in an efficient feedforward manner.The method contrasts with earlier depth-layer, quasi-3D, and optimization-based approaches.
- Scene flow: Scene-flow methods estimate dense 3D motion using multi-view, RGB-D, or stereo inputs, often incorporating local rigidity priors.Examples include rigid-motion embeddings for soft grouping and volumetric warp functions built from sparse transformations.
3. Method
SpatialTracker tracks query pixels in 3D by lifting image pixels with monocular depth and encoding each frame as a compact triplane representation. A transformer iteratively predicts trajectories, while visibility modeling, sliding windows, rigidity embeddings, and ARAP regularization support long-range tracking and rigid-part segmentation.
- Iterative Trajectory Prediction: A transformer iteratively predicts 3D trajectories and features for query points within sliding temporal windows, then projects final trajectories back to 2D.Overlapping windows propagate trajectories across long videos, while an MLP predicts point visibility at the final iteration.
- Triplane Encoding: 2D query pixels are lifted into 3D using monocular depth, image features, and positional embeddings attached to featured 3D points.The resulting point clouds encode geometry and appearance before triplane construction.
- Triplane Encoding: Triplane maps provide compact, dense features for arbitrary 3D points without the computational cost of voxelized 3D volumes.Features are sampled from three orthogonal planes through projection and bilinear interpolation, then fused by addition.
- As Rigid As Possible Constraint: The method learns rigidity embeddings that softly group trajectories with shared rigid motion and applies ARAP regularization within each rigid cluster.The ARAP loss preserves distances between highly rigid point pairs and supplies gradients for both trajectories and rigidity embeddings.
- As Rigid As Possible Constraint: Spectral clustering of trajectory affinities produces meaningful segmentations of rigid parts.The rigidity embedding is learned self-supervisedly and supports segmentation of different rigid components.
- Training: Training combines iterative 3D trajectory supervision, visibility cross-entropy, and ARAP regularization in a weighted total loss.The weighting coefficients are set to α = 10 and β = 0.1 in practice.
4. Experiments
Experiments evaluate SpatialTracker in 2D and 3D tracking settings, across multiple benchmarks, qualitative videos, baselines, and ablations of ARAP regularization and monocular depth estimators.
- 2D Tracking Evaluation: SpatialTracker is evaluated for long-range 2D tracking on TAP-Vid, BADJA, and PointOdyssey, using benchmark-specific trajectory, accuracy, and survival metrics.The evaluation compares against TAP-Net, PIPs, OmniMotion, TAPIR, and CoTracker.
- 2D Tracking Evaluation: Across TAP-Vid datasets, SpatialTracker consistently outperforms all baselines except OmniMotion, which achieves the best RGB-Stacking results with costly test-time optimization.TAP-Vid reports average position accuracy, Average Jaccard, and Occlusion Accuracy.
- 2D Tracking Evaluation: On BADJA, SpatialTracker is competitive in 3px accuracy and surpasses every baseline by a large margin in segment-based accuracy.BADJA contains seven videos of moving animals with annotated keypoints.
- 2D Tracking Evaluation: On PointOdyssey, SpatialTracker outperforms baselines across all reported 2D metrics, while more accurate ground-truth depth further improves performance.The reported metrics include Median Trajectory Error, average position accuracy, and survival rate.
- 2D Tracking Evaluation: Qualitative comparisons show better handling of complex human motion with self-occlusions, rigid groups, and small rapidly moving objects under occlusion than CoTracker.The comparisons use challenging videos from DAVIS and Internet footage.
- 3D Tracking Evaluation: For 3D tracking, SpatialTracker outperforms Chained RAFT-3D and Lifted CoTracker across ATE3D, δ0.1, and δ0.2 by a large margin.The evaluation uses 231 PointOdyssey testing sequences of 24 frames.
- Ablation Study: Removing the ARAP loss substantially reduces performance, while ZoeDepth gives the best tested depth-estimator results compared with MiDaS and DPT.The depth-estimator analysis is conducted on TAP-Vid-DAVIS, and ZoeDepth is the default model.
5. Conclusion and Discussion
SpatialTracker uses a 3D representation with a learnable ARAP constraint to identify rigid groups and estimates 3D trajectories through triplane features. It achieves superior performance on challenging real-world tracking scenarios, but depends on off-the-shelf monocular depth estimators whose accuracy may affect tracking.
- SpatialTracker estimates 3D trajectories with triplane representations and a learnable ARAP constraint that identifies rigid groups and enforces within-group rigidity.The framework is designed to leverage motion regularity in 3D.
- Experiments show superior performance over existing baselines and applicability to challenging real-world scenarios.
- The model relies on off-the-shelf monocular depth estimators, whose accuracy may affect final tracking performance.The authors anticipate that advances in monocular reconstruction will improve motion estimation.