Source-linked AI summary
Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
Jiahao Lu, Jiayi Xu, Wenbo Hu, Ruijie Zhu, Chengfeng Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu
TL;DR
Track4World addresses the challenge of estimating every pixel’s 3D trajectory from monocular video, where existing methods are sparse or slow. It uses feedforward global scene representations and 2D-3D flow correlation, outperforming existing baselines in dense world-centric 3D tracking.
Problem
Estimating all pixels’ 3D motion from monocular video is difficult because single-view reconstruction is ill-posed and existing methods are sparse or computationally slow.
Method
Track4World combines a VGGT-style global 3D scene representation with 2D-3D correlation and joint supervision to estimate dense flows and trajectories feedforward.
Results
Track4World consistently outperforms existing baselines in robust, dense 3D tracking within the world-centric coordinate system.
Takeaways & Limitations
World-centric tracking decouples camera ego-motion from object dynamics, producing spatially stable backgrounds and physically coherent absolute motions.
Takeaways & Limitations
The model depends on labor-intensive, limited-scale captured 4D motion datasets and may struggle with extreme poses or unrepresented topological changes.
Abstract
from arXiv · showhide
Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive performance, but are limited to either tracking sparse points on the first frame or a slow optimization-based framework for dense tracking. In this paper, we propose a feedforward model, called Track4World, enabling an efficient holistic 3D tracking of every pixel in the world-centric coordinate system. Built on the global 3D scene representation encoded by a VGGT-style ViT, Track4World applies a novel 3D correlation scheme to simultaneously estimate the pixel-wise 2D and 3D dense flow between arbitrary frame pairs. The estimated scene flow, along with the reconstructed 3D geometry, enables subsequent efficient 3D tracking of every pixel of this video. Extensive experiments on multiple benchmarks demonstrate that our approach consistently outperforms existing methods in 2D/3D flow estimation and 3D tracking, highlighting its robustness and scalability for real-world 4D reconstruction tasks.
1. Introduction
Track4World addresses the challenge of dense, world-centric 3D tracking for every pixel in monocular videos with a feedforward framework. It combines pairwise scene-flow estimation, efficient 2D-to-3D correlation, joint supervision, and global temporal context to produce holistic tracking.
- Motivation: Monocular 4D reconstruction is important for robotics, animation production, and physical law inference but remains difficult because single-view 3D recovery is ill-posed.Tracking 3D points across time adds further complexity.
- Limitations of Prior Work: Existing tracking-based methods generally track only first-frame points, while dense alternatives can be computationally expensive and temporally inconsistent.TrackingWorld handles all frames but relies on multiple modalities and cannot learn joint spatiotemporal priors.
- Approach: Track4World estimates pairwise scene flows between arbitrary frames and uses them to construct 3D tracking for arbitrary pixels.Decomposing continuous tracking into pairwise flow estimation reduces computational demands.
- Approach: The model uses sparse anchor points with learned upsampling to recover dense motion, avoiding iterative updates at the original image resolution.This sparse-to-dense design manages the overhead of all-pixel scene-flow estimation.
- Core Design: Its 2D-to-3D correlation mechanism avoids heavy 3D spatial correlation, supports joint 2D-3D supervision, and leverages abundant 2D flow data.The design aims to reduce dependence on scarce 3D ground-truth annotations and improve generalization.
- Core Design: Global temporal context enables arbitrary-pair flow estimation and helps resolve ambiguities and errors that arise from isolated frame pairs.The framework processes the entire video sequence simultaneously.
- Results: Track4World fuses pairwise 3D trajectories into holistic world-centric tracking and consistently outperforms existing baselines in extensive experiments.The reported outcome covers robust dense 3D tracking in the world coordinate system.
2. Related Work
Related work spans video geometry estimation and joint geometry-motion estimation, while Track4World targets feedforward dense tracking of every pixel with explicit correlations and a 2D-lifted design.
- Video Geometry Estimation: Video geometry estimation has evolved from optimization-based refinement and differentiable SLAM toward data-driven video-level point-map prediction.
- Joint Geometry and Motion Estimation: Joint geometry and motion estimation is harder than geometry estimation alone because it requires consistent spatial-temporal reasoning.
- Track4World: Track4World proposes a feedforward framework that estimates pixel-wise 2D and 3D dense flow between arbitrary frame pairs.The resulting motion and reconstructed geometry support continuous trajectories for every pixel in a global coordinate system.
- Concurrent Work: Track4World distinguishes itself through explicit feature correlations and a novel 2D-lifted correlation mechanism for motion prediction.
3. Method
Track4World extracts global scene representations and predicts dense pairwise scene flow through coupled 2D and 3D correlation updates. Sparse anchors, image-plane correlations, learned upsampling, and direct coordinate lookup make dense estimation more efficient.
- Global Scene Representation: A finetuned ViT-based geometry encoder extracts geometric features, camera-centric point clouds, and camera poses from the video.These global scene representations support subsequent arbitrary-pair scene-flow prediction.
- Scene Flow Decoder: The decoder predicts dense 2D-3D scene flow between arbitrary frame pairs, which is later used for world-centric 3D tracking.
- Correlation Design: The correlation mechanism compares prior methods’ explicit 3D searches and cross-attention with Track4World’s image-plane-anchored 3D updates.The proposed design targets improved computational efficiency and access to abundant 2D training data.
- Sparse-to-Dense Estimation: Flow estimation operates on sparse anchor points and uses iterative updates rather than processing the full-resolution image directly.The 2D and 3D flows are initialized and refined sequentially in a coupled scheme.
- 2D Iterative Correlation: The 2D update uses geometric and semantic feature-correlation volumes sampled around target-image neighborhoods.
- Dense Flow Recovery: Low-resolution flow maps are upsampled to full resolution using learned pixel shuffle and decoded contextual weights.The method combines precise intermediate 2D flow with Z-axis displacement from 3D flow.
4. Experiments
Experiments evaluate Track4World across flow estimation, tracking, geometry, pose, and ablation benchmarks, including in-domain and out-of-domain data. Qualitative results visualize dense camera-centric and world-centric tracking, including newly emerging objects.
- Experiments cover 2D/3D flow estimation, 2D/3D tracking, point-map prediction, and camera-pose estimation across multiple benchmarks.
- Flow and scene-flow estimation: Evaluation includes Kubric-3D val, KITTI, and BlinkVision, with short-range and long-range frame gaps for Kubric-3D flow assessment.
- Comparisons: Comparisons include optical-flow, scene-flow, joint geometry-flow, tracking, and concurrent methods across the reported tasks.
- 3D tracking estimation: 3D tracking evaluation uses PointOdyssey, ADT, PStudio, and DriveTrack, reporting APD for temporal windows of 16 and 50 frames.
- Ablation study: Ablations show that removing 2D supervision causes EPE3D to reach 0.6511, while removing target lifting reduces performance.
- Qualitative visualization: Qualitative visualizations include dense all-frame tracking of existing and newly emerging objects, plus world-centric trajectories with stable backgrounds and coherent dynamic motions.
5. Conclusion
Track4World is presented as an efficient feedforward foundation model for dense, world-centric 3D tracking of every pixel. Its global geometry and correlation-based flow estimation support pixel-wise trajectory extraction across videos.
- Track4World combines a VGGT-style global 3D representation with a 3D correlation mechanism for dense 2D and 3D flow estimation.
- The framework extracts efficient pixel-wise 3D trajectories in a world-centric coordinate system across entire monocular videos.
A. Overview
The supplementary material organizes implementation, architecture, and objective-function details for the proposed flow-estimation system.
- Section B covers implementation details, Section D describes the flow-estimation architecture, and Section E presents unified joint-flow objectives.
B. Implementation Details
Track4World uses staged training with separate geometry and motion-estimation phases, trained with AdamW-based optimization on eight GPUs.
- Training stages: Stage one trains geometry estimation on diverse depth- and pose-supervised datasets, while stage two freezes geometry and trains the motion-estimation module.
- Optimization: Training uses eight 40GB GPUs, with each stage running for 100,000 steps and using a peak or initial learning rate of 1 × 10^-4.
C. Finetuning Geometry Encoder
Track4World fine-tunes a backbone-agnostic geometry encoder to produce temporally consistent global scene representations. Its video-level reconstruction loss aligns predicted point clouds with ground truth while addressing scale and focal-length ambiguity.
- Encoder design: A backbone-agnostic encoder processes video frames into a global scene representation and is fine-tuned for temporal consistency.Monocular models receive global attention layers and camera pose tokens, while 3D reconstruction models update intermediate layers.
- Reconstruction loss: The video-level reconstruction loss estimates a global scale and translation to align predicted point clouds with ground truth.This alignment addresses inherent scale and focal-length ambiguity while promoting temporal coherence.
- Geometric constraints: An affine-invariant pairwise pose loss, translation alignment, magnitude regularization, normal consistency, and local geometry losses further constrain reconstruction.Together, these terms form the overall training objective for geometric fidelity and stable optimization.
D. Model Architecture Details
The architecture iteratively refines dense motion and visibility using recurrent features, correlation cues, temporal aggregation, and a dedicated 3D flow head. Its 3D flow updates combine 2D motion, geometry, temporal dependencies, and prior flow.
- Recurrent refinement: The GRU update operator aggregates correlation volumes, motion-change gradients, semantic context, recurrent features, and visibility confidence.A fused hidden state is refined through temporal aggregator layers before producing updated recurrent features.
- 3D motion refinement: The hidden feature is temporally refined and fused with the 3D flow prior before an MLP head outputs the final update.This refinement jointly leverages 2D motion features, 3D source information, point correlations, temporal dependencies, and prior flow.
- Temporal aggregation: Temporal aggregator layers refine the hidden state along the time dimension before lightweight heads update motion and visibility fields.The recurrent feature is projected, then separate MLP heads predict 2D motion and visibility changes.
- 3D flow head: The 3D flow head aggregates prepared features to predict a 3D flow update at each iteration.Inputs include lifted source features, target point coordinates, point correlations, and a 3D flow prior.
E. Losses for Flow and Tracking Estimation
Track4World uses unified supervision for pairwise scene flow and long-term tracking, combining trajectory, visibility, confidence, and 3D smoothness losses across iterative predictions. It also refines camera poses from static 3D tracks and evaluates geometry, motion, and pose accuracy.
- Unified supervision: A unified training objective supervises pairwise scene flow and long-term tracking across variable temporal intervals and frame counts.The shared point-wise correspondence structure enables consistent supervision across both dataset types.
- 2D branch losses: The 2D objective exponentially weights iterative predictions and combines trajectory, visibility, and confidence losses.The weighting factor emphasizes later updates, while hyperparameters balance visibility and confidence terms.
- 2D branch losses: Visible points receive full trajectory-loss weight, whereas occluded or invisible points receive weight 0.2.This weighting prioritizes visible-point accuracy while retaining weaker supervision for invisible points.
- 2D branch losses: The confidence loss uses binary cross-entropy to supervise track reliability based on whether trajectory error falls within tolerance δ.The model predicts a confidence score distinct from visibility.
- 3D branch losses: The 3D branch combines trajectory and scene-flow smoothness losses across iterative updates, using sampled anchor points to reduce computational cost.The smoothness term enforces local rigidity by comparing each sampled anchor flow with its K nearest neighbors.
- 3D branch losses: The smoothness loss samples M anchor points with M ≪ N and measures squared L2 differences between each anchor flow and neighboring sampled flows.The passage reports improved scene-flow accuracy from this loss in Fig. S1.
- Camera refinement: Camera poses are refined from static 3D tracks after dynamic-mask segmentation, with clips optimized in parallel and merged into a globally consistent trajectory.The evaluation covers geometry estimation, motion estimation, and camera-pose accuracy using valid pixels or points.
G.1. Depth and Geometry Metrics
The paper evaluates depth, geometry, scene flow, 2D motion, tracking, visibility, and camera trajectories with complementary accuracy and consistency metrics. These metrics quantify both pointwise errors and threshold-based success rates.
- Depth metrics: Abs Rel measures mean absolute relative depth error, while δ < 1.25 reports the percentage of predictions within a 1.25 depth ratio threshold.The two metrics capture continuous relative error and threshold accuracy.
- 3D motion metrics: EPE3D measures average Euclidean scene-flow error in meters, while AccS and AccR report strict and relaxed 3D accuracy rates.AccS uses tight thresholds, whereas AccR evaluates robustness to larger motions with relaxed thresholds.
- 2D motion metrics: EPE2D measures average pixel-coordinate error, while AccS2D and AccR2D report success under 1-pixel and 3-pixel thresholds.The strict and relaxed metrics quantify precise and coarse tracking capability.
- Tracking and visibility metrics: APD and AJ measure position accuracy relative to ground-truth depth, while δvis_avg focuses on accuracy for visible points.AJ averages thresholded position accuracy over sequences and incorporates spatial accuracy.
- Camera-pose metrics: OA evaluates binary visibility classification, while ATE measures global camera-trajectory consistency after similarity or rigid alignment.RTE measures local drift over fixed intervals or distances, and RRE measures rotation discrepancy across frame pairs.
H. More Ablation Study
Additional experiments show that Track4World’s geometry and pose components materially affect performance, while visualizations demonstrate coherent dense motion and tracking. The method remains limited by the scale and coverage of captured 4D training data.
- Geometry encoder: Removing regularization causes a catastrophic drop to Abs Rel 0.3530, while removing local loss slightly degrades performance to Abs Rel 0.1053.These ablations indicate that regularization stabilizes convergence and local loss refines geometric predictions.
- Test-time pose refinement: Test-time pose optimization further improves Track4World’s camera accuracy, with Ours (FF.+Opt.) exceeding MegaSaM at lower computational cost.Without optimization, Ours (FF.) consistently outperforms STV2 on Sintel camera-pose accuracy.
- Scene flow visualization: Compared with ZeroMSF, Track4World produces smoother spatial transitions and higher temporal consistency in geometry and motion.The visualization evaluates pairwise scene-flow predictions through deformed point maps.
- Dense tracking visualization: Qualitative results cover scene flow, dense first-frame 2D and 3D tracking, per-pixel tracking across frames, and world-centric dense tracking.The supplementary visualizations include camera-centric and world-centric views of dense tracking behavior.
- Limitations and future work: Captured 4D motion datasets are labor-intensive and limited in scale, constraining generalization to extreme poses or complex topological changes absent from training.The authors propose synthetic data and unsupervised or semi-supervised learning as future directions.