PaperlayerView exact revision on arXiv

Source-linked AI summary

TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking

Jisu Nam, Jahyeok Koo, Soowon Son, Jaewoo Jung, Honggyu An, Junhwa Hur, Seungryong Kim

arXiv:2605.12587v1cs.CV

TL;DR

Dense 3D tracking needs reference-anchored motion reasoning, which conflicts with the frame-anchored outputs of video diffusion transformers. TrackCraft3R resolves this mismatch with dual latents and temporal RoPE, achieving state-of-the-art benchmark performance while running 1.3× faster and using 4.6× less peak memory than DELTAv2.

  • Problem

    Video diffusion transformers provide spatio-temporal priors but produce frame-anchored outputs, whereas dense 3D tracking requires consistently following reference-frame points across time.

  • Method

    TrackCraft3R uses geometry and first-frame-anchored track latents with temporal RoPE to predict reference-anchored pointmaps and visibility in one forward pass.

  • Results

    1.3× faster and 4.6× less peak memory than DELTAv2, while achieving state-of-the-art performance across standard sparse and dense 3D tracking benchmarks.

  • Takeaways & Limitations

    The results demonstrate that video diffusion transformers can support feed-forward, reference-anchored dense 3D tracking with robustness to large motions and long videos.

  • Takeaways & Limitations

    TrackCraft3R depends on external per-frame depth and camera pose, so its accuracy is bounded by the quality of the input geometry.

Abstract

from arXiv · show

Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.

1 Introduction

Dense 3D tracking needs reference-anchored, world-coordinate trajectories, but existing approaches lack either real-world temporal priors or compatibility with video DiTs’ frame-anchored outputs. TrackCraft3R resolves this mismatch with a feed-forward video-DiT tracker using dual latents and temporal RoPE alignment, achieving state-of-the-art tracking with improved efficiency and robustness.

  • Motivation: Dense 3D trajectories support robotic manipulation, dynamic scene reconstruction, and controllable video generation, while world-coordinate reasoning separates object motion from camera ego-motion.Accurate tracking requires canceling camera motion in a 3D world coordinate frame.
  • Challenge: Existing trackers either iteratively learn from synthetic 4D data or fine-tune reconstruction models whose static-image training lacks rich real-world temporal priors.Video diffusion transformers offer spatio-temporal priors learned from internet-scale videos, but their outputs are frame-anchored rather than reference-anchored.
  • Method: TrackCraft3R is the first feed-forward method to repurpose a video diffusion transformer for dense 3D tracking from monocular video.Given frame-anchored reconstruction pointmaps, it predicts a first-frame-anchored tracking pointmap and visibility for every first-frame pixel in one forward pass.
  • Method: TrackCraft3R converts frame-anchored generation into reference-anchored tracking through dual geometry-and-track latents and temporal RoPE alignment.Geometry latents encode per-frame RGB and pointmaps, while first-frame track latents serve as dense query points across time.
  • Results: TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks while running 1.3× faster and using 4.6× less peak memory than DELTAv2.The method also demonstrates robustness to large motions, long videos, large temporal strides, and validated design choices through extensive ablations.

2 Related Work

Prior 3D trackers either extend iterative 2D tracking with monocular depth and camera motion or fine-tune reconstruction models on synthetic 4D data, leaving gaps in real-world temporal priors. TrackCraft3R addresses the mismatch between frame-anchored video diffusion outputs and reference-anchored dense 3D tracking by directly predicting reference-anchored trajectories in one pass.

  • 3D Point Tracking: Existing 3D point trackers extend iterative 2D methods using monocular depth, camera coordinates, or camera poses to compensate for camera motion in world coordinates.These approaches recover long-range motion trajectories through iterative tracking procedures.
  • 3D Point Tracking: Recent feed-forward trackers fine-tune pretrained 3D reconstruction models on synthetic 4D data, gaining spatial priors but lacking strong temporal priors from real-world video dynamics.MotionCrafter instead repurposes a video diffusion U-Net to incorporate temporal priors, but predicts frame-anchored scene flow between adjacent frames.
  • Video Diffusion Models for Frame-Anchored Perception: Video diffusion models have been repurposed for perception tasks including depth, surface normals, dense correspondence, optical flow, and temporally consistent video depth estimation.These methods provide robust spatio-temporal priors but produce frame-anchored outputs tied to individual frames’ content and timestamps.
  • Video Diffusion Models for Frame-Anchored Perception: TrackCraft3R is presented as the first video-DiT method for reference-anchored dense 3D tracking, directly producing a tracking pointmap in one forward pass.This avoids the temporal chaining required by MotionCrafter’s adjacent-frame, frame-anchored scene-flow predictions, which can accumulate errors under occlusion.

3 Preliminaries

This section introduces the latent video representation, video DiT architecture, feed-forward regression use, and 3D RoPE for relative spatio-temporal attention. VAEs preserve the first frame while temporally downsampling subsequent frames and can encode geometric modalities.

  • Variational Autoencoder (VAE): A VAE encodes a video into a spatially and temporally downsampled latent representation while preserving the first frame, and a decoder reconstructs the video.Temporal downsampling is applied only to the F frames after the first.
  • Variational Autoencoder (VAE): Pre-trained RGB-video VAEs can encode and decode pointmaps, depth maps, and camera rays, enabling diffusion models to predict geometric modalities in latent space.
  • Video Diffusion Transformers (DiTs): Video DiTs patchify and project the latent, then use rectified flow matching and full 3D attention to predict velocity along a noise-to-data interpolation.
  • Video Diffusion Transformers (DiTs): The work repurposes the video DiT as a feed-forward regressor rather than a multi-step denoiser, enabling efficient inference without iterative sampling.
  • 3D Rotary Positional Embedding (3D RoPE): 3D RoPE applies axis-specific rotations to temporal and spatial channel groups, making attention depend on relative 3D positions and favoring tokens with similar temporal indices.

4 Video Diffusion Transformer for Dense 3D Tracking

TrackCraft3R repurposes a video diffusion transformer into a single-pass dense 3D tracker that predicts reference-anchored pointmaps and visibility from frame-anchored geometry. Its dual-latent representation and temporal RoPE alignment convert frame-anchored generation into reference-anchored tracking.

  • Pointmap representations: TrackCraft3R predicts the 3D positions and visibility of every point from reference frame I0 across time in a shared world coordinate frame.The tracking pointmap follows the same physical points originally observed in I0, while each visibility value indicates whether that point is visible at the target timestamp.
  • Dual-Latent Representation: The dual-latent representation combines per-timestamp geometry latents with first-frame-anchored track latents that act as dense tracking queries.Geometry latents concatenate RGB and pointmap features, while the track latent replicates the first-frame geometry latent across timestamps and is processed jointly with geometry tokens.
  • Temporal RoPE Alignment: Temporal RoPE alignment assigns each track latent the same timestamp index as its target geometry latent, encouraging attention to retrieve the corresponding 3D position.Attention from r5 is predominantly localized on g5, and within g5 it aligns with the same physical point under motion.
  • Trajectory and Visibility Prediction: The model decodes track-latent outputs into tracking pointmaps and visibility maps, using separate latent halves for pointmap and visibility prediction.A fixed diffusion timestep of zero and a null text prompt convert the video DiT into a one-step regressor.

5 Experiment

TrackCraft3R achieves state-of-the-art 3D tracking across sparse and dense benchmarks, outperforming iterative and feed-forward baselines. It also remains robust under large motion and long videos while improving inference efficiency and benefiting from stronger geometry and model adaptation.

  • Qualitative Comparison: TrackCraft3R produces accurate dense trajectories under large object dynamics and occlusion, where DELTAv2 often fails on real-world ITTO videos.The comparison is shown qualitatively in Figure 4.
  • Robustness to Large Motion and Long Videos: TrackCraft3R’s performance drops much more slowly than DELTAv2 as temporal stride or video length increases.Figure 5 evaluates APD3D and AJ under strides from 1 to 12 and sequence lengths from 12 to 120 frames.
  • Ablation Studies: Pre-trained spatio-temporal priors, reference anchoring, temporal RoPE alignment, residual displacement, higher LoRA rank, VAE fine-tuning, and stronger input geometry each improve tracking performance.Random initialization degrades all metrics; ground-truth geometry improves all metrics without retraining, while higher LoRA rank and Stage 2 VAE unfreezing provide further gains.
  • Inference Efficiency: 1.3× faster and uses 4.6× less peak memory than DELTAv2 for 12-frame clips on a single NVIDIA A6000 GPU.TrackCraft3R uses feed-forward prediction, whereas DELTA and DELTAv2 perform six-step iterative refinement and construct 4D correlation features.

6 Conclusion

TrackCraft3R repurposes a video diffusion transformer into a single-pass dense 3D tracker by converting frame-anchored generation into reference-anchored tracking. This is enabled by dual-latent representations and temporal RoPE alignment with LoRA fine-tuning.

  • 6 Conclusion: TrackCraft3R is the first method to repurpose a video diffusion transformer as a single-pass dense 3D tracker.
  • 6 Conclusion: Its dual-latent representation couples per-frame geometry latents with first-frame-anchored track latents, while temporal RoPE alignment specifies each track latent’s target timestamp.Together with LoRA fine-tuning, these designs convert video DiTs’ per-frame generative paradigm into reference-anchored dense tracking.

A Training Datasets · B Comparison with Lifted Dense 2D Tracker · C Comparison with Sparse 3D Trackers

TrackCraft3R is trained on four synthetic video datasets and consistently outperforms lifted dense 2D and sparse 3D trackers on 3D tracking benchmarks. Comparisons use AJ, APD3D, and OA after Sim(3) alignment, with ViPE depth and camera poses used for fair 3D evaluation.

  • A Training Datasets: The training mixture comprises Kubric, DynamicReplica, PointOdyssey, and TartanAir synthetic datasets.Kubric, DynamicReplica, and PointOdyssey provide RGB, depth, camera parameters, and 3D trajectories, while TartanAir contains static scenes.
  • A Training Datasets: Kubric sequences are rendered at 480×832 resolution for 81 frames, with dense trajectories extracted from the first frame.The rendering protocol follows prior work and uses 6K sequences.
  • A Training Datasets: DynamicReplica and PointOdyssey provide sparse 3D trajectories derived from mesh vertices for training.These datasets also provide RGB, depth, and camera parameters.
  • B Comparison with Lifted Dense 2D Tracker: TrackCraft3R consistently outperforms lifted AllTracker, achieving higher overall AJ, APD3D, and OA across all sparse and dense benchmarks.AllTracker’s estimated dense 2D tracks are unprojected into 3D using ViPE depth and camera poses.
  • C Comparison with Sparse 3D Trackers: TrackCraft3R outperforms both SpatialTrackerV2 and TAPIP3D, achieving the best average AJ, APD3D, and OA on sparse 3D tracking benchmarks.All methods use ViPE depth and camera poses for fair comparison.
  • C Comparison with Sparse 3D Trackers: The sparse-tracker evaluation compares TrackCraft3R with SpatialTrackerV2, TAPIP3D, and V-DPM using AJ, APD3D, and OA after Sim(3) alignment.The reported comparison tables identify the best and second-best results by highlighting them separately.

D Comparison with V-DPM

Against V-DPM, TrackCraft3R achieves competitive sparse-tracking accuracy with higher AJ, while requiring far less 3D/4D supervision and substantially less inference time and memory. Its efficiency advantage grows with clip length, supported by single-pass latent prediction and interleaved inference.

  • Benchmark comparison: TrackCraft3R + DA3 and TrackCraft3R + V-DPM outperform V-DPM in AJ and OA, while V-DPM achieves slightly higher APD3D on sparse tracking benchmarks.The comparison evaluates the first 24 frames and includes TrackCraft3R using V-DPM’s frame-anchored reconstruction pointmaps.
  • Dataset scale: TrackCraft3R is fine-tuned on 4 synthetic 3D/4D datasets, compared with V-DPM’s 23 3D/4D-annotated datasets, while retaining competitive APD3D and higher AJ.TrackCraft3R starts from Wan2.1-T2V, pretrained on billions of generic web images and videos without 3D annotations.
  • Inference efficiency: At 23 frames, TrackCraft3R runs 6.6× faster and uses 2.3× less memory than V-DPM; at 12 frames, the gains are 3.2× speed and 1.7× memory.The measurements use 448 × 448 resolution, 12- and 23-frame clips, and a single A6000 GPU.
  • Inference efficiency: TrackCraft3R predicts all trajectories in one compressed-latent feed-forward pass, yielding O(L) runtime and O(1) peak memory with interleaved inference for longer clips.V-DPM instead invokes a time-conditioned decoder once per timestamp, incurring O(L2) time and O(L) memory.
  • Summary: The comparison highlights compatibility with any 3D geometry estimator and substantial efficiency gains, while trading a small amount of point accuracy for data efficiency.Using V-DPM’s reconstruction maps as input accesses its 3D point predictions but not its full learned tracking capability.

E Additional Attention Visualization

The attention visualizations show that temporal RoPE alignment reliably directs each track latent to its corresponding timestamp, while full 3D attention establishes spatial correspondences between track and geometry latents under motion.

  • Temporal alignment between track and geometry latents: 29.0% in Fig. 6 and 30.1% in Fig. 7 of attention mass goes to the temporally aligned geometry latent, verifying temporal RoPE as a timestamp-identification signal.Each track latent assigns its highest attention to the corresponding geometry latent when averaged across transformer layers.
  • Correspondence within aligned latents: Full 3D attention establishes reliable spatial correspondences between track and geometry latents under motion, including the moving baseball example.Some layers focus on RoPE-initialized positions, while others identify correspondences between the same physical points.

F Additional Qualitative Results

This section presents additional qualitative results comparing TrackCraft3R with DELTAv2 on ITTO and DAVIS videos, with results shown in Figs. 10 and 11.

  • F Additional Qualitative Results: Additional qualitative results are presented for TrackCraft3R on ITTO and DAVIS videos.
  • F Additional Qualitative Results: The qualitative evaluation compares TrackCraft3R with DELTAv2.
  • F Additional Qualitative Results: The comparisons are shown in Figs. 10 and 11.

G Limitations and Future Work

TrackCraft3R’s accuracy is bounded by the quality of externally supplied per-frame depth and camera pose, while future work could jointly generate videos and 3D tracks. The authors also encourage responsible use for real-world videos involving people.

  • Limitations: TrackCraft3R’s accuracy is bounded by the quality of external per-frame depth and camera-pose estimates, although the design can benefit from advances in 3D foundation models.The method relies on geometry from external 3D foundation models, consistent with common world-coordinate tracking conventions.
  • Future Work: Future work could jointly generate videos and 3D tracks within a single video DiT, supporting unified generation and dense 4D perception for robotic manipulation.Generated videos and tracks could serve as intermediate representations for action prediction.
  • Limitations: The authors encourage responsible use when applying TrackCraft3R to real-world videos involving people, including individual tracking.
Loading 2605.12587v1…