PaperlayerView exact revision on arXiv

Source-linked AI summary

TAPIP3D: Tracking Any Point in Persistent 3D Geometry

Bowei Zhang, Lei Ke, Adam W. Harley, Katerina Fragkiadaki

arXiv:2504.14717v3cs.CVcs.LG

TL;DR

Long-term 3D point tracking must handle occlusions and camera-induced motion while modeling dynamics in 3D. TAPIP3D lifts video features into camera- or world-centric 3D feature clouds, uses neighborhood-to-neighborhood attention, and iteratively refines trajectories. It outperforms previous methods on 3D tracking metrics, with state-of-the-art performance when ground-truth or sensor depth is available and competitive performance with estimated depth.

  • Problem

    Existing point trackers mainly operate in pixel or UVD space, although apparent video motion often comes from camera movement and real-world dynamics unfold in 3D.

  • Method

    TAPIP3D lifts 2D video features into camera- or world-centric spatio-temporal 3D feature clouds and iteratively updates trajectories with Neighborhood-to-Neighborhood attention.

  • Results

    TAPIP3D outperforms previous methods on 3D tracking metrics, achieving state-of-the-art performance with ground-truth or sensor depth and competitive performance with estimated depth.

  • Takeaways & Limitations

    Compensating for camera motion in world-centric coordinates makes 3D tracks more predictable and supports strong 3D tracking performance.

  • Takeaways & Limitations

    Performance depends on geometrically consistent depth across frames, and estimated-depth failures can degrade tracking quality.

Abstract

from arXiv · show

We introduce TAPIP3D, a novel approach for long-term 3D point tracking in monocular RGB and RGB-D videos. TAPIP3D represents videos as camera-stabilized spatio-temporal feature clouds, leveraging depth and camera motion information to lift 2D video features into a 3D world space where camera movement is effectively canceled out. Within this stabilized 3D representation, TAPIP3D iteratively refines multi-frame motion estimates, enabling robust point tracking over long time horizons. To handle the irregular structure of 3D point distributions, we propose a 3D Neighborhood-to-Neighborhood (N2N) attention mechanism - a 3D-aware contextualization strategy that builds informative, spatially coherent feature neighborhoods to support precise trajectory estimation. Our 3D-centric formulation significantly improves performance over existing 3D point tracking methods and even surpasses state-of-the-art 2D pixel trackers in accuracy when reliable depth is available. The model supports inference in both camera-centric (unstabilized) and world-centric (stabilized) coordinates, with experiments showing that compensating for camera motion leads to substantial gains in tracking robustness. By replacing the conventional 2D square correlation windows used in prior 2D and 3D trackers with a spatially grounded 3D attention mechanism, TAPIP3D achieves strong and consistent results across multiple 3D point tracking benchmarks. Project Page: https://tapip3d.github.io

1 Introduction

TAPIP3D addresses long-term 3D point tracking by representing video features in 3D and compensating for camera motion. It uses spatio-temporal attention and evaluates camera- and world-centric tracking across established benchmarks.

  • TAPIP3D asks whether depth and camera-pose estimation can improve 3D point tracking by compensating for camera motion.
  • The method lifts 2D feature vectors into corresponding XYZ coordinates to construct spatio-temporal 3D feature clouds.
  • Camera-stabilized world-space trajectories are smoother and more linear than trajectories in camera-dependent UVD space.
  • TAPIP3D tracks points in either camera or world 3D feature space using 3D attentions and Neighborhood-to-Neighborhood cross-attention.
  • Across TAPVid3D, LSFOdyssey, Dynamic Replica, and DexYCB, TAPIP3D outperforms previous methods on 3D tracking metrics, especially with accurate depth.
  • The method achieves state-of-the-art performance with ground-truth or sensor depth and competitive performance with estimated depth.

2 Related Work

Prior work spans 2D and UVD-based point tracking, reconstruction-based 3D tracking, scene flow estimation, and learning-based 3D reconstruction. These approaches differ in representation, temporal scope, and computational requirements.

  • 2D and 3D Point Tracking: Recent 2D point trackers estimate multi-frame trajectories using iterative refinement, multi-point context, broader correlation contexts, and post-hoc densification.
  • 2D and 3D Point Tracking: SpatialTracker extends point tracking to 3D with Triplane features, while DELTA and SceneTracker operate in UVD coordinates.
  • Point Tracking via Reconstruction: Dynamic NeRF- and Gaussian-based reconstruction methods extract 3D tracks through per-video test-time optimization and are computationally expensive.
  • Scene Flow Estimation: Scene flow methods estimate 3D motion between point-cloud pairs, but linking flow estimates terminates at pixel occlusions by design.
  • Learning-Based 3D Foundation Models: Learning-based reconstruction models such as MoGe, DUSt3R, MonST3R, and MegaSaM predict dense 3D structure from images or videos.

3 Method

TAPIP3D represents RGB-D videos as multi-scale 3D-enriched feature clouds and iteratively refines point trajectories. It uses camera/world coordinate representations and 3D neighborhood attention to provide spatially grounded context for tracking.

  • Overview: TAPIP3D featurizes RGB-D inputs into per-timestep feature maps and initializes 3D query tracks before iterative refinement.The model outputs 3D trajectories for query points specified by XYZ coordinates and uses 3D-based feature lookups during refinement.
  • Tracking space: Camera poses transform points into either camera space or the first camera’s world coordinate system, allowing camera motion to be compensated during tracking.Using XYZ coordinates rather than UVD coordinates helps leverage 3D priors and makes tracks more predictable when camera motion is disentangled from scene motion.
  • Video representation: Each 2D feature-map cell stores a C-dimensional feature vector and an unprojected 3D (X, Y, Z) position, forming a sequence of feature clouds.The representation retains a 2D grid in memory while treating the data as featurized point clouds.
  • 3D Neighborhood-to-Neighborhood Attention: For each trajectory, TAPIP3D retrieves local 3D k-nearest neighbors as support and context tokens, using relative offsets with positional embeddings.Support neighbors come from the query’s originating timestep, while context neighbors are retrieved at each timestep from the current estimated coordinate.
  • 3D Neighborhood-to-Neighborhood Attention: Bi-directional cross-attention contextualizes support and context neighborhoods, and multi-scale processing concatenates the resulting neighborhood information into trajectory-wise vectors.The architecture then combines neighborhood features with motion, projected 2D coordinates, occlusion estimates, and timestep embeddings in trajectory tokens.

4 Experiments

TAPIP3D is evaluated across real-world and synthetic 3D point-tracking benchmarks, depth modalities, coordinate systems, and architectural ablations. Results consistently favor world-space tracking, 3D neighborhood selection, and Neighborhood-to-Neighborhood attention.

  • Evaluation setup: TAPIP3D is evaluated on real-world and synthetic benchmarks using sensor, estimated, and ground-truth depth, with both camera- and world-coordinate inference.The evaluation includes TAPVid3D, DexYCB, LSFOdyssey, and Dynamic Replica, while 2D tracks are obtained by projecting inferred 3D tracks.
  • Benchmark results: TAPIP3D-world consistently outperforms prior methods across TAPVid3D subsets, including a 2.5-point AJ3D and 3.5-point APD3D gain over DELTA on ADT.ADT contains significant camera motion, and TAPIP3D-world also outperforms the camera-coordinate variant.
  • Benchmark results: AJ3D improves from 26.4 to 30.3 on DexYCB-Pt over the previous best method using sensor depth.SpaTracker performs below depth-lifted CoTracker3 on this benchmark.
  • Benchmark results: On LSFOdyssey and Dynamic Replica, TAPIP3D-world leads competing 3D trackers with estimated or ground-truth depth and surpasses CoTracker3’s 2D accuracy with ground-truth depth.Improved depth quality produces a substantially larger APD3D gain for TAPIP3D than for DELTA or SpaTracker.
  • Ablations: 3D k-NN raises AJ3D from 27.7 to 29.8 compared with fixed 2D neighbors by filtering irrelevant 2D neighbors using 3D geometry.The ablation is conducted on DexYCB-Pt with sensor depth.
  • Ablations: Neighborhood-to-Neighborhood attention raises APD3D from 75.1 to 84.1 by incorporating support-point context and reducing matching ambiguities.World XYZ tracking also outperforms UV+D and UV+log(D) alternatives, with world XYZ performing best on AJ3D.

5 Limitations

TAPIP3D depends on geometrically consistent depth across frames, and inaccurate or unavailable high-quality depth can degrade tracking quality, especially on 2D metrics.

  • Depth-map fidelity constrains TAPIP3D because the method operates in XYZ space and requires stable, consistent geometry across frames.Depth flickering or incorrect surface connections can produce degraded tracking quality.
  • Without high-quality sensor depth, TAPIP3D may perform worse on 2D metrics than UVD-space or purely 2D tracking methods.Its 2D trajectories are projected from 3D estimates and depend on geometrically consistent data.

6 Conclusion

TAPIP3D performs multi-frame 3D point tracking with spatio-temporal 3D feature clouds and neighborhood-to-neighborhood attention in camera-centric or world-centric coordinates. It achieves state-of-the-art performance with ground-truth depth and competitive performance with estimated depth.

  • TAPIP3D represents video as spatio-temporal 3D feature clouds in camera-centric or world-centric coordinates.It lifts 2D video features into structured 3D space using depth and camera motion information.
  • Neighborhood-to-neighborhood attention contextualizes estimated tracks, while iterative updates refine multi-frame trajectories.
  • TAPIP3D achieves state-of-the-art 3D point tracking with ground-truth depth and competitive performance with estimated depth.The evaluation covers established synthetic and real-world tracking benchmarks.
  • By using depth and camera motion information, TAPIP3D addresses limitations of prior tracking methods in handling large camera motion.

Supplementary Material

The supplementary material expands the paper with training, evaluation, inference-speed, visualization, ablation, and additional benchmark details.

  • Section A provides further training details, including a memory design that reduces VRAM usage.
  • Section B explains the evaluation setup, while Section C reports inference-speed details.
  • Section D presents visualizations using stereo depth enabled by FoundationStereo.
  • Section E provides additional ablation results, and Section G reports evaluation on TAPVid-DAVIS.
  • Section H directs readers to extensive visual comparisons on the project page.

A Additional Training Details

Training uses iterative refinement with discounted losses and a gradient-detachment strategy that keeps memory usage constant across iterations and windows while reducing VRAM consumption.

  • Training: Training performs Mtrain = 4 iterative refinements per window and discounts each iteration’s loss by γMtrain−m.The total loss sums discounted losses across iterations and windows, then scales by 0.005 to mitigate unstable early gradients.
  • Training: Table 6 lists the training hyperparameters.
  • Memory-saving strategy: The memory-saving strategy detaches gradients of predicted coordinates and visibilities after each iteration.This allows each window and iteration loss to be computed and backpropagated independently.
  • Memory-saving strategy: VRAM usage remains constant regardless of the number of iterations and windows, with minimal training-speed impact and no performance sacrifice.
  • Memory-saving strategy: VRAM usage decreases from over 48GB, which caused GPU memory overflow, to approximately 20GB.

B Evaluation Details

TAPIP3D is evaluated with standard point-tracking metrics, varied depth and camera inputs, and both camera- and world-centric tracking. The study also measures efficiency and selects the N2N neighborhood size as a performance–compute trade-off.

  • Backward trajectories are obtained by reversing each video and rerunning inference because the windowed models predict only forward trajectories.
  • Evaluation augments query points with 16 × 16 grid support queries, lifts them into 3D, and uses ground-truth intrinsics when available.
  • The model normalizes spatio-temporal feature-cloud coordinates using a 3D-point scale factor and bounds evaluation depths to reduce extreme-estimate effects.
  • 11.3 FPS is achieved on 32-frame sequences with 1,024 queries, making TAPIP3D approximately 1.3× slower than SpaTracker but much faster than optimization-based methods.
  • Geometry estimation dominates monocular-video runtime: MegaSaM takes 121.1s on a DexYCB sequence, versus 6.4s for TAPIP3D tracking.
  • Stereo videos are processed by estimating disparity with FoundationStereo, then unprojecting image features into camera-centric spatio-temporal feature clouds.
  • Estimated disparity can be combined with camera poses such as MegaSaM to produce world-centric tracks in addition to camera-centric results.
  • Using K = 32 neighbors balances N2N performance and efficiency because its O(TNK^2D + TNKD^2) complexity makes attention a major bottleneck.

E.2 Robustness to Image Blur

The blur experiment evaluates TAPIP3D under increasing Gaussian image degradation while re-estimating depth, and compares its 3D tracking robustness with DELTA and camera-pose perturbation results.

  • Gaussian blur is tested at σ ∈ {0.0, 0.5, 1.0, 2.0}, with depth re-estimated for each blurred sequence using MegaSaM.
  • TAPIP3D achieves higher 3D accuracy than DELTA at every tested blur level, including heavy blur.
  • Under heavy blur, predicted 3D points remain aligned with scene geometry even as MegaSaM depth estimates become less consistent across frames.
  • Camera-pose robustness is separately tested by adding random-axis rotations with Gaussian angular noise of σ ∈ {0.5°, 1.0°, 2.0°}.
  • TAPIP3D-world degrades gracefully with increasing pose noise and still outperforms DELTA at σ = 2.0°.

F Evaluation with Sparse Depth

The sparse-depth evaluation constructs a Waymo benchmark by densifying LiDAR measurements and finds TAPIP3D-camera most accurate among the compared methods.

  • The benchmark contains 50 Waymo Open Dataset sequences with depth maps formed by nearest-neighbor interpolation of sparse LiDAR measurements.
  • TAPIP3D-camera achieves the best 3D accuracy among TAPIP3D-camera, DELTA, and CoTracker3 on the sparse-depth benchmark.
  • The results indicate that TAPIP3D remains usable and competitive when only sparse depth is available.

G TAP-Vid Evaluation

On TAP-Vid-DAVIS, TAPIP3D-world reports lower 2D tracking scores than recent state-of-the-art 2D trackers, consistent with its 3D-tracking design and monocular projection limits.

  • TAPIP3D-world achieves AJ 58.9, APD 71.2, and OA 89.6 on the TAP-Vid-DAVIS 2D benchmark.
  • These scores are lower than those of recent state-of-the-art 2D trackers.
  • The method is not designed for state-of-the-art 2D tracking, and projected monocular 3D tracks cannot achieve sub-pixel accuracy.
  • MegaSaM failures on some sequences can produce severely noisy depth estimates and unstable point clouds, further affecting performance.
Loading 2504.14717v3…