PaperlayerView exact revision on arXiv

Source-linked AI summary

Track Everything Everywhere Fast and Robustly

Yunzhou Song, Jiahui Lei, Ziyun Wang, Lingjie Liu, Kostas Daniilidis

arXiv:2403.17931v1cs.CV

TL;DR

Long-term pixel tracking underpins spatial intelligence, while the global MLP-like deformation network in OmniMotion is a computational bottleneck. The paper introduces CaDeX++ and visual-foundation-model regularization; it reports 90% less computational time and improved tracking accuracy and robustness over OmniMotion.

  • Problem

    Long-term pixel tracking supports spatial intelligence, but querying OmniMotion’s global MLP-like deformation network is a computational bottleneck.

    Source: §1 · pp. 1–2

  • Method

    CaDeX++ uses a local feature grid and nonlinear invertible blocks, while depth-based geometry and DINOv2 long-term semantics regularize tracking optimization.

    Source: §1 · p. 3

  • Results

    90% less computational time, with state-of-the-art test-time tracking performance and improved accuracy and robustness over OmniMotion.

    Source: §1 · pp. 2–3, §5 · p. 14

  • Takeaways & Limitations

    Long-term pixel tracking supports downstream tasks including 3D reconstruction and video recognition.

    Source: §1 · p. 1

    Tracking performance is dominated by the precision and quality of input depth and pixel correspondence.

    Source: D Limitation and Future Research · p. 19

Abstract

from arXiv · show

We propose a novel test-time optimization approach for efficiently and robustly tracking any pixel at any time in a video. The latest state-of-the-art optimization-based tracking technique, OmniMotion, requires a prohibitively long optimization time, rendering it impractical for downstream applications. OmniMotion is sensitive to the choice of random seeds, leading to unstable convergence. To improve efficiency and robustness, we introduce a novel invertible deformation network, CaDeX++, which factorizes the function representation into a local spatial-temporal feature grid and enhances the expressivity of the coupling blocks with non-linear functions. While CaDeX++ incorporates a stronger geometric bias within its architectural design, it also takes advantage of the inductive bias provided by the vision foundation models. Our system utilizes monocular depth estimation to represent scene geometry and enhances the objective by incorporating DINOv2 long-term semantics to regulate the optimization process. Our experiments demonstrate a substantial improvement in training speed (more than \textbf{10 times} faster), robustness, and accuracy in tracking over the SoTA optimization-based method OmniMotion.

1 Introduction

Long-term pixel tracking associates observations across time and supports tasks from 3D reconstruction to video recognition, but existing methods face trade-offs in correspondence density and long-range consistency. The paper introduces CaDeX++ and vision-foundation-model priors to improve test-time tracking efficiency, robustness, and accuracy over OmniMotion [34].

1 Introduction

  • Long-term pixel tracking associates visual information across continuous observations and underpins tasks including 3D reconstruction and video recognition..Source: §1 · p. 1
  • Feature-based methods support long-horizon matching but are often sparse, while optical-flow methods are dense but struggle with long-range motion and occlusion.The paper also identifies slow convergence and unpredictable reconstruction quality as problems for OmniMotion [34] under insufficient constraints and random initialization.Source: §1 · p. 2
  • CaDeX++ combines a local feature grid and nonlinear functions with depth-based geometry and DINOv2 [25] long-term semantics to regularize tracking optimization.The depth map is initialized from monocular metric depth estimation, while DINOv2 correspondence adds long-term association information absent from OmniMotion's short-term optical-flow fitting.Source: §1 · pp. 2–3
  • Training time is reduced by more than 90%, with reported stabilization and tracking-performance improvements over OmniMotion [34].The paper also describes its tracking as faster and more robust than OmniMotion [34].Source: §1 · p. 3

2 Related Work

Related work spans sparse feature-based correspondence, dense optical flow, learned trackers, and test-time optimization. These approaches differ in how they represent tracks and handle long-range consistency.

2 Related Work

  • Feature descriptors support long-horizon matching but are sparse, whereas optical flow provides dense correspondence but can lack long-term consistency.Classical feature examples include ORB, SIFT, and SURF; RAFT uses recurrent networks to simulate optimization steps in optical flow.Source: §2 · p. 3
  • Particle Video connects short-term flow to preserve dense long-range motion, and later trackers build on particle-video updates or learned trajectory refinement.PIPs uses an MLP-Mixer; TAPIR [9] refines trajectories from TAP-Net initialization, while MFT selects high-confidence flow chains using uncertainty and occlusion estimates.Source: §2 · pp. 3–4
  • The overview describes lifting a query pixel with an optimizable depth map, deforming it through canonical space, and fitting with RAFT [33] flow plus DINOv2 [25] correspondence.RAFT supplies short-term dense flow, while DINOv2 supplies long-term sparse correspondence.Source: Figure 2 · p. 4
  • OmniMotion [34] lifts pixels into 3D and jointly optimizes canonical-space scene representation and invertible mappings to local scenes.The method represents tracking as per-pixel trajectories in a canonical 3D volume with bijective mappings to quasi-3D local scenes.Source: §2 · p. 4

3 Method

The method represents long-term pixel tracks with an invertible deformation field and optimizes it using depth-based geometry and short- and long-term correspondences. CaDeX++ combines local spatial-temporal features with nonlinear coupling maps while preserving invertibility.

3.1 Preliminaries

  • The tracking function maps a query pixel and destination time to a position and visibility, fitting the deformation against optical flow and rendering errors.The method lifts points along camera rays, deforms them into a shared canonical space, and maps them to the target frame.Source: §3.1 · pp. 4–5

3.2 CaDeX++: Non-linear and Local Invertible NVPs

  • CaDeX++ factorizes global latent codes into local spatial-temporal feature grids and uses monotonic piecewise nonlinear maps to increase expressivity while retaining invertibility.The factorization avoids dependence on the coordinate being changed, preserving the coupling block’s invertibility.Source: §3.2 · pp. 6–7

3.3 Optimization with Depth Prior

  • The method replaces radiance-field reconstruction with optimizable depth maps initialized by monocular metric depth estimates, regularizing them toward their initialization.This design addresses ambiguous geometry under small camera baselines and replaces an inefficient, under-constrained radiance field.Source: §3.3 · p. 7

3.4 Incorporation of Long-term Semantics

  • Filtered DINOv2 matches add sparse long-term semantic correspondences to the short-term optical-flow optimization targets.Candidates are mutually nearest patch matches, then filtered using within-frame feature self-similarity.Source: §3.4 · pp. 7–8, Figure 4 · p. 7

3.5 Training and Inference

  • Optimization samples flow and semantic matches, fits pixel position and depth consistency, regularizes optimized depths near initialization, and infers visibility by comparing warped-point and target-depth values.The loss also includes depth regularization, and occlusion is detected when the warped point lies behind the target depth beyond a threshold.Source: §3.5 · pp. 8–9

4 Experiments

Experiments compare the method with feed-forward and optimization-based trackers on DAVIS and RGB-Stacking, assessing accuracy, runtime, ablations, qualitative behavior, and robustness.

4.1 Experiment Setup

  • Evaluation follows OmniMotion [34] protocols on 480p inputs with metrics at 256x256, using real-scene DAVIS videos and synthetic RGB-Stacking videos with object motion and a static camera.Source: §4.1 · p. 9

Qualitative comparison

Quantitative comparisons

  • On DAVIS, the method has the best temporal coherence among all methods and higher position precision than other optimization methods, comparable to feed-forward methods; on RGB-Stacking, it beats feed-forward methods.Source: Quantitative comparisons · pp. 10–11, Table 1 · p. 11

Baselines

  • Comparisons span feed-forward trackers and per-scene optimization methods; OmniMotion [34] fits invertible mappings to canonical space using an optimizable NeRF volume.Source: Baselines · pp. 9–10, Table 1 · p. 11

Quantitative comparisons

4.3 Ablation Study

  • Ablations attribute precision and convergence gains to optimizable depth maps, trajectory precision gains to long-term supervision, and faster convergence to CaDeX++; without long-term supervision, frequent large occlusions defeat tracking.Source: §4.3 · pp. 11–12, Figure 7 · p. 12, Figure 6 · p. 12

4.4 Further Comparison with CoTracker and OmniMotion

  • CoTracker [14] fails on unseen settings, including low-frame-rate relative motion and textured backgrounds, while this method tracks reasonably in the two reported cases.Source: §4.4 · pp. 12–13
  • OmniMotion [34] produces high-variance fitting errors across random seeds, whereas this method remains stable; the authors attribute robustness to depth-prior regularization.Source: §4.4 · p. 14, Table 4 · p. 14, Figure 9 · p. 14

5 Conclusion

The approach computes long-term pixel trajectories using a local-grid invertible block and foundational models to bootstrap semantic consistency with short-term flow. It reports state-of-the-art tracking test time while reducing computational time by 90%.

5 Conclusion

  • The method achieves state-of-the-art performance in optimizing tracking test time while reducing computational time by 90%.Compared with OmniMotion, it tracks pixels faster and more robustly.Source: §5 · p. 14
  • A novel invertible block with a local grid boosts the expressivity of the mapping functions.The approach also uses foundational models to bootstrap long-term semantic consistency with short-term flow consistency.Source: §5 · p. 14
  • The approach computes long-term trajectories of pixels from video, targeting computational efficiency and robustness.The conclusion identifies these as key shortcomings of previous work.Source: §5 · p. 14

A CaDeX++

CaDeX++ uses temporal and spatial feature grids with a tiny MLP to parameterize control-point biases and outlier slopes. Its control points are aggregated separately for negative and positive α values, with linear extrapolation outside their range.

A CaDeX++

  • The temporal feature grid uses three resolutions, while the spatial feature grid uses two resolutions, each with specified feature dimensions.Temporal resolutions are T/20, T/4, and 13T/20 with dimension 16; spatial resolutions are 12 and 96 with dimension 32 each.Source: A CaDeX++ · p. 17
  • A tiny MLP predicts positive incremental control-point biases and positive outlier slopes.The implementation uses two hidden layers for the tiny MLP.Source: A CaDeX++ · p. 17
  • Negative-α control-point coordinates are formed by summing their incremental biases, while positive-α coordinates are aggregated separately.The two sets of control points are defined through separate cumulative sums of their α and β increments.Source: A CaDeX++ · p. 17
  • Inputs outside the leftmost or rightmost control point are mapped by linear extrapolation using the corresponding outlier slope.The outlier slope is denoted k_m.Source: A CaDeX++ · p. 18

B Preparing Long-term Correspondence

Long-term correspondences combine flow sampled from nearby query frames with DINOv2-based coarse matches outside that neighborhood. Three filters retain representative matches while removing noisy ones.

B Preparing Long-term Correspondence

  • Training samples flow within a 12-frame neighborhood and searches long-term correspondence beyond 10 frames, using low-resolution DINOv2 feature maps for coarse matches.The method applies three strong filters to remove noisy matches and retain representative ones.Source: B Preparing Long-term Correspondence · p. 18
  • Mutual Maximum filtering retains a pair only when each point is the other frame’s best match, with feature cosine similarity above 0.75.The similarity is computed between the paired points’ features.Source: B Preparing Long-term Correspondence · p. 18
  • The background filter keeps points with fewer than 100 similar points in their feature map, using a similarity threshold of 0.55.The filter counts points exceeding the similarity threshold.Source: B Preparing Long-term Correspondence · p. 18
  • The local noise filter selects points whose summed feature similarity across an 11 × 11 neighborhood exceeds 30.The neighborhood is centered on the candidate point.Source: B Preparing Long-term Correspondence · p. 18

C Optimization Based on CoTraker

The DAVIS optimization experiment uses CoTracker output as part of its training supervision, and its results are presented in Table 5.

C Optimization Based on CoTraker

  • CoTracker output is used as part of the training supervision for each scene.The reported optimization experiment is on the DAVIS dataset.Source: C Optimization Based on CoTraker · p. 19
  • The optimization results on DAVIS are presented in Table 5.The table caption identifies its contents as optimization results on DAVIS with CoTracker output.Source: C Optimization Based on CoTraker · p. 19, Table 5 · p. 19
  • Table 5 reports optimization on DAVIS with CoTracker output.The supplied passage identifies the table’s scope but does not state its cell values.Source: Table 5 · p. 19

D Limitation and Future Research

Tracking performance is dominated by the quality of input depth and pixel correspondence. The current architecture primarily addresses 2D pixel tracking, leaving its potential in other tasks for future investigation.

D Limitation and Future Research

Loading 2403.17931v1…