Source-linked AI summary
Tracking Everything Everywhere All at Once
Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, Noah Snavely
TL;DR
Dense, long-range video tracking must handle occlusions and maintain globally consistent trajectories beyond the limited contexts of prior methods. OmniMotion addresses this with a test-time-optimized quasi-3D canonical volume and local-canonical bijections, achieving significant qualitative and quantitative improvements over prior state-of-the-art methods. Its current limitations include difficult rapid or highly non-rigid motion, initialization sensitivity, and computational expense.
Problem
Existing motion methods struggle to estimate dense, long-range trajectories while tracking through occlusions and maintaining global consistency.
Method
OmniMotion optimizes a quasi-3D canonical volume with per-frame local-canonical bijections to represent complete video motion without explicitly disentangling camera and scene motion.
Results
OmniMotion achieves significant qualitative and quantitative improvements over prior state-of-the-art methods.
Takeaways & Limitations
The representation produces globally consistent, smooth long-range motion through occlusions for videos with varying camera setups and scene dynamics.
Takeaways & Limitations
The method struggles with rapid or highly non-rigid motion and thin structures, can be initialization-sensitive, and is computationally expensive.
Abstract
from arXiv · showhide
We present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively. See our project page for more results: http://omnimotion.github.io/
1. Introduction
Dense, long-range pixel tracking remains difficult because existing methods use incomplete motion representations or limited temporal and spatial context. OmniMotion jointly estimates globally consistent, full-length trajectories for every pixel, including through occlusions, and achieves state-of-the-art TAP-Vid performance.
- Motivation: Sparse tracking covers only selected interest points, while pairwise optical flow does not capture long-range motion trajectories.Together, these limitations leave neither approach fully modeling video motion.
- Motivation: Limited temporal and spatial context can accumulate errors and produce spatiotemporal inconsistencies over long trajectories.Prior methods may also lose tracking during occlusions.
- Motivation: The central challenges are accurate tracking across long sequences, tracking through occlusions, and maintaining spatial and temporal coherence.
- OmniMotion: OmniMotion jointly estimates full-length motion trajectories for every pixel using all video information and a quasi-3D canonical volume with local-canonical bijections.The representation models combined camera and scene motion and guarantees cycle consistency.
- Results: On the TAP video tracking benchmark, OmniMotion achieves state-of-the-art performance and outperforms prior methods by a large margin.
2. Related Work
Prior work spans sparse tracking, optical flow, feature matching, pixel-level tracking, global video optimization, and neural video representations. These approaches provide useful capabilities but retain limitations in dense coverage, long-range consistency, occlusion handling, or general dynamic scenes.
- Sparse feature tracking: Sparse feature tracking establishes long-range correspondence but is limited to distinctive interest points and often restricted to rigid scenes.
- Optical flow: Optical flow provides precise consecutive-frame motion, but chaining pairwise estimates into long trajectories causes drift and fails for long-range estimation.Multi-frame flow methods address some two-frame limitations but still struggle with long-range motion.
- Feature matching: Pairwise feature matching can estimate dense correspondences between distant frames, but typically omits temporal context.
- Pixel-level long-range tracking: PIPs tracks through occlusions within an 8-frame window, but longer videos require chaining that is prone to drift and loses points occluded beyond that window.
- Video-based motion optimization: Particle Video produces semi-dense long-range trajectories but treats an occluded entity as a different particle when it reappears.Combinatorial assignment methods can track through occlusion and generate longer trajectories.
- Neural video representations: Prior neural video representations mainly target novel view synthesis and video decomposition, while dynamic novel-view systems require known camera poses and can produce erroneous motion.
3. Overview
OmniMotion is a test-time optimization method that uses frames and noisy pairwise motion estimates to construct a globally consistent representation of an entire video. After optimization, it produces smooth trajectories for arbitrary pixels across the full sequence and tracks through occlusions.
- Overview: OmniMotion takes video frames and pairwise noisy motion estimates, such as optical flow fields, as input to optimize a complete motion representation.
- Overview: The optimized representation can be queried at any pixel in any frame to produce a smooth, accurate trajectory across the full video.
- Overview: The method identifies occlusions and tracks points through them.
4. OmniMotion representation
OmniMotion represents video motion with a quasi-3D canonical volume and bijections between each frame’s local space and canonical space. This global representation supports cycle-consistent motion computation, occlusion reasoning, and flexible camera–scene dynamics without explicit dynamic 3D reconstruction.
- Motivation: Fixed depth-separated layers cannot represent general 3D motion, while explicit dynamic 3D reconstruction is extremely ill-posed.
- Representation: OmniMotion uses a canonical 3D volume mapped to each frame’s local volume through neural local-canonical bijections.The mappings capture camera and scene motion without disentangling them.
- Representation: The quasi-3D relaxation avoids ambiguities in dynamic reconstruction while retaining properties needed for consistent long-term tracking through occlusions.
- Canonical 3D volume: The canonical volume stores density and color at 3D coordinates, enabling surface tracking, occlusion reasoning, and photometric-loss computation.
- 3D bijections: A local 3D point maps to canonical coordinate u through T_i and then to another frame through the inverse mapping T_j^-1, yielding x_j.
- 3D bijections: Bijective mappings make correspondences cycle consistent because they arise from the same canonical point.
- Motion computation: For a query pixel, OmniMotion lifts samples along a ray to 3D, maps them to the target frame, alpha-composites them, and projects the result back to 2D.A fixed orthographic camera is used because camera motion is subsumed by the bijections; multiple sampled surfaces support occlusion handling.
5. Optimization
OmniMotion optimizes a complete, globally consistent motion estimate from video and noisy pairwise correspondences. Its losses combine flow supervision, photometric consistency, temporal smoothness, and hard-example sampling to address sparse or imbalanced motion evidence.
- OmniMotion takes a video sequence and noisy correspondence predictions to generate a complete, globally consistent motion estimate.The method can use pairwise correspondences from RAFT or TAP-Net as guidance.
- The primary flow loss minimizes mean absolute error between predicted and supervising input flows.Predicted flow is obtained from optimized point positions, while supervising flow comes from optical-flow estimates.
- Photometric loss minimizes mean squared error between predicted and observed frame colors.The canonical representation stores color, enabling this consistency term during optimization.
- Temporal regularization penalizes large 3D accelerations by mapping sampled points into adjacent frames.This encourages smooth estimated 3D motion across time.
- The combined objective weights flow, photometric, and regularization losses through λ coefficients.The weights control the relative importance of the individual terms.
- Hard-example mining samples high-flow-error regions more frequently to counter the dominance of reliable background correspondences.Errors are computed from cached predictions against input flows, using consecutive frames where supervision is assumed most reliable.
6. Evaluation
OmniMotion is evaluated on TAP-Vid against dense correspondence, optical-flow, trajectory, and test-time optimization baselines using position, occlusion, and temporal-coherence measures. It achieves the best overall performance, including stronger long-video and occlusion tracking, while ablations and qualitative results examine its design and behavior.
- Benchmarks and metrics: TAP-Vid evaluation reports position accuracy, occlusion accuracy, and temporal coherence across real and synthetic long-video tracking datasets.The benchmark includes fully annotated tracks and labels points as occluded when not visible.
- Quantitative comparisons: OmniMotion achieves the best position accuracy, occlusion accuracy, and temporal coherence consistently across different datasets.The method is evaluated in two variants optimized from TAP-Net or RAFT pairwise correspondences.
- Qualitative comparisons: Qualitative DAVIS results show successful tracking through occlusions, occlusion detection, plausible occluded-point locations, and handling of large camera-motion parallax.Baseline methods fail on the highlighted occlusion events.
- Quantitative comparisons: OmniMotion improves temporal coherence over direct non-adjacent correspondence methods through its globally consistent representation.Compared with chained methods, it performs especially well on longer videos, where accumulated errors affect tracking.
- Quantitative comparisons: Unlike chaining methods and PIPs, OmniMotion tracks points through extended occlusions and remains robust on longer trajectories.PIPs can lose a point when occlusion lasts beyond its eight-frame temporal window.
- Ablations and analysis: The ablation study tests replacing the invertible mapping with separate forward and backward networks, removing the strict cycle-consistency guarantees of the proposed bijections.The variant fails to construct a meaningful canonical space and has limited representational capacity.
7. Limitations
The method has limitations with difficult motion, initialization, and computational scaling. These constraints affect accuracy, optimization reliability, and practical efficiency.
- Motion and correspondence limits: Rapid, highly non-rigid motion and thin structures can prevent accurate global motion estimation.Pairwise correspondence methods may fail to provide enough reliable correspondences in these scenarios.
- Optimization limits: Initialization sensitivity on difficult videos can lead optimization to sub-optimal local minima.Observed failures include incorrect surface ordering or duplicated objects in canonical space that may be difficult to correct.
- Computational limits: Exhaustively computing all pairwise flows scales quadratically with sequence length and makes the method computationally expensive.The authors suggest more efficient matching alternatives could improve scalability.
- Computational limits: The neural implicit representation also entails a relatively long optimization process, limiting efficiency for longer sequences.Recent research may help accelerate this optimization and enable further scaling.
8. Conclusion
The paper introduces OmniMotion, a test-time optimization method for complete, globally consistent video motion. It models varied camera and scene dynamics, tracks through occlusions, and improves substantially over prior state-of-the-art methods.
- Conclusion: OmniMotion represents video motion with a quasi-3D canonical volume and per-frame local-canonical bijections.The representation supports varying camera setups and scene dynamics.
- Conclusion: OmniMotion produces complete, globally consistent, smooth long-range motion trajectories through occlusions.The method estimates motion for an entire video rather than only limited temporal windows.
- Conclusion: The method achieves significant qualitative and quantitative improvements over prior state-of-the-art methods.
A. Preparing pairwise correspondences
The method prepares global supervision by exhaustively collecting pairwise correspondences, filtering them for consistency, and recovering useful signals near occlusions. Reliable correspondences can then be chained to densify supervision across distant frames.
- A. Preparing pairwise correspondences: All pairwise correspondences between frames are exhaustively computed and filtered using cycle and appearance consistency checks.Appearance filtering uses dense DINO features for frame pairs more than three frames apart.
- A. Preparing pairwise correspondences: Distant-frame optical flow is initialized from the previous target frame to improve its predictions.Despite this initialization, distant-frame estimates can still contain significant errors requiring filtering.
- A. Preparing pairwise correspondences: Appearance checks remove spurious correspondences that pass cycle consistency by locking deforming foreground regions onto background motion.Correspondences with DINO feature cosine similarity below 0.5 are filtered out.
- A. Preparing pairwise correspondences: A two-pass cycle check identifies reliable flows in occluded regions when standard forward-backward consistency fails.Such correspondences bypass cycle filtering when they span fewer than three frames, particularly for RAFT-based variants.
- A. Preparing pairwise correspondences: After both cycle and appearance checks, filtered RAFT correspondences remain reliable across target frames with increasing temporal distance.
- A. Preparing pairwise correspondences: Chaining reliable correspondences densifies supervision between distant frames whose direct flow estimates were discarded.This is especially beneficial for sequences with rapid motion or large displacements.
B. Additional ablations
The additional ablation study evaluates an alternative plain 2D motion representation on DAVIS. This comparison tests replacing OmniMotion’s quasi-3D bijection representation with a positional-encoded MLP.
- B. Additional ablations: The Plain 2D ablation replaces the quasi-3D bijection representation with an eight-layer, 256-neuron MLP.The MLP receives query pixel location, query time, and target time, with eight-frequency positional encoding.
C. Additional implementation details
Additional implementation details describe the training losses, sampling strategy, invertible mapping architecture, and dataset-consistent evaluation setup.
- Losses: An auxiliary gradient loss compares predicted and observed color differences between randomly sampled, potentially distant pixel pairs.The same gradient loss is also applied to flow predictions to improve spatial consistency.
- Training schedule: Training samples correspondences from frame pairs with a maximum interval of 20, then expands the window by one every 2k steps.This progressively increases the temporal range used during training.
- Losses: The flow loss Lflo is a frame-interval-weighted MAE, while λpho increases from 0 to 10 over the first 50k steps and λreg is 20.The photometric loss is delayed because it refines motion more effectively after large motion errors have been reduced.
- Rendering and sampling: Each ray uses stratified sampling of K = 32 points between near and far depths, with cross-volume mappings encouraged to remain within the predefined depth range.At inference, the corresponding location is selected from the sample with the largest alpha value rather than alpha compositing.
- Network architecture: The invertible mapping network Mθ uses six affine coupling layers with alternating split patterns.Each learnable coupling component is an MLP conditioned on a frame latent code and input coordinates.
- Evaluation details: For Deformable Sprites evaluation, inverse correspondences are approximated by nearest-neighbor search in atlas space, with occlusion estimates handled separately.Target-frame points are mapped to the atlas and matched to the closest atlas coordinate for the source point.