PaperlayerView exact revision on arXiv

Source-linked AI summary

TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement

Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, Andrew Zisserman

arXiv:2306.08637v2cs.CV

TL;DR

Tracking arbitrary physical surface points through video requires both occlusion-aware matching and temporally consistent refinement. TAPIR combines independent per-frame initialization with iterative local refinement, substantially improving TAP-Vid results, extending to high-resolution videos, and enabling plausible image-based animations.

  • Problem

    Tracking Any Point requires locating queried physical surface points across video frames and identifying when they are occluded, despite challenges including limited real-world ground truth.

  • Method

    TAPIR uses per-frame global matching for initialization followed by iterative refinement of trajectories and query features from local correlations.

  • Results

    TAPIR improves over prior works by approximately 20% on the cited TAP-Vid comparisons and substantially reduces inference runtime on Kinetics.

  • Takeaways & Limitations

    TAPIR supports fast long-term tracking, higher-resolution video extension, and plausible animation generation from static images.

  • Takeaways & Limitations

    TAPIR can struggle with precise foreground-background boundaries and large appearance changes; the public Kubric-trained version may also underperform with moving cameras or points leaving the frame.

Abstract

from arXiv · show

We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS. Our model facilitates fast inference on long and high-resolution video sequences. On a modern GPU, our implementation has the capacity to track points faster than real-time, and can be flexibly extended to higher-resolution videos. Given the high-quality trajectories extracted from a large dataset, we demonstrate a proof-of-concept diffusion model which generates trajectories from static images, enabling plausible animations. Visualizations, source code, and pretrained models can be found on our project webpage.

1. Introduction

TAPIR addresses long-term physical point tracking by combining occlusion-robust per-frame search with temporally informed refinement. It substantially improves TAP-Vid performance and supports trajectory-based animation from still images.

  • Problem: TAP formulates point tracking as locating queried surface points across visible video frames while identifying occlusions.The task targets physical correspondence rather than only consecutive-frame motion or selected keypoints.
  • Challenges: TAPIR must handle occlusions, point reappearance, long visible intervals, and limited real-world supervision with synthetic-to-real transfer.These challenges motivate both search and temporal integration in the model.
  • Approach: TAPIR combines a coarse, occlusion-robust match computed independently per frame with fine iterative refinement using local spatio-temporal information.Its design integrates the complementary strengths of global search and temporal refinement.
  • Results: Approximately 20% improvement is reported on TAP-Vid-DAVIS over TAP-Net and on TAP-Vid-Kinetics over PIPs, with substantially reduced inference runtime on Kinetics.The comparison is benchmark-specific: DAVIS uses TAP-Net as the reference, while Kinetics uses PIPs.
  • Applications: A proof-of-concept model uses TAPIR trajectories to generate plausible animations from individual images.The paper also analyzes architectural choices and releases the resulting model and resources.

2. Related Work

Prior correspondence methods address dense pairwise motion, sparse reconstruction keypoints, semantic landmarks, or long-term tracking, but differ from tracking arbitrary physical surface points through videos.

  • Optical Flow: Optical flow estimates dense motion between subsequent frames but lacks a simple mechanism for tracking across long videos.Ground truth is difficult to obtain, so evaluation often uses synthetic scenes.
  • Keypoint Correspondence: Keypoint correspondence matches sparse, easily trackable features across image pairs, typically for structure-from-motion rather than arbitrary points.These methods generally ignore occlusion because geometric filtering can remove errors in SfM.
  • Semantic Keypoint Tracking: Semantic keypoint tracking follows landmarks whose appearance can vary substantially with pose and therefore often relies on large supervised datasets.Some work tracks surface meshes, and motion-based sim2real motivates improved surface tracking.
  • Long-Term Physical Tracking: Long-term physical point tracking has earlier hand-engineered methods, while TAPIR builds on the deep-learning approaches TAP-Net and PIPs.TAP-Net supplies per-frame global search, whereas PIPs contributes local refinement and smoothing.
  • TAPIR Context: TAPIR’s architecture uses global per-frame matching followed by local temporal refinement, as summarized in its two-stage design.The figure omits multi-scale pyramids for simplicity.

3. TAPIR Model

TAPIR initializes tracks by matching query features independently in every frame, then iteratively refines positions, occlusion, uncertainty, and query features using local multi-scale temporal correlations.

  • Track Initialization: For each frame, TAPIR estimates a 2D point location, occlusion probability, and location-uncertainty probability.The model first compares query features with all frame features to obtain an occlusion-robust initial estimate.
  • Design Rationale: TAPIR’s design combines TAP-Net-style per-frame initialization with PIPs-style refinement while adding temporal convolution and other components.The combination is presented as bridging the merits of both prior architectures.
  • Track Initialization: The initialization stage computes a coarse cost volume from query-to-frame feature dot products and converts each frame’s heatmap into a differentiable point estimate.A spatial soft argmax with thresholding suppresses spurious matches but limits original-resolution localization to a few pixels.
  • Track Initialization: TAPIR predicts uncertainty separately so visible points with substantially incorrect locations can be distinguished from ordinary occlusions.A prediction is considered useful when it lies within distance threshold δ of the ground truth.
  • Iterative Refinement: Each refinement iteration extracts local score maps around the trajectory across multiple spatial-pyramid resolutions and processes many frames jointly.The network receives positions, query features, and flattened local score maps, then predicts updates that integrate information across time.
  • Iterative Refinement: A convolutional refinement network updates position, occlusion, uncertainty, and query features, with reusable parameters across iterations.Its blocks combine 1×1 and depthwise convolutions, translating channel-mixing and within-channel operations from the PIPs MLP-Mixer.
  • Feature Design: High-precision refinement uses 7 × 7 local neighborhoods and multi-resolution score maps, including a stride-4 map to reduce train/test resolution mismatch.The stride-4 map increases memory pressure but improves training consistency with high-resolution feature extraction.
  • Training Dataset: Training data include a modified panning dataset and Kubric MOVi-E to address the synthetic-to-real gap caused by limited camera-motion diversity.Kubric’s camera looks toward a fixed workspace center, unlike real panning scenes.

4. Extension to High-Resolution Videos

TAPIR extends from 256 × 256 training videos to higher-resolution inputs through an image pyramid, using coarse initialization followed by refinement at progressively larger resolutions.

  • High-Resolution Extension: TAPIR handles higher-resolution videos by initializing at 256 × 256 and refining the track at logarithmically spaced pyramid levels up to the original resolution.Each level uses the preceding position estimate while reinitializing occlusion and uncertainty to avoid overconfidence.
  • High-Resolution Extension: Applying per-frame initialization directly to a much larger image grid can create false positives because the model is tuned for a 32 × 32 feature grid.The pyramid limits this issue while preserving the feature-comparison structure that supports resolution scaling.

5. Experiments

TAPIR is evaluated on diverse TAP-Vid datasets, where it improves substantially over prior methods, remains effective under query-first evaluation, and supports high-resolution, fast inference. Ablations show that its architectural components and refinement design contribute non-trivially to performance.

  • Benchmark and evaluation: TAP-Vid covers four datasets with distinct challenges, including complex motion, scale changes, cuts, camera shake, textureless regions, and synthetic point tracks.The benchmark includes DAVIS, Kinetics, RGB Stacking, and Kubric MOVi-E.
  • Quantitative comparison: 19.3% absolute improvement over PIPs on DAVIS and 10.6% over TAP-Net on Kinetics demonstrate TAPIR’s substantial benchmark gains.The paper notes that Average Jaccard combines position and occlusion accuracy, making these absolute improvements significant.
  • Quantitative comparison: Under query-first evaluation, TAPIR preserves the same relative advantage, outperforming TAP-Net by 10% on Kinetics, 20% on DAVIS, and 3% on RGB-Stacking.Performance is overall lower than in the strided evaluation because query-first points have more intervening frames before prediction.
  • High-resolution and runtime: TAPIR processes 50 queried points at roughly 150 frames per second on 256x256 DAVIS video, while partitioning across 8 TPU-v3 devices enables higher-resolution evaluation.The reported high-resolution limits are 1080p for DAVIS and 720p for Kinetics.
  • High-resolution and runtime: Higher image resolution improves point localization but makes occlusion prediction less accurate, leaving multi-resolution combination as future work.The authors attribute the occlusion change possibly to the larger contextual information available at higher resolution.
  • Ablation studies: Ablations find that depthwise convolution, uncertainty estimation, higher-resolution features, and TAP-Net initialization each contribute to TAPIR’s performance.Depthwise convolution matters more on DAVIS, while depthwise convolution and uncertainty estimation contribute roughly equally on Kinetics.
  • Ablation studies: TAPIR performs best with 4–5 refinement iterations, and additional iterations can reduce performance, possibly through oversmoothing.The paper uses 4 iterations and suggests stronger TAP-Net initialization may require fewer iterations to converge.

6. Open-Source Version

The open-source TAPIR release adds tuned architectural, constant, and training modifications, and provides model variants for different camera-motion settings.

  • Open-Source Version: The open-source TAPIR model introduces modifications to the backbone, model and training constants, and training setup.These changes follow comprehensive hyperparameter tuning intended to produce the most powerful released model.
  • Open-Source Version: The open-sourced TAPIR version improves substantially over the major reported model, particularly on the RGB-Stacking dataset.
  • Open-Source Version: The released backbone uses a Pre-ResNet18 without max pooling, with four ResNet layers having dimensions [64, 128, 256, 256].The layers use two ResNet blocks each and strides [1].
  • Open-Source Version: The public version uses three pyramid layers, four iterative refinements, a softmax temperature of 20.0, and an expected distance threshold of 6.The temperature was found to perform marginally better than 10.0.
  • Open-Source Version: Training runs for 50,000 steps with AdamW, 1,000 warm-up steps, a base learning rate of 0.001, cosine scheduling, and weight decay 0.1.Each TPU device uses a batch size of 8.
  • Open-Source Version: TAPIR is trained on a modified panning dataset and Kubric MOVi-E, with separate versions offered for applications involving static or moving cameras.The Kubric-trained model performs well with static cameras but can be suboptimal with moving cameras, especially when points leave the frame.

7. Animating still images with TAPIR

The paper uses TAPIR trajectories as an explicit motion representation for diffusion-based animation of still images. A trajectory model predicts motion, while a pixel model renders appearance conditioned on those trajectories.

  • Animating still images with TAPIR: Animating still images is challenging because realistic motion is difficult to generate and video modeling is computationally expensive.The section asks whether explicit surface motion can improve generated videos.
  • Animating still images with TAPIR: The pipeline pairs a diffusion model that generates dense trajectories with a second diffusion model that generates pixels from the image and trajectories.At test time, predicted trajectories dictate the motion and the pixel model produces appearance consistent with the first frame.
  • Animating still images with TAPIR: The trajectory model encodes the conditioning image with a stride-4 ConvNet and uses a 2D U-Net to denoise trajectory representations stacked across frames.The representation includes occlusion, relative and absolute positions, and 32 Fourier features.
  • Animating still images with TAPIR: Trajectory samples visualize motion by coloring paths from dark purple at their starting points toward brighter yellow with distance.The examples depict alternative motions such as cutting, whole-body translation, circular arm movement, head turning, and arm raising.
  • Animating still images with TAPIR: The model generates multiple physically plausible motions for each image and performs rudimentary human pose estimation without supervision.This demonstrates transfer from TAPIR trained without humans and a trajectory model trained on video rather than still images.

8. Conclusion

TAPIR combines matching and refinement for point tracking, achieving stable, occlusion-robust predictions that scale to high-resolution video. The paper also demonstrates still-image animation, while identifying remaining visual limitations.

  • Conclusion: TAPIR introduces a two-stage Tracking Any Point model comprising a matching stage and a refinement stage.
  • Conclusion: The model provides stable predictions, occlusion robustness, and scalability to high-resolution videos.
  • Conclusion: TAPIR supports a proof-of-concept for animating still frames, proposed as a foundation for future graphics research.
  • Conclusion: TAPIR can have difficulty precisely perceiving foreground-background boundaries and may struggle with large appearance changes.

A. Runtime Analysis

The runtime study evaluates TAP-Net, PIPs, and TAPIR on DAVIS under varying query counts and video lengths. TAP-Net and TAPIR exploit parallelization, whereas PIPs scales linearly with processed points and frames.

  • Runtime Analysis: An RNN provides relatively little benefit for this form of temporal integration, without producing a measured detriment.The result is identified as an area for future research.
  • Runtime Analysis: The runtime experiment uses the 256x256 DAVIS horsejump-high video, a single V100 GPU, five runs, 10–50 query points, and 8–50 frames.Query points are randomly sampled from the first frame, and videos are truncated to the tested lengths.
  • Runtime Analysis: TAP-Net and TAPIR provide fast inference through parallelization, with runtime independent of query count in the tested range and better-than-linear scaling across frames.Feature computation and GPU overhead dominate runtime for the evaluated query counts, although parts of both algorithms remain linear in frame count.
  • Runtime Analysis: PIPs exhibits a linear increase in computational time with the number of points and frames processed.

B. More Ablations

This section quantitatively examines architectural design choices in TAPIR. It asks whether simple alternatives can process full videos and whether within-channel computation is effective.

  • The analysis evaluates four open questions about TAPIR’s architectural design decisions.
  • One question is whether simple, fast alternatives to chaining can process the full video between query and output frames.
  • Another question concerns the role of within-channel computation, including depthwise convolutions and within-channel mixer layers.

B.1. Recurrent Neural Networks

The ablations examine recurrent processing, convolution choices, pyramid depth, and temporal shifting. Recurrent processing provides little improvement, while simpler architectural choices often preserve competitive performance.

  • Recurrent Neural Networks: PIPs’ chaining can use intervening frames to maintain tracks when appearance changes without occlusions, but temporal continuity may backfire under occlusions.
  • Recurrent Neural Networks: TAPIR adds an across-time RNN after cost-volume computation to investigate whether recurrent processing can provide chaining-like temporal continuity.
  • Recurrent Neural Networks: The Conv RNN summarizes global motion and propagates matches while avoiding direct feature access and excessive latent state that could encourage overfitting.
  • Recurrent Neural Networks: Gating changes performance by only 0.4% on Kinetics and -0.2% on DAVIS, while no tested RNN improved over this configuration.
  • Depthwise versus Dense Convolution: Depthwise convolution performs slightly better than dense 1D convolution despite the dense alternative nearly quadrupling refinement-network parameters.
  • Feature Pyramids and Time-Shifting: Two pyramid levels save computation while providing competitive performance, and removing TSM causes negligible losses of 0.1% on Kinetics and 0.3% on DAVIS.

C. Implementation Details

TAPIR combines stride-based feature extraction, global matching, and iterative refinement. Its implementation includes temporal normalization, convolutional updates, occlusion prediction, and memory-conscious query batching.

  • Feature Extraction: The bulk experimental backbone is a TSM-ResNet, using stride-8 and stride-4 features normalized across channels.
  • Initialization: The initialization computes query-to-video dot products, then uses embedding convolutions, spatial soft argmax, and an MLP to estimate positions, occlusion, and uncertainty.
  • Refinement: Each refinement iteration predicts trajectory and query-feature updates, reuses network parameters, and can be repeated as many times as desired.
  • Refinement: The refinement network follows PIPs with 12 residual blocks combining pointwise and depthwise units, expanding channels to 2048 before reducing them to 512.
  • Training: Training samples 256 query points per video, but refinement uses batches of 32 query points to avoid running out of memory.

C.2. Training Dataset

The paper modifies synthetic training data to include camera panning, then uses TAPIR trajectories to train trajectory and video diffusion models. Memory constraints motivate per-frame video reconstruction.

  • Training Dataset: The new dataset addresses TAPIR failures on background points that move offscreen early in panning videos.
  • Training Dataset: The modified MOVi-E camera follows a random linear look-at trajectory near the workspace center, introducing panning while keeping objects mostly in view.
  • Diffusion Models: The diffusion pipeline chains two models without image pretraining: a trajectory model and a video model.
  • Diffusion Models: TAPIR processes 24-frame, 256×256 clips with a dense 64×64 first-frame query grid to provide training trajectories.
  • Diffusion Models: The trajectory model predicts trajectories from the first image, while the video model uses that image and trajectories to reconstruct later frames.
  • Video Model: Because full-clip reconstruction is memory-prohibitive, training reconstructs one frame at a time using warped image features, warped pixels, and nearby noisy frames.
  • Video Model: Multiple image warps help the model infer patch contributions and undo aliasing when patches overlap.
Loading 2306.08637v2…