PaperlayerView exact revision on arXiv

Source-linked AI summary

TAP-Vid: A Benchmark for Tracking Any Point in a Video

Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, Yi Yang

arXiv:2211.03726v2cs.CVstat.ML

TL;DR

The paper addresses the lack of a benchmark for tracking arbitrary physical surface points over long video clips, a capability relevant to understanding deformation and motion. It formalizes Tracking Any Point, builds TAP-Vid from real and synthetic data using semi-automatic annotation, and introduces TAP-Net. TAP-Net trained on synthetic data outperforms prior methods on the benchmark, while TAP remains bounded by transparent or liquid surfaces and validation-domain issues.

  • Problem

    Long-term tracking of arbitrary points on generic physical surfaces lacked a benchmark, despite its relevance to surface deformation, 3D shape, physical properties, and interactions.

  • Method

    The paper formalizes TAP, constructs TAP-Vid from human-annotated real videos and synthetic tracks, and uses optical-flow-assisted annotation plus an end-to-end TAP-Net model.

  • Results

    TAP-Net trained on synthetic data outperforms prior methods on all four TAP-Vid datasets by a large margin.

  • Takeaways & Limitations

    TAP-Vid provides a benchmark for long-term, class-agnostic point tracking across real and synthetic video domains.

  • Takeaways & Limitations

    TAP excludes liquids and transparent objects, and Kubric validation may not correlate well with other datasets because of domain overfitting.

Abstract

from arXiv · show

Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences about 3D shape, physical properties and object interactions. While the problem of tracking arbitrary physical points on surfaces over longer video clips has received some attention, no dataset or benchmark for evaluation existed, until now. In this paper, we first formalize the problem, naming it tracking any point (TAP). We introduce a companion benchmark, TAP-Vid, which is composed of both real-world videos with accurate human annotations of point tracks, and synthetic videos with perfect ground-truth point tracks. Central to the construction of our benchmark is a novel semi-automatic crowdsourced pipeline which uses optical flow estimates to compensate for easier, short-term motion like camera shake, allowing annotators to focus on harder sections of video. We validate our pipeline on synthetic data and propose a simple end-to-end point tracking model TAP-Net, showing that it outperforms all prior methods on our benchmark when trained on synthetic data.

1 Introduction

The paper formalizes Tracking Any Point (TAP), addressing long-term motion estimation for arbitrary physical surface points, including deformation and occlusion. It introduces a semi-automatic annotation pipeline and a benchmark combining real and synthetic data.

  • Motion understanding benefits from estimating how generic physical surface points move, accelerate, turn, and deform over time.
  • Existing trackers incompletely address arbitrary surface motion: boxes and segments lose deformation information, optical flow is pairwise, and semantic keypoints are not scalable.
  • TAP formalizes long-term tracking from a pixel query, predicting the same surface point’s trajectory and its visibility across a video.
  • The benchmark combines real videos with human annotations and synthetic videos with perfect point tracks for training and evaluation.
  • The annotation pipeline labels 1,189 Kinetics videos and 30 DAVIS videos, requiring roughly 3.3 annotator hours for 30 points across a 10-second clip.
  • The evaluation dataset contains 31,951 points tracked across 1,219 real videos, alongside baseline comparisons and evidence of transfer to JHMDB.

2 Related Work

Prior correspondence methods address parts of point tracking but differ in scope, robustness, or data requirements. TAP-Vid instead evaluates arbitrary class-agnostic surface points over longer videos.

  • Earlier long-term surface-point trackers were hand-engineered and brittle, while optical flow generally estimates motion only between image pairs.
  • Structure-from-motion relies on sparse keypoint matches, and related robotic-manipulation methods often train on large datasets depicting a single object.
  • PIPs targets long-term video point tracking but relies mainly on BADJA, which has just 9 videos and tracks joints rather than surface points.
  • Smart annotation systems have accelerated segmentation and object tracking by proposing labels that human annotators can accept or reject.
  • TAP-Vid compares against existing datasets using precise human-annotated tracks of arbitrary class-agnostic points over 10-second durations.

3 Dataset Overview

TAP-Vid defines point-track outputs for queried video points and combines real, synthetic, and simulated-robotics datasets. Its sampling strategy emphasizes diverse points across many videos, while Kubric supports arbitrary point sampling.

  • Given query points (x, y, t), algorithms output per-frame positions (x_t, y_t) and binary occlusion values o_t.
  • Synthetic datasets provide perfect tracks but imperfect realism, whereas real datasets provide human-annotated tracks from diverse videos.
  • The benchmark samples a few points from many videos because evaluation requires a sufficiently random sample, not exhaustive point labeling.
  • TAP-Vid-Kinetics uses diverse 10-second, 250-frame clips from Kinetics-700, including moving objects, camera motion, and non-ideal lighting.
  • TAP-Vid-DAVIS contains 30 challenging videos, with up to 5 objects and 5 points per object annotated at 1080p before evaluation resizing.
  • TAP-Vid-Kubric uses simulated scenes with roughly 20 objects and supplies both supervised training data and evaluation videos.
  • TAP-Vid-RGB-Stacking contains 50 simulated robotic episodes with 30 simulator-tracked points sampled per video.

4 Real-World Dataset Construction

TAP-Vid constructs real-world point tracks through a three-stage crowdsourced workflow combining object selection, optical-flow-assisted annotation, and iterative refinement. The benchmark evaluates both tracking accuracy and occlusion prediction with metrics that jointly assess visible-point locations and occlusions.

  • Iterative refinement: A second annotator checks and corrects every submitted point track until the latest annotator agrees with earlier labels.Refinement typically involves 4–5 annotators and takes approximately 3.3 hours for a 10-second video.
  • Annotation interface: The interface combines video visualization, frame-navigation controls, information panels, and buttons for adding or submitting point tracks.Tracks use ENTER, MOVE, and EXIT points, with ENTER points restarting annotation after occlusion.
  • Point annotation: Optical-flow track assistance propagates point estimates between frames, while annotators correct drift and mark occlusions.The method uses RAFT flow and finds a shortest path whose motion agrees with the flow between manually selected points.
  • Evaluation metrics: Evaluation uses Occlusion Accuracy, visible-point threshold accuracy, and Average Jaccard to combine location and occlusion performance.Thresholded position accuracy averages across 1, 2, 4, 8, and 16 pixels on 256x256 images.
  • Evaluation protocol: Validation guidance recommends synthetic Kubric for training but warns that domain overfitting can weaken its correlation with other datasets.For extensive experimentation, the authors recommend DAVIS for validation and reserving other datasets for testing.

5 Dataset Analysis

The dataset contains long, diverse point trajectories, and the annotation pipeline achieves high agreement with synthetic ground truth and between independent human raters. These analyses support the reliability of the benchmark annotations.

  • Point trajectory statistics: At least 85% of videos contain at least five trajectory clusters, indicating diverse point motion within videos.Clusters group approximately 30 trajectories per video using mean-centered distances between non-occluded points.
  • Point trajectory statistics: The dataset covers trajectories with diameters extending up to 55% of image dimensions and represents 532 unique Kinetics-700 action categories.Trajectory diameter is the maximum distance between any two positions of a point over its trajectory.
  • Human annotation quality: With optical-flow assistance on Kubric, over 99% of annotated points fall within 8 pixels of ground truth, while annotation speed improves by 28%.Accuracy rates are also 96% within 4 pixels and 87% within 2 pixels; time falls from 50 to 36 minutes per video.
  • Human annotation quality: Independent human raters agree on 95.5% of occlusion labels and 92.5% of locations at a 4-pixel threshold.Their tracks differ by an average of 1.46 pixels at 256x256 resolution.

6 Baselines

The evaluated baselines adapt existing point-tracking and optical-flow methods, but their limitations motivate TAP-Net, a simple end-to-end model built around cost volumes and separate position and occlusion prediction.

  • Baseline methods: The baseline suite adapts Kubric-VFS-Like, RAFT, and other state-of-the-art point-tracking methods with simple extensions.The implementations are intentionally kept simple as baseline comparisons.
  • Baseline performance: Existing baselines perform poorly because they omit occlusion handling, struggle with deformable objects, or transfer poorly from synthetic to real data.The benchmark structure also favors frame-pair methods and does not evaluate occlusion estimation.
  • 6.1.3 Loss Definition: The model is presented as a simple, fast, reasonable-performing end-to-end deep-learning algorithm for tracking any point.Its loss combines Huber position regression for visible points with cross-entropy occlusion classification, weighted by a trade-off parameter.
  • 6.1 TAP-Net: TAP-Net computes dense video features, compares a query feature with features across frames, and processes these comparisons as a cost volume.The cost volume is formed by comparing the interpolated query feature with spatial features at another time.
  • 6.1.2 Track Prediction: A shared network then predicts each point’s position and occlusion independently for every frame.The occlusion branch pools spatial features to regress one logit, while the position branch produces a softmax heatmap and soft argmax estimate.

7 Results

TAP-Net outperforms prior methods across the TAP-Vid datasets, while cross-dataset experiments show that Kinetics fine-tuning can improve JHMDB performance but JHMDB fine-tuning harms Kinetics performance.

  • TAP-Net outperforms all prior works on all four TAP-Vid datasets, often by a wide margin.The comparison reports average Jaccard, visible-point position accuracy, and binary occlusion accuracy.
  • RAFT performs poorly because it cannot easily detect occlusions and frame-by-frame errors accumulate during tracking.
  • PIPs performs best on TAP-Vid-DAVIS but struggles on TAP-Vid-Kinetics and TAP-Vid-RGB-Stacking.Its local-search, online design is vulnerable to long occlusions, cuts, rapid motion, and textureless objects.
  • 59.4% to 66.6%: full-resolution evaluation raises average position accuracy, while occlusion accuracy rises from 82.1% to 83%.The paper presents full-resolution evaluation as a separate challenge because competing methods use lower-resolution videos.
  • 7.1 Comparison to JHMDB: 63.4 PCK@0.1 and 80.1 PCK@0.2: Kinetics fine-tuning improves JHMDB performance over Kubric-only training.The Kubric-only values are 62.3 PCK@0.1 and 79.8 PCK@0.2.
  • 7.1 Comparison to JHMDB: 58.6 < δx_avg and 36.4 Average Jaccard: JHMDB fine-tuning reduces Kinetics performance versus 46.6 and 60.9 with synthetic-only training.The paper attributes the degradation partly to JHMDB lacking occlusion information and observes a consistent downtrend during training.

8 Conclusions

The paper introduces Tracking Any Point and the TAP-Vid benchmark, combining real human annotations with synthetic ground truth to support progress in long-term point tracking.

  • The paper introduces Tracking Any Point and the TAP-Vid dataset to advance an under-studied domain.
  • TAP remains limited to points that can be tracked reliably, excluding liquids and transparent objects and allowing occasional real-annotation errors.
  • TAP-Vid examples span Kinetics, DAVIS, Kubric, and RGB-Stacking, illustrating diverse tracked-point trajectories.
  • The project website provides ground-truth annotations, dataset downloads, processing instructions, and annotation-tool information.
  • Annotator guidelines permit any reliably trackable point and encourage use of optical-flow assistance to improve tracks.

D Simulated Dataset Generation

The synthetic TAP-Vid datasets generate point tracks from simulator geometry, while annotation examples and guidelines illustrate how points are selected and tracked across frames.

  • Synthetic point tracks are computed because Kubric and RGB-Stacking simulators do not directly expose them.Both datasets contain rigid objects, enabling tracks to be derived from pixel-space query points and object geometry.
  • Kubric samples exactly 256 query points per video, aiming for balanced coverage across objects and the background.
  • Human annotation examples cover Kinetics, DAVIS, Kubric, and RGB-Stacking using three frames per example.
  • Annotators choose visible object points, track them through occlusions, correct interpolation errors, and repeat until satisfied.
  • Guidelines show representative annotation procedures, with minor variations introduced as results and feedback accumulated.
  • RGB-Stacking samples exactly 600 first-frame query points per video, then selects 20 moving-object and 10 static-object points.

E Dataset Statistics: Agglomerative Clustering of Trajectories

The dataset assesses trajectory diversity by agglomeratively clustering point tracks using mean-centered distances, a 2-pixel threshold, and overlap requirements.

  • Agglomerative clustering starts with one cluster per trajectory and repeatedly merges clusters until inter-cluster distances exceed 2 pixels.
  • Cluster distance is the minimum distance between any two trajectories belonging to the two clusters.
  • The implementation stores trajectory coordinates and occlusions, computes pairwise distances, and merges the closest eligible clusters iteratively.
  • The clustering procedure is documented in Listing 1, while the reported metrics include Average Jaccard, position accuracy, and Occlusion Accuracy with δ = 2.

F.1 Validation on Simulated Videos

The annotation pipeline is validated on synthetic videos with known point-track ground truth and tested for accuracy, agreement, and efficiency. Optical-flow assistance improves precise tracking while reducing annotation time.

  • Validation setup: Synthetic Kubric videos provide perfect ground-truth tracks for validating human annotations under controlled conditions.Annotators labeled up to 30 points per 24-frame video, and annotated tracks were compared with ground truth.
  • Validation setup: Camera jitter and higher frame rates were added to make simulated videos more representative of realistic annotation challenges.Videos were increased from 12 FPS to 25 FPS, matching the real videos.
  • Annotation accuracy: The optical-flow track-assist tool nearly halved the error rate for points within 2 pixels and increased the within-1-pixel rate by over 25 percentage points.These improvements occurred while annotation time fell from 50 to 36 minutes per video.
  • Annotation efficiency: Annotation time decreased from 50 minutes to 36 minutes per video when using optical-flow track assistance.The tool improved accuracy despite making annotation substantially faster.
  • Human agreement: Human raters achieved 95.5% occlusion accuracy and 92.5% location agreement at a 4-pixel threshold on DAVIS point tracks.The agreement study covered 650 points across 30 videos.

G.2 Model Ablations

The ablations examine temporal backbones, coordinate prediction, query sampling, and baseline trackers. The reported results indicate that the chosen architecture and soft-argmax design are useful, while query strategy affects difficulty and existing trackers have practical limitations.

  • Model ablations: All tested architectural changes reduce performance to some extent on TAP-Vid-Kinetics.The comparisons include full time shifting, no temporal shifting, and replacing soft argmax with an MLP.
  • Model ablations: Full TSM performs worse despite its larger temporal receptive field, while removing temporal information particularly harms Average Jaccard.The authors suggest that full temporal shifting may enable memorization of Kubric motions.
  • Query sampling strategies: Strided queries sample a point every 5 frames, whereas First queries begin at the first visible frame and track only into the future.The First setting is harder because query frames are farther from output frames on average, but it supports comparisons with online methods.
  • Baseline methods: RAFT extends pairwise optical flow across multiple frames by repeatedly updating the query point, but errors accumulate and occlusions are handled only when points leave the frame.The method has no simple recovery mechanism for accumulated tracking errors.
  • Baseline methods: PIPs chains estimates across 8-frame segments and runs both forward and backward for each query point.Its refinement uses query and frame features with local-neighborhood dot products and visibility estimates.

J Semantic analysis

The semantic analysis characterizes the labeled-object distribution rather than treating TAP-Vid as a classification dataset. Kinetics categories were explicitly recorded with frequencies, while smaller datasets support qualitative inspection.

  • Object distribution: TAP-Vid is not intended for classification, but labeled-object distributions are reported to contextualize performance.The analysis distinguishes qualitative inspection for DAVIS and RGB-Stacking from explicit category labeling for Kinetics.
  • Kinetics categories: Kinetics annotators labeled object boxes before adding points, and Tables 9–12 list the named categories and their frequencies.The labeling process was informal and did not control for duplicate labels or typos.
  • Dataset comparison: DAVIS and RGB-Stacking are small enough for their object distributions to be observed qualitatively.The passage contrasts these datasets with the explicit category-frequency analysis performed for Kinetics.
Loading 2211.03726v2…