Source-linked AI summary
BootsTAP: Bootstrapped Training for Tracking-Any-Point
Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, João Carreira, Andrew Zisserman
TL;DR
Real-world TAP training data is scarce, while synthetic training data has limited variety. BootsTAP bootstraps a student-teacher model on unlabeled real videos using trajectory consistency, and reports state-of-the-art point-tracking benchmark performance. Its training assumptions concern points on solid, opaque surfaces.
Problem
Real-world TAP ground truth is scarce because manual labeling is arduous and imprecise and 3D sensing is limited in availability.
Source: §1 · p. 2Method
Results
Bootstrapped training achieves state-of-the-art point-tracking benchmark performance, surpassing prior state of the art.
Source: §1 · p. 3, §6 · p. 17Takeaways & Limitations
Consistency under affine transformations and query-point choice enables improvement on unlabeled data without complex spatial- or temporal-smoothness priors.
Source: §6 · p. 17The consistency assumptions are stated for points on solid, opaque surfaces.
Source: §3 · p. 5
Abstract
from arXiv · showhide
To endow models with greater understanding of physics and motion, it is useful to enable them to perceive how solid surfaces move and deform in real scenes. This can be formalized as Tracking-Any-Point (TAP), which requires the algorithm to track any point on solid surfaces in a video, potentially densely in space and time. Large-scale groundtruth training data for TAP is only available in simulation, which currently has a limited variety of objects and motion. In this work, we demonstrate how large-scale, unlabeled, uncurated real-world data can improve a TAP model with minimal architectural changes, using a selfsupervised student-teacher setup. We demonstrate state-of-the-art performance on the TAP-Vid benchmark surpassing previous results by a wide margin: for example, TAP-Vid-DAVIS performance improves from 61.3% to 67.4%, and TAP-Vid-Kinetics from 57.2% to 62.5%. For visualizations, see our project webpage at https://bootstap.github.io/
1 Introduction
Tracking-Any-Point represents precise motion by following queried points across video frames, but real-world training labels are scarce or costly. BootsTAP uses unlabeled videos and teacher-student consistency to improve a TAP model, reporting new state-of-the-art benchmark results with minimal architectural changes.
1 Introduction
- TAP takes a video and query points, then tracks those points across other frames and marks them occluded when they are not visible.Its outputs describe motion and deformation of points on solid surfaces over time.Source: §1 · pp. 1–2
- Real-world TAP training data is limited because manual labels are arduous and imprecise, while 3D sensing is available only in limited scenarios and quantity.Prior state-of-the-art methods therefore relied on synthetic data.Source: §1 · p. 2
- A supervised synthetic-data model initializes teacher and student networks; the teacher supplies pseudo-labels, and the student learns from a transformed, corrupted video view.The teacher is an exponential moving average of the student, and the student’s predictions are mapped back before the self-supervised loss is applied.Source: §1 · pp. 2–3, Figure 1 · p. 2
- The work presents a large-scale unlabeled-video training pipeline and reports new state-of-the-art point-tracking results with minimal architectural changes.It uses consistency under spatial transformations and invariance to query-point choice as supervisory signals.Source: §1 · p. 3
2 Related Work
Prior point-tracking work has used supervised synthetic data and several self-supervised correspondence signals, each with constraints. BootsTAP combines multi-frame student-teacher training with transformed views and cycle-consistency filtering.
Tracking-Any-Point
- Existing state-of-the-art point trackers mainly train with synthetic data, which has a large domain gap from the real world.TAP provides dense, long-duration point tracks that can support tasks including video editing, 3D estimation, object segmentation, camera tracking, and robotics.Source: §2 · p. 3
Self-supervised correspondence via photometric loss
- Photometric self-supervision can produce false matches under occlusion, lighting changes, or repeated and constant textures, prompting methods to add complex priors and losses.Local appearance matching can better capture motion in high-resolution video by resolving detailed textures.Source: §2 · p. 4
Temporal continuity and cycle-consistency
- Temporal cycle-consistency can yield tracking features, but it fails to provide useful supervision in challenging situations such as occlusions..Source: §2 · p. 4
Semi-supervised correspondence
- Prior pseudo-labeling methods for longer-term tracking include approaches that do not retrain a general TAP model and a reconstruction-based algorithm that combines supervised and adversarial losses.The latter is described as complex and reports 59.8 on TAP-Vid-DAVIS < δx avg, versus 78.1 for this work.Source: §2 · p. 4
- The paper’s pipeline generates teacher pseudo-label trajectories, transforms and corrupts the student’s video view, maps student tracks back, and filters trajectories that fail cycle consistency.This illustration also shows queries sampled along the teacher’s trajectory.Source: Figure 2 · p. 5
3 Method
BootsTAP trains a student against teacher-generated pseudo-labels on unlabeled videos, relying on trajectory consistency while guarding against trivial solutions. It transforms the student’s view, samples queries along teacher tracks, and filters unreliable trajectories before applying self-supervised and supervised losses.
3 Method
- A student-teacher setup is used because backpropagating through both views can produce trivial solutions, while the teacher receives no gradients that might corrupt its predictions.The approach assumes solid, opaque surface tracks are equivariant to spatial transformations and invariant to non-spatial changes, with each trajectory independent of its query point.Source: §3 · pp. 5–6
- After pretraining, the method adds five 2D convolutional residual layers to the backbone and derives pseudo-label targets from teacher predictions.The added layers roughly double backbone parameters and are initialized to the identity.Source: §3 · p. 6
- The self-supervised loss derives from TAPIR’s position, occlusion, and uncertainty losses, treating teacher pseudo-labels as ground truth and using the teacher’s final refined prediction.Refined predictions supervise unrefined student predictions, encouraging stronger features that enable faster convergence.Source: §3 · pp. 6–7
- The student receives a harder video view with smoothly varying affine transformations, lower resolution, and JPEG corruption; its predictions are mapped back to the original coordinates for loss computation.The degradations remove texture information, encouraging tracking from higher-level cues rather than low-level texture matching.Source: §3 · p. 7
- The student query is sampled from a visible point on the teacher’s trajectory, and trajectories are excluded when the student’s prediction at the initial time is too far from the teacher’s query.The cycle-consistency distance threshold is 4 pixels; the same query is sampled with probability 0.5, with the remaining probability assigned across other visible teacher-predicted points.Source: §3 · pp. 7–8
- Training continues on Kubric with the supervised TAPIR loss to avoid catastrophic forgetting, using separate Adam optimizer parameters and learning rates for the supervised and self-supervised tasks.The related approach [57] freezes its computed tracks and fine-tunes on a target dataset, whereas this work describes training across domains without requiring fine-tuning.Source: §3 · p. 8
4 Experiments
The experiments train a TAPIR-based model on unlabeled online videos alongside Kubric and evaluate it zero-shot on TAP-Vid and RoboTAP. Bootstrapping improves benchmark performance, while ablations identify important choices in transformations, query sampling, training setup, and data.
4.1 Training datasets
- Training uses over 15 million 24-frame clips from publicly available online videos alongside standard Kubric training.The data pipeline selects 60-fps videos, filters overlays and shot changes, and samples five clips per video.Source: §4 · p. 8, §4.1 · p. 9
4.2 Evaluation datasets
- Evaluation is zero-shot on five ground-truth datasets, with qualitative evaluation also reported for RoboCAT-NIST and Libero, which lack ground truth.The quantitative datasets include Kinetics, DAVIS, RGB-Stacking, and RoboTAP; metrics include occlusion accuracy, localization accuracy, and Average Jaccard.Source: §4.2 · p. 9
4.3 Results
- Over 5% absolute performance improvement on RoboTAP accompanies wide-margin Kinetics gains; the method also beats CoTracker on DAVIS, chiefly through occlusion accuracy.The comparisons use query-first metrics on Table 2 and benchmark metrics on Table 1.Source: §4.3 · pp. 9–10, Table 2 · p. 10, Table 1 · p. 10
- BootsTAPIR improves qualitative tracking on ambiguous textures, object boundaries, visible-but-occluded points, textureless grippers, scale changes, and shiny objects.On RoboCAT-NIST, where ground truth is unavailable, tracks are smoother and occlusion predictions more accurate; results are described as comparable on Libero.Source: §4.3 · p. 11, Figure 3 · p. 12, Figure 4 · p. 13
4.4 Ablations
- Removing affine transformations causes large performance drops, while JPEG augmentation helps especially on Kinetics and varied query points outperform using only the teacher query.The varied-query gains in localization increase < δx avg from 77.5 to 77.9 on DAVIS strided and from 66.8 to 67.7 on Kinetics q_first; DAVIS benefits depend on sampling the teacher query more often.Source: §4.4 · pp. 11–13
- Removing occlusion-confidence filtering has little effect on DAVIS but degrades Kinetics, while cycle-consistency filtering performs slightly better on DAVIS.The authors note that correctly removing bad teacher tracks remains an open problem.Source: §4.4 · p. 13
- Self-supervised training adds 1.2% on DAVIS and almost 2% on Kinetics over longer Kubric-only training; backpropagating through both models instead of using EMA and stop-gradient collapses real-world performance.The ablation base also included an occlusion-loss mask and a 3D-ConvNet; the latter slightly helped DAVIS but harmed Kinetics and was removed for compatibility with causal TAPIR models.Source: §4.4 · pp. 11, 13, 14
- Training on 2- or 6-frame clips performs poorly, while a 1% data subset harms Kinetics but improves DAVIS.The authors suggest extra temporal context may let the teacher correct more errors and that the algorithm can remain effective with less data.Source: §4.4 · p. 14
5 Higher Resolution and Public Release
The released model adds longer and higher-resolution training data and reports gains across high-resolution, causal, and additional benchmark evaluations. The section also notes a background-tracking bias that can cause thin foreground objects to be lost.
5 Higher Resolution and Public Release
- The release fixes a pixel-misaligned crop; snap-to-occluder harms TAPIR but improves BootsTAP.The authors suggest snapping may compensate for a background-tracking bias in bootstrapping.Source: §5 · pp. 14–15
- The extra training tasks include 512×512, 24-frame clips and 150-frame, 256×256 videos at 30 frames per second.The 512×512 tasks use Kubric and the same real-world video dataset.Source: §5 · p. 15
- At 512×512 evaluation, performance improves by 1.2% on Kinetics and 2.8% on DAVIS.Further test-time resolution did not improve results.Source: Table 6 · p. 16, §5 · p. 16
- Bootstrapping can lose thin foreground objects as reliable background tracks become self-reinforcing.The authors identify finding a more principled solution as future work.Source: §5 · pp. 15–16
- The causal model improves performance by 4.6% on Kinetics and 3.0$ on DAVIS under query-first evaluation.The model is trained using the full release setup, including extra high-resolution and long-clip data.Source: §5 · p. 16, Table 7 · p. 17
- On the Perception Test, BootsTAPIR shows a similar-magnitude gap over prior results.The benchmark uses point tracks annotated on videos of unusual situations filmed by participants.Source: §5 · p. 16, Table 8 · p. 17
- Informal A100 benchmarking reports 10,000-point inference in 5.6 seconds and causal tracking of 400 points at 30.1 frames per second.Both results use 256×256 video; the first uses a 50-frame clip.Source: §5 · p. 17
6 Conclusion
BootsTAP uses consistency principles on unlabeled data to improve TAP performance without complex motion-smoothness priors. Its loss procedure uses teacher predictions and transformed student views to compute self-supervised track losses.
6 Conclusion
- The conclusion reports that query-point invariance and non-spatial corruption invariance, alongside affine-transform equivariance, improve TAP performance on unlabeled data.The formulation avoids spatial smoothness of motion and temporal smoothness of tracks, and the authors report continued improvement with longer training.Source: §6 · p. 17
- Training remains computationally expensive, and single-point correspondence estimates cannot elegantly handle duplicated or rotationally symmetric objects with ambiguous correspondence.The authors frame this as a limitation of the approach.Source: §6 · p. 17
A Summary of the approach
- The loss procedure samples teacher queries, predicts teacher tracks and occlusions, and uses those predictions as pseudo-labels for the student.The student receives a transformed video view and query, with predicted coordinates transformed back to the original space.Source: A Summary of the approach · p. 22
B Implementation details
The transformation scheme mimics plausible camera motion and zooming by applying smoothly varying spatial transformations that preserve reasonable content coverage and aspect ratios.
B.1 Distribution over transformations
- Frame-wise affine transformations vary smoothly over time to mimic simple, plausible camera motion and zooming.Crop dimensions and top-left coordinates are interpolated between sampled start- and end-frame values.Source: §B.1 · p. 23
- Start- and end-frame crop dimensions are sampled to cover 60%–100% of the input while favoring aspect ratios near 1.The sampled area is uniform on [0.6, 1.0], and the procedure samples crop dimensions and top-left coordinates.Source: §B.1 · p. 23
- Each sampled transform rescales a frame and places it in a zero-valued canvas, with the transform applied after frame augmentation.The query coordinates receive the corresponding transformation as well.Source: §B.1 · pp. 23–24
B.2 Training details
Training combines Kubric and real videos, uses added residual blocks, and follows a cosine learning-rate schedule with a lower rate for PIPs mixer steps.
B.2 Training details
- Each device trains on 4 Kubric videos and 2 real videos per batch for 200,000 iterations across 256 A100 GPUs.The learning rate peaks at 2e-4 after 1,000 warmup steps.Source: §B.2 · p. 24
- The model adds five residual blocks to the stride-8, 256-channel backbone, each using two sequential 3 × 3 convolutions with channel expansion factor 4.The added layers are placed on top of the backbone.Source: §B.2 · p. 24
- A cosine schedule uses a 2e-4 peak learning rate, while PIPs mixer steps use one-fifth the backbone learning rate.The schedule includes 1,000 warmup steps; other hyperparameters match TAPIR.Source: §B.2 · p. 24
B.3 Libero finetuning
BootsTAP is further trained on Libero alongside Kubric and internet videos, and the authors report improved tracking despite the domain gap with internet data.
B.3 Libero finetuning
- The Libero evaluation uses the gripper view, which contains large and difficult motions, and the internet-video-trained model already improves results substantially.The authors note a large domain gap between Libero data and internet videos.Source: §B.3 · p. 24
- Further self-supervised training on Libero, Kubric, and internet videos improves results despite using no labels.The model is trained for another 50K steps, with an update weight of 0.2 for each self-supervised task.Source: §B.3 · p. 24
- The additional training uses the full set of demonstrated trajectories and sums updates from separate Kubric, internet-video, and Libero optimizers.All tasks use the gripper view; other parameters are kept the same between Libero and internet-video training.Source: §B.3 · p. 24
B.4 Snap-to-occluder
The appendix describes a query-point adjustment intended to favor foreground tracking and compares 2D and 3D backbones, retaining the 2D backbone for the final model.
C Comparison with and without a 3D ConvNet Backbone
- The 3D backbone slightly improves TAP-Vid DAVIS strided position accuracy but harms occlusion accuracy, and significantly degrades Kinetics query_first performance.The authors retain a 2D backbone for the final model, noting the optimal choice may depend on the downstream application.Source: Table 9 · p. 25, C Comparison with and without a 3D… · p. 25
B.4 Snap-to-occluder
- Snap-to-occluder replaces some background query points with neighboring foreground points so the model must track the foreground object.A neighbor qualifies as front-side when its depth exceeds the query point’s by more than 5%; replacement occurs with 50% probability.Source: §B.4 · pp. 24–25
C Comparison with and without a 3D ConvNet Backbone
- The alternative 3D ConvNet converts each residual block’s first convolution to 3×3×3, giving features a 21-frame temporal receptive field.The 2D feature extractor processes each frame independently and cannot use temporal cues for feature extraction.Source: C Comparison with and without a 3D… · p. 25
D Evaluation Datasets
The evaluation suite spans internet and benchmark videos, synthetic stacking scenes, and robotics tasks, with varied track availability, duration, and visual difficulty.
D Evaluation Datasets
- TAP-Vid-Kinetics contains 1K diverse action videos with about 26 tracked points each and challenging cuts, motion, clutter, and lighting.The videos are about 10 seconds long and come from the Kinetics-700-2020 validation set [7].Source: D Evaluation Datasets · p. 26
- TAP-Vid-DAVIS extends 30 DAVIS 2017 validation videos with about 22 point tracks each for tracking evaluation.The videos use the same human annotation process as TAP-Vid-Kinetics; DAVIS is a video object segmentation benchmark [49].Source: D Evaluation Datasets · p. 26
- TAP-Vid-RGB-Stacking provides 50 Kubric-generated synthetic videos, each with 30 annotated tracks across 250 frames.The videos simulate a robotic stacking environment.Source: D Evaluation Datasets · p. 26
- RoboTAP contains 265 real-world manipulation videos, averaging about 272 frames and 44 tracks, with textureless symmetric objects far out-of-domain for training data.The videos are far out-of-domain relative to Kubric and the online lifestyle videos used for self-supervised learning.Source: D Evaluation Datasets · p. 26
- RoboCAT-NIST demonstrates gear insertion with symmetric, textureless gears, but has no human ground-truth tracks.It uses teleoperated robot arms, three gear sizes, and a 3-peg base; videos were processed to 64 frames at 222 × 296 resolution [6].Source: D Evaluation Datasets · p. 26
- Libero contains variable-length synthetic robot-arm demonstrations for diverse tasks, with no ground-truth tracks.The sequences are 128×128 and intended for imitation learning; point tracking has previously proved useful for robotic manipulation [69].Source: D Evaluation Datasets · p. 26
D.1 Evaluation metrics
Evaluation uses three metrics for tracking and occlusion quality, alongside two query modes that differ in when queries are sampled and which time direction is measured.
D.1 Evaluation metrics
- Average Jaccard is the benchmark target, combining occlusion and position accuracy across five pixel thresholds.At each threshold, it is the fraction of true positives over true positives, false positives, and false negatives.Source: §D.1 · pp. 26–27
- < δx_avg averages visible-point position accuracy over thresholds of 1, 2, 4, 8, and 16 pixels, while Occlusion Accuracy averages framewise binary occlusion-classification accuracy.Position accuracy counts visible points closer than the threshold to ground truth.Source: §D.1 · p. 26
- Strided mode queries visible points every five frames and evaluates forward and backward tracking; query-first mode queries first-visible points and evaluates future tracking only.Both modes are used for TAP-Vid evaluation.Source: §D.1 · p. 27