CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos

1 INTRODUCTION

• Recent point trackers use transformer-based designs for recovering video correspondences, with joint tracking improving quality for partially occluded points.Examples include PIPs, TAP-Vid, TAPIR, and CoTracker.Source: §1 · pp. 1–2
• The method targets a gap caused by difficult real-video annotation, costly synthetic-data scaling, and residual sim-to-real differences.High-quality training data is important because trackers must handle ambiguous cases such as occlusions and fast motion.Source: §1 · p. 2
• The paper studies whether the millions of real videos and complex semi-supervised recipe used by BootsTAPIR are necessary for strong point tracking.It also compares competing tracker designs whose individual components were previously shown to help but whose necessity remained unclear.Source: §1 · p. 2
• CoTracker3 outperforms BootsTAPIR on TAP-Vid and Dynamic Replica while using three orders of magnitude fewer unlabelled videos and a simpler training protocol.The model also studies performance as the amount of real training video increases.Source: §1 · p. 2
• CoTracker3 combines ideas from prior trackers while removing global matching and simplifying other components, helping assess which design choices are necessary.Its architecture can operate offline with a single window or online with sliding windows when trained accordingly.Source: §1 · p. 2

2 RELATED WORK

• Point-tracking research progressed from PIPs and TAPIR’s correlation and matching designs to CoTracker’s joint tracking, LocoTrack’s 4D correlations, and other specialized architectures.CoTracker3 further simplifies this framework while retaining joint tracking for occluded points and running 27% faster than LocoTrack.Source: §2 · pp. 2–3
• Most point trackers use automatically annotated synthetic videos because precise pixel-level annotation of real tracks is difficult.This paradigm includes methods trained solely on Kubric.Source: §2 · p. 3
• Alternative semi-supervised methods use photometric consistency, cycle consistency, temporal continuity, colorization, or pseudo-label refinement, but can require extra losses, occlusion reasoning, augmentation, or suffer saturation.These approaches provide training signals without full point-track supervision.Source: §2 · p. 3
• BootsTAPIR fine-tunes on 15 million real videos with augmentations, masks, EMA, and retained synthetic supervision, whereas CoTracker3 removes these requirements.CoTracker3 also avoids ground-truth supervised data during self-supervised fine-tuning and uses only a small fraction of BootsTAPIR’s real videos.Source: §2 · p. 3

3.1 TRAINING USING UNLABELLED VIDEOS

• CoTracker3 trains a student with pseudo-labels from multiple frozen teachers on real videos, using 1,000× less data than BootsTAPIR.Teachers include CoTracker3 online and offline, CoTracker, and TAPIR; the student is pretrained on synthetic data.Source: §3.1 · p. 4

3.2 COTRACKER3 MODEL

• The tracker computes multiscale convolutional feature maps and 4D correlations between query-point neighborhoods and current track estimates.Correlation features are projected before entering the transformer.Source: §3.2 · p. 5

3.3 MODEL TRAINING

• Training supervises visible and occluded tracks with discounted Huber losses, while confidence and visibility use per-iteration BCE losses.For pseudo-labelled videos, confidence and visibility are not supervised directly and their separate prediction layer is frozen.Source: §3.3 · p. 6
• The online model processes videos in overlapping forward-only windows, whereas the offline model tracks in both directions and handles occluded points better.Offline tracking improves long-term visible-point tracking but is memory-bound; online tracking can operate indefinitely in real time.Source: §3.2 · p. 5, §3.3 · p. 6

3.2 COTRACKER3 MODEL

• CoTracker3 removes global matching and replaces LocoTrack’s correlation module with a simpler MLP while iteratively updating tracks, confidence, and visibility.The model uses 4D correlations, Fourier displacement embeddings, and a transformer for incremental updates.Source: §3.2 · pp. 5–6, §3.4 · p. 6

3.4 DISCUSSION

• CoTracker3 has 2× fewer parameters than CoTracker and is 27% faster than LocoTrack despite using cross-track attention.The efficiency gains are attributed to removing global matching and using an MLP for correlations.Source: §3.4 · pp. 6–7
• 15k additional real videos make CoTracker3 online outperform all other methods on TAP-Vid, including BootsTAPIR trained on 1,000× more real videos.The comparison uses synthetic Kubric pretraining plus 15k real videos.Source: Table 1 · p. 7

4.1 COMPARISON TO THE STATE-OF-THE-ART

• With 15k real videos, CoTracker3 exceeds state-of-the-art results across all metrics on DAVIS, RGB-Stacking, and Kinetics, and on AJ and OA for RoboTAP.It outperforms BootsTAPIR despite using 15k rather than 15M additional real videos.Source: §4.1 · p. 8, Table 2 · p. 8
• Offline CoTracker3 tracks occluded points better than its online version because access to all frames enables interpolation behind occlusions.On Dynamic Replica, the online model is already better than other methods when trained only on Kubric; adding real data improves occluded tracking only for the offline model.Source: §4.1 · p. 8

4.2 SCALING EXPERIMENTS

• CoTracker3, LocoTrack, and CoTracker all improve as pseudo-labeled real-video data increases, with CoTracker3 online, offline, and LocoTrack tending to plateau after 30k videos.All models improve with 0.1k videos, while CoTracker continues improving up to possibly 100k; CoTracker3 self-training adds 1.2 average TAP-Vid points.Source: §4.2 · p. 9

4.3 ABLATIONS

• Cross-track attention improves Dynamic Replica by +5.1 for occluded points versus +1.6 for visible points by inferring hidden positions from visible tracks.Independent point tracking cannot use this information.Source: §4.3 · p. 9, Table 3 · p. 9
• Using all four teacher models performs better than removing any teacher, showing that even weaker teachers provide complementary knowledge.Random teacher sampling outperforms aggregating teacher predictions by their mean or median.Source: §4.3 · p. 9, Table 5 · p. 9
• SIFT is consistently strong among point-sampling strategies, while the sampling choice does not significantly affect performance overall.Freezing the confidence and visibility head improves TAP-Vid AJ by +0.8 and OA by +3.9 on average.Source: §4.3 · p. 10, Table 6 · p. 10, Table 7 · p. 10

• CoTracker3 surpasses state-of-the-art trackers on TAP-Vid and other benchmarks while using 1,000× fewer videos than trackers trained on massive real-video collections.Joint tracking improves occlusion handling, particularly in the offline model.Source: §5 · p. 10

• Both model variants are pre-trained on synthetic TAP-Vid-Kubric, with online training using 64-frame videos and offline training using sequences of 30–60 frames.The online model uses 384 query points, while the offline model uses 512 points sampled uniformly in time.Source: A IMPLEMENTATION DETAILS · p. 14
• Real-video scaling uses caption-filtered Internet-like videos, a reduced learning rate, 15,000 iterations, and 384 SIFT-sampled tracks from eight random frames.Frame sampling is biased toward the beginning of each video.Source: A IMPLEMENTATION DETAILS · p. 14
• During TAP-Vid inference, additional grid-based support points provide context for online and scaled offline models, while the Kubric-trained offline model uses 1,000 uniformly sampled support points.Source: A IMPLEMENTATION DETAILS · p. 14
• The scaling pipeline samples query points, predicts tracks with student and teacher models, computes prediction differences, and updates the student.It is illustrated as a sequential pseudo-labeling process.Source: Figure 5 · p. 14

• Randomly sampling one teacher per batch yields better TAP-Vid δavg than supervising with the mean or median of all teachers’ predictions.Source: Table 8 · p. 15
• Repeatedly scaling CoTracker3 offline produces a slight additional improvement when the scaled model becomes one of the teachers.Source: Table 9 · p. 15
• For a fixed set of 15k videos, training longer trades increased computation against improved results, with the iteration count matched to the number of videos.Source: Table 10 · p. 15
• CoTracker3 is approximately 30% faster than LocoTrack while retaining linear time complexity as the number of tracked points increases.Proxy tokens preserve linear complexity despite group attention between tracks.Source: B PERFORMANCE · pp. 14–15, Figure 6 · p. 15

• Adding more teachers consistently improves the student model, but averaging their predictions instead of sampling a random teacher does not improve performance.Source: C ADDITIONAL EXPERIMENTS · p. 15

Repeated scaling

• Repeated scaling, where a newly trained student becomes a teacher in a second round, produces only slight metric improvements.This suggests the first scaling round already distilled most knowledge from the other teachers.Source: C ADDITIONAL EXPERIMENTS · p. 16

Convergence behavior during scaling

• Increasing dataset iterations improves TAP-Vid performance with diminishing returns until further iterations reach a saturation point.The reported training schedule uses a number of iterations equal to the number of training videos, with batch size 32.Source: C ADDITIONAL EXPERIMENTS · p. 16

• Further improvement requires stronger or more diverse teachers because student performance saturates after absorbing knowledge from the available teachers.Source: D LIMITATIONS · p. 16