Source-linked AI summary
CoTracker: It is Better to Track Together
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht
TL;DR
Long-term point tracking must handle dependencies among points, occlusions, and points leaving the camera view, while many existing trackers treat points independently. CoTracker jointly tracks points with a transformer and proxy tokens in causal sliding windows, achieving strong benchmark performance and long-term tracking, though synthetic training limits generalization to complex scenes.
Problem
Existing point trackers usually treat points independently even though points can have strong dependencies, while integrating optical flow over time accumulates drift.
Method
CoTracker jointly tracks points with transformer attention, proxy tokens, overlapping causal windows, and unrolled recurrent training.
Results
CoTracker achieves state-of-the-art performance on standard tracking benchmarks and tracks points through occlusions, out of view, and across hundreds of frames.
Takeaways & Limitations
Joint and recurrent tracking supports long-term point tracking while proxy tokens enable tracking very large numbers of points simultaneously.
Takeaways & Limitations
Because CoTracker is trained purely on synthetic data, it sometimes generalizes poorly to complex scenes with reflections and shadows.
Abstract
from arXiv · showhide
We introduce CoTracker, a transformer-based model that tracks a large number of 2D points in long video sequences. Differently from most existing approaches that track points independently, CoTracker tracks them jointly, accounting for their dependencies. We show that joint tracking significantly improves tracking accuracy and robustness, and allows CoTracker to track occluded points and points outside of the camera view. We also introduce several innovations for this class of trackers, including using token proxies that significantly improve memory efficiency and allow CoTracker to track 70k points jointly and simultaneously at inference on a single GPU. CoTracker is an online algorithm that operates causally on short windows. However, it is trained utilizing unrolled windows as a recurrent network, maintaining tracks for long periods of time even when points are occluded or leave the field of view. Quantitatively, CoTracker substantially outperforms prior trackers on standard point-tracking benchmarks.
1 Introduction
CoTracker addresses the limitations of independent point tracking by jointly estimating many tracks and exploiting dependencies between points. Its transformer architecture, proxy tokens, and recurrent sliding-window training support accurate, scalable, long-term tracking.
- Motivation: Point tracking estimates point motion over extended periods, but conventional trackers usually treat sparse points as statistically independent despite dependencies between points.Optical flow instead estimates instantaneous velocities jointly for all points.
- Contribution: CoTracker jointly estimates a very large number of tracks using transformer attention between tracks, improving accuracy especially when points are occluded.Additional support points can expand the tracker's contextual information.
- Architecture: Proxy tokens replace expensive full self-attention between tracks with efficient cross-attention, reducing memory complexity and enabling near-dense tracking on one GPU.The proxies are processed like a small number of additional tracks.
- Long-term tracking: CoTracker operates online on overlapping short windows whose refined predictions are passed forward, while unrolled training optimizes this recurrent application.This design supports tracking through long occlusions.
- Joint tracking: Joint tracking produces better foreground and background tracks than non-joint tracking, which can make background points follow object motion.The comparison is illustrated in Fig. 2, with background points in cyan and foreground points in magenta.
- Evaluation: CoTracker is evaluated on TAP-Vid-DAVIS, TAP-Vid-RGB-Stacking, PointOdyssey, and DynamicReplica, achieving state-of-the-art performance on several benchmarks.The model works well for single points and excels for groups of points.
2 Related work
Prior work addresses instantaneous or short-horizon motion, but long-term point tracking remains distinct from optical flow and independent point trackers. CoTracker builds on these areas while targeting long-term, joint tracking of points in videos.
- Optical flow: Optical-flow models estimate dense instantaneous motion, whereas CoTracker applies related 4D cost volumes and iterative updates to point tracking.The distinction is between instantaneous motion estimation and tracking over longer horizons.
- Optical flow: Transformer-based optical-flow methods include Flowformer, which tokenizes 4D cost volumes, and GMFlow, which uses self-attention for refinement.These approaches remain part of the optical-flow literature rather than long-term point tracking.
- Long-term tracking: Applying optical flow to tracking by integrating predictions over time accumulates drift, motivating architectures designed for longer tracking horizons.CoTracker is presented as one such architecture.
- Multi-frame optical flow: Multi-frame optical-flow methods produce dense flow across several frames but are not designed for long-term tracking or points occluded for long periods.Earlier approaches also used Kalman filtering for temporal consistency.
- Visual object tracking: Visual object tracking has studied joint tracking and occlusion handling, but CoTracker focuses on points, including background points, rather than objects.The related work includes both handcrafted joint trackers and deep visual object trackers.
- Point tracking: PIPs++ addresses long-term tracking but still tracks points independently, while OmniMotion requires costly test-time optimization unsuitable for many practical online applications.PIPs introduced sliding-window tracking with restarts from the last visible frame but could not track beyond one window.
3 CoTracker
CoTracker formulates 2D point tracking as joint, iterative estimation over time and tracks, using transformer attention to exchange information among points. Sliding windows, proxy tracks, and recurrent unrolling make long-video tracking computationally tractable while supporting occlusions and out-of-view points.
- 3 CoTracker: CoTracker predicts 2D point trajectories and visibility flags throughout an RGB video from queried starting locations and times.Each track begins at its query time, and visibility indicates whether the point is visible or occluded in each valid frame.
- 3.1 Transformer formulation: Each transformer update progressively refines locations and appearance, while visibility is predicted once after the final update.The update adds ΔP and ΔQ to the current estimates; iterative visibility updates did not improve performance because accurate visibility requires an accurate location.
- 3.2 Transformer architecture and proxy tokens: Proxy tracks reduce track-attention cost to O(NK + K^2 + T^2) by letting regular tracks attend to K learned proxy tokens instead of one another.The proxies are concatenated at the transformer input and discarded at its output, with K much smaller than N.
- 3.3 Windowed inference and unrolled training: Overlapping windows pass refined predictions forward, enabling recurrent unrolling over arbitrarily long videos and tracking through long occlusions or outside the camera view.At test time, the windowed transformer can be unrolled beyond the number of windows used in training.
4 Experiments
CoTracker is evaluated on standard real and synthetic point-tracking benchmarks, with experiments examining joint tracking, recurrent training, proxy-token scalability, support points, and limitations. The results show stronger performance for occluded-point tracking, long-term propagation, and large-scale inference, while synthetic training limits generalization to complex scenes.
- Comparisons to the State of the Art: CoTracker is evaluated on TAP-Vid-DAVIS, TAP-Vid-RGB-Stacking, PointOdyssey, and Dynamic Replica after training on synthetic TAP-Vid-Kubric sequences.The benchmarks cover real and synthetic videos, causal evaluation protocols, and long-term sequences.
- Comparisons to the State of the Art: CoTracker’s occluded-point accuracy exceeds its visible-point advantage on Dynamic Replica, indicating stronger gains for invisible points.The paper attributes this strength to jointly tracking groups of points.
- Ablations: 62.4 →68.9 (+10.4%) avg accuracy results from joint tracking on Dynamic Replica when cross-track attention is retained.The ablation replaces cross-track attention with time-attention layers at comparable model size.
- Ablations: 18 AJ points are lost when unrolled training is disabled, showing its importance for tracking over periods longer than the training sequences.The reported long-term extension exceeds 10× the sequence length used in training.
- Ablations: ×7.4 more points fit within an 80 GB memory budget with proxy tokens, while inference becomes ×7 faster at the maximum tracked-point count.The system can track a 263 × 263 grid, with best results using 64 proxy tokens.
- Ablations: Local support points help substantially more than global points, while combining global and local context works best for camera and object motion.The authors associate local gains with dependencies among points on the same object.
- Limitations: CoTracker sometimes tracks shadows with their casting objects and does not always generalize to complex scenes containing reflections and shadows.This limitation follows from training exclusively on synthetic data and may be desirable for some applications but not others.
5 Conclusions
CoTracker jointly tracks several points while accounting for their dependencies. It achieves state-of-the-art performance on standard benchmarks, including long-term tracking through occlusions and outside the camera view.
- CoTracker tracks several points jointly while accounting for their dependencies.
- CoTracker achieves state-of-the-art performance on standard tracking benchmarks, often by a substantial margin.
- The tracker can maintain tracks through occlusions and points leaving the field of view, even for hundreds of frames.
- Its flexible, memory-efficient transformer architecture allows future integration of functionalities such as 3D reconstruction.
A Additional ablations
The ablations examine support-point configurations and sliding-window sizes. Local support grids improve results, while matched training and inference windows provide the best performance under the available training-sequence constraints.
- Support points: Adding a local grid yields significantly better results than adding points on a global regular grid.
- Support points: A configuration of 25 global and 64 local support points is optimal for TAP-Vid-DAVIS in terms of speed and accuracy.The selected configuration is used for evaluation on all TAP-Vid benchmarks.
- Support points: Differences across support-point hyperparameters affect performance by less than one percentage point for a wide variety of configurations.
- Training sliding window size: A training sliding window of 8 is optimal because Kubric training sequences contain only 24 frames, creating a trade-off between context and prediction propagation.
- Inference sliding window size: The model performs best when inference and training sliding-window sizes are identical.
B Efficiency
On a 50-frame 256×256 video, CoTracker is slower than TAPIR but faster than PIPs++, while achieving better accuracy than both. On TAP-Vid-Kinetics, its results are comparable to TAPIR.
- On a 50-frame 256×256 video, CoTracker is slower than TAPIR but faster than PIPs++, while achieving better accuracy than both.The comparison was run on an A40 GPU.
- On TAP-Vid-Kinetics, CoTracker and TAPIR achieve comparable results despite TAPIR being an offline method with a matching module.
C Evaluation on TAP-Vid-Kinetics
TAP-Vid-Kinetics contains approximately 250-frame videos, including discontinuous multi-shot videos that violate assumptions made by most trackers, including CoTracker. CoTracker and TAPIR therefore achieve comparable performance on this benchmark.
- TAP-Vid-Kinetics contains 1,144 videos of approximately 250 frames, and some are discontinuous combinations of video chunks.
- The discontinuous multi-shot videos do not satisfy CoTracker's continuous-video design assumptions.
- CoTracker and TAPIR achieve comparable results on this benchmark, although TAPIR includes a matching stage suited to combined video chunks.
D Implementation Details
CoTracker’s implementation uses a CNN feature extractor, sliding-window inference, iterative refinement, and recurrent-style training. The design balances tracking speed and accuracy while supporting occlusion augmentation and varied trajectories.
- Feature CNN: The feature CNN downsamples 384×512 frames by a factor of 4 and outputs 128-channel features.It uses residual blocks and convolutional layers, matching the CNN used in PIPs.
- Sliding windows: Sliding windows pass masks and visibility logits between overlapping windows to indicate where predictions are required.Points not yet visible in the current window are masked during cross-track attention.
- Efficiency: Efficiency is evaluated by processing 50-frame, 256 × 256 videos while varying the number of tracked points on an NVIDIA A40 48GB GPU.The reported measure is the time required to process the video.
- Iterative updates: CoTracker is trained with 4 iterative updates and evaluated with 6, providing a stated trade-off between speed and accuracy.Performance remains stable for 4–8 updates before slowly degrading.
- Training: Training samples 768 trajectories per batch for 50,000 iterations using AdamW and a 5e−4 learning rate.Batches contain 32 samples distributed across 32 GPUs.
- Augmentations: Training augmentations vary color, blur, occlusion appearance, and spatial scale to diversify the inputs.Occlusions are simulated with mean-colored or image-derived rectangular patches.
E Broader societal impact
Point tracking and optical flow are fundamental low-level computer-vision tasks that support many downstream applications. The paper states that point tracking has no direct societal impact, while indirect effects may arise through computer-vision applications.
- Applications: Motion estimation underlies computer-vision tasks including 3D reconstruction and video object segmentation.The passage describes it as an important component across many application areas.
- Scope: Point tracking is presented as a fundamental low-level computer-vision task.The passage places point tracking alongside optical flow as forms of motion estimation.
- Societal impact: The paper states that point tracking has no direct societal impact, though effects may materialize through downstream computer-vision applications.The passage does not characterize those indirect effects as positive or negative in this excerpt.
F Technical note
The technical note formalizes how neural-network layers map discrete tensor indices and continuous coordinates, including filter, interpolation, and sampling operations. These mappings allow features at transformed coordinates to be related back to the input image.
- Coordinate representation: A tensor element is interpreted as information about a continuous image coordinate, linking discrete indices to spatial locations.The tensor’s components are treated as unit tiles covering a continuous interval.
- Layer mappings: Neural-network layers establish affine mappings between indices and coordinates across tensors.The mapping is written as i1 = α12i2 + β12, with an equivalent coordinate mapping.
- Composed mappings: Chains of layers compose these mappings so coordinates in any layer can be referred back to the input image.The recurrence provides a direct input-referenced mapping through a sequence of layers.
- Sampling layers: Bilinear sampling reads values from an input tensor at arbitrary coordinates represented in a chosen coordinate range.The sampling formulation distinguishes normalized coordinates and alternative conventions for align_corners.
- Filter-like layers: For filter-like layers, an output pixel corresponds to the center of the input range used by the filter.The range is determined by filter size F, padding P, and stride S.
- Interpolation layers: Interpolation mappings depend on whether align_corners is enabled, changing how output extrema correspond to input pixels or coordinates.The note separately describes True and False conventions for interpolation and sampling.