Source-linked AI summary
AllTracker: Efficient Dense Point Tracking at High Resolution
Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, Suya You, Rares Ambrus, Katerina Fragkiadaki, Leonidas J. Guibas
TL;DR
Long-range point tracking must handle temporal changes, occlusions, and dense high-resolution correspondence without sacrificing spatial or temporal awareness. AllTracker formulates the task as multi-frame optical flow, using recurrent low-resolution refinement with spatial convolutions and temporal attention; it achieves state-of-the-art dense high-resolution tracking, including 66.1 average accuracy versus 65.0 for the closest competitor. The authors also report limitations in short-range optical-flow estimation and temporal-window size.
Problem
Long-range point tracking remains difficult because chained optical flow accumulates drift and occlusions, while direct long-range correspondence faces increasing variation in perspective, illumination, and scene geometry.
Method
AllTracker casts point tracking as multi-frame long-range optical flow, combining recurrent low-resolution refinement, spatial convolutions, pixel-aligned temporal attention, and joint optical-flow and point-tracking supervision.
Results
66.1 versus 65.0 average accuracy: AllTracker outperforms other models on average across the reported point-tracking benchmarks and produces dense tracking at high resolution.
Takeaways & Limitations
AllTracker makes sparse point trackers of similar speed and accuracy redundant while retaining dense high-resolution output.
Takeaways & Limitations
The model appears to underfit short-range motion estimation, and its temporal window size may remain constrained by available GPU memory.
Abstract
from arXiv · showhide
We introduce AllTracker: a model that estimates long-range point tracks by way of estimating the flow field between a query frame and every other frame of a video. Unlike existing point tracking methods, our approach delivers high-resolution and dense (all-pixel) correspondence fields, which can be visualized as flow maps. Unlike existing optical flow methods, our approach corresponds one frame to hundreds of subsequent frames, rather than just the next frame. We develop a new architecture for this task, blending techniques from existing work in optical flow and point tracking: the model performs iterative inference on low-resolution grids of correspondence estimates, propagating information spatially via 2D convolution layers, and propagating information temporally via pixel-aligned attention layers. The model is fast and parameter-efficient (16 million parameters), and delivers state-of-the-art point tracking accuracy at high resolution (i.e., tracking 768x1024 pixels, on a 40G GPU). A benefit of our design is that we can train jointly on optical flow datasets and point tracking datasets, and we find that doing so is crucial for top performance. We provide an extensive ablation study on our architecture details and training recipe, making it clear which details matter most. Our code and model weights are available at https://alltracker.github.io
1. Introduction
AllTracker addresses the difficulty of converting optical flow into long-range, dense point tracks by estimating flow from a query frame to every other frame. Its architecture combines temporal priors, spatial awareness, and joint training across optical-flow and point-tracking datasets.
- Long-range point tracking is difficult because chained flow accumulates drift and must handle occlusions, while direct long-range flow faces changing perspective, illumination, and scene geometry.
- Recent point trackers improve temporal reasoning and occlusion handling but trade away spatial awareness, sparse output, or high-resolution accuracy.
- AllTracker casts point tracking as multi-frame long-range optical flow, combining learnable temporal priors with high-resolution spatial awareness.
- The model combines recurrent iterative refinement, spatial cross-correlations, per-pixel temporal modules, low-resolution processing, and final upsampling.
- Joint training on optical-flow and point-tracking datasets, together with a long training schedule, is reported as crucial for top performance.
- The resulting tracker operates at high resolution and full density, providing a state-of-the-art point-tracking model.
2. Related Work
Related work spans iterative optical-flow estimation, flow-based tracking, and point trackers that add temporal context without dense spatial output. AllTracker builds on these directions while using multi-frame dense flow and joint supervision across relevant datasets.
- Optical flow: Optical-flow methods estimate motion between consecutive frames through iterative correspondence refinement using appearance costs and smoothness or learned priors.
- Optical flow: SEA-RAFT represents the optical-flow state of the art described here, refining low-resolution flows with correlations and 2D convolutions before full-resolution upsampling.
- Flow-based point tracking: Flow-based trackers commonly chain flow vectors, while newer methods use multistep flows or learned temporal priors to address drift and occlusions.
- Flow-based point tracking: DTF and DELTA also relate a reference frame to other frames, but approximate global spatial message passing with attention to sparse anchor or centroid tokens.
- Point trackers without flow: Point trackers without flow use temporal windows, multipoint context, improved initialization, wider correlations, transformer designs, or post-hoc densification.
- Training data and self-supervision: Training practice combines mostly synthetic optical-flow and point-tracking datasets, while AllTracker avoids pseudo-labeling and adds synthetic data through flow supervision.
- AllTracker: AllTracker’s architecture computes query-aligned multi-scale cost volumes, recurrently refines correspondence fields, and upsamples them into high-resolution flow maps.
3. AllTracker
AllTracker processes video subsequences around a selected query frame, iteratively refining low-resolution correspondence, visibility, and confidence estimates before producing dense full-resolution outputs. Spatial convolutions and pixel-aligned temporal attention propagate information across the window, with supervision supporting both dense flow and sparse tracks.
- The model takes a video and query-frame index, outputting full-timespan tracking for every pixel in the query frame.
- Its output has shape T, H, W, 4, containing two flow-offset channels plus visibility and confidence estimates.
- AllTracker processes sliding windows of length S, initializes low-resolution outputs, iteratively revises them, upsamples to full resolution, and reuses overlapping estimates.
- The encoder compresses each subsequence from S, H, W, 3 to S, H/8, W/8, D, with D = 256 and S = 16 in the described setup.
- The query feature map is tiled across the subsequence, and multi-scale correlations provide appearance-based correspondence cues.
- The iterative refinement stage updates tracking coordinates, visibility, and confidence from motion, feature, and correlation representations.
- Interleaved 2D ConvNeXt blocks and pixel-aligned temporal transformer blocks propagate information spatially and temporally across corresponding query pixels.
- The model applies additive revisions, performs pixel-shuffle upsampling, and shares the recurrent refinement weights across four iterations.
4. Experiments
AllTracker is evaluated across diverse point-tracking benchmarks, resolutions, efficiency settings, and ablations. It achieves strong accuracy while combining dense high-resolution tracking with optical-flow-like speed and memory efficiency.
- Metrics and benchmarks: δavg averages δk over k ∈{1, 2, 4, 8, 16} and measures trajectory accuracy, with higher values indicating closer agreement with ground truth.The metric has a maximum value of 100 and is prioritized for interpretability and robustness to outliers.
- Metrics and benchmarks: Nine benchmarks cover animals, YouTube videos, surveillance, egocentric recordings, and robotics, with videos trimmed to at most 600 frames.The evaluation uses point annotations across BADJA, Horse10, TAP-Vid, CroHD, EgoPoints, RGB-Stacking, and RoboTAP.
- Main results: 66.1 vs. 65.0 δavg: full AllTracker outperforms the closest CoTracker3 variant on average across the evaluated datasets.CoTracker3 wins on CroHD, Davis, and DriveTrack, while AllTracker wins on the remaining datasets and overall.
- High-resolution performance: At 768 × 1024, AllTracker-Tiny outperforms CoTracker3, while AllTracker processes 786,432 tracks on a 40G A100 GPU.CoTracker3 encountered out-of-memory errors until evaluation was moved to a 96G H100 GPU, despite tracking sparse points.
- Speed: 57.9 FPS and 62.6 δavg: streaming AllTracker runs in real time at 512 × 512, compared with 21.4 FPS and 62.2 δavg for BootsTAPIR.The streaming sliding-window strategy incurs an accuracy penalty relative to the model’s normal 66.1 δavg.
- Ablations: A transformer-based temporal module outperforms mixer- and convolution-based alternatives, while pre-trained ConvNeXt and the selected refinement and correlation settings perform best in ablations.The reported configuration uses 3 refinement blocks and radius-4 correlations at 5 scales; L1 also outperforms Huber in the tested setup, 56.8 vs. 54.9.
5. Conclusion and Limitations
AllTracker treats point tracking as multi-frame optical flow, achieving state-of-the-art dense, high-resolution point tracking while retaining limitations in short-range optical flow and temporal coverage.
- AllTracker treats point tracking as multi-frame optical flow and produces dense output at high resolution.
- AllTracker delivers state-of-the-art performance on point tracking benchmarks and makes sparse point trackers of similar speed and accuracy redundant.
- AllTracker does not outperform state-of-the-art optical flow methods on optical flow estimation, suggesting underfitting on short-range motion.
- Larger temporal windows may help resolve longer occlusions, while future work could add physical and common-sense motion constraints.
A. Additional model details
The model’s refinement architecture combines convolutional processing for spatial information with pixel-aligned temporal processing, using recurrent iterative updates and memory-saving design choices.
- Recurrent module: The recurrent module processes correlation and motion fields with parallel 2-layer CNNs, concatenates their features, and uses a reduced hidden state.The hidden-state split reduces output channels from 260 to 132 and may stabilize recurrence.
- Model layers: The architecture uses ConvNeXt blocks for feature processing and 8-head pixel-aligned transformer blocks for temporal modeling.
- Iterative refinement: The refinement block consolidates visibility, confidence, correlation, motion, and appearance features before updating features, visibility, confidence, and motion.This refinement process is iterated 4 times with shared weights.
- Unlike SEA-RAFT, the model omits direct regression of an initial optical-flow estimate primarily to save memory.The paper also notes that large frame gaps can make flow estimation impossible without temporal context.
B. Additional training details
Additional training details describe mixed-precision implementation, resource-controlled settings, joint handling of optical-flow and point-tracking data, and the evaluation protocol used for comparison.
- The model is trained with mixed precision in PyTorch using bfloat16.
- Training uses dense supervision from optical-flow datasets and between 256 and 6144 valid tracks from point-tracking samples.Optical-flow data does not include visibility labels.
- Batch size, video length, input resolution, and refinement steps are adjusted to control speed and memory within an 8x A100 40G budget.For 24-frame videos, the setting is batch size 1, resolution 384 × 512, and 4 refinement steps; for 56-frame videos, it is resolution 256 × 384 and 3 steps.
- The CoTracker3 comparison evaluates δavg across nine benchmarks at 384 × 512, with parameter counts reported in millions.
- The dense correspondence visualizations compare ground-truth trajectories with estimated first-to-last-frame correspondence fields from different models.RAFT and SEA-RAFT use only the first and last frames, unlike methods using broader temporal information.
- The training configuration splits eight GPUs between 24-frame and 56-frame videos during the first Kubric-only stage.
- Applying sigmoid before direct BCE loss was found to be more numerically stable than BCE with logits.
- No architecture modifications are required for joint optical-flow and point-tracking training, although temporal attention is redundant for optical flow.The temporal transformer remains enabled because its MLP layers still participate in processing.
C. Additional baseline details
The additional baseline details identify evaluation metrics and describe how optical-flow and point-tracking methods differ in their experimental behavior and query handling.
- CoTracker-style performance depends on how query points are grouped, with same-object query groups tending to be tracked more accurately.
- Table 40 reports optical-flow end-point error on the official SINTEL test benchmark using the EPE-All metric.
- Table 37 reports optical-flow end-point error for CVO Final and Extended test sets, separately for visible and occluded pixels.
D. Additional optical flow results
Additional evaluations show that AllTracker performs best on long CVO sequences and runs faster than DOT, but does not achieve state-of-the-art optical flow accuracy.
- AllTracker’s optical flow is less accurate than SEA-RAFT but comparable to RAFT or GMFlow on SINTEL.
- On long CVO sequences, AllTracker performs best, while DOT performs best on short sequences.
- AllTracker is 3x the speed of DOT on CVO.
- Attaining top optical-flow and point-tracking performance with one model remains an open challenge.
E. Additional ablation details
The ablation validation setup uses subsampled, truncated datasets for rapid experimentation, while four recurrent refinement iterations are used in the main evaluation.
- Validation dataset: The validation dataset combines BADJA, CroHD, TAPVid-Davis, DriveTrack, Horse10, and RoboTAP for ablation studies.
- Validation dataset: The ablation studies subsample the validation datasets and exclude some available data to obtain a quick but reliable performance estimate.
- Inference steps: 4 recurrent refinement iterations are used in the main evaluation, after accuracy rises quickly and then plateaus.
- Validation dataset: A truncated regime uses videos of at most 300 frames and 100,000 training steps on Kubric, allowing most ablations to finish within 24 hours.
F. Additional qualitative results
Long-range flow estimates are obtained by tracking every first-frame pixel with point-tracker baselines, while qualitative visualizations reveal distinct strengths and failures across methods.
- Evaluation setup: Point-tracker baselines estimate long-range flow by querying every pixel in the video’s first frame.
- Evaluation setup: Optical-flow baselines estimate long-range flow by pairing the first frame with each subsequent frame, producing T −1 frame pairs.
- Qualitative findings: When foreground displacement is large, flow models often describe only the background instead of the dynamic foreground.
- Qualitative findings: PIPs++ and LocoTrack often lack spatial smoothness, whereas flow models generally preserve it; CoTracker3 fails occasionally.
- Qualitative findings: AllTracker’s outputs appear smooth and accurate, consistent with combining 2D flow processing and point-tracker temporal coherence.