PaperlayerView exact revision on arXiv

Source-linked AI summary

CoWTracker: Tracking by Warping instead of Correlation

Zihang Lai, Eldar Insafutdinov, Edgar Sucar, Andrea Vedaldi

arXiv:2602.04877v1cs.CV

TL;DR

Dense point tracking commonly relies on cost volumes whose computation scales quadratically with image resolution. CoWTracker replaces them with iterative warping and transformer refinement, achieving state-of-the-art tracking results and strong optical-flow performance with the same model. The results support a unified warping-based approach to tracking and optical flow.

  • Problem

    Cost-volume matching scales quadratically with image resolution, and the paper asks whether optical flow’s warping-based alternative can transfer to dense point tracking.

    Source: §1 · p. 2

  • Method

    CoWTracker iteratively warps target-frame features to the query frame using current displacement estimates and refines the tracks with a transformer.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    CoWTracker achieves state-of-the-art performance on several dense point tracking benchmarks and performs strongly on optical flow, sometimes outperforming specialized methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The warp-based head scales linearly with spatial resolution and transfers competitively to optical flow without flow-specific training.

    Source: §6 · p. 9

    The method may fail under extreme viewpoint changes, long-range full occlusions, or severe specularities.

    Source: §5 · p. 9

Abstract

from arXiv · show

Dense point tracking is a fundamental problem in computer vision, with applications ranging from video analysis to robotic manipulation. State-of-the-art trackers typically rely on cost volumes to match features across frames, but this approach incurs quadratic complexity in spatial resolution, limiting scalability and efficiency. In this paper, we propose \method, a novel dense point tracker that eschews cost volumes in favor of warping. Inspired by recent advances in optical flow, our approach iteratively refines track estimates by warping features from the target frame to the query frame based on the current estimate. Combined with a transformer architecture that performs joint spatiotemporal reasoning across all tracks, our design establishes long-range correspondences without computing feature correlations. Our model is simple and achieves state-of-the-art performance on standard dense point tracking benchmarks, including TAP-Vid-DAVIS, TAP-Vid-Kinetics, and Robo-TAP. Remarkably, the model also excels at optical flow, sometimes outperforming specialized methods on the Sintel, KITTI, and Spring benchmarks. These results suggest that warping-based architectures can unify dense point tracking and optical flow estimation.

1. Introduction

Cost volumes are central to recent point trackers but scale quadratically with image resolution, motivating a warping-based alternative. CoWTracker applies iterative warping and transformer reasoning to dense point tracking, with reported strength on tracking and optical-flow benchmarks.

1. Introduction

  • In the roller-coaster example, CoWTracker follows tracks near boundaries and through occlusions, while AllTracker struggles with thin structures and fails on the coaster’s front half.The figure starts from a grid of query points subsampled by 8×.Source: Figure 2 · p. 2
  • Matching each feature to every feature in another image makes cost-volume computation scale quadratically with image resolution.Recent optical-flow work showed cost volumes can be discarded at test time or replaced with warping.Source: §1 · p. 2
  • CoWTracker replaces cost volumes with iterative warping: it estimates dense tracks, warps features from other frames to a reference frame, then refines tracks.Its transformer uses separate self-attention along spatial and temporal dimensions.Source: §1 · p. 2
  • Although each warping step pairs a source location with one target location, transformer self-attention can still reason globally about correspondences.The paper connects this global reasoning to joint tracking, where self-attention operates across tracks.Source: §1 · p. 2
  • CoWTracker achieves state-of-the-art performance on several dense point tracking benchmarks and performs strongly on optical flow, sometimes outperforming specialized methods.The reported benchmarks include TAP-Vid-DAVIS, TAP-Vid-Kinetics, and Robo-TAP for tracking, and Sintel, KITTI, and Spring for optical flow.Source: §1 · p. 2

2. Related work

Prior point-tracking work spans correlation-based refinement, joint reasoning across tracks, transformer formulations, and increasingly dense or geometry-aware approaches. CoWTracker departs from the cost-volume paradigm used by most prior TAP approaches in favor of warping.

2. Related work

  • PIPs combines correlation maps inspired by RAFT with transformer-based refinement of track estimates.TAP-Vid formalized TAP and introduced TAP-Net; TAPIR combines global matching with local refinement, while PIPs++ targets longer-term tracking.Source: §2 · p. 3
  • CoTracker uses correlations between simultaneous tracks to improve robustness under occlusion and out-of-frame motion; DOT densifies its outputs.CoTracker3 simplifies the architecture and adds a data-efficient self-training regime.Source: §2 · p. 3
  • Most prior TAP approaches use cost volumes, whereas this work opts for a purely warping-based tracker.The related-work discussion describes the alternative as simple, efficient, and accurate.Source: §2 · p. 3
  • TAPTR treats points as decoder queries, while TAPTRv2 removes cost volumes for sparse tracking but is difficult to extend to dense tracking because each query outputs one displacement.Other approaches include 4D correlation features in LocoTrack and geometry-based track validation in VGGSfM and SceneTracker.Source: §2 · p. 3

3. Method

CoWTracker replaces cost-volume matching with iterative feature warping and transformer updates; the supplied benchmark caption reports best mean performance across four datasets and three metrics.

3. Method

  • The tracker combines a feature-producing backbone, a DPT upsampler, and a warping-based tracker to compute dense tracks.Source: §3 · p. 3

3.1. Problem Definition

  • For each query pixel, the tracker estimates its location in every target frame from its starting location in the query frame, and predicts visibility and confidence.Source: §3.1 · p. 3

3.2. CoWTracker Overview

  • CoWTracker avoids cost volumes by iteratively warping target-frame features using current displacements, then updating tracks with a transformer that alternates spatial and temporal attention.Track correlations help compensate for the absence of an explicit cost-volume search.Source: §3.2 · p. 3, §3.4 · p. 5

3.3. Features

  • The DPT upsampler raises feature resolution without constructing a correlation tensor or pyramid, supporting tracking of thin structures and near boundaries.Source: §3.3 · p. 4

3.4. Warping-only Tracker

  • CoWTracker achieves the best mean performance across four datasets and all three reported metrics, outperforming prior dense methods and strong sparse TAP baselines.The table caption specifies training only on Kubric data.Source: Table 1 · p. 5
  • The tracker head pairs cross-frame features only through warping, so its cost scales linearly with target count, feature resolution, and update iterations.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate CoWTracker on dense point tracking and optical flow, using the same model for both tasks. The model outperforms the strongest dense prior across point-tracking datasets and shows strong zero-shot flow results.

4.1. Implementation details

  • The model uses pretrained VGGT features, a DPT-style upsampler, and a tracker run for five iterations at stride 2.VGGT patch-embedding layers are frozen while the remaining backbone, upsampler, and tracker are finetuned; training uses Kubric data and visible- and occluded-track losses.Source: §4.1 · p. 5

4.2. Point Tracking

  • Point tracking is evaluated on TAP-Vid Kinetics, DAVIS, and RGB-Stacking, plus RoboTAP, using OA, δavg, and AJ.The datasets include in-the-wild videos, synthetic robotic sequences, and real-world robotic manipulation videos.Source: §4.2 · p. 5

4.2.2. Qualitative Results

  • During a full BMX-sequence occlusion, CoWTracker maintains a coherent track and re-locks onto the rider, unlike DELTA and AllTracker.The figure rows compare the methods, and lower-right boxes show frame numbers; CoWTracker also localizes thin structures after the occlusion.Source: Figure 4 · p. 6, §4.2.2 · p. 6

4.2.1. Quantitative Results

  • CoWTracker surpasses AllTracker across four point-tracking datasets and all three metrics.Average gains are 2.4 AJ, 1.3 δavg, and 1.8 OA; against AllTracker (Kub), controlled-data gains rise to 3.3 / 2.2 / 3.0 in AJ/δavg/OA, with OA gains of 3.0 over AllTracker (Kub) and 1.8 over AllTracker (Kub+Mix), reaching +4.3 on DAVIS.Source: §4.2.1 · p. 6

4.3.2. Qualitative Results

  • Sintel flow predictions closely match ground truth under large motion, occlusions, and clutter; refinement gains improve from K = 1 to K = 2 and then saturate around K = 5–6.The test-time iteration ablation reports modest additional gains from OA.Source: Figure 5 · p. 7, §4.3.2 · p. 7, Figure 7 · p. 8

4.3. Optical Flow

  • For optical-flow transfer, the same point-tracking model processes frame pairs without flow-specific training, tuning, or configuration changes.Evaluation covers Sintel, KITTI-2015, and Spring using standard protocols and metrics.Source: §4.3 · pp. 6–7

4.3.1. Quantitative Results

  • On Sintel, CoWTracker reaches 0.78/1.48 EPE on Clean/Final, improving over the next-best specialized model by 17%/20%; on KITTI-2015 it reaches 1.04 EPE and 4.87% Fl-all.Against WAFT twins-a2, KITTI-2015 EPE and Fl-all are reduced by 9.6% and 7.9%, respectively.Source: §4.3.1 · p. 7
  • CoWTracker has lower Sintel EPE than SEA-RAFT across motion bins, with a 0.16 versus 0.30 slope and 46% lower EPE in the highest-motion bin.On low-motion Spring, it achieves 0.17 EPE and 0.75% 1px, but the authors caution that small gaps near the noise scale limit conclusions from Spring alone.Source: §4.3.1 · p. 7

4.4. Ablation Studies

The ablations test backbone, feature indexing, temporal attention, iterative refinement, and head design. They favor stronger backbones, higher-resolution indexing, temporal reasoning, multiple refinement steps, and explicit warping.

Backbone choice

  • VGGT achieves the best δavg on all three datasets, improving over Pi3 by +1.7 δavg on average.The CoTracker ConvNet backbone performs clearly worse, while ViT and Pi3 work reasonably well under the same warping-only head.Source: §4.4 · p. 8

Upsampling for high-resolution indexing

  • The DPT upsampler performs best, gaining +5.5 δavg on DAVIS and +3.4 δavg on RoboTAP over using no upsampler.Removing upsampling hurts; bilinear and LoftUp-style cross-attention help, with LoftUp-style attention helping slightly more.Source: §4.4 · p. 8

Indexing resolution vs. patch size

  • At 1/2 stride, indexing reaches 78.0 δavg on DAVIS and 83.4 on RoboTAP, outperforming coarser 1/4–1/16 settings despite compute-matched patch sizes.The ablation varies indexing resolution while adjusting transformer patch size to roughly match compute.Source: §4.4 · p. 8

Spatial vs. spatio-temporal tracker network

  • Temporal attention improves δavg by 11.7 on RGB-Stacking and 11.2 on RoboTAP, both long-sequence settings of up to 600 frames.The comparison is between a spatial-only image transformer and the spatio-temporal tracker.Source: §4.4 · p. 8

Iterative refinement

  • Multiple refinement steps improve δavg by +6.6 on DAVIS and +4.0 on RoboTAP over a single-pass head.The comparison tests the iterative refinement scheme against a single-pass head.Source: §4.4 · p. 8

Head design

  • The non-warping head loses 23.4 δavg on DAVIS and 9.6 δavg on RoboTAP, while CoWTracker compares favorably with an AllTracker head on the same backbone.The authors hypothesize that direct backbone-feature use and higher operating resolution explain the comparison.Source: §4.4 · p. 8

Refinement steps

  • Increasing refinement steps consistently boosts performance, with the largest gain from K = 1 to K = 2 and diminishing returns thereafter.Metrics stabilize around K = 5, the adopted default.Source: §4.4 · p. 8

5. Limitations

CoWTracker has limitations in difficult visual conditions, throughput, refinement, and training-data scope.

Visual conditions

  • The method may fail under extreme viewpoint changes, long-range full occlusions, or severe specularities.These conditions mark a boundary on the settings addressed by the current method.Source: §5 · p. 9

Throughput and backbone

  • Overall throughput depends heavily on the chosen backbone, and VGGT’s quadratic video-length complexity requires longer clips to be processed in chunks.The iterative warping head itself is lightweight, with small marginal overhead.Source: §5 · p. 9

Refinement

  • Iterative refinement tends to saturate after five to six steps, indicating a ceiling on the model’s self-correction capability.Source: §5 · p. 9

Training data

  • The model is trained exclusively on synthetic Kubric data, while diverse real-world data remains underleveraged.The authors suggest natural videos could improve robustness to lighting and noise patterns that are difficult to simulate.Source: §5 · p. 9

6. Conclusions

CoWTracker replaces cost volumes with iterative warping-based refinement. It achieves state-of-the-art tracking on TAP-Vid and RoboTAP and transfers competitively to optical flow without flow-specific training.

6. Conclusions

  • CoWTracker replaces cost volumes with iterative warping-based refinement, scales linearly with spatial resolution, and attains state-of-the-art tracking on TAP-Vid and RoboTAP.The same model transfers competitively to optical flow without flow-specific training.Source: §6 · p. 9

7. Implementation Details

The implementation extracts frame features and initializes tracks and hidden state, then iteratively warps features and refines tracks without constructing a correlation or cost volume.

7.1. Pseudo-code of CoWTracker

  • At each of K iterations, the update loop warps features, concatenates original and warped features with tracks and hidden state, updates the hidden state, and refines tracks with a linear head.It begins with backbone features for all frames and replicated anchor-frame features for all queries; no correlation or cost volume is constructed.Source: §7.1 · p. 11

8. Extended Quantitative Results

CoWTracker leads the extended AllTracker benchmark suite; the section also examines its runtime and the memory trade-offs of cost-volume-free tracking.

8.1. Extended Evaluation on the AllTracker Benchmark Suite

  • The extended suite adds DriveTrack, EgoPoints, and Horse10, evaluated with AllTracker’s protocol and splits using δavg averaged over sequences.Source: §8.1 · p. 11
  • Figure 8 sketches an algorithm that iteratively refines tracks across frames through repeated warping and updates.Source: Figure 8 · p. 11
  • 73.6 average δavg gives CoWTracker the suite’s best overall result, +2.3 points over the strongest AllTracker, and it is best or tied-best on every dataset.The comparison covers point trackers and optical-flow baselines, including the three added benchmarks.Source: Table 4 · p. 11, §8.1 · p. 11

8.2. Runtime Analysis

  • For clips ≤40 frames, CoWTracker reaches around 6–7 million points per second and over 30 frames per second; longer videos reach around 3.5 million points per second and nearly 20 frames per second.Source: Table 5 · p. 12, §8.2 · p. 12
  • The tracker scales approximately linearly with video length, while the VGGT backbone’s quadratic growth drives long-sequence runtime; chunking trades about 1% in δ accuracy for a substantial runtime reduction.The backbone is described as essentially orthogonal to the tracker, so more efficient backbone designs would improve end-to-end speed.Source: §8.2 · p. 12

8.3. Memory Analysis

  • Cost-volume memory grows roughly 16× when image resolution doubles; CoWTracker’s warping head avoids cost volumes and runs on stride 1/2 features using only a few gigabytes.Stride 1/2 cost volumes would require hundreds of gigabytes in the stated setting, while prior dense trackers often use stride 1/8 and lose fine details.Source: Figure 9 · p. 12, §8.3 · p. 12

9. Extended Qualitative Results

The qualitative results show consistent tracking in challenging motion and across varied video domains. The same tracking weights also produce optical flow on datasets not used for optical-flow training.

9.1. Comparison to Existing Methods

  • CoWTracker maintains tracks through a diver’s non-rigid motion and appearance changes, and follows a small drone through out-of-frame motion and re-entry where DELTA and AllTracker lose or drift from targets.The authors attribute small-object tracking to high-resolution features preserving fine spatial details.Source: §9.1 · pp. 12–13, Figure 10 · p. 13

9.2. Generalization Across Domains

  • Without fine-tuning, CoWTracker tracks across egocentric, self-driving, and robotics videos, maintaining accurate tracks for up to 600 frames in egocentric and robotics examples.Source: §9.2 · p. 13, Figure 11 · p. 14

9.3. Zero-Shot Optical Flow Estimation

  • Using the same tracking weights without optical-flow training, CoWTracker produces flow on Spring and KITTI with sharp motion boundaries and alignment to ground truth.The displayed Spring and KITTI results extend the paper’s Sintel analysis.Source: §9.3 · pp. 13–14, Figure 12 · p. 14

Related papers

Loading 2602.04877v1…