Source-linked AI summary
COTR: Correspondence Transformer for Matching Across Images
Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, Kwang Moo Yi
TL;DR
Image correspondence must support both sparse keypoint matching and dense pixel mappings across challenging image changes. COTR uses a functional transformer architecture with recursive zoom-in inference, and is reported to perform at or above the state of the art across several correspondence datasets and tasks without retraining.
Problem
Image correspondence spans sparse and dense regimes, which traditionally use different priors and methods for applications such as stereo and optical flow.
Method
COTR predicts a correspondence x′ from a query point x and two images using a transformer that learns local and global priors, with recursive zoom-ins refining estimates.
Results
COTR achieves performance at or above the state of the art on HPatches, KITTI, ETH3D, and one IMC2020 scene across sparse and dense correspondence tasks.
Takeaways & Limitations
The functional architecture provides one correspondence framework that addresses both sparse and dense matching problems without retraining for a specific dataset.
Takeaways & Limitations
Image Matching Challenge comparisons lack exact GLU-Net+GOCor numbers because those results and implementation were unavailable at submission.
Abstract
from arXiv · showhide
We propose a novel framework for finding correspondences in images based on a deep neural network that, given two images and a query point in one of them, finds its correspondence in the other. By doing so, one has the option to query only the points of interest and retrieve sparse correspondences, or to query all points in an image and obtain dense mappings. Importantly, in order to capture both local and global priors, and to let our model relate between image regions using the most relevant among said priors, we realize our network using a transformer. At inference time, we apply our correspondence network by recursively zooming in around the estimates, yielding a multiscale pipeline able to provide highly-accurate correspondences. Our method significantly outperforms the state of the art on both sparse and dense correspondence problems on multiple datasets and tasks, ranging from wide-baseline stereo to optical flow, without any retraining for a specific dataset. We commit to releasing data, code, and all the tools necessary to train from scratch and ensure reproducibility.
1. Introduction
COTR bridges sparse and dense image correspondence by querying arbitrary points with a transformer that learns to use local and global priors. Recursive multiscale inference refines estimates, supporting accurate matching across varied motion and correspondence settings.
- Image correspondence supports applications including camera calibration, optical flow, Structure from Motion, visual localization, point tracking, and human pose estimation.
- Sparse methods match extracted keypoints for global camera motion, whereas dense methods map every pixel and exploit local smoothness for small temporal changes.Dense methods can also recover correspondences in texture-less areas by using contextual smoothness.
- COTR bridges sparse and dense matching with a network that learns to express and select global and local priors through transformer attention.The transformer can model discontinuities such as optical-flow changes across object boundaries while relating different image locations.
- COTR represents correspondence as x′ = FΦ(x | I, I′), allowing arbitrary query points and producing only as many matches as requested.Unlike dense methods, it learns smoothness implicitly and can handle large camera motion effectively.
- COTR is presented as the first transformer-based approach for accurate image correspondences.
- Recursive multiscale inference repeatedly zooms around estimates to compute highly accurate correspondences.The approach is presented as combining dense and sparse strengths, with transformer replacement by an MLP used to substantiate the design.
2. Related works
Prior correspondence systems are commonly divided into multistage sparse pipelines and dense optical-flow methods, while functional deep networks provide a related alternative. COTR is positioned as an end-to-end approach that avoids this division and targets both correspondence regimes.
- Sparse methods: Sparse correspondence pipelines typically detect keypoints, describe features, and match descriptors with ratio tests or robust outlier-resistant matchers.Robust matching may use RANSAC, consensus or motion heuristics, or learned methods.
- Dense methods: Dense methods generally target optical flow with small displacements, using local neighborhoods or global smoothness to estimate correspondences.Earlier methods for large baselines and appearance changes recovered depth and camera motion but performed below sparse methods.
- Comparison: COTR differs from sparse pipelines by learning feature extraction and matching end-to-end without dividing the process into multiple stages.
- Functional methods using deep learning: Functional deep-learning methods represent outputs as functions of query points, and prior work applied this idea to signed distance fields and incomplete-shape correspondences.That research reported state-of-the-art performance for functional methods, though it was not directly about image correspondence.
- The related-work discussion notes a concurrent relevant work for feature-less image matching that appeared shortly after COTR became public.
3. Method
COTR formulates correspondence as a queryable function implemented with a transformer that relates features across images. Recursive zoom-in inference refines estimates, while cycle consistency handles scale differences, invalid matches, and arbitrary image sizes.
- 3.1. Problem formulation: COTR represents correspondence as a parametric function that maps a query location in one image to its corresponding location in another.The training objective combines correspondence error with a cycle-consistency constraint.
- 3.2. Network architecture: A shared CNN extracts 16 × 16 × 256 features from each resized image, which are concatenated spatially and augmented with positional encoding.The resulting 16 × 32 × 256 context map is processed by the transformer.
- 3.2. Network architecture: The transformer encoder relates locations within and across images, while its decoder processes encoded query points and a fully connected layer predicts correspondences.Spatial concatenation enables self-attention within images and cross-attention between images without creating artificial same-location channel relationships.
- 3.2. Network architecture: Multiple query points can be processed simultaneously, with self-attention between queries disabled so their correspondence estimates remain independent.The decoder and final prediction layer translate the queries into multiple output coordinates.
- 3.3. Inference: At inference, COTR recursively applies the same network to zoomed-in crops, using a factor of two for four refinement steps to improve localization.This addresses the coarse localization caused by heavily downsampled feature maps, with a trade-off between computation and refinement depth.
- 3.3. Inference: Cycle-consistency filtering rejects correspondences with error greater than τ_cycle = 5 pixels, while scale compensation uses estimated co-visible regions.For arbitrary image sizes, the method initially resizes images to 256 × 256 and later crops and resizes patches around estimated points.
4. Results
COTR is evaluated without retraining across dense correspondence, optical flow, wide-baseline stereo, and object-centric scenes. It achieves strong accuracy across these settings, while interpolation, rejected queries, and task-specific input assumptions define important boundaries.
- Evaluation settings: COTR is evaluated without retraining on HPatches, KITTI, ETH3D, and IMC2020 across dense and sparse correspondence tasks.The evaluation covers planar matching, autonomous-driving scenes, increasing-baseline correspondence, and wide-baseline stereo pose estimation.
- HPatches: COTR provides the best HPatches results with and without interpolation except at PCK-1px, where it remains close to the best baseline.The authors caution that PCK-1px is affected by violations of the planar assumption in several scenes.
- KITTI: COTR outperforms KITTI baselines by a large margin, while its interpolated version is comparable to RAFT and slightly better than GLU-Net+GOCor.The reported KITTI metrics are AEPE and the flow outlier ratio on KITTI-2012 and KITTI-2015.
- KITTI: 81.8% of queried KITTI locations return confident results; among rejected queries, 67.8% fall outside the other image, while interpolation remains the main limitation.The authors attribute minor boundary errors to interpolation and suggest CNN-based interpolation as future work.
- KITTI: COTR recovers motion for objects moving in different directions, unlike GLU-Net’s bias toward a single global motion.Qualitative KITTI examples include nearby cars moving oppositely or partially occluding one another.
- ETH3D: COTR produces the most accurate ETH3D correspondences across settings and increasingly outperforms other methods as the frame interval, baseline, and difficulty increase.It ties LiteFlowNet at a three-frame difference and shows its strongest advantage at larger baselines.
- Object-centric scenes: COTR estimates dense correspondences between objects moving in different directions despite object-centric images differing substantially from its training data.The authors present this result as evidence of the approach’s generality.
5. Conclusions and future work
COTR combines a functional correspondence network with recursive inference to address both sparse and dense matching. It performs on par with or above state-of-the-art methods across several benchmarks and tasks.
- COTR introduces a functional network designed to address both sparse and dense image-matching problems.
- Recursive inference across multiple scales computes highly accurate correspondences.
- The zooming analysis shows that end pixel error decreases as more zoom-in is applied on HPatches.
- COTR achieves performance on par with or above the state of the art on HPatches, KITTI, ETH3D, and one IMC2020 scene.
Supplementary Material
The supplementary experiments compare positional-encoding choices through training and validation losses. Linear positional encoding converges stably, whereas log-linear encoding produces divergence and early termination.
- Log-linear positional encoding produces unstable training and validation losses for COTR.
- Training with log-linear positional encoding is terminated earlier because the loss diverges.
- Linear positional encoding yields training and validation losses that slowly converge to a stable status.
A. Compute
COTR’s recursive functional design and transformer architecture impose substantial computational demands. The prototype processes points individually, while batching and tiling are proposed to reduce inference cost.
- COTR has significant compute requirements because its functional, recursive design is coupled with a transformer.
- The nonoptimized prototype queries one point at a time and achieves 35 correspondences per second on an NVIDIA RTX 3090 GPU.
- Tiling and batching queries could reduce operations and substantially speed up inference without a significant preliminary performance drop.
B. Log-linear vs Linear
COTR’s positional-encoding experiments find linear frequency growth important for stable optimization. The commonly used log-linear strategy fails to converge, and the reason remains unresolved.
- Linear positional encoding is important because it enables COTR to converge successfully.
- COTR fails to converge when using the commonly used log-linear positional-encoding strategy.
- The authors suspect the difference reflects the correspondence task’s lack of very high-frequency components, but leave further investigation for future work.
C. Architectural details for COTR
COTR uses a ResNet50-based CNN backbone followed by transformer encoder and decoder components.
- ResNet50 layer3 features provide a 1024-channel map after the fourth downsampling step.A 1 × 1 convolution projects these features to 256 channels to reduce transformer computation.
- The transformer uses six encoder layers and six decoder layers.
- Each encoder layer contains an 8-head self-attention module.
D. Architectural details for the MLP variant
The MLP variant replaces COTR’s transformer processing with global image latent vectors and a coordinate-regression MLP.
- Max pooling extracts a global latent vector from each image’s 256-channel feature map.Without global pooling, a fully connected alternative quickly produced degenerate estimates at the image centre.
- A 3-layer MLP regresses correspondence coordinates from concatenated image latents and the positional-encoded query point.Its input is 768-dimensional, and each linear layer has 256 neurons followed by ReLU activations.
E. Comparing with RAFT [65]
The comparison evaluates RAFT across multiple datasets, with COTR remaining competitive on KITTI and outperforming RAFT on the other reported datasets.
- The comparison table covers KITTI 2012, KITTI 2015, and HPatches.
- The comparison table also includes the Image Matching Challenge.
- On KITTI, sparse COTR performs best, while COTR with interpolation is roughly on par with RAFT.
- On other datasets, COTR outperforms RAFT by a large margin.
- RAFT requires two input images of the same size and is evaluated after resizing HPatches and Image Matching Challenge images to 1024×1024.