Source-linked AI summary
Learning Feature Descriptors using Camera Pose Supervision
Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, Noah Snavely
TL;DR
Learned descriptor training typically requires pixel-level correspondences that are difficult to obtain at scale. This paper learns descriptors from relative camera poses using epipolar-constraint losses and a differentiable architecture, achieving state-of-the-art performance across geometric tasks while matching or outperforming fully supervised methods.
Problem
Learned descriptors improve correspondence estimation but may generalize poorly to unseen real-world scenarios, while pixel-level correspondence annotations are difficult to acquire at scale.
Method
The framework learns descriptors solely from relative camera poses using losses based on epipolar constraints and an efficient architecture that makes correspondence differentiable.
Results
The method achieves state-of-the-art performance across geometric tasks and outperforms fully supervised counterparts without correspondence annotations during training.
Takeaways & Limitations
Camera poses alone can suffice for learning strong descriptors, while the resulting dense descriptors can support downstream tasks with existing keypoint detectors.
Takeaways & Limitations
Synthetic homography-warped image pairs do not capture the full range of geometric and photometric variations observed in real images.
Abstract
from arXiv · showhide
Recent research on learned visual descriptors has shown promising improvements in correspondence estimation, a key component of many 3D vision tasks. However, existing descriptor learning frameworks typically require ground-truth correspondences between feature points for training, which are challenging to acquire at scale. In this paper we propose a novel weakly-supervised framework that can learn feature descriptors solely from relative camera poses between images. To do so, we devise both a new loss function that exploits the epipolar constraint given by camera poses, and a new model architecture that makes the whole pipeline differentiable and efficient. Because we no longer need pixel-level ground-truth correspondences, our framework opens up the possibility of training on much larger and more diverse datasets for better and unbiased descriptors. We call the resulting descriptors CAmera Pose Supervised, or CAPS, descriptors. Though trained with weak supervision, CAPS descriptors outperform even prior fully-supervised descriptors and achieve state-of-the-art performance on a variety of geometric tasks. Project Page: https://qianqianwang68.github.io/CAPS/
1 Introduction
CAPS addresses limited and potentially biased correspondence training data by learning descriptors from relative camera poses, using epipolar constraints and a differentiable, efficient architecture. The resulting descriptors support strong geometric-task performance despite weak supervision.
- Motivation: Learned descriptors can generalize poorly to unseen real-world scenarios, while accurate dense correspondence data are difficult to collect at scale.Existing SfM alternatives provide sparse, potentially keypoint-biased correspondences, and homography-warped pairs miss real-world geometric and photometric variation.
- Approach: CAPS relaxes the requirement for pixel-level ground-truth correspondences by learning descriptors solely from relative camera poses.Relative poses can come from sensors such as IMUs and GPS or be estimated using SfM pipelines.
- Approach: Relative camera poses are translated into epipolar constraints that supervise predicted pixel correspondences.The framework addresses the inability of standard triplet and contrastive losses to use camera poses directly.
- Results: CAPS descriptors perform on par with or better than prior fully supervised state-of-the-art methods across a range of geometric tasks.The trained system generates dense descriptors for arbitrary input images and can be combined with existing keypoint detectors.
- Approach: A novel loss function and efficient network architecture make correspondence locations differentiable with respect to descriptors during training.The architecture includes a differentiable matching pipeline designed for efficient learning.
2 Related Work
Prior descriptor learning largely relies on metric-learning objectives, while correspondence methods use differentiable matching in narrower search settings. CAPS instead combines epipolar supervision with differentiable dense matching for wide-baseline descriptor learning.
- Descriptor Learning: Descriptor-learning methods commonly use pairwise, triplet, or structured losses to separate matching and non-matching points in feature space.Existing approaches include both patch-based and dense descriptor methods.
- Correspondence Learning: Differentiable correlation and cost-volume layers support stereo and optical-flow correspondence, but those tasks search along scanlines or within local patches.Wide-baseline matching requires searching over the whole image, motivating an efficient coarse-to-fine architecture.
- Epipolar Constraint: Prior epipolar approaches either target semantic keypoints or generate pseudoground-truth correspondences through a non-differentiable process.The paper positions CAPS as enabling differentiable dense descriptor learning with epipolar constraints.
- Epipolar Constraint: CAPS uses epipolar loss to measure predicted-correspondence distance from the ground-truth epipolar line and cycle consistency loss to constrain forward-backward matches.These losses are illustrated for a query point, its predicted correspondence, and the returned point after forward-backward matching.
3 Method
The method learns dense descriptors from camera-pose-derived epipolar constraints and cycle consistency, using differentiable matching so correspondence locations support end-to-end training. A coarse-to-fine architecture improves efficiency while preserving high-resolution descriptors, and uncertainty weighting reduces the influence of unreliable points.
- 3.1 Loss Formulation: Relative camera poses are converted into epipolar constraints, and predicted matches are trained with complementary epipolar and cycle consistency losses.The epipolar term places predictions on the appropriate epipolar line, while cycle consistency encourages forward-backward mappings to return near the query point and suppresses other line-consistent outputs.
- 3.2 Differentiable Matching Layer: A differentiable matching layer correlates dense descriptors, converts the resulting 2D softmax distribution into a correspondence expectation, and enables end-to-end training.Unlike nearest-neighbor matching, the expected correspondence location is differentiable with respect to the descriptor-producing network.
- 3.2 Differentiable Matching Layer: Uncertainty weighting downweights infeasible or non-discriminative query points whose missing correspondences could otherwise provide incorrect training signals.Uncertainty is derived from the total variance, the trace of the covariance matrix of the predicted correspondence distribution, rather than from extra network parameters.
- 3.3 Coarse-to-Fine Architecture: The coarse-to-fine architecture searches globally at coarse resolution, restricts fine matching to a local window, and applies losses at both levels.This reduces the cost of full-map correlation while training coarse and fine features simultaneously.
- 3.3 Coarse-to-Fine Architecture: Concatenated coarse- and fine-level descriptors form hierarchical representations that capture abstract and detailed information, while coarse-to-fine reasoning improves efficiency and matching accuracy.The paper reports that the architecture preserves descriptor resolution without evaluating full correlation between large feature maps.
- 3.4 Discussion: The epipolar constraint provides sufficient supervision by eliminating many geometrically invalid matches, while true correspondences are more likely to have similar feature encodings.Camera-pose-trained descriptors outperform prior fully supervised methods, and correspondence-supervised training performs even better.
4 Experimental Results
CAPS is evaluated on sparse and dense feature matching, homography and relative pose estimation, and 3D reconstruction. Across these tasks, CAPS generally performs competitively or better than established descriptors, while ablations support the roles of epipolar supervision and coarse-to-fine architecture.
- Feature Matching Results: CAPS achieves the best overall sparse matching performance with SuperPoint keypoints and competitive performance with SIFT keypoints on HPatches.With the same detectors, CAPS improves over SIFT + ContextDesc and SuperPoint.
- Feature Matching Results: CAPS achieves the overall best dense matching performance on HPatches, except against R2D2 at small thresholds of ≤4px.The comparison uses mean PCK; CAPS is evaluated with 4x downsampled descriptor maps, while the cited R2D2 implementation uses full-resolution maps.
- Downstream Tasks: CAPS improves homography estimation over SIFT and SuperPoint descriptors, achieving the overall best performance with SuperPoint keypoints without annotated correspondence training.Accuracy is evaluated using corner correctness at 1, 3, and 5 pixel thresholds.
- Downstream Tasks: CAPS improves relative pose estimation over SIFT and SuperPoint, although R2D2 outperforms CAPS with SuperPoint keypoints on ScanNet.Relative pose is estimated from mutual nearest-neighbor matches through essential-matrix estimation and decomposition.
- Downstream Tasks: CAPS provides competitive SfM performance, matching or exceeding baselines in reconstruction completeness but not achieving the lowest reprojection error.The paper attributes this pattern to a trade-off between reconstruction completeness and reprojection error.
- Ablation Analysis: Ablations show that epipolar loss and the coarse-to-fine architecture are important: cycle consistency alone fails, while coarse-to-fine substantially improves performance.The architecture may reduce multimodal matching issues through local-window correspondence and capture hierarchical global and local information.
5 Conclusion
The framework learns descriptors from camera pose supervision alone, using epipolar-constraint losses and an efficient differentiable architecture. It achieves state-of-the-art performance across geometric tasks while outperforming fully supervised counterparts, and motivates further work on transformation invariance and combined losses.
- Camera pose supervision alone trains the proposed descriptor-learning framework without correspondence annotations.
- Epipolar-constraint losses and an efficient architecture make correspondence differentiable for learning.
- State-of-the-art performance across geometric tasks surpasses fully supervised counterparts without correspondence annotations during training.
- Future work targets improved invariance to large geometric transformations and investigates combining pose supervision with traditional metric-learning losses.