Source-linked AI summary
Learning to Find Good Correspondences
Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, Pascal Fua
TL;DR
Wide-baseline stereo requires selecting reliable matches from noisy sparse correspondences while recovering relative camera pose. The paper learns this jointly with a compact MLP architecture using Context Normalization and a weighted eight-point formulation. It reports substantial state-of-the-art gains, including strong generalization with little training data, while requiring known camera intrinsics.
Problem
Wide-baseline pose estimation is limited by unreliable correspondences and sparse outlier rejection, while most existing methods use non-differentiable handcrafted techniques.
Method
A deep network assigns inlier-likelihood weights to putative correspondences, jointly classifies them and estimates the essential matrix through a weighted eight-point algorithm.
Results
The method outperforms the state of the art by a significant margin across challenging indoor and outdoor datasets, including with limited training data.
Takeaways & Limitations
Sparse, keypoint-based robust estimation can remain effective for wide-baseline stereo and generalize to unseen scenes with weak supervision.
Takeaways & Limitations
The solution requires known camera intrinsics, and its weak supervision does not perfectly identify correspondence labels.
Abstract
from arXiv · showhide
We develop a deep architecture to learn to find good correspondences for wide-baseline stereo. Given a set of putative sparse matches and the camera intrinsics, we train our network in an end-to-end fashion to label the correspondences as inliers or outliers, while simultaneously using them to recover the relative pose, as encoded by the essential matrix. Our architecture is based on a multi-layer perceptron operating on pixel coordinates rather than directly on the image, and is thus simple and small. We introduce a novel normalization technique, called Context Normalization, which allows us to process each data point separately while imbuing it with global information, and also makes the network invariant to the order of the correspondences. Our experiments on multiple challenging datasets demonstrate that our method is able to drastically improve the state of the art with little training data.
1. Introduction
Wide-baseline relative-pose estimation remains difficult because appearance changes and repetitive structures produce unreliable correspondences. The paper proposes a deep, correspondence-based network that jointly selects geometrically consistent matches and estimates camera motion.
- Wide baselines, repetitive structures, and illumination changes make relative camera-motion recovery difficult.
- Improving local feature matching does not necessarily improve pose estimation, making reliable inlier selection a key bottleneck.
- Sparse outlier rejection has received little attention, with most existing methods relying on non-differentiable handcrafted techniques.
- The proposed deep network jointly classifies correspondences as retain or reject and regresses camera motion using epipolar constraints.
- MLPs process correspondences independently, while Context Normalization injects global information and makes the network invariant to input order.
- The method reportedly doubles state-of-the-art performance, generalizes across indoor and outdoor datasets, and remains effective with only 59 training images.
2. Related Work
Prior correspondence pipelines rely on sparse matching, small-subset hypothesis generation, or dense image-based networks, each with important limitations. The paper positions its approach as a way to use global correspondence information for differentiable stereo outlier rejection.
- Traditional handcrafted methods: Traditional methods use sparse keypoints, reject outliers with RANSAC, and estimate the essential matrix with five- or eight-point algorithms.
- Traditional handcrafted methods: RANSAC remains the de facto standard despite improvements from methods such as MLESAC, PROSAC, USAC, and LMEDS.
- Traditional handcrafted methods: Small-subset hypothesis generation is sub-optimal when wide-baseline image pairs contain many outliers, because most hypotheses become useless.
- Learning-based methods: The architecture accepts all correspondences, processes each independently with shared Perceptrons, and uses Context Normalization to embed global information.
- Learning-based methods: Piece-wise smoothness assumptions can be violated in practice, limiting methods such as GMS despite improvements over traditional matching.
- Learning-based methods: Dense correspondence networks perform well on constrained video stereo but can be harmful with occlusions or large baselines.
- Learning-based methods: DSAC provides differentiable outlier rejection for monocular pose estimation but is difficult to extend to stereo and does not significantly surpass RANSAC.
- Learning-based methods: The proposed method differs from prior annotated homography-based work by requiring only essential matrices and working in real-world scenarios.
3. Method
The method learns geometrically consistent correspondences and camera motion jointly from sparse putative matches, using a weighted eight-point formulation and Context Normalization to incorporate global context while preserving permutation invariance.
- Problem formulation: The network assigns weights to putative correspondences, jointly classifies inliers and outliers, and regresses the essential matrix for relative camera motion.The weights determine each correspondence’s influence in a weighted reformulation of the eight-point algorithm.
- Problem formulation: Local features are matched by descriptor distance, after which descriptors are discarded and only normalized location quadruplets enter the network.Camera intrinsics normalize coordinates to [−1, 1] for better-conditioned optimization.
- Network architecture: Applying MLPs independently to correspondences makes the network invariant to input order, while Context Normalization frames each correspondence using the global distribution of the set.The architecture is inspired by PointNet but uses feature-map normalization rather than an explicitly concatenated global feature.
- Network architecture: Context Normalization normalizes feature maps across correspondences for each image pair, embedding scene geometry and camera motion into point-wise MLP features.Unlike standard normalization methods focused primarily on convergence, this operation supplies contextual information to correspondence-wise processing.
- Learning objective: Training uses a differentiable hybrid loss combining correspondence classification with essential-matrix regression from the weighted eight-point algorithm.The regression term compares the predicted essential matrix with the ground truth while accounting for the possible sign flip of the estimate.
- Test-time processing: At test time, RANSAC can further remove residual outliers among correspondences labeled as inliers by the network.The authors report that this performs much better than stand-alone RANSAC, supporting the use of learned filtering before sampling.
4. Experimental Results
The experiments evaluate the proposed correspondence-learning approach across datasets, optimization choices, Context Normalization, and post-processing. It outperforms prior methods across challenging outdoor and indoor scenes, generalizes to unseen scenes, and remains effective with limited training data.
- Datasets and evaluation: The evaluation covers outdoor YFCC100M collections, indoor SUN3D sequences, and additional small sequences used to test generalization.Training, validation, and test splits are formed for the evaluated sequences, with image pairs selected using reconstruction overlap or visibility.
- Ablation study: 12–24%: the hybrid approach combining correspondence classification and essential-matrix regression outperforms the other optimization variants on Sacre Coeur.Direct essential-matrix regression and Essential, which omit classification, perform worse when enough keypoints are available; the gap is larger at smaller error thresholds.
- Context Normalization: Context Normalization is necessary for strong performance, with the proposed approach using it performing much better than the same approach without it and better than PointNet.The comparison evaluates models with and without the final RANSAC stage.
- Efficiency: 17x faster: the network plus RANSAC processes 2k matches in substantially less time than a RANSAC loop using all matches, while returning about 300 inliers.A GPU forward pass takes 13 ms and RANSAC adds 9 ms, compared with 373 ms for the full RANSAC loop.
- Known scenes: More than twice the previous state-of-the-art mAP: the method achieves its strongest results on feature-rich outdoor collections, while also outperforming other methods with SIFT.The performance gap widens at stricter error thresholds.
- Generalization: A single model outperforms the state of the art on unseen outdoor and indoor scenes, using 2000 outdoor and 500 indoor training images, and can remain competitive with 59 images.Outdoor generalization is strong; indoor generalization is weaker, probably because of the capabilities of SIFT and LIFT in indoor scenes.
5. Conclusion
The paper presents a sparse, single-shot deep network for relative motion estimation that outperforms the state of the art with few images and limited supervision, while requiring known camera intrinsics.
- The proposed method recovers relative motion between two images with a sparse deep network in a single shot.
- The solution requires known camera intrinsics.
A.1. Dataset details
The appendix describes dataset splits and labels used for evaluation, including figures reporting same-sequence and cross-sequence results for Outdoors data.
- Dataset details: SUN3D uses nine sequences for training and fifteen previously used sequences only for testing.Images are split into disjoint training, validation, and test subsets in a 60-20-20 ratio.
- Dataset details: The appendix labels each dataset set from a-u, and the best model concatenates datasets marked ♦ and △.
- Dataset details: Figure 9 reports results for a model trained and tested on ‘Reichstag’.
- Dataset details: Figure 10 averages results for a model trained on ‘Reichstag’ and tested on the other ‘Outdoors’ sequences.The tested sequences are a-c, e, f, and ♦.
A.2. Training with limited data
Using only the 59-image ‘Reichstag’ training sequence, the method outperforms all baselines on same-sequence and cross-sequence evaluation.
- Training with limited data: 59 training images suffice for the ‘Reichstag’ experiment, with 8 validation and 8 testing images.Visibility constraints still yield over 1500 training image pairs and about 35 validation and testing pairs each.
- Training with limited data: The model outperforms all baselines when trained and tested on ‘Reichstag’ and when generalized to other ‘Outdoors’ sequences.LIFT plus RANSAC and GMS plus RANSAC are the closest competitors.
- Training with limited data: The best results use LIFT features, consistent with the paper’s previous observations.
A.3. Per-sequence results
Per-sequence evaluation shows that the proposed approach outperforms every baseline across Outdoors and Indoors testing subsets, with feature rankings differing by subset.
- Per-sequence results: The approach outperforms every baseline across the reported testing sequences.Outdoors datasets a-f are averaged in ∗, while Indoors datasets g-u are averaged in ‡.
- Per-sequence results: LIFT performs better than SIFT on the Outdoors subset, whereas SIFT performs better than LIFT on the Indoors subset.
- Per-sequence results: DeMoN performs well only on Outdoors sequences e and f and completely fails on YFCC100M sequences.
- Per-sequence results: G3DR performs even worse, suggesting that sparse methods are preferable for photo-tourism datasets.
A.4. Ransac post-processing
The section evaluates RANSAC as test-time post-processing and reports comparisons with the proposed method across outdoor and indoor sequences and multiple error thresholds.
- RANSAC post-processing improves performance and speed relative to the method without post-processing.The improvement is available only at test time because training requires differentiability.
- Results are shown at error thresholds of 5°, 10°, and 20°.
- Figure 11 reports results for every sequence in the Outdoors subset and the Indoors subset.The figure also marks average performance over each subset.
- Table 2 compares stand-alone RANSAC with the proposed method using either the 8-point algorithm or RANSAC for post-processing.Both SIFT and LIFT use 2k keypoints in this comparison.
A.5. Differentiating through eigendecomposition
The paper differentiates through eigendecomposition using TensorFlow's implementation, with the derivative expressed for a matrix decomposed into eigenvectors and eigenvalues.
- The implementation differentiates through eigendecomposition using TensorFlow.The paper refers readers to prior work for further details.
- For X = UΣU⊤, U is the matrix of eigenvectors and Σ is a diagonal matrix containing the eigenvalues.
- Msym is defined as 1/2(M⊤+ M), and σi denotes the i-th eigenvalue.