PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Find Good Correspondences

Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, Pascal Fua

arXiv:1711.05971v2cs.CV

TL;DR

Selecting reliable matches remains a challenge in wide-baseline stereo, and better point matching does not necessarily improve pose estimation. This paper jointly classifies putative correspondences and estimates relative pose using a sparse network with Context Normalization, and reports state-of-the-art performance with little training data.

  • Problem

    Improvements on point-matching benchmarks do not necessarily improve pose estimation, leaving the selection of the most useful correspondences as a research challenge.

    Source: §1 · p. 1

  • Method

    The method uses a deep network to classify putative correspondences and regress camera motion jointly, with Context Normalization providing global context across order-invariant correspondence processing.

    Source: §1 · pp. 1–2

  • Results

    A single model outperforms current methods on drastically different indoor and outdoor datasets, with state-of-the-art outdoor performance reported using only 59 training images.

    Source: §1 · p. 2, §4.5.2 · p. 8

  • Takeaways & Limitations

    The conclusion is that sparse, keypoint-based robust estimation can remain relevant alongside dense, black-box formulations.

    Source: §5 · p. 8

    The method requires known camera intrinsics.

    Source: §5 · p. 8

Abstract

from arXiv · show

We develop a deep architecture to learn to find good correspondences for wide-baseline stereo. Given a set of putative sparse matches and the camera intrinsics, we train our network in an end-to-end fashion to label the correspondences as inliers or outliers, while simultaneously using them to recover the relative pose, as encoded by the essential matrix. Our architecture is based on a multi-layer perceptron operating on pixel coordinates rather than directly on the image, and is thus simple and small. We introduce a novel normalization technique, called Context Normalization, which allows us to process each data point separately while imbuing it with global information, and also makes the network invariant to the order of the correspondences. Our experiments on multiple challenging datasets demonstrate that our method is able to drastically improve the state of the art with little training data.

1. Introduction

The paper addresses wide-baseline pose recovery by learning to select geometrically consistent correspondences and estimate camera motion jointly. Its architecture uses correspondence-wise MLPs and Context Normalization to combine per-match processing with information from the full match set.

Introduction

  • Wide baselines, repetitive structures, and illumination changes make relative-pose recovery difficult, and better point-matching benchmarks do not necessarily improve pose estimation.This motivates selecting the correspondences most useful for pose recovery rather than focusing only on establishing matches.Source: §1 · p. 1
  • DSAC [4] is described as the only recent differentiable approach to sparse outlier rejection, but it mimics RANSAC and targets 3D-to-2D rather than stereo correspondences.Most algorithms still rely on non-differentiable handcrafted techniques for this problem.Source: §1 · p. 1
  • The network jointly classifies correspondences as retain or reject and regresses camera motion using epipolar constraints and a weighted reformulation of the 8-point algorithm.It assumes known camera intrinsics and represents camera motion with the essential matrix.Source: §1 · pp. 1–2
  • Context Normalization normalizes feature-map distributions across correspondences at each perceptron, giving each match global context while preserving order invariance.The paper reports that this non-parametric operation was simpler and more effective for its purposes than PointNet’s global context features [21].Source: §1 · p. 2
  • The method can double state-of-the-art performance and still outperform current methods on challenging outdoor scenes with only 59 training images.It also requires only weak supervision through essential matrices.Source: §1 · p. 2
  • A single keypoint-based model generalizes to unseen scenes and outperforms current methods on drastically different indoor and outdoor datasets.The paper contrasts this generalization with image-based dense methods.Source: §1 · p. 2

2. Related Work

Related work covers sparse keypoint-and-RANSAC pipelines and learned correspondence methods, whose limitations motivate approaches that use broader correspondence information for pose estimation.

Traditional handcrafted methods

  • Traditional pipelines use sparse keypoints such as SIFT, reject outliers with RANSAC, and recover the essential matrix with 5-point [20] or 8-point [18] algorithms.MLESAC, PROSAC, USAC, and LMEDS address outlier rejection, but RANSAC remains the de facto standard.Source: §2 · p. 2
  • Small-subset hypothesis generation, such as the 5-point algorithm's use of five correspondences, makes most hypotheses useless when large-baseline pairs contain many outliers.Large baselines and imaging changes increase the outlier share, undermining these approaches.Source: §2 · p. 2
  • Methods and GMS jointly reject outliers and estimate global motion using piece-wise smoothness, but that assumption is often violated in practice.GMS divides images into grids and forms initial matches between grid cells.Source: §2 · p. 2

Learning-based methods

  • Dense image-input methods [32] [31] [37] [36] show promise on constrained video stereo, but remain far from solving general correspondence and can be harmful with occlusions or large baselines.The passage identifies such occlusions and baselines as commonplace in photo-tourism scenes.Source: §2 · p. 2
  • DSAC [4] introduces differentiable outlier rejection for monocular pose, but its hypothesis-evaluation strategy is difficult to extend to stereo and does not significantly outperform RANSAC.The passage characterizes DSAC as replacing RANSAC with a differentiable counterpart for end-to-end training.Source: §2 · p. 2
  • The dual network in [9] assumes homography and needs ground-truth object-corner annotations, whereas our method trains from essential matrices and processes correspondences permutation-invariantly.Figure 2 describes shared-weight perceptrons processing each correspondence independently, with Context Normalization embedding global information.Source: §2 · p. 2, Figure 2 · p. 2

3. Method

The method predicts inlier weights for sparse image correspondences and uses them in a differentiable weighted eight-point algorithm to estimate the essential matrix. Context Normalization lets a correspondence-wise network use information from the full set, while classification and pose-regression losses train it jointly.

3. Method

  • A network assigns each putative match an inlier weight, then uses the weights for essential-matrix estimation, jointly performing correspondence classification and pose regression.The relative camera rotation and translation can be recovered from the resulting essential matrix by singular value decomposition.Source: §3 · p. 3

3.1. Problem Formulation

  • The input is formed by nearest-descriptor matches between local features such as SIFT or LIFT [35]; normalized pixel-coordinate quadruplets are retained while descriptors are discarded.Camera intrinsics normalize coordinates to [−1, 1].Source: §3.1 · p. 3

3.2. Network Architecture

  • Context Normalization normalizes feature maps across correspondences within each image pair, allowing an MLP to process matches separately while embedding scene geometry and camera motion.The network is a 12-layer ResNet with 128-neuron shared perceptrons, Context and Batch Normalization, and a truncated tanh output that can assign outliers zero weight.Source: §3.2 · pp. 3–4

3.3. Computing the Essential Matrix

  • The weighted eight-point algorithm gives each correspondence’s row in the constraint matrix its predicted weight, yielding a differentiable essential-matrix estimate that ignores zero-weight matches.The method uses a self-adjoint eigendecomposition differentiable with respect to weights, followed by rank-2 projection for the final estimate.Source: §3.3 · p. 4

3.4. Learning with Classification and Regression

  • Training combines per-correspondence classification with essential-matrix regression; weak labels use a symmetric epipolar-distance threshold of 10^-2, despite allowing some near-line outliers.The regression term penalizes deviation from the ground-truth essential matrix; the weak supervisory signal avoids exhaustive correspondence annotation or requiring depth data.Source: §3.4 · pp. 4–5

3.5. RANSAC at Test Time

  • At test time, RANSAC can filter residual outliers among the network’s predicted inliers, since differentiability is unnecessary during inference.The passage states that this performs much better than stand-alone RANSAC.Source: §3.5 · p. 5

4. Experimental Results

Experiments evaluate the method on outdoor and indoor correspondence tasks, test its design choices, and compare it with keypoint-based and dense baselines. The results show strong performance on known scenes and generalization to unseen scenes, including with limited training data.

4.1. Datasets

  • Evaluation covers five YFCC100M outdoor sequences and SUN3D indoor sequences, with Fountain and HerzJesu used only to test generalization.SUN3D includes nine sequences for training and testing plus 15 test-only sequences; videos are subsampled by a factor of 10. Training sequences use disjoint 60/20/20 image splits, with 1,000 pairs for validation and testing.Source: §4.1 · p. 5

4.3.1 Hybrid Approach vs Classification vs Regression

  • The hybrid formulation outperforms classification-only, essential-matrix regression, and direct regression by 12–24% when enough keypoints are available.Essential and Direct perform worse than Classification; the gap is larger at smaller error thresholds.Source: Figure 4 · p. 6, §4.3.1 · pp. 6–7

4.2. Evaluation Protocols

  • The evaluation compares keypoint-based methods including RANSAC, MLESAC, LMEDS, and GMS, as well as dense methods G3DR and DeMoN.Pose error is the angular difference in degrees; mAP is the area under the normalized precision curve up to 5°, 10°, or 20°, using the same threshold for rotation and translation.Source: §4.2 · p. 6

4.3.2 Context Normalization vs Context Feature

  • With Context Normalization, our approach performs much better than PointNet, while removing Context Normalization makes it perform poorly.The comparison uses a similarly complex PointNet architecture trained with the same hybrid loss; results are reported with and without the final RANSAC stage.Source: Figure 5 · p. 6, §4.3.2 · p. 7

4.5.1 Performance on Known Scenes

  • On known outdoor scenes, LIFT gives more than twice the previous state-of-the-art mAP, while SIFT also outperforms other methods; indoors, our method beats all compared methods, including dense ones.For indoors, SIFT performs better than LIFT; on outdoors, the advantage widens at stricter error thresholds.Source: Figure 6 · p. 7, §4.5.1 · pp. 7–8

4.5.2 Generalization to Unknown Scenes

  • With one model trained on 2,000 outdoor and 500 indoor images, our method outperforms the state of the art on both unseen-scene subsets.The outdoor margin is significant, while the indoor margin is small; the model is tested on different scenes from those used for training.Source: Figure 7 · p. 7, §4.5.2 · p. 8

4.4. Post-processing with RANSAC

  • For 2k matches, our network takes 13 ms on GPU or 25 ms on CPU, and with RANSAC the method is 17x faster than a full RANSAC loop.The network returns about 300 inliers; RANSAC removes about 100 more in 9 ms, while a full RANSAC loop takes 373 ms and returns about 300 inliers.Source: §4.4 · p. 7

5. Conclusion

The paper proposes a sparse, single-shot deep-network technique for recovering relative motion between two images and reports that it outperforms the state of the art with few images and limited supervision. The method requires known camera intrinsics; future work will investigate using the fundamental matrix instead.

5. Conclusion

  • The method outperforms the state of the art by a significant margin with few images and limited supervision.The conclusion describes the approach as a single-shot technique for recovering relative motion between two images.Source: §5 · p. 8
  • The proposed relative-motion technique is sparse, keeping keypoint-based robust estimation relevant alongside dense, black-box formulations.It uses a deep network to recover relative motion between two images in a single shot.Source: §5 · p. 8
  • The solution requires known intrinsics; future work will investigate using the fundamental matrix, which may require extra normalization or regularization for regression stability.The authors anticipate numerical stability problems in the regression component of the hybrid loss under this change.Source: §5 · p. 8

A. Supplementary Appendix for “Learning to Find Good Correspondences” · A.1. Dataset details

The appendix describes dataset selection, splitting, and labeling for training and testing, and identifies the dataset combination used to train the best model.

A.1. Dataset details

  • SUN3D uses 9 training sequences and 15 sequences previously used by [31] for testing only.The 15 testing sequences are marked with ‡.Source: §A.1 · p. 10
  • Image numbers reported for SUN3D are counted after subsampling its video sequences by a factor of 10.Source: §A.1 · p. 10
  • Images in each set are split into disjoint training, validation, and test subsets in a 60-20-20 ratio.The passage points to Section 4.1 for the split procedure.Source: §A.1 · p. 10
  • For sets marked ‡, up to 500 images use a 0-0-100 split, and no model is trained on them.Source: §A.1 · p. 10
  • The best model is trained by concatenating datasets marked ♦ and △.Source: §A.1 · p. 10
  • Table 1 lists the datasets, whose sets receive labels a-u for use in Section A.3.Source: Table 1 · p. 10, §A.1 · p. 10
  • Figure 9 presents results for a model trained and tested on Reichstag.Source: Figure 9 · p. 11

A.2. Training with limited data

With only the Reichstag sequence for training, the method still outperforms baselines when generalizing to other Outdoors scenes. LIFT features give the best results.

A.2. Training with limited data

  • With limited-data training, the method beats GMS by 65%-200% relative and RANSAC by about 50% relative at different error thresholds on other Outdoors scenes.These results use the concatenation of St. Peter’s and Brown 1, the paper’s best model.Source: §A.2 · p. 11
  • LIFT features give the best results, and the method outperforms all baselines, with LIFT plus RANSAC and GMS plus RANSAC the closest competitors.The passage says this is consistent with the paper’s previous observations.Source: §A.2 · p. 11
  • Training uses Reichstag, with 59 training, 8 validation, and 8 test images; visibility constraints still yield over 1,500 training pairs and about 35 pairs for each held-out split.The experiments evaluate on the same sequence and on every other Outdoors sequence, following the earlier protocols.Source: §A.2 · p. 11

A.3. Per-sequence results

Per-sequence results show that our approach outperforms every baseline, with the relative performance of LIFT and SIFT differing between outdoor and indoor subsets.

A.3. Per-sequence results

  • Across testing sequences and error thresholds, our approach outperforms every baseline; LIFT beats SIFT outdoors, while SIFT beats LIFT indoors.Fig. 11 reports separate results for each testing sequence using models trained on one sequence from each data type; outdoor datasets a–f and indoor datasets g–u are averaged separately.Source: §A.3 · p. 11
  • DeMoN performs well only on outdoor sequences e and f and completely fails on YFCC100M, while G3DR [36] performs even worse.These results hint that sparse methods are preferable for photo-tourism datasets.Source: §A.3 · p. 11

A.4. Ransac post-processing

RANSAC post-processing is reported to improve both performance and speed over stand-alone RANSAC. The appendix compares stand-alone RANSAC with the method using either the 8-point algorithm or RANSAC for post-processing, noting the boost is available only at test time.

A.4. Ransac post-processing

  • The generalization experiments compare stand-alone RANSAC with the method using either the 8-point algorithm or RANSAC for post-processing.Both SIFT and LIFT use 2k keypoints.Source: §A.4 · pp. 11, 13, Table 2 · p. 13
  • The post-processing boost is possible only at test time because training requires differentiability.Source: §A.4 · p. 13
  • RANSAC post-processing greatly improves both performance and speed over stand-alone RANSAC.The comparison includes the method using RANSAC for post-processing.Source: §A.4 · p. 11, Table 2 · p. 13

A.5. Differentiating through eigendecomposition

The method differentiates through eigendecomposition using TensorFlow and refers to for details of the eigenvector derivative.

A.5. Differentiating through eigendecomposition

  • The implementation relies on TensorFlow to differentiate through eigendecomposition, with further details referred to.For X = UΣU⊤, U contains eigenvectors and Σ is diagonal with eigenvalues.Source: §A.5 · p. 13
  • The derivative definition uses Msym = 1/2(M⊤ + M), and σ_i denotes the i-th eigenvalue.Source: §A.5 · p. 13

Related papers

Loading 1711.05971v2…