PaperlayerView exact revision on arXiv

Source-linked AI summary

To Learn or Not to Learn: Visual Localization from Essential Matrices

Qunjie Zhou, Torsten Sattler, Marc Pollefeys, Laura Leal-Taixe

arXiv:1908.01293v2cs.CV

TL;DR

Accurate visual localization often relies on scene-specific representations, while relative-pose methods promise easier adaptation but remain less accurate. The paper proposes an essential-matrix localization framework and compares classical and learned estimators, finding that the classical approach achieves state-of-the-art performance while learned alternatives expose specific generalization and accuracy limits.

  • Problem

    Relative-pose localization offers flexible scene adaptation, but the paper asks why learning-based approaches do not perform well in this setting.

    Source: §I · p. 1

  • Method

    The paper proposes a 3D-model-free localization framework that uses retrieved images, estimated essential matrices, and RANSAC to recover absolute poses, comparing classical and learned estimators.

    Source: §I · p. 1, §III · p. 2

  • Results

    The SIFT-based approach is competitive with complex state-of-the-art methods, while essential-matrix regression generalizes poorly and learned features in the hybrid approach remain less accurate than SIFT+5pt.

    Source: §I · p. 1, B. Analyzing Relative Pose Regression… · p. 6, C. Discussion · p. 6

  • Takeaways & Limitations

    The analysis identifies relative-pose regression layers as a principal generalization failure and shows that learned features and matches also need improvement for accuracy.

    Source: B. Analyzing Relative Pose Regression… · p. 6, C. Discussion · p. 6, §VI · p. 6

Abstract

from arXiv · show

Visual localization is the problem of estimating a camera within a scene and a key component in computer vision applications such as self-driving cars and Mixed Reality. State-of-the-art approaches for accurate visual localization use scene-specific representations, resulting in the overhead of constructing these models when applying the techniques to new scenes. Recently, deep learning-based approaches based on relative pose estimation have been proposed, carrying the promise of easily adapting to new scenes. However, it has been shown such approaches are currently significantly less accurate than state-of-the-art approaches. In this paper, we are interested in analyzing this behavior. To this end, we propose a novel framework for visual localization from relative poses. Using a classical feature-based approach within this framework, we show state-of-the-art performance. Replacing the classical approach with learned alternatives at various levels, we then identify the reasons for why deep learned approaches do not perform well. Based on our analysis, we make recommendations for future work.

I. INTRODUCTION

Accurate localization commonly relies on scene-specific 3D representations, while image-database and relative-pose approaches offer more flexible adaptation. The paper analyzes why learned relative-pose localization remains inaccurate and compares learned alternatives within a shared framework.

I. INTRODUCTION

  • Scene-specific 3D representations support accurate pose estimation but must be built for each scene and do not generalize to unseen scenes.They can represent geometry explicitly as a point cloud or implicitly in CNN weights.Source: §I · p. 1
  • Relative-pose localization offers a simpler, more flexible scene representation and easier adaptation to new scenes, motivating analysis of why learned approaches perform poorly.Image databases with associated camera poses can be built by adding posed images.Source: §I · p. 1
  • The framework compares classical SIFT features, direct essential-matrix regression, and learned feature matching, finding the SIFT-based approach competitive with more complex state-of-the-art methods.The learned regression approach remains significantly worse than SIFT and does not generalize to unseen scenes; learned features also produce less accurate results in the hybrid approach.Source: §I · p. 1

II. RELATED WORK

Prior localization work spans 3D-feature matching, image-database retrieval, and learned absolute or relative pose estimation. This paper builds on retrieval and relative-pose methods while examining the accuracy limits of learning the full pipeline.

II. RELATED WORK

  • Feature-based localization can be accurate, but direct methods face scalability limits from memory use and ambiguities, while compression can reduce the number of localized images.Indirect methods retrieve relevant images and match descriptors against points visible in them.Source: §II · pp. 1–2
  • The paper extends relative-pose triangulation beyond the two-database-image limit of with a more general RANSAC-based approach.The pipeline uses known database-image poses and relative poses to estimate the query pose without requiring a stored 3D scene representation.Source: §II · p. 2
  • Learned 3D-matching and absolute-pose approaches generally use scene-specific representations, while this paper focuses on single-image localization rather than sequence-based pose regression.The cited sequence-based methods can improve performance over individual-image methods.Source: §II · p. 2
  • Relative-pose learning is expected to generalize across scenes, but prior approaches include depth-supervised or image-synthesis-supervised methods and a planar-motion setting that is nontrivial to extend to 6DOF with larger baselines.The approach in has limited generalization to new scenes.Source: §II · p. 2
  • Compared with [35], the paper regresses essential matrices, handles pose ambiguity with a novel RANSAC scheme, and finds that learning the whole pipeline is not the most accurate solution.The pose-loss weighting in [35] may need adjustment per scene.Source: §II · p. 2

III. ESSENTIAL MATRIX BASED LOCALIZATION

The proposed model-free pipeline retrieves database images, estimates query-to-database essential matrices, and combines them with known database poses to recover the query’s absolute pose. Essential-matrix regression avoids a scene-dependent pose-loss weighting issue, with ambiguities handled in RANSAC.

III. ESSENTIAL MATRIX BASED LOCALIZATION

  • The pipeline retrieves relevant database images, estimates an essential matrix for each query-image pair, then uses the retrieved images’ known poses and those matrices to estimate the query’s absolute pose.The pipeline is agnostic to how the essential matrices are estimated.Source: §III · p. 2, Figure 1 · p. 3
  • Regressing essential matrices removes the scene-dependent weighting issue of direct relative-pose regression, while the resulting pose ambiguities are handled inside a RANSAC loop.Direct decomposition of an essential matrix into relative poses produces ambiguities.Source: §III · p. 2
  • The framework is 3D-model-free and uses retrieved images with known camera poses; rotations may be represented by matrices or quaternions.The pipeline uses essential matrices in its RANSAC scheme to recover absolute poses.Source: §III · pp. 2–3

A. Retrieving Relevant Database Images

The pipeline retrieves database images and selects pairs to balance triangulation geometry with visual overlap. It then estimates pairwise essential matrices and resolves their pose ambiguities using multiple image pairs.

A. Retrieving Relevant Database Images

  • Database images are represented by 4096-dimensional DenseVLAD descriptors, chosen for their reported generalization to unseen scenes.The passage contrasts DenseVLAD with other learned retrieval pipelines.Source: A. Retrieving Relevant Database Images · p. 3
  • Selecting only the top-ranked retrieved images can yield similar camera viewpoints; iterative distance constraints instead seek larger triangulation angles while retaining enough visual overlap.RANSAC robustly handles pairs depicting a different scene part from the query.Source: A. Retrieving Relevant Database Images · p. 3

B. Pairwise Relative Pose Estimation

  • Each pair’s essential matrix yields four candidate relative poses; without feature matches, the usual cheirality test cannot select the correct candidate for directly regressed poses.The candidates are (R,t), (R,−t), (R′,t), and (R′,−t).Source: B. Pairwise Relative Pose Estimation · p. 3
  • Triangulation across multiple relative poses determines query position despite translation-sign flips, while the rotation choice is resolved by comparing absolute-rotation predictions from two image pairs.The position can be uniquely determined from at least two image pairs.Source: B. Pairwise Relative Pose Estimation · p. 3

C. Absolute Pose Estimation via RANSAC

The method estimates a query camera pose by triangulating rays from relative poses, then uses RANSAC to reject inconsistent image pairs.

C. Absolute Pose Estimation via RANSAC

  • Two image pairs provide relative rotations and translation directions for triangulating the query position and estimating its absolute rotation.The triangulation requires non-collinear camera centers; with multiple database images, the problematic case is when all images lie exactly on a line.Source: C. Absolute Pose Estimation via RANSAC · p. 3
  • RANSAC samples two image pairs to form an absolute-pose hypothesis, then treats a pair as an inlier when its translation-direction angle is below αmax.For each pair, the method selects the relative rotation most consistent with the hypothesis before testing translation consistency.Source: C. Absolute Pose Estimation via RANSAC · p. 3
  • Local optimization accounts for noisy relative-pose predictions, and RANSAC returns the pose with the most inliers.Source: C. Absolute Pose Estimation via RANSAC · p. 3

IV. ESSENTIAL MATRIX ESTIMATION

The section compares handcrafted and learned essential-matrix estimation within a localization pipeline, motivated by the accuracy gap for end-to-end learned localization. Its learned approach regresses essential matrices rather than directly balancing rotation and translation losses.

IV. ESSENTIAL MATRIX ESTIMATION

  • The section frames learned essential-matrix estimation as one point on a spectrum from handcrafted to data-driven approaches to investigate why learned localization underperforms classic methods.Source: §IV · pp. 3–4

A. Feature-based: SIFT + 5-Point Solver

B. Learning-based: Direct Regression via EssNet

  • Essential-matrix regression eliminates the scene-dependent β weighting required by relative-pose regression and outperforms [35].Rotation and translation are typically measured in degrees and meters, respectively, motivating the weighting parameter.Source: B. Learning-based: Direct Regression… · p. 4
  • The learned methods use a ResNet34 siamese network with either fixed feature matching in EssNet or learnable neighborhood-consensus matching in NC-EssNet.The matching score map feeds regression layers that predict a 9D essential-matrix vector, which is projected to a valid essential matrix; training minimizes Euclidean distance to the ground-truth matrix.Source: B. Learning-based: Direct Regression… · p. 4

C. Hybrid: Learnable Matching + 5-Point Solver

The hybrid method learns feature extraction and matching, then uses a classical geometric solver to estimate the essential matrix.

C. Hybrid: Learnable Matching + 5-Point Solver

  • Learned feature extraction and matching produce 2D-2D correspondences, while a 5-point solver inside RANSAC computes the essential matrix.The approach is equivalent in architecture to NC-EssNet without its regression layers.Source: C. Hybrid: Learnable Matching +… · p. 4

V. EXPERIMENTS

The experiments evaluate the proposed localization approach and use handcrafted and learned essential-matrix estimators to examine learned methods’ weaknesses.

V. EXPERIMENTS

  • The handcrafted-feature, RANSAC-based essential-matrix approach achieves state-of-the-art performance; learned alternatives are used to analyze their weaknesses.Source: §V · p. 4
  • Evaluation uses Cambridge Landmarks and 7 Scenes, reporting per-dataset averages of median absolute position error in meters and rotation error in degrees.Source: §V · p. 4
  • EssNet and NC-EssNet share training settings, use a ResNet34 pretrained on ImageNet, and are evaluated per scene after training on pairs from all scenes.A single network tests all Cambridge Landmarks sequences; training uses validation-based early stopping and selects the model with best validation accuracy.Source: §V · p. 5
  • Testing retrieves the top five training images with DenseVLAD and applies scene-specific distance constraints; RANSAC thresholds t1 and t2 are selected by grid search.The table presents the thresholds used in the experiments.Source: §V · p. 5, Table I · p. 5

A. Comparison with State-of-the-Art

SIFT+5Pt outperforms image-retrieval, absolute-pose-regression, and relative-pose-regression methods, and is competitive with structure-based methods that require scene-specific models.

A. Comparison with State-of-the-Art

B. Analyzing Relative Pose Regression (RPR)

NC-EssNet performs strongly among pose-regression methods, but cross-scene tests reveal a substantial accuracy gap; hybrid experiments point to regression layers, rather than features, as the generalization weakness.

B. Analyzing Relative Pose Regression (RPR)

  • The analysis uses EssNet and NC-EssNet as RPR methods within the essential-matrix pipeline to investigate their generalization behavior.Source: B. Analyzing Relative Pose Regression… · pp. 5–6
  • NC-EssNet performs better than all APR approaches except MapNet+PGO, is competitive with RelocNet, and outperforms Relative PN.MapNet+PGO uses external GPS information; AnchorNet is more accurate but is trained explicitly per scene.Source: B. Analyzing Relative Pose Regression… · p. 5
  • Cross-scene evaluation shows a substantial pose-accuracy gap for EssNet and NC-EssNet versus same-scene training, especially compared with SIFT+5Pt.The authors conclude that EssNet and NC-EssNet fail to learn a general underlying principle; similar behavior is reported for prior RPR methods.Source: B. Analyzing Relative Pose Regression… · p. 6, Table III · p. 6
  • In the hybrid approach, localization accuracy varies little across feature extractors trained with different strategies, indicating that the regression layers drive the observed generalization failure.The hybrid keeps the NC matching layer trained on unrelated ivd fixed while varying feature-extractor training.Source: B. Analyzing Relative Pose Regression… · p. 6

C. Discussion

The discussion links learned pose regression’s generalization and accuracy problems to implicit matching and coarse feature localization. The conclusion reports state-of-the-art results for the framework but says purely data-driven pose regression generalizes poorly.

C. Discussion

  • Explicitly learning feature matching improves generalization over direct relative-pose regression, but its poses remain less accurate than SIFT+5pt.The discussion relates the remaining inaccuracy to CNN features being coarsely localized to image patches.Source: C. Discussion · p. 6, Table IV · p. 6

VI. CONCLUSION

  • The essential-matrix framework achieves state-of-the-art results, while the purely data-driven approach does not generalize well.Source: §VI · p. 6
  • The learned features and matches generalize quite well, but using them directly for pose estimation is less accurate than the handcrafted pipeline.The authors conclude that more research is needed before data-driven localization is both accurate and easily generalizes to new scenes.Source: §VI · p. 6
Loading 1908.01293v2…