PaperlayerView exact revision on arXiv

Source-linked AI summary

DKM: Dense Kernelized Feature Matching for Geometry Estimation

Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, Michael Felsberg

arXiv:2202.00667v3cs.CVcs.LG

TL;DR

Feature matching for two-view geometry remains challenging under large image variations, and prior dense methods had not matched sparse or semi-sparse performance. DKM introduces a dense matching architecture with kernelized global matching, refined warps, and learned certainty, achieving state-of-the-art geometry estimation results.

  • Problem

    Prior dense feature-matching methods failed to rival sparse or semi-sparse methods for two-view geometry estimation despite dense matching's goal of finding all correspondences.

  • Method

    DKM combines a kernelized global matcher, warp refinement using stacked feature maps and depthwise kernels, and dense certainty estimation with balanced warp sampling.

  • Results

    +8.9 AUC@5° on MegaDepth-1500, with DKM outperforming both dense and sparse methods in two-view geometry estimation.

  • Takeaways & Limitations

    DKM establishes a dense feature-matching approach that achieves state-of-the-art results across multiple geometry-estimation benchmarks.

  • Takeaways & Limitations

    Dense warp refinement is unimodal, creating challenges at discontinuous warps such as depth boundaries, while small objects bordering the sky can remain overly uncertain.

Abstract

from arXiv · show

Feature matching is a challenging computer vision task that involves finding correspondences between two images of a 3D scene. In this paper we consider the dense approach instead of the more common sparse paradigm, thus striving to find all correspondences. Perhaps counter-intuitively, dense methods have previously shown inferior performance to their sparse and semi-sparse counterparts for estimation of two-view geometry. This changes with our novel dense method, which outperforms both dense and sparse methods on geometry estimation. The novelty is threefold: First, we propose a kernel regression global matcher. Secondly, we propose warp refinement through stacked feature maps and depthwise convolution kernels. Thirdly, we propose learning dense confidence through consistent depth and a balanced sampling approach for dense confidence maps. Through extensive experiments we confirm that our proposed dense method, \textbf{D}ense \textbf{K}ernelized Feature \textbf{M}atching, sets a new state-of-the-art on multiple geometry estimation benchmarks. In particular, we achieve an improvement on MegaDepth-1500 of +4.9 and +8.9 AUC$@5^{\circ}$ compared to the best previous sparse method and dense method respectively. Our code is provided at https://github.com/Parskatt/dkm

1. Introduction

Feature matching supplies correspondences for two-view geometry estimation, but challenging viewpoint, illumination, time, and seasonal changes make reliable matches difficult. DKM addresses the prior performance gap of dense methods and outperforms sparse and dense approaches on geometry estimation.

  • Two-view geometry estimation supports 3D reconstruction, SLAM, and visual relocalisation through matched pixel pairs.
  • Challenging image pairs can vary substantially in viewpoint, illumination, time of day, and season.
  • Sparse keypoint methods struggle because reliable, repeatable keypoint localization is difficult in challenging scenes.
  • Dense matching avoids coarse-grid repeatability issues and provides affine matches, but earlier dense methods did not rival sparse or semi-sparse geometry-estimation performance.
  • DKM combines improved global matching, warp refinement, dense certainty estimation, and balanced warp sampling to outperform dense and sparse methods in homography and relative pose estimation.
  • +8.9 AUC@5° improvement over the best previous dense method is reported on MegaDepth-1500.

2. Related Work

Prior global matching methods use descriptor distances, conditioned descriptors, or coarse detector-free grids, while dense methods regress a warp from a global correlation volume. DKM formulates matching with a Gaussian Process approach.

  • Traditional global matching computes pair-wise descriptor distances for detected keypoints and extracts matches with mutual nearest neighbours.
  • Graph neural networks can condition descriptors jointly, while optimal transport can replace mutual-nearest-neighbour extraction.
  • Detector-free methods avoid keypoint detection by performing global matching on a coarse grid.
  • Dense methods must produce a dense warp, typically regressed from a global 4D-correlation volume.
  • DKM proposes a Gaussian Process formulation of the matching problem.

3. Method

DKM estimates scene geometry through dense matching by extracting multiscale features, predicting and refining dense warps, learning certainty, and sampling reliable matches for robust estimation.

  • Overview: The five-stage pipeline extracts a shared-weight multistride feature pyramid, establishes coarse global matches, refines the warp, estimates certainty, and applies robust geometry solvers.The final stage uses RANSAC with minimal solvers.
  • Global Matcher: The global matcher formulates coordinate matching as Gaussian Process regression over embedded features, then decodes the posterior into a dense warp and certainty.The decoder predicts coordinates in a canonical grid and match-validity logits.
  • Global Matcher: Cosine coordinate embeddings address multimodality in coordinate regression, while the GP posterior supplies a probabilistic representation for decoding.The method assumes independent embedding dimensions, yielding a block-diagonal kernel, and uses an exponential cosine similarity kernel.
  • Qualitative Behavior: The global matcher and refiner produce accurate warps and certainty under extreme perspective, illustrating robustness to large viewpoint changes.Figure 5 visualizes forward and reverse certainty-weighted warps.
  • Certainty and Sampling: Dense certainty is learned by classifying depth consistency, then used with balanced sampling to select reliable and diverse matches for geometry estimation.Balanced sampling first draws certainty-weighted matches, estimates density in four-dimensional match space, and reweights toward scene diversity.

4. State-of-the-Art Comparison

DKM achieves state-of-the-art geometry estimation across outdoor homography and pose benchmarks, outperforming prior sparse and dense methods.

  • HPatches Homography: +3.6 AUC@3px over the best previous method on HPatches homography estimation.HPatches evaluates planar scenes under homographies, with AUC as the metric.
  • MegaDepth-1500 Pose Estimation: +4.9 AUC@5° over ASpanFormer on MegaDepth-1500 pose estimation.DKM also exceeds the previous best dense method, PDC-Net+, by +8.9 AUC@5°.
  • MegaDepth-1500 Pose Estimation: +8.9 AUC@5° over PDC-Net+ on MegaDepth-1500 pose estimation.The comparison is against the best previous dense method under the stated benchmark protocol.
  • ScanNet-1500 Pose Estimation: +4.0 AUC@5° over the previous best sparse method on ScanNet-1500 pose estimation.Compared with the previous best dense method, the reported gain is +9.3.

5. Ablation Study

The ablations attribute DKM’s performance to its kernelized global matcher, depthwise warp refiners, balanced certainty-based sampling, and sufficiently high resolution, while identifying failure near depth discontinuities.

  • Global Matcher: +1.1 AUC@5° from replacing the baseline correlation-volume regressor with the kernelized global matcher.The cosine coordinate-embedding approach outperforms the linear regression alternative.
  • Warp Refiners: +4.8 AUC@5° from depthwise warp refiners over the baseline.The stacked-feature-map input representation adds a further +1.5 AUC@5°.
  • Match Sampling: +2.0 AUC@5° from balanced match sampling during estimation.The ablation also shows that a certainty estimate is needed for the sampling strategy.
  • Resolution: Higher training resolution improves estimation performance, with gains when increasing from 384 × 512 to 540 × 720.The study concludes that sufficiently high resolution is important for accurate estimation.
  • Failure Case: Warp refinement can struggle near depth discontinuities, and DKM’s certainty estimate can occasionally be overly uncertain.This representative failure case describes limitations of the unimodal refinement and certainty prediction.
  • Bidirectional DKM: +1.0 AUC@5° from using bidirectional DKM with reverse-warp matches.The comparison evaluates the impact of bidirectional matching on two-view pose estimation.

6. Conclusion

DKM is presented as a dense feature-matching method achieving state-of-the-art two-view geometry estimation. Its contributions combine kernelized global matching, depthwise-kernel warp refinement, dense confidence learning, and balanced warp sampling, while limitations remain at discontinuous warps and small objects bordering the sky.

  • A kernelized global matcher and embedding decoder provide robust coarse matches.
  • Warp refinement uses large depth-wise kernels on stacked feature maps.
  • Dense confidence maps are learned by classifying consistent depth, with balanced sampling for dense warps.
  • +8.9 AUC@5◦ gains are reported on the MegaDepth-1500 benchmark.
  • Limitations: The unimodal dense warp refinement faces challenges at discontinuities such as depth boundaries, while small objects bordering the sky can receive excessive uncertainty.

Supplementary Material for

The supplementary material includes qualitative DKM warp and certainty examples, including a pair from Piazza San Marco.

  • The document identifies DKM as Dense Kernelized Feature Matching for Geometry Estimation.
  • A qualitative Piazza San Marco pair is shown with DKM warp and certainty.

A.1. MegaDepth-8-Scenes Pose Estimation

MegaDepth-8-Scenes broadens evaluation beyond MegaDepth-1500 by sampling 1600 pairs from eight scenes, while retaining the same protocol. DKM achieves state-of-the-art results against both sparse and dense methods.

  • 1600 pairs from 8 different scenes are sampled to test whether MegaDepth-1500 results hold in a wider setting.
  • The eight scenes include Piazza San Marco, Sagrada Familia, Lincoln Memorial Statue, British Museum, Tower of London, Florence Cathedral, Milan Cathedral, and Mount Rushmore.
  • The MegaDepth-8-Scenes benchmark uses the same protocol as MegaDepth-1500.
  • +3.3 AUC@5◦ relative performance is achieved over the previous best sparse method.
  • +8.7 percentage points are achieved over the previous best dense method.

A.2. St. Paul’s Cathedral

On St. Paul’s Cathedral, DKM is compared with COTR and ECO-TR using the ECO-TR evaluation protocol. The reported comparison favors DKM by +8.0 mAA@5◦, alongside qualitative examples from multiple scenes.

  • DKM is compared with COTR and ECO-TR on the St. Paul’s Cathedral scene.
  • The evaluation uses the ECO-TR protocol.
  • The supplementary material includes qualitative examples from Sagrada Familia, British Museum, Florence Cathedral, and Mount Rushmore.
  • +8.0 mAA@5◦ is reported for DKM over COTR and ECO-TR.
  • Qualitative examples are presented for St. Paul’s Cathedral and St. Peter’s Basilica.
  • Pose estimation results on MegaDepth-8-Scenes are measured in AUC, where higher is better, with sparse methods in the top section and dense methods in the bottom section.

B.2. HPatches

DKM performs well on HPatches planar scenes despite not being trained for planar scenes, with qualitative examples shown for the v bird and v graffiti pairs.

  • B.2. HPatches: DKM performs very well on HPatches planar scenes despite not being trained for planar scenes.
  • B.2. HPatches: Qualitative results are presented for the HPatches planar scene v bird pair.
  • B.2. HPatches: Qualitative results are presented for the HPatches planar scene v graffiti pair.

C. Additional Failure Cases

DKM fails completely in an example with an extreme lack of texture, but predicts very low certainty for that pair.

  • C. Additional Failure Cases: DKM completely fails on a pair with an extreme lack of unique local textures.The authors note that unique global patterns exist, so the matching is not ill-defined.
  • C. Additional Failure Cases: The model predicts very low certainty for the failure case, indicating a well-calibrated uncertainty estimate.
  • C. Additional Failure Cases: Figure 17 shows a qualitative example of DKM’s indoor model on the ScanNet-1500 benchmark.
Loading 2202.00667v3…