PaperlayerView exact revision on arXiv

Source-linked AI summary

DKM: Dense Kernelized Feature Matching for Geometry Estimation

Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, Michael Felsberg

arXiv:2202.00667v3cs.CVcs.LG

TL;DR

Prior dense matching methods had not matched sparse and semi-sparse methods on geometry estimation. DKM introduces a dense matching approach and reports state-of-the-art geometry estimation, including a +8.9 AUC@5° gain on MegaDepth-1500.

  • Problem

    Prior dense matching approaches had not achieved performance rivaling sparse or semi-sparse methods on geometry estimation.

    Source: §1 · p. 2

  • Method

    DKM combines a kernelized global matcher and warp refinement with dense certainty estimation from consistent depth and balanced sampling.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    DKM achieves state-of-the-art two-view geometry estimation, with gains of +8.9 AUC@5° on MegaDepth-1500.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The results show that dense matching can outperform sparse and dense prior methods in homography and two-view relative pose estimation.

    Source: §1 · p. 2

    DKM’s unimodal dense warp refinement poses challenges at discontinuities such as depth boundaries.

    Source: §6 · p. 8

Abstract

from arXiv · show

Feature matching is a challenging computer vision task that involves finding correspondences between two images of a 3D scene. In this paper we consider the dense approach instead of the more common sparse paradigm, thus striving to find all correspondences. Perhaps counter-intuitively, dense methods have previously shown inferior performance to their sparse and semi-sparse counterparts for estimation of two-view geometry. This changes with our novel dense method, which outperforms both dense and sparse methods on geometry estimation. The novelty is threefold: First, we propose a kernel regression global matcher. Secondly, we propose warp refinement through stacked feature maps and depthwise convolution kernels. Thirdly, we propose learning dense confidence through consistent depth and a balanced sampling approach for dense confidence maps. Through extensive experiments we confirm that our proposed dense method, \textbf{D}ense \textbf{K}ernelized Feature \textbf{M}atching, sets a new state-of-the-art on multiple geometry estimation benchmarks. In particular, we achieve an improvement on MegaDepth-1500 of +4.9 and +8.9 AUC$@5^{\circ}$ compared to the best previous sparse method and dense method respectively. Our code is provided at https://github.com/Parskatt/dkm

1. Introduction

The introduction frames feature matching as the first step in two-view geometry estimation, where large viewpoint and appearance changes make reliable correspondences difficult. DKM addresses limitations of sparse and prior dense methods with a dense matcher, refinement, and certainty-and-sampling design.

Motivation

  • Dense matching avoids coarse-scale sparse-match grid-artifact issues and provides affine matches, but previous dense approaches had not rivaled sparse or semi-sparse methods on geometry estimation.Sparse keypoint localization can be difficult in challenging scenes, causing matching and estimation errors.Source: §1 · pp. 1–2

State-of-the-Art

  • +8.9 AUC@5◦ on MegaDepth-1500 over the best previous dense method is the reported improvement in estimation results.The paper also reports that its method outperforms dense and sparse methods in homography and two-view relative pose estimation.Source: §1 · p. 2

Global Matcher

  • DKM proposes a kernelized global matcher and embedding decoder to produce robust coarse matches.The paper identifies Table 4 as the ablation of their performance gains.Source: §1 · p. 2

Warp Refiners

  • DKM refines warps with large depthwise separable kernels, stacked feature maps, and local correlation to improve precision.The paper identifies Table 5 as the ablation of the refinement stage's performance impact.Source: §1 · p. 2

Certainty and Sampling

  • DKM predicts dense certainty from consistent depth and uses balanced sampling for dense matches.The paper identifies Table 6 as the ablation of the certainty and sampling gains.Source: §1 · p. 2

2. Related Work

Prior work addresses global matching, dense warp refinement, and match certainty, while DKM proposes a Gaussian Process matcher, stacked-feature refinement, and certainty based on 3D consistency with balanced sampling.

Global Matching

  • DKM formulates global matching as a Gaussian Process problem, contrasting with descriptor-distance matching, graph-based descriptor conditioning, and coarse-grid detector-free approaches.Dense methods typically predict a warp by regression over a global 4D-correlation volume.Source: §2 · p. 2

Match Refinement

  • DKM refines dense warps using stacked feature maps and large depth-wise convolution kernels, whereas prior dense refinement uses local correlation.Detector-free methods instead refine sparse matches by patch-based CNN regression or transformers.Source: §2 · p. 2

Match Certainty and Sampling

  • DKM estimates certainty from 3D reconstruction consistency and uses balanced sampling to provide diverse matches for geometry estimation.The approach responds to overconfident estimates on unmatchable pairs and the importance of well-distributed matches for pose estimation.Source: §2 · p. 2

3. Method

DKM estimates two-view geometry by extracting multiscale features, predicting and refining a dense warp with certainty, then sampling matches for robust estimation.

3.1. Preliminaries

  • The framework predicts a dense warp and certainty, samples a subset of matches, and uses a robust estimator such as RANSAC to infer geometry.The certainty is zero for unmatchable pixels, and matches are sampled without replacement.Source: §3.1 · pp. 3–4

3.2. Constructing the Global Matcher Gθ

  • The global matcher casts matching as embedded coordinate regression using Gaussian Process regression; its embeddings preserve multimodality, and an embedding decoder predicts canonical-grid coordinates plus per-pixel match-validity logits.Global matchers use stride-32 and stride-16 backbone features; the stride-16 embedding decoder receives context feature maps from the stride-32 decoder.Source: §3.2 · p. 4

3.3. Refining the Warp with Rθ

  • Warp refiners concatenate warped features from image B and use local correlation in B; their blocks use 5x5 depthwise separable kernels followed by 1x1 convolutions.The warp and certainty are upsampled between scales, and refiners predict residual warp offsets and certainty-logit offsets.Source: §3.3 · p. 5

3.4. Certainty Estimation and Sampling for Geometry Estimation

  • Certainty training labels matches consistent under ground-truth depth, while balanced sampling thresholds certainty at 0.05 and reweights an initial sample by reciprocal KDE in 4D match space.The depth consistency constraint uses α = 0.05; the KDE reweighting is intended to produce scene-balanced samples.Source: §3.4 · pp. 5–6

3.5. Loss Formulation

  • Training combines per-stride warp regression and certainty losses, using ℓ2 warp distance and unweighted binary cross entropy against the consistent-depth mask.The method detaches gradients between scales and sets fine-stride certainty targets to zero when the coarse warp exceeds a distance threshold from ground truth.Source: §3.5 · p. 6

4. State-of-the-Art Comparison

DKM improves on prior sparse and dense methods across outdoor and indoor geometry estimation benchmarks. The evaluation also reports HPatches homography results and additional outdoor comparisons.

4. State-of-the-Art Comparison

  • DKM is trained and evaluated separately for outdoor and indoor geometry estimation; evaluation averages 5 benchmark runs and samples at most 5,000 matches.Source: §4 · p. 6

4.1. Training Details

  • Training uses AdamW for 250,000 steps, with MegaDepth outdoor images at 540 × 720 and ScanNet [8] indoor images at 480 × 640.The learning rate decays by a factor of 0.2 at steps 166,666 and 225,000.Source: §4.1 · p. 6

4.2. Outdoor Geometry Estimation

  • +3.6 AUC@3px over the best previous method on HPatches homography estimation.The evaluation follows LoFTR’s protocol, resizing images so their shorter side is 480 pixels.Source: §4.2 · p. 6, Table 1 · p. 7
  • DKM shows major improvements on a new benchmark of 8 diverse MegaDepth scenes and large improvements over COTR [19] and ECO-TR [41] on St. Paul’s Cathedral.Details of these comparisons are in supplementary sections A.1 and A.2.Source: §4.2 · p. 7

4.3. Indoor Geometry Estimation

  • +4.0 AUC@5◦ over the previous best sparse method and +9.3 over the previous best dense method on ScanNet-1500.ScanNet includes challenging sequences with low-texture regions and large perspective changes.Source: Table 3 · p. 7, §4.3 · p. 7

5. Ablation Study

The ablations test how the global matcher, warp-refinement design and features, match sampling, input resolution, and bidirectional matching affect geometry-estimation performance.

Global Matcher

  • The proposed global matcher improves performance by +1.1 AUC@5◦ over a strong baseline, while linear regression underperforms cosine coordinate embeddings.The comparison tests kernelized regression with an embedding decoder against a correlation-volume regressor.Source: §5 · p. 7, Table 4 · p. 7

Warp Refiners

  • Depthwise warp refiners outperform the baseline by +4.8 AUC@5, while the input representation improves performance by +1.5 AUC@5.The comparison ablates both the refiner architecture and the features used.Source: §5 · p. 8, Table 5 · p. 8

Match Sampling

  • Balanced match sampling improves estimation by +2.0 AUC@5, and comparison with sampling without certainty shows the need for a certainty estimate.Balancing uses the reciprocal of the KDE estimate.Source: §5 · p. 8, Table 6 · p. 8

Resolution

  • Increasing input resolution from 384 × 512 to 540 × 720 improves performance by +1.3 AUC@5◦.The study finds that resolution must be sufficiently high for accurate estimation.Source: §5 · p. 8, Table 7 · p. 8

Bidirectionality

  • For two-view pose estimation, bidirectional DKM yields an improvement of +1.0 AUC@5° using reverse-warp matches; AUC is higher-is-better.Source: §5 · p. 8, Table 8 · p. 8

6. Conclusion

DKM is a dense feature-matching approach for two-view geometry estimation, combining a kernelized global matcher, depth-wise warp refinement, and dense-confidence learning. Its experiments report a +8.9 AUC@5◦ gain on MegaDepth-1500, while the conclusion notes limitations in warp refinement and confidence estimation.

6. Conclusion

  • DKM reports +8.9 AUC@5◦ on the MegaDepth-1500 benchmark.The conclusion describes this as a gain but does not specify the comparison baseline in the supplied passages.Source: §6 · p. 8
  • DKM combines a kernelized-regressor global matcher, large depth-wise kernels on stacked feature maps for warp refinement, and dense-confidence learning through consistent-depth classification and balanced sampling.These are the three contributions summarized in the conclusion.Source: §6 · p. 8

Limitations

  • DKM’s unimodal warp refinement struggles with discontinuities at depth boundaries, and its confidence maps are overly uncertain for small objects bordering the sky.The paper suggests the confidence issue may stem from classifying consistent depth rather than predicting model uncertainty, as in PDCNet.Source: §6 · p. 8

Supplementary Material for · A. Additional State-of-the-Art Comparison

The supplementary material includes a qualitative example of a Piazza San Marco image pair with DKM warp and certainty.

Supplementary Material for

  • Figure 8 shows the Piazza San Marco (0008) pair with DKM warp and certainty.Source: Figure 8 · p. 12

A.1. MegaDepth-8-Scenes Pose Estimation

The MegaDepth-8-Scenes benchmark extends evaluation to 1,600 image pairs from eight scenes, where DKM again achieves state-of-the-art pose-estimation results.

A.1. MegaDepth-8-Scenes Pose Estimation

  • The benchmark samples 1,600 pairs from eight scenes to test whether MegaDepth-1500 results generalize beyond its two scenes.The authors call this benchmark MegaDepth-8-Scenes.Source: §A.1 · p. 12
  • +3.3 AUC@5° over the previous best sparse method and +8.7 percentage points over the previous best dense method on MegaDepth-8-Scenes.Results use the same protocol as MegaDepth-1500.Source: §A.1 · p. 12

A.2. St. Paul’s Cathedral · B. Further Qualitative Examples · B.1. MegaDepth-1500

On St. Paul’s Cathedral, DKM outperforms COTR and ECO-TR by +8.0 mAA@5°. The block also documents qualitative DKM warp-and-certainty examples and identifies Megadepth-8-Scenes pose-estimation results.

A.2. St. Paul’s Cathedral

  • Qualitative examples show DKM warp and certainty for Sagrada Familia and British Museum image pairs.Source: Figure 9 · p. 12, Figure 10 · p. 12
  • +8.0 mAA@5°: DKM outperforms COTR and ECO-TR on St. Paul’s Cathedral.The comparison uses ECO-TR’s evaluation protocol.Source: §A.2 · p. 12

B.1. MegaDepth-1500

  • A qualitative example is presented for the St. Peter’s Basilica (0015) scene; Table 9 reports Megadepth-8-Scenes pose-estimation AUC, where higher is better.Source: §B.1 · p. 12, Table 9 · p. 13
  • A qualitative example shows DKM warp and certainty for a Florence Cathedral (0032) image pair.Source: Figure 11 · p. 13
  • A qualitative example shows DKM warp and certainty for a Mount Rushmore (1589) image pair.Source: Figure 12 · p. 13

B.2. HPatches · B.3. ScanNet

The appendix shows DKM performing well qualitatively on HPatches despite not being trained for planar scenes, and also presents examples from St. Paul’s Cathedral, St. Peter’s Basilica, and ScanNet-1500.

B.2. HPatches

  • Despite not being trained for planar scenes, DKM performs very well qualitatively on HPatches.Figures 15 and 16 show results on the planar scenes v bird and v graffiti.Source: §B.2 · p. 13, Figure 15 · p. 14, Figure 16 · p. 14
  • The St. Paul’s Cathedral benchmark table reports pose-estimation mAA, averaged with estimated standard deviation over five runs.The supplied passage gives the table’s metric and reporting protocol, but no result values.Source: Table 10 · p. 13
  • A qualitative example shows DKM’s warp and certainty on the St. Paul’s Cathedral benchmark.This is identified as Figure 13.Source: Figure 13 · p. 13
  • A qualitative example shows DKM’s warp and certainty on a pair from the St. Peter’s Basilica (0015) scene.This is identified as Figure 14.Source: Figure 14 · p. 13

B.3. ScanNet

  • The appendix presents a qualitative example of DKM’s indoor model on the ScanNet-1500 benchmark.The passage identifies this example as Figure 17.Source: §B.3 · p. 13

B.2. HPatches

  • Figure 15 presents a DKM result on the HPatches planar scene v bird.The caption identifies the scene but does not provide a quantitative result.Source: Figure 15 · p. 14
  • Figure 16 presents a DKM result on the HPatches planar scene v graffiti.The caption identifies the scene but does not provide a quantitative result.Source: Figure 16 · p. 14

C. Additional Failure Cases

DKM can completely fail when local textures lack unique features, yet it predicts very low certainty for this case.

C. Additional Failure Cases

  • DKM completely fails on a pair lacking unique local textures, despite unique global patterns, and estimates very low certainty.The authors suggest the lack of unique local textures causes the failure; the low certainty indicates a well-calibrated uncertainty estimate.Source: C. Additional Failure Cases · p. 14, Figure 18 · p. 14

Related papers

Loading 2202.00667v3…