PaperlayerView exact revision on arXiv

Source-linked AI summary

RoMa: Robust Dense Feature Matching

Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, Michael Felsberg

arXiv:2305.15404v2cs.CV

TL;DR

Dense feature matching must remain robust to challenging real-world changes, but existing coarse features rely on limited 3D-supervised data. RoMa combines frozen DINOv2 coarse features, specialized ConvNet fine features, a transformer anchor-probability decoder, and improved losses, achieving a 36% gain on WxBS and setting a new state-of-the-art.

  • Problem

    Existing dense matchers commonly learn coarse features with expensive, limited 3D supervision, creating a challenge for robust real-world matching.

  • Method

    RoMa combines frozen DINOv2 coarse features with specialized ConvNet fine features, a transformer decoder predicting anchor probabilities, and regression-by-classification with robust regression.

  • Results

    36% gain on the difficult WxBS benchmark accompanies major gains across diverse benchmarks, establishing RoMa as a new state-of-the-art.

  • Takeaways & Limitations

    RoMa’s largest gains occur on the most difficult benchmarks, supporting its robustness for dense feature matching.

  • Takeaways & Limitations

    RoMa optimizes dense feature matching indirectly rather than directly training for downstream two-view geometry, localization, or 3D reconstruction tasks.

Abstract

from arXiv · show

Feature matching is an important computer vision task that involves estimating correspondences between two images of a 3D scene, and dense methods estimate all such correspondences. The aim is to learn a robust model, i.e., a model able to match under challenging real-world changes. In this work, we propose such a model, leveraging frozen pretrained features from the foundation model DINOv2. Although these features are significantly more robust than local features trained from scratch, they are inherently coarse. We therefore combine them with specialized ConvNet fine features, creating a precisely localizable feature pyramid. To further improve robustness, we propose a tailored transformer match decoder that predicts anchor probabilities, which enables it to express multimodality. Finally, we propose an improved loss formulation through regression-by-classification with subsequent robust regression. We conduct a comprehensive set of experiments that show that our method, RoMa, achieves significant gains, setting a new state-of-the-art. In particular, we achieve a 36% improvement on the extremely challenging WxBS benchmark. Code is provided at https://github.com/Parskatt/RoMa

1. Introduction

RoMa targets robust dense feature matching by combining frozen DINOv2 coarse features with specialized ConvNet fine features, a multimodal anchor-probability decoder, and stage-specific losses. Extensive ablations and benchmark experiments show that RoMa sets a new state-of-the-art, including a 36% gain on WxBS.

  • Introduction: Features specialized exclusively for coarse matching or refinement significantly outperform features trained jointly for both tasks.This finding supports the architectural separation between coarse and fine feature roles.
  • Introduction: RoMa combines frozen DINOv2 coarse features with specialized ConvNet fine features to create a robust, precisely localizable feature pyramid.This addresses DINOv2’s lack of fine features while retaining its strong general features.
  • Introduction: RoMa uses a Transformer-based match decoder that predicts anchor probabilities instead of coordinates, enabling multimodal coarse matching distributions.The design reflects the observation that coarse matching is often multimodal, whereas conditional refinement is more likely unimodal.
  • Introduction: RoMa trains coarse global matches with regression-by-classification and refinement with robust regression, separating the two loss paradigms.The loss division is motivated by different coarse-stage and refinement-stage matching distributions.
  • Introduction: 36% gain on the difficult WxBS benchmark demonstrates RoMa’s strong performance on extremely challenging real-world matching cases.The paper reports extensive ablations and state-of-the-art experiments across diverse, competitive benchmarks.

2. Related Work

Related work spans keypoint-based, detector-free, and dense feature matching, alongside foundation-model visual features and robust regression-by-classification techniques.

  • Feature Matching: Feature matching has traditionally used keypoint detection and description, while newer methods perform coarse dense matching with refinement or estimate dense warps for matchable pixel pairs.Detector-free methods replace keypoint detection with coarse-scale dense matching and mutual-nearest-neighbor extraction before refinement.
  • Foundation Models: Self-supervised foundation models pretrained on large datasets have shown potential for learning general-purpose visual features, with self-supervised ViT features capturing more distinct information than supervised features.The passage attributes this observation to label-free self-distillation and also mentions iBOT’s masked-image modeling within self-distillation.
  • Robust Regression: Robust losses transition continuously between concentrated inlier and broad outlier distributions, while regression by classification casts regression as classification through techniques such as binning.The passage notes applications of robust losses in optical flow, robust smoothing, and loss functions.

3. Method

RoMa combines frozen DINOv2 coarse features with specialized ConvNet fine features, then decodes matches through anchor probabilities and refines them progressively to full resolution. Its design addresses the tradeoff between coarse robustness and precise localizability while reducing overfitting and computational cost.

  • 3.1 Preliminaries: Dense matching estimates a correspondence warp and matchability for every pixel, typically through global coarse matching followed by fine refinement.The model represents correspondences probabilistically as conditional and joint distributions over image coordinates.
  • 3.3 Match Decoder: RoMa replaces direct warp regression with a Transformer match decoder that predicts anchor probabilities, enabling multimodal correspondence distributions.The decoder improves the full approach particularly when paired with RoMa’s proposed loss, while refinement predicts residual warp and certainty offsets across multiple scales.
  • 3.2 Feature Encoders: RoMa decouples coarse and fine encoders, using DINOv2 for coarse features and specialized ConvNet features for precise localization.DINOv2 provides only stride-14 features, motivating a separate fine-feature encoder.
  • 3.2 Feature Encoders: Freezing DINOv2 reduces overfitting, improves robustness, and lowers computational and memory requirements, while encoder specialization significantly increases performance.The fixed representation prevents adaptation to the training set, and separate encoders specialize in coarse matching versus refinement.
  • 3.2 Feature Encoders: VGG19, although weaker than ResNet50 for coarse matching, significantly outperforms it as a fine feature, revealing tension between localizability and robustness.This motivates choosing fine features independently from the robust coarse encoder.

4. Experiments

Experiments evaluate RoMa through ablations and diverse two-view geometry benchmarks, showing improved performance over prior work across datasets while incurring a modest runtime increase over DKM.

  • Evaluation protocol: Performance is measured by PCK, the percentage of ground-truth correspondences whose estimated matches have EPE below a pixel threshold.The validation set uses random MegaDepth scene pairs with overlap > 0.
  • Ablations: The ablation progressively specializes fine features and replaces RN50 with VGG19 before adding the Transformer match decoder.Setup II removes weight sharing, while Setup III uses VGG19 fine features and further improves performance despite weaker coarse-feature results.
  • Two-view geometry benchmarks: 26% relative error reduction is achieved on IMC2022, while WxBS performance gains 36% over prior work.The evaluation samples 10,000 balanced correspondences for estimation and reports consistent improvements across benchmarks.
  • InLoc: RoMa shows large improvements over all previous approaches on the InLoc Visual Localization benchmark, setting a new state-of-the-art.Evaluation uses the HLoc pipeline and follows DKM for correspondence sampling.
  • Runtime: 7% runtime increase is observed for RoMa versus DKM, rising from 186.3 to 198.8 ms per pair at 560×560 resolution and batch size 8.The comparison runs on an RTX6000 GPU.

5. Conclusion

RoMa combines frozen DINOv2 coarse features with specialized ConvNet fine features, a transformer decoder predicting anchor probabilities, and regression-by-classification to produce robust, precisely localizable dense matching. The authors identify supervised correspondences and indirect downstream-task optimization as limitations, while noting pretrained features improve generalization and direct downstream training could improve performance.

  • Contributions: RoMa combines frozen DINOv2 coarse features with specialized ConvNet fine features to form a robust, precisely localizable feature pyramid.The coarse features provide robustness, while the specialized fine features improve localization.
  • Contributions: The tailored transformer match decoder predicts anchor probabilities instead of directly regressing coordinates.This decoder is presented as an additional performance improvement in RoMa.
  • Contributions: RoMa also uses an improved loss formulation based on regression-by-classification.The supplied passage truncates immediately after introducing this formulation.
  • Limitations: Supervised correspondences limit the amount of usable training data, while pretrained frozen foundation-model features improve generalization.The authors describe pretrained features as a remedy for the data limitation.
  • Limitations: Dense feature-matching training optimizes downstream two-view geometry, localization, or 3D reconstruction only indirectly.The authors suggest direct training on downstream tasks could improve performance.

A. Further Details on Frozen Feature Evaluation

Frozen feature evaluation uses an exponential cosine kernel with inverse temperature 10 and trains only a single linear layer using the proposed regression-by-classification loss.

  • Evaluation setup: 10 is the inverse temperature used for the exponential cosine kernel adopted from DKM.The kernel is used in frozen feature evaluation.
  • Evaluation setup: Only a single linear layer is trained because the backbone remains frozen.Training uses the same split and learning rates as the main experiments.
  • Evaluation setup: The evaluation uses the regression-by-classification loss proposed in Section 3.4.The setup also includes a qualitative example of estimated warps in Figure 5.

B. Further Architectural Details

RoMa combines multiscale fine features, projected DINOv2 patch features, and a Gaussian-process global matcher. Its refinement stack uses five convolutional refiners operating across multiple strides and incorporating local correlation and previous-warp information.

  • Encoders: Fine features use strides {1, 2, 4, 8}, dimensions {64, 128, 256, 512}, and linear-plus-batchnorm projections to {9, 64, 256, 512}.
  • Encoders: DINOv2 ViT-L-14 patch features exclude the cls token, use dimension 1024, and are projected with batchnorm to dimension 512.The patches have size 14.
  • Global Matcher: The global matcher uses DKM’s Gaussian-process match encoder with an exponential cosine kernel, inverse temperature 10, and a 512-dimensional embedding space.The Gaussian process predicts a posterior over embedded coordinates in the other image.
  • Refiners: Five refiners at strides {1, 2, 4, 8, 14} each contain 8 convolutional blocks with internal dimensions {24, 144, 569, 1137, 1377}.
  • Refiners: Refiners consume stacked feature maps, local correlations around the previous warp, and a linear encoding of that warp, producing a B × Hs × Ws × (2 + 1) tensor.The local-correlation sizes are {0, 0, 5, 7, 15}; the output contains the warp and a logit offset.

C. Qualitative Comparison on WxBS

On the WxBS benchmark, RoMa is qualitatively more robust than DKM, matching image pairs despite seasonal, illumination, scale, and viewpoint changes.

  • C. Qualitative Comparison on WxBS: RoMa is more robust than DKM on WxBS, where DKM fails on multiple image pairs.The comparison is shown qualitatively in Figure 6.
  • C. Qualitative Comparison on WxBS: RoMa matches pairs under seasonal, extreme illumination, extreme scale, and viewpoint changes.These challenging conditions include season changes, extreme illumination, and extreme scale and viewpoint variation.

D. Further Details on Metrics

This section defines the evaluation metrics for Image Matching Challenge 2022 and MegaDepth/ScanNet, including threshold-based pose accuracy and AUC-based evaluation.

  • Image Matching Challenge 2022: Image Matching Challenge 2022 mAA compares the estimated fundamental matrix with hidden ground truth using rotation error in degrees and translation error in meters.A pose is accurate only when it satisfies both thresholds.
  • Image Matching Challenge 2022: mAA averages accuracy across ten uniformly spaced rotation-and-translation threshold pairs and images balanced across scenes.
  • MegaDepth/ScanNet: MegaDepth/ScanNet evaluation uses an AUC metric.The supplied passage ends immediately after introducing this metric.

E. Further Details on Theoretical Model

The theoretical model of matchability has a straightforward connection to scale-space theory. Applying a Gaussian kernel jointly to the matching distribution yields the diffusion process used in the paper.

  • The model of matchability connects straightforwardly to scale-space theory.
  • Image scale-space is parameterized by a parameter s.
  • Applying a Gaussian kernel jointly to the matching distribution yields the paper’s diffusion process.

F. Further Details on Match Sampling

Because dense warps provide too many correspondences for sparse relative pose estimators, RoMa follows DKM by using balanced sampling based on the estimated match distribution.

  • Sampling motivation: Dense feature matching produces a warp and certainty, whereas downstream robust relative pose estimators assume sparse correspondences.Using all warp correspondences is possible in principle but not practical.
  • Balanced sampling: Balanced sampling reweights samples by the reciprocal of a KDE estimate of the match distribution pθ(xA, xB).This increases matches in less certain regions, which prior work showed improves performance.
  • Balanced sampling: RoMa follows DKM and uses balanced sampling to produce a sparse set of matches.The method addresses the computational cost of using all correspondences from the dense warp.
Loading 2305.15404v2…