PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniGlue: Generalizable Feature Matching with Foundation Model Guidance

Hanwen Jiang, Arjun Karpur, Bingyi Cao, Qixing Huang, Andre Araujo

arXiv:2405.12979v1cs.CV

TL;DR

Learnable image matchers can perform well on conventional benchmarks but often generalize poorly to new image domains, limiting their potential for real-world use. OmniGlue combines foundation-model guidance with position-guided attention that separates spatial information from matching descriptors. Across six novel domains, it reports average relative gains of 20.9% over SuperGlue and 9.5% over LightGlue for pose and registration accuracy at the tightest thresholds.

  • Problem

    Learnable matching methods often lose performance on out-of-domain images, while collecting high-quality correspondence annotations for every domain is costly.

    Source: §1 · pp. 1–2

  • Method

    OmniGlue uses broad foundation-model visual knowledge to guide feature propagation and a keypoint-position-guided attention mechanism that separates position information from matching descriptors.

    Source: §1 · p. 2, §3.2 · pp. 3–4

  • Results

    Across six novel domains, OmniGlue reports average relative gains of 20.9% over SuperGlue and 9.5% over LightGlue for pose and registration accuracy at the tightest thresholds.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    OmniGlue improves cross-domain generalization and can also be adapted to a target domain using a limited amount of data.

    Source: §1 · p. 2, §5 · p. 8

    An experiment found that combining the two features reduced performance, and how to merge them effectively remains an open problem, likely due to DINO’s coarse granularity.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

The image matching field has been witnessing a continuous emergence of novel learnable feature matching techniques, with ever-improving performance on conventional benchmarks. However, our investigation shows that despite these gains, their potential for real-world applications is restricted by their limited generalization capabilities to novel image domains. In this paper, we introduce OmniGlue, the first learnable image matcher that is designed with generalization as a core principle. OmniGlue leverages broad knowledge from a vision foundation model to guide the feature matching process, boosting generalization to domains not seen at training time. Additionally, we propose a novel keypoint position-guided attention mechanism which disentangles spatial and appearance information, leading to enhanced matching descriptors. We perform comprehensive experiments on a suite of $7$ datasets with varied image domains, including scene-level, object-centric and aerial images. OmniGlue's novel components lead to relative gains on unseen domains of $20.9\%$ with respect to a directly comparable reference model, while also outperforming the recent LightGlue method by $9.5\%$ relatively.Code and model can be found at https://hwjiang1510.github.io/OmniGlue

1. Introduction

OmniGlue is designed to improve learnable image matching’s transfer to domains not seen during training, where existing methods can lose performance. It combines foundation-model guidance with position-guided attention and reports gains across novel domains.

1. Introduction

  • Learnable matchers can perform well on conventional benchmarks yet drop substantially on out-of-domain images, while high-quality correspondence annotations are costly to collect for every domain.The introduction cites object-centric and aerial captures as examples of out-of-domain data.Source: §1 · pp. 1–2
  • OmniGlue uses broad visual knowledge from DINO to guide inter-image feature propagation, downweighting irrelevant keypoints and encouraging fusion from potentially matchable regions.The foundation model provides coarse guidance when a specialized matcher struggles with domain shift.Source: §1 · p. 2
  • OmniGlue’s keypoint-position-guided attention is designed to avoid specializing too strongly in the training distribution of keypoints and relative-pose transformations, supporting generalization across diverse visual domains.Its strategy for leveraging keypoint positional encoding boosts cross-domain transfer by up to 6.1% (14.9% relatively).Source: §1 · p. 2
  • Across 7 datasets from varied domains, comprehensive experiments show OmniGlue’s strong improvements, with relative gains of 20.9% on average in all novel domains.Source: §1 · p. 2
  • Foundation-model guidance improves pose estimation accuracy by up to 5.8% (14.4% relatively) on domains not observed during training.The reported contribution attributes this gain to leveraging broad visual knowledge in the learnable matching process.Source: §1 · p. 2

2. Related Work

Related work describes a shift from hand-crafted to learnable local matching, while emphasizing that learned methods can specialize to training domains. OmniGlue builds on sparse matching and uses foundation-model guidance and separated positional information to target transfer across domains.

Generalizable Local Feature Matching

  • Hand-crafted methods such as SIFT, SURF, and ORB remain in use across domains because many learnable methods are specialized to data-rich training domains and generalize poorly elsewhere.Learnable matchers improve performance over hand-crafted systems but can make the full matching pipeline more domain-specific.Source: §2 · p. 2

Sparse Learnable Matching

  • Sparse methods such as SuperGlue entangle local descriptors with keypoint positions, making matching overly dependent on learned positional patterns.OmniGlue separates these during feature propagation to reduce reliance on position patterns across domains.Source: §2 · p. 2

(Semi-)Dense Learnable Matching

  • Some dense methods generalize worse on new domains than sparse methods, so OmniGlue focuses on sparse matching with domain-agnostic local descriptors.Dense methods can benefit from end-to-end learning and perform better in the training domain.Source: §2 · pp. 2–3

Matching with Additional Image Representations

  • Foundation models provide coarse patch-level matching signals, but their matching accuracy is limited on keypoints or regions without strong semantic information.OmniGlue instead uses DINOv2 features to identify potentially related regions and guide attention-based feature refinement.Source: §2 · p. 3, Figure 2 · p. 3

3. OmniGlue

OmniGlue combines SuperPoint’s fine-grained features with DINOv2’s broad visual knowledge to guide keypoint matching. Its information propagation uses DINOv2-guided inter-image graphs and separates positional context from the descriptors used for matching.

3.1. Model Overview

  • OmniGlue extracts SuperPoint and DINOv2 features, builds intra- and inter-image keypoint graphs, propagates information with self- and cross-attention, then applies optimal matching layers.DINOv2 provides coarse similarity guidance, while SuperPoint provides fine-grained matching features.Source: §3.1 · p. 3

3.2. OmniGlue Details

  • DINOv2 similarities select the top half of candidate keypoints for each inter-image connection, pruning the dense pairwise graph without forcing matches to stay within that candidate set.The authors note DINOv2 may be incorrect in some cases.Source: §3.2 · p. 4
  • Intra-image graphs densely connect keypoints within each image, while inter-image graphs guide cross-attention; both DINOv2 and keypoint positions provide guidance during propagation.The propagation module uses one self-attention layer on intra-image graphs and one cross-attention layer on inter-image graphs per block.Source: §3.2 · p. 4
  • Position-guided attention uses positional features to compute attention weights but keeps the attention update composed only of local descriptor components.This lets the model use spatial context without incorporating positional information into the final descriptors used for matching.Source: §3.2 · pp. 4–5

3.3. Comparison Against SuperGlue and LightGlue

  • Unlike SuperGlue, OmniGlue separates positional and appearance features; unlike LightGlue, it uses foundation-model guidance to support transfer to unseen image domains.LightGlue also removes positional dependence from updated descriptors, but uses rotary positional encodings only in self-attention layers.Source: §3.3 · p. 5

4. Experiments

OmniGlue is evaluated across varied datasets and matching tasks, with emphasis on zero-shot generalization beyond training domains. Its strongest reported comparison is an average 20.9% relative gain over SuperGlue and 9.5% over LightGlue on six novel domains, alongside ablations of its proposed components.

4.1. Experimental Setup

  • The evaluation spans synthetic homography, outdoor scenes, object-centric images, indoor scenes, and aerial pairs, covering correspondence estimation, camera pose estimation, and aerial image registration.The datasets include both real and rendered imagery, varied camera baselines, and cases with different backgrounds.Source: §4.1 · pp. 5–6

4.2. Results

  • On six novel domains, OmniGlue achieves average relative gains of 20.9% over SuperGlue and 9.5% over LightGlue, with larger gains over LightGlue on GSO-Hard, NAVI-Wild, and DeepAerial.Table 3 reports in-domain AUC and out-of-domain pose accuracy/PCK; the prose specifies the averaged gains for pose and registration accuracy at the tightest thresholds.Source: Table 3 · p. 7, §4.2 · p. 7
  • On MegaDepth-1500, OmniGlue achieves a 12.3% relative gain over SuperGlue in pose AUC at 5° while remaining comparable to LightGlue.For transfer from SH100 to SH200, OmniGlue also improves precision by 12% and recall by 14% over SuperGlue; from SH200 to MegaDepth, it improves recall by 15%.Source: §4.2 · pp. 6–7
  • With limited GSO fine-tuning data, OmniGlue improves on both test subsets using 10 training instances and achieves an approximately 10% relative gain over SuperGlue across experiments.The Table 4 setup uses sneaker instances for training and evaluates on all 17 GSO categories.Source: §4.2 · pp. 7–8, Table 4 · p. 8

4.3. Ablation Study and Insights

  • Ablations indicate that foundation-model guidance improves out-of-domain generalization, while position-guided attention improves both in-domain and out-of-domain performance.Removing positional embeddings improves generalization but reduces in-domain performance; merging DINO features with local descriptors reduces performance, and the effective merging approach remains open.Source: §4.3 · p. 8, Table 5 · p. 8
  • For DINO guidance, selecting the top half of keypoints in the other image to build inter-image graphs is reported as the optimal choice.Applying DINO guidance to both intra- and inter-image graphs performs worse than the reported approach.Source: §4.3 · p. 8

5. Conclusions and Future Work

OmniGlue combines foundation-model guidance with a disentangled keypoint-position attention module, and the authors report improved cross-domain generalization. They also identify limited-data fine-tuning as an adaptation route and unannotated target-domain data as future work.

5. Conclusions and Future Work

  • OmniGlue combines broad foundation-model visual knowledge for graph building with an attention module that addresses descriptor-position entanglement.The authors describe these as its novel components for generalizable matching.Source: §5 · p. 8
  • OmniGlue outperforms prior work with better cross-domain generalization.Source: §5 · p. 8
  • OmniGlue can be adapted to a target domain using a limited amount of fine-tuning data.Source: §5 · p. 8
  • The authors propose exploring unannotated target-domain data and better architectures and data strategies toward a foundational matching model.Source: §5 · p. 8

A. Additional Model Details

OmniGlue is trained for 750,000 iterations, and the implementation uses DINOv2 features extracted from resized images with a ViT-14-base backbone. The model also uses an improved positional embedding scheme from LFM-3D.

A. Additional Model Details

B. Target Domain Visualization

The target-domain examples span object and scene imagery, including Google Scanned Objects, NAVI, ScanNet-1500, and DeepAerial. The paper characterizes this range as challenging for generalizable image matching.

B. Target Domain Visualization

C. Area Under Curve (AUC) Pose Results

The appendix reports pose AUC performance in Table 6 and says pose accuracy is reported in the main paper because out-of-domain performance is limited.

C. Area Under Curve (AUC) Pose Results

D. Latency analysis.

OmniGlue’s added modules are reported to preserve latency relative to SuperGlue, with DINOv2’s extra computation balanced by graph-pruning savings. Empirically, OmniGlue runs at a similar frame rate to SuperGlue without graph pruning.

D. Latency analysis.

  • DINOv2 adds computation, but its features prune graphs and reduce computation; OmniGlue’s novel modules do not hurt latency compared with SuperGlue.Source: D. Latency analysis · p. 9
  • At a 630-pixel input width and typical 16:9 aspect ratio, the estimated DINOv2 computation and graph-pruning savings are balanced.The analysis simplifies the added and saved computation to O(n2k2).Source: D. Latency analysis · p. 9
  • OmniGlue runs at a similar frame rate to SuperGlue without graph pruning, measured on an NVIDIA A40 GPU with FlashAttention.The result was reproduced using Glue-Factory.Source: D. Latency analysis · p. 9

E. Additional Qualitative Results

Figure 7 compares OmniGlue with MNN and SuperGlue across image pairs and feature configurations. The authors report improved OmniGlue matches for pairs with significant viewing-condition changes across varied domains.

E. Additional Qualitative Results

Related papers

Loading 2405.12979v1…