PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation

Philipp Lindenberger, Paul-Edouard Sarlin, Ondrej Miksik, Marc Pollefeys

arXiv:2610.32976v1cs.CV

TL;DR

Sparse matching is efficient but detector-limited, while dense matching is robust but expensive, and learned matchers also need to reject visually similar non-overlapping pairs. FlexGlue matches sparse queries against dense context and jointly predicts matches and covisibility. Across the reported benchmarks, it achieves strong reconstruction and disambiguation results while maintaining competitive computational efficiency.

  • Problem

    Sparse matchers are limited by detector repeatability, dense matchers are costly, and learned matchers trained on covisible pairs can miss visually similar non-overlapping pairs.

    Source: §1 · p. 2

  • Method

    FlexGlue matches sparse query points against dense image context and jointly predicts subpixel correspondences and whether an image pair is covisible.

    Source: §1 · p. 2

  • Results

    FlexGlue achieves the highest accuracy on ETH3D and IMC2025, improves over prior work on all disambiguation benchmarks, InLoc, and HardMatch, while retaining competitive computational efficiency.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The reported results span two-view reconstruction, disambiguation, visual localization, and challenging two-view matching benchmarks.

    Source: §5 · p. 10

    Indoor hard negatives are difficult to collect at scale, and existing disambiguation training sets are outdoor, creating a domain gap for zero-shot indoor evaluation.

    Source: §3.5 · p. 6

Abstract

from arXiv · show

To estimate correspondences across images, existing approaches match either sparse interest points, which is highly efficient but limited by repeatability, or dense pixel grids, which is robust but expensive. Furthermore, these approaches are trained with only matching pairs and thus often fail to reject unrelated pairs that exhibit visual symmetries, which is equally important in applications like Structure-from-Motion (SfM). We bridge these gaps by introducing FlexGlue, a neural network for asymmetric sparse-to-dense matching and disambiguation. Given two images and coordinates of points in the first image, FlexGlue estimates the coordinates of the corresponding points in the second image while simultaneously rejecting the pair if non-matching. By searching for a limited set of correspondences among dense pixels, our approach is both fast and effective at both matching and disambiguation: it combines the sub-pixel accuracy of dense matching with the speed of sparse matching, while rejecting incorrect pairs with high precision. Experiments on multiple datasets show that FlexGlue pushes the Pareto frontier in relative pose estimation and significantly outperforms existing approaches for SfM and visual localization. FlexGlue is publicly available at https://github.com/cvg/flexglue .

1 INTRODUCTION

Image matching must balance sparse methods’ efficiency against dense methods’ robustness, while also rejecting non-overlapping pairs with visual similarities. FlexGlue matches sparse queries against dense context and jointly predicts correspondence and covisibility.

1 INTRODUCTION

  • Sparse matching is detector-limited, while dense matching is costly and semi-dense matching refines grid cells even when only sparse correspondences are needed.Sparse localization noise propagates into correspondences; dense and semi-dense methods also face resolution costs or complicate discrete tracks.Source: §1 · p. 2
  • Existing learned matchers are trained on covisible pairs, leaving them vulnerable to visually similar non-overlapping pairs that can cause catastrophic scene collapses.Separate covisibility models add architectural redundancy and cost.Source: §1 · p. 2
  • FlexGlue combines sparse-query efficiency with dense-context matching, producing subpixel correspondences while avoiding dependence on repeatable detections.A lightweight Transformer aggregates dense context, while correspondence readout and refinement operate only at query points.Source: §1 · p. 2
  • FlexGlue jointly learns to match covisible pairs and reject non-covisible pairs, and can learn disambiguation label-free from flip augmentations.Its early detection of negative pairs can accelerate matching by stopping inference on unmatchable pairs.Source: §1 · p. 2
  • The paper’s contributions are efficient sparse-to-dense matching and joint matching-disambiguation, with state-of-the-art accuracy on some two-view estimation and indoor localization benchmarks.It also reports that the joint formulation outperforms dedicated disambiguation models.Source: §1 · p. 3
  • FlexGlue reports higher throughput than disambiguation models, early stopping on unmatchable pairs, and improved SfM robustness on ETH3D and the CVPR 2025 Image Matching Challenge.The introduction also reports a large-margin improvement on its indoor doppelganger benchmark, built from LaMAR with hard-mined negatives.Source: §1 · p. 3

2 RELATED WORK

Prior methods trade efficiency, global context, robustness, and discrete-track compatibility across sparse, dense, semi-dense, and sparse-to-dense matching. FlexGlue aggregates context over dense grids while reading out correspondences at sparse queries, and combines matching with covisibility prediction.

2 RELATED WORK

  • Sparse matchers are efficient and yield discrete tracks, but their accuracy is bounded by detector repeatability and localization noise.Their attention also aggregates context only at keypoints, limiting scene understanding in textureless or repetitive regions.Source: §2 · p. 3
  • Dense matching is robust but refines correspondences across pixels, while semi-dense matching anchors them to grid cells and complicates discrete-track extraction.FlexGlue instead aggregates context on a coarse grid and reads out subpixel correspondences at sparse queries, with cost scaling linearly in query count.Source: §2 · p. 3
  • Earlier sparse-to-dense approaches lack global context, use slow recursive refinement, or restrict context in one view to sparse keypoints; FlexGlue uses both full dense grids.Source: §2 · p. 3
  • Separate disambiguation models add overhead, generalize poorly beyond training data, and were evaluated only on outdoor landmarks; FlexGlue predicts matches and covisibility in one model.It exits early on unmatchable pairs.Source: §2 · p. 3
  • Figure 3 illustrates a covisible pair and visually similar doppelgangers that FlexGlue identifies early, stopping matching for the latter.The figure shows learned-feature PCA overlays, pre-filtering correspondences, and covisibility scores after 3 and 9 layers.Source: Figure 3 · p. 4

3 METHOD

FlexGlue estimates sparse-to-dense correspondences and matchability from image-pair context, with a pair-level covisibility score for rejecting non-overlapping images. Its method combines coarse matching, local refinement, joint supervision, and an indoor doppelganger benchmark.

3 METHOD

  • FlexGlue predicts bidirectional sparse-to-dense correspondences, point matchability, and whether two images observe the same scene.Correspondences between keypoints are derived geometrically from the predicted coordinates.Source: §3 · p. 4

3.1 COARSE MATCHING

  • A frozen pretrained CNN supplies multiscale features that become dense tokens, which Transformer layers update through within-image self-attention and across-image cross-attention.A query readout and refinement head produce correspondence coordinates, confidence, matchability, and subpixel accuracy.Source: §3 · p. 4, §3.1 · p. 4
  • Windowed readout renormalizes correspondence probabilities around the nearest cell and interpolates cell centers, improving depth-discontinuity accuracy while avoiding cell quantization.The method uses 1.5-entmax to assign zero probability to unlikely cells and improve subpixel accuracy.Source: §3.1 · pp. 4–5

3.2 LOCAL REFINEMENT

  • A lightweight MLP predicts a sub-cell offset from interpolated query and coarse-match tokens, refining each correspondence with computation that scales linearly with query count.Refinement shifts correspondences by 0.6 px on average.Source: §3.2 · p. 5

3.3 VISUAL DISAMBIGUATION

  • Per-token covisibility votes are supervised at every layer, enabling early stopping when the pair score indicates confident non-overlap.The threshold decay makes early layers stop only on confident non-covisible pairs.Source: §3.3 · p. 5

3.4 SUPERVISION

  • Training samples randomly displaced queries from a uniform grid and supervises correspondence, matchability, refinement, and pair covisibility with task-specific losses.The coarse loss targets bilinear weights for the four surrounding cells; matching and refinement losses apply only to visible queries.Source: §3.4 · p. 6

3.5 INDOOR DOPPELGANGERS BENCHMARK

  • The indoor doppelganger benchmark uses LaMAR scenes and retrieval-mined, visually similar non-overlapping pairs to test disambiguation beyond outdoor settings.Pairs are labeled from mesh-based overlap; each scene contains 1,000 pairs, split evenly between positive and negative examples. Existing disambiguation training sets are outdoor, so methods are evaluated zero-shot on indoor data and show substantial performance drops.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments evaluate FlexGlue across two-view pose estimation, visual disambiguation, multi-view reconstruction, visual localization, and component ablations. Results report strong benchmark performance alongside competitive computational efficiency.

4.1 IMPLEMENTATION DETAILS

  • FlexGlue is trained on eight datasets for 50 epochs, using 150k samples per epoch and four H200 GPUs, for about four GPU-days.The same model can be trained on four RTX 4090 GPUs in about 1.5 days with gradient checkpointing.Source: §4.1 · p. 7

4.2 RELATIVE POSE ESTIMATION

  • In sparse-to-dense mode, FlexGlue is competitive with RoMav2 while substantially faster, and it achieves the highest accuracy on HardMatch.Sparse matching performs similarly to LoMa, while dense correspondence estimation improves relative pose results but lacks robustness on HardMatch.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 TWO-VIEW ESTIMATION

  • With 2048 DaD keypoints, FlexGlue has highest accuracy on every Table 2 benchmark except IMC22; gains are largest on aerial imagery, and higher resolution consistently helps.Additional RoMav2 refinement reaches 67.3 AUC@10 px on SatAst and 66.6 AUC@10◦ on TerraSky3D.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 VISUAL DISAMBIGUATION

  • FlexGlue achieves the highest accuracy on ViSym-F while being over 4× faster than DG++; it performs on par with DG on in-distribution data and excels on out-of-distribution LaMAR.It outperforms literature on LaMAR even without manually annotated doppelganger training pairs.Source: §4.4 · p. 8

4.5 MULTI-VIEW ESTIMATION

  • On ETH3D, matching plus disambiguation improves pose accuracy; on IMC2025, FlexGlue beats the winning MASt3R entry at 4096 keypoints and the leaderboard by 8 points at 10k.Early stopping hardly impairs accuracy while increasing throughput, and still outperforms RoMav2+DG++ at a fraction of its runtime.Source: §4.5 · p. 9, Table 4 · p. 9, Table 5 · p. 9

4.6 VISUAL LOCALIZATION

  • On InLoc, FlexGlue is as robust and precise as dense matchers, achieves the best results at 5 of 6 thresholds, and has much higher throughput.Source: §4.6 · pp. 9–10, Table 6 · p. 10

4.7 ABLATIONS

  • Ablations find that 1.5-entmax most improves robustness and refinement improves subpixel precision at negligible cost; covisibility-only training loses up to 9.3 AUROC points.Matching supervision helps disambiguation generalize, and additional indoor negatives mined on NaverLabs further improve LaMAR.Source: §4.7 · p. 10, Table 7 · p. 10, Table 8 · p. 10

5 CONCLUSION

FlexGlue combines sparse-to-dense correspondences with visual disambiguation to balance sparse matchers’ efficiency and dense matchers’ robustness. The conclusion reports strong results across multi-view reconstruction, disambiguation, visual localization, and challenging two-view matching.

5 CONCLUSION

  • FlexGlue achieves the highest accuracy on the ETH3D and IMC2025 multi-view reconstruction benchmarks.Source: §5 · p. 10
  • Its sparse-to-dense architecture combines uniform grid attention, localized query points, and visual disambiguation while maintaining competitive computational efficiency.Source: §5 · p. 10

A APPENDIX

  • The appendix adds inference-time and keypoint-detector results, LaMAR-DG benchmark construction, implementation and evaluation details, and qualitative examples.Source: A APPENDIX · p. 11

B.1 INFERENCE TIME & LAYER ABLATION

FlexGlue’s fixed-grid context aggregation leaves only a lightweight readout to scale with query count, making its matching cost linear. Strong matching and disambiguation performance appears early in the network, and the model can use different keypoint detectors without retraining.

B.1 INFERENCE TIME & LAYER ABLATION

  • FlexGlue’s readout scales linearly with query count, unlike quadratic sparse matchers, and it is faster than sparse matchers at large keypoint budgets and faster than RoMav2.Strong matching and disambiguation performance appears early, supporting early exit on negative pairs; Fig. 5 shows runtime increasing linearly with layer count.Source: §B.1 · p. 11, Figure 5 · p. 11

B.2 KEYPOINT DETECTORS

  • The same model can use SuperPoint, DaD, or RaCo keypoints without retraining; RaCo gives the most accurate ETH3D and IMC2025 reconstruction results.All detectors perform similarly on two-view HardMatch.Source: §B.2 · pp. 11–12

B.3 DETAILED RESULTS ON LAMAR-DG

FlexGlue remains robust on difficult LaMAR-DG pairs, including cross-day and cross-device comparisons where DG++ degrades. The reported cross-device AP is affected by a positive-negative imbalance.

B.3 DETAILED RESULTS ON LAMAR-DG

  • DG++ degrades strongly on hard, cross-day, and cross-device pairs, while FlexGlue remains robust across those groups.Source: Figure 6 · p. 12
  • FlexGlue improves most over DG++ on hard, cross-day, and cross-device LaMAR-DG pairs.Difficulty bins are tertiles of MegaLoc similarity per label, so hard negatives are highly similar and hard positives are dissimilar.Source: §B.3 · p. 12, Figure 6 · p. 12
  • High AP on cross-device pairs reflects an imbalance of positives and negatives caused by hard-negative mining, not just pairwise matching performance.Source: §B.3 · p. 12

C BENCHMARKS

LaMAR-DG uses mesh-derived overlap to curate positive and negative pairs, while ViSym-F corrects noisy test labels. The section also reports how detector choice affects reconstruction and two-view matching.

C.1 CURATION OF LAMAR-DG

  • LaMAR-DG positives have 10–80% overlap in both directions, while negatives have zero overlap; overlap is computed from mesh-rendered depth with occlusions accounted for.The benchmark uses map sessions from the CAB, HGE, and LIN buildings captured with HoloLens and iPhone devices.Source: §C.1 · p. 12
  • RaCo yields the most accurate reconstructions, while all listed detectors perform similarly on two-view matching.FlexGlue works with any detector.Source: Table 9 · p. 12
  • Hard LaMAR-DG negatives are selected from visually similar non-overlapping views, and pairs are grouped into easy, medium, and hard similarity tertiles.The benchmark has 500 positive and 500 negative pairs per building.Source: §C.1 · p. 13

C.2 VISYM MANUAL LABEL CLEANING

  • ViSym-F manually relabels the 3,180 test pairs because GPS-derived labels are noisy, often marking covisible pairs negative; ambiguous cases are excluded and the retained set is balanced.Most corrections turn negatives into positives; the original labels also miss covisible pairs under strong viewpoint changes and mark some distinct but similar facades as covisible.Source: §C.2 · p. 13, Figure 7 · p. 13, Table 10 · p. 13

D.1 TRAINING DATA

Training uses multiple datasets and augments examples with hard negatives and flipped-image negatives. The section also describes the supervision losses and three ways to convert FlexGlue predictions into matches.

D.1 TRAINING DATA

  • Final fine-tuning mixes Doppelgangers with other datasets and turns half of the latter pairs into negatives by flipping one image.The indoor variant additionally mines pairs from four NaverLabs floors and holds out a fifth for validation.Source: §D.1 · p. 13, Table 11 · p. 14

D.2 SUPERVISION

  • The coarse readout uses an α-entmax Fenchel–Young loss because cross-entropy is undefined when entmax assigns zero probability; it remains finite and is minimized when predicted probabilities match the ground-truth bilinear weights.The refinement is supervised with a generalized Charbonnier loss, and losses are averaged uniformly over layers.Source: §D.2 · p. 14

D.3 MATCHING MODES

  • Sparse-to-dense mode matches each query keypoint to its predicted correspondence, while sparse mode forms point-to-point assignments from predictions in both directions.The sparse assignments use symmetric distances and mutual nearest neighbors with a 3 px threshold.Source: §D.3 · p. 14
  • Dense mode samples up to 5,000 correspondences from dense predictions in both images, weighted by confidence.All modes discard low-confidence correspondences or those whose cycle error exceeds 3 px; the default confidence threshold is 0.1.Source: §D.3 · p. 14

E EVALUATION DETAILS

The evaluation details specify protocols and baselines for relative pose, multi-view estimation, visual disambiguation, and visual localization. They also describe the benchmark sets and the localization pipeline.

E.1 RELATIVE POSE ESTIMATION

  • Relative pose evaluation follows baseline protocols, estimates essential matrices with OpenCV RANSAC at a 0.5 px inlier threshold, and reports pose-error AUC at 5°/10°/20°.HardMatch reports mean average accuracy at 10 px from epipolar error on ground-truth correspondences.Source: §E.1 · p. 15
  • For sparse matching, all methods detect up to 4096 keypoints; pairing dense RoMa v2 with SuperPoint uses only 2048 keypoints to remain roughly comparable to sparse methods limited by keypoint repeatability.Sparse-to-dense benchmarks include CAPS and S2DNet, which perform exhaustive searches over dense features, and RCM, a deep sparse-to-dense matcher.Source: §E.1 · p. 15

E.2 MULTI-VIEW ESTIMATION

  • ETH3D multi-view comparisons include sparse matchers, LoMa-B, MASt3R-SfM with DG++ disambiguation, and COLMAP via hloc with exhaustive pair selection unless stated otherwise.Some baselines are strengthened with Doppelgangers++ disambiguation.Source: §E.2 · p. 15

E.3 VISUAL DISAMBIGUATION

  • Visual disambiguation is evaluated on Doppelgangers, manually cleaned ViSym, and LaMAR-DG, against DG and DG++ and a FlexGlue variant trained without labeled doppelganger pairs.The benchmarks cover 4,659 phototourism pairs, 3,180 phone-captured street-scene pairs, and 3,000 indoor and outdoor pairs, respectively.Source: §E.3 · p. 15

E.4 VISUAL LOCALIZATION

  • InLoc localization retrieves the top 40 database images, establishes 2D–2D matches, lifts them to 2D–3D using database depth, and estimates absolute pose with PnP.The accompanying figure shows more accurate sparse-to-dense correspondences, especially with large scale changes and low-texture areas.Source: §E.4 · pp. 15–16, Figure 8 · p. 16

F LIMITATIONS •

The limitations concern correlated matching errors, fine accuracy on easy pairs, later-layer attention, and accuracy near depth discontinuities.

F LIMITATIONS •

  • Correlated query errors from learned coarse-grid interpolation can reduce relative-pose accuracy, which assumes point errors are independent.Many errors lie along the epipolar line, suggesting FlexGlue may learn noisy epipolar geometry.Source: F LIMITATIONS • · p. 16
  • RoMa v2 produces more accurate matches than FlexGlue on easy pairs, attributed to its dense refiner.The authors suggest stronger refinement or a dedicated sparse-to-dense refiner with finer features could improve fine accuracy.Source: F LIMITATIONS • · p. 16
  • In later layers, coarse attention often improves little and cross-attention becomes very local.Source: F LIMITATIONS • · p. 16
  • At depth discontinuities, accuracy falls from over 90% to around 75% PCK@5px because interpolation can place correspondences on the wrong side of a depth jump.Windowed readout partially mitigates this on the target side; finer features may further improve accuracy.Source: F LIMITATIONS • · p. 16

G QUALITATIVE RESULTS

The qualitative results visualize FlexGlue’s learned structural cues, scene clustering, dense matches, and visual disambiguation. They also show refinement behavior and failure cases across matching comparisons.

G QUALITATIVE RESULTS

  • FlexGlue’s tokens highlight covisible areas and learn structural priors including lines, perspective, and planarity.The visualization shows the first three principal components of tokens at layer 9.Source: Figure 9 · p. 17
  • Scene graphs on IMC2025 form clean landmark clusters, though one incorrect image pair passes the visual-disambiguation filter.Colors indicate predicted clusters and edges indicate retained image pairs.Source: Figure 10 · p. 18
  • Dense matching has errors near depth discontinuities that refinement addresses; deeper layers also refine matches and fix some such errors.FlexGlue’s dense correspondences are fairly accurate despite supervision only at sparse queries.Source: Figure 11 · p. 18, Figure 12 · p. 19
  • Visual disambiguation distinguishes hard covisible pairs but can mislabel pairs under extreme appearance or viewpoint changes or missing context.An example failure is a wrong facade.Source: Figure 13 · p. 19
  • Qualitative match comparisons report match counts and precision at 3 px, using 2048 keypoints for LoMa-G and 1024 for RoMav2 and FlexGlue.The comparisons include two pairs per dataset.Source: Figure 14 · p. 20
  • Failure examples show all methods struggling with extreme rotation and self-similarity, while FlexGlue matches the wrong repeated facade instance where RoMav2 succeeds.The examples span TartanAir, Hypersim, and MegaDepth.Source: Figure 14 · p. 21
Loading 2610.32976v1…