PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperGlue: Learning Feature Matching with Graph Neural Networks

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich

arXiv:1911.11763v2cs.CV

TL;DR

Reliable feature correspondences are difficult under viewpoint, lighting, occlusion, blur, and texture changes, yet they are essential for geometric vision. SuperGlue learns context-aware matching with attention-based graph neural networks and differentiable optimal transport, achieving highly accurate indoor and outdoor pose estimation while running in real time.

  • Problem

    Estimating feature correspondences for 3D reconstruction and camera pose remains challenging under viewpoint, lighting, occlusion, blur, and texture changes.

  • Method

    SuperGlue learns a matching process over pre-existing local features using attention-based graph neural networks and differentiable optimal transport for partial assignments.

  • Results

    SuperGlue achieves significant improvement over existing approaches and highly accurate relative pose estimation on extreme wide-baseline indoor and outdoor image pairs, with 84.9% matching precision outdoors.

  • Takeaways & Limitations

    SuperGlue replaces handcrafted matching heuristics with a unified learnable middle-end that runs in real time and works with both classical and learned features.

Abstract

from arXiv · show

This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at https://github.com/magicleap/SuperGluePretrainedNetwork.

1. Introduction

SuperGlue reframes local-feature matching as learned partial assignment between two image feature sets, using attention and differentiable optimal transport to improve correspondence estimation for challenging geometric vision tasks. Combined with SuperPoint, it outperforms handcrafted and learned alternatives on indoor and outdoor pose estimation.

  • Motivation: Reliable image correspondences are essential for estimating 3D structure and camera poses in SLAM and SfM, but viewpoint, lighting, occlusion, blur, and limited texture make matching difficult.These correspondences are obtained by matching local features in a process known as data association.
  • Contribution: SuperGlue learns matching directly from pre-existing local features rather than learning task-agnostic features followed by simple matching heuristics.In SLAM, it serves as a learned middle-end between feature extraction and pose estimation or bundle adjustment.
  • Method: The method formulates correspondence estimation as a partial assignment between two local-feature sets, relaxed to differentiable optimal transport with costs predicted by a graph neural network.This formulation explicitly accommodates partial point visibility and occlusion.
  • Method: Self- and cross-attention enable SuperGlue to aggregate intra-image and inter-image context while solving the assignment optimization problem.The architecture is inspired by the Transformer.
  • Results: SuperGlue advances the state of the art in indoor and outdoor pose estimation when combined with SuperPoint, outperforming handcrafted matchers and learned inlier classifiers.The authors present this combination as a step toward end-to-end deep SLAM.

2. Related work

Prior local-feature matching pipelines combine hand-designed detection, description, nearest-neighbor matching, filtering, and geometric estimation, while recent learned methods improve descriptors or filter NN-derived matches. SuperGlue instead connects learned assignment costs, graph matching, and attention-based set aggregation to address the limitations of these approaches.

  • Classical matching pipelines detect points, compute descriptors, use nearest-neighbor search, filter incorrect matches, and estimate a geometric transformation with heuristics such as SIFT, Lowe’s ratio test, mutual checks, and neighborhood consensus.
  • Recent deep-learning methods learn sparse detectors, local descriptors, wider-context features, or inlier–outlier filters, but their matches are still estimated by nearest-neighbor search [18] [19] [32] [20] [33] [7].
  • Graph matching is commonly posed as an NP-hard quadratic assignment problem, while handcrafted local-feature costs rely on complex, brittle heuristics; Caetano et al. [9] learn costs only for a simpler linear assignment with a shallow model.
  • Attention supports permutation-aware set aggregation by focusing on specific elements and attributes, combining global and data-dependent local context unlike uniform pooling, normalization, or fixed-neighborhood methods [15] [60] [33] [32] [26].

3. The SuperGlue Architecture

SuperGlue predicts a partial, confidence-valued correspondence assignment between two sets of local features using an attentional graph neural network and an optimal matching layer. The architecture aggregates context within and across images, then uses differentiable assignment with reciprocity and permutation-equivariant inductive biases.

  • 3. The SuperGlue Architecture: SuperGlue predicts a partial soft assignment matrix with confidence values for correspondences between two sets of local features.The assignment is defined over keypoints from two images and is the network’s primary prediction target.
  • 3.1 Attentional Graph Neural Network: The attentional graph neural network combines each keypoint’s appearance and position, then alternates self- and cross-image message passing to produce matching descriptors.Self edges connect keypoints within an image, while cross edges connect keypoints across images; attention dynamically weights the aggregated messages.
  • 3.2 Optimal Matching Layer: The optimal matching layer scores every candidate pair through the inner product of matching descriptors and solves the resulting assignment problem under partial-assignment constraints.Using descriptor similarities avoids constructing a separate representation for all M × N candidate matches.
  • 3. The SuperGlue Architecture: The architecture is equivariant to permutations of keypoints and images, while its optimal-transport formulation softly enforces reciprocal matches during training.These properties provide an inductive bias aligned with the symmetry of matching and embed mutual consistency into learning.

4. Implementation details

SuperGlue is designed to work with local feature detectors and descriptors, particularly SuperPoint, and uses a 12M-parameter attention architecture with differentiable optimal transport for real-time matching. Training performs SuperPoint processing on the fly with augmented keypoints, while inference retains matches above a confidence threshold.

  • Feature integration: SuperGlue can combine with any local feature detector and descriptor, but works particularly well with SuperPoint [18], whose sparse, repeatable keypoints enable efficient matching.Visual descriptors are bilinearly sampled from SuperPoint’s semi-dense feature map.
  • Feature integration: For fair matcher comparisons, the visual descriptor network is not trained with SuperGlue unless explicitly stated, and inference retains matches using a confidence threshold of 0.2.The confidence threshold is applied at test time.
  • Architecture details: SuperGlue uses D = 256 representations, 9 alternating self- and cross-attention layers with 4 heads, and T = 100 Sinkhorn iterations.The PyTorch implementation contains 12M parameters.
  • Architecture details: 69 ms per forward pass, or 15 FPS, enables real-time matching for an indoor image pair on an NVIDIA GTX 1080 GPU.The reported runtime is for the stated indoor image-pair setting.
  • Training details: During training, SuperPoint detection and description run on the fly in batches, with random keypoints added to support efficient batching and robustness.Additional training details are provided in Appendix E.

5. Experiments

Across homography and indoor/outdoor pose-estimation experiments, SuperGlue achieves high-quality correspondences and consistently outperforms handcrafted and learned matchers. Ablations show that graph reasoning, attention, positional encoding, depth, and end-to-end descriptor training contribute to performance.

  • Homography estimation: 98% recall and high precision enable SuperGlue to recover nearly all possible homography matches while suppressing most outliers.Its correspondences are strong enough that DLT performs better than RANSAC, unlike competing outlier-rejection methods that cannot improve on nearest-neighbor matching.
  • Indoor pose estimation: Indoor evaluation targets texture-poor, self-similar scenes with complex 3D geometry and large viewpoint changes, using pose accuracy, precision, and matching score metrics.Relative poses are estimated with RANSAC, and pose accuracy is summarized by AUC at 5°, 10°, and 20° thresholds.
  • Indoor and outdoor pose estimation: SuperGlue outperforms handcrafted and learned matchers for indoor and outdoor pose estimation with both SIFT and SuperPoint features.It achieves higher pose accuracy, precision, and matching scores across the reported evaluations.
  • Outdoor pose estimation: 84.9% precision in outdoor matching reinforces SuperGlue’s ability to produce highly reliable correspondences across challenging real-world scenes.SuperGlue outperforms all baselines at every relative-pose threshold for both SuperPoint and SIFT.
  • Ablation study: The optimal matching layer improves over nearest-neighbor matching, while the graph neural network explains most gains; cross-attention, positional encoding, and depth further improve results.The ablation study finds that all SuperGlue blocks provide substantial performance gains.
  • Ablation study: 53.38 AUC@20◦ versus 51.84 results when SuperPoint descriptors are jointly trained with SuperGlue, supporting end-to-end learning beyond matching.The improvement comes from backpropagating through the SuperPoint descriptor network during SuperGlue training.

6. Conclusion · Appendix

The conclusion presents SuperGlue as an attention-based graph neural network that unifies context aggregation, matching, filtering, and partial-assignment handling through optimal transport. The appendix provides additional experiments, qualitative examples, timing results, and analyses of learned attention patterns.

  • 6. Conclusion: SuperGlue uses self-attention to expand descriptor receptive fields and cross-attention for communication between images during matching.
  • 6. Conclusion: Optimal transport enables SuperGlue to handle partial assignments and occluded points within the matching process.
  • 6. Conclusion: SuperGlue replaces handcrafted heuristics with one learnable architecture that jointly performs context aggregation, matching, and filtering.
  • 6. Conclusion: SuperGlue is presented as a step toward end-to-end deep SLAM when combined with a deep front-end.
  • 6. Conclusion: Figure 6 shows SuperGlue consistently producing more correct matches and fewer mismatches than nearest-neighbor matching with handcrafted or learned outlier rejection.
  • 6. Conclusion: Figure 7 visualizes diverse self- and cross-attention patterns that focus on global or local context, self-similarities, distinctive features, or match candidates.
  • Appendix: The appendix presents additional experimental details, quantitative results, qualitative examples, timing results, and analyses of learned attention patterns.

A. Detailed results · A.1. Homography estimation · A.2. Indoor pose estimation

Detailed evaluations show that SuperGlue dominates homography matching across correctness thresholds, generalizes from synthetic training to real HPatches data, and improves indoor pose accuracy and matching trade-offs on ScanNet. Additional visualizations cover synthetic and real homographies and difficult wide-baseline indoor pairs.

  • A.1. Homography estimation: SuperGlue matching results include qualitative evaluations on both synthetic and real homographies.These results are presented in Figure 13.
  • A.1. Homography estimation: SuperGlue achieves higher match precision and homography accuracy than alternatives across all evaluated pixel and error thresholds.Higher precision also enables more accurate homography estimation with DLT than with RANSAC.
  • A.1. Homography estimation: On HPatches, models trained only on synthetic homographies are evaluated on 295 viewpoint-change and 285 illumination-change image pairs, demonstrating real-data generalization.The evaluation reports precision and recall for both HPatches subsets.
  • A.2. Indoor pose estimation: Qualitative indoor visualizations emphasize the extreme difficulty of the wide-baseline image pairs used for evaluation.Additional SuperGlue match visualizations are shown in Figure 14.
  • A.2. Indoor pose estimation: The ScanNet evaluation analyzes methods using SuperPoint features and varies confidence to obtain pose-error and precision-versus-correct-match curves.Both precision and the number of correct matches are identified as critical for accurate and robust pose estimation.
  • A.2. Indoor pose estimation: On ScanNet, SuperGlue produces more accurate pose estimates at every evaluated error threshold and offers the best precision–correct-match trade-off.Correctness uses reprojection error with ground-truth depth at a 10-pixel threshold, or symmetric epipolar error for keypoints with invalid depth.
  • A.2. Indoor pose estimation: On YFCC100M pairs, SuperGlue consistently improves over baselines with both SIFT and SuperPoint features under approximate and exact AUC evaluation.The evaluation uses the same image pairs as OANet [71].

A.3. Outdoor pose estimation

On outdoor YFCC100M evaluations, SuperGlue consistently improves over all baselines with both SIFT and SuperPoint, while also achieving state-of-the-art Aachen Day-Night localization with fewer keypoints. Qualitative results are additionally reported on PhotoTourism and MegaDepth.

  • Qualitative results: Figure 15 provides additional qualitative results on the PhotoTourism test set and MegaDepth validation set.
  • YFCC100M: SuperGlue consistently improves over all baselines on YFCC100M evaluation pairs for both SIFT and SuperPoint, although its SIFT gain over OANet is smaller on the higher-overlap, lower-difficulty pairs.Approximate AUC overestimates accuracy but preserves the identical method ranking.
  • Aachen Day-Night: SuperGlue significantly improves SuperPoint for Aachen Day-Night visual localization, reaching new state-of-the-art results with comparably fewer keypoints.

B. SuperGlue for visual localization · C. Timing and model parameters

SuperGlue is evaluated for challenging day-night visual localization using SuperPoint matches and COLMAP-based SfM registration, while its runtime and parameterization are measured across keypoint counts. The method generalizes to extreme illumination changes and runs at real-time rates, with 12M total parameters.

  • B. SuperGlue for visual localization: SuperGlue supports absolute-pose estimation for visual localization, addressing greater viewpoint and illumination variation than phototourism datasets.The task estimates a query image’s pose relative to a 3D model.
  • B. SuperGlue for visual localization: The Aachen Day-Night evaluation uses up to 4096 SuperPoint keypoints, SuperGlue matching, a triangulated daytime SfM model, and COLMAP registration of nighttime queries.Localization is scored by the percentage of queries meeting distance and orientation thresholds.
  • B. SuperGlue for visual localization: SuperGlue generalizes well to extreme day-night illumination changes and matches repeated building-facade patterns accurately.Figure 10 visualizes correspondences between nighttime queries and daytime database images, distinguishing RANSAC inliers from outliers.
  • C. Timing and model parameters: 14.5 FPS at 512 keypoints and 11.5 FPS at 1024 keypoints demonstrate real-time SuperGlue inference.The Graph Neural Network and Optimal Matching Layer have similar computational costs.
  • C. Timing and model parameters: Runtime measurements cover SuperGlue and its Graph Neural Network and Optimal Matching Layer across different keypoint counts on an NVIDIA GeForce GTX 1080 GPU over 500 runs.These measurements provide the basis for the detailed inference-time breakdown in Figure 11.
  • C. Timing and model parameters: The Keypoint Encoder MLP has five layers and 100k parameters, while each message-update layer has 0.66M parameters; SuperGlue contains 18 layers totaling 12M parameters.The encoder maps positions through dimensions (32, 64, 128, 256, D), and the message-update MLP maps (2D, D); both use BatchNorm and ReLUs.

D. Analyzing attention

SuperGlue’s attention becomes progressively more spatially focused across layers: both self- and cross-attention spans decrease substantially, shifting from broad image regions toward local neighborhoods and likely matches.

  • Quantitative analysis: Both self- and cross-attention spans decrease by more than 10× from the first to the last layer, indicating progressively localized attention.Early layers attend broadly, whereas later self-attention focuses near each keypoint and cross-attention concentrates near the true match.
  • Quantitative analysis: Self-attention initially covers a large image area before narrowing to a small neighborhood around each keypoint.The self-attention span is computed from weighted pixel-space distances between each keypoint and other keypoints.
  • Quantitative analysis: Cross-attention initially searches broadly before narrowing to the vicinity of the final predicted match.The cross-attention span measures the average distance between the predicted match and attended keypoints.
  • Quantitative analysis: Intermediate attention layers show oscillating spans, suggesting a more complex refinement process.The trends are averaged over 100 ScanNet image pairs.

E. Experimental details

The experiments evaluate SuperGlue on synthetic homography, indoor ScanNet, and outdoor MegaDepth/PhotoTourism pose-estimation settings using defined overlap, keypoint, reprojection, and training protocols. Ablations and end-to-end training isolate the contributions of graph reasoning, cross-attention, positional encoding, and descriptor learning.

  • Homography estimation: Homography evaluation uses 1,024 640×480 image pairs, 512 SuperPoint keypoints, and a 3-pixel reprojection threshold after randomized perspective, scale, rotation, and translation transforms.Correspondences are evaluated on synthetic homographies designed to avoid bordering artifacts.
  • Indoor pose estimation: Indoor evaluation uses image pairs with overlap scores in [0.4, 0.8], while ScanNet provides more accurate camera poses than SUN3D for correspondence labeling and pose evaluation.Training samples 200 pairs per scene per epoch, similarly as in [19].
  • Outdoor pose estimation: Outdoor training samples Megadepth pairs with overlap in [0.1, 0.7], and evaluation covers all 11 PhotoTourism scenes using overlap in [0.1, 0.4] [19].Images are resized so their longest dimension is 1600 pixels.
  • Ablation study: Ablations replace the graph neural network, cross-attention, or positional encoding to test graph reasoning, cross-image interaction, and keypoint encoding while retaining specified components.The no-cross-attention model has the full model’s parameter count and behaves like a Siamese network.
  • End-to-end training: End-to-end training initializes two SuperPoint copies, freezes detection, and propagates SuperGlue gradients through the descriptor network without additional losses.The trained models, evaluation code, and image pairs are publicly available.
Loading 1911.11763v2…