PaperlayerView exact revision on arXiv

Source-linked AI summary

LightGlue: Local Feature Matching at Light Speed

Philipp Lindenberger, Paul-Edouard Sarlin, Marc Pollefeys

arXiv:2306.13643v1cs.CV

TL;DR

Sparse image matching must remain robust while meeting demanding runtime and training constraints. LightGlue revisits SuperGlue’s design with simpler, more efficient components and adaptive computation, achieving stronger efficiency–accuracy trade-offs and faster inference on easy pairs.

  • Problem

    Sparse matching is important for vision applications, but difficult visual conditions, high computational cost, and demanding Transformer training complicate robust low-latency deployment.

  • Method

    LightGlue revisits SuperGlue’s design with architecture modifications, confidence-based adaptive depth, and early pruning of points predicted to be unmatchable.

  • Results

    LightGlue is a plug-and-play SuperGlue replacement that predicts strong matches at a fraction of the runtime, with reported gains in accuracy, efficiency, and trainability.

  • Takeaways & Limitations

    Adaptive computation makes LightGlue particularly suitable for latency-sensitive applications such as SLAM and larger-scene reconstruction.

  • Takeaways & Limitations

    Larger input images could improve accuracy but would impose prohibitive and impractical runtime and memory requirements.

Abstract

from arXiv · show

We introduce LightGlue, a deep neural network that learns to match local features across images. We revisit multiple design decisions of SuperGlue, the state of the art in sparse matching, and derive simple but effective improvements. Cumulatively, they make LightGlue more efficient - in terms of both memory and computation, more accurate, and much easier to train. One key property is that LightGlue is adaptive to the difficulty of the problem: the inference is much faster on image pairs that are intuitively easy to match, for example because of a larger visual overlap or limited appearance change. This opens up exciting prospects for deploying deep matchers in latency-sensitive applications like 3D reconstruction. The code and trained models are publicly available at https://github.com/cvg/LightGlue.

1. Introduction

LightGlue addresses the computational and training limitations of deep sparse matching while adapting its computation to each image pair’s difficulty. It targets efficient, accurate matching for latency-sensitive vision applications.

  • Motivation: Sparse feature matching supports camera tracking and 3D mapping but must balance robustness against descriptor uniqueness under difficult visual conditions.Symmetries, weak texture, occlusion, viewpoint, and lighting changes make reliable, discriminative descriptions challenging.
  • Motivation: SuperGlue jointly matches points and rejects outliers robustly, but its computational cost and training demands limit low-latency and resource-constrained applications.Follow-up methods reduced efficiency costs but failed to reach SuperGlue’s performance.
  • Results: LightGlue offers fine-grained speed–accuracy control and is reported as faster and more accurate than SuperGlue, with accuracy closer to LoFTR at 8× higher speed in typical outdoor conditions.The adaptive mechanism enables earlier stopping when predictions are confident.
  • Approach: LightGlue combines architecture modifications and an adaptive mechanism that adjusts inference to image-pair difficulty, including early stopping on easy pairs.Difficulty varies with visual overlap, appearance changes, and discriminative information; unmatchable points can also be discarded early.
  • Results: Experiments characterize LightGlue as a plug-and-play SuperGlue replacement that produces strong matches at a fraction of the runtime, supporting latency-sensitive uses such as SLAM and large-scene reconstruction.The model and training code are released publicly with a permissive license.

2. Related work

Prior work established deep and dense approaches for robust matching, but efficiency, memory, and training difficulties remain central concerns. LightGlue combines efficiency-oriented Transformer innovations with adaptive processing for sparse feature matching.

  • Local feature matching: CNN-based local features improve matching accuracy and robustness, but local descriptors still require filtering and geometric verification to reject incorrect correspondences.Traditional filtering relies on heuristics, classifiers, and robust geometric model fitting.
  • Deep matchers: SuperGlue jointly matches local features and rejects outliers using Transformers and optimal transport, but its complexity grows quadratically with the number of keypoints and it is hard to train.It learns priors about scene geometry and camera motion.
  • Deep matchers: Efficiency-focused follow-up methods reduce attention cost for many keypoints but provide no gains at smaller standard input sizes and can impair robustness in challenging conditions.LightGlue instead targets typical operating conditions without compromising performance across difficulty levels.
  • Dense matchers: Dense matchers improve robustness by processing many grid points, but their higher computational cost limits input resolution and spatial correspondence accuracy.LightGlue remains sparse while being competitive with dense matchers at a fraction of the runtime.
  • Efficient Transformers: Transformer efficiency research reduces attention memory through linear formulations, latent bottlenecks, checkpointing, and memory-access optimization, though some methods impair small-input performance.These approaches provide context for LightGlue’s efficiency-oriented design.
  • Adaptive and positional designs: Adaptive-depth methods predict whether representations need further computation, while relative positional encodings aim to stabilize training and capture long-range dependencies.LightGlue adapts these ideas to 2D feature matching and reports gains in efficiency and accuracy.

3. Fast feature matching

LightGlue jointly processes sparse features with self- and cross-attention, predicts correspondences at every layer, and adapts computation by halting early and pruning unmatchable points. Its relative positional encoding, bidirectional attention, disentangled prediction head, and deep supervision target efficient, accurate matching.

  • Architecture: LightGlue stacks layers that jointly update two feature sets through self- and cross-attention, then predicts a partial assignment from their representations.Each layer contains one self-attention and one cross-attention unit, followed by a lightweight assignment head.
  • Transformer backbone: Relative rotary encoding captures learned relative positions within each image while avoiding absolute positional information.The encoding is cached across layers and is not applied to value vectors.
  • Transformer backbone: Bidirectional cross-attention computes the similarity once for both message directions, saving a factor of 2 in an O(NMd) step.Relative positions are omitted across images because they are not meaningful there.
  • Adaptive depth and width: Adaptive inference reduces computation by halting when enough point predictions are confident and pruning points that are confidently rejected early.The confidence threshold α controls the accuracy–inference-time trade-off, while point pruning reduces both search space and inference time.
  • Design improvements: LightGlue improves deeper-layer accuracy by repeatedly adding relative positional encoding rather than allowing positional information to be forgotten.The paper also attributes cleaner gradients and efficiency to disentangling similarity and matchability.
  • Correspondence prediction: LightGlue separates pairwise similarity from per-point matchability and combines them into a soft partial assignment before selecting mutually best matches above threshold τ.Unmatchable points can receive low matchability, for example when occluded.
  • Training and inference: The lightweight head enables layerwise assignment prediction and supervision, speeding convergence and allowing inference to exit after any layer.This avoids the expensive final-layer-only prediction scheme used by SuperGlue.

4. Details that matter

LightGlue uses synthetic homographies for pre-training and MegaDepth for fine-tuning, with implementation choices that improve training efficiency and fit larger batches on limited hardware. These choices reduce resources relative to SuperGlue.

  • Recipe: LightGlue is pre-trained on synthetic homographies sampled from 1M images and fine-tuned on MegaDepth, which contains 1M images of 196 tourism landmarks.MegaDepth provides camera calibration, recovered poses, and dense depth from multi-view stereo.
  • Training tricks: The training recipe reduces resources compared with SuperGlue, lowering training cost and making deep matchers more accessible.The paper presents these details as contributors to training speed, stability, and accuracy.
  • Training tricks: Incomplete MegaDepth depth maps are handled by labeling points with large epipolar error as unmatchable, while learning-rate annealing and 2k points per image improve accuracy.Gradient checkpointing and mixed precision fit 32 image pairs on one GPU with 24GB VRAM.
  • Implementation details: LightGlue uses nine layers, four attention heads, and 256-dimensional representations, with optimized runtime numbers relying on efficient self-attention.The model is trained with both SuperPoint and SIFT and remains compatible with other local features.

5. Experiments

LightGlue is evaluated across homography estimation, relative pose estimation, visual localization, and design ablations. It generally improves matching accuracy and efficiency, while adaptive computation accelerates inference according to image-pair difficulty.

  • 5.1. Homography estimation: LightGlue yields higher-precision correspondences than SuperGlue and SGMNet with similar recall, enabling more accurate homographies using DLT and competitiveness with LoFTR at 5px.Its correspondence quality makes the simple DLT solver competitive with the slower MAGSAC.
  • 5.2. Relative pose estimation: LightGlue largely outperforms SuperGlue and SGMNet on SuperPoint features and improves matching accuracy over DISK features.The method produces more accurate correspondences and relative poses while reducing inference time by 30%.
  • 5.2. Relative pose estimation: Over 2× faster than SuperGlue and SGMNet, the adaptive variant remains more accurate by detecting confident predictions early.With tuned LO-RANSAC, LightGlue can exceed some dense matchers that are 5–11 times slower; ASPANFormer is the most accurate dense matcher evaluated.
  • 5.3. Outdoor visual localization: 2.5× higher throughput than SuperGlue is achieved at similar visual-localization accuracy, rising to 4× with the optimized variant.The optimized model matches up to 4096 keypoints in real time.
  • 5.4. Insights: Compared with SuperGlue, LightGlue converges significantly faster and achieves +4% recall and +12% precision.The faster convergence makes training on new data more practical, although sufficiently long SuperGlue training can reach similar accuracies.
  • 5.4. Insights: Adaptive depth and width reduce computation case by case: easy image pairs exit after fewer layers, while unmatchable points are removed from later inputs.Easy samples achieve a 1.86× speedup, and adaptive pruning further reduces runtime for any input size; LightGlue is faster than SuperGlue and SGMNet up to 2K keypoints.

6. Conclusion

LightGlue combines attention-based matching with architectural and training improvements, producing an adaptive matcher whose computation reflects image-pair difficulty. The supplied conclusion materials also visualize its pruning behavior and feature outputs across local-feature extractors.

  • 6. Conclusion: LightGlue is a deep network for matching sparse local features that combines attention mechanisms, Transformer innovations, and self-confidence to adapt computation to pair difficulty.Its adaptive computation changes both depth and width.
  • 6. Conclusion: The adaptive visualizations show early removal of unmatchable points and reduced matching search spaces across easy, medium, and difficult image pairs.The figure also presents matchability scores and matches across the three difficulty levels.
  • 6. Conclusion: Feature-output comparisons cover SIFT+LightGlue, SuperPoint+LightGlue, and DISK+LightGlue.The comparison is shown for three local-feature extractors.

A. Image Matching Challenge

On Image Matching Challenge benchmarks, LightGlue remains competitive with SuperGlue while substantially reducing matching time, and DISK features further improve accuracy. On IMC 2023, SuperPoint+LightGlue also improves end-to-end Structure-from-Motion scores over SuperPoint+SuperGlue.

  • Image Matching Challenge 2020: 2.5× faster matching: LightGlue remains competitive with SuperGlue in both stereo and multi-view PhotoTourism tracks.Adaptive depth largely reduces runtime for easy image pairs.
  • Image Matching Challenge 2020: +8%/+5% AUC: DISK+LightGlue improves over the SuperPoint+LightGlue equivalent on smaller stereo and multi-view thresholds.DISK+LightGlue also predicts 30% more matches with higher epipolar precision.
  • Image Matching Challenge 2021: On IMC 2021 Phototourism, LightGlue outperforms other sparse baselines in both stereo and multi-view tasks and surpasses tuned public-leaderboard baselines.The table caption notes that LightGlue does not use test-time augmentation while some baselines do.
  • Image Matching Challenge 2023: SuperPoint+LightGlue reaches 38.4/46.1% on the IMC 2023 public/private leaderboards, versus 36.1/43.8% for SuperPoint+SuperGlue, a +2.3% improvement.The benchmark evaluates end-to-end Structure-from-Motion using camera-pose accuracy averaged over multiple thresholds.

B. Additional results

Additional evaluations show that LightGlue is faster while maintaining or improving accuracy across relative pose estimation and visual localization. Its main reported failure mode on InLoc involves repeated objects and weak retrieval rather than a general matching deficit.

  • Relative pose estimation: On a SuperGlue-disjoint MegaDepth-1800 split, LightGlue achieves higher precision, pose accuracy, and speed than existing sparse matchers while remaining competitive with dense matchers.It predicts a similar number of correspondences to existing sparse matchers.
  • Relative pose estimation: LightGlue predicts more accurate correspondences than existing sparse matchers at a fraction of the time, while LoFTR remains state-of-the-art by a 2% AUC@5° margin.The comparison uses the MegaDepth-1800 evaluation described in Table 8.
  • Outdoor visual localization: On Aachen v1.1, all methods have similar accuracy because the dataset is largely saturated, but LightGlue is far faster than all approaches.The reported variations in accuracy are insignificant.
  • Indoor visual localization: On InLoc, LightGlue is competitive with SuperGlue and more accurate at (0.25m,10°), with differences below 2% considered insignificant given the split sizes.The reported failures over SuperGlue occur in 6 of 356 images at 1m and involve repeated objects plus weak retrieval.
  • Indoor visual localization: LightGlue’s InLoc failures over SuperGlue are attributed to more matches on repeated objects, such as trash cans, combined with weak retrieval.The paper illustrates this behavior in Figure 10.

C.1. Architecture

LightGlue combines Transformer-based matching with adaptive early stopping and point pruning, supported by efficient training and synthetic homography pre-training. These mechanisms reduce computation while preserving matching accuracy under the reported settings.

  • Graph neural network: The architecture uses nine Transformer layers with self- and cross-attention, shared query/key projections in bidirectional cross-attention, and gradient checkpointing during training.The update MLP has hidden dimension 2d, while efficient self-attention reduces IO complexity.
  • Confidence classifier: LightGlue predicts per-keypoint confidence and halts inference when α=95% of states are deemed confident.Confidence predictions are supervised with binary cross-entropy and are not backpropagated into the matching states.
  • Exit criterion and point pruning: At 95% exit confidence, inference time falls by 25% on average with hardly any loss of downstream accuracy.Lowering α to 80% reduces inference time by almost 50% while retaining competitive MegaDepth accuracy.
  • Training: Training uses SuperPoint, SIFT, and DISK features, synthetic homographies, strong photometric augmentation, and 3px symmetric reprojection error for supervision.The pre-training corpus contains 170k Oxford-Paris distractor images.
  • Training: > 99% recall and > 90% precision are achieved on validation and test sets after training for up to 40 epochs or 6M image pairs.The authors report that strong photometric and perspective augmentations are crucial for robust training.

D. Timings

LightGlue’s runtime gains come from cheaper attention and assignment operations, with bidirectional cross-attention reusing a single similarity matrix. Timing measurements isolate matching computation from feature extraction and pose estimation.

  • Measurement setup: Reported timings measure matching only, excluding sparse feature extraction and robust pose estimation, and average runtime over the respective datasets.Experiments were conducted on a single RTX 3080 with 10GB VRAM.
  • Attention: Bidirectional cross-attention reduces runtime by 33% by computing the similarity matrix only once.The softmax over both directions remains the main bottleneck.
  • Assignment: The partial assignment layer is significantly faster in LightGlue and accounts for most default inference-time improvements relative to SuperGlue.The breakdown evaluates self-, cross-, and partial-assignment layers on 1024 keypoints.
  • Assignment: Cheap double-softmax and unary matchability predictions are faster than optimal transport, which requires 100 training iterations for stability.The comparison concerns solving the partial assignment problem.
  • Measurement setup: Efficient self-attention and mixed precision further reduce runtime and memory, but these improvements are excluded from fair comparisons unless explicitly stated.This isolates architectural comparisons from additional implementation optimizations.

E. Qualitative Results

The qualitative results visualize LightGlue’s matching behavior, including unmatched-point discarding, early stopping across difficulty levels, and outputs with several feature extractors.

  • Figure 8 visualizes how LightGlue discards unmatched points during matching.
  • Figure 8 illustrates LightGlue’s early stopping mechanism on easy, medium, and hard image pairs.
  • Figure 9 presents qualitative matching outputs using SIFT, SuperPoint, and DISK features.
Loading 2306.13643v1…