PaperlayerView exact revision on arXiv

Source-linked AI summary

XFeat: Accelerated Features for Lightweight Image Matching

Guilherme Potje, Felipe Cadar, Andre Araujo, Renato Martins, Erickson R. Nascimento

arXiv:2404.19174v1cs.CV

TL;DR

Resource-constrained visual correspondence needs fast, robust feature extraction and matching without sacrificing the image resolution needed for accurate correspondences. XFeat addresses this with a lightweight CNN supporting sparse and semi-dense matching, and reports faster lightweight alternatives with comparable accuracy to larger models. Its scope is bounded by evidence that naive channel pruning can compromise robustness and that shallow networks gain limited speed from reduced representation.

  • Problem

    Image matching requires efficient, robust processing on resource-constrained devices, while high-resolution inputs increase computation and existing solutions may depend on hardware-specific optimization.

  • Method

    XFeat uses a lightweight, hardware-agnostic CNN for keypoint detection and local feature extraction, with sparse matching and coarse semi-dense matching refined to pixel-level offsets.

  • Results

    XFeat achieves fast and accurate matching across three tasks, outperforming lightweight deep-learning alternatives by up to 5× in speed while remaining comparable in accuracy to larger models.

  • Takeaways & Limitations

    XFeat offers a resource-efficient alternative for downstream applications including visual localization, camera pose estimation, augmented reality, and mobile robotics.

  • Takeaways & Limitations

    Naively pruning channels can compromise robustness to varying illumination and viewpoint, while shallow networks may provide limited representational capacity and only minor speed gains.

Abstract

from arXiv · show

We introduce a lightweight and accurate architecture for resource-efficient visual correspondence. Our method, dubbed XFeat (Accelerated Features), revisits fundamental design choices in convolutional neural networks for detecting, extracting, and matching local features. Our new model satisfies a critical need for fast and robust algorithms suitable to resource-limited devices. In particular, accurate image matching requires sufficiently large image resolutions - for this reason, we keep the resolution as large as possible while limiting the number of channels in the network. Besides, our model is designed to offer the choice of matching at the sparse or semi-dense levels, each of which may be more suitable for different downstream applications, such as visual navigation and augmented reality. Our model is the first to offer semi-dense matching efficiently, leveraging a novel match refinement module that relies on coarse local descriptors. XFeat is versatile and hardware-independent, surpassing current deep learning-based local features in speed (up to 5x faster) with comparable or better accuracy, proven in pose estimation and visual localization. We showcase it running in real-time on an inexpensive laptop CPU without specialized hardware optimizations. Code and weights are available at www.verlab.dcc.ufmg.br/descriptors/xfeat_cvpr24.

1. Introduction

XFeat targets efficient, hardware-agnostic local feature extraction and matching for resource-constrained platforms. It supports both sparse and semi-dense correspondence while improving the accuracy–efficiency trade-off.

  • Motivation: Recent image-matching advances often increase computational requirements, motivating efficient feature extraction for resource-constrained platforms.Hardware-specific optimization remains cumbersome, while architectural approaches to efficient extraction are comparatively limited.
  • XFeat: XFeat is a lightweight, hardware-agnostic CNN architecture for robust keypoint detection and local feature extraction.Its design reduces computational footprint without precluding optimization for specific hardware configurations.
  • Matching versatility: XFeat supports both sparse keypoint matching and dense matching of a coarse feature map for different downstream tasks.Sparse matching suits visual localization with Structure-from-Motion maps, while dense matching can help pose estimation in poorly textured scenes.
  • Efficiency and accuracy: Up to 5× faster than lightweight deep-learning local-feature alternatives, XFeat remains comparable in accuracy to larger models such as SuperPoint and DISK.The comparison is reported for the matching accuracy–computational-efficiency trade-off.
  • Semi-dense refinement: XFeat introduces a match-refinement module that obtains pixel-level offsets from coarse semi-dense matches without requiring high-resolution features beyond local descriptors.The strategy reduces computation while achieving high matching accuracy and density.

2. Related Work

Related work increasingly improves image-matching robustness and accuracy, but often at substantial computational cost. Efficient methods reduce this burden with simpler designs, although some require hardware-specific support or remain expensive at common matching resolutions.

  • Image matching: Modern image matching spans handcrafted keypoints with learned descriptors, joint detection and description, learned matchers, and semi-dense or dense methods.Transformer-based approaches have recently improved robustness and accuracy for wide-baseline image pairs.
  • Computational cost: Recent matching methods often inflate computational demands and require adaptation for large-scale visual localization, SLAM, and structure-from-motion.The paper positions XFeat as reducing compute in sparse extraction and pixel-level semi-dense matching while retaining similar performance.
  • Efficient description: SuperPoint jointly detects keypoints and computes descriptors but can remain costly at image sizes common in image matching.SiLK instead emphasizes a simpler learned keypoint-and-descriptor strategy with comparable performance to existing methods.
  • Hardware dependence: Some efficiency approaches rely on custom compilation and processor-specific arithmetic operations, restricting applicability across diverse hardware.This motivates hardware-independent architectural efficiency rather than only low-level optimization.
  • Learned matching: LightGlue accelerates learned matching while maintaining high accuracy, yet its transformer architecture remains costly when computational efficiency is critical.The paper contrasts this with its focus on highly efficient and robust image matching for broad deployment.

3. XFeat: Accelerated Features

XFeat reduces feature-extraction cost by preserving relatively large spatial resolutions while limiting early channels, then supports both keypoint-based and semi-dense matching. Its refinement module recovers pixel-level offsets from coarse descriptor matches without high-resolution features.

  • Featherweight Network Backbone: Naively pruning channels throughout the network compromises robustness to varying illumination and viewpoint.
  • Featherweight Network Backbone: Depthwise separable convolutions provide smaller gains in shallow, high-resolution local-feature networks than in low-resolution classification or detection settings.The text attributes this to limited representational capacity and minor speed gains in shallow feature-extraction networks.
  • Featherweight Network Backbone: XFeat targets the resolution–compute bottleneck by reducing early-layer channels while retaining spatial resolution for fine-grained correspondences.The method identifies H_i × W_i as the dominant convolutional cost and reallocates channel capacity across the network.
  • Featherweight Network Backbone: The backbone uses six resolution-reducing convolutional blocks with channel depths {4, 8, 24, 64, 64, 128}, followed by multiresolution fusion.
  • Local Feature Extraction: XFeat produces a keypoint heatmap K, a 64-D dense descriptor map F, and a reliability heatmap R through separate feature heads.The keypoint head reshapes 8 × 8 image cells into 64-dimensional features and classifies 64 positions plus a dustbin.
  • Dense Matching: For semi-dense matching, nearest-neighbor coarse descriptors feed an MLP that predicts pixel-level offsets while avoiding high-resolution feature inputs.The dense feature map is at 1/8 input resolution, and the refinement module is trained jointly with the backbone.

4. Experiments

Across relative pose estimation, homography estimation, visual localization, and ablations, XFeat combines competitive accuracy with substantially lower computational cost. Its semi-dense matching and refinement design support robustness, density, and efficient deployment.

  • Relative pose estimation: 5× faster than ALIKE, XFeat achieves competitive relative-pose results on Megadepth-1500 in the sparse setting.Evaluation uses AUC, Acc@10°, MIR, inlier counts, and CPU FPS; images are resized to a 1,200-pixel maximum dimension.
  • Relative pose estimation: 9× speedup over SuperPoint and 16× speedup over DISK accompany comparable relative-pose performance on Megadepth-1500.The comparison uses the same reported Megadepth-1500 evaluation and distinguishes DISK’s heavier model and 10k-keypoint setting.
  • Relative pose estimation: XFeat generalizes better to ScanNet indoor scenes, while DISK and ALIKE show signs of bias toward landmark datasets.The reported ScanNet comparison used methods without retraining.
  • Homography estimation: XFeat provides high-quality homography estimation with a fraction of the compute, whereas ORB and SiLK fail on several illumination sequences.The HPatches evaluation uses Mean Homography Accuracy at thresholds of 3, 5, and 7 pixels.
  • Visual localization: XFeat matches leading visual-localization accuracy while running at least 9 times faster and using a more compact descriptor.On Aachen day-night, it is reported as on par with the state of the art for thresholds above 0.5 m and 5°.
  • Ablation: 11% additional inference cost makes the match-refinement module critical for XFeat* accuracy with an average of 10,000 descriptors.The ablation attributes degradation without the parallel keypoint branch to limited intermediate-embedding capacity affecting semi-dense matching and refinement.

5. Conclusion

XFeat is a lightweight CNN architecture for accelerated feature extraction supporting sparse and semi-dense image matching. Experiments and ablations show fast, accurate matching without advanced low-level hardware optimizations, targeting augmented-reality and mobile-robotics deployment.

  • XFeat supports both sparse and semi-dense image matching through a lightweight CNN architecture for accelerated feature extraction.
  • Experiments across three tasks and ablations show fast and accurate image matching without advanced low-level hardware optimizations.
  • XFeat targets efficient, general data-driven solutions for augmented reality and mobile robotics, particularly mobile applications.

A. Backbone details

The XFeat backbone uses modular basic layers and blocks to downsample spatial resolution while increasing depth. Its 23-layer design is deeper than ALIKE and SuperPoint backbones but remains faster through efficient downsampling.

  • A basic layer combines a 2D convolution, ReLU, and batch normalization, with stride 2 used when spatial downsampling is required.Kernel sizes are k = 1 or k = 3.
  • Six basic blocks progressively halve spatial resolution and increase network depth, followed by blocks for multi-resolution fusion and reliability-map prediction.Each basic block contains two or three basic layers, with the first layer performing downsampling.
  • The 23-layer backbone is deeper than ALIKE and SuperPoint backbones but achieves faster inference through its efficient downsampling strategy.
  • A single skip connection slightly improved performance and was incorporated into the final backbone design.

B. Training description

XFeat is trained on a hybrid mixture of Megadepth scenes and synthetically warped COCO images. Training uses modest memory and compute resources, though Megadepth data loading is the main speed bottleneck.

  • The training set combines Megadepth scenes and synthetically warped COCO image pairs in a 6:4 proportion.COCO images are used without labels, with synthetic warps generated during training.
  • Images are resized to 800 × 600, and the network is trained with Adam for 160,000 iterations.The initial learning rate is 3 × 10^-4, with exponential decay of 0.5 every 30,000 updates.
  • Training converges within 36 hours on one RTX 4090 GPU while using 6.5 GB of VRAM.The reported memory total includes training and on-the-fly synthetic warps performed on the GPU.
  • Disk I/O is the predominant speed bottleneck because Megadepth images and depth maps are loaded at their original resolution.The authors state that more careful data preparation could readily address this bottleneck.

C. Detailed timing analysis

The timing analysis finds that match refinement adds negligible cost after nearest-neighbor search. On a highly constrained embedded computer, XFeat is substantially faster than the compared learned methods.

  • Match refinement has negligible cost, and XFeat* matches XFeat timing with the same keypoint count.Refinement is performed after the nearest-neighbor search.
  • 1.8 FPS is achieved by XFeat on the Orange Pi Zero 3, compared with 0.16 FPS for SuperPoint and 0.58 FPS for ALIKE.All methods use 480-resolution inputs and standard PyTorch implementations without deployment optimization.
  • XFeat is the only learned method in this experiment to exceed one FPS on the constrained embedded device.The Orange Pi Zero 3 uses a Cortex-A53 ARM processor and costs $28.

D. Megadepth-1500 qualitative results

On challenging viewpoint and illumination changes, XFeat and XFeat* show strong qualitative robustness relative to baseline methods, including the larger DISK architecture.

  • XFeat and XFeat* exhibit exceptional robustness under strong viewpoint and illumination changes.The comparison is made against the baseline methods used in the main paper.
  • This robustness is observed even relative to DISK, identified as the largest CNN architecture by floating-point operations.The authors hypothesize that XFeat’s large receptive field and greater depth contribute to this behavior.
  • The proposed methods are contrasted with shallower models such as SuperPoint and ALIKE in these challenging cases.

E. ScanNet-1500 extended discussion

XFeat and XFeat* combine strong pose accuracy with faster indoor relative pose estimation, while competing methods show less consistent transfer across tasks. The authors associate this generalization with hybrid training and architectural choices.

  • E. ScanNet-1500 extended discussion: XFeat and XFeat* surpass fast and standard local feature extractors in pose accuracy while being significantly faster for indoor relative pose estimation.
  • E. ScanNet-1500 extended discussion: DISK and ALIKE perform exceptionally well at strict AUC@5° thresholds on Megadepth-1500 but are similar or worse than XFeat and SuperPoint on homography estimation and visual localization.The authors interpret this pattern as signs of overfitting in landmark imagery.
  • E. ScanNet-1500 extended discussion: The authors conjecture that hybrid training with synthetic COCO warps produces less biased local descriptors and improves generalization across downstream tasks and datasets.They suggest that this training reduces emphasis on distinctive textures common in landmark outdoor imagery.
  • E. ScanNet-1500 extended discussion: A large receptive field and increased network depth are proposed as factors helping XFeat handle indoor imagery with less distinctive content.

F. Comparison with learned matchers

XFeat* combines traditional nearest-neighbor matching with lightweight refinement, offering a faster semi-dense alternative to learned and coarse-to-fine matchers while retaining high accuracy and dense correspondences. Its limitations include paired-input refinement and reduced robustness under aggressive viewpoint changes or highly ambiguous image pairs.

  • Method: XFeat* uses paired inputs for refinement but relies on traditional nearest-neighbor search followed by lightweight match refinement, unlike learned matchers.The comparison is methodologically closest to Patch2Pix among the listed approaches.
  • Results: 22× speedup over existing semi-dense solutions is reported for XFeat*, while it surpasses Patch2Pix in accuracy and provides more matches than LightGlue.XFeat* is also faster than LightGlue in this comparison.
  • Efficiency: XFeat* can be trained on a single 8 GB GPU, whereas LoFTR requires 64 GPUs for 24 hours, highlighting a substantially lower training burden.The authors describe learned matchers as more data hungry and more expensive to train.
  • Qualitative comparison: On Megadepth-1500, XFeat and XFeat* outperform DISK in semi-dense matching with 10,000 local features at a reported 16× speedup.In sparse matching with 4,096 keypoints, XFeat is reported as 5× faster than ALIKE and 9× faster than SuperPoint.
  • Qualitative comparison: On ScanNet-1500 indoor imagery, the proposed approaches outperform DISK and ALIKE in camera pose and inlier ratio.The caption also notes that SuperPoint often outperforms DISK and ALIKE.
  • Limitations: XFeat as a local descriptor has limited robustness to aggressive viewpoint changes and highly ambiguous image pairs compared with transformer-based feature matchers.The authors suggest coupling XFeat features with lightweight transformer matchers such as LightGlue or LoFTR.
Loading 2404.19174v1…