PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperPoint: Self-Supervised Interest Point Detection and Description

Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich

arXiv:1712.07629v4cs.CV

TL;DR

Interest point detection lacks clear semantic labels and large real-image annotations, making strongly supervised learning difficult. SuperPoint uses self-supervised Homographic Adaptation to train a fully convolutional detector-descriptor, achieving strong HPatches homography estimation against established methods.

  • Problem

    Interest point detection is semantically ill-defined and lacks large labeled image databases, making strongly supervised convolutional training non-trivial.

  • Method

    SuperPoint bootstraps from synthetic shapes and unlabeled target-domain images, using self-training with multi-scale, multi-homography Homographic Adaptation.

  • Results

    On HPatches, SuperPoint outperforms LIFT and ORB, matches SIFT for homography estimation, and outperforms classical detectors under illumination changes.

  • Takeaways & Limitations

    The resulting system provides a single efficient convolutional network for interest point detection and description in geometric matching tasks.

  • Takeaways & Limitations

    Homographies accurately model viewpoint changes mainly for rotation about the camera center, distant objects, or planar scenes.

Abstract

from arXiv · show

This paper presents a self-supervised framework for training interest point detectors and descriptors suitable for a large number of multiple-view geometry problems in computer vision. As opposed to patch-based neural networks, our fully-convolutional model operates on full-sized images and jointly computes pixel-level interest point locations and associated descriptors in one forward pass. We introduce Homographic Adaptation, a multi-scale, multi-homography approach for boosting interest point detection repeatability and performing cross-domain adaptation (e.g., synthetic-to-real). Our model, when trained on the MS-COCO generic image dataset using Homographic Adaptation, is able to repeatedly detect a much richer set of interest points than the initial pre-adapted deep model and any other traditional corner detector. The final system gives rise to state-of-the-art homography estimation results on HPatches when compared to LIFT, SIFT and ORB.

1. Introduction

The paper frames interest-point detection as a foundational problem in geometric computer vision and addresses the lack of semantic ground truth with self-supervised training. It combines synthetic pretraining, Homographic Adaptation, and a descriptor subnetwork in a fully convolutional architecture.

  • Motivation: Interest points provide stable, repeatable 2D image locations for tasks including SLAM, Structure-from-Motion, camera calibration, and image matching.They are intended to remain consistent across lighting conditions and viewpoints.
  • Motivation: Human supervision is difficult because interest points lack the clear semantic definition available for tasks such as human-body keypoint estimation.The paper therefore does not formulate real-image interest-point detection as a conventional large-scale supervised problem.
  • Self-supervised training: The proposed self-supervised approach creates pseudo-ground-truth interest points in real images using the detector itself instead of large-scale human annotation.This enables training from automatically generated labels in real images.
  • Self-supervised training: MagicPoint is pretrained on millions of Synthetic Shapes examples, whose simple geometric shapes provide unambiguous interest-point locations.The resulting detector significantly outperforms traditional interest-point detectors on the synthetic dataset and performs surprisingly well on real images despite domain adaptation difficulties.
  • Homographic Adaptation: Homographic Adaptation repeatedly warps images across viewpoints and scales, boosting repeatability and generating pseudo-ground-truth points for training.The resulting detections are more repeatable and cover a larger set of image locations.
  • Descriptors: SuperPoint combines the interest-point network with a descriptor subnetwork to attach fixed-dimensional descriptors for higher-level tasks such as image matching.The combination is facilitated by the architecture’s deep convolutional stack, which extracts multi-scale features.

2. Related Work

Related work spans traditional local features, deep learned descriptors and detectors, unsupervised patch-based methods, and self-supervised geometric adaptation. SuperPoint differs by combining interest point detection and descriptor computation in a single real-time network.

  • Traditional methods: FAST framed high-speed corner detection as machine learning, while SIFT remains a prominent traditional local feature descriptor.Traditional interest point detectors have been extensively evaluated.
  • Deep learned features: UCN and DeepDesc learn to match image substructures but do not perform interest point detection, unlike LIFT’s patch-based detect-then-describe pipeline.LIFT is described as a convolutional replacement for SIFT that also includes orientation estimation.
  • SuperPoint distinction: SuperPoint is the only compared method that computes both interest points and descriptors in a single network in real-time.The qualitative comparison table identifies this as SuperPoint’s distinguishing capability.
  • Unsupervised detection: Quad-Networks addresses interest point detection without supervision but uses small image patches and a shallow 2-layer network, while TILDE applies a principle similar to Homographic Adaptation.The passage contrasts TILDE with large fully-convolutional neural networks.
  • Self-supervised adaptation: Related self-supervised approaches include equivariant landmark transform, Geometric Matching Networks, and Deep Image Homography Estimation, but they lack interest points and point correspondences.These methods use related self-supervision strategies for synthetic-to-real adaptation or global transformation estimation.

3. SuperPoint Architecture

SuperPoint is a fully convolutional network that processes full-sized images and jointly produces interest-point detections and fixed-length descriptors in one forward pass. A shared VGG-style encoder feeds separate detection and descriptor decoder heads, using efficient non-learned upsampling.

  • Network overview: A shared encoder processes the full-sized image, then splits into task-specific decoder heads for interest-point detection and description.The architecture produces both outputs in a single forward pass.
  • Shared encoder: Three max-pooling layers reduce an H × W image to Hc = H/8 and Wc = W/8, forming non-overlapping 8 × 8 cells.The encoder uses convolution, pooling, and nonlinear activation layers in a VGG-style design.
  • Interest point detector: The detector head predicts 65 channels per cell: 64 local 8 × 8 pixel regions plus a no-interest-point dustbin.A channel-wise softmax removes the dustbin, followed by reshaping from RHc×Wc×64 to RH×W.
  • Decoder design: Both decoder heads use non-learned upsampling to restore full spatial resolution, keeping the model fast and easy to train.The detector uses a reshape-based sub-pixel convolution, also called depth to space or pixel shuffle.
  • Descriptor head: The descriptor head predicts a semi-dense descriptor grid, then applies bicubic interpolation and L2 normalization to produce unit-length dense descriptors.Semi-dense prediction, such as one descriptor every 8 pixels, reduces training memory and keeps runtime tractable.

4. Synthetic Pre-Training

The authors bootstrap interest-point detection with MagicPoint, a detector trained on a large synthetic dataset because no large labeled interest-point image database exists. Synthetic Shapes uses unambiguous geometric junctions and endpoints, while MagicPoint performs strongly on synthetic images but generalizes imperfectly to natural images.

  • Synthetic Pre-Training: MagicPoint is the SuperPoint detector pathway trained on Synthetic Shapes without the descriptor head.It is later combined with Homographic Adaptation to generate pseudo-ground-truth labels for unlabeled images.
  • Synthetic Pre-Training: Synthetic Shapes provides large-scale rendered training data because no large database of interest-point-labeled images exists.The dataset contains simplified 2D quadrilaterals, triangles, lines, and ellipses.
  • Synthetic Pre-Training: Synthetic Shapes removes label ambiguity by defining interest points as Y-, L-, and T-junctions, tiny-ellipse centers, and line-segment endpoints.Homographic warps augment each rendered image, with data generated on-the-fly so the network never sees an example twice.
  • Synthetic Pre-Training: MagicPoint outperforms FAST, Harris corners, and Shi-Tomasi on Synthetic Shapes, with mean Average Precision measured on 1000 held-out images.The classical detectors struggle with imaging noise, whereas MagicPoint is described as robust to added noise.
  • Synthetic Pre-Training: MagicPoint generalizes reasonably to real images with strong corner-like structure but underperforms classical detectors in repeatability under viewpoint change across natural images.Examples of favorable scenes include tables, chairs, and windows.

5. Homographic Adaptation

Homographic Adaptation self-supervises interest point detection by aggregating predictions from randomly warped copies of unlabeled target-domain images. It uses plausible homographies and can be applied iteratively during training to improve the detector’s generalization to real images.

  • Self-supervised adaptation: The method generates pseudo-ground-truth interest point locations from unlabeled target-domain images, then trains with standard supervised learning machinery.It applies random homographies to warped copies of each input image and combines the resulting detections.
  • Self-supervised adaptation: Homographies are suitable because they model viewpoint changes for rotational camera motion, distant objects, and planar scenes without requiring 3D information.They can therefore be randomly sampled and applied directly to 2D images.
  • Aggregation: Because detectors are not perfectly homography-covariant, Homographic Adaptation averages responses over many random homographies to produce an improved detector.The aggregation empirically combines predictions that may yield different interest points under different homographies.
  • Homography sampling: Plausible homographies are composed from translation, scale, in-plane rotation, and symmetric perspective distortion sampled within predetermined ranges.An initial root center crop helps avoid bordering artifacts.
  • Homography sampling: The number of homographic warps Nh is a hyper-parameter, with experiments varying small Nh = 10, medium Nh = 100, and large Nh = 1000.The first homography is typically the identity, so Nh=1 corresponds to no adaptation.
  • Training: Applying Homographic Adaptation during training improves MagicPoint’s generalization to real images and can be repeated iteratively for continued self-supervision.The resulting model is called SuperPoint in the experiments.

6. Experimental Details

SuperPoint uses a VGG-like fully convolutional architecture with separate detector and descriptor heads. Training proceeds from a Synthetic Shapes MagicPoint base through two rounds of Homographic Adaptation on MS-COCO, followed by joint training on transformed grayscale images.

  • Model architecture: The encoder has eight 3×3 convolutional layers with widths 64-64-64-64-128-128-128-128 and 2×2 max pooling every two layers.Each decoder head contains a 3×3 convolution with 256 units, followed by a 1×1 convolution with 65 detector units or 256 descriptor units.
  • Base training: MagicPoint, which omits SuperPoint’s descriptor head, is pretrained for 200,000 iterations on procedurally rendered Synthetic Shapes data.The synthetic data is rendered on-the-fly, so the network never sees the same example twice.
  • Homographic Adaptation: Pseudo-ground-truth labels are generated from 80,000 MS-COCO 2014 training images at 240 × 320 grayscale resolution using Homographic Adaptation with Nh = 100.The adaptation is repeated a second time using the model trained after the first round.
  • Joint training: SuperPoint is jointly trained on 240×320 grayscale COCO images after applying randomly sampled, restrictive homographies designed to model pairwise matching.Extreme in-plane rotations are avoided because they are rarely observed in HPatches.
  • Optimization and augmentation: Training uses PyTorch mini-batches of 32 with ADAM at lr = 0.001 and β = (0.9, 0.999), plus noise, motion blur, and brightness augmentation.These augmentations are used to improve robustness to lighting and viewpoint changes.

7. Experiments

Experiments on HPatches evaluate SuperPoint’s runtime, repeatability, matching, and homography estimation against classical and learned baselines. Homographic Adaptation substantially improves repeatability, while SuperPoint performs strongly in illumination changes, descriptor-focused matching, and homography estimation.

  • Runtime: 11.15 ms: A single SuperPoint forward pass on 480 × 640 inputs produces detection locations and a semi-dense descriptor map on a Titan X GPU.Descriptors can be sampled without constructing the entire dense descriptor map.
  • Interest point detection: Homographic Adaptation gives MagicPoint a large repeatability boost, with SuperPoint outperforming classical detectors under illumination changes and matching them under viewpoint changes.Repeatability is measured on HPatches at 240 × 320 resolution with 300 points per image and a correct distance of ϵ = 3 pixels.
  • Homography estimation: SuperPoint outperforms LIFT and ORB and performs comparably to SIFT for HPatches homography estimation across multiple correctness thresholds.SuperPoint tends to produce more numerous, correct matches that densely cover the image, especially under illumination changes.
  • Homography estimation: ORB achieves the highest repeatability, but its detections form sparse clusters and score poorly on homography estimation, showing that repeatability alone does not ensure better matching or estimation.The qualitative comparison is illustrated in Figure 8.
  • Descriptor matching: SuperPoint scores strongly on nearest neighbor mAP and matching score, supporting the advantage of learned representations over hand-tuned representations for descriptor matching.The comparison uses LIFT, SIFT, and ORB on HPatches.

8. Conclusion · APPENDIX

The paper presents SuperPoint as a fully convolutional network for interest point detection and description, trained with self-supervised Homographic Adaptation. The authors conclude that it transfers synthetic knowledge to real images, efficiently unifies detection and description, supports geometric computer vision, and motivates future applications beyond these tasks.

  • 8. Conclusion: SuperPoint combines interest point detection and description in a single fully-convolutional neural network.The architecture is trained using the self-supervised domain adaptation framework Homographic Adaptation.
  • 8. Conclusion: The experiments demonstrate transfer of knowledge from a synthetic dataset onto real-world images.This transfer is achieved through the Homographic Adaptation framework.
  • 8. Conclusion: The resulting system performs well for geometric computer vision tasks.The supplied conclusion passage truncates the specific continuation after “geometric co”.
  • 8. Conclusion: Future work will test whether Homographic Adaptation improves models for semantic segmentation and object detection.The passage gives SegNet as an example of semantic segmentation and SSD as an example of object detection.
  • 8. Conclusion: Future work will also examine how interest point detection and description, and potentially other tasks, benefit each other.The authors state that these interactions require careful investigation.
  • 8. Conclusion: The authors believe SuperPoint can address visual data-association across 3D computer vision problems such as SLAM and SfM.They specifically identify these as potential applications of the network.
  • 8. Conclusion: A learning-based Visual SLAM front-end could enable more robust applications in robotics and augmented reality.This is presented as an expected application of the proposed approach.

A. Evaluation Metrics

Evaluation follows the protocol of, adapting detection-distance measurements to 2D point-center distances for the fully-convolutional model. The metrics assess corner detection, localization, repeatability, descriptor discrimination, joint pipeline performance, and homography estimation.

  • Corner Detection Average Precision: Corner detection uses Precision-Recall curves, Average Precision, localization error for correct detections, and repeatability rate, with threshold ε defining correctness.Average Precision ranges from 0 to 1, and larger AP is better.
  • Localization Error: Localization Error is computed only for correct detections and ranges between 0 and ε, with lower LE preferred.It complements the Average Precision analysis.
  • Repeatability: Repeatability measures whether a point detected in one image is detected in the second, using 2D point-center distances and threshold ε instead of patch overlap.This substitutes for patch-overlap evaluation because SuperPoint is fully convolutional and does not extract patches.
  • Descriptor and Pipeline Metrics: Nearest Neighbor mean Average Precision evaluates descriptor discrimination using the AUC of Precision-Recall curves across descriptor-distance thresholds, symmetrically averaged over image pairs.Matching Score evaluates the detector and descriptor together as recovered ground-truth correspondences divided by proposed features in the shared viewpoint region.
  • Homography Estimation: Homography estimation compares how estimated and ground-truth homographies transform four image corners rather than directly comparing differently scaled 3 × 3 matrices.Threshold ε denotes a correct homography, and the scores range between 0 and 1, with higher values better.

B. Additional Synthetic Shapes Experiments

On Synthetic Shapes, MagicPoint detectors were evaluated across 10 rendered geometry and negative-image categories, with results compared under clean and noisy conditions. They outperformed classical detectors across categories, while also showing limited blob-center detection capability.

  • Dataset and evaluation: The Synthetic Shapes dataset contains 10 categories of rendered geometric images with ground-truth corners, including negative images without corners.Categories include triangles, quadrilaterals, lines, cubes, checkerboards, stars, ellipses, and random noise images.
  • Noise robustness: MagicPointL and MagicPointS are relatively unaffected by imaging noise compared with the classical detectors.The evaluation computes Average Precision and Localization Error with and without added imaging noise.
  • Detection results: MagicPoint detectors outperform classical detectors in all categories, with a significant mAP gap under imaging noise.Mean Average Precision is higher-is-better, while Mean Localization Error is lower-is-better.
  • Blob detection: MagicPoint can detect blob centers when the entire shape is not too large, but blob confidences are typically lower than corner confidences.Because of the lower confidence, combining blob and corner detections in one system is cumbersome, so blob training is omitted from the main experiments.
  • Blob detection: MagicPoint detects the centers of 71 pixel rectangles, indicating that its network receptive field is at least 71 pixels.The experiment varies square widths in 96 × 96 images and measures confidence at the center blob and top-left corner.

C. Homographic Adaptation Experiment · D. Extra Qualitative Examples

The Homographic Adaptation experiment separates within-scale from across-scale aggregation and finds within-scale aggregation has the largest effect on repeatability. Additional HPatches examples compare SuperPoint with LIFT, SIFT, and ORB using correspondence visualizations.

  • C. Homographic Adaptation Experiment: Within-scale aggregation preserves interest points under rotations and translations, whereas across-scale aggregation accounts for features visible at different image resolutions.The experiment distinguishes transformations within one scale from aggregation across multiple scales.
  • C. Homographic Adaptation Experiment: The Homographic Adaptation experiment varies the number of applied homographies and separately isolates the effect of scale.The reported homography settings are small Nh = 10, medium Nh = 100, and large Nh = 1000.
  • C. Homographic Adaptation Experiment: Across-scale aggregation treats the number of scales Ns as a hyper-parameter and can weight higher-resolution pyramid levels more heavily.Ns = 1 means no multi-scale aggregation; Ns > 1 defines a multi-scale image pyramid.
  • C. Homographic Adaptation Experiment: Within-scale aggregation has the biggest effect on repeatability in held-out MS-COCO experiments.The comparison evaluates both within-scale and across-scale aggregation, with results summarized in Figure 14.
  • D. Extra Qualitative Examples: Extra qualitative HPatches examples show SuperPoint, LIFT, SIFT, and ORB matching results.The examples extend those presented earlier in Figure 8.
  • D. Extra Qualitative Examples: The HPatches visualizations mark correct correspondences with green lines, matched points with green dots, mismatched points with red dots, and points outside the shared viewpoint region with blue dots.These encodings are used in Figure 15’s extra qualitative results.
Loading 1712.07629v4…