Source-linked AI summary
LF-Net: Learning Local Features from Images
Yuki Ono, Eduard Trulls, Pascal Fua, Kwang Moo Yi
TL;DR
Local-feature learning needs an end-to-end pipeline that does not depend on hand-crafted detector outputs, but geometry-based target construction is non-differentiable. LF-Net uses a two-branch architecture with depth and camera geometry to train detector and descriptor components, and it outperforms state-of-the-art methods indoors and outdoors while reaching 60 fps for QVGA images.
Problem
Sparse pipelines may require hand-crafted detectors or separate component training, while dense-correspondence learning faces non-differentiable keypoint selection and too few cross-image matches.
Method
LF-Net combines a dense multi-scale detector with a patch descriptor and trains two network branches using depth and camera geometry to create virtual targets while keeping one branch differentiable.
Results
LF-Net outperforms the state of the art on indoor and outdoor datasets and runs at 60 fps for QVGA images.
Takeaways & Limitations
The complete local-feature pipeline can be learned from image collections without hand-crafted priors, while retaining fast inference.
Takeaways & Limitations
Training assumes depth and camera parameters and excludes image regions without depth measurements or containing occlusions.
Abstract
from arXiv · showhide
We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60+ fps for QVGA images.
1 Introduction
LF-Net addresses limitations of sparse feature pipelines by learning the detector and descriptor end-to-end without hand-crafted detector supervision. Its two-branch training uses known scene geometry to create virtual targets while preserving differentiability in the optimized branch.
- Motivation: Sparse methods either train only parts of the pipeline or require hand-crafted detectors to initialize end-to-end training.These limitations can reduce full-pipeline gains or discard image regions missed by the hand-crafted detector.
- Approach: LF-Net learns a sparse-matching pipeline end-to-end without hand-crafted detectors, using image pairs with relative pose and corresponding depth maps.The geometry can come from laser scanners or shape-from-structure algorithms, without further annotation.
- Motivation: End-to-end learning from dense correspondences is difficult because extracting multiple score-map maxima is non-differentiable.Independent keypoint extraction on both images also yields few ground-truth matches for descriptor training.
- Approach: The method creates a virtual target by detecting maxima in one image, warping selected keypoints using ground-truth geometry, and optimizing the other branch toward sharp corresponding responses.This guarantees a large pool of ground-truth matches while confining the non-differentiable operation to one branch.
2 Related work
Prior local-feature research developed hand-crafted and learned components for detection, orientation, and description, but often optimized components separately or relied on existing keypoints. LF-Net is situated within efforts to learn the full pipeline while addressing limitations of repeatability-focused evaluation and SIFT-dependent training.
- Context: Local features remain central to wide-baseline matching because they are versatile and robust to occlusions that can challenge dense methods.Dense alternatives typically work best on narrow baselines and may suffer in occluded scenes.
- Pipeline components: Feature extraction and matching typically comprise interest-point detection, orientation estimation, and descriptor creation.Many learned methods target only one component rather than the integrated pipeline.
- Hand-crafted methods: Hand-crafted approaches such as SIFT, SURF, and DAISY integrate local-feature processing while emphasizing broad functionality or computational efficiency.SIFT provided an integrated solution; SURF and DAISY addressed efficiency through filters, integral images, and convolutional computations.
- Learned methods: Deep learning renewed efforts to replace all local-feature components with convolutional networks, following earlier machine-learning work focused mainly on descriptors.Learned approaches include metric learning, convex optimization, and neural architectures for keypoints and descriptors.
- Orientations: Prior orientation-learning work predicted orientations that minimize distances between orientation-dependent descriptors, assuming keypoints came from another technique.This separates orientation estimation from keypoint extraction.
- Pipeline evaluation: Benchmark gains for individual components may not survive full-pipeline evaluation, while repeatability can select keypoints that are poor for matching.Descriptor robustness can also be unnecessary or counterproductive when patches are already aligned.
- Learned full pipelines: A fully differentiable prior architecture showed joint training is necessary but still relied on SIFT keypoints, preventing learning where SIFT fails.This motivates learning keypoint selection without that hand-crafted training prior.
3 Method
LF-Net combines a dense multi-scale detector with a patch descriptor network in an end-to-end local feature pipeline. Its two-branch training uses geometry-derived correspondences while confining non-differentiable operations to one branch.
- LF-Net architecture: LF-Net comprises a dense, multi-scale, fully convolutional detector and a descriptor network operating on patches around selected keypoints.The detector returns keypoint locations, scales, and orientations, while the descriptor network outputs local descriptors.
- Keypoint detection: The feature map is processed across five scales between 1/2 and 2, with 5×5 filters and differentiable spatial non-maximum suppression.Responses are resized to the original image size and merged into a final scale-space score map.
- Keypoint detection: Top-scoring pixels become keypoints, while local softargmax provides sub-pixel coordinates and scale softargmax predicts each keypoint’s scale.The top-K selection is non-differentiable, but gradients still flow through selected points and the softargmax refinement.
- Descriptor extraction: Descriptors are computed from normalized 32×32 patches cropped around keypoints using differentiable bilinear sampling.Each descriptor input is defined by a keypoint’s location, scale, and orientation.
- Learning LF-Net: Training uses two image branches with geometry-derived score-map and patch correspondences, combining image-level and patch-level losses while restricting non-differentiable operations to one branch.Depth, camera pose, and intrinsics warp detected points; occluded regions are excluded from optimization.
4 Experiments
Experiments evaluate LF-Net on indoor and outdoor datasets using matching score, comparing it with traditional and learned local-feature pipelines. LF-Net performs strongly outdoors, while indoor performance is affected by larger frame changes and depth inaccuracies; joint detector–descriptor training improves results.
- 4.1 Datasets: The experiments use ScanNet indoors and 25 photo-tourism collections outdoors, with incomplete sensor depth indoors and noisy reconstructed depth outdoors.Indoor evaluation uses the full set of 312 test sequences; outdoor depth is obtained with COLMAP and filtered for invalid or occluded pixels.
- 4.2 Baselines and metrics: Performance is measured by matching score: the ratio of nearest-neighbor descriptor correspondences that are correct under ground-truth geometry, using a 5-pixel localization threshold.The threshold emphasizes accurate keypoint localization in scenes with complex geometry.
- 4.3 Results on outdoors data: 9% relative improvement over SuperPoint is reported without rotation-scale estimation, while LF-Net exceeds the closest traditional-pipeline competitor, LIFT, by 45% relative with those augmentations.The no-rotation-scale setup suits mostly upright, similarly scaled images; the augmented setup compares explicit rotation and scale handling.
- 4.4 Results on indoors data: LF-Net matches SuperPoint at 10-frame differences indoors but is 6% relative worse on average for larger frame differences.The authors associate this gap with informative 3D boundaries being excluded because of Kinect depth inaccuracies.
- 4.5 Ablation study: Joint detector–descriptor training increases average matching score by 7% relative outdoors (.299 to .321) and 1% relative indoors (.445 to .450).The ablation sets λ_pair = 0 to separate detector and descriptor training; joint training improves both datasets, with a smaller indoor gain.
5 Conclusions
LF-Net learns a complete local-feature pipeline end-to-end from image collections using a two-branch training strategy with virtual targets. It outperforms the state of the art on indoor and outdoor datasets while operating at 60 fps for QVGA images.
- 5 Conclusions: LF-Net embeds the entire feature-extraction pipeline and trains end-to-end from image collections using iteratively created virtual target responses.The non-differentiable process runs in one branch while the other remains differentiable for optimization.
- 5 Conclusions: LF-Net outperforms the state of the art by a large margin on both indoor and outdoor datasets and runs at 60 fps for QVGA images.The conclusion reports the cross-dataset result together with the stated computational speed.
6 Appendix
The appendix examines cross-domain generalization, computational cost, geometric precision, and rotation robustness. LF-Net maintains strong matching performance while offering fast extraction, though precision evaluation depends on threshold and data quality.
- 6.1 Generalization performance: 18% relative performance drops from outdoor-trained models tested indoors, versus 8% from indoor-trained models tested outdoors.Matching scores change from .450 to .370 and from .321 to .295, respectively.
- 6.2 Computational cost: 62 fps extraction is achieved for 512 keypoints on QVGA frames, while VGA extraction reaches 25 fps on a Titan X PASCAL.For 1024 keypoints, the corresponding rates are 42 fps and 20 fps.
- 6.3 Geometric precision: LF-Net outperforms SIFT and SURF across pixel thresholds above sub-pixel, sometimes doubling their performance on particular datasets.The evaluation reports thresholds from 1 to 5 pixels; sub-pixel results are considered unreliable because depth estimates are noisy.
- 6.4 Results on the ‘hpatches’ dataset: On the hpatches dataset, LF-Net outperforms classical algorithms except at the sub-pixel level.The comparison is reported using matching scores across multiple pixel thresholds.
- Rotation robustness: Models learned without rotation invariance can perform well on aligned images but perform very poorly overall under applied in-plane rotations.Rotation tests use outdoor image pairs with rotations every 10° from 0° to 360°.