Source-linked AI summary
Learning to Assign Orientations to Feature Points
Kwang Moo Yi, Yannick Verdie, Pascal Fua, Vincent Lepetit
TL;DR
Feature-point orientation estimates can be unreliable under viewpoint and illumination changes, while choosing a canonical target to learn is difficult. The paper uses Siamese training to learn orientations from image patches and reports improved descriptor matching over the state of the art across datasets without retraining for each one.
Problem
Orientation estimates can be inaccurate in complex scenes, and there is no clear canonical target orientation to supervise directly.
Source: §1 · p. 1, §3.1 · p. 3Method
A CNN predicts feature-point orientations from patches, with a Siamese network learning the target implicitly and a GHH-based activation for the task.
Source: §1 · p. 2Results
The learned orientations consistently improve descriptor performance and outperform state-of-the-art descriptors on all evaluated datasets.
Source: §5 · p. 8Takeaways & Limitations
The learned orientation assignment can be paired with existing rotation-sensitive descriptors without retraining for each dataset.
Source: §1 · p. 2Descriptor gradients are not straightforward to compute: SIFT uses non-differentiable histograms, and the pooling region can change with orientation.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
We show how to train a Convolutional Neural Network to assign a canonical orientation to feature points given an image patch centered on the feature point. Our method improves feature point matching upon the state-of-the art and can be used in conjunction with any existing rotation sensitive descriptors. To avoid the tedious and almost impossible task of finding a target orientation to learn, we propose to use Siamese networks which implicitly find the optimal orientations during training. We also propose a new type of activation function for Neural Networks that generalizes the popular ReLU, maxout, and PReLU activation functions. This novel activation performs better for our task. We validate the effectiveness of our method extensively with four existing datasets, including two non-planar datasets, as well as our own dataset. We show that we outperform the state-of-the-art without the need of retraining for each dataset.
1. Introduction
The introduction identifies unreliable orientation estimates in complex scenes as a weakness of feature matching and presents learned orientation assignment as a way to improve rotation-sensitive descriptors. It outlines Siamese-network training, a new activation function, and evaluation across varied datasets without dataset-specific retraining.
1. Introduction
- The paper frames canonical orientation assignment as an important but largely overlooked step despite extensive research on feature-point detectors and descriptors.The introduction notes that SIFT’s dominant orientation is commonly assumed to work well.Source: §1 · p. 1
- Viewpoint and illumination changes can alter feature-point appearance on non-planar surfaces, producing orientation errors; rotation-invariant descriptors also discard useful rotation-sensitive information.The paper argues that better orientation assignments can improve matching with rotation-sensitive descriptors.Source: §1 · pp. 1–2
- A CNN predicts feature-point orientation from an image patch, while Siamese training treats the target orientation as an implicit variable and the descriptor as a black box.This avoids having to specify a target canonical orientation and allows use with rotation-sensitive descriptors such as SIFT, SURF, and VGG.Source: §1 · p. 2
- The proposed GHH-based activation generalizes ReLU, maxout, and PReLU and performs better for this task.The introduction identifies it as a key component of the method.Source: §1 · p. 2
- The evaluation spans planar or distant-object datasets, 3D-object datasets, and a new dataset with complex camera movements, with reported state-of-the-art improvements and no dataset-specific retraining.The new dataset includes in-plane rotations and viewpoint changes.Source: §1 · p. 2
2. Related Work
Related work covers handcrafted orientation assignment, rotation-invariant descriptors, and learned matching or descriptors. The paper positions its approach as learning orientations for general feature points while retaining rotation-sensitive descriptor information.
Orientation assignment of feature point detectors
- Handcrafted methods estimate dominant orientation in different ways, including SIFT, SURF, MOPs, ORB, and HIP, with HIP noted as noise-sensitive.The methods use gradient histograms, wavelet responses, patch gradients, image moments, or intensity differences.Source: §2 · p. 2
Rotation invariant descriptors
- Rotation-invariant descriptors include MROGH and LIOP, while BRISK and FREAK still depend on orientation estimation during extraction.Other approaches use spin-image histograms or rotation-invariant distance functions.Source: §2 · p. 2
- Although some rotation-invariant methods may outperform original SIFT, SIFT with the paper’s learned orientation estimation is reported to outperform them.The authors suggest that discarding rotation-sensitive information may explain this difference; also requires matching pairs across the whole scene.Source: §2 · p. 2
Learning-based methods
- Prior learning-based work addresses patch pose or learns patch comparisons and descriptors, but does not learn orientation assignment for general feature points.One pose method uses a separate regressor per patch, which the passage says is not viable for general feature points.Source: §2 · pp. 2–3
- Learning-based descriptors still rely on traditionally handcrafted local orientation estimates, and often train on Brown patches oriented using SfM ground truth that may not be practical or transfer reliably.The passage notes possible performance degradation on other data when orientation estimates are inaccurate.Source: §2 · p. 3
3. Method
The method learns feature-point orientations by optimizing descriptor distances for corresponding image patches, avoiding the need for target orientations. It also introduces a learnable piecewise-linear activation intended to improve orientation assignment.
3.1. Canonical Orientation as an Implicit Variable
- Heuristic targets such as SIFT’s dominant orientation did not reliably identify the ideal canonical orientation, so the method learns orientations by minimizing descriptor distances for matching feature pairs.The optimization is intended to find orientations that are reliably predictable and improve matching.Source: §3.1 · p. 3
3.2. Formalization
- Training uses pairs of patches depicting the same physical 3D point, and optimizes CNN parameters through the resulting descriptor distances without prescribing target orientations.The descriptor is treated as a black box, allowing different rotation-sensitive descriptors to be used.Source: §3.2 · p. 3
- Instead of directly predicting an angle or discretized histogram, the CNN predicts two values that are converted to an angle with arctan2.Direct angle prediction and histogram-like outputs performed poorly in early experiments; histogram outputs also require discretization and can make the network too large at fine resolutions.Source: §3.2 · pp. 3–4
- Because descriptor gradients can be difficult to compute, the method approximates them numerically using descriptors computed at orientations spaced every 5 degrees.Analytic gradients are also possible for descriptors that have an analytic form.Source: §3.2 · p. 4
3.3. Generalized Hinging Hyperplane Activation
- The proposed GHH activation learns a piecewise-linear function, generalizing maxout, ReLU, and PReLU through settings of its segment parameters.Its greater flexibility is presented as useful for learning orientations.Source: §3.3 · p. 4
4. Results
The experiments evaluate learned orientation assignments across multiple descriptors and datasets, including non-planar scenes, and compare them with existing methods. EF-VGG⋆ achieves the strongest reported matching results, while GHH improves performance over the tested activation functions.
4.1. Dataset, Training, and Evaluation Setup
- Oxford is used for training, while the other datasets test the method; Strecha and DTU provide non-planar 3D scenes, and matching is evaluated with mAP.Orientation estimation takes 0.47 milliseconds per feature point; with SIFT descriptors, the total is 1.39 milliseconds per feature point.Source: Figure 3 · p. 5, §4.1 · pp. 5–6
4.2. Descriptor Matching Performances
- 27.4% better mAP than EF-VGG is reported for EF-VGG⋆, which also ranks first across the datasets; EF-Daisy⋆ outperforms the compared methods as well.The gains from learned orientations are consistent across descriptors and exceed those from multiple orientations on average; multiple orientations increase descriptor extraction time by 34% and matching time by 79%.Source: §4.2 · pp. 6–7, Figure 5 · p. 7, Table 1 · p. 7
4.3. Performance With Different Activations
- GHH yields consistent matching gains over ReLU, Tanh, maxout, and PReLU in the EF-SIFT+ comparison.The comparison holds other parameters identical except activation type and fully connected layer output counts.Source: §4.3 · p. 7, Figure 6 · p. 8
4.4. Results on Upright and Non-upright Datasets
- On non-upright datasets, EF-Daisy⋆ and EF-VGG⋆ significantly outperform the alternatives, while on upright datasets they approach the zero-orientation upper bounds.The paper defines “Up” assignments as zero orientations; EF-VGG performs similarly to EF-VGG-Up, so inaccurate assignments are not helpful.Source: §4.4 · pp. 7–8, Figure 7 · p. 8
4.5. Application to Multi-View Stereo
- EF-VGG⋆ produces the most MVS vertices on both tested sequences, followed by EF-VGG and then EF-Daisy.For Strecha’s fountain sequence, the counts are 365261, 323979, and 260747 vertices, respectively; for DTU Scene 55, they are 95014, 86826, and 72972.Source: §4.5 · p. 8, Figure 8 · p. 8
5. Conclusion
The paper presents learned canonical orientations that improve existing descriptors, with Siamese training avoiding the need to define a target orientation. It also proposes GHH activations and identifies joint learning of orientation and descriptors as future work.
5. Conclusion
- Descriptors using learned orientations consistently improve performance and outperform state-of-the-art descriptors across all evaluated datasets.The authors also report that the GHH activation function is effective for their task.Source: §5 · p. 8
- A Siamese network learns feature-point orientations without explicitly defining a target orientation.The paper’s CNN estimates a canonical orientation from a patch around each feature point.Source: §5 · p. 8
- Jointly learning orientation assignment and a descriptor in a fully differentiable Siamese network is proposed as future work.The paper notes that its method enhanced the learning-based VGG descriptor, while full integration remains a future direction.Source: §5 · p. 8
A.1. Implementations of the Methods
The compared feature points are capped at 1,000, with detector and descriptor implementations using the stated libraries, defaults, and selected parameter settings.
A.1. Implementations of the Methods
- Feature points are ranked by detector response, and only the top 1,000 are retained.This cap is used to keep the maximum number of feature points at 1,000.Source: §A.1 · p. 11
- ORB uses nFeatures=1000 and nLevels=3, BRISK uses threshold 20, and SURF and SIFT use default parameters.ORB, SURF, and SIFT use OpenCV; BRISK is provided by its authors.Source: §A.1 · p. 11
- LIOP uses VLFeat, and the methods are implemented with Theano.The LIOP parameters are defaults; the implementation statement identifies Python Theano.Source: §A.1 · p. 11
- KAZE, EF, BiCE, FREAK, MROGH, and sGLOH use author-provided implementations with default parameters.The implementations are specified in the respective passages; the cited settings report defaults.Source: §A.1 · p. 11
- Daisy patches use four times the scale, while other Daisy parameters remain at their defaults.The four-times scale is the value used by the authors in.Source: §A.1 · p. 11
- VGG uses the pre-learned Liberty model, patches with relativeExtent 7.5, and the EF detector.The relativeExtent matches SIFT; the Liberty model is used because the other two datasets are partially included in the test set.Source: §A.1 · p. 11