Source-linked AI summary
Quad-networks: unsupervised learning to rank for interest point detection
Nikolay Savinov, Akihito Seki, Lubor Ladicky, Torsten Sattler, Marc Pollefeys
TL;DR
Interest-point detection needs sparse points that remain repeatable under domain transformations, yet what makes a point interesting is often too ambiguous for unbiased human labeling. The paper learns a transformation-invariant neural ranking from scratch and extracts its extreme quantiles as detections, achieving superior or comparable performance to DoG for RGB and RGB/depth settings.
Problem
Interest-point labels are highly ambiguous because desired points must be sparse and repeatable under viewpoint or illumination changes, including across RGB and depth modalities.
Method
A neural network learns a real-valued ranking whose ordering is repeatable under transformations, with top and bottom quantiles used as interest points.
Results
The method achieves superior or comparable performance to DoG for RGB detection from scratch and cross-modal RGB/depth detection.
Takeaways & Limitations
Unsupervised transformation-invariant ranking provides a detector-learning formulation that avoids human assessment and can be applied across RGB/RGB and RGB/depth tasks.
Takeaways & Limitations
The formulation assumes transformations preserve point correspondences, while some invariant or augmenting transformations rely on distributions from real data.
Abstract
from arXiv · showhide
Several machine learning tasks require to represent the data using only a sparse set of interest points. An ideal detector is able to find the corresponding interest points even if the data undergo a transformation typical for a given domain. Since the task is of high practical interest in computer vision, many hand-crafted solutions were proposed. In this paper, we ask a fundamental question: can we learn such detectors from scratch? Since it is often unclear what points are "interesting", human labelling cannot be used to find a truly unbiased solution. Therefore, the task requires an unsupervised formulation. We are the first to propose such a formulation: training a neural network to rank points in a transformation-invariant manner. Interest points are then extracted from the top/bottom quantiles of this ranking. We validate our approach on two tasks: standard RGB image interest point detection and challenging cross-modal interest point detection between RGB and depth images. We quantitatively show that our unsupervised method performs better or on-par with baselines.
1. Introduction
The paper frames interest-point detection as an unusually ambiguous labeling problem and proposes learning detectors without human assessment. Its method learns a transformation-invariant ranking whose extreme quantiles provide repeatable interest points.
- Motivation: Interest points should remain repeatable under substantial viewpoint or illumination changes while forming a sparse set of image locations.Such points can support correspondence matching and sparse 3D-structure or camera-position estimation.
- Motivation: Human labeling is unsuitable because what counts as interesting is unclear and annotators may select visually salient but non-repeatable corners or blobs.The ambiguity is especially severe when the desired points must work across different image modalities.
- Motivation: Cross-modal RGB/depth detection requires corresponding points despite major visual differences between natural RGB images and depth maps.Texture-driven RGB features may be absent from depth, making simple hand-crafted heuristics unreliable.
- Contribution: The paper presents unsupervised learning to rank as a way to learn interest-point detectors independently of human assessment.It organizes the paper around the formulation, image application, experimental validation, and conclusion.
- Approach: The proposed approach trains a neural network to rank points so their ordering remains repeatable under desired transformations.Interest points are extracted from the ranking’s top and bottom quantiles.
2. Related work
Related work spans unsupervised auxiliary-task learning, hand-crafted and supervised image detectors, and the unresolved challenge of cross-modal repeatability. The paper positions ranking-based detection as a general procedure that avoids unrelated auxiliary tasks and applies across RGB/RGB and RGB/depth pairs.
- Unsupervised formulation: Unlike auxiliary-task approaches in representation learning, the method uses a repeatable ranking directly rather than designing an unrelated auxiliary task.The top and bottom ranking quantiles serve as detections.
- Image detection: Existing image detectors commonly hand-craft responses for blobs, corners, or edges, while newer methods often learn subsets from hand-crafted detections.Examples include DoG, Harris, FAST, MSER, LIFT, and TILDE.
- Unsupervised formulation: The proposed procedure learns the detector from scratch, avoiding reliance on existing hand-crafted detectors in complicated cross-modal cases.This contrasts with supervised approaches such as LIFT that select subsets of DoG detections.
- Cross-modal detection: Cross-modal interest-point detection requires repeatability across different image modalities, but prior works did not propose a general solution.The paper demonstrates its procedure for both RGB/RGB and RGB/depth modality pairs.
3. Detection by ranking
The detection problem is formulated as learning a response-based ranking that preserves ordering among corresponding points under transformations. Training penalizes ranking violations with a differentiable objective, and extreme response quantiles become detections.
- Problem formulation: Objects undergo transformations with known corresponding point indexes, and the ranking constraints are imposed on those correspondences.Each point has at most one correspondence in the transformed object under the stated assumption.
- Problem formulation: The detector ranks points using a single real-valued response function H(p|w), potentially implemented by a neural network.Ranking invariance requires corresponding point pairs to preserve their relative order after transformations.
- Detection rule: Top and bottom quantiles of the response function are selected as interest points after sorting all object points by response.Observation 1 states that, when every point has a correspondence, these quantiles correspond before and after transformation.
- Optimization: The ranking objective aggregates penalties for non-positive ranking agreement across objects, transformations, and corresponding point pairs.The training sets and transformation sets are assumed finite for optimization.
- Optimization: A differentiable hinge-loss upper bound replaces the hard-to-optimize misranking-count loss.Because the resulting objective is differentiable when H is differentiable with respect to w, gradient descent can be used, although the objective remains non-convex.
4. Image interest point detector learning
The detector learns rankings that remain consistent across desired transformations, allowing sparse interest points to survive contrast filtering and non-maximum suppression. Training uses transformed correspondence quadruples to enforce invariance and augment data.
- Transformation-invariant ranking: The objective learns a response ranking whose relative ordering remains consistent across corresponding patches under desired transformations.The detector represents each point with a scalar response and optimizes ranking constraints across image transformations.
- Sparse detection: Top and bottom response quantiles provide sparse detections through contrast filtering.This follows from the ranking property when every point has a correspondence.
- Sparse detection: Local extrema are preserved across corresponding visible neighborhoods, making non-maximum suppression compatible with the learned objective.A point is a local extremum in one image if and only if its correspondence is a local extremum in the transformed image.
- Sparse detection: The resulting detector pipeline combines non-maximum suppression with contrast filtering, as in many existing detectors.The paper specifically notes that this pipeline is used by the popular DoG detector.
- Training data: Training samples images and transformations because available correspondence datasets alone do not guarantee the desired invariances or augmentation conditions.The paper highlights missing relative rotations and illumination changes as examples of training distributions that may not match testing needs.
- Training data: Random transformation quadruples enforce invariance or create augmentation by choosing whether paired elements share transformation assignments.Invariance uses (ti1, ti1, ti2, ti2), while augmentation uses (ta1, ta2, ta1, ta2); non-parametrizable effects rely on real-data distributions.
5. Experiments
The experiments evaluate learned response functions for RGB, fully unsupervised RGB, and cross-modal RGB/depth detection against DoG and related baselines. Across these settings, the learned methods generally improve repeatability or matching, including with deep convolutional models.
- Experimental settings: The experiments compare three detector settings: RGB with ground-truth correspondences, fully unsupervised RGB, and cross-modal RGB/depth detection.The cross-modal detector uses view-aligned RGBD frames, while the RGB ground-truth setup uses projected laser-scanned 3D points.
- Evaluation protocol: The evaluation replaces only the response filter while retaining non-maximum suppression, localization, and contrast filtering in the detector pipeline.Interest points are detected in scale-space, where non-maximum suppression uses a 3 × 3 × 3 neighborhood.
- Evaluation protocol: Repeatability is compared at fixed numbers of interest points using the 40% overlap threshold, avoiding advantages from detectors that simply return more points.The reported measure is the ratio of correctly detected points to detections in the image with fewer points.
- RGB results: Learned linear and non-linear response functions outperform baselines in most cases, with the non-linear model better than the linear model in the majority of cases.These results are averaged over image pairs for each transformation type.
- RGB results: The learned detectors combined with SIFT are slightly better than DoG+SIFT in most cases, but perform worse on UBC JPEG-compression data absent from training.The comparison uses the ratio of correct matches to all matches.
- Cross-modal results: The Deep Conv Net achieves overall better repeatability than baselines in the cross-modal RGB/depth experiment.DoG produces few depth detections because depth images are smooth and lack texture, whereas the learned methods produce more points through a peakier response distribution.
6. Conclusion
The paper proposes an unsupervised interest point detector that ranks object points and uses ranking quantiles as interest points, achieving superior or comparable performance to DoG in RGB and RGB-depth settings.
- The method learns a repeatable ranking of object points and extracts interest points from its top and bottom quantiles.The detector is learned without human assessment and demonstrated for images and cross-modal RGB-depth detection.
- The approach performs superiorly or comparably to DoG for RGB detection learned from scratch and for repeatability between RGB and depth modalities.
- Future work includes jointly learning descriptors with the detector and applying the method beyond images, such as video interest frame detection.