Source-linked AI summary
TILDE: A Temporally Invariant Learned DEtector
Yannick Verdie, Kwang Moo Yi, Pascal Fua, Vincent Lepetit
TL;DR
Keypoint detectors are sensitive to drastic weather and lighting changes, motivating a detector that remains repeatable across such conditions. TILDE learns stable points from same-viewpoint training images, trains a regressor to produce score-map maxima, and reports superior performance on its new benchmark while matching state-of-the-art performance on Oxford and EF.
Problem
Existing keypoint detectors can lose reliability under outdoor changes in weather, season, and time of day, and no standard benchmark tests these temporal changes.
Method
TILDE identifies repeatable keypoint candidates across same-viewpoint training images and trains a regressor whose score-map maxima define keypoints.
Results
TILDE significantly outperforms state-of-the-art methods on the new outdoor benchmark and achieves state-of-the-art performance on Oxford and EF datasets.
Takeaways & Limitations
The results demonstrate generalization from challenging outdoor conditions to the Oxford and EF datasets.
Takeaways & Limitations
The method’s scale-space extension remains a future research direction.
Abstract
from arXiv · showhide
We introduce a learning-based approach to detect repeatable keypoints under drastic imaging changes of weather and lighting conditions to which state-of-the-art keypoint detectors are surprisingly sensitive. We first identify good keypoint candidates in multiple training images taken from the same viewpoint. We then train a regressor to predict a score map whose maxima are those points so that they can be found by simple non-maximum suppression. As there are no standard datasets to test the influence of these kinds of changes, we created our own, which we will make publicly available. We will show that our method significantly outperforms the state-of-the-art methods in such challenging conditions, while still achieving state-of-the-art performance on the untrained standard Oxford dataset.
1. Introduction
TILDE addresses the sharp loss of keypoint reliability under outdoor changes in weather, season, and time of day. It learns stable points from repeated observations and reports strong performance on challenging and standard datasets.
- Motivation: Outdoor changes in weather, seasons, and time of day can severely degrade keypoint repeatability, even with illumination-invariant descriptors.This limits matching across fair and foul weather and between different times of day.
- Approach: The approach identifies potentially stable training points, trains a regressor to produce local score-map maxima, and extracts new keypoints with non-maximum suppression.The regressor is designed to handle complex appearance variation efficiently and reliably.
- Evaluation: The authors report significant gains over state-of-the-art methods on the new benchmark while retaining state-of-the-art performance on Oxford and EF datasets.They evaluate repeatability using the same measure across the new, Oxford, and EF datasets.
- Approach: TILDE learns a detector that extracts keypoints stable under drastic illumination changes caused by weather, season, and time of day.The method is designed to support matching under these challenging conditions.
- Contributions: The paper contributes a method for generating good keypoints to learn and a new benchmark dataset for outdoor images captured across times and seasons.The dataset combines AMOS webcam images with panoramic images and is intended for public release with the code.
2. Related Work
Prior keypoint detectors improve performance on standard datasets but remain vulnerable to illumination changes and context-specific limitations. TILDE proposes a generic learned detector and efficient task-specific regressor for comparison across datasets.
- Handcrafted Keypoint Detectors: Handcrafted detectors have improved on standard benchmarks but still suffer severe performance drops under illumination changes.Their advantages can also depend on specialized structures or application contexts.
- Handcrafted Keypoint Detectors: Handcrafted methods are difficult to adapt to context, limiting their flexibility for matching similar outdoor scenes.Examples include methods tailored to camera calibration or symmetric objects.
- Learned Keypoint Detectors: Earlier learned detectors primarily targeted extraction speed, depended on initial detectors, or achieved limited repeatability improvement.The related methods include FAST, FAST-ER, matchability learning, and learned filters for visual odometry.
- TILDE: TILDE proposes a generic scheme for learning keypoint detectors and a novel efficient regressor specified for this task.The paper compares it with state-of-the-art handcrafted methods and TaSK across several datasets.
- TILDE: Figure 2 summarizes the pipeline from same-viewpoint training images through regressed score maps to keypoints selected at local maxima.The figure provides the conceptual bridge between the training setup and detector output.
3. Learning a Robust Keypoint Detector
TILDE learns robust keypoint detection from aligned, same-viewpoint images captured under changing conditions. It selects repeatable locations as training positives, trains a peaked-response regressor, and uses score-map maxima for detection.
- Training Setup: The training setup assumes images of the same scene and viewpoint captured across different seasons and times of day.Locations consistently visible across these conditions are treated as candidate stable points.
- Training Samples: Positive samples are patches centered on candidate stable locations, while patches far from them provide negative samples.The regressor is trained to produce peaked responses at positives and small scores at negatives.
- Detection: Keypoints in new images are extracted by finding local maxima in the regressor’s score map and thresholding low-valued locations.This turns the learned response map into keypoints using simple post-processing.
- Data Collection: Training images come from fixed outdoor webcams in the AMOS dataset and panoramic images captured from a building.The webcam series span different times of day and seasons.
- Candidate Selection: The candidate-generation procedure retains the best 100 repeated locations, using SIFT detections whose distances are smaller than the estimated SIFT scale.Positive patches are collected from every image, including images where the original detector missed the location.
- Candidate Selection: Including missed detections as positive samples focuses learning on reliably detectable points and corrects errors of the original detector.The strategy discards locations detected only infrequently.
4. An Efficient Piece-wise Linear Regressor
The method uses a piece-wise linear GHH regressor whose image-wide responses are computed efficiently from learned filters, while an objective shapes keypoint responses and promotes temporal repeatability.
- Regressor form: The regressor is a piece-wise linear function expressed with Generalized Hinging Hyperplanes (GHH).Its parameters include linear-filter vectors and binary signs, while N and M control model complexity.
- Efficient evaluation: Applying the regressor at every image location requires image convolutions and pixel-wise maximum operators.This avoids the random image access of regression trees and the higher-order convolutions used by most CNN layers.
- Objective function: The objective separates locations near keypoints from distant locations, enforces local maxima at keypoints, and regularizes responses across time.The classification-like, shape, and temporal terms jointly define the training objective.
- Objective function: The shape regularizer constrains response shape without constraining response scale, preserving the separate role of the classification-like loss.Only filters contributing to the max-operator response are used for the shape constraint.
- Optimization: The shape term can be optimized efficiently in the Fourier domain using Parseval’s and the convolution theorems.The response-shape enforcement is generalized to arbitrary shapes, with an efficient approximation of the relevant sample operator.
- Optimization: Training first applies PCA, then greedily adds hyperplanes using trust-region Newton optimization; learned filters are approximated with separable filters for faster convolution.Experiments use N = 4 and M = 4, and meta-parameters are selected by grid-search cross-validation.
5. Results
Experiments evaluate TILDE and its regressor variants using repeatability on Webcam, Oxford, and EF datasets, plus qualitative matching and runtime comparisons. TILDE significantly outperforms compared methods on challenging temporal changes, retains state-of-the-art performance on Oxford and EF, and achieves correct matching under drastic appearance changes.
- Quantitative evaluation: Repeatability counts consistently detected keypoints across aligned images, with matches defined within 5 pixels after projection.The evaluation also modifies the measure to use nearest-neighbor association and a small fixed number of keypoints, reported as Repeatability (2%).
- Webcam dataset: TILDE significantly outperforms state-of-the-art methods on Webcam sequences, both when trained and tested on the same sequence and when tested on unseen sequences.The performance gap is reduced on unseen sequences, but TILDE still outperforms all compared methods by a significant margin.
- Oxford and EF datasets: TILDE achieves state-of-the-art performance on Oxford and EF when trained on the Chamonix Webcam sequence, without learning a new detector on those datasets.It significantly outperforms state-of-the-art methods on the outdoor Bikes, Trees, Leuven, and Rushmore scenes, and obtains good results on Boat despite not modeling scale.
- Qualitative matching: Using SIFT descriptors for all detectors, TILDE retrieves correct transformations between image pairs captured on different days under different weather conditions.The comparison uses 300 keypoints per method and evaluates keypoint detection rather than descriptor quality.
- Objective-function ablations: The best performance uses all three objective-function terms; shape regularization helps on unseen Oxford and EF, whereas temporal regularization helps on images similar to training data.The ablations are TILDE-PC with classification loss, TILDE-PT with classification and temporal regularization, and TILDE-PS with classification and shape regularization.
- Runtime: TILDE-P24 is not very far from SIFT in full-pipeline computation time on a 640×418 image.The method is highly parallelizable, although the current implementation does not use parallelization, so a better implementation could significantly reduce runtime.
6. Conclusion
TILDE learns keypoint detectors that remain reliable under drastic weather and lighting changes, and the regressors generalize across challenging and standard datasets. Scale-space extension remains a future research direction.
- TILDE introduces a learning scheme for reliable keypoint detection under drastic weather and lighting changes.
- The training-set generation method supports learning regressors for repeatable keypoints.
- The piece-wise linear regressor achieved the best result among the three learned regressors.
- TILDE significantly outperforms current state-of-the-art methods on the new benchmark and achieves state-of-the-art performance on Oxford and EF.These results demonstrate generalization capability across the evaluated datasets.
- Extending the method to scale space is identified as future research.