Source-linked AI summary
Learning Local Image Descriptors with Deep Siamese and Triplet Convolutional Networks by Minimising Global Loss Functions
Vijay Kumar B G, Gustavo Carneiro, Ian Reid
TL;DR
Local image descriptor learning lacked evidence for triplet networks and whole-training-set global losses. The paper introduces both approaches and evaluates them on UBC, reporting state-of-the-art results across different model configurations.
Problem
Local image descriptor learning had not established whether triplet networks improve siamese networks, and existing methods did not use a whole-training-set global loss.
Method
The paper trains siamese and triplet ConvNets with global losses that regulate distances among same-class and different-class patches.
Results
Triplet networks outperform siamese networks on UBC classification results, while triplet plus global losses produce the best embedding and global-loss central-surround siamese models produce the best UBC classification result.
Takeaways & Limitations
The models deliver state-of-the-art results on the UBC benchmark, which the authors attribute to better regularisation from the global loss.
Takeaways & Limitations
Preliminary results indicate that the global-loss triplet model does not outperform the models reported in Table 2.
Abstract
from arXiv · showhide
Recent innovations in training deep convolutional neural network (ConvNet) models have motivated the design of new methods to automatically learn local image descriptors. The latest deep ConvNets proposed for this task consist of a siamese network that is trained by penalising misclassification of pairs of local image patches. Current results from machine learning show that replacing this siamese by a triplet network can improve the classification accuracy in several problems, but this has yet to be demonstrated for local image descriptor learning. Moreover, current siamese and triplet networks have been trained with stochastic gradient descent that computes the gradient from individual pairs or triplets of local image patches, which can make them prone to overfitting. In this paper, we first propose the use of triplet networks for the problem of local image descriptor learning. Furthermore, we also propose the use of a global loss that minimises the overall classification error in the training set, which can improve the generalisation capability of the model. Using the UBC benchmark dataset for comparing local image descriptors, we show that the triplet network produces a more accurate embedding than the siamese network in terms of the UBC dataset errors. Moreover, we also demonstrate that a combination of the triplet and global losses produces the best embedding in the field, using this triplet network. Finally, we also show that the use of the central-surround siamese network trained with the global loss produces the best result of the field on the UBC dataset. Pre-trained models are available online at https://github.com/vijaykbg/deep-patchmatch
1. Introduction
Local image descriptors support patch-matching applications and aim to distinguish corresponding 3-D locations while remaining robust to brightness and geometric changes. This paper proposes triplet networks and global losses, then reports improved UBC benchmark results for several model combinations.
- Local image descriptors are used in wide baseline stereo, structure from motion, and image classification.
- Effective descriptors should be unique for patches from different 3-D locations and robust to brightness and geometric deformations.
- Recent deep ConvNet methods learn descriptors with pairwise siamese losses or multiclass classification losses.
- The paper proposes triplet networks and a global loss applicable to siamese and triplet architectures.The global loss reduces same-class distance and increases different-class distance while minimising variance between their distance distributions.
- On the UBC dataset, triplet networks outperform siamese networks, triplet plus global losses produce the best embedding, and global-loss central-surround siamese models produce the best classification result.
2. Related Work
Prior local image descriptor methods include metric learning, deep siamese networks, and single-network classification approaches. The paper identifies two gaps: triplet networks had not been applied to this task, and existing deep ConvNet methods did not use a whole-training-set global loss.
- Metric Learning: Metric learning methods seek a Mahalanobis distance or linear feature transform for embedding labeled points.The transform is obtained by factorising W as GG⊤ and solving a generalized eigenvalue problem.
- Metric Learning: Metric learning extensions address multimodal distributions, nearest-neighbor classification, and nonlinear transformations.
- Deep Descriptor Learning: Deep siamese ConvNets learn nonlinear descriptors by reducing distances for same-class patches and increasing distances for different-class patches.
- Deep Descriptor Learning: Earlier metric-learning methods performed significantly worse on the UBC dataset than recently proposed siamese deep ConvNets.
- Research Gap: Triplet networks had improved siamese results in other settings but had not yet been applied to local image descriptor learning.
- Research Gap: Existing deep ConvNet methods did not use the whole training set in a global loss, which the paper states can improve model generalisation.
3. Methodology
The methodology introduces triplet and siamese architectures trained with global losses that optimise distributions of matching and non-matching descriptor relationships. It also defines four models combining triplet, siamese, central-surround, and global-loss designs.
- Feature embedding: The method assumes labelled image patches and uses a shared ConvNet feature embedding f(x, θf) for descriptor learning.The embedding maps patch inputs and network parameters into an m-dimensional representation.
- Siamese networks: Siamese networks use tied two-tower ConvNets trained on labelled pairs, either minimising embedding distances or estimating pairwise similarities.Matching patches are encouraged toward smaller distances or larger similarity scores, while non-matching patches receive the opposite treatment.
- Triplet networks: Triplet networks extend siamese networks by jointly processing matching and non-matching patches with a margin-based triplet loss.The three towers are constrained to share the same network, while the triplet provides relative context for optimisation.
- Global loss: The global embedding loss reduces matching and non-matching distance variances, decreases the matching mean distance, and increases the non-matching mean distance.Its objective is to reduce the false-positive and false-negative region by separating the two distance distributions.
- Global loss: Unlike triplet-loss gradients that depend only on one triplet, global-loss gradients depend on whole-training-set statistics and may suppress outlier-driven gradients.The global loss uses matching and non-matching distribution means, providing a potential generalisation benefit.
- Proposed models: The proposed models are TNet, TLoss; TNet, TGLoss; SNet, GLoss; and CS SNet, GLoss.They respectively cover triplet loss, combined triplet and global losses, siamese global similarity loss, and central-surround siamese global similarity loss.
4. Toy Problem
A toy two-class dataset with 5% mislabeled outliers compares triplet, combined triplet-global, and global losses for learning embeddings. The global loss yields coherent nearest-neighbour labels around outliers, while triplet loss shows more overfitting.
- The toy dataset contains 80 samples from two Gaussian classes, with four labels switched to create 5% training outliers.
- Three identically configured ConvNets are trained using triplet, combined triplet-global, or global loss functions.
- Nearest-neighbour labels in embedding space are evaluated by mapping a grid of input points to the labels of their closest training samples.
- The triplet-loss embedding is more affected by outliers and appears to overfit the training data.
- The combined triplet-global loss reduces labelling mistakes, while global loss produces a coherent label map near outliers and is described as robust to them.
5. Experiments
Experiments use the UBC benchmark’s three image-patch sets and six train-test combinations to compare proposed models with state-of-the-art descriptors. Results include ROC/FPR95 evaluation and identify strong performance from global-loss variants, especially TNet-TGLoss for embeddings and CS SNet with global loss for pairwise similarity.
- Dataset: The UBC benchmark contains Yosemite, Notre Dame, and Liberty, with six combinations of one-set training and another-set testing.Each set has more than 450,000 normalized 64 × 64 patches detected using Difference of Gaussians.
- Training setup: Training uses SGD, augmentation by three rotations and two flips, and randomly sampled triplets for the siamese and triplet networks.
- Evaluation: The proposed models are evaluated with FPR95 across all six train-test combinations, using separate tables for pairwise similarity and embedding methods.ROC curves are also reported for the proposed and state-of-the-art models.
- Results: The TNet-TGLoss triplet network achieves the best result in the field for feature embedding.
- Results: The centre-surround siamese network trained with global loss produces a pairwise-similarity result almost half that of the previous state-of-the-art 2ch-2stream.
- Results: Siamese pairwise methods generally classify better than triplet embeddings, but TNet-TGLoss outperforms the Siam network and 512d-F(512) pairwise models.Pairwise methods require pairwise inputs during inference, whereas descriptor-distance computation can be simplified using cosine distance when norms equal 1.
6. Conclusions
The paper introduces triplet and siamese patch-matching models trained with a combined triplet and global loss. The global loss is argued to improve regularisation, while a triplet-plus-global extension did not improve preliminary results beyond Table 2.
- The proposed methods combine triplet or siamese networks with a global loss applied to mini-batches for patch matching.The authors describe this as the first application of both the global loss and triplet network to patch matching.
- The authors argue that the models’ superior results arise from better regularisation provided by the global loss.
- The models achieve state-of-the-art results on the UBC benchmark dataset.
- Preliminary, unshown results indicate that combining the global loss with the triplet network does not outperform the results in Table 2.