Source-linked AI summary
SoftTriple Loss: Deep Metric Learning Without Triplet Sampling
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, Rong Jin
TL;DR
Deep metric learning is hindered by the vast number of triplet constraints and limited mini-batch neighborhoods. The paper analyzes SoftMax and introduces SoftTriple with multiple centers per class, achieving consistent improvements on fine-grained benchmarks while eliminating triplet sampling. The method has a stated limitation concerning bounded-loss analysis.
Problem
Mini-batch sampling can miss the original data neighborhoods, making deep metric-learning optimization complicated and potentially sub-optimal.
Method
SoftTriple extends SoftMax's smoothed triplet formulation with multiple centers per class, encoded in the last fully connected layer to avoid triplet sampling.
Results
SoftTriple consistently improves performance over fine-grained benchmark data sets, including more than 3% R@1 improvement over the state of the art on Stanford Online Products.
Takeaways & Limitations
SoftTriple provides a classification-style SGD pipeline for learning embeddings while modeling intra-class variance without sampling triplet constraints.
Takeaways & Limitations
The analysis of the bounded loss is outside the scope of this work.
Abstract
from arXiv · showhide
Distance metric learning (DML) is to learn the embeddings where examples from the same class are closer than examples from different classes. It can be cast as an optimization problem with triplet constraints. Due to the vast number of triplet constraints, a sampling strategy is essential for DML. With the tremendous success of deep learning in classifications, it has been applied for DML. When learning embeddings with deep neural networks (DNNs), only a mini-batch of data is available at each iteration. The set of triplet constraints has to be sampled within the mini-batch. Since a mini-batch cannot capture the neighbors in the original set well, it makes the learned embeddings sub-optimal. On the contrary, optimizing SoftMax loss, which is a classification loss, with DNN shows a superior performance in certain DML tasks. It inspires us to investigate the formulation of SoftMax. Our analysis shows that SoftMax loss is equivalent to a smoothed triplet loss where each class has a single center. In real-world data, one class can contain several local clusters rather than a single one, e.g., birds of different poses. Therefore, we propose the SoftTriple loss to extend the SoftMax loss with multiple centers for each class. Compared with conventional deep metric learning algorithms, optimizing SoftTriple loss can learn the embeddings without the sampling phase by mildly increasing the size of the last fully connected layer. Experiments on the benchmark fine-grained data sets demonstrate the effectiveness of the proposed loss function. Code is available at https://github.com/idstcv/SoftTriple
1. Introduction
Deep metric learning avoids hand-crafted-feature limitations by learning embeddings with neural networks, but mini-batch constraint sampling can miss global neighborhoods. The paper analyzes SoftMax and proposes SoftTriple, which uses multiple class centers to model intra-class variance without triplet sampling.
- Hand-crafted features can lose task-relevant information, while deep metric learning learns low-dimensional embeddings directly from raw materials.
- Mini-batches poorly capture neighborhoods in large data sets, making sampling informative pairs or triplets essential but optimization complicated.A mini-batch of size m contains O(m^2) pairs and O(m^3) triplets.
- SoftMax loss is equivalent to a smoothed triplet loss using one center per class, pairing an example with its class center and a different-class center.
- SoftTriple extends SoftMax with multiple centers per class, reducing intra-class variance and better representing local clusters.The proposed design assigns examples to multiple centers encoded in the last fully connected layer.
- SoftTriple can learn embeddings without sampling triplets by encoding centers in the last fully connected layer, with the number of triplets linear in original examples.The paper also initializes many centers and uses an L2,1 norm to obtain a compact set.
2. Related Work
Prior work addresses metric-learning dimensionality, constraint complexity, and mini-batch limitations through projections, specialized constraints, sampling, and proxies. These approaches motivate learning embeddings while reducing dependence on exhaustive triplet sampling.
- Conventional metric learning often uses hand-crafted features and faces computational costs from maintaining a positive semi-definite metric matrix.The related work discusses PCA, low-rank assumptions, fewer PSD projections, and random projections as cost-reduction strategies.
- Pairwise constraints require same-class distances to be small and different-class distances to be large, whereas triplet constraints impose a margin between similar and dissimilar examples.
- Triplet constraints can reach O(n^3), but they model local-cluster geometry and intra-class variance more directly than pairwise constraints.
- Deep metric learning learns task-dependent embeddings from raw materials, but small SGD mini-batches provide limited neighborhood information.
- Proxy-based methods reduce triplet counts and can avoid cross-batch sampling, but using one proxy per class resembles SoftMax and may not represent multiple local clusters.
3. SoftTriple Loss
SoftTriple extends SoftMax-based metric learning from one center per class to multiple centers, using smoothed class similarities to better represent intra-class structure. The resulting loss preserves original triplet constraints while avoiding triplet sampling, although the number of centers trades accuracy against efficiency and may cause overfitting.
- Sampling and complexity: SoftTriple avoids triplet sampling because centers are encoded in the final fully connected layer, reducing the constraint count to linear in the number of original examples.This contrasts with conventional triplet-based methods, whose total triplets can be cubic in the number of examples.
- SoftMax and triplet constraints: SoftMax loss is equivalent to a smoothed triplet loss involving an example, its class center, and a center from another class.The entropy regularizer smooths the triplet hardness and reduces outlier influence.
- Motivation: Conventional SoftMax assumes one center per class, which cannot represent classes containing multiple local clusters and large intra-class variance.The paper motivates multiple centers as a way to capture more complex class geometry.
- SoftTriple formulation: SoftTriple assigns each class K centers and computes a relaxed, entropy-smoothed similarity before applying the loss.The multiple-center similarity is introduced first, then smoothed through an entropy-regularized formulation.
- Recovering original triplets: The center-based constraints can preserve the large-margin property of original triplets when same-class examples share their nearest center.The approximation error is bounded by the distances from examples to their corresponding centers; increasing centers can reduce this error.
- Trade-offs: More centers reduce intra-class approximation error but enlarge the final fully connected layer, slow optimization, increase computation, and may cause overfitting.The paper therefore recommends an appropriate compact center set rather than maximizing the number of centers.
- Adaptive Number of Centers: Choosing K is difficult without prior knowledge of each class distribution, so the method starts with a sufficiently large K and encourages similar centers to merge.The L2,1 regularizer shrinks the number of unique centers while retaining center diversity.
4. Experiments
Experiments on CUB-2011, Cars196, and Stanford Online Products evaluate SoftTriple for retrieval and clustering, comparing center counts, regularization, and embedding dimensions. SoftTriple consistently outperforms the compared methods while learning embeddings without triplet sampling.
- Experimental setup: Experiments use CUB-2011, Cars196, and SOP with retrieval measured by Recall@k and clustering by NMI.The backbone is Inception with batch normalization, and the reported comparisons use 64- or 512-dimensional embeddings.
- CUB-2011: SoftTriple achieves the best 64-dimensional performance on CUB-2011, improving R@1 by 10% over ProxyNCA and 2% over SoftMaxnorm.The results support using multiple centers to represent complex class geometry.
- CUB-2011: With 512-dimensional embeddings on CUB-2011, SoftTriple improves R@1 by more than 8% over HTL and about 2% over Margin.HTL uses the same Inception backbone, whereas Margin uses a stronger backbone than Inception.
- Center analysis: The regularizer shrinks redundant centers and keeps performance stable as the initial center count grows, whereas removing it causes overfitting when centers are over-parameterized.Performance improves significantly from 1 to 10 centers, then almost remains unchanged with the regularizer.
- Retrieval examples: SoftTriple improves retrieval examples by eliminating different-class images from the top results that appear with SoftMaxnorm.Different-class images are identified by red bounding boxes in the retrieval visualization.
- Cars196 and SOP: SoftTriple improves R@1 by about 2% over SoftMaxnorm on Cars196 and by 0.4% over the one-center setting on SOP.On large embeddings, it improves R@1 by 3% over HTL on Cars196 and by more than 3% over the state of the art on SOP.
5. Conclusion
SoftTriple addresses poor mini-batch triplet coverage by learning embeddings with multiple class centers and no sampling phase. Experiments on fine-grained benchmarks show consistent improvement over SoftTriple’s comparison methods.
- SoftTriple learns embeddings without sampling triplets by representing each class with multiple centers.The centers are encoded in the last fully connected layer, allowing standard SGD classification training.
- Multiple centers capture hidden data distributions better by reducing intra-class variance.
- The loss uses triplets defined through similarities between original examples and classes.
- SoftTriple consistently improves performance across fine-grained benchmark data sets.
- SoftTriple may also apply to classification because SoftMax loss is widely used for that task.The paper identifies classification evaluation as future work.