Source-linked AI summary
Hard-Aware Deeply Cascaded Embedding
Yuhui Yuan, Kuiyuan Yang, Chao Zhang
TL;DR
Deep metric learning faces impractical pair and triplet sample growth, while model-relative hard-example mining can underfit with simple models or overfit with complex ones. HDC addresses this by cascading models of increasing complexity, adaptively forwarding hard samples and updating the models that find them hard. Across five benchmarks, the authors report state-of-the-art results and controlled experiments supporting the hard-aware design.
Problem
Pair and triplet training produces O(N^2) or O(N^3) samples, while hard-example mining depends on model complexity and lacks a generally adequate model choice.
Method
HDC ensembles models with increasing complexities in a cascade, forwarding samples through progressively deeper models and back-propagating errors to models that treat them as hard.
Results
HDC achieves state-of-the-art results on five benchmarks, with extensive comparisons verifying its effectiveness for deep metric embedding.
Takeaways & Limitations
The results support adaptively matching samples with different hard levels to models of adequate complexity within an ensembled cascade.
Takeaways & Limitations
The method is verified with three cascaded models, while cascading more models and increasing complexities more smoothly remain future work.
Abstract
from arXiv · showhide
Riding on the waves of deep neural networks, deep metric learning has also achieved promising results in various tasks using triplet network or Siamese network. Though the basic goal of making images from the same category closer than the ones from different categories is intuitive, it is hard to directly optimize due to the quadratic or cubic sample size. To solve the problem, hard example mining which only focuses on a subset of samples that are considered hard is widely used. However, hard is defined relative to a model, where complex models treat most samples as easy ones and vice versa for simple models, and both are not good for training. Samples are also with different hard levels, it is hard to define a model with the just right complexity and choose hard examples adequately. This motivates us to ensemble a set of models with different complexities in cascaded manner and mine hard examples adaptively, a sample is judged by a series of models with increasing complexities and only updates models that consider the sample as a hard case. We evaluate our method on CARS196, CUB-200-2011, Stanford Online Products, VehicleID and DeepFashion datasets. Our method outperforms state-of-the-art methods by a large margin.
1. Introduction
Deep metric embedding learns spaces that bring similar images together and separate dissimilar ones, but its pairwise or tripletwise sample growth makes exhaustive optimization impractical. The paper addresses model-dependent hard-example mining by cascading models of increasing complexity and adapting which samples update which models.
- Deep metric embedding learns an embedding space that pulls similar images closer and pushes dissimilar images farther apart.
- N images can generate O(N^2) pair samples or O(N^3) triplet samples, making exhaustive consideration impossible even for moderate datasets.
- Hard-example mining is model-relative: complex models treat most samples as easy and risk overfitting, whereas simple models treat most as hard.
- The proposed method cascades models with increasing depth, forwarding each sample until a model finds it easy or the deepest model is reached.
- Errors are back-propagated only to models that consider the sample hard, while the framework ensembles models with different complexities.
2. Related Work
Related work broadly mines hard examples, designs metric-learning losses, or uses cascades and auxiliary supervision for other goals. HDC differs by assigning samples at different hard levels to models with adequate complexities while ensembling the models as one network.
- Hard-example mining: Prior hard-example methods mine hard images or triplets using a single model, limiting their ability to exploit different sample hard levels.
- Cascaded models: Cascaded computation blocks in detection reject easy background patches early to reduce computational cost, providing a related design precedent.
- Auxiliary supervision: Deeply-supervised networks add losses to output and intermediate layers to improve hidden-layer learning and alleviate gradient vanishing.
- HDC distinction: HDC assigns samples of different hard levels to models with adequate complexities and ensembles the models together as one model.
- Metric-learning losses: Other metric-learning work introduces losses for sub-clustering, position-dependent distances, or matching positive and negative similarity distributions.
3. Hard-Aware Deeply Cascaded Embedding
Hard-Aware Deeply Cascaded Embedding (HDC) handles samples with different difficulty levels using cascaded sub-networks of increasing depth and adaptive hard-example mining.
- Framework: HDC ensembles K sub-networks of different depths, with the first model using the initial computation block G1.Models are built from cascaded computation blocks and transform outputs into low-dimensional feature vectors for distance calculation.
- Framework: Pairs judged easy by the current model stop processing, while hard pairs continue through increasingly complex models until accepted or reaching model K.Each model receives pairs retained as hard by previous models, and the hard sets are denoted Pk and Nk.
- Optimization: The final HDC objective combines the losses of all cascade models, with λk weighting model k.The framework optimizes the model parameters using SGD and distributes gradients across the computation blocks and transform functions.
- Motivation: Unlike single-model mining, HDC accommodates diverse sample difficulty by cascading models with increasing complexities.The authors motivate this design because a single model is unlikely to have the right complexity for all hard levels.
- Contrastive loss: For contrastive loss, positive pairs are penalized by their distance, while negative pairs are penalized when their distance falls below margin M.Distances are computed between L2-normalized feature vectors using Euclidean distance.
- Hard-example mining: Hard examples are selected by ranking mini-batch losses and taking the top hk percent of positive and negative pairs for each model.This avoids predefined thresholds because loss distributions differ across models and change during training.
- Efficiency and implementation: Cascaded ensembling shares forward-pass computations, providing computational efficiency during both training and testing.The implementation uses mini-batch SGD and multi-batch processing to use possible pairs in a mini-batch for gradient estimation.
4. Experimental Evaluation
The experiments evaluate HDC for image retrieval across five datasets using standard train/test protocols and Recall@K or MAP. HDC improves over baselines and prior state-of-the-art methods across diverse settings, including large-category datasets and tests without bounding boxes.
- Evaluation setup: The evaluation uses MAP and Recall@K on five standard deep metric embedding datasets under their standard train/test splits.Recall@K counts a query as successful when at least one positive image appears among the top K retrieved images.
- Baseline comparison: The distance-distribution overlap decreases from GoogLeNet/pool51024 through HDC + Contrastive†384, while the corresponding LDA score increases.The compared methods are shown on the CARS196 test set.
- Baseline comparison: The baseline study compares pretrained GoogLeNet features, contrastive loss, hard-example mining, and HDC with cascaded submodels.HDC uses three 128-dimensional cascade embeddings, producing a 384-dimensional ensembled representation.
- Baseline comparison: HDC achieves the best CARS196 baseline result, while HDC + Contrastive-3†128 outperforms traditional Hard + Contrastive†128.The authors attribute this comparison to shallow modules also being trained by hard samples mined by shallow models.
- State-of-the-art comparison: HDC outperforms methods without bounding boxes on CARS196 and CUB-200-2011, and exceeds bounding-box methods on CARS196.The method also achieves state-of-the-art performance against Npairs⊞ with multiple-crop testing.
- State-of-the-art comparison: On Stanford Online Products, HDC improves Recall@1 by 4% absolute over previous state-of-the-art methods.The dataset has many categories and few images per category, unlike CARS196 and CUB-200-2011.
- State-of-the-art comparison: Multiple-crop testing raises Recall@1 from 67.7% to 70.1% on Stanford Online Products.The paper also reports large-margin improvements on DeepFashion In-shop Clothes and VehicleID, which similarly have few images per class and many classes.
5. Conclusions
The paper concludes that Hard-Aware Deeply Cascaded Embedding addresses hard-sample mining by combining models of increasing complexity, with effectiveness demonstrated across five benchmarks and a stated scope for future refinement.
- Hard-Aware Deeply Cascaded Embedding considers both sample hard levels and model complexities through a cascaded ensemble.Models with increasing complexities are used to mine samples adaptively.
- Extensive comparisons on five benchmarks verify the method’s effectiveness in learning deep metric embeddings.The authors report state-of-the-art results across five datasets and describe the method as having good generalization capability.
- The method is currently verified with three cascaded models, while future work will add more models and smoother complexity increases.The authors also plan to combine the method with other loss functions.