PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Metric Learning with BIER: Boosting Independent Embeddings Robustly

Michael Opitz, Georg Waltner, Horst Possegger, Horst Bischof

arXiv:1801.04815v1cs.CV

TL;DR

Large embeddings for image retrieval can become highly correlated and suffer accuracy saturation or decline from over-fitting. BIER partitions the embedding into an ensemble trained with online gradient boosting and diversity losses, and reports state-of-the-art retrieval performance across five datasets.

  • Problem

    Large dot-product embeddings can over-fit, causing retrieval accuracy to saturate or decline.

  • Method

    BIER partitions a CNN embedding into shared-feature learners, trains them with online gradient boosting, and uses Activation or Adversarial Loss to increase diversity.

  • Results

    BIER achieves state-of-the-art performance on CUB-200-2011, Cars-196, Stanford Online Products, In-Shop Clothes Retrieval, and VehicleID.

  • Takeaways & Limitations

    Reducing correlation among embedding learners increases ensemble retrieval accuracy, with Adversarial Loss outperforming Activation Loss.

Abstract

from arXiv · show

Learning similarity functions between image pairs with deep neural networks yields highly correlated activations of embeddings. In this work, we show how to improve the robustness of such embeddings by exploiting the independence within ensembles. To this end, we divide the last embedding layer of a deep network into an embedding ensemble and formulate training this ensemble as an online gradient boosting problem. Each learner receives a reweighted training sample from the previous learners. Further, we propose two loss functions which increase the diversity in our ensemble. These loss functions can be applied either for weight initialization or during training. Together, our contributions leverage large embedding sizes more effectively by significantly reducing correlation of the embedding and consequently increase retrieval accuracy of the embedding. Our method works with any differentiable loss function and does not introduce any additional parameters during test time. We evaluate our metric learning method on image retrieval tasks and show that it improves over state-of-the-art methods on the CUB 200-2011, Cars-196, Stanford Online Products, In-Shop Clothes Retrieval and VehicleID datasets.

1 INTRODUCTION

BIER addresses the saturation and over-fitting of large dot-product embeddings by dividing the embedding into multiple learners and training them for low correlation. It combines online gradient boosting with diversity losses and reports improved retrieval performance across several benchmarks.

  • Dot-product metric-learning accuracy can saturate or decline with large embeddings because of over-fitting.
  • BIER divides the final CNN embedding into multiple non-overlapping groups, each acting as a separate metric-learning network over shared features.
  • Online gradient boosting reweights training samples for successive learners using the loss gradient, avoiding highly correlated learners from naïve joint optimization.
  • The method adds weight-initialization and auxiliary-training losses to reduce learner correlation and improve training efficiency and accuracy.The auxiliary loss permits higher learning rates and further reduces correlation between learners.
  • BIER achieves state-of-the-art performance across CUB-200-2011, Cars-196, Stanford Online Products, In-Shop Clothes Retrieval, and VehicleID.It adds no CNN parameters and only negligible additional training and runtime cost.

2 RELATED WORK

The paper situates BIER within metric learning, boosting, and CNN-based embedding methods. It distinguishes its end-to-end online boosting approach from prior offline, multi-layer, and full-model ensemble methods while complementing existing losses and mining strategies.

  • 2.1 Metric Learning: Metric learning maps images into spaces where similar images are close and dissimilar images are far apart.
  • 2.2 Boosting for CNNs: Unlike prior offline boosting methods, BIER performs online boosting directly within end-to-end CNN training using fully connected learners over shared features.
  • 2.1.2 CNN Based Metric Learning: CNN metric-learning systems typically combine a nonlinear feature extractor with a linear embedding layer trained using pairwise, triplet, or quadruple losses.
  • 2.1.2 CNN Based Metric Learning: BIER complements existing metric-learning losses and improves performance when combined with them.
  • 2.2 Boosting for CNNs: Successive BIER learners reweight samples according to loss gradients, emphasizing harder examples, while auxiliary losses explicitly encourage embedding diversity.
  • 2.2 Boosting for CNNs: Unlike approaches training multiple full CNNs or using multiple network layers, BIER trains one shared CNN and reduces correlation among learners in a single embedding layer.

3 BOOSTING A METRIC NETWORK

BIER turns a metric CNN’s embedding into shared-feature weak learners trained with online gradient boosting, then adds diversity losses to reduce learner correlation. The resulting ensemble supports efficient similarity learning while improving diversity and reported accuracy.

  • Metric learning setup: Metric CNNs learn high-dimensional embeddings and cosine similarity from image pairs, triplets, or other structured samples.The similarity score is a dot product and is bounded between −1 and +1.
  • Metric learning setup: Binomial deviance, contrastive, and triplet losses are considered; binomial deviance benefits more from BIER in evaluation.The paper attributes this difference to binomial deviance’s smooth gradient compared with the 0-or-1 gradients of contrastive and triplet losses.
  • Online gradient boosting CNNs: Each successive learner receives a reweighted batch based on the negative loss gradient at the preceding ensemble stage, emphasizing examples with large gradients.The ensemble uses fixed boosting weights and is trained end-to-end with mini-batch SGD and momentum.
  • Online gradient boosting CNNs: BIER divides the embedding layer into non-overlapping groups, treating each group as a weak learner while sharing a feature representation.This avoids the cost of training multiple separate CNNs; the shared representation is a pre-trained ImageNet CNN in the experiments.
  • Online gradient boosting CNNs: Learner group sizes are increased for later learners because boosting makes them focus on more complex samples.The method uses a convex combination of weak learners and assigns later learners larger embedding capacity.
  • Inference: At test time, BIER L2-normalizes and boosting-weights each learner embedding before concatenation into one feature vector.The resulting distances can be computed efficiently with dot products.
  • Diversity loss functions: Activation and Adversarial Losses further diversify learners and can be used for initialization or as auxiliary training losses.Activation Loss suppresses other embeddings for a sample, whereas Adversarial Loss uses regressors and gradient reversal to minimize inter-embedding similarity.

4 EVALUATION

BIER improves retrieval by combining independently trained embedding groups, using boosting and auxiliary losses to reduce correlation. Across loss functions, embedding sizes, initialization methods, and datasets, these choices consistently improve accuracy.

  • Strength and Correlation: Activation-loss initialization improves over the baseline, while boosting further increases ensemble accuracy by training successive classifiers on reweighted samples.The resulting classifiers focus on different training examples and become less correlated.
  • Loss Functions: Binomial deviance, triplet, and contrastive losses all benefit from BIER, with the strongest performance observed for losses having smooth gradients.Triplet and contrastive gradients for negative samples are either 0 or 1, whereas binomial deviance provides continuous values between 0 and 1.
  • Number of Groups: With a fixed 512-dimensional embedding, three to four learners are optimal because more groups weaken individual learners while fewer groups increase correlation.Smaller numbers of groups produce stronger but more correlated individual embeddings.
  • Embedding Size: Larger embeddings benefit from more groups because they are more prone to overfitting and contain greater redundancy.Splitting them into smaller complementary learners improves the use of large embedding sizes.
  • Initialization: BIER's proposed initialization methods outperform orthogonal and random initialization by starting training with less-correlated learners.This makes it easier for boosting to maintain learner diversity during training.
  • Auxiliary Losses: Auxiliary losses improve BIER over initialization-only training, stabilize optimization at larger learning rates, and reduce training time.Using high learning rates without auxiliary losses makes training unstable and substantially reduces performance.
  • Auxiliary Losses: Adversarial Loss outperforms Activation Loss because it reduces classifier and feature correlation more effectively while preserving comparable individual-learner accuracy.Its ensemble reaches 57.5% versus 56.5% R@1, with classifier correlation 0.6031 versus 0.7310 and feature correlation 0.0731 versus 0.0882.
  • Comparison with the State-of-the-Art: Across five retrieval datasets, BIER consistently improves a strong baseline, while A-BIER outperforms state-of-the-art methods on all datasets.A-BIER improves over BIER by 1.5%, 6.1%, and 3–6% on the three more challenging datasets, respectively.

5 CONCLUSION

BIER trains metric-learning ensembles with online gradient boosting and diversity-promoting losses. The method reduces learner correlation, improves ensemble accuracy, and adds no test-time parameters with negligible computational cost.

  • BIER casts training an ensemble of metric CNNs with a shared feature representation as an online gradient boosting problem.
  • Two loss functions encourage diversity among ensemble learners during initialization or training.
  • The Adversarial Loss increases ensemble diversity more effectively and outperforms the earlier Activation Loss.
  • BIER significantly reduces correlation in the last hidden CNN layer and increases ensemble accuracy across experiments.
  • BIER introduces no additional test-time parameters and only negligible additional computational cost during training and testing.

APPENDIX A OVERVIEW

The appendix outlines BIER's triplet-loss training procedure and presents its implementation as an online gradient boosting algorithm. Triplets contain one positive and one negative pair, and later learners receive reweighted training data.

  • For triplet losses, BIER samples an anchor-positive pair and an anchor-negative pair satisfying their respective similarity labels.
  • The forward pass accumulates positive and negative similarity scores separately before the backward pass reweights the training set.
  • Algorithm 2 presents the online gradient boosting procedure for CNNs using triplet-based loss functions.
  • The appendix also provides the learner coefficient definition, iteration structure, forward-pass sampling, predictions, and backward-pass updates.

APPENDIX C LIST OF GROUP SIZES

Appendix C reports the group dimensionalities used when evaluating the number of groups with embeddings of dimensionality 512 and 1024. Table 14 lists the sizes of all groups.

  • The number-of-groups experiment evaluates embeddings with dimensionality 512 and 1024.
  • Table 14 lists the dimensionality of every group used in the experiments.

APPENDIX D SUMMARY OF IMPROVEMENTS

Appendix D summarizes BIER's contributions on CUB-200-2011. The initialization method, boosting-based training, and auxiliary loss functions each improve the model's final R@1 score.

  • On CUB-200-2011, the initialization method, boosting-based training, and auxiliary loss functions improve the model's final R@1 score.
  • Table 15 summarizes the impact of initialization and boosting on the CUB-200-2011 dataset.

APPENDIX E EVALUATION OF END-TO-END TRAINING

End-to-end training improves BIER’s final retrieval performance over stagewise training. The improvement is attributed to lower-layer weights benefiting from greater ensemble diversity.

  • End-to-end training significantly improves the final R@1 score compared with stagewise training.Stagewise training fixes all hidden layers after applying online boosting to a finetuned network, whereas end-to-end training updates the full network.

APPENDIX F GENERAL APPLICABILITY

BIER is evaluated beyond metric learning on CIFAR-10 classification, where independent classifier groups are trained with auxiliary diversity losses. It improves accuracy over weight decay and dropout baselines on both tested architectures.

  • BIER is tested on CIFAR-10 to assess whether it can improve CNNs beyond metric learning.The experiment targets general applicability rather than state-of-the-art CIFAR-10 accuracy.
  • The CIFAR-10 setup uses CIFAR-10-Quick and a wider architecture, selecting the number of BIER groups on validation performance.The dataset is split into 40,000 training, 10,000 validation, and 10,000 test images.
  • The last fully connected layer is divided into non-overlapping groups, each receiving a classifier and auxiliary Activation or Adversarial loss.The auxiliary losses are applied to the last hidden layer to encourage independence among groups.
  • 3.00% over weight decay and 1.10% over dropout, BIER improves CIFAR-10-Quick performance.On the larger network, BIER improves over the baseline by 2.54% and over dropout by 1.52%.
  • The preliminary CIFAR-10 results indicate that BIER generalizes to tasks beyond metric learning.The authors identify further investigation on other computer vision tasks as future work.
  • Successive learners typically perform better on harder examples than earlier learners with smaller embedding sizes.The paper uses qualitative examples to illustrate differences between learned embeddings.

APPENDIX H QUALITATIVE RESULTS

The appendix presents qualitative examples in Figures 9–13 to illustrate BIER’s effectiveness.

  • Figures 9, 10, 11, 12 and 13 provide qualitative examples illustrating BIER’s effectiveness.

Query Learner-1 Learner-2 Learner-3

Figure 8 compares qualitative retrieval results for the first three learners on CUB-200-2011. Correct and incorrect retrieved images are marked with different colors.

  • Figure 8 retrieves the most similar image for learners 1, 2 and 3 given each CUB-200-2011 query image.
  • Correct retrievals are highlighted green, while incorrect retrievals are highlighted red.

Query 1 2 3 4 5

Figures 9–13 present qualitative retrieval results across five image datasets, showing the five most similar images for each query. Correct and incorrect retrievals are marked in green and red, respectively.

  • CUB-200-2011 results retrieve the five most similar images for each query, with correct results in green and incorrect results in red.
  • Cars-196 results retrieve the five most similar images for each query, with correct results in green and incorrect results in red.
  • Stanford Online Products results retrieve the five most similar images for each query, with correct results in green and incorrect results in red.
  • In-Shop Clothes Retrieval results retrieve the five most similar images for each query, with correct results in green and incorrect results in red.
  • VehicleID results retrieve the five most similar images for each query, with correct results in green and incorrect results in red.
Loading 1801.04815v1…