PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention-based Ensemble for Deep Metric Learning

Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, Keunjoo Kwon

arXiv:1804.00382v2cs.CV

TL;DR

Deep metric learning must organize similar and dissimilar images in an embedding space, while ensemble performance depends on diverse learners. The paper uses multiple attention modules with a shared embedding function and a divergence loss to diversify learners. It reports significant improvements over state-of-the-art methods on image-retrieval benchmarks.

  • Problem

    Deep metric learning ensembles need diverse feature embeddings, but optimal architectures for producing that diversity have received limited research.

  • Method

    The method uses multiple attention masks with a shared global embedding function and a divergence loss that encourages learners to attend to different image parts.

  • Results

    The method outperforms current state-of-the-art techniques by a significant margin on several image-retrieval benchmarks.

  • Takeaways & Limitations

    Attention-based diversity improves ensemble performance and also improves each individual learner compared with the baseline.

  • Takeaways & Limitations

    For CARS-196 and CUB-200-2011, evaluation reports both original and cropped images because the datasets provide bounding boxes.

Abstract

from arXiv · show

Deep metric learning aims to learn an embedding function, modeled as deep neural network. This embedding function usually puts semantically similar images close while dissimilar images far from each other in the learned embedding space. Recently, ensemble has been applied to deep metric learning to yield state-of-the-art results. As one important aspect of ensemble, the learners should be diverse in their feature embeddings. To this end, we propose an attention-based ensemble, which uses multiple attention masks, so that each learner can attend to different parts of the object. We also propose a divergence loss, which encourages diversity among the learners. The proposed method is applied to the standard benchmarks of deep metric learning and experimental results show that it outperforms the state-of-the-art methods by a significant margin on image retrieval tasks.

1 Introduction

Deep metric learning maps images into an embedding space where similar images are close and dissimilar images are far apart. The paper motivates attention-based ensemble design because ensemble quality depends on both learner performance and embedding diversity.

  • Deep metric learning learns a neural embedding function that places semantically similar images close and dissimilar images far apart.
  • Ensemble combines multiple learners and often concatenates their embeddings to obtain a better embedding space under distance constraints.
  • Successful ensembles require both strong individual learners and diversity among their feature embeddings.
  • The proposed attention-based ensemble trains one shared embedding function with separate attention modules, encouraging learners to attend to different object locations.
  • Existing deep metric learning ensembles use approaches including cascaded models, online gradient boosting, parameter sharing, and differing initializations or training subsets.
  • Attention mechanisms have been used for object-part detection, convolutional attention, diversified visual attention, and differentiable spatial transformations.

3 Attention-based ensemble

The section formalizes deep metric learning and ensemble metrics, then introduces attention-based learners that share feature extraction while using distinct attention modules and a divergence loss to promote embedding diversity.

  • 3.1 Deep metric learning: Deep metric learning seeks an embedding that preserves similarity by placing semantically similar samples close and dissimilar samples far apart.
  • 3.2 Ensemble for deep metric learning: A classical ensemble averages distances from independently trained learner embeddings, while alternative constructions share either the spatial extractor or global embedding function.
  • 3.2 Ensemble for deep metric learning: Attention-based ensemble learners map one input to distinctive intermediate and final embedding locations while sharing a single global embedding function.
  • 3.3 Attention-based ensemble model: The proposed architecture splits feature extraction into a shared spatial extractor and global embedding function, with each learner owning an attention module that produces a same-sized mask.
  • 3.3 Attention-based ensemble model: Each mask is multiplied element-wise with shared features before global embedding, and the resulting learner embeddings are trained with metric loss plus a weighted divergence regularizer.
  • 3.4 Loss: Divergence loss increases distances between embeddings of the same image from different learners, encouraging different attended regions without directly constraining masks.

4 Implementation

The implementation uses GoogLeNet with shared spatial and intermediate features, lightweight attention modules, contrastive loss, and standard optimization and preprocessing choices.

  • Architecture: GoogLeNet provides the spatial feature extractor and global feature embedding function for ABE-M.The spatial extractor uses the max-pooling output after inception(3b), while the remaining network forms the global embedding function.
  • Architecture: The intermediate inception(4a)–inception(4e) component is shared across all learners to reduce computation, memory use, and parameters.The authors report no performance drop in preliminary experiments with this shared implementation.
  • Training: The model uses contrastive loss as its distance metric loss, with both contrastive and divergence margins set to 1.The hinge function and Euclidean distance define the contrastive-loss formulation.
  • Training: Training uses Caffe, ImageNet initialization, stochastic gradient descent with momentum 0.9, and a validation-tuned base learning rate.The final network layer and attention convolution layer are randomly initialized.
  • Preprocessing: Images are resized with aspect ratio preserved, padded to 256×256, randomly cropped and flipped during training, and center-cropped during testing.ImageNet channel means are subtracted from the inputs.
  • Training: ABE-8 experiments use batch size 64, while the default ensemble embedding size is 512 and each learner uses 512/M dimensions.The batch-size constraint is associated with the available Tesla M40 GPU memory.

5 Evaluation

Evaluation measures image-retrieval performance with Recall@K across standard datasets using established train-test splits and both original and cropped images where applicable.

  • Evaluation protocol: Recall@K equals the fraction of test queries whose top K retrieved images include at least one image with the same label.The score is averaged over the complete test set after excluding the query image itself.
  • Datasets: Experiments cover CARS-196, CUB-200-2011, Stanford Online Products, and In-shop Clothes Retrieval.These datasets span cars, birds, products, and clothing retrieval tasks.
  • Evaluation protocol: The experiments follow the train-test split used by earlier work for fair comparison with other methods.The same evaluation protocol is applied across the commonly used image-retrieval datasets.
  • Evaluation protocol: CARS-196 and CUB-200-2011 are evaluated using both original images and cropped images because both datasets provide bounding boxes.This reports performance under both image presentations.

6 Experiments

On CARS-196, ABE-M is compared with M-heads across ensemble embedding sizes, showing stronger retrieval performance alongside lower parameter counts and a scaling trend with more learners.

  • Comparison with M-heads: ABE-M outperforms M-heads by a significant margin on CARS-196 across varying ensemble embedding sizes.The comparison is presented in Table 1 and Fig. 4.
  • Efficiency: ABE-M uses fewer model parameters than M-heads because its global feature extractor is shared among learners.The additional attention modules make ABE-M require higher flops, although the difference becomes increasingly insignificant as M increases.
  • Scaling with ensemble size: ABE-M 512 ensemble performance increases as the number of learners M increases, even though each learner’s embedding size decreases to 512/M.Individual-learner performance also increases with M, unlike the corresponding M-heads case.
  • Comparison with M-heads: Table 1 reports Recall@K percentages on CARS-196, with superscripts denoting ensemble embedding size.The table compares retrieval performance across ensemble configurations and baselines.

ABE-M without divergence loss

Experiments isolate divergence loss and visualize attention masks, showing that the loss supports diverse learner embeddings and distinct object-part attention in ABE-M.

  • Diagnostic comparison: The Fig. 5 panels compare positive, negative, and self-pair cosine-similarity histograms for ABE-8 and 8-heads ensembles with or without divergence loss.Self pairs compare embeddings from different learners applied to the same image.
  • Effect of divergence loss: Without divergence loss, ABE-M performs similarly to its individual learners, whereas the full model gains substantially over them in ensemble performance.The comparison is reported on CARS-196.
  • Effect of divergence loss: With divergence loss, ABE-M learners have diverse embeddings and lower cosine similarity for self pairs from the same image.Without the loss, self-pair cosine similarity approaches 1 because learners converge toward similar embeddings.
  • Attention visualization: ABE-8 learners attend to different car regions, including the upper body, bottom, roof, tires, and lights.The visualization shows only three of eight learners and three of 480 channels because of space limitations.
  • Attention visualization: The mean attention mask focuses more on object areas than on the background.This observation is reported from the attention-mask visualization on CARS-196.

Divergence loss in M-heads

The 8-heads ensemble does not benefit from divergence loss, while ablations identify attention-module depth and branching point as important architectural choices.

  • Divergence loss: Divergence loss does not improve performance in the 8-heads ensemble.The authors hypothesize that each learner can arbitrarily compose a different metric space, making self-pair cosine similarities close to zero already.
  • Attention-module depth: Five inception blocks, spanning inception(4a) through inception(4e), give the best Recall@1.The study varies attention-module depth from one to seven inception blocks in GoogLeNet.
  • Branching point: Branching at pool3 performs best when the attention module contains five inception blocks.The branching point separates the spatial feature extractor from the global feature embedding function.
  • Branching point: Earlier branching points increase memory requirements because they deepen the global embedding function replicated across learners.The global function requires M times the memory of an individual learner in the ABE-M model.

Sensitivity to λdiv

The sensitivity study finds that λdiv strongly affects performance, with λdiv = 1 performing best; the method is evaluated against state-of-the-art retrieval systems across several benchmarks.

  • Sensitivity to λdiv: λdiv = 1 performs best for ABE-M, while lower values degrade performance quickly.The effect is measured using Recall@K.
  • Benchmark comparison: The approach performs best on all major image-retrieval benchmarks in comparisons with current state-of-the-art techniques.The comparisons cover CARS-196, CUB-200-2011, SOP, and in-shop clothes retrieval, including cropped datasets.
  • Benchmark comparison: The reported comparisons use Recall@K(%) on CUB-200-2011 and CARS-196.These results are listed in Table 4.
  • Benchmark comparison: The cropped CUB-200-2011 and CARS-196 results are reported using Recall@K(%).These results are listed in Table 5.
  • Benchmark comparison: The SOP results are reported as Recall@K(%).These results are listed in Table 6.
  • Benchmark comparison: The in-shop clothes retrieval results are reported as Recall@K(%).These results are listed in Table 7.

7 Conclusion

The paper introduces an attention-based ensemble for deep metric learning that uses diverse attention masks and divergence loss, and reports significant state-of-the-art gains across image-retrieval benchmarks.

  • Conclusion: The framework uses multiple attention-based learners so different learners attend to different image parts.The ensemble is designed around diverse feature embeddings.
  • Conclusion: Divergence loss diversifies learners’ feature embeddings by encouraging them to attend to different image parts.The conclusion reports that it improves ensemble performance and individual learner performance over the baseline.
  • Conclusion: The method outperforms current state-of-the-art techniques by a significant margin on several image-retrieval benchmarks.The listed benchmarks include CARS-196, CUB-200-2011, SOP, and in-shop clothes retrieval.

Attention-based Ensemble for Deep Metric Learning (Supplementary Material)

The supplementary material defines the ABE-M embedding construction and contrasts attention-based multi-head and multi-tail ensemble variants.

  • Supplementary experiments: Supplementary ablations compare M-heads+att and M-tails on CARS-196.Unless otherwise specified, models use ensemble embedding size 512 and batch size 64.
  • ABE-M construction: ABE-M combines a shared spatial feature extractor, learner-specific attention modules, and a global feature embedding function.The attention mask has the same size as the spatial extractor output, and element-wise multiplication combines them.
  • Alternative architectures: M-heads+att uses attention with separate global embedding functions and no divergence loss.It is an ensemble of Gm(S(x)◦Am(S(x))).
  • Alternative architectures: M-tails uses multiple tails with a shared head and omits attention modules.Its form is G(Sm(x)).
  • Supplementary results: Table 1 reports Recall@K(%) on CARS-196 with an ensemble embedding size of 512.The supplementary material also compares 8-heads and ABE-8 in Table 2.

Effect of attention module on M-heads

Attention improves the M-heads ensemble, while ABE-8 gains further from sharing the global embedding function and applying divergence loss. An alternative 8-tails architecture achieves a larger gain over 8-heads than 8-heads+att.

  • Attention module: 79.7% Recall@1 with 8-heads+att improves on 76.1% from 8-heads, isolating the attention module’s effect.The comparison controls for other factors in the ensemble design.
  • Attention module: 85.2% Recall@1 from ABE-8 further improves on 79.7% by sharing G(·) and applying divergence loss alongside attention.ABE-8 combines attention with a shared global embedding function and divergence-based regularization.
  • Ensemble architecture: 8-tails improves Recall@1 over 8-heads, with a +5.0% gain compared with +3.6% for 8-heads+att.ABE-8 further improves performance over 8-tails by using an attention module.
Loading 1804.00382v2…