PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiGrain: a unified image embedding for classes and instances

Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, Matthijs Douze

arXiv:1902.05509v2cs.CV

TL;DR

Class, instance, and copy recognition commonly use specialized, incompatible representations. MultiGrain jointly learns a compact embedding for these tasks with resolution-adaptive pooling; it reports strong classification and retrieval performance, including 79.4% top-1 accuracy at resolution 500.

  • Problem

    Class, instance, and copy recognition commonly rely on specialized, incompatible image representations, while retrieval's need for finer details makes high-resolution inputs impractical for joint end-to-end training.

    Source: §1 · p. 1, §3.5 · p. 5

  • Method

    MultiGrain jointly trains a compact embedding for classification and instance retrieval, using a generalized mean pooling layer that adapts to different test-time resolutions.

    Source: §1 · pp. 1–2

  • Results

    MultiGrain performs well on classification and retrieval and sets a pure-classification state of the art versus the same trunk; it reaches 79.4% top-1 accuracy at resolution 500.

    Source: §5 · p. 8, §4.4 · p. 8

  • Takeaways & Limitations

    The reported results show that classification and retrieval can benefit from each other.

    Source: §5 · p. 8

    At 800 pixels, the p* evaluation-resolution selection has limits: accuracy drops to 77.2% (76.2% without the p* adaptation).

    Source: §4.4 · p. 7

Abstract

from arXiv · show

MultiGrain is a network architecture producing compact vector representations that are suited both for image classification and particular object retrieval. It builds on a standard classification trunk. The top of the network produces an embedding containing coarse and fine-grained information, so that images can be recognized based on the object class, particular object, or if they are distorted copies. Our joint training is simple: we minimize a cross-entropy loss for classification and a ranking loss that determines if two images are identical up to data augmentation, with no need for additional labels. A key component of MultiGrain is a pooling layer that takes advantage of high-resolution images with a network trained at a lower resolution. When fed to a linear classifier, the learned embeddings provide state-of-the-art classification accuracy. For instance, we obtain 79.4% top-1 accuracy with a ResNet-50 learned on Imagenet, which is a +1.8% absolute improvement over the AutoAugment method. When compared with the cosine similarity, the same embeddings perform on par with the state-of-the-art for image retrieval at moderate resolutions.

1. Introduction

Class-, instance-, and copy-level recognition commonly rely on incompatible representations, and using separate embeddings increases resource use. MultiGrain instead learns one compact representation for all three tasks.

1. Introduction

  • MultiGrain aims to unify class, instance, and copy recognition in a compact representation rather than rely on incompatible specialized representations.Using narrowly optimized embeddings for multiple query types multiplies resource use.Source: §1 · p. 1
  • A generalized-mean pooling layer converts spatial maps to fixed-size vectors and supports adapting the network to different test-time resolutions.The authors report higher accuracies and say this avoids the effort of training networks for larger input resolutions.Source: §1 · p. 2
  • Joint training pairs classification cross-entropy with an instance-recognition contrastive loss, using image identity and augmentation rather than retrieval-specific labels.Source: §1 · p. 2
  • MultiGrain offers competing performance in classification and image retrieval, with a notable ResNet-50 ImageNet accuracy boost.The introduction reports the comparison without giving the accuracy value.Source: §1 · p. 2

2. Related work

Prior work uses classification-trained CNNs as retrieval features, while retrieval-specific pooling and multi-task training address distinct task demands. This section also situates repeated augmented sampling as a way to improve generalization.

2. Related work

  • CNNs trained on classification datasets became competitive feature extractors for instance retrieval [1] [9] [35].Source: §2 · p. 2
  • Retrieval-specific systems modify classification trunks with regional pooling: R-MAC aggregates regional CNN features [43], while Gordo et al. [10] [11] fine-tune such descriptors with ranking loss.Source: §2 · p. 2
  • Generalized mean pooling replaces R-MAC's regional aggregation with exponentiated whole-image feature pooling that localizes features on points of interest.Source: §2 · p. 3
  • Multi-task networks can benefit from shared capacity, but their design and training involve heuristics for parameter sharing and gradient weighting.Source: §2 · p. 3
  • Repeated augmentations place augmented copies of an image together in a batch, improving generalization with the same batch size but fewer distinct images.Hoffer et al. [19] concurrently introduced a related batch-augmented sampling strategy for larger distributed batches.Source: §2 · p. 3

3. Architecture design

MultiGrain unifies classification and instance retrieval in an end-to-end architecture, addressing differences in their pooling and batching needs. Its design combines generalized-mean pooling, joint training, repeated augmentations, PCA whitening, and larger-resolution inference.

3. Architecture design

  • MultiGrain bridges classification and retrieval requirements in a unified architecture trained end to end.The architecture addresses differences between the tasks, including retrieval-specific pooling and batching.Source: §3 · p. 3, Table 1 · p. 3

3.1. Spatial pooling operators

  • GeM pooling with p>1 emphasizes salient features, and adjusting its exponent supports resolution changes between training and test time.GeM generalizes average pooling at p=1 and spatial max pooling at p=∞; the paper applies and evaluates it for classification.Source: §3.1 · pp. 3–4

3.2. Training objective

  • The joint objective combines classification cross-entropy with a retrieval margin loss over normalized embeddings and match labels.Distance-weighted sampling supports batches of about 80–120 instances with 3–5 positive images per instance.Source: §3.2 · pp. 4–5

3.3. Batching with repeated augmentation (RA)

  • Repeated augmentations form batches from multiple augmented views of sampled images, labeling views of the same image as positive pairs.With larger batches, this scheme outperforms standard i.i.d. sampling at the same batch size and learning rate, though small batches reduce performance.Source: §3.3 · p. 5

3.4. PCA whitening

  • After training, PCA whitening uses an external unlabeled image dataset, and the classifier can be adjusted to use the whitened embeddings.The authors report that inducing decorrelation via a loss is insufficient to ensure features generalize well.Source: §3.4 · p. 5

3.5. Input sizes

  • Training at 224×224 supports larger-resolution inference by increasing the pooling exponent; Figure 3 shows clearer car locations at full resolution and fewer noisy detections at p*=3.Other p* choices can give similar results when fine-tuned at a given resolution with cross-entropy.Source: §3.5 · pp. 5–6, Figure 3 · p. 6

4. Experiments and Results

The experiments evaluate a ResNet-50-based MultiGrain model across image classification and retrieval, varying test resolution, pooling, and the loss tradeoff. Results cover classification gains, retrieval comparisons, and the settings where performance is constrained.

4.1. Experimental settings

  • Experiments train ResNet-50 on ImageNet-2012 with 224 × 224 crops and evaluate classification plus retrieval on Holidays, UKB, and C10k.C10k combines the INRIA Copydays strong subset with 10K distractor images.Source: §4.1 · p. 6

4.2. Expanding resolution with pooling exponent

  • At 224 pixels, the trained pooling exponent p=3 is best for classification; larger test scales require p*>p for both classification and retrieval.Figure 4 reports a clear interaction between test resolution and pooling exponent.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3. Analysis of the tradeoff parameter

  • 77.4% top-1 at s*=224 with λ=0.5 exceeds λ=1, while λ=0.1 performs poorly; classification becomes dominant over retrieval by epoch 120.The authors attribute the late shift to the network having learned to cancel data augmentation.Source: §4.3 · p. 7

4.4. Classification results

  • Four gains come from repeated augmentations (+0.6%, p=1), margin loss (+0.2%, p=1), p=3 pooling (+0.4%), and 500-pixel evaluation (+1.2%, reaching 78.6 top-1).At 800 pixels, accuracy falls to 77.2% (76.2% without p* adaptation), which the authors link to the larger training–testing scale discrepancy.Source: §4.4 · p. 7
  • 79.4% top-1 accuracy at resolution 500 is reported for MultiGrain, which the authors say outperforms the state of the art for ResNet-50 trained at 224.At resolution 224, MultiGrain reaches 78.2% top-1, versus 77.6% for AutoAugment and 76.7% for mixup.Source: Table 2 · p. 8, §4.4 · p. 8

4.5. Retrieval results

  • At comparable resolutions, MultiGrain improves retrieval accuracy over the ResNet-50 baseline on all datasets; 500 pixels is a faster operating point than traditional 800–1000-pixel inference.At resolution 500, the margin-loss results are slightly lower than without it, partly attributed to limited transfer from IN-aug to retrieval-dataset variations.Source: §4.5 · p. 8, Table 3 · p. 8

5. Conclusion

MultiGrain is a unified embedding for image classification and instance retrieval, built on a convolutional trunk with GeM pooling and two training heads. The paper reports strong performance on both tasks and a new classification state of the art among results using the same trunk.

5. Conclusion

  • MultiGrain combines image classification and instance retrieval in a unified embedding built on a classical convolutional trunk, GeM layer, and two training heads.The authors report that changing the pooling layer permits higher-resolution inference while retaining a smaller training resolution.Source: §5 · p. 8
  • The embeddings perform well on classification and retrieval, and set a new state of the art on pure classification among results using the same convolutional trunk.The authors conclude that retrieval and classification can benefit from each other.Source: §5 · p. 8
  • An implementation of MultiGrain is open-sourced.It is available at the repository named in the paper.Source: §5 · p. 8

Supplementary Material

The supplementary material collects an idealized data-augmentation experiment, method hyperparameters, retrieval ablations, and an additional experiment using a pretrained ConvNet.

Supplementary Material

  • Appendix A examines data-augmented batches in a toy model, while appendices B and C list method hyperparameters.Appendix C covers data-augmentation settings.Source: Supplementary Material · p. 11
  • Appendix D provides additional ablation results for retrieval.Source: Supplementary Material · p. 11
  • Appendix E applies MultiGrain ingredients to a pretrained ConvNet at almost no additional training cost and reports what appear to be the best ImageNet-2012 classification results for a ConvNet with publicly available weights.Source: Supplementary Material · p. 11

A. Data-augmented batches: toy model

This appendix examines whether placing augmented versions of images in the same batch can improve optimization and generalization. It reports improved final accuracy in a network experiment and faster convergence in an idealized toy model.

A. Data-augmented batches: toy model

  • Batch-augmented sampling improves final validation accuracy over uniform batches, despite reduced batch diversity in the initial epochs.The comparison uses cross-entropy training with the paper's schedule and p = 1 pooling; the cited figure tracks ImageNet validation accuracy.Source: A. Data-augmented batches: toy model · p. 11, Figure A.1 · p. 11
  • The toy experiment investigates whether batch-augmentation benefits extend beyond the paper's architecture, using a linear SVM and symmetry across the x-axis as label-preserving augmentation.The synthetic training data are sampled from positive and negative two-dimensional Gaussian distributions; the experiment uses batches of size two.Source: A. Data-augmented batches: toy model · pp. 11–12
  • The two sampling schedules differ in how examples are batched: uniform sampling draws distinct elements randomly, while paired sampling places an image and its augmentation together.The appendix describes the schedules as otherwise using the same optimization setup for its network comparison.Source: A. Data-augmented batches: toy model · p. 12
  • Pairing augmented examples in one batch accelerates convergence in the toy model, with accuracy averaged across 100 runs.Figure A.3 presents the test-accuracy evolution for the synthetic-data SVM.Source: A. Data-augmented batches: toy model · p. 12, Figure A.3 · p. 12
  • The idealized experiment indicates that repeated augmentation can boost optimization and generalization, reinforcing the effect of data augmentation.Source: A. Data-augmented batches: toy model · p. 12

B. Margin loss hyper-parameters

The supplementary material points to tables listing margin-loss hyperparameters and full data-augmentation transforms and parameters.

B. Margin loss hyper-parameters

C. Data augmentation hyper-parameters

Table C.1 lists the transformations and parameters used for the full data augmentation in the experiments.

C. Data augmentation hyper-parameters

D. Additional results and ablation study for Multigrain in retrieval

The ablation reports that MultiGrain’s components improve on using activations alone and remain competitive at matched resolution and complexity. It also notes limits from triplet-loss choices and classification-specific AutoAugment.

D. Additional results and ablation study for Multigrain in retrieval

E. Evaluation of off-the-shelf classifiers at higher resolutions

The section evaluates pretrained classifiers at higher resolutions by adapting the pooling exponent, and reports improved accuracy for every tested network. It also identifies untrained p > 1 GeM pooling and the cost of large-network experiments as qualifications.

E. Evaluation of off-the-shelf classifiers at higher resolutions

  • The additional classification results use off-the-shelf pretrained classifiers with standard average pooling (p = 1).Source: E. Evaluation of off-the-shelf… · p. 12
  • The strategy adapts pretrained GeM classifiers to a higher evaluation scale by finetuning p* with cross-entropy on resized ImageNet training images.Backpropagation reaches only the final classification layer before the pooling layer, limiting memory use relative to full finetuning.Source: E. Evaluation of off-the-shelf… · pp. 12–13
  • The p* adaptation uses 50,000 training images, 50 per category, and one pass through the reduced dataset.Source: E. Evaluation of off-the-shelf… · p. 13
  • For each network, some higher evaluation scale and adapted p* perform better than standard evaluation.Table E.1 reports top-1/top-5 validation accuracy at training resolution and at higher resolutions s* = 350, 400, 450, 500, with the corresponding adapted p*.Source: E. Evaluation of off-the-shelf… · p. 13, Table E.1 · p. 13
  • These networks were not trained with GeM pooling at p > 1; training large networks from scratch with p > 1 and other MultiGrain components is left for future work because it is significantly more expensive.The passage says such training is likely to yield higher values, citing scale insensitivity and better performance at larger resolution as benefits of p > 1 pooling.Source: E. Evaluation of off-the-shelf… · p. 13
Loading 1902.05509v2…