Source-linked AI summary
Hard Mixtures of Experts for Large Scale Weakly Supervised Vision
Sam Gross, Marc'Aurelio Ranzato, Arthur Szlam
TL;DR
Standard CNNs struggle to keep pace with large weakly supervised image collections, motivating models that can use more capacity. This paper routes images to experts using CNN feature clusters, and reports improved tag-prediction accuracy from larger-capacity mixtures on these datasets.
Problem
Standard CNNs cannot keep up with the scale of weakly supervised image data and show evidence of underfitting on datasets with hundreds of millions of images.
Source: §1 · p. 1Method
Results
The mixture significantly improves test loss, q@m, and p@m over base models across both decoder choices, CNN architectures, and test sampling schemes.
Source: §4.4 · p. 6Takeaways & Limitations
On weakly supervised collections of 100 million to 500 million images, the hard mixture raises tag-prediction accuracy, trains experts in parallel, and costs twice the base model to evaluate.
Source: §5 · p. 8The experts are less efficient than a standard CNN in modeling power per parameter and data usage, since each sees only a fraction of the data.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Training convolutional networks (CNN's) that fit on a single GPU with minibatch stochastic gradient descent has become effective in practice. However, there is still no effective method for training large CNN's that do not fit in the memory of a few GPU cards, or for parallelizing CNN training. In this work we show that a simple hard mixture of experts model can be efficiently trained to good effect on large scale hashtag (multilabel) prediction tasks. Mixture of experts models are not new (Jacobs et. al. 1991, Collobert et. al. 2003), but in the past, researchers have had to devise sophisticated methods to deal with data fragmentation. We show empirically that modern weakly supervised data sets are large enough to support naive partitioning schemes where each data point is assigned to a single expert. Because the experts are independent, training them in parallel is easy, and evaluation is cheap for the size of the model. Furthermore, we show that we can use a single decoding layer for all the experts, allowing a unified feature embedding space. We demonstrate that it is feasible (and in fact relatively painless) to train far larger models than could be practically trained with standard CNN architectures, and that the extra capacity can be well used on current datasets.
1. Introduction
Hand annotation is laborious, while web and social-media image collections offer the possibility of vastly larger weakly supervised training sets. The authors propose a simple mixture of experts for this setting, arguing that large models can improve tag-prediction accuracy where standard CNNs underfit.
1. Introduction
- Hand annotation is laborious, whereas web and social-media collections could provide truly gigantic training sets with weak supervision.Prior work showed that vision models can be built from weakly supervised rather than hand-annotated data.Source: §1 · p. 1
- Standard CNNs cannot keep pace with the volume of weakly supervised images and show evidence of underfitting on datasets with hundreds of millions of images.The paper notes that more images are posted to photo-sharing sites in a day than current standard CNN training pipelines can process.Source: §1 · p. 1
- The paper clusters a trained CNN’s feature space to assign each image to an expert and also proposes a shared decoder for transfer-oriented features.It reports that this simple approach can improve test accuracy by efficiently training more powerful models for weakly supervised tag prediction.Source: §1 · p. 1
2. Models
The model routes each image to one expert using clusters in a pretrained CNN’s feature space, then uses either separate decoders or a shared decoder. This design supports parallel expert training and low-cost evaluation, with trade-offs in parameter and data efficiency and without end-to-end training.
2. Models
- A pretrained CNN supplies feature representations that are clustered with K-means to assign images to experts, rather than learning the gate end to end.The model uses a hard assignment: each input is routed to one expert.Source: §2 · p. 2
- Experts can predict labels through separate decoders or produce features for a shared decoder, while each expert still predicts across the full label set.The shared decoder maps images into a shared embedding space; separate decoders suit settings focused on label prediction.Source: §2 · p. 2, Figure 1 · p. 2, Figure 2 · p. 2
2.1. Advantages and liabilities of hard mixtures of experts versus standard CNN’s
- Independent experts are easier to train in parallel, and evaluating a routed image requires computing the gate and only its selected expert.A CNN with K times as many feature maps could require a forward pass costing K times as much.Source: §2.1 · p. 2
- The mixture uses parameters and data less efficiently than a standard CNN because experts do not share interacting parameters and each sees only part of the training data.In the shared-decoder version, decoder parameters see all the data.Source: §2.1 · p. 2
2.2. Advantages and liabilities of no end-to-end training
- End-to-end training might yield more accurate models, but at this scale it remains an exceptional engineering effort with infrastructure that is not widespread.The authors describe their alternative as simple to use with a GPU because experts train independently; even serial expert training takes a small multiple of trunk training time.Source: §2.2 · p. 3
- A contemporaneous system demonstrated end-to-end mixture training at scale for NLP RNNs, whereas this work presents a simpler approach.The cited work is.Source: §2.2 · p. 3
3. Related Work
The paper builds on weakly supervised vision and mixture-of-experts research, but uses hard routing based on clustered image features rather than class-derived expert assignments. It also uses a simple expert-selection output, with a distinct shared-decoder variant.
3. Related Work
- Prior work established noisy tags and captions from large image collections as supervision for learning image features, and this paper studies larger-capacity models in the same framework.The cited examples are.Source: §3 · p. 3
- Unlike soft assignments and iterative gating approaches, this model uses one K-means clustering pass on a non-MoE CNN’s features to assign experts.Both the gate and experts are CNNs rather than SVMs; mixture models were introduced in.Source: §3 · p. 3
- Instead of routing from predicted class superclasses, the model clusters image embeddings, which better fits multilabel images and avoids consulting experts for every likely tag.The authors note that embedding clustering may also mitigate unrecoverable gating errors in single-label classification.Source: §3 · p. 3
- Prior approaches combine expert outputs through optimization, parallel layers, or weighted distributions, while this model selects one expert’s output or sends its features to a shared decoder.The cited approaches are.Source: §3 · p. 3
- Concurrent work trained a hard mixture for large-scale language modeling, while this paper addresses vision with a simpler gating scheme.Source: §3 · p. 3
4. Experiments
The experiments evaluate mixture-of-experts models on weakly supervised hashtag prediction, ImageNet classification, and transfer to labeled datasets. On YFCC100M, mixtures outperform their base models, while ImageNet shows no accuracy improvement at this scale.
4. Experiments
- Experiments span ImageNet classification, weakly supervised hashtag prediction, and transfer of fixed feature extractors to labeled datasets; the Instagram food dataset cannot be released for reproduction.The Instagram collection contains 440 million anonymized images.Source: §4 · pp. 3–4
4.4. Tag prediction results
- Mixture models significantly outperform base models in test loss, q@m, and p@m across base architectures and decoder types; increasing experts from 25 to 100 raises q@m and p@m.The ResNet-18 MoE-50 shared-decoder model has about 36 times as many parameters as its base model and costs twice as much to evaluate. Base-model train loss is nearly equal to test loss, which the authors interpret as underfitting.Source: Table 1 · p. 4, §4.4 · p. 6
4.1. Weakly Supervised Training Loss
- Tag targets are M-dimensional multi-hot vectors, and predictions come from either independent expert heads or a shared decoder.Source: §4.1 · p. 4
4.2. Model and training details
- Training used 50 centroids after projecting gater features to 256 PCA dimensions; on YFCC100M, the gater took 3–5 days on 8 GPUs and each expert under 16 hours on one GPU.Experts were trained without momentum, with the learning rate halved every five epochs.Source: §4.2 · pp. 4–5
4.3. Imagenet results
- ImageNet classification showed no accuracy improvement with the 50-expert mixture at this dataset scale.ImageNet is described as a hand-annotated dataset of 1.2 million images across roughly 1,000 categories.Source: Table 4 · p. 5, §4.3 · p. 5
4.5. Analysis of YFCC100M clusters
- Clusters vary in size but are all utilized; tag distributions range from concentrated in a few clusters to spread across many, and sparse distributions are often easier to predict.Figure 6 relates each tag’s cluster-distribution sparsity to its q@10 accuracy in ResNet-18 MoE-50.Source: §4.5 · p. 7, Figure 6 · p. 7, Figure 5 · p. 7
4.6. Transfer
- For transfer, the authors fixed the YFCC100M-trained CNN features and retrained a linear classifier for each target dataset.The decoder was removed, the remaining CNN layers were fixed, and target-dataset labels supervised the new classifier.Source: §4.6 · pp. 7–8
4.6.1 Transfer Results
- Transfer gains were reported across datasets, especially for fine-grained CUB and Oxford Flowers, but improved tag prediction did not clearly translate into improved transfer performance.A single ImageNet-trained model outperformed the YFCC100M-trained models on ImageNet.Source: §4.6.1 · p. 8
5. Conclusion
The paper concludes that large weakly supervised image collections require specialized architectures, and that a hard mixture of experts can raise tag-prediction accuracy at 100–500 million images.
5. Conclusion
- A hard mixture of experts significantly raises tag-prediction accuracy on weakly supervised collections of 100 million to 500 million images.The authors argue that training convolutional models on datasets with many billions of images will require specialized architectures unless optimization or hardware changes substantially.Source: §5 · p. 8
- Routing each image to a single expert keeps evaluation at twice the computational cost of the base model.Source: §5 · p. 8
- Independent experts are easy to train in parallel, supporting model training at this scale.Source: §5 · p. 8
- A shared-decoder version shows encouraging transfer results, allowing expert features to be used for transfer tasks.Source: §5 · p. 8