Source-linked AI summary
Diversity with Cooperation: Ensemble Methods for Few-Shot Classification
Nikita Dvornik, Cordelia Schmid, Julien Mairal
TL;DR
Few-shot classifiers must adapt to new classes from only a few annotations, while the benefits of meta-learning remain unsettled. The paper uses ensembles with cooperation and prediction-diversity strategies to reduce classifier variance. It reports state-of-the-art few-shot performance from a distilled single network without meta-learning or test-time computational overhead.
Problem
Few-shot classification requires models to adapt to new classes from very few annotated samples, while meta-learning's empirical benefits remain unclear.
Method
The paper trains ensembles of deep networks and introduces penalties that encourage cooperation while preserving prediction diversity.
Results
A single network obtained by distillation achieves state-of-the-art few-shot performance without relying on meta-learning or adding computational overhead at test time.
Takeaways & Limitations
Ensemble-based variance reduction, combining cooperation and diversity, is effective for few-shot learning, and its behavior can be retained in one distilled network.
Abstract
from arXiv · showhide
Few-shot classification consists of learning a predictive model that is able to effectively adapt to a new class, given only a few annotated samples. To solve this challenging problem, meta-learning has become a popular paradigm that advocates the ability to "learn to adapt". Recent works have shown, however, that simple learning strategies without meta-learning could be competitive. In this paper, we go a step further and show that by addressing the fundamental high-variance issue of few-shot learning classifiers, it is possible to significantly outperform current meta-learning techniques. Our approach consists of designing an ensemble of deep networks to leverage the variance of the classifiers, and introducing new strategies to encourage the networks to cooperate, while encouraging prediction diversity. Evaluation is conducted on the mini-ImageNet and CUB datasets, where we show that even a single network obtained by distillation yields state-of-the-art results.
1. Introduction
Few-shot classification requires adapting models to new classes from very few annotations, where fine-tuning often performs poorly and meta-learning's empirical benefits remain unclear. This paper targets classifier variance with ensembles that combine cooperation and diversity, outperforming state-of-the-art methods without meta-learning and retaining performance after distillation.
- The paper focuses on ensemble-based variance reduction for few-shot classifiers, training multiple CNNs and then removing their prediction layers.
- Few-shot classification evaluates adaptation to novel classes using typically one or five annotated samples, without external data or a pre-trained network.
- Fine-tuning convolutional neural networks on new tasks with very few samples has been shown to provide poor results.
- Meta-learning transfers knowledge across sampled classification tasks, but its empirical benefits remain unclear, with standard CNN training sometimes outperforming meta-learning.
- Cooperation and diversity penalties coordinate networks during training while encouraging varied predictions, improving on independently trained ensembles.
- The approach significantly outperforms current state-of-the-art few-shot techniques without meta-learning, while distillation produces a single network with minor accuracy loss using additional unlabeled data.
2. Related Work
Related work frames few-shot learning around meta-training and meta-testing, while recent evidence questions whether meta-learning is optimal. Ensemble methods reduce estimator variance through diverse predictions, and distillation can preserve ensemble behavior without test-time overhead.
- Few-shot classification: Few-shot classification separates meta-training on annotated data from meta-testing on novel categories with few labeled examples.
- Meta-learning: Meta-learning samples tasks to train models for new problems, including methods that learn network initialization or optimization rules.
- Meta-learning: Recent studies found standard feature training followed by limited meta-learning could outperform or match end-to-end meta-learning.
- Ensemble methods: Ensemble methods reduce estimator variance and can improve prediction quality by averaging diverse classifiers, even when individual classifiers perform poorly.
- Ensemble methods: Distillation trains a single network to mimic an ensemble, potentially preserving performance while removing neural-network ensemble overhead at test time.
- Ensemble methods: This paper encourages cooperation through conditional class-probability matching and finds diversity increasingly important as the ensemble grows.
3. Our Approach
The approach combines fixed-feature mean-centroid classifiers with ensembles of deep networks, using pairwise penalties to balance cooperation and prediction diversity. Ensemble predictions are averaged, and distillation compresses the ensemble into one model with relatively close performance.
- Mean-centroid classifiers: Few-shot classification uses a frozen CNN feature extractor, class prototypes formed from new-class examples, and nearest-centroid prediction.The CNN is trained on the base dataset, its final layer is removed, and the resulting features support mean-centroid classification.
- Mean-centroid classifiers: 77.28 ± 0.46% for the mean-centroid classifier versus 77.01 ± 0.50% for a parameterized alternative on 1,000 miniImageNet-test 5-shot tasks.The similar averages indicate that learning additional parameters is difficult in this very-low-sample regime.
- Learning ensembles of deep networks: Ensembles jointly train K CNNs with a pairwise penalty that can encode either prediction diversity or collaboration.Independent training still yields differing solutions because of different initializations and random seeds.
- Learning ensembles of deep networks: At meta-testing, K mean-centroid classifiers produce class probabilities that are averaged, which performs better empirically than voting.The ensemble reduces the high variance associated with few-shot classifiers.
- Encouraging diversity and cooperation: Diversity penalizes similarity among non-ground-truth class probabilities, encouraging different second-best and subsequent predictions across networks.This strategy can reduce individual-model accuracy while improving ensemble prediction accuracy, especially for larger ensembles.
- Encouraging diversity and cooperation: Symmetrized KL cooperation improves training stability, individual networks, and the ensemble, but excessive cooperation makes ensemble members too similar and reduces ensembling gains.Cooperation is mainly useful with small networks, while diversity becomes important as the ensemble grows.
- Ensemble distillation: Knowledge distillation trains a single network to match the ensemble’s average output, achieving performance relatively close to the ensemble without test-time ensemble overhead.The ensemble adds substantial training-time computation, while distillation removes the corresponding test-time overhead.
4. Experiments
Experiments evaluate cooperation, diversity, robust ensemble training, and distillation across few-shot benchmarks. Robust ensembles generally perform best, while distilled models approach ensemble performance and outperform reported baselines on mini-ImageNet and tiered-ImageNet.
- Experimental setup: Experiments use mini-ImageNet, tiered-ImageNet, and CUB under 1- and 5-shot evaluation, reporting mean accuracy over sampled tasks with confidence intervals.The standard evaluation samples 1,000 tasks, while tiered-ImageNet uses 5,000 experiments.
- Ensemble strategies: Robust ensembles generally perform best across ensemble sizes, while cooperation helps smaller ensembles and diversity benefits larger ones.On CUB, robust and diverse ensembles perform similarly at n = 20.
- Distillation: Distilled mini-ImageNet models remain close to their full ensembles, with about a 1% gap in 5-shot accuracy and slightly higher accuracy in 1-shot settings.Adding extra unannotated data reduces the 5-shot distillation gap.
- Benchmark comparisons: Distilled networks significantly outperform current meta-learning and other literature baselines on mini-ImageNet and tiered-ImageNet.The comparisons cover both 1-shot and 5-shot tasks.
- Relationship penalties: For relationship criteria, cosine similarity is slightly better for diversity, whereas positive symmetrized KL divergence performs better for cooperation.The reported diversity accuracies fall within error bars, while negative KL similarity can hurt performance at larger γ.
- Domain shift: Under mini-ImageNet-to-CUB domain shift, diverse ensembles achieve the best accuracy, while robust and distilled robust ensembles do not surpass a linear classifier on frozen features.Both robust approaches still outperform distance-based cosine classifiers.
5. Conclusions
The paper argues that ensemble methods reduce the high variance of few-shot classifiers and that cooperation complements prediction diversity. Distillation produces a single network with state-of-the-art few-shot performance without meta-learning or test-time computational overhead.
- Conclusions: Ensembling significantly reduces the high variance of distance-based few-shot classifiers.The conclusion identifies variance reduction as the central benefit of the ensemble approach.
- Conclusions: Cooperation during training is important alongside the diversity traditionally encouraged through randomization and data augmentation.The paper presents cooperation and diversity as complementary ensemble principles.
- Conclusions: Under domain shift, the reported evaluation shows that diverse ensembles outperform the robust ensemble variants and distance-based approaches.The domain-shift comparison uses models trained on mini-ImageNet and evaluated on CUB.
- Conclusions: A distilled single network achieves state-of-the-art few-shot performance without meta-learning and without computational overhead at test time.The conclusion frames distillation as preserving the practical benefit of ensembling in one model.
A. Implementation Details
Implementation details specify architectures, input resolutions, optimization schedules, and augmentation choices for the reported experiments. The configurations vary across mini-ImageNet, tiered-ImageNet, and model types.
- Architectures and optimization: ResNet18 is trained on 84x84 mini-ImageNet images with Adam and an initial learning rate of 3·10^-4.The learning rate is reduced by a factor of 10 after validation stagnation, with patience differing for individual models, ensembles, and distillation.
- Distillation: When distilling an ensemble into one network, the patience parameter is doubled relative to the corresponding training configuration.This adjustment is specified for the reported ResNet18 and WideResNet28 procedures.
- Architectures and optimization: WideResNet28 is trained on 80x80 mini-ImageNet images with Adam and an initial learning rate of 1·10^-4.Its schedule likewise reduces the learning rate after validation improvement stops.
- Architectures and optimization: Tiered-ImageNet experiments use ResNet18 with 224x224 inputs and an initial Adam learning rate of 3·10^-4.Training uses random crops, color augmentation, and weight decay λ = 1·10^-4; testing uses central crops.
B. Additional Results
The experiments analyze ensemble performance across ensemble sizes, architectures, and input resolutions on CUB and MiniImageNet. Cooperation and robust ensemble strategies show architecture- and setting-dependent advantages, while distillation provides a single-network evaluation option.
- Experimental setup: The experiments compare ensemble accuracy across ensemble sizes, network architectures, and input image resolutions.Figure A1 reports aggregated ensemble accuracy alongside the average performance of individual ensemble members.
- CUB: The CUB results report 1- and 5-shot classification accuracy across ensemble types and numbers of networks using 1000 independent test experiments.The table also includes distilled single-network results, with ’++’ denoting use of extra unannotated images for distillation.
- ResNet18 on MiniImageNet: On MiniImageNet with ResNet18 and 84x84 images, Cooperation training performs best for all ensemble sizes below 20.Other diversity-oriented strategies perform worse because individual networks are far from overfitting the training set.
- ResNet18 on MiniImageNet: For ResNet18 on 84x84 MiniImageNet images, robust ensembles perform best at n = 20, where diversity matters more.
- WideResNet28 on MiniImageNet: For WideResNet28 on 80x80 MiniImageNet images, Diverse training does not help, whereas robust ensembles outperform the other training regimes.The reported pattern supports generalization of the proposed solution across architectures.