Source-linked AI summary
Low-Shot Learning from Imaginary Data
Yu-Xiong Wang, Ross Girshick, Martial Hebert, Bharath Hariharan
TL;DR
Visual recognition systems struggle to learn novel concepts from very few labeled examples, unlike humans’ ability to imagine variations from a single image. This paper jointly trains a meta-learner and hallucinator to generate classification-useful examples, achieving state-of-the-art low-shot performance with gains up to 6 points.
Problem
Low-shot learning asks how visual recognition systems can learn novel concepts from very few labeled examples, despite conventional systems often requiring thousands per class.
Method
The method jointly trains a meta-learner and hallucinator, which generates additional examples from few-shot training data for an end-to-end classification objective.
Results
The approach achieves significant gains across multiple meta-learners, including up to 6 points for prototypical networks, and state-of-the-art benchmark performance.
Takeaways & Limitations
Hallucinated examples can improve low-shot classification across different meta-learning methods without requiring extra annotations.
Takeaways & Limitations
Joint evaluation involves base–novel accuracy tradeoffs, and hyperparameter choices can substantially change these tradeoff points without changing average performance.
Abstract
from arXiv · showhide
Humans can quickly learn new visual concepts, perhaps because they can easily visualize or imagine what novel objects look like from different views. Incorporating this ability to hallucinate novel instances of new concepts might help machine vision systems perform better low-shot learning, i.e., learning concepts from few examples. We present a novel approach to low-shot learning that uses this idea. Our approach builds on recent progress in meta-learning ("learning to learn") by combining a meta-learner with a "hallucinator" that produces additional training examples, and optimizing both models jointly. Our hallucinator can be incorporated into a variety of meta-learners and provides significant gains: up to a 6 point boost in classification accuracy when only a single training example is available, yielding state-of-the-art performance on the challenging ImageNet low-shot classification benchmark.
1. Introduction
Low-shot learning addresses the challenge of teaching visual recognition systems new concepts from very few labeled examples. The paper proposes jointly learning a classifier and hallucinator that generates classification-useful examples, leveraging shared visual variation without extra annotations.
- Modern recognition systems often require thousands of examples per class, whereas humans can learn novel visual concepts from a single example.
- Meta-learning frames low-shot learning as optimizing a learner that maps small labeled training sets to classifiers evaluated on sampled test sets.
- Shared visual variations such as pose, translation, lighting, and articulation could let systems hallucinate additional examples for novel concepts.
- General-image hallucination is difficult because generative models can suffer mode collapse and may capture modes unlike those useful for classification.
- The proposed method optimizes hallucination for classifier usefulness rather than diversity or realism, jointly incorporating it into the classification objective.
- The approach unifies meta-learning with a hallucinator, works with different meta-learners without extra annotations, and improves single-example accuracy by up to 6 points.
2. Related Work
Related work addresses low-shot learning through generative models, transformation sharing, invariant representations, and meta-learning. The paper positions its method as end-to-end transformation learning that avoids extra annotations and brittle heuristics.
- Generative approaches share priors across categories, but unconstrained-domain models still cannot capture the entirety of the data distribution.
- Transformation-based methods generate novel-class examples by applying sampled transformations, using either pose annotations or transferred transformations.
- The paper’s approach learns transformations end-to-end, avoiding both expensive annotations and carefully designed heuristic pipelines.
- Representation-based methods seek features invariant to intra-class variation, sharing features across seen and novel classes or learning them incrementally.
- Meta-learning methods directly train parametrized mappings from training sets to classifiers using architectures such as RNNs, memory-augmented networks, MLPs, gradient updates, or prototypes.
3. Meta-Learning
Meta-learning trains classification algorithms to perform well on small datasets by simulating low-shot tasks during meta-training and applying the learned algorithm to novel tasks. The paper extends this framework with hallucinated training examples and also introduces prototype matching networks.
- Supervised learning applies a parameterized model to a training set to estimate p(y|x), while the classification algorithm h maps training data and a test input to label probabilities.
- Low-shot meta-learning estimates parameters for classification algorithms that can achieve high accuracy when the training set is small.
- Meta-training samples classes and examples from a large labeled dataset, optimizing the learner on the resulting classification tasks before meta-testing on novel tasks.
- The hallucination method is general across meta-learners: it adds generated examples to the sampled training set before classification.
- Prototypical networks assign probabilities using distances to class means in a learned feature space.
- Prototype matching networks combine contextual embeddings from matching networks with class-mean representations to reduce bias toward rare classes.
- Prototype matching networks learn feature-extractor, matching, and contextual-embedding parameters jointly.
4. Meta-Learning with Learned Hallucination
The method unifies meta-learning with a hallucinator that expands few-shot training sets using generated examples, and jointly optimizes hallucination for classification utility. The augmented examples are used by a differentiable classifier, allowing the hallucinator and meta-learner to be trained together.
- Hallucination during meta-testing: The hallucinator maps real examples and noise vectors to additional labeled examples, expanding the initial few-shot training set before classification.During meta-testing, generated examples are combined with real examples, while the hallucinator parameters remain fixed during classification.
- Meta-training the hallucinator: The hallucinator is optimized for examples that improve classifier learning rather than for realism or diversity alone.The objective directly trains hallucination through a classification objective embedded in the end-to-end model.
- Meta-training the hallucinator: Meta-training samples few-shot episodes, generates examples for each class, augments the training set, and feeds it to the classification algorithm to compute the final loss.The generated set is sized so each class has exactly naug examples.
- Meta-training the hallucinator: Differentiability of the classification algorithm with respect to augmented examples enables back-propagation through the classifier to train the hallucinator.Prototypical networks provide one example because feature extraction, class means, and distance-based probabilities can be differentiable.
- Meta-training the hallucinator: Joint training lets the classifier accommodate errors in hallucinated examples while directly targeting hallucinations useful for class distinctions.The training process is agnostic to the specific meta-learning algorithm and is reported to provide gains across different meta-learners.
5. Experimental Protocol
The experiments use an ImageNet-based low-shot benchmark with disjoint validation and test classes, while addressing calibration and evaluation problems caused by base–novel class tradeoffs. The protocol varies a novel-class prior and introduces separate measurements for novel and base classes.
- Benchmark: The benchmark trains a ConvNet feature extractor on base classes with thousands of examples, then evaluates recognition on additional novel classes.The benchmark is based on ImageNet images and subsets of ImageNet classes.
- Tradeoffs between base and novel classes: Joint-label evaluation creates base–novel performance tradeoffs, allowing hyperparameters to improve one class group while worsening the other without changing average performance substantially.Different methods can therefore achieve similar averages while emphasizing different base or novel class accuracies.
- Tradeoffs between base and novel classes: Class-prior calibration updates classifier probabilities using prior knowledge about whether an image belongs to base or novel classes.The novel-class prior is denoted µ.
- Tradeoffs between base and novel classes: Varying the novel-class prior µ can raise novel-class accuracy and lower base-class accuracy while overall accuracy remains fairly stable for matching networks.The prior may be known or cross-validated, although some practical settings lack held-out categories for cross-validation.
- A new evaluation: The new evaluation protocol separates comparisons by restricting the label space to novel or base classes and reporting distinct accuracy measurements.This design aims to make novel-class comparisons more directly comparable than joint-label evaluation.
- Benchmark: Disjoint validation and testing classes prevent hallucinator, meta-learner, and novel-class-prior hyperparameters from overfitting test-time novel classes.This protocol follows the benchmark design described by Hariharan and Girshick.
6. Experiments
Experiments evaluate learned hallucination across meta-learners, baselines, label spaces, and feature representations. Hallucination substantially improves low-shot accuracy, outperforming prior methods and simpler ablations.
- Experimental setup: Experiments compare prototypical, matching, and prototype matching networks, including versions using a meta-learned hallucinator.The hallucinator architecture is a three-layer MLP, and experiments use five trials for each novel-class sample size.
- Impact of hallucination: Up to 6 points for prototypical networks and 2 points for prototype matching networks, hallucination significantly improves novel-class accuracy.Gains remain significant through n = 20 for prototypical networks and n = 5 for prototype matching networks.
- Impact of hallucination: Hallucination gains decrease after cross-validating the novel-class prior, suggesting that hallucination also provides resilience to score mis-calibration.The benefit remains significant when extensive prior cross-validation is unavailable.
- Comparison to prior work: Our approach provides almost a 2 point improvement across baselines on novel classes and similar joint-label-space improvements after prior cross-validation.The paper reports that the approach achieves state-of-the-art performance.
- Comparison to prior work: Prototype matching networks outperform matching networks on novel classes as more examples become available and perform better across the joint label space.The comparison is reported in Table 1.
- Ablations: Simple Gaussian, dropout, and weighted-average hallucinators often hurt accuracy and lag the learned approach by at least 3 points.Untrained and fixed hallucinators also hurt accuracy, indicating that useful hallucination requires meta-training.
- Visualization and limitations: Hallucinations remain fairly clustered, although the learned examples tend to cluster around class boundaries in t-SNE visualizations.Increasing hallucination diversity is identified as future work.
7. Conclusion
The paper presents a trained hallucinator that generates additional examples for low-shot learning and reports gains across multiple meta-learning methods. Its best model achieves state-of-the-art performance on a realistic benchmark, while the precise effect of hallucinated examples remains future work.
- The approach uses a trained hallucinator to generate additional examples for low-shot learning.
- The hallucinator is trained end-to-end with meta-learning and improves multiple meta-learning methods.
- The best proposed model achieves state-of-the-art performance on a realistic benchmark by a comfortable margin.
- The exact effect of the hallucinated examples remains an open direction for future work.