Source-linked AI summary
Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle
TL;DR
Few-shot classification lacks benchmarks that adequately capture heterogeneous tasks and generalization beyond individual datasets. The paper introduces Meta-Dataset, a large and diverse benchmark with realistic episode construction, evaluates multiple baselines and meta-learners, and finds that fo-Proto-MAML ranks highest on average across dataset test sets. The experiments also identify unresolved challenges involving heterogeneous training sources, meta-learning benefits, and robustness to test-episode data amounts.
Problem
Existing few-shot benchmarks are limited by homogeneous tasks, within-dataset evaluation, and insufficiently informative comparisons.
Method
Meta-Dataset combines diverse datasets with hierarchy-aware, imbalanced, and variable-way, variable-shot episode generation, and evaluates established baselines, meta-learners, and Proto-MAML.
Results
fo-Proto-MAML is the top performer by average test-set rank across datasets, with Prototypical Networks also strong and Finetune a notable competitor.
Takeaways & Limitations
Meta-Dataset provides a broader environment for studying few-shot learning across heterogeneous data sources and realistic task variation.
Takeaways & Limitations
Current models do not always improve with multiple training sources, meta-learning is not always beneficial across datasets, and robustness varies with test-episode data amount.
Abstract
from arXiv · showhide
Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose Meta-Dataset: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on Meta-Dataset, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in Meta-Dataset. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.
1 INTRODUCTION
Few-shot classification seeks to learn new classes from very few examples, but standard benchmarks are limited in realism, diversity, and discriminatory power. Meta-Dataset addresses these gaps with diverse datasets, realistic task variation, and broader evaluation of learning strategies.
- Few-shot classification learns to classify unseen examples into new classes using only a few reference examples.
- Omniglot and mini-ImageNet have become less informative because recent methods achieve similarly high accuracy on them.
- Existing benchmarks use homogeneous tasks, measure only within-dataset generalization, and ignore relationships between classes.
- Meta-Dataset combines multiple diverse datasets with hierarchy-aware class selection, realistic imbalance, and variable numbers of classes and shots.
- The paper evaluates established models, introduces baselines and a new meta-learner, and analyzes data, training-source, pretraining, and meta-learning effects.
2 FEW-SHOT CLASSIFICATION: TASK FORMULATION AND APPROACHES
Few-shot episodes train models to predict query labels from support examples, using either episodic meta-learning or non-episodic representations and adaptation. The section reviews distance-based, relation-based, gradient-based, and hybrid approaches, including Proto-MAML.
- Task Formulation: A few-shot episode contains a support set of labeled examples and a held-out query set used to evaluate generalization.
- Training Approaches: Meta-learners are trained across tasks sampled from disjoint training classes, while non-episodic methods first learn representations from all training classes.
- Non-episodic Approaches: The reviewed non-episodic baselines include k-NN, Finetune, and the cosine-classifier variant Baseline++.
- Prototypical Networks: Prototypical Networks average support embeddings into class prototypes and classify queries by Euclidean distance to the nearest prototype.
- MAML: MAML adapts parameters within each episode and learns an embedding initialization designed for rapid task adaptation.
- Proto-MAML: Proto-MAML initializes MAML’s task-specific linear layer from prototypical-network weights and biases before support-set optimization.
3 META-DATASET: A NEW FEW-SHOT CLASSIFICATION BENCHMARK
Meta-Dataset is a large, multi-dataset benchmark designed to evaluate few-shot learners on realistic, heterogeneous tasks. Its episode generator incorporates dataset diversity, class hierarchies, variable ways and shots, and class imbalance.
- Meta-Dataset changes both the data sources and episode formulation to measure progress on realistic few-shot classification tasks.
- The benchmark combines ten existing datasets and supports evaluation of generalization to entirely new datasets.
- Episodes use classes from one dataset, reserve Traffic Signs and MSCOCO entirely for evaluation, and exploit ImageNet and Omniglot hierarchies.
- Episode generation samples a dataset, selects classes according to dataset-specific procedures, and samples support and query examples.
- Query sets are class-balanced, while support sizes and per-class shots vary to produce realistic imbalance and few-shot conditions.
4 RELATED WORK
Related work situates Meta-Dataset among meta-learners, representation-based few-shot classifiers, comparison-based methods, and broader image-classification benchmarks. Unlike some related benchmarks, Meta-Dataset evaluates generalization to new classes and datasets.
- Prior few-shot methods include rapid-adaptation meta-learners, representation-based classifiers, and methods comparing support and query examples.
- The Visual Domain Decathlon uses ten datasets but evaluates held-out examples from the same datasets rather than new classes or datasets.
- Meta-Dataset is distinguished by combining multi-dataset training with evaluation of generalization to new classes and datasets.
5 EXPERIMENTS
Experiments evaluate models on META-DATASET under different training sources, task sizes, initialization schemes, and the use of meta-training. Results identify strong methods while exposing sensitivity to heterogeneous data and test-task conditions.
- Main results: fo-Proto-MAML is the top performer by average rank, while Prototypical Networks also perform strongly and Finetune remains competitive.Table 1 compares models trained on ImageNet-only and all-dataset sources using average rank, where smaller is better.
- Training on all datasets: Training on all datasets substantially improves performance for Omniglot, Quick Draw, and Aircraft compared with ImageNet-only training.The improvement is observed across models, but the result does not extend uniformly to the other test sources.
- Ways and shots analysis: Increasing ways makes tasks harder, whereas increasing shots improves classification, with models benefiting at different rates as shots increase.Prototypical Networks and fo-Proto-MAML excel in very-low-shot settings but saturate faster; Finetune, Matching Networks, and fo-MAML improve more with additional shots.
- Effect of pre-training: ImageNet-pretrained initialization is often beneficial, but it can hurt on datasets substantially different from ImageNet, including Omniglot and Quick Draw.The effect depends on the test data source and can influence the final solution’s suitability across datasets.
- Effect of meta-training: Meta-learning usually provides a small benefit with ImageNet-only training, but that benefit often disappears or reverses when training uses all datasets.The comparison uses inference-only baselines that retain the meta-learners’ inference algorithms while removing episodic meta-training.
- Fine-grainedness analysis: The experiments also analyze fine-grainedness, finding the expected difficulty trend on the train sub-graph but not on the test sub-graph.The test sub-graph is not rich enough to exhibit a clear trend, whereas train-sub-graph performance agrees with finer-grained tasks being harder.
6 CONCLUSION
The paper introduces META-DATASET as a large-scale, diverse, realistic environment for few-shot classification. Its experiments identify unresolved challenges involving heterogeneous training data, task construction, initialization, and robustness to test-task data amounts.
- Conclusion: META-DATASET provides a large-scale, diverse, and realistic environment for few-shot classification.The benchmark is intended to support exploration of models across heterogeneous data and task conditions.
- Open challenges: Future work must address unclear strategies for creating training episodes, constructing validation tasks, and choosing initialization schemes across heterogeneous data.The paper identifies these design choices as unresolved directions for meta-learning across heterogeneous sources.
- Open challenges: Current models do not consistently improve with multiple training sources, meta-learning, or different amounts of data in test episodes.Different models excel in different parts of the test-task data spectrum, and meta-learning is not always beneficial across datasets.
.1 RECOMMENDATION FOR REPORTING RESULTS ON META-DATASET
The paper recommends reporting both average performance across datasets and robustness across varying task specifications. This combination evaluates both overall accuracy and performance when the test-time number of ways and shots are unknown.
- Average performance: Report average accuracy over 600 test episodes for each method and dataset, using average rank as the evaluation metric.Results should cover models trained on ImageNet only and on all datasets.
- Robustness analysis: Also report plots measuring robustness to variations in the number of shots and ways.The paper identifies Figures 2a and 2b for ImageNet-only training and Figures 5a and 5b for training on all datasets.
- Rationale: Evaluating both averages and task variations avoids assuming that the test-time setup will be known in advance.The recommended plots assess performance under different specifications of test tasks.
- Standardization: The released source code will include scripts for generating robustness plots and automatically computing ranks.These tools are intended to help standardize reporting procedures.
.2 DETAILS OF META-DATASET’S SAMPLING ALGORITHM
This section provides a complete description of sampling-algorithm steps that were previously explained only at a higher level in the main paper.
- The appendix expands the main paper’s higher-level explanation into a complete description of selected sampling steps.
STEP 1: SAMPLING THE EPISODE’S CLASS SET
Episode class sets are sampled from eligible nodes in the ImageNet hierarchy, producing tasks that vary from fine-grained to coarse-grained classification.
- ImageNet class sampling: Sample an eligible DAG node uniformly, requiring it to span at least 5 and at most 392 leaves.The procedure is applied separately within the specified train, validation, or test split.
- ImageNet class sampling: Use all spanned leaves when there are at most 50; otherwise randomly select 50 leaves as episode classes.
- Task granularity: Sampling nodes at different heights creates tasks with different semantic granularity.Small-height nodes yield fine-grained classes, while larger-height nodes support broader, more coarse-grained episodes.
STEP 2: SAMPLING THE EPISODE’S EXAMPLES
The episode sampler constructs class-balanced query sets and variable, potentially imbalanced support sets while covering the very-low-shot regime.
- Computing the query set size: Compute query images per class from class availability, reserving enough examples for support and capping each class’s query contribution at 10.The 0.5 multiplier preserves support examples, while the minimum over classes keeps the query set class-balanced.
- Episode examples: The resulting sampling design combines class-balanced querying with variable support sizes and class-specific shot allocation.
- Computing the support set size: Sample β uniformly from (0, 1] to vary the total support-set size.This enables smaller support sets even when many images are available and keeps at least one example per selected class through ceiling rounding.
- Computing the shot of each class: Support examples are allocated across classes using noisy class proportions based on dataset image counts.The noise factor α_c is sampled from [log(0.5), log(2)), with additional examples selected proportionally when available.
.3 DATASETS
META-DATASET combines 10 image datasets spanning natural images, handwritten characters, fine-grained categories, textures, drawings, signs, and object crops. Its evaluation and analysis examine diverse training sources, task robustness, pre-training, meta-learning, and model-selection choices.
- Datasets: META-DATASET is formed from 10 image datasets with substantially different visual domains and class structures.The collection includes ImageNet, Omniglot, Aircraft, Birds, DTD, Quick Draw, Fungi, VGG Flower, Traffic Signs, and MSCOCO.
- Datasets: Omniglot uses alphabet-level splits, with 30 training and 20 evaluation alphabets, making generalization more challenging than character-level splitting.Five of the 30 training alphabets are held out for validation, leaving 25 for training.
- Experimental setup: The study evaluates convolutional, residual, and wide-residual architectures, with meta-learners generally preferring ResNet-18 over the four-layer ConvNet.Relation Networks also use an additional architecture inspired by the original Relation Networks paper.
- Results: Training on all datasets does not consistently improve generalization over ImageNet-only training, although Omniglot and Quick Draw improve across models in particular cases.Performance is slightly worse in some cases, motivating further work on leveraging diverse data sources.
- Results: Pre-training usually helps on datasets visually similar to ImageNet but may hurt on Omniglot, Quick Draw, and Aircraft, suggesting ImageNet-like specialization.Meta-training usually helps on ImageNet but sometimes hurts when conducted across all datasets.