PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning for Semi-Supervised Few-Shot Classification

Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, Richard S. Zemel

arXiv:1803.00676v1cs.LGcs.CVstat.ML

TL;DR

Few-shot classification lacks labeled data for each new problem, motivating methods that can also exploit available unlabeled examples. The paper extends Prototypical Networks with semi-supervised episode-level prototype refinement for target and distractor-class data. Across the evaluated benchmarks, the proposed models consistently improve over baselines, with Masked Soft k-Means most robust when distractors are present.

  • Problem

    Few-shot classification must learn new classes from only a handful of labels, while realistic episodes may also include unlabeled examples and distractor classes.

  • Method

    The paper proposes end-to-end semi-supervised extensions of Prototypical Networks that incorporate unlabeled examples into prototype production during episodic training and inference.

  • Results

    The proposed models consistently improve over baselines in semi-supervised settings; Masked Soft k-Means is most robust with distractors and is best in all but one case across three datasets.

  • Takeaways & Limitations

    Meta-trained Prototypical Networks can learn to leverage unlabeled examples and improve predictions in semi-supervised few-shot classification.

  • Takeaways & Limitations

    The evaluation uses strictly less label information than previously published dataset results, so those published numbers serve as upper bounds rather than directly matched baselines.

Abstract

from arXiv · show

In few-shot classification, we are interested in learning algorithms that train a classifier from only a handful of labeled examples. Recent progress in few-shot classification has featured meta-learning, in which a parameterized model for a learning algorithm is defined and trained on episodes representing different classification problems, each with a small labeled training set and its corresponding test set. In this work, we advance this few-shot classification paradigm towards a scenario where unlabeled examples are also available within each episode. We consider two situations: one where all unlabeled examples are assumed to belong to the same set of classes as the labeled examples of the episode, as well as the more challenging situation where examples from other distractor classes are also provided. To address this paradigm, we propose novel extensions of Prototypical Networks (Snell et al., 2017) that are augmented with the ability to use unlabeled examples when producing prototypes. These models are trained in an end-to-end way on episodes, to learn to leverage the unlabeled examples successfully. We evaluate these methods on versions of the Omniglot and miniImageNet benchmarks, adapted to this new framework augmented with unlabeled examples. We also propose a new split of ImageNet, consisting of a large set of classes, with a hierarchical structure. Our experiments confirm that our Prototypical Networks can learn to improve their predictions due to unlabeled examples, much like a semi-supervised algorithm would.

1 INTRODUCTION

Few-shot meta-learning addresses classification with only a handful of labels, but existing episodes omit unlabeled data that humans can use. This work introduces semi-supervised episodes, benchmarks them, and extends Prototypical Networks to leverage unlabeled examples, including distractors.

  • Few-shot classification targets new problems with only a handful of labeled examples per class.
  • Meta-learning trains transferable learning algorithms across episodes representing different classification problems.
  • Existing few-shot episodes are limited because they omit additional unlabeled data and differ from how humans learn new concepts.
  • The proposed framework evaluates episodes with unlabeled examples from target classes and, more challengingly, distractor classes.
  • The study introduces adapted Omniglot and miniImageNet benchmarks, investigates both unlabeled-data settings, and proposes three Prototypical Network extensions.
  • Experiments show that the semi-supervised variants learn to use unlabeled examples and outperform purely supervised Prototypical Networks.

2 BACKGROUND

Few-shot learning uses episodic training to learn classifiers for unseen classes from small support sets, while Prototypical Networks classify queries by distances to class prototypes learned in an embedding space.

  • Episodic few-shot learning simulates test problems by sampling classes and constructing labeled support and query sets.
  • Training updates model parameters using query-set prediction loss after feeding the support set to the model.
  • Meta-learning trains a model to act like a learning algorithm that maps labeled examples to a predictor generalizing to new examples.
  • Prototypical Networks compute one prototype per class from the support examples and classify queries by distances to those prototypes.
  • The model learns an embedding function that places same-class examples near one another and different-class examples farther apart.
  • Training minimizes average query loss across episodes, while test episodes use the learned predictor to classify queries from unseen classes.

3 SEMI-SUPERVISED FEW-SHOT LEARNING

The paper extends Prototypical Networks to semi-supervised few-shot episodes by refining class prototypes with unlabeled examples, including episodes containing distractor classes. It develops differentiable refinement strategies that can incorporate relevant examples while limiting the influence of distractors.

  • Problem setup: Semi-supervised episodes add an unlabeled set R to the labeled support set S, while query set Q remains the prediction target.The unlabeled examples may belong to episode classes or to distractor classes outside the episode.
  • Semi-supervised Prototypical Networks: The proposed models refine ordinary Prototypical Network prototypes using unlabeled examples and retain the usual query-classification training objective.Refined prototypes replace the original prototypes when classifying query examples.
  • Soft k-means refinement: Soft k-means assigns unlabeled examples softly to episode classes and incorporates them into refined prototypes through a differentiable clustering procedure.It starts from ordinary prototypes, estimates partial assignments from Euclidean distances, and updates the prototype locations.
  • Refinement depth: A single refinement step was sufficient in experiments, with additional recursive refinement failing to improve results.This finding applies to the described refinement processes, including regular soft k-means and masking.
  • Distractor handling: A distractor-cluster variant adds an additional cluster to absorb examples from irrelevant classes, using an origin-centered prototype and a learned distractor length-scale.The approach is intended to prevent distractors from partially influencing the legitimate class prototypes.
  • Soft k-means with masking: The masking variant models distractors as examples outside regions around legitimate prototypes, using learned thresholds, slopes, and soft masks to control each example’s contribution.The masking mechanism is differentiable and allows an MLP to learn whether to include or ignore unlabeled examples.

4 RELATED WORK

This work extends episodic metric-based few-shot learning toward semi-supervised episodes with unlabeled examples and transfer across disjoint training and test classes.

  • Metric learning: Metric-learning methods embed same-class examples nearby and separate-class examples far apart, then classify using support-query embedding distances.The paper extends Prototypical Networks within this framework.
  • Semi-supervised learning: Semi-supervised related work includes self-training, which adds confidently pseudo-labeled unlabeled items to the training set.The paper also relates to transductive learning, but separates unlabeled and query sets during meta-learning to avoid memorization.
  • Clustering: Clustering work with outliers aims to discover and ignore irrelevant examples so they do not shift cluster locations into poor partitions.This objective is relevant to episodes containing distractor classes.
  • Contribution: The paper extends these literatures by learning transfer from training classes Ctrain to disjoint test classes Ctest rather than training and evaluating within one dataset.This contribution places semi-supervised learning and clustering in the few-shot meta-learning setting.

5 EXPERIMENTS

The experiments evaluate semi-supervised Prototypical Network extensions across three datasets and episodes with or without distractor classes. Proposed models generally outperform baselines, with Masked Soft k-Means most robust under distractors and accuracy improving as unlabeled-set size increases.

  • Datasets: The evaluation uses Omniglot, miniImageNet, and the proposed tieredImageNet across semi-supervised few-shot settings.Omniglot has 6,492 rotated character classes; miniImageNet has 100 classes; tieredImageNet has 608 classes grouped into 34 higher-level categories.
  • Datasets: tieredImageNet separates 20 training, 6 validation, and 8 testing categories to keep training and testing classes hierarchically distinct.Its hierarchy is designed to avoid semantically close categories being split across train and test.
  • Semi-supervised setup: Each dataset is split into disjoint labeled and unlabeled images, using 10% labeled data for Omniglot and tieredImageNet and 40% for miniImageNet.The authors report averages over 10 random labeled/unlabeled splits.
  • Evaluation: Models are compared with supervised Prototypical Networks and test-time Semi-Supervised Inference in episodes with and without distractor classes.The proposed models refine prototypes during both training and testing, unlike the test-time-only refinement baseline.
  • Results: Across all three benchmarks, at least one proposed model outperforms the baselines, while all three usually outperform them without distractors.With distractors, Masked Soft k-Means is best in all but one case and approaches the no-distractor upper bound.
  • Results: Test accuracy improves clearly as unlabeled items per class increase from 0 to 25, although the models were trained with M = 5.This demonstrates extrapolation to larger unlabeled sets and improved representations after semi-supervised refinement.

6 CONCLUSION

The paper introduces semi-supervised few-shot learning with unlabeled examples in each episode, including settings with novel distractor classes, and extends Prototypical Networks for this paradigm.

  • 6 CONCLUSION: The proposed paradigm adds an unlabeled set to each few-shot learning episode, including cases where unlabeled examples come from novel distractor classes.The work also introduces tieredImageNet to support evaluation with larger, hierarchically organized data.
  • 6 CONCLUSION: Several novel Prototypical Network extensions show consistent improvements over baselines in semi-supervised settings.The models are designed to use unlabeled examples when producing class prototypes.
  • 6 CONCLUSION: The study evaluates the proposed methods on adapted Omniglot and miniImageNet benchmarks and on the new tieredImageNet split.The new split addresses the need for larger datasets with hierarchical label structure and separate labeled and unlabeled data.

A OMNIGLOT DATASET DETAILS

The Omniglot experiments use the established Vinyals et al. train/test split, add a validation split, and partition each class into disjoint labeled and unlabeled portions.

  • A OMNIGLOT DATASET DETAILS: The Omniglot experiments use the same train/test split as Vinyals et al. (2016), with a separately created validation split for hyper-parameter selection.Models are trained only on the training split.
  • A OMNIGLOT DATASET DETAILS: Training, validation, and testing are separated by alphabet, with Gurmukhi characters divided between training and testing.The supplied alphabet lists specify the members of each split.
  • A OMNIGLOT DATASET DETAILS: The split details enumerate the specific alphabets assigned to training, validation, and testing.The test set includes Kannada, Malayalam, Manipuri, Mongolian, Tibetan, and other listed alphabets.

B tieredIMAGENET DATASET DETAILS

tieredImageNet organizes ImageNet classes into higher-level categories and separates training, validation, and testing categories to control hierarchical overlap and semantic distance.

  • B tieredIMAGENET DATASET DETAILS: Each tieredImageNet high-level category contains between 10 and 30 ILSVRC-12 classes, averaging 17.8 classes.Classes with multiple parent nodes were removed to ensure separation between training and test categories.
  • B tieredIMAGENET DATASET DETAILS: Test categories span different levels of separation from training categories, from relatively similar working-dog classes to more distinct geological formations.This design varies the semantic relationship between training and test classes.
  • B tieredIMAGENET DATASET DETAILS: The dataset uses 20 training, 6 validation, and 8 testing categories drawn from 34 total categories.The category lists identify the specific category assignments for each split.

C.1 FEW-SHOT CLASSIFICATION BASELINES

The baseline experiments compare simple pixel- and CNN-feature-based classifiers, while Figure 6 examines performance as the number of unlabeled test-time items varies.

  • C.1 FEW-SHOT CLASSIFICATION BASELINES: Regular ProtoNet performs significantly better than the 1-nearest-neighbor and logistic-regression baselines on all three few-shot classification datasets.The baselines use either pixel inputs or CNN features.
  • C.1 FEW-SHOT CLASSIFICATION BASELINES: The baseline comparisons include 1-nearest neighbor and logistic regression with either pixel inputs or CNN-extracted features.The CNN features may come from randomly initialized or supervised-pretrained networks in the broader baseline table.
  • C.1 FEW-SHOT CLASSIFICATION BASELINES: Figure 6 reports test accuracy for different numbers of unlabeled items during test time.The figure is paired with a mask-value analysis for the Masked Soft k-Means model on Omniglot.

D HYPERPARAMETER DETAILS

This section provides training hyperparameters for the evaluated datasets and captions figures covering tieredImageNet categories, model performance, and Omniglot mask values.

  • Omniglot training used a learning rate of 1e-3, halved every 2K updates from update 2K, for 20K updates.These settings were adopted from the best ordinary Prototypical Network settings in Snell et al. (2017).
  • Figure 5 depicts the tieredImagenet category hierarchy, with training categories in red and test categories in blue.Each category indicates the number of associated classes.
  • Figure 6 reports tieredImageNet model performance for different numbers of unlabeled items available at test time.The chart includes test accuracy numbers.
  • Figure 7 shows mask values predicted by masked soft k-means on Omniglot.
Loading 1803.00676v1…