PaperlayerView exact revision on arXiv

Source-linked AI summary

Matching Networks for One Shot Learning

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra

arXiv:1606.04080v2cs.LGstat.ML

TL;DR

Learning new concepts from a single labelled example remains difficult for standard deep learning. Matching Networks combine attention and memory with matching train/test conditions to predict labels from a small support set without fine-tuning, improving ImageNet one-shot performance by almost 6% over Inception.

  • Problem

    Standard deep learning systems require hundreds or thousands of examples, motivating one-shot learning from a single labelled example.

  • Method

    Matching Networks use attention, memory, and episodic meta-learning to map a small labelled support set and an example to its label without fine-tuning.

  • Results

    Matching Networks improve ImageNet one-shot performance over Inception by almost 6% on the randImageNet experiment.

  • Takeaways & Limitations

    Training explicitly for one-shot learning and using non-parametric structures supports state-of-the-art performance across varied one-shot classification tasks.

  • Takeaways & Limitations

    Performance can fail when novel-label distributions diverge substantially from the training distribution.

Abstract

from arXiv · show

Learning from a few examples remains a key challenge in machine learning. Despite recent advances in important domains such as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data. In this work, we employ ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories. Our framework learns a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. We then define one-shot learning problems on vision (using Omniglot, ImageNet) and language tasks. Our algorithm improves one-shot accuracy on ImageNet from 87.6% to 93.2% and from 88.0% to 93.8% on Omniglot compared to competing approaches. We also demonstrate the usefulness of the same model on language modeling by introducing a one-shot task on the Penn Treebank.

1 Introduction

The paper introduces one-shot learning as learning a class from a single labelled example, and proposes Matching Networks with a matched train–test procedure for rapid adaptation. It also defines benchmark tasks for ImageNet and small-scale language modeling.

  • Motivation: One-shot learning aims to learn a class from a single labelled example, addressing deep learning’s need for hundreds or thousands of examples.Data augmentation and regularization alleviate overfitting in low-data regimes but do not solve the broader dependence on large datasets and many weight updates.
  • Model: Matching Nets combine attention and memory mechanisms to enable rapid learning of novel examples.The model seeks to combine parametric and non-parametric strengths, including rapid acquisition of new concepts.
  • Training procedure: The training procedure matches test conditions by showing only a few examples per class and switching tasks between minibatches.This procedure is designed to train the network for rapid learning from new tasks presented with few examples.
  • Benchmarks: The paper defines benchmark tasks for one-shot learning on ImageNet and small-scale language modeling.These tasks are intended to support evaluation of other approaches to the challenge.

2 Model

Matching Networks cast one-shot learning as a set-to-set mapping from a small labelled support set and test example to a predictive distribution, using attention over learned embeddings. They train this classifier episodically to match one-shot inference, enabling prediction on unseen classes without fine-tuning.

  • Model formulation: Matching Networks map a support set S and test example x̂ to P(ŷ|x̂, S), predicting the label with arg max_y P(y|x̂, S).The support set contains k image-label pairs, and the same parameterized network is reused for new support sets.
  • Model formulation: The classifier computes a label combination weighted by an attention mechanism, reducing to kernel density estimation or k−b-nearest neighbours under specific attention choices.These interpretations follow when attention is a kernel or assigns nonzero weight only to the nearest support examples.
  • Attention mechanism: Attention uses a softmax over cosine similarity between neural embeddings f(x̂) and g(x_i), with deep convolutional networks for images and word embeddings for language.The embedding functions may share parameters, with f = g as a possible configuration.
  • Full context embeddings: The model conditions embeddings on the entire support set, using a bidirectional LSTM for g(x_i, S) and an attention-based LSTM for f(x̂, S).The test embedding can attend over g(S) for a fixed number K of unrolling steps, potentially ignoring support elements and adding depth to attention computation.
  • Training: Training samples episodic tasks, support sets, and batches from label distributions, minimizing batch-label prediction error conditioned on each support set.This meta-learning procedure matches inference and allows the model to classify novel classes without fine-tuning, although performance degrades when the novel-label distribution diverges substantially from training.

3 Related Work

Related work connects Matching Networks to expressive attention-based neural architectures, metric learning, and prior one-shot or zero-shot learning research. The paper distinguishes its whole-support-set loss and introduces an ImageNet one-shot benchmark to address limited prior literature.

  • Expressive neural architectures: LSTMs, content-based attention, and Neural Turing Machines exemplify models extending beyond static classification of fixed vectors.These advances supported more expressive architectures across speech, translation, and program-learning tasks.
  • Metric learning: Matching Networks relate to content-based attention, kernel nearest-neighbor methods, and metric learning, especially Neighborhood Component Analysis.The cited NCA work includes a nonlinear follow-up.
  • Metric learning: The paper’s loss uses the whole support set S rather than pair-wise comparisons, making it more amenable to one-shot learning.Prior related approaches include deep convolutional siamese networks and set-based losses with less powerful metrics.
  • One-shot and zero-shot learning: Prior one-shot work inspired the paper and provided the Omniglot dataset, while zero-shot work used ImageNet and one-shot ImageNet literature remained limited.The authors address this gap through ImageNet benchmark and task definitions in the following section.

4 Experiments

Experiments evaluate Matching Networks on N-way k-shot classification with held-out classes across Omniglot, ImageNet, and a new Penn Treebank language task. The model outperforms baselines on the principal vision benchmarks, while language accuracy improves with more support examples but remains below the LSTM oracle.

  • Experimental setup: Experiments use N-way k-shot tasks, classifying disjoint unlabeled examples from N previously unseen classes given k labeled examples per class.Random performance is 1/N, and one-shot training and testing use disjoint held-out label sets.
  • Experimental setup: The evaluation spans Omniglot, ImageNet, and Penn Treebank, covering image classification and language modeling across varied data complexity, scale, and modality.Vision baselines include raw-pixel matching, classifier features, MANN, and Convolutional Siamese Nets.
  • Omniglot: Matching Networks outperform baselines on Omniglot across 1-shot and 5-shot, 5-way and 20-way settings, with more examples helping and 5-way tasks easier than 20-way tasks.Fine-tuning modestly improves the Baseline Classifier, while cosine distance can avoid fine-tuning on the small support set.
  • Omniglot: 72% accuracy is achieved by Matching Networks on disjoint 10-way MNIST after Omniglot training, compared with 63% for the Baseline Classifier and 70% for Siamese Nets.This tests transfer to a completely disjoint classification task.
  • ImageNet: Around two percentage points is the typical performance improvement from Full Contextual Embeddings on miniImageNet, with Matching Networks outperforming baselines with and without fine-tuning.Full Contextual Embeddings made no difference on Omniglot but could be evaluated sensibly on the harder miniImageNet task.

5 Conclusion

The paper introduces Matching Networks, a neural architecture and training regime designed for one-shot learning that achieves state-of-the-art performance across several one-shot classification tasks. It highlights task-specific training and non-parametric neural structures as key insights for remembering and adapting to new training sets.

  • 5 Conclusion: Matching Networks is a neural architecture whose corresponding training regime achieves state-of-the-art performance on varied one-shot classification tasks.The paper presents the architecture as its central contribution.
  • 5 Conclusion: One-shot learning is easier when the network is trained specifically to perform one-shot learning.This is identified as the first key insight of the work.
  • 5 Conclusion: Non-parametric structures help neural networks remember and adapt to new training sets within the same tasks.This is identified as the second key insight of the work.

Appendix · A Model Description

This appendix fully specifies the models that condition embedding functions f and g on the entire support set S, providing precise details omitted from the main text because similar mechanisms were previously described.

  • A Model Description: The appendix fully specifies the models used in the framework.These models condition the embedding functions f and g on the whole support set S.
  • A Model Description: The specified models condition embedding functions f and g on the whole support set S.
  • A Model Description: Precise model details were placed in the appendix because similar mechanisms had already been fully described in previous work.

A.1 The Fully Conditional Embedding f

The fully conditional embedding combines a neural embedding of the query with iterative attention-based reads over the support-set embeddings. Its processing uses K steps of an LSTM read mechanism, with content-based attention normalized over support examples.

  • Final embedding: After K reads, attLSTM(f′(x̂), g(S), K) produces h_K as the fully conditional embedding.The LSTM follows the implementation defined in [23], with x as input, h as output, and c as the cell.
  • Embedding components: The embedding uses f′, a neural network such as VGG or Inception, together with g applied to each support-set element x_i.The support-set embeddings are denoted g(S).
  • Iterative processing: The model performs K processing steps following the “Process” block described in prior work.The state after k processing steps is defined recursively in the section’s equations.
  • Attention and read-out: At each step, content-based attention reads from g(S), and the read-out is concatenated to the preceding LSTM state.The attention softmax normalizes with respect to g(x_i).

A.2 The Fully Conditional Embedding g

The fully conditional embedding g(x_i, S) encodes each support-set element with a bidirectional LSTM built around a neural network g′(x_i), while using a skip connection between inputs and outputs.

  • The Fully Conditional Embedding g: g(x_i, S) uses a bidirectional LSTM to encode support-set elements, with g′(x_i) provided by a neural network such as VGG or Inception.The embedding combines the bidirectional LSTM states with g′(x_i).
  • The Fully Conditional Embedding g: The recurrent computation follows the stated LSTM implementation, starts the backward recursion at i = |S|, and includes a skip connection between inputs and outputs.The LSTM uses x as input, h as the output after the output gate, and c as the cell.

B miniImageNet Description

miniImageNet comprises 100 randomly selected ImageNet classes, with 80 used for training and 20 held out for testing; each class contains 600 images.

  • Dataset construction: miniImageNet was constructed from 100 random ImageNet classes.The exact class identifiers are provided in the paper’s accompanying text file.
  • Train-test split: 80 classes were used for training, while the last 20 were reserved for testing and never seen during training.This split was used in the paper’s one-shot experiments.
  • Dataset contents: Each of the 100 miniImageNet classes contains exactly 600 images.The paper directs readers to a text file listing the exact images comprising each class.

C ImageNet Class Splits

The full ImageNet experiments use two defined class splits, with those classes excluded from training during the one-shot experiments described in Section 4.1.2.

  • C ImageNet Class Splits: Two class splits are defined for the full ImageNet experiments.The passages enumerate the class identifiers in the respective splits.
  • C ImageNet Class Splits: The split classes are excluded from training during the one-shot ImageNet experiments in Section 4.1.2.This exclusion establishes the class separation used for those experiments.

D PTB Class Splits

This section defines the two class splits used in the Penn Treebank experiments. These classes were excluded from training during the one-shot language experiments in section 4.1.3.

  • D PTB Class Splits: Two PTB class splits are defined for the experiments.The splits identify the classes withheld from training in the one-shot language setup.
  • D PTB Class Splits: The listed split vocabulary includes tokens such as “abortion,” “accelerating,” “acceptable,” and “airport.”These entries are presented as members of the class-split vocabulary.
  • D PTB Class Splits: The vocabulary also includes tokens such as “demonstrated,” “department,” “devaluation,” “dividends,” and “earthquakes.”The passage continues the enumeration of classes used in the PTB split definitions.
Loading 1606.04080v2…