One-shot Learning with Memory-Augmented Neural Networks

Adam Santoro; Sergey Bartunov; Matthew Botvinick; Daan Wierstra; Timothy Lillicrap

Source-linked AI summary

One-shot Learning with Memory-Augmented Neural Networks

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap

arXiv:1605.06065v1May 19, 2016cs.LG

TL;DR

One-shot learning challenges gradient-based networks because they must adapt from sparse data without inefficient relearning or catastrophic interference. The paper trains memory-augmented neural networks through meta-learning and introduces the content-based LRUA access module. On never-before-seen Omniglot classes, accuracy reached 82.8% on the second presentation and 98.1% by the tenth.

Problem
One-shot learning requires rapid inference from small data, while sequential gradient-based relearning can cause poor learning or catastrophic interference.
Method
The paper uses meta-trained memory-augmented neural networks that combine gradual representation learning with external-memory binding and content-based LRUA access.
Results
82.8% accuracy occurred on the second presentation of never-before-seen Omniglot classes, rising to 98.1% by the tenth.
Takeaways & Limitations
MANNs can rapidly assimilate task-specific information while retaining slowly learned knowledge across tasks.
Takeaways & Limitations
The tested tasks shared substantial high-level structure, so broader task ranges may reintroduce continual-learning challenges including catastrophic interference.

Abstract

from arXiv · show

Despite recent breakthroughs in the applications of deep neural networks, one setting that presents a persistent challenge is that of "one-shot learning." Traditional gradient-based networks require a lot of data to learn, often through extensive iterative training. When new data is encountered, the models must inefficiently relearn their parameters to adequately incorporate the new information without catastrophic interference. Architectures with augmented memory capacities, such as Neural Turing Machines (NTMs), offer the ability to quickly encode and retrieve new information, and hence can potentially obviate the downsides of conventional models. Here, we demonstrate the ability of a memory-augmented neural network to rapidly assimilate new data, and leverage this data to make accurate predictions after only a few samples. We also introduce a new method for accessing an external memory that focuses on memory content, unlike previous methods that additionally use memory location-based focusing mechanisms.

1. Introduction

One-shot learning requires rapid behavioral adaptation from very little data, whereas conventional gradient-based learning relies on extensive incremental training and can suffer poor learning or catastrophic interference. The paper proposes memory-augmented networks and meta-learning as a route to combining gradual task-level learning with rapid storage of new information.

One-shot learning requires abrupt behavioral shifts after single observations, unlike tasks evaluated after extensive training on large datasets.
Gradient-based relearning from sequentially presented examples can produce poor learning or catastrophic interference.
Meta-learning separates rapid within-task learning from slower knowledge accumulation across tasks.
Memory-capable networks can rapidly cache representations while weight updates shift their broader learning bias.
MANNs provide stable, element-wise addressable memory whose capacity need not be tied to the number of parameters, addressing limitations of standard recurrent architectures.
The paper combines gradient-based representation learning with external-memory binding and introduces content-focused access rather than location-based focusing.

2. Meta-Learning Task Methodology

The meta-learning setup trains a model across a distribution of episodes rather than optimizing for one dataset alone. Temporally offset labels force the model to retain samples, bind them to later-provided labels, and use those bindings for prediction.

Meta-learning minimizes expected learning cost across a distribution of datasets p(D), rather than learning cost on a single dataset D.
Each episode presents a dataset containing classification images with labels or regression inputs with values from a hidden function.
The network receives (x1, null), (x2, y1), …, (xT, yT−1) and must output yt for the current query xt.
Shuffled labels prevent weight-based sample-class bindings, requiring the model to hold samples until their labels arrive and store the resulting associations.
The task structure rewards a general scheme that binds representations to labels independently of their specific content.

3. Memory-Augmented Model

The model uses a differentiable controller with external memory for rapid reading and writing of representations. Its LRUA module selects memory locations by usage while retrieval remains content-based, favoring recent or rarely used information over sequence-oriented location addressing.

3.1. Neural Turing Machines: An NTM combines a controller with differentiable external memory and read/write heads that can encode or retrieve representations at each timestep.
3.1. Neural Turing Machines: The task presents inputs with temporally offset labels, encouraging external memory to bind sample representations to labels for later classification.
3.1. Neural Turing Machines: Given xt, the controller produces a key kt that is stored in or used to retrieve a row of memory Mt through cosine-similarity addressing.
3.1. Neural Turing Machines: The retrieved memory rt feeds the classifier and the next controller state.
3.2. Least Recently Used Access: Unlike prior NTM access combining content and location, LRUA is designed for conjunctive information coding independent of sequence.
3.2. Least Recently Used Access: LRUA writes to either the least-used or most-recently-read location, preserving recent memories or updating recently encoded information.
3.2. Least Recently Used Access: Usage weights decay previous usage and incorporate current reads, after which least-used weights identify candidate write locations.
3.2. Least Recently Used Access: Writing can overwrite a rarely used slot or the previously used slot, erasing the least-used memory when the latter is selected.

4. Experimental Results

Experiments show that the MANN rapidly learned to classify unseen Omniglot classes, scale to more classes with curriculum training, and estimate unseen functions from sparse samples. Performance depended on memory management: persistent cross-episode memory caused interference, while content-based access supported strong one-shot learning.

Omniglot Classification: After 100,000 episodes, the MANN classified never-before-seen Omniglot classes with 82.8% accuracy on the second presentation, rising to 98.1% by the tenth.The test episodes required no further learning and used classes from a disjoint test set.
Omniglot Classification: The MANN surpassed human performance on every presentation in the one-hot-label classification task and guessed above chance on first instances.The above-chance first-instance performance was attributed to matching new samples against bindings stored in external memory.
Omniglot Classification: Five-character labels enabled episodes with up to fifteen unique classes, and the MANN retained similar performance at this larger scale.The labels provided 3125 possible combinations, reducing repeated class-label assignments across episodes.
Omniglot Classification: The MANN outperformed a nearest-neighbour baseline that had unlimited memory and access to richer autoencoder features.The baseline also trained on three times as much augmented data and still performed worse than the MANN.
Persistent Memory Interference: Without wiping memory between episodes, classification became less robust, although ten-class episodes of length 75 reached comparable accuracy under some conditions.Persistent information from episodes with unique classes and labels acted as interference for the current episode.
Regression: In regression, MANN predictions tracked Gaussian-process functions after 20 samples and generalized appreciably to two- and three-dimensional inputs.Log-likelihood predictions became more accurate as samples were stored in memory, despite the GP receiving the correct generating hyperparameters.

5. Discussion & Future Work

MANNs use gradual learning for cross-task knowledge and addressable memory for rapidly binding information from new tasks. The evaluated model outperformed an LSTM on sparse-data classification and regression, while broader task diversity and active learning remain open directions.

Discussion: MANNs combine gradual background-knowledge learning with flexible memory for information specific to newly encountered tasks.Their dedicated memory is addressable and structurally independent from process-control mechanisms.
Discussion: The evaluated MANN outperformed an LSTM on two meta-learning tasks involving classification and regression with sparse training data.
Discussion: The studied tasks require flexible storage and access plus inductive transfer from longer-term experience, rather than rote retrieval alone.Inductive transfer means drawing inferences from new data using longer-term experience.
Discussion: Informal human comparisons found superior MANN performance, including at set-sizes not expected to produce comparable human performance.
Future Work: Wider task distributions could reintroduce continual-learning challenges, including catastrophic interference; active-learning tasks are another proposed direction.The current tasks shared a high degree of high-level structure, and the memory-writing procedure was selected for these tasks rather than learned.

Supplementary Information

The supplementary model is an NTM variant with a controller, external memory, read/write heads, and output distribution. It uses content-based reading, LRUA writing, and task-specific output distributions for classification and regression.

Architecture: The MANN variant comprises a controller, read and write heads, external memory, and an output distribution.The controller receives input data and also provides input to the output distribution.
Controller: The best-performing controller is an LSTM with 200 hidden units receiving concatenated current input and previous label.
Memory access: Content-based reading uses the controller cell state as a query and cosine similarity against rows of the external memory.Softmax transforms the similarity measures into read weights, which retrieve a memory vector.
Memory access: Four memory reads were used for reported results, implemented by concatenating additional read vectors with the controller hidden state.Multiple reads were not combined through interpolation.
Memory access: LRUA writes to either the most recently read location or the least recently used location to preserve or update potentially useful memories.A learnable sigmoid gate forms a convex combination of previous read and least-used weights.
Output distributions: Classification uses categorical softmax outputs, while regression predicts Gaussian mean and variance parameters for target probabilities.String labels are predicted as five independently classified five-way chunks; regression uses predictive µ and σ.

7. Classification input data

Classification episodes use augmented Omniglot images represented as flattened pixel vectors, paired with episode-specific labels. Labels may be one-hot class vectors or five-character strings encoded through five-hot chunks.

Input construction: Episodes retain 10N Omniglot samples after selecting N unique classes, with N equal to 5, 10, or 15.Class representation counts need not be equal, and some selected classes may lack representative samples.
Image augmentation: Images are randomly rotated, translated, downscaled to 20x20, and then given a randomized class-dependent rotation.The class-dependent rotation is randomized independently across episodes.
Label encoding: Class labels are randomly selected across episodes; one-hot labels have size N, while string labels use five-element chunks.The string-label representation is shown as a five-hot vector.

8. Task

Episodes contain sampled classification or regression sequences with lengths tied to the number of unique classes. Training runs for 100,000 episodes before evaluation on disjoint test classes without further weight updates.

Task setup: Episodes use 5, 10, or 15 unique classes and typically have lengths of 50, 100, or 150 respectively.Episode length is ten times the maximum number of unique classes unless explicitly changed.
Task setup: The model trains for 100,000 episodes, then enters a test phase using a disjoint Omniglot class set with weight updates stopped.The test classes are drawn from Omniglot classes 1201–1623.
Task setup: Curriculum training increases the maximum number of unique classes by one every 10,000 training episodes.Each increase also raises episode length to ten times the new maximum.

9. Parameters

The reported setup used RMSprop and selected memory, controller, optimization, and read parameters, alongside human and kNN comparison-task details.

RMSprop used a 1e−4 learning rate, 5e−1 maximum learning rate, 0.95 decay, and 0.9 momentum.
The grid search selected 128 memory slots, memory size 40, a 200-unit LSTM controller, learning rate 1e−4, and four memory reads.
Human comparison task: Human participants observed sequences of 50 images with five unique classes and entered class identities as digits without a response time limit.
kNN comparison: With no data, kNN randomly predicts one class, giving first-point accuracy probability 1/N; the stated values are 4% and 0.4% for five and 15 classes per episode.