Source-linked AI summary
Meta Networks
Tsendsuren Munkhdalai, Hong Yu
TL;DR
Neural networks face difficulty generalizing rapidly from small training sets while retaining previously learned concepts. MetaNet addresses this with meta-level knowledge, fast parameterization, and meta-learner-generated parameters, achieving strong one-shot benchmark results and supporting continual learning, though integrating multiple weight time-scales remains difficult.
Problem
Neural networks struggle to rapidly generalize to new concepts from small data while preserving previously learned patterns during continual learning.
Method
MetaNet learns task-agnostic meta knowledge and uses loss-gradient meta information to generate fast parameters for base and meta learners.
Results
MetaNet improves published one-shot results by 0.5% to 2% on Omniglot and by up to 6% accuracy on 5-way one-shot Mini-ImageNet evaluation.
Takeaways & Limitations
MetaNet supports rapid generalization and exhibits properties related to continual learning across one-shot supervised-learning settings.
Takeaways & Limitations
Integrating slow, task-level, and example-specific weights becomes difficult as neural networks contain many parameter types operating at multiple time-scales.
Abstract
from arXiv · showhide
Neural networks have been successfully applied in applications with a large amount of labeled data. However, the task of rapid generalization on new concepts with small training data while preserving performances on previously learned ones still presents a significant challenge to neural network models. In this work, we introduce a novel meta learning method, Meta Networks (MetaNet), that learns a meta-level knowledge across tasks and shifts its inductive biases via fast parameterization for rapid generalization. When evaluated on Omniglot and Mini-ImageNet benchmarks, our MetaNet models achieve a near human-level performance and outperform the baseline approaches by up to 6% accuracy. We demonstrate several appealing properties of MetaNet relating to generalization and continual learning.
1. Introduction
Meta Networks addresses rapid generalization from few examples and continual learning by combining base and meta learners with external memory and fast parameterization. Its evaluations report improved one-shot supervised-learning performance and properties related to generalization and continual learning.
- Standard neural networks typically require large labeled datasets and struggle to learn new concepts incrementally without forgetting previously learned patterns.
- MetaNet uses separate base and meta learners operating in task space and task-agnostic meta space, supported by external memory.The base learner analyzes the input task and provides higher-order meta information to the meta learner.
- The meta learner rapidly parameterizes itself and the base learner using slow, task-level fast, and example-level fast weights.External memory supports rapid learning and generalization.
- MetaNet uses representation and main task losses with distinct objectives for representation learning and the input task.
- The method was extensively evaluated on one-shot supervised-learning problems and improved state-of-the-art benchmark results while exhibiting generalization and continual-learning properties.
2. Related Work
Related work spans one-shot learning, meta optimizers, fast-weight generation, and memory-augmented neural networks. MetaNet is positioned as a meta-space approach that generates fast weights at two time-scales for one-shot generalization.
- One-shot learning addresses sequences of changing multi-class classification tasks with only one or a few labeled examples per class.Prior approaches include generative models, metric learning, nearest-neighbor methods, and memory-based systems.
- MetaNet relates to prior one-shot methods by modeling neural architectures for rapid learning and generalization across tasks.
- Meta-optimizer research trains optimization algorithms within meta learning, often focusing on tasks with large datasets, whereas MetaNet targets one-shot generalization without such data.
- Earlier fast-weight approaches generated weights for single-layer controllers, convolutional filters, or recurrent networks; MetaNet generates fast weights at two time-scales in meta space.
- MetaNet’s external memory also places it within the broader family of memory-augmented neural networks.
3. Meta Networks
MetaNet combines base and meta learners with external memory to rapidly adapt neural networks through task- and example-specific fast parameterization. Its design uses loss gradients as meta-information and integrates fast and slow weights through augmented layers.
- Meta Networks: MetaNet uses separate base and meta learners operating in task and meta spaces, respectively, with external memory supporting rapid learning.The meta learner generates fast parameters while the base learner performs the task objective.
- Meta Learner: The dynamic representation function u uses slow weights Q and task-level fast weights Q* to construct task-dependent input embeddings.The generated task-specific parameters are integrated with Q through the layer augmentation method.
- Fast Parameterization: Fast weights are generated at multiple time scales, including task-level and example-level parameterization, while slow weights are updated during training.The meta learner produces example-level weights for the base learner and task-level weights for representation learning.
- Meta Learner: Loss gradients from support examples serve as meta-information for generating fast parameters.The gradients are derived from the base learner’s task loss and used by the meta learner to generate W* parameters.
- Layer Augmentation: The layer augmentation method combines activations produced by slow and fast weights through element-wise addition.Using both weight types was essential for convergence in preliminary experiments; a fast-weight-only base learner failed to converge.
4. Results
The paper evaluates MetaNet with one-shot classification experiments across Omniglot, Mini-ImageNet, MNIST, and multiple Omniglot splits. The experiments include benchmark comparisons and ablations of fast parameterization.
- Datasets: One-shot classification experiments use Omniglot, Mini-ImageNet, and MNIST datasets.The Omniglot dataset contains 1,623 classes with 20 images per class and uses standard training and evaluation alphabet splits.
4.1. One-shot Learning Test
MetaNet was evaluated across one-shot learning benchmarks, where it improved prior results and maintained a relatively small performance decline as task difficulty increased. It also achieved the best reported Mini-ImageNet result and strong performance on the Omniglot standard split.
- Benchmark settings: MetaNet was evaluated in four benchmark settings spanning Omniglot, Mini-ImageNet, MNIST, and the Omniglot standard split.The experiments included previous and standard Omniglot splits, Mini-ImageNet, and MNIST as out-of-domain data.
- Omniglot previous split: MetaNet improved previous best results by 0.5% to 2% accuracy on 5-way through 20-way Omniglot one-shot tasks.As the number of classes increased, MetaNet’s performance drop was around 2%, compared with 3% to 15% for other models.
- Omniglot previous split: MetaNet’s performance drop was around 2% from 5-way to 20-way classification, versus 3% to 15% for other models, yielding a 2% absolute improvement on 20-way tasks.The comparison reports the relative robustness of MetaNet as task difficulty increases.
- Ablation: MetaNet’s dynamic representation learning function contributed to performance over MetaNet-, while additional task-level base-learner weights in MetaNet+ had a negative effect.The gap between MetaNet and MetaNet- increased in the out-of-domain setting.
- Mini-ImageNet: MetaNet improved the previous Mini-ImageNet one-shot result by up to 6% accuracy and obtained the best result.The evaluation used 5-way one-shot classification and reported results on the Mini-ImageNet test set.
- Omniglot standard split: On the Omniglot standard split, MetaNet slightly outperformed human performance but underperformed probabilistic programming, with a small gap among the three top results.The standard split was more difficult than the alternate split because it had fewer training classes and larger test classes.
4.2. Generalization Test
The generalization experiments tested whether MetaNet could transfer across task difficulties, base learners, and sequential problems. Results showed cross-task flexibility, effective parameterization of fixed networks, and reverse transfer with limited later forgetting.
- N-way training and K-way testing: MetaNet trained on N-way one-shot tasks was evaluated on K-way tasks with N ≠ K to test cross-task generalization without second-task training.The evaluation inserted and fixed a new softmax layer whose weights were augmented by the meta learner.
- N-way training and K-way testing: 93.07% 20-way test accuracy was obtained by MetaNet trained on 5-way tasks, exceeding Siamese Net trained on 20-way tasks and closely matching Matching Network.Models trained on harder tasks generally improved on easier test tasks; a 20-way-trained model improved the 5-way baseline by 0.6%.
- N-way training and K-way testing: MetaNet trained on 10-way tasks achieved around 65% accuracy on 100-way one-shot classification.This was reported as a preliminary experiment on more extreme test-time classification.
- Rapid parameterization of fixed-weight base learner: Replacing the base learner with a new CNN during evaluation allowed the meta learner to generate fast weights for the fixed network.The fast weights augmented the new CNN’s fixed slow weights.
- Rapid parameterization of fixed-weight base learner: The target CNN optimized within MetaNet outperformed fixed-weight CNNs, although their test accuracies converged after more one-shot learning trials.The comparison used small and large CNN base learners on Omniglot 5-way classification.
- Meta-level continual learning: MNIST training produced reverse transfer to Omniglot until 2400 trials, after which meta weights began forgetting Omniglot information and accuracy dropped from 2800 trials onward.The Omniglot performance change ranged from -1.7% to 1.24%; even after 7600 MNIST trials, the drop was only 1.7%.
5. Discussion and Future Work
MetaNet addresses rapid generalization and continual learning by acquiring generic knowledge in meta space and rapidly shifting network parameters and inductive biases. The discussion highlights flexible task adaptation while identifying integration of multiple weight timescales as a limitation.
- MetaNet acquires generic knowledge in a meta space and uses fast parameterization to shift underlying networks’ parameters and inductive biases for rapid generalization.
- The framework supports one-shot generalization from a single example and shows properties related to generalization and continual learning.
- Loss gradients serve as meta information, supporting state-of-the-art results on several one-shot supervised-learning benchmarks and flexible model behavior.
- MetaNet can alternate between softmax layers during testing and supports continual learning up to a certain point.
- Integrating slow, task-level, and example-specific weights through layer augmentation becomes difficult and can reduce performance compared with simpler configurations.
- Future applications include one-shot reinforcement or imitation learning and recurrent MetaNet models for sequence modeling and language understanding.
A. Training Details
MetaNet is trained and evaluated through episodic N-way one-shot tasks formed from disjoint training and test classes. The implementation uses convolutional base and representation learners, with fast-weight augmentation and separate meta-learning networks.
- Training and testing use disjoint training and test classes, with tasks containing an N-class support set and one image per class.
- Each task also includes L labeled examples used to update the model during the task.
- Omniglot experiments use a 64-filter, five-layer convolutional base learner and a matching dynamic representation network.
- Mini-ImageNet uses a similar CNN architecture with 32 filters, while the final three layers are augmented with fast weights for efficiency and flexibility.
- The auxiliary networks use a 20-unit single-layer LSTM and a three-layer 20-unit ReLU MLP, with parameters shared across gradient coordinates.
B. MNIST as Out-Of-Domain Data
The out-of-domain experiment trains on Omniglot and evaluates on MNIST in a 10-way one-shot setting. MetaNet achieves 74.8% accuracy, while the no-dynamic variant performs worse and MetaNet+ does not improve over the standard model.
- The experiment treats MNIST as a separate domain, training on Omniglot and testing on MNIST in 10-way one-shot classification.
- 74.8% accuracy was achieved by the standard MetaNet model on the MNIST out-of-domain task.
- 71.6% accuracy was achieved by MetaNet-, which was 0.6% and 3.2% lower than the other fast-weight variants.
- 72.3% accuracy was achieved by MetaNet+, and the experiment did not show improvement from this variant.
- Matching Net reported 72.0% accuracy in the same setup, while Neural Statistician was reported as the best result at that time.