Source-linked AI summary
One-Shot Generalization in Deep Generative Models
Danilo Jimenez Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, Daan Wierstra
TL;DR
The paper addresses one-shot generalization: understanding a new concept from a single example and generating compelling alternatives. It develops sequential generative models built on feedback and attention, combining deep representations with Bayesian reasoning. Across three tasks, the models generate compelling and diverse samples after seeing new examples once, while remaining distinct from one-shot learning because their parameters are not updated at test time.
Problem
Humans can understand new concepts and generate compelling variations from one or a few examples, but this one-shot generalization capacity challenges neural models.
Method
The paper develops sequential generative models that combine deep neural representations with approximate Bayesian reasoning through feedback and attention.
Results
Across unconditional sampling and two conditional exemplar-generation tasks, the models generate compelling and diverse samples after seeing new examples just once.
Takeaways & Limitations
Sequential generative models provide general-purpose models for one-shot machine learning and state-of-the-art deep density estimation and image generation.
Takeaways & Limitations
The models perform one-shot inference rather than one-shot learning: their parameters are not updated after each new input and they require reasonable amounts of data.
Abstract
from arXiv · showhide
Humans have an impressive ability to reason about new concepts and experiences from just a single example. In particular, humans have an ability for one-shot generalization: an ability to encounter a new concept, understand its structure, and then be able to generate compelling alternative variations of the concept. We develop machine learning systems with this important capacity by developing new deep generative models, models that combine the representational power of deep learning with the inferential power of Bayesian reasoning. We develop a class of sequential generative models that are built on the principles of feedback and attention. These two characteristics lead to generative models that are among the state-of-the art in density estimation and image generation. We demonstrate the one-shot generalization ability of our models using three tasks: unconditional sampling, generating new exemplars of a given concept, and generating new exemplars of a family of concepts. In all cases our models are able to generate compelling and diverse samples---having seen new examples just once---providing an important class of general-purpose models for one-shot machine learning.
1. Introduction
The paper targets one-shot generalization: generating compelling variations of concepts after seeing only one or a few examples. It develops sequential deep generative models that combine deep representations with approximate Bayesian reasoning, feedback, and attention.
- One-shot generalization lets a system infer the structure of a new concept from one or a few examples and generate alternative variations.
- The paper combines deep neural representations embedded in hierarchical latent-variable models with approximate Bayesian reasoning to address one-shot generalization.
- Deep generative models specify observed data through hierarchies of latent variables and use deep networks to learn rich nonlinear conditional distributions.
- Sequential generative models use feedback and attention to integrate analysis with constructed interpretations during inference and generation.Sequential latent-variable inference provides internal feedback or “thinking time” for extracting information from each data point.
- The paper demonstrates one-shot generalization through unconditional sampling, generating new exemplars of a concept, and generating exemplars from a family of concepts.
- The models generate compelling and diverse alternative image variations after seeing new examples just once.
2. Varieties of Attention
Attention selectively routes information within generative models and supports invariances, while spatially transformed attention handles object shape, size, position, scale, and rotation.
- Attention selectively routes information between parts of a model and can incorporate invariances with few additional parameters and low computational cost.
- Inference uses reading attention to transform an image into a canonical representation, while generation uses writing attention to selectively update output variables.
- Spatially-transformed attention: Spatially transformed attention provides invariance to object shape and size through general affine transformations rather than selecting fixed image patches.
- Spatially-transformed attention: Spatial transformers let generative models independently handle the position, scale, rotation, and content of generated image parts.
3. Iterative and Attentive Generative Models
The paper develops sequential deep generative models that combine latent-variable modeling with iterative feedback and attention. These models generate and infer through successive latent-variable groups, hidden-canvas updates, and attentive reading and writing, with a nonlinear multi-modal joint posterior.
- Latent-variable inference: Variational inference optimizes a free-energy objective that trades off reconstruction ability against posterior complexity.The inference model amortizes posterior inference using a deep neural network parameterized by shared variational parameters.
- Sequential generative models: Sequential generative models extend single-step latent-variable models by generating T groups of latent variables rather than all K variables at once.The model describes observations over T time steps, with latent variables generated sequentially.
- Generative process: Each generative step samples z_t, updates a hidden state and canvas, and transforms the final canvas into an observation distribution.The canvas can be updated additively or with a convolutional gated recurrent unit, while the output function maps the final canvas to likelihood parameters.
- Dependent posterior inference: Although each conditional posterior is Gaussian, nonlinear sequential dependencies make the joint posterior non-Gaussian and multi-modal, enabling more accurate inference.The multi-modal structure arises from the nonlinearities used across the sequential inference process.
- Model properties and complexity: The framework generalizes DRAW, composited VAEs, and AIR through a hidden canvas, simplified feedback connections, shared inference parameters, and spatial-transformer attention.The hidden canvas permits pre-image construction before a final corrective transformation, while omitting canvas-to-hidden feedback reduces model parameters without affecting reported performance.
4. Image Generation and Analysis
The models achieve competitive image-generation performance across MNIST, Multi-MNIST, Omniglot, and Multi-PIE, while sequential inference provides a way to allocate latent variables over time.
- The spatially transformed attention model with a CGRU hidden canvas provides the best performance among existing models on binarized MNIST.
- On 64 × 64 Multi-MNIST, additive canvas with spatially transformed attention provides a reliable way to learn the harder two-digit images.The dataset places two MNIST digits at random locations and converges more slowly than MNIST.
- Longer sequences can improve performance because each step contributes a KL-divergence term to the objective.
- KL-divergence decays toward the end of the sequence, indicating diminishing contributions from later latent variables.The sequential structure supports deciding how many latent variables to use and when to terminate computation.
- On Omniglot, sequential models outperform competing approaches, while spatially transformed attention and an additive hidden canvas perform best among the tested sequential models.The data regime includes 105 × 105 binary images across 1628 classes with only 20 images per class.
- The Multi-PIE model generates compelling grayscale faces across orientations and genders that are representative of the data.It was trained on 15 viewpoints but only 3 of 19 illumination conditions.
5. One-Shot Generalization
The paper evaluates one-shot generalization through unconditional generation, exemplar variation, and novel-alphabet generation. These tasks test progressively stronger forms of generating samples from limited context.
- The evaluation uses unconditional generation, novel variations of a given exemplar, and representative samples from a novel alphabet.These tasks test weaker to stronger forms of one-shot generalization.
1. Unconditional Generation.
Unconditional generation produces samples that reflect the characteristics of the Omniglot data, including varied styles and representative stroke patterns.
- Generated Omniglot samples show rounded patterns, line segments, and thick and thin strokes representative of the dataset.The reported likelihoods place the model among the state-of-the-art.
2. Novel variations of a given exemplar.
The model generates novel variations of unseen characters under weak and strong one-shot generalization tests. It produces visually similar variations even when test alphabets and characters are entirely absent from training, though low-data splits increase overfitting.
- Task setup: At test time, a conditional generative model receives one unseen character image and generates novel exemplars of that character.The context image x′ is used to condition generation through an attentional mechanism.
- Weak generalization: The weak test withholds three character types from each alphabet, while related characters from those alphabets remain in training.The test set contains 3,000 characters, requiring transfer within familiar alphabets.
- Strong generalization: The strong test uses 30 alphabets for training and 20 entirely unseen alphabets for testing, making generalization harder and overfitting easier.The model has seen neither the test characters nor alphabets from their families; alternative 40-10 and 45-5 splits vary difficulty.
- Results: The model generates visually similar and reasonable variations in both weak and strong tests, unlike a comparator requiring human stroke information.The approach uses only the spatial arrangement of image data as domain-specific information.
- Data regime: The 45-5 split shows a small train-test likelihood gap, whereas the other splits show a large gap and greater overfitting in low-data regimes.The figures illustrate generations for different train-test splits and unconditional samples for 52 × 52 Omniglot.
3. Representative samples from a novel alphabet.
The model is conditioned on a single sample from a novel alphabet and generates new characters consistent with that alphabet. The generated samples capture common alphabet features despite the task's subjectivity.
- Task setup: The model conditions on one to ten samples from a novel alphabet and generates new characters consistent with that alphabet.The reported experiment uses the hardest setting: one context image.
- Results: With one context image, the model generations show that it can pick up common features of the novel alphabet and use them in new characters.The task is described as highly subjective.
- Scope: The paper distinguishes one-shot generalization from one-shot learning: the model performs inference on new points without updating its parameters.One-shot learning would require a parameter update after each new input, which this model does not perform.
6. Conclusion
The paper concludes that sequential generative models using feedback and attention support one-shot generalization, density estimation, and image generation. Spatial attention improves generalization, but reasonable data remains necessary to avoid overfitting.
- Conclusion: Sequential generative models use feedback and attention to compute data probabilities through a multi-step process over groups of latent variables.The models are presented as extensions of variational auto-encoders.
- Conclusion: Spatial attention substantially improves the model’s ability to generalize, and the spatial transformer becomes the default attention mechanism for generative models.The spatial transformer supports both reading and writing.
- Conclusion: Across several tasks, the model generates compelling and diverse samples after seeing new examples only once.The conclusion presents this as evidence of one-shot generalization over a range of tasks.
- Limitations: The approach still needs a reasonable amount of data to avoid overfitting, which the authors identify as a limitation for future work.The conclusion explicitly preserves this scope boundary.
A.1. SVHN
The supplied material identifies SVHN as a 32 × 32 RGB house-number dataset and describes several attention mechanisms used in the models. These mechanisms selectively process image regions and can support sequential read or write operations.
- Dataset: SVHN consists of 32 × 32 RGB images depicting house numbers.
- Attention mechanisms: Randomized attention selects input patches without learning the selection policy, and dropout at the input layer effectively implements this mechanism.On MNIST, competitive learning requires attending to a large number of patches.
- Attention mechanisms: Error-based attention selects locations from reconstruction error, allowing each step to target regions that reduce that error.The mechanism can jump to relevant image regions and addresses limited gradient information in large, sparse images.
- Attention mechanisms: Multiple spatial attention additively combines several spatial transformers so the model can read or write at multiple locations simultaneously.The spatial transformers are conditioned on a shared context v.
- Recurrent component: The CGRU updates a hidden canvas using element-wise products, stride-1 convolutions, and sigmoid gates.The supplied description specifies 3 × 3 kernels for the relevant matrices.