Source-linked AI summary
DRAW: A Recurrent Neural Network For Image Generation
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra
TL;DR
Image generators often construct whole scenes in one shot, limiting iterative refinement and making large images difficult to scale. DRAW combines recurrent variational encoding with differentiable two-dimensional attention to build images through successive modifications, improving binarized-MNIST results and producing highly realistic house-number images. Its attention mechanism also benefits image classification.
Problem
One-shot image generation conditions all pixels on a single latent distribution, precluding iterative self-correction and making large images difficult to scale.
Method
DRAW combines recurrent encoder–decoder variational auto-encoding with fully differentiable two-dimensional attention for selective reading and writing during sequential image construction.
Results
DRAW improves the best known results for binarized MNIST generation, generates highly realistic house-number images, and its attention mechanism benefits image classification.
Takeaways & Limitations
Sequential construction and differentiable attention provide a generative architecture that produces novel images, with outputs virtually indistinguishable from real MNIST and SVHN data.
Takeaways & Limitations
Without selective attention, the architecture cannot focus on part of the input or modify only part of the canvas, a limitation identified as important for large-scale image generation.
Abstract
from arXiv · showhide
This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.
1. Introduction
DRAW replaces one-shot image generation with sequential construction by recurrent encoder and decoder networks. Its differentiable spatial attention lets the model selectively read and write image regions while retaining end-to-end gradient training.
- Most image-generation approaches condition all pixels on one latent distribution, making iterative self-correction unavailable and scaling to large images difficult.
- DRAW uses recurrent encoder and decoder networks to compress training images and reconstruct them from latent codes.The system is trained end-to-end with stochastic gradient descent using a variational upper bound on data log-likelihood.
- Instead of generating images in one pass, DRAW iteratively constructs scenes through decoder modifications that the encoder observes.
- DRAW’s fully differentiable attention mechanism selects image regions through partial glimpses while remaining trainable with standard backpropagation.The mechanism resembles selective read and write operations developed for Neural Turing Machines.
- The paper evaluates DRAW on MNIST, Street View House Numbers, and CIFAR-10, alongside a preliminary cluttered-MNIST classification experiment.
2. The DRAW Network
DRAW extends variational auto-encoders with recurrent encoder–decoder interaction, sequential latent sampling, and cumulative canvas updates. Training combines reconstruction and latent costs, while generation repeatedly samples from the prior and updates the canvas.
- 2.1. Network Architecture: DRAW uses recurrent encoder and decoder networks that exchange a sequence of latent samples, with the encoder receiving the decoder’s previous outputs.This lets the encoder condition later representations on the generation history.
- 2.1. Network Architecture: At each time-step, the decoder modifies a cumulative canvas matrix, whose final state parameterizes the image distribution.The number of time-steps T is specified in advance.
- 2.1. Network Architecture: The encoder produces a distribution Q(Z_t|henc_t) over each latent vector, implemented in the experiments as a diagonal Gaussian.Gaussian latents support unbiased, low-variance stochastic gradients through the reparameterization trick.
- 2.2. Loss Function: The total loss is the expected sum of reconstruction loss Lx and latent loss Lz, optimized with stochastic gradient descent.A single latent sample is used for each stochastic-gradient step.
- 2.2. Loss Function: Lz measures the description cost of transmitting latent samples from the prior, while Lx measures reconstructing the input given those samples.Together they correspond to expected data compression by the decoder and prior.
- 2.3. Stochastic Data Generation: Generation repeatedly samples latent variables from the prior and runs the decoder to update the canvas, without involving the encoder.After T repetitions, the generated image is sampled from the distribution parameterized by the final canvas.
3. Read and Write Operations
DRAW contrasts a full-image read/write configuration with selective attention that extracts and reconstructs localized patches using differentiable Gaussian filterbanks. Decoder outputs dynamically control each attention location, zoom, precision, and intensity, while reading and writing use related but distinct operations.
- Reading and Writing Without Attention: Without attention, the encoder receives the entire input image and the decoder modifies the entire canvas at every time-step.
- Reading and Writing Without Attention: This configuration cannot focus encoding on part of the input or restrict decoding to part of the canvas, so it provides no explicit selective attention mechanism.
- Selective Attention Model: DRAW applies an array of 2D Gaussian filters to produce an image patch with smoothly varying location and zoom while retaining gradient-descent training.
- Selective Attention Model: The grid centre and stride position the filterbank, with larger stride exposing more of the original image at lower effective resolution.
- Selective Attention Model: At each time-step, decoder outputs determine the attention centre, variance, stride, and intensity, with variance, stride, and intensity emitted in log-scale to ensure positivity.
- Reading and Writing With Attention: The read operation returns image and error patches, whereas writing uses distinct attention parameters, reversed filter transposition, and inverted intensity to place a decoder patch on the canvas.
- Reading and Writing With Attention: For colour images, the same reading and writing filters are applied independently across all three RGB channels.
4. Experimental Results
DRAW was evaluated on cluttered MNIST classification and on MNIST, SVHN, and CIFAR-10 image generation. Selective attention improved classification and binarized-MNIST generation, while generated images were novel and highly realistic on MNIST and SVHN but blurrier on CIFAR-10.
- CIFAR-10 Generation: Generated images were novel and virtually indistinguishable from real data for MNIST and SVHN, whereas CIFAR images were somewhat blurry but retained recognizable natural-scene structure.For CIFAR-10, the images captured much of the shape, colour, and composition of real photographs.
- Cluttered MNIST Classification: DRAW uses an LSTM that receives a 12 × 12 glimpse at each time-step and then classifies the digit after a fixed number of glimpses.The differentiable attention mechanism supports iterative glimpses of cluttered translated MNIST.
- Cluttered MNIST Classification: Significant improvement in test error over original RAM was achieved with a single attention patch at each time-step instead of RAM’s four patches at different zooms.The comparison concerns 100 × 100 cluttered translated MNIST classification.
- MNIST Generation: DRAW with attention considerably improves the state of the art in negative log-likelihood on binarized MNIST, while DRAW without attention performs comparably to recent generative models.The metric is average nats per image on the test set, with Table 2 reporting negative log-likelihood in nats per test-set example.
- MNIST Generation: With attention, DRAW constructs digits by tracing lines, whereas without attention it progressively sharpens a globally blurred image.The comparison describes the image-generation sequences for the two DRAW configurations.
- SVHN Generation: SVHN generations were highly realistic, and the network drew digits one at a time while moving and scaling its attention patch to vary slopes and sizes.The SVHN experiments used 54 × 54 patches extracted at random locations from 64 × 64 house-number images.
5. Conclusion
DRAW generates highly realistic natural images and improves on the best known results for binarized MNIST generation. Its differentiable attention mechanism also benefits image classification.
- DRAW generates highly realistic natural images, including photographs of house numbers.
- DRAW improves on the best known results for binarized MNIST generation.
- DRAW’s two-dimensional differentiable attention mechanism benefits image generation and image classification.