Source-linked AI summary
Generating images with recurrent adversarial networks
Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, Roland Memisevic
TL;DR
The paper develops a recurrent image generator motivated by iterative canvas updates and addresses the lack of an obvious quantitative evaluation for adversarial models. It trains the generator adversarially, evaluates models through generator–discriminator competition, and reports strong samples and superior performance under the proposed metric.
Problem
Image generation has become an important goal, but generative adversarial models lack an obvious quantitative evaluation method, while existing human-inspection approaches are subjective, expensive, and cumbersome.
Method
The paper proposes a recurrent generator that learns an image-generation procedure through structurally identical sequential updates, trains it adversarially, and compares models using competing generators and discriminators.
Results
The recurrent model produces reasonably classifiable, varied, detailed samples, and recurrent models with three or five steps outperform the one-step model across all three datasets.
Takeaways & Limitations
The proposed competition-based metric enables quantitative comparison of adversarial networks and ranks the recurrent generative model above existing state-of-the-art models under that metric.
Takeaways & Limitations
The sample-ratio evaluation is applicable only when the test ratio is approximately one; otherwise, the sample ratio is biased.
Abstract
from arXiv · showhide
Gatys et al. (2015) showed that optimizing pixels to match features in a convolutional network with respect reference image features is a way to render images of high visual quality. We show that unrolling this gradient-based optimization yields a recurrent computation that creates images by incrementally adding onto a visual "canvas". We propose a recurrent generative model inspired by this view, and show that it can be trained using adversarial training to generate very good image samples. We also propose a way to quantitatively compare adversarial networks by having the generators and discriminators of these networks compete against each other.
1. Introduction
Image generation has progressed from a diagnostic goal toward practical applications, with sequential and adversarial models improving visual quality. This paper proposes a recurrent adversarial generator that accumulates canvas updates and introduces a quantitative cross-model evaluation scheme.
- Motivation: Image generation became an objective in its own right as visual quality improved and industrial applications approached.Earlier work treated generation mainly as evidence of visual understanding.
- Related approaches: Generative models commonly use probabilistic latent-variable training or adversarial sampling with a separately trained discriminator.Variational autoencoders impose distributional properties on latent representations, whereas adversarial models train generators to produce data-like examples.
- Sequential generation: Sequential variants such as DRAW and LAPGAN generate images over multiple steps and were shown to improve visual quality over their ancestor models.DRAW accumulates updates into a canvas, while LAPGAN uses coarse-to-fine generation and upsampling.
- Proposed model: The proposed recurrent generator learns structurally identical update steps without imposing coarse-to-fine structure, attention, or variational training criteria.The recurrent network is trained to determine the generation procedure itself.
- Proposed model: Unrolling feature-matching gradient descent yields an encoder-decoder recurrent computation in which encoded canvas and reference features determine canvas updates.This connects pixel optimization to incremental recurrent image generation.
- Evaluation: The paper also introduces a cross-over battle in which generators and discriminators from two adversarial models compete to compare their relative performance quantitatively.The scheme is intended to replace or complement subjective inspection-based evaluation.
2. Background
GANs formulate generation as a game between a generator and discriminator, while convolutional architectures provide the mappings used by these networks. Their training objectives and update rules motivate the adversarial framework used in the paper.
- GAN formulation: A GAN trains generator G and discriminator D in a non-cooperative game: G produces hard-to-classify samples while D distinguishes generated from real data.The ideal target is a generator distribution matching the data distribution.
- GAN formulation: Formally, D classifies points in R^M as real or generated, while G maps z in R^K from a prior distribution into the data space R^M.The models are trained through a minmax objective over their parameters.
- Training objective: The standard adversarial objective can suffer from saturation, producing small gradients that impede generator learning.The paper motivates a reformulated objective to address this vanishing-gradient problem.
- Training objective: The reformulated objective preserves the underlying intuition, generator gradient direction, and fixed points of the original objective.Although the objectives differ, their generator updates point in the same direction.
- Training procedure: Generation samples z from a zero-mean, unit-variance Gaussian prior, passes it through G, and evaluates the resulting sample with D.The generator may be deterministic or probabilistic, although prior deployed models were deterministic.
- Training procedure: After evaluating the cost, backpropagation updates D when its predictions are wrong and updates G when D correctly identifies generated samples.The update rules reflect the two opposing minmax operators.
- Network architecture: Convolutional and convolutional-transpose layers define the discriminator and generator mappings, with shared layerwise notation for inputs, activations, weights, and biases.The paper uses “convolutional transpose” rather than “deconvolutional” for the transpose operation.
3. Model
The paper develops GRAN as a recurrent adversarial image generator inspired by sequential generation and gradient-based feature matching. It incrementally constructs samples through decoder updates, encoder states, and accumulated canvas changes.
- Connection to gradient-based generation: Gradient-based feature matching can be unrolled into recurrent computation: an encoder extracts current-canvas features, while a decoder computes canvas updates.
- Motivation: GRAN explores recurrent image generation as an intermediate between DRAW and gradient-based optimization with a generative adversarial objective.
- Generative Recurrent Adversarial Networks: GRAN's generator is a recurrent feedback loop that consumes noise and produces outputs across multiple time steps.
- Generative Recurrent Adversarial Networks: The generated updates are accumulated on a canvas, with the initial hidden state set to a zero vector and the final output scaled to (0, 1).
- Generative Recurrent Adversarial Networks: At each step, the decoder function f receives noise and the previous hidden state, produces an update, and the encoder function g represents the previous update.
4. Model Evaluation: Battle between GANs
The paper evaluates GANs by making two generator–discriminator pairs compete across models, then uses sample and test ratios to compare performance and validate the outcome.
- Battle setup: Two GAN models exchange opponents in testing: each generator attempts to fool the other model’s discriminator.The training phase preserves each model’s original generator–discriminator pairing before cross-model testing.
- Battle metrics: The test ratio measures relative generalization using discrimination of predefined test data, while discriminator overfitting can bias the sample ratio.Overfitting may increase the chance of generating samples biased toward training data.
- Battle metrics: The sample ratio measures which model’s generator fools the opponent discriminator more easily, indicating which produces more data-like samples.The discriminators classify samples generated by their opponents.
- Decision rule: The proposed metric declares M1 the winner when r_sample < 1 and r_test ≃ 1, M2 when r_sample > 1 and r_test ≃ 1, and otherwise reports a tie.The test-ratio condition is intended to ensure neither discriminator is more overfitted than the other.
- Scope: The evaluation measure, called Generative Adversarial Metric, compares adversarial generative models and partially compares models such as VAE or DRAW using GRAN discriminator error rates.For other models, the criterion is the error rate of GRAN’s discriminators on their generated samples.
5. Experiments
Experiments evaluate GRAN across datasets, time-step counts, noise strategies, sample quality, comparison models, and overfitting. Recurrent generation generally improves over the single-step baseline, while different per-step noises introduce training instability.
- GRAN performance: Across MNIST, CIFAR10, and LSUN, GRAN3 and GRAN5 outperformed GRAN1, with samples judged discernible and reasonably classifiable by humans.LSUN samples also covered varied church buildings and fine textures.
- Comparison models: On MNIST, GAM error rates for DVAE and DRAW samples were below 50%, indicating adversarial discriminators could distinguish these non-adversarial-model samples.The authors hypothesize that mean-squared-error smoothing makes samples look nicer without capturing all relevant real-image statistics.
- Overfitting: Nearest-neighbor checks found no replicated training examples, although the authors observed interpolation between training cases.They characterize possible overfitting as sensible interpolation rather than memorization, while noting generative models may struggle to generate images in certain regions.
- Intermediate generation: Intermediate samples gradually become more fine-grained and add details, despite identical architectures at every time step and no enforced coarse-to-fine structure.Color filling suggests a more complex process than simple coarse-to-fine generation.
- Time-step evaluation: Testing GRAN7 and GRAN9 samples from GRAN5 showed similar visual quality, while GRAN7 won the GAM battle.The higher-step samples appeared brighter than GRAN3 and GRAN5.
- Noise strategy: Different noises at each time step produce discernible samples but require more hyperparameter effort and tend to collapse during prolonged training.They also yield more pronounced changes between intermediate steps.
6. Conclusion
The paper introduces a recurrent adversarial generative model and a quantitative comparison metric for adversarial networks. It reports higher-quality visual samples than an equivalent single-step model and superior performance under the proposed metric.
- The paper proposes GRAN, a generative model based on adversarial training of a recurrent neural network.
- The recurrent generative model produces higher-quality visual samples than an equivalent single-step model.
- The paper introduces a metric for quantitative comparison and reports superior performance over existing state-of-the-art generative models under that metric.
Supplementary Materials
The supplementary material explains convolutional transpose operations, their gradients, the relation between input backpropagation and sequential models, and GRAN’s architecture and experiments. It also documents datasets, training choices, and ImageNet behavior.
- Additional Notes on Convolutional Transpose: A convolution’s gradient is itself a convolution and can implement the convolutional transpose.The convolutional transpose corresponds to a transpose of the replicated input matrix.
- Additional Notes on Convolutional Transpose: For stride 2, the gradient applies convolution to an upsampled input formed by inserting zeros between input values.
- Relation between sequential modeling and backpropagation with respect to the input methods: Backpropagation with respect to the input iteratively updates an input by differentiating an objective through an intermediate function.The intermediate function may comprise multiple nonlinear functions, including neural networks.
- Relation between sequential modeling and backpropagation with respect to the input methods: The content-based objective compares the hidden representation of the current input with that of a reference image, and its optimization unrolls into a recurrent computation.
- Experimental setup: GRAN is evaluated on MNIST, CIFAR10, and LSUN using normalized pixels, ADAM optimization, batch size 100, and separately selected generator and discriminator learning rates.
- Experimental setup: GRAN is compared at 1, 3, and 5 time steps, with GRAN1 equivalent to DCGAN up to one additional fully connected layer.