Source-linked AI summary
Inverting The Generator Of A Generative Adversarial Network
Antonia Creswell, Anil Anthony Bharath
TL;DR
GAN generators are difficult to invert, limiting access to latent representations useful for discriminative tasks and image manipulation. This paper proposes inversion techniques for pre-trained GANs and finds that latent encodings preserve digit identity and style while reconstructing unseen-alphabet characters well.
Problem
GANs are difficult to invert, yet accessing latent representations may support image retrieval, classification, manipulation, and analysis of learned models.
Method
The paper infers latent representations by optimizing through a pre-trained GAN's available computational graph, including batch inversion when required by batch normalization.
Results
The method preserves MNIST digit identity and style and reconstructs handwritten characters from Omniglot alphabets unseen during GAN training.
Takeaways & Limitations
Latent encodings may support applications beyond digit classification, with possible implications for one-shot learning.
Takeaways & Limitations
Prior inversion approaches require training an additional network, while deterministic generators may admit multiple latent representations for one image.
Abstract
from arXiv · showhide
Generative adversarial networks (GANs) learn to synthesise new samples from a high-dimensional distribution by passing samples drawn from a latent space through a generative network. When the high-dimensional distribution describes images of a particular data set, the network should learn to generate visually similar image samples for latent variables that are close to each other in the latent space. For tasks such as image retrieval and image classification, it may be useful to exploit the arrangement of the latent space by projecting images into it, and using this as a representation for discriminative tasks. GANs often consist of multiple layers of non-linear computations, making them very difficult to invert. This paper introduces techniques for projecting image samples into the latent space using any pre-trained GAN, provided that the computational graph is available. We evaluate these techniques on both MNIST digits and Omniglot handwritten characters. In the case of MNIST digits, we show that projections into the latent space maintain information about the style and the identity of the digit. In the case of Omniglot characters, we show that even characters from alphabets that have not been seen during training may be projected well into the latent space; this suggests that this approach may have applications in one-shot learning.
1 Introduction
The paper proposes inverting a GAN by finding latent vectors whose generated images resemble target images, without training an additional decoder network. The approach supports pre-trained generators with available computational graphs and aims to preserve MNIST style and character class while improving inversion efficiency through batches.
- 1 Introduction: Existing decoder-based inversion methods often reconstruct MNIST digits poorly, failing to preserve style and character class.They also require training a third network, preventing inversion of pre-trained networks.
- 1 Introduction: The proposed technique infers latent representations whose generated samples are visually similar to their source images and better preserve MNIST style and character class.The method is presented as an alternative to prior decoder-based approaches.
- 1 Introduction: The approach can be applied to a pre-trained generator when its computational graph is available.This avoids requiring a newly trained decoder network.
- 1 Introduction: Batches of latent vectors can be inferred from batches of images, allowing multiple images to be inverted in parallel and improving efficiency.Batch inversion may also be necessary for networks using batch normalisation.
- 1 Introduction: GAN inversion finds a latent vector z whose generated image closely resembles a target image.This maps an image from image space into the generator's latent space.
2 Method: Inverting The Generator
The method inverts a trained generator by optimizing latent variables so their generated images resemble target images, optionally using batch processing and prior-based constraints.
- Core inversion: Inversion infers z from image x by minimizing reconstruction error through the trained generator.When G's computational graph is known, gradient descent can calculate z* from the reconstruction objective.
- Core inversion: Algorithm 1 initializes z* from the prior, repeatedly computes binary cross-entropy, and updates z* with gradient descent until convergence.The procedure then returns the optimized latent representation.
- Core inversion: A deterministic generator maps each z to one image, but generator collapse can allow one image to have multiple latent representations.Thus, inversion may have multiple valid z solutions for the same x.
- Batch inversion: Mixed-batch inversion addresses batch-normalisation issues and permits multiple image samples to be inverted in parallel.The method assumes sufficiently large batches make batch statistics approximately constant, so each latent update is driven mainly by its own reconstruction loss.
- Using prior knowledge of P(Z): Prior-aware inversion clips latent values for uniform priors and can add mean- and standard-deviation penalties for Gaussian priors.For sufficiently large latent dimensions, these statistics can be estimated more meaningfully across a batch than from one sample.
3 Relation to Previous Work
The paper positions its inversion method as more broadly applicable than prior approaches by handling batch normalisation and using pixel-wise reconstruction loss.
- Differences from prior work: The method extends prior inversion work by showing that mixed batches can be inverted despite batch-normalisation effects.The cited comparison notes that these batch-normalisation problems were not discussed by Zhu et al.
- Prior handling: The approach encourages inferred latent codes to remain near the generator's training prior through hard or soft constraints.The paper evaluates this strategy for both uniform and normal priors.
- Differences from prior work: Unlike approaches using AlexNet feature loss, this method compares images with pixel-wise loss, which is presented as generic to the dataset.The paper specifically contrasts natural-scene features with datasets such as MNIST digits.
4 “Pre-trained” Models
The experiments use four GANs trained on MNIST and Omniglot, varying the latent prior to evaluate inversion across datasets and prior distributions.
- Model setup: Four models were trained by crossing two datasets, MNIST and Omniglot, with uniform and normal latent priors.This design permits comparison of clipping or regularisation under the corresponding priors.
- MNIST: MNIST contains 60k handwritten digit samples covering categories 0 through 9, split into 50k training and 10k testing examples.The networks were trained on the 50k-sample training set.
- MNIST: MNIST generators were trained for 500 iterations with batch size 128 and learning rate 0.002 using Adam updates.The training data covered all ten digit categories.
- Generations: Figure 1 presents 100 random generations from MNIST networks trained with uniform and normal priors.The figure caption also identifies Omniglot generations under both priors.
- Omniglot: Omniglot contains characters from 50 alphabets, with 30 writing systems in the background training set and 20 different systems in testing.The Omniglot generators use the same architecture as prior work, and the supplied passage continues beyond the training description.
5 Experiments
The experiments evaluate inversion by reconstruction error and visual preservation of identity and style across MNIST and Omniglot generators trained with different priors.
- Evaluation design: A valid inversion maps image x to z* such that G(z*) is close to the original image x.The experiments apply inversion to random batches of images across four generators.
- MNIST results: MNIST inversions generally preserve digit style and identity well, while Table 2 reports absolute mean reconstruction error for the tested methods.Visual inspection did not clearly establish that regularisation improved inversion.
- Evaluation design: The study compares binary cross-entropy inversion with clipping or regularisation under uniform and normal latent priors.The goal is to examine whether constraints are necessary and whether the method generalizes across priors.
- Evaluation measures: Mean absolute pixel error quantifies reconstruction quality, while visual comparisons assess preservation of digit or character identity and MNIST style.The qualitative evaluation uses pairs of original images and reconstructions.
6 Results
The proposed inversion reconstructs MNIST digits while preserving style and identity, and produces generally sharp Omniglot reconstructions even for unseen alphabets. Regularisation does not improve reconstruction in either evaluation.
- MNIST: MNIST inversions generally preserve both the style and identity of handwritten digits.The evaluation compares original images x with reconstructions G(z∗).
- Regularisation: Regularisation does not improve the reported reconstruction results for either MNIST or Omniglot.The MNIST discussion reports no visual improvement, while the Omniglot comparison likewise supports inversion without regularisation.
- Omniglot: Omniglot reconstructions are generally sharp and capture fine details such as small circles and edges.The generator was tested on characters from alphabets absent from training.
- MNIST: Figure 2 compares original MNIST images with inverted reconstructions for generators trained with uniform and normal priors.Original images appear on the right and inverted images G(z∗) on the left.
- Omniglot: One severe Omniglot failure occurs in Fig. 3(b), where the top example fails to invert the sample.This is an isolated failure described alongside otherwise generally successful reconstructions.
7 Conclusion
The paper presents a computational-graph-based process for inverting pre-trained GAN generators into latent encodings. Results on MNIST and Omniglot indicate preserved digit properties and good reconstruction of unseen-alphabet characters, while batch inversion addresses a batch-normalisation constraint.
- Conclusion: The proposed process inverts any pre-trained GAN with an available computational graph to obtain latent encodings for image samples.The generator maps latent variables to images, and inversion seeks the corresponding latent representation.
- Conclusion: For MNIST and Omniglot, regularisation was unnecessary, suggesting that the inversion approach may apply more generally.The paper presents regularisation methods for different latent priors but reports no need for them in these datasets.
- Batch inversion: Batch inversion is necessary when batch normalisation prevents reliable inversion of single image samples and may improve computational efficiency.Under reasonable assumptions, each latent sample’s update depends only on its own reconstruction error.
- Findings: MNIST inversions preserve digit identity and style, suggesting that the latent space retains both properties.The paper also reports that such latent encodings may support applications beyond digit classification.
- Findings: Omniglot characters from alphabets unseen during GAN training can be projected into latent space with good reconstructions.The paper states that this may have implications for one-shot learning.