Source-linked AI summary
Invertible Conditional GANs for image editing
Guim Perarnau, Joost van de Weijer, Bogdan Raducanu, Jose M. Álvarez
TL;DR
GANs and cGANs generate realistic images but lack a mechanism to infer a real image’s latent representation for controlled editing. The paper combines an encoder with a cGAN as an IcGAN, evaluates encoder and cGAN designs, and finds strong conditional attribute generation and reconstruction of unseen images. IcGANs enable deterministic complex modifications of real images within the evaluated scope.
Problem
GANs lack an inference mechanism for mapping a real image into latent and conditional representations needed to reconstruct and modify it.
Method
IcGANs combine an encoder that maps real images to z and y with a cGAN that regenerates images under modified conditional information.
Results
cGANs achieve overall accuracy of ∼86% for conditional attributes, while the encoder reconstructs unseen test images and IND performs slightly better qualitatively than IND-COND.
Takeaways & Limitations
IcGANs enable explicit control of complex attributes when regenerating real images, including smooth interpolation between plausible CelebA faces.
Takeaways & Limitations
For binary conditional information, the paper samples y from pdata because direct interpolation and kernel density estimation are unsuitable.
Abstract
from arXiv · showhide
Generative Adversarial Networks (GANs) have recently demonstrated to successfully approximate complex data distributions. A relevant extension of this model is conditional GANs (cGANs), where the introduction of external information allows to determine specific representations of the generated images. In this work, we evaluate encoders to inverse the mapping of a cGAN, i.e., mapping a real image into a latent space and a conditional representation. This allows, for example, to reconstruct and modify real images of faces conditioning on arbitrary attributes. Additionally, we evaluate the design of cGANs. The combination of an encoder with a cGAN, which we call Invertible cGAN (IcGAN), enables to re-generate real images with deterministic complex modifications.
1 Introduction
Complex face editing is difficult to automate realistically, while GANs lack a mechanism for mapping real images back into latent representations. IcGANs combine an encoder with a cGAN to reconstruct real images and control their attributes.
- Complex face edits such as changing smiles, hair color, or gender often require skilled human editing to remain realistic.
- GANs generate plausible images and cGANs control image features through conditional information, but GANs lack inference for real inputs.
- IcGANs combine an encoder and cGAN to map real images into latent and conditional representations for meaningful attribute modifications.
- The paper evaluates IcGANs on MNIST and CelebA by changing conditional information y to perform meaningful and realistic image edits.
2 Related work
Related work contrasts VAEs and GANs as prominent generative-modeling approaches and positions IcGANs as a conditional inversion approach extending prior encoder-based GAN research.
- VAEs regularize sampling through a prior latent space but use pixel-wise reconstruction loss, which can produce blurry images.
- GANs optimize generator realism through adversarial training, while later work improves architectures, training, feature disentanglement, and conditional generation.
- IcGANs differ from prior encoder approaches by encoding both latent and conditional representations and enabling complex variations of input images.
- The paper analyzes conditional encoding and cGAN design decisions, complementing related encoder work.
3 Background: Generative Adversarial Networks
GANs learn image distributions through competition between a generator and discriminator. Conditional GANs add information that controls modeled attributes, while latent variables represent remaining variation.
- A GAN trains generator G and discriminator D in an iterative minimax game to approximate the data distribution.
- The generator seeks to fool the discriminator, whereas the discriminator distinguishes real samples from generated ones.
- The GAN objective combines expected log discrimination of real data with expected log rejection of generated samples.
- The latent vector z is sampled from a known simple distribution such as a normal distribution.
- cGANs add conditional information y and sample generated labels y′ from a density model for conditional generation.
- Conditional information models constrained variations, while latent z encodes data variations not represented by y.
4 Invertible Conditional GANs
IcGAN combines a conditional GAN with an encoder that maps real images to latent and conditional representations, enabling controlled reconstruction and editing. The section also examines encoder designs, conditional placement, and conditional sampling for cGANs.
- 4.1 Encoder: IcGAN combines a cGAN generator with an encoder E that maps a real image x to latent representation z and conditional vector y.The generator then produces x′ = G(z, y), allowing controlled variations through the latent and conditional representations.
- 4.1 Encoder: The encoder uses separate sub-encoders Ez and Ey, trained respectively to recover latent vectors and conditional information from images.Ez is trained on generated images with latent-vector supervision, while Ey can be trained directly on real images and labels.
- 4.1 Encoder: Three encoder interaction strategies are considered: a shared encoder with two outputs, independent encoders, and an Ez encoder conditioned on Ey.These strategies are evaluated for how they leverage conditional information during encoding.
- 4.2 Conditional GAN: The cGAN study examines where to inject y and how to sample generator-side conditional information y′.The generator receives z and y′, while the discriminator receives y with real or generated samples; candidate sampling methods include kernel density estimation, interpolation, and training-set labels.
- 4.2 Conditional GAN: For binary attributes, direct interpolation can produce implausible conditional vectors, so the paper favors sampling approaches suited to the label distribution.The paper notes that interpolation is mainly suitable for real-valued attribute vectors, whereas its datasets use binary conditional information.
5 Experiments
Experiments evaluate cGAN design, encoder configurations, and IcGAN editing on MNIST and CelebA, with quantitative evaluation performed on CelebA only. The results identify effective conditional-injection and encoder choices, and demonstrate reconstruction, attribute modification, interpolation, and attribute transfer.
- 5.1 Datasets: Experiments use MNIST and CelebA, but quantitative evaluation is performed only on CelebA because it is considerably more complex.CelebA uses 18 visually relevant attributes after filtering the original 40.
- 5.2 Evaluating the conditional GAN: The cGAN reaches approximately 86% overall accuracy for generating visual representations of conditional attributes.The best configuration inserts y at the generator input and the discriminator’s first convolutional layer.
- 5.3 Evaluating the encoder: IND produces the most faithful reconstructions among the evaluated encoder configurations, slightly outperforming IND-COND qualitatively.The evaluation prioritizes visual reconstruction quality, and IND also achieves the lowest squared reconstruction loss, Le = 0.429, on 150K generated CelebA test images.
- 5.3 Evaluating the encoder: Latent-space reconstruction preserves high-level facial features while sacrificing some local details, without producing the blur associated with element-wise encoders such as VAEs.The latent representation is invariant to local details such as exact hair, eye, or face position.
- 5.4 Evaluating the IcGAN: IcGAN reconstructions preserve real-image structure while changing conditional attributes across columns, producing handwritten-style digit variations and modified CelebA faces.The procedure encodes each real image into z and y, fixes z within rows, and modifies y across columns.
- 5.4 Evaluating the IcGAN: Linear interpolation of z and y between reconstructed CelebA faces yields plausible faces and smooth transitions, while swapping y transfers attributes between faces.The results suggest that z represents pose, illumination, and background, whereas y represents unique facial features.
6 Conclusions
The paper introduces IcGANs by combining an encoder with a conditional GAN to infer latent and conditional representations from real images. It also reports design choices for conditional injection and encoder training, with satisfactory and promising CelebA results.
- 6 Conclusions: IcGANs combine an encoder with a conditional GAN to map real images to latent z and conditional y representations.This addresses GANs’ lack of an inverse mapping for real samples while supporting explicit attribute control.
- 6 Conclusions: The generator performs best when y is added at the input, while the discriminator performs best when y is added at its first layer.The conclusion also identifies two independent encoders for z and y as the best training option in the experiments.
- 6 Conclusions: Results on CelebA are described as satisfactory and promising for complex face editing.The model explicitly controls complex generated attributes through conditional information y.