Source-linked AI summary
It Takes (Only) Two: Adversarial Generator-Encoder Networks
Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
TL;DR
Existing hybrid approaches combine autoencoding and adversarial learning but add architectural complexity. AGE instead trains an encoder and generator directly through latent-distribution divergences, achieving high-quality generation, strong reconstructions, and plausible conditional colorizations. The paper concludes that this two-component approach can match or improve on more complex systems in the reported settings.
Problem
Existing approaches supporting both generation and inference typically combine autoencoders with adversarial discriminators, increasing model complexity.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Direct generator-versus-encoder adversarial training can combine generation, inference, and reconstruction without a discriminator network and with relatively quick convergence.
Source: §4 · p. 8The approach leaves room to test more complex latent distributions and alternative divergence measures.
Source: §4 · p. 9
Abstract
from arXiv · showhide
We present a new autoencoder-type architecture that is trainable in an unsupervised mode, sustains both generation and inference, and has the quality of conditional and unconditional samples boosted by adversarial learning. Unlike previous hybrids of autoencoders and adversarial networks, the adversarial game in our approach is set up directly between the encoder and the generator, and no external mappings are trained in the process of learning. The game objective compares the divergences of each of the real and the generated data distributions with the prior distribution in the latent space. We show that direct generator-vs-encoder game leads to a tight coupling of the two components, resulting in samples and reconstructions of a comparable quality to some recently-proposed more complex architectures.
1 Introduction
AGE combines autoencoder-style inference and generation with adversarial learning using only an encoder and generator. It targets lower complexity while retaining competitive sample quality, reconstruction quality, and conditional colorization.
Adversarial learning
- Unlike hybrid systems using discriminators or post-hoc inverse mappings, AGE directly trains the encoder-generator pair through divergences involving latent distributions.The generator reduces the divergence for encoded generated samples, while the encoder supplies the opposing adversarial signal.Source: §1 · pp. 1–2
Motivation
- AGE avoids the added complexity of systems containing an encoder, generator, and discriminator while retaining autoencoder and adversarial-training advantages.Reconstruction losses can optionally make the encoder-generator pair act as an autoencoder.Source: §1 · pp. 1–2
Architecture
- AGE combines two feed-forward mappings—an encoder and generator—to support both generation from a simple prior and inference into latent space.The encoder maps real and generated samples into latent space, inducing two distributions whose divergences from the prior guide learning.Source: §1 · p. 2
Evaluation
Conditional experiments
- In the conditional setting, AGE successfully tackles colorization, a problem described as difficult for GAN-based approaches.The approach is evaluated alongside its unconditional generation and reconstruction capabilities.Source: §1 · p. 2
2 Adversarial Generator-Encoder Networks
AGE networks combine generation, inference, adversarial distribution alignment, and reciprocal reconstruction using only an encoder and generator. Their objectives align real and generated distributions through a latent prior while theoretical and qualitative results support accurate generation and reconstruction.
2.1 Adversarial distribution alignment
- The adversarial criterion compares encoded real and generated distributions with a simple reference prior, making latent-space statistics replace direct high-dimensional distribution matching.The prior is chosen so divergences from empirical encoded distributions can be computed simply, including with stochastic batch approximations.Source: §2.1 · pp. 3–4
- Perfect generators are exactly the saddle-point solutions of the basic game, and the reference-prior game likewise has saddle points precisely when the generator matches the data distribution.The analysis assumes sufficiently expressive mappings and the existence of at least one perfect generator.Source: §2.1 · pp. 3–4
2.2 Encoder-generator reciprocity and reconstruction losses
- Latent- and data-space reconstruction losses encourage encoder-generator reciprocity, with the latent loss making distribution alignment sufficient for reciprocal mappings.The training objective combines the adversarial game with reciprocity losses.Source: §2.2 · p. 4, §2.3 · p. 4
2.3 Training AGE networks
- Qualitatively, AGE produces more accurate reconstructions than ALI [6], especially on Tiny ImageNet, while samples have similar visual quality.Figure 2 compares samples and reconstructions for Tiny ImageNet and SVHN, with odd columns showing real examples and even columns showing reconstructions.Source: Figure 2 · p. 5
- Training alternates generator and encoder updates; omitting the generator’s data-space reconstruction loss helps avoid the blurring characteristic of autoencoders.The encoder maximizes mismatch between encoded generated samples and the prior relative to encoded real data, while the generator removes that mismatch.Source: §2.3 · p. 5
3 Experiments
AGE is evaluated for unconditional generation and reconstruction, then for conditional colorization. Across these settings, it produces competitive samples and reconstructions while supporting diverse colorizations and faster training than ALI.
3.1 Unconditionally-trained AGE networks
- On Tiny ImageNet, AGE reconstructions have considerably higher fidelity than those obtained by ALI.Source: §3.1 · p. 6
- AGE reconstructions on CelebA are roughly comparable or better than ALI, while 10 training epochs take two hours versus six hours for ALI.Source: §3.1 · p. 6, Figure 3 · p. 7
- AGE achieves an Inception score of 5.90 ± 0.04 on CIFAR-10, exceeding ALI’s 5.34 ± 0.05 and another reported score of 4.36 ± 0.04, but below 7.72 ± 0.13.Source: §3.1 · p. 6
- AGE obtains a MNIST log likelihood of 746, above ALI’s 721, VAE’s 705.375, and GAN’s 346.679 under the reported evaluation.Source: §3.1 · p. 6
3.2 Conditional AGE network experiments.
- The conditional experiments use Lab images, condition both mappings on the lightness channel, and represent colorization with three-dimensional latent variables.Source: §3.2 · p. 7
- For conditional colorization, AGE produces plausible and diverse outputs across latent inputs, whereas pix2pix largely ignores added noise maps and yields low diversity.Source: §3.2 · p. 8, Figure 4 · p. 8
- Conditional AGE also transfers a latent color representation from one image to colorize another grayscale image.Source: §3.2 · p. 8
4 Conclusion
The paper presents AGE as a discriminator-free adversarial approach for jointly learning generation and inference. It reports high-quality samples, strong reconstructions, and diverse colorizations, while leaving latent distributions and divergence measures open for further study.
- AGE jointly learns generation and inference through an adversarial game based on distribution divergences rather than individual-sample discrimination.Source: §4 · p. 8
- Across standard datasets, AGE produces high-quality samples and reconstructions with better fidelity than [6], while generating plausible and diverse colorizations unlike GAN-based system [11].Source: §4 · p. 8
- The approach remains open to experiments with more complex latent distributions and alternative divergence measures.Source: §4 · p. 9
5 Appendix
The supplementary material provides theorem proofs and additional qualitative results across several datasets.
- The supplement restates and proves the main-text theorems and presents additional qualitative results on several datasets.Source: §5 · p. 11
A Proofs
The proofs characterize when the AGE game’s saddle points recover the target data distribution and establish alignment properties under almost-everywhere invertibility. They also note that optimal encoders need not align encoded real and generated distributions with the prior.
Assumptions
- The theory assumes an invertible almost-everywhere map between the chosen latent and data distributions, with atomless distributions admitting such a map.The latent prior may be chosen as a Gaussian or uniform distribution on a sphere.Source: A Proofs · p. 11
Theorem 1
- A saddle point of game (8) occurs if and only if the generator distribution matches the data distribution, g*(Z) = X.The proof uses nonnegativity of V1 and the condition that zero divergence implies equality of encoded distributions.Source: A Proofs · p. 11
Supporting lemmas
- If an encoder is almost everywhere invertible and encoded generated and real distributions coincide, then the generated and real data distributions themselves coincide.The argument uses almost-everywhere inversion to transfer equality from latent space back to data space.Source: A Proofs · p. 11
Theorem 2
- For game (9), every Nash equilibrium generator matches the data distribution, and any aligned generator forms a saddle point with a suitable encoder.Thus the generator–encoder game has equilibria precisely tied to distributional alignment, subject to the theorem’s assumptions.Source: A Proofs · pp. 11–12
Caveat
- Not every optimal encoder aligns encoded real and generated distributions with the reference latent distribution, because an encoder may collapse the data distribution to a few points.The theorem concerns generator distributional alignment even when individual optimal encoders have this degeneracy.Source: A Proofs · p. 12
Supporting result
- An almost-everywhere inverse of an alignment map transfers the aligned distribution back, yielding h(Q) = W.The result establishes reciprocal alignment on the supports of the two distributions.Source: A Proofs · p. 12
B Additional qualitative results.
The qualitative results compare AGE samples, reconstructions, interpolations, and conditional color transfer across image datasets. They also show how reconstruction terms affect sample diversity and inference, while adding data-space reconstruction can degrade reconstructions.
CIFAR-10 samples
- CIFAR-10 samples from the reconstruction-trained model remain diverse and support inference, unlike the distribution-alignment-only ablation.The ablated model is compared with DCGAN [23], while reconstruction-trained results are connected to Figure 6.Source: Figure 5 · p. 13
CIFAR-10 reconstructions
- CIFAR-10 reconstruction comparisons show real test examples in odd columns and their unseen reconstructions in even columns, against ALI [6].The examples were never observed during training.Source: Figure 6 · p. 13
CelebA samples
- CelebA results place real examples beside samples generated by the AGE model.The figure presents real images in panel (a) and generated samples in panel (b).Source: Figure 7 · p. 14
Latent interpolation
- CelebA latent interpolation displays a sequence between two endpoint images, with the original images positioned at the two sides.The visualization illustrates interpolation in the learned latent space.Source: Figure 8 · p. 14
Reconstruction-loss ablation
- Adding data-space reconstruction loss produces degraded reconstructions, shown with real images in odd columns and reconstructions in even columns.The paper states that this term is omitted from the generator objective in the other experiments.Source: Figure 9 · p. 14
Conditional color transfer
- Conditional AGE transfers the color scheme of the first image in each triplet onto the second image.The figure reports additional conditional color-transfer results.Source: Figure 10 · p. 15