Source-linked AI summary
Conditional Image Synthesis With Auxiliary Classifier GANs
Augustus Odena, Christopher Olah, Jonathon Shlens
TL;DR
High-resolution, globally coherent image synthesis remains difficult for GANs, especially on diverse datasets. The paper introduces auxiliary classifier GANs and evaluates their class discriminability and diversity, finding that 128×128 samples are more discriminable than naively resized low-resolution samples and that 84.7% of ImageNet classes achieve comparable diversity.
Problem
GANs struggle to generate globally coherent, high-resolution samples from datasets with high variability, while existing metrics do not fully assess intra-class diversity.
Method
The paper introduces auxiliary classifier GANs, which condition generation on class labels and train the discriminator to predict both source and class.
Results
Across ImageNet, 128×128 samples show increased global coherence, a 50% discriminability decrease after downsampling to 32×32, and diversity comparable to real data for 84.7% of classes.
Takeaways & Limitations
The results show that high-resolution class-conditional synthesis captures class information beyond naive upscaling while maintaining comparable diversity for most ImageNet classes.
Takeaways & Limitations
Only 84.7% of ImageNet classes matched real-data diversity, and training stability required distributing the classes across 100 AC-GAN models.
Abstract
from arXiv · showhide
Synthesizing high resolution photorealistic images has been a long-standing challenge in machine learning. In this paper we introduce new methods for the improved training of generative adversarial networks (GANs) for image synthesis. We construct a variant of GANs employing label conditioning that results in 128x128 resolution image samples exhibiting global coherence. We expand on previous work for image quality assessment to provide two new analyses for assessing the discriminability and diversity of samples from class-conditional image synthesis models. These analyses demonstrate that high resolution samples provide class information not present in low resolution samples. Across 1000 ImageNet classes, 128x128 samples are more than twice as discriminable as artificially resized 32x32 samples. In addition, 84.7% of the classes have samples exhibiting diversity comparable to real ImageNet data.
1. Introduction
GANs struggle with globally coherent, high-resolution synthesis, especially on highly variable datasets. This work adds latent-space structure and a specialized cost function, demonstrates 128x128 ImageNet synthesis, and introduces quantitative analyses of resolution use and perceptual variability.
- GANs struggle to generate globally coherent, high-resolution samples, particularly from datasets with high variability.
- Adding structure to the GAN latent space and a specialized cost function produces higher-quality, more globally coherent 128x128 ImageNet samples.
- The model synthesizes images for all 1000 ImageNet classes at 128x128 spatial resolution, with extension to other spatial resolutions.
- The paper measures how much synthesis models use output resolution and introduces a fast metric for perceptual variability and GAN collapsing behavior.
- The authors identify the high number of ImageNet classes as a central synthesis difficulty, provide an explicit solution, and report strong results on memorization and CIFAR-10 Inception score.
2. Background
GANs train a generator and discriminator in opposition, while side information can condition synthesis or be reconstructed by an auxiliary decoder. These extensions aim to improve class-conditional sample quality and the discriminator’s original task.
- GAN framework: A GAN trains generator G and discriminator D adversarially: G maps noise z to Xfake, while D estimates P(S | X) over image sources.The discriminator maximizes the log-likelihood of assigning the correct source.
- Conditional synthesis: Supplying class labels to both networks produces class-conditional samples and can significantly improve generated sample quality.Image captions and bounding-box localizations are richer side information that may improve quality further.
- Auxiliary reconstruction: An auxiliary decoder can make the discriminator reconstruct class labels or latent-variable subsets instead of directly receiving side information.This modification adds an auxiliary prediction task to the discriminator.
- Auxiliary reconstruction: Forcing a model to perform additional tasks is known to improve performance on its original task.The auxiliary-decoder approach applies this multitask-training principle to the discriminator.
3. AC-GANs
AC-GANs condition generation on class labels while training the discriminator to predict both source and class. The paper also reports stabilized training and uses class-based ensembles for ImageNet experiments.
- Architecture: AC-GANs generate images from a class label c and noise z, while the discriminator predicts source and class distributions.The discriminator outputs P(S | X) and P(C | X).
- Training objective: The discriminator maximizes LS + LC, whereas the generator maximizes LC −LS; z is learned independently of the class label.LS is the log-likelihood of the correct source, and LC is the class objective.
- Contributions: The AC-GAN modification produces excellent results and appears to stabilize training, alongside methods for evaluating resolution use and perceptual sample variability.The authors describe AC-GAN as one part of the paper’s technical contributions.
- ImageNet implementation: 100 AC-GANs, each trained on a 10-class split, form the ensemble used in all ImageNet experiments.The architecture permits separating large datasets into class-based subsets for separate generator and discriminator training.
4. Results
The results show that AC-GANs benefit from higher output resolution and can be evaluated using discriminability and perceptual-diversity measures. Across ImageNet classes, higher-resolution samples are more discriminable, while mode dropping is associated with lower sample quality.
- Training setup: 100 AC-GAN models were trained on 10 ImageNet classes each for 50000 mini-batches, using 128 × 128 and 64 × 64 output resolutions.Reducing variability across all 1000 classes improved training quality.
- Discriminability: 10.1% ± 2.0% accuracy was achieved at 128 × 128, versus 7.0% ± 2.0% after resizing to 64 × 64 and 5.0% ± 2.0% after resizing to 32 × 32.Downsizing decreased visual discriminability by 38% and 50%, respectively.
- Discriminability: 84.4% of ImageNet classes had higher accuracy at 128 × 128 than at 32 × 32, while the 64 × 64-trained model achieved less discriminability than the 128 × 128 model.The 64 × 64 model’s accuracies plateaued at 64 × 64 resolution.
- Diversity: Inception accuracy and MS-SSIM were anti-correlated (r2 = −0.16), with 74% of low-diversity classes having Inception accuracies ≤1%.Conversely, 78% of high-diversity classes had Inception accuracies exceeding 1%.
- Compositionality: Holding z fixed while changing class labels preserved position, layout, and background across bird classes, indicating that AC-GAN represents certain types of compositionality.The samples used eight bird classes, with identical z values across each row.
5. Discussion
The discussion highlights AC-GAN’s globally coherent ImageNet synthesis and introduces quantitative analyses of sample discriminability and diversity. It also identifies limitations in visual quality, diversity coverage, and training scalability while pointing to semi-supervised learning as an application.
- Contributions: AC-GANs generated globally coherent ImageNet samples and enabled quantitative analysis of discriminability across spatial resolutions and diversity relative to training data.The work also found samples more discriminable than those from a lower-resolution model followed by naive resizing.
- Limitations and future work: 10.1%± 2.0% average Inception accuracy for synthesized images remains below 81% for real training data.The discussion identifies improving the visual discriminability of the 128 × 128 model as an important direction, including augmenting the discriminator with a pre-trained model for supervised tasks.
- Limitations and future work: 84.7% of ImageNet classes exhibited diversity comparable to real training data, while dividing 1000 classes across 100 AC-GAN models greatly aided training stability.Building one model capable of generating samples from all 1000 classes is identified as an important next step.
- Applications: AC-GANs can support semi-supervised learning by ignoring the class-label loss component when a training image lacks a label.Image synthesis models provide classifiers with a rich prior over natural image statistics for datasets with few labels.
A. Hyperparameters
The paper summarizes separate hyperparameter settings for its ImageNet and CIFAR-10 models. The ImageNet setup defines Soft-Sigmoid and uses discriminator activation noise, while CIFAR-10 settings are selected by grid search and evaluated for Inception Score variance.
- A. Hyperparameters: ImageNet and CIFAR-10 models use separately summarized hyperparameters in Tables 1 and 2.The paper presents the ImageNet settings in Table 1 and the CIFAR-10 settings in Table 2.
- A. Hyperparameters: Soft-Sigmoid applies Softmax to K output units and Sigmoid to the remaining unit.This operation covers K+1 output units.
- A. Hyperparameters: The ImageNet discriminator uses activation noise, following Salimans et al. (2016).The passage identifies activation noise as a suggested technique from the cited work.
- A. Hyperparameters: CIFAR-10 hyperparameters are selected through grid search, training one AC-GAN per setting on the full dataset.Each trained model's samples are split into groups to estimate Inception Score variance, using an analysis described as identical to Salimans et al. (2016).