Source-linked AI summary
AutoGAN: Neural Architecture Search for Generative Adversarial Networks
Xinyu Gong, Shiyu Chang, Yifan Jiang, Zhangyang Wang
TL;DR
GAN architecture design remained relatively underexplored, while applying NAS to GANs is complicated by unstable training and the absence of a natural validation reward. AutoGAN addresses this gap with a GAN-specific RNN-guided search using shared parameters, dynamic resetting, Inception-score rewards, and progressive multi-level search. Its discovered architectures achieve competitive results against hand-crafted GANs, including state-of-the-art FID scores on CIFAR-10 and STL-10, although higher-resolution search remains computationally prohibitive.
Problem
GAN backbone architecture design has received relatively less attention, and adapting NAS to GANs is difficult because GAN training is unstable and lacks a natural validation-accuracy reward.
Method
AutoGAN defines a generator search space and uses an RNN controller with parameter sharing, dynamic resetting, Inception-score rewards, and multi-level architecture search.
Results
AutoGAN discovers architectures competitive with hand-crafted GANs, achieving FID scores of 12.42 on CIFAR-10 and 31.01 on STL-10.
Takeaways & Limitations
AutoGAN provides an initial demonstration that NAS can identify effective GAN generator architectures on CIFAR-10 and STL-10.
Takeaways & Limitations
AutoGAN has not been tested on higher-resolution synthesis because search cost becomes prohibitively high; CIFAR-10 search already takes 43 hours.
Abstract
from arXiv · showhide
Neural architecture search (NAS) has witnessed prevailing success in image classification and (very recently) segmentation tasks. In this paper, we present the first preliminary study on introducing the NAS algorithm to generative adversarial networks (GANs), dubbed AutoGAN. The marriage of NAS and GANs faces its unique challenges. We define the search space for the generator architectural variations and use an RNN controller to guide the search, with parameter sharing and dynamic-resetting to accelerate the process. Inception score is adopted as the reward, and a multi-level search strategy is introduced to perform NAS in a progressive way. Experiments validate the effectiveness of AutoGAN on the task of unconditional image generation. Specifically, our discovered architectures achieve highly competitive performance compared to current state-of-the-art hand-crafted GANs, e.g., setting new state-of-the-art FID scores of 12.42 on CIFAR-10, and 31.01 on STL-10, respectively. We also conclude with a discussion of the current limitations and future potential of AutoGAN. The code is available at https://github.com/TAMU-VITA/AutoGAN
1. Introduction
AutoGAN addresses the underexplored problem of automatically designing GAN backbones despite unstable training and the lack of a natural NAS validation reward. It combines a GAN-specific search strategy with progressive search and evaluates discovered generators competitively against hand-crafted models.
- GAN backbone design has received less attention than architecture design in classification, segmentation, and pose estimation, despite its potential importance.
- NAS methods successful in classification and segmentation cannot be naively transferred to GANs because GAN training is unstable and prone to collapse.
- AutoGAN introduces a GAN-specific architecture search scheme, presented as an early effort to bring NAS to generative adversarial networks.
- The method defines a GAN search space, uses an RNN controller, and adds parameter dynamic-resetting to accelerate architecture search.
- AutoGAN uses Inception score as its reinforcement-learning reward, while discovered models also perform favorably under FID.
- Multi-level architecture search performs bottom-up, sequential search across stages using beam search.
- 12.42 FID on CIFAR-10 and 31.01 FID on STL-10 establish new state-of-the-art FID scores for the evaluated image-generation tasks.
2. Related Work
NAS searches for task-specific neural architectures through a search space, optimization algorithm, and proxy task, but prior work had not developed NAS for generative models. GANs remain difficult targets because their training is unstable and sensitive.
- NAS aims to find task-specific neural architectures rather than relying on hand-crafted designs, with prior success in image classification and structured prediction.
- A NAS system comprises a search space, an optimization algorithm, and a proxy task for evaluating candidate architectures.
- Search spaces may use macro search over whole architectures or micro search over reusable cells, while optimization can use reinforcement learning, evolutionary, Bayesian, random, or gradient-based methods.
- Multi-level NAS preserves top-performing candidates from smaller cells and extends the search to larger cells using beam search.
- GANs use competing generator and discriminator networks, but training commonly suffers from non-convergence, mode collapse, and hyperparameter sensitivity.
3. Technical Approach
AutoGAN searches generator architectures with an RNN controller while jointly training a shared GAN, using dynamic resetting and progressive multi-level search to address instability and computational cost. Its search space specifies cell connections, blocks, normalization, upsampling, and shortcuts.
- Search strategy: AutoGAN jointly considers generator and discriminator training because fixing one network while searching the other can create capacity imbalance.
- Search strategy: An RNN controller samples generator building blocks from the defined search space to construct the generator architecture.
- Search space: Each generator cell is represented by skip connections plus convolution type, normalization, upsampling, and in-cell shortcut choices.
- Search space: The controller can select multiple skip connections from preceding cells, while the search space includes pre-activation or post-activation convolution blocks.
- Search space: The cell search space includes batch, instance, or no normalization; bilinear, nearest-neighbor, or stride-2 deconvolution upsampling; and a binary in-cell shortcut.
- Proxy task: AutoGAN uses Inception score as the reinforcement-learning reward because FID is substantially more time-consuming to calculate.
- Optimization: Parameter dynamic-resetting terminates shared-GAN training when recent generator and discriminator losses have sufficiently low standard deviation, then reinitializes the shared model while retaining controller parameters.
- Optimization: The search alternates between training shared GAN parameters with a fixed controller and training controller parameters with fixed shared parameters.
4. Experiments
Experiments evaluate AutoGAN primarily on CIFAR-10 and additionally test transfer to STL-10, using shared-GAN training settings and a fixed search schedule. The datasets differ in resolution and available training data.
- Datasets: CIFAR-10 provides 50,000 training and 10,000 test images at 32 × 32 resolution, with no data augmentation.
- Datasets: STL-10 uses 5,000 labeled training images and 100,000 unlabeled images resized to 48 × 48, also without data augmentation.
- Training details: Shared GAN training uses spectral-normalization GAN settings, hinge loss, Adam optimization, and spectral normalization only on the discriminator.
- Training details: The generator and discriminator use batch sizes of 128 and 64 respectively, while the controller uses Adam with a 3.5e−4 learning rate and entropy-weight 1e−4.
- Search schedule: AutoGAN runs for 90 search iterations, training the shared GAN for 15 epochs and the controller for 30 steps per iteration.
4.1. Results on CIFAR 10
AutoGAN discovers competitive CIFAR-10 generator architectures, with design preferences including pre-activation blocks, upsampling, no normalization, and dense skip connections. Its best model achieves state-of-the-art FID while remaining computationally efficient.
- AutoGAN discovers a generator with 3 convolution blocks, preferring pre-activation over post-activation blocks.
- AutoGAN favors nearest-neighbor or bilinear upsampling over deconvolution, which may reduce checkerboard artifacts.
- AutoGAN prefers no normalization and denser skip connections spanning 2–4 convolutional layers for multi-scale feature fusion.
- AutoGAN outperforms hand-crafted SN-GAN within the same building-block search space, supporting the importance of generator structure and the search algorithm.
- The top architecture uses 1.77G FLOPs, outperforming SN-GAN at 1.69G FLOPs and matching Progressive GAN at 6.39G FLOPs.
4.2. Transferability on STL-10
The CIFAR-10-discovered AutoGAN architecture is retrained on STL-10 with its structure fixed to test transferability. It achieves the best reported FID despite slightly lagging a competitor in Inception score.
- AutoGAN retrains the CIFAR-10-discovered architecture on STL-10 without changing its structure, testing whether it transfers across datasets.
- AutoGAN slightly trails improving MMD-GAN in Inception score but obtains the best FID result among the compared STL-10 models.
- The STL-10 evaluation uses the architecture discovered on CIFAR-10, as stated in the Table 2 caption.
4.3. Ablation Study and Analysis
The ablations examine proxy-task validity, reward choice, dynamic resetting, multi-level search, and search algorithms. Results support the proxy task, dynamic resetting, and MLAS as effective design choices, while the proposed search outperforms tested random-search variants.
- Validation of Proxy Tasks: A proxy task evaluating Inception score on the shared GAN’s child model correlates positively with true evaluation, with Spearman’s rank correlation coefficient of 0.779.This provides a fair approximation of evaluation after architectures are trained from scratch for 30 epochs.
- Comparing to Using FID as Reward: Using Inception score or reciprocal FID as the controller reward compares the resulting models’ IS and FID throughout CIFAR-10 search.The supplied passage defines the comparison setup but does not report the curves’ final outcomes.
- Dynamic Resetting: Dynamic resetting achieves comparable performance to parameter sharing alone while improving training efficiency.The comparison is motivated by the risk of wasting time training a shared model after mode collapse.
- Multi-Level Architecture Search: MLAS eventually outperforms SLAS in Inception score and trains faster because it progressively searches architectures instead of always generating at final resolution.SLAS begins with a higher Inception score, but MLAS improves progressively and has consistent training advantages.
- Comparison with Random Search: Under a 48-hour CIFAR-10 search constraint, weight-sharing random search reaches IS = 8.09 and FID = 17.34, while early-stopping random search reaches IS = 7.97 and FID = 21.39.Both random-search variants are reported as inferior to AutoGAN’s proposed search algorithm.
5. Conclusions, Limitations and Discussions
AutoGAN is presented as an early NAS approach for GANs that finds competitive architectures on CIFAR-10 and STL-10. The paper identifies limited search spaces, computational costs, discriminator search, and label conditioning as directions for improvement.
- Conclusions: AutoGAN is presented as the first effort to bring NAS into GANs and identifies effective architectures on CIFAR-10 and STL-10.The paper reports competitive image-generation results against state-of-the-art hand-crafted GANs and says ablations reveal benefits from each component.
- Discussions: GAN training instability and hyperparameter sensitivity make AutoGAN more challenging than NAS for image classification, leaving substantial room for improvement.This limitation is framed as a consequence of GAN training itself.
- Limitations and Future Work: The current search space excludes some powerful GANs and should be enlarged with additional building blocks, losses, and architectural mechanisms.Suggested extensions include attention, style-based generators, relativistic discriminators, and Wasserstein loss.
- Limitations and Future Work: Higher-resolution synthesis has not been tested, and its computational cost could become prohibitively high; CIFAR-10 search already takes 43 hours.Improving search efficiency is identified as the key challenge for scaling to settings such as ImageNet.
- Limitations and Future Work: Transfer learning from low-resolution to higher-resolution search is suggested as a possible but challenging strategy for GAN image generation.The paper notes that generation must preserve and synthesize details more demandingly than classification.
- Limitations and Future Work: AutoGAN does not yet search for better discriminators and may require alternating generator–discriminator search, which would make the problem more challenging.The paper identifies discriminator architecture search as an unrealized opportunity.
- Limitations and Future Work: Future AutoGAN systems will need to incorporate labels for settings such as conditional and semi-supervised GANs.