Source-linked AI summary
Self-Attention Generative Adversarial Networks
Han Zhang, Ian Goodfellow, Dimitris Metaxas, Augustus Odena
TL;DR
Convolutional GANs struggle to model geometric and long-range dependencies in some image classes. SAGAN adds self-attention and generator spectral normalization, achieving state-of-the-art ImageNet image synthesis with an Inception score of 52.52 versus 36.8 and FID of 18.65 versus 27.62.
Problem
Convolutional GANs struggle to capture geometric patterns and long-range dependencies in some ImageNet image classes.
Method
SAGAN adds self-attention to GANs for modeling relationships across distant image regions and applies spectral normalization to both generator and discriminator.
Results
52.52 versus 36.8 Inception score and 18.65 versus 27.62 FID establish SAGAN as state-of-the-art for class-conditional ImageNet image generation.
Takeaways & Limitations
Self-attention enables coordinated fine details and global geometric constraints, while generator spectral normalization stabilizes GAN training.
Takeaways & Limitations
The Inception score does not assess realism of details or intra-class diversity.
Abstract
from arXiv · showhide
In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN achieves the state-of-the-art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.
1. Introduction
The introduction identifies limitations of convolutional GANs in modeling some multi-class image categories and proposes SAGAN, which adds self-attention and generator spectral normalization. Experiments on ImageNet report improved synthesis metrics and attention patterns aligned with object shapes.
- Motivation: Convolutional GANs have difficulty modeling some image classes in multi-class datasets such as ImageNet.The passage contrasts this difficulty with success on classes distinguished more by texture than structure.
- Self-attention mechanism: SAGAN introduces self-attention into convolutional GANs to model long-range, multi-level dependencies across image regions.The generator coordinates fine details with distant portions of an image, while the discriminator can enforce global image-structure consistency.
- Generator conditioning: Spectral normalization is applied to the GAN generator to enforce good conditioning and improve GAN performance.This extends its prior use from the discriminator to the generator.
- Experimental results: 52.52: SAGAN boosts the best reported Inception score from 36.8 to 52.52 on ImageNet.This result is reported alongside extensive ImageNet experiments validating the proposed attention and stabilization techniques.
- Experimental results: 18.65: SAGAN reduces Fréchet Inception distance from 27.62 to 18.65 on ImageNet.Attention visualizations show that the generator leverages neighborhoods corresponding to object shapes rather than fixed local regions.
2. Related Work
Prior work spans GAN-based image generation and efforts to improve training stability and sample diversity through architectural, objective, regularization, and heuristic changes. Attention mechanisms, particularly self-attention, provide a framework for modeling global dependencies, which SAGAN applies to image representations.
- Generative Adversarial Networks: GANs have achieved success in image-to-image translation, image super-resolution, and text-to-image synthesis.
- Generative Adversarial Networks: Researchers have pursued improved GAN stability and sample diversity through new architectures, modified objectives and dynamics, regularization, and heuristic tricks.
- Attention Models: Attention mechanisms support models that capture global dependencies, while self-attention computes each sequence position’s response by attending to all positions in that sequence.
- Attention Models: SAGAN learns to efficiently find global, long-range dependencies within internal representations of images.
3. Self-Attention Generative Adversarial Networks
SAGAN adapts a non-local model to add self-attention to both the generator and discriminator, enabling long-range dependency modeling beyond convolutional local neighborhoods. Its attention module combines non-local evidence with residual input features and gradually increases non-local influence during training.
- Self-attention architecture: SAGAN introduces self-attention into both the generator and discriminator to address convolutional layers’ inefficiency in modeling long-range image dependencies.The generator and discriminator are trained alternately using the hinge version of the adversarial loss.
- Self-attention architecture: The attention module transforms hidden features into spaces f and g, then uses attention weights β_j,i to determine how strongly location i contributes to synthesizing region j.The hidden features contain C channels across N feature locations, and the attention output has the same C×N dimensionality.
- Implementation: For memory efficiency, SAGAN reduces the intermediate channel count to C/8 in all experiments without observing significant performance degradation after a few ImageNet training epochs.The reduction is implemented through learned 1×1 convolution weight matrices.
- Self-attention architecture: The module adds the input feature map to a scaled attention output, with a learnable scalar γ initialized to 0.This residual formulation lets the network initially rely on local features before increasing the contribution of non-local evidence.
4. Techniques to Stabilize the Training of GANs
The section investigates spectral normalization and TTUR as techniques for stabilizing GAN training on challenging datasets. It extends spectral normalization to the generator and advocates TTUR for slow learning caused by regularized discriminators.
- Stabilization techniques: The authors apply spectral normalization to both the generator and discriminator to stabilize GAN training.They also confirm TTUR’s effectiveness and recommend it for slow learning in regularized discriminators.
- Spectral normalization: Spectral normalization constrains the discriminator’s Lipschitz constant by restricting each layer’s spectral norm without requiring extra hyperparameter tuning.Setting all weight-layer spectral norms to 1 consistently performs well in practice.
- Spectral normalization: Generator spectral normalization prevents parameter-magnitude escalation and unusual gradients by improving generator conditioning.The authors find that normalizing both networks enables fewer discriminator updates per generator update.
- Two-timescale update rule: Regularized discriminators often slow GAN learning and typically require multiple discriminator updates, such as 5, for each generator update.TTUR uses separate learning rates for the generator and discriminator and is advocated to address this slow learning.
5. Experiments
Experiments on ImageNet evaluate GAN stabilization, self-attention placement, and class-conditional generation. SAGAN improves training behavior, benefits from attention at larger feature maps, and achieves state-of-the-art ImageNet metrics.
- Evaluation metrics: The evaluation uses Inception score (IS) and Fréchet Inception distance (FID), while noting that IS has serious limitations.Higher IS indicates better image quality, but IS primarily assesses confident class recognition and diversity across classes.
- Stabilization techniques: FID and IS begin declining at the 260k-th iteration with balanced updates, whereas SN on G/D+TTUR improves image quality monotonically throughout training.The comparison evaluates spectral normalization on the generator and discriminator together with imbalanced learning rates.
- Self-attention placement: Self-attention at middle-to-high feature maps, such as feat32 and feat64, outperforms attention at low-level maps, such as feat8 and feat16.FID improves from 22.98 for SAGAN, feat8 to 18.28 for SAGAN, feat32 because larger maps provide more evidence and conditioning freedom.
- Self-attention placement: Self-attention blocks outperform residual blocks with the same number of parameters.Replacing attention with a residual block raises FID from 22.98 to 42.13 on 8×8 maps and from 18.28 to 27.33 on 32×32 maps.
- ImageNet comparison: 52.52 is SAGAN’s Inception score, improving the best published score from 36.8; its FID is 18.65 and intra FID is 83.7.SAGAN achieves the best reported Inception score, intra FID, and FID on class-conditional ImageNet generation.
- ImageNet comparison: SAGAN’s lower intra FID advantage is greater for geometrically complex classes than for texture-dominated classes such as valley, stone wall, and coral fungus.Self-attention complements convolution for long-range geometric dependencies but behaves similarly to local convolution for simple texture.
6. Conclusion
The conclusion presents SAGAN as a self-attention GAN that models long-range dependencies and achieves state-of-the-art class-conditional ImageNet generation, while spectral normalization stabilizes training and TTUR accelerates regularized discriminator training.
- Conclusion: SAGAN incorporates self-attention into GANs to model long-range dependencies and achieves state-of-the-art class-conditional image generation on ImageNet.The conclusion identifies self-attention as effective for long-range dependency modeling.
- Conclusion: Spectral normalization applied to the generator stabilizes GAN training.The conclusion explicitly attributes improved training stability to generator spectral normalization.
- Conclusion: TTUR speeds up training of regularized discriminators.The conclusion reports TTUR as a training-speed improvement for regularized discriminators.