PaperlayerView exact revision on arXiv

Source-linked AI summary

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

Yifan Jiang, Shiyu Chang, Zhangyang Wang

arXiv:2102.07074v4cs.CV

TL;DR

The paper asks whether a strong GAN can be built entirely without convolutions, addressing the limited study of pure transformers for GAN generation. It introduces TransGAN, combining transformer-based generator and discriminator designs with grid self-attention and a specialized training recipe, and reports competitive performance across multiple datasets while scaling to higher-resolution generation. The paper also identifies remaining room for improvement on 256 × 256 and extremely high-resolution tasks.

  • Problem

    The paper investigates whether a strong GAN can be built completely free of convolutions, a setting that had received little challenge despite CNNs dominating successful GANs.

  • Method

    TransGAN uses a progressively upsampling transformer generator, a multi-scale discriminator, grid self-attention, and training techniques including augmentation, modified normalization, and relative position encoding.

  • Results

    TransGAN achieves state-of-the-art performance across multiple popular datasets and produces diverse, detailed examples for higher-resolution generation.

  • Takeaways & Limitations

    Pure-transformer GANs can provide an encouraging alternative to convolutional GAN backbones while supporting higher-resolution generative tasks.

  • Takeaways & Limitations

    The paper reports substantial room for improvement on 256 × 256 generation and identifies 1024 × 1024 generation as a future direction.

Abstract

from arXiv · show

The recent explosive interest on transformers has suggested their potential to become powerful "universal" models for computer vision tasks, such as classification, detection, and segmentation. While those attempts mainly study the discriminative models, we explore transformers on some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs). Our goal is to conduct the first pilot study in building a GAN completely free of convolutions, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution, and correspondingly a multi-scale discriminator to capture simultaneously semantic contexts and low-level textures. On top of them, we introduce the new module of grid self-attention for alleviating the memory bottleneck further, in order to scale up TransGAN to high-resolution generation. We also develop a unique training recipe including a series of techniques that can mitigate the training instability issues of TransGAN, such as data augmentation, modified normalization, and relative position encoding. Our best architecture achieves highly competitive performance compared to current state-of-the-art GANs using convolutional backbones. Specifically, TransGAN sets new state-of-the-art inception score of 10.43 and FID of 18.28 on STL-10, outperforming StyleGAN-V2. When it comes to higher-resolution (e.g. 256 x 256) generation tasks, such as on CelebA-HQ and LSUN-Church, TransGAN continues to produce diverse visual examples with high fidelity and impressive texture details. In addition, we dive deep into the transformer-based generation models to understand how their behaviors differ from convolutional ones, by visualizing training dynamics. The code is available at https://github.com/VITA-Group/TransGAN.

1 Introduction

TransGAN asks whether a strong GAN can be built entirely without convolutions, replacing conventional CNN backbones with pure transformer architectures and tailored training techniques. It combines memory-conscious generator and discriminator designs with grid self-attention to support higher-resolution generation.

  • Motivation: CNNs’ local receptive fields and spatial invariance limit efficient long-range dependency modeling and adaptation to heterogeneous visual patterns.These limitations motivate exploring self-attention and transformer architectures for GAN backbones.
  • Approach: The architecture uses a memory-friendly transformer generator, a multi-scale discriminator, and grid self-attention for higher-resolution scaling.The generator progressively increases feature resolution, while the discriminator balances global contexts and local details.
  • Training: The training recipe combines data augmentation, modified layer normalization, and relative position encoding to stabilize optimization and generalization.These techniques address instability associated with both GAN and transformer training.
  • Motivation: TransGAN investigates the first GAN built entirely from transformer architectures without convolutional layers.The paper frames this as a challenge to the prevailing reliance on CNN-based generators and discriminators.
  • Results: TransGAN reports inception score 10.43 and FID 18.28 on STL-10, while also evaluating higher-resolution generation tasks.The paper presents these results as highly competitive with convolutional GANs and reports continued generation of diverse, detailed examples at higher resolution.

2 Related Works

Prior image-generation systems largely retained convolutional backbones, while transformer research established strong representation capabilities without convolution. TransGAN extends this direction to a fully transformer-based GAN and introduces architectural mechanisms for scalable image generation.

  • Generative Adversarial Networks: GANs broadly adopted fully convolutional backbones and CNN-derived components, while research explored losses, style-based generators, and other stabilization techniques.These prior designs span image translation, enhancement, and editing applications.
  • Transformers in Computer Vision: Pure vision transformers represent images as sequences of visual words and offer strong representation capability without human-defined inductive bias.Compared with CNNs, they avoid strong feature-locality and spatial-invariance biases.
  • Transformer Modules for Image Generation: Earlier generation work used transformers for autoregressive sequence generation or sparse attention, while other systems combined transformer modules with convolutional components.These approaches did not establish a GAN completely free of convolutions.
  • TransGAN: TransGAN’s pipeline uses pure-transformer generator and discriminator stages, with grid transformer blocks for scalable high-resolution generation.The illustrated 256×256 example uses patch-based processing; practical patch sizes are normally no more than 8 × 8.

3 Technical Approach: A Journey Towards GAN with Pure Transformers

TransGAN builds a convolution-free GAN from transformer components designed to manage memory while modeling both global structure and local detail. Grid self-attention and a tailored training recipe further support scalable, stable high-resolution generation.

  • Core Transformer Blocks: TransGAN uses a transformer encoder with pre-normalized self-attention and feed-forward MLP sublayers connected residually.The encoder serves as the basic building block with minimal architectural changes.
  • 3.2 Multi-scale Discriminator: The multi-scale discriminator processes differently sized patches to extract semantic structure and texture details at multiple resolutions.Its pyramid architecture uses varied patch sizes and average-pooling downsampling between stages.
  • 3.3 Grid Self-Attention: Grid self-attention partitions high-resolution feature maps into non-overlapping grids, retaining standard self-attention at lower resolutions to balance efficiency and global awareness.It is applied above 32 × 32 resolution, with a default grid size of 16 × 16.
  • 3.3 Grid Self-Attention: Grid boundary artifacts appear early but gradually vanish with sufficient training iterations and data, producing coherent final results.The authors attribute this behavior to the discriminator’s larger multi-scale receptive field, which evaluates fidelity at different scales.
  • 3.4 Exploring the Training Recipe: The training recipe combines data augmentation, modified normalization, and relative position encoding to stabilize transformer-based GAN optimization and improve performance.Differential augmentation uses Translation, Cutout, and Color, while relative position encoding is applied to both generator and discriminator attention.

4 Experiments

Experiments evaluate TransGAN against convolutional GANs across low- and high-resolution datasets, then examine augmentation, ablations, and training dynamics. TransGAN achieves strong benchmark results, while high-resolution synthesis remains an area for improvement.

  • Benchmark comparisons: TransGAN surpasses several CNN-based GANs on CIFAR-10, exceeds StyleGAN-v2 on FID with matched augmentation, and is second only to StyleGAN-v2 in inception score.
  • Benchmark comparisons: TransGAN sets new state-of-the-art IS and FID results on STL-10, a larger and higher-resolution benchmark than CIFAR-10.The results suggest transformer-based architectures benefit more notably from larger-scale data than CNNs.
  • Benchmark comparisons: TransGAN reaches a FID score of 5.28 on CelebA at 128 × 128, slightly outperforming StyleGAN-v2 while producing diverse visual details.
  • High-resolution synthesis: 10.28 FID on CelebA-HQ 256 × 256 and 8.94 FID on LSUN Church demonstrate competitive high-resolution results with rich texture details.
  • Training techniques: Data augmentation improves TransGAN on CIFAR-10 from IS 8.36 to 9.02 and FID 22.53 to 9.26, a larger gain than for most CNN-based GANs.
  • Ablation study: Grid self-attention and the added multi-scale discriminator, modified normalization, and relative position encoding each improve ablation performance, yielding final FIDs of 5.01, 5.28, and 8.94.These values correspond to CelebA 64 × 64, CelebA 128 × 128, and LSUN Church 256 × 256, respectively.

5 Conclusions, Limitation, and Discussions of Broad Impact

TransGAN is presented as a first pure-transformer GAN pilot with competitive performance and higher-resolution scaling. The authors identify remaining challenges in 256 × 256 and larger generation, while noting both potential data-engine applications and misuse risks.

  • The model achieves state-of-the-art performance across multiple popular datasets and scales to higher-resolution generative tasks.
  • The authors identify 256 × 256 and extremely high-resolution generation, such as 1024 × 1024, as future directions.
  • The generative model may serve as a data engine for data collection and may help avoid privacy concerns through synthesized images.
  • Abuse of advanced generative models may create fake media materials, requiring future caution.

A Implementation of Data Augmentation

TransGAN uses differentiable augmentation with translation, cutout, and color operations, while AutoGAN omits translation because it harms CNN-based performance with full data.

  • TransGAN applies Translation, Cutout, and Color augmentation with probabilities 1.0, 0.3, and 1.0, respectively.
  • AutoGAN uses only Cutout and Color augmentation because Translation augmentation hurts CNN-based GAN performance when 100% data is utilized.

B Detailed Architecture Configurations

The architecture configurations specify Transformer and Grid Blocks, bicubic upsampling, feature-map shapes, and discriminator operations across datasets. The discriminator ultimately predicts real or fake from a CLS token.

  • The architecture configurations for different datasets are listed in Tables 4, 5, 6, and 7.
  • Generator configurations: A Transformer Block consists of self-attention, normalization, and feed-forward MLP components.
  • Generator configurations: A Grid Block replaces standard self-attention with Grid Self-Attention using grid size 16, while upsampling defaults to bicubic upsampling.
  • Discriminator configurations: The discriminator splits patches, applies linear transformation, concatenates stage outputs with another sequence, and adds a final CLS token.
  • Discriminator configurations: The discriminator uses only the CLS token through a classification head to predict real or fake.

C Failure Cases Analysis

TransGAN’s high-resolution failure cases are concentrated in wearing-glasses examples and side faces, pointing to data-distribution imbalance and insufficient training data as possible issues.

  • TransGAN has inferior FID scores to state-of-the-art ConvNet-based GANs on high-resolution synthesis tasks.
  • On CelebA-HQ at 256 × 256 resolution, representative failure examples are mostly from the wearing-glasses class and side faces.
  • The observed failure pattern indicates possible imbalanced data distribution and insufficient training data issues.

D Training Cost

Training TransGAN across datasets and resolutions from 32 × 32 to 256 × 256 incurs substantial computational cost, with the largest experiment taking around three days on 32 V100 GPUs.

  • Around 3 days and 32 V100 GPUs are required for the largest TransGAN experiment.

E Memory Cost Comparison

The paper compares GPU memory usage for standard self-attention and grid self-attention under controlled inference settings. Even with a reduced standard-attention model, grid self-attention uses substantially less memory.

  • Experimental setup: The comparison evaluates standard and grid self-attention inference costs on an Nvidia V100 with batch size 1 and without gradient computation.For 256 × 256 tasks, the standard-attention model was reduced to fit GPU memory.
  • Experimental setup: Architecture configurations are listed for STL-10 and CelebA at 128 × 128 resolution.The supplied table captions identify the corresponding configuration tables.
  • Memory comparison: Even after reducing the standard-attention model, it costs significantly more GPU memory than grid self-attention.The comparison includes 128 × 128 and 256 × 256 resolution tasks.

F Visual Examples

TransGAN is illustrated through latent-space interpolations and high-resolution samples on CelebA at 256 × 256 resolution. The reported examples show impressive detail and diversity.

  • Visual quality: The high-resolution visual examples produced by TransGAN show impressive details and diversity.Figures 8 and 9 provide additional high-resolution examples.
  • Latent-space interpolation: TransGAN produces latent-space interpolations on the CelebA 256 × 256 dataset.The supplied passage identifies Figure 8 as the latent-space interpolation visualization.
  • High-resolution examples: TransGAN provides high-resolution representative visual examples on CelebA at 256 × 256 resolution.The supplied passage identifies Figure 9 as the representative-example visualization.
Loading 2102.07074v4…