PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling up GANs for Text-to-Image Synthesis

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, Taesung Park

arXiv:2303.05511v2cs.CVcs.GRcs.LG

TL;DR

GANs have struggled to scale to open-world, web-scale text-to-image synthesis because naive capacity increases destabilize training. GigaGAN introduces architectural and training techniques for stable billion-scale GAN training, achieving competitive quality with much faster inference, high-resolution synthesis, and latent editing. The authors conclude that GANs remain viable for large-scale text-to-image generation, although visual quality is not yet comparable to production-grade systems.

  • Problem

    GANs have remained difficult to scale to complex open-world datasets because increasing StyleGAN2 capacity causes unstable training.

  • Method

    GigaGAN combines dynamic filter selection, self- and cross-attention, and multi-scale training in a modular two-stage GAN architecture.

  • Results

    GigaGAN achieves a zero-shot FID of 9.09 on COCO2014, generates 512px images in 0.13 seconds, and synthesizes 4K images in 3.66 seconds.

  • Takeaways & Limitations

    The results demonstrate that GANs remain a viable option for large-scale text-to-image synthesis while retaining efficient inference and controllable latent-space editing.

  • Takeaways & Limitations

    The visual quality of GigaGAN is not yet comparable to production-grade models such as DALL·E 2, particularly for photorealism and text-to-image alignment.

Abstract

from arXiv · show

The recent success of text-to-image synthesis has taken the world by storm and captured the general public's imagination. From a technical standpoint, it also marked a drastic change in the favored architecture to design generative image models. GANs used to be the de facto choice, with techniques like StyleGAN. With DALL-E 2, auto-regressive and diffusion models became the new standard for large-scale generative models overnight. This rapid shift raises a fundamental question: can we scale up GANs to benefit from large datasets like LAION? We find that naÏvely increasing the capacity of the StyleGAN architecture quickly becomes unstable. We introduce GigaGAN, a new GAN architecture that far exceeds this limit, demonstrating GANs as a viable option for text-to-image synthesis. GigaGAN offers three major advantages. First, it is orders of magnitude faster at inference time, taking only 0.13 seconds to synthesize a 512px image. Second, it can synthesize high-resolution images, for example, 16-megapixel pixels in 3.66 seconds. Finally, GigaGAN supports various latent space editing applications such as latent interpolation, style mixing, and vector arithmetic operations.

1. Introduction

GigaGAN addresses the instability and scaling challenges that have limited GANs for open-world text-to-image synthesis. Its architecture enables billion-scale training, fast inference, high-resolution synthesis, modular upsampling, and latent-space editing.

  • Motivation: Diffusion and autoregressive models deliver high quality but rely on iterative inference, whereas GANs generate images through a single forward pass.Iterative inference enables stable training with simple objectives but increases inference cost.
  • Motivation: Naively scaling StyleGAN2 destabilizes training, while complex open-world datasets remain challenging for GANs.The paper asks whether GANs can benefit from large-scale data and compute, and what barriers prevent further scaling.
  • Approach: GigaGAN stabilizes a billion-parameter GAN using filter-bank capacity scaling, attention layers, and a new multi-scale training scheme.The method combines sample-specific linear combinations of filters with image-only self-attention, image-text cross-attention, and multi-scale outputs.
  • Applications: A modular two-stage pipeline generates 64 × 64 images before upsampling to 512 × 512, and the upsampler can enhance outputs from base diffusion models.The text-conditioned GAN upsampler is described as efficient and higher-quality despite not being trained on diffusion images.
  • Results: 36× larger than StyleGAN2 and 6× larger than StyleGAN-XL and XMC-GAN, GigaGAN achieves a zero-shot FID of 9.09 on COCO2014.The reported FID is lower than those of DALL·E 2, Parti-750M, and Stable Diffusion.
  • Results: 0.13 seconds produces a 512px image, while 3.7 seconds produces a 4096px image.Figure 1 also presents selected 2K and 4K examples.

2. Related Works

Related work established strong open-world text-to-image synthesis with diffusion and autoregressive models, but their iterative sampling remains computationally expensive. GigaGAN targets this gap by applying GANs to data-rich web-scale training and efficient super-resolution.

  • Text-to-image synthesis: Diffusion and autoregressive models produce strong open-world text-to-image results but require time-consuming iterative sampling.Large-scale language encoders contribute to photorealistic diffusion outputs, while consecutive reverse processes remain expensive.
  • Text-to-image synthesis: Sampling-acceleration methods reduce steps or reuse features, but latent diffusion still performs computationally expensive consecutive reverse processes.These costs limit the use of large-scale text-to-image models in interactive applications.
  • GAN-based image synthesis: GANs have supported natural-image synthesis, text-to-image generation, image translation, and image editing, with StyleGAN-family models showing strong synthesis ability.The passage situates GigaGAN within established GAN image-synthesis research.
  • Large-scale GAN training: GigaGAN changes GAN training from relatively small closed-world datasets to extremely data-rich web-crawled text-image pairs such as LAION2B-en and COYO-700M.Prior GAN text-to-image models used datasets including CUB-200, MSCOCO, and LN-OpenImages.
  • Related GAN methods: StyleGAN-T and GALIP pursue similar large-scale GAN text-to-image goals with complementary insights.The paper identifies them as concurrent methods.
  • Super-resolution: Large-scale text-to-image systems commonly use cascaded generation and super-resolution, but diffusion-based upsamplers require many iterations.These pipelines generate at low resolution and progressively upsample to higher resolutions.

3. Method

GigaGAN scales text-to-image GANs through condition-dependent convolution filters, interleaved attention, multi-scale supervision, and a staged upsampler. These designs target instability and limited expressivity while supporting high-capacity generation.

  • Expressive convolution: GigaGAN dynamically selects convolution filters from a text-conditioned filter bank, increasing kernel expressivity without widening every convolution layer.A style vector predicts weights that aggregate multiple filters; selection occurs once per layer, so its compute is decoupled from image resolution.
  • Attention and stability: The generator interleaves self-attention and cross-attention with convolution to model long-range image relationships and local text conditioning.The approach addresses convolution’s limited receptive field while preserving the convolutional backbone.
  • Text conditioning: Text conditioning supplies both a style code and local word descriptors: global CLIP text features and latent code z produce w, while local descriptors drive cross-attention.The style code modulates synthesis, and local descriptors provide token-level conditioning to the generator.
  • Multi-scale generation: The synthesis network outputs a five-level image pyramid, with each level independently contributing to GAN training across resolutions.The pyramid levels have resolutions 64, 32, 16, 8, and 4, and multi-scale supervision improves the use of low-resolution generator blocks.
  • Discriminator design: The discriminator compares text and image branches while making multi-scale predictions, and adds CLIP-based vision-aided adversarial losses for stability.Its image branch processes an image pyramid, while text features modulate image features in the discriminator.
  • GAN-based upsampler: A separate GAN-based upsampler converts base outputs into high-resolution images, enabling a two-stage pipeline with higher-capacity 64px generation under fixed resources.The upsampler is designed as a modular component and is intended to serve as a drop-in replacement for other models’ superresolution stages.

4. Experiments

The experiments evaluate GigaGAN through component ablations, comparisons with text-to-image and distilled diffusion models, super-resolution tests, and latent-space manipulation. Results show scalable performance, fast inference, high-resolution synthesis, and controllable editing, while FID does not always imply better visual quality.

  • Effectiveness of proposed components: Each added component consistently improves the ablation metrics, whereas simply widening the StyleGAN2 baseline does not improve FID or CLIP scores.The higher-capacity final formulation achieves better performance, supporting improved scalability.
  • Evaluation caveat: A lower FID does not necessarily mean better generated-image quality, motivating further research on evaluation metrics for text-to-image models.The paper describes this issue as a possible corner case of zero-shot FID on COCO2014.
  • Super-resolution: GigaGAN's upsampler outperforms other upsamplers on realism, text alignment, and ground-truth closeness, and achieves the best IS and FID on unconditional ImageNet super-resolution.The ImageNet comparison uses a single feedforward pass.
  • Controllable image synthesis: Style mixing, prompt interpolation, and prompt mixing use disentangled latent, text-embedding, and style spaces for coarse-to-fine and layout-preserving control.Prompt mixing applies recomputed embeddings and style codes to later generator layers, while cross-attention localizes style to the target object.

5. Discussion and Limitations

The discussion concludes that GigaGAN demonstrates GAN scalability for text-to-image synthesis while remaining behind production-grade models in visual quality and alignment. The authors view the architecture as a promising design space whose performance may improve with larger models.

  • Scalability: GigaGAN scales to model sizes that enable text-to-image synthesis, answering the paper's scalability question affirmatively.The conclusion contrasts this capability with the still-limited visual quality of the current results.
  • Limitations: GigaGAN's visual quality is not yet comparable to production-grade models such as DALL·E 2, including failures in photorealism and text-to-image alignment.The cited failures use the same input prompts as the DALL·E 2 paper.
  • Implications: The architecture achieves competitive results while being orders of magnitude faster and supporting latent interpolation and stylization.The authors describe these results as extending beyond what a naïve scaling approach permits.

Appendices

The appendices provide training and evaluation details, ImageNet results with visual comparisons, and additional text-to-image samples compared with LDM, Stable Diffusion, and DALL·E 2.

  • Appendix contents: Appendix A gives training and evaluation details, Appendix B reports ImageNet results with visual comparisons, and Appendix C provides additional text-to-image comparisons.Appendix C compares samples with LDM, Stable Diffusion, and DALL·E 2.

A. Training and evaluation details

Training uses filtered web-scale image–text data, while evaluation applies specified cropping, resizing, and FID procedures across text-to-image and super-resolution settings.

  • Training data and preprocessing: GigaGAN is trained on LAION2B-en and COYO-700M after filtering image–text pairs by resolution, CLIP score, aesthetics score, and watermarks.Training images are center-cropped and resized to 64 × 64 with PIL.LANCZOS.
  • Text-to-image evaluation: Text-to-image evaluation uses 40,504 real and 30,000 generated COCO2014 validation images resized to 299 × 299 for clean-fid calculation.The same evaluation passage specifies center cropping before resizing.
  • Super-resolution evaluation: Super-resolution experiments use the stated cropping strategy and resize images to 64 × 64 for the base generator or 256 × 256 for the super-resolution stack.The procedure follows Kang et al.'s training and evaluation protocol for comparison with other generative models.
  • Metrics: Super-resolution FID uses 50k generated images without truncation tricks, with real and generated images upsampled to 299 × 299 before feature-based evaluation.Precision and Recall also uses pre-calculated real-image features and 50k generated images.

A.3. Super-resolution results

This section describes evaluation procedures for ImageNet super-resolution comparisons, including preprocessing and alignment with prior methods.

  • The model is trained on ImageNet using the configuration specified in Table A2.
  • The comparison with SR3 and LDM follows the evaluation procedures described in their respective papers.

B.1. Qualitative results

The paper evaluates a class-conditional GAN on ImageNet and reports competitive generation quality against large diffusion and transformer models.

  • The model is a class-conditional GAN evaluated on ImageNet with an apples-to-apples comparison pipeline.The design includes L2 self-attention, style-adaptive convolution kernels, matching-aware loss, and a wider synthesis network.
  • Table A1 compares 256px ImageNet synthesis against large diffusion and transformer models.Some shaded methods use a pretrained ImageNet classifier, which may favor automated metrics.

B.2. Quantitative results

On ImageNet, GigaGAN is reported to produce competitive visual quality, while its visual strengths differ from those indicated by FID alone.

  • GigaGAN appears to capture overall structure and fine-grained details better than StyleGAN-XL in the shown ImageNet examples.The examples highlight monarch wing patterns and arctic fox fur.
  • Table A2 reports the hyperparameters used for GigaGAN training.The table defines abbreviations including PD, R1, LPIPS, AdamW, and CLIP-ViT-B/32-V.
  • FID values for ADM-G-U, LDM-4-G, GigaGAN, and StyleGAN-XL are 4.01, 3.60, 3.45, and 2.32, respectively.StyleGAN-XL has the lowest FID, although the accompanying visual comparison describes weaker overall structure.

C. Text-to-image synthesis results

GigaGAN uses truncation to trade diversity for text alignment and supports several latent- and prompt-based editing operations, while remaining faster than diffusion models.

  • C.1. Truncation trick at inference: At inference, GigaGAN applies a truncation trick analogous to guidance techniques used in diffusion models.
  • C.1. Truncation trick at inference: Truncation interpolates latent vectors toward distribution means, trading diversity for fidelity; ψ = 1.0 applies no truncation and ψ = 0.0 collapses to the mean.For text-conditioned generation, the method also uses the mean latent conditioned on the prompt.
  • C.1. Truncation trick at inference: More truncation increases CLIP score while increasing FID because diversity is reduced.
  • C.2. Latent and prompt editing: GigaGAN supports style mixing by combining coarse style from one sample with fine style from another while preserving the prompt.
  • C.2. Latent and prompt editing: Prompt interpolation combines text embeddings and style vectors from different prompts to create smooth transitions with similar layouts.
  • C.2. Latent and prompt editing: Prompt mixing changes style through new text prompts applied to later generator layers, achieving layout-preserving style control.Cross-attention automatically localizes the style control.
  • C.2. Latent and prompt editing: GigaGAN produces diverse images for a given prompt across subjects and artistic styles.
  • C.3. Comparisons with diffusion models: Compared with diffusion models, GigaGAN is faster, but its outputs can fall behind in structural coherency and fine details.The examples report issues involving bed-frame legs, vase symmetry, zebra stripes, skateboard shapes, eyes, facial expression, and attire.
Loading 2303.05511v2…