PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleSwin: Transformer-based GAN for High-resolution Image Generation

Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, Baining Guo

arXiv:2112.10762v2cs.CV

TL;DR

High-resolution image generation lacks transformer GANs matching ConvNets, motivating StyleSwin. It combines Swin local attention with style-based generation, double attention, positional encoding, and wavelet discrimination, achieving state-of-the-art quality including FID 4.43 on CelebA-HQ 1024 and FID 5.07 on FFHQ-1024. The results support the promise of transformers for high-resolution synthesis, although practical throughput remains lower than StyleGAN2 and bCR is ineffective at 1024 × 1024.

  • Problem

    Transformers had not demonstrated image-generation ability on par with ConvNets at high resolutions, while efficient attention and high-resolution synthesis remained open challenges.

  • Method

    StyleSwin builds a style-based GAN with Swin local attention, double attention, sinusoidal positional encoding, and a wavelet discriminator for high-resolution synthesis.

  • Results

    StyleSwin achieves state-of-the-art quality across multiple benchmarks, with FID 4.43 on CelebA-HQ 1024 and FID 5.07 on FFHQ-1024.

  • Takeaways & Limitations

    The results support using transformers for high-resolution image generation, including self-attention at high-resolution stages for high-fidelity detail synthesis.

  • Takeaways & Limitations

    StyleSwin reaches 11.05 imgs/sec versus StyleGAN2’s 40.05 imgs/sec on a single V100 GPU, and bCR is ineffective at 1024 × 1024.

Abstract

from arXiv · show

Despite the tantalizing success in a broad of vision tasks, transformers have not yet demonstrated on-par ability as ConvNets in high-resolution image generative modeling. In this paper, we seek to explore using pure transformers to build a generative adversarial network for high-resolution image synthesis. To this end, we believe that local attention is crucial to strike the balance between computational efficiency and modeling capacity. Hence, the proposed generator adopts Swin transformer in a style-based architecture. To achieve a larger receptive field, we propose double attention which simultaneously leverages the context of the local and the shifted windows, leading to improved generation quality. Moreover, we show that offering the knowledge of the absolute position that has been lost in window-based transformers greatly benefits the generation quality. The proposed StyleSwin is scalable to high resolutions, with both the coarse geometry and fine structures benefit from the strong expressivity of transformers. However, blocking artifacts occur during high-resolution synthesis because performing the local attention in a block-wise manner may break the spatial coherency. To solve this, we empirically investigate various solutions, among which we find that employing a wavelet discriminator to examine the spectral discrepancy effectively suppresses the artifacts. Extensive experiments show the superiority over prior transformer-based GANs, especially on high resolutions, e.g., 1024x1024. The StyleSwin, without complex training strategies, excels over StyleGAN on CelebA-HQ 1024, and achieves on-par performance on FFHQ-1024, proving the promise of using transformers for high-resolution image generation. The code and models will be available at https://github.com/microsoft/StyleSwin.

1. Introduction

StyleSwin explores a transformer-based GAN for high-resolution image generation by combining efficient local attention with architectural adaptations for capacity, receptive field, and positional information. It addresses high-resolution blocking artifacts with a wavelet discriminator and achieves strong benchmark performance, including FID 4.43 on CelebA-HQ 1024 and FID 5.07 on FFHQ-1024.

  • 1. Introduction: StyleSwin uses Swin transformers’ window-based local attention to balance computational efficiency and modeling capacity for scalable high-resolution synthesis.The approach targets resolutions such as 1024 × 1024 while retaining expressive transformer representations across image scales.
  • 1. Introduction: Double attention attends to both local and shifted windows, enlarging the receptive field without much computational overhead.This adaptation is designed to improve generator capacity beyond the limited context of local attention alone.
  • 1. Introduction: Sinusoidal positional encoding supplies absolute position information missing from window-based transformers for image synthesis.The paper identifies global positioning as a key feature for generation and applies positional encoding at each layer.
  • 1. Introduction: Wavelet discrimination examines spectral discrepancies to suppress blocking artifacts caused by block-wise local attention breaking spatial coherency.The authors motivate this treatment by observing that high-resolution synthesis exhibits artifacts when attention is computed independently across blocks.
  • 1. Introduction: 4.43 FID on CelebA-HQ 1024 outperforms prior works including StyleGAN, while 5.07 FID on FFHQ-1024 approaches StyleGAN2.The method achieves these results without complex training strategies, alongside state-of-the-art quality on multiple benchmarks.

2. Related Work

Recent GAN methods have largely improved image generation through architectural designs that increase model capacity, while convolutional backbones remain prevalent. The related work positions transformer backbones as a potential alternative because they are regarded as having stronger modeling capacity.

  • 2. Related Work: Recent GANs increasingly rely on architecture design with enhanced capacity rather than primarily on adversarial-training stabilization.Earlier methods emphasized regularization and adversarial loss design, whereas later prominent methods focus on architectural improvements.
  • 2. Related Work: Contemporary GAN-based methods commonly use convolutional backbones, although transformers are regarded as superior in modeling capacity.This contrast motivates investigating transformers for high-resolution image generation.

3. Method

StyleSwin builds a scalable transformer GAN by combining Swin’s local attention with style injection, double attention, and positional information. The method addresses high-resolution blocking artifacts with artifact-suppression strategies, finding the wavelet discriminator most effective.

  • Transformer-based GAN architecture: StyleSwin uses Swin transformer blocks as a scalable generator backbone, balancing local-attention efficiency with modeling capacity for high-resolution synthesis.The generator progressively upsamples feature maps through transformer blocks, while block-wise attention has linear computational complexity in image size.
  • Style injection: Style codes mapped from latent z modulate transformer feature maps through style injection, substantially improving generative modeling capacity.The study compares AdaNorm, modulated MLP, and cross-attention variants; AdaIN provides the best FFHQ-256 generation quality, while AdaBN fails to converge.
  • Positional encoding: Relative and sinusoidal absolute positional encodings jointly provide local relative positions and global positions across scales.The method uses sinusoidal positional encoding because it admits translation invariance, while relative positional encoding operates within each transformer block.
  • Blocking artifact in high-resolution synthesis: At 1024×1024 resolution, window-wise attention produces blocking artifacts by disrupting spatial coherency, motivating dedicated suppression methods.The artifacts correlate with local-attention window size and are distinct from checkerboard artifacts caused by transposed convolution.
  • Artifact suppression: Among artifact-suppression approaches, the wavelet discriminator achieves the lowest FID and the most visually pleasing results while examining frequency discrepancies across scales.Other approaches can remove visual artifacts but may suffer from train-test gaps, reduced fine-detail synthesis, or degraded FID.

4. Experiments

Experiments evaluate StyleSwin on face and church datasets at 256×256 and 1024×1024 resolutions using FID and qualitative samples. The model achieves strong image quality, while ablations and efficiency comparisons examine its components, artifacts, and computational cost.

  • 4.1. Experiment setup: StyleSwin is evaluated on CelebA-HQ, FFHQ, and LSUN Church, covering 256×256 and 1024×1024 image synthesis.CelebA-HQ and FFHQ are tested at both resolutions, while LSUN Church is tested at 256×256.
  • 4.1. Experiment setup: FID measures distribution discrepancy between generated and real images, with lower scores indicating better generation quality.FFHQ and LSUN Church use 50,000-image validation sets; CelebA-HQ uses all training samples as the reference set.
  • 4.3. Main results: 4.43 FID on CelebA-HQ 1024 sets a record and outperforms prior works including StyleGAN.On FFHQ-1024, StyleSwin considerably closes the gap with StyleGAN2 without complex training strategies or additional regularization.
  • 4.3. Main results: StyleSwin generates diverse high-resolution faces and coherent church scenes with global geometry and high-fidelity details.Compared with StyleGAN2, it has a larger model size but lower FLOPs, indicating lower theoretical computational cost.
  • 4.4. Ablation study: Style injection, double attention, wavelet discrimination, and sinusoidal positional encoding each improve FID in the ablation study.The wavelet discriminator suppresses blocking artifacts and strengthens high-frequency supervision; bCR improves FFHQ-256 FID by 2.69 but is not effective at 1024×1024.
  • 4.4. Ablation study: Regularization schemes for high-resolution synthesis remain an open direction because bCR was not effective at 1024×1024.The authors leave high-resolution regularization schemes for future work.

5. Conclusion

The paper presents StyleSwin as a transformer-based GAN for high-resolution image generation. Its local and double attention, positional encoding, and wavelet discriminator support competitive quality across resolutions while addressing computational and artifact-related challenges.

  • 5. Conclusion: StyleSwin combines efficient local attention with double attention that compensates for its limited receptive field.The generator also uses sinusoidal positional encoding to provide global patch positioning.
  • 5. Conclusion: StyleSwin consistently outperforms leading convolutional approaches on 256×256 datasets.The conclusion attributes this to the increased expressivity of transformers.
  • 5. Conclusion: A wavelet discriminator penalizes spectral discrepancy to address blocking artifacts in high-resolution synthesis.The method ultimately approaches the performance of the best-performing ConvNets at 1024×1024 resolution.
  • 5. Conclusion: The results support further study of transformers for generative modeling.The conclusion frames StyleSwin as evidence that transformers can deliver compelling high-resolution image quality.

Appendix A. Implementation Details

Implementation uses standard adversarial training with R1 regularization, dataset-specific settings, and practical stabilization techniques. Training includes TTUR, discriminator spectral normalization, optional bCR augmentation, and substantial GPU resources for 1024×1024 synthesis.

  • Training: StyleSwin uses the standard non-saturating logistic GAN loss with an R1 gradient penalty.The discriminator measures sample realism, while the generator produces samples intended to be recognized as real.
  • Training: R1 regularization is applied every 16 iterations, with dataset-dependent weight γ.This regularization penalizes gradients on real data to encourage local stability.
  • Optimization: TTUR uses a discriminator learning rate four times larger than the generator’s, alongside spectral normalization on the discriminator.The learning rate is linearly decayed for all datasets except CelebA-HQ 1024.
  • Data augmentation: At 256×256, FFHQ and CelebA-HQ training benefits from balanced consistency regularization with flipping, color, translation, and cutout augmentations.The listed augmentation probabilities are 0.5, 1.0, 1.0, and 1.0, respectively.
  • Compute: Training StyleSwin at 1024×1024 takes about 14 days on eight 32GB Tesla V100 GPUs.The implementation uses PyTorch, with hyperparameters summarized in Table 7.

Appendix B. Detailed Architecture

StyleSwin begins from a learned constant and progressively upsamples through transformer blocks at successive resolution scales. Each scale uses two transformer blocks, including double-attention configurations.

  • Architecture: StyleSwin starts from a constant 4×4×512 input and hierarchically upsamples feature maps with transformer blocks.Two transformer blocks model each resolution scale.
  • Architecture: A double-attention block configuration specifies channel dimension, window size, and number of attention heads.For example, “512-d, 4-w, 16-h” denotes 512 channels, window size 4, and 16 attention heads.
  • Architecture: Bilinear upsampling is followed by feedforward MLPs at the indicated channel dimension.The architecture table records these operations for each generation scale.

Appendix C. The Modeling Capacity of Double Attention

The appendix evaluates double attention’s modeling capacity through image reconstruction and reports improved capacity over vanilla Swin attention.

  • Autoencoder setup: The autoencoder combines a fixed MoCo-pretrained ResNet-50 encoder with a transformer-based decoder that hierarchically upsamples a 16 × 16 feature map.The encoder preserves both low-level and high-level information before decoding.
  • Generator architectures: StyleSwin-256 and StyleSwin-1024 use the detailed generator architectures documented in Table 8.The supplied passage identifies the table but does not provide its architectural contents.
  • Modeling capacity: Double attention improves image reconstruction modeling capacity over vanilla Swin attention in the autoencoder decoder.The decoder uses transformer blocks and is trained with L1 loss.

Appendix D. Additional Quantitative Evaluation

Additional experiments compare StyleSwin with StyleGAN2 at 256 and 1024 resolution using FID, KID, and FID-Inf across the evaluated datasets.

  • LSUN Car: StyleSwin achieves comparable performance to state-of-the-art StyleGAN2 on the LSUN Car subset.Additional KID and FID-Inf evaluations are also reported for all datasets.
  • 256-resolution evaluation: Table 9 compares StyleSwin with StyleGAN2 at 256 resolution using FID, KID, and FID-Inf.The supplied passage provides the evaluation dimensions but not the individual table values.
  • 1024-resolution evaluation: Table 10 compares StyleSwin with StyleGAN2 at 1024 resolution using FID, KID, and FID-Inf.For FFHQ-1024, the table reports StyleGAN2 metrics; for CelebA-HQ 1024, it reports StyleGAN metrics.

Appendix E. More Qualitative Results

The appendix presents StyleSwin’s latent interpolations and additional samples, emphasizing smooth morphing and high-quality synthesis across faces, churches, and cars, alongside responsible-use cautions.

  • Latent code interpolation: StyleSwin produces smooth, meaningful latent interpolations spanning gender, poses, and eyeglasses.The interpolation linearly connects two randomly sampled latent codes.
  • Latent code interpolation: Figure 10 visualizes latent interpolation between the left-most and right-most FFHQ-1024 images.The figure caption identifies the resolution and endpoint images.
  • Additional image samples: FFHQ-1024 and CelebA-HQ 1024 samples exhibit diverse viewpoints, backgrounds, and accessories.These samples are presented as evidence of StyleSwin’s capacity for synthetic face generation.
  • Additional image samples: LSUN Church 256 and LSUN Car 256 samples show coherent structures, complicated materials, and high-quality light effects.The examples target complex scenes and materials.
  • Responsible use: The model may be misused for human impersonation, motivating tags or watermarks when distributing generated photos.The authors also encourage fair, representative training data and additional care for certain demographic groups.

Appendix G. Discussion of Limitation

The discussion identifies a practical gap between StyleSwin’s lower theoretical FLOPs and its lower measured throughput, while the supplied figure captions provide sample references rather than limitation findings.

  • Practical efficiency: 11.05 imgs/sec is StyleSwin’s measured throughput on one V100 GPU, versus 40.05 imgs/sec for StyleGAN2.The authors attribute the gap primarily to insufficient optimization of vision transformers compared with ConvNets.
  • Practical efficiency: StyleSwin’s theoretical FLOPs are smaller than StyleGAN2’s, but theoretical FLOPs do not translate directly into practical throughput.The discussion frames this discrepancy as a limitation of current implementation efficiency.
  • Open issue: The authors leave bCR’s ineffectiveness at 1024 × 1024 for further study.This is an additional limitation identified in the discussion.
  • Qualitative references: Figures 11–14 provide image samples for FFHQ-1024, CelebA-HQ 1024, LSUN Church 256, and LSUN Car 256.These captions document qualitative examples rather than a separate limitation result.
Loading 2112.10762v2…