Source-linked AI summary
Generative Adversarial Transformers
Drew A. Hudson, C. Lawrence Zitnick
TL;DR
Unconditional GAN synthesis still struggles with structured, compositional scenes and coordinating long-range visual dependencies. GANformer addresses this with a bipartite transformer for visual generation, achieving strong results across simulated and real-world datasets while improving efficiency, transparency, and compositionality.
Problem
Unconditional GAN synthesis of structured, compositional scenes remains less capable than synthesis of faces, single objects, and natural scenery.
Method
GANformer uses bipartite attention to iteratively exchange information between latent variables and image features, enabling efficient bidirectional long-range interactions.
Results
Across simulated and real-world datasets, GANformer achieves state-of-the-art image quality and diversity, with faster learning, greater data-efficiency, transparency, and compositionality.
Takeaways & Limitations
GANformer combines GAN and transformer strengths into an efficient visual generative model with improved latent disentanglement and a more transparent, compositional synthesis process.
Takeaways & Limitations
The paper demonstrates GANformer primarily for visual synthesis, leaving its usefulness for other tasks as an expectation rather than an evaluated result.
Abstract
from arXiv · showhide
We introduce the GANformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linear efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. We demonstrate the model's strength and robustness through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and outdoor scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data-efficiency. Further qualitative and quantitative experiments offer us an insight into the model's inner workings, revealing improved interpretability and stronger disentanglement, and illustrating the benefits and efficacy of our approach. An implementation of the model is available at https://github.com/dorarad/gansformer.
1. Introduction
The paper introduces GANformer, a bipartite transformer for visual synthesis that addresses CNN and standard-transformer limitations through bidirectional information exchange and scalable attention. Experiments report state-of-the-art performance across simulated and real-world datasets, with gains in structured multi-object scenes, data efficiency, transparency, and compositionality.
- Human perception relies on reciprocal bottom-up and top-down processing, whose interplay supports rich internal representations and vivid mental imagery.
- CNNs use one-way feed-forward processing and local receptive fields, limiting their ability to model long-range dependencies or develop holistic visual understanding.
- Bipartite attention supports long-range interactions while avoiding the quadratic complexity that limits standard transformers in computer vision.
- GANformer is a generalization of the vanilla transformer for visual synthesis that uses a bipartite structure to exchange information between generated image features and compact latent representations.
- GANformer achieves state-of-the-art performance across simulated and real-world datasets, with particularly impressive gains for highly structured multi-object scenes.
- Experiments indicate that GANformer needs fewer training steps and samples than competing approaches, while analyses and ablations support enhanced transparency and compositionality.
2. Related Work
Prior visual synthesis work is dominated by GANs, while vision transformers either miss global interactions through local attention or incur quadratic computational costs. GANformer addresses these limitations with bipartite attention between image features and a small set of latent variables, supporting efficient long-range interaction and iterative refinement.
- Generative modeling: GANs have become a leading paradigm in visual synthesis, with advances in training stability, image quality, and diversity across tasks including translation, super-resolution, and style transfer.The passage also mentions representation-related applications, but its final list is truncated.
- Vision transformers: Vision attention models commonly use local operations that miss global interactions or global transformer attention with prohibitive quadratic complexity.These two structural streams are described as common approaches for recognition, segmentation, detection, and synthesis.
- GANformer: GANformer replaces costly self-attention with bipartite attention between image features and a small collection of latent variables, enabling long-range interaction with scalable linear computation across layers.Its bipartite design aligns with transforming source latents into an output image.
- GANformer: Simplex and Duplex attention iteratively update and refine assignments between image features and latents, extending these techniques to high-resolution generative modeling.The passage characterizes this as the first exploration of these attention techniques in high-resolution generative modeling.
- Compositional generation: Unlike layered GANs that combine independently generated components through alpha-composition, GANformer generates one unified image through a cooperative process coordinating its components.The comparison concerns models that use multiple generator replicas to produce image layers.
3. The Generative Adversarial Transformer
GANformer unifies GANs and transformers through a bipartite architecture that enables efficient long-range interactions between latent variables and visual features. Its compositional latent space, bidirectional refinement, and multiplicative integration provide flexible control over global and localized visual properties.
- Compositional Latent Space: Multiple latent variables coordinate through attention to produce images cooperatively, matching the compositionality of natural scenes.Latents can control attended spatial regions, guiding the synthesis of objects and entities.
- Bipartite Transformer: GANformer uses a bipartite structure to model long-range dependencies while maintaining linear computational costs.The architecture separates image elements from latent aggregator variables and computes attention over the resulting bipartite graph.
- Bidirectional Interaction: Bidirectional interaction lets latent variables and visual features refine and interpret each other.Simplex attention propagates information from latents to images, while duplex attention supports propagation in both directions.
- Multiplicative Integration: Multiplicative integration modulates feature scale and bias, extending StyleGAN-like style control to localized semantic regions.Soft attention groups elements by proximity and content similarity, enabling flexible regional style modulation beyond StyleGAN’s global control.
- Design Properties: Together, these design choices yield high efficiency, improved latent-space disentanglement, and greater transparency of the generative process.The section presents these properties as outcomes of combining the GANformer’s architectural innovations.
4. Experiments
Experiments show that GANformer achieves strong image-synthesis performance across diverse datasets, learns faster, and exhibits semantic, disentangled representations. Ablations and distributional analyses further support its design choices and compositional modeling capabilities.
- Quantitative performance: GANformer achieves state-of-the-art image quality and diversity across FFHQ, CLEVR, LSUN-Bedrooms, and Cityscapes.The experiments evaluate both quantitative performance and qualitative behavior across human faces, multi-object scenes, and indoor and outdoor scenes.
- Quantitative performance: 9.17 FID on CLEVR versus 16.05, while LSUN-Bedrooms reaches 6.51 versus 11.53 under equal training steps.The gains are smaller on FFHQ, where image-layout diversity is naturally lower.
- Learning efficiency: Approximately 3-times fewer training steps are needed on CLEVR to produce high-quality images than for the second-best approach.Learning curves and intermediate samples indicate significantly faster learning than competing models.
- Interpretability and compositionality: Latent variables attend to coherent regions and semantic concepts, including windows, pillows, sidewalks, cars, and background.Attention visualizations and detector-based correlations indicate that different latents capture objects or constituent scene components.
- Semantic coverage: GANformer obtains better CLEVR scores across almost all semantic properties, complementing FID and Precision/Recall with structure- and semantics-focused evaluation.The analysis uses generated-scene object and attribute distributions to assess coverage of the natural CLEVR image distribution.
- Ablation studies: Earlier and higher-resolution application of bipartite attention improves performance and accelerates learning, empirically validating the design.The ablation varies the earliest and latest generator layers receiving attention; high-resolution layers particularly benefit.
5. Conclusion
The paper introduces GANformer, an efficient bipartite transformer combining top-down and bottom-up interactions for generative modeling, with strong quantitative and qualitative results. Its inductive biases aim to encourage transparency, data-efficiency, and compositionality.
- Conclusion: GANformer is a novel, efficient bipartite transformer that combines top-down and bottom-up interactions for generative modeling.The model achieved strong quantitative and qualitative results indicating robustness and efficacy.
- Conclusion: Its inductive biases aim to encourage transparency, data-efficiency, and compositionality in neural networks.
Supplementary Material
The supplementary material provides additional GANformer experiments and visualizations, including training-progress and converged sample comparisons with baseline models. It also documents implementation and optimization details and evaluates spatial compositionality of the model’s attention mechanism.
- Implementation details: Section A specifies the GANformer’s implementation details, optimization scheme, and training configuration.
- Spatial compositionality: Section B and figure 8 evaluate the spatial compositionality of the GANformer’s attention mechanism.
A. Implementation and Training Details
The models are implemented and trained under matched configurations using a TensorFlow StyleGAN codebase and StyleGAN2-derived optimization practices. Training uses 256 × 256 images, dataset-specific augmentation, and equal training durations across models.
- Implementation: All models are implemented in the StyleGAN authors’ TensorFlow codebase to match training configuration, model size, and optimization details.The GANformer and model-size settings are provided in tables 4 and 5.
- Training Configuration: Training adopts StyleGAN2 techniques, including style mixing, Xavier initialization, stochastic variation, exponential moving average weights, lazy R1 regularization, and non-saturating logistic loss.Optimization uses Adam with batch size 32 (4 × 8 gradient accumulation), equalized learning rate 0.001, β1 = 0.0, and β2 = 0.99.
- Datasets and Compute: Models are trained on 256 × 256 images for the same number of steps, roughly one week on 2 NVIDIA V100 GPUs per model.Dataset sizes are 100k CLEVR images, 70k FFHQ images, about 25k Cityscapes images, and 3M LSUN-Bedrooms images; Cityscapes and FFHQ use mirror augmentation.
B. Spatial Compositionality
The model’s spatial compositionality is quantified by measuring alignment between latent-variable attention and semantic segments across synthesized scenes. Results on LSUN-Bedrooms and Cityscapes indicate that the model decomposes images into semantically meaningful object and entity segments.
- Spatial Compositionality: A pre-trained segmentor provides semantic segmentations for synthesized scenes, enabling correlation measurement between latent-variable attention maps and semantic classes.The correlation is computed using the maximum intersection-over-union between each class segment and attention segments across model layers.
- Spatial Compositionality: The evaluation averages these maximum intersection-over-union scores over 1k images for LSUN-Bedrooms and Cityscapes.Results are presented in Figure 8 for both datasets.
- Spatial Compositionality: High correlations between semantic classes and model attention indicate decomposition into semantically meaningful segments of objects and entities.The analysis uses attention cast by latent variables as the basis for assessing compositional structure.