Source-linked AI summary
Controllable Text-to-Image Generation
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, Philip H. S. Torr
TL;DR
Text-to-image generators can produce realistic images but often change unrelated content when users modify only part of a description. ControlGAN combines word-level spatial and channel-wise attention, word-level discrimination, and perceptual loss to support targeted manipulation; experiments on CUB and COCO report effective disentanglement and stronger performance than existing methods. COCO results are weaker than CUB results because its captions are more abstract and its categories have fewer examples.
Problem
Existing text-to-image generators are typically uncontrollable, changing unrelated visual attributes when users modify words in a description.
Method
ControlGAN uses an attention-driven generator, a word-level discriminator, and perceptual loss to manipulate relevant visual attributes while preserving unrelated content.
Results
Experiments on CUB and COCO show effective attribute disentanglement and accurate image-part manipulation, with performance surpassing existing methods qualitatively and quantitatively.
Takeaways & Limitations
ControlGAN enables natural-language manipulation of specific synthetic-image attributes while preserving other content.
Takeaways & Limitations
COCO results are weaker than CUB results because the dataset has few text-image pairs and more abstract captions focused mainly on object categories.
Abstract
from arXiv · showhide
In this paper, we propose a novel controllable text-to-image generative adversarial network (ControlGAN), which can effectively synthesise high-quality images and also control parts of the image generation according to natural language descriptions. To achieve this, we introduce a word-level spatial and channel-wise attention-driven generator that can disentangle different visual attributes, and allow the model to focus on generating and manipulating subregions corresponding to the most relevant words. Also, a word-level discriminator is proposed to provide fine-grained supervisory feedback by correlating words with image regions, facilitating training an effective generator which is able to manipulate specific visual attributes without affecting the generation of other content. Furthermore, perceptual loss is adopted to reduce the randomness involved in the image generation, and to encourage the generator to manipulate specific attributes required in the modified text. Extensive experiments on benchmark datasets demonstrate that our method outperforms existing state of the art, and is able to effectively manipulate synthetic images using natural language descriptions. Code is available at https://github.com/mrlibw/ControlGAN.
1 Introduction
ControlGAN addresses the lack of controllability in text-to-image generation by manipulating specified visual attributes while preserving unrelated content. It combines attention, word-level discrimination, and perceptual loss, and reports effective attribute disentanglement and strong benchmark performance.
- Text changes in existing generative networks can alter unrelated attributes such as a bird’s pose and position, making image editing undesirable.
- ControlGAN generates images from text and manipulates object attributes without affecting other content.
- Its attention-driven generator synthesises subregions linked to the most relevant words, while progressively refining images from coarse to fine.
- A word-level discriminator correlates words with image subregions to provide fine-grained training signals for visual attributes.
- Perceptual loss reduces generation randomness and encourages preservation of visual appearance associated with unmodified text.
- Experiments on CUB and COCO report effective attribute disentanglement, accurate image-part manipulation without losing diversity, and improved qualitative and quantitative performance over existing methods.
2 Related Work
Prior work established attention-based, multi-stage, and conditional approaches for text-to-image generation and related image manipulation. ControlGAN builds on these directions by addressing limitations in spatial attention and word–image correspondence.
- Text-to-image Generation: Earlier text-to-image methods used attention, approximate Langevin sampling, conditional GANs, and coarse-to-fine generation.
- Image-to-image translation: Related image-to-image methods manipulated objects through verbal commands, latent vectors, VAE–GAN reconstruction, or text-adaptive discrimination.
- Attention: Attention has been applied across captioning, translation, object detection, and visual question answering to capture task-relevant information.
- Attention: AttnGAN used word-level spatial attention to guide generation toward subregions corresponding to relevant words.
- Attention: Spatial attention does not account for channel information, although CNN feature channels may serve different purposes and require differentiated treatment.
3 Controllable Generative Adversarial Networks
ControlGAN combines multi-stage text-to-image generation with channel-wise attention, word-level discrimination, and perceptual loss to support localized attribute manipulation while preserving unrelated content.
- Architecture: ControlGAN uses a multi-stage AttnGAN backbone with spatial and channel-wise attention to condition successive image-generation stages on word features.The framework generates images from coarse to fine, concatenating attentive word-context features with hidden visual features for the next stage.
- Channel-Wise Attention: Channel-wise attention correlates words with visual channels, enhancing relevant channels and reducing the influence of irrelevant ones.This dynamic weighting supports disentangling word attributes into different feature channels.
- Word-Level Discriminator: The word-level discriminator correlates words with image subregions to provide fine-grained feedback for independently supervising visual attributes.It aligns visual and word features, computes word-context correlations, aggregates spatial information, and sums word-level correlations.
- Perceptual Loss: Perceptual loss matches VGG feature representations of generated and real images to reduce randomness and preserve semantic consistency in unconstrained regions.The loss uses activations from a pre-trained VGG network, with feature-map dimensions defined for each selected layer.
- Objective Functions: The generator loss combines adversarial, text-image correlation, perceptual, and text-image matching losses, while adversarial terms enforce realism and text alignment.Training alternates generator and discriminator optimization across stages, with additional losses controlled by hyper-parameters.
4 Experiments
Experiments evaluate ControlGAN on CUB and COCO using quantitative metrics, qualitative comparisons, and component ablations. The results indicate improved image quality, semantic alignment, and control over localized attribute changes, with weaker outcomes on the more challenging COCO dataset.
- Experimental setup: ControlGAN is evaluated on CUB and MS COCO against StackGAN++ and AttnGAN using quantitative and qualitative experiments.The evaluation uses the datasets and comparison methods described for the experiments, with Table 1 reporting Inception Score, R-precision, and L2 reconstruction error.
- Quantitative results: ControlGAN achieves better Inception Score and R-precision values than state-of-the-art methods on CUB and competitive performance on COCO.Inception Score evaluates image quality and diversity, while R-precision measures image-text relevance.
- Quantitative results: ControlGAN obtains significantly lower L2 reconstruction error than other methods, indicating better preservation of content when modified text is used.The error is computed between images generated from original and modified text.
- Qualitative results: Qualitative results show accurate manipulation of specified visual attributes, including an out-of-distribution red-zebra query, while preserving unrelated content.The results are presented in Figure 4, whose odd and even columns correspond to original and modified text.
- Qualitative comparison: Compared with AttnGAN and StackGAN++, ControlGAN changes attributes tied to modified text while better preserving unchanged attributes, whereas the baselines often alter pose, background, or object shape.On COCO, the compared methods may fail to preserve object shape or generate reasonable images.
- Component analysis: Channel-wise attention correlates with semantic object parts, the word-level discriminator supplies fine-grained word-region supervision, and perceptual loss helps preserve content linked to unchanged text.Ablations report failures in controllable manipulation when channel-wise attention or the word-level discriminator is removed, and weaker content preservation without perceptual loss.
5 Conclusion
ControlGAN generates and manipulates images from natural-language descriptions while preserving unmodified content. Its three components support attribute disentanglement, fine-grained supervision, and reduced generation randomness.
- ControlGAN manipulates visual attributes from natural-language descriptions while preserving other generated content.
- Its attention-driven generator disentangles visual attributes, and its word-level discriminator supplies fine-grained training signals for each attribute.
- Perceptual loss reduces generation randomness and encourages reconstruction of content associated with unmodified text.