Source-linked AI summary
Object-driven Text-to-Image Synthesis via Adversarial Training
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, Jianfeng Gao
TL;DR
Complex-scene text-to-image synthesis requires representing object-level language and spatial structure beyond global sentence conditioning. Obj-GAN combines layout-conditioned object-driven generation with object-wise discrimination, and significantly outperforms prior state-of-the-art GAN models across various metrics on COCO.
Problem
Complex scenes require modeling multiple objects, their relations, and missing visual information beyond the fine-grained information captured by global sentence conditioning.
Method
Obj-GAN follows a two-step layout-image process using object-driven attention and Fast R-CNN-based object-wise discriminators.
Results
Obj-GAN significantly outperforms previous state-of-the-art GAN models across various metrics on the large-scale COCO benchmark.
Takeaways & Limitations
Object-driven attention is more robust and interpretable than traditional grid attention for generating objects in complex scenes.
Takeaways & Limitations
Grid attention can be unreliable on complex scenes when previous image features do not provide a meaningful layout.
Abstract
from arXiv · showhide
In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes. Following the two-step (layout-image) generation process, a novel object-driven attentive image generator is proposed to synthesize salient objects by paying attention to the most relevant words in the text description and the pre-generated semantic layout. In addition, a new Fast R-CNN based object-wise discriminator is proposed to provide rich object-wise discrimination signals on whether the synthesized object matches the text description and the pre-generated layout. The proposed Obj-GAN significantly outperforms the previous state of the art in various metrics on the large-scale COCO benchmark, increasing the Inception score by 27% and decreasing the FID score by 11%. A thorough comparison between the traditional grid attention and the new object-driven attention is provided through analyzing their mechanisms and visualizing their attention layers, showing insights of how the proposed model generates complex scenes in high quality.
1. Introduction
Text-to-image synthesis must connect ambiguous language with visual content, yet existing methods struggle to preserve fine-grained object information in complex scenes. Obj-GAN addresses this with object-driven attention and object-wise discrimination within a two-step layout-image process.
- Text-to-image synthesis requires handling ambiguous, incomplete language while learning across vision and language modalities.
- Existing approaches often rely on a global sentence vector, which can miss word-level information needed for high-quality images.
- Complex scenes require modeling multiple objects, their relations, and missing visual information, challenges that existing methods do not specifically address.
- Two-step layout-image generation improves semantic layout control, but without explicit word/object-level information it can produce images lacking realistic detail.
- Obj-GAN combines an object-driven attentive generator with an object-wise discriminator to synthesize high-resolution images from text and a pre-generated semantic layout.
- At each stage, the generator focuses on words relevant to each bounding-box object, while the discriminator evaluates whether each object matches the text and layout.
- Obj-GAN significantly outperforms previous state-of-the-art methods on COCO, with ablations and attention visualizations analyzing its components and generation behavior.
2. Related Work
Text-to-image research has used multiple generative paradigms, with GANs performing strongly but often underusing fine-grained language. Two-step methods add semantic layouts, while attention mechanisms connect language and visual regions.
- Text-to-image synthesis has been approached through variational inference, approximate Langevin processes, conditional PixelCNNs, and conditional GANs.
- GAN-based methods have shown better image-generation performance than other approaches, but commonly condition only on a global sentence vector.
- Two-step methods first construct semantic layouts from text or scene graphs, then synthesize images conditioned on the layouts and text.
- Traditional grid attention has been applied across vision-language tasks including captioning, question answering, text-to-image generation, and retrieval.
3. Object-driven Attentive GAN
Obj-GAN generates complex scenes through semantic layout generation followed by object-aware image synthesis. Its attention and discrimination mechanisms use object-level information to improve consistency between text, layout, and generated objects.
- Semantic layout generation: Obj-GAN first generates a semantic layout containing object class labels, bounding boxes, and shapes, then synthesizes the image conditioned on that layout.A box generator produces bounding boxes, followed by a shape generator; the image-generation step uses the object-driven attentive generator and object-wise discriminator.
- Semantic layout generation: The box generator uses an attentive seq2seq model to better capture correspondences between sentence words and object bounding boxes.Each bounding box contains an object class label and coordinates, and the attention mechanism is motivated by the correspondence between boxes and words.
- Semantic layout generation: The shape generator predicts each object’s shape from its bounding box and random noise while using bidirectional convolutional LSTM modeling and perceptual loss.Generated shapes must match object location and category while aligning with surrounding context; GAN training and perceptual loss constrain and stabilize generation.
- Attentive multistage image generation: The multistage image generator uses a base generator for a low-resolution image and refiners that attend to relevant words, class labels, and semantic-layout features.The generator concatenates patch-wise, object-wise, and label context with hidden features; additional refiners can produce images up to 256 × 256 resolution.
- Object-driven attention: Object-driven attention assigns word-based context to semantically meaningful bounding-box regions, avoiding dependence on potentially unreliable low-resolution image features.Object context is distributed across pixels inside each object region, with max-pooling used when bounding boxes overlap; unlike grid attention, it remains independent of G0 image generation.
- Object-wise discrimination: Patch-wise discriminators reduce model size and improve generated-image quality relative to global discriminators, while object-wise discrimination supplies object-level training signals.The object-wise discriminator extracts region features for each bounding box using Fast R-CNN with ROI-align before evaluating object consistency.
4. Experiments
The experiments evaluate Obj-GAN on COCO using quantitative metrics, ablations, qualitative comparisons, and attention visualizations. Obj-GAN consistently improves over prior or alternative methods, while metric limitations remain.
- Evaluation setup: COCO evaluation uses Inception score, FID, and R-precision on official 2014 train and validation splits with object annotations and five descriptions per image.The authors note that FID aligns better with human qualitative evaluation than Inception score.
- Evaluation setup: Table 1 compares methods across predicted boxes and shapes, ground-truth boxes with predicted shapes, and ground-truth boxes and shapes.The table marks the best performance separately for the three layout settings.
- Evaluation caveats: The quantitative metrics are imperfect: Inception score fails to evaluate semantic layout, while R-precision depends on a retrieval model limited in capturing object relations.The authors therefore combine quantitative evaluation with qualitative visualization.
- Comparative results: Obj-GAN consistently outperforms P-AttnGAN w/ Lyt on all three metrics across the three conditional-layout settings.The comparison uses predicted boxes and shapes, ground-truth boxes with predicted shapes, and ground-truth boxes with shapes.
- Comparative results: Object-driven attention produces richer foreground and background textures than P-AttnGAN w/ Lyt under the same layout input.The authors attribute the background benefit to stronger implicit foreground distinction and clearer regional guidance.
- Attention analysis: Grid attention can be scattered and inaccurate in complex scenes, whereas object-driven attention uses word and class embeddings and bounds context by object shapes.The paper reports improved visual quality with object-driven attention and further improvement when semantic layout generation is improved.
5. Conclusions
The paper concludes that multi-stage Obj-GANs synthesize complex scenes by progressively using fine-grained word and object information, with object-wise discriminators supplying condition-specific signals.
- Conclusion: Obj-GANs synthesize complex-scene images from text descriptions through multi-stage generation.Each generator stage uses object-driven attention to utilize fine-grained word and object information while refining the image.
- Conclusion: Fast R-CNN-based object-wise discriminators provide object-wise discrimination signals paired with conditional generator inputs.The signals correspond to the generator conditions used during synthesis.
- Conclusion: Obj-GAN significantly outperforms previous state-of-the-art GAN models on various metrics on the large-scale COCO benchmark.Extensive experiments also demonstrate effectiveness and generalization ability for complex-scene text-to-image generation.
A.1. Obj-GAN vs. the ablative versions
Additional qualitative comparisons examine Obj-GAN and its ablative versions on COCO. Spectral normalization does not produce obvious visual-quality improvement in these examples.
- Ablation study: Additional images compare Obj-GAN with its ablative versions on the COCO dataset.Figures 10 and 11 complement the earlier qualitative comparison in Figure 6.
- Ablation study: Spectral normalization produces no obvious improvement in visual quality in the reported supplementary comparisons.The observation concerns the qualitative examples shown for the ablation study.
A.2. Visualization of attention maps
The appendix provides attention-map visualizations for P-AttnGAN and Obj-GAN as a supplement to the main comparison.
- Attention visualization: Attention maps generated by P-AttnGAN and Obj-GAN are visualized as a complement to Figure 8.The appendix uses these maps to extend the main attention comparison.
A.3. Results based on the ground-truth layout
The appendix presents qualitative Obj-GAN results generated from ground-truth layouts, including comparisons with prior methods and attention visualizations.
- Ground-truth layouts are used to generate the reported Obj-GAN results.
- Figure 11 provides an overall qualitative comparison.
- Figure 12 compares Obj-GAN with P-AttnGAN by placing each method’s attention maps beside its generated image.
- Figures 13–15 show additional results based on ground-truth layouts.
A.4. Bi-LSTM text encoder, DAMSM and R-precision
This section describes the Bi-LSTM text encoder and DAMSM-based fine-grained image–text matching losses used for training and evaluation.
- DAMSM learns a joint embedding of image regions and sentence words, while its fine-grained conditional loss matches generated sub-regions with corresponding words.
- The Bi-LSTM represents each word by concatenated bidirectional hidden states and forms a global sentence vector from its final hidden states.
- The image encoder uses Inception-v3 to extract local region features and a global image feature, then maps them into the common semantic space.
- Similarity is computed between words and image regions, and attention forms a context vector for each word as a weighted sum of visual features.
- The image–text relevance aggregates word-level similarities, with γ2 controlling emphasis on the most relevant word–image pair.
- DAMSM is pretrained on real image–text pairs and also supplies the R-precision evaluation score for generated images.
A.5. Network architectures for semantic layout generation
The semantic layout generator uses an attentive encoder–decoder to predict object labels and bounding boxes from text, with Gaussian-mixture modeling for coordinates.
- The box generator improves a prior design by adding attention and uses the Bi-LSTM text encoder within an encoder–decoder framework.
- Object labels are modeled categorically, while bounding-box coordinates are modeled with Gaussian mixtures and decomposed into two bivariate mixtures.
- Decoder LSTM outputs provide the Gaussian-mixture parameters, with contextual inputs computed from the text encoder’s word features.
- The shape generator replaces transposed-convolution upsampling with an instance-normalized block to address unstable training and severe grid artifacts.
A.6. Network architectures for image generation
The image-generation architecture combines attentive generators with patch-wise, shape, and object-wise discriminators, including spectral-normalized variants.
- The architecture specifies generator and discriminator networks through shared basic blocks and fixed hyperparameters.
- Generators use Adam with learning rate 0.0002, β1 = 0.5, and β2 = 0.999.
- Separate object-wise discriminators handle small and large objects, with large objects defined by width or height exceeding one-third of the image size.
- The design also combines object-wise discriminators with spectral-normalized projection discriminators, whose comparison is illustrated in Figure 16.
- Object-wise discrimination uses Fast R-CNN region features to judge whether each object is realistic and consistent with its label and text context.