Source-linked AI summary
Semantics Disentangling for Text-to-Image Generation
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, Jing Shao
TL;DR
Text-to-image systems must reconcile diverse descriptions of the same content, which can yield inconsistent generated images. SD-GAN disentangles shared and detailed semantics through a Siamese discriminator and semantic-conditioned batch normalization, achieving state-of-the-art results on CUB and MS-COCO.
Problem
Diverse linguistic expressions for the same image make it difficult to extract consistent semantics, while existing work mainly emphasizes visual quality and resolution.
Method
SD-GAN uses a Siamese discriminator to distill semantic commons and Semantic-Conditioned Batch Normalization to preserve detailed linguistic cues during generation.
Results
SD-GAN achieves state-of-the-art text-to-image generation performance on the CUB-200 and MS-COCO datasets.
Takeaways & Limitations
Separating shared semantic content from linguistic details supports generation consistency under expression variants while retaining fine-grained visual diversity.
Abstract
from arXiv · showhide
Synthesizing photo-realistic images from text descriptions is a challenging problem. Previous studies have shown remarkable progresses on visual quality of the generated images. In this paper, we consider semantics from the input text descriptions in helping render photo-realistic images. However, diverse linguistic expressions pose challenges in extracting consistent semantics even they depict the same thing. To this end, we propose a novel photo-realistic text-to-image generation model that implicitly disentangles semantics to both fulfill the high-level semantic consistency and low-level semantic diversity. To be specific, we design (1) a Siamese mechanism in the discriminator to learn consistent high-level semantics, and (2) a visual-semantic embedding strategy by semantic-conditioned batch normalization to find diverse low-level semantics. Extensive experiments and ablation studies on CUB and MS-COCO datasets demonstrate the superiority of the proposed method in comparison to state-of-the-art methods.
1. Introduction
Text-to-image generation must handle diverse descriptions of the same image, which can produce inconsistent outputs. SD-GAN separates shared semantic content from descriptive details using Siamese discrimination and semantic-conditioned normalization.
- Text descriptions are natural and convenient, but cross-modal text-to-image generation remains challenging.
- Descriptions of the same image can vary substantially, causing generated appearances to deviate from the ground-truth and from one another.
- SD-GAN distills semantic commons from texts to improve generation consistency while retaining semantic diversities and details for fine-grained generation.
- Its Siamese discriminator compares text-paired generations with contrastive loss, reducing feature distances for intra-class pairs and enlarging them for inter-class pairs.
- Semantic-Conditioned Batch Normalization embeds detailed linguistic cues into generator feature maps to guide fine-grained visual synthesis.
- SD-GAN reports state-of-the-art performance on the CUB-200 and MS-COCO text-to-image datasets.
2. Related Works
Related work established GAN-based text-to-image synthesis and improved image generation through stacked generators and attention. These approaches provide the foundation for SD-GAN's semantics-focused generation framework.
- GANs use adversarial generator-discriminator training, and conditional GANs were shown capable of synthesizing plausible images from text descriptions.
- Stacked GAN structures generate images at different sizes through coarse-to-fine synthesis, while attention mechanisms enable word-level conditioning for fine-grained generation.
- Semantic-conditioned batch normalization layers enhance visual-semantic embedding and are implemented in GAN generators for linguistic-condition-based visual generation.
3. Semantics Disentangling Generative Adversarial Network (SD-GAN)
SD-GAN combines a Siamese discriminator with contrastive losses to distill shared semantics across descriptions, while SCBN preserves textual details through sentence- and word-level visual-semantic modulation.
- 3.1. Siamese Structure with Contrastive Losses: SD-GAN uses a Siamese scheme with paired descriptions and contrastive loss to encourage consistency between images generated from descriptions of the same image.The discriminator compares visual features from the two branches; descriptions from different images are pushed farther apart.
- 3.1. Siamese Structure with Contrastive Losses: The backbone contains a text encoder and hierarchical generator-discriminator subnets, with G0 producing 64 × 64 images and later stages producing higher resolutions.Each later generator uses the previous visual output together with the sentence feature, and each stage has an associated discriminator.
- 3.1. Siamese Structure with Contrastive Losses: The contrastive objective measures the Euclidean distance between branch visual features, using y to distinguish same-image descriptions from different-image descriptions.The feature-vector length N is set to 256 and ε is set to 1.0 in the experiments.
- 3.1. Siamese Structure with Contrastive Losses: A margin α prevents same-description outputs from becoming identical, preserving appearance variation and avoiding collapsed nonsensical visualization modes.The experiments set α = 0.1.
- 3.2. Semantic-Conditioned Batch Normalization (SCBN): SCBN conditions generator feature normalization on linguistic cues, using sentence-level MLPs and word-level visual-semantic embedding to retain semantic diversity and details.Sentence features produce channel-wise modulation, while the VSE module fuses word and visual features for local subregions.
- 3.2. Semantic-Conditioned Batch Normalization (SCBN): SCBN modifies conditional normalization by adding cue-dependent scale and shift parameters to the learned affine parameters of batch normalization.The underlying conditional normalization uses γ, β together with γc and βc to modulate visual feature maps.
4. Experiments
Experiments on CUB and MS-COCO evaluate SD-GAN against prior methods through quantitative, human, qualitative, sensitivity, and component analyses. Results support improved text-conditioned generation, semantic consistency across expression variants, and retention of linguistic details.
- Comparing with the state-of-the-arts: SD-GAN achieves inception scores of 4.67 ± .09 on CUB and 35.69 ± .50 on MS-COCO, exceeding previous best scores.The previous best scores are 4.36 ± .03 on CUB and 25.89 ± .47 on MS-COCO.
- Comparing with the state-of-the-arts: Users ranked SD-GAN images first for about 70% of presented texts, higher than StackGAN and AttnGAN.The human evaluation ranks images by relevance to the input text.
- Comparing with the state-of-the-arts: Qualitative comparisons show SD-GAN produces more recognizable and semantically meaningful images under linguistic expression variants than StackGAN and AttnGAN.Examples include gray wings on a white bird and sheep on grass.
- Component Analysis: Combining SCBN and Siamese improves scores from 4.49 to 4.67 on CUB and from 29.79 to 35.69 on MS-COCO.The Siamese structure also raises the CUB score from 4.11 to 4.51 when added to the baseline.
- Component Analysis: Visualizations indicate SCBN adds linguistic details, Siamese promotes consistency across descriptions, and their combination improves both properties.SCBN captures details such as blue wings, while Siamese can otherwise lose details such as black wings.
- Sensitivity Analysis: Changing one word or phrase modifies generated attributes such as bird color and scene, demonstrating sensitivity to subtle textual changes.Examples compare yellow versus blue birds and beach versus grass-field scenes.
- Component Analysis: SCBN layers outperform corresponding general batch-normalization layers for sentence-level and word-level linguistic cues.The comparison evaluates SCBN-sent versus BN-sent and SCBN-word versus BN-word.
5. Conclusion
The paper concludes that SD-GAN uses Siamese learning to preserve semantic consistency across expression variants and SCBN to retain linguistic details. Experiments on CUB and MS-COCO demonstrate the effectiveness of these components for text-to-image generation.
- Conclusion: SD-GAN distills semantic commons from linguistic descriptions so generated images remain consistent under expression variants.The framework uses a Siamese structure for this purpose.
- Conclusion: SCBN complements the Siamese mechanism by embedding instance linguistic cues into visual generation to preserve semantic diversities and details.The method reformulates batch normalization using linguistic cues.
- Conclusion: Extensive experiments demonstrate the effectiveness and significance of SD-GAN on CUB and the large-scale MS-COCO dataset.The conclusion reports evidence across both datasets.
6. Appendix
The appendix details the discriminator architecture and supplements the paper with qualitative comparisons and randomly sampled generated images from CUB and MS-COCO.
- Architecture of Discriminators: Figure 8 depicts discriminator architecture with convolutional visual-feature extraction followed by two Siamese branches.The architecture uses hierarchical low-to-high-resolution stages, with a discriminator following each generator stage.
- Architecture of Discriminators: The Siamese branches compare visual-feature distances d = ∥v1 − v2∥2 and dc = ∥vc1 − vc2∥2.
- Architecture of Discriminators: The D0, D1, and D2 discriminators use similar structures with different down-sampling depths and independent parameters.Their outputs are designed to have the same size.
- More Results: The appendix shows 400 randomly selected 256 × 256 generated images for each dataset and reports no obvious nonsensical modes on CUB or MS-COCO.Figures 10 and 11 provide the CUB and MS-COCO samples, respectively.
- More Results: Figure 9 compares module configurations of SD-GAN and compares SD-GAN qualitatively with StackGAN and AttnGAN using two descriptions per ground-truth image.The configurations add SCBN and Siamese components step by step.