PaperlayerView exact revision on arXiv

Source-linked AI summary

Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

Seunghoon Hong, Dingdong Yang, Jongwook Choi, Honglak Lee

arXiv:1801.05091v2cs.CV

TL;DR

Generating realistic images from complicated text descriptions remains difficult when models learn a direct text-to-pixel mapping. This paper hierarchically constructs a semantic layout with bounding boxes and object masks, then conditions image generation on that layout and the text. The resulting images are reported to be more semantically meaningful, better aligned with text, and controllable through layout editing.

  • Problem

    Existing approaches struggle to generate reasonable images for complicated real-world descriptions such as MS-COCO because direct text-to-pixel mapping must capture multiple visual concepts.

  • Method

    The model constructs a semantic layout between text and image by generating object bounding boxes, refining object shapes, and using the resulting layout to guide pixel synthesis.

  • Results

    The method generates semantically meaningful images that are well-aligned with input descriptions and substantially improves generation quality on challenging MS-COCO data.

  • Takeaways & Limitations

    Predicted layouts provide instance-wise annotations and let users control generation by adding, removing, resizing, or relocating objects.

  • Takeaways & Limitations

    The shape-generation formulation uses a random noise vector z_t ∼N(0, I).

Abstract

from arXiv · show

We propose a novel hierarchical approach for text-to-image synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple steps, in which it first constructs a semantic layout from the text by the layout generator and converts the layout to an image by the image generator. The proposed layout generator progressively constructs a semantic layout in a coarse-to-fine manner by generating object bounding boxes and refining each box by estimating object shapes inside the box. The image generator synthesizes an image conditioned on the inferred semantic layout, which provides a useful semantic structure of an image matching with the text description. Our model not only generates semantically more meaningful images, but also allows automatic annotation of generated images and user-controlled generation process by modifying the generated scene layout. We demonstrate the capability of the proposed model on challenging MS-COCO dataset and show that the model can substantially improve the image quality, interpretability of output and semantic alignment to input text over existing approaches.

1. Introduction

Existing text-to-image methods struggle with complicated real-world descriptions because direct text-to-pixel mapping must capture objects, spatial relations, and scene context simultaneously. The proposed method inserts an explicit semantic layout between text and image, enabling hierarchical generation and user control.

  • Motivation: Existing approaches have mainly been limited to simple bird and flower datasets, while complicated MS-COCO images remain challenging.Complex descriptions require reasoning about object categories, spatial configurations, and scene context.
  • Approach: The method constructs a semantic layout as an intermediate representation between text and image, encoding object instances and fine-grained scene structure.The layout includes object number, category, location, size, and shape.
  • Approach: The layout generator first estimates bounding boxes, then refines each object’s shape, while the image generator uses the resulting layout and text for pixel-level synthesis.The box, shape, and image generators are independent neural networks trained in parallel with corresponding supervisions.
  • Benefits: Predicted layouts provide instance-wise annotations and an interactive interface for modifying objects, sizes, and locations during generation.These controls support removing or adding objects and changing their size and location.
  • Results: Explicit layout conditioning produces semantically meaningful images that are well-aligned with input descriptions.The authors report substantial improvement in generation quality on the challenging MS-COCO dataset.

2. Related Work

Related work spans several generative formulations and structure-conditioned methods. This paper differs by predicting general semantic label maps rather than structure specific to a particular object type.

  • Text-to-image synthesis: Prior text-to-image methods use VAE, autoregressive, optimization-based, and conditional GAN formulations.Conditional GAN approaches condition both generator and discriminator on text, with later work increasing image resolution through two-stage generation.
  • Semantic-label image generation: Semantic-label-to-image methods translate dense pixel-wise labels into images, sometimes combining dense layouts with attribute vectors.Examples include pixel-to-pixel translation, cascaded refinement, and conditional GAN approaches.
  • Structure-conditioned generation: Structure-based generation has inferred representations such as surface normal maps, human joints, and local key-points in other domains.The paper’s method instead predicts semantic label maps as a general representation of natural images.

3. Overview

The framework progressively converts text into a semantic scene layout and then into pixels. It composes object boxes, refines their shapes into a semantic label map, and conditions image synthesis on that map and the text.

  • Semantic layout: Aggregating instance-wise masks produces a semantic label map that defines the scene’s semantic structure.The label map combines outputs from the box and shape generators.
  • Box generation: The box generator converts text embedding s into bounding boxes defining each object’s location, size, and category.The boxes form the image’s coarse layout.
  • Shape generation: The shape generator takes the generated boxes and predicts binary masks defining each object’s foreground shape.These masks refine the coarse object layout.
  • Image generation: The image generator translates the semantic label map and text embedding into pixels matching the text description.Explicit layout conditioning helps preserve detailed object shapes and makes semantic contents easier to recognize.

4. Inferring Semantic Layout from Text

The layout generator infers scene structure from text hierarchically, first composing labeled bounding boxes and then refining each object into a mask. Instance-wise and global constraints guide mask generation, with perceptual reconstruction loss supporting shape quality and training stability.

  • 4.1. Bounding Box Generation: The box generator maps text embedding s to a set of labeled bounding boxes specifying each object's location, size, and category.Its stochastic autoregressive model samples class labels and box coordinates, with an end-of-sequence indicator determining the number of objects adaptively.
  • 4.2. Shape Generation: The shape generator refines each bounding box into a binary object mask that specifies the object's shape inside the box.Bounding boxes are converted into a class-labeled binary tensor before mask prediction.
  • 4.2. Shape Generation: The recurrent shape generator uses convolutional encoding, bidirectional convolutional LSTM context, spatially tiled noise, and decoding to produce each mask.This design incorporates information from all object instances while generating masks sequentially.
  • 4.2. Shape Generation: Two conditional adversarial losses enforce compatibility between each mask and its box while encouraging globally coherent interactions among objects.The instance-wise discriminator evaluates individual masks, whereas the global discriminator evaluates their aggregated context.
  • 4.2. Shape Generation: Perceptual reconstruction loss compares real and predicted masks in a pretrained CNN feature space and significantly improves GAN stability and object-shape quality.The implementation uses VGG-19 features and combines reconstruction with the other shape-generator losses.

5. Synthesizing Images from Text and Layout

The image generator converts aggregated object masks into a semantic label map and synthesizes an image conditioned jointly on that layout and the text. Its encoder-decoder uses text-gated layout features, background text conditioning, residual blocks, and a cascaded decoder.

  • 5. Synthesizing Images from Text and Layout: Binary object masks are aggregated into a semantic label map whose active locations indicate which object class covers each pixel.The map preserves class-based spatial structure for image synthesis.
  • 5. Synthesizing Images from Text and Layout: The image generator produces an image from the semantic layout, text embedding, and random noise.The generated image is trained to conform to both the layout and the text condition.
  • 5. Synthesizing Images from Text and Layout: A convolutional encoder-decoder encodes the layout, gates its features using text attention, concatenates background text and noise, and decodes the result through residual blocks.The layout feature is gated as Ag = A ⊙ σ(S), where S is spatially replicated from the text embedding.
  • 5. Synthesizing Images from Text and Layout: The cascaded decoder enhances conditioning on layout structure and produces better object boundaries.
  • 5. Synthesizing Images from Text and Layout: The image generator is jointly trained with a discriminator using adversarial and perceptual reconstruction losses conditioned on the layout and text.The discriminator receives the generated image, semantic layout, and spatially tiled text embedding.

6. Experiments

Experiments on MS-COCO evaluate semantic alignment, recognizability, diversity, and controllability. Across quantitative, qualitative, and human evaluations, the method improves alignment and supports layout- and text-based control.

  • Experimental setup: The model is evaluated on MS-COCO, which contains complex scenes with many objects and diverse contexts.The dataset includes 164,000 training images, 80 semantic classes, instance annotations, and five descriptions per image.
  • Quantitative comparison: The proposed method substantially outperforms conditional-GAN baselines on Inception score and caption-generation metrics.Inception score measures recognizability and diversity, while caption metrics assess relevance between generated images and input text.
  • Quantitative comparison: Users selected the proposed images as most relevant in about 60% of sentences, compared with about 20% for baselines.The human evaluation used 1,000 sentences, each annotated by five users.
  • Qualitative analysis: Qualitative comparisons show more recognizable and semantically meaningful images whose reconstructed captions better align with input descriptions.Baselines often produce clear high-frequency details but fail to capture important object and scene structure.
  • Qualitative analysis: Replacing predicted bounding boxes and masks with ground-truth layouts yields gradual performance improvements, indicating errors in both layout components.Figure 5 compares predicted layouts, mixed predicted and ground-truth layouts, and fully ground-truth layouts.
  • Generation control: The model generates diverse semantic structures for fixed text while preserving object categories and counts, and adapts to caption or layout manipulations.Users can modify scene context, object category, object number, spatial composition, backgrounds, or bounding-box layouts.

7. Conclusion

The paper concludes that explicit semantic-layout inference enables hierarchical text-to-image synthesis that preserves semantic details and remains highly relevant to text. The predicted layout also supports generation control, while end-to-end layout and image training remains future work.

  • 7. Conclusion: The model hierarchically constructs semantic layouts in a coarse-to-fine manner using a series of generators.Image generation is conditioned on the explicit layout prediction.
  • 7. Conclusion: Conditioning on explicit layouts generates complicated images that preserve semantic details and remain highly relevant to text descriptions.The conclusion presents this as the main synthesis outcome.
  • 7. Conclusion: Predicted layouts can be modified to control the image-generation process.The conclusion identifies layout-based control as an additional capability.
  • 7. Conclusion: End-to-end training of layout and image generation is identified as an interesting direction for future work.This is the paper’s stated future-work scope.

A. Implementation Details

The implementation decomposes layout generation into bounding-box and shape prediction, then uses recurrent convolutional components to predict object masks. The box generator models coordinates with decomposed Gaussian mixtures, trained using Adam optimization.

  • A.1. Box Generator: The box generator samples each object bounding box from a joint probability distribution over location, size, and class.The notation defines the t-th object box as Bt = (bx_t, by_t, bw_t, bh_t, l_t).
  • A.1. Box Generator: Categorical class probabilities and Gaussian-mixture parameters are produced from LSTM outputs at each generation step.The mixture parameters are concatenated into a vector.
  • A.1. Box Generator: The box-coordinate probability is approximated with two bivariate Gaussian mixtures because a full covariance model is expensive.The decomposition reduces the number of parameters relative to a full quadrivariate Gaussian covariance matrix.
  • A.2. Shape Generator: The shape generator encodes each box tensor, combines features with a bidirectional convolutional LSTM, and predicts a binary object mask.Noise is spatially replicated and concatenated before masked residual processing and upsampling.
  • A.2. Shape Generator: Instance-wise and global discriminators receive concatenated box-and-mask representations at object and scene levels.The two discriminators share an architecture but use separate parameters.

A.3. Image Generator

The image generator converts an estimated semantic layout and text description into an image, using refinement and matching-aware training to align outputs with both conditions.

  • Architecture: The image generator combines the semantic layout with text to synthesize an image conditioned on both inputs.Its architecture uses cascaded refinement layers that repeatedly combine intermediate features with resized semantic layouts.
  • Training: Matching-aware loss adds mismatched text embeddings as fake examples during adversarial training.For a ground-truth triple (M, s, X), a random non-relevant embedding is paired with the layout and image.
  • Training: Matching-aware loss substantially improves text-conditional generation and stabilizes overall GAN training.
  • Training: The image generator is trained with Adam using learning rate 0.0002, β1 = 0.5, and β2 = 0.999.The learning rate is linearly decreased after the first 30 epochs.

B.1. Ablative Analysis

The ablation study evaluates the shape generator, perceptual losses, and attention mechanism, showing that these components support recognizable, text-conditional image generation.

  • Impact of shape generator: Removing the shape generator causes substantial performance degradation because box-only generation misses detailed object shapes.Explicitly inferring shapes improves overall image quality and interpretability of content.
  • Impact of perceptual loss in shape generator: Replacing the shape generator’s reconstruction loss with pixelwise ℓ1 loss reduces object-shape accuracy and caption generation performance.Perceptual loss produces more recognizable images.
  • Impact of perceptual loss in image generator: Perceptual loss in the image generator critically improves performance by reducing visual differences between real and synthesized images.
  • Impact of attention in image generator: Removing attention from the image generator lowers text-conditional generation performance.Attention helps generate textures and backgrounds relevant to the input text by encouraging use of text information.

B.2. More qualitative examples

Qualitative results show that the model generates text-matched layouts and images, diverse samples, and controllable scenes through semantic-layout editing.

  • Image and layout generation: Generated bounding boxes match input-text content, while object shapes capture class-specific attributes and relations.The image generator then predicts compatible object textures and backgrounds.
  • Diversity of samples: The model generates diverse samples by producing semantic layouts that remain highly related to the input description.
  • Controllable image generation: Adding new object bounding boxes creates corresponding object instances and modifies surrounding content compatibly.
  • Controllable image generation: Modifying object bounding-box sizes and locations enables controlled changes to generated scenes.
Loading 1801.05091v2…