Source-linked AI summary
Image Generation from Scene Graphs
Justin Johnson, Agrim Gupta, Li Fei-Fei
TL;DR
Existing text-to-image methods can struggle with complex descriptions containing many objects and relationships. The paper generates images from scene graphs using graph convolution, predicted layouts, cascaded refinement, and adversarial training, and reports stronger user-study results than StackGAN. Across Visual Genome and COCO-Stuff, the method generates complex images that respect input objects and relationships, though layout prediction can bottleneck image quality and the user study favors the method because graphs may include objects absent from captions.
Problem
Text-to-image methods produce strong results in limited domains but struggle to faithfully depict complex sentences containing many objects and relationships.
Method
The method processes scene graphs with graph convolution, predicts object layouts, renders them with a cascaded refinement network, and trains jointly with two discriminators.
Results
68% of user-study trials found the method better matched COCO captions than StackGAN, and its images contained 59% more recognizable objects.
Takeaways & Limitations
Conditioning on structured scene graphs enables the method to generate complex images with multiple recognizable objects while respecting specified relationships.
Takeaways & Limitations
The user study is biased toward the method because scene graphs may contain objects not mentioned in captions, and layout prediction can bottleneck image quality.
Abstract
from arXiv · showhide
To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation masks for objects, and converts the layout to an image with a cascaded refinement network. The network is trained adversarially against a pair of discriminators to ensure realistic outputs. We validate our approach on Visual Genome and COCO-Stuff, where qualitative results, ablations, and user studies demonstrate our method's ability to generate complex images with multiple objects.
1. Introduction
The paper motivates image generation as a test of visual understanding, but finds that text-based methods struggle with complex scenes. It therefore conditions generation on scene graphs to reason explicitly about objects and relationships, using a learned layout-to-image pipeline and user studies for evaluation.
- Complex sentences with many objects are difficult for leading text-to-image methods to depict faithfully.
- Scene graphs represent objects and their relationships more explicitly than the linear structure of sentences.
- The method processes scene graphs with graph convolution, predicts object bounding boxes and segmentation masks, and converts the layout into an image with a cascaded refinement network.
- The model is trained end-to-end against image-patch and object discriminators to encourage realistic images containing recognizable objects.
- The method is evaluated on Visual Genome and COCO-Stuff using qualitative results, ablations, and user studies.
- 68% of trials judged the method's images a better match to COCO captions than StackGAN, while its images contained 59% more recognizable objects.
2. Related Work
Related work spans generative image models, conditional synthesis, scene graphs, graph learning, and structured or layout-based generation. The paper builds on these directions while processing a new scene graph on each forward pass and using a CRN to render predicted layouts.
- Generative Image Models: Generative image models include GANs, variational autoencoders, and autoregressive approaches with distinct training or likelihood objectives.
- Conditional Image Synthesis: Conditional image synthesis adds inputs such as category labels, and this work follows the approach that makes the discriminator predict the label.
- Text-to-Image Synthesis: Prior text-to-image systems use GANs, multistage generation, keypoints, or multiscale autoregressive models to condition image synthesis on sentences and related structure.
- Structured Image Synthesis: The approach uses a cascaded refinement network architecture previously applied to generate high-resolution street scenes from semantic segmentation.
- Scene Graphs: Scene graphs are directed graphs whose nodes represent objects and whose edges represent relationships, with Visual Genome providing human-annotated examples.
- Deep Learning on Graphs: The paper differs from graph-embedding methods by processing a new graph on each forward pass rather than learning embeddings for nodes in one fixed graph.
3. Method
The method converts scene graphs into realistic images by propagating information through graph convolutions, constructing an object-based scene layout, and refining that layout into an image adversarially.
- Adversarial training: A pair of discriminator networks trains the generator adversarially to produce realistic images containing recognizable objects.The discriminators operate on image and object realism, respectively.
- Graph processing: The model processes scene graphs with graph convolution layers that propagate information along edges and operate on graphs of arbitrary shape.Each layer computes updated node and edge vectors from local graph neighborhoods using shared functions.
- Scene layout: Object embeddings encode information across graph objects and relationships, then predict bounding boxes and segmentation masks that form an intermediate scene layout.The layout network predicts a soft binary mask and bounding box for each object; object layouts are summed to obtain the scene layout.
- Training and inference: At training time the model uses ground-truth bounding boxes for layout construction, whereas at test time it uses predicted boxes.The overall generator takes a scene graph and noise as input and outputs an image.
- Image synthesis: The scene layout is converted into an image with a cascaded refinement network that generates at progressively increasing spatial scales.The CRN uses convolutional refinement modules with spatial resolution doubling between modules, enabling coarse-to-fine synthesis.
4. Experiments
Experiments on Visual Genome and COCO-Stuff show that the model generates complex scenes respecting graph-specified objects and relationships. Ablations, localization analyses, and user studies evaluate the contributions and semantic quality of its components.
- Datasets and setup: The model is evaluated on Visual Genome and COCO-Stuff, using human-annotated or synthetic scene graphs and 64 × 64 image generation.COCO-Stuff graphs use six geometric relationships and exclude objects covering less than 2% of an image.
- Qualitative results: Qualitative examples contain multiple instances of object categories and place objects according to relationships in the input graph.Examples include multiple sheep, busses, people, and cars, as well as spatial and semantic relations such as left of, below, and riding.
- Qualitative results: Ground-truth layouts sometimes substantially improve image quality, indicating that predicted layout quality can bottleneck the model.The paper illustrates cases where predicted object layouts differ from ground truth and where using the ground-truth layout improves output quality.
- Ablation study: Removing graph convolution degrades performance even with ground-truth layouts, showing that graph relationships contribute beyond object-position prediction.Without graph convolution, the model cannot jointly reason about different objects and predicts only one box and mask per category.
- Object localization: Without graph convolution, box variety collapses to σx = σarea = 0, whereas the full model achieves both variety and high agreement with ground-truth boxes.Graph convolution without relationships improves variety but reduces agreement with ground-truth positions; the full model combines both properties.
5. Conclusion
The paper presents an end-to-end method for generating images from structured scene graphs, enabling explicit reasoning about objects and relationships and producing complex images with recognizable objects.
- The method generates images from structured scene graphs rather than unstructured text, allowing explicit reasoning about objects and relationships.
- The approach is designed to generate complex images containing many recognizable objects.
- The authors thank several colleagues for helpful comments and suggestions.
A. Network Architecture
This section describes the exact network architectures used for all components of the model.
- The section specifies the exact network architectures for all model components.
- The architectural descriptions cover the components used throughout the image-generation model.
- The section provides implementation-level architectural detail rather than introducing a separate model objective.
A.1. Graph Convolution Layer
The model processes scene graphs with graph convolutions, predicts object layouts through boxes and masks, and converts those layouts into images with cascaded refinement.
- A.1. Graph Convolution Layer: Graph convolution layers process node and edge vectors by propagating relationship information and pooling candidate vectors for each object.The graph convolution architecture uses shared weights, allowing one layer to process graphs of any size or topology.
- A.1. Graph Convolution Layer: The graph convolution network embeds objects and relationships, then applies five layers with Din = Dout = 128 and H = 512.
- A.3–A.4. Layout Prediction: A box regression network predicts normalized object bounding boxes, while a mask regression network predicts M × M segmentation masks.Box coordinates are parameterized as (x0, y0, x1, y1) and normalized to [0, 1]; mask values lie in (0, 1).
- A.4. Mask Regression Network: The mask-network description contains an implementation error: the main text says transpose convolution, whereas Table 7 specifies upsampling and stride-1 convolutions.The authors state that this error will be corrected in the camera-ready version.
- A.5. Scene Layout: Object embeddings, predicted boxes, and masks are combined into a scene layout without learnable parameters.The scene layout has shape D × H × W with D = 128 and H × W = 64 × 64.
- A.6. Cascaded Refinement Network: A Cascaded Refinement Network converts the scene layout into an image through cascaded refinement modules that progressively increase spatial resolution.The network uses five modules, beginning with Gaussian noise of shape 32 × 2 × 2.
A.7. Batch Normalization in the Generator
The generator uses different batch-normalization behavior during training phases to retain optimization benefits while avoiding artifacts observed when switching modes at test time.
- Using batch normalization in train mode during training and eval mode at test time led to significant image artifacts.
- The generator is trained for 100K iterations with batch normalization in train mode, followed by 900K iterations in eval mode.
- Because discriminators are unused at test time, their batch normalization remains in train mode.
A.8. Object Discriminator
The model uses an object discriminator to assess object realism and category recognition, alongside an image discriminator that evaluates local image patches. Higher-resolution variants modify layout, refinement, discriminator depth, crop size, and batch size.
- Object discriminator: The object discriminator classifies cropped objects as real or fake and predicts their object categories with an auxiliary classifier.Objects are cropped from generated or real images using bounding boxes and resized to 32 × 32.
- Image discriminator: The image discriminator classifies overlapping 8 × 8 patches from real or generated images as real or fake.Its input is a complete image, and its output covers a grid of local patches.
- Higher-resolution variant: The 128 × 128 variant computes layouts at 128×128, adds a refinement module and discriminator layer, uses 64×64 object crops, and reduces batch size from 32 to 24.These changes accompany preliminary experiments with larger output images.
B. Image Loss Functions
Ablations show that the discriminators support both smoothness and realism: removing them produces overly smooth images, while removing either discriminator introduces specific quality problems. Complex indoor scene graphs remain a failure case.
- Discriminator ablations: Removing both discriminators and using only L1 loss tends to produce overly smoothed images.The ablation compares the full model with an L1-only version.
- Discriminator ablations: Removing the object discriminator makes objects less recognizable, whereas removing the image discriminator produces low-level artifacts and less realistic images overall.The two ablations expose distinct object-level and image-level quality effects.
- Failure case: All model variants struggle to synthesize a realistic image from a complex indoor scene graph.The limitation is shown as a typical failure case in the bottom row of Figure 9.
C. User Study
Two Mechanical Turk studies compare the proposed method with StackGAN on caption matching and object visibility. The evaluation uses randomized presentation, equal method exposure, 1,024 images per method, and majority opinions from three workers.
- Study design: The first study asks users which image better matches a COCO caption: a StackGAN image or an image generated from a corresponding COCO scene graph.The order of the two images is randomized in each trial.
- Study design: The second study asks users to select which COCO objects are visible in images generated by the two methods.Repeated instances of an object category count only once, and presentation order is randomized.
- Evaluation protocol: Each study evaluates 1,024 images from each method, with every image viewed by three workers and results reported by majority opinion.StackGAN outputs are downsampled from 256 × 256 to 64 × 64 before comparison with the proposed method.