Source-linked AI summary
Specifying Object Attributes and Relations in Interactive Scene Generation
Oron Ashual, Lior Wolf
TL;DR
Image synthesis needs controllable ways to specify what objects appear and where they are. The paper uses dual scene-graph embeddings for layout and appearance, yielding improved performance over existing baselines while supporting interactive, diverse, and per-object-controlled generation.
Problem
Image synthesis requires an accessible, manipulable representation of object identity, appearance, and spatial relations for specifying desired outputs.
Method
The method combines per-object location and appearance embeddings with a scene-graph-driven network and an interactive interface supporting archetype selection and semantic appearance import.
Results
The dual encoding, architecture, and loss terms lead to improved performance over existing baselines, with realistic outputs, stronger scene-graph matching, and higher diversity.
Takeaways & Limitations
Users can create novel scenes interactively, vary object placement and appearance, and duplicate objects from other images without pixel-level copying.
Takeaways & Limitations
Comparisons are limited because some baselines report only 64x64 results, one accuracy measure is incompatible, and code or competitive numbers are unavailable for others.
Abstract
from arXiv · showhide
We introduce a method for the generation of images from an input scene graph. The method separates between a layout embedding and an appearance embedding. The dual embedding leads to generated images that better match the scene graph, have higher visual quality, and support more complex scene graphs. In addition, the embedding scheme supports multiple and diverse output images per scene graph, which can be further controlled by the user. We demonstrate two modes of per-object control: (i) importing elements from other images, and (ii) navigation in the object space, by selecting an appearance archetype. Our code is publicly available at https://www.github.com/ashual/scene_generation
1. Introduction
The paper represents user intentions with scene graphs that combine object classes, spatial relations, location attributes, and controllable appearances. Its dual encoding and interactive architecture support semantic object manipulation, automatic relation inference, and diverse image generation.
- User representation: Scene graphs encode objects as classes with appearance attributes and connect them through relative spatial relations such as “above” or “left of”.Appearances can be selected from clustered archetypes or copied from a sample image.
- Dual encoding: Each object receives a location embedding and an appearance embedding, separating placement and global image features from object-specific visual characteristics.The placement representation is generated from the scene graph, while appearance information can be replaced independently.
- Graph and interface: The scene graph stores object classes, coarse 5 × 5-grid locations, ten discretized size values, and relations including “inside” and “surrounding”.The interface infers edge relations from relative object positions and renders scenes in real time.
- Architecture: The network converts scene-graph information into per-object embeddings, masks, bounding boxes, and appearance embeddings before combining them into a multidimensional tensor for image synthesis.Appearance vectors are imported from other images or selected from archetypes.
- Interactive control: The system supports interactive additions, resizing, and appearance replacement while generating corresponding scene graphs, layouts, and images.The illustrated workflow includes adding giraffes, sheep, and airplanes, enlarging objects, and changing an appearance archetype.
- Design differences: Relative to prior scene-graph generation, the method separates layout from appearance and adds stochasticity and specialized losses to increase control, diversity, and output quality.The introduction also identifies mask and counterfactual appearance discriminators, feature matching, and perceptual loss as architectural differences.
2. Previous Work
Prior work covers GAN-based conditional generation, scene-graph and layout-conditioned synthesis, text-conditioned generation, and interactive neuron manipulation. The paper situates its approach among methods that control image content through different input representations and interfaces.
- Conditional generation: Conditional image generation creates images matching a specified input, including class labels, text descriptions, or source images in image translation.Pix2pix requires paired samples from two domains, while Pix2pixHD influenced later image and video mapping systems.
- Scene-graph representations: Scene graphs have been used for image generation, text retrieval, and datasets such as COCO-stuff and the visual genome.Related synthesis work also conditions on bounding-box layouts rather than inferring layouts from scene graphs.
- Architectural context: The paper’s composite architecture includes subnetworks for scene-graph embeddings, masks, bounding boxes, appearance embeddings, multiplexing, and image reconstruction.Figure 2 summarizes the flow from scene graph to layout tensor and final image.
- Alternative conditioning: Other methods generate images from explicit bounding-box layouts at 64x64 resolution or from input sentences without constructing a scene graph as an intermediate representation.These approaches differ in whether spatial structure is supplied directly or represented through a graph.
- Interactive manipulation: An interactive GAN-dissection tool manipulates localized neurons to add, expand, reduce, or remove objects through a drawing interface.The paper characterizes its own manipulation as more semantic and more connected to spatial relations between objects.
3. Method
The method represents each scene-graph object with separate location and appearance information, then combines generated masks, boxes, and appearance embeddings to produce an image. Users can control layouts and appearances through coarse spatial attributes, imported object appearances, and class-specific archetypes.
- Scene-graph representation: Each object node combines a learned class embedding with a location vector encoding a 5 × 5 grid position and one of ten size values.Scene-graph edges encode six spatial relations between objects.
- Appearance control: Appearance embeddings are extracted by a CNN from cropped object images and can be replaced with embeddings imported from another image of the same class.The replacement is semantic rather than pixel-level and is trained using counterfactual appearance substitutions.
- Dual encoding and synthesis: A graph convolutional network converts the scene graph into per-object layout embeddings, while separate networks generate object masks and bounding boxes.Masks depend on per-object random vectors, enabling variation in generated object layouts.
- Dual encoding and synthesis: A fixed tensor-construction function combines per-object masks, boxes, appearance embeddings, and class indicators before an encoder-decoder network outputs the image.The resulting tensor has spatial dimensions H × W and channels for appearance and class information.
- Training objectives: Training uses reconstruction, box, perceptual, feature-matching, and adversarial objectives across mask, object, and image discriminators.The image discriminator operates at full and half scales, while the object discriminator evaluates cropped generated objects.
- Interactive controls: The interface supports object placement, ten displayed size levels, 100 class-specific appearance archetypes, and appearance selection or copying from other images.Archetypes are obtained by applying the appearance network to training objects and clustering each class with k-means.
4. Experiments
Experiments evaluate image quality, diversity, layout accuracy, controllability, and user preference across ground-truth and inferred layouts. The method generally outperforms baselines, supports diverse and appearance-controlled generation, and shows that sampled locations can sometimes be unrealistic.
- Experimental setup: The evaluation uses COCO-Stuff, compares against prior methods, and tests ground-truth and inferred layouts at 64×64, 128×128, and 256×256 resolutions.The dataset contains approximately 25,000 training, 1,000 validation, and 2,000 test images.
- Metrics: Image quality is assessed with inception, FID, and classification accuracy, while additional measures evaluate diversity and bounding-box placement.Higher inception, classification accuracy, diversity, and IoU/recall scores are favorable, whereas lower FID is favorable.
- Image-generation results: Our method obtains a significant lead over baselines across reported image-generation scores for both ground-truth and generated layouts.Ground-truth layouts achieve better scores than generated layouts.
- Image quality and diversity: The method produces realistic outputs, better matches test images when using ground-truth attributes, and achieves higher diversity than baselines even with fixed location and appearance attributes.Varying either location or appearance attributes adds substantial diversity.
- Controllability: Sampling appearance archetypes or location attributes produces visually diverse images, although some sampled locations are incompatible with realistic scenes; zero is the default location value.Appearance embeddings can also be copied from unseen reference images while preserving visual quality across the generated scene.
- Layout and interaction: Bounding-box placement improves over the baseline even without location attributes, and adding those attributes further improves results.The appearance-copying experiment processes a 256×256 graph in 16.3ms.
- User study: A user study with 20 students found significant improvements over the baseline in realism, scene-graph adherence, and the number of represented objects.The study used 30 random test scene graphs per participant.
- Ablation analysis: Removing any evaluated loss noticeably degrades performance; removing the perceptual loss is especially harmful, while removing the mask discriminator is the most damaging discriminator ablation.Replacing the image discriminator with the baseline’s version also reduces accuracy.
5. Conclusion
The paper presents an image-generation tool conditioned on scene graphs and optional location information, combining location and appearance embeddings to improve performance over existing baselines.
- The tool generates images from scene graphs, optionally augmented with object location information.
- Each object receives both a location embedding and an appearance embedding.
- Appearance embeddings can be extracted from another image to duplicate existing objects in a new, substantially different layout.
- The dual encoding, architecture, and loss terms together improve performance over existing baselines.
A. Network architecture
The network architecture combines graph-based scene encoding with specialized notation for convolutional, upsampling, normalization, and discriminator components.
- The graph convolutional network concatenates object embeddings with location attributes and relation embeddings before producing per-object and per-relation vectors in R128.
- The architecture uses shorthand for convolutional layers with specified filters, kernels, strides, normalization, activations, padding, and upsampling operations.
- Discriminator components are described with notation covering input and output channels, filters, strides, padding, Leaky-ReLU, instance normalization, and average pooling.
- The remaining network components are enumerated as separate mask, appearance, and reconstruction-related modules.