Text2Scene: Generating Compositional Scenes from Textual Descriptions

1. Introduction

• Text2Scene generates compositional scene representations by iteratively adding objects and predicting their attributes from visually descriptive language.Its sequence-to-sequence framework uses text and current-scene representations, attention, an object decoder, and an attribute decoder.Source: §1 · pp. 1–2, Figure 2 · p. 2
• Figure 1 contrasts input descriptions, Text2Scene outputs, and reference scenes for abstract scenes, object layouts, and synthetic composites.Source: Figure 1 · p. 1
• Text2Scene does not use GANs and is reported to outperform the best Abstract Scenes result, approach state of the art on COCO automatic metrics, and lead on human evaluations.The paper presents the sequential object-generation process as interpretable.Source: §1 · p. 1
• Text descriptions can imply attributes indirectly and encode spatial configurations involving multiple objects.For example, “Mike is surprised” should affect Mike’s facial attribute, while Jenny’s orientation depends on the positions of Mike and the duck.Source: §1 · pp. 1–2
• Under minor modifications, the framework handles cartoon-like scenes, semantic layouts for real images, and synthetic image composites.The experiments cover Abstract Scenes and COCO tasks.Source: §1 · pp. 1–2

2. Related Work

• Many recent text-to-image approaches use GANs, but synthesizing complex scenes with interacting objects remains challenging without explicitly defined interactions.Text2Scene draws on compositionality and generates scene objects sequentially without GANs.Source: §2 · p. 2
• For abstract scenes, introduced a graphical model, whereas Text2Scene trains end-to-end from sentences without a semantic parser.For synthetic image generation, it can optionally use the cascaded refinement module from [27], and predicts layouts indirectly from text rather than receiving them as input.Source: §2 · p. 2
• Unlike [14], [12], [16], and [9], Text2Scene infers object presence from text, trains end-to-end, and covers abstract scenes, semantic layouts, and composite images.The cited approaches differ in their inputs or training setup: for example, [14] receives scene graphs and [9] receives objects as inputs.Source: §2 · p. 3

3. Model

• At each step, Text2Scene attends to the text to select an object or stop, infers its attributes and relations, then grounds them on the canvas.Source: §3 · p. 3

3.1. Text Encoder

• A bidirectional GRU encodes each word with its context, and concatenated word embeddings and hidden vectors form the text features.Source: §3.1 · p. 3

3.2. Object and Attribute Decoders

• The object decoder combines spatial canvas context, text context for objects not yet added, and the previously predicted object; attribute attention then focuses on the current object to predict its location and attributes.Source: §3.2 · p. 4

3.3. Foreground Patch Embedding

• For synthetic composites, a patch-embedding network predicts an appearance vector trained to be closer to the target patch feature than to a same-category negative patch.Source: §3.3 · p. 4

3.4. Objective

• Training uses negative log-likelihood losses for objects, locations, and discrete attributes, with optional triplet loss for synthetic images and attention regularizers that encourage coverage of input words.Source: §3.4 · p. 4
• On Abstract Scenes, the full model improves every Table 1 metric except U-obj Coord; human ratings validate at least one statement 94% and at least two 64% of the time.Source: Table 1 · p. 5, Table 2 · p. 5

4. Experiments

• The experiments cover Abstract Scenes clip-art generation, COCO semantic-layout prediction, and COCO synthetic-image generation, with task-specific data and splits.Abstract Scenes uses clip-art attributes; COCO layouts use object boxes, while synthetic-image generation retrieves segmented object and stuff patches.Source: §4 · pp. 5–6

4.1. Evaluation

• Abstract-scene automatic evaluation measures object and object-pair precision and recall, attribute accuracy, and coordinate distances; layout and synthetic-image evaluation also use caption-generation metrics.Synthetic images are additionally assessed with Inception Score.Source: §4.1 · p. 6
• Abstract-scene comparisons include prior work and model variants that remove attention or regularization, or omit dedicated attribute attention.The attention variants are described as ablations of the full model.Source: §4.1 · pp. 6–7
• Human evaluation tests whether generated abstract scenes entail their associated statements and compares synthetic images with alternatives through pairwise judgments.Abstract-scene judgments are grouped by semantic frame, including single objects, object pairs, locations, and expressions.Source: §4.1 · p. 7

4.2. Results and Discussion

• Text2Scene improves over prior work and its variants on all Abstract Scenes metrics except U-obj Coord, with human evaluations favoring specific semantic information and comparable location judgments.Reference scenes also show that semantically related object pairs are harder to generate; the proposed metrics correlate with human judgments.Source: §4.2 · p. 7
• Captions generated from Text2Scene layouts are closest to captions generated from true layouts, while qualitative examples show object presence, counts, and spatial relations.The table includes ground-truth layouts as an upper bound; Figure 6 illustrates the predicted concepts.Source: Table 3 · p. 6, §4.2 · p. 7, Figure 6 · p. 7
• Without post-processing, Text2Scene beats prior methods on synthetic-image automatic metrics except AttnGAN; inpainting improves all metrics, and human evaluations favor it over alternatives including AttnGAN.Generated composites average 8.15 patches from 7.38 different source images, and the model was observed to generalize to uncommon situations.Source: Table 4 · p. 6, §4.2 · p. 8

5. Conclusions

• Experiments across Abstract Scenes and COCO show that Text2Scene can capture finer semantic concepts from visually descriptive text and generate complex scenes.The analysis includes quantitative and qualitative results across different scene-generation tasks and two domains.Source: §5 · p. 8

A.1. Text Encoder

• All experiments use a one-layer bidirectional GRU text encoder with 256 hidden dimensions per direction and fixed GloVe embeddings except for synthetic-image generation, where embeddings are fine-tuned.Word embeddings are initialized from GloVe.Source: §A.1 · p. 9

A.2. Scene Encoder

• Abstract-scene generation uses a fixed ImageNet-pretrained ResNet-34 scene encoder, while layout and synthetic-image tasks use purpose-built encoders.The latter tasks do not use RGB-image inputs for their scene encoders.Source: §A.2 · p. 9
• The layout encoder uses a categorical vocabulary of 83 tokens, while synthetic-image generation uses 98, including COCO object categories, stuff supercategories, and special tokens.The special tokens are start, end, and padding.Source: §A.2 · p. 9
• Synthetic-image inputs group category-specific color patches in sets of three channels, so the encoder begins with a depthwise separable convolution to reduce parameters and memory use.Each three-channel group is convolved into one output feature-map channel.Source: §A.2 · p. 9

A.3. Convolutional Recurrent Module

• The scene recurrent module is a one-cell ConvGRU with 3 × 3 stride-1 convolutions, 512 hidden dimensions, and spatially preserved feature maps.Its hidden state is initialized by spatially replicating the text encoder’s last hidden state.Source: §A.3 · p. 9
• The layout scene-encoder table defines its convolutional and residual-block notation, including the stride-2 downsampling in its second residual block.The table specifies that convolutional layers are followed by ReLU activations.Source: Table 6 · p. 9
• The synthetic-image scene-encoder table specifies a depthwise separable first convolution and LeakyReLU activations with negative slope 0.2.That first convolution maps each group of three channels to one output channel.Source: Table 7 · p. 9

A.4. Object and Attribute Decoders

• The two-layer object decoder predicts the next object likelihood, while the four-layer attribute CNN predicts location and attribute likelihoods across the grid.The attribute output includes location plus channels for discretized attributes or the appearance vector used for synthetic-image patch retrieval.Source: §A.4 · p. 9, Table 8 · p. 10
• At test time, the model samples the top-1 location first and collects the predicted attributes at that location; training uses likelihoods at ground-truth locations.Location likelihoods are normalized over the spatial domain with softmax.Source: §A.4 · pp. 9–10
• Text-based attention modules are defined similarly to [22], and trainable matrices compute attention scores for collecting context vectors.Source: §A.4 · p. 10
• The decoder architecture is shared across all three tasks, with 28 × 28 grids for abstract scenes and layouts and 32 × 32 for synthetic images.The model can compose on canvases of different sizes despite using fixed grid resolution.Source: §A.4 · p. 10

A.5. Foreground Patch Embedding

• Each foreground segment is represented by its color patch, binary foreground mask, and semantic context map, computed from a bounding box enlarged 50% in each direction.Source: §A.5 · p. 10
• A five-layer CNN embeds the concatenated segment inputs, and a linear regressor combines that feature with 2048-dimensional ResNet152 features to form the final vector.The convolutional feature is expected to encode shape, appearance, and context, while pretrained color-patch features help retrieve segments with strong semantics such as people.Source: §A.5 · p. 10, Table 9 · p. 10

A.6. Inpainting Network

• For COCO canvas simulation, the method retrieves alternative segments using trained embedding features rather than intersection-over-union and omits boundary elision to preserve fine details such as faces.The inpainting network follows [27] but excludes its layer-normalization layers.Source: §A.6 · p. 10

B. Optimization

• Training uses Adam with an initial learning rate of 5e−5, decayed by 0.8 every three epochs, clips gradient norm at 10, and stops when validation errors cease decreasing.Source: B. Optimization · p. 10
• The loss weights differ across abstract-scene, semantic-layout, and synthetic-image generation tasks.The listed weight tuples are (8,2,2,2,1,1,1,1), (5,2,2,2,1,0), and (5,2,2,2,1,0,10,0.5), respectively.Source: B. Optimization · p. 10
• The task-specific hyperparameters were selected to make different component losses comparable, and exploring better settings is left for future work.Source: B. Optimization · p. 10

C. User Study

• The clip-art study presents 100 test-set groups to three annotators each, who judge sentence-scene entailment as true, false, or uncertain; uncertain responses are excluded and majority opinions reported.Source: C. User Study · p. 11
• Synthetic-image comparisons pair Text2Scene outputs with SG2IM [14], HDGAN [41], or AttnGAN [36] for 500 test sentences, with five annotators judging which image better matches each sentence.Source: C. User Study · p. 11

D.1. Abstract Scene

• Generated clip-art can differ dramatically from reference scenes while still matching the input descriptions.Source: §D.1 · p. 11

D.2. Layout Generation

• Layout examples show that the model learns semantic concepts from language, including object presence, counts, and spatial relations.Source: §D.2 · p. 11

• The examples suggest Text2Scene learns semantic knowledge that helps retrieve object segments in similar contexts, beyond object presence and spatial layout.Source: §D.3 · p. 11
• Each synthetic image is composited from retrieved patch segments, with three source images shown for each example.Source: §D.3 · p. 11, Figure 11 · p. 14
• Figure 12 provides additional qualitative examples of synthetic image generation.Source: Figure 12 · p. 15