Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction

1. Introduction

• Complex caption-conditioned generation remains difficult: the passage describes good generators for complex datasets such as MS COCO as nonexistent and suggests captions may not contain enough information for an entire image.It notes that combining object annotations with intermediate bounding-box and mask generation can improve results.Source: §1 · pp. 1–2

1.1. GeNeVA Task and Datasets

• The task replaces caption-conditioned one-shot generation with iterative image generation from continual linguistic input, while leveraging advances in text-conditioned single-image generation.Source: §1 · p. 2
• GeNeVA requires the Drawer to map complex instructions to objects and relationships, then modify the existing image consistently with previous images and instructions.The Teller provides sequential instructions and can gauge progress through visual feedback.Source: §1.1 · p. 2
• Experiments use CoDraw and the introduced i-CLEVR dataset, whose linguistic descriptions guide incremental construction of CLEVR scenes.The authors use synthetic datasets to assess progress on the task and improve the iterative generation process.Source: §1.1 · p. 2

1.2. Contributions

• The paper introduces a recurrent GAN for modifying images in the context of instructional history, and reports that it can build images incrementally, move and resize objects, and outperform a non-iterative baseline.It also introduces i-CLEVR and a relational similarity metric for evaluating plausible object placement.Source: §1.2 · p. 2

• Caption-conditioned image-generation work includes Reed et al., StackGAN, Hong et al.’s three-step use of boxes and masks, and AttnGAN.The passage describes AttnGAN as using spatial attention and an image-text similarity module.Source: §2 · pp. 2–3
• ChatPainter generates one final image from a caption and dialogue, without intermediate images for each dialogue turn, unlike an artist updating a sketch from feedback.Source: §2 · p. 3
• Earlier recurrent generation approaches aimed at a single final image without external feedback; Lin et al. also use fixed object templates and do not modify existing objects.Yang et al. recursively generate background, foreground, and mask components.Source: §2 · p. 3
• GeNeVA’s distinctive task is to recursively generate and modify intermediate images from continual text instructions while keeping each image consistent with past instructions.Source: §2 · p. 3

3.1. Model

• Each generated image is conditioned on the prior canvas representation and a context-aware encoding of the current instruction and history.The prior image is encoded separately, while a recurrent unit updates the context-aware condition from the current instruction and previous condition.Source: §3.1 · pp. 3–4
• The discriminator fuses features from current and previous images to focus on modification quality, and is conditioned on the context-aware instruction state.The model tests element-wise subtraction and concatenation as feature-fusion options.Source: §3.1 · p. 4
• The discriminator also scores real images paired with wrong instructions and predicts which objects are present at the current step.The auxiliary object-detection task uses binary cross-entropy over possible objects.Source: §3.1 · pp. 4–5
• The generator objective combines adversarial training with an auxiliary object-detection loss, while a zero-centered gradient penalty regularizes the discriminator on real data.The generator loss weights the auxiliary loss by β; the gradient penalty uses weighting factor γ.Source: §3.1 · p. 5

3.2. Implementation Details

• Implementation uses ResBlocks, 16×16 self-attention, and discriminator spectral normalization; training teacher-forces prior frames and updates GAN parameters each time step.The image encoder, recurrent condition unit, and text encoder are updated once per sequence; the text encoder and recurrent unit use the discriminator objective only.Source: §3.2 · p. 5

4. Datasets

• CoDraw is the only publicly available dataset identified for GeNeVA, so the authors introduce i-CLEVR for the task.GeNeVA requires textual drawing instructions and corresponding ground-truth images for each instruction.Source: §4 · p. 5

4.1. CoDraw

• CoDraw contains 9,993 clip-art-like scenes with 58 object types and Teller–Drawer dialogue accompanying canvas updates.Preprocessing concatenates turns until the image changes, so each retained turn corresponds to an object being added or removed.Source: §4.1 · p. 5

4.2. i-CLEVR

• Each i-CLEVR example has five image-instruction pairs, with instructions specifying an object's shape, color, and position relative to existing objects.Instructions sometimes refer to the most recently added object as “it,” requiring context from the sequence.Source: §4.2 · p. 5
• i-CLEVR contains 10,000 sequences totaling 50,000 images and instructions, split into 6,000 training and 2,000 each for validation and testing.Each sequence begins from an empty background canvas.Source: §4.2 · p. 6

5.1. Evaluation Metrics

• Evaluation uses object detection and scene-graph relations; relational similarity is a lower bound because valid instruction-compatible graphs can differ from the ground-truth layout.The graph metric compares relations for objects detected in both images and penalizes differences in positioning even when the instructions permit alternate graphs.Source: §5.1 · p. 6

5.2. Ablation Study

• The ablation tests adversarial-only Baseline, Mismatch, G prior, Aux, discriminator fusion by concatenation or subtraction, and a single-step Non-iterative model.The Non-iterative variant concatenates all instructions into one instruction; the tested component sets are summarized in Table 2.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3. Results

• Subtraction fusion outperforms the other tested models and improves almost all metrics on both datasets; iterative generation also outperforms the non-iterative baseline.The comparison is reported in the ablation results; the authors attribute subtraction fusion's advantage to encouraging the discriminator to focus on modifications.Source: Table 1 · p. 7, §5.3 · p. 8
• Baseline performs weakest across both datasets, while adding the wrong-pair discriminator term improves all metrics over Baseline.The authors attribute the Baseline weakness to rebuilding each image from scratch without consistency between successive generations.Source: §5.3 · p. 7
• G prior improves all i-CLEVR metrics but has mixed effects on CoDraw; Aux trades precision against recall and relational similarity differently across datasets.G prior slightly improves CoDraw recall and relational similarity while reducing precision; Aux improves CoDraw recall and relational similarity but lowers precision, with the opposite pattern for i-CLEVR.Source: §5.3 · p. 7
• On CoDraw, generations capture coarse scene details but struggle with fine details and instructions adding several objects; i-CLEVR generations capture colors and spatial relations accurately.The CoDraw examples include difficulty with tiny objects, facial expressions, and poses.Source: §5.3 · p. 8
• On crowded i-CLEVR scenes, the model sometimes fails to add a fifth object; with an intermediate ground-truth image as the test-time starting image, it can still place objects correctly.The intermediate-image experiment is reported in the appendix.Source: §5.3 · p. 8

6. Conclusion and Future Work

• The recurrent GAN draws reasonable images iteratively and significantly outperforms the non-iterative baseline.The authors identify photo-realistic images, transitions, and instruction annotations as prohibitively expensive to collect, leaving no appropriate public dataset.Source: §6 · p. 8
• Future work could make the system more useful by allowing it to ask users for clarification when needed.The authors also state that scaling the task to photo-realistic images requires datasets of images, transitions, and linguistic instructions.Source: §6 · p. 8

A. Object detector and localizer network

• The GeNeVA evaluation metrics rely on the object detector and localizer network, which therefore needs high detection and localization performance.The paper reports mean test-set Precision, Recall, and F1-Score for the detector and localizer, plus centroid NRMSE; higher is better for the first three and lower for NRMSE.Source: A. Object detector and localizer network · p. 10, Table 3 · p. 10

B. Relational Similarity metric: rsim

• Front-behind and left-right relationships are determined from predicted object-center coordinates.Source: §B.1 · p. 10
• The object detector and localizer predicts centers for both ground-truth and generated images.Source: §B.1 · p. 10
• Predicted centers are used instead of renderer perspective coordinates because those are available only for i-CLEVR ground truth.Source: §B.1 · p. 10

B.2. Appropriateness for evaluation

• The detector predicts binary class presence and the localizer regresses one centroid per class because each dataset image contains at most one instance per class.The authors say this design breaks with multiple instances of a class or frequent object misclassification.Source: §B.2 · p. 10
• Over-generation is rare in qualitative assessment, and the object detector accuracy is very high.Source: §B.2 · p. 10
• Multiple instances per class are out-of-distribution, and the authors report no observed over-generation failure; rsim is position-sensitive and would need a union-of-detections denominator for such datasets.Source: §B.2 · p. 10

B.3. Shortcomings

• Quantitative measures for attributes such as “boy kicking” are missing, and the commonly used Inception Score and FID do not account for attributes.The authors identify an attribute-aware evaluation metric as a valuable future contribution.Source: §B.3 · p. 10

B.4. Comparison with existing metrics

• SSM scores exact reconstructions and covers attributes and spatial relations, but lower detector accuracy for some attributes reduces metric reliability.The paper instead focuses on generating complete images and rewarding plausible generations with correct left-right and front-behind relationships.Source: §B.4 · p. 10

B.5. Qualitative evaluation

• Figure 6 shows generated examples spanning rsim scores from 0 to 1 to illustrate how the metric reflects matched spatial relationships.Source: §B.5 · p. 11

C. Generation Examples

• Selected best-model examples cover CoDraw and i-CLEVR, while random CoDraw examples compare models from the ablation study.The comparisons include Baseline, Mismatch, G prior, Aux, D Concat, D Subtract, and Non-iterative models.Source: C. Generation Examples · p. 11

D. Generalization to new background images

• After training with an empty background, GeNeVA-GAN placed a desired object with the correct location, color, and shape on an object-containing test background.The authors say this demonstrates generalization to a background not used for training and recognition of existing objects without instruction history.Source: D. Generalization to new background… · p. 11

E. i-CLEVR Dataset Generation

• Each step renders an object with a unique shape-and-color combination; later objects are placed visibly and at a minimum distance from others.Objects have one of three shapes and one of eight colors, with fixed material and size; the first object is centered.Source: E. i-CLEVR Dataset Generation · p. 11
• The second instruction uses a template naming an object's color and shape and specifying its relative depth and horizontal position.Source: E. i-CLEVR Dataset Generation · p. 11
• From the third instruction onward, an object's position is described relative to two randomly chosen existing objects.Source: E. i-CLEVR Dataset Generation · p. 11

F. Additional implementation details

• Each instruction uses 300-dimensional GloVe embeddings encoded by a bidirectional GRU into a 1024-dimensional representation.Source: F. Additional implementation details · p. 11
• The higher-level GRU state dimensions and conditioning augmentation output are each 1024.Source: F. Additional implementation details · p. 11
• Training used batch size 32 on two NVIDIA P100 GPUs, with gradient norm clipped at 50 and image-encoding learning rates of 0.006.Generator and discriminator optimizer betas were (0.0, 0.9), and weight decay was 0.Source: F. Additional implementation details · p. 11

• The bi-directional-GRU trained over GloVe embeddings performed better than skip-thought encoding, so the paper used the former.Source: G. Additional language encoder… · p. 11
• Passing previous-image features directly to the Generator made it easier for the model to generate an accurate image than passing the image through the language encoder.Source: G. Additional language encoder… · p. 11

G. Additional language encoder experiments

• In an untrained setting with a different initial image, GeNeVA-GAN adds the desired object at the correct location and preserves the canvas and relationships to existing objects.The experiment indicates the model understands existing objects while adding new ones with the requested properties.Source: Figure 16 · p. 21