Generating Illustrated Instructions

1. Introduction

• LLMs can provide personalized instructions and adapt after mistakes, but they cannot generate visuals needed for tasks involving visual inspection.Examples include judging when onions are golden or looking for bubbles in a submerged tire.Source: §1 · p. 1
• The paper defines Illustrated Instructions as steps with visualizations for a user's task and develops desiderata and metrics to assess the generations.The metrics draw on prior work in instructional-article assessment and image generation.Source: §1 · p. 1
• StackedDiffusion combines pretrained LLMs and finetuned text-to-image diffusion, using spatial tiling, concatenated text embeddings, and step-positional encodings without adding learnable parameters.It is trained on stacks of instructional images and can produce useful illustrations with limited instructional data.Source: §1 · p. 2
• 30% of cases, StackedDiffusion outperforms human-generated articles, and it strongly surpasses baseline approaches and state-of-the-art models.The authors report this outcome from ablations and human evaluations.Source: §1 · p. 2

2. Related Work

• Prior instructional-data work centers on understanding rather than generating full multimodal articles; concurrent autoregressive multi-image methods require extra parameters, while StackedDiffusion uses pretrained T2I priors for consistency without them [2] [57].The paper positions its focus as generation of complete articles with text and illustrations.Source: §2 · p. 2

3. Illustrated Instructions

• The task maps a goal to natural-language step descriptions and a corresponding illustration for each step.The paper argues that simply generating step text and then generating each image independently misses the task's additional requirements.Source: §3 · pp. 2–3
• Goal faithfulness requires every image to reflect the overall goal, but goal alignment alone does not make an image a useful step illustration.The colored-ice example shows images depicting the finished goal rather than the requested intermediate steps.Source: §3 · p. 3
• Step faithfulness requires each image to depict its step, rather than allowing the goal text to eclipse the step-specific details.Baseline T2I images in the example all reflect the goal instead of the requested steps.Source: §3 · p. 3
• Cross-image consistency requires images in one generation to agree; otherwise details such as ice color or image style can change between steps.The figure caption describes a naive frozen T2I model that fails to capture both goal and step and can vary the ice color between images.Source: §3 · p. 3, Figure 2 · p. 3

4. StackedDiffusion

• Separate goal and step encodings, marked with step-positional encodings, let the model use each text independently or in combination.The paper reports that concatenating goal and step text as a string could cause steps or the goal to be ignored and risk encoder truncation.Source: §4 · p. 3
• Spatial tiling lets the denoising U-Net generate all step images jointly, allowing information to pass between them for cross-image consistency.The images’ latents are tiled as if they were one image; the approach leverages the U-Net’s learned prior for within-image consistency.Source: §4 · p. 4
• At inference, classifier-free guidance generates a latent conditioned on the goal and N step texts, which is split and decoded into N images.The paper uses tiling along one spatial dimension for simplicity and reports that alternative tiling strategies do not significantly affect performance.Source: §4 · p. 4
• The model uses a pretrained T2I U-Net and text encoder, with the U-Net finetuned on stacked inputs and the text encoder kept frozen.The stacked conditioning increases U-Net input spatial resolution and is modeled using its existing parameters.Source: §4 · p. 4
• A zero signal-to-noise ratio at the final training diffusion timestep is used to mitigate the train–test difference for high-resolution latents.The paper describes this difference as signal remaining in training inputs at the final timestep while inference starts from pure noise.Source: §4 · p. 4

5.1. Illustrated Instructions Dataset

• The study repurposes VGSI’s WikiHow articles, using a held-out recipe subset because some high-level tasks lack visually consistent steps.VGSI articles contain a goal, 6–10 natural-language steps, and associated illustrations.Source: §5.1 · p. 5

5.3. Baselines

• 78% and 66.6% human preferences over frozen and finetuned T2I baselines, respectively; StackedDiffusion improves step faithfulness and consistency while maintaining high goal faithfulness.Human evaluators also preferred its images to ground-truth images 18.5% of the time.Source: Table 1 · p. 5, §5.3 · p. 6

5.2. Metrics

• Goal faithfulness, step faithfulness, and cross-image consistency are measured with CLIP- or DINO-based metrics; human preference is the primary overall-quality measure.CLIP features miss visual distinctions such as color, object count, and style, motivating DINO features for consistency.Source: §5.2 · pp. 5–6

5.5. Comparison to prior work

• In fully generated-output human evaluations, annotators preferred StackedDiffusion to handwritten, manually illustrated ground truth 30% of the time; it beat frozen or fine-tuned T2I baselines by more than 66% win-rate.Against StackedDiffusion, GILL had a 0% human-evaluation win rate and GILL++ 3.9%; StackedDiffusion was preferred over retrieval-based approaches, supporting the conclusion that its articles cannot simply be retrieved from the training corpus.Source: Table 2 · p. 7, §5.5 · pp. 7–8

5.4. Ablations

• More training data improves faithfulness and consistency, while removing 0SNR or step-positional encoding lowers step faithfulness; evaluators preferred six steps to four and slightly preferred it to eight.Step faithfulness falls from 61.5% to 52.8% without 0SNR and to 49.8% without step-positional encoding; the paper attributes the limited advantage of eight steps to insufficient long-article data.Source: Figure 7 · p. 6, §5.4 · p. 7

5.6. Applications

• The full system supports personalized instructions, suggests specific goals from situational context, and adapts instructions after user errors—capabilities fixed articles do not provide.Examples include dietary customization, suggesting apple crisps for someone with many apples, and offering alternative steps after a mistake.Source: Figure 8 · p. 7, §5.6 · p. 8, Figure 9 · p. 8

6. Limitations and Conclusions

• Generated Illustrated Instructions still have a gap with ground-truth data.The paper does not quantify that gap in this passage.Source: §6 · p. 8
• Future work could examine how obtaining instructional data at greater scale influences StackedDiffusion.Source: §6 · p. 8
• The authors identify newer T2I architectures for better faithfulness and quality, and text-to-video clips for each step, as future directions.Source: §6 · p. 8

7. Related Work: Text-to-Video Generation

• Many text-to-video works generate multiple frames but are typically limited to short timescales with only minor visual changes between frames.Source: §7 · p. 12
• Some recent methods generate longer videos, but at the cost of substantial parameter overhead over their base T2I models.Source: §7 · p. 12
• Factorized spatial and temporal attention suits settings with little frame-to-frame change, unlike illustrated instructions’ larger changes across consecutive images.Source: §7 · p. 12

8. Tiling Comparison

• The (3x2) tiling orientation achieved a 45.9% human-evaluation win rate, with GF 79.3, SF 60.7, and CIC 50.5; results were similar to the comparison orientation.The authors conclude that tiling strategy does not substantially affect performance.Source: §8 · p. 12

9. Latent Diffusion Models Preliminaries

• Latent diffusion generates samples by gradually denoising in a VAE-induced latent space, then decodes them into images.The passage says this objective matches the paper’s Eq. 3.Source: §9 · p. 12

10. LLM inference prompt engineering

• The LLM prompt requests a goal followed by concise, one-line numbered headline steps, with no other text and at most six steps.For goal-only input, the prompt is phrased as “How can I {GOAL}?”Source: §10 · p. 12

11. Data

• The dataset was regrouped by goal before resplitting, avoiding the original split’s distribution of images from one goal across training and validation.Training has 95,328 goals and 476,053 step-image pairs; validation has 1,711 recipe goals and 9,473 pairs.Source: §11 · p. 12

12. Metrics

• Goal Faithfulness measures whether an image matches its goal text more than three randomly selected other goals, using CLIP similarity choices.The vision-language model is fixed to evaluate generated data.Source: §12 · p. 12
• Step Faithfulness measures whether an image matches its step text more than other steps for the same goal; comparisons are meaningful for fixed step count N.The passage notes that the metric tends to decrease as the number of steps increases.Source: §12 · p. 13
• Cross-Image Consistency uses DINO similarity between step images to assess visual consistency within a sequence.DINO is used instead of CLIP because visual features such as color and object count matter for consistency.Source: §12 · p. 13
• FID is computed against the 9,473 validation images as the reference distribution, using clean-fid.Source: §12 · p. 13
• Human evaluation compares article pairs for 140 fixed validation goals, using three annotators per comparison and majority wins after removing ties.On average, 80% of annotators agreed with the decision for a sample.Source: §12 · p. 13

13. Additional Diffusion Baselines

• The goal+step joint-encoding ablation scores GF 91.6, SF 50.2, and CIC 51.6.The passage says finetuning improves SF over the frozen model, while CIC increases slightly as goal information’s influence on all images is reduced.Source: §13 · p. 13
• The step-only baseline scores GF 59.5, SF 77.2, and CIC 53.1, and strongly under-performs the final model in human evaluations.It has no goal information; its higher SF comes with much lower GF and substantially worse CIC.Source: §13 · p. 13

14. Training/Inference Cost

• StackedDiffusion took 9.7 hours to train to convergence on 8 A100 GPUs, versus 7 hours for single-step finetuning.The passage says inference time is about the same as a single-step model requiring N forward passes.Source: §14 · p. 13

15. StackedDiffusion implementation Details

• The in-house U-Net uses a T5-XXL text encoder and changes block output channels to (320, 640, 1280, 1280).Its text condition passes through attention, normalization, and projection layers before entering the U-Net.Source: §15 · p. 13

16. GILL Comparison Details

• GILL’s outputs were typically too short for full articles, often comprising one step or image, and the authors report low text quality.Prompt variations and parameter changes did not produce longer outputs adequate to illustrate a goal.Source: §16 · p. 15
• Even with fixed LLM-generated text, GILL++ image generations remained inferior to StackedDiffusion’s.The comparison was intended to assess GILL’s image generation and retrieval capabilities separately from its language modeling.Source: §16 · p. 15

17.1. LLM + CLIP

• The LLM+CLIP baseline retrieves the training image most similar to each LLM-generated step text, so its images can come from different goals.Source: §17.1 · p. 15

17.2. Goal Retrieval

• The goal-retrieval baseline selects the closest training article by CLIP text similarity and reuses its steps and associated images.The baseline is intended to test whether the model generates new articles rather than reproducing the closest training article.Source: §17.2 · p. 15

18. Additional Qualitative Results

• The qualitative examples include StackedDiffusion articles outside the recipe domain, including furniture repair.Source: §18 · p. 15

• The model can render a bowl similar to the previous one even when the instruction calls for “another” bowl.Source: §19 · p. 15, Figure 21 · p. 17
• A 3x3 cupcake grid can become a 2x4 grid because geometric arrangements are treated as similar despite differing counts.Source: §19 · p. 15, Figure 21 · p. 17
• Over-consistency can undermine step faithfulness: cauliflower appears in the steamer in Step 2 before the text says to add it in Step 3.Source: Figure 22 · p. 17