StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation

• Story continuation conditions subsequent images on an initial scene, which can provide visual information about new story elements that captions do not fully describe.The paper introduces DiDeMoSV and reformats PororoSV and FlintstonesSV for this task.Source: §1 · pp. 2–3
• The retrofitted model adds layers to copy relevant elements from the initial scene and a self-attention block that supplies global story context during frame generation.The additional modules are trained from scratch during story-continuation finetuning.Source: §1 · p. 3
• Prompt tuning freezes pretrained weights and learns task-specific embeddings, making adaptation time- and memory-efficient.Source: §1 · p. 3
• StoryDALL-E outperforms StoryGANc on three story-continuation datasets across several metrics.Source: §1 · p. 3
• The copying mechanism improves correlation with the source image, supporting better visual-story continuity and generation of low-resource and unseen characters.Source: §1 · pp. 3–4

• DALL-E generates text and image tokens, while VideoGPT adapts its architecture for video generation from a first frame but trains from scratch; this work retrofits pretrained weights for story continuation.Source: §2 · p. 4
• Existing story-visualization datasets feature recurring characters, motivating more diverse test cases and the proposed story-continuation task and dataset.Source: §2 · p. 4
• StoryGAN-based models combine recurrent text encoding, image generation, and image and story discriminators, with prior work adding alignment, discriminator, and input-structure changes.The paper uses prior models as starting points and modifies them for story continuation.Source: §2 · p. 4
• Parameter-efficient methods train added components while keeping pretrained weights frozen; this paper applies prompt tuning and retrofitting to pretrained image synthesis.The related-work passage also describes retrofitting a pretrained language model with cross-attention layers to retrieve relevant tokens.Source: §2 · pp. 4–5

3.1 Story Continuation

• Story continuation gives the model a source frame so it can use character and setting information when generating subsequent images.Story visualization models otherwise rely on training-set images, while narrative captions may omit visual attributes.Source: §3.1 · p. 5

3.2 StoryDALL-E

• StoryDALL-E adds a global story encoder for narrative context and cross-attention layers that use source-frame embeddings to condition generation.The source-frame embeddings serve as keys and values, while the preceding self-attention output serves as the query.Source: §3.2 · pp. 6–7
• The self-attention story encoder contextualizes all caption embeddings in parallel and adds positional information for each target frame.The resulting story embeddings are prepended to the current caption’s word embeddings.Source: §3.2 · pp. 6–7
• Prompt-tuning freezes pretrained weights and optimizes task-specific prompt vectors and StoryDALL-E modules, while full-model fine-tuning updates pretrained weights.The prompt is appended to the caption embeddings alongside global story embeddings.Source: §3.2 · pp. 7–8

3.3 StoryGANc

• StoryGANc adapts the StoryGAN framework to accept the source frame, combining a caption encoder, recurrent image generator, and image and story discriminators.The story discriminator judges generated image sequences, while the image discriminator evaluates individual images.Source: §3.3 · pp. 8–9

4 Datasets

• PororoSV and FlintstonesSV contain recurring characters, so their evaluations cannot measure models’ generalization to new characters.PororoSV has nine main characters and FlintstonesSV has seven, appearing throughout their respective datasets.Source: §4 · p. 9
• DiDeMoSV adds diverse real-world video-caption scenarios, with 11,550/2,707/3,378 training, validation, and test samples, each containing three consecutive frames.Its 10,000 short clips have more than 40,000 temporally localized descriptions.Source: §4 · p. 9
• Existing story-visualization datasets can be adapted by using each sequence’s first frame as the source and its remaining frames as targets.PororoSV and FlintstonesSV samples contain five consecutive frames.Source: §4 · pp. 9–10

5 Experiments

• Experiments initialize StoryDALL-E from 1.3B-parameter minDALL-E and train it for five epochs, while StoryGANc trains for 120 epochs.StoryGANc uses CLIP or distilBERT text encoders; models are trained on single A6000 GPUs.Source: §5 · p. 10
• Evaluation uses FID on all three datasets and character-classification F1 and frame accuracy on PororoSV and FlintstonesSV.FID measures distance between ground-truth and generated-image feature vectors; lower FID and higher F1 and frame accuracy are better.Source: §5 · p. 10, Table 1 · p. 11

6 Results

• Fully finetuned StoryDALL-E improves FID over StoryGANc on PororoSV and FlintstonesSV and by a wide margin on DiDeMoSV, while character-classification scores vary by dataset and tuning setting.For PororoSV, character classification drops by 6% with finetuning and 14% with prompt tuning; frame accuracy remains low for all models.Source: §6 · pp. 11–12
• Removing cross-attention sharply increases FID across datasets, whereas removing global story embeddings causes smaller FID drops; removing both also sharply increases FID.Without cross-attention, the model no longer receives the initial image; without story embeddings, it retains that image but loses global story context.Source: §6 · p. 12, Table 2 · p. 12

6.1 Human Evaluation

• Human ratings favor StoryDALL-E over StoryGANc for visual quality and relevance on most dataset comparisons, with FlintstonesSV relevance as the exception.StoryDALL-E also wins on consistency for PororoSV and DiDeMoSV, though the reported absolute consistency and relevance results leave room for improvement.Source: Table 3 · p. 12, §6.1 · p. 13

7.1 Qualitative Analysis

• StoryDALL-E generates higher-quality images than StoryGANc in sampled examples, but its outputs still fall short of human-produced images, especially in frames with multiple characters.StoryGANc generations for DiDeMoSV are often incomprehensible; StoryDALL-E can draw on pretrained knowledge and source-frame copying for real-world concepts.Source: §7.1 · p. 14, Figure 5 · p. 15
• With source-frame guidance, StoryDALL-E better reproduces the shape and form of scarce PororoSV characters and partially copies visual traits of non-recurring FlintstonesSV characters.Examples include a purple skirt and a blue uniform.Source: §7.1 · p. 14

7.2 Retro-fitted Cross-Attention

• Retro cross-attention uses the source frame as key/value and the target frame as query, emphasizing relevant source regions for copying visual attributes.Attention is concentrated near the pink bird in one example and follows a diagonal when source and target frames are nearly similar.Source: §7.2 · pp. 14–15, Figure 5 · p. 15

8 New Results & Demo with DALL-E Mega

• DALL-E Mega-based StoryDALL-E adds decoder cross-attention for source-image copying and a global story encoder for sequential frame generation, then fully finetunes the encoder and decoder.DALL-E Mega uses an encoder-decoder architecture and was pretrained on 15 million Conceptual Captions images.Source: §8 · p. 15, Table 4 · p. 16
• The fully finetuned mega-StoryDALL-E improves FID by up to 3% over StoryDALL-E across datasets, with smaller improvements in character-classification scores.Table 4 reports FID, character-classification F1, and frame accuracy on PororoSV, FlintstonesSV, and DiDeMoSV.Source: §8 · p. 15, Table 4 · p. 16
• The Pororo-trained model is available in an in-browser demo, where it handles stories with up to three characters across frames but struggles with more than three.It copies source-image elements across generated frames and performs best on overtly visual actions; generated scenes also tend to reflect Pororo’s snowy-village setting.Source: §8 · pp. 15–16, Figure 6 · p. 16

9 Conclusion

• StoryDALL-E adapts pretrained transformer-based text-to-image models for story continuation and outperforms GAN-based models on the story continuation datasets.The paper also introduces DiDeMoSV and reports improved results using the larger DALL-E Mega model.Source: §9 · p. 17

A.1 Story Visualization

• Story visualization generates a corresponding image sequence from temporally ordered captions that describe the same narrative.The paper notes that captions may not provide enough detail for models to depict new characters or other story elements.Source: §A.1 · p. 21

A.2 Pretrained Text-to-Image Synthesis Models (DALL-E)

• DALL-E combines a VQVAE that converts images into discrete tokens with an autoregressive transformer that models text and image tokens jointly.The transformer predicts image tokens using the text and previously predicted image tokens.Source: §A.2 · pp. 21–22

B.1 StoryDALL-E

• StoryDALL-E’s best tested retro-layer density places one retro layer in every third minDALL-E self-attention block.The densest tested setting increased model parameters by nearly 60%.Source: §B.1 · p. 22
• StoryDALL-E trains with a combined text and image cross-entropy objective, using caption and image sequence lengths of 64 and 256.These lengths are specified for the model’s text and image sequences, respectively.Source: §B.1 · p. 22

B.2 StoryGANc

• StoryGANc uses a pretrained language model to encode captions with context from other story timesteps.For RoBERTa, captions are concatenated and supplied at each timestep; for CLIP, a self-attention block contextualizes timestep representations.Source: §B.2 · p. 23
• StoryGANc matches story-representation patches to source-frame patches, uses the selected patches in generation, and trains with image and story discriminators.The story discriminator uses 3D convolution over generated images; the image discriminator evaluates individual images.Source: §B.2 · pp. 23–24, Figure 7 · p. 24

C.1 Dataset Construction

• DiDeMoSV is built by ranking 10 sampled frames per clip against each caption, then forming overlapping three-frame sequences and cartoonizing the selected images.The dataset is derived from DiDeMo video-captioning data, and video-level train, validation, and test splits are kept separate.Source: C Dataset Construction · p. 24, §C.1 · p. 25

D Experimental Details

• The study uses open-source minDALL-E (1.3 billion parameters, trained on 14 million text-image pairs from CC3M and CC12M), adapts it to StoryDALL-E, then prompt-tunes or fine-tunes the retrofitted model on target datasets.Source: D Experimental Details · p. 26
• StoryDALL-E and mega-StoryDALL-E train for five epochs, while LM-StoryGAN trains for 120 epochs.StoryDALL-E learning rates are 1e-04 for fine-tuning and 5e-04 for prompt-tuning; full-model fine-tuning uses 1e-05.Source: D Experimental Details · p. 26
• Evaluation uses FID across datasets and character F1 and frame accuracy on PororoSV and FlintstonesSV, but not DiDeMoSV.The character-classification metrics are omitted for DiDeMoSV because it lacks recurring characters.Source: D Experimental Details · p. 26

E Additional Results

• Fully fine-tuned StoryDALL-E performs best across all three validation datasets on FID, while its character scores are close to StoryGANc on FlintstonesSV and relatively lower on PororoSV.Manual analysis nevertheless finds better visual quality for StoryDALL-E; the authors note that classifier scores may favor fine character details over shape and form.Source: E Additional Results · p. 27
• StoryDALL-E is preferred for visual quality and caption alignment, while attribute-consistency wins and losses are nearly uniform on PororoSV and DiDeMoSV and favor it on FlintstonesSV.The authors attribute the visual-quality preference to higher-resolution VQGAN-VAE images and caption alignment to DALL-E’s pretrained knowledge.Source: E Additional Results · p. 27

F.1 Story Visualization vs. Story Continuation

• In the illustrated PororoSV example, caption-only VLCStoryGAN places the story indoors, whereas source-conditioned StoryGANc continues the snowy-field setting.The authors connect the difference to captions lacking setting and background details.Source: §F.1 · pp. 28–29, Figure 8 · p. 28

F.2 Correlation between Source and Generated Images

• Source-image conditioning increases source-to-generated-image cosine similarity by up to 0.3 points across all three datasets.The comparison uses CLIP image embeddings and reports mean and standard deviation; Table 6 summarizes the conditioned and unconditioned results.Source: §F.2 · p. 29, Table 6 · p. 29

F.3 Semantic Analysis of the DiDeMoSV dataset.

• Many DiDeMoSV verbs describe movement or vision, which the authors relate to simple event chains relevant to story continuation.Examples include “walks,” “turns,” “see,” and “looks.”Source: §F.3 · p. 29
• YOLOv3 object analysis excludes frequent DiDeMoSV nouns such as camera because they lack corresponding model classes; at the default confidence threshold of 0.25, it generates predictions for only 76% of DiDeMoSV images.The person class is dominant in both datasets.Source: §F.3 · p. 30
• DiDeMoSV’s caption and frame contents span actions, objects, and settings, and the authors present its diversity as a challenging benchmark alongside PororoSV and FlintstonesSV.The noun, verb, and object-frequency plots summarize these dataset contents.Source: §F.3 · pp. 29–30, Figure 9 · p. 31