SEED-Story: Multimodal Long Story Generation with Large Language Model

• Multimodal story generation must comprehend intertwined images and narrative text while maintaining coherent plots, characters, and visual styles.Source: §1 · p. 2
• SEED-Story leverages an MLLM’s multimodal comprehension to generate narrative text and images aligned with that narrative.Source: §1 · p. 2
• The model predicts text tokens and image features, then adapts an SD-XL-based de-tokenizer to preserve character and style consistency.Source: §1 · p. 2
• Multimodal attention sink retains selected beginning and image-boundary tokens, enabling efficient generation of stories with sequences longer than those used for training.Source: §1 · p. 3
• StoryStream is a large-scale, high-resolution dataset with longer sequences and detailed narratives, and SEED-Story achieves superior image-style consistency, story engagement, and image-text coherence.Source: §1 · p. 3
• The paper’s contributions are SEED-Story, multimodal attention sink, and StoryStream for multimodal story generation.Source: §1 · p. 3

• Table 1 compares datasets by image count, resolution, story length, and average text length per sentence.Source: Table 1 · p. 4
• Personalized story visualization generates consistent images from provided captions, whereas multimodal story generation creates consecutive storylines and corresponding images.Source: §2 · pp. 3–4
• Multimodal story generation must predict story developments and illustrations autoregressively while incorporating previous results as context.Source: §2 · p. 4
• The paper adopts a closed-set setting and identifies diverse scenarios and generalization to unseen characters as future directions.Source: §2 · p. 4

MLLM for Multimodal Story Generation

• StoryGPTV requires additional character and object masks, while MM-interleaved uses complex multi-scale attention that makes long multimodal stories difficult to generate.Source: §2 · p. 4

Visual Story Dataset

• Existing datasets trade off realism, character consistency, resolution, scale, or narrative simplicity; StoryStream targets global consistency, scale, high resolution, and narrative-rich text.Source: §2 · p. 4

3.1 STORY GENERATION WITH MULTIMODAL LARGE LANGUAGE MODEL

• ViT features are passed through a learnable module into SD-XL’s U-Net, using open-world image-text pairs and story data to preserve image information.Source: §3.1 · pp. 4–5
• SEED-Story’s three-stage pipeline trains an SD-XL de-tokenizer, instruction-tunes an MLLM on text and image features, and adapts the de-tokenizer for consistent outputs.Stage 1 reconstructs images from ViT features; Stage 2 uses next-word prediction and image-feature regression; Stage 3 tunes SD-XL with regressed features.Source: Figure 3 · p. 5
• Instruction tuning samples random-length stories and trains next-token prediction plus cosine-similarity regression between MLLM outputs and target image features.A LoRA module is used during SEED-Story fine-tuning.Source: §3.1 · p. 5
• De-tokenizer adaptation aligns style and texture by training only SD-XL, avoiding optimization conflicts and reducing memory requirements.Source: §3.1 · p. 6

3.2 LONG STORY GENERATION WITH MULTIMODAL ATTENTION SINK

• Long-story generation is difficult because extended interleaved datasets are rare, short-training/long-testing causes degradation, and longer contexts increase memory and computation.Source: §3.2 · p. 6
• Sliding-window attention disrupts KV-cache token relationships, while standard attention sink is less effective for multimodal contexts.Source: §3.2 · p. 6
• Attention analysis identifies starting, punctuation, BoI, and EoI tokens as focal points, with MLLMs especially attending to image tokens near BoI and EoI.Source: §3.2 · pp. 6–7
• Multimodal attention sink retains starting tokens and image tokens near BoI and EoI, enabling high-quality images with low computational cost.Source: §3.2 · p. 7

4.1 DATASET CONSTRUCTION

• StoryStream combines cartoon keyframes, subtitles, and detailed image descriptions, then uses GPT-4 to generate narrative texts for training story-generation models.The construction pipeline aggregates groups of visual and textual elements with series background information.Source: §4.1 · p. 8

4.2 KEY FEATURES

• 257,850 released images make StoryStream a large-scale dataset featuring consistent main characters.The dataset comprises three subsets.Source: §4.2 · p. 8
• 480x768 images provide substantially higher resolution than existing datasets’ 128x128 images.Source: §4.2 · p. 8
• StoryStream uses abstract, detailed, narrative, story-toned text rather than simple name-and-action descriptions.The dataset is designed to provide richer narrative content for story visualization.Source: §4.2 · p. 8
• Stories contain up to 30 images with corresponding texts that convey cohesive progression through extended narratives.Source: §4.2 · p. 8
• With the same initial image, different starting texts can produce different storylines, including or omitting the man in the yellow hat.Source: Figure 7 · p. 9

5.1 QUANTITATIVE RESULTS

• SEED-Story outperforms MM-Interleaved in story engagement and image-text coherence, with slightly higher preference for image style consistency.FID evaluates visual quality, while GPT-4V comparisons assess style consistency, story engagement, and image-text coherence.Source: §5.1 · p. 9

5.2 QUALITATIVE RESULTS

• Qualitative results show that SEED-Story generates different plots and corresponding illustrations from different user-provided beginnings.Additional examples are reported in the appendix.Source: §5.2 · p. 10

5.3 MULTIMODAL ATTENTION SINK

• Without multimodal attention sink, alternative attention mechanisms produce meaningless images as long-story inference extends beyond training length.Figure 8 visualizes this degradation in later frames.Source: §5.3 · p. 10, Figure 8 · p. 10
• Multimodal attention sink consistently produces high-quality long-story images while improving efficiency over dense attention with only modest extra time and memory versus window attention and vanilla attention sink.The added cost comes from retaining extra image tokens in the key-value cache, and Table 2 supports the conclusion.Source: §5.3 · p. 10, Table 2 · p. 10

6 CONCLUSION

• SEED-Story uses an MLLM to generate long multimodal stories with rich narrative text and contextually relevant images.Source: §6 · p. 10
• The multimodal attention sink enables efficient generation of long sequences beyond the training sequence length.Source: §6 · p. 10
• StoryStream is presented as a high-quality dataset for training and benchmarking multimodal story generation.Source: §6 · p. 10

A USER STUDY

• SEED-Story clearly outperforms the baseline in image generation ability and text-to-image coherence, with slightly higher preference for story text quality.The user study compares 125 samples across image quality, style consistency, diversity, story engagement, and image-story coherence.Source: A USER STUDY · p. 15, Table 3 · p. 15

B ABLATION STUDY ON DE-TOKENIZER ADAPTATION

• De-tokenizer adaptation improves image style and character consistency and enhances image quality according to FID scores.Before adaptation, images retain semantic relevance, backgrounds, and characters but exhibit texture distortion and style inconsistency.Source: B ABLATION STUDY ON DE-TOKENIZER… · p. 15, Table 4 · p. 15
• Before the third stage, generated images preserve semantic information and mostly correct backgrounds and characters but have low-quality textures and inconsistent cartoon style.Figure A compares images with and without de-tokenizer adaptation.Source: B ABLATION STUDY ON DE-TOKENIZER… · p. 15

C.1 VISUAL TOKENIZATION AND DE-TOKENIZATION

• 64 learnable queries process 256×4096 ViT image features, while a second Q-Former maps MLLM outputs to SD-XL conditioning embeddings.Images are resized to 448×448 before ViT encoding; the visual feature shape is [256, 4096].Source: §C.1 · pp. 15–16

C.2 INSTRUCTION TUNING

• Instruction tuning samples a random story length and computes losses on the final target sequence after preceding story sequences and the user prompt.The format includes a beginning-of-story marker, user prompt, preceding sequences, a target sequence, and an end-of-sequence marker.Source: §C.2 · p. 16

C.3 DE-TOKENIZER ADAPTATION

• The SD-XL de-tokenizer is fully fine-tuned while MLLM parameters remain fixed, using three epochs with bf16 training and regularization.The stage uses a 1 × 10−4 learning rate and 0.03 weight decay on four NVIDIA-A800 GPUs.Source: §C.3 · p. 16

D.1 ATTENTION MAP VISUALIZATION

• Attention visualizations consistently focus on “0” tokens, punctuation, and tokens adjacent to image boundaries across lengths, heads, and layers.These patterns provide empirical support for the token-preservation design used in multimodal attention sink.Source: §D.1 · p. 16

D.2 STATISTICS ON TOKEN’S ATTENTION

• Aggregating the top 10 keys by occurrence reveals four recurring key-token categories—“0” tokens, punctuation, image tokens adjacent to BOI, and image tokens adjacent to EOI—supporting the multimodal attention-sink mechanism.The punctuation category includes “,”, “.”, and “;”, while image-token categories include BOI/IMG04 and IMG57–IMG63 near EOI.Source: §D.2 · pp. 16–17
• Image tokens near BOI and EOI repeatedly appear among highly attended keys, alongside punctuation and “0” tokens.The listed boundary categories include BOI and IMG04, plus IMG57–IMG63 and the added EOI.Source: §D.2 · p. 16

E STORY VISUALIZATION COMPARISON

• In visualization-only comparisons, SEED-Story shows better style and character consistency and higher quality than the baselines.The comparison adapts SEED-Story for visualization-only generation and evaluates it against StoryGen and LDM; quantitative evaluation is reported in Table 6.Source: E STORY VISUALIZATION COMPARISON · p. 17, Table 6 · p. 17, Figure C · p. 18

F MULTIMODAL STORY GENERATION RESULTS

• Additional results report that SEED-Story continues producing story images and text with high quality.The section presents further multimodal generation examples in Figures F, G, and H.Source: F MULTIMODAL STORY GENERATION RESULTS · p. 18
• The Rabbids Invasion visualization result provides an example of the model’s story-generation output.Figure D specifically presents a story visualization result on Rabbids Invasion data.Source: Figure D · p. 19
• SEED-Story generates long sequences with engaging plots and vivid images.The reported examples include story visualization results for Rabbids Invasion and The Land Before Time.Source: F MULTIMODAL STORY GENERATION RESULTS · p. 19, Figure D · p. 19

G.1 COMPARATIVE EVALUATION

• SEED-Story and MM-interleaved are compared on 180 five-segment stories using GPT-4 or GPT-4V win-rate judgments from a shared image-text starting point.The five-segment limit accommodates GPT-4V’s maximum of ten images per input session.Source: §G.1 · p. 19

G.2 SCORE EVALUATION

• Direct GPT-4 scoring evaluates generated stories out of 10 for style consistency, story engagement, and text-image coherence, with results reported in Table 7.The evaluation prompt instructs the judge to avoid length-related bias and follow a strict score format.Source: §G.2 · p. 23, Table 7 · p. 23

H STORY VIDEO

• The multimodal generation model’s images are animated and synchronized with audio to create a narrative video available in the supplementary materials.This video-generation technique showcases the model’s multimodal outputs in an audiovisual format.Source: H STORY VIDEO · p. 23

I.1 CURIOUS GEORGE

• Curious George follows a curious monkey guided by the Man with the Yellow Hat through educational adventures involving exploration, experimentation, and problem-solving.The series is described as teaching preschoolers about math, science, and engineering.Source: §I.1 · p. 23
• The project uses Curious George materials under license and states that its noncommercial educational research complies with PBS terms and applicable intellectual-property law.The materials are intended solely for research, with accurate citation and attribution of PBS content.Source: §I.1 · p. 23

I.2 RABBIDS INVASION

• Rabbids Invasion provides subsets derived from the French-American animated series and its characters’ humorous family-oriented adventures.The series debuted on France 3 in 2013 and has multiple seasons and global reach.Source: §I.2 · p. 24

I.3 THE LAND BEFORE TIME

• The Land Before Time supplies subsets from websites associated with its long-running dinosaur franchise, whose stories emphasize friendship, teamwork, community, and perseverance.The franchise began with a 1988 film and expanded across sequels, television, games, and merchandising.Source: §I.3 · p. 24

I.4 APPRECIATION

• Data from Curious George, Rabbids Invasion, and The Land Before Time enhanced narrative-generation models while supporting children’s imagination, interest in learning, literacy, and cognitive skills.The project frames animated storytelling as an educational and entertaining context for interactive content.Source: §I.4 · p. 24

J BROADER IMPACTS

• The project addresses potential copyright risks through strict licensing, legal-compliance measures, and user education on responsible intellectual-property use.Planned measures include user guidelines, training sessions, and support for navigating copyright law.Source: J BROADER IMPACTS · pp. 24–25

• Validation does not include realistic or in-the-wild storytelling conditions, leaving robustness to noisy and practical data uncertain.Broader test conditions could assess robustness and adaptability outside controlled settings.Source: K LIMITATIONS · p. 25
• Training is restricted to animation datasets without broad stylistic coverage, limiting generalization to unrepresented visual and narrative styles.Expanding styles, genres, and visual aesthetics could address this boundary.Source: K LIMITATIONS · p. 25