Source-linked AI summary
StoryMem: Multi-shot Long Video Storytelling with Memory
Kaiwen Zhang, Liming Jiang, Angtian Wang, Jacob Zhiyuan Fang, Tiancheng Zhi, Qing Yan, Hao Kang, Xin Lu, Xingang Pan
TL;DR
StoryMem targets the difficulty of generating cinematic, coherent multi-shot videos over long durations. It adapts pretrained single-shot diffusion models through iterative memory-conditioned shot synthesis, and experiments show superior cross-shot consistency while retaining aesthetic quality and prompt adherence.
Problem
Long-form video storytelling requires multi-shot generation with cinematic quality and long-range consistency across characters, scenes, and narrative progression.
Method
StoryMem maintains a dynamically updated keyframe memory bank and injects it into pretrained single-shot video diffusion models through M2V conditioning, negative RoPE shifts, and LoRA fine-tuning.
Results
StoryMem achieves superior cross-shot consistency over previous methods while preserving high aesthetic quality and prompt adherence on the introduced ST-Bench benchmark.
Takeaways & Limitations
Explicit visual memory enables coherent minute-long narrative video generation while retaining the strengths of pretrained single-shot video diffusion models.
Takeaways & Limitations
StoryMem may struggle with ambiguous visual-memory retrieval in complex multi-character scenarios and with fully smooth transitions when adjacent shots have large motion discrepancies.
Abstract
from arXiv · showhide
Visual storytelling requires generating multi-shot videos with cinematic quality and long-range consistency. Inspired by human memory, we propose StoryMem, a paradigm that reformulates long-form video storytelling as iterative shot synthesis conditioned on explicit visual memory, transforming pre-trained single-shot video diffusion models into multi-shot storytellers. This is achieved by a novel Memory-to-Video (M2V) design, which maintains a compact and dynamically updated memory bank of keyframes from historical generated shots. The stored memory is then injected into single-shot video diffusion models via latent concatenation and negative RoPE shifts with only LoRA fine-tuning. A semantic keyframe selection strategy, together with aesthetic preference filtering, further ensures informative and stable memory throughout generation. Moreover, the proposed framework naturally accommodates smooth shot transitions and customized story generation applications. To facilitate evaluation, we introduce ST-Bench, a diverse benchmark for multi-shot video storytelling. Extensive experiments demonstrate that StoryMem achieves superior cross-shot consistency over previous methods while preserving high aesthetic quality and prompt adherence, marking a significant step toward coherent minute-long video storytelling.
1 Introduction
StoryMem addresses the challenge of coherent minute-long, multi-shot storytelling by conditioning shot-by-shot generation on explicit visual memory. Its memory-conditioned adaptation of single-shot diffusion models improves cross-shot consistency while preserving visual quality and prompt adherence.
- StoryMem addresses the challenge of maintaining character, environment, style, and narrative coherence across minute-long, multi-shot videos.
- The method reformulates long-form storytelling as iterative shot synthesis conditioned on past keyframes and per-shot descriptions.
- StoryMem injects a compact memory bank into pretrained single-shot diffusion models using latent concatenation, negative RoPE shifts, and LoRA fine-tuning.
- Semantic keyframe selection and aesthetic preference filtering preserve informative, reliable visual memory for subsequent shot generation.
2 Related Work
Prior work either extends image-based pipelines to video or jointly models multiple shots, while video-generation memory mechanisms mainly target controllable world simulations. StoryMem instead introduces explicit visual memory for general video generation to preserve consistency across minute-long stories.
- Keyframe-based methods extend image storytelling pipelines into videos but use decoupled shot generation that limits temporal context propagation.
- Joint multi-shot methods improve cross-shot coherence but can incur quadratic computational costs and require large-scale multi-shot retraining.
- Existing video-generation memory mechanisms primarily support spatial consistency in controllable world simulations using auxiliary actions or camera poses.
- StoryMem introduces explicit memory into general video generation models to retain essential characters and background scenes throughout minute-long generation.
3 Methodology
StoryMem extends pretrained single-shot video diffusion models into coherent multi-shot generation by conditioning each shot on a compact, evolving visual memory. Its M2V design combines memory-aware conditioning, negative RoPE shifts, selective keyframe updates, and aesthetic filtering.
- Problem Formulation: StoryMem generates each shot from its text description and an evolving memory that summarizes prior characters, scenes, and stylistic information.The memory is represented by selected keyframes and supports cross-shot consistency and narrative coherence.
- Memory-to-Video: The M2V mechanism injects memory into a pretrained image-conditioned video model by encoding memory frames as latents and concatenating them with zero-filled generative-frame latents.The combined noisy latent, conditional latent, and mask are processed by the DiT for velocity prediction.
- Memory-to-Video: Negative RoPE assigns memory latents negative temporal indices, treating them as preceding events while preserving the current shot’s positions from zero.This lets the DiT attend across past memory and current video in a unified temporal space.
- Data Curation and Training: Unlike joint long-video methods, StoryMem can be trained on single-shot videos grouped by visual similarity or consistent identity, then applied through dynamic memory extraction and updating.This design avoids directly conditioning on all previous shots, whose temporal redundancy makes generation inefficient.
- Memory Extraction and Update: Semantic keyframe selection uses CLIP similarity to retain distinct frames, while HPSv3 filtering removes blurred or noisy candidates and preserves reliable visual memories.The selection process fixes the first frame and adds later frames when similarity falls below a dynamic threshold.
- Extensions and Qualitative Results: Qualitative comparisons show coherent multi-shot videos aligned with per-shot descriptions, while keyframe-based baselines lose long-term character and scene consistency.The framework is also described as adaptable to other orthogonal video-generation paradigms.
4 Experiments
StoryMem is evaluated against independent-shot, keyframe-based, and other multi-shot baselines on ST-Bench using qualitative, quantitative, user-study, application, and ablation analyses. It improves cross-shot consistency while retaining strong visual quality and prompt following, with specific gains from memory selection, aesthetic filtering, and memory management.
- Benchmark: ST-Bench provides 300 detailed prompts across 30 long story scripts with diverse styles, scenes, characters, events, shot types, and camera movements.The benchmark was introduced because existing evaluation resources did not provide a suitable standard for multi-scene, multi-shot video generation.
- Qualitative Results: Qualitatively, StoryMem preserves character identity, clothing, and street-scene appearance across multiple shot transitions better than the baselines.In Shot 5, it retrieves contextual information from Shot 2, whereas other methods show mismatched identities, altered outfits, or inconsistent scenes.
- Quantitative Results: StoryMem outperforms the base model by 28.7% and HoloCine by 9.4% in overall cross-shot consistency, while achieving the highest aesthetic score and best global semantic alignment.Its single-shot prompt-following score is slightly lower under the MI2V setting, which adds control for smoother shot transitions.
- User Study: Human evaluators strongly prefer StoryMem over all baselines across most dimensions, while independently generated Wan2.2 videos retain single-shot quality but perform worse on other criteria.The user study addresses aspects that automatic metrics may miss, including character-level consistency and transition naturalness.
- Extended Application: StoryMem supports customized story generation by using reference images as initial memory and selectively preserving relevant generated context during narrative progression.The framework can also work with specialized reference-preserving techniques, although those techniques are outside the study’s focus.
- Ablation Study: Removing aesthetic filtering introduces low-quality or uninformative memory frames and produces the lowest aesthetic score, while replacing the memory sink with a sliding window degrades long-range fidelity.A naive first-frame strategy also misses newly introduced characters, causing inconsistent appearance and outfits.
5 Conclusion
StoryMem uses explicit visual memory to support coherent multi-shot storytelling, but remains challenged by ambiguous retrieval in complex multi-character scenes and abrupt motion changes between shots.
- StoryMem augments single-shot diffusion models with explicit visual memory for shot-by-shot synthesis of minute-long narrative videos.The framework combines latent concatenation, negative RoPE shifts, lightweight LoRA fine-tuning, and memory extraction and updating.
- The method delivers robust cross-shot coherence while preserving aesthetic quality and prompt adherence, outperforming state-of-the-art methods.
- Purely visual memory can ambiguously retrieve context in complex multi-character scenarios, causing inconsistent character appearance across shots.
- Large motion discrepancies between adjacent shots can still produce insufficiently smooth transitions.
A More Qualitative Results
Additional qualitative results demonstrate StoryMem’s effectiveness across diverse visual styles and its versatility for story video generation.
- Additional examples illustrate StoryMem’s effectiveness across diverse visual styles.Figure 8 provides another qualitative comparison supporting this observation.
- Further generated results highlight StoryMem’s versatility in story video generation.Figure 9 presents additional examples from the approach.
B More Details of ST-Bench
ST-Bench contains 30 GPT-5-generated long-form story scripts producing 300 video prompts under a unified structured schema.
- ST-Bench comprises 30 long-form story scripts generated using GPT-5, yielding 300 video prompts in total.
- Each story follows a unified JSON schema covering an overview, 1-4 scenes, 8-12 shot-level prompts, optional first-frame prompts, and cut indicators.First-frame prompts are included only for keyframe-based two-stage methods.
C Limitation and Future Work
StoryMem’s limitations include ambiguous visual-memory retrieval and imperfect transitions when adjacent shots differ substantially in motion, while its supplementary materials provide qualitative examples and script-generation details.
- C Limitation and Future Work: Visual memory may retrieve the wrong context in complex multi-character scenes because it lacks textual metadata, causing inconsistent character appearance across shots.The authors attribute this limitation to the cross-attention-based Wan2.2 architecture and suggest more explicit character descriptions as a mitigation.
- C Limitation and Future Work: Single-frame connections do not convey video speed information, so large motion-speed differences can make adjacent shot transitions appear unnatural.The authors propose overlapping more frames across consecutive shots when no scene cut is intended.
- C Limitation and Future Work: Additional qualitative figures illustrate StoryMem’s results across diverse visual styles and story-generation examples.
- C Limitation and Future Work: The ST-Bench script prompt limits stories to 1-4 scenes and no more than 12 prompts, with each prompt representing a five-second video clip.The instructions also discourage overly complex text rendering, extreme motions, and audio-dependent effects.
- C Limitation and Future Work: Figures 10 and 11 document the system prompt and an example story script used in ST-Bench.