PaperlayerView exact revision on arXiv

Source-linked AI summary

StoryMem: Multi-shot Long Video Storytelling with Memory

Kaiwen Zhang, Liming Jiang, Angtian Wang, Jacob Zhiyuan Fang, Tiancheng Zhi, Qing Yan, Hao Kang, Xin Lu, Xingang Pan

arXiv:2512.19539v1cs.CV

TL;DR

Generating coherent minute-long visual narratives remains challenging because storytelling requires consistency across shots and scenes. StoryMem generates shots iteratively with explicit visual memory and reports stronger cross-shot consistency on ST-Bench, exceeding the base model by 28.7% and HoloCine by 9.4% in overall consistency.

  • Problem

    Generating minute-long multi-shot narratives remains challenging because storytelling requires coherence across shots and scenes, including consistent characters, environments, style, and narrative flow.

    Source: §1 · p. 1

  • Method

    StoryMem synthesizes stories shot by shot, conditioning a compact bank of keyframes from previously generated shots as explicit visual memory.

    Source: §1 · pp. 2–3

  • Results

    StoryMem leads baselines on Cross-shot Consistency, exceeding the base model by 28.7% and HoloCine by 9.4% in overall consistency.

    Source: §4.3 · p. 10

  • Takeaways & Limitations

    The authors position StoryMem as a step toward coherent minute-long storytelling, bridging single-shot video diffusion and long-form visual storytelling.

    Source: §1 · p. 3

    StoryMem may struggle with ambiguous memory retrieval in complex multi-character scenes and fully smooth transitions when adjacent shots have large motion discrepancies.

    Source: §5 · p. 11

Abstract

from arXiv · show

Visual storytelling requires generating multi-shot videos with cinematic quality and long-range consistency. Inspired by human memory, we propose StoryMem, a paradigm that reformulates long-form video storytelling as iterative shot synthesis conditioned on explicit visual memory, transforming pre-trained single-shot video diffusion models into multi-shot storytellers. This is achieved by a novel Memory-to-Video (M2V) design, which maintains a compact and dynamically updated memory bank of keyframes from historical generated shots. The stored memory is then injected into single-shot video diffusion models via latent concatenation and negative RoPE shifts with only LoRA fine-tuning. A semantic keyframe selection strategy, together with aesthetic preference filtering, further ensures informative and stable memory throughout generation. Moreover, the proposed framework naturally accommodates smooth shot transitions and customized story generation applications. To facilitate evaluation, we introduce ST-Bench, a diverse benchmark for multi-shot video storytelling. Extensive experiments demonstrate that StoryMem achieves superior cross-shot consistency over previous methods while preserving high aesthetic quality and prompt adherence, marking a significant step toward coherent minute-long video storytelling.

1 Introduction

StoryMem reframes long-form video storytelling as iterative shot generation guided by explicit visual memory, adapting pretrained single-shot models to support cross-shot coherence. It introduces ST-Bench and reports improved cross-shot consistency while retaining visual fidelity and prompt adherence.

Introduction

  • Long-form storytelling remains challenging because joint-shot models incur high costs, while independent keyframe-based pipelines lack cross-shot context and produce inconsistent details and rigid transitions.The introduction describes joint methods including LCT [8] and decoupled keyframe-based methods [13] [29] [45] [53] [56].Source: §1 · pp. 1–2
  • StoryMem generates stories shot by shot, using a compact memory bank of past keyframes to guide consistent visual semantics and scene evolution.The memory preserves context such as characters, scenes, and stylistic cues from previously generated shots.Source: §1 · pp. 2–3
  • Memory-to-Video combines memory latent concatenation with a negative RoPE shift, and uses LoRA fine-tuning on semantically coherent short clips to extend pretrained I2V models.The paper says this approach supports long-term dependencies and cross-shot consistency while preserving the visual quality of the pretrained models.Source: §1 · p. 3
  • Semantic keyframe selection using CLIP [36] features and HPSv3 [30] aesthetic filtering keep memory compact and reliable as it is dynamically extracted, updated, and injected for each shot.Source: §1 · p. 3
  • StoryMem also supports natural scene transitions with I2V control and customized reference-to-video generation by incorporating reference images into memory.Source: §1 · p. 3
  • ST-Bench covers diverse multi-scene, multi-shot narratives, and experiments report superior cross-shot consistency while retaining high visual fidelity and prompt adherence over state-of-the-art methods.Source: §1 · p. 3

2 Related Work

Related work addresses story consistency through keyframes, joint multi-shot modeling, or specialized video-world-model memory. StoryMem instead uses explicit memory for general-purpose, shot-by-shot video generation.

Keyframe-based Story Generation

  • Keyframe-based video storytelling expands generated story images into clips, but consistency remains limited to keyframes and shots are largely independent, with often-rigid transitions [13] [29] [45] [53] [56].StoryMem’s memory-to-video framework instead generates shots sequentially while preserving context across shots with minimal computational overhead.Source: §2 · p. 3

Multi-shot Long Video Generation

  • Existing multi-shot methods build on pretrained single-shot models; LCT [8] uses full attention and interleaved 3D RoRE for coherence at quadratic cost, while later methods improve efficiency with token compression or sparse attention [47] [2] [17] [31].These methods require large-scale multi-shot training, which can degrade quality relative to the pretrained single-shot base; StoryMem instead uses lightweight LoRA fine-tuning on short clips and can adapt to I2V and R2V.Source: §2 · p. 3

Memory Mechanisms in Video Generation

  • Prior video-generation memory methods target spatial consistency in controllable world simulations, using inference-time training, 3D modeling [24], or camera-based retrieval and cross-attention [48] [51].Their reliance on auxiliary inputs such as actions or camera poses limits application to general-purpose video generation.Source: §2 · p. 3
  • StoryMem introduces an explicit memory mechanism into general video-generation models to retain essential characters and background scenes across minute-long generation.The paper characterizes this as the first such mechanism for general video generation.Source: §2 · p. 4

3 Methodology

StoryMem turns single-shot video diffusion into multi-shot storytelling through memory-conditioned shot synthesis and a dynamically updated bank of keyframes. Its methodology defines the memory-based generation process, implements it with M2V, and extends it to support transitions and user-controlled initialization.

3.1 Preliminary

  • M2V extends the base model’s masked, first-frame-conditioned latent diffusion architecture to condition generation on memory keyframes.The noisy latent, conditional latent, and mask are concatenated along the channel dimension, with the mask marking preserved and generated frames.Source: §3.1 · p. 5

3.2 Problem Formulation

  • Rather than condition each shot on all prior video or independent global keyframes, StoryMem uses compact keyframe memory to summarize prior visual context and support coherent autoregressive generation.The memory can represent characters, scenes, and style across shots.Source: §3.2 · pp. 5–6

3.3 Memory-to-Video (M2V)

  • M2V conditions video diffusion on memory latents and uses negative RoPE indices to place past keyframes before the current shot in a shared temporal space.Training uses single-shot targets conditioned on frames sampled from related clips; at inference, only the newly generated segment is decoded.Source: §3.3 · p. 6

3.4 Memory Extraction and Update

  • A dynamic memory bank retains semantically distinct, aesthetically reliable keyframes, combining fixed long-term anchors with a sliding window of recent memories.Keyframes are selected using CLIP similarity and filtered with HPSv3; when capacity is reached, the oldest short-term memories are discarded.Source: §3.4 · pp. 6–7

3.5 Extension to MI2V and MR2V

  • M2V can reuse the previous shot’s last frame as the next shot’s first frame when no scene cut is specified, while reference images can initialize memory for customized generation.The framework is also described as adaptable to other orthogonal video-generation paradigms.Source: §3.5 · pp. 7–8

4 Experiments

Experiments evaluate StoryMem on the newly introduced ST-Bench and report stronger cross-shot consistency than baselines, alongside user-study evidence and ablations of its memory components.

4.1 Implementation Details

  • StoryMem uses Wan2.2-I2V-A14B with rank-128 LoRA fine-tuning and trains on 400K five-second clips; inference limits memory to three sink frames and three keyframes per shot.Training samples 1–10 memory frames and uses a negative RoPE shift offset of 5; inference applies semantic-similarity and aesthetic-score thresholds of 0.9 and 3.0.Source: §4.1 · p. 8

4.2 ST-Bench

  • ST-Bench provides 300 detailed video prompts from 30 diverse story scripts, each with 8–12 shot prompts, story overviews, and scene-cut indicators.First-frame prompts are included only for two-stage keyframe-based methods.Source: §4.2 · pp. 8–9

4.3 Evaluation

  • In qualitative comparisons, StoryMem maintains character identity, outfits, and scene appearance across shots, while pretrained and keyframe-based baselines lose consistency and HoloCine degrades in visual quality.StoryMem retrieves context from Shot 2 to keep the street scene consistent in Shot 5.Source: §4.3 · p. 9
  • Human evaluators strongly prefer StoryMem over all baselines across most dimensions, while independently generated Wan2.2 shots perform well on single-shot quality but worse on other dimensions.The user study uses pairwise comparisons because existing metrics do not fully capture character-level consistency or transition naturalness.Source: §4.3 · p. 10
  • Using reference images as initial memory enables customized stories that preserve generated scenes while selectively reusing relevant context, and the framework is compatible with specialized reference-preserving techniques.Figure 5 illustrates the customized-generation application.Source: Figure 5 · p. 9, §4.3 · p. 10

4.4 Ablation Study

  • Ablations show that first-frame-only memory misses newly introduced characters, removing aesthetic filtering admits low-quality or uninformative frames, and sliding-window memory reduces long-range visual fidelity.The full method performs best on most metrics, with a slight drop in single-shot prompt following under initial-memory constraints.Source: §4.4 · pp. 10–11, Figure 6 · p. 10, Table 2 · p. 11

5 Conclusion

StoryMem uses explicit visual memory to extend single-shot diffusion models into a shot-by-shot framework for minute-long storytelling. Experiments show robust cross-shot coherence while preserving aesthetic quality and prompt adherence, but complex multi-character scenes and large motion differences remain challenging.

5 Conclusion

  • StoryMem outperforms state-of-the-art methods in cross-shot coherence while preserving aesthetic quality and prompt adherence.The conclusion reports these outcomes from extensive experiments.Source: §5 · p. 11
  • M2V augments single-shot diffusion models with explicit visual memory via latent concatenation, negative RoPE shifts, and lightweight LoRA fine-tuning.Combined with memory extraction and update, it enables shot-by-shot synthesis of minute-long narrative videos.Source: §5 · p. 11
  • StoryMem may struggle when visual memory becomes ambiguous in complex multi-character scenarios or when adjacent shots have large motion discrepancies.Future work will explore entity-aware memory representations and improved transition modeling for richer, more fluid narratives.Source: §5 · p. 11

Appendix · A More Qualitative Results

Additional qualitative results illustrate StoryMem’s effectiveness across diverse visual styles and its versatility in story-video generation.

A More Qualitative Results

  • Figures 8 and 9 add qualitative examples, with Figure 8 illustrating StoryMem’s superiority across diverse visual styles and Figure 9 highlighting its versatility in story-video generation.Source: A More Qualitative Results · p. 16

B More Details of ST-Bench

ST-Bench is a standardized benchmark of long-form story scripts and video prompts, designed to support diverse generation paradigms. Its structured scripts specify shot content and transitions, and all scripts will be released for future research.

B More Details of ST-Bench

  • ST-Bench contains 30 GPT-5-generated long-form story scripts yielding 300 video prompts, spanning realistic and fairy-tale styles, ancient and modern settings, and Western and Eastern aesthetics.The benchmark standardizes inputs across generation paradigms.Source: B More Details of ST-Bench · p. 16
  • Each script follows a unified JSON schema with a story overview, 1–4 scenes, 8–12 shot prompts, optional first-frame prompts, and shot-transition cut indicators.First-frame prompts are included only for keyframe-based two-stage methods; cut indicators mark hard scene transitions or smooth continuations.Source: B More Details of ST-Bench · p. 16
  • A structured system instruction limits shot descriptions to 1–4 concise sentences covering character appearance, simple motions, scene layout, mood, and lightweight camera guidance.Figures 10 and 11 show the instruction and an example script, respectively; the example excludes first-frame prompts.Source: B More Details of ST-Bench · p. 16
  • All ST-Bench story scripts will be released to support future research.The benchmark section also provides a complete example story script, excluding first-frame prompts.Source: B More Details of ST-Bench · p. 16

C Limitation and Future Work

StoryMem’s limitations are ambiguous visual-memory retrieval in complex multi-character scenes and occasionally unnatural transitions between shots with different motion speeds. The authors propose entity-aware memory and overlapping frames as future improvements.

C Limitation and Future Work

  • Without textual metadata, visual memory may retrieve the wrong character context in complex scenes; explicit descriptions help, and entity-aware memory is a planned remedy.The Wan2.2 cross-attention-based DiT architecture, rather than more flexible MMDiT [5] [20], constrains memory updates to visual information.Source: C Limitation and Future Work · p. 16
  • Single-frame connections can leave transitions unnatural when adjacent shots differ greatly in motion speed; overlapping frames where no scene cut is intended is proposed to improve smoothness.Autoregressive generation and MI2V alleviate rigid transitions but do not convey video speed information.Source: C Limitation and Future Work · p. 16

Related papers

Loading 2512.19539v1…