PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoMemory: Toward Consistent Video Generation via Memory Integration

Jinsong Zhou, Yihua Du, Xinli Xu, Luozhou Wang, Zijie Zhuang, Yehang Zhang, Shuaibo Li, Xiaojun Hu, Bolan Su, Ying-cong Chen

arXiv:2601.03655v1cs.CV

TL;DR

Narrative video generation still struggles to preserve entity identity and appearance across changing scenes and long gaps. VideoMemory tracks and updates entity states across shots with a Dynamic Memory Bank, and achieves strong cross-shot coherence and perceptual realism on a 54-case benchmark.

  • Problem

    Narrative video generation lacks persistent explicit entity representations, leaving long-range consistency across shots an unresolved challenge.

    Source: §1 · p. 2

  • Method

    VideoMemory combines narrative decomposition and visual synthesis with a Dynamic Memory Bank that stores and updates entity visual and semantic states across shots.

    Source: §1 · p. 2

  • Results

    VideoMemory outperforms prior methods on character, prop, and background consistency metrics, with one setting exception, and its advantage grows with sequence length.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The 54-case benchmark provides a structured test of identity preservation while allowing intentional narrative changes.

    Source: §1 · p. 3

    Prior temporal-generation approaches still struggle to recreate previously seen entities and props with stable attributes.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Maintaining consistent characters, props, and environments across multiple shots is a central challenge in narrative video generation. Existing models can produce high-quality short clips but often fail to preserve entity identity and appearance when scenes change or when entities reappear after long temporal gaps. We present VideoMemory, an entity-centric framework that integrates narrative planning with visual generation through a Dynamic Memory Bank. Given a structured script, a multi-agent system decomposes the narrative into shots, retrieves entity representations from memory, and synthesizes keyframes and videos conditioned on these retrieved states. The Dynamic Memory Bank stores explicit visual and semantic descriptors for characters, props, and backgrounds, and is updated after each shot to reflect story-driven changes while preserving identity. This retrieval-update mechanism enables consistent portrayal of entities across distant shots and supports coherent long-form generation. To evaluate this setting, we construct a 54-case multi-shot consistency benchmark covering character-, prop-, and background-persistent scenarios. Extensive experiments show that VideoMemory achieves strong entity-level coherence and high perceptual quality across diverse narrative sequences.

1. Introduction

VideoMemory addresses long-range entity consistency in narrative video generation with a Dynamic Memory Bank that retrieves and updates character, prop, and background states across shots. The paper also introduces a 54-case benchmark for evaluating consistency across persistent entity types.

1. Introduction

  • Story-driven generation requires characters, props, and environments to remain consistent across distant shots despite changing viewpoints, lighting, and narrative contexts.Unlike standalone short clips, recurring entities may also undergo intentional story-driven changes.Source: §1 · p. 2
  • Prior methods improve planning, joint-shot context, or frame-level retrieval, but lack explicit entity states for reliable identity recovery after long gaps.The approaches include plan-then-generate pipelines [12] [19] [44], one-pass multi-shot generation [9] [24] [35], and memory-inspired methods [37] [41] [42].Source: §1 · p. 2
  • Inspired by film and television production, VideoMemory treats recurring entities as story assets whose appearance inventory is updated as the narrative unfolds.The proposed memory is intended to preserve recognizability while reflecting plot-driven changes.Source: §1 · p. 2
  • VideoMemory uses a Dynamic Memory Bank to retrieve entity states before each shot and update them afterward, coordinating storyboard, memory, and visualization agents.The bank stores explicit visual and semantic representations for characters, props, and backgrounds; updates can reflect narrative changes while preserving identity, without full-frame history caching.Source: §1 · pp. 2–3
  • The 54-case benchmark covers character-, prop-, and background-persistent scenarios with varying shot lengths and tests identity preservation alongside intentional narrative changes.The paper reports strong cross-shot coherence, stable long-range entity continuity, and high perceptual realism across all categories.Source: §1 · p. 3

2. Related Work

Related work advances story-driven video generation and memory-based long-video generation, but persistent entity appearance and structure remain challenging across shots and long temporal gaps. VideoMemory addresses these limitations with explicit memory retrieval and updates.

2.1. Story-Driven Video Generation

  • Keyframe-first methods use image-to-video generation to improve character consistency across shots, while LLM-integrated methods plan narrative elements for more consistent story visualization [8] [14] [18] [23] [36] [45] [12, 13, 19, 21, 38, 39, 44].Source: §2.1 · p. 3
  • Multi-shot methods generate consecutive shots in one forward pass using specialized architectures and cinematic-transition datasets, yet still struggle to reproduce entities and props with stable attributes [9, 16, 34, 35].Source: §2.1 · p. 3
  • The proposed dynamic Memory Bank stores entities by semantic attributes and key descriptors, then retrieves and updates them during generation to reproduce characters and objects across shots more faithfully.Source: §2.1 · p. 3

2.2. Memory in Video Generation

  • Diffusion Transformer-based models, including HunyuanVideo [17] and Wan 2.1&2.2 [33], produce high-fidelity, temporally stable short videos but often show repetition and content drift in long-duration extrapolation [43].Source: §2.2 · p. 3
  • Long-video methods extend duration but still fail to preserve the appearance and structure of objects that reappear after long gaps, exposing memory degradation in diffusion-based models [1, 3, 4, 15, 20, 32, 40].Source: §2.2 · p. 3
  • FramePack [42], Context-as-Memory [41], and a dynamically updated 3D point-cloud memory [37] provide prior-content references, but face high maintenance costs and limited consistency, especially in object geometry.Source: §2.2 · p. 3
  • The proposed Memory Bank retrieves and updates key characters and objects while storing only critical entities, reducing memory overhead and improving long-term consistency in appearance and structure.Source: §2.2 · p. 3

3. Method

VideoMemory maps a script synopsis to narratively coherent multi-shot video by planning shots, retrieving or creating entity references, and generating keyframes and videos through a shot-by-shot, memory-guided pipeline. Its Dynamic Memory Bank supports consistent character, prop, and background appearances while allowing story-driven changes over time.

3.1. Task Formulation

  • The task is to map a script synopsis to multi-shot video that is visually plausible and narratively coherent while preserving character identity, prop state, and compatible backgrounds across time.The mapping combines high-level planning of sub-scripts, scenes, shots, and camera parameters with visual realization subject to long-range entity constraints.Source: §3.1 · p. 4

3.2. Multi-Agent Cooperation

  • VideoMemory divides generation into storyboard, memory, and visualization agents that plan structured shots, identify entities and retrieve or create references, then compose keyframes and synthesize shot videos.The Storyboard Agent uses an LLM to refine the synopsis into beats and shots with spatial setup, characters, events, framing, and action order.Source: §3.2 · pp. 4–5

3.3. Dynamic Memory Bank

  • The Dynamic Memory Bank reuses references for recurring entities and creates updated appearances from historical references when needed, preserving identity through changes such as aging.It maintains separate character, prop, and background banks and retrieves semantically compatible entries rather than relying on exact text matches.Source: §3.3 · p. 5

3.4. Keyframe & Video Generation

  • Using memory-guided keyframes expanded by an image-to-video model yields multi-shot video with stable character identities, persistent props, and coherent environments over long temporal ranges.The Visualization Agent conditions keyframe generation on entity references, then uses the keyframe and shot description to generate each shot video before concatenating shots in narrative order.Source: §3.4 · p. 6

4. Experiments

VideoMemory’s Dynamic Memory Bank supports consistent entity generation across multi-shot narratives, and experiments report strong quantitative, human-preference, and qualitative results. The benchmark tests persistent characters, props, or backgrounds across narratives of varying shot lengths.

4.1. Experimental Setup

  • Experiments use Gemini 2.5 Pro, Gemini 2.5 Flash-Image, and Wan 2.2 I2V-14B [33], comparing T2V [33] [35] and keyframe-first baselines [14] [45] [44].Consistency uses DINOv2 similarity for detected face regions, target prop regions, or full frames; all methods receive the same per-shot descriptions, and keyframe-first baselines share the Wan animation model.Source: §4.1 · pp. 6–7
  • The 54-case benchmark tests character-, prop-, or background-persistence across 4-, 8-, and 12-shot three-act narratives, varying the other two factors.Its design targets long-horizon entity persistence, which existing single-shot and short-clip benchmarks are ill-suited to evaluate.Source: §4.1 · pp. 6–7

4.2. Quantitative Results

  • VideoMemory leads character, prop, and background consistency metrics, with one setting exception; its advantage grows with shot length, and aggregate averages confirm greater stability.Some baselines could not produce videos at the required shot counts for longer settings. The authors attribute long-horizon coherence to the Dynamic Memory Bank.Source: Table 1 · p. 7, §4.2 · p. 7
  • Experts preferred VideoMemory over all five baselines, with preference rates reaching 95.8% for character and background consistency.Five participants compared videos across 270 randomly selected pairs covering the 54 benchmark cases.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. Qualitative Analysis

  • Across character, prop, and background cases, baselines show identity drift, while VideoMemory preserves character appearance, the red kite, and garage layout across shots.The examples illustrate how entity-centric memory prevents visually disruptive drift in multi-shot narratives.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.4. Ablation Studies

  • Removing the Dynamic Memory Bank drops background consistency from 0.72 to 0.28; reintroducing the Character, Prop, and Background Banks incrementally restores stability.The qualitative ablation also shows entity drift without memory and progressive stabilization as each bank is added.Source: Table 3 · p. 8, §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

VideoMemory combines an entity-centric Dynamic Memory Bank with a multi-agent script-to-video pipeline to support long-range entity continuity and story-driven variation across shots.

5. Conclusion

  • VideoMemory explicitly maintains and updates narrative entities’ visual and semantic states across shots through a Dynamic Memory Bank.The framework targets script-to-multi-shot video generation.Source: §5 · p. 8
  • Its multi-agent pipeline integrates narrative decomposition and visual synthesis, supporting coherent long-range entity continuity while accommodating story-driven variations.Source: §5 · p. 8
  • The authors introduced a 54-case benchmark for character-, prop-, and background-level consistency, and report robust entity coherence and high perceptual quality across diverse narrative settings.Source: §5 · p. 8

A. Full Videos

The paper directs readers to its project page for complete videos.

A. Full Videos

B. Evaluation Metrics

VideoMemory evaluates cross-shot consistency with three subclass-specific metrics that share a reference-based protocol using each shot’s middle frame. Scores are computed per sequence and averaged across benchmark cases in the corresponding subclass and shot-length setting.

B. Evaluation Metrics

  • Three complementary metrics measure Character-, Prop-, and Background Consistency on their corresponding persistent benchmark subsets.The metrics are designed to assess cross-shot consistency in a factorized and interpretable manner.Source: B. Evaluation Metrics · p. 9
  • Each metric uses the middle frame of every shot, comparing subsequent-shot visual features with features from the first shot’s middle frame.The comparison is cosine similarity, and the three metrics differ only in how visual features are extracted.Source: B. Evaluation Metrics · p. 9
  • For each sequence, similarities are aggregated using a normalization rule, then averaged across benchmark cases for the matching subclass and shot-length setting.Character, Prop, and Background Consistency are reported on their respective persistent subsets.Source: B. Evaluation Metrics · p. 9

B.1. Character Consistency

Character consistency is measured by comparing face descriptors from later shots with a first-shot reference; higher similarity indicates stronger identity preservation despite changes in pose, expression, and illumination.

B.1. Character Consistency

  • For character-persistent cases, the metric detects and crops faces in middle frames with Mediapipe, extracts and aggregates DINOv2 descriptors, then compares each shot with the first-shot reference by cosine similarity.Higher scores indicate consistent recognition of the same identity across shots while remaining robust to pose, expression, and local illumination changes.Source: §B.1 · p. 9

B.2. Prop Consistency … C. System Prompts for Multi-Agent Pipeline

The appendix defines prop and background consistency metrics, explains how variable shot counts affect scoring, and documents the multi-agent prompts for reproducible, consistency-focused generation. The prompts specify storyboard decomposition, dynamic memory management, and keyframe and video generation.

B.2. Prop Consistency

  • Prop consistency compares DINOv2 descriptors from Grounded SAM -localized prop regions across detected shots, aggregating their similarities with the same normalization strategy.The metric assesses visual appearance, including color, shape, and texture, despite changes in viewpoint or context.Source: §B.2 · p. 9

B.3. Background Consistency

  • Background consistency compares full-frame DINOv2 descriptors between the reference and subsequent shots without spatial segmentation.It measures preservation of global background properties such as layout, large-scale structures, and lighting when the environment is meant to remain fixed.Source: §B.3 · p. 9

B.4. Handling Variable Shot Counts

  • When a method generates too many shots, scoring uses only the first requested shots; when it generates too few, missing shots count as zero similarity through normalization by the requested non-reference count.For example, six generated shots against an eight-shot script yield five similarities divided by seven.Source: §B.4 · p. 9

C. System Prompts for Multi-Agent Pipeline

  • The system prompts specify structural and consistency-related agent behavior for reproducibility while excluding aesthetic or cinematographic directives.The appendix presents the prompts as aligned with the main paper.Source: C. System Prompts for Multi-Agent… · p. 10
  • The Storyboard Agent turns the synopsis into an ordered JSON shot list that faithfully identifies on-screen characters, props, settings, and time jumps without adding story elements or visual-style directions.Each shot records scene, plot, characters, key props, and environment information.Source: C. System Prompts for Multi-Agent… · p. 10
  • For each shot, the Memory Agent extracts entities and their current states, reuses suitable references or creates new ones, and updates the Dynamic Memory Bank when meaningful changes occur.New references can use earlier images of the same entity to preserve identity; unchanged states reuse the previous entry.Source: C. System Prompts for Multi-Agent… · pp. 10–11
  • For each shot, the Visualization Agent generates a keyframe and a video clip from it, using entity references and prompts that preserve identity, appearance, and layout without adding story elements or visual-style directions.The keyframe is treated as the video’s first frame, and the video prompt describes the shot’s essential action and its evolution over time.Source: C. System Prompts for Multi-Agent… · p. 11

D. Prompt for Benchmark Story Generation

The benchmark prompt specifies constraints on story structure, shot count, and element persistence to evaluate cross-shot consistency across varying narrative complexities.

D. Prompt for Benchmark Story Generation

  • The prompt generates 54 scripts: six each for the nine combinations of 4, 8, or 12 shots and character-, prop-, or scene-persistence.The benchmark varies both total shot count and which core element persists.Source: D. Prompt for Benchmark Story Generation · p. 12
  • Each story must distribute its plot across three explicitly divided acts, with shot allocation adjusted to the selected total.The prompt gives approximate act shares of 20–30%, 40–60%, and 20–30%.Source: D. Prompt for Benchmark Story Generation · p. 12
  • Scripts allow only one adult character and one visually prominent, bright-colored prop, and must remain family-friendly without violent, sexual, gory, or disturbing content.The prop must be at least backpack- or large-book-sized and easy to distinguish from the background.Source: D. Prompt for Benchmark Story Generation · p. 12
  • Shot text must use natural, on-screen screenplay descriptions without camera terminology, with each shot as a numbered standalone paragraph and no bullets.The output contains only shot content, and each paragraph begins with SHOT followed by its number.Source: D. Prompt for Benchmark Story Generation · p. 12
  • Each script follows one persistence condition: the persistent character, prop, or scene remains consistent while the other two elements vary in presence or setting.For character-persistent and prop-persistent scripts, the scene changes in every shot; scene-persistent scripts keep one location while character and prop status varies.Source: D. Prompt for Benchmark Story Generation · p. 12
  • At first appearance, the character and prop receive detailed parenthetical descriptions; their appearances must remain unchanged, with no repeated parenthetical descriptions later.The character description includes age, appearance, and clothing; the prop description includes appearance, bright color, and relative size.Source: D. Prompt for Benchmark Story Generation · pp. 12–13

E. User Study

The study assesses whether quantitative consistency gains are perceptible to human viewers through expert comparisons; VideoMemory was strongly preferred across character, prop, and background consistency.

Participants

  • Five expert raters familiar with generative models, video editing, and cinematography evaluated cross-shot consistency rather than aesthetics or story quality.All participants were fluent in English.Source: E. User Study · p. 13

Stimuli

  • The study compared its method with five baselines in 270 method pairs across 54 scripts; each pair shared the script, shot breakdown, and text prompts, differing only by generation method.Videos were shown at their original resolution and aspect ratio and trimmed to the same duration used in the quantitative benchmark.Source: E. User Study · p. 13

Interface

  • Each trial used an anonymized, side-by-side Method A/Method B display with the script shown, replay controls, and a forced-choice response.Participants could play, pause, and replay videos as often as they wished before selecting a method.Source: E. User Study · p. 13

Task and questions

  • Trials asked which method better maintained character, prop, or background consistency, with subclass question types sampled uniformly and no tie option.Participants were instructed to choose the better-matching video even when the difference was subtle.Source: E. User Study · p. 13

Design and procedure

  • Each participant evaluated 270 pairs in randomized trial order and randomized left/right assignment, remotely, with breaks allowed and no time limit after a practice block.The practice block used example pairs outside the benchmark.Source: E. User Study · p. 13

Aggregation

  • Preference rates reached 95.8% for character and background consistency, and VideoMemory was strongly preferred across all three subclasses.Preference ratios measured selections of VideoMemory over each baseline for each script, baseline, and consistency subclass.Source: E. User Study · p. 13

F. Successful Cases

Additional qualitative examples show character, prop, and background consistency across changing settings, lighting, camera angles, and actions. The section also presents the perceptual user-study interface.

Successful Cases

  • In each user-study trial, participants read a benchmark script, watched two anonymized videos side by side, and chose which better maintained the specified consistency type.Figure 7 shows a character-consistency question as an example.Source: Figure 7 · p. 14
  • Figure 8 illustrates character identity across indoor, outdoor, and transit settings; a hard hat retaining its appearance; and a subway platform remaining stable across characters and actions.The examples correspond to character, prop, and background consistency, respectively.Source: F. Successful Cases · p. 13, Figure 8 · p. 15

Related papers

Loading 2601.03655v1…