Source-linked AI summary
VideoMemory: Toward Consistent Video Generation via Memory Integration
Jinsong Zhou, Yihua Du, Xinli Xu, Luozhou Wang, Zijie Zhuang, Yehang Zhang, Shuaibo Li, Xiaojun Hu, Bolan Su, Ying-cong Chen
TL;DR
Narrative video generation still struggles to preserve characters, props, and environments across scene changes and long temporal gaps. VideoMemory combines script-based narrative planning with an entity-aware Dynamic Memory Bank that retrieves and updates visual and semantic states during generation. On a 54-case benchmark, it achieves robust entity coherence and high perceptual quality across diverse narrative settings, with strong user preference and ablation evidence supporting the memory design.
Problem
Narrative video methods lack persistent entity representations, making identity and appearance unreliable when characters or props reappear after long gaps.
Method
VideoMemory uses a multi-agent script-to-video pipeline with a Dynamic Memory Bank that retrieves and updates explicit visual and semantic states for characters, props, and backgrounds.
Results
VideoMemory delivers robust entity coherence and high perceptual quality across diverse narrative settings on a 54-case benchmark, with preference rates as high as 95.8 % for character and background consistency.
Takeaways & Limitations
Explicit entity memory supports coherent long-range continuity while allowing story-driven changes in characters, props, and environments.
Takeaways & Limitations
The evaluated setting focuses on 54 structured cases spanning character-, prop-, and background-persistent scenarios, while prior approaches can struggle with object-level consistency.
Abstract
from arXiv · showhide
Maintaining consistent characters, props, and environments across multiple shots is a central challenge in narrative video generation. Existing models can produce high-quality short clips but often fail to preserve entity identity and appearance when scenes change or when entities reappear after long temporal gaps. We present VideoMemory, an entity-centric framework that integrates narrative planning with visual generation through a Dynamic Memory Bank. Given a structured script, a multi-agent system decomposes the narrative into shots, retrieves entity representations from memory, and synthesizes keyframes and videos conditioned on these retrieved states. The Dynamic Memory Bank stores explicit visual and semantic descriptors for characters, props, and backgrounds, and is updated after each shot to reflect story-driven changes while preserving identity. This retrieval-update mechanism enables consistent portrayal of entities across distant shots and supports coherent long-form generation. To evaluate this setting, we construct a 54-case multi-shot consistency benchmark covering character-, prop-, and background-persistent scenarios. Extensive experiments show that VideoMemory achieves strong entity-level coherence and high perceptual quality across diverse narrative sequences.
1. Introduction
Narrative video generation requires persistent entity representations because characters, props, and environments must remain recognizable across changing and distant shots. VideoMemory addresses this with an entity-aware Dynamic Memory Bank integrated into script decomposition, retrieval, visualization, and post-shot updates.
- Story-level generation must preserve characters, signature props, and environments across scenes and shots, imposing stricter requirements than standalone short-clip synthesis.
- Existing approaches provide planning or broader temporal context but lack explicit persistent entity states, causing identity drift when entities reappear after long gaps.
- VideoMemory stores evolving visual and semantic states for characters, props, and backgrounds in an explicit Dynamic Memory Bank.
- A multi-agent pipeline expands scripts into shots, retrieves required entity states before generation, synthesizes conditioned keyframes and videos, then updates memory after each shot.
- The 54-case benchmark covers character-, prop-, and background-persistent scenarios, and experiments report strong cross-shot coherence, long-range continuity, and perceptual realism.
2. Related Work
Related work improves narrative planning, temporal smoothness, and long-video duration, but object-level consistency after long temporal gaps remains difficult. VideoMemory addresses this gap with explicit retrieval and efficient updates for critical entities.
- Storyboard and keyframe-first methods improve visual planning and character continuity, while language-model pipelines provide greater control over narrative elements.
- Multi-shot generation models consecutive shots jointly or sequentially, typically using specialized architectures and cinematic-transition datasets.
- Despite improved temporal smoothness, existing approaches struggle to recreate previously appearing entities or props with stable attributes.
- Long-video methods extend duration but still fail to preserve identical object appearance and structure after long temporal gaps, revealing memory degradation in diffusion-based models.
- VideoMemory stores critical entities using explicit retrieval and update modules, reducing memory overhead while improving long-term appearance and structural consistency.
3. Method
VideoMemory coordinates storyboard planning, entity-aware memory retrieval or creation, and keyframe-to-video synthesis in a shot-by-shot multi-agent pipeline. Its Dynamic Memory Bank stores explicit entity states and updates them across shots to preserve identity while allowing narrative changes.
- Task formulation: VideoMemory maps a script synopsis to a multi-shot video through high-level planning and memory-guided visual realization.The process decomposes the synopsis into structured shots, then materializes each shot as a keyframe and video while respecting entity constraints.
- Storyboard Agent: The Storyboard Agent expands the synopsis into ordered shot descriptions specifying scenes, actions, characters, props, environments, and temporal information.These descriptions provide the narrative scaffold for downstream modules.
- Memory Agent: The Memory Agent identifies each shot’s characters, props, and background elements, represents their semantic states, and retrieves or creates corresponding reference images.It centralizes entity analysis and memory interaction so consistency decisions remain explicit and inspectable.
- Visualization Agent: The Visualization Agent combines shot descriptions with retrieved entity references to compose keyframes and drive an image-to-video model.Keyframes encode composition, poses, and background layout while preserving memorized entity appearances.
- Dynamic Memory Bank: The Dynamic Memory Bank maintains specialized character, prop, and background entries containing entity identifiers, attribute states, and reference images.Semantic matching retrieves compatible entries; failed retrieval triggers generation using current attributes and historical references, after which the new state is inserted.
- Dynamic Memory Bank: Repeated retrieve-or-update operations preserve recognizable characters, props, and environments across long temporal ranges while accommodating changes such as aging, damage, or re-decoration.All keyframes are guided by the shared memory before I2V expansion and shot concatenation.
4. Experiments
VideoMemory is evaluated on structured multi-shot cases using consistency metrics, user preferences, qualitative comparisons, and ablations. Across these evaluations, it maintains entity identity more reliably than baselines, especially over longer sequences, while its Dynamic Memory Bank is central to the gains.
- Experimental Setup: The benchmark contains 54 cases spanning character-, prop-, and background-persistent scenarios across 4-, 8-, and 12-shot narratives.Each case varies the two non-persistent factors across shots to test whether models decouple the target entity from changing context.
- Experimental Setup: VideoMemory is compared with native text-to-video and keyframe-first baselines using the same per-shot textual descriptions and a shared animation backbone for keyframe-first methods.The fixed visual synthesis backbone is Wan 2.2 I2V-14B, while Gemini models handle narrative reasoning and keyframe generation.
- Quantitative Results: VideoMemory outperforms prior methods on character, prop, and background consistency, with its advantage becoming more pronounced as sequence length increases.Some baselines could not produce videos at the required shot counts for longer settings, yielding incomplete scores.
- Quantitative Results: 95.8 % is the highest reported user preference rate for VideoMemory on character and background consistency comparisons against baselines.Five participants evaluated 270 randomly selected video pairs in forced-choice comparisons covering all 54 benchmark cases and five baselines.
- Qualitative Analysis: Qualitative examples show baselines changing character appearance, morphing a red kite, or rearranging a garage, while VideoMemory preserves entity identity across distant shots.The comparison covers character, prop, and background-persistent cases.
- Ablation Studies: Removing the Dynamic Memory Bank drops background consistency from 0.72 to 0.28, while reintroducing the Character, Prop, and Background Banks progressively restores stability.The full model produces the most holistically stable output, and the results are averaged over 8-shot cases.
5. Conclusion
VideoMemory uses a Dynamic Memory Bank to maintain and update narrative entities across shots within a multi-agent script-to-video pipeline. Its 54-case benchmark and empirical results support robust entity coherence and perceptual quality across diverse narrative settings.
- 5. Conclusion: VideoMemory explicitly maintains and updates the visual and semantic states of narrative entities across shots.The framework integrates this memory with multi-agent narrative decomposition and visual synthesis.
- 5. Conclusion: The 54-case benchmark targets character-, prop-, and background-level consistency in story-driven multi-shot generation.
- 5. Conclusion: Empirical results demonstrate robust entity coherence and high perceptual quality across diverse narrative settings.
A. Full Videos
Complete videos are available through the project page.
- The paper directs readers to the project page for complete videos.
- The project page is the stated destination for viewing the complete videos.
- No further details about the complete videos are provided in this section.
B. Evaluation Metrics
The evaluation uses three complementary, reference-based consistency metrics tailored to character-, prop-, and background-persistent benchmark subsets. Scores are computed from middle-frame features and averaged across benchmark cases for each shot-length setting.
- Three complementary metrics separately evaluate character, prop, and background consistency across their corresponding benchmark subclasses.The metrics share a reference-based protocol but differ in visual feature extraction.
- The shared protocol makes cross-shot consistency factorized and interpretable across the three benchmark subclasses.
- Each sequence score compares middle-frame features from subsequent shots with the first shot’s middle frame, then averages scores across benchmark cases.The final metric is reported for each subclass and shot-length setting.
B.1. Character Consistency
The system combines script-to-shot decomposition, dynamic memory management, and reference-conditioned keyframe and video generation to preserve entity identity across shots. Character, prop, and background consistency are evaluated through feature similarity on middle frames, with variable shot counts handled against the requested sequence length.
- Character Consistency: The Character-persistent metric detects visible faces, extracts DINOv2 face features, and compares each shot with the first shot’s character descriptor.Higher similarity indicates more consistent identity recognition despite pose, expression, and illumination changes.
- Character Consistency: Prop consistency localizes the designated object with Grounded SAM and compares DINOv2 descriptors across detected middle frames.The metric targets stability in color, shape, and texture under viewpoint or context changes.
- Character Consistency: Background consistency compares full-middle-frame DINOv2 features to assess preservation of layout, large-scale structures, and lighting.Unlike the character and prop metrics, it uses no spatial segmentation.
- Character Consistency: When a model generates too many shots, evaluation discards extras and scores only the first Nreq shots.This keeps evaluation within the script’s prescribed shot window.
- Character Consistency: When a model generates too few shots, available similarities are normalized by Nreq −1, treating missing shots as zero similarity.For Nreq = 8 and Nout = 6, five similarities are divided by seven.
- Character Consistency: The multi-agent pipeline expands scripts into shot descriptions, retrieves or creates entity references, and generates keyframes and videos conditioned on those references.The agents process one shot at a time while maintaining characters, props, and scenes across references.
D. Prompt for Benchmark Story Generation
The benchmark-generation prompt produces 54 structured story scripts spanning nine combinations of shot count and persistent element. It enforces screenplay-style formatting, controlled entity persistence, and variation in the nonpersistent elements.
- The benchmark contains 54 scripts covering 4-, 8-, and 12-shot settings for character-, prop-, and scene-persistent stories.The design uses six samples for each of the nine combinations.
- Each story follows a three-act structure distributed rationally across the selected number of shots.The prompt gives approximate allocation ranges for the three acts.
- Scripts use natural screenplay descriptions, numbered shot paragraphs, one character and one prop, and family-friendly content constraints.First appearances receive detailed descriptions, while later appearances must preserve the established character and prop appearance.
- Character-persistent stories keep one character unchanged while varying scenes and prop status across shots.
- Prop-persistent stories keep one prop unchanged while varying scenes and character status across shots.
- Scene-persistent stories keep all shots in one location while varying character and prop status across shots.The prompt illustrates variation through combinations such as character-plus-prop, character-only, prop-only, or neither.
E. User Study
The user study evaluated whether VideoMemory’s consistency gains are perceptible to human observers across character, prop, and background persistence tasks. Five expert raters compared anonymized method pairs under controlled, randomized conditions, and VideoMemory was strongly preferred across subclasses.
- Participants: Five expert raters evaluated cross-shot consistency rather than overall aesthetics or story quality.Participants were fluent in English and familiar with basic cinematography concepts.
- Stimuli: Each trial compared videos generated from the same script, shot breakdown, and prompts, varying only the generation method.The study included 270 one-to-one comparisons between VideoMemory and five baselines across 54 benchmark scripts.
- Interface and task: Participants viewed anonymized videos side-by-side and answered one subclass-specific forced-choice consistency question.Questions targeted character, prop, or background consistency across all shots.
- Design and procedure: Trial order and method-side assignment were randomized, while participants could replay videos without a time limit.Each participant evaluated 270 pairs remotely and could take breaks between trials.
- Results: Preference rates reached 95.8% for character and background consistency, with VideoMemory strongly preferred across all subclasses.The authors report that human preferences aligned with the quantitative metrics.
F. Successful Cases
Additional qualitative examples illustrate VideoMemory’s ability to preserve characters, props, and backgrounds across changing settings, lighting, viewpoints, and actions.
- Character Consistency: Character consistency preserves a protagonist’s identity across indoor rooms, outdoor parks, and public transit.The example demonstrates identity preservation across diverse environments.
- Prop Consistency: Prop consistency maintains a bright orange hard hat’s color and structure despite changing lighting and camera angles.The object remains visually stable while presentation conditions vary.
- Background Consistency: Background consistency keeps a subway platform stable as a fixed backdrop for multiple characters and actions.The scene remains consistent while activity changes in the foreground.
User Study
The study interface presents a benchmark script alongside two anonymized videos and asks viewers to compare consistency for a specific entity class. Additional examples cover character, prop, and background persistence.
- Study interface: Each user-study trial shows the English script, two anonymized videos, and a subclass-specific consistency question.Participants choose between Method A and Method B using a forced-choice interface.
- Example story: One example follows a composer using a distinctive bright red notebook as a creative catalyst while traveling.The passage identifies the notebook as the story’s persistent prop.
- Qualitative examples: The qualitative examples span character consistency across changing environments, prop consistency for a hard hat, and background consistency in a subway station.The three stories correspond to the benchmark’s character-, prop-, and background-consistency modes.