Source-linked AI summary
Memento: Reconstruct to Remember for Consistent Long Video Generation
Xuan Wei, Longbin Ji, Guan Wang, Xiangrui Liu, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Qingqi Hong
TL;DR
Long-form video generation must preserve recurring subject identities across shots, but existing memory selection does not explicitly verify identity-critical evidence. Memento addresses this with reconstruction-supervised dual-query memory and achieves state-of-the-art consistency, coherence, and visual quality.
Problem
Long-form video generation lacks explicit verification that historical memory preserves identity-critical evidence needed for recurring subject consistency across shots.
Method
Memento jointly trains shot-level autoregressive generation with memory-based subject reconstruction and dual-query retrieval for long-term identity evidence and short-term continuity.
Results
0.7338 inter-shot consistency, outperforming StoryMem (0.6606), while achieving superior long-term subject consistency, cross-shot coherence, and visual quality.
Takeaways & Limitations
Memento explicitly preserves long-term identity evidence while maintaining short-term contextual continuity and extending to long-form generation without architectural modification.
Takeaways & Limitations
Autoregressive generation may propagate degraded shots through memory, and physical plausibility remains limited by the video backbone’s lack of explicit physics modeling.
Abstract
from arXiv · showhide
Long-form video generation requires recurring subjects to remain consistent across various shots, viewpoints, motions, and scene transitions. Existing temporal decomposition methods improve scalability by generating videos shot by shot. However, they mainly focus on optimizing plausible next-shot continuations without verifying whether the historical memory preserves identity-critical subject evidence. Consequently, as generation proceeds, recurring subjects may be diluted, overwritten, or forgotten. In this paper, we propose Memento, a subject-reconstruction-guided framework that treats subject preservation as an explicit identity grounding problem, based on the premise that a memory bank faithfully preserving a subject should support reconstructing that subject from memory alone. Specifically, Memento jointly trains autoregressive next-shot generation with memory-based subject reconstruction, recovering target appearances using historical memory and global story captions. To disentangle long-range subject evidence from short-range cues, Memento introduces a dual-query memory mechanism, where one query retrieves identity-relevant memory and the other selects short-context keyframes for coherent continuation. Additionally, a subject-aware cinematic data pipeline provides precise reconstruction supervision via consistent, pronoun-free subject descriptions. Experiments demonstrate that Memento achieves state-of-the-art performance in long-term subject consistency, cross-shot coherence, and visual quality.
1 Introduction
Long-form video generation requires recurring subjects to preserve their identities across shots, viewpoints, motions, and scene transitions, but existing memory strategies may dilute identity-critical evidence. Memento addresses this as an identity-grounding problem through memory-based subject reconstruction, dual-query retrieval, and subject-aware supervision.
- 1 Introduction: Long-form video generation must preserve recurring subject identities across multiple shots, viewpoints, poses, motions, and scene transitions.Existing temporal decomposition improves scalability and cross-shot coordination, but memory selection often prioritizes generic relevance, saliency, or short-term context over identity preservation.
- 1 Introduction: Memento explicitly supervises identity preservation by reconstructing target subject appearances from historical memory and the global story caption without direct visual prompts.This reframes subject consistency as an identity-grounding problem rather than only plausible local continuation.
- 1 Introduction: Memento disentangles memory retrieval with dual queries for long-range identity evidence and short-range visual cues supporting local shot generation.Long-range evidence remains stable across distant shots and scene changes, whereas short-range cues capture recent layout, motion tendency, and camera continuity.
- 1 Introduction: Memento generates videos autoregressively at the shot level while maintaining a fixed-length historical memory bank derived from prior memory tokens and the most recent shot’s visual features.The framework combines this memory design with a global story caption and per-shot captions.
- 1 Introduction: A subject-aware cinematic data pipeline provides consistent, pronoun-free subject descriptions and reconstruction targets, while experiments show superior long-term subject consistency, cross-shot coherence, and visual quality.The reported evaluation spans diverse long-form scenarios.
2 Related Work
Related work advances long-form video generation through storyboard-based, joint multi-shot, and memory-conditioned autoregressive paradigms, but each faces limitations in coupling, scalability, or subject preservation. Memento addresses these gaps by explicitly grounding long-form consistency in memory-based subject reconstruction with dual-query retrieval and joint next-shot training.
- Background: Short-shot diffusion, T2V, and I2V models achieve high visual fidelity and natural motion but lack explicit mechanisms for consistency across multiple shots.These models are primarily designed for temporally bounded single scenes.
- Storyboard-based generation: Storyboard-based methods animate sparse keyframes efficiently, but weak clip coupling can cause rigid transitions, detail drift, and limited narrative coherence.Consistency is enforced mostly at the keyframe level.
- Joint multi-shot generation: Joint multi-shot methods improve global interaction and shot-level alignment through cross-shot attention, yet context-window limits and rapidly increasing memory and computation costs restrict open-ended generation.These costs grow as the number of shots increases.
- Memory-conditioned autoregressive generation: Memory-conditioned autoregressive methods improve scalability by generating shots sequentially from compact historical memory, but existing selection emphasizes generic relevance or short-term context compression rather than explicit subject preservation.StoryMem selects keyframes using semantic and aesthetic criteria, while OneStory compresses historical context for next-shot generation.
- Memento: Memento formulates long-form consistency as subject grounding in memory, separating long-context subject evidence from short-context cues and jointly training next-shot generation with memory-based subject reconstruction.This encourages identity-critical information retention while preserving local visual continuity.
3 Method
Memento generates long videos autoregressively at the shot level while explicitly preserving recurring subjects through memory-based reconstruction. Its method combines subject-aware supervision, dual-path memory retrieval, and joint next-shot generation with identity reconstruction.
- Framework Overview: Memento uses a compact historical memory bank to preserve persistent subject evidence while retaining recent visual context for local shot continuity.Videos are generated autoregressively shot by shot, with optional use of the previous shot’s final frame for smoother transitions.
- Subject-Aware Data Curation: The subject-aware data pipeline creates global story, per-shot, and reconstruction captions using fixed, pronoun-free references for recurring subjects.Reconstruction-target captions pair subject-salient frames with unambiguous descriptions to supervise identity preservation.
- Dual-Query Memory: Dual-query retrieval separates story-conditioned selection of long-range subject evidence from shot-conditioned selection of short-context references for upcoming generation.Each path selects top-K tokens from a shared candidate pool, reducing competition between persistent identity information and transient scene context.
- Memory Bank Construction: Memento maintains a fixed-length memory by combining prior memory with features from the latest generated shot, preserving selected historical and newly observed evidence.A pretrained VAE encodes frames into latent features before candidate selection.
- Subject-Anchored Multi-Task Training: Subject-anchored multi-task training jointly optimizes next-shot diffusion denoising and memory-based subject reconstruction using the shared memory-conditioned generator.Next-shot denoising conditions on the memory bank, global story caption, and current shot caption, adding identity supervision without extra training stages.
4 Experiments
Memento is evaluated against representative multi-shot video generation methods using automatic metrics, qualitative comparisons, and a user study. It achieves strong long-term subject consistency and narrative coherence, while ablations support reconstruction-guided training and disentangled memory as key components.
- Quantitative evaluation: Memento achieves the highest inter-shot, intra-shot, and inter-scene subject consistency, with scores of 0.7338, 0.8578, and 0.7268, respectively.It also obtains the best story-level and shot-level semantic consistency, at 0.3063 and 0.2893.
- Qualitative evaluation: Qualitative comparisons show that Memento preserves subject appearance more stably than StoryMem and HoloCine across viewpoints, lighting, backgrounds, and scene transitions.Reconstruction frames provide explicit subject anchors, reducing identity drift during story progression.
- User study: In a user study over 30 cases with 10 participants, Memento wins cross-shot consistency comparisons against StoryMem by 57.3% and HoloCine by 69.0%.It also leads in prompt following, with win rates of 60.3% against StoryMem and 63.0% against HoloCine.
- Ablation study: Adding subject reconstruction raises inter-shot subject consistency from 0.7227 to 0.7489, whereas adding disentangled memory changes it to 0.7338.Disentangled memory nonetheless improves semantic, background, intra-shot, and inter-scene consistency by separating long-term identity evidence from short-term cues.
- Advanced capabilities: Memento also supports age-consistent identity preservation while maintaining facial structure, hairstyle tendency, and overall appearance across controllable temporal changes.This indicates separation of persistent identity cues from temporal variation rather than simple copying of a fixed reference.
5 Conclusion
Memento preserves long-term subject identity while maintaining short-term continuity through dual-query memory retrieval and memory-based reconstruction, supported by subject-aware data curation. Its limitations are error propagation from degraded memory writes and limited physical plausibility from the video backbone.
- Conclusion: Memento combines dual-query memory retrieval with memory-based subject reconstruction to preserve long-term identity evidence during next-shot generation.The framework also maintains short-term contextual continuity.
- Conclusion: A subject-aware data curation pipeline supports Memento’s cross-shot subject consistency, narrative coherence, and visual quality.The framework naturally extends to longer video generation.
- Limitations: Autoregressive generation may propagate errors when degraded shots are written into memory.This is identified as Memento’s first limitation.
- Limitations: Physical plausibility remains limited because the video backbone lacks explicit modeling of gravity, object permanence, and rigid-body dynamics.Future work will study robust memory filtering and physics-aware generation.
A Appendix · B Training Details · C Additional Data Details
The appendix reports the curated dataset’s scale, sequence structure, and duration, alongside the tools and captioning workload used for annotation. It also references detailed caption statistics in Table 3.
- C Additional Data Details: The curated dataset contains 2,033 video sequences and 20,227 clips.
- C Additional Data Details: Each sequence contains 9.95 clips on average, ranging from 6 to 16 clips.
- C Additional Data Details: Average sequence duration is 45.9 seconds, with a 42.9-second median and a 30-to-65-second range.
- C Additional Data Details: Annotation uses Qwen3-VL-8B for caption generation, PaddleOCR for OCR detection, and ByteTrack for identifying reconstruction frames.
- C Additional Data Details: Qwen3-VL-8B is called 26,252 times across sequence captioning, clip captioning, and reconstruction captioning.
- C Additional Data Details: Table 3 presents detailed caption statistics for the curated dataset and generated captions.Caption length is measured by character count.
D Evaluation Metrics
The benchmark evaluates semantic, background, subject, aesthetic, and cross-shot/scene consistency, with cosine-similarity metrics computed from L2-normalized feature vectors unless otherwise specified.
- Evaluation Scope: The evaluation covers semantic consistency, background consistency, subject consistency, aesthetic quality, and cross-shot/scene consistency.These dimensions define the benchmark’s evaluation scope.
- Metric Definitions: The paper provides detailed definitions for the evaluation metrics used in its benchmark.The metrics are designed to assess multiple forms of consistency and visual quality.
- Similarity Computation: Unless otherwise specified, all feature vectors are L2-normalized before cosine similarity is computed.This normalization establishes the default procedure for similarity-based evaluation.
D.1 Semantic Consistency · D.2 Background Consistency · D.3 Subject Consistency
The evaluation measures semantic alignment, background stability, and recurring-subject identity consistency using ViCLIP, CLIP, and DINOv2 features, respectively. Metrics combine normalized feature similarities at scene and/or video levels, with subject consistency additionally separating short-term smoothness from long-range identity preservation.
- D.1 Semantic Consistency: Semantic consistency uses ViCLIP to measure alignment between generated video content and textual prompts.The evaluation uses the OpenGVLab/ViCLIP-B-16-hf checkpoint as the video-text encoder.
- D.1 Semantic Consistency: Global semantic consistency samples 24 frames per scene, averages normalized 8-frame clip features, and compares them with concatenated scene prompts.Video and text representations are compared after feature aggregation and L2 normalization.
- D.1 Semantic Consistency: Shot-level semantic consistency independently averages ViCLIP features from 8-frame clips and reports mean cosine similarity with each shot’s prompt.The score is averaged over all shots.
- D.2 Background Consistency: The background score averages cosine similarity between adjacent frames sampled at a fixed CLIP_SAMPLE_INTERVAL.This procedure follows CLIP encoding and L2 normalization of each sampled frame.
- D.2 Background Consistency: Background consistency is reported as intra-scene consistency within scenes and intra-video consistency across frames concatenated over the story.These provide scene-level and video-level views of temporal background stability.
- D.3 Subject Consistency: Subject consistency evaluates recurring-identity stability with DINOv2 ViT-B/14 features and combines consecutive-frame similarity with similarity to the first frame.The first term measures short-term temporal smoothness, while the second measures long-range identity preservation; the metric is reported at scene and video levels.
D.4 Aesthetic Quality … D.7 Inter-Scene Consistency
The evaluation measures aesthetic quality and consistency at frame, shot, scene, and transition levels. It combines normalized CLIP-based aesthetics with ViCLIP- and DINOv2-based similarity metrics for local and long-range preservation.
- D.4 Aesthetic Quality: Aesthetic quality uses OpenAI CLIP ViT-L/14 features and a LAION aesthetic MLP predictor to score each generated frame.CLIP features are L2-normalized before aesthetic prediction.
- D.4 Aesthetic Quality: The final aesthetic score averages normalized frame-level scores, while standard deviation measures stability across frames.The original score range [1, 10] is normalized to [0, 1].
- D.5 Inter-Shot Consistency: ViCLIP-based inter-shot consistency averages cosine similarities between adjacent shot-level video features extracted from uniformly sampled frames.The metric evaluates preservation of visual and identity information across adjacent shots.
- D.5 Inter-Shot Consistency: DINOv2-based inter-shot subject consistency averages cosine similarities between adjacent shots represented by meanpooled sampled-frame features.This variant focuses on identity-related consistency across neighboring shots.
- D.5 Inter-Shot Consistency: Character-grouped inter-shot consistency groups shots by [Person X] tags and averages pairwise ViCLIP similarities within and across character groups.This score specifically evaluates recurring-subject consistency.
- D.6 Intra-Shot Consistency: Intra-shot consistency corresponds to intra-scene subject and background consistency, measuring stability of subjects, backgrounds, and layouts across local temporal frames.It evaluates temporal coherence within the same scene or shot.
- D.7 Inter-Scene Consistency: Inter-scene consistency averages ViCLIP cosine similarities between adjacent scene-level feature vectors encoded from entire scenes.Higher scores indicate better preservation of subject and visual identity across scene boundaries.