Source-linked AI summary
Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing
Dohun Lee, Chun-Hao Paul Huang, Xuelin Chen, Jong Chul Ye, Duygu Ceylan, Hyeonho Jeong
TL;DR
Existing video-to-video models often handle sequential edits independently, creating cross-turn consistency problems as regions drift or are overwritten. Memory-V2V uses prior edits as structured constraints through task-aware retrieval and relevance-aware compression, improving consistency across novel-view synthesis and long-video editing while reducing computation. Its scope remains tied to the limitations of the underlying editor and continuous single-shot training data.
Problem
Existing video editing models often treat sequential turns independently, causing previously generated regions to drift or be overwritten and limiting cross-turn consistency.
Method
Memory-V2V incorporates prior editing history as structured constraints using task-aware retrieval, relevance-aware tokenization, and adaptive token merging.
Results
Memory-V2V substantially improves cross-iteration consistency on video novel view synthesis and text-guided long video editing while outperforming state-of-the-art baselines in single-turn editing.
Takeaways & Limitations
Memory-V2V enables scalable and consistent iterative editing across diverse video tasks without linear growth in computational cost.
Takeaways & Limitations
Memory-V2V inherits limitations such as large view changes from its underlying single-turn editor and may struggle with multi-shot videos containing abrupt scene transitions.
Abstract
from arXiv · showhide
Video-to-video diffusion models achieve impressive single-turn editing performance, but practical editing workflows are inherently iterative. When edits are applied sequentially, existing models treat each turn independently, often causing previously generated regions to drift or be overwritten. We identify this failure mode as the problem of cross-turn consistency in multi-turn video editing. We introduce Memory-V2V, a memory-augmented framework that treats prior edits as structured constraints for subsequent generations. Memory-V2V maintains an external memory of previous outputs, retrieves task-relevant edits, and integrates them through relevance-aware tokenization and adaptive compression. These technical ingredients enable scalable conditioning without linear growth in computation. We demonstrate Memory-V2V on iterative video novel view synthesis and text-guided long video editing. Memory-V2V substantially enhances cross-turn consistency while maintaining visual quality, outperforming strong baselines with modest overhead.
1 Introduction
Multi-turn video editing exposes a cross-turn consistency problem: independently generated sequential edits can drift or overwrite earlier results. Memory-V2V addresses this by treating prior edits as selectively retrieved, compressed constraints and improves consistency across two editing tasks.
- Motivation: Multi-turn workflows progressively refine video outputs across multiple interactions, requiring attribute-level control over subjects, motion, or viewpoint.
- Motivation: Sequential edits can produce inconsistent novel-view regions and gradual global appearance drift across independently edited video segments.An apple changed into an orange can appear as visually different oranges across segments.
- Memory-V2V: Memory-V2V treats previous outputs as structured external constraints that selectively enforce consistency across editing iterations rather than as temporal context.
- Memory-V2V: Task-aware retrieval selects relevant prior edits, while dynamic tokenization and adaptive token merging allocate capacity according to memory relevance.The approach retrieves edits using task-specific signals and compresses less relevant memory more aggressively.
- Evaluation: Memory-V2V improves cross-iteration consistency in video novel view synthesis and text-guided long video editing while outperforming state-of-the-art baselines in single-turn editing.
2 Related Work
Prior video editing and generation methods use optimization, explicit visual cues, or memory mechanisms, but long-video segmentation and limited temporal context leave consistency challenges unresolved. Memory-V2V targets this gap in iterative video editing.
- Video novel view synthesis: Video novel view synthesis seeks plausible unseen-camera videos while preserving 3D structure and temporal dynamics, but paired multi-view video data are scarce.Prior approaches therefore use test-time optimization or scene-specific overfitting of monocular video generators.
- Video novel view synthesis: Point-cloud renderings provide geometric proxies for target camera trajectories, but iterative point-cloud refinement does not generalize to dynamic scenes with non-rigid motion.
- Text-guided video editing: Text-guided editing methods use deterministic inversion, test-time optimization, or explicit cues such as depth and optical flow, yet can introduce temporal artifacts.
- Text-guided video editing: Instruction-based video-to-video models preserve subject identity and motion but have limited temporal context, making independent long-video segment editing prone to appearance inconsistencies.
- Long video generation with memory: Memory-based long-video generators address past-generation information through external caches, but Memory-V2V incorporates prior editing history into the current conditioning input for cross-edit consistency.
3 Memory-V2V
Memory-V2V extends video-to-video diffusion to multi-turn editing by using prior outputs as selectively retrieved, relevance-weighted constraints. Dynamic tokenization and adaptive merging preserve important details while limiting the computational cost of conditioning on editing history.
- Framework overview: Memory-V2V conditions each new edit on the current input and a selectively retrieved subset of previously edited videos rather than the full history.The retrieved videos are stored in an external cache and selected as the top-k most relevant memories.
- Evaluation: The framework is evaluated on video novel view synthesis and text-guided long video editing, improving cross-iteration consistency and single-turn editing performance over state-of-the-art baselines.The paper reports these improvements while using selective retrieval and compression to avoid conditioning on the entire editing history.
- Video retrieval: Video novel view synthesis retrieves memories by geometric field-of-view similarity between the target and cached camera trajectories.VideoFOV constructs frame-level visibility from sampled viewing directions, unions these into video-level FOVs, and combines overlap and containment scores for ranking.
- Dynamic tokenization: Dynamic tokenization assigns finer compression to more relevant videos and stronger compression to less relevant retrieved videos.The 1×2×2 tokenizer processes the user input, 1×4×4 processes the top-3 retrieved videos, and 1×8×8 processes the remaining videos.
- Adaptive token merging: Adaptive token merging uses a learnable convolutional operator to compact tokens from frames with low attention responsiveness.The reduction factor scales with the number of conditional videos, and adaptive merging preserves generation quality better than discarding low-importance tokens.
- Text-guided long-video editing: Memory-V2V supports text-guided long-video editing by reformulating videos longer than the base model’s context window as memory-aware iterative editing.Training avoids long paired videos by extending only target videos from short source–target pairs, and edited segments are stitched into one output video.
4 Experiments
Experiments evaluate Memory-V2V on iterative video novel view synthesis and text-guided long video editing, with ablations examining consistency, token responsiveness, compression, and computational cost. Across these evaluations, the method preserves consistency and visual quality while reducing the cost of scalable conditioning.
- Evaluation Setup: Memory-V2V is evaluated for iterative video novel view synthesis and text-guided long video editing, using finetuned extensions of ReCamMaster and LucyEdit.The novel-view model uses synthetic multi-camera videos, while long-video editing uses Señorita data with temporally extended targets.
- Video Novel View Synthesis: Memory-V2V maintains consistency across all evaluated novel-view iterations while preserving visual quality and improving camera adherence.Base models lose cross-consistency as iterations increase; sequential ReCam preserves only some pairwise relationships.
- Component Ablation: VideoFOV retrieval with dynamic tokenization preserves long-term consistency between the 1st and 5th generations, while adaptive token merging maintains efficiency without visible degradation.The component ablation attributes consistency gains to retrieval and dynamic tokenization, and computational savings to adaptive token merging.
- Token Responsiveness: Highly responsive conditioning tokens are necessary for cross-consistency, while merging low-responsive tokens preserves consistency and visual fidelity better than discarding them.Discarding low-responsive tokens introduces artifacts and can degrade fine visual details and motion.
- Computational Cost: Over 90% reductions in FLOPs and latency are achieved versus naive context-window scaling, with adaptive token merging providing a further 30% reduction.The computational gains increase as the number of memory videos grows.
5 Conclusion
The paper formulates multi-turn video editing as a consistency challenge and introduces Memory-V2V, which uses prior edits as structured constraints. It concludes that task-aware retrieval and relevance-aware compression support scalable iterative editing, while acknowledging limits inherited from the base model and training data.
- Conclusion: Memory-V2V treats prior edits as structured constraints and combines task-aware retrieval with relevance-aware compression for iterative video editing.The framework is designed to maintain consistency across diverse video tasks without treating prior outputs as ordinary temporal context.
- Limitations: Memory-V2V inherits the underlying single-turn editor's limitations, including difficulty with large view changes.Its training data contains continuous single-shot videos, so multi-shot long videos with abrupt scene transitions may remain challenging.
- Preliminary: Flow-based training progressively transforms samples from a prior distribution toward the target data distribution through linear interpolations and reverse integration.The flow ODE defines the velocity-guided transformation, while inference uses predicted velocity to guide an initial noise sample.
B.1 Video retrieval and adaptive token merging experiments.
The experiment evaluates retrieval and adaptive token merging for long-term consistency, finding that retrieval improves cross-iteration consistency while token merging reduces runtime without harming quality or consistency.
- Video retrieval improves cross-iteration multi-view consistency in fifth-iteration outputs.MEt3R compares videos generated in the first and fifth iterations to assess preservation over long editing sequences.
- Adaptive token merging significantly reduces runtime without degrading generation quality or consistency.
B.2 Token reduction experiments.
The token reduction experiment uses overlapping camera trajectories at the first and fifth iterations, with random trajectories in between, and evaluates fifth-iteration outputs.
- During the third iteration, tokens from 50% of frames are merged or discarded according to responsiveness.High-responsiveness manipulation uses the top 50% of frames by responsiveness, while low-responsiveness manipulation uses the bottom 50%.
- Frames geometrically aligned with the target trajectory generally exhibit higher responsiveness than unrelated frames.Responsiveness is averaged across transformer blocks for related and unrelated historical frames.
C Proof-of-Concept Experiment: Ideal Context Encoder
The proof-of-concept tests whether context encoders can preserve novel-view content across two editing turns. Video-latent context performs best, whereas CUT3R and LVSM states lack sufficient fine-grained information.
- The experiment tests whether encoded first-turn output E(x1) can preserve consistency between novel content in x1 and x2.The second turn conditions on the original source video and encoded context from the first generated video.
- Three pretrained context encoders are evaluated: CUT3R, LVSM, and a video VAE.Encoded representations are patchified to match DiT video-token dimensionality, and the encoder is frozen during finetuning.
- CUT3R and LVSM state tokens fall short of encoding detailed appearance in newly generated regions.
- The same video latent space achieves the best quality and consistency, motivating its adoption for context representation.
C.1 Context Encoder Setup
The context-encoder setup examines recurrent geometric states, novel-view latents, and retrieval representations as conditioning signals. It also specifies sphere-based visibility computation and semantic segment retrieval procedures.
- CUT3R Context Encoder: CUT3R maintains a recurrent geometric state that accumulates 3D cues across video frames.The state is projected into transformer latent space and injected through a dedicated conditioning branch for diffusion generation.
- CUT3R Context Encoder: CUT3R state conditioning fails to provide sufficiently informative guidance because reconstruction from the state is coarse and blurry.
- LVSM Context Encoder: LVSM jointly encodes input images and Plücker ray embeddings into a shared feature space for novel-view synthesis.Its tokens and an initial latent token are processed by an encoder, producing a latent state used by the model’s renderer.
- LVSM Context Encoder: Conditioning diffusion generation on the LVSM latent state provides little benefit because it lacks transferable, fine-grained geometric cues.
- VideoFOV Retrieval: VideoFOV retrieval samples 64,800 directions on a unit sphere and marks points visible when projections lie within image bounds with positive depth.Visibility sets are accumulated across frames to compute video-level field of view and relevance scores.
- VideoFOV Retrieval: The VideoFOV workflow ranks retrieved videos by descending relevance relative to a target query video.
- Text-Guided Retrieval: Text-guided long-video retrieval ranks segments using cosine similarity between averaged DINOv2 frame descriptors.The most recently edited segment is always included to maintain temporal coherence between consecutive output segments.
E Additional Training and Inference Details
This section describes positional, camera, and memory-conditioning strategies for stable multi-turn video generation, including RoPE assignments, noise-based training, and per-video camera embeddings.
- Positional embedding (RoPE) design: Disjoint temporal RoPE ranges separate target, user-input, and memory videos as the conditioning set expands across editing iterations.The ranges are τ_tgt={0,…,T−1}, τ_usr={T,…,2T−1}, and τ_mem={2T,…,3T−1}.
- Mitigating training-inference gap: Memory-token noise and RoPE dropout train the model to prioritize user-input videos while allowing memory videos to provide an independent conditioning signal.Memory tokens receive Gaussian perturbations, while the user-input RoPE range is disabled with probability p during training.
- Mitigating training-inference gap: The resulting RoPE formulation supports controlled information flow from user-input videos to target videos while mitigating error propagation through accumulated memory videos.This training procedure is designed for stability as memory videos accumulate across turns.
- Camera conditioning strategy: Camera extrinsics are embedded separately for each video, enabling the model to distinguish heterogeneous viewpoints across target, user-input, and memory videos.Each camera trajectory is represented by per-frame 3 × 4 rigid transformations and projected into feature space through E_c(·).
- Dataset construction: The long-form editing dataset extends Señorita-2M editing pairs because its original clips contain only 33 frames, insufficient for long-horizon editing.The construction preserves stable local editing pairs while extending their temporal duration.
- Positional embedding (RoPE) design: Long-video editing assigns separate RoPE ranges to the target, immediately preceding, and remaining memory segments to preserve recent continuity alongside broader historical context.The preceding segment receives τ_prev={T,…,2T−1}, while other memory segments receive τ_mem={2T,…,3T−1}.
- Mitigating training-inference gap: Training-inference mismatch arises because training places generated memory frames after targets, whereas inference uses previously generated memory frames before targets.The section addresses this ordering discrepancy through RoPE design and training procedures.
- Failure cases: Memory-V2V can struggle with long videos containing multiple shots and large scene transitions.Figure 17 identifies this setting as a failure case.
F Additional Qualitative Results
Additional qualitative results are provided for both multi-turn video novel view synthesis and text-guided long video editing.
- Additional results: Further qualitative results cover multi-turn video novel view synthesis and text-guided long video editing.The results are referenced in Figures 18 and 19.
G Additional Discussion
The discussion identifies data-related limitations in multi-shot and long-video settings, while the appendices provide additional results and retrieval algorithm details.
- Limitations: Training uses only continuous single-shot videos, limiting robustness to real long-form content with abrupt shot transitions.The model may propagate objects or textures from a preceding shot into the next.
- Limitations: Imperfect generated training extensions can accumulate deviations in memory and produce visible appearance drift during repeated denoising.The cited artifacts include flickering, tone changes, and blur at real-to-synthesized junctions.
- Future directions: The authors propose multi-shot datasets and higher-quality long-video pairs as directions for addressing the identified limitation.They also mention diffusion distillation or autoregressive generation frameworks as future work for improving interactivity.
- Retrieval algorithms: VideoFOV Retrieval selects top-k memory videos using spherical field-of-view overlap scores.The algorithm takes target-video camera parameters, sphere samples, a balance parameter λ, and top-k as inputs.
- Retrieval algorithms: DINOv2-based Segment Similarity Ranking sorts candidate segments using descriptor similarity and can enforce the most recent segment first.The procedure outputs sorted indices and similarity scores.