Source-linked AI summary
PermaVid: Consistent Video Generation Across Edits via Disentangled Context Memory
Shuai Yang, Bingjie Gao, Ziwei Liu, Jiaqi Wang, Dahua Lin, Tong Wu
TL;DR
Long-term video consistency becomes difficult when edits change appearance or layout, especially across revisited viewpoints. PermaVid disentangles semantic appearance from geometric structure in multimodal memory and updates retrieval according to edit scope. Experiments show stronger semantic and structural consistency than existing methods under global and local edits.
Problem
Existing video-generation memories entangle changing semantic appearance with geometry, limiting long-term consistency across revisited viewpoints and edits.
Method
PermaVid uses separate RGB and depth memories with edit-aware updates and retrieval to refresh appearance while preserving reusable geometry.
Results
PermaVid achieves the best reported PSNR, SSIM, and LPIPS results under both global and local edits, while also leading semantic consistency and VBench-Avg.
Takeaways & Limitations
The framework preserves reusable geometric structure while propagating updated global or local appearance across time and revisited viewpoints.
Abstract
from arXiv · showhide
Consistent video generation under editing operations requires persistence: when edits modify scene appearance or layout, subsequent generations should remain coherent across time and viewpoints. However, existing memory designs struggle to maintain long-term consistency after such modifications, as stored contexts may become outdated or invalid. To address this, we propose PermaVid, a novel framework built upon a multi-modal context memory that disentangles spatial context into semantic appearance and geometric structure, together with an edit-aware memory update and retrieval strategy that keeps memory evolution aligned with subsequent observations. Specifically, we develop two complementary memory banks: an RGB context memory that captures appearance-aware observations while implicitly encoding geometry, and a depth context memory that preserves geometry-only structure disentangled from semantics. Building on this design, we introduce a memory-guided video generation model that performs multi-modal feature fusion under reference conditions drawn from mixed-modality memory contexts. Experiments demonstrate that our method maintains strong long-term semantic and structural consistency after edits, significantly outperforming state-of-the-art methods.
1 Introduction
Long-term visual consistency remains challenging in camera-moving and edited video generation because previously observed spatial context can become outdated after appearance changes. PermaVid addresses this by disentangling semantic appearance from geometric structure in a multi-modal context memory with separate RGB and depth representations.
- Motivation: Long-term visual consistency is difficult when moving cameras revisit previously observed regions, especially across extended temporal horizons.The challenge arises in controllable video generation and editing settings involving camera motion and revisited viewpoints.
- Existing approaches: Existing memory methods model temporal context or retrieve pose-conditioned features to improve temporal coherence and viewpoint consistency.These strategies store latent states or historical frames, or reuse visual information observed under similar viewpoints.
- Key insight: Spatial context separates into semantic appearance and geometric structure, which differ because appearance changes with edits, lighting, or style while geometry is typically more stable.This distinction motivates storing the two forms of spatial information separately.
- PermaVid: PermaVid introduces a disentangled multi-modal context memory for consistent video generation under editing operations.The framework models semantic appearance and geometric structure as separate but complementary memory representations.
- PermaVid: PermaVid maintains an RGB-based memory for appearance information and a depth-based memory for geometric structure.The two memory types implement the proposed separation between semantic appearance and geometry.
2 Related Works
Related work advances video generation through diffusion-model improvements, camera-conditioned control, and memory mechanisms that preserve temporal and cross-view coherence. These directions progressively address realism, controllability, geometric consistency, and historical-context retention.
- Video Generation: Diffusion models dominate video generation, evolving from U-Net–based models to Transformer-based frameworks and autoregressive step-wise generation.Research also improves performance through learning strategies, data curation, and prompt engineering.
- Camera-Controlled Video Generation: Camera-controlled methods use pose conditions or spatial constraints to enable scene exploration, viewpoint traversal, and revisiting while improving cross-view geometric coherence.These approaches partially alleviate geometric inconsistency through pose-conditioned generation or spatially aware representations.
- Memory Mechanisms: Memory-based methods retain historical information through temporal memory, feature caching, spatial retrieval, compressed keyframes, and spatiotemporal context representations.Recent approaches additionally use pose-conditioned retrieval and external memory structures to preserve long-term context.
3 Method
PermaVid disentangles semantic appearance from geometric structure in a multi-modal memory, then updates and retrieves references according to edit type, affected regions, and camera trajectories. A memory-guided DiT fuses compact RGB-depth references to generate consistent videos across time, viewpoints, and edits.
- Disentangled Context Memory: PermaVid separates spatial context into semantic appearance At and geometric structure Gt, storing them in complementary RGB and depth memory banks.Appearance covers identity, texture, color, illumination, and style; geometry covers layout, shape, and spatial relationships.
- Disentangled Context Memory: RGB memory supplies high-fidelity appearance references, while depth memory preserves geometry that remains useful when RGB content becomes outdated after edits.Depth context is largely invariant to texture, lighting, and style changes.
- Edit-Aware Memory Update: Global edits invalidate all RGB contexts and advance the semantic version, whereas local edits invalidate only units whose view footprints overlap the affected region.Depth memory is retained for global edits and local edits that do not change geometry.
- Edit-Aware Memory Retrieval: Retrieval selects memory that is spatially relevant and edit-valid, enforcing the current semantic version for RGB while retrieving depth by spatial relevance alone.At most B diverse, nonredundant memory units cover the target camera trajectory.
- Edit-Aware Memory Retrieval: This update-and-retrieval mechanism refreshes outdated appearance globally, refreshes only affected local regions, and reuses valid geometry without stale appearance references.The design aligns memory evolution with editing operations.
- Memory-Guided Generation: A memory-guided DiT injects camera and text conditions while a dedicated branch encodes independently VAE-processed RGB-depth references with relative positional encoding.Distributed and cascaded Context Blocks incorporate sparse observations sampled from different timestamps.
- Training Data: The UE-Mem dataset provides 4k videos of 1000 frames across 100 Unreal Engine scenes, with RGB-depth sequences and accurate 6-DoF camera poses.An automatic Unreal Engine 5 synthesis pipeline uses navigation and pose-tracking policies to create revisiting trajectories.
4 Experiments · 4.1 Implementation Details
The experiments implement PermaVid as a memory-guided video generation model based on Wan2.1-14B. Training uses variable-length clips at fixed 480 × 832 resolution, with the pretrained VACE memory-context branch frozen while only the main DiT and camera encoder are trained.
- 4.1 Implementation Details: The memory-guided video generation model is built on the Wan2.1-14B architecture.
- 4.1 Implementation Details: Training samples video clips with randomly varied lengths between 25 and 81 frames.
- 4.1 Implementation Details: All training videos use a fixed resolution of 480 × 832.
- 4.1 Implementation Details: The memory context branch is pretrained from VACE and retained to preserve strong multi-modal feature perception.
- 4.1 Implementation Details: The pretrained memory context branch remains frozen during training.
- 4.1 Implementation Details: Optimization updates only the parameters of the main DiT and camera encoder.
4.2 Metrics and Baselines.
The evaluation compares PermaVid with four state-of-the-art methods on a 200-image benchmark spanning diverse scene types and challenging camera trajectories with viewpoint revisiting.
- Baselines: The comparison includes HY-Worldplay, HY-Gamecraft, Matrix-Game-2.0, and VMem.These are identified as existing state-of-the-art methods for consistent video generation across edits.
- Benchmark: The benchmark contains 200 images sourced from free websites and AI-generated images.The images cover both realistic and stylized scenes.
- Evaluation Protocol: The evaluation set covers indoor and outdoor environments with random and complex camera trajectories involving viewpoint revisiting.
4.3 Qualitative Comparison
PermaVid is qualitatively evaluated for structural and semantic consistency under global and local edits. It preserves geometry while propagating edited appearance globally and recalls post-edit local content during viewpoint revisits.
- Global edits: Under global edits, PermaVid maintains stable geometric structure while propagating the edited semantic appearance across time and viewpoints.After a global edit at frame 240, revisiting views retain stable spatial layouts, building placements, and clock-tower geometry while the scene semantics are updated.
- Global edits: PermaVid preserves structural consistency when the camera revisits previously observed viewpoints after a global edit.The evaluation focuses on whether underlying geometry remains stable despite altered overall semantic appearance.
- Local edits: Under local edits, PermaVid consistently recalls the post-edit content at revisited viewpoints while preserving structural integrity and updated local semantics.This view-recall evaluation examines whether edited regions are correctly retained when the camera returns to the same viewpoints.
4.4 Quantitative Comparison
PermaVid achieves the strongest quantitative consistency under both global and local edits. It leads structural, semantic, recall-consistency, and visual-quality metrics, whereas baselines degrade or fail to update edited semantics.
- Global edits: Under global semantic edits, PermaVid achieves the best PSNR, SSIM, and LPIPS, indicating strong geometric-structure preservation.These metrics evaluate structural consistency within view recall consistency.
- Global edits: PermaVid significantly outperforms all baselines in semantic consistency measured by CLIP-Vid, effectively propagating edited appearance.HY-WorldPlay preserves structure but fails to update semantics, while VMem, HY-GameCraft, and Matrix-Game-2 degrade in both aspects.
- Local edits: Under local edits, PermaVid achieves the best PSNR, SSIM, and LPIPS, indicating accurate recall of edited local content at revisited viewpoints.The evaluation measures long-term view recall consistency and visual quality after local edits.
- Local edits: PermaVid also delivers the best visual quality according to VBench-Avg, while baseline methods show obvious degradation in view recall consistency.The degradation reflects baselines' inability to reliably recall edited content.
4.5 Ablation Study
The ablation compares disentangled and non-disentangled context memory settings. The disentangled design uses edit-aware retrieval to combine RGB and depth contexts while separately handling appearance and geometry.
- Disentangled Context Memory: The ablation contrasts settings with and without the proposed disentangled context memory.The comparison is between “w/ Disentangled Contexts” and “w/o Disentangled Contexts.”
- Disentangled Context Memory: The disentangled setting retrieves RGB-only, depth-only, or mixed-modality reference contexts according to the edit-aware retrieval strategy.This multi-modal memory design supports different reference-context modalities depending on the edit.
- Disentangled Context Memory: It separates semantic appearance from geometric structure, enabling geometry reuse while selectively updating outdated appearance.The passage describes this separation as the mechanism for handling edits in the proposed memory design.
4.6 Memory Overhead Analysis
The memory mechanism adds only marginal runtime overhead during long-horizon generation. Inference is dominated by the video generation backbone, while depth estimation and memory retrieval remain negligible throughout the sequence.
- Overhead profiling: The analysis profiles component runtime ratios and absolute memory-retrieval time during long-horizon generation with a large-loop camera trajectory.This evaluates overhead across the full generation sequence.
- Retrieval scaling: Memory retrieval time gradually increases as historical contexts accumulate but remains at the millisecond level.The measured growth still indicates only marginal runtime overhead.
- Component runtime: The video generation backbone consistently accounts for nearly all inference time.Depth estimation and memory retrieval remain negligible across the entire sequence.
5 Conclusion
PermaVid preserves long-term video consistency across editing operations through a multi-modal context memory that separates semantic appearance from geometric structure. Its edit-aware updating and retrieval selectively maintain context aligned with the latest edited state.
- PermaVid targets long-term consistency in video generation across editing operations.
- Its multi-modal context memory disentangles spatial context into semantic appearance and geometric structure.
- An edit-aware memory update and retrieval strategy selectively updates and reuses cross-modal memory to keep retrieved context aligned with the latest edited state.