Source-linked AI summary
EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editing
Yehonathan Litman, Shikun Liu, Dario Seyb, Nicholas Milef, Yang Zhou, Carl Marshall, Shubham Tulsiani, Caleb Leak
TL;DR
Existing generative video inpainting methods process full spatiotemporal context even for localized edits, making them computationally expensive. EditCtrl uses masked-token local processing and lightweight global temporal guidance through adapters on a frozen video model. It reports improved quality and efficiency, including more than 50× speedup over generative baselines, while enabling multi-region editing and temporal propagation.
Problem
Full-attention video inpainting methods process the entire spatiotemporal context regardless of the edit region, creating significant computational costs for localized edits.
Method
EditCtrl uses a sparse local encoder for masked-region tokens and a lightweight temporal global context embedder, trained through adapters on a frozen pretrained model.
Results
EditCtrl improves quality and efficiency over state-of-the-art methods, with more than a 50× speedup over generative baselines while matching or slightly exceeding full-attention quality and prompt alignment.
Takeaways & Limitations
The disentangled design supports arbitrary-resolution editing, simultaneous multi-region prompts, and real-time propagation into future frames.
Takeaways & Limitations
The video VAE degrades background context, the local encoder struggles with very fast motion, and tiled VAE encoding and decoding constrain 4K throughput.
Abstract
from arXiv · showhide
High-fidelity generative video editing has seen significant quality improvements by leveraging pre-trained video foundation models. However, their computational cost is a major bottleneck, as they are often designed to inefficiently process the full video context regardless of the inpainting mask's size, even for sparse, localized edits. In this paper, we introduce EditCtrl, an efficient video inpainting control framework that focuses computation only where it is needed. Our approach features a novel local video context module that operates solely on masked tokens, yielding a computational cost proportional to the edit size. This local-first generation is then guided by a lightweight temporal global context embedder that ensures video-wide context consistency with minimal overhead. Not only is EditCtrl 10 times more compute efficient than state-of-the-art generative editing methods, it even improves editing quality compared to methods designed with full-attention. Finally, we showcase how EditCtrl unlocks new capabilities, including multi-region editing with text prompts and autoregressive content propagation.
1 Introduction
Generative video inpainting must create semantically rich, temporally coherent edits without paying full-video computation for localized changes. EditCtrl addresses this with disentangled local and global control through lightweight adapters, supporting efficient and interactive editing.
- Generative video inpainting replaces arbitrary video regions with high-fidelity, contextually consistent content, but data-driven methods struggle with new objects and large scene replacements.
- Full-attention inpainting methods process the entire spatiotemporal video context regardless of whether each region requires editing, creating significant computational cost.
- EditCtrl scales computation with the edit region by using a sparse local context encoder for masked tokens and a lightweight temporal global embedder for video-wide cues.
- Lightweight adapters on a frozen base model preserve pretrained generative quality and compatibility with distilled and autoregressive model variants.
- EditCtrl matches its full-attention base model and outperforms other state-of-the-art methods while supporting arbitrary-resolution editing, multi-region inpainting, and real-time content propagation.
2 Related Works
Prior video inpainting and acceleration methods improve efficiency or quality but remain limited in complex, text-conditioned, temporally controlled editing. EditCtrl combines disentangled local and global control with adapter-based integration for efficient, flexible video editing.
- Early video inpainting methods efficiently target regions but struggle with complex motion, masked regions, and coherent semantic content conditioned on text prompts.
- Video-generation acceleration approaches reduce diffusion steps, use sparse attention, or apply linear transformers, but these are alternative routes to computational efficiency.
- EditCtrl extends mask-focused computation to video while learning local adapters instead of fully fine-tuning the model, preserving integration with additional control variants.
- EditCtrl treats local and global context as high-frequency and low-frequency control components, respectively, within spatiotemporal video editing.
- Lightweight adapters disentangle sparse local and global mask components, accelerating editing with minimal base-model interference and enabling multi-prompt regional editing and temporal content propagation.
3 Methodology
EditCtrl separates local masked-region processing from lightweight global temporal conditioning, using adapters with a frozen video diffusion backbone. This design scales computation with the edit area while supporting coherent editing, multiple regions, and future-frame propagation.
- Local Context Encoder: The framework derives local context from encoded background content concatenated with down-sampled edit masks, then injects local features into selected transformer layers.Mask dilation expands the selected neighborhood to improve blending around edited regions.
- Disentangled Editing Architecture: EditCtrl processes only masked-region tokens locally while encoding global temporal context separately to preserve coherent video edits.The local encoder operates on selected context tokens, while the global embedder supplies video-wide scene and temporal cues.
- Adapter-Based Finetuning: EditCtrl fine-tunes lightweight adapters on a frozen pretrained diffusion model, preserving the backbone’s generative capabilities and compatibility with model variants.The architecture is designed to compose with adapters pretrained for full-attention tasks.
- Global Context Embedder: A temporal global context embedder compresses video-wide scene cues and injects attention features into text-conditioned representations with minimal compute overhead.The global representation is produced from a spatially down-sampled background and remains invariant to the original video resolution.
- Training Strategy: The local encoder is trained before the global embedder through a piecewise loss because jointly training both modules from the start is unstable.The staged schedule uses the local loss before switching to the combined objective after a predefined number of iterations.
- Interactive Editing: EditCtrl’s masked-region computation supports high-resolution editing, simultaneous distinct edits, and real-time propagation into future frames.Separate regions can be batch-processed and merged into their output latent locations, while propagated background supplies context when future frames are unavailable.
4 Experiments
EditCtrl is evaluated on diverse video editing and inpainting datasets against generative, non-generative, and full-attention baselines. It achieves strong quality and prompt alignment with substantially lower compute, while ablations and interactive applications examine its components and capabilities.
- Experimental Setup: EditCtrl is evaluated on VPBench-Edit, DAVIS, and VPBench-Inp using quality, preservation, semantic-alignment, and throughput metrics.The test sets contain 45 editing videos and 150 inpainting videos, each six seconds long.
- Ablation: Both local and global adapters improve editing quality, with their combination greatly increasing throughput and even exceeding the full-attention model in edit quality.Removing adapter components harms quality, while the complete design remains comparable to full-attention editing.
- Experimental Setup: The comparison includes state-of-the-art generative and non-generative methods, including VACE, which uses full attention.ReVideo additionally requires the first frame to be edited through an image-inpainting backbone.
- Results: EditCtrl generates videos that better align with prompts and preserve backgrounds while using more than a 50× speedup over generative baselines.It also matches or slightly exceeds the quality and prompt alignment of its full-attention baseline with proportional acceleration.
- Interactive Editing: Independent masked-region generation enables simultaneous multi-region editing with distinct text prompts in a single batch-inference pass.This supports complex edits across multiple non-contiguous masks.
- Interactive Editing: Replacing the frozen base model with an autoregressive video diffusion model enables coherent content propagation across subsequent frames while reducing compute further.Mask propagation can use optical flow or camera pose for augmented-reality editing.
5 Conclusion
EditCtrl improves video editing quality and efficiency by generating only where needed, while retaining several practical limitations tied to the VAE, motion, mask size, and high-resolution processing.
- Conclusion: EditCtrl improves over state-of-the-art methods in quality and efficiency while introducing interactive video editing functionality.The method generatively edits only where needed.
- Limitations: The VAE significantly degrades background context and contributes to artifacts in videos with very fast motion.The local encoder also struggles when spatiotemporal local context shifts rapidly.
- Limitations: VAE encoding and decoding are not an end-to-end throughput bottleneck at 480 × 720 but become limiting for 4K videos because tiling is required under VRAM constraints.Future work could integrate additional temporal information such as motion.
A Additional Visualizations
Additional visualizations extend the reported demonstrations to multi-prompt editing, content propagation, video editing, and video inpainting on multiple datasets.
- Additional Visualizations: Additional video editing and inpainting comparisons are provided for VPBench and DAVIS.Figures 12–15 cover editing examples and inpainting comparisons across the two benchmarks.
B Additional Details
Additional details examine autoregressive propagation, ultrawide-video transferability, failure modes, augmented-reality use, and further visual comparisons.
- Content Propagation: A distilled autoregressive base model lets EditCtrl edit videos of any length using sliding-window attention over local frames.This extends generation indefinitely with some quality loss from denoising newly incoming noisy latents.
- Content Propagation: Optical-flow propagation and edge blending help generated content match incoming video motion before the future frame is displayed.Waiting for future frames would provide more accurate local context but would not support the same anticipatory workflow.
- Transferability: The fixed 256×256 global embedder transfers to challenging 21:9 ultrawide timelapse footage and composes with an aesthetic style LoRA.The example indicates that downsampled global context captures spatial and temporal information in extreme aspect ratios.
- Failure Modes: At high motion, EditCtrl can introduce artifacts and incorrect scene interactions, while very large masks can cause excessive content changes.The high-motion artifacts also appear in the full-attention VACE baseline and are attributed to the VAE.
- Augmented Reality: EditCtrl supports real-time augmented-reality editing by propagating content to match user movement and projecting it when the frame is displayed.The figure specifically highlights low latency and motion-aligned propagation.