Source-linked AI summary
NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing
Tianlin Pan, Jiayi Dai, Chenpu Yuan, Zhengyao Lv, Binxin Yang, Hubery Yin, Chen Li, Jing Lyu, Caifeng Shan, Chenyang Si
TL;DR
Local video editing is limited by scarce paired data and architectures that struggle to preserve source dynamics. NOVA decouples semantic control from dense synthesis, and experiments report superior performance across editing-specific and general quality metrics without per-video fine-tuning.
Problem
Local video editing faces scarce high-quality paired data, while first-frame-guided architectures can compound discrepancies into structural drift and temporal incoherence.
Method
NOVA uses sparse edited keyframes for semantic guidance and dense original-video information for motion and texture preservation, with unpaired learning components for temporal coherence and motion reconstruction.
Results
NOVA quantitatively surpasses recent approaches across all editing-specific and general quality metrics without per-video fine-tuning.
Takeaways & Limitations
Sparse Control, Dense Synthesis offers a paradigm for balancing user-guided semantic changes with preservation of original video dynamics.
Takeaways & Limitations
First-frame-guided architectures remain fragile because small discrepancies between an initial edit and source dynamics can compound over time.
Abstract
from arXiv · showhide
Recent video editing models have achieved impressive results, but most still require large-scale paired datasets. Collecting such naturally aligned pairs at scale remains highly challenging and constitutes a critical bottleneck, especially for local video editing data. Existing workarounds transfer image editing to video through global motion control for pair-free video editing, but such designs struggle with background and temporal consistency. In this paper, we propose NOVA: Sparse Control \& Dense Synthesis, a new framework for unpaired video editing. Specifically, the sparse branch provides semantic guidance through user-edited keyframes distributed across the video, and the dense branch continuously incorporates motion and texture information from the original video to maintain high fidelity and coherence. Moreover, we introduce a degradation-simulation training strategy that enables the model to learn motion reconstruction and temporal consistency by training on artificially degraded videos, thus eliminating the need for paired data. Our extensive experiments demonstrate that NOVA outperforms existing approaches in edit fidelity, motion preservation, and temporal coherence.
1. Introduction
Video editing progress is constrained by scarce paired data, especially for local edits. NOVA addresses this with sparse semantic control, dense source-video synthesis, and degradation-simulation training for unpaired learning.
- Motivation: Local video editing is hindered by scarce high-quality paired data and the difficulty of creating realistic before-and-after examples.Synthetic local-edit datasets can contain artifacts and inconsistencies that limit model generalization.
- NOVA: NOVA uses edited keyframes as temporal anchors, providing stronger semantic constraints than a single edited frame.The sparse branch encodes multiple keyframes to guide where and how edits are applied.
- NOVA: The Dense Branch encodes the unedited source video and injects motion cues and background details through multi-level cross-attention.This reduces hallucination and texture drift in non-keyframe regions.
- Training and inference: NOVA trains on artificially degraded videos to learn temporal restoration, texture propagation, and frame-to-frame consistency without paired data.At inference, keyframes are edited sequentially with reference to the first edited frame to reduce flicker and discontinuities.
- Results: NOVA quantitatively surpasses recent approaches across editing-specific and general quality metrics without per-video fine-tuning.Ablations identify dense synthesis as important for background preservation and consistency-aware inference as essential for temporal coherence.
2. Related Work
Prior video-editing methods use diffusion, synthetic data, frame guidance, or per-video adaptation, but remain limited by local-edit artifacts, paired-data demands, and computational costs. NOVA’s training and inference pipelines combine sparse keyframe control with dense source-video information.
- Diffusion-based editing: Diffusion-based video editing methods often perform well globally but struggle with artifacts and inconsistent edits in targeted local regions.Local editing requires spatially targeted modifications that are difficult to represent in synthetic paired data.
- Data and adaptation limits: Existing schemes commonly require either costly per-video fine-tuning or large-scale paired video data that is difficult to acquire.NOVA instead decouples control and synthesis to learn from unpaired data while preserving source fidelity.
- Frame-guided editing: Naive multi-keyframe approaches can produce inconsistent background textures and implausible motion in non-keyframes without access to the original video.The failure reflects hallucinated content between edited anchors.
- Frame-guided editing: Frame-guided methods propagate edits from an initial frame, while other approaches add motion cues or per-video motion adapters to improve consistency.These methods can require significant computational resources and may struggle with addition and removal tasks.
- NOVA pipelines: NOVA’s training pipeline combines interpolated keyframe references for sparse control with cut-and-paste synthetic edits as dense synthesis targets.The model’s sparse and dense branches interact through cross-attention.
- NOVA pipelines: At inference, sequentially edited keyframes are interpolated into a degraded reference for the sparse branch, while the original video enters the dense branch.This preserves access to original motion and texture information during generation.
3. Methodology
NOVA decouples semantic control from source-video synthesis through edited keyframes and dense motion and texture cues. Its unpaired training simulates degradation and synthetic source inputs to learn coherence, restoration, and fidelity.
- Existing first-frame-guided methods entangle appearance generation with motion preservation, allowing discrepancies to compound into structural drift and temporal incoherence.
- NOVA uses sparse edited keyframes for semantic guidance while dense synthesis preserves motion and texture from the original video.The architecture includes a main denoising branch, a sparse control branch, and a dense synthesis branch.
- Cross-attention lets the main branch query dense-branch keys and values, injecting fine-grained motion cues and background details without directly fusing dense features.The dense branch supplies motion and texture information, while the sparse branch provides the reference condition from edited keyframes.
- NOVA creates degraded keyframe references using localized stochastic corruptions and linear interpolation between adjacent keyframes to simulate motion and appearance inconsistencies.The degradation operator includes Gaussian blurring and random affine transformations.
- A Source Fidelity Pipeline generates pseudo source videos by pasting sampled content through moving masks, supplying dense motion and texture references without paired data.The masks undergo random translation, rotation, and scaling transformations.
- The inference pipeline interpolates edited keyframes into a reference video while the original unedited video feeds the dense branch, preserving source dynamics during editing.The interpolated reference conditions the sparse branch, and the original video provides motion and texture information to the dense branch.
4. Experiments
NOVA is evaluated against recent frame-guided video editing methods using multiple quality, fidelity, and consistency measures, alongside ablations of its key components and robustness.
- Evaluation metrics: Background SSIM, Motion Smoothness, and Background Consistency complement CLIP-based evaluation for editing and preservation quality.Background SSIM targets unedited regions identified with SAM2 masks.
- Dense Branch ablation: Removing the Dense Branch introduces hallucinated background details, whereas retaining source information improves background reconstruction and per-frame CLIP similarity.The dense branch can recover sharper details even from degraded inputs, indicating guided synthesis beyond direct texture copying.
- Quantitative comparison: NOVA is compared with recent frame-guided methods on multiple metrics without per-video fine-tuning.The comparison includes AnyV2V, I2VEdit, LoRA-Edit, VACE, and Senorita-2M.
- Consistency-aware inference: Referencing the first edited frame during keyframe editing substantially improves styling consistency over independent keyframe edits.The independent baseline produces noticeable inconsistencies in the added window.
- Model sensitivity: Replacing FLUX.1 Kontext with Qwen-Image-Edit leaves the overall pipeline applicable, supporting generality across keyframe editing models.The ablation tests sensitivity to the choice of image editing model.
- Keyframe interval robustness: The pipeline remains robust when inference keyframe intervals vary from the training interval of 10 to 8, 16, and 20 frames.These results indicate flexibility across different levels of guidance sparsity.
5. Conclusion
NOVA presents Sparse Control, Dense Synthesis as a dual-branch paradigm for balancing user-guided edits with preservation of original video dynamics. Its main stated limitation is dependence on the quality of edited keyframe anchors.
- Conclusion: NOVA combines multi-keyframe sparse guidance with dense source-detail injection to reduce motion and texture hallucination in non-edited regions.The framework is positioned as a direction for high-quality and efficient video editing.
- Limitations: Performance depends on edited keyframe quality, and producing high-fidelity anchors in one pass may require user iteration with current image editing models.The context-aware pipeline improves temporal consistency but does not remove this anchor-quality constraint.
A. Details of Experiments on Naive Multi-Keyframe Guidance
Preliminary VACE experiments show that naive sparse multi-keyframe references cause severe temporal instability, while separating appearance and motion guidance yields more stable outputs.
- Setup: VACE uses a reference video for semantic or structural guidance and a mask video to mark preserved versus regenerated regions.Reference inputs may include edited RGB frames, depth maps, or pose skeletons.
- Naive guidance: A naive configuration placing edited keyframes among gray filler frames causes severe flickering and inconsistent motion in VACE.The instability appears in both the 1.3B and 14B variants and is attributed to sparse discontinuous references absent from training.
- Refined guidance: The refined configuration keeps edited RGB content only at the first keyframe, uses depth maps at later keyframes, and marks only the first frame for preservation.This separates appearance anchoring from structural and motion guidance.
- Outcome: Decoupling appearance from motion guidance produces visibly more stable VACE outputs with reduced flickering.The first edited frame supplies target appearance while later depth cues help preserve original motion dynamics.
B. Details of Training Pipelines
NOVA generates unpaired training signals through anchored degraded references and synthetic source videos, using controlled masks, motion, and appearance transformations.
- Source Fidelity Pipeline: The Source Fidelity Pipeline is a pseudo-source generation procedure for dense-branch training.It constructs a pseudo-source video from a filler video and a moving synthetic mask.
- Source Fidelity Pipeline: The procedure updates mask position and rotation over time, handles boundary collisions, extracts patches, and composites them into the target video.The resulting video combines masked filler-video content with unmasked target-video content.
- Anchored Control Pipeline: The Anchored Control Pipeline always includes the start and end frames, samples additional keyframes, and applies stochastic geometric and appearance degradations.The degradations include zoom-stretch transforms and localized blur.
- Training signals: Together, the pipelines create degraded edited references and spatially misaligned pseudo-source videos for sparse and dense supervision without paired data.Interpolation between keyframes introduces motion degradation and simulated appearance inconsistencies.