PaperlayerView exact revision on arXiv

Source-linked AI summary

Vera: A Layered Diffusion Model for Content-Preserving Video Editing

Hongkai Zheng, Ta-Ying Cheng, Benjamin Klein, Yisong Yue, Zhuoning Yuan

arXiv:2606.23610v1cs.CV

TL;DR

Video editing methods that regenerate every pixel can alter characters and backgrounds that should remain unchanged. Vera generates an edit layer and alpha matte for compositing, and outperforms leading open-source models in content preservation while remaining competitive in edit quality.

  • Problem

    Existing video editing methods often regenerate every pixel, limiting preservation of unchanged characters and background scenes.

  • Method

    Vera generates an edit layer and alpha matte for compositing with the source video, using layered DiTs with joint self-attention and curated layered data.

  • Results

    Vera outperforms leading open-source video editing models in content preservation while remaining competitive in edit quality across quantitative and human evaluations.

  • Takeaways & Limitations

    Layer separation supports faithful preservation and natural composition while enabling editable outputs for downstream iterative refinement.

  • Takeaways & Limitations

    Evaluation covers only object addition and background replacement, while broader editing operations require suitable layered supervision.

Abstract

from arXiv · show

Video diffusion models have enabled remarkable progress in video generation and editing. However, content preservation remains a core challenge: existing methods regenerate every pixel and often alter elements that should remain unchanged, such as characters or background scenes. We introduce Vera, a layered diffusion framework for content-preserving video editing. Instead of regenerating the entire video, Vera generates an edit layer along with an alpha matte for compositing with the source video, separating creative editing from content preservation by design. To encourage coherent composition with the source video, we extend the text-to-video DiT into a Mixture-of-Transformers (MoT) architecture, with separate DiTs for each layer that interact through joint self-attention. To support the training of Vera, we further construct a high-quality layered dataset with accurate alpha mattes, diverse scenes and dynamics, and visual effects. Across our quantitative benchmark and human preference study, Vera outperforms leading open-source video editing models in content preservation while remaining competitive in edit quality, using 486K frames of layered training data.

1 Introduction

Vera addresses the content-preservation problem in video editing by generating an edit layer and alpha matte for compositing with the source video. It combines this layered framework with a high-quality dataset and achieves stronger content preservation than leading open-source editors while remaining competitive in edit quality.

  • Motivation and Challenges: Existing video editing methods regenerate the entire video despite regional constraints, allowing inadvertent changes to unrelated content.These approaches use regional constraints and mask conditioning but remain within an end-to-end diffusion paradigm.
  • Motivation and Challenges: Layered video editing must precisely align the edit layer and alpha matte while matching the source video's camera motion, lighting, spatial layout, and scale.These requirements make natural compositing challenging and leave layered video editing underexplored.
  • Vera Framework: Vera generates a separate RGBA edit layer and alpha matte that composite with the source video, explicitly preserving original content while enabling natural composition.Given a source video and editing instruction, Vera produces an edit layer, an alpha matte, and a composite video.
  • Dataset and Evaluation: Vera's layered training dataset combines synthetic composites, realistic single- and multi-object scenes, interactive visual effects, and a benchmark covering diverse motion dynamics and scene complexity.The dataset is designed to support high-quality layered video editing across varied scenes and dynamics.
  • Dataset and Evaluation: 486K frames of layered training data support results showing Vera outperforms leading open-source video editing models in content preservation while remaining competitive in edit quality.The comparison spans both a quantitative benchmark and a human preference study.

2 Related Work

Diffusion models have advanced high-fidelity image synthesis and video generation, motivating prompt-driven video editing and end-to-end visual-effects generation. Recent work increasingly uses layered representations for more granular control over scene composition and transparent video generation.

  • Diffusion-based generation: Diffusion models transformed generative vision, advancing high-fidelity image synthesis and video generation, including controllable text-to-video systems.The cited work includes proprietary Runway Gen models and Google Veo 3.
  • Video editing: Video-editing research now supports temporally consistent, prompt-driven edits and increasingly targets end-to-end visual-effects generation for structural and stylistic scene changes.These developments also motivate datasets for training and evaluating complex editing frameworks.
  • Vera: Vera addresses video-editing limitations with an end-to-end layered framework that generates both the edited video and its corresponding alpha matte.Its architecture is presented as distinct from standard fine-tuning and VACE-style context-adapter approaches.
  • Layered generation: Layer-wise image methods generate transparent or decomposed layers, including foreground RGBA and background layers, to provide more granular control and disentangled multi-object editing.Examples include LayerDiffuse, LayerFusion, LayerDecomp, LayerEdit, and Qwen-Image-Layered.
  • Layered video generation: Layered video methods extend this direction through per-layer prompting, alpha-channel output, and shifted alpha distributions for improved RGBA video generation.Examples include LayerFlow, Transpixeler, and Wan-Alpha.

3 Method

Vera formulates content-preserving editing as joint generation of an edit layer, alpha matte, and composite video for compositing with the source. Its latent diffusion model uses separate interacting DiTs for these distinct outputs and is trained on a curated layered video dataset.

  • Layered formulation: Vera generates an edit layer V_edit and alpha matte A_edit from the source video and conditioning signal, then composites them with preserved content.The formulation treats preserved content, the edit layer, and alpha matte as separate components of the output video.
  • Layered formulation: The method approximates V_preserved ≈ V_src when preserved content has minimal semi-transparent regions, while allowing generative edit layers with large semi-transparent regions.This setting covers common applications such as background replacement and object addition; edit-layer effects can include shadows, reflections, and glasses.
  • Latent diffusion model: Vera jointly models V_edit, A_edit, and V_composite with latent flow-matching diffusion, encoding videos with the Wan2.1 VAE and representing alpha mattes as grayscale videos.The model predicts the three output latents conditioned on the source-video latent and conditioning signal.
  • Mixture-of-Transformers: Three separate DiTs, one per output, use joint self-attention to model interactions among the edit, alpha, and composite layers while preserving branch specialization.Each DiT retains its own QKV projections and FFN weights, while tokens from all branches are concatenated for self-attention.
  • Layered dataset: 486K frames at 832 × 480 resolution form the layered dataset, comprising roughly 6K four-video samples constructed from open-source videos.Each sample contains an input video, edit layer, alpha matte, and composite target video, covering background change and object addition.

4 Experiments · 4.1 Experimental Setup · 4.2 Comparison with Existing Methods

Vera is evaluated on background-change and object-addition tasks using diverse test videos and comparisons with recent open-source editors. It delivers stronger content preservation than baselines, remains competitive in edit quality, and is preferred by humans for preservation and instruction compliance.

  • 4 Experiments: Vera is evaluated on background change and object addition to assess content preservation, video quality, and instruction compliance.The study includes qualitative and quantitative comparisons, followed by ablations and a human preference study.
  • 4.1 Experimental Setup · Training: 8,000 training steps use batch size 16, 832 × 480 resolution, and 49-frame clips for Vera-1.3B and Vera-14B.The variants total 3.9B and 42B parameters, respectively, and are trained with randomly dropped mask inputs.
  • 4.1 Experimental Setup · Evaluation: 141 test video–prompt pairs span motion, camera movement, object count, and scene complexity across object addition and background change.The evaluation draws from Pexels, DAVIS, and VACEBench; supported models receive coarse SAM 2 object-mask videos for background change.
  • 4.1 Experimental Setup · Baselines: Seven recent open-source video editing models provide baselines across general instruction-based, region-constrained, and mask-conditioned editing categories.The comparison includes Ditto, Lucy-edit, ICVE, ReCo, and VACE among the listed methods.
  • 4.2 Comparison with Existing Methods · Quantitative Results: 7.1 dB is Vera-14B’s object-addition PSNR gain over the strongest baseline, while its background-change gain is 4.5 dB.Vera-1.3B gains 6.3 dB and 3.5 dB on the same tasks, while reducing 1−SSIM and LPIPS by more than half.
  • 4.2 Comparison with Existing Methods · User Study: Vera-1.3B is preferred over all baselines for content preservation and instruction compliance, but loses video-quality preference to VACE-14B and Ditto on background change.The human study uses side-by-side 2AFC trials comparing Vera-1.3B with randomly sampled baselines across three dimensions, using 486K layered-training frames.
  • 4.2 Comparison with Existing Methods · Qualitative Comparisons: Qualitative comparisons show baselines distorting untouched faces and bodies, merging inserted objects with foreground subjects, and bleeding object attributes into surrounding regions.Vera instead preserves original content while maintaining high-quality edits; it also matches matting methods on YouTubeMatte without specialized matting loss or matting-only training.

4.3 Ablation Study

Vera’s ablations attribute its content preservation and composition to the layered representation, composite branch, MoT architecture, tailored branch learning rates, and diverse training data. Together, these choices improve preservation while maintaining competitive composition and instruction-following.

  • 4.3.1 Layered Paradigm: Layered editing improves preservation by 2.8–5.0 dB PSNR over 1.3B V2V baselines while retaining competitive composition and instruction-following.Vera and Vera-no-comp use the same training data and gradient steps as the V2V baselines.
  • 4.3.2 Composite Branch: Removing the composite branch raises raw preservation slightly but reduces object-addition composition CS from 3.46 to 2.85 and IS from 3.97 to 3.54.The composite layer regularizes the edit and alpha branches and promotes cross-layer harmonization.
  • 4.3.3 MoT Architecture: The MoT architecture uses separate DiTs for edit, alpha, and composite layers with joint self-attention, unlike a single DiT concatenating all layer tokens.Table 3 also varies channel versus sequence concatenation for input-video conditioning.
  • 4.3.4 Learning Rates: Increasing the alpha branch learning rate to 10× the 10^-5 base rate notably improves content preservation, while further increasing the composite branch rate affects the default configuration.Table 5 evaluates branch-specific learning-rate multiples for Vera-1.3B.
  • Training Data: Synthetic data provides accurate alpha supervision for fine structures such as hair but lacks diversity in object scale, camera motion, and scene layout.This limitation leaves the model without supervision for matching those properties and weakens generalization.
  • Training Data: Adding realistic single-object and multi-object data improves preservation and composition, with object-addition PSNR increasing from 19.1 to 24.8 dB.The multi-object data targets complex scenes and interactive effects, though it causes some regression on background change.

5 Conclusion and Limitations

Vera introduces jointly generated edit layers, alpha mattes, and composite videos to separate editable content from preserved source content and support iterative refinement. Its limitations include higher inference cost, narrow evaluation coverage, and an approximate preservation inference procedure.

  • Conclusion: Vera jointly produces an edit layer, alpha matte, and composite video, separating what to generate from what to preserve.The resulting editable layers can support iterative refinement in downstream editing workflows.
  • Limitations: Vera-1.3B is roughly 3× slower than VACE because jointly generating three layers increases inference cost.This comparison is reported in Appendix Table 7.
  • Limitations: Evaluation is limited to object addition and background replacement.Extending Vera to relighting, complex visual effects, and broader editing operations will require layered supervision capturing the corresponding interactions.
  • Limitations: The inference procedure approximates 𝑉preserved with 𝑉src.The supplied passage identifies this approximation as a remaining limitation, but its consequences are truncated.

A Supplementary Material · A.1 Training data · A.1.1 Data pipeline overview

Vera’s layered training data combines VideoMatte240K with real-world Pexels and Mixkit videos through multi-stage pipelines for object addition and background change. These pipelines integrate automated processing with human annotation, while VideoMatte240K skips collection and matting because it already provides precise alpha mattes.

  • A.1.1 Data pipeline overview: The training data comes from VideoMatte240K and real-world videos collected from Pexels and Mixkit.Both source types support construction of layered training data with alpha mattes.
  • A.1.1 Data pipeline overview: Both data pipelines combine segmentation, matting, inpainting, and VLM-based captioning with human annotation and quality filtering.Annotators identify suitable objects, provide SAM2 point prompts, and filter low-quality results.
  • A.1.1 Data pipeline overview: VideoMatte240K skips video collection/filtering and matting because it already supplies high-quality alpha mattes with precise boundaries.Its pipeline proceeds directly from object removal onward.
  • A.1.1 Data pipeline overview: Figure 5 presents the complete processing chains for object addition and background change, while Table 10 enumerates the data sources.The overview distinguishes the two pipelines and the special handling of VideoMatte240K.
  • A.1.1 Data pipeline overview: The object-addition pipeline has six stages beginning with collection and filtering of high-quality, diverse Pexels and Mixkit videos.The videos cover varied scenes and objects and are preprocessed for sufficient resolution and visual diversity.
  • A.1.1 Data pipeline overview: In the object-addition pipeline, annotators select suitable objects and prompt SAM2 to produce segmentation masks for matting.Human-provided point prompts initiate object segmentation before subsequent processing stages.
  • A.1.1 Data pipeline overview: The background-change pipeline shares video collection and matting with object addition and often reuses the resulting alpha mattes.It diverges after these shared stages to process background edits.
  • A.1.1 Data pipeline overview: Background change removes identified objects with Casper-1.3B while preserving their visual effects, such as shadows and reflections.This treatment differs from object addition because the object’s visual effects remain relevant to the edited background.

A.2 Model details

Vera extends Wan2.1 T2V DiT into a three-DiT Mixture-of-Transformers architecture whose layers interact through joint self-attention. This design increases computation and generation time, while training uses layer-specific optimization and mask dropout.

  • Architecture: Vera uses separate DiTs for the edit layer, alpha matte, and composite video, coupled through joint self-attention.The transformer depth is inherited from Wan2.1: 30 layers for the 1.3B variant and 40 for the 14B variant.
  • Architecture: Because each layer has independent parameters, Vera’s total parameter count is roughly three times the base T2V model.Per-step FLOPs increase by more than 3× because joint self-attention processes the combined token sequence, whose attention cost grows quadratically with length.
  • Efficiency: Vera-1.3B generates a video in roughly 8.3 min on one A100 with 21.8 GB peak VRAM, about 3× slower than VACE.The overhead comes from joint self-attention over all three layers, and standard techniques such as kernel fusion and sequence parallelism can reduce it.
  • Training details: Training uses Adam, a 200-step linear warmup, equally weighted losses across the three output layers, and randomly dropped mask conditioning.Mask dropout rates are 0.3 for background change and 0.9 for object addition, with individual base learning rates selected by the per-layer ablation.

A.3 Alpha matte quality

Vera predicts competitive alpha mattes on YouTubeMatte despite using no specialized matting loss or matting-only training. Vera-14B performs on par with MatAnyone, exceeds raw SAM2 masks, and trails only VideoMaMa.

  • A.3 Alpha matte quality: Vera-14B’s predicted alpha mattes are competitive with dedicated video matting methods on YouTubeMatte despite no specialized matting loss or matting-only training.The evaluation uses YouTubeMatte resized to 832 × 480 and compares Vera with MatAnyone, raw SAM2 masks, and VideoMaMa.
  • A.3 Alpha matte quality: Vera-14B is on par with MatAnyone, performs well above the raw SAM2 mask, and trails only VideoMaMa.

A.4 User study

The human preference study used randomized 2AFC trials comparing anonymized edited videos, with difference heatmaps supporting content-preservation judgments. Annotators evaluated content preservation, video quality, and instruction compliance across up to 608 trials.

  • Study protocol: Each 2AFC trial displayed the source video, editing instruction, and two anonymized edited videos side by side.Vera and the baseline were randomly assigned to the left or right position to avoid positional bias.
  • Study protocol: Difference heatmaps accompanied both edited videos, highlighting pixel-level deviations from the source for content-preservation judgments.Annotators selected preferences for content preservation, video quality, and instruction compliance.
  • Study protocol: 19 annotators each received 32 pairwise trials, yielding a maximum of 608 trials.The study excluded responses from annotators who re…
Loading 2606.23610v1…