Source-linked AI summary
MagicProp: Diffusion-based Video Editing via Motion-aware Appearance Propagation
Hanshu Yan, Jun Hao Liew, Long Mai, Shanchuan Lin, Jiashi Feng
TL;DR
MagicProp addresses video appearance editing while preserving motion by separating reference-frame editing from motion-aware propagation. Its autoregressive diffusion model generates remaining frames using the reference, target motion, and previous edited appearance, and experiments report flexible, high-quality editing of arbitrary regions.
Problem
Video appearance editing must support changes to content or style in local regions or across a whole video while preserving its motion.
Source: §1 · p. 2Method
MagicProp edits one frame with image-editing methods, then uses PropDPM to autoregressively generate remaining frames conditioned on the appearance reference, target depth, and previous edited frame.
Source: §1 · p. 2, §3.2 · p. 6Results
Experiments demonstrate that MagicProp flexibly edits arbitrary video regions and generates high-quality results across editing scenarios.
Source: §1 · p. 2Takeaways & Limitations
The framework combines flexible reference-frame editing with motion-aware propagation to maintain temporal consistency across frames.
Source: §1 · p. 2Autoregressive error accumulation can cause over-smoothing and artifacts beyond 30 frames, and later frames may lose detail or shift in color.
Source: §3.3 · p. 6, §4 · p. 7
Abstract
from arXiv · showhide
This paper addresses the issue of modifying the visual appearance of videos while preserving their motion. A novel framework, named MagicProp, is proposed, which disentangles the video editing process into two stages: appearance editing and motion-aware appearance propagation. In the first stage, MagicProp selects a single frame from the input video and applies image-editing techniques to modify the content and/or style of the frame. The flexibility of these techniques enables the editing of arbitrary regions within the frame. In the second stage, MagicProp employs the edited frame as an appearance reference and generates the remaining frames using an autoregressive rendering approach. To achieve this, a diffusion-based conditional generation model, called PropDPM, is developed, which synthesizes the target frame by conditioning on the reference appearance, the target motion, and its previous appearance. The autoregressive editing approach ensures temporal consistency in the resulting videos. Overall, MagicProp combines the flexibility of image-editing techniques with the superior temporal consistency of autoregressive modeling, enabling flexible editing of object types and aesthetic styles in arbitrary regions of input videos while maintaining good temporal consistency across frames. Extensive experiments in various video editing scenarios demonstrate the effectiveness of MagicProp.
1 INTRODUCTION
MagicProp addresses video appearance editing while preserving motion, separating reference-frame editing from motion-aware propagation. Its experiments demonstrate flexible editing of arbitrary regions with high-quality results.
- The framework responds to the trade-off between flexible image-based editing and the temporal consistency sought by sequence-based methods.Image models can edit arbitrary regions but struggle with temporal consistency; sequence methods can constrain motion but may be harder to control locally.Source: §1 · p. 2
- MagicProp edits one selected frame, then autoregressively renders remaining frames conditioned on the edited reference and the video's motion sequence.The motion condition can be depth maps from the input video.Source: §1 · p. 2
- Its conditional image-diffusion model uses the previous frame, target depth, and reference appearance to synthesize each target frame.A lightweight adapter injects semantic- and pixel-level reference information, while zero-terminal SNR training supports color and style matching.Source: §1 · p. 2
- Experiments demonstrate that MagicProp can flexibly edit arbitrary regions of a video and generate high-quality results.The experiments cover local object and background editing as well as global stylization.Source: §1 · p. 2
2 RELATED WORKS AND PRELIMINARIES
Prior video-editing methods trade flexible local control against temporal consistency, while diffusion models provide a probabilistic basis for conditional generation. The paper defines video-editing notation using latent frames, appearance instructions, and depth sequences to represent motion.
2.1 RELATED WORKS
- Per-frame image editing is flexible, but appearance can vary across frames; sequential video models improve temporal consistency while limiting local editability and image quality.Cross-frame attention can improve shape and style consistency, while correspondence-based warping may fail under occlusion; sequential approaches may require costly per-video optimization or rely on text prompts.Source: §2.1 · p. 3
- MagicProp separates appearance editing from propagation, combining image-editing methods and localization masks with autoregressive future-frame synthesis.This design addresses the difficulty of specifying a particular local region with text prompts alone.Source: §2.1 · p. 3
2.2 PRELIMINARIES
- DDPMs model data distributions by reversing a Markov chain of learned Gaussian transitions that begins from a standard normal distribution.For conditional generation, the denoising network can take additional control signals; training reduces to noise estimation.Source: §2.2 · p. 4
- Video-editing notation represents a video as RGB frames encoded one by one into a spatially downsampled latent sequence.The framework denotes appearance instructions as text, masks, or visual references, and represents source motion with estimated depth maps.Source: §2.2 · p. 4
3 METHOD
MagicProp separates video appearance editing from motion-aware propagation: it edits one reference frame, then autoregressively synthesizes the remaining frames using motion and appearance conditions. PropDPM combines these conditions to preserve the source layout while carrying the edited appearance through the video.
3 METHOD
- MagicProp divides video editing into appearance editing followed by motion-aware propagation, targeting altered appearance while retaining the source video’s motion.The framework is motivated by the consistency of scenes and objects across frames in short videos.Source: §3 · p. 4
3.1 APPEARANCE EDITING
- A selected source frame is edited with image-editing methods, optionally using ControlNet, a text prompt, and a mask, then serves as the appearance reference.The editing can change object types or visual styles in a local region or across the whole frame.Source: §3.1 · p. 5
3.2 MOTION-AWARE APPEARANCE PROPAGATION
- PropDPM renders each remaining frame autoregressively from the reference appearance, target depth map, and previous edited frame.The generated layout follows the source video’s depth sequence, which the paper says preserves motion and temporal consistency.Source: §3.2 · p. 6
3.3 MODEL DESIGN OF PROPDPM
- Autoregressive error accumulation can degrade later frames, causing smoothing and color shifts; a fixed reference adaptor and zero-terminal-SNR training are proposed to mitigate this.The fixed reference serves as an anchor, while zero-terminal-SNR is described as improving frame quality at each iteration.Source: §3.3 · p. 6
3.3.1 APPEARANCE ADAPTOR
- The appearance adaptor fuses spatially corresponding VAE latents and CLIP class and patch tokens into features supplied to cross-attention.The fused features retain global and local reference information for matching the generated frame’s appearance.Source: §3.3.1 · pp. 6–7
3.3.2 ZERO-TERMINAL-SNR NOISE SCHEDULE
- Zero-Terminal-SNR forces terminal SNR to zero and trains the model to predict v-values rather than noise; the paper reports that this alleviates color shifting.The stated motivation is to bridge the gap between inference starting from pure noise and the strongest noise handled by vanilla DDPM training.Source: §3.3.2 · p. 7
3.3.3 TRAINING
- PropDPM starts from Stable-Diffusion-v1.5 and is trained on sampled WebVid-10M videos plus a self-collected dataset, using three-frame reference, previous-frame, and target triplets.The training data comprises 1 million sampled public videos and 20,000 self-collected high-resolution videos without watermarks.Source: §3.3.3 · p. 7
4 APPLICATION
MagicProp supports global and localized video appearance edits, with masks used to select regions. The paper reports robust editing up to 30 frames, while longer videos may exhibit accumulated autoregressive degradation.
4 APPLICATION
- MagicProp uses masks and ControlNet to edit selected video regions, supporting arbitrary-region edits.Masks may be user-provided or extracted by a segmentation model such as Segment-Anything.Source: §4 · p. 7
- Masks can come from users or a segmentation model, allowing either manual or automated localization of the edited parts.The paper names Segment-Anything as an example segmentation model.Source: §4 · p. 7
- Videos up to 30 frames are reported to edit robustly; beyond 30 frames, over-smoothing and artifacts may arise from autoregressive error accumulation.The authors identify processing longer videos as future work.Source: §4 · p. 7
A SUPPLEMENTARY
The supplementary implementation describes an adaptor that fuses CLIP image embeddings with VAE latents. Its code adjusts their spatial and channel dimensions before combining them into tokens.
A SUPPLEMENTARY
- The appearance adaptor is implemented to fuse class-level and pixel-level information from the edited reference frame.The code describes combining CLIP embeddings with VAE latents.Source: A SUPPLEMENTARY · p. 10
- The code reshapes CLIP patch embeddings into a spatial feature map and downsamples them before fusion.It removes the class token from the patch sequence before reshaping the remaining tokens.Source: A SUPPLEMENTARY · p. 11
- The implementation adjusts VAE latent channels and spatial dimensions, concatenates the result with CLIP patch features, and mixes the combined features.The listed code returns the fused features together with the untouched CLIP class token.Source: A SUPPLEMENTARY · p. 11