ReVideo: Remake a Video with Motion and Content Control

• Trajectory-guided and box-guided video generation methods do not support motion editing, while motion-transfer methods cannot modify the transferred motion.Source: §1 · p. 2
• Unedited content can dominate sparse trajectory signals, making it difficult to learn motion control and preserve unedited regions while adding customized motion.The paper identifies training imbalance and condition coupling as the causes of this challenge.Source: §1 · p. 2
• ReVideo uses first-frame modification for content editing and trajectory lines for motion control, with a three-stage training strategy and spatiotemporal adaptive fusion module.The method is designed to support editing in multiple areas in parallel.Source: §1 · pp. 2–3
• The paper presents local editing of both content and motion as a first attempt and says the method extends to multi-area editing.Source: §1 · p. 2
• ReVideo performs well across local content changes with motion preserved, motion customization with content preserved, and joint content-and-motion editing.The introduction points to Fig. 1 for examples.Source: §1 · p. 3

2.1 Controllable Image and Video Generation

• Controllable image and video generation uses text, image, and external control modules, including trajectory- and box-based motion guidance.These approaches train extra motion controllers on pretrained video diffusion models.Source: §2.1 · p. 3

2.2 Diffusion-based Video Editing

• Training-free and one-shot diffusion editing can preserve general video structure, but the paper describes these methods as better suited to global edits such as style transfer than precise editing.Source: §2.2 · p. 3
• For precise editing, first-frame modification and selected-area regeneration offer local controls, while text instructions can struggle to locate targets in complex scenarios.The paper's goal is local content and motion modification while keeping unedited content unchanged.Source: §2.2 · p. 3
• Figure 2 depicts two potential structures for injecting motion and content control.Source: Figure 2 · p. 4

3.1 Preliminaries

• Stable Video Diffusion generates a video sequence from a reference image by iteratively denoising latent representations with a conditional 3D UNet.The reference image is injected through CLIP cross-attention and by concatenating its VAE latent with each frame’s latent.Source: §3.1 · p. 4

3.2 Task Formulation and Some Insights

• The task controls edited content through a modified first frame, preserved content through the masked original video, and edited-area motion through trajectory maps.Each motion map has two channels for tracked-point movement in horizontal and vertical directions relative to the previous frame.Source: §3.2 · p. 4
• Unedited content carries easier-to-learn inter-frame motion, so the model can ignore sparse trajectories; toy experiments also find that separate control branches alone do not resolve this coupling.Motion-only prior training improves motion control, but adding unedited-content control weakens its accuracy.Source: §3.2 · p. 5

3.3 Coarse-to-fine Training Strategy

• The three training stages first establish a motion prior, then decouple motion and content using edited and unedited regions from different videos, and finally remove boundary artifacts on normal video data.In the last stage, only temporal self-attention key and value embeddings are fine-tuned to preserve the earlier control prior while removing block artifacts.Source: §3.3 · p. 6

3.4 Spatiotemporal Adaptive Fusion Module

• The spatiotemporal adaptive fusion module predicts Γ from the editing mask and timestep to weight content and motion conditions instead of directly summing them.Γ assigns more motion weight inside the editing region and more content weight in unedited areas, while adjusting across sampling steps.Source: §3.4 · p. 7

4.1 Implementation Details

• Training uses SVD and WebVid, with three stages optimized for 40K, 30K, and 20K iterations, respectively.The setup uses four NVIDIA A100 GPUs, resolution 512×320, and CoTracker-extracted trajectories.Source: §4.1 · p. 7

4.2 Comparison

• Figure 6 compares InsV2V, AnyV2V, Pika, and ReVideo; ReVideo propagates edited content while allowing motion customization.The prose reports distortions or editing challenges for competing methods in some scenarios, including static edited-content motion for AnyV2V and Pika.Source: §4.2 · p. 7, Figure 6 · p. 8
• ReVideo outperforms InsV2V and AnyV2V across all evaluation terms, and Pika in text alignment and human evaluation.It scores slightly below Pika on temporal consistency and unedited-content quality.Source: §4.2 · pp. 8–9

4.3 Ablation Study

• Directly summing content and motion controls is less accurate on complex trajectories such as wavy lines than using SAFM.SAFM helps decouple the controls in the editing area for more accurate trajectory guidance.Source: §4.3 · p. 9
• Removing SAFM's time condition produces editing-area boundary artifacts, while selective temporal-layer key and value tuning preserves motion control during stage-3 deblocking.Tuning spatial layers still triggers motion-control neglect; the selective temporal tuning also harmoniously fuses edited and unedited areas.Source: §4.3 · p. 9

5 Conclusion

• ReVideo targets local video editing of visual content and motion, where directly training both controls can cause motion control to be ignored.The paper addresses this coupling with three-stage coarse-to-fine training and spatiotemporal adaptive fusion.Source: §5 · p. 9
• The method supports content-only, motion-only, and combined editing, and extends to multi-area editing without specific training.The paper describes this as its conclusion from extensive experiments.Source: §5 · p. 10
• Regeneration quality is limited by the base model, and a non-ideal SVD generation prior can produce unexpected editing artifacts.Source: §5 · p. 10

A Appendix

• Figure 8 depicts the trajectory-sampling pipeline used in ReVideo training.Source: Figure 8 · p. 14

A.1 Details of Trajectory Sampling

• A grid sparsifies dense tracking points to produce the initial candidate points for sampling.Source: §A.1 · p. 14
• The mean tracking length sets a threshold for filtering out points with large motion; normalized lengths then define probabilities for sampling 1 to 10 trajectories.Source: §A.1 · p. 14
• Figure 9 is identified as a robustness visualization for irregular editing areas.Source: Figure 9 · p. 14

• Despite training on rectangular areas, ReVideo maintains stable content and motion editing in hand-drawn irregular areas.Source: §A.2 · p. 14

• Each encoder uses three convolution-and-downsampling sub-blocks to map its condition map to the size of latent z_t.The content and motion encoders have the same low-cost structure.Source: §A.3 · p. 14

• Additional examples span object addition, trajectory changes, content edits that preserve motion, and multi-region editing, with content and motion control coexisting harmoniously.Source: §A.4 · p. 15

A.5 Necessity of Fine-tuning the Base Model

• Jointly fine-tuning the base model can alleviate artifacts in complex scenes and highly motion-coupled edited and unedited areas.Without base-model fine-tuning, the examples show content disjunction such as a misaligned tree trunk and artifacts from motion conflict.Source: §A.5 · p. 16

A.6 More Frame Editing

• On a 9-second, 90-frame video, ReVideo broadcasts the first-frame edit while maintaining motion consistency, but error accumulation affects editing quality.A sliding-window strategy uses the previous window’s last edited frame as the current reference; a more powerful base model may alleviate error accumulation.Source: §A.6 · p. 16, Figure 12 · p. 17

A.7 More Discussion of Pika in Adding New Object

• All five Pika attempts to add a plane failed even with text-consistency strength set highest; first-frame editing followed by video-wide broadcasting accurately specifies local content.Source: §A.7 · p. 16, Figure 13 · p. 17