Source-linked AI summary
SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing
Xinyao Zhang, Wenkai Dong, Yuxin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong Wang
TL;DR
Instruction-guided video editing struggles to combine precise semantic changes with faithful temporal motion, while reliance on explicit external priors limits inherent semantic-motion learning. SAMA factorizes the problem into Semantic Anchoring and Motion Alignment, trains them through factorized pre-training followed by supervised fine-tuning, and achieves strong zero-shot behavior plus state-of-the-art open-source performance competitive with commercial systems.
Problem
Instruction-guided video editing must balance precise semantic modifications with faithful motion preservation, but existing approaches rely heavily on explicit external priors.
Method
SAMA separates semantic anchoring from motion modeling, using semantic-token prediction, motion-centric restoration pre-training, and subsequent supervised fine-tuning.
Results
SAMA achieves state-of-the-art performance among open-source models, competitive results against leading commercial systems, and strong zero-shot editing behavior from factorized pre-training alone.
Takeaways & Limitations
Factorized pre-training can produce strong instruction-guided video editing without paired editing data and supports learning disentangled semantic and motion representations.
Abstract
from arXiv · showhide
Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and datasets will be released.
1 Introduction
SAMA factorizes instruction-guided video editing into semantic structure planning and motion modeling to balance precise edits with temporally coherent motion. It combines Semantic Anchoring, Motion Alignment, and two-stage training, achieving strong performance among open-source models and competitiveness with commercial systems.
- Instruction-guided video editing must apply fine-grained semantic changes while preserving coherent subject, background, and camera motion.Current models can trade semantic fidelity for temporal consistency, producing artifacts, identity drift, or texture popping.
- SAMA factorizes editing into semantic structure planning and motion modeling, reducing reliance on explicit external priors.The framework treats the two capabilities as complementary rather than forcing one undifferentiated process.
- Semantic Anchoring predicts semantic tokens with video latents to support instruction-aware planning while retaining high-fidelity latent rendering.It uses sparse anchor frames because semantic edits are typically sparse and temporally stable.
- Motion Alignment uses motion-centric video restoration tasks so the backbone internalizes coherent temporal dynamics from raw videos.The approach is designed to learn motion without requiring explicit editing supervision.
- SAMA’s factorized pre-training alone induces strong zero-shot editing behavior before supervised fine-tuning on paired video-editing data.Fine-tuning subsequently addresses residual semantic–motion conflicts and improves visual fidelity.
- SAMA achieves state-of-the-art performance among open-source video-editing models and remains competitive with leading commercial systems.The reported comparisons include systems such as Kling-Omni and Runway.
2 Related Work
Prior work spans zero-shot, data-driven, unified, semantic-alignment, and self-supervised approaches to instruction-guided video editing and video representation learning. SAMA applies semantic-alignment regularization and motion-centric pretext tasks to improve instruction following and temporal consistency without heavy test-time optimization.
- Instruction-guided video editing: Early instruction-guided video editors repurposed pretrained text-to-image diffusion models for videos with added temporal modeling.These methods commonly followed zero-shot or one-/few-shot paradigms.
- Instruction-guided video editing: Large instruction-guided video-editing datasets shifted recent research toward end-to-end data-driven models.Examples include Señorita-2M, InsViE-1M, Ditto-1M, ReCo-Data, and OpenVE-3M.
- Unified video editing: Unified and in-context methods represent diverse editing operations within shared video, source, and multimodal condition sequences.UNIC avoids task-specific adapters and DDIM inversion by learning editing behaviors in context.
- Unified video representations: Unified video understanding and generation frameworks provide strong representations for video content and motion dynamics.The cited examples include Omni-Video, InstructX, UniVideo, and VINO.
- Semantic alignment: Semantic-alignment methods transfer encoder or relational knowledge into image and video generation models to improve representation alignment.Related work includes feature alignment, self-distillation, and spatio-temporal token-relation alignment.
- SAMA’s positioning: SAMA applies semantic-alignment regularization to instruction-guided video editing, improving instruction following and temporal consistency without heavy test-time optimization.Its Motion Alignment also adopts lightweight self-supervised restoration objectives motivated by speed learning, spatiotemporal puzzles, and reconstruction.
3 Method
SAMA factorizes instruction-guided video editing into semantic anchoring and motion alignment within a video diffusion framework. It combines sparse-frame semantic planning, motion-centric restoration pre-training, and supervised fine-tuning to improve editing fidelity and temporal stability.
- SAMA uses a video diffusion model to generate an edited target video that follows the instruction while preserving realistic motion and non-edited content.
- Semantic Anchoring: Semantic Anchoring predicts semantic tokens from sparse anchor frames and injects them into the denoising sequence alongside target video latents.The semantic tokens are denoised jointly with target latents and then passed through a prediction head.
- Semantic Anchoring: The overall objective combines flow-matching loss with a Semantic Anchoring semantic-prediction loss.The semantic prediction objective uses an L1 loss between predicted and extracted anchor tokens.
- Motion Alignment: Motion Alignment applies source-video transformations that require recovery of missing content, playback speed, or spatio-temporal ordering.The three perturbations are cube inpainting, speed perturbation, and tube shuffle.
- Training Strategies: SAMA trains in two stages: factorized pre-training without paired video-editing data, followed by supervised fine-tuning on paired video-editing datasets.The first stage uses image-editing pairs and text-to-video data, while the second stage mixes paired video-editing data with a small portion of image-editing data.
- Training Strategies: Stage 0 already provides strong zero-shot video-editing capability, while Stage 1 further improves edit fidelity and benchmark performance with paired supervision.The two-stage design separates semantic anchoring and motion alignment from scarce paired video-editing data.
4 Experiments
SAMA is evaluated against open- and closed-source video-editing systems across three benchmarks, with experiments covering comparisons, qualitative behavior, zero-shot editing, and module ablations.
- Experimental settings: SAMA is compared with closed- and open-source systems on VIE-Bench, OpenVE-Bench, and ReCo-Bench.The evaluation includes multiple commercial and open-source baselines.
- Quantitative comparisons: SAMA consistently outperforms existing open-source models across most metrics and remains competitive with closed-source systems.It achieves the best overall performance on Swap/Change and Remove, with strong results across OpenVE-Bench and ReCo-Bench.
- Qualitative comparisons: SAMA demonstrates stronger instruction adherence and temporal consistency across diverse editing types in qualitative comparisons.Examples include relative-position and attribute constraints, object replacement, temporal alignment after removal, and identity preservation during stylization.
- Zero-shot evaluation: Factorized pre-training alone produces strong zero-shot editing across Replace, Add, Remove, Style, and Hybrid tasks while largely preserving non-edited content.The zero-shot setting uses no video-editing training data and evaluates editing instructions directly at inference.
- Semantic Anchoring ablation: Semantic Anchoring accelerates diffusion-loss reduction, stabilizes training, and improves mean scores across all VIE-Bench tasks.With the same training steps, SA also produces higher-quality edits earlier than the baseline.
- Motion Alignment ablation: Motion Alignment improves temporal consistency under fast motion and alleviates motion blur, while combining SA and MA raises the overall score by 0.783 over the baseline.MA alone improves the overall score by 0.399 over the baseline, and the two components are reported as complementary.
5 Conclusion
SAMA separates semantic anchoring and motion alignment within a DiT, achieving strong benchmark performance and zero-shot editing while retaining future challenges in long-video and fast-motion settings.
- Conclusion: SAMA separates semantic anchoring and motion alignment within a DiT for instruction-guided video editing.Semantic anchoring uses semantic-token prediction at anchor frames, while motion alignment uses motion-centric restoration pre-training on text-to-video data.
- Conclusion: SAMA achieves state-of-the-art performance among open-source methods and competitive results against commercial systems on three benchmarks.The conclusion cites VIE-Bench, OpenVE-Bench, and ReCo-Bench.
- Conclusion: SAMA exhibits strong zero-shot editing behavior, suggesting that robust instruction following can emerge from disentangled semantic and motion representations.The stated future directions are long-video editing, fast-motion scenarios, and stronger semantic tokenization.
A Discussion on Type Embeddings vs. Shifted RoPE
Type embeddings provide a lightweight way to distinguish source, target, and semantic tokens while preserving the pretrained backbone’s positional encoding.
- Type embeddings: Learned type embeddings distinguish source-video latents, target-video latents, and semantic tokens throughout all training stages.They encode token identity without altering the backbone’s positional encoding.
- Type embeddings: Type embeddings introduce a smaller perturbation to the pretrained prior than shifted RoPE and empirically yield faster convergence.The design is presented as explicit yet lightweight.
B VLM-based Data Filtering Details
The appendix describes VLM-based data filtering and evaluation criteria for instruction following, visual quality, content preservation, and motion consistency.
- Data filtering: A VLM judge scores samples from 1–10 across instruction following, visual quality, content preservation, and video motion consistency.The appendix states that Qwen2.5-VL-72B is used with three inference turns and averaged scores.
- Evaluation criteria: Instruction following measures how well the edited video implements the user’s specific instruction.The scale ranges from perfectly following all aspects to not following the instruction.
- Evaluation criteria: Visual quality measures whether the edited video is visually coherent and natural-looking.The rubric ranges from seamless editing to obvious problems.
- Evaluation criteria: Motion consistency measures whether edited videos preserve the original motion of the main object and background.The rubric treats style, object, and appearance changes as acceptable when assessing motion consistency.
- Evaluation criteria: Content preservation measures whether unrelated content from the original video remains unchanged after editing.The rubric ranges from perfect preservation to changes affecting much unrelated content.
C Task-Specific Settings for Text-to-Video Pretext Tasks
The text-to-video setup evaluates Motion Alignment with either no pretext task or three motion-centric pretext tasks: Cube Inpainting, Speed Perturbation, and Tube Shuffle. Each task uses a specified corruption or permutation to train temporal understanding.
- The setup compares text-to-video training with no pretext task against three motion-centric pretext tasks.The tasks are Cube Inpainting, Speed Perturbation, and Tube Shuffle.
- Cube Inpainting masks 30% of each video for reconstruction.
- Speed Perturbation applies 2× temporal acceleration to the video.
- Tube Shuffle divides videos into 2×2×2 spatiotemporal tubes and randomly shuffles them.
D Pretext Prediction Visualization
The pretext-prediction visualizations show the model reconstructing masked regions, correcting temporal perturbations, and restoring coherent structure after tube permutation.
- The model plausibly completes masked spatio-temporal regions in Cube Inpainting.
- The model recovers more natural motion dynamics from temporally perturbed inputs in Speed Perturbation.
- The model restores coherent spatio-temporal structure after tube permutation in Tube Shuffle.
- These qualitative predictions indicate that the pretext objectives encourage the backbone to learn motion-related restoration abilities.
E More qualitative results
Additional qualitative comparisons across VIE-Bench, OpenVE-Bench, and ReCo-Bench emphasize semantic instruction grounding and motion consistency. The method is described as producing consistent and visually appealing results across varied scenarios.
- The left column presents examples emphasizing semantic understanding and instruction grounding.
- The right column presents cases emphasizing improved motion consistency and temporal alignment.
- Figure 9 visualizes predictions for Cube Inpainting, Speed Perturbation, and Tube Shuffle.
- The additional comparisons cover VIE-Bench, OpenVE-Bench, and ReCo-Bench.The corresponding visual results are presented in Figures 10–12.