Source-linked AI summary
MoCha:End-to-End Video Character Replacement without Structural Guidance
Zhengbo Xu, Jie Ma, Ziheng Wang, Zhan Peng, Jun Liang, Jing Li
TL;DR
Video character replacement is constrained by paired-data scarcity and reconstruction-based methods that require per-frame masks and structural guidance. MoCha instead uses a single arbitrary-frame mask, multimodal in-context learning, condition-aware RoPE, RL-based facial post-training, and a three-source data pipeline. Extensive experiments show that it significantly outperforms existing methods across identity preservation, temporal consistency, and facial-expression fidelity.
Problem
Existing video character-replacement methods rely on per-frame masks and structural guidance, while strictly aligned paired videos are difficult to obtain.
Method
MoCha is an end-to-end framework using one arbitrary-frame mask without structural guidance, with condition-aware RoPE, RL-based facial post-training, and three-source paired-data construction.
Results
MoCha significantly outperforms current state-of-the-art methods in identity preservation, temporal consistency, and facial-expression fidelity.
Takeaways & Limitations
MoCha transfers original character motion and facial expression while requiring only a single arbitrary-frame mask.
Abstract
from arXiv · showhide
Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha
1. Introduction
MoCha addresses the limitations of reconstruction-based video character replacement by using only one arbitrary-frame mask and no structural guidance. It combines multimodal in-context learning, condition-aware RoPE, RL-based post-training, and a specialized paired-data pipeline.
- Existing methods require dense per-frame masks and structural guidance, limiting performance in occlusions, unusual poses, and character interactions.Their reconstruction-based paradigm can struggle when source-video information is difficult to preserve.
- MoCha performs end-to-end character replacement using only a single mask from an arbitrary source-video frame, without structural guidance.The framework harnesses video diffusion models’ tracking ability to transfer character motion and expressions.
- MoCha integrates video content, a frame mask, and a reference identity through in-context learning, using condition-aware RoPE to fuse the inputs.The method decouples source motion and facial expressions from the background before transferring them to the new identity.
- MoCha uses an RL-based post-training stage to further enhance facial consistency with the reference identity.The training overview presents in-context learning and post-training as two stages.
- Training data are constructed from UE5-rendered paired videos, expression-driven portrait animations, and augmented existing video-mask pairs.The pipeline is designed to provide high-quality paired data for character replacement.
- Extensive experiments report improvements over current state-of-the-art methods in identity preservation, temporal consistency, and facial-expression fidelity.
2. Related Works
Related work centers on in-context learning for video generation and reconstruction-based video character replacement. Existing replacement methods combine masked-region generation with reference identities and, in some cases, structural or facial guidance.
- Video diffusion models support in-context learning by concatenating conditioning inputs along the temporal dimension for downstream generation tasks.
- A dominant character-replacement paradigm generates a specified character inside a masked target-video region.
- HunyuanCustom concatenates a reference image with the masked target area, while VACE adds depth or skeleton guidance to maintain character movement.
- Wan-Animate adds a face encoder to extract fine-grained detail for maintaining facial expressions.
3. Method
MoCha formulates video character replacement as in-context generation using a source video, a single-frame mask, and reference images. Condition-aware positional encoding and facial-reward post-training support flexible conditioning and improved identity consistency.
- Preliminary: MoCha uses Rectified Flow to train a video diffusion model that predicts latent velocity from noisy latents and iteratively decodes output video from noise.The input video is compressed by a VAE; inference starts from pure noise and follows a decreasing timestep list.
- In-Context Learning: MoCha encodes the target video, source video, single-frame mask, and reference images into visual tokens for conditional generation.The tokenized conditions are concatenated with target tokens along the frame dimension and processed by a DiT backbone.
- In-Context Learning: Condition-aware RoPE assigns shared frame indices to source and target videos, while references and arbitrary mask frames receive specialized indices.The mask index varies with the designated frame number, enabling flexible arbitrary frame mask selection.
- In-Context Learning: This conditioning design supports variable generation length, multiple reference images, and arbitrary frame mask selection.These capabilities follow from the proposed in-context learning scheme and condition-aware RoPE.
- Identity-Enhancing Post-Training: An RL-based post-training stage optimizes facial consistency between generated and ground-truth videos using a facial reward function.Backpropagation is restricted to the final K sampling steps to reduce memory use and accelerate optimization.
4. Dataset
MoCha constructs paired training data from rendered scenes, expression-driven portrait animation, and augmented real-world video-mask pairs. These sources address the need for aligned character replacements while adding facial-expression fidelity and realism.
- Dataset Construction: MoCha requires strictly aligned paired videos sharing motion, facial expressions, and background dynamics, which are difficult to obtain in real-world settings.The data pipeline aggregates three distinct sources to address this limitation.
- Rendered Data: UE5 renders paired character videos by substituting characters while preserving scene parameters, and also provides masks and multi-pose reference images.The pipeline combines virtual scenes, characters, motions, expressions, and procedural camera trajectories.
- Dataset Construction: The training pipeline contains three sources: UE5-rendered data, expression-driven portrait animation data, and augmented data synthesized from video-mask pairs.The three-source design is summarized in Figure 4.
- Expression-Driven Data: Expression-driven paired data uses Flux inpainting to replace foreground characters and LivePortrait to animate both images with the same facial-driven video.This dataset is constructed to enhance facial expression fidelity.
- Augmented Data: Real-world videos from VIVID-10M and VPData augment the synthetic data, with YOLOv12 filtering non-human videos.The augmented source provides existing video-mask pairs and helps address synthetic artifacts and limited realism.
5. Experiments
MoCha is evaluated against state-of-the-art methods on synthetic and real-world benchmarks, with ablations examining data sources, post-training, tracking, complex scenarios, and broader applications. It achieves strong identity preservation, temporal consistency, facial fidelity, and realism while requiring only a single-frame mask.
- Qualitative Results: MoCha preserves reference identity, original motion, environment details, and lighting more effectively than methods with motion, integration, or lighting limitations.Kling and HunyuanCustom preserve reference characteristics but lose original motion; Wan-Animate and VACE preserve clothing but struggle with lighting and shading.
- Quantitative Results: MoCha achieves state-of-the-art performance across reference-based metrics and superior quality and consistency on selected VBench dimensions.The synthetic benchmark uses SSIM, LPIPS, and PSNR; the real-world benchmark evaluates six VBench dimensions.
- Ablation on Real-Human Data: Real-human training data improves facial appearance and expression alignment while mitigating synthetic artifacts and increasing character realism and fidelity.The evaluated real-human data comprises expression-driven data and augmented video-mask data.
- Ablation on Identity-Enhancing Post-Training: RL-based identity-enhancing post-training improves facial identity preservation while maintaining the original generation quality.The strategy uses reward LoRA to address imperfect identity consistency relative to the reference image.
- Tracking and Complex Scenarios: MoCha tracks the selected character across the video using only a single-frame mask, and produces realistic, high-quality results in complex interactions and effects.The reported complex cases include object interaction, multiple-character interaction, and challenging visual effects.
- Application beyond Character Replacement: MoCha also supports non-human subject replacement and can be applied to face swapping and virtual try-on through edited reference characters.These applications are presented as evidence of broader practical utility beyond character replacement.
6. Conclusion
MoCha is an end-to-end video character replacement framework that uses a single arbitrary frame mask instead of dense structural guidance. It combines in-context learning, condition-aware RoPE, RL-based post-training, and paired-data construction, and reports strong performance and broader editing applications.
- 6. Conclusion: MoCha replaces dense guidance with a single arbitrary frame mask by leveraging video diffusion tracking.The framework transfers original character motion and facial expression through efficient in-context learning.
- 6. Conclusion: Condition-aware RoPE and RL-based post-training adapt multimodal inputs and enhance facial identity.The post-training is guided by a differentiable facial function.
- 6. Conclusion: Extensive experiments show that MoCha significantly outperforms existing methods and extends to face swapping and virtual try-on.The conclusion also describes these applications as evidence of wide applicability beyond video character replacement.