Source-linked AI summary
DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning
Mingshuang Luo, Shuang Liang, Zhengkun Rong, Yuxuan Luo, Tianshu Hu, Ruibing Hou, Hong Chang, Yong Li, Yuan Zhang, Mingyuan Gao
TL;DR
Character image animation must transfer motion while preserving identity, yet existing methods face a motion–identity trade-off and depend heavily on explicit pose priors. DreamActor-M2 reframes conditioning as in-context learning, combines pose-based and self-bootstrapped RGB-driven stages, and reports state-of-the-art visual fidelity and cross-domain generalization on AW Bench, with remaining difficulty in complex interacting rotations.
Problem
Existing character animation methods struggle to jointly achieve identity preservation, motion fidelity, and cross-domain generalization because motion injection creates a trade-off and explicit pose priors limit dynamics and character coverage.
Method
DreamActor-M2 unifies reference appearance and motion as spatiotemporal visual context, then uses self-bootstrapped pseudo-paired data to train end-to-end RGB-driven animation without explicit pose supervision.
Results
DreamActor-M2 achieves state-of-the-art visual fidelity and cross-domain generalization, supported by extensive experiments and the AW Bench benchmark.
Takeaways & Limitations
The framework provides a unified animation paradigm that extends motion transfer to diverse characters and challenging scenarios, including direct transfer from raw videos.
Takeaways & Limitations
DreamActor-M2 occasionally struggles with complex interactions such as two characters rotating around each other, attributed to scarce motion-trajectory-crossing training data.
Abstract
from arXiv · showhide
Character image animation aims to synthesize high-fidelity videos by transferring motion from a driving sequence to a static reference image. Despite recent advancements, existing methods suffer from two fundamental challenges: (1) suboptimal motion injection strategies that lead to a trade-off between identity preservation and motion consistency, manifesting as a "see-saw", and (2) an over-reliance on explicit pose priors (e.g., skeletons), which inadequately capture intricate dynamics and hinder generalization to arbitrary, non-humanoid characters. To address these challenges, we present DreamActor-M2, a universal animation framework that reimagines motion conditioning as an in-context learning problem. Our approach follows a two-stage paradigm. First, we bridge the input modality gap by fusing reference appearance and motion cues into a unified latent space, enabling the model to jointly reason about spatial identity and temporal dynamics by leveraging the generative prior of foundational models. Second, we introduce a self-bootstrapped data synthesis pipeline that curates pseudo cross-identity training pairs, facilitating a seamless transition from pose-dependent control to direct, end-to-end RGB-driven animation. This strategy significantly enhances generalization across diverse characters and motion scenarios. To facilitate comprehensive evaluation, we further introduce AW Bench, a versatile benchmark encompassing a wide spectrum of characters types and motion scenarios. Extensive experiments demonstrate that DreamActor-M2 achieves state-of-the-art performance, delivering superior visual fidelity and robust cross-domain generalization. Project Page: https://grisoon.github.io/DreamActor-M2/
1 Introduction
DreamActor-M2 addresses the identity–motion trade-off and pose-prior limitations in character image animation through in-context motion conditioning and self-bootstrapped training. The framework and AW Bench support reported gains in visual fidelity and cross-domain generalization.
- Character image animation transfers motion from a driving sequence to a static reference image to synthesize high-fidelity videos.
- Existing methods struggle to jointly preserve identity, maintain motion fidelity, and generalize across domains.Their motion injection strategies create a see-saw between identity preservation and motion consistency, while explicit pose reliance limits applicability.
- DreamActor-M2 spatiotemporally concatenates motion controls with the reference image, forming a unified representation interpreted as visual context by a pretrained video backbone.This preserves the original architecture while leveraging foundation-model generative priors.
- The two-stage framework progresses from pose-based conditioning with MLLM motion-semantic guidance to end-to-end RGB-driven animation without explicit pose supervision.A self-bootstrapped pipeline uses the pose-based variant to create pseudo-paired cross-identity training data.
- AW Bench covers diverse character categories and motion types for more rigorous character image animation evaluation.
- Extensive experiments report state-of-the-art visual fidelity and cross-domain generalization.
2 Related Work
Prior work uses latent video diffusion backbones, explicit pose guidance, raw RGB motion representations, and emerging in-context learning strategies. These approaches improve animation in specific settings but retain limitations involving identity leakage, motion fidelity, generalization, computational cost, or dependence on structured motion.
- Latent Video Diffusion Models: Seedance 1.0 is adopted as DreamActor-M2’s pretrained backbone because character image animation aligns with image-to-video generation.
- Pose Guidance in Character Image Animation: Pose-guided methods inject 2D skeleton or SMPL signals, but same-identity training can entangle identity appearance with motion features.
- Pose Guidance in Character Image Animation: Some pose-guided designs compromise fine-grained motion semantics and motion fidelity despite capturing global motion patterns.
- Pose Guidance in Character Image Animation: Raw-RGB approaches avoid explicit motion representations but can limit generalization or add substantial per-video computational overhead.DreamVideo requires one model per identity, while FlexiAct requires a dedicated frequency-aware embedding for each driving video.
- In-Context Learning: In-context learning remains relatively underexplored in video generation, and existing contextual approaches retain task-specific or representation-dependent constraints.TIC-FT does not address character-animation challenges, while SCAIL depends heavily on accurate 3D motion representations.
3 Approach
DreamActor-M2 reframes motion conditioning as spatiotemporal in-context learning, combining reference appearance and driving cues without lossy motion compression. Its two-stage framework progresses from pose-based conditioning to RGB-driven animation through self-bootstrapped pseudo-pairs.
- Motivation: Existing injection methods trade identity preservation against motion consistency: pose alignment can leak shape, while cross-attention can discard fine-grained dynamics.Temporal concatenation also lacks frame-wise spatial correspondence, weakening motion details and reconstruction quality.
- 3.2 Motion Injection via In-Context Learning: Spatiotemporal in-context injection spatially combines the reference with the first motion frame, masks later reference regions, and stacks frames temporally.The resulting composite is projected into latent space with masks and noise before entering the diffusion transformer.
- 3.3 Pose-Based DreamActor-M2: Pose-based DreamActor-M2 reconstructs a video from its first frame and extracted 2D pose sequence under self-supervised training.Random bone scaling and bounding-box normalization perturb structural cues while preserving motion patterns and scale invariance.
- 3.3 Pose-Based DreamActor-M2: Target-oriented MLLM text guidance restores fine-grained motion semantics weakened by pose augmentation while jointly incorporating appearance descriptions.The framework uses lightweight LoRA adaptation while freezing the backbone and excluding the text branch from adaptation.
- 3.4 End-to-End Training Paradigm: End-to-End DreamActor-M2 removes explicit pose-estimator dependence by training on self-bootstrapped pseudo-pairs generated from the pose-based model.The synthesized video preserves source motion while adopting a novel identity, and direct RGB training bypasses intermediate pose representations.
4 AW Bench
AW Bench is introduced to evaluate universal character animation across diverse identities and motion categories beyond existing human-focused evaluation settings. It includes human and non-human driving videos, varied capture conditions, and diverse reference subjects.
- Motivation: Existing evaluation datasets and setups do not adequately support universal animation across human and non-human subjects.The benchmark is designed for driving videos and reference images spanning broader subject types.
- Benchmark Composition: AW Bench covers a wide range of motion types and reference identities using 100 driving videos and 200 reference images.Human motions vary by body region, age group, activity, and camera setting.
- Benchmark Composition: The non-human corpus includes animals and animated characters, while the benchmark also explores multi-subject driving scenarios.These categories extend evaluation beyond conventional human character animation.
5 Experiments
Experiments evaluate DreamActor-M2 on diverse animation scenarios using automatic and human-aligned assessments, qualitative comparisons, product comparisons, and ablations. The results show strong visual fidelity, generalization, and robustness across character and motion domains.
- Quantitative Comparison: DreamActor-M2 improves every evaluation dimension and surpasses existing baselines by a substantial margin in human evaluation.Agreement between automated and human results supports the framework’s generation quality and robustness.
- Product Comparison: +9.66% GSB lead over Kling 2.6, with further leads of +43.66% over Kling-O1, +51.43% over Wan2.2-Animate, and +57.04% over DreamActor-M1.The comparison uses the same testing dataset and reports competitive or leading subjective performance against platform-level products.
- Qualitative Results: Qualitative comparisons show identity preservation, motion alignment, fine-grained motion capture, and accurate motion semantics across heterogeneous scenarios.The method also handles one-to-many driving, where competitors commonly show artifacts or structural collapse.
- Generalization: DreamActor-M2 generalizes to cross-morphology, non-human driving sources, and multi-person scenarios without structural collapse.It synthesizes plausible lower-body motion in Half2Full tasks while maintaining upper-body synchronization and supports Animal2Animal, Cartoon2Cartoon, One2Multi, and Multi2Multi transfers.
- Ablation Study: Spatiotemporal injection outperforms temporal injection, while pose augmentation consistently improves generation quality and preserves intricate structural details.The ablations particularly highlight improved spatial fidelity for details such as hand gestures.
6 Conclusion
DreamActor-M2 unifies reference and motion signals through spatiotemporal in-context learning, then extends this representation to direct end-to-end motion transfer. Experiments on AW Bench support its fidelity and generalization across diverse scenarios.
- 6 Conclusion: DreamActor-M2 integrates motion and reference signals into a unified representation through spatiotemporal in-context learning.The design leverages the pre-trained backbone’s generative priors for character animation.
- 6 Conclusion: The unified representation enables direct end-to-end motion transfer from raw videos without explicit pose estimation.This extends the framework to increasingly diverse and challenging animation scenarios.
- 6 Conclusion: AW Bench experiments demonstrate robust visual fidelity and cross-domain generalization for DreamActor-M2.The benchmark provides evidence for a unified character-animation paradigm across diverse scenarios.
7 Limitations and Future Works
DreamActor-M2 remains challenged by complex multi-person interactions, particularly when characters rotate around each other. The authors attribute this limitation primarily to scarce training data containing motion trajectory crossings.
- 7 Limitations and Future Works: Complex interactions such as two characters rotating around each other remain difficult for DreamActor-M2.The authors attribute this issue primarily to scarce training data featuring motion trajectory crossing and plan more diverse multi-person datasets.
8 Ethics considerations
Human-related images and videos introduce social risks because the framework can animate people and create fake videos. The authors propose restricting access and following responsible-use guidelines to reduce misuse.
- Human image animation can be misused to create fake videos of people, although existing detection tools can identify these fakes.
- The authors call for clear ethical rules and responsible usage guidelines to reduce risks from the technology.
- The authors state that access to the core models and code will be strictly restricted to prevent misuse.
- The images and videos used in the data and experiments come from publicly available sources, with a contact channel for removal concerns.