Source-linked AI summary
MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model
Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, Yinqiang Zheng
TL;DR
Existing image animation methods either focus on specific domains or provide limited motion control for open-domain image-to-video generation. MOFA-Video adds sparse motion controls through adapters on a pretrained video diffusion model. The paper reports advantages over current state-of-the-art methods across various applications, while noting limits under substantial content changes or motion guidance.
Problem
In-domain animation methods are limited to specific categories, while image-to-video diffusion can change image content or produce simple motion, leaving users needing more controllable open-domain animation.
Source: §1 · p. 2Method
MOFA-Video uses domain-aware adapters to turn sparse motion hints into dense motion fields and warped image features for a pretrained video diffusion model.
Source: §1 · p. 3Results
Experiments show advantages over current state-of-the-art methods for various applications of unified, fine-grained motion-field control.
Source: §5 · p. 14Takeaways & Limitations
Adapters for different motion domains can be used together for fine-grained control within a unified image-to-video framework.
Source: §1 · p. 3, §5 · p. 14The method struggles to generate content far from the input image and may produce blurriness or structure loss under large motion guidance.
Source: §4.3 · p. 14
Abstract
from arXiv · showhide
We present MOFA-Video, an advanced controllable image animation method that generates video from the given image using various additional controllable signals (such as human landmarks reference, manual trajectories, and another even provided video) or their combinations. This is different from previous methods which only can work on a specific motion domain or show weak control abilities with diffusion prior. To achieve our goal, we design several domain-aware motion field adapters (\ie, MOFA-Adapters) to control the generated motions in the video generation pipeline. For MOFA-Adapters, we consider the temporal motion consistency of the video and generate the dense motion flow from the given sparse control conditions first, and then, the multi-scale features of the given image are wrapped as a guided feature for stable video diffusion generation. We naively train two motion adapters for the manual trajectories and the human landmarks individually since they both contain sparse information about the control. After training, the MOFA-Adapters in different domains can also work together for more controllable video generation. Project Page: https://myniuuu.github.io/MOFA_Video/
1 Introduction
Earlier animation methods often specialized in a narrow object domain, while diffusion-based image-to-video methods opened up broader imagery but offered limited control. MOFA-Video adds sparse motion guidance through domain-aware adapters on a pretrained video diffusion model.
1 Introduction
- Domain-specific animation methods offer control within limited object categories, but their learned natural-animation priors do not generalize to diverse images.Examples include fluids, human hair, and human bodies or faces.Source: §1 · p. 2
- Diffusion-based image-to-video methods support open-domain animation, but may alter image content or produce only simple text-driven or idle motion.These limitations constrain their use for real-world animation tasks that need more control.Source: §1 · p. 2
- Sparse key-points or key-frames provide a shared control handle for formulating animations across motion domains.The paper uses this framing to motivate controllable animation of in-the-wild images.Source: §1 · pp. 2–3
- MOFA-Video adds domain-aware adapters to pretrained Stable Video Diffusion, converting sparse motion hints into dense fields and warped image features for generation.The approach models frame-wise motion explicitly and is designed for temporal consistency.Source: §1 · p. 3
- Adapters trained for different sparse-control domains can be combined for joint control of human faces, background objects, and camera movement.The video diffusion model remains fixed while adapters provide the additional motion controls.Source: §1 · p. 3
2 Related Work
Prior controllable animation work often targeted specific domains, while image-to-video diffusion broadened image animation but left motion control limited. MOFA-Video proposes a unified framework for motion domains on a pretrained video diffusion model.
2.1 Controllable Image Animation
- Earlier controllable animation methods focused on specific domains such as fluids or human pose and face, whereas MOFA-Video models motion from different domains in one framework.The framework is intended to work both singly and jointly in a pretrained video diffusion model.Source: §2.1 · pp. 3–4
2.2 Image-to-Video Diffusion Models.
- Image-to-video diffusion methods mainly use text motion control; MOFA-Video instead uses domain-specific adapters for motion hints, including human face or body signals.The related-work discussion contrasts this scope with methods focused on natural object motion or with weak spatial correction.Source: §2.2 · p. 4
3 Method
MOFA-Video adds sparse motion controls from multiple domains to a frozen Stable Video Diffusion model. Its adapters turn those controls into dense motion guidance and support individual or combined controls at inference.
3 Method
- The unified framework uses domain-specific MOFA-Adapters to control video generation from sparse motion signals, individually or jointly.The adapters are added to a frozen Stable Video Diffusion model.Source: §3 · pp. 4–5, §3.1 · p. 5
3.1 Generative Motion Field Adapter (MOFA-Adapter)
- The adapter generates dense motion fields from sparse hints, warps multi-scale reference-image features, and fuses them into the video diffusion model’s features.Its components are a reference encoder, sparse-to-dense motion generator, and feature-fusion encoder.Source: §3.1 · pp. 5–6, Figure 3 · p. 5
3.2 Training MOFA-Adapters on Stable Video Diffusion
- Training derives sparse motion vectors by sampling dense optical flow, or maps facial landmark displacement from the first frame into sparse flow.The adapter is optimized to reconstruct video clips from the first frame and sampled sparse motion vectors using the pre-trained diffusion model.Source: §3.2 · pp. 6–7
3.3 Inferences
- Inference supports user-drawn object or camera trajectories, masked motion brushes, portrait animation from facial landmarks, longer videos, and combined adapters without retraining.Longer clips use overlapping 14-frame groups with 7-frame overlap and average predicted noise across groups; facial landmarks can also come from another facial video or manual blendshapes.Source: §3.3 · pp. 7–9, Figure 6 · p. 9, Figure 7 · p. 9
4 Experiment
MOFA-Video is evaluated on trajectory-based control, portrait animation, and longer-video inference, with comparisons to specialized methods and ablations of its design. The reported limitations are difficulty generating content far from the input image and possible artifacts under large motion guidance.
4.1 Comparison with Other State-of-the-Art Methods
- MOFA-Video achieves better performance on all reported metrics than DragNUWA for trajectory-based animation and the best results on all metrics for portrait animation.User-study volunteers also preferred the method across the evaluated aspects.Source: Table 1 · p. 10, Table 2 · p. 10, §4.1 · pp. 11–12
- The trajectory-based adapter also controls camera motion, achieving comparable results to MotionCtrl with fewer visual artifacts and larger motion magnitudes.Fixed optical-flow patterns can directly specify camera motion.Source: Figure 8 · p. 10, §4.1 · p. 11
4.2 Ablation Studies
- The full model achieves the best controllability and synthesis quality among three ablated network designs.Without spatial warping, control is imprecise; without dense-flow feature control, artifacts appear; without tuning, dense flow over-controls generation.Source: Figure 10 · p. 12, §4.2 · p. 13
- Periodic sampling addresses error accumulation and temporal inconsistency in long-video inference by blending neighboring groups’ latent features.Conditioning on the latest generated frame causes later-stage errors, while conditioning on the first frame leaves flicker between groups.Source: §4.2 · pp. 13–14, Figure 11 · p. 13
- Applying the trajectory-trained model directly to portrait animation produces unnatural facial expressions, supporting domain-specific tuning.Portrait results also address unaligned inputs, body motion, paste-back artifacts, and synthesis up to 1024 × 1024.Source: §4.1 · p. 12, §4.2 · p. 14, Figure 12 · p. 14
4.3 Limitations
- The method struggles to generate content far from the input image and may produce blurriness or structure loss under large motion guidance.The authors attribute the content-generation boundary to the video diffusion model’s training on limited video clips.Source: §4.3 · p. 14
5 Conclusion
The conclusion presents MOFA-Video as a pipeline for controllable image-to-video animation across multiple motion domains, using a sparse-to-dense adapter. Experiments are reported to show advantages over current methods across applications.
5 Conclusion
- MOFA-Video supports generic, controllable image-to-video animation from multiple motion domains, including handcrafted trajectories, dense flows, and human key-points.Its sparse-to-dense MOFA-Adapter controls generated motion, with fine-grained control unified as flow motion fields.Source: §5 · p. 14
- The detailed MOFA-Adapter architecture is illustrated in Fig. 13.The supplied figure passage identifies the architecture but provides no further component details.Source: Figure 13 · p. 15
A.1 More Architecture Details of MOFA-Adapter
The MOFA-Adapter combines sparse-to-dense motion generation with reference and fusion encoders, using forward warping in feature space.
A.1 More Architecture Details of MOFA-Adapter
- MOFA-Adapter comprises an S2D network, Reference Encoder, and Fusion Encoder, with forward warping for feature-space spatial warping.The Fusion Encoder has the same architecture as the Stable Video Diffusion encoder.Source: §A.1 · p. 15
A.2 More Training Details
The appendix reports separate training setups for trajectory and portrait adapters, and describes combining adapters through mask-aware feature selection.
A.2 More Training Details
- The trajectory adapter is trained on WebVid-10M for 100,000 iterations; the portrait adapter uses 5,889 portrait videos for 200,000 iterations.Both use AdamW with a learning rate of 2 × 10^-5; batch sizes are 8 for trajectories and 1 for portraits.Source: §A.2 · p. 15
A.3 Inference via Multiple MOFA-Adapters
- For combined controls, a user-designated mask selects landmark-adapter features inside its region and trajectory-adapter features outside it before the frozen SVD generates the output.The signals are routed through their respective adapters separately.Source: §A.3 · pp. 15–16
B More Visual Results
The supplementary section presents additional trajectory, camera-motion, portrait, and multi-adapter animation results, with video examples in the supplementary demo.
B More Visual Results
- The section provides more visual results, with video examples in the supplementary demo.Source: B More Visual Results · p. 16
B.1 Trajectory-based Image Animation
- Additional trajectory-based image animation results are shown in Fig. 14.Source: §B.1 · p. 16
Camera Motion Control
- Basic optical-flow patterns provide camera-motion control alongside handcrafted trajectories, with results shown in Fig. 15.Source: Camera Motion Control · p. 16
B.2 Portrait Image Animation
- More portrait image animation results are shown in Fig. 16.Source: §B.2 · p. 16
B.3 Multi-MOFA Adapters
- Fig. 17 presents additional control results using Multi-MOFA Adapters.Source: §B.3 · p. 16
C Comparison Results
The comparison section reports portrait-quality and lip-synchronization results, discusses trajectory comparisons, and contrasts motion-brush controls with Gen-2.
C.1 Trajectory-based
- Additional trajectory-based comparison results with DragNUWA are shown in Fig. 18.Source: §C.1 · p. 16
Motion Brush
- MOFA-Video combines regional masks with trajectories for advanced nonlinear motions, while Gen-2 supports only basic mask-plus-direction controls.Source: Motion Brush · p. 16
C.2 Portrait Image Animation
- LPIPS is 0.2099 for MOFA-Video versus 0.2308 for SadTalker, while lip synchronization is comparable.Source: §C.2 · p. 17
D Ablation Study Results
The ablations compare network-structure variants, and the supplementary video demo includes ablation studies and method limitations.
D Ablation Study Results
- The full method gives the most balanced results across metrics; removing explicit motion warping produces much worse results on all metrics.Source: D Ablation Study Results · p. 17
- Without MOFA-Adapter tuning, motions are very limited; the no-S2D variant ranks second-best but its LPIPS indicates mismatch with motion guidance.Source: D Ablation Study Results · p. 17
E Video Demo
- The supplementary video demo includes a brief introduction, video results, ablations, and method limitations.Source: E Video Demo · p. 17
F Limitations
MOFA-Video struggles to generate or control content substantially different from the input image and may show artifacts under extensive motion guidance.
F Limitations
- Generating or controlling content substantially different from the input image remains difficult, which the authors attribute to the video diffusion model's limited training clips.Source: F Limitations · p. 17
- Extensive motion guidance may produce blurriness or loss of structure.Source: F Limitations · p. 17