Source-linked AI summary
Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling
Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, Hongsheng Li
TL;DR
Existing image-to-video methods struggle with temporal consistency and fine-grained motion control, especially for large motion. Motion-I2V factorizes generation into diffusion-based motion prediction and motion-guided rendering, with controls for sparse trajectories and regions. It reports more consistent large-motion videos, prompt-following improvements, and support for zero-shot video-to-video translation.
Problem
Existing I2V methods have limited temporal attention and struggle with temporal consistency, while text-only prompting provides insufficient fine-grained motion control.
Method
Motion-I2V predicts reference-pixel trajectories with a diffusion-based motion field model, then uses motion-augmented temporal attention to render frames guided by those trajectories.
Results
Motion-I2V outperforms compared methods on prompt-following and generates more consistent videos under larger motions, while also supporting qualitative large-motion generation.
Takeaways & Limitations
Sparse trajectory ControlNet and region masks provide fine-grained motion and region control, while the rendering stage supports zero-shot video-to-video translation.
Takeaways & Limitations
The method tends to generate medium-brightness videos because its noise schedule does not enforce zero signal-to-noise ratio at the final timestep, creating a training-test discrepancy that may limit generalization.
Abstract
from arXiv · showhide
We introduce Motion-I2V, a novel framework for consistent and controllable image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image's pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image's feature to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even at the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region annotations. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V's second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation. Please see our project page at https://xiaoyushi97.github.io/Motion-I2V/.
1. Introduction
Motion-I2V addresses temporal inconsistency and limited controllability in open-domain image-to-video generation by explicitly modeling motion before rendering video frames.
- Motivation: Open-domain I2V must preserve the reference image’s appearance while modeling plausible, complicated spatial-temporal dynamics.Existing approaches often extend text-to-image diffusion models with 1-D temporal attention, whose narrow receptive field makes consistency difficult under large motion.
- Framework: Motion-I2V factorizes image-to-video generation into explicit motion prediction followed by consistent video synthesis.The first stage predicts pixel-wise trajectories from the reference image and textual instruction; the second generates animation using those predicted dynamics.
- Controllability: A ControlNet converts sparse trajectory annotations into plausible dense trajectories, enabling precise user control over object motion.The framework also supports region-specific animation through motion masks and motion brush controls.
- Applications: Motion-I2V’s second stage also supports video-to-video translation by propagating a transformed first frame using source-video trajectories.Users can stylize the first frame with image-to-image tools and retain motion from the source video.
2. Related Work
Prior image-animation and diffusion-based I2V methods provide category-specific or open-domain generation but commonly rely on simultaneous motion modeling and rendering with limited temporal attention.
- Image Animation: Earlier image-animation methods target natural dynamics, faces, or bodies, while some use optical flow and warping-based rendering.These approaches motivate incorporating explicit motion modeling into modern generative models.
- Motion-I2V: Motion-I2V is organized as two stages: explicit motion generation followed by motion-guided I2V.The overview separates motion prediction from frame synthesis rather than learning the complete image-to-video mapping at once.
- Diffusion-based I2V: Recent diffusion-based methods extend text-to-image models with vanilla 1-D temporal attention for open-domain image animation.Motion-I2V enlarges the temporal receptive field through explicit motion prediction.
- Diffusion-based I2V: VideoComposer, I2VGen-XL, and DynamiCrafter typically handle motion modeling and video generation simultaneously, which the paper associates with unrealistic motion and temporal inconsistency.Motion-I2V instead decouples motion modeling from video rendering.
- Motion Modeling: Optical flow estimates displacement fields between consecutive frames, while point tracking computes a query point’s trajectory across video frames.These are established representations and methods for modeling object movement.
3. Method
Motion-I2V factorizes image-to-video generation into explicit motion prediction followed by motion-guided video rendering. Its motion-augmented temporal attention propagates reference features according to predicted trajectories while retaining the diffusion model’s generative priors.
- Motivation: I2V must generate plausible motion while preserving the reference image’s visual appearance, but limited 1-D temporal attention makes temporal consistency difficult.The vanilla video latent diffusion model attends across frames at corresponding spatial locations, restricting its temporal receptive field for complex spatial-temporal dynamics.
- Stage 1: Motion Prediction: Motion-I2V predicts pixel-wise motion fields in stage one, representing each reference pixel’s displacement toward future frames.The predictor is adapted from a pretrained Stable Diffusion model and uses the reference image plus text prompt as conditions.
- Stage 1: Motion Prediction: The motion-field predictor is trained through staged adaptation from an image latent diffusion model to a video latent diffusion model, then full-model fine-tuning.FlowFormer++ and DOT provide optical-flow and multi-frame trajectory estimates as training targets, while flow maps are encoded with an optical-flow VAE.
- Stage 2: Video Rendering: Stage two uses motion-augmented temporal attention to forward-warp reference features with predicted fields and interleave them with synthesized-frame features.Warped features are incorporated into the temporal attention sequence, enlarging its receptive field under motion guidance.
- Stage 2: Video Rendering: The rendering stage preserves a clean reference latent while denoising the remaining video latents, providing direct reference-image conditioning throughout synthesis.The clean reference code is concatenated with noisy future-frame codes at every denoising timestep.
4. Fine-grained Control of Motion-I2V Generation
Motion-I2V provides fine-grained motion control by converting sparse trajectory and region annotations into motion guidance for image-to-video generation. Users can specify trajectories, animated regions, or both, while the framework also supports video-to-video translation.
- 4.1. Sparse Trajectory Guided I2V: Sparse trajectory guidance lets users draw one or more pixel-wise trajectories that are transformed into dense displacement fields for motion-controlled I2V.A first-stage ControlNet interprets sparse inputs and supplies dense motion fields to the second stage.
- 4.1. Sparse Trajectory Guided I2V: Trajectory ControlNet takes sparse trajectories and a binary mask as conditions and outputs dense optical flow maps.
- 4.2. Region-Specific I2V: Region-specific I2V animates only user-specified regions, leaving unmasked regions static.The region control is implemented by setting sparse displacement inputs to zero and using a mask to distinguish specified from non-specified areas.
- 4.2. Region-Specific I2V: Motion-I2V supports combining motion trajectories with motion-brush region masks for more controllable generation.
- 4.3. Zero-Shot Video-to-Video Translation: The framework supports video-to-video translation by transferring source-video displacement fields to a first frame converted into a target artistic style.Dense point tracking predicts the source-video displacement fields used to animate the converted frame.
5. Experiments
Experiments evaluate Motion-I2V on diverse image-to-video content, compare it with open-source methods, and test zero-shot video-to-video translation and architectural choices. The reported results emphasize instruction following, temporal consistency, relatively large motion, and stabilization from explicit motion modeling.
- Video-to-Video Translation: The second stage supports zero-shot video-to-video translation by animating a style-transformed source frame with source-video motions.The first frame is transformed using existing image-to-image tools before animation.
- Experimental Setup: The test set contains 80 Pixabay images spanning human activity, animals, vehicles, natural scenes, and AI-generated images.Prompts describe image content and possible motion, while CLIP text-image logits measure prompt consistency.
- Comparison with Other Methods: Motion-I2V outperforms VideoComposer, I2VGen-XL, and DynamiCrafter on the prompt-following metric.The comparison uses open-sourced state-of-the-art methods.
- Comparison with Other Methods: Motion-I2V generates more consistent videos with larger motions than the compared methods.Table 1 reports best instruction-following ability and temporal consistency alongside relatively large motions.
- Ablation Study: Without stage 1, inference becomes unstable and produces crashed results, while motion fields and adaptive feature injection improve consistency and avoid extreme distortions.The ablation compares removing stage 1 with using predicted motion fields and attention-based injection.
6. Limitations and Conclusions
The paper concludes that explicit motion modeling and motion-guided temporal attention support consistent, controllable image-to-video generation, including large-motion cases. It also identifies medium-brightness outputs as a limitation linked to the noise schedule.
- Limitations: The method tends to generate videos of medium brightness because the noise schedule does not enforce zero signal-to-noise ratio at the final timestep.The authors suggest newer Zero-SNR schedulers may alleviate the training-test discrepancy and generalization limit.
- Conclusions: Qualitative comparisons report that Motion-I2V preserves reference-image identity under large motion, whereas Gen2 produces severe distortion.DynamiCrafter and Pika 1.0 tend to generate much smaller motions.
- Conclusions: The method enlarges temporal receptive fields with motion-guided temporal attention to improve temporal modeling beyond vanilla 1-D attention.The conclusion presents this as the second-stage design for video rendering.
- Conclusions: Training a first-stage ControlNet enables sparse trajectory control and supports region-specific animation with custom motion masks.The region-specific control is named motion brush.