PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionPro: A Precise Motion Controller for Image-to-Video Generation

Zhongwei Zhang, Fuchen Long, Zhaofan Qiu, Yingwei Pan, Wu Liu, Ting Yao, Tao Mei

arXiv:2505.20287v1cs.CVcs.MM

TL;DR

Interactive I2V motion control remains challenging because Gaussian-extended trajectories can be coarse and trajectory-only conditions can confuse object and camera motion. MotionPro combines region-wise trajectories and motion masks to guide denoising, and experiments report improved video quality and motion-trajectory alignment, alongside a benchmark for evaluation.

  • Problem

    Large-kernel Gaussian trajectory extension can make motion control coarse, and trajectory-only conditions may not distinguish object motion from camera motion.

    Source: §1 · p. 2

  • Method

    MotionPro combines trajectories sampled from local flow regions with a motion mask and uses both to guide video denoising through feature modulation.

    Source: §3.2 · p. 4, §5 · p. 8

  • Results

    Experiments on WebVid-10M and MC-Bench demonstrate superiority in video quality and motion-trajectory alignment; on WebVid-10M, MotionPro attains FVD 59.88, outperforming MOFA-Video by 27.82.

    Source: §1 · p. 2, §4.2 · p. 5

  • Takeaways & Limitations

    MC-Bench provides 1.1K user-annotated image-trajectory pairs for evaluating fine-grained and object-level I2V motion control.

    Source: §1 · p. 2

Abstract

from arXiv · show

Animating images with interactive motion control has garnered popularity for image-to-video (I2V) generation. Modern approaches typically rely on large Gaussian kernels to extend motion trajectories as condition without explicitly defining movement region, leading to coarse motion control and failing to disentangle object and camera moving. To alleviate these, we present MotionPro, a precise motion controller that novelly leverages region-wise trajectory and motion mask to regulate fine-grained motion synthesis and identify target motion category (i.e., object or camera moving), respectively. Technically, MotionPro first estimates the flow maps on each training video via a tracking model, and then samples the region-wise trajectories to simulate inference scenario. Instead of extending flow through large Gaussian kernels, our region-wise trajectory approach enables more precise control by directly utilizing trajectories within local regions, thereby effectively characterizing fine-grained movements. A motion mask is simultaneously derived from the predicted flow maps to capture the holistic motion dynamics of the movement regions. To pursue natural motion control, MotionPro further strengthens video denoising by incorporating both region-wise trajectories and motion mask through feature modulation. More remarkably, we meticulously construct a benchmark, i.e., MC-Bench, with 1.1K user-annotated image-trajectory pairs, for the evaluation of both fine-grained and object-level I2V motion control. Extensive experiments conducted on WebVid-10M and MC-Bench demonstrate the effectiveness of MotionPro. Please refer to our project page for more results: https://zhw-zhang.github.io/MotionPro-page/.

1. Introduction

MotionPro addresses coarse motion control and object-versus-camera ambiguity by combining local trajectories with a motion mask. It also introduces MC-Bench for evaluating fine-grained and object-level I2V control.

1. Introduction

  • The target setting uses a reference image, sparse trajectory, and region mask so users can interactively guide natural motion aligned with the supplied trajectory.This is presented as a user-friendly alternative to requiring conditions such as depth, edges, or optical flow.Source: §1 · p. 1
  • Large-kernel Gaussian filtering can blur fine-grained motion details, while trajectory-only control can confuse object motion with camera motion.For example, the same planet trajectory could indicate a moving planet or a downward-moving camera.Source: §1 · pp. 1–2
  • MotionPro samples trajectories from local flow regions and uses a flow-derived motion mask to represent motion areas and identify the target motion category.The two control signals guide denoising through predicted affine parameters that modulate video latent codes.Source: §1 · p. 2
  • MC-Bench provides 1.1K user-annotated image-trajectory pairs for evaluating fine-grained and object-level I2V motion control.The paper reports experiments verifying MotionPro's superiority in video quality and motion-trajectory alignment.Source: §1 · p. 2

2. Related Work

Prior I2V work supplies image conditions for video synthesis, while controllable video methods have explored depth, edges, optical flow, boxes, and trajectories. MotionPro uses region-wise trajectories and motion masks as integrated controls for I2V generation.

2. Related Work

  • I2V systems use a static image as the condition for temporally coherent video synthesis, and MotionPro selects SVD as its base architecture.Prior approaches include injecting image latents or CLIP features into denoising and using cascaded diffusion for high-resolution generation.Source: §2 · p. 2
  • Trajectory-based methods use Gaussian filtering or sparse-to-dense estimation, but trajectory-only control can limit fine-grained motion and fail to distinguish object from camera movement.Motion-I2V and MOFA-Video use two-stage trajectory densification; MOFA-Video also uses a movement-region mask as post-processing.Source: §2 · p. 2
  • The overview depicts MotionPro extracting trajectory and mask controls, encoding them into multi-scale features, and injecting them into SVD's 3D-UNet through feature modulation.It also places LoRA layers in transformer attention modules to improve motion-trajectory alignment.Source: Figure 2 · p. 3
  • MotionPro integrates region-wise trajectories and motion masks as conditional inputs for motion-region-aware video generation, rather than applying the mask only after generation.The paper notes that post-processing flow masks can occasionally coincide with unnatural motion and localized distortions.Source: §2 · p. 3

3. Our Approach

MotionPro adds region-wise trajectories and a motion mask to SVD-based I2V denoising, then injects their encoded features into 3D-UNet for interactive motion control.

3.1. Preliminaries: Stable Video Diffusion

  • SVD predicts clean video latent codes from noised latents, noise level, and reference image; MotionPro adds trajectory and mask conditions to refine denoising.Source: §3.1 · p. 3

3.2. Motion Condition Generation

  • MotionPro intersects visibility masks, masks the flow maps, then samples local-region trajectories with randomized mask ratios to cover varied motion-selection scenarios.The mask ratio is sampled from [rmin, 1.0].Source: §3.2 · p. 4
  • The motion mask thresholds average flow magnitude above 1 and repeats the resulting binary map across frames to represent motion regions.It is intended to specify motion category and support global motion correlation.Source: §3.2 · p. 4

3.3. Motion Control Learning

  • A lightweight encoder embeds concatenated controls at multiple scales; each scale predicts γ and β to modulate normalized 3D-UNet features with a residual skip.Temporal convolutions are zero-initialized so γ and β start at zero, supporting stable optimization.Source: §3.3 · p. 4
  • LoRA layers in all spatial-temporal transformer attention modules add trainable residual weight updates while retaining the pretrained model’s motion prior.The update is expressed as W′ = W + AB^T.Source: §3.3 · p. 5

3.4. Inference Pipeline of MotionPro

  • At inference, users brush a motion region and draw a trajectory; MotionPro pads the trajectory locally and guides denoising with both controls.The user-provided brush mask supplies the motion mask.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate MotionPro on WebVid-10M and MC-Bench, including fine-grained and object-level motion control. The reported results favor MotionPro, while ablations examine local-region size, mask ratio, and feature injection.

4.1. Experimental Settings

  • For evaluation, WebVid-10M uses 1K of its 5K validation videos, with trajectories sampled at a 15% ratio and the first frame as input; MC-Bench provides 1.1K reference images with user-annotated trajectories.Because MC-Bench lacks ground-truth video, FVD and FID are inapplicable; MD-Img and MD-Vid measure alignment between generated motion and input trajectory.Source: §4.1 · p. 5

4.2. Evaluation on Fine-grained Motion Control

  • On WebVid-10M, MotionPro reaches 59.88 FVD, outperforming MOFA-Video by 27.82; on MC-Bench it improves MD-Img and MD-Vid over baselines.Generated-flow magnitude is 8.95 for MotionPro versus 4.95 for MOFA-Video.Source: §4.2 · pp. 5–6, Table 2 · p. 5

4.3. Evaluation on Object-level Motion Control

  • On MC-Bench object-level control, MotionPro achieves 10.48 MD-Img and 8.59 MD-Vid, reducing Mean Distance versus DragAnything by 1.82 and 2.78, respectively.The table reports object-level motion-control results.Source: Table 3 · p. 6, §4.3 · p. 6

4.2. Evaluation on Fine-grained Motion Control

  • MC-Bench results show MotionPro aligns fine-grained generated motion with user trajectories, while examples contrast its natural movements with baseline misinterpretations and local distortions.The comparisons include DragNUWA’s camera-motion misinterpretations and MOFA-Video’s local-part distortion.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3. Evaluation on Object-level Motion Control

  • For object-level control, MotionPro matches input trajectories while maintaining natural dynamics; examples include complicated and counterintuitive trajectory alignments.The examples include a round-trip sun trajectory and a train moving backward.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4. Ablation Study on MotionPro

  • Using k = 8 gives more natural motion and precise alignment; larger regions extend trajectories too broadly, while MD-Vid gains become marginal by k = 16.Frame Consistency varies by less than 0.01 across the tested region sizes.Source: §4.4 · pp. 7–8, Figure 7 · p. 8
  • Setting rmin to 0.95 empirically gives the best motion-trajectory alignment, balancing sparse inference-like training signals against optimization with scarce trajectories.Frame Consistency is insensitive to rmin in both motion-control settings.Source: Figure 6 · p. 7, §4.4 · p. 8
  • Adaptive feature modulation achieves better MD-Vid than feature concatenation or summation, which require strict spatial-temporal alignment for information exchange.Modulation regulates features through estimated scale and bias.Source: §4.4 · p. 8, Figure 8 · p. 8

5. Conclusions

MotionPro combines region-wise trajectories and a motion mask through feature modulation to regulate video denoising for precise motion control. The paper introduces MC-Bench and reports experiments on WebVid-10M and MC-Bench validating the method.

  • MotionPro uses local region-wise trajectories and a flow-derived motion mask to guide video denoising through feature modulation for natural motion synthesis and precise trajectory alignment.Source: §5 · p. 8
  • MC-Bench contains 1.1K user-annotated image-trajectory pairs for evaluating fine-grained and object-level motion control.Source: §5 · p. 8
  • Experiments on WebVid-10M and MC-Bench validate MotionPro’s superiority over state-of-the-art approaches, as reported by the authors.Source: §5 · p. 8

6. Supplementary Material

The supplement documents MC-Bench construction and evaluation protocols, then reports human-evaluation and ablation results, mask robustness, camera-control applications, and runtime.

6.1. Dataset Details of MC-Bench

  • MC-Bench contains 412 reference images and 1.1K user-annotated trajectories, split into 460 fine-grained and 680 object-level image-trajectory pairs.The images span varied visual content; 72 come from DragBench and 340 were added.Source: §6.1 · p. 12

6.2. Baseline Choices and Experimental Details

  • Fine-grained MC-Bench comparisons use DragDiffusion and MOFA-Video, while object-level comparisons use MOFA-Video and DragAnything.DragNUWA is excluded from fine-grained MC-Bench evaluation because it lacks motion-region input; static points outside the mask help DragAnything distinguish object from camera motion.Source: §6.2 · p. 12

6.3. Human Evaluation

  • The human study sampled 200 generated videos across both control settings and asked 32 Amazon MTurk evaluators to choose the best model output for each shared input.Evaluation considered motion quality, temporal coherence, and trajectory alignment.Source: §6.3 · p. 12
  • MotionPro outperforms all baselines in user-preference ratios across three criteria for both fine-grained and object-level motion control.The criteria are motion quality, temporal coherence, and trajectory alignment.Source: §6.3 · p. 12, Table 4 · p. 13

6.4. Ablation on control signals.

  • The two ablated control-signal variants have WebVid-10M FVD scores of 73.7 and 66.2, both worse than MotionPro’s 59.88.The variants replace region-wise trajectories with random trajectories or disable the motion mask.Source: §6.4 · p. 12

6.5. Robustness of motion mask

  • At inference, the motion mask need only indicate a rough dynamic region, not a precisely aligned shape.The reported robustness is attributed to training with estimated masks rather than precise ground-truth masks.Source: §6.5 · pp. 12–13

6.6. Application: Camera Control

  • Without additional training, MotionPro supports simultaneous object-and-camera motion control and camera control from a pose sequence.For pose-driven control, the sequence is converted into sparse trajectories using estimated depth, a lifted point cloud, and reprojection under the new poses.Source: §6.6 · p. 13

6.7. Runtime Comparison

  • For 16-frame videos at 512 × 320 on one NVIDIA H100 GPU, MotionPro runs in 17 seconds, comparable to DragNUWA (27), MOFA-Video (15), and DragAnything (32), and faster than DragDiffusion (320).The comparison reports runtime in seconds.Source: §6.7 · p. 13

Related papers

Loading 2505.20287v1…