PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent

Xinyao Liao, Xianfang Zeng, Liao Wang, Gang Yu, Guosheng Lin, Chi Zhang

arXiv:2502.03207v2cs.CVcs.GR

TL;DR

Fine-grained text control of I2V motion remains challenging, as existing approaches struggle with precise element-level control or require expert-supplied motion inputs. MotionAgent translates text into object trajectories and camera extrinsics, composes them into unified optical flow, and uses that flow to guide video generation. The authors report improved motion accuracy on their VBench subset and better camera-motion control with comparable video quality.

  • Problem

    Existing I2V models have difficulty achieving fine-grained motion control from text, while some control methods require expert knowledge to supply trajectories or camera extrinsics.

    Source: §1 · pp. 1–2

  • Method

    MotionAgent parses text into object trajectories and camera extrinsics, composes them into unified optical flow, and uses an optical flow adapter to condition an I2V diffusion model.

    Source: §1 · p. 2

  • Results

    MotionAgent achieves the best motion accuracy on its VBench subset and improves camera-motion control with comparable video quality to advanced models.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported results support fine-grained text-only control of object movement and camera motion in I2V generation.

    Source: §1 · p. 2, §5 · p. 8

    The composition assumes the first frame’s camera coordinate serves as the world coordinate.

    Source: §10 · p. 14

Abstract

from arXiv · show

We propose MotionAgent, enabling fine-grained motion control for text-guided image-to-video generation. The key technique is the motion field agent that converts motion information in text prompts into explicit motion fields, providing flexible and precise motion guidance. Specifically, the agent extracts the object movement and camera motion described in the text and converts them into object trajectories and camera extrinsics, respectively. An analytical optical flow composition module integrates these motion representations in 3D space and projects them into a unified optical flow. An optical flow adapter takes the flow to control the base image-to-video diffusion model for generating fine-grained controlled videos. The significant improvement in the Video-Text Camera Motion metrics on VBench indicates that our method achieves precise control over camera motion. We construct a subset of VBench to evaluate the alignment of motion information in the text and the generated video, outperforming other advanced models on motion generation accuracy.

1. Introduction

MotionAgent addresses the difficulty of fine-grained text control in image-to-video generation by translating prompt-described motion into explicit representations. It combines object and camera motion control and reports stronger motion accuracy on its VBench subset.

1. Introduction

  • MotionAgent turns text-described object movement and camera motion into trajectories and camera extrinsics, then composes them into unified optical flow to guide an I2V diffusion model.A feedback loop lets the agent revise its actions based on the generated video.Source: §1 · p. 2
  • MotionAgent achieves the best motion accuracy on its VBench subset and improves camera-motion control while maintaining comparable video quality to advanced models.The authors also report better text-motion alignment and high quality in a user study.Source: §1 · p. 2

2.1. Video Generation via Agent

  • Agent-based video work spans controllable generation, zero-shot generation, and long-video generation, while Motionzero parses prompts and warps intermediate features using motion priors.Other examples include animation-making, multi-agent collaboration, feedback-based planning, and conversational camera navigation.Source: §2.1 · p. 2

2.2. Text-guided I2V Generation

  • Text-guided I2V models use text encoders or jointly learned text and visual tokens, but motion control depends on training data that often lacks motion-rich labels.The cited datasets commonly describe frame content with little motion information.Source: §2.2 · p. 2

2.3. I2V Generation via Motion Control Module

  • Existing motion-control methods use specialized modules or explicit motion representations, but many control one motion type or require users to supply trajectories or camera extrinsics.MotionAgent instead accepts text and uses 3D optical-flow composition to control object movement and camera motion together.Source: §2.3 · pp. 2–3

3. Method

The method converts prompt-described object and camera motion into explicit trajectories and extrinsics, composes them as unified optical flow, and uses that flow to condition an I2V diffusion model.

3.1. Motion Field Agent

  • The motion field agent parses prompts into object movement and camera motion, then plots trajectories and generates camera extrinsics, with optional video-based revision.The pipeline’s four steps are illustrated in Figure 2.Source: §3.1 · p. 3, Figure 2 · p. 4

3.1.1. Video Motion Decomposition

  • Separating prompt motion into object and camera descriptions decouples the motion information for independent control of each type.Source: §3.1.1 · p. 3

3.1.2. Object Trajectory Plotting

  • For object control, the agent identifies objects and plots trajectories with lengths and curvature suited to the described movement, selecting points from an image grid.Detection and segmentation help locate objects and establish trajectory starting points.Source: §3.1.2 · p. 3

3.1.3. Camera Extrinsics Generation

  • The agent generates camera extrinsics from the camera-motion text and input image, constraining translation to (−1, 1) before rescaling it using estimated depth.The image informs the magnitude of camera motion.Source: §3.1.3 · p. 4

3.1.4. Rethinking

  • An optional feedback loop analyzes the generated video and revisits earlier actions to correct misalignment between the prompt and video.Source: §3.1.4 · p. 4

3.2. Controllable I2V Generation

  • The generation model uses trajectories and camera extrinsics as intermediate representations to guide fine-grained object and camera motion in the video.The optical-flow adapter conditions the base I2V diffusion model on the composed motion representation.Source: §3 · p. 3, §3.2 · p. 4

3.2.1. Analytical Optical Flow Composition

  • The composition module lifts image points into 3D, applies object-motion offsets, reprojects them using camera extrinsics, and derives unified flow containing both motion types.The first-frame camera coordinate serves as the world coordinate, and camera translation is scaled according to maximum depth range.Source: §3.2.1 · pp. 4–5

3.2.2. Optical Flow Adapter Tuning

  • The adapter is fine-tuned on unified flow because its original training on real flow may produce degraded videos when given geometrically unrealistic unified flow.Training examples are prepared by estimating real flow and camera extrinsics, isolating object motion, sampling trajectories, and recomputing unified flow.Source: §3.2.2 · p. 5

4. Experiments

MotionAgent is evaluated on general and controllable image-to-video generation, user preference, and component ablations. Results report strong motion control, with ablations examining object identification, flow composition, adapter tuning, and rethinking.

4.1. Implementation Details

  • Experiments use SVD as the frozen base model and tune the optical-flow adapter; controllable-generation evaluation reports Object Movement Q&A, Complex Camera Motion, and Overall Score.Training uses 24 frames sampled at stride 4, 512×512 resolution, and a learning rate of 2×10−5.Source: §4.1 · p. 5

4.2.1. General I2V Generation

  • 81.91% on Video-Text Camera Motion is significantly higher than other methods, while general I2V results rank among the top on most metrics.Compared with SVD, MotionAgent improves consistency, smoothness, and aesthetic quality; its dynamic degree varies with the motion detail in the prompt.Source: §4.2.1 · p. 5, Table 1 · p. 6

4.2.2. Controllable I2V Generation

  • MotionAgent achieves the best results in controllable video generation, including the most accurate object movement control and significantly better complex camera motion scores than other methods.The authors attribute stronger scores than Motion-I2V to more precise motion representation in the generated optical flow.Source: Table 2 · p. 6, §4.2.2 · pp. 6–7

4.2.3. User Study

  • In a user study, participants ranked videos from four methods on visual quality and prompt-motion semantic alignment; across 1,500 ratings from 50 valid responses, our method ranked highest on both dimensions.Source: §4.2.3 · p. 7

4.3.1. Object Identification

  • Grounded-SAM object identification performs significantly better than multi-round dialogue, whose misjudgments lower object-movement scores and dynamic degree.Because object and camera motion are decomposed, object-identification errors do not significantly affect camera motion.Source: §4.3.1 · pp. 7–8, Table 3 · p. 8

4.3.2. Optical Composition and Adapter Fine-tuning

  • Analytical flow composition scores higher on object movement and complex camera motion than directly adding flows, which can create unreasonable regions and obscure object flow.The untuned primitive adapter performs poorly on complex camera motion when given unified object-and-camera flow maps; the full module has the highest dynamic-degree score.Source: §4.3.2 · p. 8, Table 4 · p. 8

4.3.3. Different LLMs and Base I2V Generation model

  • Replacing GPT-4o with QWen2 or LLama3 achieves almost the same performance, and the method is also tested by integrating it with Motion-I2V.The Motion-I2V integration replaces its Flow U-Net and supplies unified optical flow maps to its I2V U-Net.Source: §4.3.3 · p. 8, Table 4 · p. 8

4.3.4. Rethinking

  • Rethinking improves complex camera-motion metrics and refines object trajectories by reviewing generated video and correcting motion paths.The illustrated example amplifies camera translation along the z-axis.Source: §4.3.4 · p. 8, Figure 7 · p. 8

5. Conclusion

MotionAgent targets precise text-only control of image-to-video generation. It converts text-described motion into explicit representations and unified optical flow, then fine-tunes an adapter; experiments report that it significantly outperforms other text-guided I2V models.

5. Conclusion

  • MotionAgent addresses the challenge of precisely controlling image-to-video generation using only text input.The work introduces a motion field agent, analytical optical-flow composition, an optical-flow adapter, and a rethinking mechanism.Source: §5 · p. 8
  • The motion field agent converts text-described motion into object trajectories and camera extrinsics, which are integrated into unified optical-flow maps for controllable generation.A rethinking mechanism lets the agent correct prior actions according to generated results.Source: §5 · p. 8
  • The authors introduce a video-motion benchmark and report that MotionAgent significantly outperforms other text-guided I2V models.Source: §5 · p. 8

7. Object Trajectory Plotting Module

The module plots trajectories through grid selection, then interpolates them for frame-level positions; an ablation reports better evaluation results than direct offset generation in both metrics.

7.1. Details of Trajectory Plotting

  • The agent specifies trajectory points using grid labels and one of nine within-grid locations, with one to four optional intermediate points.The start and end points are also specified by grid label and within-grid location.Source: §7.1 · p. 13
  • After the trajectory is plotted, interpolation supplies a position for each frame, which is used to calculate dense optical flow.The grid divides the image into small square areas, each subdivided into nine subareas.Source: §7.1 · p. 13, Figure 8 · p. 13

7.2. Ablation Study

  • Grid selection outperforms direct offset generation in both trajectory-plotting metrics.The authors attribute this to better image-layout understanding and easier agent use; direct offsets may produce unsuitable trajectory lengths and lower dynamic degree.Source: Table 5 · p. 13, §7.2 · p. 13

8. Rethinking Module

The rethinking step revises earlier motion outputs using generated-video feedback. It improves most VBench metrics, notably Video-Text Camera Motion, while shortening trajectories may slightly reduce Dynamic Degree.

8. Rethinking Module

  • The agent corrects earlier camera-motion errors and refines object trajectories and camera-motion range to reduce artifacts.Figure 9 illustrates the agent correcting a previous error response and confirming a correct action, which facilitates better video results.Source: §8 · pp. 13–14
  • Shortening object trajectories during rethinking may explain the slight decrease in Dynamic Degree.Source: §8 · p. 13
  • Rethinking raises scores across most VBench metrics, with notable improvement in Video-Text Camera Motion.The reported results also describe improved video-quality metrics.Source: §8 · p. 13, Table 6 · p. 14

9. Complex and Ambiguous Prompts.

The evaluation rewrites motion prompts to test complex and ambiguous descriptions. MotionAgent remains competitive, though ambiguous prompts reduce performance on both motion metrics.

9. Complex and Ambiguous Prompts.

  • On complex prompts, object-movement performance remains comparable while camera-motion performance declines slightly relative to simple prompts.Source: §9 · p. 14
  • Ambiguous prompts degrade both object-movement and camera-motion performance relative to simple prompts.Source: §9 · p. 14
  • Despite these changes, MotionAgent outperforms other I2V methods in semantic alignment between motion prompts and generated videos.Source: §9 · p. 14

10. Training Data Preparation

Training preparation estimates camera motion, depth, and real optical flow from videos, then derives object-motion flow and composes unified flow maps for adapter fine-tuning.

10. Training Data Preparation

  • Training videos are separated into foreground and background, and the dynamic foreground is removed before estimating camera extrinsics, depth, and real optical flow.The preparation uses BiRefNet for segmentation, DROID-SLAM for extrinsics, Metric3D for depth, and Unimatch for real optical flow.Source: §10 · p. 14
  • Object-motion flow is estimated by advancing first-frame pixel positions with real optical flow, then reprojecting them into the first-frame coordinates using depth and camera extrinsics.The method assumes the first-frame camera coordinates serve as world coordinates; the reprojected positions contain only object movement.Source: §10 · p. 14
  • Sampled object-motion flow yields sparse trajectories that are composed into unified flow for adapter fine-tuning; maps exceeding an error threshold are replaced by real flow.Source: §10 · p. 14

11. Metrics Details

The section defines metrics for general I2V quality and controllable I2V motion alignment. Table 7 covers complex and ambiguous controllable I2V results.

11. Metrics Details

  • Controllable I2V evaluation reports Object Movement Q&A, Complex Camera Motion, and Overall Score.Object Movement Q&A measures text–object-movement consistency; Complex Camera Motion measures alignment between complex camera movement and text.Source: §11 · pp. 14–15
  • Video-Text Camera Motion measures camera-motion consistency with input text, while Motion Smoothness assesses smoothness and physical plausibility.Other listed metrics assess subject and background consistency, aesthetics, dynamic degree, and overall I2V quality.Source: §11 · p. 14
  • Table 7 presents results for complex and ambiguous controllable I2V generation.Its caption states the evaluation scope but supplies no result values.Source: Table 7 · p. 15

12. Dynamic Degree

MotionAgent has a relatively lower Dynamic Degree on VBench’s original prompt, which the authors attribute to its precise motion control.

12. Dynamic Degree

  • The authors report a relatively lower Dynamic Degree on VBench’s original prompt.They attribute the decrease to precise control: mentioned objects move accurately, while unmentioned objects remain as static as possible.Source: §12 · p. 15
  • The authors explain the lower score by keeping objects not mentioned in the text as static as possible.Objects that are mentioned move accurately.Source: §12 · p. 15
  • Figure 10 compares MotionAgent with DynamiCrafter and illustrates MotionAgent following the prompt’s motion information precisely.Source: §12 · p. 15

13. Qualitative Results

The qualitative results show fine-grained controlled videos, rethinking outcomes, intermediate representations, and comparisons with other models.

13.1. Fine-grained Controllable Video Results

13.2. Rethinking Results

  • In Figure 12, rethinking corrects inaccurate object movement for a complex prompt and improves video quality for an ambiguous prompt.Source: §13.2 · p. 15

13.3. Visualization of Intermediate Representations

  • Figure 12 visualizes intermediate representations generated by the motion field agent, alongside rethinking and optical-flow results.Source: §13.3 · p. 15, Figure 12 · p. 15

13.4. Qualitative Comparison Results

  • Figure 13 presents more results in comparisons with DynamiCrafter, CogVideoX [68], and Pyramid Flow [27].Source: §13.4 · p. 15

14. User Study Interface

The user study presents participants with shuffled videos to rank, while the accompanying prompts and figures document the study interface and examples.

15. Prompts

  • Tables 8–10 provide prompt templates for object-trajectory plotting, camera-extrinsics generation, and rethinking.The supplied table titles identify the object-trajectory, camera-extrinsics, and rethinking templates.Source: §15 · p. 16, Table 8 · p. 19, Table 9 · p. 20
  • Figure 13 presents additional benchmark comparisons, with motion described in the text shown in bold.Source: Figure 13 · p. 17

14. User Study Interface

  • Participants evaluate 30 video groups and answer two sub-questions per group; four methods’ videos are shuffled for ranking.Rankings are collected twice for specified requirements, then averaged across methods and evaluation dimensions.Source: §14 · p. 16, Figure 14 · p. 18

Related papers

Loading 2502.03207v2…