PaperlayerView exact revision on arXiv

Source-linked AI summary

MotiMotion: Motion-Controlled Video Generation with Visual Reasoning

Lee Hsin-Ying, Hanwen Jiang, Yiqun Mei, Jing Shi, Ming-Hsuan Yang, Zhixin Shu

arXiv:2605.22818v1cs.CV

TL;DR

Motion-controlled video models often rigidly follow sparse, imprecise, and causally incomplete trajectories, missing plausible secondary effects. MotiMotion uses training-free VLM reasoning followed by confidence-aware generation, and reports more plausible interactions and higher preference on MotiBench. Its motion plans still do not guarantee pixel-level precision, and generated video may carry broader generative-model misuse risks.

  • Problem

    Existing motion-control methods rely on user trajectories that can be sparse, imprecise, and causally incomplete, making physical and logical consequences difficult to specify.

  • Method

    MotiMotion uses a training-free VLM to refine primary trajectories and propose secondary motions, then modulates generation guidance according to trajectory confidence.

  • Results

    MotiBench VLM evaluations and human studies report higher physical realism and logical plausibility, with greater user preference than existing motion-control methods.

  • Takeaways & Limitations

    Reasoning over visual context and adaptively trusting motion guidance supports more plausible object behavior and interaction in motion-controlled video.

  • Takeaways & Limitations

    VLM-predicted trajectories can suffer spatial jitter and inaccuracies, so the reasoning framework does not guarantee pixel-level precision.

Abstract

from arXiv · show

Current motion-controlled image-to-video generation models rigidly follow user-provided trajectories that are often sparse, imprecise, and causally incomplete. Such reliance often yields unnatural or implausible outcomes, especially by missing secondary causal consequences. To address this, we introduce MotiMotion, a novel framework that reformulates motion control as a reasoning-then-generation problem. To encourage causally grounded and commonsense-consistent interactions, we leverage a training-free vision-language reasoner to refine image-space coordinates of primary trajectories and to hallucinate plausible secondary motions. To further improve motion naturalness, we propose a confidence-aware control scheme that modulates guidance strength, enabling the model to closely follow high-confidence plans while correcting artifacts under low-confidence inputs with its internal generative priors. To support systematic evaluation, we curate a new image-to-video benchmark, MotiBench, consisting of interaction-centric scenes where new events are triggered by motion. Both VLM-based evaluation and a human study on MotiBench demonstrate that MotiMotion produces videos with more plausible object behaviors and interaction, and is preferred over existing approaches.

1. Introduction

Existing motion-controlled video models depend on sparse, literal trajectories that omit physical and causal consequences. MotiMotion instead reasons over user intent and visual context before generation, and evaluation on MotiBench reports stronger physical and logical plausibility.

  • Motivation: Users struggle to specify physically valid trajectories for complex, interactive, or causal motions.Examples include hinged-door kinematics, gravity-driven acceleration, synchronized objects, and chain reactions.
  • Motivation: Existing motion-control mechanisms treat sparse user trajectories as ground truth rather than simulating realistic motion.This ignores physical and logical gaps left by imperfect inputs.
  • Motivation: Visual context is needed to anticipate implicit consequences such as domino chain reactions or coupled clock-hand motion.Without such reasoning, generated videos can violate physical reality or miss causal effects.
  • MotiMotion: MotiMotion uses a training-free VLM planning stage to translate user input into detailed narrations and physically plausible trajectories before diffusion generation.The resulting controls are intended to align video synthesis with world knowledge, common sense, and physical principles.
  • Evaluation: MotiBench evaluation reports higher physical realism and logical plausibility for MotiMotion, with ablations confirming contributions from prompt reasoning and motion refinement.The benchmark combines automatic VLM-based evaluation and human studies.

2. Related Work

Prior work controls video motion through trajectories, reasoning modules, or physics-aware objectives, but MotiMotion combines modular VLM reasoning with adaptive, editable motion control.

  • Motion-Controlled Video Generation: Existing motion-controlled video methods generally enforce adherence to user trajectories, whereas MotiMotion treats input as high-level intent.Its control strength can adapt to trajectory precision.
  • Generation with Reasoning and Planning: Modular video frameworks separate reasoning from generation, enabling inspection and intervention in intermediate motion plans.MotiMotion follows this modular paradigm rather than an opaque unified model.
  • Physics-Aware Generation: Physics-aware generation has used physics solvers, synthetic data, object properties, preference optimization, geometric feedback, and inference-time alignment.These approaches pursue physical plausibility through explicit physical cues or alignment mechanisms.
  • Physics-Aware Generation: MotiMotion integrates VLMs as physics reasoners while retaining flexible spatial-temporal control and editability.This distinguishes its modular approach from methods that embed physics into training or inference.

3. Method

MotiMotion combines a flow-matching video generator with VLM-based motion reasoning and confidence-aware conditioning. The design expands sparse inputs into causal plans while allowing the generator to rely on learned dynamics when guidance is unreliable.

  • Overview: The method targets the gap between abstract user intent and physically plausible visual dynamics with VLM planning and a flow-matching generator.The two components expand sparse inputs into dense plans and follow them across varying precision.
  • Base Video Generator: The base generator conditions diffusion-transformer video synthesis on dense point trajectories represented as spatiotemporal heatmaps.Motion maps are projected into latent space and injected alongside noisy video and reference-image latents.
  • Base Video Generator: The generator uses a 3D VAE, T5 text encoder, and Diffusion Transformer with image-to-video conditioning through concatenated latents and attention layers.Motion cues are available to the transformer from the initial tokenization layer.
  • Reasoning: The VLM interprets trajectories, an overlaid image, and an optional prompt to produce narrative prompts and refined trajectories.Refinements preserve spatial intent while adjusting timing and adding secondary consequences such as deformations or splashes.
  • Reasoning: Iterative VLM review can watch generated videos and refine prior motion predictions until users or the VLM find the result satisfactory.This supports repeated correction of motion plans.
  • Limitations: VLM-predicted trajectories can contain spatial jitter and inaccuracies, while manual inputs are sparse, discrete, and over-smoothed.The reasoning stage improves semantic and physical logicality but does not guarantee pixel-level precision.
  • Confidence-Aware Control: Confidence-aware control scales motion conditioning so reliable trajectories are followed closely while imprecise inputs permit the generator’s prior to add natural details.Training simulates affine distortion, temporal linearization, and over-smoothing across confidence levels.

4. Experiments

Experiments evaluate MotiMotion on the interaction-focused MotiBench using VLM judgments, human preferences, qualitative comparisons, cross-method verification, and ablations. Results consistently favor reasoning and confidence-aware control for physical realism, semantic consistency, and plausible interactions.

  • MotiBench: MotiBench contains pre-event physical interaction scenes designed to test whether motion triggers plausible larger physical responses.The benchmark includes manually designed trajectories and coarse prompts across diverse trigger types, including support failure, connection removal, and mechanical release.
  • Main Results: MotiMotion achieves the highest VLM-judged scores across physical realism, photorealism, and semantic consistency on MotiBench.Table 1 compares MotiMotion with previous motion-controlled video generation methods.
  • Main Results: MotiMotion is consistently favored in both automatic and human evaluation, with qualitative comparisons showing more physically plausible outcomes and better intended interactions.The 2AFC evaluation covers object property, interaction, and overall preference against two baselines.
  • Evaluation Protocol: MotiMotion is evaluated with an automated VLM judge and a human two-alternative forced-choice study focused on physical fidelity.The VLM protocol assesses object property, interaction, and overall plausibility; the user study compares generated videos side by side.
  • Cross-Method Verification: Reasoning consistently improves other motion-controlled video methods, particularly on physical realism and semantic consistency.Cross-method verification tests whether the reasoning module generalizes beyond MotiMotion’s primary generator.
  • Ablation Study: Both reasoning and confidence-aware control improve physical realism, photorealism, and semantic consistency, while confidence-aware control corrects artifacts from mismatched trajectories.Examples include correcting unnatural domino motion and seesaw distortion caused by imprecise predicted trajectories.
  • Ablation Study: Prompt reasoning substantially improves physical realism and semantic consistency even without user text prompts.This result indicates that the gain comes from reasoning over image- and trajectory-conditioned inputs.
  • Ablation Study: An iterative reasoning-generation loop corrects a clock-motion failure by revising trajectories and adding static points to prevent camera-motion misinterpretation.The correction sequence is demonstrated only in the illustrated example, not in the other experiments.

5. Conclusion

MotiMotion reasons over user intention and visual context before generating video, transforming raw trajectories into natural dynamics and producing plausible motion. The framework is presented as a foundation for intelligent, realistic interaction in digital environments.

  • MotiMotion reasons over user intention and visual context to generate plausible motion.Its core pipeline predicts motion and transforms raw trajectories into natural dynamics.
  • The framework uses a reasoning-then-generation pipeline to connect motion prediction with video synthesis.
  • MotiMotion is highly preferred in user studies and by VLM agents.

Impact Statement

The paper acknowledges societal risks associated with enhanced motion control in generative video. It recommends layered safeguards and public digital literacy to reduce potential misuse.

  • Enhanced motion control could potentially be misused to create deceptive or harmful content.
  • The paper advocates digital watermarking, rigorous content filtering, and restricted model distribution as safeguards.
  • Public digital literacy is identified as a critical defense against synthetic-media misuse.

A.1. Trajectory Degradation

Trajectory Degradation synthetically corrupts training trajectories according to sampled reliability, scaling geometric, structural, temporal, and smoothing distortions with degradation intensity. Higher intensity produces less faithful trajectory inputs.

  • A.1. Trajectory Degradation: Trajectory Degradation samples reliability s ∈[0, 1] and inversely links it to noise level.High scores preserve the original trajectory, while low scores introduce geometric and structural distortions.
  • A.1. Trajectory Degradation: Degradation intensity I linearly scales the magnitude of three sequential corruption operations.
  • A.1. Trajectory Degradation: Affine perturbation simulates camera instability or tracking drift through rotation, scaling, and translation around the trajectory centroid.
  • A.1. Trajectory Degradation: Temporal subsampling reduces trajectories to keypoints and reconstructs them with linear interpolation to simulate low-fidelity or sparse tracking.
  • A.1. Trajectory Degradation: As I increases, trajectories become increasingly piecewise linear and lose high-frequency motion nuances.
  • A.1. Trajectory Degradation: Smoothing applies a Savitzky-Golay filter whose window length increases with intensity, aggressively removing fine-grained details from low-confidence inputs.

A.2. Training and Inference

The appendix specifies optimization settings, inference costs, and MotiBench’s example coverage. MotiBench spans diverse interaction scenarios, including collisions, constraints, flows, mechanisms, and common objects.

  • A.2. Training and Inference: The video generation model is trained with Adam, 100 warm-up steps, a constant learning-rate schedule, and 0.03 weight decay.
  • A.2. Training and Inference: Prompt and motion reasoning takes about 50 seconds and costs 0.07 USD, while prompt-only reasoning takes about 15 seconds and costs 0.02 USD.
  • A.2. Training and Inference: MotiBench includes examples across collision, constraint change, flow, tool mechanism, and common-object scenarios.
  • A.2. Training and Inference: 30% (19 samples) of MotiBench samples involve multi-object interaction.

B.2. Additional Results

Additional results show how MotiMotion refines user trajectories, proposes missing motions, and uses confidence-aware control to reduce errors from imperfect reasoning. One-stage reasoning provides comparable performance to two-stage reasoning with substantially lower computational and time costs.

  • The VLM refines user trajectories and proposes new trajectories, with green, red, and blue lines denoting user input, refined input, and model predictions.
  • The motion-following evaluation reports performance comparable to previous work using FVD and endpoint error across settings with 4 and 16 trajectories.The evaluation follows the Motion Prompting protocol.
  • Observed reasoning failures include incorrect motion prediction, inaccurate grounding, and incomplete trajectories.
  • Confidence-aware control reduces error propagation from inaccurate reasoning outputs.The generator learns to follow high-confidence trajectories more strictly and rely more on its pretrained motion prior when inputs are unreliable.
  • MotiMotion can reason without input trajectories by having the VLM propose all trajectories for motion-triggered image editing scenarios.The examples use MotionEdit and confidence-aware control.
  • One-stage reasoning is comparable in performance to two-stage reasoning while requiring much less computation and time.Two-stage reasoning adds another reasoning and generation pass, increasing latency.

B.3. Evaluation Details

The evaluation combines a 12-participant two-alternative forced-choice study with weighted preference scoring that emphasizes the strength of qualitative improvements. The resulting metric is designed to distinguish substantial physical gains from marginal wins.

  • The user study recruits 12 participants, each answering 30 questions comparing MotiMotion against two baselines.Questions are sampled from paired combinations of the methods.
  • Weighted scoring gives stronger improvements more influence than slight improvements instead of treating all wins as equal.Major collision fixes receive more weight than minor texture sharpening.
  • The weighted preference rate aggregates decisive wins by assigning weights of 1, 2, and 3 to slight, moderate, and strong wins, while ties receive zero.
  • A 65% weighted preference score means Model A captured 65% of total preference points, with strong wins contributing substantially more.The metric is intended to reflect qualitative superiority rather than merely frequent marginal advantages.

C. Discussion

MotiBench targets sparse-trigger interaction scenarios in which models must infer downstream effects from coarse user intent. Its focused design is supported by benchmark evaluation, human study, and cross-backbone comparison, while practical deployment remains limited by latency and external API dependence.

  • MotiBench is a complementary benchmark for pre-event, underdetermined interactions, rather than a general motion-following benchmark or replacement for DAVIS.It evaluates inferred downstream consequences beyond trajectory fidelity.
  • The benchmark contains 62 manually curated pre-event interaction images covering collision, support, constraint changes, flow, tool mechanisms, and multi-object interactions.Each sample provides an imminent event and only coarse user intent, leaving consequences to be inferred.
  • Benchmark evaluation, human study, and cross-backbone comparison support the validity of MotiBench.
  • Reasoning latency and dependence on external APIs may limit practical applicability, especially for real-time control.The paper identifies smaller local VLMs and distillation as promising directions.
Loading 2605.22818v1…