PaperlayerView exact revision on arXiv

Source-linked AI summary

MoRight: Motion Control Done Right

Shaowei Liu, Xuanchi Ren, Tianchang Shen, Huan Ling, Saurabh Gupta, Shenlong Wang, Sanja Fidler, Jun Gao

arXiv:2604.07348v1cs.CVcs.AIcs.GRcs.LGcs.RO

TL;DR

Existing motion-controlled video methods entangle camera and object motion and do not model causal relationships between object motions. MoRight disentangles those controls and models active and passive motion; across three benchmarks, it outperforms existing methods in generation quality, motion controllability, and interaction awareness.

  • Problem

    Existing motion-controlled video methods conflate camera and object motion and model trajectories as displacements without representing object-to-object consequences.

    Source: §1 · p. 2

  • Method

    MoRight combines a dual-stream canonical-view motion formulation with active/passive motion decomposition to support viewpoint control and forward and inverse action–consequence reasoning.

    Source: §1 · p. 2

  • Results

    Across three benchmarks, MoRight outperforms existing methods in generation quality, motion controllability, and interaction awareness.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Users can specify active motions for scene evolution or desired passive outcomes for plausible action recovery, while freely adjusting the camera viewpoint.

    Source: §1 · p. 2

    MoRight is designed for common smooth camera trajectories, and drastic camera motion can degrade predicted interaction dynamics.

    Source: §4.7 · p. 15

Abstract

from arXiv · show

Generating motion-controlled videos--where user-specified actions drive physically plausible scene dynamics under freely chosen viewpoints--demands two capabilities: (1) disentangled motion control, allowing users to separately control the object motion and adjust camera viewpoint; and (2) motion causality, ensuring that user-driven actions trigger coherent reactions from other objects rather than merely displacing pixels. Existing methods fall short on both fronts: they entangle camera and object motion into a single tracking signal and treat motion as kinematic displacement without modeling causal relationships between object motion. We introduce MoRight, a unified framework that addresses both limitations through disentangled motion modeling. Object motion is specified in a canonical static-view and transferred to an arbitrary target camera viewpoint via temporal cross-view attention, enabling disentangled camera and object control. We further decompose motion into active (user-driven) and passive (consequence) components, training the model to learn motion causality from data. At inference, users can either supply active motion and MoRight predicts consequences (forward reasoning), or specify desired passive outcomes and MoRight recovers plausible driving actions (inverse reasoning), all while freely adjusting the camera viewpoint. Experiments on three benchmarks demonstrate state-of-the-art performance in generation quality, motion controllability, and interaction awareness.

1. Introduction

MoRight addresses entangled camera–object control and missing action–consequence reasoning with a unified video-generation framework. It lets users control camera and object motion independently and predict or recover actions and their consequences.

Introduction

  • MoRight targets joint camera–object controllability and causally consistent scene dynamics for interactive video generation.The introduction motivates these capabilities for embodied AI, world models, and immersive content creation.Source: §1 · p. 2
  • Existing displacement-based methods [10] [15] [24] [68] [70] [75] entangle camera and object motion and largely ignore causal consequences between objects.They follow pixel trajectories, leaving users to specify effects for all objects even when actions such as pushing a cup cause further motion or collisions.Source: §1 · p. 2
  • A dual-stream formulation specifies object trajectories in a canonical static view and transfers them to target viewpoints through temporal cross-view attention.This allows users to draw trajectories in the image plane while independently adjusting the camera viewpoint.Source: §1 · p. 2
  • During training, MoRight separates user-driven active motion from passive causal outcomes, enabling forward prediction and inverse recovery of plausible actions.Given active motion, it predicts scene evolution; given desired passive outcomes, it recovers actions that could produce them.Source: §1 · p. 2
  • MoRight outperforms existing methods on three benchmarks in generation quality, motion controllability, and interaction awareness.The benchmarks cover diverse interaction scenarios.Source: §1 · p. 2

2. Related Work

Related work identifies limitations in pixel-space motion control, camera–object decoupling, and causal video generation. MoRight addresses them through disentangled motion modeling and learned action–response dynamics.

2.1. Motion-Controlled Video Generation

  • Pixel-space trajectory controls [15] [24] [38] [42] [62] [63] [91] entangle camera and object motion and often require crafted or manually annotated physically plausible paths.MoRight accepts simple strokes or sparse tracklets and completes them into coherent, interaction-aware dynamics.Source: §2.1 · p. 3

2.2. Camera–Object Motion Disentanglement

  • Camera–object decoupling methods [14] [26] [62] [91] typically require depth, 3D trajectories, or segmentation masks and pre-warp signals, limiting single-frame image-to-video use.MoRight uses a canonical static-view branch and cross-view attention to transfer object dynamics to target viewpoints without explicit 3D supervision or precomputed scene decomposition.Source: §2.2 · p. 3

2.3. Interaction and Causal Reasoning in Video Generation

  • Physics-engine methods [11] [44] [50] and force-vector controls [25] target specific interactions, while language-model pipelines [45] [57] [76] [81] risk error propagation through intermediate representations.MoRight learns latent action–response structure from video, separates active and passive motion, and performs pixel-level forward consequence prediction or inverse action inference in one forward pass.Source: §2.3 · p. 3

3. Approach

MoRight separates object motion from camera viewpoint through canonical-view transfer and models causal active/passive interactions to predict consequences or recover driving actions.

3.1. Disentangled Camera-Object Motion Control

  • A shared-weight dual-stream model anchors user motion in a static canonical view, conditions the target stream on camera poses, and transfers motion through cross-view self-attention during joint denoising.Motion and camera encodings are injected at each transformer block; the canonical stream acts as a virtual anchor while the target stream is provided to the user.Source: §3.1 · pp. 4–5, Figure 2 · p. 4

3.2. Motion Causality Modeling

  • Motion dropout trains the model to infer whichever active or passive component is absent from full-video supervision, enabling forward action-to-reaction and inverse reaction-to-action reasoning.Active motion is the intentional action; passive motion is the reaction of other objects.Source: §3.2 · p. 6

3.3. Training Data Curation

  • The curation pipeline estimates depth, camera poses, and dense tracks, canonicalizes trajectories to the first frame, and uses a vision-language model and segmentation masks to assign tracks to active or passive objects.Component-specific captions are also generated, with only one component's caption provided during training to prevent information leakage.Source: §3.3 · pp. 6–7
  • Training data combines synthetic static/moving-camera pairs and camera-operation augmentations with mixed single-view real videos and SyncCamMaster graphics to broaden camera coverage and mitigate synthetic artifacts.The synthetic moving-camera videos provide supervision for the second stream; mixed real videos include static-camera duplicates and videos supervising only the target stream.Source: §3.3 · p. 7

3.4. Training and Inference

  • Training uses pixel- or patch-level trajectories and masks occluded or missing tracks; inference accepts sparse strokes, optional text, and target poses, applies depth-based visibility masking, and returns the target stream.Randomly varying trajectory granularity supports fine-grained pixel control and object-level manipulation, while track masking simulates occlusion and tracking failures.Source: §3.4 · p. 7
  • On DynPose-100K and Cooking, MoRight achieves comparable visual quality and more accurate motion control despite weaker inputs than tracking-based methods.MoRight uses first-frame reprojected trajectories and camera poses, while tracking-based methods require privileged foreground/background tracks.Source: Table 1 · p. 8

4. Experiments

Experiments evaluate MoRight’s disentangled camera–object control and motion-causality reasoning across video benchmarks, human preferences, ablations, and identified limitations.

4.1. Implementation Details

  • The model is fine-tuned from Wan2.1-14B for 15K iterations on 64 GPUs, updating only the camera encoder, trajectory encoder, and self-attention layers.Training uses 76K static-view videos, 43K synthesized paired dynamic-view videos, and 3.4K synthetic interaction videos, all processed at 480p.Source: §4.1 · p. 8

4.2. Experiment Settings

  • MoRight’s control evaluation covers DynPose-100K, WISA, and real-world cooking videos, using video-quality, camera-accuracy, motion-accuracy, and motion-realism metrics.The evaluation uses 50 videos from each dataset; metrics include PSNR, SSIM, FID, FVD, camera rotation and translation errors, EPE, PC, and SA.Source: §4.2 · p. 8

4.3. Disentangled Camera-Object Motion Control

  • On DynPose-100K [61], WanMove [15] achieves the best overall numbers; MoRight is slightly behind, has comparable controllability to methods using privileged future-frame tracking information, and achieves the best EPE for object-motion accuracy; on Cooking, MoRight leads in visual quality and motion-control accuracy.Source: §4.3 · pp. 9, 11

4.4. Motion Causality Modeling

  • MoRight achieves the highest PC score on WISA and the best FID and FVD on both WISA and Cooking, while its SA is slightly lower than full-prompt baselines but remains comparable.For SA evaluation, passive-motion descriptions are removed from MoRight’s input to avoid information leakage; baselines retain prompts containing both actions and outcomes.Source: Table 2 · p. 11, §4.4 · p. 12

4.6. Ablation Studies

  • Ablations show that removing the canonical-view anchor or motion reasoning degrades control or interaction quality, while the full model remains robust across motion-input granularities and active or passive inputs.The cascaded pipeline accumulates control errors, and paired-only training slightly degrades camera accuracy; removing motion reasoning increases FID/FVD and reduces PC.Source: Table 3 · p. 11, §4.6 · pp. 13, 15

4.5. Human Perceptual Evaluation

  • Participants preferred MoRight in 53.5% of controllability, 54.6% of motion-realism, and 55.9% of photorealism evaluations, exceeding ATI and WanMove on all three criteria.The study collected 330 evaluations per criterion from 11 participants.Source: Figure 8 · p. 13, §4.5 · p. 13

4.7. Limitation Analysis

  • Limitations include implausible interaction outcomes, unnatural motion from occluded or sparse tracks, physical inconsistencies, hallucinated later-frame content, and degraded dynamics under drastic camera motion.The camera control is designed for common smooth trajectories, and drastic camera changes can degrade predicted interaction dynamics.Source: Figure 9 · p. 14, §4.7 · p. 15

5. Conclusion

MoRight combines independent control of object motion and camera viewpoints with causal modeling of action–response dynamics. It supports both predicting scene outcomes from user-driven motion and inferring actions from desired outcomes.

5. Conclusion

  • Experiments on DynPose-100K, WISA, and Cooking show strong performance in generation quality, motion control, and interaction awareness.The conclusion presents MoRight as a step toward more interactive and physically grounded video generation.Source: §5 · p. 15
  • MoRight’s dual-stream design independently controls object trajectories and camera viewpoints, addressing entangled camera and object motion.This design enables users to specify object motion separately from camera viewpoints.Source: §5 · p. 15
  • MoRight decomposes motion into user-driven active and consequential passive components to learn action–response dynamics.This addresses the limited causal reasoning of prior motion-controlled methods.Source: §5 · p. 15
  • At inference, MoRight predicts scene outcomes from active motion or infers actions from desired passive results.These capabilities provide forward prediction and inverse reasoning, respectively.Source: §5 · p. 15

A. Implementation Details · A.1. Network Architecture

MoRight adapts Wan2.1 I2V-14B with two-view video inputs and separate camera and tracking conditions, training selected encoders and attention layers while freezing the rest of the network.

A.1. Network Architecture

  • The model encodes two-view videos, concatenates their tokens temporally, and uses shared spatial RoPE embeddings with distinct temporal indices.It builds on Wan2.1 I2V-14B [67].Source: §A.1 · p. 16
  • A lightweight temporal encoder processes trajectory maps with RMSNorm, SiLU, and two 3 × 1 × 1 Conv3D layers that downsample time 4× to match Wan latent resolution.Source: §A.1 · p. 16
  • Camera control follows Gen3C [60], warping the first frame with the camera trajectory and VAE-encoding it into the same latent space and resolution.Source: §A.1 · p. 16
  • Camera and tracking features are projected to Wan's 5120-dimensional hidden space and added before self-attention; training updates these encoders and self-attention layers while freezing the rest.Source: §A.1 · p. 16

A.2. Training Data Curation

MoRight’s training-data curation identifies active and passive objects and motions, rewrites captions to separate motion types, and selects nearly static-camera videos for paired multi-view data.

A.2. Training Data Curation

  • Qwen3-VL [3] identifies active and passive objects, SAM2 [59] segments them, and Qwen3 [3] rewrites captions so each describes only one motion type.The original and rewritten captions are sampled with equal probability during training to encourage inference of plausible interaction consequences.Source: §A.2 · p. 16
  • Paired multi-view data comes from collected Internet videos with nearly static cameras, filtered to maximum rotation of 0.5° and translation of 5 mm using ViPE camera poses [36].Source: §A.2 · p. 16

A.3. Training · A.4. Inference

Training augments motion tracks and sampling to improve robustness and causality learning, while inference provides interactive control of object trajectories and camera viewpoints from a single image.

A.3. Training

  • Training randomly simplifies trajectories, supplies active or passive tracks, drops visible tracks, and truncates tracks to simulate partial observations.The respective probabilities are 0.5, 0.8/0.2 for active/passive tracks, and 0.2 for track dropout.Source: §A.3 · p. 16
  • Training samples 500–2000 tracks per iteration, fixes inference at 1500 tracks, and samples single-view data less often than multi-view data.The sampling balance is intended to support camera–object disentanglement and prevent overfitting to single-view motion patterns.Source: §A.3 · p. 17

A.4. Inference

  • At inference, users select objects and specify coarse shared-object trajectories or fine-grained sparse point tracks from the first frame.The interactive demo also allows trajectories to be drawn on the first frame from a single input image.Source: §A.4 · p. 17, Figure 11 · p. 17
  • For interaction-driven editing, users can apply push, directional pull, and reach-to-target primitives, or define arbitrary 2D passive-motion trajectories.The paper illustrates inverse reasoning with straight-line passive trajectories in different directions.Source: §A.4 · p. 17
  • Users specify camera motion independently through a sequence of camera poses, with the first frame treated as the identity pose.This supports editing object dynamics and camera viewpoints separately during generation.Source: §A.4 · p. 17
  • The interface visualizes trajectories across time and checks occlusion using first-frame depth estimated by MoGe.These tools support interactive editing of object dynamics and camera viewpoints.Source: §A.4 · p. 17

A.5. Evaluation

Evaluation uses a 50-clip cooking benchmark alongside automatic motion-quality measures and human judgments of controllability, realism, and visual quality.

A.5. Evaluation

  • The Cooking Benchmark contains 50 YouTube cooking clips spanning kitchen environments and hand–object interactions such as pushing, cutting, and picking.It is designed as a testbed for interactive motion reasoning.Source: §A.5 · p. 17
  • Motion quality is evaluated with Physical Commonsense (PC) for real-world physical behavior and Semantic Adherence (SA) for consistency with the input text prompt.SA uses each video’s original caption as the prompt; an automatic evaluation rater scores the generated videos, with normalized scores reported across methods.Source: §A.5 · pp. 17–18
  • Human evaluation compares videos on controllability, motion realism, and photorealism across 30 randomly selected video sets with shuffled candidate order.Participants select the best video for each criterion, with ties allowed or “None” if no video satisfies it.Source: Figure 12 · p. 18, §A.5 · p. 18

B. More Qualitative Results … B.3. Qualitative Comparison

The qualitative results show MoRight supporting forward and inverse causal reasoning, maintaining object dynamics across camera controls, and producing more controllable, coherent interactions than ATI and WanMove despite using weaker inputs.

B.1. Interactive Motion Generation

  • MoRight predicts plausible scene consequences from specified active motion in forward reasoning and infers feasible driving actions from passive outcomes in inverse reasoning.Figure 13 visualizes input action and passive-trajectory tracks over generated static reference-view videos.Source: §B.1 · p. 18, Figure 13 · p. 19
  • The causal-reasoning examples distinguish user actions from passive trajectories by overlaying both types of input tracks on generated frames.Colored tracks represent active actions or passive trajectories in the static reference-view visualizations.Source: §B.1 · p. 18, Figure 13 · p. 19

B.3. Qualitative Comparison

  • With first-frame 2D trajectories rather than privileged 3D tracks and full interaction trajectories, MoRight shows stronger controllability and more coherent interactions than ATI [68] and WanMove [15].Camera motion is introduced in the second stream of MoRight’s dual-stream architecture, while the baselines use depth-projected, pixel-aligned per-frame tracks.Source: Figure 15 · p. 21, §B.3 · p. 22

B.2. Disentangled Camera-Object Control

  • Three object motions crossed with orbit-left, zoom-in, and zoom-out camera motions produce nine videos, with each group holding object motion fixed while varying viewpoint.The examples demonstrate separate object and camera control.Source: Figure 14 · p. 20, §B.2 · p. 22
  • Object dynamics remain consistent across different camera viewpoints under the same object motion, although stochastic interaction generation causes minor variations.The variations occur across videos sharing an object motion but using different camera motions.Source: Figure 14 · p. 20, §B.2 · p. 22

B.3. Qualitative Comparison

  • The qualitative comparison contrasts MoRight’s first-frame 2D motion input with baselines that require privileged depth trajectories projected into pixel-aligned per-frame tracks and full active and passive trajectories.The comparison includes ATI [68] and WanMove [15].Source: Figure 15 · p. 21, §B.3 · p. 22

Related papers

Loading 2604.07348v1…