Source-linked AI summary
MoRight: Motion Control Done Right
Shaowei Liu, Xuanchi Ren, Tianchang Shen, Huan Ling, Saurabh Gupta, Shenlong Wang, Sanja Fidler, Jun Gao
TL;DR
Existing motion-controlled video methods entangle camera and object motion and treat motion as kinematic displacement without modeling causal interactions. MoRight uses canonical-view object modeling, cross-view transfer, and active/passive motion decomposition to support disentangled control and forward or inverse reasoning. Across three benchmarks, it achieves strong performance in generation quality, motion control, and interaction awareness, while remaining limited by several failure cases and difficulty with drastic camera motion.
Problem
Existing methods entangle camera and object motion and largely ignore causal relationships between objects, limiting unified motion-controlled video generation.
Method
MoRight models object motion in a canonical static view, transfers it to target viewpoints through cross-view attention, and decomposes motion into active and passive components.
Results
MoRight outperforms existing methods across three benchmarks in generation quality, motion controllability, and interaction awareness.
Takeaways & Limitations
The framework supports independent camera and object control together with forward prediction of consequences and inverse recovery of actions from desired outcomes.
Takeaways & Limitations
MoRight can produce incorrect interactions, unnatural motion under temporally sparse trajectories, physically inconsistent outcomes, hallucinated content, and degraded dynamics under drastic camera motion.
Abstract
from arXiv · showhide
Generating motion-controlled videos--where user-specified actions drive physically plausible scene dynamics under freely chosen viewpoints--demands two capabilities: (1) disentangled motion control, allowing users to separately control the object motion and adjust camera viewpoint; and (2) motion causality, ensuring that user-driven actions trigger coherent reactions from other objects rather than merely displacing pixels. Existing methods fall short on both fronts: they entangle camera and object motion into a single tracking signal and treat motion as kinematic displacement without modeling causal relationships between object motion. We introduce MoRight, a unified framework that addresses both limitations through disentangled motion modeling. Object motion is specified in a canonical static-view and transferred to an arbitrary target camera viewpoint via temporal cross-view attention, enabling disentangled camera and object control. We further decompose motion into active (user-driven) and passive (consequence) components, training the model to learn motion causality from data. At inference, users can either supply active motion and MoRight predicts consequences (forward reasoning), or specify desired passive outcomes and MoRight recovers plausible driving actions (inverse reasoning), all while freely adjusting the camera viewpoint. Experiments on three benchmarks demonstrate state-of-the-art performance in generation quality, motion controllability, and interaction awareness.
1. Introduction
MoRight addresses the entanglement of camera and object motion and the lack of causal reasoning in controllable video generation. Its unified framework enables disentangled viewpoint and object control, models action–consequence relationships, and supports forward and inverse reasoning.
- Existing methods entangle camera and object motion, making joint control ambiguous because viewpoint changes alter pixel trajectories.
- They also treat user-specified motion as kinematic displacement, largely ignoring causal consequences among interacting objects.
- Across three benchmarks, MoRight outperforms existing methods in generation quality, motion controllability, and interaction awareness.
- MoRight uses a disentangled framework that lets users specify object trajectories while freely adjusting camera viewpoints for coherent video generation.
- The model decomposes motion into active user-driven and passive consequence components to produce consistent scene dynamics.
- MoRight supports forward reasoning from active motion to future scene evolution and inverse reasoning from desired passive outcomes to plausible actions.
2. Related Work
Prior controllable video methods commonly rely on pixel-space trajectories, which entangle object and camera motion and may require privileged geometric information. MoRight instead uses a canonical static-view branch and cross-view attention without explicit 3D supervision or pre-computed scene decomposition.
- Trajectory-based methods use pixel-space signals that inherently entangle object motion with viewpoint changes.
- Methods that decouple camera and object motion often rely on privileged depth, 3D trajectories, or foreground–background masks.
- These assumptions require the full video sequence or 3D motion in advance, limiting applicability when only one reference frame is available.
- MoRight represents object dynamics in a canonical static view and transfers them to arbitrary viewpoints without explicit 3D supervision or pre-computed scene decomposition.
- Physics-engine and explicit-action approaches model specific interaction phenomena but are constrained by domain-specific designs and simulation requirements.
3. Approach
MoRight uses a dual-stream formulation to separate object motion from camera viewpoint, transferring motion from a canonical view to a target view. It also models active and passive motion components so the model can generate plausible consequences and support forward and inverse reasoning.
- Disentangled Camera-Object Motion Control: The canonical stream learns object motion in a fixed view, while cross-view self-attention transfers that information to the target viewpoint.Camera and motion conditions are injected into each transformer block before the streams exchange information through self-attention.
- Disentangled Camera-Object Motion Control: MoRight jointly denoises a canonical stream containing object motion and a target stream containing object and camera motion, reusing shared DiT weights.The target-stream output is presented to the user during inference.
- Disentangled Camera-Object Motion Control: Camera conditioning uses warped frames based on camera pose and estimated depth, whereas the canonical stream uses identity warping.Motion is encoded as a per-pixel trajectory map; the target stream receives an empty motion condition.
- Motion Causality Modeling: Motion causality decomposes foreground-object tracks into active intentional actions and passive consequential reactions.Examples include a hand pushing an object and the pushed object sliding in response.
- Motion Causality Modeling: During training, motion dropout removes one component while supervision uses the full video, encouraging the model to infer missing action or consequence rather than replaying trajectories.The dropped tracking condition is selected using a dropout probability p.
- Motion Causality Modeling: At inference, MoRight supports forward reasoning from action to reaction and inverse reasoning from a desired reaction to a plausible driving action.These reasoning modes operate alongside the framework’s disentangled camera and object-motion control.
4. Experiments
MoRight is evaluated on diverse datasets and comparisons spanning camera control, interactive motion reasoning, controllability, and visual quality. It achieves strong results while retaining robustness across input conditions, but remains limited by implausible interactions, sparse tracks, physical inconsistencies, hallucinations, and drastic camera motion.
- Experiment Settings: Three evaluation datasets cover dynamic viewpoints, physical interactions, and complex hand-object cooking scenarios.DynPose-100K, WISA, and 50 real-world cooking videos are used for evaluation.
- Experiment Settings: MoRight uses canonical-frame reprojected trajectories, unlike baselines requiring dense future pixel tracks for motion control.All methods share the Wan2.1-14B backbone in the reported comparison setup.
- Quantitative Results: MoRight achieves the best EPE for object motion accuracy and the best overall visual quality and motion-control performance on Cooking.On DynPose-100K, WanMove has the best overall numbers, while MoRight remains comparably controllable despite not using privileged future-frame tracking.
- Interactive Motion Generation: MoRight achieves the highest PC score on WISA and the best FID and FVD on both WISA and Cooking, while remaining comparable on SA under active-only prompts.The evaluation uses video quality metrics FID and FVD and motion-realism metrics PC and SA.
- Human Perceptual Evaluation: Human evaluators preferred MoRight in 53.5% of controllability, 54.6% of motion-realism, and 55.9% of photorealism judgments.These preference rates exceeded ATI and WanMove across all three criteria.
- Ablation and Limitation Analysis: Ablations show that removing the fixed-view branch or motion reasoning degrades control, while performance remains stable across motion granularities and active or passive inputs.The full model achieves the best overall ablation performance, whereas errors can arise from incorrect interactions, occlusion-induced sparsity, disappearing objects, or hallucinated content.
5. Conclusion
MoRight unifies disentangled camera–object motion control with causal reasoning for controllable, interaction-aware video generation. It supports forward prediction from active motion and inverse reasoning from desired passive outcomes, with strong performance across three benchmarks.
- MoRight independently controls object trajectories and camera viewpoints through a dual-stream design.This addresses prior entanglement between camera and object motion.
- MoRight decomposes motion into active user-driven and passive consequence components to learn action–response dynamics.
- At inference, MoRight supports forward prediction from active motion and inverse reasoning from desired passive results.
- MoRight shows strong performance in generation quality, motion control, and interaction awareness on DynPose-100K, WISA, and Cooking.
A.1. Network Architecture
The network uses two-view video streams with shared spatial embeddings and distinct temporal indices, while camera and tracking features condition Wan2.1 transformer blocks.
- MoRight encodes two-view videos and concatenates their tokens along the temporal dimension before processing them with Wan2.1.
- The two streams share spatial RoPE embeddings but use different temporal indices to distinguish their temporal roles.
- A lightweight temporal encoder processes trajectory maps using RMSNorm, SiLU, and two 3 × 1 × 1 Conv3D layers.The Conv3D layers downsample the temporal dimension by 4× to match Wan2.1.
- Camera and tracking features are projected to the 5120-dimensional Wan hidden space and added to video tokens before self-attention.
- Training updates the lightweight temporal, camera, tracking, and self-attention components while freezing the rest of the network.
A.2. Training Data Curation
Training data curation identifies active and passive objects and rewrites captions so each describes one motion type. Qwen3-VL performs identification and caption rewriting, while SAM2 segments video objects.
- Qwen3-VL identifies active and passive objects and their motion in training videos.
- SAM2 provides video object segmentation for the objects identified during curation.
- Qwen3 rewrites captions by decomposing object motion into active or passive descriptions.Each rewritten caption contains only one motion type.
- The Qwen3-VL curation prompt requires valid JSON output from a video-understanding specialist.
A.3. Training
Training uses trajectory simplification, active/passive track sampling, motion dropout, and multi-view sampling controls to improve robustness and camera–object disentanglement. The system also provides interactive trajectory and camera editing from a single image.
- Training: Training randomly simplifies input trajectories with probability 0.5 by averaging tracks so each object shares one trajectory.
- Training: Active and passive motion tracks are provided with probabilities 0.8 and 0.2, respectively, to encourage causal motion reasoning.
- Training: Training uses motion dropout and controls multi-view versus single-view sampling to support robustness and camera–object disentanglement.Single-view data is sampled at a lower rate to reduce overfitting to single-view motion patterns.
- Training: Inference fixes the number of input tracks at 1500 for consistent evaluation.The training range is between 500 and 2000 tracks per iteration.
- Interactive control: The interactive interface lets users draw object trajectories on the first frame while specifying camera motion independently through camera poses.It supports coarse or fine-grained motion editing, trajectory visualization, and occlusion checking.
A.5. Evaluation
MoRight is evaluated on interactive cooking videos using automatic physical and semantic metrics, alongside human perceptual judgments of controllability, realism, and photorealism.
- The Cooking Benchmark contains 50 real-world YouTube video clips featuring diverse hand–object interactions and manipulation behaviors.Examples include pushing, cutting, and picking across varied kitchen environments.
- Physical Commonsense measures whether generated videos follow real-world physical behaviors.
- Human evaluation asks participants to select the best video for controllability, motion realism, and photorealism, allowing ties or None selections.
- Semantic Adherence measures consistency with the input text prompt using each video’s original caption.An automatic evaluation rater produces normalized scores across methods.
B.1. Interactive Motion Generation
MoRight demonstrates causal interaction reasoning and disentangled camera–object control through forward and inverse motion generation, varied viewpoints, and qualitative comparisons with ATI and WanMove.
- Interactive Motion Generation: MoRight predicts plausible scene consequences from active motion and recovers feasible driving actions from desired passive outcomes.These correspond to forward and inverse causal reasoning, respectively.
- Disentangled Camera-Object Control: Nine videos combine three object motions with three camera motions, including orbit-left, zoom-in, and zoom-out.The results are organized into three groups sharing object-motion settings.
- Disentangled Camera-Object Control: Holding object motion fixed while varying camera viewpoints maintains consistent object dynamics, with minor stochastic variations.
- Qualitative Comparison: Compared with ATI and WanMove, MoRight uses only first-frame 2D motion trajectories rather than privileged 3D tracks and full interaction trajectories.Camera motion is introduced through the second stream of its dual-stream architecture.
- Qualitative Comparison: Despite weaker inputs, MoRight produces stronger controllability and more coherent interactions while maintaining disentangled camera–object motion.