Source-linked AI summary
Motion Prompting: Controlling Video Generation with Motion Trajectories
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, Andrew Owens, Deqing Sun
TL;DR
Video-generation text prompts do not precisely capture dynamic motion, motivating a more expressive control representation. The paper trains a track-conditioned video model using sparse or dense motion trajectories and expands high-level requests into detailed prompts. It demonstrates broad applications, strong quantitative and human-evaluation results, and emergent behaviors, while noting failures from both motion conditioning and the underlying video model.
Problem
Text prompts struggle to express the trajectory, timing, acceleration, and synchronized movements needed for granular motion control.
Method
The paper trains a ControlNet adapter conditioned on flexible spatio-temporal motion trajectories and uses motion prompt expansion to convert high-level requests into detailed prompts.
Results
The unified approach supports object and camera control, motion transfer, image editing, and model probing, with quantitative and human evaluations indicating strong performance.
Takeaways & Limitations
Motion prompts provide a flexible interface for controlling video motion and probing learned physics and world knowledge in video models.
Takeaways & Limitations
Failures arise from both motion conditioning or prompting and the underlying video model, including distorted objects and implausible new object formation.
Abstract
from arXiv · showhide
Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions. To this end, we train a video generation model conditioned on spatio-temporally sparse or dense motion trajectories. In contrast to prior motion conditioning work, this flexible representation can encode any number of trajectories, object-specific or global scene motion, and temporally sparse motion; due to its flexibility we refer to this conditioning as motion prompts. While users may directly specify sparse trajectories, we also show how to translate high-level user requests into detailed, semi-dense motion prompts, a process we term motion prompt expansion. We demonstrate the versatility of our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing. Our results showcase emergent behaviors, such as realistic physics, suggesting the potential of motion prompts for probing video models and interacting with future generative world models. Finally, we evaluate quantitatively, conduct a human study, and demonstrate strong performance. Video results are available on our webpage: https://motion-prompting.github.io/
1. Introduction
The paper treats motion as a complementary control signal for video generation because text underspecifies temporal details. It introduces flexible motion prompts and motion prompt expansion, demonstrating broad control capabilities and strong evaluation results.
- Motivation: Text prompts struggle to specify motion nuances such as exact trajectories, acceleration, timing, and synchronized movements.The paper argues that these details are better conveyed through motion itself.
- Motion prompts: Spatio-temporally sparse or dense trajectories encode arbitrary numbers of point motions, object-specific or global motion, and temporally sparse constraints.This representation tracks point movement and visibility throughout a video.
- Motion prompt expansion: Motion prompt expansion translates high-level requests into detailed trajectories, bridging user goals and the motion representation.Examples include moving a camera around a plane or rotating an object.
- Applications: The framework supports object and camera control, simultaneous motion control, motion transfer, image interaction, and drag-based image editing.These capabilities are demonstrated using a single trained model and varied user inputs.
- Evaluation: The authors report quantitative and human evaluations against baselines, with the model performing well.The paper also presents ablations to validate design choices.
2. Related Work
Related work adapts video generation models to motion conditioning through varied representations, architectures, losses, and training strategies. This paper positions trajectory-based prompts as flexible while emphasizing a simpler single-stage training recipe.
- Video diffusion models: Video generation models have been conditioned on natural language or static images and are also viewed as potential world simulators.Prior work includes visual planning for embodied agents.
- Motion-conditioned video generation: Motion-conditioned methods adapt pretrained video models using techniques including LoRA, DreamBooth, sparse motions, specialized losses, architectures, and multi-stage fine-tuning.Several approaches require complicated engineering for stable training and convergence.
- Motion-conditioned video generation: The proposed model uses single-stage training with uniformly sampled dense trajectories and generalizes to sparse and dense trajectories at inference.The authors contrast this recipe with methods requiring separate adapters, custom losses, or data-filtering pipelines.
- Alternative control signals: Other approaches use entity-centric signals such as bounding boxes, masks, human pose, camera pose, or entity-centric masks for zero-shot adaptation.The paper presents motion prompts as a more flexible motion-control interface.
- Motion representations: Optical flow can accumulate errors and lacks occlusion handling, whereas point trajectories support occlusions and sparse or dense tracking over arbitrary temporal durations.These properties motivate trajectories for the paper’s motion-control application.
3. Method
The method conditions a Lumiere-based video diffusion model on point-track motion prompts encoded in a space-time volume, supporting sparse or dense motion across objects and scenes. A ControlNet is trained on automatically extracted tracks, and the resulting model generalizes across track layouts, counts, and starting times.
- 3.1. Motion Prompts: The model accepts a first frame, text prompt, and point-track motion prompt, enabling object control, camera control, motion transfer, and drag-based image editing.These capabilities are handled under one unified model.
- 3.1. Motion Prompts: Point trajectories encode sparse or dense motion, object motion, scene motion, and occlusions through visibility flags.Each track is represented across space and time, allowing a unified motion-control interface.
- 3.2. Architecture: Tracks are encoded as unique embeddings written into visible track locations in a T × H × W × C volume, with other locations set to zero.Visibility suppresses embeddings for off-screen or occluded tracks; colliding track embeddings are summed.
- 3.4. Training: The model is trained with ControlNet on 2.2M videos containing densely extracted tracks and predicted occlusions, sampling a random number of tracks per video.The videos are resized to 128 × 128, with 16,384 extracted tracks per video.
- 3.4. Training: Training exhibits sudden convergence, while diffusion loss is not correlated with performance in following the supplied tracks.The model can shift from ignoring conditioning to fully using it within a short number of training steps.
- 3.4. Training: The model generalizes to spatially localized tracks, more or fewer tracks than used in training, and tracks that begin after the first frame.The authors hypothesize that convolutional inductive biases and trajectory diversity contribute to this generalization.
4. Motion Prompts
Motion prompts support interactive image manipulation, camera and object control, motion composition, motion transfer, and probing model failures. Prompt expansion and derived trajectories translate simple user inputs into richer controls, while experiments reveal both emergent behaviors and practical limitations.
- 4.1. “Interacting” with an Image: The system converts mouse drags into configurable grids of point tracks, including static tracks that keep image backgrounds still.Users can choose grid density and size at inference time, and partial tracks support multiple drags at different locations and times.
- 4.1. “Interacting” with an Image: Interactive motion prompts produce direct and emergent behaviors, including moving heads, tossing hair, sweeping sand, and predicting motion after brief conditioning.These interactions are used to probe the learned physics and world understanding of the video prior.
- 4.1. “Interacting” with an Image: Drag-based image editing applies user-supplied drags to alter objects while optionally keeping image regions static.The paper presents qualitative results for this application in Figure 4.
- 4.3. Camera Control with Depth: Camera control projects a depth-estimated point cloud through specified camera poses to create tracks, enabling orbiting, upward arcing, and mouse-driven camera motion.The method uses monocular depth estimation and can add occlusion flags through z-buffering.
- 4.3. Camera Control with Depth: Although the model is not trained or conditioned on camera poses, motion prompts yield compelling camera control from general motion training.The authors use this result to argue that specific capabilities can be elicited from general motion conditioning.
- 4.4. Composing Motions: Composing object and camera tracks enables simultaneous control, but the two-dimensional displacement approximation can fail under extreme camera motion.The authors report that the composition works well for small to moderate camera motion without specific training for this capability.
- 4.5. Motion Transfer: Motion transfer extracts tracks from a source video and applies them to a different image, including puppeteering a macaque and transferring chewing motion to trees.The method is described as robust to fairly out-of-domain target images.
- 4.6. Failures, Limitations, and Probing Models: Failures arise from motion conditioning or prompting and from the underlying video model, with examples including distorted anatomy and spontaneous chess-piece creation.These failures motivate using motion prompts to probe limitations in learned video representations.
5. Quantitative Results
The paper evaluates track-conditioned video generation quantitatively and through a human study, finding strong performance against recent baselines. Ablations further show that dense-track training is most effective, particularly when many tracks are used.
- The evaluation measures video appearance with PSNR, SSIM, LPIPS, and FVD, and motion adherence with EPE on DAVIS.
- Our model outperforms the baselines in almost all cases, while DragAnything sometimes achieves lower EPE with fewer tracks but produces visual artifacts.The artifacts are reflected in DragAnything’s weaker PSNR, SSIM, LPIPS, and FVD results.
- The human study compares motion adherence, motion realism, and visual quality using 180 two-alternative forced-choice questions.The study uses 30 single-trajectory inputs and reports win rates with 95% confidence intervals.
- When motion and appearance are considered together, participants prefer our approach over baselines in all categories.
- Dense-track training is most effective, especially for large track counts, and also performs better for sparse tracks in the ablation.The authors hypothesize that dense training provides more useful training signal and generalizes to sparse tracks.
6. Conclusion
The paper concludes that flexible motion prompts support motion-conditioned video generation across sparse and dense camera, object, and scene motion. Motion prompt expansion broadens this framework to high-level requests and applications including motion control, transfer, and image editing.
- Motion prompts use spatio-temporal trajectories to encode arbitrary motion complexity for cameras, objects, or full scenes.
- Motion prompt expansion translates high-level motion requests into detailed motion prompts.
- The approach supports motion control, motion transfer, interacting with images, and image editing.
A.1. Architecture and Training
The model is trained with a ControlNet conditioned on motion tracks and a Lumiere-based video-generation architecture. Training uses a large internal video dataset and dense precomputed trajectories.
- The ControlNet copies Lumiere’s encoder stack, adds zero convolutions, and accepts a T × H × W × C conditioning signal with T = 80, H = W = 128, and C = 64.
- The model is trained for 70,000 steps with Adafactor at a learning rate of 1 × 10^-4 without learning-rate decay.
- The spatial super-resolution model produces 1024 × 1024, 80-frame videos at 16 frames per second without motion-conditioning fine-tuning.
- Training uses an internal dataset of 2.2 million videos with 16,384 precomputed tracks per video.
A.2. Qualitative Results
The qualitative pipeline converts mouse gestures, geometric transformations, depth, and camera poses into motion tracks for video generation. Track subsampling balances explicit control against the model’s learned video priors across camera control and motion transfer.
- The GUI records 80 frames of mouse input to match the model’s five-second output at 16 frames per second.
- Mouse drags become grids of tracks whose stride, size, persistence, and pinning can be configured by the user.
- Mouse motion can be converted into spherical tracks by projecting rotating 3D points onto 2D image trajectories.
- Camera control uses monocular depth to unproject the first frame into a point cloud, project it through camera poses, and determine visibility with z-buffering.
- Camera and motion-transfer prompts use dense tracks that are randomly subsampled because too many suppress learned priors while too few provide insufficient control.
- For most depth-based prompts, 1024 tracks provide a balance between control and emergent video-prior effects.
- The qualitative evaluation uses first frames, tracks, and text prompts on DAVIS, with accommodations for baseline inputs and human-study comparisons.
B. Training Observations
Training loss does not directly track model performance, and the model can suddenly shift from ignoring the control signal to fully adhering to it within a few epochs.
- Training loss does not directly correlate with model performance.The authors report that test losses do not begin decreasing until step 20,000, while training loss remains difficult to interpret.
- The model exhibits sudden convergence from ignoring the control signal to full adherence within a few epochs.
- Test losses begin decreasing only after step 20,000, despite the inscrutable training-loss behavior.
C. Full Quantitative Results
The supplementary evaluation covers DAVIS videos across multiple trajectory counts and measures both visual appearance and motion. Additional applications demonstrate control through human pose, motion magnification, and space-time motion visualizations.
- DAVIS evaluations cover trajectory counts N = {1, 16, 512, 2048}, with supplementary results also reported for N = 4 and N = 64.
- Quantitative evaluation measures generated-video appearance with PSNR, SSIM, LPIPS, and FVD, and motion with EPE.Each method is trained from a different base model.
- Human-motion control uses pose-estimated keypoints, applies motions to selected keypoints, and feeds them to the model.The pose is estimated with an off-the-shelf model.
- Motion magnification tracks an input video, smooths the tracks with a spatial-temporal Gaussian blur, magnifies them, and conditions generation on the first frame and magnified tracks.
- The motion-magnification visualization shows the first frame of two videos and space-time slices through a blue line at different magnification factors.