Source-linked AI summary
MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance
Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, Zuxuan Wu
TL;DR
Trajectory-controlled video generation lacks flexible control formats and public resources for training and evaluation. MagicMotion addresses these gaps with dense-to-sparse trajectory guidance, MagicData, and MagicBench, and outperforms prior approaches across reported metrics and object-count categories.
Problem
Existing trajectory-controlled video methods provide limited control formats and face challenges with complex motion, object consistency, and multi-object control.
Method
MagicMotion uses Trajectory ControlNet and progressive training to support masks, boxes, and sparse boxes for image-to-video generation.
Results
MagicMotion outperforms previous approaches across metrics on MagicBench and DAVIS and achieves the best results across MagicBench object-number categories.
Takeaways & Limitations
MagicData and MagicBench provide public trajectory-annotated videos and evaluation across video quality, trajectory accuracy, and controlled-object counts.
Abstract
from arXiv · showhide
Recent advances in video generation have led to remarkable improvements in visual quality and temporal coherence. Upon this, trajectory-controllable video generation has emerged to enable precise object motion control through explicitly defined spatial paths. However, existing methods struggle with complex object movements and multi-object motion control, resulting in imprecise trajectory adherence, poor object consistency, and compromised visual quality. Furthermore, these methods only support trajectory control in a single format, limiting their applicability in diverse scenarios. Additionally, there is no publicly available dataset or benchmark specifically tailored for trajectory-controllable video generation, hindering robust training and systematic evaluation. To address these challenges, we introduce MagicMotion, a novel image-to-video generation framework that enables trajectory control through three levels of conditions from dense to sparse: masks, bounding boxes, and sparse boxes. Given an input image and trajectories, MagicMotion seamlessly animates objects along defined trajectories while maintaining object consistency and visual quality. Furthermore, we present MagicData, a large-scale trajectory-controlled video dataset, along with an automated pipeline for annotation and filtering. We also introduce MagicBench, a comprehensive benchmark that assesses both video quality and trajectory control accuracy across different numbers of objects. Extensive experiments demonstrate that MagicMotion outperforms previous methods across various metrics. Our project page are publicly available at https://quanhaol.github.io/magicmotion-site.
1. Introduction
MagicMotion addresses limitations in trajectory-controlled video generation with dense-to-sparse control, while introducing public data and evaluation resources for systematic training and benchmarking.
- Fine-grained trajectory control is needed because text-to-video methods lack precise control over object and camera movement.
- Existing trajectory methods use single-format control signals, creating a trade-off between sparse flexibility and dense control over object shape and size.
- MagicMotion integrates trajectory information through Trajectory ControlNet and supports masks, boxes, and sparse boxes with progressive training.
- MagicData provides 23K annotated video samples, while MagicBench evaluates video quality and trajectory accuracy across six object-count categories.
- The paper identifies masks, boxes, and sparse boxes as three trajectory-control signals and presents MagicData and MagicBench as public resources.
2. Related Works
Related work progresses from diffusion-based video generation toward trajectory control, but existing approaches still lack precise object and camera trajectory guidance.
- Video Diffusion Models: Diffusion models accelerated video generation, while early systems such as AnimateDiff and SimDA inserted temporal layers into pretrained text-to-image models.
- Video Diffusion Models: UNet-based methods struggle with long, high-quality videos, whereas DiT-based systems generate longer high-quality videos but rely mainly on text or image guidance.
- Trajectory Controllable Video Generation: Trajectory-controllable methods guide object and camera motion using optical flow maps or point maps processed through trajectory encoders.
3. Method
MagicMotion combines a DiT-based image-to-video model with Trajectory ControlNet, progressive dense-to-sparse training, latent segmentation supervision, and dedicated data and benchmark pipelines.
- Overview: MagicMotion generates a video from an input image and trajectory maps, with the output aligned to the provided trajectories.
- Base I2V generation model: The base I2V model encodes images and videos with a pretrained 3D VAE, denoises latent representations through DiT blocks, and decodes the result into video.
- Trajectory ControlNet: Trajectory ControlNet encodes trajectory information with trainable DiT copies and injects it into corresponding base-model blocks through zero-initialized convolutions.
- Dense-to-Sparse Training Procedure: Progressive training transfers weights across stages to support dense masks, bounding boxes, and sparse boxes, while fixing the first trajectory frame as a segmentation mask.
- Latent Segmentation Loss: The latent segmentation head predicts masks from DiT features without decoding, and latent segment loss compares predicted and ground-truth mask latents.
- Data Pipeline: MagicData addresses the lack of public trajectory annotations with a curation-and-filtering pipeline that yields 23K videos containing dense and sparse annotations.
- MagicBench: MagicBench contains 600 videos grouped into six object-count categories and evaluates both video quality and trajectory accuracy.
4. Experiment
Experiments evaluate MagicMotion against prior trajectory-controllable methods on video quality, trajectory accuracy, object counts, and component ablations. MagicMotion consistently follows specified motion while preserving video quality, and its dataset and training designs improve control and object-shape fidelity.
- Comparison with Other Approaches: MagicMotion is evaluated on MagicBench and DAVIS against seven public trajectory-controllable image-to-video methods.Evaluation uses annotated trajectory inputs and compares both quantitative and qualitative results.
- Comparison with Other Approaches: MagicMotion outperforms all previous approaches across every reported metric on both MagicBench and DAVIS.It also achieves the best results across categories defined by the number of controlled objects.
- Comparison with Other Approaches: MagicMotion smoothly follows specified trajectories while preserving high video quality, unlike baselines exhibiting shape deformation, artifacts, or fine-detail inconsistencies.The qualitative comparison includes failures in object-shape preservation and subject consistency among competing methods.
- Ablation Studies: Models trained on MagicData outperform those trained on the ablation dataset across all metrics.Without MagicData, the model can generate an unexpected child instead of moving the intended boy while maintaining video quality.
- Ablation Studies: Removing progressive training weakens object-shape perception and trajectory accuracy, while latent segment loss similarly improves fine-grained shape comprehension under sparse control.The progressive-training ablation distorts a woman’s head, and omitting latent segment loss reduces shape understanding and control precision.
5. Conclusion
MagicMotion integrates trajectory information into a diffusion transformer and supports dense-to-sparse control through progressive training. The paper also contributes MagicData and MagicBench, with experiments on MagicBench and DAVIS demonstrating superiority over previous works.
- Method: MagicMotion uses a ControlNet-like architecture to integrate trajectory information into a diffusion transformer.Progressive training supports masks, bounding boxes, and sparse boxes, while latent segment loss improves fine-grained shape perception under sparse conditions.
- Resources: The paper introduces MagicData, a high-quality annotated dataset, and MagicBench, a benchmark measuring video quality, trajectory accuracy, and controlled-object counts.MagicData is created through a data curation and filtering pipeline.
- Results: Experiments on MagicBench and DAVIS demonstrate MagicMotion’s superiority compared with previous works.
A. Ablations on Latent Segment Loss
The latent segment loss ablation shows that this loss improves the model’s ability to capture object shapes and maintain accurate trajectory control under sparse guidance.
- Ablation on Latent Segment Loss: Removing latent segment loss significantly reduces object-shape capture and trajectory-control accuracy.Without the loss, the woman’s arm in the generated video appears incomplete.
B. Additional Experiments
Additional experiments apply MagicMotion to camera-motion control, video editing, and different trajectories from one input image. These settings demonstrate control over camera effects, edited content, and multiple object paths.
- Camera Motion Control: MagicMotion controls camera rotation, zoom, and pan through bounding-box trajectories.Rotating boxes simulates camera rotation, changing box size controls perceived distance, and shifting boxes creates opposite camera motion.
- Video Editing: MagicMotion supports video editing by combining an edited first frame with segment-mask trajectory guidance.The experiments transform a black swan into a diamond swan and place a camel in a palace setting.
- Same Input Image with Different Trajectories: MagicMotion animates different objects along designated paths from the same input image.Examples include two bears, two fish, and the moon following separate motion trajectories.
C. Latent Segment Masks
MagicMotion Stage 3 predicts latent segmentation masks from sparse bounding-box conditions, including through frames without trajectory information.
- Stage 3 predicts latent segmentation masks for each frame from sparse bounding-box conditions.The masks are predicted across dynamic scenes rather than supplied directly as dense guidance.
- The predictions remain accurate when some frames have no trajectory information.Examples include a man standing up toward a robot and a boy’s head sinking into water.
D. Additional Comparison results
Additional comparisons report MagicMotion’s backbone context, broad performance across object counts, and qualitative control advantages over prior methods.
- Table 4 compares each method’s backbone together with supported video-generation length and resolution.
- MagicMotion consistently outperforms other methods across all MagicBench metrics, especially with many moving objects.The reported margin increases as the number of controlled objects becomes larger.
- Qualitative comparisons show accurate trajectory control and high-quality videos, whereas other methods exhibit significant defects.The comparisons cover cat, witch, elephant, robot, and tiger-motion scenarios.
E. Additional Ablation Results.
Ablations show that MagicData, progressive training, and latent segment loss each address distinct visual or motion-quality failures.
- Without MagicData training, the generated woman has an extra hand.
- Without progressive training, turning produces severe defects, including a second face appearing where the hair should be.
- Camera movements can be controlled by setting specific trajectory conditions.
- Without latent segment loss, the woman’s lipstick is distorted into a rectangular shape.
F. More Details on MagicData
Supplementary details describe MagicData and MagicBench settings, along with additional examples of editing, trajectory control, latent-mask prediction, and qualitative comparisons.
- MagicData videos average 346 frames and typically measure 999 pixels high by 1503 pixels wide.During training, videos are resized to 48 frames at 720p.
- Stage 1 can move foreground objects along an origin video’s trajectory after FLUX edits the first-frame image.
- MagicBench videos are sampled to 49 frames and resized to 720p for evaluation, with six classes based on annotated foreground-object counts.One example video is provided for each object-number category.
- MagicMotion can generate videos from the same input image using different trajectories.
- The supplementary figures include latent-mask visualizations, quantitative comparisons for one-to-more-than-five moving objects, and qualitative comparison cases.The quantitative tables cover object-count groups 1/2, 3/4, and 5/above 5.