Source-linked AI summary
Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance
Ruihang Chu, Yefei He, Zhekai Chen, Shiwei Zhang, Xiaogang Xu, Bin Xia, Dingdong Wang, Hongwei Yi, Xihui Liu, Hengshuang Zhao, Yu Liu, Yingya Zhang, Yujiu Yang
TL;DR
Motion-controllable video generation remains limited by coarse control and restricted scalability. Wan-Move edits image condition features using latent point-trajectory guidance without architectural changes, and its experiments report high-quality 5-second, 480p videos with motion controllability on par with Kling 1.5 Pro while introducing the MoveBench benchmark. Prolonged trajectory occlusion remains a control limitation.
Problem
Existing motion-controllable video methods suffer from coarse control granularity and limited scalability, motivating more precise and practical motion control.
Method
Wan-Move projects dense point trajectories into latent space and propagates first-frame features along them to update image condition features without auxiliary motion modules or architectural changes.
Results
Wan-Move generates high-quality 5-second, 480p videos with motion controllability on par with Kling 1.5 Pro, with extensive experiments on MoveBench and public datasets.
Takeaways & Limitations
MoveBench provides diverse, long-duration, hybrid-verified motion annotations, while Wan-Move offers scalable motion control through fine-tuning existing I2V models.
Takeaways & Limitations
Prolonged point-track absence caused by occlusion may lead to loss of control, although short-term occlusions can be recovered after reappearance.
Abstract
from arXiv · showhide
We present Wan-Move, a simple and scalable framework that brings motion control to video generative models. Existing motion-controllable methods typically suffer from coarse control granularity and limited scalability, leaving their outputs insufficient for practical use. We narrow this gap by achieving precise and high-quality motion control. Our core idea is to directly make the original condition features motion-aware for guiding video synthesis. To this end, we first represent object motions with dense point trajectories, allowing fine-grained control over the scene. We then project these trajectories into latent space and propagate the first frame's features along each trajectory, producing an aligned spatiotemporal feature map that tells how each scene element should move. This feature map serves as the updated latent condition, which is naturally integrated into the off-the-shelf image-to-video model, e.g., Wan-I2V-14B, as motion guidance without any architecture change. It removes the need for auxiliary motion encoders and makes fine-tuning base models easily scalable. Through scaled training, Wan-Move generates 5-second, 480p videos whose motion controllability rivals Kling 1.5 Pro's commercial Motion Brush, as indicated by user studies. To support comprehensive evaluation, we further design MoveBench, a rigorously curated benchmark featuring diverse content categories and hybrid-verified annotations. It is distinguished by larger data volume, longer video durations, and high-quality motion annotations. Extensive experiments on MoveBench and the public dataset consistently show Wan-Move's superior motion quality. Code, models, and benchmark data are made publicly available.
1 Introduction
Wan-Move addresses coarse, hard-to-scale motion control by editing image condition features with dense trajectory guidance, while MoveBench enables broad evaluation. Experiments report diverse motion-control tasks and commercial-grade results with scaled training.
- 1 Introduction: Wan-Move injects motion guidance by editing image condition features, adding no auxiliary motion-processing modules and enabling scalable fine-tuning of existing I2V backbones.The framework updates the latent guide with both appearance and motion information for video generation.
- 1 Introduction: Dense point trajectories provide fine-grained local and global motion control, while latent feature replication propagates first-frame context along each trajectory.Trajectories are transferred from pixel space into latent coordinates before copied first-frame features guide later frames.
- 1 Introduction: MoveBench contains over 1,000 five-second cases across 54 content categories, with point trajectories and segmentation masks verified through human labeling and SAM predictions.The benchmark is designed to cover broad scenarios, long-range dynamics, and multiple-object motion.
- 1 Introduction: Extensive experiments on MoveBench and public datasets show that Wan-Move supports diverse motion-control tasks and delivers commercial-grade results with scaled training.The reported evaluation covers both the introduced benchmark and public datasets.
2 Related Work
Prior motion-controllable video methods use varied architectures and guidance signals, but fine-grained control, scalability, and evaluation coverage remain constrained. Wan-Move instead updates native I2V condition features without auxiliary modules, alongside a broader benchmark design.
- 2 Related Work: Video-generation research has progressed from 3D U-Net diffusion models toward architectures intended to model longer sequences and spatiotemporal dependencies.The passage contrasts CNN-based limitations in long-range spatiotemporal modeling with subsequent architectural advances.
- 2 Related Work: Training-free methods can degrade on fine-grained or multi-object motion, while fine-tuning methods introduce diverse motion signals and integration techniques.The related-work landscape includes noisy-latent optimization, attention manipulation, and motion-conditioned fine-tuning.
- 2 Related Work: Wan-Move uses native I2V features to mitigate limitations of auxiliary motion modules, identity embeddings, and non-native representations.The approach avoids additional modules such as identity embeddings while operating within the I2V foundation model.
- 2 Related Work: Existing motion-control benchmarks are often small, task-specific, short, or sparsely annotated, limiting evaluation of long-range dynamics and multi-object interactions.DAVIS and VIPSeg are cited as examples of short clips with sparse annotations, while MagicBench categorizes samples mainly by object count.
3 Method
Wan-Move injects point-trajectory motion guidance directly into latent image-condition features, avoiding auxiliary motion modules while retaining the original I2V pipeline. Training uses curated videos and sampled trajectories, and inference combines text, image, and trajectory conditions.
- 3.2 Latent Trajectory Guidance: Wan-Move converts point trajectories into latent coordinates and propagates first-frame features along them to create motion-aware condition features.The latent positions are spatially mapped and temporally averaged, then first-frame features are replicated at subsequent trajectory locations.
- 3.2 Latent Trajectory Guidance: The method injects motion by editing the existing image condition feature, eliminating explicit motion-condition features, injection modules, and auxiliary motion encoders.This design supports fine-grained trajectory control without changing the I2V architecture.
- 3.3 Training and Inference: Training samples dense 32×32 point tracks and randomly selects 1–200 trajectories in 95% of iterations, while dropping motion conditions in 5% to preserve I2V capability.Selected first-frame features are replicated into subsequent zero-padded frames.
- 3.3 Training and Inference: At inference, Wan-Move conditions generation on a text prompt, first-frame image, and sparse or dense point trajectories.Text and image context are encoded separately, while the VAE-derived image feature receives latent feature replication before generation.
4 MoveBench
MoveBench is a large, curated benchmark for motion-controllable video generation with 1,018 five-second, 480×832 videos and precise annotations. Its reported comparisons indicate that Wan-Move improves both visual fidelity and motion quality across all listed metrics.
- 4 MoveBench: MoveBench contains 1,018 videos at 480×832 resolution and five-second duration for comprehensive motion-control evaluation.The videos are selected from a free-license source and address the small scale and short duration of prior benchmarks.
- 4 MoveBench: The benchmark uses a four-stage curation pipeline, 54 content categories, and 15–25 representative examples per category to balance diversity and quality.Videos are quality-filtered, cropped to 480p, uniformly sampled to 81 frames, and categorized before manual selection.
- 4 MoveBench: Wan-Move consistently yields substantial improvements in visual fidelity and motion quality across all metrics on MoveBench and DAVIS.Table 1 reports performance comparisons against recent motion-controllable methods.
5 Experiment
Wan-Move is evaluated on single- and multi-object motion, guidance strategies, scaling, and generalization, with comparisons against prior methods and commercial-quality benchmarks. Across these experiments, it delivers precise motion control and competitive video quality while retaining scalable training and broad application coverage.
- 5.2 Main Results: Wan-Move achieves the best single-object motion-control performance among compared methods, attaining the lowest EPE on MoveBench and DAVIS.The comparison includes ImageConductor, Levitor, MagicMotion, and Tora, with the latter methods reporting similar EPE values of 3.2–3.4.
- 5.2 Main Results: Wan-Move achieves significantly lower FVD and EPE than other methods for multi-object motion control on 192 MoveBench cases.The result indicates precise adherence to motion constraints in more complex multi-object scenarios.
- 5.2 Main Results: In the human study, Wan-Move exceeds 96% win rates against Tora across motion accuracy, motion quality, and visual quality.The evaluation uses 50 samples per method and 20 participants; the supplied passage also begins a comparison with Kling 1.5 Pro.
- 5.3 Ablation Study: Latent feature replication produces the highest PSNR of 17.8 and lowest EPE of 2.6 among tested guidance strategies.Pixel replication yields EPE 3.7 and generation failures, while random track embeddings weaken quality and produce stiff motion near tracked points.
- 5.3 Ablation Study: Simple condition concatenation matches ControlNet’s motion-controllable performance while avoiding its additional 225 seconds of inference latency.This supports Wan-Move’s architecture-free integration strategy.
- 5.3 Ablation Study: Performance improves as training tracks increase to N=200, but further increases worsen EPE because dense training tracks mismatch sparse evaluation tracks.The optimal setting jointly achieves the best reported SSIM and EPE.
- 5.3 Ablation Study: Wan-Move outperforms MagicMotion and Tora under matched CogVideoX1.5-5B-I2V backbones and training-data scales.The comparison uses variants trained on 23K and 630K samples and reports results on MoveBench.
- 5.4 Applications: Point trajectories support object, multi-object, camera, and other motion-control applications across diverse scenarios.Camera control can use directly dragged background points or point clouds projected along camera pose trajectories.
6 Conclusion and Discussion
Wan-Move combines latent feature replication with point-trajectory guidance to provide precise, scalable motion control without architectural changes. Its experiments support 5-second, 480p generation with motion controllability comparable to Kling 1.5 Pro, while prolonged occlusion remains a limitation.
- Conclusion: Wan-Move generates high-quality 5-second, 480p videos with motion controllability on par with Kling 1.5 Pro’s Motion Brush.The framework uses latent-coordinate trajectory guidance and first-frame feature replication without an extra motion encoder or architectural change.
- Limitations and broader impacts: Prolonged trajectory absence can cause loss of motion control, although short-term occlusions may recover when points reappear.The method also has dual-use potential because it produces realistic, controllable videos.
8 Implementation Details
The implementation builds MoveBench through large-scale dataset filtering, categorization, and interactive trajectory annotation. Its training and annotation pipeline emphasizes motion-aware captions, reviewed masks, and scalable model execution.
- Dataset construction: The benchmark’s training-dataset composition is documented across sources including Panda70M, Pixabay, Pexels, and YouTube.Pexels videos are separated from MoveBench videos to prevent data leakage.
- Dataset processing: Training videos are captioned with a Qwen2.5-VL prompt that emphasizes motion and camera attributes while preserving scene descriptions.The same captioning prompt is applied to MoveBench videos to support semantic context and physically plausible motion generation.
- Benchmark construction: MoveBench organizes filtered videos into 54 content categories using sampled-frame SigLip features, clustering, and manual selection.Category labels are automatically captioned with Qwen2.5-VL before representative videos are selected.
- Interactive labeling: Interactive annotation lets users select motion points, review SAM-generated masks, exclude irrelevant regions with negative points, and assign object IDs to multiple trajectories.The interface is designed to isolate articulated motions and small objects for precise annotation.
- Training details: Wan-Move uses Wan-I2V-14B with FSDP-wrapped DiT and umT5 components, bfloat16 parameters, AdamW optimization, and 30,000 training steps.Inference follows Wan’s sampling scheme with 50 sampling steps.
9 Additional Experiments
Additional experiments examine trajectory overlap, training-sampling strategies, temporary invisibility, and failure cases. The results support robust handling of temporary disappearance, while revealing degradation from prolonged disappearance, crowded scenes, implausible trajectories, and tracking errors.
- Trajectory overlap: Randomly selecting one first-frame feature for overlapping trajectories yields lower FVD and EPE than feature averaging.The authors hypothesize that averaging overlapping features causes information loss.
- Training strategies: Directly sampling 1–200 point trajectories achieves comparable EPE and lower FVD than dense-to-sampling training.This simplifies training while retaining generalization across trajectory counts.
- Temporary invisibility: Wan-Move maintains stable generation quality when trajectories are temporarily invisible.The authors attribute this capability to similar training cases and inherent model generalization.
- Failure cases: Persistent trajectory disappearance causes control degradation, while crowded scenes, physically invalid trajectories, and erroneous tracking points produce additional failures.These conditions can lead to visual artifacts or out-of-distribution predictions.
10 Qualitative Visualizations
Wan-Move supports accurate motion transfer, camera control, video motion copying, and 3D object rotation through trajectory-based conditioning. Qualitative comparisons indicate strong motion accuracy, visual quality, appearance preservation, and object-level alignment.
- More Motion Transfer Results: Wan-Move achieves nearly identical appearance quality and motion alignment to original videos under dense trajectory conditions and the same first frame.The implementation uses 1,024 dense point trajectories for these motion-transfer visualizations.
- Qualitative Comparisons: Wan-Move demonstrates superior motion accuracy and visual quality in qualitative comparisons with Tora and Kling 1.5 Pro.Major motion-control failures or visual artifacts are marked with red boxes in the comparison figure.
- 3D Object Rotation: Wan-Move supports 3D object rotation by transforming depth-based 3D positions and reprojecting them into 2D trajectory conditions.The resulting trajectories serve as conditioning inputs for rotating objects in generated videos.
- Camera Control: Different point trajectories enable flexible camera control, including linear displacement, dolly-in, and dolly-out movements.Camera-aligned trajectories are obtained by projecting a depth-derived point cloud along a predefined camera path and applying z-buffering.
- More Motion Transfer Results: Dense point trajectories preserve high-fidelity appearance and object-level motion alignment even under complex environmental conditions.The demonstrated motion-copying setup uses examples such as 1,024-point trajectories.