Source-linked AI summary
ATI: Any Trajectory Instruction for Controllable Video Generation
Angtian Wang, Haibin Huang, Jacob Zhiyuan Fang, Yiding Yang, Chongyang Ma
TL;DR
Precise control over motion remains challenging, while existing approaches often handle camera, object, and local motion with separate specialized designs. ATI represents these forms of motion as trajectories and injects them into pretrained video-model latents; the authors report better controllability and visual quality than prior modular methods and commercial systems.
Problem
Precise motion control is challenging, and existing approaches often handle camera movement, object translation, and local deformation through separate specialized modules.
Source: §1 · p. 1Method
ATI uses a motion injector to project user-specified trajectories for camera, object, and local motion into the latent space of pretrained image-to-video models.
Source: §1 · p. 2Results
Takeaways & Limitations
A shared trajectory-based interface simplifies the user workflow for specifying different forms of video motion control.
Source: §1 · p. 2ATI can struggle with very rapid point movements and trajectories that require an object to disassemble, sometimes producing unnatural distortions.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
We propose a unified framework for motion control in video generation that seamlessly integrates camera movement, object-level translation, and fine-grained local motion using trajectory-based inputs. In contrast to prior methods that address these motion types through separate modules or task-specific designs, our approach offers a cohesive solution by projecting user-defined trajectories into the latent space of pre-trained image-to-video generation models via a lightweight motion injector. Users can specify keypoints and their motion paths to control localized deformations, entire object motion, virtual camera dynamics, or combinations of these. The injected trajectory signals guide the generative process to produce temporally consistent and semantically aligned motion sequences. Our framework demonstrates superior performance across multiple video motion control tasks, including stylized motion effects (e.g., motion brushes), dynamic viewpoint changes, and precise local motion manipulation. Experiments show that our method provides significantly better controllability and visual quality compared to prior approaches and commercial solutions, while remaining broadly compatible with various state-of-the-art video generation backbones. Project page: https://anytraj.github.io/.
1. Introduction
ATI frames camera, object, and local motion as trajectories of selected scene points, addressing motion control that prior systems split across specialized modules [21] [26] [31] [32] [41]. A motion injector projects those paths into pretrained video-model latents, and the framework is reported to improve motion quality and control while supporting different backbones.
1. Introduction
- The Figure 2 pipeline takes an image and trajectories, injects point-wise paths into latent conditioning, then decodes video from a DiT-denoised latent.Source: Figure 2 · p. 2
- Prior approaches often use separate specialized modules for camera movement, object translation, and local deformation, fragmenting workflows and limiting creative expression [21] [26] [31] [32] [41].Source: §1 · p. 2
- ATI unifies camera movement, object-level motion, and local deformation by representing each as trajectories of user-selected key points.Source: §1 · p. 2
- A motion injector projects user-specified trajectories into the latent space of pretrained image-to-video models, without requiring base-model retraining [24] [28].Source: §1 · p. 2
- ATI is reported to improve control precision and visual quality over academic methods and commercial products, with compatibility across different video-generation backbones.Source: §1 · p. 2
2. Related Work
Related work covers camera-path conditioning, object-motion control, and methods that jointly control camera and object motion. These approaches use varied representations, including scene reconstruction, motion fields, boxes, depth, and trajectory-aware modules.
2. Related Work
- Camera-control methods encode paths with Plücker coordinates [42] or use single-image 3D reconstruction to guide viewpoint generation [40] [7].Source: §2 · p. 2
- Object-motion approaches use sparse or dense optical flow [38] [16] [33] [27], learned dense motion fields [21] [25], bounding boxes [30], or depth and clustering [29].Source: §2 · pp. 2–3
- Interactive or trajectory-aware modules extend object-motion control [20] [19], while a training-free approach separates “out-of-place” and “in-place” animation.Source: §2 · p. 3
- Joint camera-and-object control is addressed by explicit concurrent-control modules [31], motion tracks [8], a 3D-aware representation [5], and unified disentangled control [43].Source: §2 · p. 3
3. Method
ATI represents camera, object, and local motion as trajectories of selected points and injects trajectory signals into a pretrained image-to-video model. Its method combines Gaussian feature-level guidance with tail dropout training to reduce distortions when trajectories end early.
3. Method
- ATI unifies camera, object-level, and local motion control by encoding point trajectories into the latent space of a pretrained image-to-video diffusion model.The framework uses a Gaussian-based motion injector to provide continuous control over object and camera dynamics.Source: §3 · p. 3
3.1. Conditional Video Generation Models
- Conditional video generation models progressively add Gaussian noise to videos, then learn to reverse the process by predicting that noise given a conditioning signal.The report uses Seaweed-7B and Wan2.1-14B as base models, with denoising implemented using a DiT architecture.Source: §3.1 · p. 3
3.2. Gaussian Model for Trajectory Instruction of Feature
- The method encodes the input image with a VAE, bilinearly samples a feature at each trajectory’s initial position, then spreads it across neighboring latent pixels with Gaussian masks.The masks are centered at trajectory locations in subsequent frames and provide fine-grained control to the generator.Source: §3.2 · p. 3
3.3. Tail Dropout Regularization
- Tail dropout reduces distortions and unintended occluders at early-ending trajectories by truncating training trajectories and teaching that missing future points need not imply occlusion.During training, a dropout frame is sampled with probability 0.2, after which point visibility is set to zero.Source: §3.3 · p. 4
3.4. Data Collection
- Training data comes from 2.4 million selected clips, after processing 5 million high-quality clips and retaining those with strong object motion.TAP-Net tracks 120 uniformly sampled points per clip; training randomly selects 1 to 20 points for each video.Source: §3.4 · p. 4
4. Experiments
ATI is integrated with two video-generation backbones and evaluated through qualitative examples and a trajectory-following test. The examples show camera, object, and combined motion control, alongside cases where trajectory following is less reliable.
4.1. Implementation Details
- ATI is integrated with Seaweed-7B and Wan2.1-14B by injecting trajectory instructions before patchification and fine-tuning all DiT parameters for 50,000 iterations.Both models reach satisfactory trajectory following after 15,000 iterations; five-second 480p generation takes approximately 8 GPU-minutes.Source: §4.1 · p. 4
- For Wan2.1, black frames after the initial image provide VAE features that are blended with trajectory-instruction features.Source: §4.1 · pp. 4–5
4.2. Qualitative Results
- Radially outward point motion creates a smooth zoom-in; adding horizontal translation targets a region, while anchoring the subject can reproduce a dolly-zoom.Source: §4.2 · p. 6
- Combined camera and object trajectories can be created by editing camera paths for object motion or applying camera transformations to object paths.Source: §4.2 · p. 6
- Seaweed ATI shows slightly better trajectory-instruction following, while Wan2.1 ATI shows richer motion at unconstrained locations.The authors suggest the difference may relate to how the input latent is zeroconditioned.Source: §4.2 · p. 6
- Very rapid point motion can hinder trajectory following, while trajectories requiring object disassembly can cause failures or unnatural distortions.Source: §4.2 · p. 6
- ATI handles intersecting trajectories and can satisfy instructions through alternative realistic solutions, such as rotating the camera instead of deforming an object implausibly.Source: §4.2 · p. 7
4.3. Quantitative Results
- The quantitative test uses 100 image–trajectory pairs and evaluates tracked points with Acc@0.01, Acc@0.05, and Appearance Rate.TAP-Net tracks points from first-frame inputs; the accuracy thresholds are fractions of the image diagonal, and Appearance Rate measures visibility prediction when a trajectory is present.Source: §4.3 · p. 7
5. Conclusion
ATI combines camera, object, and local motion control in one latent-space injection module. The conclusion reports improved controllability and visual quality over prior modular methods and commercial systems, and identifies more physically and instruction-aligned object motion as future work.
5. Conclusion
- ATI unifies camera movement, object translation, and fine-grained local motion through a single latent-space injection module.Source: §5 · p. 7
- The conclusion reports that ATI outperforms prior modular methods and commercial systems in controllability and visual quality.Source: §5 · p. 7
- Future work will improve object motion to better follow real-world physics and user inputs.Source: §5 · p. 8