Source-linked AI summary
Tora: Trajectory-oriented Diffusion Transformer for Video Generation
Zhenghao Zhang, Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, Weizhi Wang
TL;DR
Existing video-generation methods provide limited evidence for scalable, controllable motion, especially in longer sequences. Tora integrates trajectory processing and motion fusion into a DiT-based framework, and experiments report stronger motion fidelity, including 3–5 times higher trajectory accuracy and approximately 30–40% better FVD in 128-frame tests.
Problem
Existing controllable-video methods show limitations on longer sequences, motivating scalable video generation with robust motion control.
Method
Tora uses a Trajectory Extractor to encode trajectories as hierarchical spacetime motion patches and a Motion-guidance Fuser to integrate them into DiT blocks.
Results
Tora achieves 3-5 times higher trajectory accuracy and approximately 30-40% better FVD than UNet-based methods in 128-frame tests.
Takeaways & Limitations
Tora generates trajectory-adherent videos across diverse durations, aspect ratios, and resolutions, including up to 204 frames at 720p.
Abstract
from arXiv · showhide
Recent advancements in Diffusion Transformer (DiT) have demonstrated remarkable proficiency in producing high-quality video content. Nonetheless, the potential of transformer-based diffusion models for effectively generating videos with controllable motion remains an area of limited exploration. This paper introduces Tora, the first trajectory-oriented DiT framework that concurrently integrates textual, visual, and trajectory conditions, thereby enabling scalable video generation with effective motion guidance. Specifically, Tora consists of a Trajectory Extractor (TE), a Spatial-Temporal DiT, and a Motion-guidance Fuser (MGF). The TE encodes arbitrary trajectories into hierarchical spacetime motion patches with a 3D motion compression network. The MGF integrates the motion patches into the DiT blocks to generate consistent videos that accurately follow designated trajectories. Our design aligns seamlessly with DiT's scalability, allowing precise control of video content's dynamics with diverse durations, aspect ratios, and resolutions. Extensive experiments demonstrate that Tora excels in achieving high motion fidelity compared to the foundational DiT model, while also accurately simulating the complex movements of the physical world. Code is made available at https://github.com/alibaba/Tora .
1 Alibaba Cloud Computing 2 Fudan University
Tora demonstrates video generation guided by trajectories, images, and text, including whimsical scenes such as a floating teacup and a sunglasses-wearing fox.
- The examples include a delicate porcelain teacup floating upward on a linen tablecloth.
- Another example depicts a stylish fox wearing sunglasses walking across a candy-filled Mars landscape.
- Tora supports video guidance from arbitrary trajectories, images, texts, or combinations thereof.
1. Introduction
The introduction identifies a need for longer videos with robust motion control and presents Tora as a scalable DiT solution integrating trajectories with text and images. Its reported contributions include flexible generation and strong motion-control performance.
- UNet-based controllable-video methods are restricted to 16 frames at fixed, lower resolution, producing distortions during substantial positional shifts.
- Tora integrates text, images, and trajectories in a DiT model for scalable video generation with robust motion control.
- The proposed trajectory extractor converts arbitrary trajectories into hierarchical spacetime motion patches, while the motion-guidance fuser integrates them within DiT blocks.
- Training uses high-quality video-text pairs, trajectory extraction, motion filtering, and adapter-like training of temporal blocks with the motion modules.
- Tora enables motion-manipulable videos across varying aspect ratios, extending to 204 frames and 720p resolution.
- Experiments report state-of-the-art accuracy in controlling object motions and superiority in simulating physical-world movements.
2. Related Work
Related work covers video diffusion and motion-control methods, while emphasizing persistent long-sequence artifacts and Tora’s integration of trajectories into a DiT architecture.
- Video diffusion models commonly add temporal convolutional and attention layers to pretrained image diffusion models but face capacity and scalability limits for long videos.
- Motion-control methods use reference videos, depth maps, sketches, motion vectors, trajectories, or bounding information to guide generated videos.
- Its Trajectory Extractor embeds trajectory vectors into the video-patch latent space and extracts hierarchical motion features.
- Existing methods produce noticeable motion-consistency and visual-presentation artifacts on longer sequences.
- Tora integrates trajectories into the DiT architecture to enable closer adherence to physical-world motion.
3. Methodology
Tora combines a Spatial-Temporal DiT with trajectory extraction and motion-guidance fusion to support scalable, trajectory-controlled video generation. Its method converts trajectories into motion patches aligned with video latents and evaluates multiple fusion designs for integrating them into DiT blocks.
- Spatial-Temporal DiT: Tora uses OpenSora’s Spatial-Temporal DiT, whose alternating spatial and temporal blocks process variable-length video sequences.The architecture reduces video dimensions with a video autoencoder, patchifies the latent representation, and applies spatial and temporal attention.
- Trajectory Extractor: The Trajectory Extractor converts framewise spatial trajectories into trajectory maps and spacetime motion patches in the video latent space.The trajectory representation begins with positions and offsets, then uses matching patch sizes and convolutional layers to produce motion patches.
- Motion-guidance Fuser: Tora integrates trajectory conditioning through three Motion-guidance Fuser designs: extra-channel connections, adaptive normalization, and cross-attention.These variants inject motion patches into each Spatial-Temporal DiT block using concatenation, scale-and-shift modulation, or attention.
- Motion-guidance Fuser: Adaptive normalization provides the best performance and computational efficiency among the evaluated fusion architectures.The paper therefore uses the adaptive norm layer as the default Motion-guidance Fuser unless otherwise specified.
- Data Processing: The method uses structured video filtering and motion-focused processing to construct training data with consistent object motion.Processing includes scene-based clipping, removal of invalid or low-quality videos, and filtering of camera-dominated motion.
4. Experiments
Experiments show that Tora maintains stronger trajectory control and visual quality than UNet-based and OpenSora-based alternatives, especially for longer videos. Ablations identify 3D motion-VAE compression, adaptive-norm fusion, and scaling as important design choices.
- Results: Tora achieves 3-5 times higher trajectory accuracy and approximately 30-40% better FVD than UNet-based methods in 128-frame tests.UNet-based methods show increasing deviations with longer sequences, producing motion blur and object deformation.
- Results: Tora improves motion control and visual performance over OpenSora at most settings, while suppressing temporal artifacts and motion blur.OpenSora produces random object trajectories, whereas OpenSora-based DragNUWA suffers from motion-representation incompatibility and about 5% lower visual quality than baseline.
- Results: Trajectory error increases gradually with duration for Tora, unlike the larger deviations observed in UNet models across resolutions and durations.The results indicate that Tora maintains trajectory control over longer durations.
- Ablation study: The customized 3D motion VAE achieves the best trajectory-compression results among the evaluated methods.Mid-frame sampling is vulnerable to flow errors and patch dissimilarity, while average pooling dilutes trajectory direction and magnitude.
- Ablation study: Adaptive-norm fusion achieves the lowest FVD and Trajectory Error with the highest computational efficiency among the evaluated MGF variants.Its dynamic feature adaptation and temporal modulation support motion integration without strict alignment.
- Ablation study: Placing MGF in the Temporal DiT block reduces Trajectory Error from 23.39 to 14.25.The two-stage training strategy also improves adaptability to sparse and dense motion patterns.
- Scaling motion-control ability: Increasing model scale and training data improves motion control, confirming compatibility between the motion modules and DiT scaling.The study transfers the modules to 2B- and 5B-parameter CogVideoX architectures.
5. Conclusion
Tora is a trajectory-oriented DiT framework that encodes arbitrary trajectories into spacetime motion patches for controllable video generation. It supports high-quality, trajectory-adherent videos across diverse durations, aspect ratios, and resolutions.
- Conclusion: Tora encodes arbitrary trajectories into spacetime motion patches aligned with DiT scaling properties.The framework is designed to simulate physical-world movements more realistically.
- Conclusion: Tora generates trajectory-adherent videos across diverse durations, aspect ratios, and resolutions, including up to 204 frames at 720p.The paper presents this capability as evidence of versatility and robustness across motion patterns.
Appendix
The supplementary material provides additional results, dataset information, and analyses supporting the paper’s findings and conclusions.
- Appendix: The appendix contains qualitative results, a DiT-based user study, data-processing and dataset details, prompt refinement, and motion-VAE training information.It includes dataset quantity and duration details alongside additional analyses.
6. Qualitative Comparisons
Qualitative comparisons show that trajectory control improves motion coherence, object fidelity, and naturalness over text-only motion control. Among compression methods, the proposed 3D VAE best preserves relevant motion information and produces fluid, coherent movement.
- Comparison with OpenSora: Tora’s appropriate trajectories produce more coherent and stable motion than OpenSora under identical text and image conditions.The comparison emphasizes motion that simulates real-world physics.
- Comparison with OpenSora: Trajectory control improves video fluidity and preserves foreground-object fidelity during long, complex motions.The improvement addresses simultaneous object movement, swinging, and circling.
- Comparison with OpenSora: Tora maintains object shapes during oscillating, swirling, and multi-object motions where OpenSora shows deformation or flickering.Examples include a teddy bear, a rose, and two jellyfish moving in opposite directions.
- Trajectory compression methods: Key-frame sampling can misalign video and motion patches during rapid motion, reducing trajectory tracking accuracy and visual fluidity.Although it captures essential motion, the patch misalignment harms generated-object tracking.
- Trajectory compression methods: The proposed 3D VAE compresses trajectories into the video latent space and best preserves successive-frame motion, improving fluidity and coherence.It is trained on a large dataset with flow annotations to extract relevant motion features for guidance.
7. User Study with DiT-based methods
The user study compares Tora with four video-generation systems using pairwise judgments of physics simulation, sensory quality, and instruction adherence. Participants provide trajectories for Tora in response to text prompts, including multi-object and non-linear motions.
- User study: Ten human volunteers evaluate Tora, OpenSora-v1.2, CogVideoX-2B, Vidu, and Kling v1.0 using pairwise comparisons.The criteria are physics simulation, sensory quality, and instruction adherence.
- User study: Tora’s evaluation uses participant-drawn trajectories in response to the supplied text prompts.The prompts include upward and downward jellyfish motion, circular rose motion, and teddy-bear oscillation.
- Evaluation criteria: The comparison covers physics simulation, sensory quality, and instruction following across the evaluated systems.Table 5 summarizes win rates against OpenSora-v1.2, CogVideoX, Vidu, and Kling.
- Evaluation prompts: The qualitative prompts test distinct motion patterns, including opposing vertical movement, circular motion, and side-to-side oscillation.The prompt set includes descriptive scene context around each requested movement.
8. Data Pre-processing
Tora’s preprocessing pipeline filters videos for validity, resolution, camera stability, and suitable object motion before trajectory training. The resulting process constructs a high-quality training dataset, while trajectory compression methods are visually compared separately.
- Filtering workflow: The preprocessing workflow removes invalid videos and retains videos with aesthetic scores above 5.5 and flow scores above 3.Invalidity checks include decoding failures and zero-duration videos.
- Filtering workflow: Videos below 720p are removed to ensure adequate clarity and detail for subsequent study.Resolution filtering establishes 720p as the minimum standard.
- Filtering workflow: Camera-motion filtering removes scenes where camera movement could obscure primary-subject motion.The process uses a camera-motion detector and motion segmentor, with zoom thresholds between 0.4 and 0.6.
- Motion filtering: Foreground optical-flow analysis excludes mostly static scenes while retaining dramatic-motion videos with probability (1 − flow score/100).This balances insufficient motion against optical-flow deviations caused by extreme object movement.
- Outcome: The filtering and processing steps produce a high-quality video dataset for subsequent training.Figure 9 separately compares key-frame sampling, average pooling, and 3D VAE trajectory compression.
9. Dataset Details
The training dataset combines public and internal video sources, then supports multiscale, mixed-duration training through resolution-duration buckets. Dataset statistics describe its composition, while evaluation data is selected for pronounced object motion under mostly stable cameras.
- Training dataset: The dataset begins with 2.6M videos from Panda-70M, Mixart, and internal sources, yielding 631k eligible training videos after preprocessing.The source composition and filtered total are reported as dataset-level statistics.
- Training dataset: The dataset overview reports training-video durations, resolutions, FPS, frame counts, and caption lengths.Tables 6 and 7 summarize these statistical properties, while Figure 10 shows distribution information.
- Training strategy: Multiscale and mixed-duration training assigns videos to buckets defined by resolution and duration.Videos with different aspect ratios are assigned according to their bucket attributes.
- Training strategy: Random downsampling and bucket-specific batch sizes support training across scales while balancing GPU load.The bucket configurations are presented in Figure 10c.
- Evaluation dataset: The evaluation dataset uses object-segmentation videos filtered for predominantly stable cameras and pronounced object motion.This selection is intended to improve the reliability of motion-focused assessments.
10. Prompt Refinement
Tora refines testing prompts into detailed English descriptions and evaluates motion using trajectories derived from annotated objects. The supplied passages also describe motion-VAE training and reconstruction evaluation.
- 10. Prompt Refinement: Prompt refinement begins with a task description for converting simple user inputs into more suitable video-generation prompts.
- 10. Prompt Refinement: Evaluation videos use annotated object-mask centers as trajectory points, with the number of trajectories matching the number of annotated objects.
- 10. Prompt Refinement: The examples provide detailed descriptions of subjects, settings, appearance, lighting, focus, actions, and exclusions.
- 10. Prompt Refinement: Refined prompts should describe all visible objects, support video regeneration, contain no more than six sentences, and be written in English.
- 10. Prompt Refinement: GPT-4o processes testing captions after receiving the task description, producing captions aligned with the refinement objectives.
- 10. Prompt Refinement: The prompt-refinement procedure is illustrated with inputs including flying fish, fighting cats, a polar bear on the Great Wall, and other scenes.
- 11. Motion VAE Training: The motion-specific 3D VAE is refined because pretrained video VAEs may inadequately encode optical-flow features across domains.
- 11. Motion VAE Training: Motion-VAE reconstruction and controllability are evaluated with PSNR, SSIM, and Trajectory Error against a pure video VAE.