Source-linked AI summary
Distilling Dense Trajectory Teachers For Large-Motion Video World Models
Shuofeng Sun, Linqing Zhao, Haibin Yan
TL;DR
Video world models often struggle to follow sparse, large-step camera trajectories because training favors gradual viewpoint changes. DTTD uses dense-trajectory teacher observations, geometry-aware rewards, and a cumulative curriculum; the authors report improved camera following and cross-frame 3D consistency while maintaining visual quality.
Problem
Models trained on gradual viewpoint changes often produce nearly static videos, inaccurate camera following, or geometric drift on sparse trajectories with large pose-to-pose changes.
Source: §1 · p. 2Method
DTTD uses pose-aligned observations from frozen-model dense-trajectory videos, geometry-aware evaluation, and a cumulative curriculum to optimize generations on increasingly large camera motions.
Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 6Results
DTTD improves camera following and cross-frame 3D consistency while maintaining visual quality, without additional video data or an external generation teacher.
Source: §5 · p. 9Takeaways & Limitations
The framework transfers dense-trajectory capability to sparse, large-step motions while retaining visual quality.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Video world models are expected to generate videos that accurately follow specified camera trajectories while maintaining consistent 3D structures. However, existing models are predominantly trained on densely sampled videos with small inter-frame camera motions. When applied to sparse, large step trajectories, they often produce frozen frames, inaccurate camera motion, and severe geometric drift. We attribute this degradation to a mismatch between the gradual viewpoint transitions seen during training and the large viewpoint changes required at inference time. To address this problem, we propose Dense Trajectory Teacher Distillation (DTTD), a post-training framework that transfers a model's generation capability from dense trajectories to large camera motions. Given a sparse trajectory, DTTD interpolates additional camera poses to construct a dense trajectory covering the same path. The frozen base model then generates reliable teacher videos along this familiar dense trajectory, and pose aligned teacher observations are used to evaluate candidate videos generated under the original sparse trajectory. In this way, the model serves as its own generation teacher, requiring neither additional video data nor a stronger external video generator. We further combine dense teacher supervision with geometry aware rewards that assess cross frame consistency and camera trajectory following, and use the resulting preferences for offline preference optimization. To stably expand the model's motion range, we introduce a cumulative curriculum that progressively incorporates larger motion steps while retaining simpler trajectories to reduce capability forgetting.
1 INTRODUCTION
Sparse, large-step camera trajectories expose a mismatch with models trained on gradual viewpoint changes, while DTTD transfers dense-trajectory capability through teacher supervision and geometry-aware evaluation.
1 INTRODUCTION
- Camera-controllable world models must follow prescribed camera motion and preserve scene geometry for reliable environment rollouts and other applications.The introduction names embodied intelligence, autonomous driving, and interactive scene exploration as relevant settings.Source: §1 · p. 1
- Prior post-training methods use geometric rewards or vision-language feedback, but largely assume test trajectories remain near the training distribution.The paper identifies limited attention to camera-motion magnitude as a gap in this work.Source: §1 · p. 1
- Models trained on gradual viewpoint changes may generate nearly static videos, miss specified motion, or show geometric drift when adjacent camera poses are far apart.The paper attributes this degradation to trajectory-distribution mismatch and the need to infer newly visible content and perspective changes in one step.Source: §1 · p. 2
- DTTD uses frozen-model videos from a dense sampling of the same camera path as pose-aligned references for evaluating sparse-trajectory generations.A cumulative curriculum progressively adds larger motions while retaining simpler trajectories to mitigate capability forgetting.Source: §1 · p. 2
- Geometry-aware evaluation combines relative-trajectory error and cross-frame reprojection consistency to rank candidates and guide offline preference optimization.The signals are intended to identify incorrect motion, structural drift, inconsistent parallax, and deformation.Source: §1 · p. 2
- DTTD improves geometric consistency, camera following, and visual quality for both small-step and large-step trajectories with lightweight LoRA fine-tuning.The conclusion describes the combined framework as requiring neither additional video data nor an external generation teacher.Source: §1 · p. 3
2 RELATED WORKS
Related work connects video models’ temporal compression to dense-trajectory training and describes post-training alignment methods that address visual or geometric feedback.
2 RELATED WORKS
- Video diffusion models often compress spatial and temporal dimensions, relying on neighboring frames’ redundancy and thereby favoring densely sampled trajectories with small camera transformations.Large viewpoint changes reduce neighboring-frame redundancy and can fall outside the effective training distribution of temporally compressed models.Source: §2 · p. 3
- Video alignment methods include reward-based policy optimization and preference optimization, generally using feedback on visual quality, motion, or semantic consistency.Examples include DDPO [Black et al., 2024], Diffusion-DPO [Wallace et al., 2024], Flow-GRPO [Liu et al., 2026], and DanceGRPO [Xue et al., 2025].Source: §2 · p. 3
- Geometry-oriented alignment methods evaluate cross-frame consistency or camera motion using geometry models, epipolar constraints, or vision-language feedback.Examples are VideoGPA [Du et al., 2026], Epipolar-DPO [Kupyn et al., 2025], and World-R1 [Wang et al., 2026].Source: §2 · p. 3
3 METHOD
DTTD evaluates sparse-trajectory video candidates against dense-trajectory teacher observations and complementary geometry-based rewards. It uses the resulting preferences in a cumulative curriculum to expand the model’s motion range while retaining simpler trajectories.
3.1 PRELIMINARIES
- Flow-DPO uses preferred and dispreferred videos generated under the same condition, encouraging the trainable model to assign greater relative preference to the preferred video.The condition comprises the initial image and camera trajectory; preference strength is controlled by β.Source: §3.1 · pp. 3–4
3.2 DENSE TRAJECTORY TEACHERS.
- Trajectories with different motion magnitudes share a camera path; the frozen base model generates dense-trajectory teacher videos for evaluating student frames at corresponding poses.Depth and VAE-feature distances between student and teacher frames are averaged across shared poses and teacher videos to form the dense-trajectory reward.Source: §3.2 · pp. 4–6
- A cumulative motion curriculum progressively adds larger sampling intervals while retaining simpler trajectories, addressing scarce reliable preference pairs and the risk of forgetting simpler-motion capability.The stages use s=0.25, then s∈{0.25,0.5}, then s∈{0.25,0.5,1}.Source: §3.2 · p. 6
3.3 GEOMETRY FOUNDATION MODEL EVALUATION
- A frozen geometry model supplies complementary rewards: cross-frame reprojection consistency assesses structural consistency, while Sim(3)-aligned trajectory comparison assesses camera following.The trajectory reward identifies static generation and camera motions with incorrect directions or magnitudes.Source: §3.3 · p. 6
3.4 TRAINING
- DTTD constructs preference pairs by ranking candidate videos using dense-teacher, geometric-consistency, and trajectory-following rewards, then applies offline Flow-DPO updates to LoRA parameters.The highest- and lowest-scoring candidates are paired only when their reward gap exceeds a threshold.Source: §3.4 · p. 6
4 EXPERIMENTS
Experiments on ScanNet evaluate DTTD against the base model and preference-optimization baselines across trajectory sampling intervals. DTTD improves visual quality, camera following, and 3D consistency, while ablations show gains from dense-trajectory evaluation, geometry evaluation, and cumulative curriculum learning.
4.1 EXPERIMENT SETUP
- ScanNet preference pairs are constructed by downsampling original trajectories and interpolating dense trajectories, yielding 10K training pairs without requiring real dense trajectories.Methods are evaluated on the ScanNet test set.Source: §4.1 · p. 7
- The comparison uses Wan2.1-Fun-V1.1-1.3B-Control-Camera and applies preference optimization to the compared methods with the same data and number of training steps.DTTD uses a three-stage cumulative curriculum over s = 0.25, then s ∈ {0.25, 0.5}, then s ∈ {0.25, 0.5, 1}; only LoRA modules are updated.Source: §4.1 · p. 7, §4.2 · p. 8
4.2 MAIN RESULTS
- DTTD achieves the best overall VBench performance, with clearer gains at s = 0.5 and s = 1 in subject consistency, background consistency, aesthetic quality, and imaging quality.As sampling increases from s = 0.125 to s = 1, base-model subject consistency falls from 0.8701 to 0.8023, while DTTD falls from 0.9030 to 0.8851; DTTD's Dynamic Degree is slightly lower than some baselines.Source: §4.2 · p. 8
- DTTD achieves the best reported CamMC, CamMC Rot, CamMC Trans, and ATE rotation metrics across motion magnitudes, with larger advantages at larger motions.The comparison covers multiple motion magnitudes; the source describes ATE rotation as more stringent for following capability and ATE translation as more robust to monocular scale ambiguity.Source: §4.2 · p. 8
- DTTD has the best 3DCS and MVCS across motion magnitudes, while Epipolar-DPO leads the Epipolar metric and DTTD ranks second.Against the base model, DTTD reduces 3DCS by approximately 16.2%, 15.7%, and 15.7% at s = 0.125, 0.5, and 1, respectively; it reduces Epipolar error by approximately 34.1%, 14.6%, and 19.7%.Source: §4.2 · p. 8, Table 3 · p. 8
- Figure 3 visualizes DTTD generations at s = 0.125 in its first two rows and s = 1 in its last two rows.Figure 4 compares methods, with its first three rows at s = 0.125 and last three rows at s = 1.Source: §4.2 · pp. 8–9, Figure 3 · p. 8, Figure 4 · p. 9
4.3 ABLATION STUDIES
- Ablations show that dense-trajectory evaluation improves 3DCS and CamMC across motion intervals, geometry evaluation adds gains, and cumulative curriculum performs best overall.The curriculum particularly improves imaging quality at larger motions, while retaining simpler trajectories to mitigate forgetting.Source: §4.3 · p. 9, Table 4 · p. 9
5 CONCLUSION
DTTD transfers a video world model's dense-trajectory capability to sparse, large-step camera motions without additional video data or an external generation teacher. Combined with geometry-aware evaluation and cumulative curriculum learning, it improves camera following and cross-frame 3D consistency while maintaining visual quality.
5 CONCLUSION
- DTTD transfers a video world model's dense-trajectory capability to sparse, large-step camera motions without additional video data or an external generation teacher.The framework combines geometry-aware evaluation and cumulative curriculum learning to improve camera following and cross-frame 3D consistency while maintaining visual quality.Source: §5 · p. 9
A.1 MORE ABLATION STUDIES
The ablations find complementary contributions from dense-teacher depth and feature supervision, and from reprojection and relative-trajectory rewards. The cumulative curriculum performs best across motion magnitudes while retaining easier trajectories as larger motions are introduced.
Dense Trajectory Reward Ablation
- Combining depth and feature supervision achieves the best overall geometric and camera-following performance, with each term contributing complementary information.Depth supervision yields a larger 3DCS improvement, while feature supervision gives a stronger imaging-quality gain under larger motion.Source: §A.1 · p. 12
Geometry Reward Ablation
- Combining reprojection and relative-trajectory rewards achieves the best overall performance across motion magnitudes, with complementary geometric signals.Reprojection mainly improves 3DCS by suppressing structural drift and inconsistent parallax; relative-trajectory reward produces a larger CamMC gain.Source: §A.1 · p. 12
Curriculum Learning Ablation
- The cumulative curriculum achieves the best overall results across motion magnitudes by retaining easier trajectories while gradually introducing larger motions.Direct training on the largest interval performs poorly on dense trajectories, while non-cumulative training loses some performance after discarding earlier, easier trajectories.Source: §A.1 · pp. 12–13