Source-linked AI summary
TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models
Mark YU, Wenbo Hu, Jinbo Xing, Ying Shan
TL;DR
Monocular videos lack practical, reliable support for freely redirecting camera trajectories while preserving coherent dynamic content. TrajectoryCrafter separates geometric view transformation from stochastic generation, combines point-cloud and source-video conditioning, and uses hybrid data curation; evaluations report superior high-fidelity novel-trajectory synthesis and robust generalization.
Problem
Existing approaches struggle with plausible novel trajectories for dynamic monocular videos, often requiring synchronized multiview data or suffering occlusion and domain-gap limitations.
Method
TrajectoryCrafter decouples view transformation from content generation, combines point cloud renders and source videos in a dual-stream diffusion model, and trains on hybrid monocular and static multiview data.
Results
Quantitative and qualitative evaluations report superior high-fidelity novel-trajectory videos and robust generalization across diverse scenes.
Takeaways & Limitations
The framework enables user-directed camera-trajectory redirection from casual or AI-generated monocular footage while maintaining 4D consistency with the source video.
Takeaways & Limitations
The method struggles with very large-range trajectories, depends on depth-estimation accuracy, and incurs relatively high inference overhead from multi-step denoising.
Abstract
from arXiv · showhide
We present TrajectoryCrafter, a novel approach to redirect camera trajectories for monocular videos. By disentangling deterministic view transformations from stochastic content generation, our method achieves precise control over user-specified camera trajectories. We propose a novel dual-stream conditional video diffusion model that concurrently integrates point cloud renders and source videos as conditions, ensuring accurate view transformations and coherent 4D content generation. Instead of leveraging scarce multi-view videos, we curate a hybrid training dataset combining web-scale monocular videos with static multi-view datasets, by our innovative double-reprojection strategy, significantly fostering robust generalization across diverse scenes. Extensive evaluations on multi-view and large-scale monocular videos demonstrate the superior performance of our method.
1. Introduction
TrajectoryCrafter redirects camera trajectories in monocular videos by separating view transformation from content generation. It combines dual-stream conditioning with hybrid training data to produce high-fidelity, generalizable novel-trajectory videos.
- TrajectoryCrafter generates high-fidelity videos with user-defined camera trajectories from monocular inputs.
- The framework explicitly decouples deterministic view transformation from stochastic content generation for precise trajectory control.Dynamic point cloud renders model 3D transformations, while diffusion handles content generation.
- Its dual-stream conditional diffusion model integrates point cloud renders and source videos to support trajectory accuracy and coherent 4D generation.The point cloud branch supplies view transformations; newly introduced source-video layers address occlusions and geometric distortions.
- Quantitative and qualitative evaluations on synchronized multi-view and large-scale monocular datasets report superior performance and robust generalization across diverse scenes.Ablations also support the efficacy of the dual-stream conditioning and dataset curation strategies.
- The training strategy combines dynamic web-scale monocular videos with static multi-view resources to improve generalization and robustness across diverse scenes.
2. Related Work
Prior work reconstructs dynamic novel views mainly from synchronized multi-view videos or adapts generative models using synthetic or specialized data. These approaches face practical, realism, domain-gap, or free-view limitations for monocular dynamic videos.
- Dynamic novel-view reconstruction remains challenging because temporal variation makes 4D scene reconstruction harder than static scene synthesis.
- Most reconstruction approaches rely on synchronized multi-view videos, which remain inaccessible to typical users.
- Some monocular-video methods fine-tune on generated data, risking limited applicability to real-world environments.
- Synthetic multiview training can create domain disparities with real videos, while other methods lack free-view generation or require per-video LoRA adaptation.
3. Method
TrajectoryCrafter lifts monocular videos into dynamic point clouds for explicit trajectory control, then uses dual-stream diffusion to restore high-fidelity, 4D-consistent content. Double reprojection and static multi-view data expand training beyond scarce synchronized multiview videos.
- Video Diffusion Background: Video diffusion models progressively add noise to clean videos and iteratively denoise latent representations to generate final videos.DiT blocks process compressed video and text tokens, while a VAE decodes denoised tokens.
- View Transformation via Dynamic Point Cloud: The method estimates monocular depth and lifts each source-video frame into a dynamic point cloud before rendering views along a target camera trajectory.
- View Transformation via Dynamic Point Cloud: Point cloud renders accurately capture geometric relations and view transformations but contain holes from occlusions and out-of-frame regions.Rendered masks identify these missing regions and guide subsequent video generation.
- Dual-stream Conditional Video Diffusion: The dual-stream model uses point cloud renders and masks for trajectory control while Ref-DiT cross-attention transfers source-video appearance details into spatially misaligned view tokens.
- Dataset Curation and Training Scheme: Static multi-view datasets supply source-video, target-video, and point-cloud-render triplets for training reference-token transfer without relying exclusively on scarce multiview video collections.
- Dataset Curation and Training Scheme: Double reprojection transforms monocular videos into training pairs whose reprojected targets align with the original view while retaining occlusion holes like point cloud renders.The strategy uses randomly sampled view transformations and their inverses to create large-scale, diverse training data.
4. Experiments
TrajectoryCrafter is evaluated on synchronized multi-view and in-the-wild monocular videos, including quantitative comparisons, qualitative analyses, and ablations of conditioning and training data. Results show strong fidelity, trajectory alignment, source consistency, and generalization, while depth errors and very large trajectories remain limitations.
- Evaluation setup: The multi-view benchmark uses five refined iPhone scenes and evaluates synthesized videos with PSNR, SSIM, and LPIPS against a fixed-camera target.The source is a casually captured video; the second target is incomplete, and two scenes are discarded because of camera and LiDAR errors.
- Multi-view evaluation: TrajectoryCrafter consistently outperforms GCD, ViewCrafter, and Shape-of-motion across all reported metrics on the multi-view benchmark.The comparison includes diffusion-based generative baselines and a reconstruction-based 4D novel-view synthesis method.
- Qualitative comparison: For “zoom-in and orbit to the right,” GCD shows artifacts and motion inaccuracies, while ViewCrafter exhibits content drift and color discrepancies.TrajectoryCrafter instead achieves higher visual fidelity, better target-trajectory alignment, and stronger source-video consistency.
- Conditioning ablation: Ref-DiT blocks significantly improve novel-trajectory video quality over removing Ref-DiT or directly concatenating source videos with point-cloud renders.Direct concatenation is spatially misaligned with the target view, whereas the full model improves fidelity and content consistency across all reported metrics.
- Training-data ablation: Mixed training with dynamic monocular and static multi-view data improves meaningful occlusion content and motion consistency compared with removing either data type.The ablation also highlights differences in occlusions, geometric distortions, and motion consistency.
- Limitations: Very large-range trajectories, inaccurate monocular depth, and multi-step diffusion inference constrain the method’s scope or efficiency.The paper specifically identifies 360-degree viewpoints, propagated depth errors, and relatively high computational overhead as limitations.
5. Conclusion
TrajectoryCrafter redirects monocular videos along desired camera trajectories while maintaining 4D consistency. Its dual-stream conditioning and hybrid training corpus support accurate transformations, coherent generation, and generalization across diverse scenarios.
- Conclusion: TrajectoryCrafter regenerates monocular videos with desired camera trajectories while maintaining 4D consistency with the source.It is intended for casual captures and AI-generated footage.
- Conclusion: The dual-stream diffusion model integrates point-cloud renders and source videos to support accurate view transformations and coherent 4D content generation.The conditioning mechanism combines geometric and source-video information.
- Conclusion: A hybrid corpus of large-scale dynamic monocular videos and static multi-view datasets enhances generalization across diverse scenarios.This strategy avoids relying solely on scarce synchronized multi-view video datasets.
- Conclusion: Extensive evaluations confirm high-fidelity outputs with novel camera trajectories on multi-view and large-scale monocular videos.The conclusion reports performance across both benchmark types.