Source-linked AI summary
DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos
Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, Ying Shan
TL;DR
Open-world video depth estimation must handle diverse appearance, motion, camera movement, and video length without relying on camera poses or optical flow. DepthCrafter trains a video-to-depth diffusion model with three stages and realistic and synthetic paired data, then uses segment-wise inference and stitching for long videos. It achieves state-of-the-art zero-shot performance and supports downstream depth-based visual effects and conditional video generation.
Problem
Open-world video depth estimation requires temporally consistent, detailed depth across diverse content, motion, camera movement, and lengths without supplementary information.
Method
DepthCrafter trains a video-to-depth model from a pre-trained image-to-video diffusion model using three-stage training, paired realistic and synthetic datasets, and segment-wise inference with seamless stitching.
Results
DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings across diverse datasets.
Takeaways & Limitations
DepthCrafter generates temporally consistent depth for videos from static images to extremely long sequences and facilitates depth-based visual effects and conditional video generation.
Takeaways & Limitations
The method has relatively high computation and memory consumption, requiring around 24GB GPU memory for 110-frame segments at 1024 × 576 resolution.
Abstract
from arXiv · showhide
Estimating video depth in open-world scenarios is challenging due to the diversity of videos in appearance, content motion, camera movement, and length. We present DepthCrafter, an innovative method for generating temporally consistent long depth sequences with intricate details for open-world videos, without requiring any supplementary information such as camera poses or optical flow. The generalization ability to open-world videos is achieved by training the video-to-depth model from a pre-trained image-to-video diffusion model, through our meticulously designed three-stage training strategy. Our training approach enables the model to generate depth sequences with variable lengths at one time, up to 110 frames, and harvest both precise depth details and rich content diversity from realistic and synthetic datasets. We also propose an inference strategy that can process extremely long videos through segment-wise estimation and seamless stitching. Comprehensive evaluations on multiple datasets reveal that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings. Furthermore, DepthCrafter facilitates various downstream applications, including depth-based visual effects and conditional video generation.
1. Introduction
DepthCrafter addresses open-world video depth estimation by generating temporally consistent, detailed depth sequences without supplementary information. It combines a video diffusion model, paired realistic and synthetic datasets, long-context training, and inference for extremely long videos, achieving state-of-the-art zero-shot performance.
- Monocular depth estimation is important for mixed reality, AI-generated content, autonomous driving, and robotics, but single views cannot uniquely determine scene depth.
- Image-based depth models applied directly to videos produce temporal flickering, while existing video methods are sensitive to dynamic content and camera-pose quality.
- DepthCrafter uses a video diffusion model trained from a pre-trained image-to-video model to estimate detailed, temporally consistent depth for diverse open-world videos without additional information.
- Its three-stage training combines realistic datasets for content diversity with synthetic datasets for precise depth details and supports variable temporal context up to 110 frames.
- Zero-shot evaluations across indoor, outdoor, static, dynamic, realistic, and synthetic datasets show state-of-the-art performance, while downstream applications include depth-based visual effects and conditional video generation.
2. Related Work
Prior work includes image depth estimation, video depth estimation, and video diffusion models. DepthCrafter builds on video diffusion to estimate consistent depth sequences without the camera poses or optical flow often required by existing approaches.
- Image depth estimation: Image depth methods predict depth from single images, but their generalization to diverse open-world scenes is limited by restricted training data.
- Video depth estimation: Video depth methods use either test-time optimization or feed-forward prediction, with optimization approaches typically requiring camera poses or optical flow.
- Video diffusion models: DepthCrafter formulates video depth estimation as conditional diffusion generation and trains a video-to-depth model from a pre-trained image-to-video diffusion model.
- Video diffusion models: Video diffusion models provide the prior for high-fidelity consistent video depth estimation in DepthCrafter.
3. Method
DepthCrafter formulates video depth estimation as conditional diffusion generation, using paired realistic and synthetic data, staged training, and segment-wise inference to produce temporally consistent depth for videos of varied lengths.
- Formulation: DepthCrafter estimates temporally consistent depth sequences from open-world videos despite variation in appearance, motion, camera movement, and length.The formulation targets d conditioned on video v without requiring supplementary camera poses or optical flow.
- Formulation: The model is trained as a video-to-depth diffusion system initialized from a pre-trained image-to-video diffusion model.The denoiser progressively recovers clean depth data from noisy latent representations under video conditioning.
- Latent representation: DepthCrafter uses low-dimensional latent diffusion with shared affine-invariant scale and shift across frames to preserve temporal consistency.A VAE encodes videos or depth sequences into latents; depth sequences are replicated to three channels for encoding and averaged after decoding.
- Video conditioning: Video conditioning adapts SVD by injecting CLIP embeddings frame by frame through cross-attention, aligning generated depth with video content.The adapted mechanism provides the denoiser with semantic information from the video frames.
- Dataset construction: Training combines approximately 200K realistic paired sequences with approximately 3K fine-grained synthetic annotations spanning diverse scenes and motions.The realistic sequences are 50–200 frames long, while the synthetic sequences are 150 frames long.
- Long-video inference: The training strategy samples sequence lengths up to 110 frames, while inference handles longer videos by overlapping segments, noise initialization, and latent interpolation.Previous-segment denoised latents anchor overlapping-frame depth scale and shift before consecutive segments are stitched.
4. Experiments
DepthCrafter is evaluated across diverse zero-shot depth-estimation settings, with quantitative, qualitative, ablation, application, and limitation analyses. Results show strong temporal consistency, detail preservation, generalization across lengths and scenes, and downstream utility, alongside substantial computational demands.
- Evaluation setup: The evaluation spans synthetic and realistic, indoor and outdoor, static and dynamic videos, plus single-image and in-the-wild settings.Testing uses four video datasets, a single-image dataset, DAVIS, and in-the-wild videos under zero-shot conditions.
- Quantitative results: DepthCrafter achieves state-of-the-art performance across four video datasets, outperforming representative single-image and video depth methods.Comparisons include Marigold, Depth-Anything, Depth-Anything-V2, NVDS, and ChronoDepth.
- Qualitative results: Qualitative profiles show temporally consistent, fine-grained depth across long open-world videos, while NVDS and Depth-Anything exhibit zigzag flickering artifacts.Examples include sequences ranging from 90 to 195 frames.
- Ablation studies: The full inference strategy produces smooth depth in both static and dynamic regions, whereas baseline variants retain flickering or jagged temporal profiles.Initialization alleviates artifacts in static regions but does not remove jaggies in dynamic regions without stitching.
- Applications and limitations: DepthCrafter supports fog effects and depth-conditioned video generation, but its large model and iterative denoising require relatively high computation and memory.Inference uses 465.84 ms per frame and around 24GB GPU memory for 1024 × 576 video.
5. Conclusion
DepthCrafter uses video diffusion models to estimate temporally consistent, detailed depth across diverse open-world videos without additional information. It supports variable-length inputs and downstream depth-based applications.
- DepthCrafter leverages video diffusion models for open-world video depth estimation without additional information.
- The method generates temporally consistent depth sequences with fine-grained details across diverse content, motion, and camera movement.
- DepthCrafter supports variable-length videos, from one-frame static images to extremely long videos.
- Extensive zero-shot evaluations demonstrate state-of-the-art performance in open-world video depth estimation.
- The method facilitates depth-based visual effects and conditional video generation.
for Open-world Videos
The supplementary material provides implementation details and additional evaluations, while directing readers to interactive visual results and promising released code and models.
- Additional implementation details and evaluations are provided in the supplementary material.
- Interactive videos on the project webpage are recommended for examining the generated depth sequences’ visual quality.
- The authors state that they would release the code and model for further research and applications.
A.1. Data Preparation
DepthCrafter represents depth in the disparity domain and targets relative depth estimation by normalizing sequence disparities to [0, 1]. Cached video and depth latents reduce training overhead.
- DepthCrafter represents depth in the disparity domain and targets relative depth estimation.
- Disparity values are normalized to [0, 1] using the sequence’s maximum and minimum disparities.
- Because training updates only the U-Net while freezing the VAE, video and depth latents can be pre-processed and cached.
- The caching mechanism significantly reduces the training burden.
A.2. Training Details
DepthCrafter follows the EDM framework, using preconditioning functions to scale inputs and outputs, modulate skip connections, and condition denoising on noise levels. Training samples noise levels from a log-normal distribution and uses memory-saving infrastructure.
- DepthCrafter follows the EDM framework for training.
- cin and cout scale input and output magnitudes, cskip modulates the skip connection, and cnoise maps σt to a denoiser conditioning input.
- The λσt term applies per-sample loss weighting to balance different noise levels.
- Training randomly samples σt from a log-normal distribution following the EDM framework’s relevant noise range.
- Training uses eight NVIDIA A100 GPUs, batch size 8, learning rate 10^-5, ZeRO-2, gradient checkpointing, and mixed precision.The first and third stages use around 40GB per device, while the second uses around 80GB.
A.3. Benchmark Evaluation Details
The evaluation recompiles image-based benchmarks into video formats and adapts depth metrics to enforce consistency across frames. Inference studies examine denoising as a speed–quality trade-off, finding that five steps provide a practical setting.
- Benchmark Construction: ScanNet V2, KITTI, and Bonn are reformatted into video sequences for video depth evaluation.ScanNet uses 90 frames, KITTI uses 110 frames, and Bonn supplements indoor evaluation with dynamic content.
- Evaluation Metrics: Scale-and-shift alignment is performed per video rather than per image so depth values remain consistent across frames.AbsRel and δ1 are retained, while least-square alignment is applied to each predicted depth sequence against ground truth.
- Inference Setting: Five denoising steps balance inference speed and depth estimation quality after improvements saturate beyond five steps.Even one step performs well, while additional steps consistently refine structural details before the gains level off.
B.2. Effectiveness of Training Stages
The supplementary ablation evaluates the three-stage training strategy across multiple datasets and separately examines classifier-free guidance. The results show strong performance with partial training and improved visual detail with guidance.
- Effectiveness of Training Stages: The first two training stages already outperform strong baselines across Sintel, ScanNet, KITTI, and Bonn.The comparison includes Marigold and Depth-Anything-V2 as reference methods.
- Classifier-Free Guidance: Classifier-free guidance improves visual details in generated depth sequences, especially in structures with fine-grained detail.The effect is illustrated on an example frame from KITTI and compared against results without guidance.