Source-linked AI summary
DVD: Deterministic Video Depth Estimation with Generative Priors
Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He, Zixin Zhang, Haodong Li, Yihao Liang, Kanghao Chen, Bin Ren, Xu Zheng, Shuai Yang, Kun Zhou, Yinchuan Li, Nicu Sebe, Ying-Cong Chen
TL;DR
Video depth estimation must balance generative priors against geometric stability and data demands. DVD deterministically adapts pre-trained video diffusion models into single-pass depth regressors using a structural timestep anchor, latent manifold rectification, and global affine coherence. It reports state-of-the-art zero-shot geometric fidelity and temporal coherence while using 163× less task-specific data, with a stated limitation for extreme long-video dynamics.
Problem
Video depth estimation is constrained by generative hallucinations and scale drift versus discriminative models’ need for massive labeled data to resolve semantic ambiguity.
Method
DVD deterministically adapts pre-trained video diffusion priors into single-pass depth regressors using a timestep anchor, LMR, and global affine coherence.
Results
163× less task-specific data accompanies DVD’s state-of-the-art zero-shot geometric fidelity and temporal coherence.
Takeaways & Limitations
DVD provides an efficient, scalable route for using video foundation-model geometric priors in zero-shot video depth estimation.
Takeaways & Limitations
Extreme dynamics, prolonged occlusions, illumination shifts, or erratic camera motion can temporarily violate DVD’s global affine assumption and cause scale inconsistencies.
Abstract
from arXiv · showhide
Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-video inference without requiring complex temporal alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across benchmarks. Furthermore, DVD successfully unlocks the profound geometric priors implicit in video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.
1 Introduction
Video depth estimation must reconcile precise frame-level geometry with temporal consistency, but generative and discriminative paradigms expose complementary bottlenecks. DVD deterministically adapts pre-trained video diffusion models with three designs that target stability, detail, and long-video scalability.
- Motivation: Video depth estimation requires precise geometric reasoning and temporal consistency under camera motion and dynamic objects.Maintaining temporal consistency without sacrificing high-frequency geometric details remains a persistent real-world bottleneck.
- Motivation: Generative models introduce stochastic uncertainty and geometric hallucination, whereas discriminative models face semantic ambiguity and heavy annotation requirements.These limitations hinder scalability, reproducibility, and practical deployment.
- Our Solution: DVD deterministically adapts pre-trained video diffusion models into single-pass regressors that map RGB inputs to geometrically consistent depth.The approach uses semantic priors to address motion-induced ambiguity while avoiding generative hallucination through regression.
- Our Solution: DVD repurposes the diffusion timestep as a structural anchor, applies latent manifold rectification, and exploits global affine coherence for long-video inference.These mechanisms balance geometric stability with detail, restore spatial and temporal consistency, and support robust long-video processing.
- Empirical Validation: DVD achieves state-of-the-art zero-shot geometric fidelity and temporal coherence across four real-world benchmarks.The framework also reports robust long-video inference and generalization to unconstrained open-world domains.
- Empirical Validation: DVD unlocks pre-trained world priors with less than 1% of leading baselines’ task-specific training data while maintaining comparable inference speed.Figure 1 reports this efficiency alongside superior performance on short and long videos averaged across KITTI, ScanNet, and Bonn.
2 Related Work
Video depth methods mainly use discriminative regression or generative diffusion, each offering useful capabilities but imposing distinct limitations. DVD instead repurposes video foundation-model priors through deterministic adaptation for a broader video-depth setting.
- Monocular Depth Estimation: Discriminative video-depth methods use temporal modules or flow refinement but typically require large supervision and lack generative priors for semantic ambiguity.Their limitations are especially relevant in textureless or motion-blurred regions.
- Monocular Depth Estimation: Generative diffusion methods provide rich spatio-temporal priors and zero-shot generalization by adapting pre-trained diffusion models to depth estimation.Video-specific approaches further adapt these priors to temporal dynamics.
- Monocular Depth Estimation: Stochastic multi-step sampling in generative methods introduces high latency and geometric hallucinations.DVD addresses this bottleneck through deterministic adaptation.
- Video Diffusion Models: Modern video diffusion transformers encode physical-dynamics and 3D-consistency priors that DVD repurposes for deterministic depth regression.These models function as foundation models with geometric and dynamic information useful for video depth.
- More Video Depth Methods: Specialized methods target settings such as real-time streaming efficiency, whereas DVD explores a more general video-depth direction.These approaches are described as complementary rather than direct baselines for DVD’s core formulation.
3 Preliminary
DVD formulates video depth estimation as deterministic latent-space prediction using a frozen VAE and a video diffusion backbone. This contrasts with conventional rectified-flow sampling, replacing iterative integration with a single forward pass.
- Problem Formulation: Video depth estimation maps an RGB sequence x to a corresponding depth sequence d across F frames.The formulation operates on compressed latent representations rather than directly on pixel sequences.
- Problem Formulation: A frozen VAE encoder projects RGB and depth sequences into a unified latent space, and a decoder reconstructs the predicted depth.The learned mapping is Φ: z_x 7→ z_d, followed by reconstruction through the frozen decoder.
- Role of t in Rectified Flow: Rectified flow defines a linear interpolation between data and Gaussian noise using timestep t and trains a network to predict the flow velocity.Standard inference solves an ODE numerically from t = 1 to t = 0.
- Deterministic Adaptation: Deterministic adaptation replaces iterative ODE integration with direct depth prediction from RGB latent z_x conditioned on timestep t.The backbone computes ẑ_d = F_θ(z_x, t) in one forward pass.
4 Methodology
DVD deterministically adapts video diffusion backbones for depth estimation through timestep anchoring, latent manifold rectification, and affine-coherent long-video inference. These designs balance geometric stability with detail, restore spatio-temporal structure lost under regression, and align overlapping windows efficiently.
- Timestep as Structural Anchor: DVD repurposes the diffusion timestep as a persistent structural anchor that balances low-frequency geometric stability with high-frequency detail.The deterministic mapping replaces dynamic timestep sampling with a fixed conditioning state.
- Timestep as Structural Anchor: τ = 0.5 balances global coherence and local sharpness, whereas τ = 0.8 is stable but blurry and τ = 0.0 is sharp but unstable.The fidelity-stability trade-off persists after fine-tuning, and removing timestep conditioning degrades performance.
- Latent Manifold Rectification: Point-wise regression drives predictions toward conditional means, suppressing high-frequency differentials and causing boundary erosion and motion flickering.This mean collapse is amplified in spatio-temporal settings.
- Latent Manifold Rectification: LMR aligns predicted and target spatial and temporal differentials in latent space, restoring sharp boundaries and coherent motion without heavy auxiliary modules.Spatial gradient alignment explicitly penalizes low-frequency latent collapse.
- Global Affine Coherence: DVD’s deterministic regression eliminates stochastic output uncertainty, while global affine coherence makes inter-window discrepancies recoverable as scale-shift transformations.VAE decoding primarily induces global affine variations rather than local spatial distortions.
- Global Affine Coherence: A parameter-free affine calibration aligns overlapping depth windows and enables seamless long-video inference without feature matching, flow estimation, or recurrent temporal modules.The calibration is broadcast across the current window and overlap frames are blended linearly.
5 Experiments
DVD consistently improves geometric fidelity, temporal coherence, boundary quality, data efficiency, and long-video stability across the reported experiments. Ablations attribute these gains to timestep conditioning, latent manifold rectification, deterministic regression, and joint image-video training.
- Main Results: DVD achieves the lowest AbsRel on ScanNet (5.5) and KITTI (6.7), outperforming generative and discriminative baselines across standard benchmarks.
- Main Results: DVD yields 5.3 vs. 8.5 AbsRel against DepthCrafter on Bonn and raises ScanNet B-F1 to 0.259 versus VDA’s 0.210.The reported improvements extend to long-video evaluation and fine-grained boundary precision.
- Efficiency and Scalability: Training on 367K frames surpasses VDA trained on 60M frames, using less than 1/160 of its dataset.
- Efficiency and Scalability: DVD maintains inference speed comparable to VDA while delivering superior accuracy by bypassing iterative generative sampling.
- Efficiency and Scalability: DVD preserves global scale consistency across disjoint temporal windows, while competing methods exhibit scale drift or semantic ambiguity on long videos.Its affine-alignment mechanism maintains structural persistence as sequence length increases, whereas baseline metrics degrade more visibly.
- Framework Analysis: The structural anchor reaches its optimal balance at τ = 0.5; on KITTI, AbsRel changes from 13.8 at τ = 0.0 to 6.7 at τ = 0.5.Outdoor scenes are more sensitive, while τ ≥0.9 causes severe performance collapse across both datasets.
- Framework Analysis: Adding spatial and temporal latent gradients reduces AbsRel from 8.5 to 7.3 and raises B-F1 from 0.210 to 0.259.The differential constraints are reported to rectify regression-induced mean collapse and restore sharp boundaries and coherent motion.
- Framework Analysis: Deterministic adaptation reduces AbsRel from 9.7 with T = 10 to 7.3 with T = 1, outperforming multi-step stochastic sampling.
6 Conclusion
DVD deterministically adapts pre-trained video diffusion priors for single-pass depth estimation through a structural anchor, latent manifold rectification, and global affine coherence. It achieves state-of-the-art geometric fidelity and temporal coherence with 163× less task-specific training data than leading baselines.
- DVD deterministically adapts pre-trained video diffusion priors for single-pass depth estimation.
- Its three designs combine timestep-driven structural anchoring, LMR against spatio-temporal mean collapse, and affine-alignment long-video inference.
- DVD achieves state-of-the-art geometric fidelity and temporal coherence while using 163× less task-specific training data than leading baselines.
A Limitations and Future Work
DVD’s limitations arise in extreme long-video dynamics, real-time deployment, and compressed latent resolution. The authors propose broader temporal modeling, efficient architectures, and higher-resolution representations as future directions.
- Boundary Conditions of Long Videos: Extreme dynamics can introduce local non-linear distortions that temporarily overpower the global affine assumption, causing scale inconsistencies.The cited cases include prolonged occlusions, rapid illumination shifts, and erratic camera motions.
- Constraints for Real-Time Deployment: True real-time inference at latency-critical rates such as ≥10Hz remains challenging despite bypassing stochastic sampling.DVD still relies on a massive video DiT backbone, including Wan2.1-1.3B.
- Resolution Limits of VAE: An 8× downsampled VAE latent space inherently limits recovery of ultra-thin geometric structures at native resolutions.The authors suggest higher-resolution latent spaces or VAE-free tokenization to improve geometric fidelity.
B More Details of Global Affine Coherence
Overlapping-window predictions exhibit a strictly linear depth relationship, supporting the assumption that inter-window discrepancies are primarily global scale and shift differences.
- Global Affine Coherence: The strictly linear relationship between unaligned overlapping-window predictions corroborates global affine coherence.The analysis attributes the discrepancy primarily to deterministic VAE decoding and characterizes it through global scale and shift.
C More Implementation Details
DVD preserves pretrained video priors through parameter-efficient adaptation and evaluates temporal stability, efficiency, and cross-backbone behavior under reproducible settings.
- Training Details: DVD freezes original weights and applies LoRA exclusively to attention blocks for deterministic regression.The training details cover VAE compression, LoRA configuration, optimization, and joint-training loss weights.
- Hyperparameters: The implementation uses video batch size 16, image batch size 128, 480 × 640 resolution, a 45-frame window, and stride 9 at inference.These settings are listed among DVD’s hyperparameter configurations.
- Experimental Details: Evaluation partitions videos by duration to separate short-term geometric fidelity from long-term structural persistence.Inference efficiency is benchmarked on a single NVIDIA RTX A6000 under identical environments.
- Inference Efficiency: TensorRT and related accelerators remain future directions for real-time edge applications.The paper presents these as possible extensions of the single-pass inference pipeline.
- Cross-Backbone Generalization: DVD on CogVideoX-5B retains sharp high-frequency geometry despite using a different foundation architecture.Figure 13 compares DVD with the leading baseline, while Table 6 reports the corresponding cross-backbone setting.
D More Analysis
Additional analyses examine structural anchoring, latent rectification, overlap size, LoRA capacity, and long-sequence behavior. Together, they identify settings that preserve geometric accuracy and detail while exposing scale-drift boundaries.
- Structural Anchor: τ = 0.5 provides optimal conditioning on KITTI for DVD adapted to CogVideoX-5B.Extreme timesteps degrade geometry, and the cross-backbone result corroborates the structural-anchor analysis.
- Latent Manifold Rectification: LMR achieves AbsRel 7.3 and B-F1 0.259, jointly improving global accuracy and boundary precision.Multi-scale gradient matching reaches B-F1 0.257 but only AbsRel 8.2, whereas LMR addresses both objectives together.
- Overlap Size: Overlap O = 3 yields AbsRel 7.9, while larger overlap improves geometric accuracy with diminishing returns and added relative-time cost.A larger overlap provides more pixels for affine estimation but increases computational overhead.
- Long-Video Failure Analysis: Across an 1100-frame sequence with massive scene transitions, DVD retains sharper local structure although both DVD and VDA suffer global scale drift.The reported local fidelity includes hand details in Frames #800–#900.
- LoRA Rank: LoRA rank 512 reduces AbsRel to 7.3, while rank 1024 offers only marginal gains.Very low ranks miss high-frequency detail, whereas full fine-tuning can overfit limited training data and weaken pretrained priors.
E Failure Case Analysis
DVD’s affine coherence depends on overlap between adjacent temporal windows, so extreme long videos with abrupt scene changes can produce global scale drift. Despite this limitation, DVD preserves local high-frequency geometric detail better than the cited discriminative baseline.
- Failure boundary: Adjacent-window overlap underpins DVD’s affine coherence, limiting scale consistency in long videos with massive ego-motion or abrupt scene transitions.The limitation arises in unconstrained long-video scenarios where neighboring windows lack sufficient geometric continuity.
- Failure boundary: A 1100-frame sequence with disjoint indoor and outdoor scenes causes DVD to lose a unified absolute depth range across distant frames.Frames with zero visual overlap, such as Frame #1 and Frame #500, exhibit global scale drift.
- Comparison and scope: VDA exhibits identical or more severe temporal scale degradation under extreme dynamics, indicating that infinite-length metric anchoring remains unresolved across the field.The passage frames this limitation as an open challenge rather than a flaw unique to DVD.
- Comparison and scope: Even amid global scale drift, DVD preserves sharper local structures than VDA, including laptop-screen edges and intricate hand geometries.The cited examples span Frame #1 and Frames #800–#900.
F Exhibition Board
DVD is exhibited across diverse open-world video domains, including natural, architectural, dynamic, stylized, and AI-generated content. The exhibition includes qualitative demonstrations on both short and long videos.
- Open-world coverage: Qualitative results cover natural landscapes, complex architecture, humans, animals, animations, video games, and AI-generated videos.These examples are presented to demonstrate DVD’s zero-shot generalization across diverse open-world domains.
- Short videos: Figures 15–17 provide additional demonstrations on short videos.The figures extend the short-video qualitative exhibition.
- Long videos: Figures 18–20 provide additional demonstrations on long videos.The figures extend the long-video qualitative exhibition.