PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Temporally Consistent Video Depth from Video Diffusion Priors

Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Vitor Guizilini, Yue Wang, Matteo Poggi, Yiyi Liao

arXiv:2406.01493v4cs.CV

TL;DR

Streamed video depth estimation must combine per-frame spatial accuracy with cross-frame consistency, yet single-image depth predictions can flicker and existing streamed approaches may lack cross-clip context. ChronoDepth uses video diffusion priors and context-aware training and inference for within- and across-clip context. It reports state-of-the-art temporal consistency while maintaining comparable spatial accuracy, with a noted AbsRel degradation in scenes containing abundant dynamic objects.

  • Problem

    Video depth estimation requires spatial accuracy and temporal consistency, but frame-independent single-image methods lack context and are prone to flicker.

    Source: §1 · p. 1

  • Method

    ChronoDepth fine-tunes video diffusion for within-clip context and uses sliding-window inference that reuses prior predicted depth in overlaps without adding noise.

    Source: §1 · p. 2

  • Results

    ChronoDepth achieves state-of-the-art temporal consistency on open-world video benchmarks while maintaining spatial accuracy comparable with state-of-the-art single-image depth methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that video generation priors can yield a depth predictor with superior temporal consistency and comparable spatial accuracy.

    Source: §5 · p. 8

    ChronoDepth shows slight AbsRel degradation on Sintel scenes with abundant dynamic objects, attributed to limited moving objects in training data.

    Source: F. Limitation · p. 15

Abstract

from arXiv · show

This work addresses the challenge of streamed video depth estimation, which expects not only per-frame accuracy but, more importantly, cross-frame consistency. We argue that sharing contextual information between frames or clips is pivotal in fostering temporal consistency. Therefore, we reformulate depth prediction into a conditional generation problem to provide contextual information within a clip and across clips. Specifically, we propose a consistent context-aware training and inference strategy for arbitrarily long videos to provide cross-clip context. We sample independent noise levels for each frame within a clip during training while using a sliding window strategy and initializing overlapping frames with previously predicted frames without adding noise. Moreover, we design an effective training strategy to provide context within a clip. Extensive experimental results validate our design choices and demonstrate the superiority of our approach, dubbed ChronoDepth. Project page: https://xdimlab.github.io/ChronoDepth/.

1. Introduction

Video depth estimation must balance spatial accuracy with temporal consistency, but frame-independent predictions and limited context between streamed clips leave flicker unresolved. ChronoDepth uses video diffusion to provide context within and across clips, and achieves strong temporal consistency while maintaining comparable spatial accuracy.

1. Introduction

  • Single-image depth methods assume frames are independent, leaving them prone to flicker despite advances in spatial precision.Video applications including AR/VR, robotics, and video editing require both spatial accuracy and temporal consistency.Source: §1 · p. 1
  • Existing attention-based methods aggregate context within clips but cannot fully encode context between clips, while test-time training relies on precise camera poses and long optimization.These limitations motivate context-aware inference for arbitrarily long, streamed video.Source: §1 · p. 1
  • The method carries previously predicted depth frames into overlapping windows without adding noise, using different noise levels per frame during training to support denoising across varying noise levels.This is designed to keep cross-clip context consistent, unlike replacement with noisy versions of prior predictions.Source: §1 · p. 2
  • ChronoDepth fine-tunes video diffusion models for within-clip context and identifies sequential spatial-then-temporal training, with spatial layers frozen during temporal training, as an effective strategy.The training exploration draws on both image and video depth datasets.Source: §1 · p. 2
  • ChronoDepth achieves state-of-the-art temporal consistency on open-world video benchmarks while maintaining spatial accuracy comparable with state-of-the-art single-image depth methods.The reported comparison covers both temporal consistency and spatial accuracy.Source: §1 · p. 2

2. Related Work

Related work spans discriminative and generative monocular depth estimation, video depth methods, and video diffusion. Across these areas, the cited approaches address generalization or temporal context, while generative depth methods still exhibit temporal inconsistency.

Related Work

  • Discriminative monocular depth moved from scenario-specific metric models [20] toward affine-invariant relative depth for generalization across environments [55] [56].More recent methods also pursue generalizable metric depth through camera-aware transformations or prompts [32] [80] [29] [53].Source: §2 · p. 2
  • Generative monocular depth adapts pretrained models through approaches including LoRA, self-supervised pretraining [59], and UNet fine-tuning [40], but retains temporal flicker.Other cited approaches combine depth with surface-normal prediction [22] or use Flow Matching [27].Source: §2 · p. 2
  • Video depth methods include pose-based cost volumes [28] [47] [71], test-time training [43] [48] [85], recurrent networks [83], and attention mechanisms [69].The passage notes pose dependence for cost-volume and test-time-training approaches, and limited generalizability for test-time training.Source: §2 · p. 3
  • NVDS temporally refines single-frame predictions, while DepthCrafter [33] uses replacement-trick latent interpolation and can incur inconsistency from noise variation.The latter is described as concurrent work.Source: §2 · p. 3
  • Video diffusion adds temporal layers to image-generation frameworks, while Diffusion Forcing [15] focuses on generation and uses causal architectures with recurrent hidden states.ChronoDepth instead targets deterministic video depth with attention and no additional hidden states.Source: §2 · p. 3

3. Method

ChronoDepth repurposes Stable Video Diffusion for depth estimation on arbitrary-length videos, combining framewise noise training with prior-depth context across streamed clips.

3.1. Naive Diffusion Formulation

  • RGB-conditioned SVD denoises depth latents in VAE space, but independently initialized overlapping clips exchange no context and can flicker.The replacement trick adds noise to prior overlap depths, causing conditioning to vary across sampling steps and inconsistencies in geometry and scale.Source: §3.1 · pp. 3–4

3.2. Consistent Context-Aware Strategy

  • Independent noise levels are sampled per training frame, and later clips reuse previously predicted overlap depths to keep context consistent during sampling.A small conditioning noise accounts for uncertainty in prior predictions and mitigates long-term compounding errors.Source: §3.2 · pp. 4–5

3.3. Multi-Stage Training Protocol

  • Training uses single-frame datasets throughout and random clip lengths, then trains spatial layers before freezing them for temporal-layer tuning.The authors present sequential spatial-temporal fine-tuning as favoring temporal consistency while maintaining spatial accuracy.Source: §3.3 · pp. 5–6

4. Experimental Results

ChronoDepth is evaluated as a streamed video depth estimator across training protocols, evaluation settings, benchmark comparisons, and ablations. The experiments report strong temporal consistency with spatial accuracy comparable to state-of-the-art single-image methods.

4.1. Training Protocol

  • Training uses 39K single-frame samples and 938 video sequences, with 20K single-frame pretraining steps followed by 18K video fine-tuning steps using clip lengths sampled from 1 to 5 frames.Inference uses clips of length 10 with 5 overlapping frames.Source: §4.1 · p. 6

4.2. Evaluation Protocol

  • Evaluation spans KITTI-360, ScanNet++, and Sintel, fitting one global scale and shift per video and measuring AbsRel, δ1, and adjacent-frame MFC.Ground-truth poses and optical flow are used only to compute MFC.Source: §4.2 · pp. 6–7

4.3. Comparison with State-of-the-Art

  • 98% relative improvement on KITTI-360 MFC; ChronoDepth retains spatial accuracy comparable to state-of-the-art single-image methods and outperforms NVDS and DepthCrafter in spatial accuracy and temporal consistency.Figure 5’s y-t slices show smooth depth over time for ChronoDepth, while single-frame methods flicker and video competitors show per-clip artifacts.Source: Table 1 · p. 7, §4.3 · pp. 7–8, Figure 5 · p. 7

4.4. Ablation Studies

  • Combining single-frame and multi-frame training data improves spatial accuracy; random clip-length sampling reduces performance degradation and mitigates overfitting, while sequential spatial-then-temporal training improves consistency.Sequential training also improves spatial accuracy on ScanNet++; KITTI-360 AbsRel and δ1 are less sensitive to that choice.Source: §4.4 · p. 8
  • Consistent context-aware inference improves both spatial accuracy and temporal consistency, whereas naive sliding windows and the replacement trick retain per-clip flicker.The y-t slices in Figure 6 visualize the three inference strategies.Source: §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

ChronoDepth prioritizes temporal consistency using video generation priors while maintaining comparable spatial accuracy. The paper presents its empirical insights as groundwork for future research on video generation models for depth estimation.

5. Conclusion

  • ChronoDepth uses video generation priors to prioritize temporal consistency while maintaining spatial accuracy comparable to existing methods.The authors present their empirical insights into harnessing video generation models for depth estimation as groundwork for future investigations.Source: §5 · p. 8

Supplementary Material

The supplementary material includes an ablation of σϵ and additional qualitative results on ScanNet++.

Supplementary Material

  • Figure S1 reports accuracy and consistency metrics on KITTI-360 for different σϵ values.Source: Figure S1 · p. 13
  • Figure S2 presents additional qualitative video depth comparisons on ScanNet++.Source: Figure S2 · p. 13

A. Details on Evaluation Protocol

The evaluation protocol selects 200 KITTI-360 frames and measures multi-frame consistency using adjacent-frame depth comparisons.

A. Details on Evaluation Protocol

  • For KITTI-360 evaluation, the authors use frames 300–500, yielding 200 frames because initial frames lack ground-truth poses.Source: A. Details on Evaluation Protocol · p. 13
  • MFC compares an adjacent-frame depth map with the other frame’s depth after unprojecting, pose-transforming, and reprojecting the first map; invalid pixels are masked.In practice, the metric is calculated on adjacent frames.Source: A. Details on Evaluation Protocol · p. 13

B. Detailed Proof on Mathematical Rigor and Fluctuating Guidance

The section describes context-conditioned sampling of future depth latents and a guidance term that varies across sampling steps.

B. Detailed Proof on Mathematical Rigor and Fluctuating Guidance

C. Speed and Compute Comparison

ChronoDepth is reported as faster than Marigold and DepthCrafter and as using less memory than DepthAnything v2.

C. Speed and Compute Comparison

D. Additional Ablation

The additional ablation identifies a middle noise level for overlapping frames, while further qualitative comparisons report spatial accuracy and smoother temporal slices.

D. Additional Ablation

  • An excessively small or large overlap-frame noise level degrades spatial and temporal performance; the authors select σϵ = −4.Source: D. Additional Ablation · p. 14

E. More Qualitative Results

  • Across KITTI-360, ScanNet++, and Bonn, ChronoDepth’s spatial accuracy is comparable to or better than state-of-the-art models.Source: E. More Qualitative Results · p. 14
  • Y-t slices from most methods show high-frequency artifacts, whereas ChronoDepth’s slices are smoother, supporting its stronger temporal consistency.Source: E. More Qualitative Results · p. 14

F. Limitation

ChronoDepth is reported as robust to rapid camera motion and long videos, but its AbsRel degrades slightly in dynamic-object-rich scenes.

F. Limitation

  • ChronoDepth is robust to rapid ego-camera motion in ScanNet++ and long videos in KITTI-360.Source: F. Limitation · p. 14
  • In Sintel scenes with abundant dynamic objects, AbsRel degrades slightly, which the authors attribute to limited moving objects in training data.The authors note that training data could be extended.Source: F. Limitation · p. 15

Related papers

Loading 2406.01493v4…