PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Depth Anything: Consistent Depth Estimation for Super-Long Videos

Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, Bingyi Kang

arXiv:2501.12375v3cs.CVcs.AI

TL;DR

Video depth foundation models remain temporally inconsistent, while prior remedies are limited to short videos or require efficiency trade-offs. Video Depth Anything adapts Depth Anything V2 with spatial-temporal modeling, temporal gradient matching, and key-frame-based long-video inference. It achieves state-of-the-art video-depth performance across spatial accuracy, temporal consistency, and computational efficiency, while supporting videos lasting several minutes.

  • Problem

    Depth Anything V2 generalizes well for detailed static-image depth but suffers from temporal inconsistency in videos, while existing remedies are limited to short videos or require efficiency trade-offs.

  • Method

    Video Depth Anything replaces the Depth Anything V2 head with a spatial-temporal head, trains temporal gradient matching, and uses key-frame-based segment-wise inference with depth stitching.

  • Results

    Video Depth Anything achieves state-of-the-art spatial accuracy, temporal consistency, and computational efficiency, producing high-quality depth for videos lasting several minutes.

  • Takeaways & Limitations

    The model extends foundation-model depth capabilities to arbitrarily long videos without sacrificing generalization, detail, or computational efficiency.

  • Takeaways & Limitations

    Training relies primarily on public video-depth datasets, which may limit capabilities, and streaming-video handling remains challenging.

Abstract

from arXiv · show

Depth Anything has achieved remarkable success in monocular depth estimation with strong generalization ability. However, it suffers from temporal inconsistency in videos, hindering its practical applications. Various methods have been proposed to alleviate this issue by leveraging video generation models or introducing priors from optical flow and camera poses. Nonetheless, these methods are only applicable to short videos (< 10 seconds) and require a trade-off between quality and computational efficiency. We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos (over several minutes) without sacrificing efficiency. We base our model on Depth Anything V2 and replace its head with an efficient spatial-temporal head. We design a straightforward yet effective temporal consistency loss by constraining the temporal depth gradient, eliminating the need for additional geometric priors. The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Moreover, a novel key-frame-based strategy is developed for long video inference. Experiments show that our model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization ability. Comprehensive evaluations on multiple video benchmarks demonstrate that our approach sets a new state-of-the-art in zero-shot video depth estimation. We offer models of different scales to support a range of scenarios, with our smallest model capable of real-time performance at 30 FPS.

1. Introduction

Video Depth Anything extends Depth Anything V2 to temporally stable depth estimation for arbitrarily long videos while retaining generalization, detail, and efficiency. Its spatial-temporal head, temporal gradient matching loss, and long-video inference strategy support strong spatial and temporal results.

  • Motivation: Depth Anything V2 generalizes well and preserves rich details efficiently, but flickering and motion blur limit its use on videos requiring temporal consistency.These limitations affect applications including robotics, augmented reality, and advanced video editing.
  • Related approaches: Existing approaches use test-time optimization, plug-in modules, optical flow, or camera poses, creating inference overhead or dependence on geometric priors.The supplied passages characterize these approaches as efforts to address temporal inconsistency before Video Depth Anything.
  • Approach: Video Depth Anything preserves Depth Anything V2’s generalization, detail, and efficiency while providing temporal stability for arbitrarily long videos without geometric or video-generation priors.This is the paper’s central design goal and contribution.
  • Method: A lightweight spatial-temporal head adds temporal attention at each spatial position, while temporal gradient matching constrains predicted depth changes across time.Temporal attention is confined to the head, and the loss matches temporal depth gradients to ground-truth gradients.
  • Long-video inference: The long-video strategy uses overlapping frames and key frames from previous clips, progressively interpolating overlaps between consecutive windows for smoothness.The described construction forms 32-frame windows from eight overlapping frames and two prior key frames.
  • Results: On five video datasets, the model reaches state-of-the-art spatial accuracy on four, beats all baselines in temporal consistency, and is more efficient than video-diffusion methods.It also estimates consistent depth for videos lasting several minutes and shows only a marginal image-depth drop on one dataset versus Depth Anything V2.
  • Contributions: The contributions include a new transformation of Depth Anything for arbitrarily long videos, a prior-free temporal-consistency loss, and computationally efficient video depth estimation.The contribution summary describes the method as state of the art in both spatial and temporal video depth estimation.

2. Related Work

Prior monocular depth models improved from narrow in-domain training toward broad zero-shot generalization and detailed predictions, but static-image training leaves Depth Anything V2 temporally inconsistent. Video-depth research consequently moved from inefficient test-time optimization toward learned methods using motion or pose information.

  • Monocular depth estimation: Early monocular depth models trained on in-domain datasets and generalized poorly to zero-shot settings because their training data lacked diversity.MiDaS improved generalization through multi-dataset mixed training with affine-invariant alignment.
  • Monocular depth estimation: MiDaS improved generalization, while later large-scale synthetic-data training and distillation on 62 million unlabeled images improved generalization and geometric detail.The passage identifies Depth Anything V2 as trained exclusively on static images and therefore lacking temporal consistency.
  • Consistent video depth estimation: Consistent video-depth methods initially relied on test-time training, which was impractical for applications because of low efficiency.More recent learning-based methods use optical-flow feature warping or relative camera poses to model temporal relationships.

3. Video Depth Anything

Video Depth Anything extends Depth Anything V2 with temporal modeling, a temporal-gradient loss, and key-frame-based inference for consistent depth across long videos.

  • 3.1. Architecture: The model uses a frozen Depth Anything V2 encoder with a trainable spatio-temporal head for video and image inputs.The head inserts temporal layers into the DPT structure while retaining its original design.
  • 3.1. Architecture: Joint training combines ground-truth video depth with unlabeled images carrying teacher-generated pseudo labels.This strategy addresses the limited availability of video depth data while preserving image-depth training.
  • 3.2. Temporal Gradient Matching loss: Temporal Gradient Matching constrains changes in predicted depth between adjacent frames to match ground-truth depth changes without requiring optical flow.The loss is evaluated at corresponding image coordinates rather than flow-derived correspondences.
  • 3.2. Temporal Gradient Matching loss: The video-depth objective combines temporal-gradient matching with a scale- and shift-invariant image loss to balance temporal consistency and single-frame structure.The temporal loss is restricted to regions where the ground-truth depth change is below 0.05.
  • 3.3. Inference strategy for super-long sequence: Long-video inference uses overlapping frames and backward-subsampled key frames, then aligns and interpolates predictions across consecutive windows.The default configuration uses N = 32, To = 8, Tk = 2, and Δk = 12; key frames provide earlier-window content while limiting accumulated drift.

4. Experiments

Experiments evaluate Video Depth Anything on zero-shot video and image depth tasks, long-video frame lengths, qualitative comparisons, efficiency, and ablations of temporal losses and inference strategies.

  • Evaluation Setup: Five video datasets cover indoor, outdoor, and wild scenes, with evaluation extending to 500 frames.The study also evaluates static-image depth on five image benchmarks using geometric and temporal metrics for video depth.
  • Zero-shot Depth Estimation: Video Depth Anything achieves state-of-the-art results across long-video datasets in geometric and temporal metrics, surpassing leading methods by approximately 10% on KITTI, Scannet, and Bonn geometric accuracy.The comparison includes NVDS, ChronoDepth, DepthCrafter, DepthAnyVideo, Depth Anything V2, and NVDS + DAv2.
  • Long-video Evaluation: Across 110–500 frames, the model significantly outperforms DepthCrafter on all evaluated datasets and frame lengths, with minimal metric degradation as video length increases.It surpasses DepthAnyVideo on Scannet and NYUv2 and is comparable on Bonn for 110- and 192-frame evaluations.
  • Qualitative Comparisons: Qualitative results show better geometric accuracy and temporal consistency than DepthCrafter on 500-frame Scannet and Bonn videos, while short-video comparisons expose competing depth errors or inconsistencies.Figure 5 uses temporal depth profiles; Figure 6 marks incorrect estimates in red and inconsistent estimates in blue.
  • Experimental Settings: The experiments use VDA-S with a window size of 16 unless otherwise specified, and metrics without dataset names are means across datasets.These settings qualify the reported ablation results and aggregate metrics.
  • Ablation Studies: Temporal gradient matching with SSI significantly outperforms optical-flow warping with SSI on geometric and stability metrics while eliminating dependence on optical flow.VideoAlign variants have good geometric metrics but poor video stability; TGM+SSI is comparable to SE+SSI.

5. Conclusion

Video Depth Anything combines temporal modeling, temporal gradient matching, and keyframe-based inference to estimate consistent depth in arbitrarily long videos. Experiments report state-of-the-art spatial accuracy, temporal consistency, and computational efficiency, including videos lasting several minutes.

  • 5. Conclusion: Video Depth Anything is built on Depth Anything V2 with a spatial-temporal head that applies temporal self-attention to feature maps.The model introduces temporal interactions while retaining the foundation model as its base.
  • 5. Conclusion: Image dataset distillation is evaluated alongside video-only training as an ablation of the training strategy.Table 6 contrasts video-only training with joint video-and-image training using image-level distillation.
  • 5. Conclusion: A temporal gradient matching loss enforces temporal consistency without geometric or generative priors.The loss constrains temporal depth behavior as part of the model’s consistency design.
  • 5. Conclusion: Keyframe-based segment-wise inference with depth stitching enables depth estimation for arbitrarily long videos.The conclusion identifies this strategy as the component supporting long-video inference.
  • 5. Conclusion: The model achieves state-of-the-art spatial accuracy, temporal consistency, and computational efficiency, producing high-quality depth for videos lasting several minutes.The reported conclusion covers all three evaluation dimensions and the demonstrated video duration.

1. More Qualitative Results

Qualitative evaluations show that Video Depth Anything preserves strong static-image depth quality while improving depth accuracy and temporal behavior in long videos.

  • Static image results: Static-image predictions from Video Depth Anything are comparable to Depth-Anything-V2 and superior to DepthCrafter and Depth Any Video in complex scenes.DepthCrafter performs poorly on oil paintings, transparent objects, glass, and water.
  • Evaluation video results: DepthCrafter exhibits depth drift in long videos, whereas Video Depth Anything shows superior depth accuracy in the highlighted comparisons.All predicted depths are aligned to ground-truth video depths using the evaluation procedure.

2. Short video depth quantitative results

Short-video experiments compare Video Depth Anything with DepthCrafter and Depth Any Video across KITTI, Bonn, and Scannet under matched evaluation settings.

  • Quantitative comparisons: Approximately 7% advantage over both DepthCrafter and Depth Any Video is reported on Scannet.The comparison uses 90-frame videos for Scannet.
  • Efficiency: The proposed model uses substantially fewer parameters and less video-depth training data than the compared methods.The authors present this as evidence of the method’s efficiency.

3. Limitations and future work

The authors identify limited training-data quantity and difficulty with streaming videos as the main boundaries of the current model.

  • Limitations: Training primarily on publicly available video-depth datasets may limit model capabilities because of the available data quantity.The authors expect more data could improve performance and enable backbone fine-tuning.
  • Future work: Despite improved computational efficiency over baselines, the model still faces challenges in handling streaming videos.Streaming-video support is left for future work.

4. More Details of Pipeline

Additional details describe temporal-layer placement, feature reshaping, supervision data, training configuration, and evaluation settings used by Video Depth Anything.

  • Spatiotemporal head: Two temporal layers are inserted after Reassemble layers at the two smallest resolutions, and two precede the final Fusion layers.This placement specifies how the four temporal layers are integrated into the spatiotemporal head.
  • Feature reshaping: Before each temporal layer, features are reshaped to (B × Hf × Wf)×N ×C and restored to (B × N) × C × Hf × Wf afterward.B is batch size, N is frame count, Hf and Wf are feature dimensions, and C is the channel count.
  • Image distillation: Image distillation uses a ViT-giant teacher trained on synthetic datasets, with the distillation loss matching the spatial loss for video-depth data.This follows the approach used in Depth Anything V2.
  • Training data: Video training uses TartanAir, VKITTI, PointOdyssey, and IRS, totaling 0.55 million precisely annotated frames, plus 0.18 million wild binocular frames.A further 0.62 million frames from real-world unlabeled datasets are used for single-image supervision.
  • Training procedure: Training starts from Depth Anything V2 weights and uses two stages, combining video-depth supervision with unlabeled single-image supervision.Synthetic-only models labeled “-Syn” are trained in a single stage.

5. More Details of Evaluation

The evaluation uses five video datasets spanning indoor, outdoor, dynamic, and wild scenes, alongside five image benchmarks. Video metrics measure geometric accuracy and temporal stability after scale-and-shift alignment to ground-truth depth.

  • Evaluation datasets: Five video datasets cover static indoor, dynamic indoor, outdoor, and wild scenes, with evaluations including long-video settings.The datasets are KITTI, ScanNet, Bonn, NYUv2, and Sintel.
  • Evaluation datasets: Five image benchmarks evaluate static-image depth, with all evaluation videos and images excluded from training datasets.The image benchmarks are KITTI, Sintel, NYUv2, ETH3D, and DIODE.
  • Evaluation metrics: Video predictions are aligned to ground-truth inverse depth using optimal scale and shift before computing depth metrics.The alignment uses least squares over the entire inferred video.
  • Evaluation metrics: Geometric accuracy is measured with Absolute Relative Error and δ1, while temporal stability uses video-specific temporal metrics.The supplied passage introduces the temporal metric family but truncates its full description.

6. Applications

The paper demonstrates applications in 3D video conversion and dense point-cloud generation. These applications use temporally consistent depth to produce smoother 3D effects and more regular reconstructions across frames.

  • Dense point-cloud generation: For KITTI sequences spanning approximately 5 seconds, the model generates a clean and regular dense point cloud from multiple frames.The point cloud is formed by aligning the first frame, applying a global affine transformation, and merging per-frame point clouds.
  • 3D video conversion: The model converts DAVIS videos into 3D videos with smoother and more consistent 3D effects than monocular depth models.The application is illustrated in Figure 11.
  • Dense point-cloud generation: Compared with DepthCrafter and DepthAnyVideo, the model avoids the discontinuous layers, noisy outliers, and distant-view distortion described for those baselines.The comparison concerns dense point-cloud quality on KITTI.
  • Wild-video point clouds: For wild DAVIS videos, camera intrinsics and aligned scale-and-shift parameters are obtained from the first frame using MoGe, with distortions marked in red boxes.Figure 13 compares the proposed method with DepthCrafter and DepthAnyVideo.
Loading 2501.12375v3…