Source-linked AI summary
Robust Consistent Video Depth Estimation
Johannes Kopf, Xuejian Rong, Jia-Bin Huang
TL;DR
Consistent dense depth and camera poses remain difficult to estimate from challenging monocular videos without reliable external poses. The method combines learned single-image depth with joint pose/deformation optimization and geometry-aware filtering, achieving robust reconstruction and better performance across test beds.
Problem
Accurate, geometrically consistent dense depth is difficult for casually captured monocular videos with noise, shake, blur, rolling shutter, and dynamic objects.
Method
The method combines a single-image CNN depth prior with joint camera-pose and depth-alignment optimization, flexible deformations, and geometry-aware filtering.
Results
The method achieves robust reconstruction without input poses and reports better performance across diverse test beds, including quantitative evaluation on Sintel.
Takeaways & Limitations
The algorithm supports consistent depth reconstruction for challenging dynamic videos casually captured with hand-held devices.
Takeaways & Limitations
Residual wobble remains in aligned depth maps, while denser constraints cause excessive Ceres Solver memory use and sharply reduce performance.
Abstract
from arXiv · showhide
We present an algorithm for estimating consistent dense depth maps and camera poses from a monocular video. We integrate a learning-based depth prior, in the form of a convolutional neural network trained for single-image depth estimation, with geometric optimization, to estimate a smooth camera trajectory as well as detailed and stable depth reconstruction. Our algorithm combines two complementary techniques: (1) flexible deformation-splines for low-frequency large-scale alignment and (2) geometry-aware depth filtering for high-frequency alignment of fine depth details. In contrast to prior approaches, our method does not require camera poses as input and achieves robust reconstruction for challenging hand-held cell phone captures containing a significant amount of noise, shake, motion blur, and rolling shutter deformations. Our method quantitatively outperforms state-of-the-arts on the Sintel benchmark for both depth and pose estimations and attains favorable qualitative results across diverse wild datasets.
1. Introduction
Consistent video depth is important but difficult to estimate from casually captured monocular videos because sensor, camera, scene, and motion degradations undermine geometric reconstruction. The proposed method combines learned depth priors with geometric alignment and filtering, without requiring input poses.
- Motivation: Casually captured videos combine noise, low-light artifacts, rolling-shutter deformation, shake, blur, dynamic objects, weak texture, repetition, and occlusion.These conditions make accurate and temporally consistent dense depth estimation challenging.
- Motivation: Traditional SFM and MVS pipelines often fail on these challenges, producing missing, noisy, or unusable reconstructions.SFM may fail before later processing, while MVS can contain substantial holes and noise even when SFM succeeds.
- Prior limitations: Learning-based depth methods handle many difficult situations through learned priors, but their predictions remain scale-ambiguous and geometrically inconsistent.The learned estimates are plausible without necessarily being accurate in a geometrically consistent sense.
- Prior limitations: Hybrid methods combine learned priors with geometry but commonly depend on precise per-frame poses, making failures catastrophic when SFM produces missing or erroneous poses.This dependence limits robustness for real-world videos with difficult motion or dynamic content.
- Proposed approach: The proposed algorithm jointly optimizes camera parameters and depth alignment, using spatially varying splines for large-scale alignment and a geometry-aware filter for fine details.Automatic segmentation masks relax alignment constraints around people, vehicles, and animals, while the filter removes residual jitter without blurring aligned details.
2. Related Work
Prior work spans geometry-based reconstruction, learned single-image depth, video depth estimation, visual odometry, and temporal consistency. The paper’s overview places its method at the intersection of learned depth, pose optimization, deformation-based alignment, and geometry-aware temporal filtering.
- Multi-view stereo: MVS estimates depth from multiple viewpoints, commonly using an SFM pipeline followed by triangulation and bundle adjustment.Learning-based MVS methods augment this geometry with data-driven priors, while conventional MVS assumes static scenes.
- Single-image depth estimation: Single-image depth research uses supervised, synthetic, crowdsourced, pseudo-ground-truth, movie-based, and self-supervised training sources.These approaches provide learned depth priors from diverse forms of supervision.
- Video-based depth estimation: Video depth methods combine camera-motion estimation and multi-view reconstruction, but many perform well only for static scenes.Motion or semantic segmentation is used by some methods to constrain moving-object depth.
- Visual odometry: Visual odometry and SLAM estimate relative camera poses from image sequences, yet generic scenes remain challenging for conventional geometric methods.These methods vary by direct versus feature-based processing and sparse versus dense reconstruction.
- Temporal consistency: Temporal consistency methods constrain video outputs explicitly with optical flow or implicitly with recurrent networks.The paper’s depth filter also operates along flow trajectories, but incorporates geometric awareness.
- Paper overview: The proposed pipeline takes monocular color video, predicts per-frame depth, jointly aligns poses and deformations, then filters fine-scale depth details.Green frames denote inputs and yellow frames denote outputs in the overview.
3. Overview
The paper removes the need for externally supplied camera poses by jointly optimizing pose parameters and depth alignment. Flexible deformation handles low-frequency inconsistencies, while geometry-aware filtering improves high-frequency detail.
- Motivation: CVD requires precise camera parameters computed with SFM, which can fail to produce, register, or accurately estimate poses.Such failures can prevent the downstream optimization from operating reliably.
- Contribution: The paper replaces test-time fine-tuning with joint optimization of camera parameters and depth alignment.This directly removes CVD’s dependence on input poses.
- Optimization: Pose optimization is sensitive to inaccurate depth because depth misalignments become noisy errors in the estimated camera trajectory.Alternating optimization uses SGD for CNN depth parameters and global continuous optimization for camera parameters.
- Deformation alignment: A spatially varying bilinear spline replaces per-frame camera scale to align initially inaccurate depth estimates more flexibly.Improved depth alignment supports smoother and more accurate pose trajectories.
- Depth filtering: A geometry-aware depth filter resolves high-frequency inconsistencies by low-pass filtering reprojected depth along flow trajectories.Because deformation first aligns the input, filtering can recover fine details instead of blurring them.
4. Method
The method jointly estimates camera poses and depth without requiring poses as input. Flexible deformation splines provide low-frequency alignment, while geometry-aware filtering restores fine-scale depth details and reduces residual jitter.
- Pose and depth formulation: The approach builds on CNN single-image depth priors, optical flow, and geometric reprojection to optimize depth and camera parameters from video.Camera intrinsics, extrinsics, and per-frame scales participate in the geometric formulation.
- Pose and depth formulation: Pose parameters and depth are alternately optimized with different machinery because camera trajectories require global continuous optimization while CNN depth uses SGD.Alternating optimization keeps one quantity fixed while optimizing the other.
- Pose and depth formulation: Initial depth misalignment can produce noisy poses and prevent convergence, motivating alignment improvements before fine depth refinement.The resulting feedback loop can degrade both pose and depth estimates.
- Pose and depth with flexible deformation: A spatially varying bilinear spline replaces per-frame scale coefficients, using grid handles and bilinear interpolation to correct large-scale depth misalignments.The deformation model is smooth and flexible across the image.
- Pose and depth with flexible deformation: Flexible deformation improves depth alignment, avoids jittery camera trajectories, and removes the need to iterate between pose optimization and fine-tuning.The method is also described as considerably faster after this change.
- Geometry-aware depth filtering: A geometry-aware spatio-temporal filter follows flow trajectories and reprojects depths from other frames to recover fine-scale details while preserving depth edges.The filter uses edge-preserving weights and complements the preceding low-frequency alignment.
- Implementation details: The proposed ratio loss avoids bias toward shrinking or growing scene scale and is reused to compute depth similarity in the filter.This addresses the small-depth bias of naive Euclidean distance.
5. Experimental Results
The method is evaluated on Sintel for depth and pose, with comparisons against state-of-the-art methods and analysis of failure cases and residual wobble. It achieves favorable pose accuracy, while limitations remain in baseline completeness and aligned-depth stability.
- The evaluation uses 23 Sintel sequences with ground-truth depth and camera poses, covering Clean and Final versions.
- Depth predictions are median-scaled to ground truth, exclude depths above 80 meters, and are compared using standard error and accuracy metrics.
- COLMAP fails on 11 of 23 sequences, causing CVD to lack results there and requiring sorted pixel-wise error curves instead of standard mean metrics.
- The proposed method outperforms MiDaS-v2 across mean ATE, RPE-T, and RPE-R, and significantly outperforms DeepV2D on both Sintel categories.
- Figure 5 compares sorted translational and rotational frame-pair RPE distributions, showing more accurate pose predictions for the proposed method than competing methods.
- Residual wobble remains in aligned depth maps; replacing spline deformation with a pixel-based deformation field would require denser constraints that exhaust the current solver’s memory.
6. Conclusions
The paper presents a general monocular-video optimization algorithm for consistent depth estimation without input poses or inference-time fine-tuning. It reports robust reconstruction on challenging handheld dynamic videos and better performance across diverse test beds.
- The algorithm estimates consistent monocular-video depth without requiring input camera poses or inference-time fine-tuning.
- It provides robust reconstruction for challenging dynamic videos captured casually with handheld devices.
- The method achieves better performance across diverse test beds.