Source-linked AI summary
Consistent Video Depth Estimation
Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, Johannes Kopf
TL;DR
Dense monocular-video depth estimation must handle capture artifacts, weak geometric constraints, and dynamic objects without sacrificing temporal consistency. The paper fine-tunes a pre-trained single-image depth network at test time using SfM-derived geometric constraints, producing geometrically consistent depth with plausible detail in less constrained regions. The method compares favorably with state-of-the-art approaches and supports visual effects, while remaining limited by unreliable poses, flow, and significant dynamic motion.
Problem
Dense and accurate depth from handheld monocular video is difficult because video artifacts and dynamic objects challenge existing reconstruction methods.
Method
The method fine-tunes a pre-trained single-image depth network at test time using geometric constraints extracted from a video reconstruction.
Results
The method produces geometrically consistent and temporally coherent depth across entire videos and compares favorably against several state-of-the-art depth estimation algorithms.
Takeaways & Limitations
Improved depth quality and consistency enable compelling video-based visual effects that interact with dense scene content.
Takeaways & Limitations
The method is limited by unreliable COLMAP poses and flow, and it breaks down under significant dynamic scene motion.
Abstract
from arXiv · showhide
We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Unlike the ad-hoc priors in classical reconstruction, we use a learning-based prior, i.e., a convolutional neural network trained for single-image depth estimation. At test time, we fine-tune this network to satisfy the geometric constraints of a particular input video, while retaining its ability to synthesize plausible depth details in parts of the video that are less constrained. We show through quantitative validation that our method achieves higher accuracy and a higher degree of geometric consistency than previous monocular reconstruction methods. Visually, our results appear more stable. Our algorithm is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion. The improved quality of the reconstruction enables several applications, such as scene reconstruction and advanced video-based visual effects.
1 INTRODUCTION
Dense, accurate reconstruction from handheld monocular video is difficult because video-specific capture artifacts and dynamic objects undermine traditional geometry-based methods. The paper combines geometric constraints with learned single-image depth priors, using test-time fine-tuning to improve consistency while retaining plausible detail synthesis.
- Handheld phone video is attractive for spontaneous, large-scale capture, but fully dense and accurate reconstruction remains difficult.
- Video introduces noise, shake, motion blur, rolling-shutter deformation, small baselines, and dynamic objects beyond standard reconstruction challenges.
- Traditional multi-view reconstruction can be geometrically accurate when matches are correct, but noisy matches and heuristic regularization often produce incorrect geometry or holes.
- Single-image depth methods synthesize plausible depth in weakly constrained regions and handle dynamic scenes, but their frame-wise predictions flicker and become geometrically inconsistent.
- The method fine-tunes a single-image depth network at test time using video-derived geometric constraints, combining accurate constrained regions with plausible details elsewhere.
- The improved quality and consistency enable applications including fully automatic video special effects that interact with dense scene content.
2 RELATED WORK
Prior work addresses monocular depth, temporal consistency, and multi-view reconstruction through separate learning or geometric strategies, each with important limitations. This paper emphasizes global geometric consistency while using single-image models to handle dynamic or weakly constrained content.
- Supervised monocular depth estimation: Supervised monocular depth methods learn from image data, but training requires ground-truth depth maps that are difficult to acquire.
- Self-supervised monocular depth estimation: Self-supervised methods commonly minimize photometric reprojection error, yet photometric losses can permit inconsistent geometry and perform poorly on temporally distant frames.
- Multi-view reconstruction: Multi-view stereo achieves strong reconstruction from multiple viewpoints but assumes static scenes, producing errors or dropping pixels for dynamic objects.
- Depth from video: Existing video-to-depth systems use warped-frame cost volumes but do not account for dynamically moving objects, whereas this method estimates depth from fine-tuned single-image models.
- Temporal consistency: Temporal-consistency methods add temporal losses or post-processing to reduce flicker in independently generated video outputs.
- Contribution: The paper targets globally consistent depth rather than only per-frame accuracy, yielding detailed reconstructions with greater temporal smoothness for video applications.
3 OVERVIEW
The system first obtains camera poses and geometric constraints from a monocular video, then fine-tunes a pre-trained single-image depth network on sampled frame pairs. This produces dense, temporally coherent depth while preserving learned regularization in weakly constrained regions.
- Motivation: Casually captured videos combine handheld-camera artifacts, poor lighting, blur, and moving objects that challenge depth reconstruction.
- Motivation: Traditional reconstruction often leaves holes or noisy depth in problematic regions, while single-image methods produce plausible but temporally flickering predictions.
- Test-time Training: Test-time training samples frame pairs, compares correspondences with depth-based reprojections, and backpropagates spatial and disparity losses through shared network weights.
- Overview: The method combines single-image depth priors with geometric constraints extracted from video so the network learns consistency for a particular sequence.
- Pre-processing: Pre-processing uses COLMAP-based structure from motion, with people segmentation to improve pose estimation and matching under dynamic motion.
- Pre-processing: Because monocular reconstruction is scale-ambiguous, the SfM reconstruction is adjusted to match the learning-based depth estimate in a robust average sense.
- Results: Long-term frame constraints improve depth sharpness and temporal stability, while final depth maps remain coherent across the video and delineate occluding boundaries.
4 PRE-PROCESSING
Pre-processing establishes camera geometry and compatible scales, then selects a linear-sized set of frame pairs for dense optical-flow constraints.
- Camera registration: COLMAP estimates each frame’s camera intrinsics, extrinsics, and semi-dense multi-view stereo depth.
- Camera registration: People are segmented with Mask R-CNN and excluded from feature extraction to reduce reconstruction errors from dynamic objects.
- Scale calibration: A global scale adjustment makes structure-from-motion and learning-based depth values compatible by multiplying camera translations.
- Frame sampling: A hierarchical scheme reduces all O(N^2) frame pairs to O(N), combining consecutive pairs with progressively sparser higher-level samples.
- Optical flow estimation: Homography alignment precedes FlowNet2 estimation, while forward-backward checks remove unreliable pixels and low-overlap frame pairs.
5 TEST-TIME TRAINING ON INPUT VIDEO
The method fine-tunes a pretrained depth network using optical-flow and reprojection constraints, combining image-space and disparity consistency while addressing dynamic motion.
- Geometric loss: Optical flow identifies corresponding pixels, whose agreement with depth-reprojected points provides a geometric consistency signal.
- Geometric loss: The geometric loss contains image-space and disparity terms that penalize disagreement between flow mappings and depth reprojections.
- Geometric loss: For each valid flow pixel, the pair loss combines both terms with balancing coefficient λ = 0.1.
- Dynamic motion: Consistent motion aligned with epipolar geometry can produce wrong depth, whereas conflicting motion is empirically tolerated during test-time training.
- Optimization: Standard backpropagation fine-tunes pretrained network weights, transferring plausible single-image depth to regions difficult for traditional reconstruction.
6 RESULTS AND EVALUATION
The evaluation uses custom stereo video and public datasets to compare accuracy, consistency, ablations, and applications. The method performs favorably overall, especially on instability and drift, but remains limited by unreliable poses and extreme motion.
- Experimental Setup: The evaluation combines custom stereo videos containing static scenes and gentle object motion with TUM, ScanNet, and KITTI benchmarks.The custom test set addresses the lack of suitable existing datasets covering both static and mildly dynamic video scenes.
- Comparative Evaluation: Our method compares favorably against prior algorithms on photometric error, instability, and drift across completeness levels.The strongest gains are reported for instability and drift, highlighting temporal and geometric consistency.
- Comparative Evaluation: Visual comparisons show that multi-view stereo creates holes, single-image methods flicker, and short-window video methods drift, whereas our reconstruction is dense and temporally consistent.The comparison uses sample frames and scanline slices through color or depth spatio-temporal volumes.
- Ablation Study: Long-term frame constraints improve depth sharpness and temporal stability, while disparity loss reduces flicker but can blur results with only consecutive pairs.The ablation isolates the trade-off between temporal stabilization and sharpness under short temporal windows.
- Public Benchmarks: On public benchmarks, the method improves substantially over the Li et al. baseline and is competitive with state-of-the-art methods, but is slightly below DeepV2D on ScanNet.A separate monocular benchmark shows improvement on more than 80% of testing frames without clear average improvement because large dynamic objects can corrupt COLMAP poses.
7 CONCLUSIONS
The paper presents a test-time fine-tuning strategy that combines geometric constraints from conventional multi-view reconstruction with a pre-trained single-image depth network. The resulting estimates are geometrically consistent across an entire monocular video and compare favorably with several state-of-the-art algorithms, enabling video-based visual effects.
- The method fine-tunes a pre-trained single-image depth network using geometric constraints extracted from conventional multi-view reconstruction.
- The network produces geometrically consistent depth estimates across the entire video.
- The method compares favorably against several state-of-the-art depth estimation algorithms based on extensive quantitative and qualitative evaluation.
- Consistent video depth estimation enables compelling video-based visual effects.