PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepV2D: Video to Depth with Differentiable Structure from Motion

Zachary Teed, Jia Deng

arXiv:1812.04605v4cs.CV

TL;DR

Depth from video requires estimating both scene depth and camera motion. DeepV2D makes classical geometric stages differentiable, alternates their updates during inference, and reports strong performance across datasets, tasks, and cross-dataset settings.

  • Problem

    Existing approaches separately exploit neural representations or geometric principles for estimating depth and motion from video.

  • Method

    DeepV2D composes differentiable classical geometric algorithms into Motion and Depth Modules that alternate camera-motion and depth updates.

  • Results

    DeepV2D outperforms strong methods across varied datasets and tasks and achieves superior cross-dataset generalizability.

  • Takeaways & Limitations

    Differentiable geometric constraints support accurate depth estimation and generalization across datasets and tasks.

Abstract

from arXiv · show

We propose DeepV2D, an end-to-end deep learning architecture for predicting depth from video. DeepV2D combines the representation ability of neural networks with the geometric principles governing image formation. We compose a collection of classical geometric algorithms, which are converted into trainable modules and combined into an end-to-end differentiable architecture. DeepV2D interleaves two stages: motion estimation and depth estimation. During inference, motion and depth estimation are alternated and converge to accurate depth. Code is available https://github.com/princeton-vl/DeepV2D.

1 INTRODUCTION

DeepV2D combines differentiable classical geometry with neural representations to predict depth from video. It alternates motion and depth estimation, incorporating differentiable correspondence, PnP, and multiview stereo.

  • Motivation: Video-to-depth estimation traditionally uses Structure from Motion to jointly optimize 3D structure and camera motion before multiview stereo reconstruction.Deep networks also predict depth from single images, stereo images, or frame collections, often using learned visual cues.
  • Approach: DeepV2D converts classical geometric algorithms into differentiable network modules within an end-to-end trainable architecture.Its pipeline separates motion estimation and depth estimation into corresponding Motion and Depth Modules.
  • Inference and novelty: DeepV2D alternates depth and camera-motion updates at test time as block coordinate descent.The decomposition differs from joint optimization approaches and permits direct optimization over per-pixel depth rather than a limited depth basis.
  • Motion estimation: Flow-SE3 estimates dense 2D correspondence and unrolls one PnP iteration with Gauss-Newton updates over SE3 to refine camera motion.The updated motion is fed back into Flow-SE3 for finer pose estimation.
  • Depth estimation: The Depth Module formulates multiview stereo as a feed-forward network that builds cost volumes over video frames using trainable feature extraction and matching.This module takes camera motion as input and predicts depth.
  • Results: DeepV2D outperforms strong methods across varied datasets and tasks, with experiments reporting rapid convergence and superior cross-dataset generalizability.The cited comparisons include DeepTAM, DeMoN, BANet, and MVSNet.

2 RELATED WORK

Related work combines deep learning with multiview geometry for depth, motion, and reconstruction. DeepV2D differs by differentiating a classical SfM pipeline, including PnP, while separating motion and depth estimation.

  • Depth estimation: Stereo-depth networks extract learned features, build cost volumes, and apply 3D convolutions for feature matching.These architectures target depth estimation from rectified stereo images.
  • Motion estimation: Prior motion networks typically use generic components, whereas DeepV2D formulates motion estimation as a least-squares optimization problem.Its Flow-SE3 module maps 2D correspondence to a 6-dof camera-motion update.
  • Joint depth and motion: DeMoN and DeepTAM combine motion estimation and multiview reconstruction in trainable pipelines, with DeMoN using two frames and DeepTAM supporting a variable number of frames.Both separate depth and motion estimation in their respective architectures.
  • Comparison with BA-Net: Compared with BA-Net’s joint optimization over camera motion and depth-basis coefficients, DeepV2D uses block coordinate descent and directly optimizes per-pixel depth.Its depth module uses multiview cost-volume processing.

3 APPROACH

DeepV2D predicts dense depth from calibrated video by alternating trainable depth and camera-motion modules. Its depth module uses differentiable multiview stereo, while its motion module estimates pose corrections through learned correspondences and geometric optimization.

  • Full System: DeepV2D alternates a Depth Module and Motion Module during inference to update depth and camera motion estimates.The depth module takes camera motion as input, while the motion module takes depth as input and outputs a motion correction.
  • Depth Module: The Depth Module builds a cost volume over learned features from multiple viewpoints and predicts a dense keyframe depth map.It extracts 2D features, backprojects them using candidate depths and camera poses, pools information across views, and applies 3D stereo matching.
  • Depth Module: Differentiable bilinear sampling makes each cost volume differentiable with respect to camera pose.Candidate depths are used to reproject features from other frames into the keyframe coordinate system.
  • Motion Module: Flow-SE3 estimates dense 2D correspondences and unrolls one Gauss-Newton PnP iteration to produce an incremental SE3 camera-motion update.The updated motion can be fed back into Flow-SE3 for a finer pose update.
  • Motion Module: The Motion Module predicts residual flow and confidence maps from aligned feature pairs, then uses weighted reprojection errors in a least-squares optimization.Confidence weights are mapped to (0, 1) and determine the contribution of individual error terms.
  • Motion Module: Pose optimization can use all frame pairs jointly or only pairs connecting a selected keyframe to the remaining frames.Global optimization uses N×N-1 frame pairs, whereas keyframe optimization uses pairs of the form (1, j).

4 EXPERIMENTS

DeepV2D is evaluated across indoor and outdoor benchmarks for depth, pose, convergence, cross-dataset generalization, and tracking. It consistently outperforms existing methods while supporting variable initialization, input views, and inference iterations.

  • Benchmark results: DeepV2D reports strong depth results across NYU, ScanNet, SUN3D, and KITTI, including the highest accuracy on ScanNet and SUN3D without training on either dataset.The evaluation uses scale-matched depth because Structure from Motion cannot recover absolute scene scale.
  • NYU: DeepV2D outperforms existing single-view and multiview methods on NYU, while single-image depth initialization provides a slight performance improvement.The comparison includes DORN, DenseDepth, COLMAP, DfUSMC, MVSNet, and DeMoN.
  • Convergence and generalization: DeepV2D converges in few iterations, improves consistently with additional frames beyond its four-frame training input, and does not require retraining for variable views or iterations.The results support the effectiveness of alternating depth and motion estimation and attribute generalization to learned feature matching and correspondence.
  • ScanNet: DeepV2D outperforms BA-Net on ScanNet even when trained only on NYU, and training on ScanNet produces better performance.The ScanNet evaluation measures both depth and pose accuracy.
  • SUN3D: On SUN3D, DeepV2D outperforms DeepTAM and DeMoN without SUN3D training, despite DeepTAM receiving dataset poses during inference.DeepTAM evaluates its depth module in isolation, whereas DeepV2D estimates camera poses during inference.
  • KITTI and tracking: DeepV2D outperforms previous KITTI methods across all metrics and achieves lower translational rmse [m/s] than DeepTAM on most TUM-RGBD sequences.The system can be used as a basic SLAM system for tracking evaluation.

5 CONCLUSION

DeepV2D composes classical geometric algorithms into a fully differentiable deep learning pipeline. The architecture is flexible and performs well across varied tasks and datasets.

  • DeepV2D is a deep learning architecture built by composing classical geometric algorithms into a fully differentiable pipeline.
  • The architecture combines classical geometry with deep learning within one trainable system.
  • DeepV2D performs well across a variety of tasks and datasets.

A APPENDIX

The appendix describes differentiable least-squares optimization for pose estimation. It uses Gauss-Newton updates, Jacobians, linear-system solves, and backpropagation through the update.

  • A.1 LS-OPTIMIZATION LAYER: The optimization defines residual errors and combines them into a weighted objective function.
  • A.1 LS-OPTIMIZATION LAYER: Gauss-Newton updates minimize a second-order approximation of the objective over pose parameters.
  • A.1 LS-OPTIMIZATION LAYER: Each pose parameter is 6-dimensional, so optimizing N poses updates 6N variables.
  • A.1 LS-OPTIMIZATION LAYER: The update solves Hξ = −b with H = J^T WJ and b = J^T Wr(ξ1, ..., ξN).
  • A.1 LS-OPTIMIZATION LAYER: Training propagates gradients through the Gauss-Newton update, using Cholesky decomposition for the positive-definite system.

B TRAINING DETAILS

DeepV2D training uses a two-stage procedure: motion pretraining followed by joint end-to-end optimization. Experiments use short sampled video clips, perturbations, and image augmentations.

  • B TRAINING DETAILS: All network components are trained from scratch, with gradient checkpointing used to reduce memory and increase batch size.
  • B TRAINING DETAILS: NYU and ScanNet use 4-frame clips, while KITTI uses 5-frame clips sampled around a keyframe.
  • B TRAINING DETAILS: Training first optimizes the Motion Module using interpolated ground-truth depth, then jointly trains motion and depth end-to-end.
  • B TRAINING DETAILS: Data augmentation adjusts brightness and gamma, randomly scales image channels, and perturbs input camera poses.

C TIMING AND MEMORY USAGE

The timing and memory experiments compare DeepV2D variants using 8-frame inputs, with a single-frame FCRN baseline. Shallower or lower-resolution variants reduce resources but can reduce accuracy.

  • C TIMING AND MEMORY USAGE: Timing and peak memory are evaluated for different DeepV2D versions using 8-frame video sequences.
  • C TIMING AND MEMORY USAGE: The FCRN baseline differs by using a single frame instead of an 8-frame sequence.
  • C TIMING AND MEMORY USAGE: A single-hourglass variant increases Abs-Rel from 0.065 to 0.071, indicating a performance benefit from stacking hourglass networks.
  • C TIMING AND MEMORY USAGE: Half-resolution images decrease memory usage and inference time but slightly decrease accuracy.
  • C TIMING AND MEMORY USAGE: Replacing the 3D stereo network with correlation and a 2D encoder-decoder increases Abs-Rel from 0.065 to 0.135.

D ADDITIONAL TRACKING INFORMATION

Tracking results are reported for all Freiburg 1 sequences, with DeepV2D outperforming DeepTAM and DVO on most sequences and achieving the best average translational RMSE.

  • Table 7 reports tracking results for every sequence in the Freiburg 1 dataset.
  • DeepV2D outperforms DeepTAM and DVO on 12 of 16 RGB-D benchmark sequences.
  • DeepV2D achieves a lower translational RMSE averaged over all sequences than DeepTAM and DVO.The benchmark metric is translational RMSE [m/s].

E CAMERA POSE ABLATIONS

The camera-pose ablations show that classical SfM degrades depth accuracy, while DeepV2D’s learned motion estimates improve MVS-based reconstruction and the full alternating system.

  • The ablation replaces the trained motion module with alternative pose-estimation methods and also evaluates MVSNet with each SfM implementation.COLMAP and OpenMVG are evaluated only on sequences where they converge without error.
  • Replacing DeepV2D’s motion module with SfM degrades depth performance for both MVSNet and DeepV2D.Table 8 evaluates the impact of pose-estimation choices on depth accuracy.
  • Classical SfM is not robust enough to consistently produce accurate poses, leading to large test-set errors.
  • MVSNet performs better with poses estimated by DeepV2D than with alternative pose estimates, but remains below the full system.

F ADDITIONAL RESULTS

Additional visual results illustrate depth predictions on KITTI and NYU, including NYU comparisons with prior methods using seven-frame video clips.

  • Figure 6 visualizes DeepV2D depth predictions on the KITTI dataset.
  • Figure 7 presents additional NYU depth results using 7-frame video clips.
  • The NYU results are compared with Laina et al. (2016) and Ummenhofer et al. (2017).

G NETWORK ARCHITECTURES

The network architectures use learned modules for motion estimation and depth processing, combining feature extraction, residual flow estimation, and multiscale hourglass processing.

  • Motion Module Architecture: The motion module encodes each input image into a dense 1/4-resolution feature map before estimating residual flow and weights between feature-map pairs.Residual flow is predicted by an encoder-decoder network with skip connections formed by concatenating feature maps.
  • Depth Module Architecture: The depth module applies a 2D encoder to each video image, using residual convolutions and four nested hourglass modules across multiple scales.The nested hourglass feature dimensions are 64-128-192-256.
Loading 1812.04605v4…