Source-linked AI summary
Learning Depth from Monocular Videos using Direct Methods
Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, Simon Lucey
TL;DR
The paper addresses the gap between monocular-video and stereo-based depth learning, where scale ambiguity and learned pose prediction remain challenges. It introduces depth normalization and a differentiable direct visual odometry pose module, including a hybrid initialization-and-refinement scheme. The hybrid method provides the best reported performance among the tested pose strategies and achieves results comparable to calibrated-binocular training, while non-rigid scenes remain a bottleneck.
Problem
Monocular-video depth learning seeks to use abundant varied video without depth or stereo ground truth, but scale ambiguity and pose estimation limit existing approaches.
Method
The paper uses scale normalization and a differentiable Direct Visual Odometry module to estimate pose from predicted depth, with optional Pose-CNN initialization for refinement.
Results
The hybrid Pose-CNN initialization and DDVO refinement gives the best results, with performance comparable to Godard et al.'s calibrated-binocular method on KITTI.
Takeaways & Limitations
Direct visual odometry can replace a learned pose predictor while reducing learnable parameters, and combining initialization with DDVO improves pose prediction performance.
Takeaways & Limitations
The approach does not model world non-rigidity and therefore performs poorly on articulated objects such as bikers and pedestrians.
Abstract
from arXiv · showhide
The ability to predict depth from a single image - using recent advances in CNNs - is of increasing interest to the vision community. Unsupervised strategies to learning are particularly appealing as they can utilize much larger and varied monocular video datasets during learning without the need for ground truth depth or stereo. In previous works, separate pose and depth CNN predictors had to be determined such that their joint outputs minimized the photometric error. Inspired by recent advances in direct visual odometry (DVO), we argue that the depth CNN predictor can be learned without a pose CNN predictor. Further, we demonstrate empirically that incorporation of a differentiable implementation of DVO, along with a novel depth normalization strategy - substantially improves performance over state of the art that use monocular videos for training.
1. Introduction
The paper targets the performance gap between monocular-video and calibrated-stereo depth learning by replacing learned pose prediction with differentiable direct visual odometry and addressing scale ambiguity.
- Monocular-video training avoids depth or stereo ground truth but lags methods trained with calibrated stereo pairs.The paper frames this performance gap as its central focus while noting that monocular video is easier to obtain and more varied.
- The paper also reports that its depth predictions recover finer details, including tree trunks and advertising boards, than Zhou et al. on KITTI.
- Existing monocular methods use separate pose and depth CNNs, but the authors argue they inadequately address scale ambiguity and may cause training divergence.The paper identifies scale-sensitive depth regularization as the source of the problem.
- The proposed DDVO module estimates pose deterministically from dense depth and image reconstruction, without learnable pose parameters.Its differentiable implementation propagates training gradients from pose estimation to the depth estimator.
- A hybrid procedure initializes DDVO with a pretrained Pose-CNN and refines pose through DDVO, outperforming either module alone.The hybrid method achieves results comparable to Godard et al., a state-of-the-art method trained with calibrated binocular pairs on KITTI.
2. Learning depth estimation from videos
The paper learns inverse depth from monocular videos by directly optimizing photometric and smoothness losses, addressing scale ambiguity and replacing or augmenting learned pose prediction with differentiable visual odometry.
- The goal is to learn a CNN that predicts inverse depth from single images using monocular video without ground-truth depth or stereo.
- A two-step SfM-supervision procedure is theoretically sub-optimal because training on its estimated depth maps need not minimize the original SfM cost.
- 2.1. Scale ambiguity: Monocular depth and pose have scale ambiguity: photometric loss is scale-invariant, while the smoothness prior is scale-sensitive.
- 2.1. Scale ambiguity: Without normalization, inverse-depth scale decreases toward zero, smoothness is penalized less, structure mistakes increase, and training diverges.
- 2.1. Scale ambiguity: Normalizing each depth-CNN output by its mean stabilizes inverse-depth scale and removes the loss function’s scale-sensitive problem.
- 2.2. Modeling pose predictor: Differentiable DVO uses the current dense depth map to estimate pose without learnable parameters and supplies additional back-propagation signals to depth prediction.
- 2.2. Modeling pose predictor: A hybrid procedure initializes DDVO with a pretrained Pose-CNN, and the authors report better performance than using Pose-CNN or DDVO alone.
3. Differentiable direct visual odometry
The differentiable direct visual odometry module estimates camera pose by minimizing photometric error from image intensities, depth, and geometric warping, then propagates gradients through this process.
- 3.1. Direct visual odometry: Direct visual odometry takes a reference image, its depth map, a source image, and known camera intrinsics as inputs.
- 3.1. Direct visual odometry: The reference-to-source projection uses inverse depth, camera translation, exponential-coordinate rotation, and image-plane projection.
- 3.1. Direct visual odometry: It minimizes photometric error between the reference image and the source image warped according to the camera pose.
- 3.1. Direct visual odometry: Gauss–Newton optimization with inverse compositional updates avoids recomputing the Jacobian and Hessian at every iteration.
- 3.1. Direct visual odometry: Out-of-view projected pixels are excluded using a binary diagonal weight matrix before computing the pose update.
- 3.1. Direct visual odometry: The pose estimate is iteratively updated and composed until convergence.
- 3.1. Direct visual odometry: Direct methods are restricted by small inter-frame motion, so the implementation uses image-depth pyramids and coarse-to-fine updates.
- 3.2. Differentiable DVO: The differentiable implementation forms its regressor deterministically and differentiates through the geometric transformation updates.
4. Training loss
The training objective combines multi-scale appearance dissimilarity with inverse-depth smoothness, using bidirectional frame reconstruction and strategies intended to preserve depth sharpness.
- The loss combines appearance dissimilarity and prior smoothness terms, aggregated across four output scales.Appearance compares pixel-wise correspondences in triplets, while the prior encourages smooth inverse-depth predictions.
- Bidirectional warping compares the first and last images with the second, then reverses the comparisons to augment training at low computational cost.
- Photometric supervision uses L1 loss at coarser scales and combines L1 with single-scale SSIM at the finest scale.
- Second-order inverse-depth gradients encourage flat slopes, while downsampling and coarse-scale smoothness terms support sharper depth boundaries.
5. Experiments
Experiments evaluate the method on KITTI and its transfer to Make3D across multiple pose, normalization, and initialization settings. Depth normalization improves the Pose-CNN configuration, while Pose-CNN initialization followed by DDVO gives the strongest unsupervised KITTI results; non-rigid scenes remain a limitation.
- Experimental setup: Experiments train on 56 monocular KITTI sequences and evaluate on 697 Eigen-split test images using established cropped-region depth measures.The training data comprise 28 training sequences captured with stereo pairs, yielding two monocular sequences per sequence.
- Experimental setup: The training set removes static frames with mean optical-flow magnitude below 1 pixel and uses three-frame clips resized to 128 × 416 pixels.
- Architectures and training: The Depth-CNN uses a U-Net-like encoder-decoder with skip connections and multi-scale sigmoid inverse-depth outputs, scaled by 10 with an added 0.01.
- Architectures and training: The Pose-CNN consumes three concatenated frames and predicts exponential coordinates instead of Euler angles; training uses smoothness weight 0.01, Adam, learning rate 0.0001, and batch size 1.
- KITTI results: The baseline without inverse-depth normalization diverges after 10 epochs, so its result is reported at the 10th epoch.
- KITTI results: Depth normalization significantly improves all measures with the same architecture, bringing Ours (Pose-CNN) close to stereo-trained Godard et al. and beyond Zhou et al.
- KITTI results: Pose-CNN-initialized DDVO improves both standalone configurations and achieves the best results among unsupervised methods trained on KITTI.
- Make3D results: Direct KITTI-to-Make3D transfer without tuning generalizes moderately well, achieving similar or better results than other state-of-the-art approaches on a central crop.
6. Discussion
The paper identifies scale ambiguity as essential in monocular-video training and finds that hybrid Pose-CNN initialization with DDVO refinement gives the best results. The approach remains limited by its treatment of non-rigid scenes.
- Scale ambiguity must be considered when training depth from monocular videos.
- The method uses fewer learned pose parameters by replacing learned pose prediction with Direct Visual Odometry.
- Table 2 evaluates KITTI-trained models on Make3D without Make3D training data, using a central crop and depths below 70 meters.
- Pose-CNN initialization followed by DDVO pose refinement gives the best results.
- The approach does not model world non-rigidity and performs poorly on articulated objects such as bikers and pedestrians.
1. Differentiable vs non-differentiabel direct visual odometry module
The paper compares non-differentiable and differentiable direct visual odometry within monocular-video depth training. Differentiable DVO converges faster and avoids the poor local minimum observed with the non-differentiable alternative.
- The non-differentiable DVO variant estimates pose first and treats it as constant during depth optimization.
- The toy experiment finds that DVO converges more slowly than DDVO and becomes stuck in a poor local minimum.
- The paper concludes that differentiable DVO is necessary for training a depth estimator from monocular videos.
2. Estimate pose from depth prediction
The paper tests predicted depth as initialization for direct visual odometry and finds that its improved depth accuracy produces significantly better pose estimation.
- Direct visual odometry evaluated on the paper’s depth maps yields significantly better pose estimation than when evaluated on Zhou et al.’s depth maps.
3. Video demo on KITTI sequence
A video demo compares the paper’s depth estimation with Zhou et al.’s method on a KITTI sequence, using models pretrained on Cityscapes and fine-tuned on KITTI.
- The demo compares both methods on a KITTI Eigen test-split sequence after Cityscapes pretraining and KITTI fine-tuning.