Source-linked AI summary
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, Ian Reid
TL;DR
The paper addresses the need for vast annotated datasets and limited cross-domain generalization in supervised single-view depth estimation. It trains a CNN from scratch using photometric reconstruction from stereo pairs with known camera motion, rather than ground-truth depths. Trained on less than half of KITTI, the network achieves performance comparable to state-of-the-art supervised methods.
Problem
Supervised single-view depth estimation requires vast annotated data, while existing networks generalize poorly outside their training domain.
Method
A CNN predicts source-image depth from stereo pairs with known camera motion, using differentiable photometric reconstruction of the source as the training loss.
Results
Trained on less than half of KITTI, the network achieves performance comparable to current state-of-the-art supervised single-view depth methods.
Takeaways & Limitations
Single-view depth prediction can be trained end-to-end from scratch in a fully unsupervised manner using stereo-rig data without manual depth annotation.
Takeaways & Limitations
The chosen smoothness regularizer can over-smooth estimated motion and may produce less sharp depth maps than edge-preserving alternatives.
Abstract
from arXiv · showhide
A significant weakness of most current deep Convolutional Neural Networks is the need to train them using vast amounts of manu- ally labelled data. In this work we propose a unsupervised framework to learn a deep convolutional neural network for single view depth predic- tion, without requiring a pre-training stage or annotated ground truth depths. We achieve this by training the network in a manner analogous to an autoencoder. At training time we consider a pair of images, source and target, with small, known camera motion between the two such as a stereo pair. We train the convolutional encoder for the task of predicting the depth map for the source image. To do so, we explicitly generate an inverse warp of the target image using the predicted depth and known inter-view displacement, to reconstruct the source image; the photomet- ric error in the reconstruction is the reconstruction loss for the encoder. The acquisition of this training data is considerably simpler than for equivalent systems, requiring no manual annotation, nor calibration of depth sensor to camera. We show that our network trained on less than half of the KITTI dataset (without any further augmentation) gives com- parable performance to that of the state of art supervised methods for single view depth estimation.
1 Introduction
Single-view depth estimation is limited by the cost, sparsity, and domain dependence of supervised depth data. The paper proposes training a depth-prediction CNN end-to-end from unlabeled stereo imagery using visual geometry.
- Motivation: Supervised CNNs require large manually labeled datasets, while outdoor single-view depth training additionally needs expensive hardware and careful acquisition.KITTI depth maps are sparse, with less than 5% density, and have a limited reliable depth range.
- Contribution: The framework aims toward in-situ and lifelong learning by enabling networks to be trained or tuned with continuously acquired real imagery.The authors identify unsupervised training regimes that match or exceed supervised performance as an important step toward these settings.
- Motivation: Supervised depth networks do not generalize well outside their immediate domain, requiring separately trained models for indoor and street scenes.Transferring the task to another domain would require a new aligned RGB-D dataset and retraining.
- Motivation: Generating proxy disparity labels with standard stereo can bake stereo algorithms’ systematic errors into the learned predictor.Sensor flare, motion blur, lighting changes, and shadows are present in real images but are rarely handled adequately by standard stereo algorithms.
- Contribution: The proposed approach trains a single-view depth CNN from scratch and fully unsupervised using real, unannotated imagery captured with a stereo rig.The method uses an autoencoder-inspired framework that leverages visual geometry rather than manual depth annotation.
2 Approach
The approach trains a CNN for single-view depth from stereo or other image pairs with known motion, replacing learned decoding with geometric inverse warping and photometric reconstruction.
- Training data: Training uses image pairs with known camera motion, such as rectified stereo pairs or images from a moving SLAM system.These data are described as easier to acquire than calibrated depthmaps and aligned images.
- Geometric reconstruction: The CNN predicts a depth map from the source image, and the target image is inverse-warped using that prediction and the inter-view displacement.For a rectified stereo rig, disparity is parameterized as fB/di(x).
- Geometric reconstruction: The reconstruction loss is the differentiable photometric difference between the source image and the inverse-warped target image.The paper states that this loss is correlated with prediction error and can rank depth maps without ground-truth labels.
- Autoencoder formulation: The method is analogous to an autoencoder, but uses a fixed geometric image warp instead of a learned decoder.This removes the need to learn the decoder while making the reconstruction loss encourage correct depth.
- Regularization: Because photometric loss is ambiguous in homogeneous regions, the method adds L2 regularization on disparity discontinuities to obtain a unique depth map.The regularization strength is denoted by γ, and the selected regularizer is acknowledged to oversmooth estimated motion.
- Consistency and assumptions: The loss is related to optic-flow costs, while shared CNN parameters enforce consistency across thousands of correlated training images.The paper assumes a fixed rectified stereo setup for training simplicity but states that the method generalizes to different calibrated stereos.
3 Coarse-to-fine training with skip architecture
The training procedure addresses warp linearization limits with a coarse-to-fine, iteratively warped network that upsamples and refines disparity predictions using earlier CNN features.
- Warp linearization: Back-propagation requires linearizing the warp image around the current disparity estimate using a Taylor expansion.The linearization uses the horizontal gradient of the warp image evaluated at the previous disparity iterate.
- Coarse-to-fine optimization: Because linearization is valid only for small disparity updates, larger motions are handled with coarse-to-fine estimation and iterative warping.This strategy is described as established in stereo and optic-flow optimization.
- Skip refinement: The architecture combines predictions from deeper layers with inputs from pooling layers through fully convolutional upsampling blocks.Figure 3 identifies F1 and F2 as upsampling blocks combining predictions from L7 and L8 with P3 and P2 inputs.
- Skip refinement: The network upsamples coarse disparities with bilinear interpolation and refines them by adding zero-initialized 1 × 1 convolution outputs.Earlier CNN features are fused with upsampled predictions to recover finer image details.
4 Network Architecture
The encoder follows AlexNet through C5, replaces its fully connected layer with a fully convolutional layer, and uses skip connections to preserve spatial detail during upsampling.
- Encoder design: The deep convolutional encoder is similar to AlexNet through the C5 layer.The fully connected layer is replaced rather than retained.
- Fully convolutional layer: The replacement layer uses 2048 convolution filters of size 5 × 5, reducing network parameters and allowing variable-size test inputs.A 5 × 18 convolution is also noted as an alternative for increased capacity.
- Spatial preservation: The fully convolutional design preserves spatial information and enables stage-wise upsampling after the L7 output.This supports the coarse-to-fine stereopsis-based autoencoder.
- Skip architecture: Skip connections combine coarse depth predictions with local image information to refine details lost in later CNN layers.The design follows the observation that finer image details are lost in the last layers of deep convolutional networks.
5 Experiments
Experiments on KITTI evaluate training stages, augmentation, comparisons with supervised methods, and stereo-based baselines. The unsupervised framework achieves competitive depth prediction while improving with higher-resolution training and augmentation.
- Experimental setup: The KITTI evaluation uses 23,488 stereo pairs, with 22,600 for training and the remainder for validation, without ground-truth depths.Testing uses 697 images, and predictions are upscaled for evaluation.
- Training procedure: The network is trained in multiple stages from coarse to finer resolutions, adding upsampling layers iteratively.Each upsampling layer increases prediction resolution by approximately 2.
- Effect of upsampling: Higher-resolution training gradually improves disparity estimation, while skip connections provide diminishing returns after two upsampling layers.At layer L10, the version without skip connections outperforms its skip-connected counterpart.
- Fine tuning with augmentation: 8× data augmentation improves reconstruction quality and object-edge localization during fine-tuning.The authors highlight this as useful because new stereo-rig data can be collected cheaply in the wild.
- Comparison with state-of-the-art: The method achieves the lowest RMS and square relative error on KITTI and performs on par with state-of-the-art methods on the other evaluation measures.Eigen et al. obtains slightly lower log RMS, a metric directly optimized by the supervised comparison methods.
- Comparison with baselines: The randomly initialized, fully unsupervised network achieves on-par results with supervised methods and visually sharper details for nearby objects.Compared with, traffic lights, trucks, and pedestrians are more detailed in the proposed inverse-depth predictions.
- Comparison with stereo baselines: The proposed framework outperforms the HS→CNN baseline on all accuracy measures and performs on par with HS-stereo overall.HS-stereo estimates closer points more precisely, but its edge depths can be worse than the single-view predictions.
- Proxy-supervision analysis: SGM→CNN performs on par with fully supervised single-view depth estimation but remains less accurate than the proposed approach.The authors associate this gap with systematic holes in the SGM stereo reconstructions and find correlated errors between SGM-stereo and SGM→CNN.
6 Related work
The paper situates its geometry-based unsupervised depth estimator among approaches using synthetic data, learned stereo features, novel-view synthesis, and camera-motion learning.
- Synthetic and semi-synthetic datasets aim to reduce annotation requirements but may not capture the full variability of real-world images.
- Learned stereo methods combine CNN features with classical stereo, but rely on local matching and ground-truth data.
- Novel-view synthesis methods infer depth-like maps as intermediate representations but do not evaluate the computed depth maps directly.
- Camera-motion learning regresses 6DOF poses, whereas this work trains a CNN for image synthesis and single-view depth estimation.
7 Conclusions
The paper presents an end-to-end, fully unsupervised single-view depth network trained from scratch using stereo-rig imagery. On less than half of KITTI, it achieves performance comparable to state-of-the-art supervised methods, while several extensions remain untested.
- The proposed network is the first described as trainable end-to-end from scratch without supervision using data captured by a stereo rig.
- Training on less than half of KITTI yielded performance comparable to current state-of-the-art supervised single-view depth methods.
- The authors propose continuous in-the-wild stereo training, KITTI augmentation, and monocular SLAM motion as natural extensions.
- The hypothesis that learned low-level features will help classification remains to be proven experimentally.