PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields

Fayao Liu, Chunhua Shen, Guosheng Lin, Ian Reid

arXiv:1502.07411v6cs.CV

TL;DR

Single-image depth estimation is challenging because monocular scenes are ambiguous and prior methods often rely on geometric priors, extra information, or hand-crafted features. The paper combines deep CNNs with continuous CRFs in unified models, including a faster fully convolutional variant, and reports state-of-the-art performance on indoor and outdoor datasets.

  • Problem

    Single-monocular-image depth estimation is an ill-posed problem, with prior approaches often relying on geometric priors, extra information, or hand-crafted features.

  • Method

    The paper jointly learns CNN-based unary and pairwise potentials in a continuous CRF framework and introduces a fully convolutional superpixel-pooling variant.

  • Results

    The proposed method outperforms state-of-the-art depth estimation methods on both indoor and outdoor scene datasets.

  • Takeaways & Limitations

    Deep CNNs and continuous CRFs provide a unified framework for single-image depth estimation, while the faster variant enables deeper networks for better performance.

  • Takeaways & Limitations

    The method does not exploit geometric cues.

Abstract

from arXiv · show

In this article, we tackle the problem of depth estimation from single monocular images. Compared with depth estimation using multiple images such as stereo depth perception, depth from monocular images is much more challenging. Prior work typically focuses on exploiting geometric priors or additional sources of information, most using hand-crafted features. Recently, there is mounting evidence that features from deep convolutional neural networks (CNN) set new records for various vision applications. On the other hand, considering the continuous characteristic of the depth values, depth estimations can be naturally formulated as a continuous conditional random field (CRF) learning problem. Therefore, here we present a deep convolutional neural field model for estimating depths from single monocular images, aiming to jointly explore the capacity of deep CNN and continuous CRF. In particular, we propose a deep structured learning scheme which learns the unary and pairwise potentials of continuous CRF in a unified deep CNN framework. We then further propose an equally effective model based on fully convolutional networks and a novel superpixel pooling method, which is $\sim 10$ times faster, to speedup the patch-wise convolutions in the deep model. With this more efficient model, we are able to design deeper networks to pursue better performance. Experiments on both indoor and outdoor scene datasets demonstrate that the proposed method outperforms state-of-the-art depth estimation approaches.

1 INTRODUCTION

The paper addresses the ill-posed problem of estimating depth from a single monocular image by combining deep CNNs with continuous CRFs. It introduces unified and faster models that outperform state-of-the-art methods on indoor and outdoor datasets.

  • Motivation: Single-image depth estimation is ill-posed because one captured scene can correspond to numerous real-world scenarios.The task is important for vision applications using RGB datasets and outdoor settings where sensor depth can be noisy.
  • Related work: Prior approaches rely on geometric assumptions, retrieved candidate images, extra information, or hand-crafted features, limiting generality or efficiency.Some methods also require approximate learning and inference procedures.
  • Contribution: The proposed DCNF formulates depth estimation as continuous CRF learning without geometric priors or extra information, jointly modeling CNN-based unary and pairwise potentials.The continuous formulation permits analytical partition functions and unified deep structured learning.
  • Contribution: DCNF-FCSP combines fully convolutional networks with superpixel pooling to preserve object boundaries while accelerating training and prediction by ∼10 times.The efficiency improvement enables deeper networks while producing similar prediction accuracy.
  • Results: The proposed method outperforms state-of-the-art depth estimation methods on both indoor and outdoor scene datasets.The paper contrasts its CRF-based structured modeling with direct CNN regression and related hybrid approaches.

2 DEEP CONVOLUTIONAL NEURAL FIELDS

This section introduces the detailed DCNF model for depth estimation from a single image and establishes notation for its mathematical presentation.

  • The section presents the details of the deep convolutional neural field model for single-image depth estimation.
  • Boldfaced uppercase letters denote matrices, while boldfaced lowercase letters denote column vectors.
  • The symbol 0 denotes a column vector whose elements are all zero.

2.1 Overview

The model represents monocular depth estimation over superpixels with a continuous CRF whose unary and pairwise potentials are jointly learned in a CNN framework. It predicts superpixel depths through CNN regression and neighboring-superpixel relations, with MAP inference available in closed form.

  • 2.1 Overview: The model represents an image as superpixel nodes, each assigned the depth of its centroid.The depth vector contains one continuous value for each of the n superpixels.
  • 2.1 Overview: The continuous CRF supports MAP depth prediction through closed-form solutions.The model formulates prediction as MAP inference, and the continuous formulation permits analytic treatment of the relevant integral.
  • 2.1 Overview: The energy combines unary potentials for individual superpixels with pairwise potentials over neighboring superpixels.The unary term regresses a superpixel depth, while the pairwise term encourages similar depths for neighboring regions with similar appearances.
  • 2.1 Overview: The unary network processes patches centered on superpixel centroids, while the pairwise network processes similarity vectors for neighboring pairs.The unary CNN outputs one regressed depth per superpixel; pairwise outputs are learned from neighboring-superpixel similarities.
  • 2.1 Overview: The unary potentials are coordinate-independent, providing translation invariance, and the pairwise potentials explicitly model neighboring-superpixel relations.These design choices are stated as core properties of the model.

2.2 Potential functions

The model constructs unary potentials from CNN depth regressions and pairwise potentials from learned similarities between neighboring superpixels. Together, these terms encode local depth estimates and appearance-based smoothness.

  • 2.2.1 Unary potential: The unary potential is a least-squares penalty between a superpixel’s depth and its CNN-regressed value.The regressed depth z_p is parameterized by the CNN parameters θ.
  • 2.2.1 Unary potential: The unary CNN uses five convolutional and four fully connected layers on resized 224×224 patches centered at superpixel centroids.CNN parameters are shared across all superpixels.
  • 2.2.2 Pairwise potential: The pairwise potential uses K similarity observations to encourage smoothness between neighboring superpixels.The pairwise network produces an output R_pq for each neighboring pair.
  • 2.2.2 Pairwise potential: The pairwise coefficients β_k are constrained by β_k ≥ 0 as a sufficient condition for an integrable partition function.The paper notes that this condition is sufficient but not necessary.
  • 2.2.2 Pairwise potential: The three pairwise similarities measure colour difference, colour-histogram difference, and texture disparity using local binary patterns.These observations are computed for neighboring superpixel pairs.

2.3 Learning

Learning minimizes the regularized negative conditional log-likelihood using exact continuous-CRF calculations and back-propagated gradients. Prediction reduces to solving a linear system with a closed-form MAP solution.

  • 2.3 Learning: The energy is expressed using a regularized Laplacian matrix A = I + D − R.D − R is the graph Laplacian, with R formed from pairwise affinities.
  • 2.3 Learning: Positive definiteness of A and the quadratic energy enable analytic calculation of the continuous CRF partition-function integral.This analytic calculation underlies the tractable probability distribution and likelihood.
  • 2.3 Learning: Training minimizes regularized negative conditional log-likelihood over the network parameters.The objective includes weight decay parameters λ1 and λ2.
  • 2.3 Learning: SGD-based back propagation computes gradients for both unary parameters θ and pairwise parameters β.The pairwise gradient uses matrix derivatives involving J and the trace operator.
  • 2.3.2 Depth prediction: The MAP prediction is y⋆ = A−1z, and the pairwise-free case reduces to the CNN baseline y⋆ = z.The inverse product can instead be obtained by solving a linear system rather than explicitly computing A−1.

2.4 Speeding up training using fully convolutional networks and superpixel pooling

DCNF-FCSP replaces redundant patch-wise convolutions with a fully convolutional network and superpixel pooling, preserving the CRF-based unary/pairwise structure while substantially reducing computation.

  • DCNF-FCSP overview: The model feeds fully convolutional feature maps and superpixel segmentation into pooling, then uses three fully connected layers to produce unary output z.The unary output z and pairwise output R define the CRF potentials optimized by negative log-likelihood.
  • DCNF-FCSP overview: DCNF-FCSP performs convolution over the entire image once instead of hundreds of overlapping superpixel patches.This reduces redundant computation and GPU memory usage.
  • DCNF-FCSP overview: The resulting model provides around 10 times training speedup with almost the same prediction accuracy as DCNF.Its efficiency enables deeper network designs for better performance.
  • Superpixel pooling: Superpixel pooling associates convolutional maps with input superpixels by upsampling, masking, and average pooling within each region.The resulting n superpixel feature vectors have d dimensions and support unary-potential construction.
  • Fully convolutional networks: The fully convolutional network accepts arbitrarily sized images and outputs spatial convolution maps for dense prediction.Its use here targets the patch-wise convolutions that dominate the original DCNF computation.
  • Superpixel pooling: The pooling layer establishes associations between convolutional feature maps and superpixels through a linear forward transformation with computable backward gradients.This preserves superpixel boundary information while retaining efficient convolutional processing.

2.5 Implementation details

The implementation uses MatConvNet on a GTX 780 GPU and initializes early convolutional layers from ImageNet-trained models before pre-training and whole-network fine-tuning.

  • Implementation details: Training uses VLFeat MatConvNet with modifications on a desktop equipped with an NVIDIA GTX 780 GPU and 6GB memory.The original DCNF has approximately 40 million parameters, while DCNF-FCSP has around 5.8 million before its very deep design reaches around 20 million.
  • Implementation details: DCNF initializes its first six unary layers from an ImageNet-trained CNN, then pre-trains the remaining network before fine-tuning.The pre-training uses momentum 0.9, weight decay 0.0005, and a learning rate initialized at 0.0001.
  • Implementation details: DCNF-FCSP initializes its first five layers from the same ImageNet-trained model and follows the same pre-training and whole-model fine-tuning protocol.Its momentum and weight-decay settings match those used for DCNF.

3 EXPERIMENTS

Experiments evaluate accuracy, efficiency, and qualitative depth predictions across indoor and outdoor datasets using standard depth-error measures.

  • Experimental design: The experiments compare DCNF with baselines, compare DCNF against DCNF-FCSP, and evaluate deeper DCNF-FCSP networks against state-of-the-art methods.The datasets are NYU v2, Make3D, and KITTI.
  • Computational efficiency: Training-time comparisons measure whole-model forward-plus-backward seconds per NYU v2 image across different numbers of superpixels.The comparison is presented for DCNF and DCNF-FCSP.
  • Computational efficiency: Prediction-time comparisons measure whole-model network forward seconds per NYU v2 image across different numbers of superpixels.The reported comparison concerns depth prediction rather than training.
  • Evaluation measures: Evaluation uses average relative error, root mean squared error, average log10 error, and threshold accuracy.Threshold accuracy is reported as the percentage of pixels satisfying the specified depth-ratio condition.
  • Qualitative evaluation: Qualitative NYUD2 comparisons depict depth with color, where red denotes far regions and blue denotes close regions, and report sharper transitions aligned with local details.The predictions are shown as visual comparisons rather than numerical measurements.

3.1 Baseline comparisons

Baseline experiments compare regression and smoothness variants with the jointly trained DCNF models, which achieve the strongest reported performance while DCNF-FCSP remains comparable to DCNF.

  • Baseline methods: The baseline set includes SVR, SVR with smoothness, unary-only regression, and unary-only regression with smoothness.The smoothness variants add pairwise terms during prediction or use smoothness alongside unary regression.
  • Qualitative comparisons: Qualitative depth outputs include Make3D and KITTI examples, with red representing far depth and blue representing close depth.The predictions are displayed in log scale.
  • NYU v2: On NYU v2, unary-only deep regression outperforms the SVR baseline, while the jointly learned DCNF achieves the best performance.Fine-tuning the whole network provides an additional performance gain.
  • DCNF versus DCNF-FCSP: DCNF-FCSP performs very close to DCNF on NYU v2 and generally on par with it on Make3D.The comparisons are reported in Tables 3 and 4.
  • Make3D: Make3D evaluation is constrained because far-away objects are mapped to a maximum depth of 81 meters.The protocol therefore reports errors both below 70 meters and over the entire image.
  • Make3D: On Make3D, full DCNF with whole-network training performs best among the compared baseline methods.Evaluation follows criteria that separately exclude depths at least 70 meters and include the entire image.

3.2 DCNF vs. DCNF-FCSP

DCNF-FCSP preserves prediction accuracy while substantially improving computational efficiency over DCNF. Its speed and scalability enable deeper networks for better performance.

  • DCNF-FCSP performs very close to DCNF in prediction accuracy.
  • DCNF-FCSP provides approximately 10 times speedup over the patch-wise model.The speedup becomes more significant as the number of superpixels increases.
  • DCNF-FCSP is much faster and more scalable than DCNF for whole-image depth prediction.The comparison measures forward time for processing one image.
  • The improved efficiency enables designing deeper networks for better performance.

3.3 State-of-the-art comparisons

The proposed models achieve strong state-of-the-art performance across NYU v2, Make3D, and KITTI comparisons. Results also show competitive performance using standard or relatively limited training data, without some competing methods’ extra data or post-processing.

  • Network design: The VGG-16-based deeper network replaces the earlier AlexNet component and adds convolutional layers for unary potentials.Training first pre-trains the remaining layers with transferred layers fixed, then fine-tunes the whole network.
  • NYU v2: On NYU v2, the method outperforms classic methods, joint depth-semantic learning, and Eigen et al. across all reported metrics.It uses the standard training set of 795 images, whereas Eigen et al. use millions of additional labelled images.
  • Make3D: On Make3D, DCNF-FCSP with whole-network training ranks first overall and outperforms compared methods by large margins.The method achieves better relative error without the ad-hoc sky-pixel post-processing used for one comparison.
  • Make3D: On Make3D, the deeper model improves C1 error but degrades C2 error relative to the smaller DCNF-FCSP network.The paper attributes this difference to far-away objects sharing a single maximum-depth value in the dataset.
  • KITTI: On KITTI, the method achieves the best RMS error using 700 training images.The test set contains 697 images from 28 scenes, and competing Eigen results use millions of additional training images.

4 CONCLUSION

The paper presents a deep convolutional neural field that combines deep CNNs with continuous CRFs for monocular depth estimation. Its efficient fully convolutional superpixel model enables deeper networks, and experiments show state-of-the-art performance indoors and outdoors, while geometric cues remain unused.

  • The proposed model unifies deep CNNs and continuous CRFs for single-image depth estimation.
  • Continuous CRF log-likelihood optimization is solved by back propagation without approximations, while MAP prediction has a closed-form solution.
  • The fully convolutional superpixel model is equally effective and provides orders-of-magnitude faster training.This efficiency enables deeper networks for better performance.
  • The method outperforms state-of-the-art methods on both indoor and outdoor scene datasets.
  • The main limitation is that the method does not exploit geometric cues.The paper identifies incorporating geometric cues as future work.
Loading 1502.07411v6…