PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks

Yuanzhouhan Cao, Zifeng Wu, Chunhua Shen

arXiv:1605.02305v3cs.CV

TL;DR

Monocular depth estimation is difficult because continuous regression lacks a confidence representation for pixels with different depth uncertainty. The paper discretizes depths into bins and predicts them with fully convolutional residual networks, using information gain and fully connected CRFs; it reports strong indoor and outdoor performance and state-of-the-art results.

  • Problem

    Continuous depth regression does not provide confidence information even though different image pixels have different distributions of possible depths.

  • Method

    The paper discretizes continuous depths into bins, predicts pixel-wise depth labels with fully convolutional residual networks, and uses information gain plus fully connected CRFs.

  • Results

    The approach achieves state-of-the-art performance on indoor and outdoor datasets, while discretizing ground-truth depths has negligible performance effect.

  • Takeaways & Limitations

    Depth classification provides confidence distributions that can guide training and post-processing, supporting the paper’s conclusion that discretization performs well.

Abstract

from arXiv · show

Depth estimation from single monocular images is a key component of scene understanding and has benefited largely from deep convolutional neural networks (CNN) recently. In this article, we take advantage of the recent deep residual networks and propose a simple yet effective approach to this problem. We formulate depth estimation as a pixel-wise classification task. Specifically, we first discretize the continuous depth values into multiple bins and label the bins according to their depth range. Then we train fully convolutional deep residual networks to predict the depth label of each pixel. Performing discrete depth label classification instead of continuous depth value regression allows us to predict a confidence in the form of probability distribution. We further apply fully-connected conditional random fields (CRF) as a post processing step to enforce local smoothness interactions, which improves the results. We evaluate our approach on both indoor and outdoor datasets and achieve state-of-the-art performance.

I. INTRODUCTION

The paper reframes monocular depth estimation as pixel-wise classification using deep residual CNNs, enabling confidence estimates and CRF-based refinement. It targets limitations of regression-based methods while preserving dense predictions for general scenes.

  • Existing methods commonly formulate monocular depth estimation as structured regression and minimize L2 distance to ground-truth depths.
  • Classification naturally yields a probability distribution over depth labels, unlike typical regression outputs that provide mean predictions without confidence.The authors contrast this with constrained regression and Monte-Carlo dropout, which require additional constraints or multiple evaluation passes.
  • The method uses information gain during training so confident predictions near the ground-truth depth contribute more to parameter updates.Fully connected CRFs subsequently improve low-confidence pixel estimates using interactions with other pixels.
  • Deep residual learning addresses vanishing gradients by learning residual mappings through shortcut connections.The paper uses this framework because deeper networks and fine-level details benefit depth estimation.
  • The proposed model accepts arbitrarily sized images, outputs dense score maps, and applies fully connected CRFs to obtain final depth estimates.
  • Unlike related methods, the approach uses classification and confidence-aware processing, then applies bilinear interpolation rather than an up-sampling scheme for high-resolution predictions.

III. PROPOSED METHOD

The proposed architecture combines fully convolutional dense prediction with deep residual networks. It uses shortcut-based building blocks, supports two network depths, and restores the prediction map to input resolution by interpolation.

  • The method replaces fully connected layers with convolutional layers so arbitrarily sized images produce dense, down-sampled predictions.
  • Residual networks address vanishing gradients through shortcut connections while enabling substantially deeper architectures than VGG.The paper considers residual networks with up to 152 layers and describes residual rather than direct mapping learning.
  • Identity-mapping shortcuts add the input directly to the residual mapping, requiring matching dimensions for element-wise addition.
  • Linear-projection shortcuts apply Ws to the input to match its dimensions with the residual mapping.
  • The architecture feeds images through convolutional and pooling layers followed by four convolution blocks containing projection and identity building blocks.The paper evaluates 101-layer and 152-layer residual architectures.
  • Pooling and stride-2 convolutions down-sample predictions by a factor of 8, after which bilinear interpolation restores the input image size.

B. Loss function

The loss treats depth estimation as classification by discretizing continuous depths into logarithmic bins. An information gain matrix weights nearby depth labels more favorably, while the resulting distributions also support CRF post-processing.

  • Continuous depth values are uniformly discretized into logarithmic bins, whose labels encode the corresponding depth ranges.The model uses a pixel-wise multinomial logistic loss for the resulting classification task.
  • For each pixel, P(D|z_i) is computed from the final convolutional outputs across B discretization bins.
  • The loss includes an information gain matrix H with elements H(p, q) = exp[−α(p−q)^2], weighting depth labels according to their distance from the ground-truth label.This encourages nearby predicted labels to contribute more to parameter updates.
  • At prediction time, each pixel receives the center depth value of its selected bin, while the full probability distribution provides prediction confidence.
  • The confidence distribution can be incorporated into fully connected CRFs during post-processing.

C. Fully connected conditional random fields

The model applies a fully connected CRF after CNN prediction to refine dense depth labels by combining unary evidence with pairwise interactions across all pixels.

  • The fully connected CRF post-processes the CNN output by modeling dependencies among every pair of pixels.Its energy comprises unary and pairwise potentials.
  • The unary potential uses each pixel’s logistic loss to score its predicted depth label.
  • The pairwise potential penalizes differences between pixel depth labels, with larger penalties for labels farther apart.The penalty is defined as Δ(D_i, D_j) = |D_i−D_j|.
  • Appearance kernels use pixel position and color, reflecting that nearby, similarly colored pixels are likely to share a depth range.The nearness and color-similarity scales are controlled by σα and σβ.
  • A smoothness kernel removes small isolated regions, with its scale controlled by σγ.

IV. EXPERIMENTS

The experiments evaluate the approach on indoor NYUD2 and outdoor KITTI datasets, testing discretization, component contributions, and comparisons with prior methods using standard depth metrics.

  • The evaluation uses two benchmark RGB-D datasets: indoor NYUD2 and outdoor KITTI.
  • The experiments compare discrete depth-label classification with continuous depth-value regression.
  • The study evaluates the contribution of different components in the proposed approach.
  • The method is compared with state-of-the-art approaches on both indoor and outdoor scenes.
  • Reported metrics include root mean squared error, average log10 error, root mean squared log error, and threshold accuracy.Threshold accuracy is the percentage of predictions whose ground-truth-to-prediction depth ratio is below a specified threshold.

A. Depth label classification vs. depth value regression

The study evaluates discretizing continuous depth values before classification and finds that this conversion has negligible effect, with log-space discretization producing lower error than linear discretization. It then compares discrete classification with conventional regression across NYUD2 and KITTI settings and analyzes depth ranges.

  • Discretization analysis: Discretization errors decrease with more bins and plateau at a negligible amount; log-space discretization yields lower error than linear-space discretization.The comparison uses discretized NYUD2 ground-truth depths in linear and log space.
  • Discretization analysis: All discretized ground-truth depths reach 100% accuracy except linear discretization into 10 bins at threshold 1.25.The result supports reformulating depth estimation from regression as classification.
  • Classification versus regression: Table I compares continuous depth regression with discrete depth-label classification on the NYUD2 and KITTI datasets using different numbers of discretization bins.The experiment uses a 101-layer residual network and standard training and validation splits.
  • Classification versus regression: Regression tends toward mean depth values, which can increase errors for depths that are very far from or very close to the camera.The classification approach with information gain is evaluated as a way to alleviate this problem.
  • Classification versus regression: The NYUD2 range analysis divides ground-truth depths into 0m-3m, 3m-7m, and 7m-10m.The analysis reports results separately for these three depth ranges.
  • Classification versus regression: Table III compares NYUD2 and KITTI results with and without information gain matrices.For each dataset, the first row omits the matrix and the second includes it.

B. Component evaluation

The component analysis evaluates the information gain matrix, fully connected CRFs, and network architectures on NYUD2 and KITTI using standard dataset splits and log-space depth discretization.

  • Component evaluation: The analysis covers information gain matrices, fully connected CRFs, and network architectures on both NYUD2 and KITTI.These experiments use the standard NYUD2 training set of 795 images and 654 test images.
  • Component evaluation: Continuous depth values are discretized into 100 bins in log space for the component evaluation.The stated setup applies this discretization to the experiments described for NYUD2 and KITTI.

1) Benefit of information gain matrix:

The information gain matrix is evaluated as a component of the loss function on indoor and outdoor depth estimation datasets.

  • Benefit of information gain matrix: The information gain matrix improves depth-estimation performance on both the indoor NYUD2 dataset and the outdoor KITTI dataset.The ResNet101 model is trained with and without the matrix; performance is reported as insensitive to α in these experiments.
  • Benefit of information gain matrix: The experiments set α to 0.2 for NYUD2 and 0.5 for KITTI.The authors report that performance is not sensitive to α in these experiments.

2) Benefit of fully connected CRFs:

The study evaluates fully connected CRFs as post-processing and compares residual networks with a VGG16 baseline. CRFs improve indoor and outdoor depth estimation, while residual networks outperform VGG16 on NYUD2.

  • Benefit of fully connected CRFs: Table IV compares NYUD2 and KITTI results without CRFs against results with CRFs as post-processing.For each dataset, the first row omits CRFs and the following row includes them.
  • Network comparisons: Table V reports NYUD2 results for VGG16, ResNet101, and ResNet152 and lists their total parameter counts.The table organizes the VGG16 result first, followed by the two residual-network results.
  • Benefit of fully connected CRFs: Fully connected CRFs improve depth estimation for both indoor NYUD2 and outdoor KITTI scenes.The comparison applies CRFs as post-processing after training ResNet101 models.
  • Network comparisons: Residual networks outperform the VGG16 baseline on NYUD2, reinforcing the importance of network depth.The comparison includes ResNet101 and ResNet152; ResNet152 improves little over ResNet101 because the 795-image training set causes overfitting.

C. State-of-the-art comparisons

The approach is evaluated against recent depth-estimation methods on NYUD2 and KITTI, achieving state-of-the-art performance across the reported evaluation metrics.

  • NYUD2: On NYUD2, the deep fully convolutional residual network with depth-label classification achieves state-of-the-art performance on four evaluation metrics.The evaluation uses the standard 654 test images and fully connected CRFs as post-processing.
  • Comparisons: The NYUD2 and KITTI evaluations compare the proposed method with multiple recent architectures, including DCNF, NRF, multi-scale, and upsampling-based methods.The NYUD2 table includes results from recent depth-estimation models and the proposed approach as the final row.
  • KITTI: On KITTI, the method outperforms the other compared methods significantly under maximum-depth caps of 80 meters and 50 meters.The comparison uses the same training set as [30] but only left images, with results reported for both depth caps.

3) Cross-dataset evaluation:

Cross-dataset evaluation trains on raw NYUD2 and tests on SUN RGB-D, where the method achieves satisfactory results and outperforms other compared methods. The conclusion identifies several previously explored techniques as potential improvements.

  • Cross-dataset evaluation: Training on raw NYUD2 and testing on SUN RGB-D evaluates the method across datasets, excluding SUN RGB-D test images originating from NYUD2.The evaluation randomly selects 500 SUN RGB-D test images and compares against Liu et al. and Laina et al.
  • Results: On SUN RGB-D, the method achieves satisfactory results and outperforms the other compared methods.The comparison uses released trained models and evaluation codes from the competing authors.
  • Future improvements: The authors identify multi-scale inputs, mid-layer concatenation, and prediction-map upsampling as directions expected to improve the network.These techniques are left for future work.
Loading 1605.02305v3…