Source-linked AI summary
Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation
Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, Nicu Sebe
TL;DR
The paper addresses monocular depth estimation from a single still image, where accurately predicting depth from RGB data is challenging. It fuses multi-scale CNN side outputs with continuous CRFs using unified and cascaded models whose mean-field updates are implemented as end-to-end trainable sequential networks. Experiments report more accurate depth maps than traditional multiscale approaches and state-of-the-art performance on public benchmarks.
Problem
Accurately predicting pixel-level depth from a single RGB image remains challenging despite CNN-based monocular depth estimation.
Method
Continuous CRFs fuse complementary multi-scale CNN side outputs through a unified multi-scale model or a cascade of scale-specific CRFs.
Results
The CRF-based models produce more accurate depth maps than traditional multiscale approaches and outperform state-of-the-art depth-estimation methods on public benchmarks.
Takeaways & Limitations
The framework supports end-to-end training, works with different pretrained CNN architectures, and is reported as useful for pixel-level prediction of continuous variables beyond depth.
Takeaways & Limitations
The cascade model assumes that estimated depth at a given scale depends only on the depth estimated at the previous scale.
Abstract
from arXiv · showhide
This paper addresses the problem of depth estimation from a single still image. Inspired by recent works on multi- scale convolutional neural networks (CNN), we propose a deep model which fuses complementary information derived from multiple CNN side outputs. Different from previous methods, the integration is obtained by means of continuous Conditional Random Fields (CRFs). In particular, we propose two different variations, one based on a cascade of multiple CRFs, the other on a unified graphical model. By designing a novel CNN implementation of mean-field updates for continuous CRFs, we show that both proposed models can be regarded as sequential deep networks and that training can be performed end-to-end. Through extensive experimental evaluation we demonstrate the effective- ness of the proposed approach and establish new state of the art results on publicly available datasets.
1. Introduction
Monocular depth estimation from RGB images is challenging, and prior work combines multi-scale CNN information or cascaded CRFs to improve pixel-level predictions. This paper proposes continuous-CRF fusion of CNN side outputs through two end-to-end trainable sequential models.
- Monocular depth prediction is formulated as pixel-level regression from RGB-depth pairs, typically using CNN architectures.
- Prior depth-estimation methods improve predictions by combining information from multiple scales through feature fusion or multi-resolution inputs.
- The paper introduces continuous CRFs to fuse complementary representations from multiple CNN side-output layers instead of averaging or concatenating them.
- The proposed models comprise a unified multi-scale CRF and a cascade of scale-specific CRFs.
- A CNN implementation of continuous-CRF mean-field updates makes both models sequential deep networks that support end-to-end training.
- Experiments report more accurate depth maps than traditional multiscale approaches and state-of-the-art performance on NYU Depth V2 and Make3D.
2. Related work
Prior monocular depth-estimation research includes handcrafted-feature methods, graphical models, and deep networks. CNN-based approaches use multi-scale prediction, multitask learning, semantic information, or CRF-based refinement, while earlier graphical-model methods did not employ deep networks.
- Monocular depth-estimation methods are categorized into handcrafted-feature, graphical-model, and deep-network approaches.
- Handcrafted-feature methods include single-image 3D reconstruction, depth transfer from similar images, and combinations of semantic labels with depth features.
- Graphical-model approaches use Bayesian frameworks, multiscale MRFs, or discrete-continuous CRFs to reconstruct or fuse depth information.
- Earlier graphical-model methods described here did not employ deep networks.
- CNN-based methods include coarse-to-fine multi-scale prediction, multitask learning, joint depth and semantic segmentation, and hierarchical CRF refinement.
3. Multi-Scale Models for Depth Estimation
The paper fuses multi-scale CNN side outputs with continuous CRFs for monocular depth estimation, presenting unified and cascade formulations implemented as sequential deep networks.
- Overview: The model combines a front-end CNN producing intermediate side outputs with a fusion block based on continuous CRFs.The side outputs are generated from multiple CNN layers and provide complementary representations for depth prediction.
- Multi-scale CRFs: The unified multi-scale CRF jointly integrates side outputs across scales while enforcing smoothness between neighboring pixels and scales.Its hidden variables represent depth values across pixels and scales, with unary and pairwise energy terms.
- Sequential deep networks: Both models use mean-field inference, whose updates are implemented with a common C-MF block for sequential deep-network construction.The C-MF implementation includes Gaussian filtering, parameter weighting, combination with side outputs, and normalization.
- Prediction: At test time, the estimated variables at the finer scale are used as the predicted depth map.For the unified model, the finer-scale variables are selected from the multi-scale CRF output; the cascade likewise produces its final estimate at the finest level.
- Cascade CRFs: The cascade model applies one scale-specific CRF at each level, passing each estimated depth map as an observation to the subsequent CRF.At scale l, the CRF uses the side output s_l and the estimate from the previous scale as observed variables.
- Training: The sequential formulations support end-to-end training of the front-end CNN and fusion module through back-propagation.The paper implements error differentials for the C-MF computations and trains the whole network jointly.
4. Experiments
Experiments evaluate the proposed CRF-based multiscale fusion models on NYU Depth V2 and Make3D, comparing fusion schemes, architectures, scale counts, ordering, and state-of-the-art methods. The results consistently favor CRF-based multiscale integration, with accuracy improving as scales increase and coarse-to-fine cascades outperforming reverse ordering.
- Experimental setup: The evaluation compares several front-end CNN architectures, including AlexNet, VGG16, VGG-ED, VGG-CD, and ResNet50.The architectures provide different numbers of side outputs for multiscale fusion.
- Multiscale fusion: CRF-based models produce more accurate depth maps than traditional multiscale fusion schemes, while the multi-scale model is more accurate than the cascade at higher computational cost.The comparison supports integrating complementary CNN side-output information within a graphical-model framework.
- Multiscale fusion: Performance improves when the complete models use more scales, with five-scale versions outperforming versions using one or three side-output layers.The analysis directly compares models with different numbers of side outputs.
- Cascade ordering: A coarse-to-fine stacking order leads to more accurate depth maps than the reverse order in the cascade model.The ordering comparison tests progressive refinement from inner to outer layers versus the reverse.
- Front-end architectures: ResNet50 outperforms the other tested front-end architectures, and CRF-based models significantly improve performance across all considered networks.VGG-CD is slightly better than VGG-ED, and both outperform VGG16.
- State-of-the-art comparison: The proposed approach outperforms baseline methods on NYU Depth V2 and is the second-best model with only 4.7K pretraining pairs, despite another method using 120K pairs.Using larger pretraining data also yields stronger comparisons against methods trained with Huber loss.
- State-of-the-art comparison: Compared with Laina et al. and Liu et al., the proposed models achieve higher accuracy, while total inference for one image takes about 1 second.The comparison notes that Liu et al.’s reported time excludes superpixel computation.
5. Conclusions
The paper presents continuous CRFs as a framework for fusing multi-scale CNN side-output representations for monocular depth prediction and supports end-to-end training. Experiments validate the approach and suggest its mean-field implementation may benefit other pixel-level continuous prediction tasks.
- Continuous CRFs fuse multi-scale representations from CNN side outputs for depth prediction from a single RGB image.
- The framework works with several common CNN architectures and supports end-to-end training.
- The mean-field implementation may also benefit computer-vision tasks involving pixel-level predictions of continuous variables.