Source-linked AI summary
Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networks
Arsalan Mousavian, Hamed Pirsiavash, Jana Kosecka
TL;DR
The paper addresses separate training of semantic segmentation and depth estimation by proposing a shared model for both tasks from one RGB image. It combines staged joint learning with a fully connected CRF, achieving superior semantic segmentation and comparable depth estimation on NYUDepth V2.
Problem
Semantic segmentation and depth estimation were often addressed separately, despite evidence that CNN architectures can support multiple tasks.
Method
The model uses shared multi-scale CNN features for both tasks, trains them in stages with joint fine-tuning, and integrates a fully connected CRF using semantic, depth, RGB, and label-compatibility cues.
Results
The model produces superior semantic segmentation compared with state-of-the-art methods and comparable depth estimates on NYUDepth V2.
Takeaways & Limitations
Shared feature representations allow depth estimation and semantic segmentation to be trained together, while coupling the CRF with the network exploits depth to discriminate some semantic categories.
Takeaways & Limitations
Absolute per-pixel depth is ambiguous because the input lacks sufficient information to determine it uniquely.
Abstract
from arXiv · showhide
Multi-scale deep CNNs have been used successfully for problems mapping each pixel to a label, such as depth estimation and semantic segmentation. It has also been shown that such architectures are reusable and can be used for multiple tasks. These networks are typically trained independently for each task by varying the output layer(s) and training objective. In this work we present a new model for simultaneous depth estimation and semantic segmentation from a single RGB image. Our approach demonstrates the feasibility of training parts of the model for each task and then fine tuning the full, combined model on both tasks simultaneously using a single loss function. Furthermore we couple the deep CNN with fully connected CRF, which captures the contextual relationships and interactions between the semantic and depth cues improving the accuracy of the final results. The proposed model is trained and evaluated on NYUDepth V2 dataset outperforming the state of the art methods on semantic segmentation and achieving comparable results on the task of depth estimation.
1. Introduction
The paper proposes jointly estimating depth and semantic labels from one RGB image using shared features, then refining both through a coupled CNN–CRF model. On NYUDepth V2, it reports superior semantic segmentation and comparable depth-estimation results.
- The model jointly estimates depth and semantic segmentation from a single RGB image using a shared underlying feature representation.
- A fully connected CRF is integrated with the CNN to capture contextual relationships and interactions between semantic and depth cues.
- The model is trained on NYUDepth V2 and evaluated for semantic segmentation with estimated depth, without depth, and depth estimation alone.
- The approach outperforms state-of-the-art semantic segmentation methods and achieves comparable results for depth estimation.
2. Related work
Prior work developed CNN and CRF approaches for semantic segmentation and separate CNN approaches for monocular depth estimation. This paper positions its model as the first to jointly estimate semantic labels and depth from a single RGB image with shared representations and interactions between both outputs.
- CNN-based semantic segmentation addressed pixel-to-pixel labeling through multi-layer classifiers and encoder–decoder architectures.
- Typical CNN segmentation shortcomings included limited contextual modeling and insufficiently accurate boundaries.
- Fully connected CRFs were introduced to combine CNN responses with contextual and local pixel- and edge-based information.
- Separate research addressed monocular depth estimation using CNNs, including coarse-to-fine multi-network approaches.
- The proposed model is presented as the first to jointly estimate semantic labels and depth from one RGB image using a shared representation.
3. Proposed Method
The proposed model jointly estimates semantic labels and depth from one RGB image using shared multi-scale features, a joint objective, and a fully connected CRF. Depth and semantic predictions are refined together to improve labeling while sharing computation across tasks.
- Joint architecture: A single network jointly learns semantic segmentation and depth estimation from RGB inputs through a shared underlying feature representation.The two task modules use shared multi-scale features, allowing the learned weights to support both tasks.
- Joint optimization: The training objective combines semantic and depth losses, producing per-pixel predicted labels and depth estimates before CRF refinement.The full model is further optimized after incorporating interactions between the response maps.
- Depth estimation: Depth is estimated by discretizing possible values into bins and computing a continuous value from the predicted bin probabilities.The method uses a scale-invariant loss because absolute depth is ambiguous and relative depth is more appropriate for exploiting depth discontinuities.
- Multi-scale representation: The multi-scale network uses five paths whose features are scaled and aggregated into a comprehensive representation for both tasks.Separate convolutional layers at each path extract task-specific features for semantic segmentation and depth estimation.
- CRF refinement: The fully connected CRF combines semantic unaries with spatial, RGB, and estimated-depth features to model contextual interactions between pixels.Its learned unary and pairwise terms are optimized with mean-field inference, while depth enters the CRF as an additional modality.
4. Experiments
Experiments evaluate the multi-scale network, staged training, CRF integration, depth estimation, and semantic segmentation on NYUDepth v2. Results show improved semantic segmentation and strong scale-invariant depth performance, with learned CRF weights emphasizing useful depth discontinuities.
- Training Details: The network uses five branches to compute shared multi-scale features for semantic and depth unaries.Training begins with semantic segmentation and adds depth layers before joint optimization and CRF fine-tuning.
- Qualitative Results: Qualitative outputs recover structures missing or mislabeled in ground truth, including a left window, a black chair, and depths absent from raw measurements.The figure compares input, ground-truth and predicted semantics, raw depth, and estimated depth.
- Training Details: Two-stage training produced a slightly better model than optimizing semantic and depth losses jointly from the start.Joint training reduced the objective faster but plateaued, whereas two-stage training achieved slightly better final performance.
- Depth Estimation: The method performs significantly better on scale-invariant depth RMSE, emphasizing relative depth and depth discontinuities rather than absolute depth.Depth values are clipped at 7m, discretized into 50 bins of length 0.14m, and evaluated with multiple metrics.
- Semantic Segmentation: Estimating depth alongside semantics improves mean IoU over 40 classes by 1.3%.The full model combines semantic and depth unaries with RGB and estimated-depth inputs in the CRF, and is trained end-to-end.
- Semantic Segmentation: CRF depth weights are large for class pairs where depth discontinuities help distinguish objects such as pillows from couches and sinks from counters.RGB differences are described as less informative than depth differences for these class pairs.
5. Conclusions
The paper demonstrates that semantic segmentation and depth estimation can be learned jointly through shared representations, staged training, and single-loss fine-tuning. Coupling the network with a CRF further improves performance and lets depth help distinguish semantic categories.
- Joint semantic segmentation and depth estimation use the same network, trained in stages and then fine-tuned with a single loss function.This supports simultaneous learning of both tasks through shared feature representations.
- The proposed model produces comparable depth estimates and superior semantic segmentation relative to state-of-the-art methods.
- Coupling the CRF with the deep network further improves performance and uses estimated depth to discriminate between some semantic categories.
- The results indicate that depth estimation and semantic segmentation can share feature representations and improve final performance.