Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations

Vladimir Nekrasov; Thanuja Dharmasiri; Andrew Spek; Tom Drummond; Chunhua Shen; Ian Reid

Source-linked AI summary

Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations

Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, Ian Reid

arXiv:1809.04766v2Sep 13, 2018cs.CV

TL;DR

The paper addresses the difficulty of deploying joint depth estimation and semantic segmentation in real time when annotations are uneven across modalities. It adapts a compact segmentation network and uses hard knowledge distillation from powerful task-specific models to supply missing labels. The resulting system matches or exceeds reported state-of-the-art performance while supporting efficient joint inference and dense semantic 3D reconstruction.

Problem
Robotic deployment requires one system to perform depth estimation and semantic segmentation in real time despite uneven annotation coverage across modalities.
Method
The method adapts a lightweight shared-backbone network and uses expert predictions as synthetic labels for pre-training before fine-tuning on jointly annotated data.
Results
42.02% mean IoU and 0.565m RMSE (lin) are achieved on NYUDv2 while performing both tasks simultaneously in real time.
Takeaways & Limitations
The approach extends to more tasks and datasets, and its raw depth and segmentation predictions can be used in dense semantic SLAM systems.

Abstract

from arXiv · show

Deployment of deep learning models in robotics as sensory information extractors can be a daunting task to handle, even using generic GPU cards. Here, we address three of its most prominent hurdles, namely, i) the adaptation of a single model to perform multiple tasks at once (in this work, we consider depth estimation and semantic segmentation crucial for acquiring geometric and semantic understanding of the scene), while ii) doing it in real-time, and iii) using asymmetric datasets with uneven numbers of annotations per each modality. To overcome the first two issues, we adapt a recently proposed real-time semantic segmentation network, making changes to further reduce the number of floating point operations. To approach the third issue, we embrace a simple solution based on hard knowledge distillation under the assumption of having access to a powerful `teacher' network. We showcase how our system can be easily extended to handle more tasks, and more datasets, all at once, performing depth estimation and segmentation both indoors and outdoors with a single model. Quantitatively, we achieve results equivalent to (or better than) current state-of-the-art approaches with one forward pass costing just 13ms and 6.5 GFLOPs on 640x480 inputs. This efficiency allows us to directly incorporate the raw predictions of our network into the SemanticFusion framework for dense 3D semantic reconstruction of the scene.

I. INTRODUCTION

The paper targets joint depth estimation and semantic segmentation for robotics by addressing multi-task deployment, real-time inference, and uneven annotations. It uses existing powerful models to generate missing labels and demonstrates a compact system supporting additional tasks and indoor–outdoor settings.

Motivation: Real-time multi-task deployment is difficult because depth and segmentation often lack fully shared datasets.The paper identifies both concurrent task deployment and asymmetric annotation availability as central challenges.
Approach: Powerful single-task models can generate informative predictions for missing labels, enabling training with asymmetric annotations.The approach predicts missing segmentation masks using an expert model.
Contributions: The system jointly estimates depth and semantic segmentation, with experiments extending prediction to surface normals.The authors also train one model for indoor and outdoor depth estimation and segmentation.
Contributions: The authors argue that one small network can achieve the same performance without deploying multiple expensive models.The introduction frames this as one model replacing multiple task-specific models.

II. RELATED WORK

Prior work established multi-task learning and real-time task-specific networks, but the paper focuses on combining existing task models with real-time joint inference under asymmetric annotations. Its baseline is a lightweight RefineNet–MobileNet-v2 architecture, and its approach differs from earlier depth and multi-task methods in efficiency and annotation handling.

Multi-task learning: Multi-task learning uses shared representations with task-specific parameters and can improve generalisation through inductive bias.Earlier work applied this paradigm to combinations including depth, normals, segmentation, detection, and instance segmentation.
Architecture: The network shares most parameters while retaining only two task-specific parametric layers for each task.Figure 1 presents the general joint segmentation–depth structure.
Positioning: Earlier approaches generally did not reuse existing task models and rarely achieved real-time multi-task inference.BlitzNet is identified as the exception to the real-time qualification.
Semantic segmentation: Light-Weight RefineNet on MobileNet-v2 provides the baseline because it offers real-time semantic segmentation with fewer than 4M parameters.The baseline is selected for solid PASCAL VOC performance and compactness.
Depth estimation: Depth estimation predicts each pixel’s distance from the observer, with prior work using supervised, unsupervised, and semi-supervised learning.Earlier studies also jointly predicted structural quantities such as depth, surface normals, and curvature.
Positioning: Compared with previous depth-related methods, this approach emphasizes real-time inference and effective use of asymmetric ground-truth annotations.The paper states that its smaller real-time model quantitatively outperforms those approaches.

III. METHODOLOGY

The methodology reduces the computational cost of a lightweight joint network while preserving a shared architecture for multiple tasks. It replaces the costly final high-resolution convolution and supports extending the same machinery beyond two tasks.

Multi-task extension: The method applies the same machinery to more than two tasks.The paper explicitly demonstrates this extension in later experiments.
Backbone: Light-Weight RefineNet with MobileNet-v2 is adapted as the joint depth-and-segmentation architecture.The original network uses Chained Residual Pooling blocks built from max-pooling and 1 × 1 convolutions.
Efficiency: The last high-resolution Chained Residual Pooling block accounts for more than half of the FLOPs.It operates on feature maps at one-quarter of the original resolution.
Efficiency: Replacing its final 1 × 1 convolution with a depthwise equivalent reduces the block’s computational influence.The replacement uses grouped convolution with one group per input channel.

B. Joint Semantic Segmentation and Depth Estimation

The joint model shares a backbone before branching into lightweight task-specific heads for segmentation and depth estimation. Training combines task losses, using cross-entropy for segmentation and inverse Huber loss for depth with equal weighting.

Task branches: The model branches after the last Chained Residual Pooling block and adds two convolutional layers per task.Each task receives one depthwise 1 × 1 convolution and one standard 3 × 3 convolution.
Network formulation: A shared backbone transforms the RGB image before separate segmentation and depth branches produce their predictions.The backbone output is denoted fθb(I), while task branches are gθs and gθd.
Training objective: The total loss combines segmentation and depth losses with λ set to 0.5.Segmentation uses standard softmax cross-entropy, while depth uses inverse Huber loss.
Training objective: Ground-truth segmentation masks and depth maps provide the supervision targets for the two task losses.The segmentation loss uses per-pixel class probabilities.

C. Expert Labeling for Asymmetric Annotations

The method addresses asymmetric annotations by using expert predictions as synthetic labels for missing modalities, then fine-tuning on jointly annotated data. It also transfers to multiple datasets trained with one model.

Images with annotations for only one modality create disjoint subsets of singly and jointly annotated data.
Expert predictions replace missing annotations so both task branches receive updates during training.Updating only the branch with available labels produces biased gradients and sub-optimal solutions.
Synthetic labels from an expert model pre-train the joint network before fine-tuning on images with ground-truth annotations for both tasks.The method stores predicted labels rather than logits to reduce the resources required for large 3-D tensors.
The framework extends to several datasets and uses one model for indoor NYUDv2 and outdoor KITTI settings.The experiments use NYUDv2-40 and KITTI as representative indoor and outdoor datasets.
Training uses ImageNet initialization, mini-batch SGD with momentum, frozen batch-normalization statistics, and a tenfold learning-rate reduction after synthetic-annotation pre-training.Speed measurements average 100 forward passes on a workstation with a single GTX1080Ti GPU.

A. NYUDv2

On NYUDv2, the system trains with synthetic semantic annotations from a large depth-labeled set before fine-tuning on 795 jointly annotated images. It achieves competitive joint segmentation and depth results while operating in real time.

NYUDv2 contains 40 semantic labels, 1,449 RGB images with segmentation and depth annotations, and over 300,000 raw training images with depth annotations.The authors use fewer than 10% of the raw depth data and annotate 25K images for segmentation with a teacher network.
The model is pre-trained on synthetic annotations from the large set and fine-tuned on the original 795-image training set.
42.02% mean iou and 0.565m RMSE (lin) are achieved on the validation set while performing both tasks simultaneously in real time.The reported result outperforms several large models; detailed depth metrics appear in Table II.
Fig. 2 compares qualitative predictions with NYUDv2 ground truth, where black and dark-blue pixels indicate missing segmentation and depth annotations.
Joint training on only the small annotated set provides no significant task-specific benefit and substantially harms semantic segmentation.Additional depth annotations improve depth estimation and, with gradient accumulation, semantic segmentation; the proposed method performs better without changing the optimization algorithm.
Table III reports ablation settings through joint-annotation counts and task-specific and shared-parameter update frequencies.

B. KITTI

The KITTI setup combines sparse semantic annotations with larger depth-annotated data, using teacher predictions to supply missing labels. After pre-training and fine-tuning, the compact model performs both tasks competitively with low latency.

KITTI provides 100 semantically annotated training images and 46 test images, with no ground-truth depth maps for testing.
20,000 depth-annotated images are used for training, while ResNet-38 supplies missing semantic labels and a depth-only network supplies missing depth labels.
After pre-training on the large set and fine-tuning on 100 examples, the approach outperforms competing methods across segmentation and depth metrics.
Qualitative KITTI results omit ground-truth depth visualisations because the available depth maps are sparse.

V. EXTENSIONS

The extensions demonstrate that the approach can be applied to a single model handling more tasks or multiple datasets and scenarios. The section emphasizes direct reuse of the same simple training scheme.

The approach is extended to one model performing three tasks and to two tasks across indoor and outdoor datasets.
The extensions directly reuse the training scheme from the preceding section without targeting high performance numbers.

A. Single Model - Three Tasks

A third surface-normals branch extends the joint model beyond depth and segmentation. The resulting three-task network remains real-time, with depth largely preserved but segmentation substantially degraded.

A surface-normals branch is added alongside the depth and segmentation branches, using two convolutional layers.
Surface normals use a normalised negative-dot-product loss and a 10-times larger learning rate for their parameters.
The three-task model achieves practically unchanged depth performance but suffers a significant semantic-segmentation performance drop.
Despite the segmentation drop, all three tasks remain competitive and run simultaneously in real time.

B. Single Model - Two Datasets, Two Tasks

A single network is trained to perform depth estimation and semantic segmentation on NYUD and KITTI together. It matches separate per-dataset networks without substantially increasing model capacity and can feed predictions into SemanticFusion.

The joint model concatenates NYUD and KITTI and predicts 46 segmentation labels: 40 from NYUD and 6 from KITTI-6.
Pre-training and fine-tuning are performed on unions of the datasets’ large and small training sets.
The combined network matches the separate-dataset approach on both datasets without a substantial increase in model capacity.
SemanticFusion uses tracked SLAM poses to produce a dense 3D semantic map from frame-level predictions.
The network’s indoor depth and segmentation predictions can replace ground-truth depth and a separate segmentation CNN in SemanticFusion.

VI. CONCLUSION

The paper presents joint depth estimation and semantic segmentation as a practical route to efficient visual information extraction for robotic applications. Its compact model supports dense 3D semantic reconstruction and can be extended to additional tasks and datasets.

Efficient visual information extraction is presented as crucial for further development and deployment of robots and autonomous vehicles.
The method reuses large existing models to generate synthetic labels for pre-training a compact model.
The approach can be extended to handle more tasks and datasets simultaneously.
Raw depth and segmentation predictions can be used within available dense SLAM systems for 3D semantic reconstruction.
Future work considers incorporating expert uncertainty, operating without a reliable expert, and reducing training annotations without sacrificing performance.