PaperlayerView exact revision on arXiv

Source-linked AI summary

PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing

Dan Xu, Wanli Ouyang, Xiaogang Wang, Nicu Sebe

arXiv:1805.04409v1cs.CV

TL;DR

The paper addresses simultaneous depth estimation and scene parsing, where conventional multi-task methods directly optimize multiple final tasks. PAD-Net predicts intermediate auxiliary tasks and distills their multi-modal outputs into the final tasks, achieving strong results on NYUD-v2 and Cityscapes.

  • Problem

    Simultaneous depth estimation and scene parsing require effective joint learning, while existing approaches directly optimize final tasks from shared input data and may generalize poorly across tasks.

  • Method

    PAD-Net predicts intermediate depth, surface-normal, semantic, and contour tasks, then fuses their predictions through multi-modal distillation modules to guide final depth and scene-parsing outputs.

  • Results

    PAD-Net achieves state-of-the-art results on NYUD-v2 for both depth estimation and scene parsing, with very competitive Cityscapes scene-parsing performance.

  • Takeaways & Limitations

    Intermediate auxiliary predictions can provide supervision and multi-modal guidance that improves simultaneous depth estimation and scene parsing from RGB input.

Abstract

from arXiv · show

Depth estimation and scene parsing are two particularly important tasks in visual scene understanding. In this paper we tackle the problem of simultaneous depth estimation and scene parsing in a joint CNN. The task can be typically treated as a deep multi-task learning problem [42]. Different from previous methods directly optimizing multiple tasks given the input training data, this paper proposes a novel multi-task guided prediction-and-distillation network (PAD-Net), which first predicts a set of intermediate auxiliary tasks ranging from low level to high level, and then the predictions from these intermediate auxiliary tasks are utilized as multi-modal input via our proposed multi-modal distillation modules for the final tasks. During the joint learning, the intermediate tasks not only act as supervision for learning more robust deep representations but also provide rich multi-modal information for improving the final tasks. Extensive experiments are conducted on two challenging datasets (i.e. NYUD-v2 and Cityscapes) for both the depth estimation and scene parsing tasks, demonstrating the effectiveness of the proposed approach.

1. Introduction

Depth estimation and scene parsing are fundamental, correlated perception tasks, but conventional joint CNNs directly optimize final outputs and can struggle with complex multi-task learning. PAD-Net instead uses intermediate auxiliary predictions as multi-modal guidance for the final tasks.

  • Depth estimation and scene parsing are fundamental tasks for visual scene perception and understanding.
  • Typical multi-task methods directly predict both tasks from shared input data using cross-modal interactions or joint optimization objectives.
  • Intermediate auxiliary predictions provide multi-modal information and supervision for learning more robust representations and improving final tasks.
  • PAD-Net predicts intermediate depth, surface-normal, semantic, and contour outputs before producing the final depth and scene-parsing tasks.
  • PAD-Net achieves state-of-the-art results on NYUD-v2 for both tasks and very competitive Cityscapes scene-parsing performance, outperforming joint-optimization state of the arts.

2. Related Work

Prior work develops depth estimation, scene parsing, and deep multi-task learning separately or through direct joint optimization. PAD-Net is distinguished by treating intermediate task predictions as multi-modal inputs for final tasks.

  • Depth estimation and scene parsing: Depth-estimation research spans hand-crafted or graphical-model methods and newer deep-learning approaches.
  • Depth estimation and scene parsing: Scene-parsing research emphasizes fully convolutional and dilated convolutions, multi-scale feature learning, ensembling, and structured prediction.
  • Depth estimation and scene parsing: Earlier simultaneous depth and scene-parsing methods directly learn the two tasks rather than treating them as multi-modal inputs for final predictions.
  • Deep multi-task learning for vision: Deep multi-task learning has been applied across vision problems including geometric-semantic inference, face attributes, contours, segmentation, detection, and scene classification.
  • PAD-Net: PAD-Net uses intermediate predictions and a multi-modal distillation module to incorporate complementary information into final depth and scene-parsing outputs.

3. PAD-Net: Multi-Tasks Guided Prediction- and-Distillation Network

PAD-Net uses intermediate multi-task predictions to distill complementary multimodal information into separate depth-estimation and scene-parsing decoders. Its design combines auxiliary-task supervision with three alternative fusion mechanisms and RGB-only inference.

  • Approach overview: PAD-Net comprises a fully convolutional encoder, intermediate multi-task predictions, a multimodal distillation module, and task-specific decoders.The final outputs are depth and semantic parsing maps from RGB input.
  • Deep multi-task prediction: The intermediate module predicts depth, surface normals, scene parsing, and contours across continuous regression and discrete classification tasks.Intermediate predictions provide the inputs to subsequent multimodal distillation.
  • Optimization and inference: During inference, PAD-Net uses separate decoders and incorporates semantic, depth, surface-normal, and contour information from a single RGB image.The method therefore uses RGB input during both training and testing while exploiting multiple predicted modalities.
  • Deep multi-modal distillation: Module C adds attention-guided message passing, using generated attention maps to control information transferred between task feature maps.The attention mechanism selects useful information during multimodal fusion.
  • Deep multi-modal distillation: Module A concatenates feature maps, whereas Module B learns task-specific maps through message passing between predictions.Module A feeds one fused representation to different final-task decoders; Module B refines separate representations for each final task.
  • Optimization and inference: The decoders upsample distilled feature maps for pixel-level prediction, while training combines four task losses with corresponding weights.Contour labels derive from semantic labels, and surface normals are calculated from depth, so only semantic and depth annotations are required.

4. Experiments

Experiments on NYUD-v2 and Cityscapes evaluate PAD-Net through diagnostics and comparisons with state-of-the-art methods for depth estimation and scene parsing. The results show that multi-modal distillation and multi-task-guided prediction improve both final tasks over direct multi-task baselines.

  • Experimental Setup: Experiments use NYUD-v2 and Cityscapes, with depth and semantic labels; Cityscapes depth is not quantitatively evaluated because only disparity maps are provided.NYUD-v2 contains 1449 RGB-depth pairs, while Cityscapes provides 19 semantic classes and pre-computed disparity maps.
  • Diagnostic Experiments: Directly optimizing depth estimation and scene parsing slightly helps on NYUD-v2 but decreases performance on Cityscapes.The diagnostic compares Front-end + DE with Front-end + DE + SP.
  • Diagnostic Experiments: Attention-guided distillation improves module C over module B, while message passing improves performance over the naive combination method.The paper evaluates three multi-modal distillation modules.
  • Diagnostic Experiments: PAD-Net Distillation C + DE + SP outperforms versions with only one final task on both depth estimation and scene parsing.The result indicates that jointly learning both final tasks yields more effective features within the proposed design.
  • Diagnostic Experiments: 2.9% worse rel and 2.8% worse mIoU occur with multiple deep supervision than with MTDN-full; adding more intermediate predictions improves final performance.MTDN-full is also 3.3% better on rel and 3.7% better on mIoU than MTDN-inp0.
  • State-of-the-art Comparison: PAD-Net with ResNet-50 achieves the best depth-estimation performance across all reported metrics on NYUD-v2 without extra training data.It also outperforms Joint HCRF and Jafari et al., which jointly learn depth and scene parsing.
  • State-of-the-art Comparison: 3.7 points separate PAD-Net-ResNet50 from RefineNet-Res152 on NYUD-v2 mIoU, while Cityscapes mIoU reaches 0.803 and exceeds PSPNet by 1.9 points.The Cityscapes comparison uses only the fine-annotation training set.

5. Conclusion

PAD-Net predicts intermediate multi-task outputs and uses them to guide the final depth estimation and scene parsing tasks. Experiments showed effectiveness across NYUD-v2 and Cityscapes, including state-of-the-art or top performance.

  • PAD-Net first predicts several intermediate auxiliary tasks, then uses their predictions to facilitate optimization of the final main tasks.The auxiliary predictions provide multi-task guidance within the network.
  • Three multi-modal distillation modules are developed to use the multi-task predictions more effectively.
  • Experiments on NYUD-v2 and Cityscapes demonstrated PAD-Net's effectiveness for simultaneous depth estimation and scene parsing.
  • PAD-Net achieved new state-of-the-art results on both depth estimation and scene parsing on NYUD-v2.
  • PAD-Net achieved top performance on the Cityscapes scene parsing task.
Loading 1805.04409v1…