PaperlayerView exact revision on arXiv

Source-linked AI summary

OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion

Ruihang Miao, Weizhou Liu, Mingrui Chen, Zheng Gong, Weixin Xu, Chen Hu, Shuchang Zhou

arXiv:2302.13540v1cs.CV

TL;DR

Pure-vision SSC struggles to recover complete geometry and semantics, while accurate depth is important for geometric reconstruction. OccDepth uses stereo depth cues and depth distillation for SSC, and outperforms RGB-inferred baselines while remaining comparable to some 2.5D/3D-input methods.

  • Problem

    Recovering complete 3D geometry and semantics from visual images is challenging, and pure-vision solutions frequently lack the accurate depth needed for geometric reconstruction.

    Source: §1 · p. 2

  • Method

    OccDepth uses Stereo-SFA to fuse stereo-derived depth-aware features and OAD to distill depth knowledge into 3D features.

    Source: §1 · p. 2

  • Results

    OccDepth outperforms current RGB-inferred baselines on all classes and is comparable to some 2.5D/3D-input methods across indoor and outdoor scenes.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The SemanticTartanAir results verify improvements from stereo input in indoor scenes.

    Source: §5 · p. 8

    Stereo-SFA increases computation by 42%, while OAD improves mIoU with almost no added computation.

    Source: Architectural Components · p. 6

Abstract

from arXiv · show

3D Semantic Scene Completion (SSC) can provide dense geometric and semantic scene representations, which can be applied in the field of autonomous driving and robotic systems. It is challenging to estimate the complete geometry and semantics of a scene solely from visual images, and accurate depth information is crucial for restoring 3D geometry. In this paper, we propose the first stereo SSC method named OccDepth, which fully exploits implicit depth information from stereo images (or RGBD images) to help the recovery of 3D geometric structures. The Stereo Soft Feature Assignment (Stereo-SFA) module is proposed to better fuse 3D depth-aware features by implicitly learning the correlation between stereo images. In particular, when the input are RGBD image, a virtual stereo images can be generated through original RGB image and depth map. Besides, the Occupancy Aware Depth (OAD) module is used to obtain geometry-aware 3D features by knowledge distillation using pre-trained depth models. In addition, a reformed TartanAir benchmark, named SemanticTartanAir, is provided in this paper for further testing our OccDepth method on SSC task. Compared with the state-of-the-art RGB-inferred SSC method, extensive experiments on SemanticKITTI show that our OccDepth method achieves superior performance with improving +4.82% mIoU, of which +2.49% mIoU comes from stereo images and +2.33% mIoU comes from our proposed depth-aware method. Our code and trained models are available at https://github.com/megvii-research/OccDepth.

1 Introduction

OccDepth uses implicit depth from stereo images to improve 3D semantic scene completion, targeting the gap between vision-only and 3D-input methods. It combines stereo feature fusion and depth distillation, and introduces SemanticTartanAir for evaluation.

1 Introduction

  • The work targets the gap between vision-only and 3D-input SSC because accurate depth is crucial for reconstructing 3D geometry.The authors note that pure-vision solutions frequently lack accurate depth information.Source: §1 · p. 2
  • SemanticTartanAir, a reformed TartanAir benchmark, is introduced to evaluate stereo-input SSC in virtual scenes with real ground truth.Source: §1 · p. 2
  • Stereo-SFA implicitly learns correlations between stereo images to fuse 3D depth-aware features.OccDepth is introduced as the first stereo 3D SSC method with vision-only input.Source: §1 · p. 2
  • OAD distills knowledge from pre-trained depth models to produce depth-aware 3D features.Source: §1 · p. 2
  • OccDepth outperforms vision-only baselines by a large margin and is close to 3D-input solutions.The paper reports superior performance on SemanticKITTI, NYUv2, and SemanticTartanAir.Source: §1 · p. 2

2 Related works

Related work spans image-based 3D reconstruction, scene completion, and semantic completion, while this paper focuses on completing scene geometry and semantics. Prior SSC approaches use RGB-D, geometric inputs, or RGB images, and depth distillation provides a separate context for OccDepth’s design.

3D Reconstruction From Images

  • Image-based 3D reconstruction ranges from shape-from-shading and structure-from-motion to neural radiance fields.SceneRF [Cao and de Charette, 2022b] uses probabilistic ray sampling and a spherical U-Net for large-scene reconstruction from a single view.Source: §2 · p. 2

Scene Completion

  • Scene completion infers complete scene geometry from images or sparse 3D inputs, while semantic instance completion predicts object instances and their reconstructions from partial observations.Source: §2 · p. 2

3D Semantic Scene Completion

  • SSC methods include RGB-D and geometric-input approaches, alongside RGB-only methods such as Dahnert et al. and MonoScene [Cao and de Charette, 2022a].The paper focuses on completing both geometry and semantics using multiview RGB images for indoor and outdoor scenes.Source: §2 · p. 2

Knowledge Distillation

  • Knowledge distillation transfers learned knowledge through outputs, intermediate features, or neuron activations; OccDepth distinguishes its depth teaching from BEVDepth’s sparse LiDAR depth supervision.OccDepth uses explicit dense multiview stereo depth to teach OAD’s depth-prediction capability.Source: §2 · p. 2

3 Methods

OccDepth lifts stereo image features into 3D voxels, adds depth-based occupancy information, and predicts voxel geometry and semantics. Its method combines stereo feature fusion, depth distillation, task-specific losses, and overfitting countermeasures.

3.1 Overview

  • Stereo-SFA lifts image features into 3D voxels, while OAD adds depth information before a 3D U-Net predicts geometry and semantics.The stereo depth network supplies supervision during training.Source: §3.1 · p. 2, Figure 2 · p. 3

3.2 Stereo Soft Feature Assignment Module

  • Stereo images provide implicit depth that partly addresses monocular scale ambiguity, and Stereo-SFA fuses voxel-projected features using left-right feature correlation.The authors report that this mapping outperforms directly using stereo depth or combining left and right 2D features.Source: §3.2 · p. 3

Module Structure

  • OAD converts predicted depth distributions into voxel-space occupancy probabilities and uses them as spatial priors for voxel features.It uses LID depth discretization, which the authors say gives a more refined expression for near space.Source: Module Structure · p. 4

Stereo Depth Distillation

  • Stereo depth distillation explicitly supervises the predicted depth feature using dense ground-truth depth generated by a stereo depth network.Min pooling and a one-hot operator align the ground-truth depth before binary cross-entropy is calculated.Source: Stereo Depth Distillation · p. 4

3.4 Losses

  • The losses separate occupancy and semantic prediction, and training also includes depth and MonoScene losses.The paper describes the semantic and geometric losses as supporting accuracy and the completion loss as improving completion ability.Source: §3.4 · p. 4

3.5 Tricks for Mitigating Over-fitting

  • To mitigate overfitting from many 3D convolution parameters and limited training data, OccDepth uses 2D pretraining, data augmentation, and a decaying semantic-loss weight.The semantic-loss weight decreases with training steps but is bounded below by 0.2.Source: §3.5 · p. 5

4 Experiments

Experiments evaluate OccDepth across SemanticKITTI, NYUv2, and SemanticTartanAir, comparing it with vision-only and 2.5D/3D-input methods and examining its components. Results include quantitative benchmark comparisons, qualitative predictions, and module ablations.

4.1 Datasets and Metric

  • SemanticKITTI uses 256×256×32 voxels and 21 classes; NYUv2 uses downsampled 60×36×60 evaluation grids, while SemanticTartanAir covers indoor office scenes with 120×48×120 grids.SemanticKITTI quantitative results use the hidden test set; its ablations use validation data. IoU evaluates scene completion and mIoU evaluates semantic scene completion.Source: §4.1 · pp. 5–6

4.2 Comparison to State-of-the-arts

  • OccDepth improves over MonoScene by +4.82 mIoU/+10.94 IoU on SemanticKITTI and +10.84 mIoU/+17.62 IoU on SemanticTartanAir.The SemanticKITTI results use the hidden test set; SemanticTartanAir results use its test set.Source: §4.2 · p. 6
  • On SemanticKITTI, OccDepth is comparable to 2.5D/3D-input methods despite 82° versus 180° FOV; on NYUv2 it leads by +0.8 IoU/+1.7 mIoU.On SemanticTartanAir, its mIoU is close to 2.5D/3D-input methods that use ground-truth depth, while OccDepth does not.Source: Table 2 · p. 5, §4.2 · p. 6

Qualitative Results

  • Figure 5 reports better object-edge restoration indoors and better spatial and semantic predictions outdoors for OccDepth than the baselines.Indoor examples include sofa, ceiling-light, and carpet edges; outdoor examples include road signs, trunks, vehicles, and roads.Source: Qualitative Results · p. 6, Figure 5 · p. 7

Architectural Components

  • The architecture ablation reports improvements from Stereo-SFA, OAD, depth distillation, and overfitting mitigation; the final mitigation step adds +0.02 IoU/+0.51 mIoU.Stereo-SFA increases computation by 42%, while OAD improves mIoU with almost no computational increase.Source: Architectural Components · p. 6

Effectiveness of Stereo-SFA module

Effectiveness of OAD module

  • LID depth discretization achieves the highest accuracy, while SID has comparable mIoU; the authors suggest sparse lidar depth may explain lower accuracy.Lidar-projected depth images are sparse in image space, whereas LEAStereo-generated depth images are spatially dense.Source: Effectiveness of OAD module · p. 7

5 Conclusion

OccDepth is a depth-aware 3D semantic scene completion method evaluated on indoor and outdoor datasets. It is comparable to some 2.5D/3D-input methods and outperforms current RGB-inferred baselines on all classes.

5 Conclusion

  • OccDepth is comparable to some 2.5D/3D-input methods on indoor and outdoor scenes and outperforms current RGB-inferred baselines on all classes.The paper reports training on SemanticKITTI and NYUv2.Source: §5 · p. 7
  • Results on the newly proposed SemanticTartanAir indoor benchmark verify improvements from stereo input.Source: §5 · p. 8
  • OccDepth is presented as the first stereo RGB-inferred method comparable to 2.5D/3D-input SSC methods.Source: §5 · p. 8

Related papers

Loading 2302.13540v1…