PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans

Ji Hou, Angela Dai, Matthias Nießner

arXiv:1812.07003v3cs.CV

TL;DR

3D-SIS addresses 3D semantic instance segmentation in commodity RGB-D scans, where single-frame methods do not directly model spatially consistent instances across views. It jointly learns RGB and geometry features in a fully convolutional, end-to-end network, achieving over 13 mAP improvement on real-world data.

  • Problem

    Single-image methods do not directly provide spatially consistent semantic instance predictions across multi-view RGB-D scans.

  • Method

    3D-SIS jointly learns RGB and geometry features from multi-view RGB-D input to detect 3D object instances and predict per-voxel masks end-to-end.

  • Results

    Over 13 mAP improvement is reported on real-world data over existing state-of-the-art methods.

  • Takeaways & Limitations

    The fully convolutional network enables efficient single-shot inference on large 3D environments.

  • Takeaways & Limitations

    The method assumes semantic RGB-D instance labels and relies on camera poses and intrinsics for RGB back-projection.

Abstract

from arXiv · show

We introduce 3D-SIS, a novel neural network architecture for 3D semantic instance segmentation in commodity RGB-D scans. The core idea of our method is to jointly learn from both geometric and color signal, thus enabling accurate instance predictions. Rather than operate solely on 2D frames, we observe that most computer vision applications have multi-view RGB-D input available, which we leverage to construct an approach for 3D instance segmentation that effectively fuses together these multi-modal inputs. Our network leverages high-resolution RGB input by associating 2D images with the volumetric grid based on the pose alignment of the 3D reconstruction. For each image, we first extract 2D features for each pixel with a series of 2D convolutions; we then backproject the resulting feature vector to the associated voxel in the 3D grid. This combination of 2D and 3D feature learning allows significantly higher accuracy object detection and instance segmentation than state-of-the-art alternatives. We show results on both synthetic and real-world public benchmarks, achieving an improvement in mAP of over 13 on real-world data.

1. Introduction

3D-SIS addresses the limitations of single-image instance segmentation by jointly learning from multi-view RGB-D data to predict spatially consistent 3D object instances. Its fully convolutional design supports single-shot inference and substantially improves performance over state-of-the-art alternatives.

  • 3D semantic understanding supports applications requiring spatial relationships and object layouts, including robotics and mobile or AR/VR devices.
  • Single-image methods are poorly suited to semantic instance segmentation because instance associations must be recovered across RGB frames.
  • 3D-SIS jointly learns semantic features from RGB-derived color features and geometry features in reconstructed 3D scans.
  • The fully convolutional architecture enables efficient single-shot inference on large 3D environments.
  • 13.5 mAP is the reported improvement over state-of-the-art on real-world data.

2. Related Work

Prior work progressed from 2D detection and segmentation toward 3D learning using RGB-D frames, volumetric representations, and multi-view aggregation. 3D-SIS combines holistic 3D representation with learned 2D features.

  • 2D CNN-based methods advanced object detection and semantic instance segmentation, including per-pixel masks in Mask R-CNN.
  • Early 3D approaches predicted object boxes from single RGB-D frames using handcrafted or learned features.
  • Volumetric convolutional networks use implicit signed distance fields and support tasks such as 3D classification and semantic segmentation.
  • Multi-view methods aggregate 2D predictions through view pooling, projection, or CRFs, while 3D-SIS joins holistic 3D representation with 2D features.

3. Method Overview

3D-SIS performs end-to-end 3D object detection and per-voxel semantic instance mask prediction using jointly learned geometry and RGB features. The overview connects these outputs within one neural pipeline.

  • The method predicts 3D bounding box locations, class labels, and per-voxel semantic instance masks end-to-end.
  • Its architecture combines 3D convolutions over voxel-grid scan data with learned 2D features associated with the 3D representation.

4. Training Data

The training and inference setup represents scans as TSDF voxel grids aligned with RGB images through camera poses, using synthetic and real-world datasets. The network processes geometry and color jointly for scene-level predictions.

  • Data Representation: TSDF geometry and spatially associated RGB images form the input representation, linked through 6DoF reconstruction poses.
  • Data Representation: Training chunks measure 4.5m × 4.5m × 2.25m and contain 96 × 96 × 48 voxels, with five associated RGB images at 328x256 resolution.
  • Inference: Fully convolutional inference runs over entire scenes in one shot, with all available RGB images used at test time.
  • Synthetic Data: SUNCG provides synthetic training and evaluation scans, including 5519 training, 40 validation, and 86 test scenes.
  • Synthetic Data: Synthetic partial scans are generated by virtual RGB-D rendering, denser camera trajectories, TSDF volumetric fusion, and camera-pose image-to-voxel associations.
  • Real-world Data: ScanNetV2 supplies 1513 real-world RGB-D scenes with approximately 2.5 million frames, pose alignments, reconstructed models, and annotated 3D instance masks.
  • Real-world Data: The real-world split contains 1045 training, 156 validation, and 312 test scenes.
  • Network Architecture: Figure 2 shows detection and mask pipelines jointly using geometry and color, with detection results conditioning per-voxel masks inside predicted boxes.

5. Network Architecture

3D-SIS combines back-projected RGB features with 3D geometry to perform detection and per-voxel instance mask prediction in volumetric scans. Its two-backbone, fully convolutional design supports multi-view fusion and preserves spatial resolution for accurate masks.

  • Network Architecture: The architecture has separate detection and mask pipelines, each with a 3D feature-extraction backbone receiving scan geometry and back-projected RGB features.The detection pipeline adds a 3D-RPN, 3D-RoI pooling, and classification head; the mask pipeline predicts per-voxel instance masks.
  • RGB Feature Learning: 2D CNN features are backprojected from aligned RGB images into the voxel grid and max-pooled across views before fusion with geometric features.Depth, camera intrinsics, and 6DoF poses establish the pixel-to-voxel mapping; the method uses all available RGB images at test time.
  • 3D Feature Backbones: The detection backbone symmetrically processes geometry and RGB features, concatenates them, and produces multiscale feature maps for 3D box regression and classification.Small and large anchors are assigned to feature maps with different receptive fields, with anchor shapes selected by k-means.
  • Architecture Evaluation: The two-backbone structure converged more easily and produced significantly better instance segmentation than a single-backbone alternative.The synthetic evaluation measures mean average precision at IoU 0.25 over 23 classes.
  • Per-Voxel 3D Instance Segmentation: The mask backbone maintains spatial resolution through its 3D convolutions, then crops features using predicted boxes to produce class-specific voxel masks.Training retains only predictions whose boxes overlap ground truth by at least 0.5 IoU.

6. Training

Training proceeds sequentially, adding detection, classification, and mask components while retaining earlier losses and ultimately optimizing the full network end-to-end.

  • Training Schedule: The model first trains the detection backbone and 3D-RPN, then adds 3D-RoI pooling and classification before adding the mask network.Each stage retains the previous losses with a 1:1 ratio across losses.
  • Training Schedule: Sequential training resulted in more stable convergence than training the complete architecture without this staged procedure.
  • Optimization: SGD training uses learning rate 0.001, momentum 0.9, and batch sizes of 64 for 3D-RPN and 16 for classification and mask prediction.The learning rate is divided by 10 every 100k steps.
  • Optimization: The object-detection components train for 10 epochs, followed by 5 additional epochs after adding the mask backbone.The reported durations are approximately 24 hours and 16 hours, respectively, on a single Nvidia GTX1080Ti GPU.

7. Results

3D-SIS is evaluated for 3D detection and instance segmentation on synthetic SUNCG and real-world ScanNetV2 scans, where it consistently outperforms prior approaches. The method’s joint color-geometry learning and holistic 3D inference support improved benchmark performance and coherent predictions.

  • 7.2. 3D Instance Analysis on Real-World Scans: The final geo+5views model significantly outperforms previous and concurrent methods on the official ScanNetV2 hidden test set in mAP@0.5.Compared methods include Mask R-CNN, SGPN, MTML, 3D-BEVIS, and R-PointNet.
  • 7.2. 3D Instance Analysis on Real-World Scans: Qualitative ScanNetV2 comparisons show more accurate and semantically coherent predictions from joint color-geometry learning and full-scan inference.Figure 3 presents full scans above and close-ups below; instance colors need not match between ground truth and predictions.
  • 7.2. 3D Instance Analysis on Real-World Scans: mAP@0.25 on 18 classes shows significantly improved ScanNetV2 3D instance segmentation from explicit spatial mapping between geometry and 2D color features.The learned color features are extracted through 2D CNNs.
  • 7.1. 3D Instance Analysis on Synthetic Scans: 3D-SIS improves 3D detection and instance segmentation on synthetic SUNCG scans across 23 classes.The evaluation compares against SGPN, Seg-Cluster, and Mask R-CNN projected into 3D.
  • 7.1. 3D Instance Analysis on Synthetic Scans: mAP over 23 classes is significantly improved on SUNCG compared with approaches operating on individual input frames.The reported advantage combines color and geometric features with a holistic view of the 3D scene at test time.
  • 7.2. 3D Instance Analysis on Real-World Scans: 3D-SIS achieves significantly improved 3D detection on ScanNetV2 over Deep Sliding Shapes, Frustum PointNet, and Mask R-CNN projected to 3D.The comparison uses mAP over 18 classes.

8. Conclusion

3D-SIS jointly learns from RGB and geometry in multi-view RGB-D scans for end-to-end 3D instance prediction. Its fully-convolutional design supports efficient single-shot inference on large environments, with reported improvements exceeding 13 mAP on real-world data.

  • 8. Conclusion: 3D-SIS jointly learns RGB and geometry features from multi-view RGB-D input to predict 3D object instances end-to-end.The network detects object instances and infers per-voxel 3D semantic instance segmentation.
  • 8. Conclusion: The fully-convolutional network runs efficiently in a single shot on large 3D environments.The approach is designed for commodity RGB-D sensors and multi-view RGB-D scans.
  • 8. Conclusion: The method achieves significantly better 3D detection and instance segmentation results than existing state-of-the-art methods, improving mAP by over 13.The conclusion frames multi-view RGB and depth streams as relevant to applications such as autonomous cars and AR/VR.

A. Network Architecture

The architecture uses voxel-based region proposals and fully-convolutional detection and mask backbones, while 3D RoI-pooling allows full-scan processing despite fully connected classification layers.

  • A. Network Architecture: The detection and mask backbones are fully-convolutional, while the classification head uses several fully connected layers.3D RoI-pooling on the classification input permits processing full scans of varying sizes.
  • A. Network Architecture: The network architecture includes a detection backbone, 3D-RPN, classification head, mask backbone, and mask prediction components.Layer specifications for these components are detailed in Table 8.
  • A. Network Architecture: Anchor sizes for SUNCG and ScanNet region proposals are specified in voxel units at approximately 4.69cm voxel resolution.Dataset-specific anchors are determined by k-means clustering of ground-truth bounding boxes.

B. Training and Inference

3D-SIS trains on cropped scan volumes for memory and efficiency, then generalizes to full-scene inference. Its fully-convolutional design enables single-pass testing on very large scans, including entire floors or buildings.

  • B. Training and Inference: Training uses cropped scan chunks to satisfy memory and efficiency constraints, while testing generalizes to the full scene.Full-scene inference also helps predict consistent object boundaries across differing views of the same object.
  • B. Training and Inference: Fully-convolutional inference enables testing on very large scans, including entire floors or buildings, in a single forward pass.The approach is trained on scene parts but performs single-shot inference on large 3D environments.
  • B. Training and Inference: Region-proposal anchor sizes for ScanNet are listed in voxel units at approximately 4.69cm resolution.The anchors are dataset-specific implementation details for region proposal.
  • B. Training and Inference: Table 8 specifies the layers used in the detection backbone, 3D-RPN, classification head, mask backbone, and mask prediction.These components form the network used for full-scan inference.
  • B. Training and Inference: A large SUNCG scene measuring 45m x 45m can be tested in about 1 second.This runtime is attributed to the fully-convolutional architecture.

C. Additional Experiment Details

The experiments evaluate 3D-SIS across synthetic and real-world scans, including IoU 0.5 mean average precision and an ablation of input signals and anchor design. Joint color-geometry learning and full-scan inference produce stronger instance segmentation, with state-of-the-art results on the ScanNet Benchmark.

  • State-of-the-art performance is achieved on the ScanNet Benchmark in all three reported metrics.
  • Joint color-geometry feature learning enables significantly better instance segmentation performance at IoU 0.5.
  • The ScanNetV2 ablation finds that geometry and color signals complement each other, achieving the best performance together.
  • Mean average precision is evaluated at an IoU threshold of 0.5 on both ScanNetV2 real-world scans and SUNCG synthetic scans.ScanNetV2 evaluation covers 18 classes, while SUNCG evaluation covers 23 classes.
Loading 1812.07003v3…