PaperlayerView exact revision on arXiv

Source-linked AI summary

SEGCloud: Semantic Segmentation of 3D Point Clouds

Lyne P. Tchapmi, Christopher B. Choy, Iro Armeni, JunYoung Gwak, Silvio Savarese

arXiv:1710.07563v1cs.CV

TL;DR

3D-FCNNs provide useful representations for 3D semantic segmentation but produce coarse voxel predictions, while fine-grained point labeling requires preserving point-level detail and spatial consistency. SEGCloud transfers voxel predictions to raw points with trilinear interpolation and applies a differentiable FC-CRF in an end-to-end pipeline. Across four indoor and outdoor datasets, it achieves performance comparable or superior to state-of-the-art methods, while remaining constrained by the computational and memory costs of dense voxel grids.

  • Problem

    3D-FCNNs produce coarse voxel-level predictions, limiting fine-grained semantic labeling of raw 3D points.

  • Method

    SEGCloud voxelizes point clouds, predicts class distributions with a 3D-FCNN, interpolates scores to raw points, and uses a jointly optimized FC-CRF for fine-grained inference.

  • Results

    The framework achieves performance on par with or better than previous methods across NYU V2, S3DIS, KITTI, and Semantic3D.net.

  • Takeaways & Limitations

    Combining 3D-FCNN predictions with trilinear interpolation and CRF inference provides fine-grained point-cloud segmentation with spatially consistent labels.

  • Takeaways & Limitations

    Dense voxel processing faces computational and memory constraints as grid resolution increases, because 3D-FCNN memory requirements grow cubically with resolution.

Abstract

from arXiv · show

3D semantic scene labeling is fundamental to agents operating in the real world. In particular, labeling raw 3D point sets from sensors provides fine-grained semantics. Recent works leverage the capabilities of Neural Networks (NNs), but are limited to coarse voxel predictions and do not explicitly enforce global consistency. We present SEGCloud, an end-to-end framework to obtain 3D point-level segmentation that combines the advantages of NNs, trilinear interpolation(TI) and fully connected Conditional Random Fields (FC-CRF). Coarse voxel predictions from a 3D Fully Convolutional NN are transferred back to the raw 3D points via trilinear interpolation. Then the FC-CRF enforces global consistency and provides fine-grained semantics on the points. We implement the latter as a differentiable Recurrent NN to allow joint optimization. We evaluate the framework on two indoor and two outdoor 3D datasets (NYU V2, S3DIS, KITTI, Semantic3D.net), and show performance comparable or superior to the state-of-the-art on all datasets.

1. Introduction

SEGCloud addresses the coarse, voxel-level nature of 3D-FCNN predictions by transferring class scores to raw points and applying an FC-CRF for fine-grained, spatially consistent labeling. The end-to-end framework combines 3D-FCNNs, trilinear interpolation, and jointly trained CRF inference, with evaluation across indoor and outdoor datasets.

  • 3D semantic segmentation supports fine-grained object boundaries and labels for applications including robotics, autonomous driving, augmented reality, and construction.
  • 3D-FCNN predictions are coarse because regular-grid processing assigns one semantic label to every point within a voxel.Voxel size therefore limits overall accuracy and motivates additional processing for point-level segmentation.
  • SEGCloud transfers voxel-level class probabilities to raw 3D points with trilinear interpolation, then uses an FC-CRF to infer spatially consistent point labels.
  • The voxel-level 3D-FCNN and point-level CRF are connected through trilinear interpolation and trained jointly end-to-end.
  • The framework handles laser-scanner and RGB-D point clouds and is evaluated on NYU V2, S3DIS, KITTI, and Semantic3D.net.These datasets cover indoor and outdoor, partial and fully reconstructed 3D scenes.

2. Related Work

Prior 3D segmentation methods combine classifiers with graphical models or use neural networks, but SEGCloud emphasizes an end-to-end point-level framework. Its distinctive design uses deterministic trilinear interpolation and a differentiable, jointly trained CRF to refine coarse CNN outputs.

  • Related work spans neural networks for 3D data, graphical models for segmentation, and combinations of convolutional neural networks with CRFs.
  • Classifier–CRF pipelines commonly impose smoothness and consistency, but Random Forest and CRF modules are often optimized independently.
  • SEGCloud replaces hand-crafted classifier features with a 3D-FCNN that learns higher-dimensional features and provides point-level unary scores.
  • Unlike prior joint CNN–CRF work for medical scans, SEGCloud is a single-stream architecture for 3D scene point clouds that jointly optimizes the CNN and CRF.
  • Deterministic interpolation weights based on metric distance reduce learned parameters, and the jointly trained 3D-FCNN, CRF, and TI outperform a standalone 3D-FCNN.
  • The paper differentiates itself through end-to-end fine-grained segmentation, deterministic trilinear interpolation, and a jointly trained CRF for spatial consistency.

3. SEGCloud Framework

SEGCloud voxelizes 3D observations, predicts coarse voxel class distributions with a 3D-FCNN, transfers them to raw points through trilinear interpolation, and applies an FC-CRF for fine-grained, spatially consistent labeling.

  • The pipeline voxelizes 3D data, processes the grid with a 3D-FCNN, interpolates voxel scores to points, and applies a CRF for final point distributions.The entire pipeline is jointly optimized.
  • Each 3D observation includes position and available modalities such as color and sensor intensity, represented on a regular grid with 5cm cubic voxels.An occupancy channel represents empty space, alongside additional feature channels.
  • Coarse voxelization and downsampling limit CRF labeling resolution to 20cm, while finer grids increase cubic memory costs and reduce receptive-field context.These constraints motivate point-level CRF inference rather than a voxel-based CRF.
  • Trilinear interpolation transfers each point’s score from the 8 spatially closest voxel centers using distance-dependent weights.The same weights propagate gradients from the CRF back to the 3D-FCNN.
  • The CRF combines interpolated point scores with original point features; unary terms represent class distributions, while pairwise terms enforce smoothness and consistency.All CRF nodes are connected through pairwise potentials, using Gaussian kernels with spatial and optional RGB information.

4. CRF Inference and Joint Optimization

SEGCloud performs approximate variational CRF inference and represents the CRF as a recurrent neural network, allowing the CRF and 3D-FCNN to be optimized jointly.

  • Exact CRF energy minimization is intractable, so SEGCloud uses variational inference to jointly optimize the CRF and 3D-FCNN.The resulting point predictions incorporate smoothness and consistency.
  • CRF inference is implemented with convolutions, sums, and softmax operations as a recurrent neural network inside a standard CNN framework.This formulation integrates CRF inference and learning into the 3D-FCNN framework.
  • The training loss is the KL divergence between each final prediction distribution and its ground-truth distribution.The loss is defined over N observations.

5. Implementation Details

SEGCloud is implemented on GPUs with a Caffe-based trilinear interpolation layer and a 3D CRF-RNN, using augmentation and staged training to support robust optimization.

  • All 3D-FCNN, interpolation, and CRF computations run on a GPU, with trilinear interpolation implemented as a neural-network layer in Caffe.The CRF adapts an existing RNN implementation to 3D point clouds.
  • Random color augmentation, upright-axis rotation, and point subsampling increase effective dataset size by at least an order of magnitude.These transformations support invariance to rotation, viewpoint, and varying context.
  • Training first optimizes the 3D-FCNN through trilinear interpolation for 200 epochs, then jointly trains the 3D-FCNN and CRF end-to-end.The two modules are connected through the interpolation layer during the second stage.
  • CRF inference uses a permutohedral-grid approximation whose size depends on kernel bandwidth parameters, with θγ fixed at 5cm and θβ fixed at 11.A validation-set grid search selects θα.

6. Experiments

The experiments evaluate SEGCloud across indoor and outdoor 3D datasets and analyze its benchmark performance and key design components. Results show competitive performance, with measurable gains from the CRF, geometric augmentation, and trilinear interpolation.

  • Datasets: The evaluation covers Semantic3D.net, S3DIS, KITTI, and NYU V2, spanning indoor and outdoor, partial and fully reconstructed point clouds.The datasets were selected to support comparisons with existing 3D semantic-segmentation methods.
  • Benchmark results: 2.2 mIOU points and 2.28% accuracy separate SEGCloud from on Semantic3D.net’s reduced-8 challenge.It also exceeds the best method without extra ImageNet data by 7.1 mIOU points and 4.1% accuracy.
  • Benchmark results: 3.94 mIOU points and 0.83% mean accuracy separate the method from 3D Entangled Forests on NYU V2.The evaluation uses single-frame RGB-D-derived point clouds under the official split and 13-class labeling.
  • Benchmark results: On KITTI, SEGCloud performs on par with the Random Forests baseline using laser point clouds without RGB information.The comparison uses a labeled KITTI subset annotated for sensor-fusion research.
  • System analysis: 3.0-5.3% relative mIOU and 4.4-4.7% relative mean-accuracy improvements result from adding the CRF across datasets.On Semantic3D.net, trilinear interpolation already achieves state-of-the-art performance, while the CRF adds 3.1 mIOU points and 3.22%.

7. Conclusion

The framework combines 3D-FCNN, trilinear interpolation, and CRF for 3D point-cloud semantic segmentation, achieving performance on par with or better than state-of-the-art methods. The authors identify trilinear interpolation and geometric augmentation as important components, while noting that sparse convolutions could further improve the voxel-based implementation.

  • The framework combines 3D-FCNN, trilinear interpolation, and CRF to produce class labels for 3D point clouds.
  • The approach performs on par with or better than state-of-the-art methods based on neural networks, random forests, and graphical models.
  • Geometric 3D data augmentation and trilinear interpolation play a key role in the final performance.
  • The standard voxel-based 3D-FCNN implementation could be adapted to sparse convolutions for an additional performance boost.

A.1. Augmentation Procedures for 3D data

The framework uses on-the-fly color, geometric, and point-subsampling augmentations to enlarge effective training data and improve robustness to viewpoint, scale, density, and context variation. Large scenes are cropped, voxelized at 5cm resolution, and represented with occupancy and available sensor modalities.

  • On-the-fly augmentations increase randomness while avoiding additional storage.
  • Color augmentation randomly varies each RGB channel within ±2.5.
  • Geometric augmentation randomly rotates point clouds around the gravity direction and scales them within [0.9, 1.1].
  • Training-only point subsampling targets dense S3DIS and Semantic3D.net clouds to build invariance to missing points and accelerate training, while testing uses all input points.
  • Random transformations and subsampling increase effective dataset size and support invariance to rotation, viewpoint changes, and reduced or varying context.
  • Large observations are cropped to at most 5m per dimension and voxelized at 5cm resolution, with occupancy, RGB, and available intensity channels.

A.3. Training

Training proceeds in two stages: first optimizing point-wise predictions from trilinearly interpolated voxel outputs, then jointly fine-tuning the pretrained 3D-FCNN, interpolation layer, and CRF end-to-end.

  • The first stage trains the 3D-FCNN with trilinear interpolation by minimizing point-wise loss from voxel-wise predictions.
  • The two-stage procedure first optimizes point-level unary potentials without CRF and then optimizes the joint framework for fine-grained point-level labeling.
  • The second stage combines the pretrained 3D-FCNN, trilinear interpolation layer, and CRF for end-to-end training.

B. Effect of end-to-end training vs separate CRF initialization

Joint end-to-end training outperforms separate CRF initialization, with the clearest advantage reported for mean accuracy.

  • Joint training performs better than separate CRF initialization.
  • The advantage of joint training is especially pronounced for the mAcc metric.
  • The comparison initializes separate-CRF experiments using fixed parameters taken from optimal joint-training values.

C.1. Datasets

SEGCloud is evaluated across diverse indoor and outdoor point-cloud datasets acquired with multiple sensing modalities and reconstruction settings. Evaluation uses mean class accuracy and mean class IOU, alongside qualitative results comparing intermediate and final segmentations.

  • Datasets: The evaluation covers Semantic3D.net, S3DIS, KITTI, and NYU V2, spanning indoor and outdoor environments.Two datasets are indoor and two are outdoor.
  • Datasets: The datasets include laser-scanner, Kinect, and MatterPort acquisitions, with both fully reconstructed and single-frame RGB-D point clouds.S3DIS is fully reconstructed, whereas NYU V2 uses point clouds extracted from a single-frame RGB-D camera.
  • Datasets: Training-set size varies from 12 million points for KITTI to 1.9 billion points for Semantic3D.net, excluding the validation set.This range reflects substantially different evaluation scales across the benchmarks.
  • Evaluation Metrics: Performance is measured with mean class accuracy and mean class IOU, with IOU penalizing false positives in addition to accounting for true positives.The metrics are defined over classes using true positives, false negatives, ground-truth elements, and, for IOU, false positives.
  • Qualitative Results: Qualitative results show the standalone 3D-FCNN-TI segmentation alongside the final SEGCloud output for the evaluated datasets.The qualitative figures include Semantic3D.net, KITTI, S3DIS, and NYU V2 examples.
Loading 1710.07563v1…