Source-linked AI summary
Recurrent Slice Networks for 3D Segmentation of Point Clouds
Qiangui Huang, Weiyue Wang, Ulrich Neumann
TL;DR
Point-cloud segmentation must model local dependencies despite unordered inputs and the information loss or computational cost of alternative representations. RSNet uses slice pooling, RNNs, and slice unpooling to process ordered local features, and experiments report state-of-the-art performance across three benchmarks with less inference time and memory.
Problem
Unordered point clouds make local dependency modeling difficult, while alternative representations can lose information or require costly computation.
Method
RSNet combines slice pooling, RNN layers, and slice unpooling to project unordered point features into ordered sequences, model dependencies, and return features to points.
Results
RSNet surpasses previous state-of-the-art methods on S3DIS, ScanNet, and ShapeNet.
Takeaways & Limitations
The framework provides efficient local dependency modeling while requiring less inference time and memory than previous state-of-the-art methods.
Takeaways & Limitations
The ablation studies keep other settings unchanged from the baseline RSNet unless explicitly specified.
Abstract
from arXiv · showhide
Point clouds are an efficient data format for 3D data. However, existing 3D segmentation methods for point clouds either do not model local dependencies \cite{pointnet} or require added computations \cite{kd-net,pointnet2}. This work presents a novel 3D segmentation framework, RSNet\footnote{Codes are released here https://github.com/qianguih/RSNet}, to efficiently model local structures in point clouds. The key component of the RSNet is a lightweight local dependency module. It is a combination of a novel slice pooling layer, Recurrent Neural Network (RNN) layers, and a slice unpooling layer. The slice pooling layer is designed to project features of unordered points onto an ordered sequence of feature vectors so that traditional end-to-end learning algorithms (RNNs) can be applied. The performance of RSNet is validated by comprehensive experiments on the S3DIS\cite{stanford}, ScanNet\cite{scannet}, and ShapeNet \cite{shapenet} datasets. In its simplest form, RSNets surpass all previous state-of-the-art methods on these benchmarks. And comparisons against previous state-of-the-art methods \cite{pointnet, pointnet2} demonstrate the efficiency of RSNets.
1. Introduction
RSNet directly segments raw point clouds by converting unordered point features into ordered sequences for local dependency modeling. It is evaluated on S3DIS, ScanNet, and ShapeNet, where it surpasses prior state-of-the-art methods.
- Motivation: Point clouds are difficult to use with powerful end-to-end algorithms because they are unstructured and unordered.Many methods instead transform them into voxels or multi-view renderings.
- Motivation: Voxel and multi-view transformations can lose information and local context, while their associated CNNs are often time- and memory-consuming.
- RSNet: RSNet directly takes raw point clouds as input and outputs semantic labels for each point.
- RSNet: The local dependency module uses slice pooling, RNN layers, and slice unpooling to model dependencies among point-cloud features.Slice pooling produces ordered aggregated features, RNNs model their dependencies, and unpooling assigns updated features back to points.
- Evaluation: RSNets are validated on S3DIS, ScanNet, and ShapeNet, outperforming prior results and significantly improving performance on S3DIS and ScanNet.
2. Related Works
Prior 3D analysis methods use handcrafted features or transform point clouds into voxelized volumes and multi-view renderings. Point-cloud methods preserve the representation but differ in how they model local context and computational cost.
- Voxelized Volumes: Early end-to-end 3D methods converted point clouds into voxelized occupancy grids and applied 3D CNNs.Memory constraints limited input cubes and CNN depth in these methods.
- Multi-view Renderings: Multi-view methods transform 3D data into images for 2D CNN processing, reducing computational costs relative to volumetric methods but losing information during rendering.
- Point Clouds: PointNet processes points independently and aggregates them into a global feature, but does not model local geometric contexts.
- Point Clouds: PointNet++ incorporates local dependencies and hierarchical feature learning, while KD-net uses KD-tree construction.
- Point Clouds: Existing local-context methods rely on heavy extra computations whose costs grow linearly as higher local-detail resolutions are used.
3. Method
RSNet extracts independent point features, projects unordered points into ordered slice features, models their dependencies with bidirectional RNNs, and maps the results back to points. Slicing along three axes and linear-time pooling and unpooling support efficient local-context modeling.
- Independent Feature Extraction: RSNet’s input and output feature blocks independently generate point representations before and after the local dependency module.Both use multiple 1 × 1 convolution layers for independent feature representations.
- Independent Feature Extraction: The network uses three parallel slicing directions along the x, y, and z axes.
- RNN Modeling: The ordered slice sequence enables RNNs to model dependencies that cannot be directly applied to unordered point clouds.
- Slice Pooling: Slice pooling groups points into spatial slices and aggregates each slice into one feature vector, producing an ordered sequence.The slicing resolution r controls the number and spatial extent of slices.
- Efficiency and Trade-offs: Slice pooling has time complexity O(n) in the number of input points and O(1) with respect to slicing resolution r.
- Efficiency and Trade-offs: Smaller slicing resolutions preserve richer local details but create longer RNN sequences, whereas larger resolutions produce fewer slices with coarser contexts.The choice of r therefore involves a local-detail and sequence-length trade-off.
- RNN Modeling: Bidirectional RNNs allow information from one slice to affect other slices, updating features through interactions with neighboring points.
- Slice Unpooling: Slice unpooling reverses the projection and assigns updated slice features back to individual points.It stores the slice sets S to perform the reverse assignment.
4. Experiments
RSNet is evaluated on realistic and synthetic 3D segmentation benchmarks using a unified architecture and controlled ablations. It achieves strong segmentation results while preserving computational efficiency, with performance influenced by slicing resolution and recurrent-unit choice.
- RSNet is benchmarked on S3DIS, ScanNet, and ShapeNet, covering large-scale realistic scenes and synthetic 3D part segmentation.
- S3DIS segmentation: RSNet achieves state-of-the-art S3DIS performance without data augmentation, improving mean IOU by 3.01 and mean accuracy by 2.07 over prior 3D-volume methods.It improves mean IOU by 10.84 and mean accuracy by 10.44 over PointNet, with better per-category IOU in 7 of 13 categories.
- S3DIS segmentation: The local dependency module is associated with performance gains by modeling local geometric dependencies beyond PointNet’s global features and avoiding quantization-related local-detail loss.
- Ablation studies: A 2cm slicing resolution along x, y, and z performs best on S3DIS, reflecting a trade-off between preserving local details and processing longer RNN sequences.Smaller resolutions preserve more details but create longer sequences, while larger resolutions aggregate over wider spatial ranges.
- Ablation studies: GRU provides the best performance among the evaluated RNN units for RSNet.
- ScanNet and efficiency: On ScanNet, RSNet improves mean IOU by 5.09 and mean accuracy by 4.60 over PointNet++, while handling small details such as chairs, desks, and toilets.The comparison uses xyz information for both methods.
5. Conclusion
RSNet is presented as an efficient 3D segmentation framework that combines lightweight local dependency modeling with strong benchmark performance and lower inference costs.
- RSNet combines slice pooling, RNN layers, and slice unpooling in a lightweight local dependency modeling module.The framework is designed for 3D segmentation of point clouds.
- RSNet surpasses previous state-of-the-art methods on the S3DIS, ScanNet, and ShapeNet benchmarks.
- The ScanNet figure compares predictions from PointNet, PointNet++, and RSNet against ground truth.Highlighted regions identify areas of interest in the sample segmentation results.
- The ShapeNet results are reported in Table 7, including a PointNet++ variant trained without extra normal information.
- Table 8 analyzes computation between PointNet, PointNet++, and RSNet.
Appendices
The appendices provide additional experimental details and report six-fold validation results for S3DIS, including per-category IOUs.
- The appendix presents additional experimental settings, results, and discussions.
- Table 9 reports six-fold validation results on S3DIS with IOU values for each category.
A. More Results and Discussions on the S3DIS dataset
The S3DIS appendix describes the dataset inputs, compares six-fold validation performance, and examines class imbalance and median frequency balancing.
- S3DIS contains 272 indoor scenes from six areas in three buildings, annotated across 13 categories; RSNet uses nine-dimensional point features.The features comprise xyz coordinates, RGB intensities, and normalized xyz coordinates.
- RSNet outperforms PointNet by a large margin in six-fold validation while using less memory and reasonable additional inference time.
- Median frequency balancing improves mean accuracy on S3DIS but slightly decreases mean IOU.The strategy addresses the dataset’s imbalance between dominant and rare classes.
B. More Results and Discussions on the ScanNet dataset
The ScanNet appendix describes the dataset and evaluates adding RGB intensities to xyz inputs, reporting modest overall gains and improved recognition of visually confusable classes.
- ScanNet contains 1,513 scenes from the Matterport 3D sensor, with annotations for 20 categories and one background class.
- The baseline RSNet processes ScanNet points using three-dimensional xyz-coordinate features.
- Adding RGB intensities improves mean IOU by 1.81 and mean accuracy by 1.97.RGB particularly helps distinguish doors, windows, and pictures from walls.