Source-linked AI summary
Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs
Loic Landrieu, Martin Simonovsky
TL;DR
Large-scale 3D point-cloud segmentation is hindered by massive data volume and limited ability of existing approaches to process large inputs while preserving structure. The paper partitions scenes into superpoints, builds attributed superpoint graphs, and applies graph-based contextual segmentation, achieving state-of-the-art results on Semantic3D and S3DIS. It also identifies partitioning and contextual learning as areas for future improvement.
Problem
Large 3D point-cloud segmentation is challenging because scans can be extremely large, while existing deep architectures are limited by the size of inputs they can handle.
Method
The framework partitions point clouds into geometrically simple superpoints, constructs attributed graphs over them, and combines superpoint embeddings with graph convolutions for contextual segmentation.
Results
11.9, 8.8, and up to 12.4 mIoU points are gained on the Semantic3D reduced set, Semantic3D full set, and S3DIS, respectively, establishing state-of-the-art performance on two public datasets.
Takeaways & Limitations
Superpoint graphs enable deep learning on large-scale point clouds and provide a compact representation for contextual segmentation.
Takeaways & Limitations
The approach remains constrained by input-size limitations in existing deep architectures and by information loss or inhomogeneity associated with partitioning choices.
Abstract
from arXiv · showhide
We propose a novel deep learning-based framework to tackle the challenge of semantic segmentation of large-scale point clouds of millions of points. We argue that the organization of 3D point clouds can be efficiently captured by a structure called superpoint graph (SPG), derived from a partition of the scanned scene into geometrically homogeneous elements. SPGs offer a compact yet rich representation of contextual relationships between object parts, which is then exploited by a graph convolutional network. Our framework sets a new state of the art for segmenting outdoor LiDAR scans (+11.9 and +8.8 mIoU points for both Semantic3D test sets), as well as indoor scans (+12.4 mIoU points for the S3DIS dataset).
1. Introduction
Large-scale 3D point-cloud segmentation is difficult because scans are enormous and lack image-like regular structure. The paper addresses this with superpoint graphs that represent object parts and their contextual relationships for deep segmentation.
- Large 3D point clouds challenge deep learning because of their scale and lack of a regular grid structure.
- Existing approaches either lose information through 2D conversion or cannot process sufficiently large inputs at once.
- Superpoint graphs represent point clouds as interconnected geometrically simple shapes with rich edge features encoding relationships between object parts.
- The representation supports object-part classification, detailed adjacency modeling, and long-range interactions because graphs are much smaller than the original point clouds.
- The architecture combines PointNets for superpoint embeddings, graph convolutions for contextual segmentation, efficient Edge-Conditioned Convolutions, and gated recurrent units.
- 11.9 points, 8.8 points, and up to 12.4 points are the reported mIoU improvements on the Semantic3D reduced set, Semantic3D full set, and S3DIS, respectively.
2. Related Work
Prior large-scale point-cloud segmentation commonly classifies local points or voxels and then regularizes predictions, while newer deep models use set-, tree-, or graph-based processing. The paper positions graph convolutions with continuous edge attributes as a way to model contextual relationships.
- Classic systems classify points or voxels using handcrafted local features, then apply graphical models or structured optimization for spatial regularization.
- Recent deep-learning approaches for point clouds include set-based, tree-based, and graph-based architectures, but few demonstrate large-scale segmentation.
- PointNet handles large clouds with sliding windows, which constrains contextual information to small areas.
- Graph convolutions support message passing over variable-size graphs, with continuous edge attributes providing a mechanism for representing interactions.
- Unlike sequential graph postprocessing, the paper updates graph nodes in parallel and uses edge attributes for contextual modeling.
3. Method
The framework addresses the scale of LiDAR scans by partitioning point clouds into geometrically simple superpoints, representing their relationships in an SPG, and applying contextual graph learning.
- Contextual Segmentation: The SPG separates segmentation into stages whose graph is far smaller than a point-level graph, enabling long-range contextual interactions at lower scale.The framework trains the embedding and contextual segmentation stages end to end.
- Geometric Partition: The pipeline partitions the entire point cloud into geometrically simple superpoints before constructing the SPG.The partition is unsupervised and computationally efficient, and its connected components define the superpoints.
- Contextual Segmentation: Graph convolutions refine superpoint embeddings through message passing along superedges to produce contextual segmentation labels.The framework introduces input gating for GRU messages, a lightweight edge-conditioned convolution variant, and state concatenation for final logits.
- Geometric Partition: The partitioning objective uses local geometric features on a 10-nearest-neighbor graph, with regularization strength µ controlling partition coarseness.The optimization produces constant connected components that become the geometrically simple elements.
- Superpoint Graph Construction: The SPG is an oriented attributed graph whose nodes are superpoints and whose directed edges encode adjacency with rich features.Adjacency is derived from Voronoi connections, while edge features include spatial offsets and relative shape and size information.
- Superpoint Embedding: Each superpoint is embedded independently into a fixed-size vector using a compact PointNet applied to a small sampled point set.Context is deliberately supplied only in the subsequent graph-convolution stage.
4. Experiments
Experiments evaluate the framework on large outdoor and indoor point-cloud benchmarks using original-point-cloud metrics, with nearly the same compact model across datasets. Results show state-of-the-art segmentation, while ablations quantify contextual modeling, partition quality, and voxelization effects.
- Evaluation setup: The framework is evaluated on Semantic3D and S3DIS, with nearly the same compact model used for both outdoor and indoor datasets.The model requires 6 GB of GPU memory for testing and training.
- Evaluation setup: IoU, per-class accuracy, and overall accuracy are computed on the original point clouds rather than on superpoints.Overall accuracy is defined as the proportion of correctly classified points.
- Semantic3D: Nearly 12 mIoU points are gained on Semantic3D’s reduced set and nearly 9 mIoU points on its full set over recent state-of-the-art methods.The framework particularly improves the artefact class, attributed to partitioning artifacts by their singular shape.
- S3DIS: The framework performs better than other methods on average on S3DIS, with open doors classified more accurately but white boards merged with walls.The theoretical perfect-superpoint classification IoU for boards is only 51.3.
- Computation: Voxelization preprocessing significantly speeds computation and improves semantic-segmentation accuracy, while partitioning and SPG computation remain the main time costs.Inference time is measured on S3DIS Area 5 using a 4 GHz CPU and GTX 1080 Ti GPU.
- Ablation studies: Contextual segmentation gains roughly 22 mIoU points over unary predictions, but the learned model retains up to 26 points of improvement and loses about 12 points to superpoint inhomogeneity.Removing input gating or GRU-state concatenation costs about 5 mIoU points each; removing edge features reduces performance to the unary level.
5. Conclusion
The paper presents a deep learning framework for semantic segmentation of large point clouds using partitions into simple shapes and reports significant improvements over the state of the art.
- Superpoint graphs enable effective deep learning tools that otherwise could not handle the data volume.
- The method significantly improves on the state of the art on two publicly available datasets.
- Future improvements may target both partitioning and deep contextual classifiers.
A. Model Details
The model combines voxelization, geometric partitioning, PointNet superpoint embeddings, and graph-based segmentation, with dataset-specific preprocessing and training settings.
- Voxelization: Voxelization subsamples input clouds using per-voxel mean positions and observations on regular grids before nearest-neighbor interpolation back to original points.Bins are 5 cm for Semantic3D and 3 cm for S3DIS.
- Geometric Partition: Geometric partitioning balances superpoint semantic homogeneity against discriminability using dataset-specific regularization strengths.The regularization strength is µ = 0.8 for Semantic3D and µ = 0.03 for S3DIS.
- PointNet: PointNet maps sampled superpoint features to 32-dimensional embeddings using shallow MLPs, max pooling, a diameter feature, and an XY spatial transformer.The model uses np = 128 and nminp = 40.
- Segmentation Network: The segmentation network uses ECC variants, a learned filter-generating MLP, 32-dimensional embeddings, and T = 10 iterations.ECC-VV is used for Semantic3D and ECC-MV for S3DIS.
- Training: Training uses Adam with learning-rate schedules, dataset-specific epoch counts, batch size 2, and gradient clipping within [−1, 1].Semantic3D is trained for 500 epochs and S3DIS for 250 epochs.
B. CRF-ECC
The CRF-ECC module adapts mean-field inference by replacing dense Gaussian pairwise potentials with learned graph message passing over superedge connectivity.
- The method adapts CRF-RNN mean-field inference to postprocess PointNet embeddings represented as unary potentials.
- The original dense CRF pairwise term combines Gaussian kernels applied to edge features with a label compatibility matrix.
- CRF-ECC replaces the original pairwise term with a filter-generating network that generalizes message passing and compatibility transforms.
- The method uses superedge connectivity instead of a complete graph and outputs marginal probability distributions after T = 10 iterations.
C. Extended Ablation Studies
Extended ablations show that orientation handling, geometric features, sampling, long-range context, input gating, partition regularization, and superpoint size all affect performance, while removing all edge features is especially harmful.
- Spatial Transformer Network: 4 mIoU points are gained from the Spatial Transformer Network, despite orientation-variant relationships remaining in superedges.
- Geometric Features: +4 mIoU results from explicitly using geometric features that are available from the partitioning step.
- Sampling Superpoints: −4 mIoU follows from sampling too few points, while sampling too many causes −2 mIoU through overfitting.Subsampling also lowers training memory requirements, and no subsampling is performed at test time.
- Long-range Context: Capping contextual distances at 1 m or 5 m worsens performance, showing that longer-range context is useful in the SPG.
- Input Gate: A proposed input gate gives GRU a +1 mIoU advantage over the comparison, while LSTM scores 3 mIoU higher than GRU without it.
- Regularization Strength µ: The system performs reasonably across a range of regularization strengths controlling superpoint homogeneity and discriminability.
- Superpoint Sizes: 93% of points belong to embedded superpoints, and 79% belong to superpoints that are subsampled.
- Superedge Features: Removing all superedge features produces a −22 mIoU result, whereas no individual feature is crucial and offset deviation contributes most among individual features.
D. Video Illustration
The paper provides a video illustrating its method and qualitative results on the S3DIS dataset.
- A video illustrates the method and qualitative results on the S3DIS dataset.The video is available at the linked URL.