PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient 3D Semantic Segmentation with Superpoint Transformer

Damien Robert, Hugo Raguet, Loic Landrieu

arXiv:2306.08045v2cs.CV

TL;DR

Large-scale 3D semantic segmentation requires models that can handle enormous point clouds without the computational burden of conventional transformers or costly superpoint preprocessing. SPT combines fast hierarchical superpoint construction with multi-scale sparse self-attention, achieving strong results across three benchmarks while using far fewer resources. Its compact design supports practical training and deployment, though partition errors can affect classification at ambiguous object boundaries.

  • Problem

    3D segmentation models often require high computation and memory, while existing superpoint methods can incur costly preprocessing and limited interaction range.

  • Method

    SPT uses an efficient hierarchical superpoint partition and sparse self-attention to model relationships between superpoints at multiple scales.

  • Results

    SPT achieves state-of-the-art or near-state-of-the-art performance on S3DIS, KITTI-360, and DALES while using 212k parameters and substantially fewer training resources.

  • Takeaways & Limitations

    Small, tailored models such as SPT can provide a flexible and sustainable alternative for large-scale 3D learning, including applications with limited computing resources.

  • Takeaways & Limitations

    Partition errors at boundaries without obvious geometric discontinuities can lead to classification errors, while learned partition features would increase preprocessing time.

Abstract

from arXiv · show

We introduce a novel superpoint-based transformer architecture for efficient semantic segmentation of large-scale 3D scenes. Our method incorporates a fast algorithm to partition point clouds into a hierarchical superpoint structure, which makes our preprocessing 7 times faster than existing superpoint-based approaches. Additionally, we leverage a self-attention mechanism to capture the relationships between superpoints at multiple scales, leading to state-of-the-art performance on three challenging benchmark datasets: S3DIS (76.0% mIoU 6-fold validation), KITTI-360 (63.5% on Val), and DALES (79.6%). With only 212k parameters, our approach is up to 200 times more compact than other state-of-the-art models while maintaining similar performance. Furthermore, our model can be trained on a single GPU in 3 hours for a fold of the S3DIS dataset, which is 7x to 70x fewer GPU-hours than the best-performing methods. Our code and models are accessible at github.com/drprojects/superpoint_transformer.

1. Introduction

Large-scale 3D segmentation models face high resource demands and struggle to adapt computation to local scene complexity. SPT addresses these issues with hierarchical superpoints and sparse self-attention, combining efficient preprocessing with strong benchmark performance and compact models.

  • 1. Introduction: Existing vision transformers allocate computation uniformly through grids or samplings, increasing memory use, limiting point capacity, and hindering long-range interactions.Superpoint methods adapt partitions to local complexity but can be constrained by costly preprocessing and local graph convolutions.
  • 1. Introduction: SPT partitions point clouds hierarchically according to local properties and uses sparse self-attention to model superpoint relationships across multiple scales.Segmentation is formulated as classifying a small number of superpoints rather than individual points, enabling simultaneous processing of millions of points without sliding windows.
  • 1. Introduction: More than 7 times faster superpoint computation addresses the costly preprocessing that limits existing superpoint-based methods.The preprocessing time is reported as comparable to or faster than standard approaches.
  • 1. Introduction: SPT reaches state-of-the-art or near-state-of-the-art performance across S3DIS, KITTI-360, and DALES while using substantially fewer resources.The benchmarks cover indoor scanning, outdoor mobile acquisition, and city-scale aerial LiDAR.
  • 1. Introduction: 212k parameters and 70 times fewer GPU-hours than Stratified Transformer make SPT markedly more resource-efficient than recent models.SPT also represents a 200-fold parameter reduction compared with other state-of-the-art models such as PointNeXt.

2. Related Work

Prior 3D learning methods trade off accuracy, range, adaptability, and efficiency through grids, neighborhoods, or graph-based superpoints. SPT builds on these approaches with a multi-scale hierarchical partition that adapts to local geometry and supports broader interactions efficiently.

  • 2. Related Work: 3D vision transformers model long-range interactions but their K-nearest or voxel neighborhoods increase memory consumption and hinder large-scene processing.These neighborhood choices also limit the use of global context cues.
  • 2. Related Work: Superpoint-based methods reduce dense point clouds to compact representations, but earlier approaches can suffer from slow preprocessing and limited expressivity or range.SuperPointGraph, for example, operates on a single partition and uses graph convolutions.
  • 2. Related Work: SPT uses a multi-scale hierarchical structure adapted to local geometry rather than fixed-grid octrees, producing partitions that more closely conform to semantic boundaries.This structure supports modeling interactions between objects or object parts.
  • 2. Related Work: Because real-world scans can contain hundreds of millions of points, efficient 3D learning leverages sampling, sparse representations, and local similarity to reduce memory or input size.Superpoint-based methods can achieve input reductions of several orders of magnitude.

3. Method

SPT builds a multi-scale hierarchical superpoint partition and uses superpoint graphs with transformer-based attention to classify fine-scale superpoints efficiently.

  • Hierarchical partition: SPT partitions a point cloud into geometrically homogeneous superpoints at multiple scales and constructs adjacency graphs at each partition level.The hierarchy is formed by recursively partitioning component features, while adjacency links connect nearby components at the same level.
  • General architecture: The hierarchical partition adapts the architecture’s resolution levels, replacing grid, point, or graph subsampling in a U-Net-like encoder-decoder.Decoder stages pass information from coarser to finer partitions, while encoder stages pool features from child superpoints.
  • Prediction: SPT predicts labels for superpoints at the finest classified partition rather than individual 3D points, avoiding communication between the original points.The architecture classifies level P1 superpoints and uses the hierarchical decoder to produce predictions without point-level attention.
  • Self-attention: The encoder and decoder use attention-based modules to propagate contextual information between neighboring superpoints at each partition level.Attention operates on key, query, and value vectors associated with superpoints and incorporates learned adjacency features.
  • Adjacency encoding: The method also encodes superpoint adjacency using relative position, paired-point position, principal directions, and size or shape ratios.These features capture relationships that are not determined by distance alone between superpoints.

4. Experiments

SPT is evaluated on three large-scale 3D datasets for accuracy, efficiency, and component contributions. It achieves competitive or superior performance with compact models, fast preprocessing and training, while ablations identify the importance of superpoint relationships, hierarchy, and handcrafted features.

  • Performance Evaluation: SPT performs at or above state-of-the-art on two of three datasets despite using substantially fewer parameters.It beats PointNeXt-XL on S3DIS with 196× fewer parameters, outperforms MinkowskiNet and DeepViewAgg on KITTI-360, and exceeds ConvPoint by more than 12 points on DALES with over 21× fewer parameters.
  • Performance Evaluation: 26k-parameter SPT-nano matches KPConv and MinkowskiNet, extending the method’s compactness beyond the full model.SPT also improves over other superpoint-based methods by 7–14 points across datasets.
  • Preprocessing Speed: 12.4 minutes, 117 minutes, and 148 minutes are sufficient to preprocess S3DIS, KITTI-360, and DALES, respectively, using a 48-core CPU.An 8-core workstation preprocesses S3DIS in 26.6 minutes, and the implementation is 7× faster than SuperPoint Graph’s preprocessing.
  • Training Speed: SPT trains faster than non-superpoint methods while attaining similar performance, whereas Superpoint Graph saturates 6.0 mIoU points below SPT.The authors report that SPT performs on par or better than complex models with up to two orders of magnitude more parameters and longer training times.
  • Ablation Study: Removing handcrafted features worsens performance on all datasets, while removing adjacency encoding causes a 6.3-point S3DIS drop.Pruning the 50% longest edges also reduces performance, indicating that long-range superpoint relationships contribute to the model’s results.
  • Ablation Study: Using only one partition level causes significant losses across datasets, while superpoint dropout improves S3DIS and KITTI-360 by over 2.5 points.Using 1.5× or 3× fewer superpoints reduces performance by 2.2 or 4.7 mIoU points but lowers training time to 2.4 or 1.6 hours; the partition remains more than 20 points below the oracle.

5. Conclusion

The Superpoint Transformer combines superpoints and transformers for large-point-cloud segmentation, achieving state-of-the-art results with reduced training time, inference time, and model size.

  • The approach combines superpoints and transformers to achieve state-of-the-art segmentation with reduced training time, inference time, and model size.

Appendix

The appendix documents supporting materials, including visualization, source code, limitations, feature analysis, superpoint-graph construction, partitioning, and hyperparameter guidance.

  • The appendix covers visualization, source code, limitations, handcrafted features, superpoint-graph construction, partitioning, and hyperparameter selection.

A-1. Interactive Visualization

The project includes an interactive browser-based visualization tool for exploring S3DIS, KITTI-360, and DALES samples from different viewpoints and with different point attributes.

  • The visualization tool generates browser-compatible HTML files for viewing S3DIS, KITTI-360, and DALES samples from any angle.It supports different point attributes and was used to design and validate the model.

A-2. Source Code

The project publicly releases source code with reproduction instructions, pretrained models, notebooks, and an implementation based on PyTorch and related libraries.

  • The public repository provides installation guidance, commands to reproduce results, pretrained models, and ready-to-use notebooks.
  • The implementation uses PyTorch, PyTorch Geometric, PyTorch Lightning, and Hydra.

A-3. Limitations

The method’s efficiency comes with limitations tied to compressed superpoint representations, fixed partitions, and superpoint-level prediction. These constraints affect scaling, boundary recovery, end-to-end learning, and downstream point-level tasks.

  • Overfitting and Scaling: Superpoint compression can cause overfitting in larger SPT configurations unless training sets are sufficiently large, diverse, and complex.S3DIS compresses 274m points into fewer than 1.25m graph nodes.
  • Errors in the Partition: Partition errors at ambiguous boundaries, such as curbs versus roads, cannot be corrected and may cause classification errors.Replacing handcrafted partition features with learned features could improve this, but would substantially increase preprocessing time.
  • Learning Through the Partition: End-to-end learning of point and adjacency features remains open because differentiating through hard superpoint assignments is difficult and repeated partitioning could negate efficiency.The proposed direction would also reintroduce individual-point processing during training.
  • Predictions: Superpoint-level predictions may limit maximum segmentation performance, while an upsampling layer could reduce the method’s efficiency.The oracle ablation suggests improvements should primarily target superpoint classification rather than partitioning.
  • Predictions: Superpoint features may limit downstream tasks requiring 3D point features, including surface reconstruction and panoptic segmentation.The paper suggests exploring adaptations that operate at the superpoint level.

A-4. Handcrafted Features

The method uses handcrafted radiometric, geometric, spatial, and adjacency features computed during preprocessing to construct and represent hierarchical superpoints. These descriptors encode local properties and relationships while keeping preprocessing efficient.

  • Feature Computation: Handcrafted features are computed once during preprocessing, and optimized implementation reduces this step to a few minutes.They support both hierarchical partition construction and superpoint representation.
  • Visualization: The interactive visualization displays point clouds by position, semantic labels, geometric features, radiometry, predictions, and partitions.Figure A-2 specifically illustrates the computed local geometric descriptors and distance to ground.
  • Point Features: Point features combine normalized radiometry with PCA-based local geometry and elevation, while training also uses normalized position.Radiometry includes RGB or intensity; geometric descriptors use 50-nearest neighbors.
  • Adjacency Features: Each superpoint-graph edge uses 18 adjacency features describing interfaces, ratios, and relative pose between neighboring superpoints.Interface features use a faster approximate gap heuristic instead of full Delaunay triangulation.
  • Adjacency Features: Adjacency descriptors are mapped to key, query, and value vectors for every encoder and decoder level, with oriented graph edges because some features are asymmetric.Mean-offset and ratio features induce edge orientation.

A-5. Influence of Handcrafted Features

Ablations show that radiometric and adjacency features are broadly important, while geometric features matter especially for lower-resolution or noisier outdoor datasets. Using larger multi-room S3DIS samples also improves performance through longer-range context.

  • Point Features: Removing geometric features causes a 4.1 mIoU drop on KITTI-360, compared with 0.7 mIoU on S3DIS.The authors associate the stronger outdoor benefit with lower resolution and higher noise.
  • Adjacency Features: Removing all adjacency features causes a 3.0 to 6.3 mIoU drop across datasets, highlighting their role in superpoint contextualization.Pose features are particularly useful for S3DIS adjacency relationships.
  • S3DIS Room Partition: Reconstructing each S3DIS floor from absolute room positions increases performance by 3.8 points by enabling large multi-room samples.Room-normalized coordinates can overfit layouts that differ from the room-based partition.

A-6. Superpoint-Graphs Computation

The method builds multilevel superpoint graphs from feature-based hierarchical partitions and computes adjacency efficiently with GPU heuristics. Its design balances partition coarseness, contextual modeling, and resource use, with ablations supporting multilevel supervision and compact representations.

  • Graph Computation: Adjacent superpoints are identified with an approximate gap heuristic that restricts costly computations using centroid distances and radius bounds.All superpoint-graph computation steps are implemented on the GPU.
  • Hierarchical Partition: The hierarchical partition repeatedly aggregates increasingly coarse, geometrically homogeneous superpoints using regularized graph-based piecewise-constant approximations.The initial partition uses point features and later levels operate on coarser component graphs.
  • Hierarchical Partition: The partition coarseness uses regularization strengths, with the experiments choosing |P0| / |P1| approximately 30 as an efficiency–precision compromise.Finer partitions can improve results but increase training time and memory.
  • Supervision: Adding dist-Pi supervision consistently improves performance, while supervising P1 with label distributions instead of dominant labels reduces performance.The results support treating P1 superpoints as sufficiently pure for dominant-label supervision.
  • Detailed Results: SPT performs close to state-of-the-art across datasets while remaining smaller and faster to train, with spatially regular predictions and long-range interaction modeling.The reported class-wise analysis notes remaining room for improvement on smaller objects.
Loading 2306.08045v2…