PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Point Transformer

Chunghyun Park, Yoonwoo Jeong, Minsu Cho, Jaesik Park

arXiv:2112.04702v2cs.CV

TL;DR

Processing large-scale 3D scenes remains difficult because existing cropping, stitching, and coordinate-approximation methods increase inference costs or degrade outputs. Fast Point Transformer addresses this with lightweight local self-attention, continuous positional encoding, and voxel hashing, achieving competitive accuracy across 3D segmentation and detection while delivering 129 times faster inference than Point Transformer for S3DIS segmentation.

  • Problem

    Large-scale 3D scene understanding requires both fast inference and high accuracy, but existing local or approximate approaches can increase inference time or degrade final predictions.

  • Method

    Fast Point Transformer combines lightweight local self-attention, continuous positional encoding, centroid-aware voxelization and devoxelization, and voxel hashing for large-scale 3D point clouds.

  • Results

    The approach is competitive with the best voxel-based method across 3D semantic segmentation and detection, while achieving 129 times faster inference than Point Transformer on S3DIS segmentation.

  • Takeaways & Limitations

    Fast Point Transformer provides a speed–accuracy trade-off for large-scale 3D scene understanding and produces more coherent predictions than previous voxel-based approaches.

Abstract

from arXiv · show

The recent success of neural networks enables a better interpretation of 3D point clouds, but processing a large-scale 3D scene remains a challenging problem. Most current approaches divide a large-scale scene into small regions and combine the local predictions together. However, this scheme inevitably involves additional stages for pre- and post-processing and may also degrade the final output due to predictions in a local perspective. This paper introduces Fast Point Transformer that consists of a new lightweight self-attention layer. Our approach encodes continuous 3D coordinates, and the voxel hashing-based architecture boosts computational efficiency. The proposed method is demonstrated with 3D semantic segmentation and 3D detection. The accuracy of our approach is competitive to the best voxel-based method, and our network achieves 129 times faster inference time than the state-of-the-art, Point Transformer, with a reasonable accuracy trade-off in 3D semantic segmentation on S3DIS dataset.

POSTECH GSAI & CSE

Fast Point Transformer processes large-scale scenes with local self-attention and one-shot inference. On S3DIS semantic segmentation, it averages 0.14 seconds per scene and is 129 times faster than Point Transformer.

  • 0.14 seconds per scene is the network’s average inference time on S3DIS semantic segmentation.
  • 129 times faster than Point Transformer is the reported inference-speed advantage on S3DIS semantic segmentation.
  • One-shot scene inference avoids searching for point-wise neighbors.

1. Introduction

The paper targets the difficulty of achieving fast, accurate understanding of large-scale 3D scenes without costly local-scene processing. Fast Point Transformer combines continuous positional encoding, lightweight local self-attention, and voxel hashing, showing competitive accuracy and substantially faster inference across segmentation and detection settings.

  • Motivation: Large-scale 3D scene understanding remains challenging because processing scenes as a whole requires extensive computation and memory.
  • Motivation: Cropping and stitching scenes or approximating point coordinates can increase inference time or degrade outputs through local or approximate predictions.
  • Contributions: Fast Point Transformer encodes continuous positional information using voxel hashing, centroid-aware voxelization and devoxelization, and lightweight local self-attention.
  • Results: The method shows competitive semantic-segmentation accuracy, better ScanNet detection mAP than listed point- or voxel-based baselines, and more coherent predictions than previous voxel-based approaches.
  • Results: 129 times faster inference than Point Transformer is reported for 3D semantic segmentation on S3DIS, with a reasonable accuracy trade-off.

2. Related Work

Related work spans point-based, voxel-based, hybrid, and attention-based approaches for 3D scene understanding. Fast Point Transformer is presented within the move toward local self-attention for computationally feasible point-cloud processing.

  • Point-based methods: PointNet and PointNet++ use MLP-based point processing, with PointNet++ adding hierarchical sampling strategies.
  • Fast Point Transformer: Figure 2 depicts an architecture that voxelizes input points, processes non-empty voxels and centroids, and produces output points and features.
  • Attention-based methods: Global self-attention is generally infeasible for 3D vision because of costly operations, motivating local self-attention methods.

3. Fast Point Transformer

Fast Point Transformer processes point clouds through centroid-aware voxelization, lightweight local self-attention, and centroid-aware devoxelization. Continuous positional encodings preserve point geometry while voxel hashing and decomposed encodings improve efficiency.

  • Overview: Fast Point Transformer uses centroid-aware voxelization, lightweight self-attention, and centroid-aware devoxelization as its three processing steps.Voxelization creates voxel coordinates, features, and centroids; attention updates voxel features; devoxelization returns point features.
  • Lightweight self-attention: Voxel hashing supports efficient neighborhood queries for local self-attention, with O(1) complexity for a single query in the described procedure.Point-based k-nearest-neighbor construction is described as O(N log N), whereas voxel hashing is presented as faster for large-scale point clouds.
  • Centroid-aware devoxelization: Centroid-aware devoxelization combines each output voxel feature with the corresponding point’s learnable positional embedding to produce point-level features.An MLP maps the combined voxel feature and positional encoding to the output point feature.
  • Centroid-aware voxelization: Centroid-aware voxelization encodes each point’s offset from its voxel centroid to reduce geometric information loss from quantization.The learnable encoding is concatenated with the raw point feature before aggregation into the voxel feature.
  • Lightweight self-attention: Decomposing centroid-relative positions separates continuous absolute offsets from discretized relative voxel positions, reducing positional-encoding space complexity from O(IKD) to O(ID + KD).The continuous term uses centroid-to-voxel offsets, while the discretized term uses relative voxel coordinates with at most K positions.
  • Lightweight self-attention: The LSA layer forms centroid-aware voxel features, computes attention with relative voxel positional encodings, and uses cosine similarity rather than softmax.The design is intended to handle sparse voxel neighborhoods and learn coherent representations under rigid transformations.

4. Experiments

Experiments evaluate Fast Point Transformer on large-scale 3D datasets for segmentation and detection, emphasizing consistency, accuracy, efficiency, and ablations. The method produces coherent predictions, competitive accuracy, and substantially faster inference than point-based baselines.

  • Datasets: S3DIS contains six areas and 271 room scenes; evaluation uses Area 5 while training uses the other splits without cropping scenes into small blocks.
  • Consistency Test: CScore measures prediction agreement under rigid transformations, averaging pointwise agreement across scenes, points, and transformations.The evaluation uses 41 transformations: 26 translations and 15 rotations around the gravity axis.
  • Consistency Test: Fast Point Transformer produces more coherent feature representations than MinkowskiNet42 and relieves quantization artifacts.The comparison uses both quantitative Table 1 results and qualitative Figure 4 visualizations.
  • 3D Semantic Segmentation: Fast Point Transformer processes large-scale scenes at least 83 times faster than point-based methods and 129 times faster than Point Transformer.Against MinkowskiNet42†, it gains 1.4 absolute percentage points in mean IoU with comparable speed.
  • Model Size: Fast Point Transformer remains accurate after parameter reduction, outperforming counterpart MinkowskiNet models with 71.5% fewer network parameters.The authors attribute this robustness to the lightweight self-attention layer and its dynamic kernel weights.
  • Ablation Study: Removing continuous positional information or either positional encoding degrades performance, supporting the role of centroid-aware voxelization and devoxelization.The ablations indicate that these components preserve continuous geometric information from the input point cloud.
  • Ablation Study: Cosine attention handles varying neighbor counts more effectively than softmax, while adding query-key similarity does not improve the LSA layer.
  • Ablation Study: Increasing the local window size degrades performance because learnable relative-position tokens encounter sparse 3D data.The authors suggest positional-encoding concatenation or a shared mapping layer as possible remedies.

5. Conclusion

The paper presents Fast Point Transformer for 3D semantic segmentation and detection, reporting competitive accuracy and substantially faster inference. It also identifies small-voxel performance and architectural design as areas for future improvement.

  • Fast Point Transformer is demonstrated on 3D semantic segmentation and 3D detection, with competitive accuracy to the best voxel-based method.
  • 129 times faster inference than Point Transformer is achieved with a reasonable accuracy trade-off on S3DIS semantic segmentation.
  • Small voxel sizes remain an area for improvement, and future work will explore architectures beyond convolution-oriented U-shaped designs.

A. Appendix

The appendix provides additional details and results for Fast Point Transformer.

  • The appendix supplies additional details and results for the proposed Fast Point Transformer.

A.1. Experimental Details

The appendix documents reproducible training, hardware, and architecture settings for the experiments. It specifies shared parameterization and residual-block configurations for MinkowskiNet and Fast Point Transformer.

  • Training Details: Training experiments use fixed random seeds and shared segmentation configurations across ScanNet and S3DIS, with SGD momentum and weight decay settings documented.
  • Experimental Environment: Latency measurements use a single NVIDIA Geforce RTX 3090 GPU and Intel Core i7-5930K CPU with specified CUDA, cuDNN, PyTorch, and MinkowskiEngine versions.
  • Network Architectures: MinkowskiNet42 and Fast Point Transformer are configured with similar parameter counts of 37.9M, while smaller models reduce residual blocks following MinkowskiNet’s official code.
  • Network Architectures: S1 through S16 denote tensor strides in the feature-map hierarchy, and Table A1 gives the residual-block counts used by both model families.

A.2. Analysis on Centroid-aware Voxelization

The centroid-aware voxelization preserves continuous intra-voxel positional information and improves color reconstruction over conventional voxelization. The method is also evaluated for robustness when inference uses a larger voxel size than training.

  • PSNR improves by 1.27 with centroid-aware voxelization over conventional voxelization in ScanNet color reconstruction.Both methods use the same MinkowskiNet setting and reconstruct RGB colors from input point clouds.
  • Fast Point Transformer is more robust to inference voxel size than MinkowskiNet42†.The conclusion is based on the relative performance drops reported in the voxel-size robustness experiment.
  • The robustness experiment compares mIoU after changing inference voxel size from 4cm during training to 5cm during inference.The comparison uses MinkowskiNet42† and Fast Point Transformer on S3DIS.

A.3. Additional Experimental Results

Additional experiments examine architecture parity, memory efficiency, model-size robustness, and segmentation performance across voxel-based and hybrid baselines. The results support lower memory use, fast inference, and competitive accuracy, with performance depending on receptive field and voxel size.

  • Architecture and memory: Both MinkowskiNet42 and Fast Point Transformer architectures use the same number of learnable parameters.The architectural comparison highlights the proposed lightweight self-attention layer in Fast Point Transformer.
  • Architecture and memory: O(ID + KD) decomposition yields almost constant memory usage, unlike O(IKD), which grows with the number of voxel centroids and neighbors.A local window size of 7 causes an out-of-memory error on a 24GB RTX 3090 without decomposition.
  • Architecture and memory: The LSA layer saves memory and preserves fast inference time while achieving almost identical mIoU to exact approaches.These results are reported in the sequential-computation versus decomposition comparison.
  • Model size: 70.5% mIoU is achieved by the smaller Fast Point Transformer at 2cm voxel size, compared with 68.6% for smaller MinkowskiNet.Both methods show their largest performance drop at 2cm, which the authors hypothesize relates to a reduced receptive field.

A.4. Time Complexity Analysis

The analysis contrasts neighbor-search complexity across point-based and voxel hashing-based methods. Voxel hashing uses linear preparation and constant-time lookup per query, while point-based alternatives require more expensive search procedures or preparation.

  • Voxel hashing: MinkowskiNet and Fast Point Transformer use the same voxel-hashing neighbor-search process.The analysis treats Fast Point Transformer as representative for this process.
  • Voxel hashing: Voxel hashing constructs a hash table in O(N) time and performs inference lookup in O(M) time.Each point is inserted with O(1) cost during construction, and each query uses an O(1) hash lookup.
  • Point-based search: KPConv requires k-d-tree preparation with O(N log N) complexity and inference complexity O(KM log N).The preparation inserts N training points, while inference searches K neighbors for M query points.
  • Point-based search: PointWeb uses brute-force neighbor search with O(MNK) inference complexity.The algorithm scans N training points for each of M queries while maintaining K nearest neighbors.
  • Point-based search: PAConv and Point Transformer require no neighbor-search preparation, but their inference procedure is represented with O(MN log K) complexity.The analysis sets preparation time to constant and uses PAConv as the representative method.

A.5. Qualitative Results

Qualitative ScanNet results compare consistency scores, semantic labels, and object-detection boxes for Fast Point Transformer against MinkowskiNet-based alternatives. The figures use matched voxel-size settings where specified and include ground truth references.

  • Qualitative comparison: The paper reports that Fast Point Transformer produces more accurate semantic labels and bounding boxes qualitatively than the compared MinkowskiNet-based results.This comparison is stated alongside the consistency results.
  • Consistency scores: CScore visualizations compare MinkowskiNet42† and Fast Point Transformer using a 10cm training voxel size.The figure presents the input point cloud alongside each model’s point-wise consistency scores.
  • Semantic segmentation: Semantic-segmentation visualizations arrange input, MinkowskiNet42† predictions, Fast Point Transformer predictions, and ground truth in four columns.Both models are trained with voxel size 10cm.
  • 3D object detection: Object-detection visualizations compare VoteNet with MinkowskiNet and Fast Point Transformer backbones against ground-truth bounding boxes.The three columns show MinkowskiNet-backbone predictions, Fast Point Transformer-backbone predictions, and ground truth.
Loading 2112.04702v2…