Fast Point Transformer

• 0.14 seconds per scene is the average inference time, making Fast Point Transformer 129 times faster than Point Transformer on S3DIS semantic segmentation.Source: Figure 1 · p. 1
• Fast Point Transformer processes large-scale scenes with local self-attention in one shot, without searching for point-wise neighbors.Source: Figure 1 · p. 1

• Large-scale scene methods face a speed–accuracy challenge: cropping and stitching or approximating point coordinates can increase inference time or degrade output.Source: §1 · p. 1
• Centroid-aware voxelization and devoxelization preserve continuous-coordinate embeddings, reduce quantization artifacts, and support coherent dense predictions under rigid transformations.Source: §1 · p. 2
• Fast Point Transformer achieves competitive semantic-segmentation accuracy across voxel-hashing configurations and improves ScanNet detection mAP as a VoteNet backbone over point- and voxel-based baselines.Source: §1 · p. 2
• The proposed CScore evaluation shows that the model produces more coherent predictions under rigid transformations than previous voxel-based approaches.Source: §1 · p. 2
• The lightweight local self-attention module learns continuous positional information while reducing space complexity, and replaces convolutional layers in a U-shaped network without manual point grouping.Source: §1 · p. 2
• 129 times faster inference than Point Transformer, with a reasonable accuracy trade-off in 3D semantic segmentation on S3DIS.Source: §1 · pp. 1–2

Point-based methods

• Point-based methods progress from PointNet [27] and PointNet++ [28] to convolution and local attention, but often incur heavy large-scene overhead from k-nearest-neighbor search or k-d tree construction.KPConv [38] uses continuous-space kernel points and a k-d tree, while Mao et al. [23] use discretized kernels but convolve over every point, creating a large-scene bottleneck.Source: §2 · p. 2

Voxel-based methods

• Sparse convolution [6] [10] finds valid neighboring voxels in O(1), but voxelization can cause quantization artifacts and features inconsistent with voxel size.Mao et al. [24] use local and dilated attention to enlarge receptive fields in a voxel-based transformer.Source: §2 · p. 2

Hybrid methods

• Hybrid methods [22] [34] [45] combine fast voxel neighbor search with point-based fine-geometry features, but storing both feature types raises computation and memory costs.Point-based layers such as mini-PointNet are added to voxel-based methods to mitigate voxelization artifacts.Source: §2 · p. 2

Attention-based networks

• Because global self-attention is costly for 3D vision, recent models [11] [24] use local self-attention [3] [30] [39]; point-based models [11] use k-nearest-neighbor search to handle irregularity and improve performance.Local self-attention is also used by Guo et al. [11] and Zhao et al. to learn richer representations than fixed kernel-based methods [23] [38].Source: §2 · pp. 2–3

3.1. Overview

• The architecture processes point clouds through centroid-aware voxelization, lightweight self-attention, and centroid-aware devoxelization.Learnable positional embeddings encode point-to-centroid offsets during voxelization and help assign voxel features back to continuous points.Source: §3.1 · p. 3

3.2. Centroid-aware Voxel & Devoxelization

• Centroid-aware voxelization encodes each point’s offset from its voxel centroid before pooling, and devoxelization uses that encoding to recover point-level features.This differs from voxel methods that use barycentric interpolation or pool features without intra-voxel coordinates.Source: §3.2 · pp. 3–4

3.3. Lightweight Self-Attention

• Voxel hashing queries neighbors in O(N), while decomposing continuous centroid offsets reduces positional-encoding space from O(IKD) to O(ID + KD).The paper contrasts hashing with point-based k-nearest-neighbor search at O(N log N).Source: §3.3 · p. 4
• The LSA layer uses cosine similarity to handle sparse voxels and variable neighborhood sizes, and is reported to outperform sparse convolution on several tasks.The paper also reports more coherent representations under rigid transformations than sparse convolution.Source: §3.3 · pp. 4–5

3.4. Network Architecture

• LSA layers can replace convolution layers in sparse CNN architectures, and the paper implements semantic segmentation with a U-Net.ResNet and U-Net are given as examples of architectures that can be adapted.Source: §3.4 · p. 5

4.3. Consistency Test

• With 41 rigid transformations on ScanNet validation, Fast Point Transformer produces more coherent predictions than MinkowskiNet42† and relieves voxelization artifacts.The test uses 26 translations and 15 rotations around the gravity axis; CScore measures agreement between predictions for original and transformed points.Source: §4.3 · pp. 5–6, Table 1 · p. 6, Figure 4 · p. 6

4.4. 3D Semantic Segmentation

• Fast Point Transformer processes S3DIS scenes at least 83 times faster than point-based methods, exceeds MinkowskiNet42† by 1.4 mIoU points, and is 129 times faster than Point Transformer.The comparison measures per-scene inference latency; Fast Point Transformer uses a single feed-forward operation, while Point Transformer uses multiple inferences.Source: §4.4 · p. 6, Table 2 · p. 7
• On ScanNet, Fast Point Transformer exceeds MinkowskiNet42† in mIoU by 0.2, 3.3, and 5.4 points at voxel sizes of 2cm, 5cm, and 10cm, respectively.The authors interpret these results as evidence of greater robustness to quantization error.Source: §4.4 · p. 6, Table 3 · p. 7
• Fast Point Transformer maintains consistent accuracy with 71.5% fewer parameters than MinkowskiNet, and its 10cm model outperforms MinkowskiNet’s 5cm model.The authors attribute its resilience to parameter reduction to the lightweight self-attention layer’s dynamic kernel weights.Source: §4.4 · p. 7, Table 4 · p. 7
• Full positional encodings achieve the best mIoU; removing either encoding or centroid-aware voxelization or devoxelization degrades performance, showing their role in preserving continuous geometry.Using δ_abs additionally does not improve MinkowskiNet42†, whereas the self-attention mechanism is a more suitable way to use it than sparse convolution.Source: §4.4 · pp. 7–8, Table 5 · p. 8
• Cosine attention handles varying neighbor counts better than softmax, while adding query-key similarity does not improve the lightweight self-attention layer.The ablation uses ScanNet validation with a 10cm voxel size.Source: §4.4 · p. 8, Table 6 · p. 8
• Increasing the local window size degrades performance because learnable relative-position tokens are sparse in 3D data.The authors suggest concatenated positional encodings or a shared mapping layer as possible inductive biases to address this limitation.Source: §4.4 · p. 8, Table 7 · p. 8

4.5. 3D Object Detection

• VoteNet with a Fast Point Transformer backbone outperforms the detection baselines by a large margin on ScanNet validation.The detection network, including its voting module, is unchanged; the authors attribute the result to the backbone’s continuous positional encodings.Source: §4.5 · p. 8, Table 8 · p. 8

5. Conclusion

• 129 times faster inference than Point Transformer, with a reasonable accuracy trade-off in 3D semantic segmentation; Fast Point Transformer is competitive with the best voxel-based method [6].These results were demonstrated on large-scale 3D datasets.Source: §5 · p. 8
• The method was demonstrated on 3D semantic segmentation and 3D detection, while performance at small voxel sizes remains an area for improvement.The authors propose exploring architectures beyond U-shaped architectures [31], which were initially designed for convolutional layers.Source: §5 · p. 8
• The authors state that their code and data are going to be publicly available.Source: §5 · p. 8

A. Appendix

• The appendix adds details and results for the proposed Fast Point Transformer method.Source: A. Appendix · p. 9

Training details

• Segmentation models use batch size 8, with 100k iterations for ScanNet and 40k for S3DIS, using SGD and a warm-up/cosine learning-rate schedule.Momentum is 0.9, weight decay is 0.0001, and the learning rate decreases from 0.1 to 0.Source: §A.1 · p. 9

Latency evaluation

• Latency is measured at batch size 1 on a single NVIDIA GeForce RTX 3090 with CUDA 11.0, cuDNN 8.2.1, PyTorch 1.7.1, and MinkowskiEngine 0.5.4.The host CPU is an Intel Core i7-5930K at 3.50 GHz.Source: §A.1 · p. 9

Network architectures

• MinkowskiNet42 [6] and Fast Point Transformer are matched at 37.9M parameters; smaller models modify residual-block counts, with counts specified in Table A1.The same small-model residual-block configuration is applied to both architectures.Source: §A.1 · p. 9, Table A1 · p. 9

Color reconstruction

• Centroid-aware voxelization achieves a PSNR 1.27 higher than conventional voxelization for RGB color reconstruction on ScanNet validation.The comparison uses MinkowskiNet with the same settings, optimized by l2-difference between input and reconstructed colors.Source: Table A2 · p. 9, §A.2 · p. 9

Robustness to voxel size

• Fast Point Transformer is more robust than MinkowskiNet42† when inference voxel size increases from 4cm to 5cm on S3DIS.Both models were trained with 4cm voxels; the comparison measures the performance drop in mIoU.Source: §A.2 · p. 9, Table A3 · p. 9

Decomposition of positional encodings

• The proposed decomposition has O(ID + KD) space complexity and almost constant memory usage, unlike O(IKD) without decomposition, which grows with neighbor count.For local window size 7, the non-decomposed model ran out of memory on a single 24GB NVIDIA Geforce RTX 3090; LSA saves memory while preserving fast inference and nearly identical mIoU to exact approaches.Source: §A.3 · pp. 9–10, Table A4 · p. 10, Table A5 · p. 10

mIoU vs. model size

• At 2 cm voxels, the smaller Fast Point Transformer achieves 70.5% mIoU, while smaller MinkowskiNet achieves 68.6%.Both models show their largest performance drop at 2 cm; the authors hypothesize that fewer residual blocks reduce the receptive field too much for scene recognition.Source: §A.3 · pp. 10–11, Table A6 · p. 10

Additional speed and accuracy results

• At 4 cm, Fast Point Transformer outperforms MinkowskiNet42† by 0.4 mIoU with 4.7 times faster speed.The passage also reports a large latency gap between point-based methods [18] [27] [38] [42] and voxel hashing-based methods [6].Source: §A.3 · p. 11

A.4. Time Complexity Analysis

• Hash-table construction takes O(N) and inference takes O(M), using O(1) insertion and lookup per point or query.MinkowskiNet [6] and Fast Point Transformer use the same voxel-hashing neighbor-search process.Source: §A.4 · p. 11
• KPConv [38] takes O(N log N) to construct its tree and O(KM log N) for inference.The analysis uses KPConv's official code.Source: §A.4 · p. 11
• PointWeb uses brute-force neighbor search with inference complexity O(MNK).The analysis describes its algorithm as searching the k nearest neighbors by brute force.Source: §A.4 · p. 12
• PAConv [42] and Point Transformer require constant-time preparation and share an inference algorithm with complexity O(MN log K).PAConv is used as the representative method for their shared neighbor-search algorithm.Source: §A.4 · p. 12

A.5. Qualitative Results

• Figure A2 compares point-wise consistency scores for MinkowskiNet42† and Fast Point Transformer on ScanNet, with both models trained at 10cm voxel size.The section also identifies consistency scores as one of its qualitative results.Source: §A.5 · p. 12, Figure A2 · p. 13
• Figure A3 shows ScanNet semantic predictions from MinkowskiNet42† and Fast Point Transformer alongside the input point cloud and ground truth, using 10cm voxels for both models.Source: Figure A3 · p. 14
• Figure A4 compares ScanNet bounding boxes predicted by VoteNet [26] with MinkowskiNet and Fast Point Transformer backbones against ground truth.Source: Figure A4 · p. 14

A.6. Notations

• The notation defines M(i) as point indices in voxel i, N(i) as neighboring voxel indices, and output voxels as having centroids.Source: §A.6 · p. 15
• The centroid-to-point positional encoding is denoted e_n, and δ_enc denotes an encoding layer used in that encoding.Source: §A.6 · p. 15
• The notation uses o_n for an output point feature, a(·) for an attention operation, and g_i for a centroid-aware voxel feature.Source: §A.6 · p. 15