Source-linked AI summary
OctFormer: Octree-based Transformers for 3D Point Clouds
Peng-Shuai Wang
TL;DR
Point cloud transformers are difficult to scale because global attention has quadratic complexity and conventional local windows contain highly variable numbers of points. OctFormer uses octree ordering to form equal-sized point groups with flexible shapes, adds dilated attention, and reports faster computation plus state-of-the-art benchmark performance.
Problem
Quadratic transformer attention and highly variable point counts in local windows hinder efficient processing of large-scale point clouds.
Method
OctFormer uses octree-sorted features to create irregular local windows with fixed point counts and introduces dilated octree attention for a larger receptive field.
Results
OctFormer achieves state-of-the-art performance across several 3D segmentation and detection benchmarks and runs over 17 times faster than other methods at 200k elements.
Takeaways & Limitations
OctFormer provides a general, scalable backbone for 3D point cloud segmentation and object detection with linear computational complexity.
Takeaways & Limitations
OctFormer is prone to overfit on small-scale datasets and underperforms PointMLP on ModelNet40 classification without voting.
Abstract
from arXiv · showhide
We propose octree-based transformers, named OctFormer, for 3D point cloud learning. OctFormer can not only serve as a general and effective backbone for 3D point cloud segmentation and object detection but also have linear complexity and is scalable for large-scale point clouds. The key challenge in applying transformers to point clouds is reducing the quadratic, thus overwhelming, computation complexity of attentions. To combat this issue, several works divide point clouds into non-overlapping windows and constrain attentions in each local window. However, the point number in each window varies greatly, impeding the efficient execution on GPU. Observing that attentions are robust to the shapes of local windows, we propose a novel octree attention, which leverages sorted shuffled keys of octrees to partition point clouds into local windows containing a fixed number of points while permitting shapes of windows to change freely. And we also introduce dilated octree attention to expand the receptive field further. Our octree attention can be implemented in 10 lines of code with open-sourced libraries and runs 17 times faster than other point cloud attentions when the point number exceeds 200k. Built upon the octree attention, OctFormer can be easily scaled up and achieves state-of-the-art performances on a series of 3D segmentation and detection benchmarks, surpassing previous sparse-voxel-based CNNs and point cloud transformers in terms of both efficiency and effectiveness. Notably, on the challenging ScanNet200 dataset, OctFormer outperforms sparse-voxel-based CNNs by 7.3 in mIoU. Our code and trained models are available at https://wang-ps.github.io/octformer.
1 INTRODUCTION
OctFormer addresses the quadratic complexity and GPU inefficiency of point cloud transformers with octree-based attention that uses equal-sized point groups and scalable receptive fields. It achieves faster computation and state-of-the-art segmentation and detection results across several benchmarks.
- Motivation: Point cloud transformers face quadratic attention complexity, limiting their applicability to large-scale point clouds.Global attention is especially problematic on scene-scale datasets, while downsampled feature maps can weaken network capability and reduce performance.
- Method: OctFormer partitions octree-ordered point features into irregular local windows with a fixed number of points, enabling standard and parallelizable attention operations.Sorted shuffled octree keys place spatially close nodes contiguously, and padding makes the point count divisible by the chosen group size.
- Efficiency: 17 times faster is the reported speed of one OctFormer transformer block than previous point transformer blocks at 200k involved elements.The attention can be implemented in 10 lines using open-sourced libraries and standard operators.
- Results: OctFormer achieves the best validation performance on ScanNet, SUN RGB-D, and ScanNet200 benchmarks, surpassing sparse-voxel CNNs and point cloud transformers.On ScanNet200 segmentation, its mIoU exceeds MinkowskiNet by 7.3 and LGround by 5.4.
2 RELATED WORK
Prior 3D learning methods use voxel, point, or transformer architectures with differing efficiency and neighborhood-processing trade-offs. OctFormer contributes a hierarchical transformer whose octree attention uses variable window shapes while maintaining equal point counts for efficient computation.
- Voxel-based CNNs: Voxel-based CNNs improve efficiency over full-voxel CNNs by restricting operations to non-empty sparse voxels and using octrees or hash tables.Full-voxel CNNs are limited by cubic computational and memory costs with voxel resolution.
- Point-based Networks: Point-based networks process raw unordered point clouds using permutation-invariant operations, continuous convolution kernels, or adaptive weights.They commonly use farthest point or grid-based sampling to progressively downsample point clouds.
- OctFormer: OctFormer avoids expensive k-nearest-neighbor search and farthest point sampling while retaining a hierarchical network architecture.Its octree attention is designed for efficient point cloud feature processing.
- OctFormer: OctFormer’s key innovation is using variable local-window shapes while maintaining a fixed number of points in each window.This design targets efficient attention without requiring regular cubic windows.
- Point Cloud Transformers: Point Transformer applies vector attention to local neighborhoods, but its computation remains costly because pooling uses farthest point sampling.Point Transformer V2 replaces that sampling with grid-based sampling and adds grouped vector attention and positional encoding multipliers.
- Point Cloud Transformers: Window-based point cloud transformers restrict attention to non-overlapping windows, but differing point counts complicate batching and GPU execution.SST and SWFormer group windows with similar point counts, whereas other designs use different strategies to address imbalance.
3 OCTREE-BASED TRANSFORMERS
OctFormer combines hierarchical octree processing with attention restricted to fixed-size point groups, preserving linear scaling while allowing irregular window shapes and expanded receptive fields. Its design uses sorted octree keys for efficient partitioning, conditional positional encoding, and parallel attention execution.
- Architecture: OctFormer converts normalized point clouds into octrees, embeds leaf features, and processes them through hierarchical stages of attention blocks and downsampling modules.Each stage reduces spatial resolution and generally doubles feature channels; a lightweight FPN supports segmentation and detection.
- Octree Attention: O(N^2) attention is restricted to local windows containing K points, reducing computation to linear complexity in N.Conventional cubic windows can contain highly variable point counts, such as 48 on average versus 343 maximum on ScanNet with window size 7.
- Octree Attention: Sorted shuffled keys place octree subtrees contiguously, enabling tensor reshaping and transposing to partition non-empty-node features into fixed-size irregular windows.Features are zero-padded until the total count is divisible by K, typically 32, and padded elements are masked during parallel attention.
- Dilated Attention: Dilated octree attention pads and reshapes features using dilation D, with D set to 1 or 4, to enlarge the receptive field beyond local windows.Window attention otherwise limits information propagation between different windows.
- Positional Encoding: Conditional positional encoding applies octree-based depthwise convolution and batch normalization before attention, improving performance with fewer parameters than relative positional bias.The paper reports this improvement in an ablation study.
- Efficiency: Octree attention can be implemented in 10 lines using open-sourced libraries and PyTorch multi-head attention.The implementation applies attention to windows in parallel while masking padded elements.
4 EXPERIMENTS
Experiments show that OctFormer is effective across semantic segmentation and object detection while remaining efficient, scalable, and competitive with or superior to prior methods. Ablations further identify favorable effects from larger models, finer voxels, larger attention windows, dilation, data augmentation, and conditional positional encoding.
- Evaluation setup: OctFormer is evaluated on large-scale 3D semantic segmentation and object detection tasks, including ScanNet, ScanNet200, and SUN RGB-D.The experiments use ScanNet and ScanNet200 for segmentation and adapt FCAF3D for SUN RGB-D detection.
- Semantic segmentation: 7.3 higher mIoU than MinkowskiNet and 5.4 higher than LGround are achieved on ScanNet200, despite OctFormer training from scratch without additional data.OctFormer is also consistently better across the Head, Common, and Tail category groups.
- Model scaling: 74.1 mIoU with 18M parameters is achieved by OctFormer-Small on ScanNet validation, rising to 75.2 with voting and surpassing larger sparse-voxel CNNs.OctFormer variants scale well as model size increases, with validation mIoU and training loss improving steadily.
- Ablation studies: 0.9 mIoU improvement is obtained with 1cm rather than 2cm voxels, while larger attention point counts and dilation improve capacity-related measures.mIoU improves until dilation reaches 4; data augmentation reduces overfitting, and removing conditional positional encoding decreases mIoU by 8.0.
- Object detection: Best mAP@0.5 is achieved on SUN RGB-D, with average performance remaining 0.7 higher than CAGroup3D after replacing the FCAF3D backbone.The comparison isolates backbone advantages because CAGroup3D primarily changes the detection head.
5 CONCLUSION
OctFormer is presented as a general, efficient backbone for 3D point-cloud understanding, centered on octree attention and its dilated variant. The paper identifies limitations involving small datasets, positional encoding flexibility, cross-attention, and untested 3D generation applications.
- Conclusion: OctFormer provides a general backbone for 3D point-cloud understanding, with octree attention and a dilated variant as its core components.The paper reports efficiency and state-of-the-art performance on semantic segmentation and 3D object detection benchmarks.
- Conclusion: The paper acknowledges that future methods will eventually surpass OctFormer on performance leaderboards, while emphasizing its simplified octree attention and unified network design.The authors associate these design properties with broader accessibility and future large-scale 3D-model pretraining possibilities.
- Limitations and Future Work: The authors expect future work to explore unsupervised pretraining for small-scale datasets, alternative positional encodings, and extensions from self-attention to cross-attention.These directions address overfitting, flexibility constraints from CPE, and the absence of octree cross-attention.
- Limitations and Future Work: OctFormer is focused on point-cloud understanding, leaving applications to cross-modal 3D content creation as an open direction.Suggested applications include 3D shape generation conditioned on images, sketches, or text.