PaperlayerView exact revision on arXiv

Source-linked AI summary

Point Transformer V2: Grouped Vector Attention and Partition-based Pooling

Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, Hengshuang Zhao

arXiv:2210.05666v2cs.CV

TL;DR

PTv1's vector attention and pooling mechanisms have limitations in parameter efficiency, positional encoding, and spatial alignment. PTv2 addresses these issues with grouped vector attention, strengthened position encoding, and partition-based pooling, achieving state-of-the-art performance on several 3D point-cloud benchmarks.

  • Problem

    PTv1's vector-attention weight encoding grows drastically with depth and channel count, while prior position encoding and sampling-based pooling do not fully exploit 3D geometry or provide spatially aligned aggregation.

  • Method

    PTv2 combines grouped vector attention with a grouped weight encoding layer, an additional position encoding multiplier, and partition-based pooling.

  • Results

    PTv2 achieves state-of-the-art performance on point-cloud classification and semantic segmentation benchmarks.

  • Takeaways & Limitations

    PTv2 provides a powerful and efficient transformer-based backbone for 3D point-cloud understanding.

Abstract

from arXiv · show

As a pioneering work exploring transformer architecture for 3D point cloud understanding, Point Transformer achieves impressive results on multiple highly competitive benchmarks. In this work, we analyze the limitations of the Point Transformer and propose our powerful and efficient Point Transformer V2 model with novel designs that overcome the limitations of previous work. In particular, we first propose group vector attention, which is more effective than the previous version of vector attention. Inheriting the advantages of both learnable weight encoding and multi-head attention, we present a highly effective implementation of grouped vector attention with a novel grouped weight encoding layer. We also strengthen the position information for attention by an additional position encoding multiplier. Furthermore, we design novel and lightweight partition-based pooling methods which enable better spatial alignment and more efficient sampling. Extensive experiments show that our model achieves better performance than its predecessor and achieves state-of-the-art on several challenging 3D point cloud understanding benchmarks, including 3D point cloud segmentation on ScanNet v2 and S3DIS and 3D point cloud classification on ModelNet40. Our code will be available at https://github.com/Gofinge/PointTransformerV2.

1 Introduction

PTv2 addresses limitations of PTv1 with grouped vector attention, strengthened position encoding, and partition-based pooling. Experiments report improved performance over predecessor methods and state-of-the-art results across 3D understanding tasks.

  • 1 Introduction: PTv2 is proposed as an efficient backbone that analyzes and overcomes limitations of Point Transformer V1 through several architectural improvements.The paper identifies grouped vector attention, improved position encoding, and partition-based pooling as its principal novel designs.
  • 1 Introduction: PTv2 introduces grouped vector attention with shared group weights to improve efficiency while retaining advantages of vector and multi-head attention.The design addresses the drastic parameter growth of PTv1's vector-attention weight encoding as depth and channel count increase.
  • 1 Introduction: An additional position encoding multiplier strengthens geometric positional relationships for 3D point-cloud reasoning.The paper motivates this design by emphasizing that positional relationships among 3D points are more critical than those among 2D pixels.
  • 1 Introduction: Partition-based pooling directly fuses points within non-overlapping uniform-grid partitions for more efficient and spatially better-aligned aggregation.This replaces the combination of sampling and neighbor querying used by previous pooling approaches.
  • 1 Introduction: PTv2 outperforms predecessor works and achieves state-of-the-art performance on various 3D understanding tasks.The authors report extensive analysis and controlled experiments validating the proposed designs.

2 Related Works

Point-cloud transformers apply attention to irregular 3D data, with PTv1 using local neighborhood attention to reduce the costs of global attention. Figure 1 contrasts PTv1's mechanisms with PTv2's grouped attention and partition-based pooling.

  • 2 Related Works: Point-cloud understanding methods include projection-based, voxel-based, and point-based networks for processing irregular 3D inputs.Projection-based methods map points to image planes, while voxel-based methods transform them into regular voxel representations.
  • 2 Related Works: Point-cloud transformers emerged alongside vision transformers, with PCT applying global attention and PTv1 applying local attention between neighboring points.The passage associates PCT's global attention with memory and computational limitations, while PTv1 alleviates the memory problem through local attention.
  • 2 Related Works: Figure 1 compares PTv1 and PTv2 across attention, position encoding, pooling, and unpooling mechanisms.PTv1 uses vector attention with position encoding and sampling/interpolation-based pooling, whereas PTv2 uses grouped vector attention with improved position encoding and partition-based pooling/unpooling.

3 Point Transformer V2

Point Transformer V2 revisits PTv1’s local attention, vector-attention parameter growth, position encoding, and pooling limitations. It introduces grouped vector attention, strengthened positional encoding, and partition-based pooling to improve efficiency and spatial alignment.

  • Overview: PTv2 revisits vector self-attention and introduces improved grouped attention, position encoding, and pooling modules.The proposed architecture is presented as a set of improved modules upon PTv1.
  • Local attention: Local attention restricts each point’s computation to a reference subset because global attention over large 3D scenes is computationally heavy.PTv1 uses k nearest neighbors as the reference set, and experiments report neighborhood attention as more effective than shifted-grid attention.
  • Vector attention: Vector attention encodes query-key relations into channel-wise weights that re-weight value features before aggregation.The relation function can be subtraction, and the learnable weight encoding may be implemented with an MLP.
  • Grouped vector attention: PTv1’s weight-encoding parameters grow drastically with depth and channel count, restricting efficiency, generalization, and model depth.This motivates replacing full channel-wise encoding with grouped vector attention.
  • Grouped vector attention: Grouped vector attention divides value channels into groups, shares one scalar weight within each group, and reduces weight-encoding parameters.GVA uses g output channels instead of c, while its grouped weight encoding can include grouped linear projection, normalization, activation, and a fully connected layer for inter-group exchange.
  • Position encoding and pooling: PTv2 strengthens positional relations by multiplying the relation vector with a relative-position encoding, while partition-based pooling addresses misalignment in sampling-based pooling.The multiplier uses a learned function of relative positions, and prior pooling combines sampling with neighbor queries that are time-consuming and spatially misaligned.

4 Experiments

PTv2 is evaluated on semantic segmentation, shape classification, ablations, and model complexity. Across these experiments, it improves performance while its grouped attention and grid-based pooling also improve efficiency.

  • Semantic Segmentation: PTv2 outperforms prior methods on all reported semantic-segmentation metrics and exceeds PTv1 by 4.8% mIoU on ScanNet v2 validation.The evaluation covers ScanNet v2 and S3DIS.
  • Shape Classification: PTv2 achieves state-of-the-art performance on ModelNet40 shape classification.The reported metrics are class-average accuracy and overall accuracy on the test set.
  • Ablation Study: Grouped vector attention consistently outperforms multi-head self-attention with both shifted-grid and neighborhood attention.Neighborhood attention also performs significantly better than shifted-grid attention for non-uniform point clouds.
  • Ablation Study: Adding PTv2 components raises the ScanNet v2 validation baseline from 72.3% to 75.4% mIOU, with grouped vector attention, position encoding multiplier, and partition-based pooling evaluated incrementally.The original PTv1 baseline is 70.6%, while structural and data-processing adjustments raise the comparison baseline to 72.3%.
  • Model Complexity and Latency: Partition-based pooling achieves higher mIoU than sampling-based pooling, and grid pooling is faster while maintaining higher mIoUs than FPS-kNN and Grid-kNN.Larger grid sizes can reduce points at each stage and save memory without significantly affecting overall performance.
  • Model Complexity and Latency: Grouped vector attention improves performance and slightly reduces execution time, while grid pooling further speeds the network and improves generalization.The position encoding multiplier increases model parameters but improves performance; PTv2 remains lightweight compared with MinkUNet42's 37.9M parameters.

5 Conclusion

PTv2 is presented as a powerful and efficient transformer backbone for 3D point cloud understanding, with state-of-the-art performance on classification and semantic segmentation benchmarks.

  • PTv2 introduces grouped vector attention, improved position encoding, and partition-based pooling as improvements over PTv1.
  • PTv2 achieves state-of-the-art performance on point cloud classification and semantic segmentation benchmarks.
  • The appendix provides additional experiment details and results.

A Experiment Details

This section describes the model architectures and experimental settings used for each dataset.

  • The section describes the model architectures adopted in the experiments.
  • The section describes the experimental settings for each dataset in detail.
  • The architecture and setting descriptions support the experiments reported in the paper.

A.1 Model Architecture

The experiments use separate network architectures for semantic segmentation and shape classification, with stage configurations specifying sampled points and feature dimensions.

  • Tuples under each stage block specify sampled points and feature dimensions of attention blocks.
  • The number of sampled points is determined by grid sizes specified in the main paper.
  • Figure 5 presents network architectures for semantic segmentation and shape classification.

A.2 Experiment Setting

The experiments use specified software and hardware, licensed datasets, dataset-specific preprocessing and augmentation, and task-specific training choices.

  • The section introduces the software and hardware environment used for the experiments.
  • The software environment uses CUDA 11.1, cuDNN 8.0.5, PyTorch 1.10.1, four Nvidia RTX A6000 GPUs, and two Intel Xeon Platinum 8180 CPUs.
  • ScanNet v2 is MIT-licensed, while S3DIS and ModelNet40 permit academic use only.
  • S3DIS and ModelNet40 follow PTv1 preprocessing with slight augmentation adjustments, while ScanNet v2 adds estimated normal vectors as input features.
  • AdamW is used for segmentation, cosine annealing performs better on ScanNet v2, and cross-entropy loss is used throughout.

B Additional Quantitative Results

This section reports additional quantitative analyses, including decoder-depth ablations and experimental settings documented in accompanying tables. Attention blocks improve decoding, while increasing decoder depth does not improve performance.

  • Experimental Settings: The additional quantitative-results section includes data-augmentation and training-setting tables.
  • Decoder Depth: At least one attention block in each decoding stage significantly improves model performance.The improvement is more evident when using mapping pooling.
  • Decoder Depth: Increasing each decoder block’s depth from 1 to 2 does not improve performance.The reported comparison uses interpolation and mapping unpooling methods.
  • Decoder Depth: Table 12 reports results for different decoder depths and two upsampling strategies using mIoU%.

B.2 Ablation Study on Position Encoding Multiplier

The ablation examines how the position encoding multiplier interacts with PTv1 and PTv2. Its effect differs across architectures because PTv1 overfits, whereas GVA in PTv2 reduces overfitting and supports learning positional relations.

  • Position Encoding Multiplier: PE Multiplier does not work well with PTv1 because PTv1 already overfits the training set.Adding capacity to PTv1 does not improve performance in this setting.
  • Position Encoding Multiplier: On PTv2, GVA reduces overfitting and enhances generalization, allowing PE Multiplier to learn complex point-cloud positional relations.

B.3 Comparison of Pooling Methods

The section compares FPS-kNN, Grid-kNN, and grid-based partition pooling, including their sampling, neighbor-query, spatial-overlap, and timing characteristics. Synthetic benchmarks use equal sampling ratios and measure combined pooling and unpooling time.

  • Pooling Methods: FPS-kNN samples points with farthest point sampling and then queries k nearest neighbors for pooling.
  • Pooling Methods: Grid-kNN uses uniform-grid sampling followed by kNN indexing, which can produce uncontrollable overlap in pooling receptive fields.
  • Synthetic Benchmark: Synthetic benchmarks generate uniformly random points in a unit cube and keep sampling ratio r constant across pooling methods.
  • Runtime Comparison: The comparison measures combined pooling and unpooling time, including FPS-kNN pooling and unpooling execution time in milliseconds.
Loading 2210.05666v2…