PaperlayerView exact revision on arXiv

Source-linked AI summary

Point Transformer

Nico Engel, Vasileios Belagiannis, Klaus Dietmayer

arXiv:2011.00931v2cs.CV

TL;DR

Unordered point sets require representations that preserve point relations without depending on input order. Point Transformer combines learned point selection with local-global multi-head attention, achieving 92.8% classification accuracy while outperforming attention-based methods and matching state-of-the-art methods.

  • Problem

    Standard Transformer attention is order-dependent, motivating permutation-invariant point-set representations that preserve geometric relations and shape information.

  • Method

    Point Transformer uses multi-head attention, learned Top-K point selection through SortNet, and local-global attention to relate local features with global shape context.

  • Results

    92.8% classification accuracy outperforms attention-based methods and matches state-of-the-art methods on the reported shape-classification benchmark.

  • Takeaways & Limitations

    The model produces sorted, permutation-invariant feature lists for shape classification and part segmentation while relating local geometry to global point-cloud structure.

  • Takeaways & Limitations

    Multi-head attention has quadratic complexity, O(N^2 · d_m), with respect to the number of input points.

Abstract

from arXiv · show

In this work, we present Point Transformer, a deep neural network that operates directly on unordered and unstructured point sets. We design Point Transformer to extract local and global features and relate both representations by introducing the local-global attention mechanism, which aims to capture spatial point relations and shape information. For that purpose, we propose SortNet, as part of the Point Transformer, which induces input permutation invariance by selecting points based on a learned score. The output of Point Transformer is a sorted and permutation invariant feature list that can directly be incorporated into common computer vision applications. We evaluate our approach on standard classification and part segmentation benchmarks to demonstrate competitive results compared to the prior work. Code is publicly available at: https://github.com/engelnico/point-transformer

I. INTRODUCTION

Point Transformer addresses the challenges of processing unordered, dynamic, and unstructured point sets by replacing set pooling with attention-based, permutation-invariant representations. Its SortNet module selects important points using learned scores, while local-global attention relates geometric and shape information for computer vision tasks.

  • I. INTRODUCTION: The method targets three core point-set difficulties: permutation invariance, dynamic cardinality, and unstructured inputs requiring robust processing.Converting point sets to voxel grids or rendered views can lose shape information, while fixed-length latent pooling may inadequately account for input cardinality.
  • I. INTRODUCTION: Local-global attention relates local and global features to capture point-set geometric dependencies and object shape information.The resulting representation is intended for common computer vision applications.
  • I. INTRODUCTION: Point Transformer operates directly on unordered and unstructured point sets using multi-head attention.The architecture is designed to avoid the fixed-size bottleneck associated with traditional set pooling while preserving permutation-invariant representations.
  • I. INTRODUCTION: SortNet induces permutation invariance by selecting important points according to learned scores, producing ordered subsets that replace set pooling.These top-k selections encode local geometric and spatial relations.
  • I. INTRODUCTION: Point Transformer is evaluated on two standard benchmarks, where it delivers competitive results.The introduction frames this evaluation as a test of the proposed attention-based alternative to traditional set pooling.

II. RELATED WORK · A. POINT SET PROCESSING

This section surveys approaches for processing irregular, unordered 3D point sets, including voxel-based, view-based, direct point-set, and point-convolution methods. It highlights PointNet’s permutation invariance and global representation while noting its loss of local structures and relations.

  • II. RELATED WORK: The related-work discussion surveys approaches that process 3D points and are related to this work.
  • A. POINT SET PROCESSING: Point clouds are irregular, unordered, and variable-sized, so prior methods transform them into ordered representations such as voxel grids for 3D CNN processing.
  • A. POINT SET PROCESSING: View-based methods render point clouds into structured 2D images for traditional CNNs [12],, while multi-view methods use multiple angles to address viewpoint occlusion [11], [12],,.
  • A. POINT SET PROCESSING: PointNet [13] operates directly on 3D point sets and achieves input-permutation invariance, eliminating the need for structured-representation conversion.
  • A. POINT SET PROCESSING: PointNet applies shared-weight MLPs per point and symmetric pooling, such as max pooling, to produce a global permutation-invariant feature representation.
  • A. POINT SET PROCESSING: Because max pooling destroys local structures and relations, PointNet cannot encode local information, despite establishing a standard used by many state-of-the-art approaches [1],.
  • A. POINT SET PROCESSING: Point-cloud convolution research defines operations for irregular 3D point sets, including KPConv, SpiderCNN, and PointCNN, which achieve state-of-the-art performance across tasks.

B. ATTENTION · III. FUNDAMENTALS

The section traces attention from natural-language processing to point-cloud networks, emphasizing permutation-invariant set processing and the use of point relations and shape information for visual tasks. It formulates point-cloud processing as a set function invariant to input permutations.

  • B. ATTENTION: Attention originated in natural language processing, [33], where sequential encoder-decoder RNNs struggled to model dependencies between distant inputs.Bahdanau et al. introduced attention to address this limitation by considering the whole input sequence.
  • B. ATTENTION: Attention-based networks were adapted to point clouds by treating points as sequences, achieving strong results in machine translation and point-cloud processing.The section frames this adaptation as a way to transfer attention-based modeling from sequence data to unordered point sets.
  • B. ATTENTION: Vinyals et al. used attention to process unordered sets and demonstrated that their network could sort numbers, but addressed only generic sets.The present approach instead targets different point-cloud tasks involving shape and geometry information.
  • B. ATTENTION: ShapeContextNet combines shape-context selection and feature aggregation through trainable self-attention, but still relies on manually selected kernels sensitive to irregularities.Its hierarchical shape-context operations act as convolutional building blocks, while self-attention reduces manual parameter tuning.
  • B. ATTENTION: Point2Sequence extracts local features with an LSTM-based attention module and forms a global vector through set pooling, but its sequence-to-sequence design is more computationally complex than multi-head attention.It also uses max pooling, distinguishing it from the method discussed in this paper.
  • B. ATTENTION: Tao et al. use multi-head attention with a rotation-invariant point-cloud representation, whereas Point Transformer with SortNet uses attention to select points.Their approach focuses on rotation invariance and global max pooling, while the cited contrast concerns attention-based point selection.
  • III. FUNDAMENTALS: For point sets, attention focuses on important points while the desired set function f: P → R^O maps an unordered input P to outputs invariant to input permutations.The points are treated as interdependent because they represent objects such as laser scans.
  • III. FUNDAMENTALS: The fundamental objective is to use attention to capture relations among points and shape information for visual tasks including object classification and segmentation.The input is defined as P = {p_i ∈ R^D, i = 1, . . . , N}.

A. ATTENTION · B. TRANSFORMER

Attention computes importance-weighted relations among inputs, while Transformer architectures extend this mechanism with multi-head projections and encoder-decoder processing. However, their outputs depend on input ordering, preventing direct use on unordered point sets and motivating Point Transformer.

  • A. ATTENTION: Attention maps queries, keys, and values to outputs by computing pairwise query-key scores and using them to weight the values [19].The attention output is therefore a weighted sum in which higher-scoring relations receive greater weight.
  • A. ATTENTION: Scaled dot-product attention applies softmax to QK^T/√d_k, stabilizing relation weighting among input points.The values V are weighted by the resulting scores to capture point relations.
  • B. TRANSFORMER: Transformers extend attention with encoder-decoder stacks and multi-head attention, forming the basis of the Point Transformer architecture.Each head learns relations in a separate projected subspace before concatenation and output projection.
  • B. TRANSFORMER: Multi-head attention independently projects queries, keys, and values into h subspaces, applies attention in parallel, then concatenates and projects the results.Head dimensions are reduced to d_k = d_v = d_m/h to maintain computational complexity comparable to traditional attention.
  • B. TRANSFORMER: Transformer encoder and decoder layers combine multi-head attention with pointwise feed-forward layers, residual connections, and layer normalization.Multiple attention layers can be sequenced to capture higher-order dependencies.
  • B. TRANSFORMER: Because attention outputs follow the ordering of their input features, Transformer and multi-head attention are not permutation invariant and cannot directly process unordered point data.This limitation motivates the novel Point Transformer architecture for unordered point sets.
  • B. TRANSFORMER: For point processing, self-attention operates over latent features of an unordered point set, while cross-attention relates those features to a second projected set.Both representations use latent dimension d_m, and cross-attention produces N × d_m outputs.

IV. POINT TRANSFORMER

Point Transformer is a multihead-attention network for point sets that achieves permutation invariance through SortNet and relates local and global features using cross multi-head attention. Its architecture combines ordered local features, global point-set features, and task-specific classification or part-segmentation heads.

  • Architecture: Point Transformer relates local and global point-set features through cross multi-head attention while maintaining permutation invariance with SortNet.SortNet selects and orders local features, enabling the network to process unordered point sets.
  • Architecture: The architecture has three components: SortNet for ordered local feature sets, global feature generation, and local-global attention.These components correspond to the network’s local branch, global branch, and feature-relation mechanism.
  • Architecture: Two independent branches generate local features through SortNet and global features through a whole-point-set extraction network.The architecture then relates these representations and applies classification or part-segmentation heads depending on the task.
  • Input Representation: The network accepts point sets with D-dimensional points, commonly D = 3 for xyz coordinates, with optional intensity or normal features.Examples include D = 4 when lidar intensity is appended and D = 6 when point normals are included.

A. SORTNET

SortNet generates permutation-invariant local features by learning point scores from latent representations enriched with spatial and higher-order relations. It selects and orders high-scoring points, aggregates neighborhood features, and combines multiple subspace-specific SortNets into a fixed-size ordered feature set.

  • SortNet: SortNet produces permutation-invariant local features from different subspaces using a learnable score for each input point.Scores are computed after self multi-head attention and a row-wise feed-forward network reduces each feature to a scalar.
  • SortNet: The self multi-head attention layer captures spatial and higher-order relations among latent point features before score generation.The resulting scalar score incorporates spatial relations through the attention layer.
  • SortNet: SortNet selects the K ≤ N highest-scoring input points, sorts them by score, and groups neighboring points within Euclidean radius r to encode local features.A ball-query search aggregates points near each selected point, producing local features for the selected set.
  • SortNet: The selected points, scores, and grouped local features are combined to encode local characteristics while incorporating score calculation into optimization.This produces a local feature vector for the selected points.
  • SortNet: M separate SortNets process different subspaces, and their concatenated outputs form an ordered local feature set of fixed size.Because the selected set is ordered, the resulting local feature set is ordered as well.

B. GLOBAL FEATURE GENERATION

The global branch reduces the input point cloud with multiscale grouping and furthest point sampling, then aggregates neighboring features into a global representation.

  • B. GLOBAL FEATURE GENERATION: The global branch uses Qi et al.’s [7] set abstraction multiscale grouping layer, subsamples N points to N′ < N via furthest point sampling, and aggregates neighboring features of dimension d_m.This reduction is intended to save computational time and memory while producing the global representation.

C. LOCAL-GLOBAL ATTENTION

Point Transformer relates local and global point-cloud features through self- and cross-multi-head attention to capture local context and overall shape. By scoring global features against each selected local feature, it preserves whole-cloud structure without set pooling.

  • C. LOCAL-GLOBAL ATTENTION: Local-global attention applies self multi-head attention within local and global feature sets, then scores every global feature against every local feature to relate context and shape.The mechanism uses local features F_L and global features F_G, with cross-attention A_cross following the self-attention stages.
  • C. LOCAL-GLOBAL ATTENTION: The final row-wise feed-forward layer reduces the feature dimension to d′_m < d_m to decrease computational complexity.The resulting local-global attention mapping has output dimension d′_m.
  • C. LOCAL-GLOBAL ATTENTION: Scoring global features against SortNet-selected local features preserves information about the entire point cloud while retaining local characteristics, avoiding set-pooling reduction of shape representation.The local features form a subset of the input, F_L ⊆ P, and the local-global attention output includes both underlying shape and local information.

D. COMPLETE MODEL

Point Transformer combines independent local-feature extraction and global-feature generation branches through local-global attention, yielding a permutation-invariant ordered representation for downstream tasks. Shape classification maps this representation to object-class probabilities, while part segmentation uses cross multi-head attention to produce per-point class probabilities.

  • D. COMPLETE MODEL: Point Transformer combines SortNet-based local feature extraction with global feature generation and local-global attention, producing a permutation-invariant ordered representation of length K · M with reduced dimension d′_m.SortNet selects input points using learned scores from latent features, while local-global attention relates the resulting spatial signatures to global features.
  • Shape Classification: Shape classification flattens the sorted local-global attention output, reduces it row-wise to C class scores, and applies softmax probabilities.The classification head maps the fixed-size representation to one output per object class.
  • D. COMPLETE MODEL: The model is evaluated on ModelNet object classification and ShapeNet part segmentation benchmarks alongside related set-pooling and attention approaches.Table 1 defines the comparison scope but supplies no result values in the provided passage.
  • Part Segmentation: Part segmentation applies cross multi-head attention between aggregated shape-context features and each input point, relating every point to the entire cloud.Projected global features also use self multi-head attention and feed a set abstraction layer before attention with the Point Transformer output.
  • Part Segmentation: Part segmentation reduces each point representation to C class scores and applies softmax to produce per-point class probabilities.The resulting matrix has dimension R^N×d′_m before row-wise reduction to R^N×C.

V. EXPERIMENTS · A. POINT CLOUD CLASSIFICATION · B. POINT CLOUD PART SEGMENTATION

Point Transformer is evaluated on standard point-cloud classification and part-segmentation benchmarks against point-set, attention-based, and point-convolution methods [7], [13],,,, –,,. It achieves 92.8% classification accuracy, while the experiments also analyze network design components and define the segmentation evaluation scope.

  • V. EXPERIMENTS: Across the experiments, results are compared with direct point-set methods [7], [13],, attention-based methods,,, and point-cloud convolution methods –,.The network is implemented in PyTorch with the RAdam optimizer [41].
  • A. POINT CLOUD CLASSIFICATION: Point Transformer is evaluated on ModelNet40, a 40-category dataset with 9843 training and 2468 test samples.The experiments use 10,000 sampled points with normal vectors from the modified dataset of Qi et al. [7].
  • A. POINT CLOUD CLASSIFICATION: Network design analysis evaluates SortNet variants to show the effect of learnable scoring and compares sampling methods for global-feature generation.These comparisons are presented in Table 2.
  • A. POINT CLOUD CLASSIFICATION: 92.8% classification accuracy is achieved on ModelNet40, outperforming attention-based methods and matching state-of-the-art methods.Results are reported in Table 1 under the stated comparison groups.
  • B. POINT CLOUD PART SEGMENTATION: Part segmentation is evaluated on ShapeNet, containing 13,998 training samples, 2874 test samples, 16 object categories, and 50 part labels.The task predicts the class category of every point and requires understanding the underlying shape.

C. NETWORK COMPLEXITY · D. HYPERPARAMETER STUDY · E. POINT TRANSFORMER DESIGN ANALYSIS

The analyses characterize Point Transformer’s complexity, hyperparameter settings, module behavior, rotation robustness, and learned point selections. Results indicate efficient inference, robust classification under rotation, and shape-aware, complementary SortNet selections.

  • C. NETWORK COMPLEXITY: The complexity study compares network size and inference time against related approaches using experiments run on an Nvidia GeForce 1080Ti.Table 3 defines the comparison scope as network size and inference time.
  • D. HYPERPARAMETER STUDY: The hyperparameter study evaluates different numbers of SortNets and Top-K selections on ModelNet40, with settings selected through grid search.Tables 4 and 5 report the ModelNet40 combinations and the classification and part-segmentation hyperparameters, respectively.
  • E. POINT TRANSFORMER DESIGN ANALYSIS: SortNet is ablated against furthest-point and random selection to test whether learned scores select task-relevant points while preserving permutation invariance.The alternative sampling experiments remove permutation invariance, whereas the learned Top-K pipeline retains it.
  • E. POINT TRANSFORMER DESIGN ANALYSIS: Global-feature generation is ablated by replacing MSG set abstraction with alternative sampling methods, including the complete input cloud and N′ = 128 sampled points.The variants are evaluated by classification accuracy using the complete Point Transformer pipeline.
  • E. POINT TRANSFORMER DESIGN ANALYSIS: 92.3% classification accuracy under random rotations compares with 92.8% without rotation, while PointNet++ falls from 91.9% to 88.6%.Point Transformer was not trained with rotations, whereas the same input rotation was applied to PointNet++.
  • E. POINT TRANSFORMER DESIGN ANALYSIS: SortNet selects similar local regions across slightly different shapes, suggesting awareness of underlying shape, while multiple modules jointly preserve complementary object structure.With M = 8, different SortNet modules focus on different object parts and collectively retain as much of the underlying shape as possible.

VI. CONCLUSION AND FUTURE WORK

Point Transformer is a permutation-invariant neural network for irregular point clouds that uses learned point selection and local-global feature relations for shape classification and part segmentation. Future work targets improved Transformer efficiency through recent self-attention advances [44], [45].

  • VI. CONCLUSION AND FUTURE WORK: Point Transformer operates on irregular point clouds with multi-head attention and produces a sorted, permutation-invariant feature list for shape classification and part segmentation.Its core module selects points from latent feature representations using learned scores.
  • VI. CONCLUSION AND FUTURE WORK: The architecture relates local features to global point-cloud structure, exploiting context to induce shape awareness.
  • VI. CONCLUSION AND FUTURE WORK: The learned point-selection mechanism is based on importance for the specified task, while future work will improve efficiency using recent self-attention advances [44], [45].
Loading 2011.00931v2…