PVT: Point-Voxel Transformer for Point Cloud Learning

1. Introduction

• Point clouds are irregular and unordered, and point-based Transformers spend substantial time sampling and aggregating them, creating an efficiency bottleneck.The introduction frames this as a challenge for directly learning semantics from sensor-produced points.Source: §1 · p. 1
• PVT combines voxel-based local processing with point-based global processing to use their complementary feature information.The voxel branch captures coarse-grained local features, while point processing preserves fine-grained location information.Source: §1 · pp. 1–2
• SWA computes attention locally within non-overlapping 3D windows over non-empty voxels, achieving linear complexity in voxel count and bypassing empty-voxel computation.A GPU-based Rule Book stores non-empty voxel indexes, and shifted windows enlarge the receptive field.Source: §1 · p. 2
• RA incorporates relative-position representations for global point features, while large point clouds use linear-complexity EA to avoid RA’s O(N^2) memory cost.The paper reports RA is more robust to rigid transformations and improves results without additional training parameters.Source: §1 · p. 2
• PVT reaches 94.1% ModelNet40 accuracy without voting, averages 10× faster than Transformer baselines, and reports 86.6%, 69.2%, and 64.9% mIoU on Part, S3DIS, and SemanticKITTI.The segmentation scores are reported in that dataset order.Source: §1 · p. 2

2.1. Voxel-based models on points

• Voxel models offer regular data locality and computational efficiency, but voxelization can lose geometric detail and full-voxel computation grows cubically with resolution.Tree-structured voxel methods bypass empty-voxel computation, while bucketing multiple points into one voxel can make their features indistinguishable.Source: §2.1 · p. 2

2.2. Point-based models for point cloud learning

• Direct point-based methods preserve point locations, but data-structuring costs become a computational burden, especially on large point-cloud datasets.PVT addresses this burden with SWA; its architecture pairs voxel-based local aggregation with point-based global features.Source: §2.2 · p. 3, Figure 1 · p. 3

2.3. Self-attention and Transformer

• Self-attention and Transformers expanded from NLP to vision, with ViT applying Transformers to image patches and Swin adding spatial locality, hierarchy, and translation invariance.These works provide context for applying attention architectures to point clouds.Source: §2.3 · p. 3

2.4. Transformers on point cloud

• Point-cloud Transformers use global dot-product attention, offset attention, or local vector attention, but spend substantial time structuring irregular data.PVT targets this efficiency bottleneck; other Transformer work also explores equivariant processing, completion, and representation learning.Source: §2.4 · pp. 3–4

3. Method

• PVT maps input features into local voxel and global point features, then adds the complementary streams to produce point features.The voxel branch aggregates local features; the point branch directly extracts global features for each point.Source: §3 · p. 4

3.1. Voxel branch

• Fixed-size local windows make window attention linear in voxel resolution, unlike global self-attention’s quadratic complexity.SWA indexes non-empty voxels and avoids computation on empty ones; shifted windows introduce cross-window interaction.Source: §3.1 · p. 4

3.2. Point branch

• Relative Attention embeds relative position representations in self-attention, which the authors describe as generally more robust to rigid transformations.The module adds a relative-representation bias to attention scores.Source: §3.2 · p. 5
• For large-scale point clouds such as SemanticKITTI, PVT uses External Attention because Relative Attention has O(N^2) memory consumption.External Attention uses two small, shared, learnable memories and has linear complexity.Source: §3.2 · p. 5

3.3. Feature fusion

• The feature-fusion module adds the voxel branch’s local features to the point branch’s global features because they provide complementary information.The resulting point features have shape N × D.Source: §3 · p. 4, §3.3 · p. 5

3.4. Relationship to prior works

• Unlike PVCNN, PVT uses SWA rather than 3D convolution for voxel-branch local aggregation and uses Relative or External Attention rather than 1D convolution for global context.The authors describe SWA as highly efficient and the point-branch attention as providing global modeling power.Source: §3.4 · pp. 5–6
• Unlike PCT, PT1, and PT2, PVT does not explicitly find farthest points or neighboring point sets, and the authors report higher efficiency.Its voxel branch uses regular data for local features, while its point branch attends over the entire cloud.Source: §3.4 · p. 6

4. Experiments

• The experiments report latency on a GTX 2080 GPU and evaluate other indicators on a GTX 3090 GPU; segmentation uses dropout with keep probability 0.5 in its last two linear layers.The other point-cloud tasks use an architecture similar to the segmentation architecture.Source: §4 · p. 6

4.1. Shape Classification

• 94.1% accuracy and a 10× average speedup over PCT, PT1, PT2, and PointASNL give PVT the best speed–accuracy trade-off in ModelNet40 classification.Figure 3 also reports that PVT performs best in equal-time and equal-epoch comparisons.Source: Table 2 · p. 6, §4.1 · p. 7, Figure 3 · p. 7

4.2. Analysis of computational requirements

• 2.45M parameters and 1.62G FLOPs accompany PVT’s 94.1% accuracy; it spends 6.3% of runtime structuring sparse data and has the lowest computational and memory requirements among its baselines.Table 4 summarizes computational resource requirements and defines SDA as the runtime rate spent structuring sparse data.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. Object part segmentation

• 86.6% pIoU puts PVT above all baselines on ShapeNet Parts, a 2.9% improvement over PointNet.The point branch captures global shape details while the voxel branch captures large, continuous parts.Source: §4.3 · p. 7, Table 5 · p. 8

4.4. Indoor Scene Segmentation

• PVT attains 69.2% mIoU; separately, reported results substantially outperform most prior models under 6-fold cross-validation, with no cited link between the metric and condition.PVT’s predictions are described as very close to ground truth and as capturing detailed semantic structure in complex 3D scenes.Source: §4.4 · pp. 7–8

4.5. Outdoor semantic Segmentation

• For segmentation experiments, PVT outperforms BAAF by 5.0% in mIoU with 48× measured speedup, and improves mIoU over PointASNL and KPConv by 18.1% and 6.1%, respectively.The PointASNL and KPConv comparisons have 51× and 25× measured speedups, respectively.Source: §4.5 · p. 8

Impact of the voxel-based and point-based branches

• Combining the voxel and point branches improves both tasks over either branch alone; the single-branch baselines score 92.8% and 92.3% on classification.Baseline A encodes global context with the point branch; Baseline B encodes local features with the voxel branch.Source: Table 9 · p. 9, §5 · p. 9

Effect of the cyclic shifted windows scheme

• +1.0% OA on ModelNet40 with cyclic shifted windows versus no shifting, supporting cross-window information interaction in preceding layers.Source: Table 9 · p. 9, §5 · p. 9

Impact of the number of PVT blocks

• Model D saves 25% latency versus PVTfull but loses accuracy; Model E adds latency with little accuracy benefit, motivating the three-block full model.Source: Table 9 · p. 9, §5 · p. 10

Effect of Relative-attention

• 94.1% vs. 92.6% and 86.6% vs. 85.3% for RA versus MLP, with 0.9-point gains over EdgeConv in both comparisons.The study reports absolute percentage-point improvements of 1.5 and 1.3 over MLP, respectively.Source: Table 10 · p. 10, §5 · p. 10

Effect of relative position representations

• +0.8% OA/+0.4% mIoU with relative position representations on ModelNet40 and ShapeNet, respectively, compared with omitting the term.Source: Table 10 · p. 10, §5 · p. 10

• PVT uses GPU-based SWA with linear complexity in voxel resolution and avoids computation on empty voxels to reduce computation cost.Source: §6 · p. 10
• PVT combines voxel- and point-based networks and studies two self-attention variants for fine-grained global-shape features at different point-cloud scales.Source: §6 · p. 10
• Future directions include applying the primary structure to point-cloud pretraining, generation, and completion.Source: §6 · p. 10