PaperlayerView exact revision on arXiv

Source-linked AI summary

MixPT: Point Transformer with Interleaved Window and Global Attention

Minkyun Seo, Hyungtae Lim, Jaesik Park

arXiv:2610.31455v1cs.CV

TL;DR

Large point-cloud scenes increase attention cost, while window and global attention balance efficiency and scene-wide interaction differently. MixPT interleaves fixed-serialization window blocks with global blocks, using spatial pooling for large token sets; it achieves competitive benchmark accuracy with lower latency than representative baselines.

  • Problem

    At fixed spatial resolution, larger 3D scenes retain more points and tokens, raising attention cost as window and global attention trade off computational efficiency against scene-wide interaction.

    Source: §1 · p. 2

  • Method

    MixPT interleaves window and global attention with a fixed serialization pattern, applying global attention to spatially pooled tokens for large token sets and returning residual updates to the original-resolution stream.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across indoor and outdoor benchmarks, MixPT achieves competitive accuracy, and MixPT-S has lower per-forward latency than representative window and global attention baselines in semantic segmentation.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Ablations examine how MixPT controls when and at what spatial resolution global interaction occurs through block placement, pooling resolution, and repeated interleaved patterns.

    Source: §1 · p. 2, §5 · p. 9

    The best block schedule may depend on dataset, task, or model scale, and systematically identifying when to introduce global interaction remains open.

    Source: D LIMITATIONS · p. 20

Abstract

from arXiv · show

Point cloud transformers aim to capture detailed local geometry and relationships across a scene at a manageable computational cost. Two common approaches are window attention and global attention. They differ in how they enable scene-wide interaction, which refers to information exchange among tokens across a scene. Window attention reduces computational cost by restricting self-attention to groups of tokens called windows; in serialized designs, each window contains a fixed number of consecutive tokens from a spatially ordered sequence. Because each block only directly connects tokens within the same window, information exchange across windows relies on changes in window membership across blocks and hierarchical pooling. Global attention, in contrast, directly connects all tokens within a scene, but computing attention over all token pairs can be costly when the number of tokens is large. To balance computational cost and scene-wide interaction, we introduce MixPT, a point transformer that uses a mix of window and global attention blocks in an interleaved schedule, while retaining a single fixed serialization pattern for its window blocks. For scenes with a large number of tokens, MixPT applies global attention to spatially pooled tokens and returns residual updates to the original-resolution stream. Across indoor and outdoor benchmarks, MixPT achieves competitive accuracy with lower latency than representative window and global attention baselines. We further demonstrate the feasibility of scaling the attention stage in depth by repeating the interleaved pattern of window and global attention blocks.

1 INTRODUCTION

At fixed spatial resolution, larger 3D scenes retain more points and increase attention cost, while window and global attention make different trade-offs between efficiency and scene-wide interaction. MixPT interleaves them with fixed window serialization and pooled global attention, achieving competitive accuracy with lower latency than representative baselines.

1 INTRODUCTION

  • At fixed spatial resolution, larger 3D scenes retain more points, increasing token counts and attention cost.Window attention limits direct connections to tokens within each window; global attention connects all scene tokens but can be costly when token counts are large.Source: §1 · p. 2
  • MixPT interleaves global blocks with window blocks that reuse a single fixed serialization pattern, enabling direct information exchange across windows.For large token sets, global attention operates on spatially pooled tokens and returns residual updates to the original-resolution stream.Source: §1 · p. 2
  • Across indoor and outdoor benchmarks, MixPT achieves competitive accuracy, while MixPT-S has lower per-forward latency than representative window and global attention baselines.The contributions also analyze block arrangement, pooling resolution, and model scaling.Source: §1 · p. 2
  • The paper analyzes cross-window communication and scene-wide support across global blocks, and demonstrates deeper attention stages by repeating the interleaved pattern.Ablations examine block arrangement, pooling resolution, and model scaling.Source: §1 · p. 2

2 RELATED WORK

Prior point-cloud backbones established local aggregation and hierarchical processing, while transformer work explored scalable attention through local groups, serialized windows, and global tokens. This paper studies interleaving window and global attention for serialized point-cloud backbones, building on a design principle explored mainly in image and language models.

Point Cloud Backbones

  • Point-based and sparse-grid backbones developed local geometric aggregation and hierarchical processing for large-scale point-cloud understanding.Point methods use neighborhood aggregation and scalable sampling, while sparse voxel and octree approaches restrict computation to occupied regions.Source: §2 · p. 2

Point Cloud Transformers

  • Point-cloud transformers scale attention through local neighborhoods, shared groups, adaptive windows, fixed-size token groups, serialized windows, or global attention over volumetric patches.PTv3 changes serialization patterns across blocks to facilitate communication between groups; LitePT and Volt differ in the spatial resolution at which attention operates.Source: §2 · p. 3

Interleaving Window and Global Attention

  • Interleaving local and global attention has been explored in image and language models, motivating its study in serialized point-cloud backbones.Prior approaches introduce compact global context, sparse long-range connections, or full global attention at selected layers.Source: §2 · p. 3

Rotary Positional Encoding for 3D Points

  • MixPT adopts asymmetric channel allocation across the three spatial axes for 3D rotary positional embedding and examines its effect in ablations.The passage contrasts this allocation with symmetric formulations used in other point-cloud transformers.Source: §2 · p. 3

3 METHODOLOGY

The analysis contrasts how window and global attention provide cross-window communication, then motivates MixPT’s periodic global blocks. MixPT interleaves fixed-order window attention with global attention that can operate on pooled tokens, returning residual updates to the original-resolution stream.

3.1 PRELIMINARIES: REVISITING WINDOW AND GLOBAL ATTENTION

  • Window attention reduces pairwise interactions to O(NNw) by restricting attention to Nw-token windows, while global attention connects all tokens at potentially high cost.Serialized window transformers rely on changing serialization patterns and hierarchical pooling for cross-window communication.Source: §3.1 · p. 3
  • LitePT did not reach all tokens on average even in its deepest attention blocks, whereas Volt achieved full reachability from its first block; MixPT places its first global block after two window blocks.Reachability is measured from attention-stage input tokens, excluding receptive fields accumulated in preceding convolutional stages.Source: Figure 2 · p. 3, §3.1 · p. 4, Figure 3 · p. 4
  • Volt-S attention entropy is higher in early blocks and lower in several later blocks, while a selected query’s attention concentrates in localized regions later.The entropy measures how evenly attention is distributed over available keys; Figure 5 colors tokens by relative attention weight.Source: §3.1 · p. 4, Figure 4 · p. 4, Figure 5 · p. 4

3.2 MIXPT: INTERLEAVED WINDOW AND GLOBAL ATTENTION

  • MixPT’s window blocks use self-attention over fixed-size groups of consecutive serialized tokens, followed by an MLP, with a pre-norm design.The attention and MLP sublayers each use residual updates.Source: §3.2 · p. 5
  • For large token sets, MixPT pools spatial tokens for global attention and maps only attention and MLP residual updates back to the original-resolution stream.The pooling stride s=1 denotes the unpooled case; s>1 denotes pooling.Source: §3.2 · pp. 5–6, Figure 7 · p. 5
  • MixPT applies one-dimensional RoPE independently to x-, y-, and z-axis subspaces of each attention head’s query and key vectors, using asymmetric dimension allocation by default.The axis-specific dimensions sum to the per-head dimension and must each be even.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments evaluate semantic and instance segmentation, then examine serialization, attention-block arrangement, pooling stride, and RoPE allocation. MixPT shows competitive accuracy and latency trade-offs across indoor and outdoor benchmarks.

4.1 SEMANTIC SEGMENTATION

  • MixPT-S had the lowest latency across four datasets, reducing it 19% to 32% versus LitePT-S and running 2.1 to 6.9× faster than Volt-S outdoors, while matching or improving LitePT-S mIoU.The latency comparison is for single model forward passes; the mIoU comparison is without TTA.Source: Table 1 · p. 6, Table 2 · p. 6, §4.1 · p. 7
  • When MixPT was scaled by increasing model depth and per-head dimension while retaining 16 attention heads, MixPT-B and MixPT-L generally improved performance across the evaluated benchmarks.Source: §4.1 · p. 7

4.2 INSTANCE SEGMENTATION

  • MixPT-S* remained competitive with baselines on ScanNet and ScanNet200 using a common PointGroup head for backbone comparison.MixPT-S* extends MixPT-S with additional attention and convolution blocks and was trained without EMA for this task.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 ABLATION STUDIES AND ANALYSIS

  • Changing two of eight window blocks to global blocks raised mIoU to 70.0% and 76.0%, respectively, while fixed serialization reduced latency on both datasets.Fixing serialization left SemanticKITTI mIoU nearly unchanged and decreased ScanNet mIoU; periodic global attention improved fixed-serialization accuracy while preserving a latency advantage over multi-pattern windows.Source: Table 4 · p. 8, §4.3 · p. 8
  • Among eight-block schedules, WWGWWGWW offered a favorable nuScenes accuracy-latency trade-off and the highest ScanNet mIoU; more global blocks did not consistently improve accuracy.A deeper ten-block variant slightly improved nuScenes accuracy but increased parameters and latency.Source: Table 5 · p. 8, §4.3 · p. 8
  • On Waymo, pooling at s = 2 improved mIoU over full-resolution global attention; pooling lowered global-block latency on all three datasets, and s ≥2 reduced peak memory below window blocks.Waymo had the largest absolute global-block latency reduction; additional pooling yielded only modest further reductions in global-block memory.Source: §4.3 · p. 8, Figure 9 · p. 9
  • The 8:8:6 RoPE allocation matched 8:8:8 on nuScenes and outperformed it on ScanNet with fewer parameters, consistent with larger horizontal than vertical extents.Figure 10 shows representative scene and serialized-window extents for ScanNet and nuScenes.Source: Table 6 · p. 9, §4.3 · p. 9, Figure 10 · p. 9

5 CONCLUSION

MixPT combines fixed-order window attention with periodically interleaved global attention, using spatial pooling to reduce global-attention cost for large token sets. The experiments report competitive accuracy across indoor and outdoor benchmarks and lower MixPT-S latency in semantic segmentation.

5 CONCLUSION

  • MixPT interleaves fixed-order window attention with periodic global blocks, using pooled global attention for large token sets and returning residual updates to the original-resolution stream.The analysis motivates direct cross-window communication at selected layers.Source: §5 · p. 9
  • Across indoor and outdoor 3D scene-understanding benchmarks, MixPT achieved competitive accuracy, and MixPT-S had lower semantic-segmentation latency than representative window and global baselines.Source: §5 · p. 9
  • Ablations examined global-block placement, pooling resolution, and scaling through repeated interleaved patterns, illustrating control over when and at what resolution global interaction occurs.Source: §5 · p. 9

A ARCHITECTURE AND IMPLEMENTATION DETAILS

The architecture details describe model configurations, how schedules are written and extended, and how pooled global-attention updates return to the original token stream.

A ARCHITECTURE AND IMPLEMENTATION DETAILS

  • The notation (WWG)2WW expands to WWGWWGWW; MixPT-B and MixPT-L extend the schedule to (WWG)4WW.MixPT-S* appends two global attention blocks to the eight-block encoder attention stage and adds two convolutional blocks after unpooling at each decoder stage.Source: A ARCHITECTURE AND IMPLEMENTATION… · p. 16, Table A1 · p. 16
  • MixPT-B and MixPT-L increase stage widths and convolutional depths, while MixPT-S* adds attention and decoder convolution blocks for instance segmentation.Source: A ARCHITECTURE AND IMPLEMENTATION… · p. 16, Table A1 · p. 16
  • For pooling stride s > 1, channel-wise max pooling operates within grid cells after a linear projection and is followed by batch normalization.A second projection maps the global residual update to stage width; the update is broadcast to each cell’s member tokens and added using stored pooling assignments.Source: A ARCHITECTURE AND IMPLEMENTATION… · p. 16

B.1 SEMANTIC SEGMENTATION

The appendix specifies semantic-segmentation training and evaluation, efficiency measurement, instance-segmentation adaptation, and test-time fragment handling. It reports how results are produced and how evaluation cost is managed.

B.1 SEMANTIC SEGMENTATION

  • Semantic segmentation uses full supervision on four datasets with standard splits, EMA weights, and dataset-specific settings; evaluation follows PTv3 and LitePT protocols.For outdoor datasets, an initial 0.025 m grid sampling retains a mapping to original points; evaluation without TTA still uses fragment construction and aggregation.Source: §B.1 · pp. 16–17

B.2 EFFICIENCY MEASUREMENT

  • Latency is measured per voxelized-scene forward on an RTX A6000 using batch size one and FP16 autocast, with preprocessing and prediction aggregation excluded.Each input is timed three times after eight warm-up scenes, and the reported value is the median of per-scene medians.Source: §B.2 · p. 17

B.3 INSTANCE SEGMENTATION

  • Instance segmentation uses the PointGroup framework on ScanNet and ScanNet200, with backbone composition, initialization, and training varying by baseline.PTv3, LitePT-S*, and MixPT-S* are jointly trained from scratch; Volt uses a pretrained backbone and feature decoder that are fine-tuned with the PointGroup head.Source: §B.3 · pp. 17–18, Table A3 · p. 18

B.4 TEST-TIME AUGMENTATION AND FRAGMENTS

  • Each fragment requires a separate forward, and fragment forwards total 11.9×, 12.3×, and 13.1× the identity-view count on SemanticKITTI, nuScenes, and ScanNet, respectively.Main comparisons therefore use results without TTA while retaining fragment aggregation; Figure A1 summarizes mean fragments per TTA view.Source: §B.4 · p. 19, Figure A1 · p. 19

C ADDITIONAL RESULTS AND ABLATIONS

The additional results examine test-time augmentation, serialization order, and attention geometry. They report a modest advantage for Trans Z-order among tested fixed orders and describe the measured spatial ranges.

Semantic Segmentation with TTA

Serialization Order

  • Trans Z-order achieves the highest mIoU across the three tested datasets, with only modest performance variation across fixed serialization orders.Serialization overhead includes code construction and stage layout; in this implementation, Hilbert-based alternatives have higher code-construction cost.Source: C ADDITIONAL RESULTS AND ABLATIONS · p. 20, Table A5 · p. 20

Scene and Attention Geometry

  • Window and global attention ranges show a similar horizontal–vertical imbalance, but their grid-cell counts cannot be compared directly as physical distances.Window ranges use the attention-stage grid, while global ranges use the pooled grid for s > 1 or original attention-stage resolution for s = 1.Source: C ADDITIONAL RESULTS AND ABLATIONS · p. 20, Table A6 · p. 20

D LIMITATIONS

The limitations identify two open design questions: selecting an interleaving schedule and improving the efficiency–accuracy trade-off of global attention.

Block arrangement

  • The study establishes interleaving effectiveness rather than exhaustively optimizing block arrangement; the best schedule may vary by dataset, task, or model scale.A more systematic search for when to introduce global interaction remains open.Source: D LIMITATIONS · p. 20

Global-attention efficiency

  • Spatial grid pooling is the current way to reduce global-attention cost, and alternative compression methods may improve the efficiency–accuracy trade-off.The stated challenge is reducing token count while preserving information needed for effective scene-wide interaction.Source: D LIMITATIONS · p. 20

E QUALITATIVE RESULTS

The qualitative-results figures present semantic segmentation examples for SemanticKITTI and ScanNet, plus instance segmentation examples for ScanNet200.

SemanticKITTI

  • The SemanticKITTI figure places input point clouds, MixPT-S predictions, and ground truth in columns, with insets highlighting representative regions.The listed class labels include road, sidewalk, building, vegetation, and vehicles.Source: Figure A2 · p. 21, E QUALITATIVE RESULTS · p. 21

ScanNet semantic segmentation

  • The ScanNet semantic-segmentation figure compares input point clouds, MixPT-S predictions, and ground truth in columns.The listed class labels include wall, floor, cabinet, bed, and chair.Source: Figure A3 · p. 22, E QUALITATIVE RESULTS · p. 22

ScanNet200 instance segmentation

  • The ScanNet200 instance-segmentation figure compares inputs, MixPT-S* predictions, and ground truth; colors distinguish instances rather than semantic classes.Matched prediction–ground-truth pairs share colors when one-to-one matching within the same class has mask IoU > 0.5; gray marks background or unassigned points.Source: Figure A4 · p. 23
Loading 2610.31455v1…