PaperlayerView exact revision on arXiv

Source-linked AI summary

LitePT: Lighter Yet Stronger Point Transformer

Yuanwen Yue, Damien Robert, Jianyuan Wang, Sunghwan Hong, Jan Dirk Wegner, Christian Rupprecht, Konrad Schindler

arXiv:2512.13689v2cs.CV

TL;DR

3D point-cloud networks lack clear guidance on how to combine convolution and attention efficiently. LitePT studies their stage-wise roles, uses convolutions early and PointROPE-enhanced attention late, and achieves lower resource use while matching or outperforming PTv3 across tasks. The paper notes that global attention in later stages could further improve context modeling and reduce inference computation.

  • Problem

    The field lacks detailed understanding and principled guidelines for combining convolution and attention in versatile, efficient 3D point-cloud architectures.

  • Method

    LitePT uses sparse convolutions in early high-resolution stages, PointROPE-enhanced attention in later low-resolution stages, and a parameter-free 3D positional encoding.

  • Results

    LitePT uses 3.6× fewer parameters, reduces training and inference memory by 60.3% and 51.2%, and lowers training and inference latency by 34.5% and 58.8% while improving performance across several benchmarks.

  • Takeaways & Limitations

    Convolutions for low-level geometry and attention for high-level relations provide the paper’s supported design principle for lighter yet stronger point-cloud backbones.

  • Takeaways & Limitations

    Because attention is restricted to later stages, global self-attention is affordable and eliminating local grouping remains a proposed direction for future work.

Abstract

from arXiv · show

Modern neural architectures for 3D point cloud processing contain both convolutional layers and attention blocks, but the best way to assemble them remains unclear. We analyse the role of different computational blocks in 3D point cloud networks and find an intuitive behaviour: convolution is adequate to extract low-level geometry at high-resolution in early layers, where attention is expensive without bringing any benefits; attention captures high-level semantics and context in low-resolution, deep layers more efficiently, where convolution inflates the parameter count. Guided by this design principle, we propose a new, improved 3D point cloud backbone that employs convolutions in early stages and switches to attention for deeper layers. To avoid the loss of spatial layout information when discarding redundant convolution layers, we introduce a novel, parameter-free 3D positional encoding, PointROPE. The resulting LitePT model has $3.6\times$ fewer parameters, runs $2\times$ faster, and uses $2\times$ less memory than the state-of-the-art Point Transformer V3, but nonetheless matches or outperforms it on a range of tasks and datasets. Code and models are available at: https://github.com/prs-eth/LitePT.

1. Introduction

The paper addresses unclear guidance for combining convolution and attention in 3D point-cloud networks. It proposes LitePT, which assigns convolutions to early stages and PointROPE-enhanced attention to later stages, reducing resource use while improving benchmark performance.

  • The work responds to limited understanding of the relative strengths of point-cloud processing layers and how to combine them efficiently.The authors frame this as a missing set of principled architectural guidelines.
  • LitePT uses sparse convolutions in early stages and PointROPE-enhanced attention in later stages, matching each operator to the hierarchy’s processing needs.The design targets local geometry at high resolution and semantic context at lower resolution.
  • Convolution is adequate for local geometry in early high-resolution stages, while attention is more suitable and parameter-efficient for semantics and global context in later low-resolution stages.Attention is substantially more expensive in early layers because they contain many tokens.
  • 3.6× fewer parameters, 60.3% less training memory, 51.2% less inference memory, and 34.5% and 58.8% lower training and inference latency than PTv3.LitePT also improves performance across semantic segmentation, instance segmentation, and object detection benchmarks.

2. Related Work

Related work spans projection-based, voxel-based, convolutional, Transformer, and hybrid approaches for point-cloud representation. The paper focuses on combining convolution’s local-feature bias with Transformers’ long-range modeling while replacing costly learned positional encoding.

  • Point-cloud methods include projections to 2D images, voxelization followed by 3D convolution, and newer Transformer or hybrid architectures.Projection methods rely on assumptions such as relatively uniform density, coverage, and opaque surfaces.
  • Convolution captures local features, whereas Transformers model long-range dependencies, motivating hybrid architectures.The paper positions LitePT as a stage-specific combination rather than a uniform hybrid block.
  • Convolutional positional encoding improves efficiency but introduces many learnable parameters, motivating the parameter-free PointROPE alternative.PointROPE adapts rotary positional embedding to point-cloud learning.

3. Methodology

The methodology analyzes PTv3 across hierarchy levels, then builds LitePT from stage-specific convolution or attention blocks and introduces PointROPE to preserve positional information without deep convolution layers. The resulting architecture follows the observed division between local geometry and high-level semantics.

  • 3.1. Revisiting PTv3: Convolution vs. Attention: 67% of PTv3’s parameters belong to sparse convolutional positional encoding, while attention incurs its greatest cost in early high-token stages.Convolution parameter count grows toward the bottleneck, whereas attention latency decreases after hierarchical downsampling.
  • 3.1. Revisiting PTv3: Convolution vs. Attention: Removing convolution from PTv3 causes a larger performance drop than removing attention, indicating that the convolutional positional encoding also performs substantial feature extraction.The representations show local geometry in shallow stages and semantics in deeper stages regardless of the operator variant.
  • 3.2. Tailored Blocks for Different Network Stages: LitePT assigns convolution blocks to early high-resolution stages and attention blocks to deeper low-resolution stages, with an optional hand-over stage containing both.Early convolution aggregates local receptive fields efficiently; deeper attention handles semantic abstraction after token reduction.
  • 3.2. Tailored Blocks for Different Network Stages: LitePT simplifies PTv3’s block design by using convolution-only or attention-only blocks instead of repeating a combined block at every hierarchy level.This stage-dependent simplification is the central architectural departure from PTv3 and related hybrids.
  • 3.3. Point Rotary Positional Embedding: PointROPE applies axis-wise rotary positional embedding to 3D point features, preserving relative geometry without learnable parameters.The feature dimension is divided into x, y, and z subspaces, each using its corresponding coordinate.

4. Experiments

Experiments show that LitePT’s stage-specialized hybrid design improves efficiency while maintaining or exceeding PTv3 across semantic segmentation, instance segmentation, and object detection.

  • Ablation studies and analysis: Removing early attention preserves mIoU while improving efficiency, whereas removing late convolution cuts parameters with negligible mIoU change; Lc = 3 gives the best trade-off.The recommended configuration uses convolution in the first three stages and attention in the last two.
  • Ablation studies and analysis: LitePT scales favorably from LitePT-S to LitePT-L, with consistent performance gains and only modest latency and memory increases.Even LitePT-L, with twice PTv3’s parameter count, remains faster and uses less memory.
  • Ablation studies and analysis: PointROPE is necessary for performance, since removing it lowers nuScenes mIoU by 2.6 percentage points, while its frequency choice is relatively robust.The recommended base frequency is b = 100.
  • 3D semantic segmentation: LitePT improves outdoor semantic segmentation by 1.8 mIoU over competing architectures on both nuScenes and Waymo.These datasets contain challenging automotive LiDAR variation in point density and anisotropic scan patterns.
  • 3D semantic segmentation: LitePT is comparable to PTv3 on ScanNet and consistently outperforms competing methods on the larger Structured3D dataset.In data-constrained ScanNet settings, LitePT is sometimes slightly better than PTv3 and clearly exceeds PTv2.
  • 3D instance segmentation: LitePT sets a new ScanNet instance-segmentation state of the art with 64.9 mAP50, 3.2 percentage points above PTv3.On ScanNet200, LitePT is comparable to PTv3 and outperforms earlier methods, including PTv2 with similar parameter count.
  • 3D object detection: LitePT reaches the highest overall Waymo detection score and matches PTv3 closely while leading on two of three object categories.The evaluation uses single-scan LiDAR inputs and level-2 metrics.

5. Conclusion and Discussion

LitePT assigns convolutions to low-level geometry and attention to high-level relations, using PointROPE to preserve spatial layout after removing deeper convolution layers. The authors report this as a lightweight backbone and suggest global late-stage attention as future work.

  • Conclusion and Discussion: LitePT uses convolutions for low-level geometry and attention for high-level semantic abstraction and global context across the feature hierarchy.The design places only the required operations at each hierarchy level to avoid wasted computation.
  • Conclusion and Discussion: PointROPE is parameter-free and compensates for spatial layout information lost when convolutional layers are discarded.
  • Conclusion and Discussion: The conclusion positions LitePT as a generic high-performance backbone and the analysis as practical guidance for future architecture design.
  • Conclusion and Discussion: Global self-attention is affordable in LitePT’s later stages because the token count has already been reduced.The authors identify removing local grouping as future work that could strengthen long-range context modelling and reduce inference time.

A. Detailed Architecture

LitePT uses a five-stage U-Net-style encoder-decoder with skip connections, pooling, and unpooling. Its encoder uses convolution blocks in the first three stages and attention blocks in the final two.

  • A. Detailed Architecture: LitePT’s five-stage U-Net-style encoder-decoder uses skip connections, pooling between encoder stages, and unpooling between decoder stages.
  • A. Detailed Architecture: The encoder applies convolution blocks in its first three stages and attention blocks in its final two stages.This is the architecture’s stage-tailored design.
  • A. Detailed Architecture: Each convolution block combines a 3 × 3 × 3 sparse convolution, linear projection, LayerNorm, and a residual connection.

B. Detailed Experimental Settings

Experimental inputs differ by dataset setting: indoor experiments use RGB and surface normals, while outdoor experiments use coordinates and intensity, with elongation added for detection. Point clouds are first grid-downsampled with task- and setting-specific resolutions.

  • B. Detailed Experimental Settings: Indoor datasets use RGB and surface normals, whereas outdoor datasets use xyz coordinates and intensity, with elongation additionally used for object detection.
  • B. Detailed Experimental Settings: The experiments first grid-downsample point clouds, using 0.02m indoor and 0.05m outdoor grids for segmentation.
  • B. Detailed Experimental Settings: For 3D object detection, the grid sizes are 0.32m in the xy plane and 6m along the z axis.

C.1. Further Ablation on PointROPE

The PointROPE ablations examine coordinate choices, embedding subdivisions, and positional-encoding alternatives. Equal weighting across Cartesian axes performs best, while PointROPE achieves the highest mIoU with the smallest latency increase among compared encodings.

  • C.1. Further Ablation on PointROPE: The architecture figures show PointROPE attention and the detailed LitePT-S, LitePT-S*, and PTv3 pipelines used for comparison.
  • C.1. Further Ablation on PointROPE: PointROPE applies rotary positional embedding to query and key before standard scaled dot-product attention.
  • C.1. Further Ablation on PointROPE: Equal Cartesian-axis weighting outperforms uneven subdivisions, indicating that positional information across all three axes is similarly important.The tested splits include 6:6:6, 4:4:10, and 8:8:2.
  • C.1. Further Ablation on PointROPE: PointROPE achieves the highest mIoU and the smallest latency increase compared with sinusoidal absolute and relative positional encodings on nuScenes.

C.2. Cross-dataset Transfer

LitePT-S transfers better than PTv3 from Structured3D to ScanNet, suggesting that its positional encoding preserves spatial relationships across differing scenes and sensors.

  • C.2. Cross-dataset Transfer: PointROPE preserves relative spatial relationships across scenes with different layouts and sensor characteristics, supporting more transferable representations.The authors attribute the improved generalisation to reduced overfitting to dataset-specific structures.
  • C.2. Cross-dataset Transfer: 7.9 pp higher mIoU than PTv3 on ScanNet after Structured3D pretraining demonstrates stronger cross-dataset transfer.Models are evaluated with frozen encoders and linear probing on multi-scale features.
  • C.2. Cross-dataset Transfer: The transfer protocol pretrains PTv3 and LitePT-S on Structured3D, freezes their encoders, and evaluates ScanNet performance with a linear probe.Structured3D is described as the largest dataset in the benchmark suite.

C.4. Additional Results for LitePT-L

Scaling from LitePT-S to LitePT-L consistently improves performance, but the gain is larger on datasets with more points available for training.

  • C.4. Additional Results for LitePT-L: +1.8 pp mIoU on Structured3D versus +0.4 pp on nuScenes when scaling to LitePT-L shows that gains depend strongly on dataset size.Structured3D contains 9.5 billion points, compared with 1 billion for nuScenes.
  • C.4. Additional Results for LitePT-L: LitePT-L consistently improves performance over LitePT-S across the reported benchmarks.The authors relate this trend to established scaling laws and available data volume.

C.5. Benchmarking Outdoor Efficiency

On the large-scale Waymo outdoor LiDAR benchmark, LitePT-S remains more efficient than PTv3, while standard evaluation uses chunking and test-time augmentation.

  • C.5. Benchmarking Outdoor Efficiency: > 2× faster inference than PTv3 on Waymo confirms LitePT-S’s outdoor efficiency advantage.Results are averaged over the full Waymo dataset using a single RTX 4090 GPU.
  • C.5. Benchmarking Outdoor Efficiency: The main semantic-segmentation evaluation applies overlapping chunking and test-time augmentations, aggregating point labels by voting.Each augmented sample is partitioned so every point is assigned to at least one chunk.
  • C.5. Benchmarking Outdoor Efficiency: Removing chunking and test-time augmentation reduces semantic-segmentation performance by roughly 2% mIoU for both PTv3 and LitePT-S.This provides a single-pass comparison for downstream users.
Loading 2512.13689v2…