PaperlayerView exact revision on arXiv

Source-linked AI summary

DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, Liwei Wang

arXiv:2301.06051v2cs.CV

TL;DR

Sparse and irregular point clouds make efficient, deployment-friendly Transformer backbones difficult to design for 3D perception. DSVT addresses this with dynamic sparse window attention, rotated set partitioning, and learnable 3D pooling, achieving broad task performance with 27Hz TensorRT inference. The paper reports state-of-the-art performance across various benchmarks while identifying limited generality beyond outdoor autonomous-driving settings.

  • Problem

    Sparse point clouds make standard Transformer attention non-trivial, while existing approaches introduce redundant computation, unstable performance, or customized CUDA operations that complicate deployment.

  • Method

    DSVT uses sparsity-adaptive equal-sized window subsets for parallel attention, alternates X/Y partition axes for cross-set connections, and applies learnable 3D pooling for downsampling and geometric encoding.

  • Results

    DSVT achieves state-of-the-art performance across various 3D perception benchmarks and reaches 27Hz real-time inference with NVIDIA TensorRT.

  • Takeaways & Limitations

    DSVT provides a Transformer-only, deployment-friendly backbone for outdoor 3D perception without customized CUDA operations.

  • Takeaways & Limitations

    DSVT focuses on outdoor autonomous-driving detection with objects distributed on a 2D ground plane, leaving more general-purpose 3D backbones open.

Abstract

from arXiv · show

Designing an efficient yet deployment-friendly 3D backbone to handle sparse point clouds is a fundamental problem in 3D perception. Compared with the customized sparse convolution, the attention mechanism in Transformers is more appropriate for flexibly modeling long-range relationships and is easier to be deployed in real-world applications. However, due to the sparse characteristics of point clouds, it is non-trivial to apply a standard transformer on sparse points. In this paper, we present Dynamic Sparse Voxel Transformer (DSVT), a single-stride window-based voxel Transformer backbone for outdoor 3D perception. In order to efficiently process sparse points in parallel, we propose Dynamic Sparse Window Attention, which partitions a series of local regions in each window according to its sparsity and then computes the features of all regions in a fully parallel manner. To allow the cross-set connection, we design a rotated set partitioning strategy that alternates between two partitioning configurations in consecutive self-attention layers. To support effective downsampling and better encode geometric information, we also propose an attention-style 3D pooling module on sparse points, which is powerful and deployment-friendly without utilizing any customized CUDA operations. Our model achieves state-of-the-art performance with a broad range of 3D perception tasks. More importantly, DSVT can be easily deployed by TensorRT with real-time inference speed (27Hz). Code will be available at \url{https://github.com/Haiyang-W/DSVT}.

1. Introduction

DSVT addresses the challenge of applying Transformers to sparse point clouds with parallel sparse attention, rotated partitions, and deployment-friendly pooling. The resulting backbone targets outdoor 3D perception with real-time inference and broad task performance.

  • Motivation: Sparse and irregular point clouds make standard Transformer processing non-trivial, while existing solutions incur redundant computation, unstable performance, or costly custom CUDA operations.The central challenge is building an efficient, deployment-friendly 3D Transformer backbone.
  • Method: DSVT introduces Dynamic Sparse Window Attention, which divides each window’s sparse voxels into equal-sized, window-bounded subsets for parallel self-attention.The subsets dynamically reflect window sparsity, assigning more sets and computation resources to denser windows.
  • Method: Rotating subset partitioning between X-axis and Y-axis layers connects preceding subsets and strengthens intra-window feature fusion.Hybrid window partitioning additionally alternates window shapes across successive Transformer blocks for inter-window fusion and multi-scale information.
  • Method: A learnable 3D pooling operation supports sparse-feature downsampling and geometric encoding without relying on heavily optimized custom operations.The design is intended to improve deployment practicality compared with customized scatter functions or strided sparse convolutions.
  • Results: 27Hz real-time inference is achieved with a deployment-friendly Transformer backbone that uses no customized CUDA operations and can be accelerated by NVIDIA TensorRT.The paper also reports performance gains across various 3D perception tasks on Waymo and nuScenes.

2. Related Work

Prior point-cloud methods mainly use point-based or voxel-based representations, each with different computational trade-offs. Point-based approaches extract geometry directly but can be slowed by sampling, neighbor search, and feature duplication.

  • Point-based methods: Point-based methods extract geometric features directly from raw point clouds using point representations.The related-work passage groups prior 3D perception methods by their point or voxel representations.
  • Point-based methods: Point-based methods suffer from time-consuming point sampling and neighbor searching, along with computation-intensive point-wise feature duplication.
  • Voxel-based methods: Voxel-based methods convert input point clouds into regular 3D voxels before processing them, especially in outdoor autonomous-driving scenarios.

3. Methodology

DSVT is a single-stride sparse voxel Transformer backbone that addresses variable voxel sparsity through dynamically sized, parallel attention sets and rotated partitions. It also introduces attention-style 3D pooling for downsampling and geometric encoding without customized CUDA operations.

  • Backbone architecture: DSVT converts point clouds into sparse voxel tokens and applies single-stride transformer blocks with rotated-set and hybrid-window partitions.The architecture preserves X/Y feature-map scale while enabling intra- and inter-window feature propagation.
  • Dynamic Sparse Window Attention: Dynamic Sparse Window Attention reformulates each sparse window as size-equivalent, window-bounded voxel subsets for parallel self-attention.The method uses native deep-learning operations rather than self-designed CUDA operations, supporting deployment on modern GPUs.
  • Dynamic set partition: The number of subsets dynamically scales with window sparsity: S = floor(N/τ) + I[(N % τ) > 0], where τ limits non-empty voxels per set.More non-empty voxels assign more sets and computation resources to the window.
  • Dynamic set partition: Voxel features and coordinates are gathered by inner-window IDs into non-overlapping subsets, each containing τ entries for batched attention.The indexed tensors have features Fj ∈ Rτ×C and coordinates Oj ∈ Rτ×3.
  • Rotated set attention: Rotated-set attention alternates X-axis and Y-axis sorting in consecutive attention layers to connect voxels across otherwise non-overlapping subsets.The DSVT block applies multi-head self-attention after each partition configuration, with positional encoding from voxel coordinates.
  • Attention-style 3D pooling: Attention-style 3D pooling replaces sparse-region padding and MLP downsampling to improve geometric encoding while remaining compatible with full attention and deployment.The authors report that the pooling operation performs better than the pillar variant.

4. Experiments

Experiments evaluate DSVT across outdoor detection, BEV map segmentation, ablations, and deployment speed. The results show strong accuracy, benefits from the proposed modules, and real-time TensorRT inference.

  • 3D Object Detection: 72.1 mAPH on Waymo L2 makes single-frame DSVT-V +3.2 above the previous best one-stage method and +1.3 above the best two-stage approaches.
  • 3D Object Detection: 74.6 and 75.6 L2 mAPH on Waymo 2- and 4-frame settings exceed previous best one-stage multi-frame methods by +1.8 and +2.4.The corresponding L1 gains are +3.7 and +4.3.
  • 3D Object Detection: 72.7 test NDS and 68.4 mAP on nuScenes surpass PillarNet by +1.3 and +2.4, respectively.
  • BEV Map Segmentation: +3.0 improves BEV map segmentation with DSVT, while a deeper BEV backbone yields an additional +19.4 gain.The evaluation uses only lidar input and switches the 3D backbone within the BEVFusion lidar branch.
  • Ablation Studies: Attention-style 3D pooling outperforms the evaluated baselines, while masking empty voxels harms performance and suggests empty voxels encode object geometry.
  • Inference Speed: 27 Hz TensorRT inference runs nearly as fast as CenterPoint-Pillar while delivering +8.8 L2 mAPH, with 67 ms latency versus PV-RCNN++ at 113 ms.On a strong sparse-convolution baseline, DSVT-P adds +1.78 L2 mAPH and TensorRT provides 2 × faster execution.

5. Conclusion

The conclusion presents DSVT as a deployment-friendly Transformer-only backbone for 3D perception. Its sparse attention and pooling designs support parallel processing, geometric encoding, broad benchmark performance, and real-time inference.

  • DSVT is a deployment-friendly Transformer-only backbone for 3D perception.
  • Dynamic sparse window attention partitions sparse voxels into equal-sized, window-bounded subsets processed in parallel without customized CUDA operations.
  • DSVT achieves state-of-the-art performance across various 3D perception benchmarks with real-time running speed after NVIDIA TensorRT acceleration.

A. Proof of Non-overlap Set Partition

The appendix proves that dynamic set partitioning produces local voxel sets that are non-overlapping and complete. It also establishes that each set has either floor(N/S) or floor(N/S)+1 valid voxels.

  • Dynamic Set Partition: Dynamic set partition uses N non-empty voxels and maximum set allocation τ to determine the number S of local subsets.The definition identifies N and τ as the inputs governing the partition.
  • Non-overlap: Theorem A.2 proves that any two local sets do not overlap, including neighboring sets under the index construction.
  • Completeness: Theorem A.3 proves completeness: the union of Q0 through QS−1 equals the full index set U = {0, 1, ..., N −1}.
  • Equivalent Set Sizes: Theorem A.4 shows that every subset contains either floor(N/S) or floor(N/S)+1 valid and unique voxels.The proof treats the cases where the remainder is zero or nonzero.

B. Implementation Details

This section provides implementation details for the network architecture, ablation baselines, and training schemes.

  • The appendix documents the network architecture used by the implementation.
  • It specifies the baselines used in ablation studies.
  • It also describes the training schemes used for the experiments.

B.1. Network Architecture.

The architecture uses pillar- and voxel-based DSVT variants with rotated set partitioning and sparse-transformer components for outdoor 3D perception.

  • DSVT-P is a single-stride pillar-based sparse backbone with four DSVT blocks and alternating X-Axis and Y-Axis set partitions.Each block contains two DSVT layers with different set partitioning configurations.
  • DSVT-V is a voxel-based variant that follows the pillar-based framework and splits along the Z-Axis.Its input voxel size is (0.32m, 0.32m, 0.1875m).
  • For nuScenes, DSVT-P uses window size (30, 30, 1), maximum set token count 90, 8 attention heads, 128 input channels, and 256 hidden channels.

B.2.1 ResBackbone1x

The ResBackbone1x comparison preserves a single-stride sparse-convolution design and evaluates DSVT against it under matched settings and additional hyperparameter studies.

  • ResBackbone1x: ResBackbone1x replaces downsampling SparseConv blocks with conventional SubMConv to maintain a single-stride architecture.It is built upon Spconv 2.0 and uses the same network designs as VoxelResBackBone8x except for this replacement.
  • ResBackbone1x: +1.78 L2 mAPH is achieved by DSVT over the strong ResBackbone1x baseline.ResBackbone1x is otherwise matched to DSVT in detection head, loss functions, post-processing, and input voxel size.
  • Hyperparameter studies: The models are trained on 20% Waymo training data for 30 epochs in the reported set-size and network-depth experiments.Table 11 examines set size, while Table 12 examines network depth.
  • Hyperparameter studies: Increasing set size improves performance from 24 to 36 in DSVT-P and from 36 to 48 in DSVT-V, while very large sizes slightly decrease mAP/mAPH.

D. Qualitative Results

Qualitative Waymo validation results show DSVT locating 3D objects accurately in large scenes with sparse points, while its design differs from axial attention.

  • DSVT performs well on large scenes and accurately locates 3D objects with sparse points on the Waymo validation set.The paper attributes this qualitative behavior to the Transformer's large receptive field and attention-style 3D pooling's fine-grained geometric information.
  • DSVT's rotated partitioning is designed for parallel sparse-data processing and intra-window fusion, unlike axial attention's dense-data computation-cost reduction.

F. Limitation

The paper's stated limitation is scope: DSVT primarily targets outdoor autonomous-driving point clouds with objects distributed on a 2D ground plane.

  • DSVT mainly focuses on outdoor autonomous-driving 3D object detection involving cars, pedestrians, and cyclists distributed on a 2D ground plane.The paper identifies more general-purpose 3D backbones as an open problem.
Loading 2301.06051v2…