PaperlayerView exact revision on arXiv

Source-linked AI summary

Voxel Transformer for 3D Object Detection

Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, Chunjing Xu

arXiv:2109.02497v2cs.CV

TL;DR

VoTr addresses limited context in voxel-based 3D detection by using sparse voxel Transformer modules with efficient attention and querying. It consistently outperforms convolutional baselines on Waymo and KITTI while maintaining computational efficiency.

  • Problem

    Conventional 3D convolutional backbones have limited receptive fields and cannot efficiently capture the large context needed for voxel-based 3D object recognition and localization.

  • Method

    VoTr uses sparse and submanifold voxel modules with Local and Dilated Attention plus Fast Voxel Query to model long-range relationships efficiently.

  • Results

    VoTr consistently outperforms convolutional baselines, reaching 74.95% LEVEL 1 mAP for vehicle on Waymo and 82.09% mAP for moderate car on KITTI.

  • Takeaways & Limitations

    VoTr provides a general Transformer-based 3D backbone applicable to most voxel-based detectors while maintaining computational efficiency.

Abstract

from arXiv · show

We present Voxel Transformer (VoTr), a novel and effective voxel-based Transformer backbone for 3D object detection from point clouds. Conventional 3D convolutional backbones in voxel-based 3D detectors cannot efficiently capture large context information, which is crucial for object recognition and localization, owing to the limited receptive fields. In this paper, we resolve the problem by introducing a Transformer-based architecture that enables long-range relationships between voxels by self-attention. Given the fact that non-empty voxels are naturally sparse but numerous, directly applying standard Transformer on voxels is non-trivial. To this end, we propose the sparse voxel module and the submanifold voxel module, which can operate on the empty and non-empty voxel positions effectively. To further enlarge the attention range while maintaining comparable computational overhead to the convolutional counterparts, we propose two attention mechanisms for multi-head attention in those two modules: Local Attention and Dilated Attention, and we further propose Fast Voxel Query to accelerate the querying process in multi-head attention. VoTr contains a series of sparse and submanifold voxel modules and can be applied in most voxel-based detectors. Our proposed VoTr shows consistent improvement over the convolutional baselines while maintaining computational efficiency on the KITTI dataset and the Waymo Open dataset.

1. Introduction

VoTr introduces a Transformer-based voxel backbone to address the limited context of 3D convolutional backbones while handling sparse, numerous voxels efficiently. Its specialized voxel modules and attention mechanisms improve detection over convolutional baselines on Waymo and KITTI.

  • Motivation: 3D convolutional backbones have limited receptive fields, restricting context capture for sparse objects and making receptive-field expansion costly.A representative KITTI configuration reaches only (3.65m, 3.65m, 7.3m), while enlarging voxel size, kernels, stride, or depth introduces accuracy or computation trade-offs.
  • Motivation: Standard self-attention is difficult for voxels because non-empty voxels are sparse but numerous, requiring efficient attention over selected voxel locations.Non-empty voxels occupy less than 0.1% of Waymo’s voxel space, yet nearly 90k occur per frame.
  • Approach: VoTr is a Transformer-based backbone designed as a substitute for conventional 3D convolutional backbones in most voxel-based detectors.It uses a series of sparse and submanifold voxel modules to process voxel features.
  • Approach: Submanifold voxel modules attend strictly to non-empty voxels, while sparse voxel modules can produce features at empty locations.This design preserves geometric structure while allowing the represented non-empty voxel space to expand.
  • Approach: Local Attention preserves neighboring detail, Dilated Attention expands the attention range with few attending voxels, and Fast Voxel Query accelerates lookup.Fast Voxel Query uses a GPU-based hash table to store and retrieve non-empty voxels efficiently.

2. Related Work

Prior 3D detectors follow point-based or voxel-based approaches, while VoTr extends voxel-based detection with a Transformer backbone. Related Transformer methods in vision and point clouds motivate this architectural direction.

  • 3D Detection: Point-based detectors operate directly on raw point clouds, whereas voxel-based detectors rasterize point clouds into regular voxel grids before feature extraction.Point-based methods face costly sampling and aggregation on irregular points, while voxel-based methods benefit from regular structure and efficiency.
  • 3D Detection: Voxel-based systems commonly use 3D and 2D convolutional networks to extract voxel features and generate 3D proposals.Examples include VoxelNet, SECOND, HVNet, and PV-RCNN, which use different voxel or keypoint representations.
  • VoTr: VoTr is presented as a plug-in Transformer backbone for most voxel-based detectors, extending the voxel-based stream with self-attention.Its architecture is described as a series of sparse and submanifold voxel modules.
  • Transformer Methods: Transformer architectures have been applied to image classification, detection, segmentation, and 3D point clouds through attention-based backbones or point operators.Examples include Vision Transformer, DETR, SETR, MaX-DeepLab, and Point Transformer.

3. Voxel Transformer

VoTr is a voxel-based Transformer backbone that builds long-range relationships among sparse voxel features while preserving geometric structure and supporting flexible voxel-space expansion. It combines sparse and submanifold voxel modules with Local and Dilated Attention and Fast Voxel Query for efficient attention computation.

  • Overall Architecture: VoTr consists of multiple sparse and submanifold voxel modules whose features are projected to BEV for proposal generation and can support second-stage RoI refinement.The backbone includes three sparse voxel modules that downsample voxel grids and provide features at different positions and resolutions.
  • Voxel Transformer: VoTr applies multi-head self-attention to voxel features to construct long-range relationships among non-empty voxels.For each querying voxel, an attention range is determined before attending features are aggregated.
  • Voxel Transformer: Submanifold voxel modules operate only at input non-empty locations, preserving the original 3D structure.Their Transformer-style blocks include self-attention, a feed-forward layer, residual connections, batch normalization, and an additional linear projection.
  • Voxel Transformer: Sparse voxel modules can produce features at empty locations, expanding the original non-empty voxel space and supporting voxel downsampling.Because empty voxels lack input features, their query embeddings are approximated from attending features, using max pooling in this paper.
  • Attention Mechanisms: Local Attention searches neighboring voxel positions to retain fine-grained structure, whereas Dilated Attention progressively enlarges search steps to obtain broader context with few attending voxels.The attention range is designed to cover neighbors, reach far spatially, and limit attending voxels to avoid heavy computational overhead.
  • Efficient Attention Mechanism: Fast Voxel Query uses a GPU hash table to map queried voxel indices to non-empty feature indices before gathering features for multi-head attention.Hash lookup avoids scanning all non-empty voxels and uses a query-dependent cost O(NΩ), where NΩ ≪ Nsparse.

4. Experiments

VoTr is evaluated on Waymo Open and KITTI through detector replacements, benchmark comparisons, ablations, and efficiency measurements. Results show improvements over convolutional baselines while retaining practical computation efficiency.

  • Experimental Setup: Experiments evaluate VoTr on the Waymo Open and KITTI datasets using single-stage and two-stage detector architectures.VoTr-SSD replaces SECOND’s backbone, while VoTr-TSD replaces PV-RCNN’s first-stage 3D convolutional backbone; other configurations are kept unchanged for fair comparison.
  • Waymo Open Dataset: 1.05% and 3.26% LEVEL 1 mAP improvements result from replacing convolutional backbones with VoTr in SECOND and PV-RCNN on Waymo Open.VoTr variants also improve far-range LEVEL 1 mAP, with gains of 1.42% and 1.72% for VoTr-SSD and 3.37% and 4.83% for VoTr-TSD across the reported ranges.
  • KITTI Dataset: 2.29% mAP and 0.66% mAP improvements are reported for VoTr-SSD and VoTr-TSD on the moderate car class of the KITTI validation split.For the hard car class, VoTr-TSD achieves 79.14% mAP, while the validation results remain consistent with Waymo findings.
  • Ablation Studies: 2.79% moderate mAP gain is obtained with Dilated Attention compared with using only Local Attention.The ablation attributes this gain to the larger receptive fields provided by Dilated Attention.
  • Ablation Studies: 8.52% mAP drops at dropout probability 0.3, whereas increasing attending voxels from 24 to 48 improves performance by 1.19%.These ablations report detrimental effects from module dropout and richer context from involving more attending voxels.
  • Efficiency Analysis: VoTr reduces model parameters by 0.5M and maintains 14.65 Hz inference speed, adding about 20 ms latency per frame to the single-stage detector.The parameter reduction is attributed to linear projection layers, while the reported speed depends on the designed attention mechanisms and Fast Voxel Query.

5. Conclusion

VoTr is presented as a general Transformer-based backbone for voxel-based 3D detectors. Its sparse and submanifold modules, efficient attention mechanisms, and Fast Voxel Query support self-attention on sparse voxel structures.

  • 5. Conclusion: VoTr is a general Transformer-based 3D backbone applicable to most voxel-based 3D detectors.The architecture consists of sparse and submanifold voxel modules for efficient self-attention on sparse voxels.
  • 5. Conclusion: VoTr performs self-attention on sparse voxels using special attention mechanisms and Fast Voxel Query.The paper states that these components enable efficient processing of sparse voxel positions.

A. Architecture

The VoTr architecture alternates sparse and submanifold voxel modules across three downsampling stages. Sparse modules are needed after downsampling because voxel centers shift to previously empty locations.

  • A. Architecture: Three stride-2 sparse voxel modules downsample the voxel grids, with each followed by two submanifold voxel modules.The downsampled voxel centers no longer overlap the original centers because voxel size doubles during downsampling.

B. Dilated Attention

Dilated Attention is configured to obtain broad context with few attending voxels. The configurations are shared by VoTr-TSD and VoTr-SSD on KITTI.

  • B. Dilated Attention: A single self-attention layer can obtain large context information with only a few attending voxels using carefully designed Dilated Attention.The same Dilated Attention configurations are used for VoTr-TSD and VoTr-SSD on the KITTI dataset.

C. Qualitative Results

Qualitative results on KITTI and Waymo show VoTr-TSD detecting sparse and incomplete 3D objects using context captured by self-attention.

  • VoTr-TSD qualitatively detects sparse and incomplete 3D objects on KITTI and Waymo Open.The paper attributes this capability to rich context information captured by self-attention.
  • Figure 8 compares KITTI ground-truth boxes in blue with VoTr-TSD predictions in red.
  • Figure 9 compares Waymo Open ground-truth boxes in blue with VoTr-TSD predictions in red.
Loading 2109.02497v2…