PaperlayerView exact revision on arXiv

Source-linked AI summary

Embracing Single Stride 3D Object Detector with Sparse Transformer

Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, Zhaoxiang Zhang

arXiv:2112.06375v1cs.CV

TL;DR

LiDAR-based 3D detectors often inherit multi-stride downsampling despite much smaller relative object sizes than 2D scenes. The paper proposes SST, a single-stride sparse transformer that preserves resolution while addressing receptive field and computation. SST achieves state-of-the-art Waymo performance, including 83.8 LEVEL 1 AP for pedestrians on validation.

  • Problem

    3D detectors commonly downsample quantized point-cloud features despite the much smaller relative object sizes in 3D scenes, risking information loss for tiny objects.

  • Method

    SST preserves original resolution with a single-stride architecture and uses sparse regional attention to provide receptive field efficiently on sparse voxelized data.

  • Results

    SST achieves state-of-the-art performance on the Waymo Open Dataset, including 83.8 LEVEL 1 AP for pedestrians on the validation split.

  • Takeaways & Limitations

    The experiments support using smaller strides for 3D detection, while single-stride SST delivers strong performance on challenging small pedestrian objects.

  • Takeaways & Limitations

    The efficient-validation experiments use 20% of the data, a setting adopted for validation efficiency.

Abstract

from arXiv · show

In LiDAR-based 3D object detection for autonomous driving, the ratio of the object size to input scene size is significantly smaller compared to 2D detection cases. Overlooking this difference, many 3D detectors directly follow the common practice of 2D detectors, which downsample the feature maps even after quantizing the point clouds. In this paper, we start by rethinking how such multi-stride stereotype affects the LiDAR-based 3D object detectors. Our experiments point out that the downsampling operations bring few advantages, and lead to inevitable information loss. To remedy this issue, we propose Single-stride Sparse Transformer (SST) to maintain the original resolution from the beginning to the end of the network. Armed with transformers, our method addresses the problem of insufficient receptive field in single-stride architectures. It also cooperates well with the sparsity of point clouds and naturally avoids expensive computation. Eventually, our SST achieves state-of-the-art results on the large scale Waymo Open Dataset. It is worth mentioning that our method can achieve exciting performance (83.8 LEVEL 1 AP on validation split) on small object (pedestrian) detection due to the characteristic of single stride. Codes will be released at https://github.com/TuSimple/SST

1. Introduction

LiDAR-based 3D scenes contain much smaller relative objects than 2D images, yet 3D detectors commonly inherit multi-stride designs. The paper investigates single-stride detection and proposes SST to preserve resolution while addressing receptive-field and computation challenges.

  • Motivation: Waymo’s 150m×150m perception range makes pedestrians as small as 1m long, creating a key challenge for 3D detection.The paper compares this scale to an 8 × 8 pixel object in a 1200×1200 image.
  • Motivation: 3D object scales are smaller and more concentrated than 2D image scales, but 3D detectors widely inherit multi-scale downsampling and upsampling architectures.The paper attributes the concentrated 3D distribution to non-projective voxelization.
  • Single-stride challenge: Removing downsampling preserves original resolution but creates increased computation cost and reduced receptive field.These issues constrain real-time applicability and object-recognition capability, respectively.
  • Single-stride challenge: Single-stride CNN designs struggle to provide sufficient receptive field with acceptable computation cost; dilation can miss fine-grained details and larger kernels are expensive.The paper frames these constraints as a three-way design dilemma involving stride, receptive field, and computation.
  • Proposed method: SST uses local Sparse Regional Attention to capture large context on sparse voxelized data without expensive computation.Local self-attention is applied within partitioned voxel regions and stacked into the transformer-style network.
  • Results: 83.8 LEVEL 1 AP is reported for pedestrian detection on the Waymo validation split, alongside state-of-the-art performance on Waymo overall.The paper links the pedestrian result to the characteristic of the single-stride design.

2. Related Work

Prior work spans point-, voxel-, and range-view representations, transformer-based visual recognition, and small-object detection. The paper positions 3D small-object detection as insufficiently specialized compared with established 2D approaches.

  • 3D LiDAR-based detection: Point-based methods use PointNet families for feature learning from small regions of irregular points, while voxel-based convolutional representations remain popular.Range-view methods offer computational advantages, especially for long-range LiDAR sensors.
  • 3D LiDAR-based detection: Range-view and hybrid approaches complement point-based and voxel-based representations in autonomous-driving point-cloud learning.Hybrid methods investigate combinations of different representation types.
  • Transformers in visual recognition: Vision transformers use patch sequences, data-efficient training, or local attention to support visual recognition.The related work cites ViT, DeiT, and Swin-Transformer as representative examples.
  • Small-object detection: Small-object detection is challenging in 2D, where methods increase feature resolution, use scale-aware training, or apply strong augmentation while retaining multi-stride architectures.The paper states that no method was specialized for small-object detection in 3D space.

3. Discussion of Network Stride

Pilot experiments show that reducing network stride improves LiDAR-based 3D detection, but removing downsampling entirely exposes receptive-field limitations. These results motivate single-stride designs that address receptive field and computational overhead together.

  • Experimental setup: The pilot study compares PointPillars variants that differ only in network stride, using the same hyper-parameters except reduced convolution filter numbers.The stage strides are {1, 2, 4, 8}, {1, 2, 4, 4}, {1, 2, 2, 2}, and {1, 1, 1, 1} from D3 to D0; all outputs are upsampled to the original resolution in the detection head.
  • Effect of stride: Smaller strides improve performance across all three classes from D3 to D1, with a significant boost from D2 to D1.The result supports the study’s motivation that smaller strides are better for 3D detection.
  • Effect of stride: From D1 to D0, vehicle performance drops significantly, pedestrian performance drops slightly, and cyclist performance continues to improve.The authors conjecture that D0’s limited receptive field hinders vehicle improvement because pedestrians and cyclists are smaller.
  • Receptive field: Dilated convolution improves vehicle performance but decreases pedestrian and cyclist performance, indicating enlarged receptive fields can miss fine-grained details.The passage reports this trade-off in the pilot study’s comparison of receptive-field designs.
  • Receptive field: Larger 5×5 kernels consistently improve all three classes but incur the highest latency, revealing a computation–receptive-field trade-off.The study identifies sufficient receptive field as crucial while noting the computational cost of larger kernels.
  • Implications: Single-stride architecture has strong potential, but feasibility depends on addressing receptive-field shrinkage while reducing computational overhead.These are the two design requirements summarized from the experiments.

4. Methodology

SST maintains single-stride voxel resolution while using sparse regional attention to provide receptive field and computational efficiency. Region shifting extends context across regions, and sparse tokens are later restored to dense maps for existing detection heads.

  • Architecture: SST voxelizes point clouds, treats voxel features as tokens, and processes them through repeated regional grouping and Sparse Regional Attention blocks.The architecture eventually recovers a dense feature map for detection.
  • Regional attention: Regional Grouping restricts self-attention to tokens sharing a physical region, preserving local receptive field while avoiding global-attention overhead.Regions are fixed-size and non-overlapping.
  • Regional attention: SRA applies transformer operations to sparse regional voxel-token sets and computes only voxels containing actual LiDAR points.The operation uses positional encoding, multi-head self-attention, and layer normalization.
  • Efficient implementation: Regions are batched by similar token counts and padded tokens are masked, enabling parallel SRA computation without affecting valid tokens.This supports efficient implementation in common deep-learning frameworks.
  • Region Shift: Region Shift moves regions by half their dimensions before a second grouping and SRA, helping aggregate context for objects spanning multiple regions.The shift is applied along the x, y, and z dimensions.
  • Detection integration: SST returns sparse voxel tokens to dense maps, fills empty locations with zeros, and adds two 3 × 3 convolutions to fill holes near object centers.The detector head and losses follow PointPillars, using SSD, smooth L1 localization, focal classification, and direction losses.

5. Experiments

Experiments on Waymo Open Dataset evaluate SST across detector variants, object classes, ranges, receptive-field demands, localization thresholds, attention behavior, and region sizes. SST is especially strong for pedestrian detection and benefits from preserving fine-grained observations.

  • Experimental Setup: Waymo Open Dataset contains 1150 sequences, with 798 for training, 202 for validation, and 150 for testing.Each frame covers a 150m × 150m scene.
  • Experimental Setup: SST uses six Sparse Regional Attention blocks in a PointPillars-based model, with variants spanning one or three frames and one- or two-stage detection.The two-stage variants use LiDAR-RCNN for proposal refinement.
  • Efficiency: SST achieves competitive vehicle performance with a lightweight second stage and outperforms alternatives with relatively low latency.SparsePillars-SS suffers from more severe receptive-field shrinkage in the vehicle class.
  • Deep Investigation of Single Stride: 12.8 AP separates SST 1f from the PointPillars counterpart for pedestrians at 0m–30m, while SST gains 10.4 AP from one to three frames at 50m–inf.The corresponding RSN long-range improvement is 6.4 AP.
  • Deep Investigation of Single Stride: SST outperforms the PointPillars baseline for vehicles longer than 8m, supporting adequate receptive fields for large objects.The authors attribute this to the attention mechanism.
  • Localization Quality: SST 1f exceeds MVF++ under the stricter vehicle IoU threshold, and SST TS 3f surpasses 3DAL for pedestrians at both thresholds using three frames.The stricter thresholds are 0.8 for vehicles and 0.6 for pedestrians.
  • Attention and Ablation: Learned attention covers object parts while discriminating semantic classes and instances; performance is robust to region size and can improve with fewer layers.The largest local region performs best for pedestrian detection, and fewer layers can slightly improve pedestrian performance.

6. Conclusion and Limitations

The paper argues that 3D detectors do not need downsampling and presents SST as a single-stride alternative using sparse regional attention. It reports state-of-the-art Waymo performance, especially for pedestrians, while acknowledging slightly higher memory use.

  • Conclusion: Experiments empirically show that 3D object detectors do not really need downsampling.The conclusion challenges the inherited multi-stride backbone design for point-cloud data.
  • Conclusion: Sparse regional attention addresses insufficient receptive fields while avoiding expensive computation in the single-stride network.Stacked sparse attention modules form the Single-stride Sparse Transformer.
  • Conclusion: SST achieves state-of-the-art performance on the Waymo Open Dataset and remarkable results on the challenging pedestrian class.The conclusion attributes the pedestrian performance to the single-stride structure.
  • Limitations: SST uses slightly more memory than baseline models without elaborated optimization.The authors identify a more memory-friendly model as future work.

A. Submission on Test Server

On the Waymo test split, SST is comparable with leading methods for vehicles and substantially stronger for pedestrians. The reported comparisons use multi-frame settings from the official leaderboard.

  • Overall Comparison: Vehicle performance is comparable with the three selected competitive methods, while pedestrian performance significantly outperforms them.Only the best SST model is reported because of the Waymo test-server submission frequency limit.
  • Vehicle Detection: SST TS 3f achieves 80.99/80.62 and 73.08/72.74 for vehicle detection, compared with the listed multi-frame methods.The table reports paired performance values for the evaluated vehicle metrics.
  • Pedestrian Detection: SST TS 3f achieves 83.05/79.38 and 76.65/73.14 for pedestrian detection, exceeding the listed competing methods.These results are reported on the Waymo Open Dataset test split.

B.1. Insufficient Receptive Field of Submanifold Sparse Convolution (SSC)

Submanifold sparse convolution has a restricted receptive field because it does not fill empty voxels, limiting information flow between sparse voxels. Experiments show that enlarging its kernel helps but does not close the performance gap.

  • Receptive-Field Limitation: SSC cannot reach some distant active voxels that standard convolution reaches after two 3 × 3 layers.The difference arises because standard convolution aggregates through empty voxels, whereas SSC preserves sparsity.
  • Experimental Evidence: Replacing 3 × 3 standard convolutions with SSC drops vehicle 3D AP from 64.69 to 51.57.Increasing the SSC kernel to 5 × 5 and 7 × 7 raises AP to 55.40 and 56.77, respectively, but remains below the original variant.
  • Experimental Evidence: Larger SSC kernels partially recover performance but leave a substantial gap, demonstrating that receptive-field insufficiency is not solved by kernel expansion alone.The experiment uses vehicles because they require sufficient receptive field for detection.

B.2. Downsampling/Upsampling in Sparse Data

Downsampling and upsampling are difficult to transfer to sparse point-cloud data without losing sparsity or creating unresolved token-reconstruction problems. SST avoids these operations, while leaving several broader architectural improvements for future work.

  • Sparse convolution downsampling and upsampling rapidly reduce data sparsity, increasing computational overhead.
  • Token merging faces unresolved questions about merging spatially scattered tokens and efficiently recovering multiple locations during upsampling.
  • SST bypasses sparse token downsampling and upsampling by adopting a single-stride architecture.
  • Potential Improvements: The basic PointPillars framework leaves substantial room for further performance improvements.
  • Potential Improvements: SST’s lightweight LiDAR-RCNN second stage is inferior to elaborately designed RCNNs that reuse first-stage features, although those methods could be integrated.

D. Computational Complexity Compared with Convolutions

SST and convolutional architectures both have computational complexity linear in input-scene size, but SST additionally exploits sparse voxel occupancy through regional grouping. On Waymo, the measured occupancy ratio is approximately 0.09.

  • SST’s efficiency advantage comes from exploiting point-cloud sparsity and regional grouping.
  • For an h×w scene, convolutional layers with kernel size k×k and channel number C have complexity specified by Equation A.
  • SRA includes the average sparsity factor S, which is approximately 0.09 on the Waymo Open Dataset under the paper’s voxelization.
  • Both convolution and SRA complexities are O(hw), making them linear in the input-scene scale.
  • The complexity analysis assumes non-empty voxels are uniformly distributed, making the calculation approximate.
Loading 2112.06375v1…