PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Sparse 3D Object Detection

Lue Fan, Feng Wang, Naiyan Wang, Zhaoxiang Zhang

arXiv:2207.10035v2cs.CVcs.RO

TL;DR

Long-range LiDAR detection is challenged by dense feature maps whose costs grow quadratically with perception range and by missing object-center features in sparse architectures. FSD combines sparse voxel encoding, instance grouping, and sparse instance recognition to predict boxes without dense maps. It achieves state-of-the-art performance on Waymo and Argoverse 2, while being 2.4× faster than dense detectors on the 200-meter Argoverse 2 setting.

  • Problem

    Dense LiDAR detectors scale poorly to long-range perception, while sparse voxel architectures suffer from missing features at empty object centers.

  • Method

    FSD combines a sparse voxel encoder with center voting, instance grouping, and instance-wise feature extraction and prediction through SIR.

  • Results

    2.4× faster than state-of-the-art dense detectors while achieving state-of-the-art performance on the 200-meter Argoverse 2 dataset.

  • Takeaways & Limitations

    FSD provides a fully sparse detector for efficient long-range LiDAR object detection across Waymo and Argoverse 2 benchmarks.

  • Takeaways & Limitations

    The grouping strategy may be improved with more elaborate designs, which the authors leave for future work.

Abstract

from arXiv · show

As the perception range of LiDAR increases, LiDAR-based 3D object detection becomes a dominant task in the long-range perception task of autonomous driving. The mainstream 3D object detectors usually build dense feature maps in the network backbone and prediction head. However, the computational and spatial costs on the dense feature map are quadratic to the perception range, which makes them hardly scale up to the long-range setting. To enable efficient long-range LiDAR-based object detection, we build a fully sparse 3D object detector (FSD). The computational and spatial cost of FSD is roughly linear to the number of points and independent of the perception range. FSD is built upon the general sparse voxel encoder and a novel sparse instance recognition (SIR) module. SIR first groups the points into instances and then applies instance-wise feature extraction and prediction. In this way, SIR resolves the issue of center feature missing, which hinders the design of the fully sparse architecture for all center-based or anchor-based detectors. Moreover, SIR avoids the time-consuming neighbor queries in previous point-based methods by grouping points into instances. We conduct extensive experiments on the large-scale Waymo Open Dataset to reveal the working mechanism of FSD, and state-of-the-art performance is reported. To demonstrate the superiority of FSD in long-range detection, we also conduct experiments on Argoverse 2 Dataset, which has a much larger perception range ($200m$) than Waymo Open Dataset ($75m$). On such a large perception range, FSD achieves state-of-the-art performance and is 2.4$\times$ faster than the dense counterpart. Codes will be released at https://github.com/TuSimple/SST.

1 Introduction

Long-range LiDAR detection is difficult because dense feature maps scale quadratically with perception range, while fully sparse designs face missing center features. FSD addresses this with sparse instance recognition and reports state-of-the-art results, including faster long-range inference.

  • Motivation: Dense detectors become impractical beyond 200 meters because their feature-map computation and spatial costs grow quadratically with perception range.They perform well on shorter benchmarks with ranges below 75 meters.
  • Motivation: Center Feature Missing occurs when sparse voxel encoders leave object centers empty, weakening center-based or anchor-based predictions.Existing detectors address this by converting sparse voxels into dense BEV maps and diffusing features with convolutions.
  • Related challenge: Purely point-based detectors avoid dense maps but are limited by expensive neighborhood queries and aggressive downsampling that loses foreground information.These limitations have prevented many point-based methods from reaching state-of-the-art performance on large-scale point clouds.
  • FSD: FSD combines a sparse voxel encoder with center voting, instance grouping, and instance-level feature extraction and prediction through SIR.Grouping points before prediction reduces regression-target variance compared with predicting whole boxes from individual parts.
  • Results: 2.4× faster than state-of-the-art dense detectors while achieving state-of-the-art performance on the 200-meter Argoverse 2 setting.FSD also achieves state-of-the-art performance on the Waymo Open Dataset.

2 Related Work

Prior LiDAR detectors mainly use dense or semi-dense voxel representations, while point-based detectors remain fully sparse but struggle with large-scale point clouds.

  • Voxel-based dense detectors: Dense voxel detectors convert sparse point clouds into dense feature maps for voxel extraction or BEV processing.VoxelNet uses dense 3D convolution, while PIXOR and PointPillars use dense BEV convolution.
  • Voxel-based semi-dense detectors: Semi-dense detectors combine sparse 3D voxel features with dense BEV maps to enlarge receptive fields and support 2D detection heads.SECOND exemplifies this sparse-to-dense design.
  • Point-based sparse detectors: Point-based detectors are naturally fully sparse, but neighborhood queries remain unaffordable for scenes containing more than 100K points.Consequently, large-scale benchmarks remain dominated by voxel-based dense or semi-dense detectors.

3.1 Overall Architecture

FSD builds a fully sparse detection pipeline by extracting voxel features, grouping voted foreground points into instances, predicting proposals, and refining groups with a second SIR module.

  • Overall Architecture: FSD first uses a sparse voxel encoder to extract voxel features and vote object centers.This supplies the features and voting results used for subsequent instance formation.
  • Overall Architecture: Instance Point Grouping organizes foreground points into instances using the voting results.The groups provide the units for instance-level recognition.
  • Overall Architecture: SIR extracts instance and point features and generates proposals from the grouping results.The design predicts a whole object box from an instance rather than from isolated object parts.
  • Overall Architecture: A second SIR module uses proposals to correct point grouping and refine the proposals.This module is denoted SIR2.

3.2 Instance Point Grouping

FSD forms instances by classifying and voting from sparse voxel features, then connecting nearby voted centers into components that share group identities.

  • Classification and Voting: FSD concatenates voxel features with point-to-voxel-center offsets before foreground classification and center voting.The voting head predicts offsets from foreground points to their corresponding object centers.
  • Instance-level operators: Dynamic broadcast/pooling computes center-to-neighbor offsets and updates point features in parallel across instances.The operation relies on non-overlapping groups so each point belongs uniquely to one group.
  • Classification and Voting: L1 loss and Focal Loss supervise center voting and semantic foreground classification, respectively.These are denoted voting loss L_vote and semantic classification loss L_sem.
  • Connected Components Labeling: Connected Components Labeling links predicted centers whose pairwise distance is below a threshold, assigning each connected component as an instance.All points voted to the same component receive a shared group ID.

3.3 Sparse Instance Recognition

Sparse Instance Recognition groups points into non-overlapping instances, extracts instance-level features through dynamic broadcast/pooling, and makes one sparse prediction per group.

  • Preliminaries: Dynamic Broadcast/Pooling: Dynamic pooling aggregates point features within each group into a group feature, while dynamic broadcast distributes group features back to their member points.These operations are efficient because broadcasting is indexing and both operations support parallel processing with dynamic group sizes.
  • Preliminaries: Dynamic Broadcast/Pooling: Non-overlapping 3D instance groups satisfy the prerequisite for dynamic broadcast/pooling because each point belongs uniquely to one group.The instance-based grouping design makes the groups spatially non-overlapping.
  • Formulation of Sparse Instance Recognition: SIR uses group centers, pair-wise features, and group feature aggregation to construct point-based instance operators.The group center is the centroid of voted centers, and pair-wise features include relative coordinates and concatenated group-point features.
  • Formulation of Sparse Instance Recognition: SIR adopts a VFE-style stack of basic layers that repeatedly transforms point-wise features using dynamic broadcast, point processing, and pooling.The formulation supports parallel feature extraction for all instances and can be extended across multiple SIR layers.
  • Formulation of Sparse Instance Recognition: SIR predicts bounding boxes and class labels from aggregated group features, producing one sparse prediction for each non-overlapping group.Groups whose centers fall inside ground-truth boxes are positive samples, with regression predicting center offsets, sizes, and orientations.

3.4 Group Correction

Group correction uses SIR proposals to repair incorrect initial point grouping and then applies a second SIR module to refine the corrected groups and boxes.

  • Group Correction: Initial instance grouping may miss foreground points or include background clutter, so SIR proposals define corrected groups independently of previous group IDs.All points inside a proposal are assigned to the corresponding corrected group.
  • Group Correction: SIR2 refines each proposal by predicting its residual relative to the corresponding ground-truth box.Proposal-to-boundary offsets are added as point features so SIR2 can use proposal size and location.
  • Group Correction: The total training objective combines semantic, voting, regression, classification, residual, and IoU losses.The formulation omits normalization factors for simplicity.

3.5 Discussion

FSD differs from VoteNet by performing further instance-level feature extraction after voting and by retaining fine-grained features without fixed-point downsampling.

  • Discussion: After voting, FSD uses dynamic broadcast/pooling in SIR for further instance-level feature extraction, whereas VoteNet directly aggregates features around voted centers.The paper reports that FSD therefore extracts more powerful instance features.
  • Discussion: FSD processes any number of input points without fixed-point downsampling, avoiding the information loss associated with VoteNet’s aggressive scene downsampling.The paper connects this design to fine-grained point representation and efficient processing of large-scale point clouds.

4 Experiments

Experiments on Waymo and Argoverse 2 evaluate FSD against dense and sparse alternatives, probe treatments for Center Feature Missing, and measure long-range resource scaling. FSD achieves strong detection performance while retaining favorable efficiency as perception range and scene sparsity increase.

  • Comparison to State-of-the-art Methods: FSD achieves state-of-the-art performance among mainstream detectors on the Waymo Open Dataset validation split.The comparison uses single-frame point clouds without test-time augmentation or model ensembling.
  • Study of Treatments to Center Feature Missing: FSDplain suffers from high regression-target variance and low-quality predictions from hard voxels.It directly predicts boxes from individual voxels after sparse voxel encoding.
  • Study of Treatments to Center Feature Missing: Feature diffusion is sub-optimal for large objects because features may not reach centers or may be too weak for accurate predictions.SSTcenter performs much worse than FSDplain on large vehicles despite sharing the same attention-based sparse voxel encoder.
  • Study of Treatments to Center Feature Missing: FSDplain performs worst on normal-size vehicles, whereas FSDnogc and FSD outperform CenterPoint-PP especially on large vehicles.The results indicate that assignment strategy and object size affect the relative performance of Center Feature Missing treatments.
  • Study of Treatments to Center Feature Missing: SIR effectively resolves Center Feature Missing and outperforms feature diffusion on large vehicles.FSDnogc and SSTcenter share the same sparse voxel encoder, while FSDnogc replaces SSTcenter’s dense component with SIR.
  • Long-range Detection: FSD’s memory and latency increase roughly linearly with input-point count and only slightly as perception range expands, unlike dense detectors’ dramatic increase.On Argoverse 2, FSD also shows significantly lower memory and latency after uninterested regions are removed, with acceptable precision loss.

5 Conclusion

FSD targets efficient long-range 3D object detection with a fully sparse architecture and achieves competitive or state-of-the-art performance across the reported datasets. The authors note that more elaborate point-grouping strategies may improve performance in future work.

  • Conclusion: FSD is a fully sparse 3D object detector designed for efficient long-range object detection.It uses a point-based Sparse Instance Recognition module to address center feature missing.
  • Conclusion: FSD achieves competitive performance on Waymo Open Dataset and state-of-the-art performance on the long-range Argoverse 2 dataset, with faster inference than previous detectors.
  • Limitations: A more elaborately designed grouping strategy may improve performance, but its development is left for future work.
Loading 2207.10035v2…