PaperlayerView exact revision on arXiv

Source-linked AI summary

SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detection

Chen Chen, Zhe Chen, Jing Zhang, Dacheng Tao

arXiv:2201.01976v1cs.CV

TL;DR

Point-based 3D detectors trail voxel-based competitors partly because conventional down-sampling can preserve too much background and miss useful foreground points. SASA adds binary segmentation and semantics-guided sampling to retain foreground information, improving point-based detectors on KITTI and nuScenes, with comparable performance to state-of-the-art voxel-based methods.

  • Problem

    Point-based 3D detectors lag behind voxel-based methods because conventional set abstraction may retain excessive background information during down-sampling.

  • Method

    SASA adds a binary point-segmentation module and semantics-guided farthest point sampling to prioritize foreground points in set abstraction.

  • Results

    SASA improves single-stage and two-stage point-based detectors on KITTI and nuScenes, with around 2% mAP improvement for the KITTI car class.

  • Takeaways & Limitations

    SASA provides a lightweight, broadly adoptable point-set abstraction design that achieves highly boosted results and comparable performance to state-of-the-art voxel-based methods.

  • Takeaways & Limitations

    A top-K foreground-score sampling alternative selects too many points from easy objects, reducing scene coverage and causing many ground-truth objects to be ignored.

Abstract

from arXiv · show

Although point-based networks are demonstrated to be accurate for 3D point cloud modeling, they are still falling behind their voxel-based competitors in 3D detection. We observe that the prevailing set abstraction design for down-sampling points may maintain too much unimportant background information that can affect feature learning for detecting objects. To tackle this issue, we propose a novel set abstraction method named Semantics-Augmented Set Abstraction (SASA). Technically, we first add a binary segmentation module as the side output to help identify foreground points. Based on the estimated point-wise foreground scores, we then propose a semantics-guided point sampling algorithm to help retain more important foreground points during down-sampling. In practice, SASA shows to be effective in identifying valuable points related to foreground objects and improving feature learning for point-based 3D detection. Additionally, it is an easy-to-plug-in module and able to boost various point-based detectors, including single-stage and two-stage ones. Extensive experiments on the popular KITTI and nuScenes datasets validate the superiority of SASA, lifting point-based detection models to reach comparable performance to state-of-the-art voxel-based methods.

1 Introduction

Point-based 3D detectors lag behind voxel-based methods partly because conventional set abstraction can retain excessive background and miss foreground points. SASA adds point semantics to sampling, improving point-based detection across frameworks and datasets.

  • Point-based 3D detection has stagnated and fails to achieve top performance on related datasets despite flexible receptive fields.
  • Conventional set abstraction may sample excessive background points, causing insufficient foreground coverage and degraded detection.Feature propagation can recover dropped foreground points but incurs heavy memory usage and computational cost.
  • SASA adds binary point segmentation and semantics-guided farthest point sampling to retain more foreground points during down-sampling.S-FPS gives greater preference to positive points than conventional FPS.
  • SASA works with different point-based detection frameworks and improves the competitive car class by around 2% mAP on KITTI while also improving nuScenes results.The module is reported as easy to plug in and applicable to both single-stage and two-stage baselines.

3D Object Detection from Point Clouds.

3D detection methods process point clouds either by converting them into regular grids or by learning features directly from raw points. Point-based methods use set abstraction to sample points and reduce feature dimensions.

  • Grid-based detectors transform unordered 3D points into regular 2D pixels or 3D voxels for convolutional processing.
  • Point-based detectors model point clouds directly from raw points, including fully point-based detection paradigms such as PointRCNN.
  • PointNet-based feature learning uses set-abstraction layers to sample input points for dimension reduction, commonly with farthest point sampling.
  • SASA combines point segmentation, semantics-guided point sampling, and a normal PointNet++ set-abstraction layer.

3 Semantics-Augmented Set Abstraction

SASA augments set abstraction with point-wise foreground semantics and semantics-guided sampling, retaining diverse, object-focused key points for downstream detection.

  • Architecture: SASA combines point segmentation, semantics-guided point sampling, and a normal PointNet++ set-abstraction layer.The segmentation module produces foreground scores used by sampling before local grouping, MLP processing, and max pooling.
  • Point Segmentation Module: A lightweight two-layer MLP classifies each input point as foreground or background and outputs a foreground score.The score is computed from point features and passed through a sigmoid; box annotations derive the segmentation labels.
  • Semantics-guided Farthest Point Sampling: The direct top-K strategy can over-select easily identified objects, leaving distant or occluded instances uncovered and harming detection.S-FPS addresses this coverage problem by combining foreground preference with the global-scene awareness of farthest point sampling.
  • Semantics-guided Farthest Point Sampling: S-FPS selects points using semantics-weighted distances, favoring high-score foreground points while retaining FPS-style scene coverage.Its balance factor γ controls semantic weighting, with γ = 0 recovering vanilla FPS and very large γ approximating top-K selection.
  • Semantics-guided Farthest Point Sampling: S-FPS retains diverse foreground points, increases key-point density in high-score regions, and is less sensitive to distant outliers.These properties provide more object-related information for the subsequent box prediction network.

3DSSD.

The implementation integrates SASA into 3DSSD by replacing F-FPS with S-FPS, while applying it across selected abstraction levels in PointRCNN without changing its backbone structure.

  • 3DSSD: 3DSSD combines FPS and F-FPS to sample half of each layer’s key points, and SASA replaces the F-FPS branch with S-FPS.S-FPS-selected points generate vote points, while FPS-selected points provide context; other sampling settings remain unchanged.
  • 3DSSD: SASA begins at 3DSSD’s level 2 set-abstraction layer because the raw level 1 input cannot produce meaningful semantics.The segmentation loss weights for levels 2 and 3 are set to 0.01 and 0.1.
  • PointRCNN: PointRCNN uses vanilla FPS across its set-abstraction layers, while SASA is applied from level 2 through level 4 with the original backbone and feature-propagation layers retained.The segmentation loss weights for the three levels are 0.001, 0.01, and 0.1.

4 Experiments

Experiments evaluate SASA with 3DSSD and PointRCNN on KITTI and nuScenes, showing consistent gains across point-based detection settings. On KITTI, SASA achieves competitive performance against voxel-based methods, while nuScenes results also improve substantially over 3DSSD.

  • Experimental setup: Experiments evaluate SASA on KITTI and nuScenes using 3DSSD and PointRCNN baselines.The models are implemented with OpenPCDet, and the main state-of-the-art comparison uses 3DSSD with SASA.
  • KITTI results: SASA surpasses existing point-based detectors on KITTI car detection and obtains comparable results to state-of-the-art voxel-based models.Table 1 reports AP on the KITTI test set using 40 recall points.
  • KITTI results: 0.40%, 2.59%, and 2.61% AP gains over 3DSSD are obtained across the three KITTI difficulty levels.The largest improvements occur on the moderate and hard levels.
  • nuScenes and efficiency: 4.6% NDS and 2.4% mAP improvements over 3DSSD are reported on nuScenes, while SASA processes a KITTI sample in around 36ms.S-FPS also avoids pairwise feature-distance computation and its quadratic sampling cost as point counts grow.

4.4 Ablation Study

Ablations show that semantics-guided sampling, rather than segmentation alone, drives SASA’s gains and improves performance across point-based detectors. Properly balancing semantic scores avoids concentrating samples on a few easy instances.

  • Effects of Semantics-guided Point Sampling: S-FPS outperforms F-FPS across all three difficulty levels, with up to 2.30% improvement on the moderate level.S-FPS also hits 1% more ground-truth boxes than F-FPS.
  • Effects of Semantics-guided Point Sampling: Visualization shows S-FPS retains more key points within severely occluded or tiny instances, making hard examples more likely to be detected.
  • Effects of Semantics-guided Point Sampling: Segmentation modules alone have limited detection impact; most improvement comes from the point sampling algorithm.
  • Effects of Semantics Balance Factor: When γ = 100, point recall drops to 91.19% because samples crowd into easily identified instances and create imbalanced object coverage.When γ approaches 0, S-FPS degrades toward vanilla FPS; γ = 1 provides satisfactory performance across all difficulty levels.
  • Compatibility Study: SASA improves PointRCNN detection across all difficulty levels, including a 1.95% AP lead in hard mode.With top-100 RoIs, it covers nearly 2% more ground-truth boxes at IoU 0.7, and the gap approaches 10% with fewer RoIs.

5 Conclusion

SASA incorporates semantic information into PointNet set abstraction to guide point-based backbones toward potential objects. Results on KITTI and nuScenes indicate improved point recall and detection outcomes across multiple point-based detectors.

  • SASA guides point-based backbones by incorporating semantic information into the PointNet set-abstraction stage.
  • Experiments on KITTI and nuScenes indicate higher point recall during down-sampling and better detection outcomes for multiple point-based detectors.
  • The method is compatible with PointNet-based, transformer-based, and graph-neural-network models for model reduction.

A Further Analysis of Point Sampling

The analysis evaluates point sampling layer by layer using point recall and foreground rate, measuring both ground-truth coverage and the share of sampled foreground points.

  • Point recall measures the proportion of ground-truth boxes containing at least one internal sample point.
  • Foreground rate measures the proportion of sampled key points that are foreground points in each set-abstraction layer.

A.1 Analysis of Sampling Performance

Layer-wise results show that SASA substantially increases the foreground content of sampled points, while maintaining strong ground-truth coverage, especially in PointRCNN.

  • 3DSSD: In 3DSSD levels 2 and 3, S-FPS raises foreground rates to 35.23% and 31.24%, compared with about 9% for F-FPS.This provides richer foreground representations for the succeeding box-prediction network.
  • PointRCNN: In PointRCNN’s fourth set-abstraction layer, S-FPS reaches 96.36 recall with 64 key points and 31.2% foreground samples.Vanilla FPS covers 39.62% of ground-truth instances, with only 2.4% of sampled points from foreground.

A.2 Analysis of the Level 2 SA Layer

The section examines whether semantics-guided sampling remains effective in the level 2 SA layer despite potentially inaccurate point semantics from the preceding layer. Tables 5 and 6 analyze layer-wise point sampling on 3DSSD and PointRCNN.

  • Level 2 semantic accuracy: Limited receptive fields in level 1 SA may prevent level 2 segmentation from producing accurate point semantic maps.The concern arises because level 2 point features are inherited from a preceding layer with limited receptive field.
  • Level 2 semantic accuracy: The analysis tests whether S-FPS functions effectively when level 2 point-wise semantics are less accurate.This is the central question posed for the level 2 SA layer.
  • Experimental analysis: Table 5 evaluates layer-wise point sampling on 3DSSD using the KITTI validation split and the car class.The level 2 and level 3 SA layers use fusion sampling to select half of the key points with two sampling algorithms.
  • Experimental analysis: Table 6 evaluates point sampling by layer on PointRCNN using the car class of the KITTI validation split.
Loading 2201.01976v1…