PaperlayerView exact revision on arXiv

Source-linked AI summary

Focal Sparse Convolutional Networks for 3D Object Detection

Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, Jiaya Jia

arXiv:2204.12463v1cs.CVcs.LG

TL;DR

3D sparse features differ in spatial sparsity and importance, challenging convolutions that process all sparse data uniformly. The paper introduces learnable focal sparsity and an RGB-fused variant, achieving leading benchmark performance with lightweight, end-to-end modules.

  • Problem

    Existing sparse convolutions treat sparse 3D features uniformly, although their spatial sparsity and importance vary and regular and submanifold variants have complementary limitations.

  • Method

    Focals Conv predicts position-wise cubic importance to select features for dynamic dilation, while Focals Conv-F additionally uses fused RGB features.

  • Results

    The modules improve 3D detection on KITTI and nuScenes with small complexity overhead and achieve leading nuScenes test performance without bells and whistles.

  • Takeaways & Limitations

    Learnable sparsity with focal points is reported as essential for sophisticated 3D object detection, while the modules remain replaceable and end-to-end trainable.

  • Takeaways & Limitations

    Multimodal detection requiring multiple high-resolution views increases computation cost, despite a simplified image branch.

Abstract

from arXiv · show

Non-uniformed 3D sparse data, e.g., point clouds or voxels in different spatial positions, make contribution to the task of 3D object detection in different ways. Existing basic components in sparse convolutional networks (Sparse CNNs) process all sparse data, regardless of regular or submanifold sparse convolution. In this paper, we introduce two new modules to enhance the capability of Sparse CNNs, both are based on making feature sparsity learnable with position-wise importance prediction. They are focal sparse convolution (Focals Conv) and its multi-modal variant of focal sparse convolution with fusion, or Focals Conv-F for short. The new modules can readily substitute their plain counterparts in existing Sparse CNNs and be jointly trained in an end-to-end fashion. For the first time, we show that spatially learnable sparsity in sparse convolution is essential for sophisticated 3D object detection. Extensive experiments on the KITTI, nuScenes and Waymo benchmarks validate the effectiveness of our approach. Without bells and whistles, our results outperform all existing single-model entries on the nuScenes test benchmark at the paper submission time. Code and models are at https://github.com/dvlab-research/FocalsConv.

1. Introduction

3D sparse features vary in sparsity and importance, but conventional sparse convolutions treat them uniformly. The paper introduces learnable sparsity modules that selectively expand valuable features and reports gains across major 3D detection benchmarks.

  • Motivation: Regular sparse convolution expands every sparse feature, increasing computation and reducing feature distinctions between targets and background.Submanifold convolution avoids this cost but restricts information flow between disconnected features.
  • Motivation: 3D data has spatially varying sparsity and feature importance, making uniform convolution treatment unsuitable for object detection.Object sparsity also varies with distance from LIDAR sensors, while foreground and background features contribute differently.
  • Method: Focal sparse convolution predicts cubic importance maps and dilates only features deemed important into deformable output shapes.Importance is learned by an additional convolutional layer conditioned on input features.
  • Method: Focals Conv-F extends the LIDAR-only module by using fused RGB features for importance prediction and output-feature fusion.The image branch supplies appearance information and larger receptive fields.
  • Results: The modules replace plain sparse-convolution counterparts, add small complexity overhead, and improve results on KITTI and nuScenes.The paper reports leading performance on the nuScenes test split without bells and whistles.

2. Related Work

Related work adapts convolutional sampling, attention, or feature sparsity, while this paper focuses on learning output sparsity in intrinsically sparse 3D data. Its multimodal design also differs from decoration-based fusion by strengthening only predicted important features.

  • Convolutional Dynamic Mechanism: Dynamic CNN mechanisms adapt operations to inputs through kernel-shape adaptation or input attention masks.These approaches include deformable and dynamic convolutions.
  • Convolutional Dynamic Mechanism: Focal sparse convolution differs from deformable 3D methods by learning output feature spatial sparsity rather than only changing input sampling.Its target application is 3D object detection.
  • Convolutional Dynamic Mechanism: Attention-mask methods sparsify dense images, whereas this work exploits intrinsic sparsity and adapts across layers and related tasks.The cited comparison contrasts focal sparse convolution with static masks limited to 2D BEV inputs.
  • Completion-based Detectors: The paper focuses on feature learning rather than completion-based methods that expand or reconstruct foreground information.Completion approaches may rely on generated proposals or expand features before pruning.
  • Multi-modal Fusion: Unlike decoration-based multimodal methods, focal sparse convolution uses several jointly trained image layers and strengthens only predicted important LIDAR features.This avoids heavy segmentation or detection models and uniform decoration of all LIDAR features.

3. Focal Sparse Convolutional Networks

The proposed network replaces static sparse-convolution output patterns with learned, feature-dependent sparsity. It predicts importance, selects important inputs, dynamically expands their neighborhoods, and optionally fuses RGB features.

  • Sparse Convolution Review: Sparse convolution operates on input and output position sets rather than the full dense spatial grid.Its kernel aggregates only nonempty neighboring positions.
  • Sparse Convolution Review: Regular sparse convolution unions all kernel-neighbor positions, increasing feature count and computation.This expansion can double or triple sparse features and may blur valuable foreground information.
  • Sparse Convolution Review: Submanifold sparse convolution preserves input positions and efficiency but loses information flow between disconnected features.This constrains effective receptive fields in irregular point-cloud data.
  • Focal Sparse Convolution: Focal sparse convolution dynamically determines output positions instead of using static regular or submanifold patterns.Its output combines dilated neighborhoods around important inputs with the remaining input positions.
  • Focal Sparse Convolution: The module factorizes processing into cubic importance prediction, important-input selection, and dynamic output-shape generation.An additional submanifold sparse convolution with sigmoid activation predicts each input feature’s cubic importance map.
  • Focal Sparse Convolution: Important inputs are selected by thresholding the center of their predicted importance maps, with threshold values interpolating between regular and submanifold behavior.A top-k selection ratio is an alternative to thresholding.
  • Focal Sparse Convolution: Remaining unimportant features retain input positions because removing them or making all outputs dynamic destabilizes training.This preserves a submanifold-like path for those features.
  • Fusion Focal Sparse Convolution: The multimodal extension fuses aligned RGB and LIDAR features into importance prediction and selected sparse outputs.Its fusion layers are lightweight and jointly trained with the detector.

4. Experiments

Experiments across KITTI and nuScenes evaluate the proposed convolutions through framework comparisons, ablations, and runtime analysis. Results show consistent gains from learned importance-guided sparsity, with strong performance on both single-modal and multi-modal detection.

  • Experimental setup: The modules are evaluated on PV-RCNN, Voxel R-CNN, and CenterPoint across KITTI and nuScenes, using established detection metrics and multi-view fusion settings.KITTI evaluations use AP3D with R40 or R11 recall positions; nuScenes uses mAP and NDS.
  • Ablation studies: Dynamic output shapes outperform fixed regular dilation, while dilating all sparse features increases unpromising voxel features.The comparison fixes output shapes without changing other components.
  • Ablation studies: Large performance drops occur when importance-guided input selection is replaced by random sampling, validating predicted importance as a necessary component.The ablation changes only the importance selection step.
  • Ablation studies: Focals Conv improves as more backbone stages use it, with the best empirical setting applying it through the first three stages.The final stage has limited downsampled spatial extent, reducing the benefit of sparsity adaptation.
  • Main results: 61.7% mAP is achieved by Focals Conv-F on a strong multi-modal nuScenes baseline, versus 59.0% mAP for that baseline and 56.1% for the LIDAR-only baseline.The compared CenterPoint variants use the same fusion and image-feature extraction layers.
  • Main results: 70.1% mAP and 73.6% NDS are achieved by Focals Conv-F with test-time augmentations on nuScenes, while the unaugmented model reaches 67.8% mAP and 71.8% mAP.Both reported Focals Conv-F results outperform previous methods without ensembling; the augmented result reports both mAP and NDS in the passage.

5. Conclusion and Discussion

The paper presents focal sparse convolution and a multimodal extension as end-to-end solutions for LIDAR-only and multimodal 3D object detection. The methods achieve leading performance on large-scale nuScenes while learned sparsity is identified as essential for 3D object detectors.

  • Focal sparse convolution and its multimodal extension provide simple, effective end-to-end solutions for LIDAR-only and multimodal 3D object detection.
  • The paper identifies learned sparsity with focal points as essential for 3D object detectors.
  • Focal and fusion sparse CNNs achieve leading performance on the large-scale nuScenes benchmark.
  • Multimodal 3D detection with multiple views increases computation cost, even though the image branch is largely simplified.
  • The method relies on data-distribution sparsity, which might reflect biases in data collection, including biases with negative societal impacts.

A. More Implementation Details

The implementation uses open-source 3D detection codebases and voxelized point clouds, with dataset-specific spatial ranges and voxel sizes for KITTI and nuScenes.

  • The implementation is based on the open-sourced OpenPCDet and released CenterPoint code.
  • KITTI point clouds are clipped to [0, 70.4m] × [-40m, 40m] × [-3, 1]m with voxel size (0.05m, 0.05m, 0.1m).
  • nuScenes uses detection ranges [-54m, 54m] for X and Y and [-5m, 3m] for Z, with voxel size (0.075m, 0.075m, 0.2m).

A.2. Data Augmentations

The experiments apply dataset-specific geometric augmentations, with nuScenes additionally using translation noise and corresponding point-image processing. Supplementary tables report objective loss weights and Waymo improvements.

  • KITTI augmentation uses random X-axis flipping, scaling from 0.95 to 1.05, Z-axis rotation from -45° to 45°, and ground-truth sampling.
  • nuScenes augmentation uses X- and Y-axis flipping, rotation from -45° to 45°, scaling from 0.9 to 1.1, and translation noise on all three axes.
  • The supplementary material includes a table of objective loss weights for AP3D (R40) on the KITTI validation set.
  • The supplementary material includes a table reporting improvements upon CenterPoint on Waymo.

A.3. Training Settings

KITTI and nuScenes models are trained with Adam and cosine-annealed learning rates, while the backbone uses a VoxelNet-based stem and four-stage sparse architecture.

  • KITTI models are trained for 80 epochs with batch size 16, Adam, initial learning rate 0.01, weight decay 0.01, momentum 0.9, and gradient clipping at 10.
  • nuScenes models are trained for 20 epochs with batch size 32, Adam, learning rate from 1e-3 to 1e-4, weight decay 0.01, and gradient clipping at 35.
  • The backbone is based on VoxelNet and contains a stem layer plus four stages.
  • The last three backbone stages contain stride-2 regular sparse convolutional blocks for down-sampling.

B.1. Architecture settings

The evaluated backbones inherit standard sparse-CNN architectures, with Focals Conv inserted in selected late layers of early stages and Focals Conv-F restricted to Stage 1 for multimodal efficiency.

  • PV-RCNN and Voxel R-CNN: PV-RCNN and Voxel R-CNN use channel settings {16, 16, 32, 64, 64} and Subm-block counts {1, 2, 2, 2}.
  • Focal-convolution placement: Focals Conv is inserted in the last layer of Stages 1, 2, and 3, while Focals Conv-F is inserted only in the last layer of Stage 1.
  • CenterPoint: CenterPoint uses channels {16, 16, 32, 64, 128} and two repeated Subm blocks in each of its four stages.
  • CenterPoint: CenterPoint Subm blocks contain two sequential conv-bn-relu layers with a residual connection, but focal convolution is applied only to the last layer.
  • Architecture inheritance: The architecture-level settings are inherited unchanged from the original PV-RCNN, Voxel R-CNN, and CenterPoint frameworks for fair comparison.

C.1. Results on Bird’s Eye View on KITTI

The KITTI subsection reports that Focals Conv-F improves Voxel R-CNN on BEV and 3D detection, while describing the focal-loss objective and its implementation settings.

  • KITTI results: Focals Conv-F performs better than the strong Voxel R-CNN baseline on both APBEV and AP3D in moderate and hard KITTI cases.
  • Objective loss: The objective loss is implemented as focal loss, with γ = 2 used for the importance prediction.
  • Importance prediction: The probability p_i estimates whether feature i contributes any foreground objects, with the target transformation depending on y_i.
  • Objective loss: The focal-loss weight ablation varies values from {0.1, 0.5, 1.0, 2.0}; 0.5 and 1.0 are competitive, and 1.0 remains the default.

C.3. Improvements on the Waymo Open Dataset.

Experiments report non-trivial improvements from Focals Conv on Waymo and from focal modules on nuScenes, while identifying category-specific fusion degradation under ground-truth sampling.

  • Waymo results: Focals Conv brings non-trivial improvements on the Waymo dataset.
  • nuScenes results: 4.9% mAP improvement over plain CenterPoint is reported for Focals Conv-F on the nuScenes validation split.
  • Category-specific fusion effects: Fusion can leave performance unchanged or make it worse for categories such as Car, Ped, and Bar, although baseline improvements are consistent across categories.
  • Augmentation analysis: When ground-truth sampling is disabled while other transformations remain, all categories benefit from fusion; the authors associate the issue with image-level copy-paste occlusion.

C.6. Ablations on Voxel Size.

Voxel-size ablations show a best overall nuScenes validation setting, while the appropriate voxel size varies across object classes and motivates future analysis.

  • Voxel-size ablation: The overall mAP is best at voxel size (0.075, 0.075, 0.2)m on the nuScenes validation split.
  • Voxel-size ablation: The ablation changes X- and Y-axis voxel sizes from 0.05m to 0.15m in 0.025m intervals.
  • Voxel-size ablation: Proper voxel sizes vary across different classes, and the paper identifies this as requiring further analysis or a dynamic mechanism.
Loading 2204.12463v1…