PaperlayerView exact revision on arXiv

Source-linked AI summary

Virtual Sparse Convolution for Multimodal 3D Object Detection

Hai Wu, Chenglu Wen, Shaoshuai Shi, Xin Li, Cheng Wang

arXiv:2303.02314v1cs.CV

TL;DR

Virtual-point multimodal 3D detection enriches sparse LiDAR data but suffers from excessive virtual-point density and depth-completion noise. The paper introduces VirConvNet with StVD and NRConv, and its detector variants achieve strong KITTI results spanning efficient, high-precision, and semi-supervised detection.

  • Problem

    Virtual points improve distant-object geometry but are extremely dense and noisy, causing redundant computation and degraded 3D localization precision.

  • Method

    VirConvNet uses StVD to discard redundant nearby voxels and NRConv to encode voxel features in both 3D and 2D image space.

  • Results

    VirConv-L, VirConv-T, and VirConv-S outperform Voxel-RCNN by 3.43%, 4.63%, and 5.58% 3D AP (R40), respectively, on KITTI moderate cars.

  • Takeaways & Limitations

    The VirConvNet variants provide efficient, accurate, and semi-supervised multimodal 3D detection, with VirConv-T and VirConv-S ranking 2nd and 1st on KITTI.

Abstract

from arXiv · show

Recently, virtual/pseudo-point-based 3D object detection that seamlessly fuses RGB images and LiDAR data by depth completion has gained great attention. However, virtual points generated from an image are very dense, introducing a huge amount of redundant computation during detection. Meanwhile, noises brought by inaccurate depth completion significantly degrade detection precision. This paper proposes a fast yet effective backbone, termed VirConvNet, based on a new operator VirConv (Virtual Sparse Convolution), for virtual-point-based 3D object detection. VirConv consists of two key designs: (1) StVD (Stochastic Voxel Discard) and (2) NRConv (Noise-Resistant Submanifold Convolution). StVD alleviates the computation problem by discarding large amounts of nearby redundant voxels. NRConv tackles the noise problem by encoding voxel features in both 2D image and 3D LiDAR space. By integrating VirConv, we first develop an efficient pipeline VirConv-L based on an early fusion design. Then, we build a high-precision pipeline VirConv-T based on a transformed refinement scheme. Finally, we develop a semi-supervised pipeline VirConv-S based on a pseudo-label framework. On the KITTI car 3D detection test leaderboard, our VirConv-L achieves 85% AP with a fast running speed of 56ms. Our VirConv-T and VirConv-S attains a high-precision of 86.3% and 87.2% AP, and currently rank 2nd and 1st, respectively. The code is available at https://github.com/hailanyi/VirConv.

1. Introduction

Virtual points enrich sparse LiDAR observations but introduce severe density and noise challenges. VirConvNet addresses both through StVD and NRConv, supporting detectors that target efficiency, precision, and semi-supervised detection.

  • LiDAR-based detection struggles with distant objects because scans provide sparse sampling, while RGB images offer high-resolution scene context.
  • 466k virtual points can be generated from a 1242×375 KITTI image, creating about 27× more points than a LiDAR scan and imposing substantial computation.
  • Larger voxels or random downsampling reduce density but sacrifice useful shape cues from faraway virtual points, lowering detection accuracy.
  • Inaccurate depth completion introduces non-Gaussian noise, often on instance boundaries, that is difficult to distinguish in 3D and degrades localization precision.
  • StVD discards redundant nearby voxels while retaining faraway voxels, whereas NRConv encodes voxel geometry in both 3D and 2D image space to reduce noise impact.
  • VirConv supports VirConv-L, VirConv-T, and VirConv-S for efficient, high-precision, and semi-supervised 3D detection; VirConv-L runs at 56ms, while VirConv-T and VirConv-S rank 2nd and 1st on KITTI.

2. Related Work

Prior multimodal methods use RGB and LiDAR complementarity, including virtual points that complete distant-object geometry. VirConv addresses the resulting density and noise problems with StVD and NRConv within a broader detector family.

  • LiDAR-based 3D object detection: LiDAR-only detectors suffer from distant-object sparsity, motivating multimodal methods that combine RGB image features with point-cloud representations.
  • Multimodal 3D object detection: Virtual-point methods explicitly complete distant-object geometry through depth estimation, but the resulting points are extremely dense and often noisy.
  • 3D object detection with re-sampled point clouds: Prior resampling approaches use larger voxels or random downsampling, whereas StVD retains useful faraway voxels while accelerating processing by discarding others.
  • Noise handling in 3D vision: Depth-completion noise differs from traditional randomly distributed noise because it concentrates mainly on 2D instance boundaries.
  • Semi-supervised 3D object detection: The paper also places VirConv-S within pseudo-label-based semi-supervised multimodal 3D detection.

3. VirConv for Multimodal 3D Detection

VirConvNet addresses the density and noise challenges of virtual-point-based multimodal 3D detection through StVD and NRConv, then supports fast, high-precision, and semi-supervised detector variants.

  • 3.4. Detection Frameworks with VirConv: VirConvNet combines StVD, NRConv, and sparse-convolution downsampling in VirConv blocks, producing VirConv-L, VirConv-T, and VirConv-S detector pipelines.VirConv-L uses early fusion, VirConv-T uses transformed refinement with late fusion, and VirConv-S is based on a semi-supervised pseudo-label framework.
  • 3.2. Stochastic Voxel Discard: StVD discards virtual-point voxels during processing, using input sampling and training-time layer discarding to reduce computation and improve density robustness.Its bin-based sampling balances voxel selection across distances, while layer StVD uses a 15% discarding rate as training augmentation.
  • 3.3. Noise-Resistant Submanifold Convolution: NRConv implicitly distinguishes noise patterns through its 2D receptive field and suppresses noise without losing shape cues.This differs from methods that explicitly segment or remove noisy points.
  • 3.4. Detection Frameworks with VirConv: 56ms and 85.05 AP are reported for VirConv-L, while VirConv-S reaches 87.20 AP and VirConv-T outperforms other methods in KITTI 3D and BEV AP.The KITTI table reports car 3D detection results and identifies leaderboard-derived results with an asterisk.

4. Experiments

Experiments evaluate VirConvNet across KITTI validation and test settings, fusion schemes, components, StVD rates, multi-class detection, and object distances. VirConv improves accuracy while StVD reduces computation, with VirConv-T and VirConv-S achieving leading KITTI leaderboard rankings.

  • Main Results: 3.42%, 5%, and 5.68% 3D AP(R40) improvements over Voxel-RCNN were obtained by VirConv-L, VirConv-T, and VirConv-S on KITTI validation moderate-car detection.The corresponding 3D AP(R11) gains were 2.38%, 3.33%, and 3.54%.
  • Main Results: 3.43%, 4.63%, and 5.58% 3D AP(R40) improvements over Voxel-RCNN were obtained by VirConv-L, VirConv-T, and VirConv-S on KITTI test moderate-car detection.The three models also exceeded SFD by 0.29%, 1.49%, and 2.44%, while VirConv-T and VirConv-S ranked 2nd and 1st on the leaderboard at the stated date.
  • Ablation Study: 3.43%, 2.93%, and 2.65% 3D AP improvements resulted from replacing Voxel-RCNN with VirConvNet under virtual-points-only, early-fusion, and late-fusion settings.Efficiency also improved, attributed to StVD speeding computation and NRConv reducing noise impact.
  • Ablation Study: About 2× faster operation accompanied more accurate detection when StVD was enabled in VirConv-T.StVD discards about 90% of redundant voxels and simulates sparser training samples, improving detection robustness.
  • Ablation Study: A 90% input StVD rate produced the best accuracy-efficiency trade-off, while a 15% layer StVD rate produced the best detection accuracy.Higher input discard rates caused geometry-feature loss, whereas lower rates reduced efficiency.
  • Ablation Study: 88.32% to 90.29% car detection AP was achieved when NRConv was added to VirConv-T.The improvement is associated with encoding voxel features in both 3D and 2D image space to reduce inaccurate-depth noise.

3D AP

The paper evaluates 3D detection performance on the nuScenes test set and reports VirConv-enhanced multimodal detectors. The cited passages identify the dataset and compared detector configurations but do not provide their numerical results.

  • Evaluation: VirConv is evaluated on the nuScenes test set with CenterPoint + VP and TransFusion-L + VP alongside TransFusion.The experiment uses the same data augmentation strategy as TransFusion-L and trains for 30 epochs on eight Tesla V100 GPUs.
  • Results: The nuScenes experiment compares VirConv-enhanced CenterPoint + VP and TransFusion-L + VP against their stated detector configurations.The supplied passage states that results are shown in Table 6 but does not include the numerical values.

5. Conclusion

VirConvNet addresses virtual-point density and noise through VirConv, supporting three pipelines for efficient, accurate, and semi-supervised 3D detection. It achieves leading performance on KITTI car 3D object detection and BEV detection leaderboards.

  • VirConv addresses virtual-point density and noise through Stochastic Voxel Discard and Noise-Resistant Submanifold Convolution.These are the two mechanisms introduced in the VirConv operator.
  • VirConvNet supports VirConv-L, VirConv-T, and VirConv-S for efficient, accurate, and semi-supervised 3D detection, respectively.
  • VirConvNet holds the leading entry on both KITTI car 3D object detection and BEV detection leaderboards.The conclusion presents this as evidence of the method's effectiveness.
Loading 2303.02314v1…