PaperlayerView exact revision on arXiv

Source-linked AI summary

VPFNet: Improving 3D Object Detection with Virtual Point based LiDAR and Stereo Data Fusion

Hanqi Zhu, Jiajun Deng, Yu Zhang, Jianmin Ji, Qiuyu Mao, Houqiang Li, Yanyong Zhang

arXiv:2111.14382v2cs.CV

TL;DR

Sparse LiDAR points and dense stereo-image pixels are difficult to fuse because point-based aggregation undersamples image information, while dense 3D lifting is computationally demanding. VPFNet uses intermediate-density virtual points, nearby LiDAR-feature aggregation, and multimodal augmentation to address this gap. On KITTI, it reports leading results across several metrics and supports efficient processing.

  • Problem

    Resolution mismatch between sparse LiDAR points and dense image pixels limits image-feature sampling in point/voxel-level fusion, while dense 3D lifting increases processing demand.

  • Method

    VPFNet aggregates stereo-image and LiDAR features at virtual points whose density lies between that of LiDAR points and image pixels, with multimodal augmentation.

  • Results

    VPFNet delivers the best results in several KITTI test scenarios, including AP3D moderate, APBEV moderate, and APBEV hard, while improving over Voxel-RCNN on multiple metrics.

  • Takeaways & Limitations

    Virtual-point fusion supports accurate multimodal 3D detection while balancing image sampling rate and computational efficiency.

Abstract

from arXiv · show

It has been well recognized that fusing the complementary information from depth-aware LiDAR point clouds and semantic-rich stereo images would benefit 3D object detection. Nevertheless, it is not trivial to explore the inherently unnatural interaction between sparse 3D points and dense 2D pixels. To ease this difficulty, the recent proposals generally project the 3D points onto the 2D image plane to sample the image data and then aggregate the data at the points. However, this approach often suffers from the mismatch between the resolution of point clouds and RGB images, leading to sub-optimal performance. Specifically, taking the sparse points as the multi-modal data aggregation locations causes severe information loss for high-resolution images, which in turn undermines the effectiveness of multi-sensor fusion. In this paper, we present VPFNet -- a new architecture that cleverly aligns and aggregates the point cloud and image data at the `virtual' points. Particularly, with their density lying between that of the 3D points and 2D pixels, the virtual points can nicely bridge the resolution gap between the two sensors, and thus preserve more information for processing. Moreover, we also investigate the data augmentation techniques that can be applied to both point clouds and RGB images, as the data augmentation has made non-negligible contribution towards 3D object detectors to date. We have conducted extensive experiments on KITTI dataset, and have observed good performance compared to the state-of-the-art methods. Remarkably, our VPFNet achieves 83.21\% moderate 3D AP and 91.86\% moderate BEV AP on the KITTI test set, ranking the 1st since May 21th, 2021. The network design also takes computation efficiency into consideration -- we can achieve a FPS of 15 on a single NVIDIA RTX 2080Ti GPU. The code will be made available for reproduction and further investigation.

I. INTRODUCTION

VPFNet addresses resolution mismatch in stereo-LiDAR fusion by aggregating multimodal features at intermediate-density virtual points. On KITTI, it reports strong detection performance while maintaining efficient processing.

  • RGB images provide semantic detail, whereas LiDAR supplies precise depth but lacks color, texture, and fine-grained shape information.
  • Point/voxel-level fusion suffers resolution mismatch because dense image pixels greatly outnumber sparse LiDAR points.
  • 20,000 LiDAR points versus more than 300,000 image pixels yields roughly a 6.7% image-feature sampling rate when points determine aggregation locations.
  • Virtual points with density between LiDAR points and image pixels bridge the resolution gap and increase image-data sampling.
  • Voxel-RoI pooling aggregates K nearby LiDAR features at each virtual point, replicating sparse features into denser point-cloud representations.
  • VPFNet combines virtual-point fusion with multimodal augmentation, including random proposal resizing, weighted branches, and cut-n-paste sampling.
  • 83.21% AP3D moderate, 91.86% APBEV moderate, and 86.94% APBEV hard are reported as top-ranked KITTI test results.
  • 15 FPS processing is reported for the lightweight VPFNet design.

II. RELATED WORK

Prior work spans LiDAR-only, stereo-only, and multimodal detectors, with multimodal fusion operating at object or point/voxel granularity. VPFNet targets the resolution mismatch that limits existing fine-grained fusion and extends augmentation to multimodal data.

  • LiDAR-only 3D object detection: LiDAR-only detectors use point-based or voxel-based representations, trading preserved locations against memory locality and efficiency.
  • Stereo-only 3D object detection: Stereo-only methods either generate pseudo-LiDAR points, which can be time-consuming, or construct 3D cost volumes compressed toward BEV.
  • LiDAR and camera fusion: Multimodal detectors fuse features at object level or at finer point/voxel level.
  • LiDAR and camera fusion: Point/voxel fusion methods can lose image information because sparse LiDAR samples represent only a small portion of dense pixels.
  • LiDAR and camera fusion: Lifting image features into dense 3D space can increase computation and require downsampling that loses 3D information.
  • VPFNet: VPFNet uses intermediate-density virtual points to bridge the resolution gap while balancing image sampling rate and computational efficiency.
  • Augmentation: The method also applies cut-n-paste object sampling using prediction masks to improve multimodal detection performance.

III. PRELIMINARIES ON POINT/VOXEL-LEVEL LIDAR-CAMERA FUSION

Point/voxel-level LiDAR-camera fusion projects 3D points into the image, samples corresponding image features, and aggregates them before detection. Existing formulations commonly concatenate sampled image data with point features.

  • A typical fusion pipeline performs 3D-to-2D projection, 2D feature sampling, multimodal aggregation, and 3D object detection.
  • Existing methods use original or downsampled LiDAR points as multimodal aggregation locations and project them onto the image plane.
  • The projection uses point coordinates together with camera intrinsics and LiDAR-to-camera rotation and translation.
  • Image features at projected coordinates are sampled by bilinear interpolation and concatenated with the point’s location and feature.
  • PointPainting forms an aggregated vector from point coordinates, intensity, and a sampled image segmentation mask.
  • The aggregated feature vectors are passed to detection heads that produce box classification and regression outputs.

IV. VIRTUAL-POINT FUSION ( VPFNet)

VPFNet fuses stereo-image and LiDAR features at virtual points to bridge their resolution gap while balancing detection accuracy and computation efficiency. Its architecture combines virtual-point selection, image association, LiDAR aggregation, weighted multi-branch prediction, and multimodal augmentation.

  • VPFNet aggregates stereo-image and LiDAR data at virtual points denser than LiDAR points but sparser than image pixels.This design targets the resolution gap between the two sensors.
  • The network generates 3D proposals with a 3D backbone, extracts image features with a separate 2D backbone, and derives virtual points from proposal grids.Random proposal resizing is applied during training.
  • Image features are projected to virtual points, while modified voxel-RoI pooling aggregates nearby LiDAR features in a multi-scale manner.The resulting multimodal features are processed through sparse convolutions.
  • Weighted main and auxiliary branches predict final detections while reducing single-modality dominance and over-fitting.The branches receive different supervision weights.
  • Multimodal augmentation is included to transform stereo images and point clouds consistently during training.The design overview treats augmentation as a dedicated training-stage component.

B. Virtual Points Selection

VPFNet selects virtual points from randomly resized 3D proposals rather than using sparse LiDAR points directly. Grid sampling increases feature density while retaining proposal-centered 3D context.

  • Virtual points address low image-feature sampling caused by the resolution imbalance between sparse LiDAR and dense images.The paper reports that faraway and black objects especially exhibit low LiDAR sample density.
  • The 3D backbone generates proposals and feature maps, and selected proposals are expanded by 0.8m in every dimension for contextual coverage.Proposals undergo NMS and are retained when their IoU exceeds a preset threshold.
  • VPFNet randomly perturbs every proposal dimension to improve robustness and reduce over-fitting from accurate point-cloud proposals.Noise is sampled independently from a uniform distribution for the seven box dimensions.
  • After resizing, proposals are divided into 3D grids whose vertices become virtual points.Example grid dimensions 12 × 8 × 22 produce more than 2K foreground samples, compared with typically hundreds of LiDAR samples.

C. Associating Image Features to Virtual Points

VPFNet associates image and LiDAR features at virtual points through projection, sparse image-feature processing, and nearest-neighbor voxel aggregation. The resulting virtual-point representation combines multi-scale features from both sensors.

  • Sampled left and right image features are attached to each virtual point and then voxelized for sparse-convolution processing.Six standard 3 × 3 sparse-convolution blocks produce a compact image feature map.
  • Modified voxel-RoI pooling queries multiple nearby LiDAR features at each virtual point to compensate for sparse point density.The operation effectively replicates point features and runs separately across multimodal, multi-scale feature maps.
  • For each virtual point, a voxel query groups K neighboring features and aggregates them using MLP transformations followed by max pooling.The pooled representation combines location offsets and voxel features.
  • The final virtual-point feature has dimensions Gx × Gy × Gz × 3C, where 3C sums the channels of the three input feature maps.

E. Weighted Multi-Branch Combination

VPFNet combines virtual-point image features with LiDAR features through detection and auxiliary branches, while extending augmentation to stereo data and controlling occlusion during object insertion.

  • Weighted Multi-Branch Combination: The detection head uses aggregated virtual-point features to predict box regression and 3D IoU results.The flattened feature vector is reduced to 512 dimensions by MLPs before prediction.
  • Weighted Multi-Branch Combination: The auxiliary branch predicts box regression from aggregated image features to encourage them to encode 3D context.Its loss is combined with the main branch using separate weights.
  • Data Augmentation: VPFNet extends cut-n-paste augmentation to stereo data by generating foreground masks and associating predicted masks across image pairs.A Hungarian-algorithm cost matrix uses object locations, box heights, and scores for left-right matching.
  • Data Augmentation: Matched stereo pairs are associated with ground-truth images before selecting calibration-compatible scenes for object sampling.The sampled objects are inserted with their point clouds and masks while preserving consistency.
  • Data Augmentation: Separate 2D and 3D occlusion thresholds prevent unnecessarily challenging scenes during augmentation, followed by depth-based rearrangement and insertion.Sampled objects are ordered by ascending depth to simulate occlusion.

V. EVALUATION

The evaluation uses KITTI car detection under the updated 40-recall-point protocol and compares VPFNet with published methods and LiDAR-only baselines.

  • Dataset and Metrics: KITTI provides 7,481 training frames and 7,518 testing frames with camera images, Velodyne HDL-64E point clouds, and annotations for three object classes.The dataset defines easy, moderate, and hard difficulty levels using object size, occlusion, and truncation.
  • Test-Set Comparison: VPFNet achieves the highest 3D moderate, BEV moderate, and BEV hard precision on the KITTI test set, and ranks first among published methods for 3D hard.The comparison uses 3D and BEV precision evaluated at 40 recall points on the KITTI server.
  • Validation Results: The KITTI validation comparison reports pronounced improvement for VPFNet over baseline networks, especially on 3D metrics.VPFNet-BASE denotes the LiDAR-only version, while Voxel-RCNN is the adopted point-cloud backbone.
  • Dataset and Metrics: Evaluation focuses on car detection using 3D Average Precision with a 3D IoU threshold of 0.7.The car category is selected because it has more annotated boxes and provides more stable results.

B. Implementation

The implementation follows a KITTI train-validation split and applies geometric and ground-truth sampling augmentations jointly to the multimodal scene.

  • Data Split: The KITTI training set is split into 3,712 training frames from 96 scenarios and 3,769 validation frames from 45 scenarios.LiDAR points are projected into the camera view and points outside that view are discarded.
  • Data Augmentation: The augmentation pipeline globally rotates scenes, flips them with probability 0.5, and scales each object uniformly between 0.95 and 1.05.These strategies are combined with ground-truth box sampling.
  • Data Augmentation: Ground-truth sampling generates foreground masks, associates stereo-ground-truth triplets, and randomly pastes 15 triplets into new scenes.The pasted data includes multimodal object content and applies the modified occlusion procedure.

3) Network setting:

VPFNet uses a voxel-based 3D backbone, lightweight 2D image processing, virtual-point discretization, and weighted detection losses for multimodal prediction.

  • Network Architecture: The 3D backbone voxelizes raw points at [.05, .05, .1]m resolution and processes them with sparse convolutions before BEV proposal generation.A 2D RPN generates 3D region proposals from the BEV features.
  • Network Architecture: The 2D backbone uses two convolution blocks and virtual points are sampled at resolutions 16, 8, and 22 along width, length, and height.The proposals are randomly rotated and resized before virtual-point discretization.
  • Training and Inference: Training samples 40 proposals at IoU threshold 0.7, whereas evaluation keeps 20 proposals at IoU threshold 0.1 and uses a 0.1 score threshold.The network is trained for 80 epochs with ADAM on eight RTX 2080Ti GPUs.
  • Loss Design: The regression loss represents boxes with (x, y, z, w, l, h, θ) and computes residuals between ground-truth and anchor boxes.The region proposal loss uses focal loss for anchor classification and Smooth L1 loss for anchor regression.
  • Loss Design: The combined second-stage loss adds IoU prediction loss to weighted Smooth L1 regression losses from the main and auxiliary branches.The branch weights are denoted WV and WA.

C. VPFNet Detection Results on KITTI Test Set

VPFNet delivers strong KITTI test-set performance, with gains concentrated in moderate and hard cases. Ablations indicate that virtual-point image sampling is especially important, while image fusion improves height estimation across depth ranges.

  • KITTI test-set performance: 83.08% AP3D moderate, 91.86% APBEV moderate, and 86.94% APBEV hard were the best results in three of six KITTI categories.VPFNet also achieved the best published AP3D hard result, while its easy-category performance was not best because fusion benefits are less clear for nearby, minimally occluded objects.
  • KITTI test-set performance: +1.59% AP3D moderate and +3.03% APBEV moderate were VPFNet's gains over Voxel-RCNN, with additional gains on hard cases.The reported improvements were +1.14% for AP3D hard and +0.81% for APBEV hard.
  • Ablation studies: Virtual point image sampling contributed most to Mod3D performance among the four ablated components.Random 3D proposal resizing, right-image input, and multi-branch loss also each contributed to overall performance.
  • Ablation studies: Decreasing virtual-point density or using fewer sparse-convolution layers hurt performance in the evaluated 32-beam setting.
  • Ablation studies: Height estimation improved across all depth ranges, especially at 30–50 meters, according to vertical 1D IoU between ground-truth and predicted 2D boxes.The comparison is against VPFNet-base.

F. Qualitative Results

Qualitative comparisons show that VPFNet improves BEV and 3D box estimates over VPFNet-base in occluded, far-away, and near-object cases. The design combines virtual-point fusion with lightweight processing and multimodal components.

  • Qualitative BEV results: VPFNet improves BEV rotation estimation for an occluded car and improves rotation or box-size estimation for far-away and occluded objects.Figure 8 compares VPFNet-base with VPFNet using ground-truth and predicted boxes.
  • Qualitative 3D results: VPFNet improves box height when LiDAR beams cover only a car's middle and reduces orientation drift under severe occlusion.The activated virtual-point features are mainly located on the car roof and four wheels in the illustrated case.
  • Qualitative 3D results: VPFNet can slightly improve detection even for near objects, alongside improvements shown for far and occluded situations.
  • Efficiency: The submitted configuration uses 16 × 8 × 22 grid blocks and 6 × 6 × 6 query points, achieving 63.6 ms inference time and 15.7 FPS on an RTX 2080Ti.Each 3D proposal contains fewer than 140 actual LiDAR points on average.
  • Method design: Virtual points bridge LiDAR and image resolution while aggregating nearby point features, and the final design uses weighted detection heads to avoid single-modality dominance.The conclusion also reports that multimodal augmentation further boosts performance and that the method supports 15.7 FPS.
Loading 2111.14382v2…