Source-linked AI summary
Multimodal Virtual Point 3D Detection
Tianwei Yin, Xingyi Zhou, Philipp Krähenbühl
TL;DR
Small and distant objects are poorly sampled by costly, low-resolution Lidar despite being clearly visible in RGB images, while RGB lacks depth. MVP uses 2D detections and nearby Lidar depth to generate dense semantic 3D virtual points that augment standard Lidar detectors, improving CenterPoint by 6.6 mAP on nuScenes and reaching 66.4 mAP and 70.5 NDS without ensembles or test-time augmentation.
Problem
Lidar provides accurate depth but is costly and low-resolution at long range, while RGB resolves small and distant objects without directly providing scene depth.
Method
MVP uses 2D detections to create dense semantic 3D virtual points by lifting RGB pixels with nearby Lidar depth and combining them with the original point cloud.
Results
6.6 mAP improvement over a strong CenterPoint baseline is reported on nuScenes, with a best model reaching 66.4 mAP and 70.5 NDS without ensembles or test-time augmentation.
Takeaways & Limitations
MVP significantly improves a strong Lidar-only CenterPoint detector and integrates with current or future 3D detection algorithms.
Takeaways & Limitations
MVP assumes virtual points share the depth of nearby Lidar measurements, which may not hold for non-planar objects such as cars.
Abstract
from arXiv · showhide
Lidar-based sensing drives current autonomous vehicles. Despite rapid progress, current Lidar sensors still lag two decades behind traditional color cameras in terms of resolution and cost. For autonomous driving, this means that large objects close to the sensors are easily visible, but far-away or small objects comprise only one measurement or two. This is an issue, especially when these objects turn out to be driving hazards. On the other hand, these same objects are clearly visible in onboard RGB sensors. In this work, we present an approach to seamlessly fuse RGB sensors into Lidar-based 3D recognition. Our approach takes a set of 2D detections to generate dense 3D virtual points to augment an otherwise sparse 3D point cloud. These virtual points naturally integrate into any standard Lidar-based 3D detectors along with regular Lidar measurements. The resulting multi-modal detector is simple and effective. Experimental results on the large-scale nuScenes dataset show that our framework improves a strong CenterPoint baseline by a significant 6.6 mAP, and outperforms competing fusion approaches. Code and more visualizations are available at https://tianweiy.github.io/mvp/
1 Introduction
The paper addresses Lidar’s costly, low-resolution sensing of small and distant objects by fusing RGB measurements into dense 3D virtual points. MVP augments sparse Lidar inputs and improves CenterPoint-based 3D detection on nuScenes.
- Lidar provides accurate depth but is costly and has at least two orders of magnitude lower input resolution than a $50 RGB sensor.Small or distant objects may produce only one or two Lidar measurements while appearing as hundreds of RGB pixels.
- MVP lifts RGB measurements into 3D virtual points using nearby Lidar depth, then combines them with original measurements for standard 3D detection.The framework uses 2D detections, instance frustums, image-space depth completion, and a center-based 3D detector.
- 2D detections are accurate for small objects, while virtual points reduce density imbalance by assigning objects at different distances the same number of points.The method is described as a plug-and-play module for existing or new 2D or 3D detectors.
- 6.6 mAP improvements over a strong CenterPoint baseline are reported after adding multimodal virtual points.The reported result is on the large-scale nuScenes dataset.
- 66.4 mAP and 70.5 NDS are achieved without ensembles or test-time augmentation, outperforming competing non-ensembled methods at submission.
2 Related work
Related work spans image-based 2D detection, Lidar-based 3D detection, camera-based 3D detection, multimodal fusion, and point-cloud augmentation. MVP builds on these lines while using Lidar measurements for depth-supported virtual points.
- 2D detectors range from two-stage RCNN methods and one-shot YOLO, SSD, and RetinaNet to anchor-free CenterNet and CornerNet.The paper uses CenterNet as its 2D detector.
- Lidar-based 3D detectors estimate rotated 3D boxes from point clouds using voxel, sparse-convolution, pillar, or related representations.The related methods include VoxelNet, SECOND, PIXOR, and PointPillars.
- Monocular 3D detectors predict 3D boxes from images but achieve about half the mAP of standard Lidar-based baselines on cited benchmarks.Pseudo-Lidar methods similarly construct virtual point clouds but rely on noisy stereo depth, whereas MVP uses Lidar measurements.
- Frustum PointNet and Frustum ConvNet use image detections to identify point-cloud regions, but their performance is fundamentally limited by 2D detection quality.
- Point-cloud augmentation methods densify sparse Lidar data through learned point reconstruction or image-based depth completion.MVP builds upon depth-completion methods for multimodal 3D detection.
3 Preliminary
The preliminary sections define the paper’s 2D and 3D detection representations and the geometric correspondence between Lidar points and camera pixels. Perspective projection is non-invertible without depth information.
- A 2D detector maps an image I to object boxes, class scores, and optionally instance masks.CenterNet detects objects through class-specific heatmap peaks, while CenterNet2 adds cascade RoI heads for instance segmentation.
- A 3D detector maps point cloud P, containing location and reflectance, to a set of rotated 3D bounding boxes.The paper builds on CenterPoint and experiments with VoxelNet and PointPillars backbones.
- VoxelNet voxelizes points and processes pooled voxel features with sparse 3D convolutions, whereas PointPillars processes bird’s-eye-view pillars with 2D convolutions.
- Lidar-to-RGB correspondence applies an SE(3) transformation followed by perspective projection and division into image coordinates.The transformations are homogeneous and time-dependent.
- Perspective division makes the Lidar-to-RGB mapping surjective and non-invertible, so RGB-to-Lidar recovery requires Lidar depth measurements.
4 Multimodal Virtual Point
MVP generates dense semantic 3D virtual points from masked 2D detections and nearby Lidar depth, then integrates them with real points in a standard 3D detector. Separate pooling preserves distinctions between virtual and real measurements.
- Virtual Point Generation: The algorithm outputs multimodal virtual points containing 3D location and semantic feature information for each detected object.Its inputs include a Lidar point cloud, instance masks, semantic features, coordinate transformations, camera projection, and τ.
- Virtual Point Generation: MVP projects Lidar points into each detected instance mask to form an object frustum, discarding measurements outside detection masks.Each frustum stores projected image coordinates and associated depths.
- Virtual Point Generation: For each instance mask, MVP uniformly samples τ 2D points, retrieves nearest-neighbor frustum depths, unprojects them into 3D, and appends semantic features.The semantic feature combines the detected class’s one-hot encoding with the detection objectness score.
- Virtual Point Generation: Virtual points are generated by sampling masked 2D locations, unprojecting them with nearest-neighbor depth, and adding them to each object’s virtual-point set.
- Virtual Point 3D detection: MVP separately averages virtual and real point features within each voxel, then concatenates the averages before 3D convolution.This avoids zero-padding, density imbalance, and blurring of more precise real measurements caused by joint averaging.
- Virtual Point 3D detection: Dense virtual points also enrich second-stage refinement by providing richer bird’s-eye-view information around predicted box surfaces.
5 Experiments
Experiments evaluate MVP on nuScenes using standard 2D and 3D detection components, compare it with prior methods, and test component contributions, distance-specific performance, robustness, depth interpolation, and KITTI generalization.
- 5.1 State-of-the-art Comparison: MVP achieves 66.4 mAP and 70.5 NDS on the nuScenes test set without ensembles or test-time augmentation.The model is reported as outperforming competing non-ensembled methods on the leaderboard at submission.
- 5.1 State-of-the-art Comparison: MVP outperforms the strong CenterPoint baseline by 8.4 mAP and 5.0 NDS on nuScenes.Improvements are reported consistently across object categories, including 20.6 mAP for Bicycle and 16.3 mAP for motorcycle.
- 5.2 Ablation Studies: The 2D CenterNet detector localizes small and medium objects significantly better than the Lidar-based 3D CenterPoint detector in image-space AP.The comparison uses COCO-style AP with 2D IoU thresholds from 0.5 to 0.95.
- 5.2 Ablation Studies: Adding multimodal virtual points improves the Lidar-only baseline by 6.3 mAP with VoxelNet and 10.4 mAP with PointPillars.The component analysis evaluates MVP inputs and voxelization choices on the nuScenes validation set.
- 5.2 Ablation Studies: MVP improves over the Lidar-only baseline by 6.6 mAP overall and 10.1 mAP for faraway objects, while improving over PointPainting by 1.1 mAP for faraway objects.All three distance-comparison entries use a VoxelNet backbone and divide objects into 0–15m, 15–30m, and 30–50m ranges.
- 5.2 Ablation Studies: MVP loses only 0.8 NDS when instance-segmentation inputs are degraded by 9 points, and it also improves vehicle and cyclist detection on KITTI.The KITTI experiment reports gains of 0.5 mAP for vehicle and 2.3 mAP for cyclist.
6 Discussion and conclusions
The paper presents MVP as a multimodal virtual-point approach that lifts RGB measurements into dense 3D point clouds near target objects, improving localization and regression. It also identifies depth, feature-use, information-transfer, safety, and privacy limitations of camera–Lidar fusion.
- Discussion and conclusions: Figure 4 visualizes raw point clouds in blue, detected objects in green bounding boxes, and enclosed Lidar points in red.
- Discussion and conclusions: MVP lifts RGB measurements into 3D virtual points using close-by Lidar measurements, producing high-resolution point clouds near target objects.The framework is intended to integrate with current or future 3D detection algorithms.
- Discussion and conclusions: The method assumes virtual points share the depth of nearby Lidar measurements, which may not hold for non-planar objects such as cars.The authors propose learning-based shape and pose inference as future work.
- Discussion and conclusions: Current refinement modules use only bird’s-eye-view features, while position-and-class MVP features omit pose information and may bottleneck 2D-to-3D transfer.The authors suggest point- or voxel-based two-stage detectors as a possible way to exploit virtual points more fully.
- Discussion and conclusions: Camera–Lidar fusion may require more safety testing because the modalities respond differently to weather, geography, and day–night changes.A low sun can distract the RGB sensor and consequently the 3D detector through MVPs.
- Discussion and conclusions: Greater reliance on color sensors introduces privacy risks because people identifiable in images may be exposed to mass surveillance.