Source-linked AI summary
PointPainting: Sequential Fusion for 3D Object Detection
Sourabh Vora, Alex H. Lang, Bassam Helou, Oscar Beijbom
TL;DR
Lidar-only methods had outperformed fusion methods despite camera and lidar offering complementary information for 3D detection. PointPainting sequentially projects lidar points into image semantic-segmentation outputs and appends the resulting scores before lidar detection. Across KITTI and nuScenes, it improved multiple lidar detectors, reached state-of-the-art KITTI performance with PointRCNN, and supported pipelined processing without performance loss in ablation.
Problem
Camera and lidar provide complementary information, yet lidar-only methods outperform fusion methods on major 3D detection benchmarks.
Method
PointPainting projects each lidar point into an image semantic-segmentation network's output, appends the class activations, and feeds the painted cloud to a lidar detector.
Results
PointPainting improved three lidar detectors across KITTI and nuScenes, with painted PointRCNN achieving state of the art on KITTI and Painted PointPillars+ improving by 6.3 mAP on nuScenes.
Takeaways & Limitations
The framework is general across segmentation and lidar networks, improves performance across classes, and can reduce latency through pipelining.
Takeaways & Limitations
Because PointPainting is sequential, it cannot always be optimized end-to-end for the final 3D detection task, implying possible performance sub-optimality in theory.
Abstract
from arXiv · showhide
Camera and lidar are important sensor modalities for robotics in general and self-driving cars in particular. The sensors provide complementary information offering an opportunity for tight sensor-fusion. Surprisingly, lidar-only methods outperform fusion methods on the main benchmark datasets, suggesting a gap in the literature. In this work, we propose PointPainting: a sequential fusion method to fill this gap. PointPainting works by projecting lidar points into the output of an image-only semantic segmentation network and appending the class scores to each point. The appended (painted) point cloud can then be fed to any lidar-only method. Experiments show large improvements on three different state-of-the art methods, Point-RCNN, VoxelNet and PointPillars on the KITTI and nuScenes datasets. The painted version of PointRCNN represents a new state of the art on the KITTI leaderboard for the bird's-eye view detection task. In ablation, we study how the effects of Painting depends on the quality and format of the semantic segmentation output, and demonstrate how latency can be minimized through pipelining.
1. Introduction
PointPainting addresses the gap between complementary camera-lidar sensing and the stronger performance of lidar-only detectors by sequentially appending image semantics to lidar points. It is designed as a general fusion method compatible with different lidar architectures, while improving benchmark performance and supporting pipelined operation.
- Camera images provide fine-grained texture and color, while lidar provides accurate range but low resolution, motivating fusion that combines their complementary information.
- Lidar-only detectors had surpassed earlier fusion methods on major benchmarks, creating a central challenge for effective camera-lidar integration.
- PointPainting projects each lidar point into image segmentation outputs and concatenates the resulting channel-wise activations with the point's intensity.
- The painted point cloud can be processed by different lidar detectors without restricting the 3D detection architecture, requiring pseudo-point-cloud construction, or limiting maximum recall.
- The method achieved state-of-the-art KITTI results with painted PointRCNN, improved Painted PointPillars+ by 6.3 mAP on nuScenes, and supports low-latency pipelining without affecting performance in ablation.
2. PointPainting Architecture
PointPainting uses image semantic-segmentation scores to augment lidar points before applying a lidar-based 3D detector. Its sequential pipeline preserves compatibility with diverse lidar networks while supporting standalone semantic outputs and potentially lower latency through pipelining.
- Architecture: The architecture performs semantic segmentation, paints lidar points with pixelwise class scores, and applies a lidar-based 3D object detector.The three stages are image segmentation, score-based lidar fusion, and 3D detection.
- Image Based Semantics Network: Image segmentation outputs compact per-pixel image features that are easier to train and faster to infer than full 3D detection.The paper also notes that semantic-segmentation outputs can serve independent robotics tasks.
- PointPainting: Each lidar point is projected into the image, and the relevant pixel’s segmentation-score vector is appended to form a painted point.The projection uses sensor transformations and a camera matrix; overlapping-camera projections are resolved by randomly selecting one score vector.
- Lidar Detection: The method changes only the input point dimension, allowing decorated clouds to feed lidar networks that learn an encoder.The paper demonstrates compatibility with PointPillars, VoxelNet, and PointRCNN, spanning distinct detector architectures and representations.
- Lidar Detection: 69.86 mAP is achieved by Painted PointRCNN on the KITTI test BEV detection benchmark, establishing a new state of the art.The table defines the Painting delta as Painted PointRCNN minus PointRCNN and reports corresponding 3D results in supplementary material.
3. Experimental setup
The experiments evaluate PointPainting on KITTI and nuScenes using multiple lidar detectors and dataset-specific semantic-segmentation systems. The setup includes class-specific preprocessing, detector input-dimension changes, and an improved PointPillars+ baseline for nuScenes.
- Datasets: Experiments are conducted on the KITTI and nuScenes datasets, which provide synchronized lidar and camera data for 3D detection.KITTI evaluates cars, pedestrians, and cyclists; nuScenes evaluates ten object classes using six cameras alongside lidar.
- KITTI: KITTI uses 7,481 training samples and 7,518 test samples, with a 784-sample minival split and 6,733 remaining training samples.The KITTI benchmark annotates cars, pedestrians, and cyclists when visible in the image.
- nuScenes: nuScenes contains 28,130 training, 6,019 validation, and 6,008 test samples from 1,000 annotated 20-second scenes.The evaluation covers cars, trucks, buses, trailers, construction vehicles, pedestrians, motorcycles, bicycles, traffic cones, and barriers.
- Semantics Network Details: KITTI segmentation uses DeepLabv3+, while nuScenes uses a custom network trained with nuImages data.KITTI cyclist labels require mapping rider-and-bike semantics after painting and suppressing parked bikes to background.
- Lidar Network Details: The study uses PointPillars, VoxelNet, and PointRCNN, whose painted variants receive segmentation scores as additional point features.These detectors differ in stage design and use pillars, voxels, or point-wise features.
- Lidar Network Details: 10% mAP is gained by the improved PointPillars+ setup over the previously published PointPillars configuration on nuScenes.The improvements modify pillar resolution, network architecture, attribute estimation, sample weighting, and data augmentation.
4. Results
PointPainting improves multiple lidar detectors across KITTI and nuScenes, including state-of-the-art KITTI performance and a substantial nuScenes gain. Improvements extend across classes and help resolve qualitative lidar detection failures.
- PointPainting improved PointPillars, VoxelNet, and PointRCNN on the KITTI validation set.24 of 27 class-stratum-detector comparisons improved, with especially large changes for pedestrians and cyclists.
- Painted PointRCNN improved average precision for every KITTI test-set class across all difficulty strata.It established state-of-the-art performance on mAP and cyclist AP.
- +6.3 mAP was achieved by Painted PointPillars+ on the nuScenes test set despite a stronger baseline.Painted PointPillars+ was reported as the leading realtime method on nuScenes.
- Every nuScenes detection class received an AP boost from PointPainting.Traffic cones gained +16.8 AP, while trailers and construction vehicles gained less because segmentation recall was lower for those classes.
- Qualitative examples show painting recovering missed cyclists, improving vehicle orientation estimates, and helping distinguish pedestrians from poles.
5. Ablation Studies
Ablations show that detection benefits increase with semantic segmentation quality, while segmentation labels and scores perform similarly. Pipelining previous-image semantics removes added latency without reducing detection performance.
- 5.2. Latency: 0.75 ms of latency was added by consecutive matching for Painted PointPillars, with no detection-performance drop.
- 5.1. Dependency on Semantics: Improved semantic segmentation mIoU led to improved 3D object detection performance.An oracle using ground-truth 3D boxes improved detection by +27 mAP, indicating substantial headroom from better semantics.
- 5.1. Dependency on Semantics: Using segmentation labels instead of scores changed NDS by 0 and increased mAP by +0.4.The gain was marginal and within training noise.
- 5.1. Dependency on Semantics: The ablations suggest prioritizing segmentation quality over segmentation-output representation.
- 5.2. Latency: Consecutive matching pipelines previous-image segmentation scores into the lidar network to avoid waiting for current-image results.
6. Conclusion
The paper presents PointPainting as a sequential method that paints lidar points with image-based semantics. It reports state-of-the-art results across KITTI and nuScenes with multiple lidar networks and emphasizes the framework’s flexibility.
- PointPainting paints lidar point clouds with image-based semantics in a sequential fusion framework.
- The framework combines outputs from any segmentation network with any lidar network.
- PointPainting produces state-of-the-art results on the KITTI and nuScenes challenges with multiple lidar networks.
A. PointPainting: 3D results
PointPainting substantially improves KITTI 3D detection on validation data, but test-set effects differ by class. Pedestrian and cyclist performance improves consistently, whereas car performance drops substantially.
- Validation Set: PointPainting substantially improved KITTI validation-set 3D detection performance.
- Test Set: KITTI test-set 3D results improved consistently for pedestrians and cyclists across easy, medium, and hard strata.
- Test Set: KITTI test-set 3D detection for cars dropped substantially with PointPainting.The paper attributes this possibility to overfitting on a small minival set.
B. PointPainting Latency
Consecutive matching minimizes PointPainting’s added latency without reducing detection performance. In Painted PointPillars, the total overhead is 0.75 ms, supporting real-time camera-lidar fusion.
- Consecutive matching minimizes PointPainting’s latency without any drop in detection performance.
- 0.15 ms is added by projection when transforming and projecting point clouds to obtain segmentation scores.
- 0.6 ms is added during encoding because Painted PointPillars processes 18-dimensional decorated points instead of 7-dimensional points.
- 0.75 ms is the total additional latency introduced by Painted PointPillars over PointPillars with consecutive matching.
C. nuImages Semantic Segmentation
The nuImages semantic segmentation network achieves a validation mIoU of 0.65, with performance varying across object classes. It performs best on cars and worst on construction vehicles and trailers.
- 0.65 mIoU was achieved on the nuImages validation set by the trained semantic segmentation network.
- The segmentation network performs best on the car class and worst on construction vehicle and trailer classes.