Source-linked AI summary
YOLO3D: End-to-end real-time 3D Oriented Object Bounding Box Detection from LiDAR Point Cloud
Waleed Ali, Sherif Abdelkarim, Mohamed Zahran, Mahmoud Zidan, Ahmad El Sallab
TL;DR
Real-time 3D object detection from LiDAR is important for automated driving but remains algorithmically challenging. The paper extends YOLO V2 with direct regression for oriented 3D boxes, including yaw, center coordinates, and height, and reports 40 fps on KITTI using a Titan X GPU.
Problem
Real-time 3D object bounding-box detection and classification from LiDAR point clouds remains a strong challenge despite its importance for automated driving.
Method
The paper extends YOLO V2 into an end-to-end, one-shot LiDAR detector that directly regresses oriented 3D box geometry and class labels from bird-view input.
Results
40 fps and 75.3% mAP at 0.5 IoU are reported on KITTI for moderate cars.
Takeaways & Limitations
The results support using single-shot detectors for real-time LiDAR-based 3D oriented bounding-box detection.
Takeaways & Limitations
Performance drops significantly as the IoU threshold increases because the predicted boxes remain difficult to align precisely with objects.
Abstract
from arXiv · showhide
Object detection and classification in 3D is a key task in Automated Driving (AD). LiDAR sensors are employed to provide the 3D point cloud reconstruction of the surrounding environment, while the task of 3D object bounding box detection in real time remains a strong algorithmic challenge. In this paper, we build on the success of the one-shot regression meta-architecture in the 2D perspective image space and extend it to generate oriented 3D object bounding boxes from LiDAR point cloud. Our main contribution is in extending the loss function of YOLO v2 to include the yaw angle, the 3D box center in Cartesian coordinates and the height of the box as a direct regression problem. This formulation enables real-time performance, which is essential for automated driving. Our results are showing promising figures on KITTI benchmark, achieving real-time performance (40 fps) on Titan X GPU.
1 Introduction
The paper targets real-time 3D object detection for automated driving using LiDAR point clouds. It extends YOLO V2 with direct regression for oriented 3D boxes and evaluates the approach on KITTI.
- LiDAR provides accurate depth for environment perception, but its lack of camera color features complicates classification from point clouds alone.
- Real-time inference is essential for automated driving, motivating the use of single-shot detectors such as YOLO and SSD.
- The method feeds a bird-view representation of the LiDAR point cloud into a YOLO-based network adapted to sparse input.
- The network directly predicts 3D box center coordinates, dimensions, orientation, confidence, and object class without region proposals.
- The study evaluates real-time performance and accuracy on KITTI using a Titan X GPU, including grid-map resolution and IoU operating-point analysis.
2 Related Work
Related work spans LiDAR-only, camera-only, and sensor-fusion approaches, with differing preprocessing and trade-offs. This paper uses a LiDAR-only bird-view representation and directly regresses orientation, height, and 3D center coordinates.
- LiDAR-only detection benefits from accurate depth, while camera-only methods provide rich visual information and sensor-fusion methods combine both sources.
- Existing 3D detection systems preprocess point clouds through bird-view or front-view projections, depth maps, or voxel grids.
- The proposed system follows the LiDAR-only paradigm by projecting point clouds into a bird-view grid while retaining 3D information.
- Orientation methods based on box-side direction can fail for pedestrians, while separate component regression does not guarantee angle correlation.
- Object detection for automated driving requires efficient inference because computational complexity must be controlled while maintaining accuracy.
- Unlike prior component-based orientation regression, this work directly regresses orientation and also predicts height and z-center rather than using size-based heuristics.
3 Approach
The approach projects sparse LiDAR point clouds into bird’s-eye-view height and density maps, then extends YOLO-based regression to predict oriented 3D boxes. It directly models yaw, 3D center coordinates, and box height while adapting anchors and loss terms to LiDAR data.
- Point Cloud Representation: The input uses two bird’s-eye-view grid maps: maximum point height and point density.Each height-map cell stores the highest associated point, while density increases with the number of points in the cell.
- Yaw Angle Regression: Bounding-box orientation is normalized from -π to π into -1 to 1 and directly regressed as a single output.The yaw loss uses mean squared error between ground-truth and predicted angles; tanh activation did not improve over linear activation.
- 3D Bounding Box Regression: The model adds direct regressions for the 3D box center’s z coordinate and the box height.The z coordinate is mapped within one vertical grid cell because object elevations vary less than x and y; height is predicted analogously to width and length.
- Anchors Calculation: Anchors use mean 3D box dimensions for each object class instead of k-means clustering.This choice reflects the lower within-class variability of box dimensions in bird’s-eye-view grid maps.
- Combined Loss for 3D OBB: The combined loss extends YOLO’s 2D loss with 3D coordinates and dimensions, yaw, confidence, and class terms.It combines mean squared errors for x, y, z, width, length, height, and angle with confidence loss and class cross-entropy.
4 Experiments and Results
The experiments evaluate an end-to-end YOLO-v2-based LiDAR detector on KITTI, including input representation, training setup, IoU behavior, object-class results, and grid-map resolution effects. The system achieves real-time speed but exhibits localization and resolution-dependent trade-offs.
- Experimental setup: The model uses a modified YOLO-v2 architecture with changed downsampling, no skip connection, and added yaw, z-center, and height regression terms.Its two input channels represent maximum height and point density, while the network predicts 3D box properties and classes.
- Experimental setup: KITTI point clouds are projected into a 0.1m-per-pixel bird-view grid covering 30.4m laterally and 60.8m forward, producing 608x608 channels.Points outside the labeled image plane are filtered to avoid contradictory training information.
- KITTI results: Performance drops significantly as the IoU threshold increases, indicating difficulty aligning predicted boxes perfectly with objects.The authors describe this localization issue as inherited from YOLO versions.
- KITTI results: 40 fps and 75.3% mAP at 0.5 IoU are reported for moderate cars on KITTI using one network and a two-channel bird-view input.The validation results use approximately 40% of the KITTI training set.
- Grid-map resolution: Grid-map resolution affects memory, inference time, and performance, with inference time doubling from 16.9ms to 30.8ms when resolution changes from 0.15m/pixel to 0.1m/pixel.The paper attributes this increase to the grid-map area growing quadratically with map length or width.
5 Conclusions
The paper presents an end-to-end real-time LiDAR system that extends YOLO-v2 for 3D oriented bounding-box detection and classification. It reports real-time performance while identifying a cost in localization accuracy.
- Contributions: The system directly regresses yaw, 3D box centers, and dimensions without region proposals or heuristics.Evaluation uses KITTI at different IoU thresholds and grid-map resolutions to identify operating points for speed and accuracy.
- Conclusions: The results suggest that single-shot detectors can predict 3D boxes while maintaining real-time performance, at a cost in box localization accuracy.The paper also evaluates operating IoU thresholds and grid-map resolutions.