Source-linked AI summary
Complex-YOLO: Real-time 3D Object Detection on Point Clouds
Martin Simon, Stefan Milz, Karl Amende, Horst-Michael Gross
TL;DR
Real-time 3D detection from sparse Lidar is important for autonomous-driving perception but difficult to perform efficiently. Complex-YOLO extends YOLOv2 with a Lidar-only bird’s-eye-view representation and E-RPN orientation regression, achieving high KITTI accuracy with substantially greater efficiency. The approach simultaneously detects multiple classes, while its current 3D evaluation relies on predefined or ground-truth-derived height information.
Problem
Sparse Lidar point clouds require efficient 3D object detection for autonomous-driving perception, while existing point-cloud approaches have limited real-time efficiency.
Method
Complex-YOLO extends YOLOv2 with a single Lidar-only bird’s-eye-view RGB map and an E-RPN that uses real and imaginary regression components for orientation estimation.
Results
Complex-YOLO achieves KITTI accuracy comparable to leading methods while running more than five times faster, and detects all eight KITTI classes simultaneously.
Takeaways & Limitations
The model provides real-time Lidar-only multiclass 3D detection, including deployment at 4 fps on an NVIDIA TX2 embedded platform.
Takeaways & Limitations
The method does not directly regress height, so its 3D benchmark uses fixed spatial height information or class-based predefined heights.
Abstract
from arXiv · showhide
Lidar based 3D object detection is inevitable for autonomous driving, because it directly links to environmental understanding and therefore builds the base for prediction and motion planning. The capacity of inferencing highly sparse 3D data in real-time is an ill-posed problem for lots of other application areas besides automated vehicles, e.g. augmented reality, personal robotics or industrial automation. We introduce Complex-YOLO, a state of the art real-time 3D object detection network on point clouds only. In this work, we describe a network that expands YOLOv2, a fast 2D standard object detector for RGB images, by a specific complex regression strategy to estimate multi-class 3D boxes in Cartesian space. Thus, we propose a specific Euler-Region-Proposal Network (E-RPN) to estimate the pose of the object by adding an imaginary and a real fraction to the regression network. This ends up in a closed complex space and avoids singularities, which occur by single angle estimations. The E-RPN supports to generalize well during training. Our experiments on the KITTI benchmark suite show that we outperform current leading methods for 3D object detection specifically in terms of efficiency. We achieve state of the art results for cars, pedestrians and cyclists by being more than five times faster than the fastest competitor. Further, our model is capable of estimating all eight KITTI-classes, including Vans, Trucks or sitting pedestrians simultaneously with high accuracy.
1 Introduction
Lidar point clouds provide direct 3D distance measurements for autonomous-driving perception, but their sparsity and unordered structure make real-time object detection challenging. Complex-YOLO addresses this gap with a Lidar-only bird’s-eye-view detector designed for efficient multiclass 3D box prediction.
- Lidar directly measures surrounding-object distances, supporting accurate estimation of 3D position and heading for autonomous-driving perception.
- Point clouds are sparse, variably dense, unordered, and locally interacting, creating challenges for invariant 3D processing.
- Deep-learning point-cloud detectors generally process raw points, convert them into voxels or image stacks, or fuse multiple representations.
- Existing leading approaches use cameras, multiple consecutive networks, or computationally costly processing, limiting efficiency to roughly 4–7 fps in cited examples.
- Complex-YOLO uses a single Lidar-only bird’s-eye-view RGB map rather than multi-view fusion to target real-time efficiency.
- The proposed 3D YOLOv2 extension uses E-RPN complex-valued orientation regression and predicts all eight KITTI classes simultaneously.
2 Complex-YOLO
Complex-YOLO converts sparse Lidar point clouds into a birds-eye-view RGB-map and applies a simplified YOLOv2 CNN with an E-RPN to estimate multi-class oriented 3D boxes. Its complex angle regression and single-pass design target accurate orientation estimation and real-time efficiency.
- Point Cloud Preprocessing: The preprocessing projects Velodyne point clouds into an 80m x 40m birds-eye-view RGB-map with 8cm grid resolution.The three channels encode maximum height, intensity, and normalized point density.
- Network Architecture: The network uses the birds-eye-view RGB-map as input to a simplified YOLOv2 CNN extended with complex angle regression and an E-RPN.The E-RPN predicts 3D position, width, length, probability, class scores, and orientation from the incoming feature map.
- Network Architecture: Each 32x16 grid cell predicts five objects, with 75 output features per cell including regression, probability, and class-score features.The predictions use three size priors and two angle directions as anchor-box priors without increasing the number of predictions.
- Complex Angle Regression: The E-RPN represents orientation with imaginary and real regression fractions and computes the angle using arctan2(t_im, t_re).This closed complex-space representation is intended to avoid singularities and improve angle generalization.
- Loss Function: The training loss combines the YOLO loss with an Euler regression loss that compares predicted and ground-truth complex numbers on the unit circle.The coordinate term is scaled for stable early convergence, and rotated-box IoU uses 2D polygon intersection and union.
- Efficiency Design: A single end-to-end network predicts all bounding boxes in one inference pass, producing higher frame rates than sliding-window proposal methods while retaining comparable mAP.The authors evaluate the approach on KITTI using a Titan X and an NVIDIA TX2 board.
3 Training & Experiments
Complex-YOLO was evaluated on KITTI using a birds-eye-view representation of Velodyne point clouds, with training, validation, and runtime comparisons designed around real-time detection.
- Dataset: KITTI provides 7,481 training samples and 7,518 test samples with Velodyne point clouds, evaluating cars, pedestrians, and cyclists across three difficulty levels.The difficulty levels consider object size, distance, occlusion, and truncation.
- Training: An 85%/15% training-validation split was used because the model was trained from scratch and targeted multi-class predictions.The implementation used stochastic gradient descent with weight decay 0.0005 and momentum 0.9.
- Data limitations: The annotation distribution inherits two blind spots from the camera field of view, and more than 90% of objects have orientations similar to the recording car.Pedestrian representation below 15% is described as disadvantageous.
- Protocol: The evaluation filters objects not visible in the image plane and ignores some objects beyond 40m from the sensor.The official KITTI IoU thresholds are 0.7 for cars and 0.5 for pedestrians and cyclists.
- Runtime results: About 0.02s runtime on a Titan X made Complex-YOLO five times faster than AVOD, more than 10 times faster than VoxelNet, and faster than MV3D.MV3D required 18 times as long, while the model achieved 4fps on an embedded TX2 platform.
3D Object Detection.
Table 3 compares 3D object-detection APs for Complex-YOLO and current leading methods under the KITTI evaluation setup.
- 3D Object Detection: Table 3 reports APs in percent for 3D object detection, comparing the authors’ split validation set with leading methods evaluated on the official KITTI test set.The supplied passage specifies the comparison setup but does not include the table’s individual AP values.
4 Conclusion
The paper presents Complex-YOLO as a real-time, Lidar-only 3D detector that uses E-RPN for orientation estimation and supports simultaneous multi-class prediction.
- Conclusion: Complex-YOLO is presented as a real-time deep-learning model for 3D object detection using Lidar point clouds without additional sensors.The model reports more than 50 fps on an NVIDIA Titan X.
- Conclusion: E-RPN estimates object orientations through real and imaginary regression components in a closed complex space without singularities.The paper connects this design to robust angle prediction.
- Conclusion: The approach detects eight KITTI classes simultaneously in one forward path and reaches 4 fps on an NVIDIA TX2 embedded platform.The listed classes include cars, vans, pedestrians, cyclists, trucks, trams, sitting pedestrians, and miscellaneous objects.