Source-linked AI summary
Complexer-YOLO: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds
Martin Simon, Karl Amende, Andrea Kraus, Jens Honer, Timo Sämann, Hauke Kaulbersch, Stefan Milz, Horst Michael Gross
TL;DR
Autonomous-driving systems need precise, real-time 3D environmental perception, but Lidar point clouds are sparse and unordered. Complexer-YOLO fuses voxelized Lidar with visual semantic segmentation, adds SRTs and online tracking, and reports state-of-the-art performance across related categories while remaining real-time.
Problem
Autonomous driving requires precise real-time 3D environmental perception, while Lidar point clouds are unordered, sparse, and variably dense.
Method
Complexer-YOLO fuses voxelized Lidar features with camera-based semantic segmentation, predicts 3D boxes, evaluates them with SRTs, and applies online multitarget tracking.
Results
The pipeline reports state-of-the-art results for semantic segmentation, 3D detection, and multitarget tracking while maintaining real-time capability.
Takeaways & Limitations
The architecture provides a complete tracked real-time 3D detection pipeline operating on point clouds fused with visual semantic segmentation.
Takeaways & Limitations
IoU can assign a perfect match to equally sized and positioned boxes with opposite orientations, while exact rotated 3D IoU is time-consuming.
Abstract
from arXiv · showhide
Accurate detection of 3D objects is a fundamental problem in computer vision and has an enormous impact on autonomous cars, augmented/virtual reality and many applications in robotics. In this work we present a novel fusion of neural network based state-of-the-art 3D detector and visual semantic segmentation in the context of autonomous driving. Additionally, we introduce Scale-Rotation-Translation score (SRTs), a fast and highly parameterizable evaluation metric for comparison of object detections, which speeds up our inference time up to 20\% and halves training time. On top, we apply state-of-the-art online multi target feature tracking on the object measurements to further increase accuracy and robustness utilizing temporal information. Our experiments on KITTI show that we achieve same results as state-of-the-art in all related categories, while maintaining the performance and accuracy trade-off and still run in real-time. Furthermore, our model is the first one that fuses visual semantic with 3D object detection.
1. Introduction
Complexer-YOLO targets real-time autonomous-driving perception by combining voxelized Lidar with visual semantic features for 3D detection and tracking. It also introduces SRTs to evaluate 3D boxes while preserving real-time performance.
- Motivation: Real-time environmental perception is required for highly automated vehicles operating across conceivable scenarios.The paper emphasizes precise perception and real-time performance as requirements for full automation.
- Approach: Complexer-YOLO fuses Lidar point clouds with camera-based semantic segmentation to produce semantic point-cloud inputs for 3D detection.Visual class features are generated by fast semantic segmentation and incorporated into the voxelized input.
- Approach: The detector extends regression to predict 3D box heights and z-offsets, enabling targets to be treated in three dimensions.These parameters are incorporated into the multi-part loss function.
- Evaluation: SRTs evaluates 3D detections through scale, rotation, and translation while considering object yaw, width, height, and length.The metric is presented as a faster alternative to IoU for 3D boxes.
- Tracking: The complete pipeline combines frame-wise 3D multiclass predictions with online feature tracking and 3D object-instance tracking.The tracking pipeline uses Lidar and RGB streams, a joint object and extended target model, and an environmental model.
- Results: The authors report state-of-the-art results for semantic segmentation, 3D detection, and multitarget tracking while maintaining real-time capability.They also state that the pipeline can be brought directly into self-driving cars perceiving urban scenes.
2. Related Work
Related work covers CNN-based detection, semantic segmentation, 3D object detection, sensor fusion, and online multitarget tracking. It also introduces random-finite-set formulations for unordered targets and measurements.
- Object Detection: Two-stage detectors use proposals and region-of-interest features, whereas one-stage detectors directly predict detections with a runtime advantage.One-stage detectors are described as generally capable of real-time performance, while often trailing two-stage detectors in accuracy.
- 3D Detection: CNN-based 3D detection methods process stereo images, voxelized point clouds, projected depth maps, or fused camera and point-cloud representations.Examples include voxel feature encoding, front-view projections, multiview representations, and viewing-frustum aggregation.
- Semantic Segmentation: Semantic segmentation classifies each image pixel into a predefined class, with architectures balancing accuracy against runtime.ENet uses an encoder-decoder structure to reduce computational effort, and channel pruning can improve its efficiency.
- Multitarget Tracking: Online multitarget tracking associates detections across frames without access to future frames, making it usable in real-world scenarios.The tracking task is presented as detecting objects and then associating identical objects between frames.
- Random-Finite-Set Tracking: Targets and measurements are modeled as unordered labeled multitarget Bernoulli random finite sets.A Bernoulli random finite set is either empty or contains one element, while labels support individual trajectory estimation.
- Random-Finite-Set Tracking: The multitarget Bayes filter approximates the multitarget distribution at each time step using measurement and transition densities.The formulation includes Chapman–Kolmogorov prediction and models measurements as target-generated or clutter with unknown origin.
3. Joint Detection and Extended Target Model
The system voxelizes semantic point clouds for 3D detection, then tracks detected boxes with an extended-target Bayesian model. It replaces rotated-box IoU with the parameterizable SRTs metric, combining scale, translation, and rotation scores.
- Voxel based Complex-YOLO: The detection pipeline uses 49 convolutional layers, residual connections, and regression targets for object height and ground offset.Max-pooling layers are replaced by stride-2 convolutions, and height and ground offset are included in the multi-part loss.
- Scale-Rotation-Translation score: SRTs combines independent scaling, rotation, and translation scores into a parameterizable 3D-box evaluation metric.The translation component uses object-center distance normalized by object size, while the metric includes a penalty when objects do not intersect.
- Extended target model in the LMB RFS: The tracker interprets YOLO boxes as Gaussian-noise-corrupted measurements and predicts targets with an Unscented Kalman filter under a coordinated-turn model.Measurements contain box center, dimensions, and orientation; target association uses update likelihoods and can spawn a new target when nonassignment probability exceeds Pna.
4. Experiments
Experiments evaluate Complexer-YOLO on KITTI across detection, orientation, tracking, runtime, qualitative behavior, ablations, and training objectives. The results show real-time operation, semantic-feature benefits, SRTs efficiency gains, and evaluation limitations tied to ignored objects and rotation modeling.
- Evaluation setup: KITTI evaluation covers orientation, BEV, 3D detection, object tracking, runtime, qualitative sequences, and ablation studies.The official test set is used for benchmark submissions, while validation data supports input-feature ablations.
- Detection and tracking results: 79.31% AP for Car in moderate difficulty and 79.08% Orientation are reported, while BEV and 3D detection reach 66.07% and 49.44%, respectively.The 2D-based evaluation produces state-of-the-art tracking results in real time, but the authors identify inconsistencies with BEV and 3D results.
- Evaluation limitations: The benchmark can miscount detections because ignored Dontcare objects are treated as false positives and refined 2D pedestrian boxes may not match reprojected 3D boxes.These inconsistencies are especially consequential for evaluating detections produced in 3D space through 2D image-plane projections.
- Qualitative results: Semantic voxel features support accurate rotated 3D boxes across multiple classes and detection of small pedestrians or cyclists when objects have sufficient spatial separation.Qualitative results are visualized after projecting detections into camera space with pixel-wise semantic segmentation overlays.
- Ablation study: 21 height channels achieve similar mAP at IoU 0.7 to 51 channels while providing the best runtime–accuracy trade-off on the tested hardware.Runtime increases slightly beyond 21 height channels, suggesting limited use of finer-grained height information.
- Ablation study: 1.3% average improvement follows filtering detections with fewer than 13 points and less than 52m radial distance, but BEV Car moderate AP decreases by 4.8%.The filter removes some ignored detections as well as correct detections, with stronger effects in moderate settings.
- Ablation study: Restricting rotations to anchor-defined subsections decreases accuracy, leaving the potential complexity reduction for the learning task as an open question.The authors state that further investigation is required.
5. Conclusion
Complexer-YOLO is a tracked real-time 3D detector that fuses point-cloud data with visual semantic segmentation. It achieves competitive KITTI benchmark performance while introducing a flexible object-comparison metric.
- Complexer-YOLO operates as a tracked real-time 3D object detector on point clouds fused with visual semantic segmentation.
- Its detection results show competitive performance on KITTI benchmarks compared to state-of-the-art methods.
- The paper introduces SRTs, a flexible and simplified metric for comparing objects that addresses limitations of IoU.