Source-linked AI summary
CNN based Road User Detection using the 3D Radar Cube
Andras Palffy, Jiaao Dong, Julian F. P. Kooij, Dariu M. Gavrila
TL;DR
Existing radar road-user detection methods do not simultaneously avoid clustering-dependent classification and support low-latency urban driving. This paper combines target-level features with cropped 3D radar-cube data for individual target classification, then performs class-aware clustering, achieving higher target-wise and object-wise F1 scores than the baseline.
Problem
Existing methods do not simultaneously avoid error-prone clustering for classification and operate within one or two radar sweeps, or 75–150 ms, for urban driving.
Method
A CNN classifies each radar target from target-level features and a cropped 3D radar-cube block, after which clustering uses predicted class information to form object proposals.
Results
The method reaches 0.70 target-wise average F1 versus 0.68 for Schumann and 0.56 object-wise average F1 versus 0.48.
Takeaways & Limitations
Low-level radar features and ensembling are important, while class-specific clustering supports the method’s overall object-detection advantage.
Takeaways & Limitations
A single mis-classified radar target, probably reflected by a speed bump, produced a false-positive pedestrian detection.
Abstract
from arXiv · showhide
This letter presents a novel radar based, single-frame, multi-class detection method for moving road users (pedestrian, cyclist, car), which utilizes low-level radar cube data. The method provides class information both on the radar target- and object-level. Radar targets are classified individually after extending the target features with a cropped block of the 3D radar cube around their positions, thereby capturing the motion of moving parts in the local velocity distribution. A Convolutional Neural Network (CNN) is proposed for this classification step. Afterwards, object proposals are generated with a clustering step, which not only considers the radar targets' positions and velocities, but their calculated class scores as well. In experiments on a real-life dataset we demonstrate that our method outperforms the state-of-the-art methods both target- and object-wise by reaching an average of 0.70 (baseline: 0.68) target-wise and 0.56 (baseline: 0.48) object-wise F1 score. Furthermore, we examine the importance of the used features in an ablation study.
I. INTRODUCTION
The paper combines accurate target-level radar features with full 3D radar-cube information to classify individual road-user reflections before class-aware clustering. This addresses weaknesses of cluster-wise classification while producing both target- and object-level class information.
- Automotive radar offers robustness to rain, snow, and darkness while directly measuring radial velocity through the Doppler effect.
- Radar targets provide accurate range, azimuth, reflectivity, and radial speed, but individual reflections are insufficient to classify complete objects.
- The 3D radar cube represents reflectivity across range, azimuth, and velocity, capturing local speed distributions from moving limbs or wheels.
- Low-level radar features offer richer Doppler information but have lower range and azimuth resolution and unresolved phase ambiguity.
- The proposed RTCnet fuses target-level features with a cropped full 3D radar-cube block to classify each radar target before object clustering.
- Target-level labels support intermediate sensor fusion, while class-specific clustering produces segmented and classified object proposals for later fusion.
II. RELATED WORK
Prior radar road-user detectors commonly classify clusters, but clustering can merge or split objects and requires parameters shared across classes. The paper targets low-latency, single-frame detection without clustering-dependent classification.
- Earlier work includes static-environment radar methods, but this paper focuses on moving road users.
- Many detectors cluster radar targets with DBSCAN and classify clusters using engineered features or learned classifiers.
- Clustering can merge separate objects, split one object, and require shared parameters despite class-specific spatial and velocity profiles.
- Target-wise classification has been explored, but prior methods used multiple frames or focused only on large road users because radar sweeps are sparse.
- Low-level radar approaches have used Doppler-time, Range-Doppler, and multi-frame spectrograms for road-user classification.
- No reviewed method both avoids error-prone clustering for classification and operates within one or two radar sweeps, or 75–150 ms, for urban driving.
- The proposed contribution combines target-level and low-level data in a CNN and reports target-wise and object-wise performance on a real-world dataset.
III. PROPOSED METHOD
The method fuses target-level radar measurements with a cropped 3D radar-cube neighborhood and classifies each radar target individually using RTCnet.
- Radar targets are mapped into the radar cube, and a smaller block is cropped around each target across range, azimuth, and Doppler.
- RTCnet combines the fused low-level and target-level data to classify radar targets individually.
- The network is organized into three parts that encode spatial and Doppler information for target classification.
A. Pre-processing
Pre-processing selects dynamic radar targets, aligns their target-level measurements with radar-cube bins, and uses RTCnet modules to encode local spatial and Doppler structure.
- A. Pre-processing: A single frame of radar targets and radar-cube data is fetched, and low compensated absolute velocity targets are filtered as static.
- A. Pre-processing: Remaining dynamic targets are connected to corresponding range, azimuth, and Doppler bins before network processing.
- RTCnet: RTCnet uses three modules, beginning by encoding the target neighborhood’s Doppler distribution without extending range or azimuth dimensions.
- RTCnet: A later module extracts class information from the speed distribution around each target using one-dimensional convolutions along Doppler.
3) Score calculation:
RTCnet combines convolutional radar-cube processing with target-level features to produce class scores, using either direct multi-class prediction or binary-classifier ensembling.
- The third module concatenates flattened radar-cube features with target-level features before producing class scores.The target-level inputs are r, α, vr, and RCS; two fully connected layers with 128 nodes each are used.
- RTCnet supports direct multi-class classification through an output layer with four class nodes.The four outputs correspond to the paper’s multi-class setup.
- An ensemble alternative trains 10 One-vs-All and One-vs-One binary classifiers for individual classes and class pairs.The final prediction scores depend on voting across these binary classifiers.
- Bootstrapped ensembling of multi-class classifiers performed worse than the evaluated ensemble approach.
D. Object Clustering
The method forms object proposals by clustering individually classified moving radar targets, while evaluation compares predicted and annotated objects using target-based overlap.
- D. Object Clustering: Class-specific DBSCAN clusters radar targets using spatial position, velocity, and predicted class information.Bike, pedestrian, and car predictions are clustered in separate steps using spatial and velocity thresholds.
- D. Object Clustering: The experiments use a real-world urban-driving dataset recorded with radar, stereo camera, and vehicle odometry.The dataset contains approximately one hour of driving and includes both target-level and low-level radar outputs.
- D. Object Clustering: Training data contain more than 30/15/9 × 10^3 pedestrian/cyclist/car instances, with independently driven training and testing routes.Annotations were automatically generated and manually corrected, and radar frames were augmented by mirroring and Gaussian noise.
- D. Object Clustering: Target-wise evaluation counts a correctly classified radar target as a true positive, while the study also tests feature ablations and the complete clustering pipeline.Only moving radar targets are included, and object-level evaluation uses target-count-based overlap.
- D. Object Clustering: Object-detection evaluation defines true positives as predictions with Intersection Over Union of at least 0.5 against an annotated object.Intersection and union are calculated by counting radar targets, and duplicate detections count as false positives.
A. Baselines
The study compares RTCnet with reimplemented clustering-based baselines, using optimized sensor-specific parameters and target- and object-level evaluation.
- A. Baselines: Schumann is selected as the multi-object, multi-class baseline because it offers small latency.
- A. Baselines: Prophet is used as a second baseline despite being a single-class pedestrian detector, because its negative data include cars, dogs, and cyclists.
- A. Baselines: Both baselines are reimplemented with DBSCAN clustering and cluster-wise classification, then trained on the study’s training set.
- A. Baselines: DBSCAN spatial and velocity thresholds are independently optimized on validation data for each baseline.The optimized settings are reported alongside the class-specific clustering parameters.
- A. Baselines: RTCnet outperforms the baselines on average in target-wise F1, while ablations show benefits from ensembling and low-level data.
B. Implementation
Implementation uses fixed radar-cube crops, feature normalization, supervised CNN training, and a low-latency inference setup.
- B. Implementation: Radar-cube crops use dimensions L = W = 5 and H = 32, with a 0.3 m/s threshold filtering static objects.DBSCAN and cluster-merging thresholds are also specified for the class-specific clustering stage.
- B. Implementation: Features are normalized feature-wise to zero mean and unit standard deviation for target-level inputs and the full radar cube.The normalized training statistics are reused during inference.
- B. Implementation: The network is trained in PyTorch with cross-entropy loss for 10 epochs.
- B. Implementation: Inference takes approximately 0.04 s on the reported high-end PC, including moving-target processing, binary classifiers, and ensembling.
1) Target classification:
RTCnet classifies individual radar targets using target-level and radar-cube information, achieving stronger overall target-wise performance than the cluster-wise baselines. Object-wise evaluation also favors RTCnet overall, despite slightly weaker cyclist performance.
- RTCnet reached a target-wise macro-average F1 score of 0.70, outperforming the two cluster-wise baselines.
- Removing low-level features reduced the target-wise macro-average F1 score from 0.70 to 0.61.
- The target-wise evaluation examined performance across distance using 5 m bins and reported the number of training targets alongside F1 scores.
- RTCnet slightly underperformed Schumann on cyclists but outperformed it on pedestrians, cars, and object-wise average F1.Object-wise scores were 0.59 vs. 0.60 for cyclists, 0.61 vs. 0.54 for pedestrians, 0.47 vs. 0.31 for cars, and 0.56 vs. 0.48 overall.
D. Discussion
The discussion attributes RTCnet’s gains to individual target classification and low-level velocity information, while class-specific clustering improves object proposals. Remaining errors arise from reflections, confusing road-user signatures, and single-frame clustering trade-offs.
- RTCnet avoids dependence on initial clustering and can classify objects represented by a single radar target.
- Removing low-level data reduced average target-wise F1 from 0.70 to 0.61, supporting the value of relative velocity distributions.
- Reflections, unusual vehicles, similar car–cyclist signatures, and nearby strong reflections produce target-classification errors.
- Class-specific clustering parameters let RTCnet separate pedestrians while keeping vehicles together in challenging cases.
- Side-by-side cyclists may be detected as a car, while single-target pedestrian settings can create false positives.
VI. CONCLUSIONS AND FUTURE WORK
The paper proposes a single-frame, multi-class road-user detector that combines cropped 3D radar-cube features with target-level features and clustering. Experiments show gains in target-wise and object-wise classification, while future work targets clustering, temporal integration, and sensor fusion.
- The method applies a CNN to cropped 3D radar-cube blocks and target-level features before clustering radar targets into object proposals.
- Future work includes more advanced clustering, temporal integration or tracking, and additional camera or LiDAR modalities.