Source-linked AI summary
RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization
Yizhou Wang, Zhongyu Jiang, Yudong Li, Jenq-Neng Hwang, Guanbin Xing, Hui Liu
TL;DR
Vision-based detection suffers under adverse lighting and weather, motivating robust object detection from radar data alone. RODNet processes RF images with specialized modules and camera-radar fusion supervision, achieving 85.98% AP and 87.86% AR against baseline methods.
Problem
Vision-based detection suffers under adverse lighting and weather, motivating exploration of pure radar perception for autonomous and assisted driving.
Method
RODNet detects objects from radar RF images using M-Net and temporal deformable convolution, with training supervision from camera-radar fusion and generated confidence maps.
Results
85.98% AP and 87.86% AR were achieved, outperforming baseline methods and sustaining performance on medium and hard testing sets.
Takeaways & Limitations
RODNet accurately and robustly detects multiple object classes across parking lots, campus roads, and city streets under varied lighting conditions.
Abstract
from arXiv · showhide
Various autonomous or assisted driving strategies have been facilitated through the accurate and reliable perception of the environment around a vehicle. Among the commonly used sensors, radar has usually been considered as a robust and cost-effective solution even in adverse driving scenarios, e.g., weak/strong lighting or bad weather. Instead of considering to fuse the unreliable information from all available sensors, perception from pure radar data becomes a valuable alternative that is worth exploring. In this paper, we propose a deep radar object detection network, named RODNet, which is cross-supervised by a camera-radar fused algorithm without laborious annotation efforts, to effectively detect objects from the radio frequency (RF) images in real-time. First, the raw signals captured by millimeter-wave radars are transformed to RF images in range-azimuth coordinates. Second, our proposed RODNet takes a sequence of RF images as the input to predict the likelihood of objects in the radar field of view (FoV). Two customized modules are also added to handle multi-chirp information and object relative motion. Instead of using human-labeled ground truth for training, the proposed RODNet is cross-supervised by a novel 3D localization of detected objects using a camera-radar fusion (CRF) strategy in the training stage. Finally, we propose a method to evaluate the object detection performance of the RODNet. Due to no existing public dataset available for our task, we create a new dataset, named CRUW, which contains synchronized RGB and RF image sequences in various driving scenarios. With intensive experiments, our proposed cross-supervised RODNet achieves 86% average precision and 88% average recall of object detection performance, which shows the robustness to noisy scenarios in various driving conditions.
I. INTRODUCTION
The paper presents radar-only object detection for robust perception in adverse conditions, using cross-modal camera-radar supervision instead of laborious manual annotation. It contributes RODNet, specialized RF-image processing modules, the CRUW dataset, and an evaluation method.
- Radar remains reliable across varied weather and lighting, motivating object detection based solely on radar signals rather than relying on sensor fusion.
- RODNet uses a teacher pipeline that fuses camera and radar to estimate object classes and 3D locations, supervising a student network that uses only RF images.
- M-Net and temporal deformable convolution exploit multi-chirp information and object relative motion in RF image sequences.
- The CRF supervision framework avoids laborious manual labels by generating radar object annotations through camera-radar fusion and 3D localization.
- The work introduces CRUW, a synchronized camera-radar dataset, and a new evaluation method for RF-image radar object detection.
II. RELATED WORKS
Prior work spans image, LiDAR, radar, and cross-modal perception, while this paper develops radar-only object detection using RF signals and a dedicated detection pipeline.
- Image and vision-based methods: Image-based detectors identify object classes and precise locations from RGB images, while tracking-by-detection associates detections across consecutive frames.The TNT tracker is implemented to improve subsequent 3D localization by recovering or correcting missing and erroneous detections.
- 3D localization: 3D localization methods estimate vehicle structures such as 3D bounding boxes or keypoints, or use monocular structure-from-motion approaches.
- Radar perception: Radar classification methods use handcrafted features or neural networks, but they assume an object has already been identified and do not perform object detection.
- Cross-modal learning: Cross-modal learning transfers information between modalities, while RF-Pose uses camera-generated annotations for WiFi-based human pose estimation.Radar object detection is more challenging because it requires object classification while accounting for motion and texture information.
- Datasets and proposed direction: Most driving datasets lack radar signals, motivating the CRUW dataset and the paper’s radar object detection pipeline.The pipeline section describes RF-image preprocessing, RODNet components, customized modules, and location-based non-maximum suppression.
A. Radar Signal Processing and Properties
The paper converts complex FMCW radar signals into RF image sequences and designs RODNet to handle their temporal, spatial-resolution, and complex-valued properties.
- RF image representation: RF images represent radar reflections in range-azimuth coordinates, with azimuth on the x-axis and range on the y-axis.Two FFT stages estimate reflection range and azimuth after low-pass filtering, producing image-like sequences compatible with CNNs.
- Radar properties: Radar RF data contain rich Doppler-based motion information linked to object texture, size, and shape, requiring multiple consecutive frames as input.
- Radar properties: Radar resolution is high in range but low in azimuth because of hardware constraints such as antenna count and spacing.
- Radar properties: Radar signals are complex-valued and encode frequency and phase information, which is unusual for typical CNN architectures.
- RODNet design: RODNet therefore must extract temporal information, handle multiple spatial scales, and process complex-number data.Its architecture uses a 3D CNN autoencoder with skip connections and temporal inception convolutions, taking RF-image snippets as input.
- RODNet outputs and training: RODNet predicts class-specific confidence maps from RF-image snippets and trains them with binary cross-entropy against teacher-derived binary annotations.
C. M-Net Module
M-Net merges information from multiple radar chirps before RODNet processes temporal RF-image snippets. The module extracts chirp-wise temporal features and max-pools them into one feature map per radar frame.
- C. M-Net Module: M-Net integrates all n chirps from one radar frame before the resulting frame features enter RODNet.For n = 1, one chirp is randomly selected; for n > 1, M-Net merges the chirps.
- C. M-Net Module: A temporal convolution extracts motion features across chirps, followed by temporal max-pooling that produces one merged feature map.The operation is described as resembling Doppler-compensated FFT while remaining trainable end-to-end.
- C. M-Net Module: The merged features from every frame are concatenated along time and sent to the subsequent RODNet components.
- D. Temporal Deformable Convolution: TDC addresses object motion within RF snippets because classical 3D convolution extracts features from a fixed regular cuboid.TDC generalizes deformable convolution to 3D CNNs by adding learnable offsets to receptive-field locations.
- D. Temporal Deformable Convolution: TDC offsets deform locations within each temporal frame rather than shifting receptive locations across frames.The offset vectors are 2D, giving the overall offset field dimension (2N, T, H, W).
E. Post-processing by Location-based NMS
RODNet converts ConfMap predictions into detections with location-based NMS. OLS measures similarity using distance, object scale, and class-specific tolerance before nearby lower-confidence peaks are suppressed.
- E. Post-processing by Location-based NMS: OLS replaces IoU for RF detections by measuring similarity between points using distance, scale, and class information.Distance d is measured in meters, s represents object distance from the radar, and κcls specifies per-class error tolerance.
- E. Post-processing by Location-based NMS: L-NMS first collects 8-neighbor peaks across all class channels, then retains the highest-confidence peak and removes nearby peaks exceeding the OLS threshold.The selection and suppression cycle repeats until no candidate peaks remain.
- E. Post-processing by Location-based NMS: The CRF teacher pipeline provides object classes and more accurate 3D locations than the camera-only and radar-only alternatives.
- E. Post-processing by Location-based NMS: Overlapping RF snippets can produce multiple ConfMap predictions for one radar frame, which are averaged to trade inference speed for robustness.
IV. CROSS-MODAL SUPERVISION
The teacher pipeline generates object classes and 3D locations by combining camera-based estimates with radar detections. These fused annotations supervise RODNet in radar range-azimuth coordinates.
- Camera-Only Supervision: Camera-only annotations can be inaccurate after projection because sensor-system bias misaligns RF peaks with object centers and cameras degrade under adverse conditions.These limitations motivate camera-radar fusion for more reliable self-annotation.
- Radar Assistance: CFAR detects peaks in RF images but cannot assign object classes and often produces many false positives.Consequently, CFAR detections alone cannot serve as supervision.
- Camera-Radar Fusion: The CRF pipeline projects camera-estimated classes and 3D locations into radar range-azimuth coordinates before combining them with CFAR peaks.The transformation aligns camera BEV coordinates with the calibrated radar origin.
- Probabilistic Fusion: CRF forms separate camera and radar probability maps and fuses them by element-wise product.Fused annotations are obtained by detecting peaks in the resulting per-class probability maps.
P CRF
The fused CRF probability maps are converted into final annotations through peak detection. The paper later evaluates the localization accuracy of both camera-only and CRF annotations.
- Fused annotations are derived by detecting peaks in the fused CRF probability maps.
- The paper compares the mean 3D localization accuracy of camera-only and CRF annotations on the CRUW dataset.
C. ConfMap Generation
RODNet represents radar object locations with class-specific confidence maps and is trained and evaluated using the CRUW RF-image dataset. Experiments report strong radar-only detection across varied scenarios and conditions.
- ConfMap Generation: ConfMaps represent object locations in range-azimuth coordinates with separate channels for classes such as cars, pedestrians, and cyclists.The representation is designed to provide a network-compatible target for radar object detection.
- CRUW Dataset: CRUW contains synchronized camera-radar data in RF-image format collected across campus roads, city streets, highways, and parking lots.The dataset includes about 400K frames collected over 3.5 hours at 30 FPS.
- CRUW Dataset: Vision-hard test sequences include weak or strong lighting, blur, and other poor-image-quality conditions.These sequences are reserved for testing radar reliability when vision techniques are likely to fail.
- Evaluation: AP and AR are the main radar detection metrics, computed across OLS thresholds from 0.5 to 0.9.OLS replaces image-based IoU for matching detections with ground truth.
- Results: 85.98% AP and 87.86% AR are achieved by RODNet, outperforming radar-only baselines and sustaining performance on medium and hard test sets.The reported model includes CRF supervision, M-Net, TDC, and temporal inception CNN.
- Results: RODNet qualitatively localizes and classifies multiple objects across parking lots, campus roads, and city streets under different lighting conditions.
C. Ablation Studies
The ablation study examines how RODNet components affect detection performance and computational cost. It also considers the trade-off between overlapping temporal input frames, robustness, and inference speed.
- Ablation Setup: The study evaluates combinations of backbone, supervision, and functional modules to measure their performance influence and computational complexity.
- Backbone: About 5% better performance is obtained with the hourglass backbone than with the vanilla backbone.
- Supervision: About 8% improvement is obtained when RODNet is trained with CRF supervision.
- Customized Modules: Approximately 1%−2% improvement is obtained from each of M-Net, TDC, and temporal inception.
- Speed Trade-off: More overlapped RF frames produce more robust detections but increase inference time during real-time implementation.Inference time below 100 ms is used as the real-time criterion.
2) RF Snippet Length:
RODNet’s detection performance depends on the temporal length and chirp count of its RF input, with overly short or long snippets limiting feature extraction and larger chirp counts eventually saturating.
- RF Snippet Length: Short RF snippets produce low AP because they contain insufficient temporal radar information.
- RF Snippet Length: AP eventually drops when the RF snippet becomes too long to extract efficient radar-detection features.
- Number of Chirps: Performance improves as the number of chirps increases, but becomes flat beyond 8 chirps.
- Qualitative Results: RODNet qualitatively detects objects in difficult night, occlusion, truncation, strong-lighting, and noisy-background conditions.
- Conclusion: The conclusion describes RODNet as a radar-only detector cross-supervised by camera-radar fusion for varied autonomous and assisted-driving scenarios.
APPENDIX A NETWORK ARCHITECTURE IMPLEMENTATION
The appendix specifies RODNet’s vanilla and hourglass backbones, temporal modules, implementation settings, and the teacher-pipeline components used for cross-supervision.
- Network Architecture: RODNet uses vanilla and hourglass backbones, with Conv3D, temporal deformable convolution, transpose 3D convolution, and skip-connection layers.
- Temporal Inception Convolution: Temporal inception convolution layers use 160 input and output channels.
- Implementation: Training takes about 4 days for the vanilla model, 8 days for the hourglass model, and 10 days for the hourglass model with temporal inception layers.
- Teacher Pipeline: The teacher pipeline includes image-based detection, depth estimation, semantic segmentation, and multi-object tracking.
APPENDIX C TEMPORAL DEFORMABLE CONVOLUTION
The temporal deformable convolution appendix describes gradient computation for the offset field and bilinear interpolation over the 3D feature map.
- Gradient Computation: The offset-field gradient is computed according to the temporal deformable convolution formulation.
- Bilinear Interpolation: The interpolation formulation enumerates integer locations q in the 3D feature map and applies a bilinear kernel G to the 2D offset field.
- Bilinear Interpolation: The kernel uses g(a, b) = max(0, 1 −|a −b|).
- Qualitative Scope: Qualitative results cover a 0-25m, ±90° radar FoV across parking lots, campus roads, and city streets under different lighting conditions.
- Failure Cases: Failure cases include missed nearby pedestrians, missed cars with abnormal surfaces or occluding reflections, and false positives from traffic signs.