Source-linked AI summary
A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection
Felix Nobis, Maximilian Geisslinger, Markus Weber, Johannes Betz, Markus Lienkamp
TL;DR
Camera detection is vulnerable to severe weather and low-light noise, motivating radar-camera fusion. CRF-Net projects sparse radar into image space, learns the fusion depth, and uses BlackIn; fusion outperforms the image-only baseline across two datasets, with gains affected by filtering and dataset conditions.
Problem
Camera sensors have limited quality in severe weather and sparse lighting, while radar is more robust to these conditions.
Method
CRF-Net fuses projected sparse radar and camera data within a RetinaNet-based network, learning fusion depth and using BlackIn to focus training on radar.
Results
Fusion outperforms the image-only baseline on two datasets, including a 12.96 %-point mAP advantage on nuScenes with annotation- and ground-truth-filtered data.
Takeaways & Limitations
Neural fusion of radar and camera data can augment the detection score of a state-of-the-art image-only object-detection network.
Takeaways & Limitations
Robustness to spatial and temporal sensor miscalibration remains to be evaluated, and additional adverse-weather datasets are needed to study that assumption.
Abstract
from arXiv · showhide
Object detection in camera images, using deep learning has been proven successfully in recent years. Rising detection rates and computationally efficient network structures are pushing this technique towards application in production vehicles. Nevertheless, the sensor quality of the camera is limited in severe weather conditions and through increased sensor noise in sparsely lit areas and at night. Our approach enhances current 2D object detection networks by fusing camera data and projected sparse radar data in the network layers. The proposed CameraRadarFusionNet (CRF-Net) automatically learns at which level the fusion of the sensor data is most beneficial for the detection result. Additionally, we introduce BlackIn, a training strategy inspired by Dropout, which focuses the learning on a specific sensor type. We show that the fusion network is able to outperform a state-of-the-art image-only network for two different datasets. The code for this research will be made available to the public at: https://github.com/TUMFTM/CameraRadarFusionNet.
I. INTRODUCTION
Camera-based detection is accurate but vulnerable to adverse weather and low-light noise. The paper investigates radar-camera fusion to improve object detection reliability in these conditions.
- Camera object detection is accurate, but heavy rain, fog, and sparse lighting reduce camera visibility or increase noise.
- Radar is more robust than cameras to lighting changes, rain, and fog, while directly measuring object distance and radial velocity.
- The proposed network jointly processes camera and radar data to improve object detection accuracy.
- CRF-Net detects objects more reliably on the nuScenes and TUM datasets, while the paper also discusses limitations and future development.
II. RELATED WORK
Prior work adapts neural networks and sensor-fusion representations across modalities, including radar and lidar. This paper focuses on low-level radar-camera fusion in image space and learns where fusion is most beneficial.
- Single-shot object-detection architectures aim to balance accurate detection with reasonably low computational time for real-time applications.
- Earlier fusion methods project lidar or radar into image-related representations, guide image detection, or associate independent sensor detections.
- Radar data is transformed onto a regular grid before neural-network processing, following the grid-conversion approach used for unstructured lidar points.
- Raw-level fusion of image and radar data has received limited investigation, motivating further research into this fusion setting.
- The paper projects radar onto the camera's vertical image plane, learns the network depth for beneficial fusion, and uses 2D labels to facilitate training-data generation.
III. RADAR DATA PREPROCESSING
The preprocessing pipeline projects sparse radar returns into the camera image plane, represents radar characteristics as channels, and addresses calibration, missing height, sparsity, and clutter. Filtering choices expose important data-quality constraints.
- Radar azimuth, distance, and radar cross section are projected from the ground plane into a perpendicular image plane and stored as augmented-image pixel values.
- Pixels without radar returns receive zero-valued projected radar channels, preserving the sparse nature of the radar input.
- nuScenes provides calibration matrices, whereas the TUM fish-eye data requires a nonlinear calibration method for world-to-image mapping.
- Radar detections are extended vertically from an assumed ground-plane origin because radar provides no return height information.
- The front radar averages 57 detections per cycle in nuScenes mini, creating a density mismatch with the camera and motivating careful fusion design.
- The evaluation compares raw noisy radar with annotation-filtered and ground-truth-filtered data because clutter includes ghost, irrelevant, and ground detections.
- Spatial miscalibration, asynchronous sensing, radar inaccuracies, and 13-cycle aggregation can misalign relevant detections with ground-truth boxes.
IV. NETWORK FUSION ARCHITECTURE
CRF-Net extends RetinaNet with projected radar channels and learns where to fuse them within the network. BlackIn randomly removes camera input during training to increase learning from sparse radar data.
- CRF-Net builds on RetinaNet with a VGG backbone, augmented radar channels, 2D bounding-box regression, classification scores, and focal loss.
- Early concatenation may be suboptimal because radar measurements and camera pixels differ in semantic information content.
- The fusion architecture adapts weights across network depths so radar features can be fused where they have the greatest impact.
- The high-level CRF-Net structure is presented as the architecture for fusing camera and radar data.
- BlackIn deactivates all camera-input neurons for random training steps in 20% of training images.
- Removing camera input pushes the network to rely more on radar and learn the information value of sparse radar independently of dense camera data.
V. EXPERIMENTS AND RESULTS
The evaluation compares CameraRadarFusionNet (CRF-Net) with an adapted RetinaNet image-only baseline on nuScenes and a second dataset collected for this research.
- CRF-Net is evaluated against an adapted RetinaNet baseline on nuScenes and the TUM dataset.
A. Datasets
The experiments use nuScenes and a second TUM dataset, with aligned object classes but different annotation and camera characteristics.
- nuScenes data come from Boston and Singapore and are evaluated with condensed object classes, with and without ground-truth filters.
- Table I reports the number of objects per class in nuScenes and the TUM dataset.
- The TUM dataset uses the same evaluation classes, 2D CVAT annotations, no applicable 3D ground-truth filter, and halved RetinaNet anchor sizes.
B. Training
Training balances nuScenes scene conditions, adjusts image resolution and class weighting, uses pretrained VGG features, and runs separate training schedules for both datasets.
- nuScenes is split 60:20:20 to balance day, rain, and night scenes, using 360 x 640 inputs; TUM uses 720 x 1280 fish-eye images.
- Object classes are weighted according to their dataset-specific frequencies for mean Average Precision calculation.
- VGG weights are pretrained on ImageNet, camera channels are scaled to [-127.5,127.5], radar channels remain unscaled, and TUM receives augmentation.
- nuScenes training runs for 25 epochs, while TUM training runs for 50 epochs, both with batch size 1.
C. Evaluation
CRF-Net generally improves detection over the image baseline, with the largest reported gain under filtered radar conditions, while radar preprocessing adds execution time and noisy detections remain a limitation.
- 12.96 %-points is the CRF-Net mAP advantage over the image baseline when annotation and ground-truth radar filters are applied on nuScenes.
- Removing radar metadata such as distance and RCS lowers mAP, indicating these characteristics contribute to the detection result.
- 1.4 %-points is the fusion network’s performance gain over the baseline on the TUM dataset.
- Figure 4 qualitatively compares baseline and CRF-Net detections, with the baseline missing the pedestrian on the left.
- The authors motivate future filtering of noisy radar detections before or inside the fusion network for real-world application.
- The baseline processes a 360 x 640 image in 33 ms, compared with 43 ms for CRF-Net plus 56 ms for radar projection and channel generation.
VI. CONCLUSIONS AND OUTLOOK
CRF-Net fuses camera and radar data in a neural network and augments detection scores, while highlighting open directions for improving radar processing, sensor robustness, sensing hardware, and multimodal evaluation.
- Conclusions: CRF-Net fuses camera and radar data for road-vehicle object detection and can augment the detection score of a state-of-the-art object detection network.The paper identifies neural fusion of radar and camera data as a direction requiring further network-architecture research.
- Outlook: Future work will process radar data before fusion to filter noise and investigate robustness against spatial and temporal sensor miscalibration.These issues are presented as areas needing further evaluation or development.
- Outlook: Additional adverse-weather datasets are needed to study the potential of multimodal neural fusion for driving in such conditions.The paper frames adverse-weather robustness as an assumption requiring dedicated datasets for evaluation.
- Outlook: Adding lidar could further increase detection accuracy but would also add complexity through additional layers or new design concepts.This trade-off motivates further exploration of additional sensor modalities.
- Outlook: High-resolution or imaging radars are expected to increase radar information density, reduce clutter, and improve the approach's detection results.The expected hardware benefit is described as a future development rather than a demonstrated result in this study.
CONTRIBUTIONS AND ACKNOWLEDGMENTS
The research was conceived collaboratively, with implementation, thesis work, experimental results, critical revision, and project conception distributed among the listed contributors.
- Contributions: Felix Nobis initiated the paper's idea and contributed essentially to its conception and content.
- Contributions: Maximilian Geisslinger and Markus Weber contributed through their master theses, conception, implementation, and experimental results.
- Contributions: Johannes Betz critically revised the paper.
- Contributions: Markus Lienkamp contributed to the research project's conception and critically revised the paper for important intellectual content.