Source-linked AI summary
Distant Vehicle Detection Using Radar and Vision
Simon Chadwick, Will Maddern, Paul Newman
TL;DR
Distant vehicles are hard to detect because they appear only a few pixels wide, despite the need for timely autonomous-driving decisions. The paper combines monocular vision with radar and uses nearby cameras of different focal lengths to generate training labels. The resulting fusion methods improve detection of smaller vehicles, while multi-camera labeling improves performance over the original detector.
Problem
Autonomous vehicles need accurate distant-vehicle detection for timely decisions, but image-based CNN detectors struggle with small objects.
Method
The paper trains a detector that fuses monocular images with radar scans and automatically generates labels by combining detections from cameras with different focal lengths.
Results
Radar fusion methods significantly improve detection of smaller vehicles, and all three trained networks outperform the original detector's average precision of 0.118.
Takeaways & Limitations
Combining radar with vision and using multi-camera labels improves vehicle detection performance in the reported experiments.
Abstract
from arXiv · showhide
For autonomous vehicles to be able to operate successfully they need to be aware of other vehicles with sufficient time to make safe, stable plans. Given the possible closing speeds between two vehicles, this necessitates the ability to accurately detect distant vehicles. Many current image-based object detectors using convolutional neural networks exhibit excellent performance on existing datasets such as KITTI. However, the performance of these networks falls when detecting small (distant) objects. We demonstrate that incorporating radar data can boost performance in these difficult situations. We also introduce an efficient automated method for training data generation using cameras of different focal lengths.
I. INTRODUCTION
Distant vehicles are difficult to detect because they occupy only a few pixels, while radar provides complementary motion and weather robustness. The paper therefore combines radar with monocular vision and generates training labels from cameras with different focal lengths.
- Distant vehicles may need to be detected early because high closing speeds leave little time for safe decisions.
- CNN-based detectors perform well broadly but struggle to accurately detect small objects.
- Radar complements vision through weather robustness, range-independent measurement accuracy, and Doppler-based velocity, despite size ambiguity and clutter vulnerabilities.
- The proposed detector uses monocular images and radar scans, addressing the lack of radar in major autonomous-driving datasets.
- Different-focal-length cameras support distant-object labeling by transferring labels from the long focal length view into the canonical image.
II. RELATED WORK
Prior work spans convolutional detectors, automated label generation, and multimodal sensing, while this paper collects synchronized camera and radar data for its own dataset.
- Two-stage detectors such as Faster R-CNN achieve state-of-the-art performance but are computationally expensive, whereas one-stage methods operate in real time with a performance penalty.
- Pre-training is common for object detection, although careful network design can produce state-of-the-art results without it.
- Automated labeling has used visual odometry, temporal consistency, and existing object detectors, with some approaches still requiring manual labeling.
- Prior multimodal systems have fused cameras with LIDAR or combined cameras, LIDAR, and radar for object detection.
- The paper collects synchronized stereo, long-focal-length RGB cameras, and dual-beam pulse-Doppler radar for dataset creation.
A. Object detector
The detector improves distant-vehicle labels by transferring detections from a long-focal-length camera to a nearby wide-angle camera, then combining detections in their overlapping region.
- Label generation: Detections from the long-focal-length camera are transferred to the deployment camera without requiring object range.The cameras’ centres are approximated as coincident, enabling image-point redrawing using their intrinsic matrices and relative rotation.
- Label generation: The transfer approximation introduces an error determined by camera separation, the new camera’s focal length, and object distance.With fA = 625px, d = 0.032, and Z = 20m, the bounding-box offset is ϵ = 1px.
- Combining detections: In the overlapping image region, detections from the higher-resolution long-focal-length camera are used, while short-camera detections are filtered at the border.Short-camera detections overlapping the joint region beyond τ = 0.5 are discarded.
- Combining detections: The combined labels include vehicle and pedestrian detections transferred and merged across the two focal-length views.Figure 3 shows short-camera boxes on the left, long-camera boxes in the middle, and the combined set on the right; the overlap is outlined in green.
- Combining detections: Because distant objects often lie near the image centre, they commonly fall within the cameras’ overlapping region, limiting missed distant objects.The paper notes that the number missed is relatively small compared with the ratio of the two image areas.
C. Radar
Radar targets are projected into the camera image as sparse range and range-rate channels alongside RGB data, providing motion and location cues despite noise and missed detections.
- Radar representation: Each radar target supplies range, bearing, radial velocity, and amplitude, and is projected into camera CA as range and range-rate image channels.Targets are marked as small circles rather than single pixels to increase their influence during learning.
- Radar representation: Platform ego-motion is subtracted from each target’s range rate before projection using stereo visual odometry estimates.Because radar and cameras are unsynchronised, the closest ego-motion estimate is used for each radar scan.
- Radar projection: The projected radar locations are computed with camera projection and radar-to-camera extrinsic calibration matrices applied to homogeneous target coordinates.The projection uses PA, TAR, and XR as defined in the radar projection description.
- Radar limitations: Radar is sparse, noisy, and inconsistent: moving vehicles may be missed, while spurious high range-rate targets can briefly appear.Nevertheless, the signal contains sufficient information to guide vehicle location.
D. Sub-sampling
The data pipeline reduces temporal redundancy by retaining temporally aligned radar-image pairs and subsampling the resulting sequence.
- Temporal sampling: Frames are selected when radar and image timestamps differ by no more than 10ms, then subsampled by a factor of five.Together with the radar-image alignment, this produces approximately 1:10 subsampling from the original radar frequency.
E. Final dataset
The final dataset contains automatically labelled vehicle and pedestrian data collected with cameras and radar, while the model uses a compact SSD-based fusion architecture.
- Dataset construction: 25,076 labelled images were generated from six daytime drives covering urban, suburban, and highway settings.The split contains 17,553 training, 2,508 validation, and 5,015 testing images.
- Dataset scope: The work focuses on vehicles because the training data contain substantially more vehicles than pedestrians and radar performs poorly on pedestrians.The training set contains 53,019 vehicles and 5,559 pedestrians.
- Training data: Automatically generated training examples combine camera detections with radar range and range-rate representations.The radar visualizations include a range image and a range-rate image alongside the camera image and bounding boxes.
- Network design: The detector builds on SSD with ResNet-18 blocks and fuses radar and RGB features through separate radar branches.The tested branch designs concatenate features after the second image block or add them element-wise after the first.
- Detection pipeline: The network produces dense predictions from multiscale features using classification and bounding-box regression losses followed by non-maxima suppression.The final detections are limited to the 200 most confident predictions with an NMS threshold of 0.45.
A. Small objects
Training addresses small-object detection through feature-map box duplication, modality normalization, augmentation, and a two-branch network configuration.
- Small-object handling: Default boxes are duplicated at subdivisions of feature-map cells to counteract SSD’s degradation on smaller objects.This increases the number of candidate boxes assigned to small-object locations.
- Training procedure: Models are trained from scratch with ADAM for 50k iterations using batches of 16.The optimizer settings include β1 = 0.9, β2 = 0.999, ϵ = 10^-8, and learning rate λ = 10^-4.
- Network configuration: The concatenation-fusion network uses a separate radar branch alongside the RGB image branch.For RGB-only networks, the radar branch is removed.
- Training procedure: Image augmentation includes horizontal flips, random crops, resizing, and hue and saturation changes to reduce overfitting.Cropping retains between 0.6 and 1.0 of the full image dimensions before resizing.
- Training procedure: Each modality is scaled using its training-set mean and standard deviation so feature scales remain consistent across modalities.The normalization is applied separately to each sensing modality.
B. Evaluation metrics
Detection performance is measured with AP at an IoU threshold of 0.5 and evaluated by object-size categories, while label quality is compared against hand-labelled frames.
- Metrics: Average precision is computed at an intersection-over-union threshold of 0.5 using the PASCAL VOC2012 integration rule.Higher IoU thresholds are considered especially challenging for objects only a few pixels in size.
- Metrics: Performance is evaluated separately across object-size categories defined by the fraction of image area occupied.This follows the general structure of KITTI’s size-based evaluation while using an area-fraction measure.
- Label evaluation: Generated labels are assessed against 161 hand-labelled ground-truth frames from the test set.The hand-labelling interface uses multi-camera bounding-box transfer to label very small distant objects.
- Label evaluation: Combined multi-camera detections substantially improve automated-label quality, particularly for small vehicles.The comparison uses precision-recall curves generated from probabilities produced by the original detector.
- Label sources: The narrow-field-of-view camera supplies most of the small-object labels when compared with the wide-angle camera.This comparison is performed both on hand-labelled frames and across the entire test set.
- Limitation: The automatically labelled dataset has high precision but low recall, so it cannot be treated as ground truth.The authors nevertheless cite training evidence that missing labels may have a smaller effect on performance than expected.
B. Trained network performance on generated test set
Networks trained on the generated data improve detection of smaller vehicles, with concatenation fusion improving performance across all evaluated size subsets.
- Results: Both radar-fusion methods significantly improve detection of smaller vehicles on the generated test set.The comparison is reported in Table II by object size.
- Results: The concatenation approach improves performance across all object-size subsets.The size-dependent performance trend is illustrated in Fig. 8.
- Interpretation: Radar provides a distinctive signature for moving objects that complements RGB context when identifying small distant objects.The authors specifically associate the improvement with radar’s motion information and image context such as road position.
C. Trained network performance on hand-labelled test set
On the hand-labelled test set, both radar-fusion methods improved detection performance, with element-wise fusion outperforming concatenation; results were lower than on generated data.
- Both radar fusion methods increase performance on the hand-labelled test set.
- Element-wise radar fusion outperforms the concatenation method.
- Results on hand-labelled data are generally lower than on generated data, partly because the ground-truth set contains more very small objects.
- All three networks significantly outperform the original detector, whose average precision is 0.118.
VII. CONCLUSIONS AND FURTHER WORK
The paper concludes that multi-camera automatic labelling enables radar–vision detection improvements over the original detector. It identifies radar representation, temporal processing, and automatically generated label noise as areas for further work.
- Conclusions: The proposed system combines automatic multi-camera labelling with radar–camera fusion to improve vehicle detection performance.
- Further work: Future work may use richer radar representations, consecutive frames, and methods designed to handle noise in automatically generated labels.
- Conclusions: Radar improves recall for small and medium-sized vehicles in precision–recall evaluation on automatically generated labels.
- Evaluation: The generated test set contains proportionally fewer extremely small objects than the hand-labelled ground truth set.