Source-linked AI summary
CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformer
Youngseok Kim, Sanmin Kim, Jun Won Choi, Dongsuk Kum
TL;DR
Camera-radar 3D detection has been limited by heuristic late fusion and the sensors’ differing spatial characteristics. CRAFT performs proposal-level early fusion using polar-coordinate radar association and cross-attention feature exchange, achieving state-of-the-art nuScenes performance. The method’s camera 3D detector is treated as an external component rather than a contribution of the paper.
Problem
Late fusion of separate camera and radar detections cannot fully exploit complementary sensor information, while early fusion must handle radar ambiguity and differing coordinate systems.
Method
CRAFT associates image proposals with radar points using uncertainty-aware adaptive thresholds in polar coordinates, then exchanges camera and radar information through cross-attention fusion layers.
Results
41.1% mAP and 52.3% NDS on the nuScenes test set establish state-of-the-art performance and significantly improve the camera-only baseline.
Takeaways & Limitations
Soft polar association and spatio-contextual fusion robustly complement camera and radar information, including when image proposals are inaccurate or radar returns are sparse.
Takeaways & Limitations
The camera 3D detector is outside the paper’s scope, so improvements in camera detection could further improve the reported results.
Abstract
from arXiv · showhide
Camera and radar sensors have significant advantages in cost, reliability, and maintenance compared to LiDAR. Existing fusion methods often fuse the outputs of single modalities at the result-level, called the late fusion strategy. This can benefit from using off-the-shelf single sensor detection algorithms, but late fusion cannot fully exploit the complementary properties of sensors, thus having limited performance despite the huge potential of camera-radar fusion. Here we propose a novel proposal-level early fusion approach that effectively exploits both spatial and contextual properties of camera and radar for 3D object detection. Our fusion framework first associates image proposal with radar points in the polar coordinate system to efficiently handle the discrepancy between the coordinate system and spatial properties. Using this as a first stage, following consecutive cross-attention based feature fusion layers adaptively exchange spatio-contextual information between camera and radar, leading to a robust and attentive fusion. Our camera-radar fusion approach achieves the state-of-the-art 41.1% mAP and 52.3% NDS on the nuScenes test set, which is 8.7 and 10.8 points higher than the camera-only baseline, as well as yielding competitive performance on the LiDAR method.
1 Introduction
Camera-radar fusion is promising because the sensors offer complementary spatial and contextual information, but existing late-fusion approaches do not fully exploit it. CRAFT addresses this with proposal-level early fusion tailored to radar ambiguity and sensor-coordinate differences.
- Motivation: Camera and radar combine low-cost, reliable sensing with complementary semantic, range, weather-robustness, and depth-related properties.Cameras provide dense, azimuthally accurate pixels and rich semantics, while radar offers long-range detection and weather robustness.
- Limitations of Existing Fusion: Late fusion combines separate detection outputs using heuristic logic, creating performance-reliability trade-offs when sensor predictions disagree.Such post-processing cannot fully exploit the potential of either sensor.
- Challenge: Camera-radar early fusion is difficult because radar measurements are ambiguous and the sensors have different coordinate systems and spatial characteristics.Camera-LiDAR fusion strategies cannot be directly adapted because radar points may not accurately correspond to objects.
- CRAFT: CRAFT associates image proposals with radar points through Soft Polar Association using uncertainty-aware adaptive thresholds in polar coordinates.The framework then uses cross-attention-based fusion to exchange spatial and contextual information between modalities.
- Results: 41.1% mAP and 52.3% NDS are achieved on the nuScenes test set, significantly improving the camera-only baseline with marginal additional computation cost.The framework is presented as a proposal-level early-fusion approach using Soft Polar Association and a Spatio-Contextual Fusion Transformer.
2 Related Work
Prior 3D detection research includes image-view and bird’s-eye-view camera methods, radar representations, and camera-LiDAR fusion strategies. Camera-radar detection remains relatively underexplored, with existing methods insufficiently addressing the sensors’ spatial properties.
- Camera 3D Detection: Image-view methods extend 2D detectors with 3D regression branches but suffer from inaccurate depth estimation.BEV methods instead transform image features into a top-down representation using depth distributions or learned queries.
- Radar 3D Detection: Radar can be represented as 2-D FFTs, range-azimuth-Doppler tensors, or affordable radar point clouds.Naively applying LiDAR methods to radar is inappropriate, and radar remains insufficiently investigated for autonomous driving.
- Camera-LiDAR Fusion: Camera-LiDAR fusion methods span result-, proposal-, and point-level strategies that exploit increasingly fine-grained intermediate features.Their assumptions about accurate LiDAR measurements limit direct adaptation to radar.
- Camera-Radar Fusion: Existing camera-radar detectors use approaches including inverse perspective mapping, AVOD adaptation, and gated or frustum-based region fusion.These methods have not thoroughly considered the spatial properties of camera and radar.
- CRAFT: CRAFT uses modality-specific backbones, camera-generated 3D proposals, Soft Polar Association, and a Spatio-Contextual Fusion Transformer.The architecture is organized around proposal generation, radar association, and cross-modal feature fusion.
3 Method
CRAFT performs proposal-level early fusion by associating image proposals with radar points in polar coordinates and exchanging spatial and contextual information through cross-attention feature encoders.
- Backbones and Camera 3D Object Detector: CRAFT takes multi-camera images and radar point clouds as input, producing camera features, image proposals, and radar point features for fusion.Camera proposals encode 3D boxes and features, while radar points include locations and properties such as radar cross-section and compensated radial velocity.
- Soft Polar Association: The Soft Polar Association module associates radar points with image proposals using adaptive azimuthal and radial thresholds in polar coordinates.Thresholds are derived from proposal corners, range, and image depth variance to favor informative foreground points while excluding clutter.
- Spatio-Contextual Fusion Transformer: The Spatio-Contextual Fusion Transformer uses cross-attention to exchange spatial and contextual information between camera and radar features.Its encoders address the difficulty of determining which image pixels correspond to radar points during fusion.
- Spatio-Contextual Fusion Transformer: Image-to-Radar encoding projects radar points into the image plane and adaptively samples nearby patched image features.Patch size varies with radar distance, and deformable multi-head cross-attention selects informative image features around each projected radar location.
- Spatio-Contextual Fusion Transformer: Radar-to-Image encoding operates in polar coordinates and produces radar-encoded image proposal features used to refine proposal predictions.The fusion head predicts fusion scores, location offsets, center-ness, and velocity; zero-valued attention entries handle proposals without object-reflected radar returns.
4 Experiments
Experiments evaluate CRAFT on nuScenes using standard detection metrics and ablations of fusion, association, and coordinate choices. CRAFT outperforms competing methods, with gains linked to localization, velocity estimation, and robustness across distances and radar sparsity.
- Comparison with State-of-the-Arts: CRAFT outperforms all competing camera-radar and single-frame camera methods on the nuScenes test set.It also has more than two times faster inference speed than other methods.
- Comparison with State-of-the-Arts: +17.2% versus +4.0% on Car and +17.3% versus +2.2% on Bus show larger gains than CenterFusion with similar camera detector architecture.Improvements are higher for metallic objects with more valid radar returns, while trailers and construction vehicles gain less.
- Ablation Studies: +6.6% over radar-only fusion and +2.4% over feature concatenation show that the fusion transformer improves AP, especially at the strict 0.5m threshold.The study attributes this to using image features with attention-based fusion, improving recall and reducing localization error.
- Ablation Studies: Soft Polar Association provides the best trade-off between association recall and precision, whereas RoIPool misses valid radar points and large-radius Ball Query includes clutter.Using all points without association causes substantial computational burden and out-of-memory in the reported setting.
- Ablation Studies: Polar-coordinate association and regression make spatial information easier to learn by minimizing image-proposal error variance in the azimuthal direction.The coordinate-system ablation compares Cartesian and polar settings for point association and regression targets.
- Analysis: 32.2% improvement is achieved for objects farther than 35m, where depth estimation becomes more inaccurate and radar measurements provide greater benefit.The analysis uses a 1m matching threshold and compares object-distance groups.
- Analysis: +6.2% improvement with no or few valid radar points and +28.7% with more points indicate robustness to radar sparsity.CRAFT maintains improvements across different numbers of valid radar points returned from objects.
- Qualitative Results: Qualitative results show CRAFT refining image proposals with more accurate localization when valid radar returns are available.Samples without valid associated radar returns are predicted by the camera-only path.
5 Conclusions
The paper combines an improved camera detector with radar processing and spatio-contextual fusion to support robust camera-radar 3D detection.
- 5 Conclusions: CRAFT combines soft polar association with spatio-contextual fusion to complement camera and radar information and improve prediction robustness.The conclusion reports more accurate prediction given an inaccurate image proposal and state-of-the-art performance on nuScenes.
- Camera 3D Detector: CRAFT uses an improved CenterNet camera-only baseline, CRAFT-I, with modified detection heads, label assignment, and post-processing.The detector adds depth-variance prediction and uncertainty-aware regression, and assigns objects to projected 3D centers.
- Camera 3D Detector: Depth variance is converted into depth confidence and combined with class confidence to estimate 3D confidence.Lower confidence reflects higher estimated depth variance and is intended to better represent localization probability.
- Radar Feature Extraction: Radar features are extracted with a PointNet++ backbone because radar points are sparse and vary substantially across environments.The design uses set abstraction and feature propagation while discarding farthest-point sampling.
- Spatio-Contextual Fusion Transformer: The spatio-contextual fusion transformer exchanges information between image proposals and associated radar points through image-to-radar and radar-to-image feature encoders.The image-to-radar encoder uses deformable multi-head cross-attention, while the radar-to-image encoder omits radar elevation before positional embedding.
B Implementation Details
Implementation uses temporally accumulated, motion-compensated radar inputs and confidence-gated fusion outputs, trained with detection and auxiliary point-wise losses.
- Radar Processing: Six radar sweeps covering approximately 0.5 seconds are transformed to the current time using ego-motion and Doppler compensation.Radar points are restricted to 55m, with random sampling or duplication to match the input limit; RCS and compensated velocity are used as features.
- Detection Post-processing: Fusion predictions use a 0.3 fusion-score threshold and combine image, fusion, and center-ness scores by geometric averaging.High-scoring proposals receive predicted offsets and updated velocity, while unassociated or low-scoring proposals remain unchanged.
- Training Objective: The total objective combines detection losses with an auxiliary binary cross-entropy loss for point-wise classification.Regression losses apply only when an image proposal has at least one valid radar return from an object.
C Detailed Experimental Settings
The experiments specify optimizer settings, modality-aware augmentation, and table conventions for detailed comparisons and distance-threshold analysis.
- Optimization: CRAFT-I is trained with AdamW for 140 epochs using scheduled learning-rate reductions at epochs 90 and 120.The initial learning rate is 2.5 × 10^-4, with batch size 64, weight decay 0.0001, and gradient clipping at norm 35.
- Data Augmentation: The polar-coordinate fusion scheme permits modality-specific augmentation while preserving camera-radar geometric consistency.Image training uses horizontal flipping, crop and resize, and color jittering, alongside augmentation of the radar and fusion components.
- Comparison Conventions: Table 5 marks LiDAR, camera, and radar methods as L, C, and R, and identifies initialization, extra depth-pretraining data, multi-frame input, and image-flipping augmentation.These symbols distinguish experimental settings when comparing methods.
- Distance Analysis: Table 6 compares detailed Car-class performance under different distance thresholds.The table is intended for distance-dependent analysis of detection performance.
D Additional Experimental Results
Additional experiments compare CRAFT with camera, radar, and LiDAR methods, including models using depth pretraining or temporal image input.
- Additional State-of-the-Art Comparison: CRAFT achieves the lowest translation error among the compared DD3D-pretrained methods and outperforms the previous LiDAR method PointPillars.The comparison covers nuScenes test-set results and additional CRAFT-I and CRAFT validation results.
- Temporal Comparisons: Multi-sequence BEVFormer improves over its single-frame version through lower velocity error, indicating a temporal-information direction for future CRAFT development.The reported gains are 7% NDS and 3–4% mAP for the multi-frame model.
D.2 Performance with Different Thresholds
CRAFT improves camera-only performance across distance thresholds and achieves comparable performance with PointPillars at strict thresholds.
- 32.3% and 23.4% improvements over the camera-only baseline occur at 0.5m and 1m distance thresholds, respectively.
- 28.1% and 21.2% improvements over the previous camera-radar state of the art occur at 0.5m and 1m thresholds, respectively.
- CRAFT consistently improves the CRAFT-I baseline at all evaluated distance thresholds.CenterFusion instead degrades from its CenterNet baseline at 2m and 4m thresholds.
- CRAFT achieves comparable performance with PointPillars even at strict distance thresholds.The comparison indicates camera-radar fusion can approach LiDAR-method performance under stringent matching.
D.3 Additional Per-Class Analysis
Per-class performance depends on radar availability and scene context: occlusion and non-metallic objects reduce useful radar associations and make detection harder.
- Trailers have fewer radar points than buses, 38.3 versus 89.0 on average, and show less significant improvement because they are often self-occluded near buildings.
- Construction vehicles have fewer radar points than other vehicle classes because they are often occluded by barriers or walls.
- Radar returns are scarce for pedestrians, bicycles, traffic cones, and barriers because these objects are often non-metallic.
- Traffic cones have radar associations for 70% of image proposals, but only 57% are valid associations and the remainder are clutter.
D.4 Additional Experiments for Design Choices
Detection-head ablations show that offset and velocity regression help, whereas dimension and rotation heads hurt performance under the chosen radar representation.
- CRAFT uses location-offset and velocity heads, class-specific 3D anchors with size offsets, and CenterNet-style BEV rotation estimation.The anchor-free dimension approach failed to converge in this setting.
- Removing the offset and velocity heads significantly degrades localization and velocity-estimation performance.
- Adding dimension and rotation heads degrades performance rather than improving it.
- The authors attribute the dimension and rotation degradation to irregular, shapeless radar points and leave radar-based improvement for future work.
E Additional Qualitative Results
Qualitative results show that fusion improves localization across varied conditions, while attention visualizations illustrate adaptive exchange between image and radar features. Difficult cases remain under challenging visibility, distance, and occlusion conditions.
- Additional Qualitative Results: Fusion often corrects inaccurate camera-only localization, with especially clear improvement at night.
- Additional Qualitative Results: CRAFT still produces false negatives for faraway or fenced vehicles, dimming headlamps, and heavily occluded trailers with few radar returns.
- Attention Visualization: The Image-to-Radar encoder gradually focuses on object pixels to extract semantic features.
- Attention Visualization: The Radar-to-Image encoder attends more strongly to objects using softly associated radar points and predicts offsets accurately.
- Additional Qualitative Results: Figure 6 compares ground truth, camera-only, and fused predictions across complex, rainy, and night environments within a 55m visualization range.