PaperlayerView exact revision on arXiv

Source-linked AI summary

CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception

Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, Dongsuk Kum

arXiv:2304.00670v3cs.CVcs.AIcs.RO

TL;DR

Camera-only 3D perception lacks accurate localization and robustness to illumination and weather, motivating camera-radar fusion. CRN transforms image features into BEV with radar assistance and adaptively fuses modalities, achieving state-of-the-art multi-task performance with high efficiency.

  • Problem

    Camera-based 3D perception is semantically rich but has limited localization accuracy and sensitivity to illumination and weather, while camera-radar fusion remains insufficiently investigated.

  • Method

    CRN uses radar-assisted view transformation followed by multi-modal deformable feature aggregation to generate a unified, spatially accurate BEV representation.

  • Results

    CRN achieves LiDAR-level performance across 3D detection, tracking, and BEV segmentation while maintaining high efficiency and state-of-the-art performance on multiple tasks.

  • Takeaways & Limitations

    Camera-radar fusion can provide accurate, robust, and efficient BEV-based 3D perception suitable for long-range real-time operation.

  • Takeaways & Limitations

    Radar measurements are noisy, ambiguous, sparse, and lack elevation, so the method must adaptively handle these limitations.

Abstract

from arXiv · show

Autonomous driving requires an accurate and fast 3D perception system that includes 3D object detection, tracking, and segmentation. Although recent low-cost camera-based approaches have shown promising results, they are susceptible to poor illumination or bad weather conditions and have a large localization error. Hence, fusing camera with low-cost radar, which provides precise long-range measurement and operates reliably in all environments, is promising but has not yet been thoroughly investigated. In this paper, we propose Camera Radar Net (CRN), a novel camera-radar fusion framework that generates a semantically rich and spatially accurate bird's-eye-view (BEV) feature map for various tasks. To overcome the lack of spatial information in an image, we transform perspective view image features to BEV with the help of sparse but accurate radar points. We further aggregate image and radar feature maps in BEV using multi-modal deformable attention designed to tackle the spatial misalignment between inputs. CRN with real-time setting operates at 20 FPS while achieving comparable performance to LiDAR detectors on nuScenes, and even outperforms at a far distance on 100m setting. Moreover, CRN with offline setting yields 62.4% NDS, 57.5% mAP on nuScenes test set and ranks first among all camera and camera-radar 3D object detectors.

1. Introduction

CRN addresses camera-based 3D perception's localization and environmental weaknesses by fusing camera and radar into a BEV representation. Its two-stage design targets accurate transformation, adaptive fusion, and efficient multi-task perception.

  • The framework is motivated by the need for reliable BEV features that support downstream autonomous-driving tasks while reducing reliance on costly, maintenance-intensive LiDAR.
  • Radar complements cameras with long-range, weather-robust measurements, but its sparse, noisy, and ambiguous observations complicate fusion.The paper notes typical automotive radar can perceive up to 200m, while radar point clouds are much sparser than LiDAR.
  • Camera-only BEV methods provide rich semantics but have limited localization accuracy and are sensitive to illumination and weather conditions.
  • CRN first uses radar-assisted view transformation, then adaptively aggregates BEV image and radar features to produce a spatially accurate representation.This two-stage design is intended to address perspective-to-BEV localization and spatial misalignment between modalities.
  • CRN fuses camera and radar to achieve LiDAR-level performance across 3D detection, tracking, and BEV segmentation while maintaining efficiency.The paper identifies accuracy, robustness to missing single-sensor inputs, and marginal extra computational cost as its three main contributions.

2. Related Work

Prior work develops camera-based, point-based, and camera-point 3D perception, but camera-only methods remain limited by localization, robustness, and real-time constraints. CRN instead unifies image and radar information in BEV using radar-assisted transformation and adaptive fusion.

  • Camera-based methods transform perspective-view image features into BEV using explicit depth estimation or learned mappings, but the view discrepancy remains a central challenge.
  • Camera-only BEV methods extend to detection, tracking, segmentation, and prediction, but remain limited by localization accuracy, environmental sensitivity, and real-time cost.High performance often requires large image inputs and backbones, which slows inference.
  • LiDAR dominates point-based 3D perception, whereas radar point-based perception is less investigated and trails because radar is cluttered and lacks contextual information.
  • CRN produces a unified BEV representation by transforming image features with radar and then aggregating the resulting image and radar feature maps.The architecture uses Radar-assisted View Transformation followed by Multi-modal Feature Aggregation for downstream BEV tasks.
  • Existing camera-point fusion methods address sensor discrepancy by projecting 3D points or predictions into images, while some camera-radar methods use radar to improve depth estimation.

3. Camera Radar Net

CRN builds a unified BEV representation by using radar to improve image view transformation and multi-modal deformable attention to fuse spatially misaligned camera and radar features. Its pipeline encodes image and radar context, transforms them into BEV, and supports downstream perception tasks with sparse aggregation for efficiency.

  • Radar-assisted View Transformation: Dense image depth distributions and sparse radar occupancy jointly produce spatially accurate image context features before BEV pooling.The image context feature is collapsed along height because radar does not provide reliable elevation, and the resulting camera and radar frustum features are transformed into one BEV space.
  • Radar-assisted View Transformation: CRN first transforms image features into BEV with radar measurements, then aggregates camera and radar features through multi-modal feature aggregation.Radar-assisted view transformation uses radar points and occupancy to improve spatial accuracy, while subsequent aggregation addresses residual feature misalignment.
  • Radar-assisted View Transformation: Sparse radar points preserve metric depth during projection into camera views, are voxelized into frustum pillars, and are encoded as radar context features and occupancy.Because radar lacks reliable elevation, the method uses pillar-style representation and predicts radar occupancy with a sigmoid.
  • Multi-modal Feature Aggregation: Multi-modal deformable cross attention separately learns attention weights and sampling offsets for image and radar features while reducing fusion complexity from quadratic to linear in BEV size.The method samples K ≪ N keys, giving complexity O(2N + NK) rather than O(N^2).
  • Multi-modal Feature Aggregation: Fusion produces more reliable BEV features when image inputs miss occluded or distant vehicles and radar inputs contain wall clutter or low-reflectivity pedestrian failures.The visualization contrasts modality-specific failures with the fused feature map generated by MFA.
  • Sparse Aggregation: Top-k sparse aggregation further reduces computation by selecting Nk ≪ XY high-confidence queries using camera depth and radar occupancy.The selected queries are drawn from the combined BEV query features according to max(DI, OP).

4. Experiments

Experiments show that CRN performs strongly across detection, tracking, segmentation, long-range perception, sensor failures, and adverse conditions, while its RVT and MFA components provide measurable gains.

  • 3D Object Detection: CRN ranks first among camera-only and camera-radar detection methods and outperforms the LiDAR CenterPoint-P baseline.The comparison includes faster inference, with a small 256 × 704, R18 configuration outperforming larger camera models in mAP.
  • 3D Object Tracking and BEV Segmentation: CRN outperforms published camera-only tracking methods and achieves state-of-the-art BEV segmentation performance.Tracking improves AMOTA while reducing AMOTP and identity switches; segmentation remains real-time compared with larger camera configurations.
  • View Transformation: +8.2% NDS and +11.6% mAP are achieved by RVT over depth-only view transformation.RVT combines dense depth with sparse radar range measurements, while naive radar replacement severely degrades performance.
  • Feature Aggregation: MFA outperforms deeper convolutional BEVFusion aggregation by adaptively handling spatial misalignment between modalities.The comparison supports multi-modal deformable cross attention as the effective aggregation mechanism.
  • Scaling Up Perception Range: CRN outperforms LiDAR especially beyond 30m while retaining much faster inference, and sparse aggregation further improves speed with comparable performance.The analysis extends perception and evaluation ranges from 51.2m to 102.4m.
  • Robustness and Conditions: CRN remains competitive under sensor failures and improves performance in varied lighting and weather conditions.With radar unavailable, CRN drops 5.6% versus BEVDepth’s 15.0% drop; fusion also yields more than 10 mAP improvement over camera-only conditions.

5. Conclusion

CRN generates a contextually rich and spatially accurate BEV representation by fusing camera and radar information for multi-task 3D perception. Its multi-modal deformable attention addresses spatial misalignment and supports adaptive fusion.

  • CRN generates contextually rich and spatially accurate BEV scene representations for accurate, robust, and efficient multi-task 3D perception.
  • CRN’s attention module first receives query elements and flattened multi-modal BEV feature maps, then aggregates their features through deformable cross attention.
  • MDCA uses modality-specific value projections, attention weights, and sampling offsets to support robust fusion across image and radar feature maps.
  • CRN uses multi-modal deformable attention to inspect multiple modalities and sampling points, overcoming spatial misalignment and enabling adaptive fusion.

C. Implementation Details

The implementation extracts multi-scale image features, voxelizes radar points into a matching frustum representation, and applies multi-scale deformable attention to produce fused BEV features. Temporal fusion concatenates aligned features from previous timestamps.

  • The camera stream uses four feature-map strides, SECONDFPN concatenation, and a depth distribution with 112 uniform bins over 2.0–58.0m.
  • Radar points are projected into image coordinates, voxelized in frustum coordinates, and represented on an 8× downsampled pillar canvas for feature extraction.
  • Six MFA layers with eight attention heads and four sampling points fuse single-frame camera and radar inputs into a BEV feature map.Fused features from the previous three timestamps are aligned to the current timestamp and concatenated for the submission.
  • The standard BEV detector range is [−51.2, 51.2]m with a 128 × 128 feature map, while segmentation uses a 200 × 200 grid at 0.5m resolution.

C.2. Training Settings

Training uses end-to-end optimization with AdamW, ImageNet-pretrained image backbones, synchronized perspective-view augmentation, and separate BEV augmentation. Rotation augmentation is omitted in perspective view because it can harm radar-assisted view transformation.

  • All models train end-to-end for 24 epochs with AdamW, using ImageNet-pretrained image backbones and specified ResNet or ConvNeXt settings.
  • Table 10 provides the training settings used for the main results.
  • Perspective-view image and radar inputs share resize, crop, and horizontal-flip augmentation, while rotation is omitted because it can adversely affect RVT.
  • BEV augmentation applies random X/Y flips, rotation in [−π/8, π/8], and scaling in [0.95, 1.05], without GT-AUG.

C.3. Baselines for Ablation Studies

The ablation setup compares camera-only, point-only, and camera-point fusion baselines, adapting LiDAR-oriented implementations to radar inputs. Long-range experiments extend perception and class ranges while disabling point filtering consistently across methods.

  • The study evaluates BEVDepth, CenterPoint, and BEVFusion as camera-only, point-only, and camera-point fusion baselines, respectively.
  • CenterPoint and BEVFusion replace LiDAR inputs with radar features without network modification, while BEVFusion combines BEVDepth and CenterPoint-Pillar features using one convolution layer.
  • Long-range models extend perception to [−102.4, 102.4]m, double camera depth bins to 224, and use a 256 × 256 BEV feature map.
  • Long-range evaluation doubles nuScenes class ranges and disables point-in-box filtering for all methods, despite possible input–annotation inconsistency.

D.1. Per-Class Analysis

CRN improves camera-radar detection across object classes, with particularly broad gains beyond metallic road users.

  • CRN shows significant gains for metallic road classes including car, truck, bus, and motorcycle over camera-only baselines.
  • CRN also achieves substantial improvements for non-metallic classes including pedestrian, bicycle, traffic cone, and barrier.
  • The per-class comparison is reported on the nuScenes validation set against CenterNet, CRAFT-I, and BEVDepth camera baselines.

D.2. Design Decisions

CRN design studies show benefits from temporal aggregation and sparse feature querying, while exposing trade-offs between efficiency and localization quality.

  • Temporal Frames: A one-second temporal interval performs better than the 0.5-second interval used in previous approaches.CRN aggregates temporal BEV maps through concatenation followed by convolutional layers.
  • Temporal Frames: Multiple temporal frames significantly improve mAP, mATE, and mAVE, with performance gains saturating at four frames.The method therefore uses four frames, although orientation error remains unstable.
  • Sparse Aggregation: Sparse aggregation reduces computation linearly with the number of queried feature grids, but top-k sparsification degrades localization-related true-positive metrics.The reported complexity is O(2N_k + N_kK), and ATE, AOE, and AVE decline more noticeably than AP.
  • Sparse Aggregation: 76.4% lower MFA latency results from reducing sparse aggregation queries from the full BEV grid to 4096 queries.Latency falls from 21.01 ms to 4.96 ms on a 256 × 256 BEV grid.

E. Additional Qualitative Results

Qualitative results show CRN detecting distant and partially occluded objects while producing robust BEV segmentation across lighting, weather, and road geometries.

  • 3D Object Detection: CRN detects objects at very far distances across complex scenarios, including strongly occluded or poorly illuminated cases.It can also detect some objects without radar returns using image information alone.
  • BEV Segmentation: CRN maintains stable segmentation under varied lighting and weather, predicting complete vehicle occupancy shapes even with partial visibility.
  • BEV Segmentation: The model predicts complex drivable-region shapes under occlusion and across nearby and faraway distances.
  • 3D Object Detection: Figure 7 presents day, rainy, and night detection examples with ground truth, predictions, radar points, and highlighted failures.
  • BEV Segmentation: Figure 8 compares camera-view inputs, ground-truth BEV segmentation, and CRN predictions across road shapes and weather conditions.
Loading 2304.00670v3…