PaperlayerView exact revision on arXiv

Source-linked AI summary

CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection

Ramin Nabati, Hairong Qi

arXiv:2011.04841v1cs.CV

TL;DR

Autonomous-vehicle perception needs robust 3D detection and velocity estimation, but radar’s sparse and inaccurate geometry complicates fusion with cameras. CenterFusion uses middle fusion with center-based image detections, frustum-based radar association, and radar feature maps; on nuScenes, it outperforms camera-based methods and improves velocity estimation without temporal information.

  • Problem

    Radar provides long-range, weather-robust velocity sensing, but its sparse point clouds and inaccurate or missing vertical measurements limit direct geometric use in 3D detection.

  • Method

    CenterFusion detects image center points, associates radar detections through a frustum-based method, and fuses radar feature maps with image features to estimate 3D properties.

  • Results

    CenterFusion outperforms previous camera-based methods on nuScenes 3D detection and significantly improves velocity estimation without additional temporal information.

  • Takeaways & Limitations

    Radar information can complement image features in a middle-fusion system for robust 3D object detection and velocity estimation.

  • Takeaways & Limitations

    Radar association remains constrained by inaccurate height measurements, which can place detections outside the corresponding object’s RoI frustum.

Abstract

from arXiv · show

The perception system in autonomous vehicles is responsible for detecting and tracking the surrounding objects. This is usually done by taking advantage of several sensing modalities to increase robustness and accuracy, which makes sensor fusion a crucial part of the perception system. In this paper, we focus on the problem of radar and camera sensor fusion and propose a middle-fusion approach to exploit both radar and camera data for 3D object detection. Our approach, called CenterFusion, first uses a center point detection network to detect objects by identifying their center points on the image. It then solves the key data association problem using a novel frustum-based method to associate the radar detections to their corresponding object's center point. The associated radar detections are used to generate radar-based feature maps to complement the image features, and regress to object properties such as depth, rotation and velocity. We evaluate CenterFusion on the challenging nuScenes dataset, where it improves the overall nuScenes Detection Score (NDS) of the state-of-the-art camera-based algorithm by more than 12%. We further show that CenterFusion significantly improves the velocity estimation accuracy without using any additional temporal information. The code is available at https://github.com/mrnabati/CenterFusion .

1. Introduction

Autonomous-vehicle perception benefits from sensor fusion, but radar-camera integration must address sparse, incomplete radar geometry and robust data association. CenterFusion uses middle fusion to associate radar with image detections, enrich image features, and improve 3D detection and velocity estimation.

  • Multiple sensing modalities can increase perception robustness and accuracy, motivating sensor fusion for autonomous-vehicle detection and tracking.
  • LiDAR-camera systems provide complementary depth and appearance information, but both sensors are vulnerable to adverse weather and lack instantaneous velocity without temporal information.
  • Radar offers long-range detection, weather robustness, and Doppler-based velocity estimates without temporal information, but its point clouds are sparse and geometrically incomplete.
  • CenterFusion proposes middle fusion that associates radar detections with preliminary image-based detections, generates radar feature maps, and combines them with image features.
  • On nuScenes, CenterFusion outperforms previous camera-based 3D detection methods and significantly improves velocity estimation without additional temporal information.

2. Related Work

Prior 3D detection research includes monocular camera and LiDAR-based methods, while sensor-fusion work has primarily combined LiDAR and cameras. Fewer studies fuse radar with other sensors, using approaches that project or otherwise incorporate radar measurements for detection and velocity estimation.

  • 2. Related Work: Monocular 3D detection methods estimate object 3D boxes from a single camera using depth, geometric, or shape-prior approaches.
  • 2. Related Work: LiDAR-based methods commonly represent point clouds with 3D voxels or 2D projections, while PointRCNN operates directly on raw point clouds.
  • 2. Related Work: Most sensor-fusion methods combine LiDAR and cameras, extracting complementary views or point-level features for 3D proposals and object-property prediction.
  • 2. Related Work: Radar-fusion studies have used early or late fusion, projected radar detections, radar-generated proposals, and image augmentation for detection, depth, or velocity estimation.

3. Preliminary

CenterNet detects object centers from camera images and regresses 3D properties, while radar provides position and radial-velocity measurements that can be associated through image-derived frustums.

  • Radar Point Cloud: Automotive radar reports azimuth, radial distance, and instantaneous radial velocity, but its velocity may differ from the object’s actual motion vector.Radar is robust in adverse weather and can detect objects at long range, while vertical measurements are often inaccurate or unavailable.
  • Radar Point Cloud: Radar detections are represented as egocentric 3D points with position and compensated radial-velocity components.Three radar sweeps from the preceding 0.25 seconds are aggregated and transformed using nuScenes calibration parameters.
  • CenterNet: CenterNet predicts object center heatmaps from images, using heatmap peaks as center-point detections.Ground-truth heatmaps are generated from 2D box centers with a Gaussian kernel whose spread adapts to object size.
  • Frustum Association: Frustum association uses an image detection’s 3D bounding box to identify radar detections within the object’s region of interest.The figure shows the image detection, generated ROI frustum, and its BEV containing radar detections; δ enlarges the frustum during testing.
  • CenterNet: Separate regression heads estimate object depth, dimensions, and orientation directly from detected center points.Depth is transformed back to the original depth domain, while dimensions are regressed in meters.

4. CenterFusion

CenterFusion fuses camera and radar through center-based association: radar detections are linked to image objects with 3D frustums, expanded into pillars, and converted into features for refined 3D prediction.

  • 4. CenterFusion: CenterFusion uses middle fusion to associate radar detections with image-based object centers and re-estimate 3D properties from both modalities.The refined predictions include depth, velocity, rotation, and attributes.
  • Radar Association: Radar association is necessary because radar points cannot be robustly matched to image objects through a simple 2D-box projection.The method addresses the lack of one-to-one correspondence between radar detections and image objects.
  • Radar Association: The frustum method filters radar detections using an object’s 2D box, estimated depth, dimensions, and rotation.Training uses the ground-truth 3D box; testing uses the estimated 3D box and enlarges the frustum with δ to tolerate depth error.
  • Radar Association: Only the closest radar detection inside an object’s frustum is associated, separating overlapping objects in 3D space.The approach does not eliminate errors caused by inaccurate radar height measurements, which can place detections outside the correct frustum.
  • Pillar Expansion: Pillar expansion gives each radar point fixed 3D extent, allowing detections to associate when all or part of a pillar lies inside the frustum.This preprocessing step addresses inaccurate radar height information by providing a spatially extended representation.
  • Radar Feature Extraction: Associated radar depth and velocity populate three heatmap channels that are concatenated with image features for secondary regression heads.The heads refine depth and rotation and estimate velocity and class-specific attributes.

5. Implementation Details

The implementation uses a DLA-backed CenterNet baseline, trains radar-enabled regression heads separately, and applies fixed image, radar, augmentation, and loss configurations.

  • Network: The detection network uses pretrained CenterNet with a DLA backbone because it trains substantially faster than the stronger Hourglass alternative while retaining reasonable performance.DLA increases feature-map resolution through iterative deep aggregation layers.
  • Training: The released CenterNet model is trained for 140 epochs, velocity and attribute heads for 30 epochs, and radar-enabled secondary heads for 60 additional epochs.The secondary heads are trained on image and radar features with batch size 26 on two Nvidia P5000 GPUs.
  • Data Processing: Training and testing use images resized from 1600×900 to 800×450 pixels, with synchronized image-radar augmentation during training.Training applies horizontal flipping and random shifts; testing uses flip averaging but not multi-scale augmentation.
  • Losses: L1 loss is used for most regression heads, focal loss for the center heatmap, and BCE loss for attribute regression.

6. Results

CenterFusion outperforms camera-based methods on nuScenes overall detection quality and improves 3D box fitting and velocity estimation, particularly for distant objects.

  • CenterFusion achieves the highest nuScenes Detection Score among compared methods on both test and validation splits.The comparison includes camera-based models and the LiDAR-based InfoFocus method.
  • 12.25% and 16.9% relative NDS increases over CenterNet and MonoDIS, respectively, are reported on the test split.
  • CenterNet has higher test-set mAP with an Hourglass backbone, but CenterFusion performs better on the validation split with the shared DLA backbone.
  • CenterFusion improves validation mAP by 5.6% for motorcycles and 4.0% for cars compared with CenterNet using the DLA backbone.
  • CenterFusion produces better-fitting 3D boxes than CenterNet in most cases, especially for objects at larger distances.The qualitative comparison includes four scenes in camera and bird’s-eye-view representations.
  • CenterFusion’s estimated velocity vectors show significant improvement over CenterNet in the qualitative scenes.

7. Ablation Study

The ablation study shows progressively larger improvements as pillar expansion, frustum association, and flip testing are combined, while bicycle mAP slightly declines.

  • 15.4% relative NDS and 1.0% absolute mAP improvements result from pillar expansion with simple radar-to-image association.Radar pillars are mapped to equivalent 2D bounding boxes and filled with corresponding depth and velocity values.
  • Frustum association alone improves NDS by 25.9% relatively and mAP by 2.0% without first converting radar points to pillars.
  • Combining pillar expansion and frustum association yields 35.5% relative NDS and 4.3% absolute mAP improvements over baseline.
  • Flip testing adds 3.3% NDS and 3.9% mAP improvements, producing total gains of 37.8% and 8.4% over baseline.
  • Pillar expansion and frustum association improve mAP for most object classes, but bicycle mAP is 0.5% lower than CenterNet.

8. Conclusion

CenterFusion combines radar and camera information for robust 3D object detection, using frustum-based association and radar-based feature maps to complement image features. It was evaluated on nuScenes and outperformed state-of-the-art camera-based methods.

  • CenterFusion exploits radar information for robust 3D object detection through a radar-camera middle-fusion algorithm.
  • Frustum-based association uses preliminary detections to generate 3D RoI frustums, associate radar detections, and map them to object centers in the image.
  • Pillar expansion compensates for inaccurate radar height information by converting radar points into fixed-size 3D pillars.
  • CenterFusion was evaluated on the nuScenes 3D detection benchmark and outperformed state-of-the-art camera-based object detection methods.
  • Figure 5 compares CenterFusion and CenterNet qualitatively in camera view and BEV using detection boxes, ground truth boxes, radar points, and velocity vectors.
Loading 2011.04841v1…