PaperlayerView exact revision on arXiv

Source-linked AI summary

LXL: LiDAR Excluded Lean 3D Object Detection with 4D Imaging Radar and Camera Fusion

Weiyi Xiong, Jianan Liu, Tao Huang, Qing-Long Han, Yuxuan Xia, Bing Zhu

arXiv:2307.00724v4cs.CV

TL;DR

Sparse, noisy 4D radar point clouds and limited research on camera-radar fusion motivate LXL. The model enhances sampling-based image view transformation with image depth distributions and radar occupancy grids. It outperforms state-of-the-art 3D detection methods on VoD and TJ4DRadSet, and the authors identify robustness as a future improvement target.

  • Problem

    4D radar point clouds are sparser and noisier than LiDAR, and camera–4D radar fusion has received limited attention.

    Source: §I · p. 2, C. 3D Object Detection with 4D Imaging… · p. 3

  • Method

    LXL fuses camera and 4D imaging radar features, using predicted image depth distributions and radar 3D occupancy grids to assist image view transformation.

    Source: §I · p. 2, A. Overall Architecture · p. 3

  • Results

    LXL outperforms state-of-the-art 3D object detection models on VoD and TJ4DRadSet by 6.7% and 2.5%, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors conclude that sampling has substantial room for improvement and that a small enhancement can boost view transformation significantly.

    Source: §V · p. 12

    The authors identify robustness as a future improvement target and propose attention-based transformers for adaptive interaction between modalities.

    Source: §V · p. 12

Abstract

from arXiv · show

As an emerging technology and a relatively affordable device, the 4D imaging radar has already been confirmed effective in performing 3D object detection in autonomous driving. Nevertheless, the sparsity and noisiness of 4D radar point clouds hinder further performance improvement, and in-depth studies about its fusion with other modalities are lacking. On the other hand, as a new image view transformation strategy, "sampling" has been applied in a few image-based detectors and shown to outperform the widely applied "depth-based splatting" proposed in Lift-Splat-Shoot (LSS), even without image depth prediction. However, the potential of "sampling" is not fully unleashed. This paper investigates the "sampling" view transformation strategy on the camera and 4D imaging radar fusion-based 3D object detection. LiDAR Excluded Lean (LXL) model, predicted image depth distribution maps and radar 3D occupancy grids are generated from image perspective view (PV) features and radar bird's eye view (BEV) features, respectively. They are sent to the core of LXL, called "radar occupancy-assisted depth-based sampling", to aid image view transformation. We demonstrated that more accurate view transformation can be performed by introducing image depths and radar information to enhance the "sampling" strategy. Experiments on VoD and TJ4DRadSet datasets show that the proposed method outperforms the state-of-the-art 3D object detection methods by a significant margin without bells and whistles. Ablation studies demonstrate that our method performs the best among different enhancement settings.

I. INTRODUCTION

The introduction motivates camera–4D imaging radar fusion and the underexplored use of depth and radar information to enhance sampling. It presents LXL and reports gains over state-of-the-art methods on two datasets.

  • 4D imaging radar point clouds are higher-resolution than conventional radar data, but remain less dense and lower quality than LiDAR point clouds.Source: §I · p. 1
  • Camera and 4D imaging radar fusion has received limited attention, while sampling with predicted depths and other modalities remains underexplored.Source: §I · p. 2
  • LXL enhances sampling by combining image depth distributions and radar 3D occupancy grids to assist image feature lifting.The proposed radar occupancy-assisted depth-based sampling strategy uses bi-linear sampling for predefined voxels, followed by parallel depth- and occupancy-assisted operations.Source: §I · p. 2
  • 6.7% and 2.5%: LXL outperforms state-of-the-art models on VoD and TJ4DRadSet, respectively.Source: §I · p. 2

II. RELATED WORK

The related work reviews camera-based 3D detection and camera fusion with ordinary automotive radar. It describes camera methods that estimate depth or transform image features, alongside varied radar-camera fusion strategies.

A. 3D Object Detection with Cameras

  • Camera-based 3D detection includes direct box estimation, PV-to-BEV transformation, and lifting pixels into points or voxels for 3D detection.Lifting methods may use estimated depth distributions or assume a uniform distribution; BEV projection can alleviate occlusion and facilitate fusion.Source: A. 3D Object Detection with Cameras · p. 2

B. 3D Object Detection with Camera and Ordinary Automotive Radar Fusion

  • Ordinary automotive radar lacks height information, motivating radar-camera fusion to improve 3D object detection [29][30].Source: B. 3D Object Detection with Camera and… · pp. 2–3
  • Radar-camera fusion methods include voxel concatenation, point-cloud fusion, spatial-temporal encoding, cross-attention, and result-level fusion [32][33][34][35][22][36][37].CRN uses radar occupancy maps to assist depth-based PV-to-BEV transformation; RADIANT merges camera and radar depth predictions.Source: B. 3D Object Detection with Camera and… · p. 3

C. 3D Object Detection with 4D Imaging Radars

The section reviews 4D radar detection and fusion, noting that radar point clouds are typically sparser and noisier than LiDAR. LXL combines radar and image branches, with occupancy and depth information assisting image view transformation before feature fusion and detection.

C. 3D Object Detection with 4D Imaging Radars

A. Overall Architecture

  • LXL generates radar BEV features and occupancy grids alongside image PV features and depth distributions, then transforms and fuses image BEV features with radar BEV features for box prediction.The architecture includes radar and image branches, a view transformation module, a fusion module, and a detection head.Source: A. Overall Architecture · p. 3, Figure 1 · p. 4

B. Radar Branch

The radar branch extracts BEV features and predicts 3D occupancy grids, while the image branch produces multi-scale PV features and depth distributions. The view transformation module lifts image features into 3D and compresses height, using depth and radar occupancy assistance.

B. Radar Branch

  • Radar point clouds are pillarized and processed into BEV features that feed both feature fusion and prediction of radar 3D occupancy grids.The occupancy grid has a predefined height, and the paper states that occupancy prediction receives no direct supervision during training.Source: B. Radar Branch · p. 4

C. Image Branch

  • The image branch uses multi-scale PV features and depth-bin classification to produce per-pixel depth probability distributions for view transformation.The image backbone and neck follow YOLOX architecture, using CSPNet and PAN.Source: C. Image Branch · p. 4
  • Sampling lifts image PV features into 3D voxels, where sampled depth probabilities and radar occupancy grids provide parallel assistance before concatenation and height compression.Source: C. Image Branch · p. 4, B. Radar Branch · p. 4, Figure 2 · p. 5

D. View Transformation

The view-transformation module enhances sampling with predicted image-depth distributions and radar 3D occupancy grids. It lifts sampled image features into radar-coordinate voxels and combines depth-assisted and radar-assisted features into image BEV features.

D. View Transformation

  • The proposed radar occupancy-assisted depth-based sampling enhances feature lifting with predicted multi-scale image-depth distributions and radar 3D occupancy grids.The paper describes this as its first use of both forms of assistance to enhance sampling.Source: D. View Transformation · p. 5
  • Sampling projects predefined 3D voxels onto the image plane and bilinearly combines nearby pixel features; the authors chose it over splatting, citing Simple-BEV.Voxel centers are projected using the radar-to-image transformation and camera intrinsic matrices.Source: D. View Transformation · p. 5
  • Depth-assisted lifting samples image-depth probabilities for radar-coordinate voxels and multiplies them with image 3D features.The probabilities are interpolated from nearby image frustum voxels before weighting the image features.Source: D. View Transformation · pp. 5–6
  • Radar-assisted lifting uses predicted 3D occupancy grids rather than sparse, noisy radar points as direct image-depth supervision.The occupancy grids are generated from radar BEV features in radar coordinates; the authors note this avoids coordinate transformation and pillarization steps used by CRN.Source: D. View Transformation · p. 6
  • The two lifted feature sets are concatenated across channels, summed across levels, and reshaped for convolution to produce image BEV features.The resulting image BEV features are the output of the view-transformation module.Source: D. View Transformation · p. 6

E. Multi-modal Fusion and Detection Head

The fusion module combines radar and image BEV features at a shared resolution. The fused features feed a detection head that predicts 3D bounding boxes.

E. Multi-modal Fusion and Detection Head

  • Radar and image BEV features share a resolution, allowing concatenation and convolutional fusion before detection.The resulting fused BEV features are passed to the detection head.Source: E. Multi-modal Fusion and Detection Head · p. 6
  • The detection head uses CenterPoint to generate categorywise heatmaps and predict 3D bounding boxes, but the design is not limited to that head or fusion strategy.The authors note that attention-based fusion and an anchor-based detection head could also be used.Source: E. Multi-modal Fusion and Detection Head · p. 6

A. Dataset and Evaluation Metrics

The study evaluates 4D radar and camera detection on VoD and TJ4DRadSet, which provide sensor data, 3D box annotations, and sensor-coordinate transformations. Evaluation uses dataset-specific metrics and official splits, with VoD results limited to validation.

A. Dataset and Evaluation Metrics

  • Both datasets provide camera images, 4D radar and LiDAR points, 3D bounding-box and tracking-ID annotations, and inter-sensor transformations.VoD experiments use cars, pedestrians, and cyclists; TJ4DRadSet adds trucks and includes more varied driving scenarios.Source: A. Dataset and Evaluation Metrics · p. 6
  • VoD has 5,139 training and 1,296 validation frames, and evaluation uses validation only because its official test server is not released.The TJ4DRadSet split has 5,717 training frames and 2,040 test frames.Source: A. Dataset and Evaluation Metrics · pp. 6–7
  • VoD evaluation reports EAA AP and driving-corridor RoI AP, while TJ4DRadSet reports 3D AP and BEV AP within 70 meters.VoD RoI is defined as -4m < x < 4m and z < 25m in camera coordinates; IoU thresholds vary by class.Source: A. Dataset and Evaluation Metrics · p. 7

B. Implementation Details

The implementation uses MMDetection3D and CenterPoint with dataset-specific point-cloud ranges. Training normalizes sensor inputs, filters data outside the image view, and uses frozen pretrained image-backbone parameters.

B. Implementation Details

  • The implementation uses MMDetection3D, CenterPoint, and a 160 × 160 radar BEV resolution on VoD.Radar pillars are 0.16m × 0.16m, and the feature-extractor stride is 2.Source: B. Implementation Details · p. 7
  • For TJ4DRadSet, PCR is set to 0 < x < 69.12m, −39.68m < y < 39.68m, and −4m < z < 2m; other hyperparameters match VoD, and truck NMS threshold is 12m.Source: B. Implementation Details · p. 7
  • Images and radar points are normalized using training-set means and standard deviations; points and box centers outside the image view are filtered.The model trains for 80 epochs with AdamW and StepLR, while the pretrained YOLOX image backbone and neck are frozen.Source: B. Implementation Details · p. 7

C. Results and Analysis

LXL improves radar-camera detection relative to other fusion systems and the radar-only baseline, with performance varying by lighting and distance. The analysis also compares fusion with LiDAR detection and reports an inference-speed cost.

Results on VoD

  • LXL-R has relatively high close-range car and cyclist RoI AP, but limited pedestrian RoI AP because pedestrians are small in BEV and weakly reflected.The authors also report that the radar-only model performs poorly on EAA.Source: C. Results and Analysis · p. 7
  • Against RCFusion, LXL achieves higher detection accuracy across almost all categories and evaluation regions, with gains of 10.5% for pedestrians, 8.8% for cyclists, and 0.6% for cars.The authors attribute the larger pedestrian and cyclist gains partly to LXL’s center-based head, which is skilled at detecting small objects.Source: C. Results and Analysis · p. 8

Inference speed and visual results

  • On a single V100, multimodal methods require more than 100 ms (FPS < 10), unlike real-time LXL-R; fusion also detects some radar-missed or camera-occluded objects.Figure 3 compares LXL predictions with LXL-R and ground truth on VoD validation frames.Source: Figure 3 · p. 8, C. Results and Analysis · p. 9

LiDAR comparison

  • PointPillars-LiDAR exceeds LXL-R by 19.1% in detection accuracy, while fusion narrows the gap to 9.6% and outperforms PointPillars-LiDAR for cyclists.Table III compares the methods on the VoD validation set.Source: Table III · p. 9, C. Results and Analysis · p. 9

Results on TJ4DRadSet

Lighting analysis

  • In dark scenarios, image fusion gains 2.3% on 3D mAP, while excessive illumination degrades performance.The lighting analysis divides TJ4DRadSet into dark, standard, and Shiny subsets, comprising approximately 15%, 60%, and 25% of the test set, respectively.Source: C. Results and Analysis · p. 9, Table V · p. 10

Distance analysis

  • LXL detects better than LXL-R at almost all distances, although performance falls as distance increases; far-away objects benefit more from fusion than medium-range objects.The authors link the decline with distance to radar-point sparsity and the greater benefit for far objects to fewer missed detections and more true positives.Source: C. Results and Analysis · p. 9, Table VI · p. 10

D. Ablation Study

The VoD ablations compare image-feature lifting strategies and radar assistance in the image branch.

D. Ablation Study

  • Sampling outperforms splatting on EAA APs, although splatting is slightly better on RoI AP; far-distance sparsity contributes to splatting’s weaker broader-range performance.Sampling associates each camera-visible 3D voxel with an image feature, while splatting can leave far-away BEV grids empty.Source: D. Ablation Study · p. 11, Table VII · p. 11, Figure 5 · p. 11
  • Radar-point depth supervision yields only a slight detection improvement over no radar assistance, because sparse, noisy points make depth targets inaccurate and depth-network convergence difficult.The alternative assigns projected radar depths to nearest pixels, averaging depths when multiple radar points map to one pixel.Source: D. Ablation Study · p. 11
  • The 3D Occupancy Grids (CRN) alternative uses 3D grids rather than CRN’s 2D maps because 4D radar measures elevation, and it uses sampling rather than splatting.Its occupancy grids are generated in image coordinates, then resampled back to radar coordinates before being combined with 3D image features.Source: D. Ablation Study · p. 11
  • Radar occupancy-assisted sampling offers performance and inference-speed advantages over the 3D Occupancy Grids (CRN) alternative.The alternative is more affected by sparse radar points and requires time-consuming projection and voxelization; LXL predicts radar 3D occupancy grids directly in radar coordinates.Source: D. Ablation Study · p. 11

V. CONCLUSION

LXL is proposed for camera and 4D imaging radar fusion-based 3D object detection. The conclusion attributes its large-margin advantage over existing works mainly to radar occupancy-assisted depth-based sampling.

V. CONCLUSION

  • LXL is a proposed camera and 4D imaging radar fusion model for 3D object detection.Source: §V · p. 11
  • LXL outperforms existing works by a large margin, mainly through view transformation aided by predicted image depth distributions and radar 3D occupancy grids.The conclusion says this design can significantly improve the sampling strategy’s view transformation.Source: §V · p. 12
  • Future work includes improving robustness through attention-based transformers for adaptive interaction between modalities and applying LXL to planning and control tasks.Source: §V · p. 12
Loading 2307.00724v4…