PaperlayerView exact revision on arXiv

Source-linked AI summary

K-Radar: 4D Radar Object Detection for Autonomous Driving in Various Weather Conditions

Dong-Hee Paek, Seung-Hyun Kong, Kevin Tirta Wijaya

arXiv:2206.08171v4cs.CVcs.AI

TL;DR

Existing Radar datasets are relatively small and usually lack elevation, limiting evidence for data-driven 3D Radar perception. K-Radar introduces a large-scale, multimodal 4DRT benchmark with 3D labels and baseline networks, showing that height information matters and 4D Radar remains robust in adverse weather. Its forward-only Radar field of view is a stated scope limitation.

  • Problem

    Existing Radar datasets are relatively small, and conventional 3DRT lacks elevation information needed for 3D bounding-box estimation.

  • Method

    The paper constructs K-Radar with 4DRT, calibrated auxiliary sensors, 3D bounding-box labels, and 4DRT-based baseline neural networks.

  • Results

    4DRT height information improves detection, while the 4D Radar baseline is hardly affected by adverse weather compared with the Lidar-based baseline.

  • Takeaways & Limitations

    K-Radar supplies a large-scale benchmark and development toolkit for studying 4DRT-based 3D object detection and related autonomous-driving tasks.

  • Takeaways & Limitations

    K-Radar records 4DRT only in the forward direction with a 107-degree field of view because dense four-dimensional measurements require substantially more storage.

Abstract

from arXiv · show

Unlike RGB cameras that use visible light bands (384$\sim$769 THz) and Lidars that use infrared bands (361$\sim$331 THz), Radars use relatively longer wavelength radio bands (77$\sim$81 GHz), resulting in robust measurements in adverse weathers. Unfortunately, existing Radar datasets only contain a relatively small number of samples compared to the existing camera and Lidar datasets. This may hinder the development of sophisticated data-driven deep learning techniques for Radar-based perception. Moreover, most of the existing Radar datasets only provide 3D Radar tensor (3DRT) data that contain power measurements along the Doppler, range, and azimuth dimensions. As there is no elevation information, it is challenging to estimate the 3D bounding box of an object from 3DRT. In this work, we introduce KAIST-Radar (K-Radar), a novel large-scale object detection dataset and benchmark that contains 35K frames of 4D Radar tensor (4DRT) data with power measurements along the Doppler, range, azimuth, and elevation dimensions, together with carefully annotated 3D bounding box labels of objects on the roads. K-Radar includes challenging driving conditions such as adverse weathers (fog, rain, and snow) on various road structures (urban, suburban roads, alleyways, and highways). In addition to the 4DRT, we provide auxiliary measurements from carefully calibrated high-resolution Lidars, surround stereo cameras, and RTK-GPS. We also provide 4DRT-based object detection baseline neural networks (baseline NNs) and show that the height information is crucial for 3D object detection. And by comparing the baseline NN with a similarly-structured Lidar-based neural network, we demonstrate that 4D Radar is a more robust sensor for adverse weather conditions. All codes are available at https://github.com/kaist-avelab/k-radar.

1 Introduction

K-Radar addresses limited Radar data and missing elevation information by introducing a large-scale 4DRT dataset and benchmark for 3D object detection across diverse conditions. Baseline experiments show that height information improves detection and that 4D Radar is robust in adverse weather.

  • Dataset and benchmark: K-Radar provides a large-scale 4DRT-based dataset and benchmark for 3D object detection across diverse road, time, and weather conditions.The dataset includes 35k frames and auxiliary Lidar, stereo-camera, and RTK-GPS/IMU measurements.
  • Dataset and benchmark: 4DRT preserves elevation alongside Doppler, range, and azimuth measurements, addressing the missing height information that limits conventional 3DRT-based detection.Existing 3DRT-based works are limited to 2D BEV detection because they lack elevation information.
  • Dataset and benchmark: The dataset uses calibrated high-resolution Lidar point clouds to support accurate 3D bounding-box annotation and transformation into the Radar coordinate frame.Spatial and temporal calibration parameters correct sensor separations and asynchronous measurements.
  • Baseline and findings: The 4DRT-based baseline NN outperforms a Lidar-based network in 3D detection, especially under adverse weather conditions.The authors also provide training, evaluation, labeling, calibration, and visualization devkits.

2 Related Works

Existing autonomous-driving datasets span multiple sensor modalities but often trade off scale, environmental diversity, Radar tensor density, annotation dimensionality, or adverse-weather coverage. K-Radar is positioned as a large-scale dataset combining dense 4DRT, 3D labels, and diverse conditions.

  • Existing datasets: KITTI offers calibrated camera and Lidar data with 3D labels, but its 15K frames are relatively limited in sample count and diversity.The frames were collected mostly in urban areas during daytime.
  • Radar datasets: Existing 3DRT datasets provide limited spatial annotations because their Radar tensors lack height information.CARRADA provides labels for up to two objects, while Zenar, RADIATE, and RADDet provide 2D BEV boxes and CRUW provides 2D point locations.
  • Radar datasets: VoD and Asytx provide 3D boxes with 4DRPCs, but do not release dense 4DRTs and contain relatively few frames.Their datasets contain 8.7K and 0.5K frames, respectively.
  • K-Radar position: K-Radar is presented as the first large-scale dataset combining 4DRT measurements, diverse conditions, and 3D bounding-box labels.It is designed to address gaps in both Radar representation and environmental coverage.
  • Weather coverage: Adverse-weather datasets also differ in sensor coverage: BDD100K provides only RGB front images, whereas RADIATE provides only 32-channel low-resolution Lidar.K-Radar combines 4DRT, high-resolution Lidar, and 360-degree RGB stereo imagery.

3 K-Radar

K-Radar is a multimodal 4DRT dataset and benchmark designed for 3D object detection across varied driving conditions. It preserves elevation information, supports calibrated annotation, and provides baseline networks that process height-aware or height-agnostic radar representations.

  • Dataset and benchmark: K-Radar provides multimodal measurements and 3D object-detection baselines for diverse autonomous-driving conditions.The dataset includes 4DRT, lidar, cameras, and calibrated sensor data; its baselines directly consume 4DRT.
  • Data collection and distribution: 35K frames are collected across adverse weather, day/night conditions, and urban, suburban, highway, and other road structures.The collection process balances conditions across training and test sets.
  • Annotation and object distribution: K-Radar contains 93.3K 3D bounding-box labels for five object classes within specified longitudinal and lateral ranges.Objects are annotated only in front of the ego-vehicle, and most objects lie within 60 m.
  • 4DRT representation: 4DRT records Doppler, range, azimuth, and elevation power measurements, unlike 3DRT, which lacks height information.The paper visualizes 4DRT by reducing Doppler, converting polar coordinates to Cartesian coordinates, and producing BEV, front-view, and side-view heatmaps.
  • Calibration and annotation: High-resolution lidar supports accurate 3D box annotation, while calibration transfers labels into the radar coordinate frame and supports multimodal alignment.The annotation workflow also uses BEV-2D when adverse weather removes lidar measurements.
  • Baseline networks: RTNH uses 3D sparse convolutions to retain height information, whereas RTN uses 2D convolutions that omit it.Both networks share preprocessing, backbone, neck, and head stages, but differ in their backbones.

4 Experiment

Experiments compare height-aware and height-agnostic radar baselines, then compare radar with a similarly structured lidar detector under different weather conditions. The results support the importance of elevation and the robustness of 4D radar in adverse weather.

  • Experimental design: The experiments compare RTNH with RTN to test height information and compare RTNH with PointPillars to test weather robustness.The target class is sedan, and evaluation uses BEV and 3D Average Precision.
  • RTN versus RTNH: 7.32% higher AP3D and 7.72% higher APBEV are achieved by RTNH than RTN.RTNH also uses less GPU memory because of sparse convolutions.
  • Radar versus lidar: Lidar-based PointPillars loses 33.5% and 29.6% performance in BEV and 3D detection under sleet versus normal weather.Under heavy snow, the corresponding drops are 25.8% and 22.6%.
  • Radar versus lidar: RTNH performance is better or similar in sleet and heavy snow than in normal conditions for both BEV and 3D detection.This contrasts with the substantial weather-related degradation reported for PointPillars.

5 Limitation and Conclusion

K-Radar offers a large multimodal 4DRT benchmark with annotations and demonstrates height-aware, adverse-weather-robust radar detection. Its main scope limitation is forward-only radar coverage.

  • Limitation: K-Radar’s 4DRT measurements cover only a 107-degree forward field of view rather than the 360-degree coverage of lidar and cameras.Recording surround 4DRT would require prohibitively large memory because dense four-dimensional measurements total roughly 12 TB.
  • Conclusion: 35K frames include 4DRT, lidar, surround cameras, and RTK-IMU data, with 93.3K labels covering five object classes up to 120 m.The dataset also provides tracking IDs and supports object tracking, SLAM, and other perception tasks.
  • Conclusion: Experiments demonstrate that 4DRT height information is important for 3D detection and that 4D radar remains robust under adverse weather.The authors position K-Radar as a resource for accelerating 4DRT-based autonomous-driving perception research.

A.1 Additional samples of the K-Radar dataset and explanation of LPCs for each weather condition

Additional materials document K-Radar’s weather-dependent sensor behavior, dataset organization, licensing, and privacy safeguards. Lidar measurements can disappear in sleet or heavy snow, while the dataset remains distributed across 58 sequences.

  • Weather-dependent lidar behavior: Lidar measurements for some objects disappear in sleet and heavy snow but remain partially available in rain and light snow.Sleet freezes onto the lidar surface, while heavy snow accumulates after forward driving and blocks measurements.
  • Dataset organization: K-Radar contains 35K frames divided into 58 sequences collected under different weather, road, and time conditions.Sequence details and dataset composition are provided in Tables 5 and 6.
  • Dataset organization: Each sequence contains 12 compressed folders describing data types, extensions, sizes, and usage.The folder structure is documented in Table 6.
  • License and privacy: The dataset uses CC BY-NC-ND licensing, while the released code uses Apache License 2.0.These licenses apply separately to the dataset and code.
  • License and privacy: Pedestrian, bicycle, and motorcycle image sequences have blurred faces to protect privacy.The authors also report that these faces are not recognizable in the sequences.

B Details of the sensor suite

K-Radar uses waterproofed sensors for safe data collection in adverse weather and documents their operational specifications.

  • The sensor suite uses waterproofed sensors rated IP66 or higher for adverse-weather data collection.Table 7 summarizes each sensor’s model name, output format, resolution, maximum operating distance, field of view, and frame rate.

C Criteria for weather conditions, road structures, and collecting time

K-Radar establishes sequence conditions using detailed criteria covering weather, road structures, and collection time.

  • Sequence conditions are established according to the detailed criteria listed in Table 8.

D.1 Details of annotation

K-Radar annotates 3D bounding boxes using calibrated Lidar point clouds, with a fallback process for scenes lacking object measurements. Calibration aligns Radar tensor and Lidar data using visual and spatial cues, while camera-Lidar calibration also supports dense depth-map generation.

  • Calibrated Lidar point clouds support accurate 3D bounding-box annotation, with maximum calibration and annotation errors of 0.5 cm and 0.7 cm, respectively.The annotation program provides 1.4 cm-per-pixel resolution and is included in the published development kits.
  • When object Lidar measurements are absent, annotators use overlapped calibrated BEV-2D and dash-camera images to create 3D bounding-box annotations.
  • 4DRT and Lidar point clouds are calibrated using BEV-2D and Lidar visualizations, sensor-placement information, and near-field objects to match temporal and spatial offsets.The process uses clearly visualized objects within about 30 m as calibration clues.
  • Lidar-camera calibration estimates extrinsic, lens-distortion, and intrinsic parameters, using a scanned 3D ego-vehicle model and a GUI for parameter refinement.Projected Lidar points are overlaid on camera images to inspect calibration results.
  • Projected sparse Lidar points provide ground-truth pixel depths, while interpolation produces dense depth maps for depth-estimation tasks.

E.1 Neck and head

The baseline detection heads operate on concatenated feature maps, while the height-aware RTNH backbone preserves 4DRT height information through 3D sparse convolutions before producing BEV features.

  • Neck and head: The neck converts multiple-resolution feature maps to a common size with TransposeConv2D and concatenates them before prediction.The resulting feature map is organized by channels and BEV grid dimensions.
  • Neck and head: Each grid predicts classification and regression outputs using two anchor yaw angles and eight bounding-box parameters per anchor.The configuration yields NCLS = 3 and NREG = 16.
  • Neck and head: RTNH uses 3D sparse and submanifold convolutions to reduce GPU memory while preserving height information from 4DRT.Dense transformed feature maps are converted into BEV representations with height information encoded before concatenation.
  • Neck and head: RTN uses a 2D dense convolution backbone without height encoding and evaluates ResNet50 and ResNext101 variants.The 2D-DCB-ResNext101 variant is reported as the representative RTN result because it performs better than the alternative.

F Qualitative results of RTNH and PointPillars with additional discussion in various conditions

RTNH and PointPillars are compared qualitatively across clear and precipitating conditions, showing robust 4D Radar detection, particularly when Lidar measurements degrade in sleet and heavy snow.

  • RTNH produces similar or more robust detection than PointPillars in non-precipitating road environments with multiple vehicles.The comparison includes normal and overcast conditions and emphasizes robustness to missed detections.
  • 4D Radar has similar or more robust detection performance than Lidar across varied road environments without precipitation.
  • Normal-condition AP can be lower than overcast AP because normal scenes contain more difficult alleyway situations and parked vehicles.The authors also attribute the pattern to lighting-insensitive 4D Radar and Lidar measurements.
  • Lidar measurements remain reliable in rain and light snow but degrade in sleet and heavy snow when the sensor surface is covered by frost or snow.This degradation is visible in the LPC measurements and reflected in PointPillars results.
  • Under adverse precipitation, the results demonstrate that 4D Radar is more robust than Lidar, without claiming RTNH is a better architecture than PointPillars.RTNH consumes 4DRT, whereas PointPillars consumes LPC.

G Consideration of the K-Radar dataset as a pre-training dataset for other Radar tensor datasets

The paper considers K-Radar for pre-training other Radar tensor datasets but cautions that differing sensor, resolution, and geographic distributions limit expected transfer gains. It also documents released tools and resources for experimentation, visualization, calibration, labeling, and tracking.

  • Pre-training limitations: K-Radar pre-training does not directly guarantee strong improvement on RADIATE because the datasets have inherently different characteristics.
  • Pre-training limitations: Different Radar types produce different power-measurement distributions, which can degrade performance when training and target data distributions differ.
  • Pre-training limitations: RADIATE has 0.175m resolution compared with K-Radar's 0.46m, creating a mismatch that can adversely affect detection performance.
  • Pre-training limitations: K-Radar and RADIATE also differ geographically: K-Radar data comes from South Korea with right-side driving, while RADIATE comes from the U.K. with left-side driving.
  • Pre-training limitations: Because these differences also apply to other Radar tensor datasets, performance improvement from reciprocal K-Radar pre-training is difficult to expect.
  • Additional evaluation: Table 10 extends RTNH evaluation to wider spatial ranges and both Sedan and Bus or Truck classes.The evaluated ranges are x: 0–72m, y: -16–16m, and z: -2–7.6m.
  • Additional evaluation: Bus or Truck performance is marked '-' for rain and sleet because those classes are absent under these conditions.
  • Released resources: The released devkits include modular training, evaluation, reasoning, visualization, inference, labeling, calibration, and tracking resources.GUI-based programs support visualization and neural-network inference on large amounts of data.
Loading 2206.08171v4…