Source-linked AI summary
RadarScenes: A Real-World Radar Point Cloud Data Set for Automotive Applications
Ole Schumann, Markus Hahn, Nicolas Scheiner, Fabio Weishaupt, Julius F. Tilly, Jürgen Dickmann, Christian Wöhler
TL;DR
RadarScenes addresses the limited and difficult-to-compare data available for radar-based machine learning by introducing a large, diverse automotive radar point-cloud dataset with manual point-wise annotations. It records 118.9 million radar points over 100.1 km and 4.3 hours, while providing track identities and an evaluation scheme for comparing radar perception methods. Its scope is limited to radar perception or multi-sensor pretraining because other sensor modalities are absent apart from documentation imagery and odometry.
Problem
Limited radar data makes radar-based machine-learning development and comparison difficult, while existing datasets often lack sufficient size, diversity, or usable point-cloud annotations.
Method
The paper constructs RadarScenes from manually labeled detections recorded by four radar sensors, assigning semantic and track labels and proposing a common evaluation scheme.
Results
118.9 million radar points were collected over 100.1 km and 4.3 hours, with point-wise labels supporting radar perception algorithms focused on moving road users.
Takeaways & Limitations
RadarScenes provides a public, diverse resource for developing and comparing radar perception algorithms, including methods using sequences of measurements.
Takeaways & Limitations
The absence of other sensor modalities, apart from documentation camera and odometry data, limits use to radar perception or multi-sensor pretraining.
Abstract
from arXiv · showhide
A new automotive radar data set with measurements and point-wise annotations from more than four hours of driving is presented. Data provided by four series radar sensors mounted on one test vehicle were recorded and the individual detections of dynamic objects were manually grouped to clusters and labeled afterwards. The purpose of this data set is to enable the development of novel (machine learning-based) radar perception algorithms with the focus on moving road users. Images of the recorded sequences were captured using a documentary camera. For the evaluation of future object detection and classification algorithms, proposals for score calculation are made so that researchers can evaluate their algorithms on a common basis. Additional information as well as download instructions can be found on the website of the data set: www.radar-scenes.com.
I. INTRODUCTION
Radar is valuable for automotive perception because it remains robust in adverse weather and directly measures relative velocity, but radar-based semantic understanding lacks the data resources available to vision. RadarScenes addresses this gap with a large, diverse, publicly available point-cloud dataset.
- Radar combines robustness in adverse weather with direct relative-velocity measurement and moderate costs, supporting its role in automotive sensor stacks.
- Radar-based semantic scene understanding remains less mature than image-based automotive perception, which has benefited from strong algorithms and large open datasets.
- Limited publicly available radar data makes it difficult to develop radar-based machine-learning algorithms and compare results across methods.
- Existing automotive radar datasets often lack sufficient size, diversity, or sensor quality.
- RadarScenes provides 11 object classes, over 7000 manually labeled road users, 100 km of diverse street scenarios, and public access for radar point-cloud perception tasks.
II. RELATED WORK
Existing radar datasets trade off scenario diversity, sensor characteristics, temporal continuity, annotation completeness, and Doppler information. RadarScenes is presented as a more diversified automotive-radar dataset, while its lack of additional sensor modalities limits its scope.
- Many existing radar datasets are small, use specialized sensors, or impose other limitations that reduce their suitability for broad radar perception research.
- nuScenes and Zendar provide real-world scenarios, but their radar outputs are sparse for other road users, limiting purely radar-based evaluation.
- Astyx and NLOS-Radar use restricted object or scenario coverage, while Astyx also lacks sufficient scenarios and temporal continuity for time-based algorithms.
- Oxford Radar RobotCar, MulRan, and RADIATE produce dense radar images with rotating sensors, but their installations require substantial space and roof mounting.
- Some rotating-radar datasets omit Doppler information, reducing their usability for automotive applications where radial-velocity measurement is valuable.
- CARRADA and CRUW provide annotations for radar spectra rather than full point clouds, and only a small fraction of CRUW sequences contain labels.
- RadarScenes is described as the most diversified higher-resolution automotive-radar dataset, but its absence of other sensor modalities limits use to radar perception or multi-sensor pretraining.
III. DATA SET
The dataset overview pairs bird’s-eye-view radar plots with corresponding documentation-camera images to provide a general view of recorded scenes.
- Figure 2 presents bird’s-eye-view radar plots alongside corresponding documentation-camera images.
A. Measurement Setup
The measurement setup uses four 77 GHz series-production radar sensors in near-range mode, records vehicle motion for ego-motion compensation, and supplies camera imagery for documentation.
- Four 77 GHz series-production automotive radar sensors recorded the dataset using near-range mode with detection up to 100 m.
- Each radar sensor covers a ±60° field of view in near-range mode.
- The vehicle records position, orientation, velocity, and yaw rate so radar measurements can be ego-motion compensated and transformed into global coordinates.
- A passenger-cabin camera behind the windshield provides optical images for documenting the recorded scenarios.
B. Labeling
RadarScenes uses expert manual annotation to label moving road users, preserving semantic classes and temporal identities while accounting for radar-specific ambiguities and measurement artifacts.
- Manual annotation: Expert annotators manually label radar detections to ensure high-quality annotations without relying on camera, lidar, or instructed-road-user labels.The task requires identifying detections corresponding to objects of interest and maintaining realistic, consistent object proportions.
- Class labels: The data set focuses on moving road users and assigns eleven semantic object classes, with an optional mapping to five coarser categories for more balanced tasks.The coarse mapping supports classification and detection settings requiring a more balanced label distribution.
- Temporal identity: Each dynamic-object detection receives a label id for semantic class and a track id that links detections of the same real-world object across recording time.A new track id is assigned after more than 500 ms of occlusion or stopped motion.
- Radar-specific labeling: The annotations deliberately estimate object dimensions over the full period of observed detections rather than assigning every nearby non-zero-Doppler detection to the object.This can make labels appear visually inconsistent at first glance, although the apparent mismatch is often misleading.
- Radar-specific labeling: Non-zero Doppler does not by itself identify moving-object detections because mirror effects, measurement ambiguities, and sidelobe false positives can produce misleading velocities.Such detections may retain the default static label, so Doppler thresholding alone is insufficient.
C. Statistics
RadarScenes provides extensive measurements across urban and rural routes, with statistics reported by object class, sensor scan, and detection range.
- Dataset scale: 118.9 million radar points were measured over 100.1 km of urban and rural routes.The recordings span 4.3 h, with individual scenarios lasting 13 s to 4 min.
- Dataset scale: The data set contains measurements from scenarios collected under varied conditions, while excluding static-environment detections from the per-class annotated statistics.Static detections comprise more than 90% of all measured points and serve as the fallback class.
- Class statistics: Table II reports class statistics before and after mapping the original labels to coarser categories.The upper and lower portions correspond to the pre-mapping and post-mapping distributions, respectively.
- Class statistics: Figure 4 summarizes annotated-point and object counts, their distribution across sensor scans, and normalized class-range distributions up to 100 m.The five main object classes are compared across these three views.
D. Data Set Structure
Each sequence stores radar and odometry measurements together in an HDF5 file, with separate tables describing ego motion and individual radar detections.
- File organization: Each sequence is stored in one HDF5 file containing separate radar and odometry tables.The format keeps the two modalities associated within each sequence.
- Odometry table: Each odometry row records a timestamp together with the ego vehicle’s global position and orientation.These values describe the vehicle state at that timestamp.
- Radar table: Each radar row stores a detection timestamp and positions in global, ego-vehicle, and sensor coordinate systems, along with ego-motion-compensated measurements.The row structure links each radar detection to multiple spatial reference frames.
IV. EVALUATION
RadarScenes supports multiple radar perception tasks and proposes evaluation measures intended to make results from independent studies more comparable.
- Supported tasks: RadarScenes can support object detection, object formation, tracking, classification, semantic instance segmentation, and grid mapping.The annotations enable direct assessment of these application types.
- Evaluation: The authors suggest common evaluation measures to ease comparison between independent studies.The appropriate metric depends on the specific problem being solved.
Object Detection and Instance Segmentation
The evaluation treats object detection as point-level instance segmentation and addresses comparison difficulties caused by sparse radar data and differing scan accumulation. It recommends standardized class mappings and complementary metrics while noting radar-specific IoU limitations.
- Radar sparsity and differing accumulation windows make comparisons difficult, while point-wise labels provide no bounding boxes with real object dimensions.
- Object detectors are evaluated as instance-segmentation models by comparing predicted and ground-truth instances, class labels, and point-level overlap at each time step.
- The recommended mapped-label evaluation uses five main object classes, excluding animal and other to reduce training and validation imbalance.
- At least mAP50 should be reported, alongside metrics such as LAMR and macro-averaged class F1 to improve comparability.
- Lower IoU thresholds such as 0.3 may be useful because ground or multipath detections can make seemingly accurate radar predictions miss strict IoU targets.
- Classification evaluates object clusters rather than whole scenes and uses an additional rejection class for clusters identified as clutter without ground-truth overlap.
Semantic Segmentation
Semantic segmentation is evaluated with point-wise macro-averaged F1 across five main classes and static background, supplemented by classwise scores.
- Semantic segmentation scores use point-wise macro-averaged F1 across five main classes and one static background class.
- Additional classwise scores provide more detailed interpretation of semantic-segmentation performance.
V. CONCLUSION
The paper introduces RadarScenes as a large automotive radar dataset with point-wise labels for moving road users and supporting scene imagery. It also proposes an evaluation scheme to compare algorithms across several radar perception tasks and accumulation strategies.
- RadarScenes provides a new automotive radar dataset with point-wise semantic and track annotations for moving road users.
- The dataset includes documentary-camera images to make recorded scenes easier to understand.
- A detailed evaluation scheme supports comparisons among techniques using different accumulation schemes.
- The proposed benchmark covers object detection, clustering, classification, and semantic or instance segmentation.