Source-linked AI summary
CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations
A. Ouaknine, A. Newson, J. Rebut, F. Tupin, P. Pérez
TL;DR
Autonomous-driving research lacks large, annotated datasets containing synchronized raw radar alongside other sensors, even though radar provides radial velocity and resilience to darkness and weather. CARRADA introduces such a dataset with range-angle-Doppler annotations, a camera-based semi-automatic annotation method, and a radar semantic-segmentation baseline. The dataset and code support supervised radar research across detection, segmentation, tracking, and sensor fusion.
Problem
Autonomous-driving deep learning lacks datasets combining synchronized raw radar with detailed annotations, limiting available data for radar scene-understanding tasks.
Method
The paper constructs CARRADA, generates radar annotations semi-automatically from camera and physical information, and evaluates an FCN-based semantic-segmentation baseline.
Results
The baseline learns meaningful representations that detect and classify moving-object shapes in raw range-angle and range-Doppler radar representations.
Takeaways & Limitations
CARRADA provides synchronized video, raw radar representations, multiple annotation types, and instance identities for supervised radar perception research.
Abstract
from arXiv · showhide
High quality perception is essential for autonomous driving (AD) systems. To reach the accuracy and robustness that are required by such systems, several types of sensors must be combined. Currently, mostly cameras and laser scanners (lidar) are deployed to build a representation of the world around the vehicle. While radar sensors have been used for a long time in the automotive industry, they are still under-used for AD despite their appealing characteristics (notably, their ability to measure the relative speed of obstacles and to operate even in adverse weather conditions). To a large extent, this situation is due to the relative lack of automotive datasets with real radar signals that are both raw and annotated. In this work, we introduce CARRADA, a dataset of synchronized camera and radar recordings with range-angle-Doppler annotations. We also present a semi-automatic annotation approach, which was used to annotate the dataset, and a radar semantic segmentation baseline, which we evaluate on several metrics. Both our code and dataset are available online.
I. INTRODUCTION
CARRADA addresses the need for robust autonomous-driving scene understanding by introducing synchronized camera and raw-radar data with detailed annotations, an annotation method, and a segmentation baseline.
- Motivation: Large annotated datasets are needed to train deep-learning models for autonomous-driving tasks such as object detection and semantic segmentation.The passage identifies expanding open annotated datasets as a key challenge for improving autonomous-driving performance.
- Motivation: Radar complements cameras and lidar because it measures radial velocity and remains usable in darkness and adverse weather, despite noisy signals and low angular resolution.These interpretation difficulties have contributed to cameras and lidars being preferred.
- Contributions: CARRADA provides synchronized camera data, raw radar sequences, range-angle and range-Doppler annotations, object categories, unique identifiers, and sparse-point, bounding-box, and dense-mask labels.The dataset supports object detection, semantic segmentation, and tracking in raw radar signals.
- Contributions: The paper presents a semi-automatic radar-annotation method that uses camera information instead of lidar to reduce annotation time and cost.A radar semantic-segmentation baseline is also proposed and evaluated using established metrics.
- Organization: The paper covers related work, radar background, dataset acquisition, annotation with temporal tracking, radar semantic segmentation, dataset limitations, and conclusions.These topics are organized across Sections II–VII.
A. Related work
Prior autonomous-driving datasets largely center on camera and lidar data, while radar datasets are often processed, small, or sparsely annotated. CARRADA targets the missing combination of synchronized raw radar, multiple representations, and task-oriented annotations.
- Radar applications: Driving research has used radar representations selected for tasks including classification, odometry, detection, occupancy-grid segmentation, and sensor fusion.The cited work spans Doppler spectrograms, range-angle, and range-Doppler representations.
- Existing datasets: Existing autonomous-driving datasets commonly provide camera video or lidar point clouds with bounding boxes and semantic-segmentation masks.The passage frames annotated data as necessary for deep-learning-based scene understanding.
- Existing datasets: None of the established datasets described provides raw radar recordings synchronized with other sensors and annotated for scene understanding.Recent radar datasets are characterized as processed or barely annotated, although raw data retain information about all reflective objects.
- Radar datasets: nuScenes provides radar alongside lidar and cameras, but its radar release is processed, non-annotated, and limited to tens of points per frame.Other examples are limited in scale, representation, or annotation scope.
- Gap addressed: CARRADA claims that range-angle and range-Doppler raw radar data had not previously been released together with annotations for detection, segmentation, and tracking.The paper also reports no related deep-learning work exploiting both annotation types simultaneously.
B. Radar sensor and effects
Automotive radar infers reflector range, radial velocity, and angle from transmitted and received signals. Its recorded 3D tensor is transformed with FFT operations into range, Doppler, and angle-resolved representations whose bins are set by sensor resolution.
- Radar sensing: Radar compares transmitted and received electromagnetic waves to infer an object’s distance, radial velocity, azimuth angle, and elevation.Automotive radars commonly use MIMO systems with multiple transmitter and receiver antennas.
- Range and velocity effects: FMCW radar linearly modulates a chirp’s frequency, and reflection-induced phase or time delay provides access to the reflector’s distance.The received signal is described through its phase after reflection.
- Range and velocity effects: The Doppler effect arises from motion-dependent phase variation, so the frequency shift between transmitted and received signals provides radial speed.The radial velocity is defined as v_R = dr/dt, and the Doppler frequency depends linearly on relative reflector speed.
- Signal representation: Consecutive filtered intermediate-frequency signals form a time-domain 3D tensor indexed by chirp, chirp sampling, and Tx/Rx antenna pairs.This frame buffer is the input to subsequent frequency-domain processing.
- Angle estimation: A MIMO radar’s inter-antenna phase differences encode orientation, with azimuth inferred from phase variation between adjacent receiver pairs.The cited relation connects azimuth phase difference with antenna spacing and angle.
- Signal processing: A 3D-FFT applies Range-FFT, Doppler-FFT, and Angle-FFT operations to resolve range, radial velocity, and angle, respectively.The resulting bins are discretized according to radar resolution, including range, velocity, and angle resolution.
III. DATASET
CARRADA is a synchronized camera–radar dataset recorded for urban-driving scenarios, providing object annotations across range-angle and range-Doppler representations.
- Dataset acquisition: The acquisition setup uses an FMCW radar and camera mounted on a stationary car, with synchronized frame rates and calibrated Cartesian coordinates.The radar uses 2 transmitters and 4 receivers, producing 8 virtual antennas.
- Dataset composition: The dataset contains eight object configurations distributed across three proposed split parts.Their distribution is expressed in frame numbers.
- Dataset statistics: Mean radial-velocity distributions are provided for each object category, with velocities represented as signed values in the annotations.Negative values indicate motion away from the radar, while positive values indicate approach.
- Scenarios: One or two objects move simultaneously along varied trajectories, including approaching, receding, and lateral motion, to simulate urban-driving scenarios.Each moving object is tracked as an instance throughout its sequence.
- Annotations: Each object signature is annotated in both range-angle and range-Doppler representations using sparse points, boxes, and dense masks.Instances also receive an identification number, category, and localization.
IV. PIPELINE FOR ANNOTATION GENERATION
Because radar representations are noisy physical shapes rather than natural images, CARRADA uses a semi-automatic video-based pipeline to generate radar annotations.
- Motivation: Automotive radar representations contain objects with varying shapes and sizes while encoding physical measurements, making high-quality annotation nontrivial.The pipeline addresses this difficulty by using video frames as its basis.
- Approach: The proposed annotation pipeline is semi-automatic and based on video frames to produce labels for radar representations.Its purpose is to generate annotations despite the interpretive difficulty of radar data.
A. From vision to physical measurements
The pipeline converts camera detections into radar-aligned physical measurements, estimates radial velocity from temporal motion, and propagates the resulting annotations through radar sequences.
- From synchronized sensing: Synchronized camera and radar recordings provide visual priors for object category, real-world position, and radial velocity.These physical estimates initialize radar annotations and tracking across the sequence.
- Visual detection and tracking: Mask R-CNN supplies semantic masks and bounding boxes, while SORT tracks each detected instance using box overlap across frames.Both mask and box predictions are used to compute the object center of mass and track instances.
- Coordinate estimation: The pipeline projects each segmented object’s center of mass to its bottom-most mask pixel and treats that ground-localized pixel as the reference point.Camera intrinsic and extrinsic parameters then map image coordinates to real-world coordinates.
- Velocity estimation: The velocity vector is estimated from real-world positions separated by δt = 1 second, using consecutive frames at times t − δt and t.The resulting motion is used to calculate radial velocity relative to the radar.
- Radar-aligned measurement: Radial velocity corresponds to the projection of the object’s velocity vector onto the line connecting the radar and the object.This relationship is illustrated through camera-derived motion and radar representations.
B. DoA clustering and centroid tracking
The annotation pipeline converts radar reflections into a sparse DoA-Doppler point cloud, automatically selects a Mean Shift bandwidth, and tracks each instance cluster through time.
- B. DoA clustering and centroid tracking: The range-angle radar scene is transformed into a Cartesian DoA point cloud and enhanced with each point’s Doppler value.CFAR filtering retains high-intensity points while considering local relationships.
- B. DoA clustering and centroid tracking: The closest cluster to the computer-vision feature point is assigned to the target, and its centroid is iteratively propagated through previous and subsequent frames.The centroid supplies the tracked points I_t−1 and I_t+1 across the sequence.
- B. DoA clustering and centroid tracking: Mean Shift iteratively updates points toward weighted local means, grouping points that converge to nearby final locations into clusters.The kernel is a spherical Gaussian centered at each point, with bandwidth σ.
- B. DoA clustering and centroid tracking: Because clustering depends strongly on bandwidth and changing target reflectivity, the method evaluates ordered bandwidths and selects the cluster with the most stable point distribution.Stability is assessed by comparing fitted cluster distributions with Jensen-Shannon divergence.
C. Projections and annotations
The selected radar cluster is projected into range-Doppler and range-angle coordinates, then converted into sparse points, boxes, and dense masks for supervised learning.
- C. Projections and annotations: The optimal cluster associated with I_t is treated as the tracked object and receives a category from the image segmentation model.Its points are projected using radial velocity and real-world coordinates.
- C. Projections and annotations: The cluster is projected into range-Doppler and range-angle representations, producing sparse-point annotations in both modalities.The projections use radial velocity and Cartesian-to-polar coordinate conversion, respectively.
- C. Projections and annotations: Bounding boxes are formed as axis-aligned rectangles spanning the minimum and maximum coordinates of each projected point set.The same construction applies to either range-Doppler or range-angle points.
- C. Projections and annotations: Dense masks are generated by dilating each sparse annotation with a circular structuring element of radius r.The mask is the union of disks centered on the sparse points.
- C. Projections and annotations: The resulting annotations support the proposed radar semantic-segmentation baseline.The baseline is trained and evaluated on the annotation types described above.
V. BASELINE
The baseline applies FCN architectures to range-Doppler and range-angle radar representations, training dense masks and evaluating predictions against dense and sparse annotations.
- V. BASELINE: FCN-32s, FCN-16s, and FCN-8s use progressively more multiscale feature maps and skip connections to predict full-resolution segmentation masks.The output classifies each input bin as background, pedestrian, cyclist, or car.
- V. BASELINE: Models are evaluated with IoU, PP, and PR, aggregating category metrics using arithmetic and harmonic means across three random-seed trainings.The evaluation covers both radar representations.
- V. BASELINE: Dense-mask training is evaluated on dense masks and sparse points, but IoU is omitted for sparse points because localization is not assessed consistently.Sparse-point arithmetic and harmonic means cover three object classes, whereas dense-mask means cover four classes.
- V. BASELINE: The baseline learns meaningful representations and detects and classifies moving-object shapes even with sparse-point annotations.The authors report weaker range-angle performance than range-Doppler because low angular resolution produces less precise annotations.
- V. BASELINE: Range-angle performance is lower than range-Doppler performance, while the results remain promising without modeling the temporal dimension of object signatures.The paper suggests Cartesian transformation as a possible extension for range-angle representations.
VI. DISCUSSIONS
CARRADA provides precise annotations for supervised radar tasks, while its annotation pipeline and baseline leave clear opportunities for temporal, sparse-point, and detection-oriented extensions.
- VI. DISCUSSIONS: CARRADA provides precise annotations for exploring a range of supervised learning tasks, alongside a semantic-segmentation baseline trained on dense masks.The dataset is positioned as a resource for supervised radar research.
- VI. DISCUSSIONS: Temporal information and joint training with dense masks and sparse points are proposed as extensions to the current baseline.The authors also suggest optimizing architectures and loss functions for sparse ground-truth points.
- VI. DISCUSSIONS: Bounding-box annotations could support object detection, although off-the-shelf detectors are not adapted to radar representations and their unusual object-signature sizes.This identifies a task opportunity together with an architecture mismatch.
VII. CONCLUSION
CARRADA provides synchronized video and raw radar representations with rich object annotations, supported by a semisupervised annotation pipeline and public code and data.
- CARRADA combines synchronized video frames with raw range-angle and range-Doppler radar representations.
- Radar signatures receive sparse-point, bounding-box, dense-mask, and unique-identity annotations for object localization and categorization.
- A semisupervised algorithm generates the annotations from visual and physical knowledge.
- The annotation pipeline is intended for similar camera-radar recordings and supports research in radar sensing and multisensor fusion.
- The dataset, annotation code, and visualization code are publicly available.