PaperlayerView exact revision on arXiv

Source-linked AI summary

RAMP-CNN: A Novel Neural Network for Enhanced Automotive Radar Object Recognition

Xiangyu Gao, Guanbin Xing, Sumit Roy, Hui Liu

arXiv:2011.08981v2eess.SPcs.AIcs.CVcs.LGcs.RO

TL;DR

Reliable radar object recognition is constrained by limited angular resolution and the lack of prior methods jointly processing spatial and temporal information. RAMP-CNN uses 3D radar cube sequences and multiple network perspectives to exploit both types of information, improving recognition across tested scenarios while increasing complexity.

  • Problem

    Limited antenna counts can cause poor angular resolution, while prior neural-network methods had not jointly processed spatial and temporal radar information for object recognition.

  • Method

    RAMP-CNN inputs 3D radar cube sequences to estimate object range, azimuth angle, and class while exploiting temporal chirp information and spatial changes across frames.

  • Results

    RAMP-CNN achieves significant performance improvement over prior works across parking lot, curbside, and on-road scenarios, with 97.38% AP and 98.37% AR in the parking lot.

  • Takeaways & Limitations

    The model establishes a new state-of-the-art baseline for radar object recognition, particularly improving performance for moving objects through temporal and spatial information.

  • Takeaways & Limitations

    In some hard cases, RAMP-CNN may still provide poor radar object recognition for autonomous driving.

Abstract

from arXiv · show

Millimeter-wave radars are being increasingly integrated into commercial vehicles to support new advanced driver-assistance systems by enabling robust and high-performance object detection, localization, as well as recognition - a key component of new environmental perception. In this paper, we propose a novel radar multiple-perspectives convolutional neural network (RAMP-CNN) that extracts the location and class of objects based on further processing of the range-velocity-angle (RVA) heatmap sequences. To bypass the complexity of 4D convolutional neural networks (NN), we propose to combine several lower-dimension NN models within our RAMP-CNN model that nonetheless approaches the performance upper-bound with lower complexity. The extensive experiments show that the proposed RAMP-CNN model achieves better average recall and average precision than prior works in all testing scenarios. Besides, the RAMP-CNN model is validated to work robustly under nighttime, which enables low-cost radars as a potential substitute for pure optical sensing under severe conditions.

I. INTRODUCTION

Automotive radar offers robust range, velocity, and angle sensing when optical sensors degrade, but limited angular resolution and the impracticality of 4D convolution motivate a lower-complexity temporal-spatial approach. RAMP-CNN processes multi-frame radar representations through multiple perspectives and is reported to improve recognition performance, including at night.

  • Motivation: Millimeter-wave radar provides object detection and localization that are largely independent of environmental conditions, supporting perception when cameras and lidar degrade.The paper specifically highlights nighttime, glaring sunlight, snow, rain, and fog as challenging optical-sensing conditions.
  • Motivation: Limited antenna integration can cause poor angular resolution, so prior methods rely on temporal movement patterns when objects are not spatially separable.Targets at the same distance and radial velocity may remain unresolved in angle with compact radar hardware.
  • Gap and contribution: Prior neural-network approaches had not jointly processed spatial and temporal information from sequences of 3D radar cubes.The paper identifies this joint use of multi-frame radar-cube information as its fundamental contribution.
  • Proposed approach: RAMP-CNN slices each RVA heatmap into range-angle, range-velocity, and velocity-angle views, processes the resulting sequences with parallel lower-dimension models, and fuses their features.This design avoids impractical 4D convolution while combining multiple perspective-specific feature bases.
  • Data and training: The radar-camera system uses synchronized camera data to provide object locations and classes for supervised radar training under good light and weather conditions.Camera assistance is used to create training supervision rather than to replace radar during the described recognition pipeline.
  • Reported contributions: RAMP-CNN is reported to outperform prior works and to operate robustly at nighttime, while also using radar-specific augmentation and a loss function emphasizing velocity-related features.The paper presents these elements as contributions validated through extensive testing and ablation analysis.

II. RELATED WORK

Prior radar-recognition methods use varied input formats and often separate localization from classification, while radar preprocessing estimates range, velocity, and angle from FMCW signals. These approaches motivate jointly exploiting spatial, temporal, and multiperspective radar information.

  • Prior radar-recognition methods: Prior methods use STFT, range-angle, range-Doppler-angle, or range-velocity-angle inputs for radar object recognition.Reported systems include micro-Doppler extraction, YOLO-based detection and classification, and CNN-based class prediction.
  • Prior radar-recognition methods: 93% recognition accuracy was reported for three-class discrimination using STFT intensity heatmaps.The classes were car, pedestrian, and cyclist.
  • Prior radar-recognition methods: Existing approaches differ in how they preserve radar information, including temporal patterns, spatial structure, or selected dimensions of the radar spectrum.RODNet uses range-angle heatmap sequences, while other methods condense or emphasize particular radar dimensions.
  • Radar data preprocessing: FMCW preprocessing applies FFT operations to estimate range, radial velocity, and arrival angle from radar measurements.Range FFT uses beat-frequency peaks, Velocity FFT estimates Doppler velocity across chirps, and Angle FFT resolves receiver-array arrival angles.
  • Radar data preprocessing: The adopted 2-transmit, 4-receive TDM-MIMO configuration forms an 8-element virtual array, with approximately 15° boresight angle resolution.Motion-induced phase errors are compensated on virtual antennas before Angle FFT.

IV. RAMP-CNN MODEL: A CONVOLUTIONAL NEURAL NETWORK FOR RADAR DATA

RAMP-CNN combines three convolutional autoencoders that process range-angle, range-velocity, and velocity-angle heatmap sequences. Their complementary spatial and temporal features are fused for radar object recognition while retaining phase information where relevant.

  • RAMP-CNN architecture: RAMP-CNN uses three convolutional autoencoders to extract features from RA, RV, and VA heatmap sequences.The three perspectives provide complementary representations for the final recognition decision.
  • RAMP-CNN architecture: The RA autoencoder processes complex-valued sequences with 3D convolutions to capture spatial patterns and their temporal changes across frames.Real and imaginary components are represented as separate channels, preserving phase information.
  • Physical significance of network design: A pedestrian’s six-frame range-velocity sequence shows small location movement but substantial changes in velocity patterns.The figure uses velocity on the x-axis and range on the y-axis, illustrating temporal motion information.
  • RAMP-CNN architecture: The RV and VA autoencoders process absolute-valued heatmap sequences and extract perspective-specific distributions together with movement patterns over time.The RV and VA heatmaps are formed by summing power over the omitted RVA dimension.
  • Network details: Each CAE uses six 3D convolutional layers and three 3D transposed convolutional layers to summarize spatio-temporal patterns.The model adopts a 3D convolutional-deconvolutional structure for high-level feature representation.
  • Feature fusion: RAMP-CNN aligns VA and RV features with the RA domain, concatenates the feature bases along channels, and feeds them to the classifier.VA features are condensed over velocity and replicated along range, while RV features are replicated along angle.

Convolution Layers after Feature Fusion Module:

After feature fusion, RAMP-CNN uses 3D convolutional layers to process temporal, range, angle, and channel information, while its loss design emphasizes detections from all three radar perspectives.

  • Convolution Layers after Feature Fusion Module:: The fused features pass through a 3D inception layer and an ordinary 3D convolutional layer for recognition decisions.The ordinary layer operates across time, range, and angle, while the inception layer operates across channels, range, and angle.
  • Convolution Layers after Feature Fusion Module:: The inception layer uses (3, 5, 5), (3, 3, 3), and dilated (3, 1, 21) kernels to capture multiscale features and broad angular context.The dilated angular kernel covers almost all angle cells and reduces complexity, addressing side-lobe false alarms.
  • Camera Supervision: Camera-derived object locations and classes supervise RAMP-CNN during training, while radar operates independently during testing.The synchronized radar-camera system supplies range, azimuth angle, and class information from camera processing under good conditions.
  • Camera Supervision: RAMP-CNN predicts centerpoint heatmaps for pedestrian, cyclist, and car classes across time, range, azimuth angle, and class dimensions.A value of 1 denotes a detected center point, while 0 represents background.
  • D. Loss Function for All-perspectives Learning: The training objective is a modified penalty-reduced pixelwise logistic regression with focal loss, adding κ = 4 to focus more strongly on object regions.The loss uses α = 2 and β = 4, with normalization by the number of ground-truth objects.
  • D. Loss Function for All-perspectives Learning: A second loss term sets the range-angle input to zero, forcing the network to learn supplementary information from range-velocity and velocity-angle perspectives.The final loss is a weighted sum of the original and perspective-masked terms, with γ = 0.5.

V. RADAR DATA AUGMENTATION ALGORITHMS

The paper introduces radar-specific augmentation operations that account for polar geometry, range-dependent energy loss, and nonuniform angular resolution rather than directly applying standard image transformations.

  • V. RADAR DATA AUGMENTATION ALGORITHMS: Radar augmentation must account for complex inputs, energy loss with range, and nonuniform angular resolution in the angular domain.These differences limit the direct use of conventional RGB-image augmentation methods.
  • Horizontal Flipping: Horizontal flipping swaps the left and right portions of the radar cube along azimuth because radar has symmetric angular resolution and antenna gain.This operation can therefore be applied directly to radar data similarly to image flipping.
  • Translating in Range: Range translation transforms polar radar data into Cartesian coordinates before shifting the target along the y-axis while keeping the x-axis fixed.Interpolation and down-sampling are required because the polar-to-Cartesian relation is nonlinear.
  • Translating in Range: Range translation also requires range-dependent energy compensation and antenna-phase adjustment after shifting the target.The phase changes because the target azimuth changes during the operation.
  • Translating in Angle: Angle translation applies a predefined azimuth shift, equivalent to rotating the target around the origin in Cartesian coordinates.The operation maps shifted Cartesian targets back to the polar angular spectrum and compensates antenna-gain loss.

Interpolating:

Interpolation fills blank regions created by radar translations, using low-amplitude environmental noise to imitate areas without objects and support augmented training data.

  • Interpolating:: Interpolation fills the white-stripe gaps left by translating operations in the radar cube.The method uses environmental noise rather than zeros to represent regions where no object is present.
  • Interpolating:: The environmental-noise samples are selected from the smallest 5% of amplitudes in the 3D radar cube.This provides low-amplitude background content for the blank regions.
  • Dataset and Platform: The collected UWCR dataset combines camera images and raw radar data from vehicle-mounted platforms across multiple driving scenarios.Collection emphasized conditions where cameras are largely ineffective, including challenging light conditions.
  • Dataset and Platform: The platform uses two FLIR cameras and two perpendicular TI radars, although the reported experiments use only the horizontal-array radar data.The horizontal and vertical arrays provide range-azimuth and range-elevation information, respectively.
  • Camera Supervision: Camera detection and depth-estimation outputs are calibrated and saved as ground truth for training and evaluation.The synchronized cameras provide semantic object locations and classes for radar supervision.
  • Radar Preprocessing: The preprocessing applies 3-D FFTs with 128 points per range, angle, and velocity dimension across 16 input frames.The resulting input size is 128 × 128 × 128 × 16 in range bins, angle bins, velocity bins, and frame number.

Data Augmentation:

Augmented RVA sequences are used only during training, while evaluation compares RAMP-CNN with prior radar-recognition models across multiple scenarios using average precision and average recall.

  • Data Augmentation:: The training pipeline applies flipping, range translation, angle translation, and mixing to processed RVA heatmap sequences.Augmented data is stored locally and excluded from testing to reduce overfitting.
  • Dataset Split: Nighttime data is placed entirely in the test set because low-light camera images cannot provide ground-truth labels.Nighttime performance is therefore evaluated qualitatively rather than with numerical metrics.
  • Dataset Split: The test set contains parking-lot, curbside, on-road, and nighttime scenarios with separate data distributions.The training-set count excludes augmented data, which is included in the complete training set used for RAMP-CNN.
  • Evaluation Setup: RAMP-CNN is compared with RODNet-CDC, RODNet-HG, and CDMC, covering state-of-the-art radar-recognition and micro-Doppler-based approaches.The competing models are retrained according to the reported experimental procedure.
  • Evaluation Metrics: Average precision and average recall measure correctly located and classified instances against false alarms and missed or misclassified detections.Predicted center points are filtered with CFAR at threshold 0.2, and localization uses an object-size-adaptive distance threshold.
  • E. Evaluation Results: 97.38% AP and 98.37% AR are achieved by RAMP-CNN on the parking-lot test set, beating all prior works.The parking-lot set contains 9,900 frames and is described as relatively easy because its background is clean and objects are few.

The Parking Lot Scenario:

RAMP-CNN is evaluated across multiple test scenarios, including challenging on-road and nighttime conditions, and its components are examined through ablation and complexity analyses.

  • Curbside scenario: RAMP-CNN achieves 79.25% AP and 84.21% AR on the curbside test set, improving over RODNet-CDC’s 64.01% AP and 71.97% AR.The curbside set contains 7200 frames with pedestrians, cyclists, and cars, including simultaneously appearing and closely spaced objects.
  • On-road scenario: RAMP-CNN improves AP by 10% and AR by 6% over RODNet-CDC on the challenging on-road test set.The set includes strong environmental reflections, many cars, and crowded pedestrians across 4398 frames.
  • Nighttime scenario: RAMP-CNN performs similarly at nighttime and daytime, although nighttime performance is not numerically evaluated because ground-truth labeling is difficult.The result supports radar robustness to sunlight changes under nighttime conditions.
  • Impact of adding temporal information: For moving objects, RAMP-CNN reaches about 80% AP and 84% AR, outperforming baselines by about 15% AP and 13% AR; static-object performance is around 67% AP and 70% AR.The comparison supports the usefulness of temporal information for moving-object recognition.
  • Ablation study: Data augmentation improves RAMP-CNN by 12% AP and 8% AR in curbside tests and by 4% AP and 2% AR in on-road tests.The ablation compares the complete model with a version without data augmentation.
  • Ablation study: The proposed training loss yields around 4% AP and 3% AR improvements in parking lot, curbside, and on-road scenarios.The loss is intended to encourage learning of more Doppler-related features.
  • Complexity analysis: CNN time complexity is measured from the FLOPs required by convolutional layers, while frame-level prediction time is also measured across models.Overall CNN complexity is treated as the sum of the complexities of all convolutional layers.

Space complexity:

RAMP-CNN substantially reduces complexity relative to a 4D model, while improving recognition over prior methods at a higher cost than RODNet-CDC. The paper also reports nighttime operation and identifies remaining performance and data-collection boundaries.

  • Space complexity comprises model parameters and the memory occupied by feature-map outputs across layers.
  • RAMP-CNN uses almost 100 times fewer FLOPs, around half the parameters, and 35 times less feature-map memory than 4D-CDC.
  • RAMP-CNN’s performance improvement over RODNet-CDC comes with around 3 times higher time and space complexity.
  • RAMP-CNN improves recognition across parking-lot, curbside, and on-road scenarios, but hard cases may remain inadequate for autonomous driving.The authors suggest higher spatial resolution or radar platforms with more antennas as future improvement paths.
  • The model is verified at nighttime, while application to fog, rain, or snow remains future work because local data collection in those conditions is difficult.
  • RAMP-CNN exploits temporal chirp information and changing spatial information across frames, particularly improving performance for moving objects.

A. Translating in Range

The range-translation augmentation models how moving a target in range changes FMCW radar signals and range spectra. It uses range-bin shifts together with range-dependent amplitude and phase adjustments.

  • A. Translating in Range: FMCW de-chirped signals are sampled by quadrature ADC, replacing continuous time t with sample time i/fs.
  • A. Translating in Range: The Range FFT converts digitized de-chirped signals into a range profile indexed by range bin mr, with range mapped to bin indices.
  • A. Translating in Range: A target translated by Δr changes its range from r1 to r1 + Δr while retaining the range-translation setup defined for radar augmentation.
  • A. Translating in Range: The translated range spectrum is obtained by shifting the original spectrum by Δmr1 cells and scaling amplitude and phase according to the range change.

B. Translating in Angle

The angle-translation augmentation models a target shifted in azimuth while preserving range. It transforms the range-angle spectrum through angular-bin shifts and angle-dependent antenna-gain effects.

  • B. Translating in Angle: The range-angle spectrum is computed by applying Range FFT and Angle FFT to the digitized de-chirped signal.
  • B. Translating in Angle: The resulting spectrum uses range-bin mr and angle-bin mθ, with mappings between physical range or azimuth and their corresponding bin indices.
  • B. Translating in Angle: A target translated by Δθ keeps range r1 and changes azimuth from θ1 to θ1 + Δθ.
  • B. Translating in Angle: Angle translation shifts the original range-angle spectrum by Δmθ1 angular cells, with the signal also incorporating an antenna-gain ratio for the changed angle.
Loading 2011.08981v2…