PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-View Radar Semantic Segmentation

Arthur Ouaknine, Alasdair Newson, Patrick Pérez, Florence Tupin, Julien Rebut

arXiv:2103.16214v2cs.CV

TL;DR

Radar scene understanding remains an emerging task, despite radar’s ability to provide information that complements other sensors. This paper proposes lightweight multi-view radar segmentation architectures and associated losses; its methods outperform competing architectures, while cyclists and pedestrians remain difficult to distinguish.

  • Problem

    Radar scene understanding is still in its infancy, although radar provides information that can compensate for weaknesses of other sensors.

    Source: §1 · p. 1

  • Method

    The paper introduces lightweight architectures and loss terms for segmenting radar views while preserving coherence between multi-view predictions.

    Source: §1 · p. 2

  • Results

    The proposed methods significantly outperform competing architectures specialized either in natural-image semantic segmentation or radar scene understanding.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors report that both information from the RAD tensor and its temporal evolution are important for localization, delineation, and relative-velocity estimation.

    Source: §5 · p. 8

    Cyclists and pedestrians remain difficult to distinguish.

    Source: §5 · p. 8

Abstract

from arXiv · show

Understanding the scene around the ego-vehicle is key to assisted and autonomous driving. Nowadays, this is mostly conducted using cameras and laser scanners, despite their reduced performances in adverse weather conditions. Automotive radars are low-cost active sensors that measure properties of surrounding objects, including their relative speed, and have the key advantage of not being impacted by rain, snow or fog. However, they are seldom used for scene understanding due to the size and complexity of radar raw data and the lack of annotated datasets. Fortunately, recent open-sourced datasets have opened up research on classification, object detection and semantic segmentation with raw radar signals using end-to-end trainable models. In this work, we propose several novel architectures, and their associated losses, which analyse multiple "views" of the range-angle-Doppler radar tensor to segment it semantically. Experiments conducted on the recent CARRADA dataset demonstrate that our best model outperforms alternative models, derived either from the semantic segmentation of natural images or from radar scene understanding, while requiring significantly fewer parameters. Both our code and trained models are available at https://github.com/valeoai/MVRSS.

1. Introduction

Radar offers relative-speed information and can complement other sensors, but radar scene understanding remains less mature than camera and lidar applications. The paper proposes multi-view radar semantic segmentation to address the challenges of large, noisy radar data.

1. Introduction

  • Radar scene understanding is still in its infancy, although radar can provide information that compensates for weaknesses of other sensors.The range-angle-Doppler representation contains object signatures with enough detail to distinguish them.Source: §1 · p. 1
  • Semantic segmentation suits radar object signatures because their sizes vary greatly and they may be mixed by the sensor’s resolution.Source: §1 · p. 2
  • The proposed approach segments range-angle and range-Doppler views, using lightweight architectures and loss terms that preserve coherence between their predictions.Together, the two views suffice to deduce object localization and relative speed.Source: Figure 1 · p. 1, §1 · p. 2
  • Experiments report that the best model outperforms the other methods considered for radar semantic segmentation in quantitative and qualitative evaluations.Source: §1 · p. 2

2. Background

Automotive radar processing produces a complex range-angle-Doppler tensor, which can be aggregated into two-dimensional views. Prior work explores point clouds and individual or multiple views, but the paper identifies multi-view radar semantic segmentation as an unaddressed task.

2.1. Automotive radar sensing

  • FMCW radar processing applies Range-, Doppler-, and Angle-FFTs to form a complex range-angle-Doppler tensor whose axes discretize the corresponding physical measurements.Conventional FMCW radars usually do not provide this tensor because estimating it is computationally intensive.Source: §2.1 · p. 2
  • Averaging RAD intensities along one dimension yields RA, RD, and AD views; this representation reduces data size by a factor of 50.The view transformation aims to compress the data and reduce noise.Source: §2.1 · p. 2

2.2. Related work

  • Prior point-cloud and single-view approaches have information limitations, while earlier multi-view radar work addressed detection or localization rather than semantic segmentation.The authors state that, to their knowledge, no previous work addressed multi-view radar semantic segmentation.Source: §2.2 · pp. 2–3

3. Methods and architectures

The section reviews image- and radar-based alternatives, then presents multi-view architectures and losses for jointly segmenting radar views.

3.1. Image-based competing methods

  • FCN, U-Net, and DeepLabv3+ are image-segmentation alternatives; DeepLabv3+ uses atrous convolutions and ASPP to learn multi-scale features [27] [40] [10].FCN-8s had the best performance when FCN was used for radar segmentation [34].Source: §3.1 · p. 3

3.2. Radar-based competing methods

  • RSS-Net targets radar semantic segmentation, while RAMP-CNN aggregates radar data into views for range-angle object detection [22] [16].RSS-Net uses an encoder, ASPP, and up-sampling decoder; RAMP-CNN processes views with separate encoder-decoders and 3D convolutional layers.Source: §3.2 · p. 3

3.3. Proposed multi-view methods

  • MV-Net stacks temporal RD and RA inputs, fuses encoder features in a shared latent space, and decodes both views; AD is inferred rather than segmented.Its shared representation is processed by separate decoders that predict RD and RA segmentation maps.Source: §3.3 · pp. 3–4
  • MVA-Net adds ASPP modules; variant (b) also encodes AD features and supplies their multi-scale features to both RD and RA decoders.MVA-Net(a) processes RD and RA views; MVA-Net(b) adds the AD encoding branch.Source: §3.3 · p. 4
  • TMVA-Net extends MVA-Net with three view encoders and ASPP modules, using 3D convolutions in each encoder’s first block to learn spatiotemporal features with limited parameter growth.The architecture feeds the shared latent space and both decoders with features from the encoding backbones and dedicated ASPP modules.Source: §3.3 · pp. 4–5

3.4. Losses

  • Weighted cross-entropy counters class imbalance with inverse-frequency class weights, while Soft Dice addresses small object signatures.The paper notes that background and speckle noise dominate object signatures in radar segmentation.Source: §3.4 · p. 5
  • The coherence loss compares each view’s maximum class probabilities at each range bin, encouraging matching classes at matching distances.It is defined as the mean squared error between the two maximum range probability vectors.Source: §3.4 · p. 5
  • The final objective combines RD and RA weighted cross-entropy and Soft Dice losses with the coherence loss; its weighting factors are set empirically.The combination is intended to bring together the distinct strengths of the three loss terms.Source: §3.4 · pp. 5–6

4. Experiments

Experiments on CARRADA compare multi-view radar segmentation methods with image- and radar-based alternatives, using both quantitative and qualitative evaluation. TMVA-Net achieves the best reported scores on both views, while ablations assess architecture and loss choices.

4.1. Datasets and evaluation metrics

  • CARRADA provides synchronized radar and camera recordings across 30 sequences, with dense annotations for four classes in both RD and RA views.The additional in-house urban scenes lack radar annotations, so they support qualitative but not quantitative evaluation.Source: §4.1 · p. 6
  • The experiments report mIoU and mDice, averaging per-class IoU and Dice scores, respectively.The evaluation uses the authors’ CARRADA train, validation, and test splits.Source: §4.1 · p. 6

4.2. Implementation of competing methods

  • Competing image-segmentation architectures were used as-is, while RSS-Net and RAMP-CNN were modified for the radar inputs and tasks.RAMP-CNN’s fusion and output layers were adapted for RD segmentation.Source: §4.2 · p. 6

4.3. Training and results

  • TMVA-Net achieves the best mDice and mIoU scores on both RD and RA segmentation, and offers the best performance–parameter trade-off.It is also the only compared method besides MV-Net that segments both views with one model; the other methods require two.Source: Figure 3 · p. 6, §4.3 · p. 7, Table 1 · p. 7
  • TMVA-Net performs best across both views and metrics among the proposed architectures, with spatio-temporal features helping distinguish objects from speckle noise.The ablations report gains from ASPP modules and an additional AD-view encoder before the TMVA-Net comparison.Source: Table 2 · p. 7, §4.3 · p. 7
  • Adding CoL to wCE+SDice improves RA segmentation but slightly reduces RD performance, while qualitative examples show better cross-view object coherence.In the illustrated scene, TMVA-Net with CoL localizes and classifies both objects across the RD and RA views.Source: §4.3 · p. 8, Table 3 · p. 8, Figure 4 · p. 8

5. Conclusions and perspectives

The paper presents lightweight multi-view radar segmentation methods that localize and delineate objects while estimating relative velocity, and reports that they outperform competing architectures. Cyclists and pedestrians remain difficult to distinguish, and the authors identify them as a focus for future work.

5. Conclusions and perspectives

  • The proposed methods significantly outperform competing natural-image segmentation and radar scene-understanding architectures.Preliminary experiments also qualitatively indicate better generalization to new complex urban scenes.Source: §5 · p. 8
  • RAD tensor information and its temporal evolution are both important for radar scene segmentation tasks.The methods localize and delineate objects while simultaneously determining their relative velocity.Source: §5 · p. 8
  • The authors hope the work encourages larger dataset releases and exploration of new radar segmentation architectures.Source: §5 · p. 8
  • Cyclists and pedestrians remain difficult to distinguish and are a focus for future work.The authors also suggest exploring radar properties for tensor aggregation and class-specific data augmentation.Source: §5 · p. 8

A. RAD tensor visualisation

The RAD tensor is visualized as groups of two-dimensional slices, each corresponding to a discretized bin along the third axis. The slices show redundant signal information and substantial noise.

A. RAD tensor visualisation

B. Detailed multi-view architectures

The architecture details for MV-Net and TMVA-Net specify the operations used in each layer and their parameters. The notation covers operation dimensionality, and the ASPP and stride parameters receive additional clarifications.

B. Detailed multi-view architectures

C. Coherence loss

The coherence loss is intended to keep predictions consistent across different views of the same scene. Figure 6 illustrates the procedure used to construct it.

C. Coherence loss

D. Pre-processing and training procedures

The training procedures use model-specific learning-rate schedules and empirically chosen loss weights. Competing models also differ from the proposed models in input resizing and normalization.

D. Pre-processing and training procedures

  • Learning rates decay exponentially by γ = 0.9, with an epoch step specific to each model.The training parameters are detailed in Table 6.Source: D. Pre-processing and training… · p. 11
  • Competing methods use Cross Entropy, except RSS-Net, while the proposed methods combine weighted Cross Entropy, Soft Dice, and coherence losses.The proposed loss weights are λwCE = 1, λSDice = 10, and λCoL = 5, set empirically.Source: D. Pre-processing and training… · p. 11
  • Competing architectures resize the 256 × 64 range-Doppler input to 256 × 256, whereas the proposed architectures adapt down-sampling layers to avoid resizing.The range-angle view is 256 × 256 and needs no resizing; all methods use vertical and horizontal flips for augmentation.Source: D. Pre-processing and training… · p. 11
  • Competing methods normalize each view using local batch statistics, while the proposed methods use global statistics from the CARRADA dataset.Source: D. Pre-processing and training… · p. 11

E. Quantitative results

TMVA-Net provides the best performance–parameter-count trade-off for both range-Doppler and range-angle semantic segmentation tasks, assessed using mIoU.

E. Quantitative results

F. Qualitative results

Qualitative examples compare segmentation across RD and RA views on CARRADA test scenes and in-house urban scenes. TMVA-Net with coherence loss is singled out for jointly segmenting objects across both views.

F. Qualitative results

  • Only TMVA-Net with coherence loss localizes and classifies both objects across RD and RA in the first CARRADA example and segments both views well in the second.In scene 2, TMVA-Net without coherence loss predicts pedestrian and cyclist categories for pixels of the same object.Source: F. Qualitative results · p. 11, Figure 8 · p. 15
  • In-house examples use a different range resolution and have empty ground-truth masks; TMVA-Net variants localize objects across views, while only the coherence-loss variant distinguishes pedestrians in both.Without coherence loss, TMVA-Net detects more car signals; the authors interpret the examples as suggesting generalization from CARRADA.Source: F. Qualitative results · pp. 11–12, Figure 9 · p. 16
  • The RAD tensor is visualized as range-Doppler, range-angle, and angle-Doppler slices, each corresponding to a discretized bin of the third axis.The range-Doppler slices show redundant signal information and substantial noise within each group.Source: Figure 5 · p. 12
  • The coherence loss compares RD and RA segmentation maps by max-pooling each along its unshared axis, then taking the mean squared error between the resulting vectors.Source: Figure 6 · p. 12
  • Figure 7 plots mIoU against parameter count; other methods need two models to segment both views, doubling their actual parameters, unlike MV-Net and TMVA-Net.Source: Figure 7 · p. 12
  • MV-Net stacks q=2 past frames into 3 input channels, whereas TMVA-Net treats q=4 as a sequence with 5 channels; Table 6 counts one model for ours and two for other methods.The architecture tables describe models that take RD and RA inputs and predict segmentation maps for both views.Source: Table 4 · p. 13, Table 5 · p. 14, Table 6 · p. 15
Loading 2103.16214v2…