PaperlayerView exact revision on arXiv

Source-linked AI summary

RadarGen: Automotive Radar Point Cloud Generation from Cameras

Tomer Borreda, Fangqiang Ding, Sanja Fidler, Shengyu Huang, Or Litany

arXiv:2512.17897v2cs.CVcs.AIcs.LGcs.RO

TL;DR

Radar generation from images remains challenging because radar measurements are stochastic and carry sparse, sensor-specific structure that is difficult to infer visually. RadarGen uses conditional diffusion over radar BEV representations with visual cues, and reports outperforming its proposed baseline on geometric, attribute, and distribution metrics.

  • Problem

    Radar’s stochastic measurements and distinctive sparse, RCS, and Doppler characteristics remain underrepresented in generative simulators, while processed point clouds are the practical training data.

    Source: §1 · p. 2

  • Method

    RadarGen learns a conditional distribution over radar observations from camera imagery, using BEV radar representations and BEV-aligned visual conditioning cues.

    Source: §1 · pp. 2–3, §4.2 · p. 9

  • Results

    RadarGen outperforms the proposed baseline on a comprehensive suite of geometric, attribute, and distribution metrics.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Generated radar can be interpreted by detectors trained on real data, and the paper demonstrates scene editing as an application.

    Source: §1 · p. 3

    Performance depends on upstream foundation models and is limited in challenging conditions such as low-light scenes, strong reflections, and camera occlusion.

    Source: §6 · p. 15

Abstract

from arXiv · show

We present RadarGen, a diffusion model for synthesizing realistic automotive radar point clouds from multi-view camera imagery. RadarGen adapts efficient image-latent diffusion to the radar domain by representing radar measurements in bird's-eye-view form that encodes spatial structure together with radar cross section (RCS) and Doppler attributes. A lightweight recovery step reconstructs point clouds from the generated maps. To better align generation with the visual scene, RadarGen incorporates BEV-aligned depth, semantic, and motion cues extracted from pretrained foundation models, which guide the stochastic generation process toward physically plausible radar patterns. Conditioning on images makes the approach broadly compatible, in principle, with existing visual datasets and simulation frameworks, offering a scalable direction for multimodal generative simulation. Evaluations on large-scale driving data show that RadarGen captures characteristic radar measurement distributions and reduces the gap to perception models trained on real data, marking a step toward unified generative simulation across sensing modalities.

1 Introduction

Generative driving simulators have advanced for visual data and LiDAR, but radar remains comparatively underexplored despite its distinct sparse, RCS, and Doppler measurements. RadarGen synthesizes probabilistic radar point clouds from multi-view images using BEV representations and visual priors, and introduces metrics for evaluating them.

Motivation

  • Radar remains an open frontier for generative simulation, which has largely focused on visual data and only recently extended to LiDAR.Current neural simulators therefore do not reproduce radar’s distinctive sparsity, RCS, and Doppler characteristics.Source: §1 · p. 2

Challenges

  • Radar’s stochastic measurements are difficult to infer from vision, while commonly available processed point clouds remain the practical training representation.Raw radar recording is memory-intensive, and proprietary processing is closed and lossy.Source: §1 · p. 2

Approach

  • RadarGen learns a distribution of radar observations conditioned on camera imagery, producing diverse, semantically consistent measurements rather than a deterministic prediction.Image conditioning is intended to leverage existing visual data and simulators.Source: §1 · p. 2

Evaluation and applications

  • The paper introduces metrics for geometric fidelity, radar-attribute fidelity, and distribution similarity, and reports that generated radar can be interpreted by detectors trained on real data.It also identifies scene editing as an application.Source: §1 · p. 3

Approach

  • RadarGen represents location, RCS, and Doppler in BEV maps and conditions generation on BEV-aligned depth, semantic, and motion priors from foundation models.It builds on SANA [83] to handle the multi-channel representation and conditioning cues efficiently.Source: §1 · p. 3

2 Related Work

Prior radar simulators trade physical detail against scalability, while data-driven and generative approaches differ in their conditioning, scene coverage, and suitability for sparse radar point clouds.

2.1 Physics-based radar simulation

  • Physics-based radar simulation ranges from computationally intensive Maxwell-equation solvers to more practical ray-tracing and graphics-based proxies, but relies on manually created assets.These assets demand substantial engineering effort to cover long-tail conditions.Source: §2.1 · p. 4

2.2 Data-driven radar simulation

  • Data-driven radar methods include scene-specific NeRF and Gaussian-splatting reconstructions, conditional GANs and VAEs, and visual-conditioned radar generators, but prior approaches have limits in transfer, scalability, or stochasticity.Some driving methods target raw radar data unavailable at large scale; point-cloud mappings from LiDAR/RGB are deterministic, while the figure notes radar is harder to interpret in camera view than BEV.Source: §2.2 · p. 4, Figure 2 · p. 5

2.3 Generative point cloud models

  • General point-cloud generators include unconditional, text-conditioned, image-conditioned object-centric, and scene-level LiDAR methods, but dense LiDAR range-image approaches do not transfer to sparse, non-uniform radar.The related work describes existing scene-level LiDAR generation as generally lacking image-based conditional design.Source: §2.3 · p. 5

3 Preliminaries

Radar point clouds are sparse, non-grid-aligned sets with sensor-specific attributes, motivating BEV rather than range-image representations. Scene-level image-conditioned generation also calls for an efficient diffusion architecture.

3.1 Challenges in radar point cloud generation

  • Radar detections are sparse, unordered, and non-uniformly sampled, with RCS and Doppler attributes, so they cannot be effectively represented as dense range images.The paper adopts multi-image BEV as a scalable alternative suited to this structure.Source: §3.1 · pp. 5–6

3.2 Efficient diffusion models

  • Image-conditioned scene-level point-cloud synthesis requires an efficient diffusion architecture because multi-image BEV representations are high-dimensional.SANA uses 32x autoencoder compression and O(N) linear attention rather than standard LDMs’ 8x compression and O(N^2) self-attention.Source: §3.2 · p. 6

4 Method

RadarGen converts sparse radar detections into image-compatible BEV maps, conditions latent diffusion on scene-aligned visual cues, and recovers a sparse point cloud from generated maps.

4.1 Representing radar as images

  • Radar points are projected into BEV and encoded as density, RCS, and Doppler maps, with Gaussian smoothing for density and nearest-detection values for attributes.The maps are replicated across RGB channels and encoded independently as diffusion supervision targets.Source: §4.1 · p. 7

4.2 Conditional radar generation

  • Pretrained depth, segmentation, and optical-flow models provide visual cues that are projected into BEV to align with radar targets.Appearance and semantic maps retain image-derived color; the radial-velocity map derives motion cues from consecutive views.Source: §4.2 · p. 8
  • The diffusion model learns radar-map latents conditioned on spatially aligned BEV cues, generating diverse radar realizations through joint denoising of density, RCS, and Doppler maps.A shared-attention DiT processes the map latents together, with modality identifiers preserving channel-specific statistics.Source: §4.2 · p. 9

4.3 Recovering the sparse radar point cloud

  • A known Gaussian blur kernel makes recovery a nonnegative, L1-regularized deconvolution problem, optimized with IRL1 and FISTA before thresholding recovered coordinates.The recovered locations are used to retrieve RCS and Doppler values from their corresponding maps.Source: §4.3 · pp. 9–10

4.4 Implementation details

  • RadarGen uses a modified SANA DiT with its pretrained v1.1 autoencoder, trained for two days on eight L40 GPUs.The setup uses a 512 × 512 grid, filters points to ±50m, and drops each condition with 10% probability during training.Source: §4.4 · p. 10

5 Experiments

RadarGen broadly outperforms its baseline on radar-generation metrics and shows scene editing and conditioning ablations, while results also expose a gap to real-data perception performance.

5 Experiments

  • The evaluation uses 431 MAN TruckScenes clips for training and 49 for evaluation, with annotated frames used for evaluation and metrics covering geometry, attributes, and distribution.The framework reports metrics for both entire scenes and foreground points within annotated object boxes.Source: §5 · p. 11

5.1 Radar point cloud generation

  • RadarGen broadly outperforms the baseline, with a stated exception for CD Full in Entire Area; qualitative examples also show closer shape, distribution, and count agreement with ground truth.The qualitative comparison uses t and t + ∆t for RadarGen, while the baseline uses t alone.Source: Figure 5 · p. 11, §5.1 · p. 12
  • A detector scored NDS 0.30 on RadarGen point clouds versus 0.48 on real data, while it found nearly zero valid objects in baseline point clouds.Despite a 0.66 hit rate within true bounding boxes, overall detection remained below real-data performance.Source: §5.1 · p. 12

5.2 Scene editing

  • Editing input images supports radar point-cloud augmentation through object replacement, removal, and insertion, including removal of points newly occluded by a replacement truck.The detector successfully perceives the newly generated truck points in the replacement example.Source: Figure 6 · p. 12, §5.2 · p. 13, §5.1 · p. 13

5.3 Ablation study and analysis

  • Removing segmentation causes the largest reported conditioning degradation, while removing velocity or appearance primarily worsens Doppler MMD.The authors report that removing appearance also degrades motion-profile realism despite retaining segmentation.Source: §5.3 · p. 13
  • BEV conditioning gives better overall geometric fidelity than direct multi-view conditioning, which improves Entire Area radar-attribute MMD but requires over 3× runtime.The BEV-conditioned model trained for two days, versus nine days for the multi-view model.Source: §5.3 · pp. 13–14
  • Increasing density-map bandwidth reduces autoencoder reconstruction error but excessive smoothing harms point-cloud recovery, so the authors select σ = 2.Deconvolution yields the best point-cloud quality across tested bandwidths and both ground-truth and autoencoder-reconstructed maps.Source: §5.3 · p. 14, Figure 7 · p. 14

6 Conclusion and Future Work

RadarGen generates radar point clouds from multi-view cameras using foundation-model BEV conditioning and deconvolution recovery. The authors report broad baseline gains and identify challenging visual conditions and unseen-region hallucinations as limitations.

6 Conclusion and Future Work

  • RadarGen combines foundation-model BEV conditioning, dense radar-map generation, and deconvolution recovery, and broadly outperforms its proposed baseline across geometric, attribute, and distribution metrics.The paper also demonstrates image-editing-based augmentation and reports promising downstream detector results.Source: §6 · p. 14
  • Future work will explore video input, text-based conditioning, and training across multiple datasets and radar configurations.Source: §6 · p. 15
  • Performance is tied to upstream foundation-model capability, and points generated in camera-invisible regions can hallucinate; explicit radar mechanisms are not modeled.The authors state that explicit modeling requires large-scale raw data and detailed radar specifications and internal mechanisms, often undisclosed.Source: §6 · p. 15

Supplementary Materials for RadarGen: Automotive Radar Point Cloud

The supplementary material contains metric formulations, implementation details, and additional experimental results.

Supplementary Materials for RadarGen: Automotive Radar Point Cloud

A Evaluation Metrics

The evaluation compares synthetic and ground-truth point clouds over the full ±50 m range and within selected foreground boxes. Its metrics cover geometric fidelity, attribute accuracy, point-count agreement, and distribution similarity.

A Evaluation Metrics

  • Evaluation separates the full point cloud within ±50 m from annotated boxes containing visible Cars, Trucks, or Trailers.Foreground boxes require more than 60% camera visibility.Source: A Evaluation Metrics · p. 21

A.1 Chamfer Distance (CD)

  • Chamfer Distance measures bidirectional point-set distance, using locations for CD-Loc and normalized locations plus RCS and Doppler for CD-Full.CD is reported over all point-cloud pairs for the Entire Area and valid boxes for the Foreground.Source: §A.1 · p. 22

A.2 Point Cloud IoU@1m

  • Point Cloud IoU@1m compares point-cloud overlap using a 1 m matching threshold and is averaged over Entire Area pairs.The metric adopts the definition from [64].Source: §A.2 · p. 22

A.3 Density Similarity

  • Density Similarity compares the point counts in each box, assigning 1 when both clouds are empty and otherwise using the smaller-to-larger count ratio.It is reported for Foreground boxes, averaged across boxes.Source: §A.3 · pp. 22–23

A.4 Bounding Box Hit Rate

  • Hit Rate is the fraction of ground-truth-positive boxes that also contain at least one synthetic point.It is averaged across Foreground boxes.Source: §A.4 · p. 23

A.5 Distance-Attribute (DA)

  • Distance-Attribute jointly scores locations, RCS, and Doppler, counting a match only within thresholds and using global assignment to avoid greedy-order ambiguity.Thresholds are 1 m, 8 dBsm, and 2.5 m/s for location, RCS, and Doppler, respectively.Source: §A.5 · p. 23

A.6 Maximum Mean Discrepancy (MMD)

  • Maximum Mean Discrepancy measures distribution similarity separately for location, RCS, and Doppler using a five-scale Gaussian RBF kernel.Foreground points are aggregated by vehicle class after centering and rotation alignment; Entire Area clouds are compared pairwise.Source: §A.6 · pp. 23–24

B.1 Additional details on BEV representation

  • BEV conditioning maps are normalized to image-range values, while filtering removes points on object edges, in sky regions, or above 5 m.The authors describe BEV conditioning as interpretable because it makes the information available to the model explicit.Source: §B.1 · p. 24

B.2 Additional details on training

These supplementary sections detail RadarGen’s training and inference setup, direct multi-view conditioning, point-cloud recovery, qualitative results, and a low-light limitation.

B.2 Additional details on training

  • RadarGen training uses SANA’s [83] pretrained 600M-parameter weights and runs for 65k steps over two days on eight L40 GPUs.Training uses a per-GPU batch size of 16 and a learning rate of 10^-4; required BEV images are precomputed.Source: §B.2 · p. 25

B.3 Additional details on inference

  • Single-timestep inference takes approximately 10.5 seconds on one L40 GPU, including 9 seconds for BEV conditioning, 1 second for diffusion, and 0.5 seconds for recovery.Inference uses 20 sampling steps, a null prompt, and no guidance scale.Source: §B.3 · p. 25

B.4 Additional details on the baseline model

  • The RGB2Point baseline predicts 1,024 five-channel points and uses unnormalized Chamfer Distance after its normalized “CD Full” objective failed to converge.Its architecture is adapted to predict RCS and Doppler attributes.Source: §B.4 · p. 25

B.5 Additional details on VoxelNeXt

  • VoxelNeXt is adapted for radar input and trained for 77k steps, with flipping, rotation, translation, and scaling augmentations over a [−50m, 50m] point-cloud range.Training takes nine hours on eight L40 GPUs with a per-GPU batch size of 32.Source: §B.5 · pp. 25–26

B.6 Additional details on direct multi-view conditioning

  • Direct multi-view conditioning uses shared self-attention across 11 radar and camera latent tensors, with image latents paired with Plücker coordinates and modality indicators.The 11 tensors comprise three radar maps, four camera images at time t, and four at time t + 1.Source: §B.6 · p. 26

C.1 Additional qualitative results

  • Qualitative examples cover highway, rural, and urban scenes, show variation across random seeds, and include supplementary scene-generation videos..Source: §C.1 · p. 26

C.2 Additional radar PCL recovery results

  • Deconvolution recovery covers relevant regions while providing sufficient point density, unlike Random sampling’s gaps and Peak sampling’s sparse output.Random sampling misses the moving car in one example and creates excessive density elsewhere; Peak selects local maxima in a 3 × 3 window.Source: §C.2 · p. 26

C.3 Additional information on limitations

  • In a low-light night scene, pretrained foundation models struggle to recognize vehicles and estimate velocities; RadarGen was not trained on such scenarios.The section presents a qualitative comparison with ground-truth radar.Source: §C.3 · p. 26

C.4 Additional detection results

Detection results compare ground-truth and generated samples, and mixed real-synthetic training matches the real-only NDS on real-data evaluation.

C.4 Additional detection results

  • Table 1 compares a trained detector on ground-truth data with samples from RadarGen and the baseline across Car, Truck, and Trailer classes..Source: §C.4 · p. 27, Table 1 · p. 27
  • NDS is 0.48 for VoxelNeXt trained on real and synthetic data, equal to real-only training when evaluated on real data.Synthetic samples are included with a 50% probability during training.Source: §C.4 · p. 27

C.5 nuScenes results

On nuScenes, RadarGen broadly outperforms the baseline in quantitative metrics and produces point clouds that visually resemble ground truth in shape, distribution, and count.

C.5 nuScenes results

  • nuScenes has five radar sensors and averages a few hundred points per cloud, compared with a few thousand in MAN TruckScenes..Source: §C.5 · p. 27
  • The nuScenes RGB2Point baseline predicts a fixed set of 208 points, approximately the dataset’s average point-cloud count..Source: §C.5 · p. 27
  • RadarGen broadly outperforms the baseline on nuScenes geometric, radar-attribute, and distribution-similarity metrics.Table 2 reports geometric fidelity (CD, IoU, Density Similarity, Hit Rate), radar attribute fidelity (DA Recall, Precision, F1), and distribution similarity (MMD).Source: §C.5 · p. 27, Table 2 · p. 28
  • RadarGen’s nuScenes point clouds closely match ground truth in shape, distribution, and count, with a significant advantage over the baseline in four visual examples..Source: §C.5 · p. 27

C.6 LiDAR and range image representation

Radar range images are sparse, and passing them through pretrained autoencoders substantially degrades point-cloud reconstruction. Their depth-valued pixels also prevent directly applying the Gaussian smoothing used for BEV point maps.

C.6 LiDAR and range image representation

  • LiDAR generation commonly uses range images or rays tied to range-image pixels, from which point clouds can be reconstructed using the depth-like representation.Radar can also be viewed from the sensor perspective, but its sparsity makes accurate range-image reconstruction harder.Source: §C.6 · p. 28
  • Radar range images are significantly sparser than LiDAR range maps, making accurate reconstruction harder; one illustrated example’s CD rises from 0.21m to 2.97m after SANA’s AE.The figure also compares camera and bird’s-eye views and shows the grayscale radar range image used for the AE.Source: §C.6 · p. 28, Figure 8 · p. 36
  • 0.31m to 3.72m CD after SANA’s AE, and 0.28m to 4.91m CD after LiDAR Diffusion’s AE, show degraded radar range-image reconstruction.For the SANA AE, the MSE between the AE input and output is 0.020; values are averaged over the MAN TruckScenes mini train set.Source: §C.6 · p. 28
  • Applying the BEV maps’ Gaussian kernel to range-image pixels would discard depth information and corrupt contributions from nearby points.The range-image pixels encode point depth, unlike the BEV occupancy representation.Source: §C.6 · p. 28
  • Changing the diffusion seed yields multiple point-cloud sets for one scene, while deconvolution recovery maintains coverage and density where needed.The recovery comparison reports inconsistent density for random sampling and low density for peak recovery; the BEV conditioning figure presents appearance, semantic, and relative radial-velocity maps.Source: Figure 3 · p. 31, Figure 4 · p. 32, Figure 5 · p. 33
  • In the shown low-light night scene, foundation models struggle to recognize vehicles and estimate velocities, and RadarGen was not trained on such scenarios.Source: Figure 6 · p. 34
  • RadarGen shows higher geometric and attribute fidelity to ground truth than the baseline in NuScenes, highway, rural, and urban examples; the baseline uses only time t.RadarGen uses inputs t and t + ∆t; the NuScenes comparison is shown in Fig. 7, with highway and rural examples in Fig. 1 and urban examples in Fig. 2.Source: Figure 1 · p. 29, Figure 2 · p. 30, Figure 7 · p. 35

Related papers

Loading 2512.17897v2…