Source-linked AI summary
mmIR: Frequency-Space Inverse Rendering for 3D Millimeter-Wave Radar ADC Synthesis
Adnan Armouti, Yixuan Gao, Rajalakshmi Nandakumar
TL;DR
High-resolution 3D raw FMCW ADC data is scarce because commodity sensors and public datasets provide limited angular or signal information. mmIR fits a differentiable physics-based FMCW model to real captures using LiDAR-derived meshes, then re-renders dense virtual apertures. It achieves higher RA correlation than Sionna-RT, transfers across radar configurations without retraining, and produces LiDAR-validated single-frame 3D occupancy.
Problem
Commodity radar arrays and public datasets limit angular resolution and access to high-resolution 3D raw FMCW ADC data.
Method
mmIR performs LiDAR-assisted inverse rendering by fitting materials, normals, and antenna patterns through a differentiable phase-coherent MIMO FMCW forward model, then re-rendering dense virtual apertures.
Results
0.914 mean Pearson RA correlation versus 0.307 for Sionna-RT is achieved across 13 ColoRadar scenes, with 0.554 correlation after cross-sensor transfer without retraining.
Takeaways & Limitations
Fitted scenes can synthesize high-resolution 3D radar data, including LiDAR-validated single-frame 3D occupancy from dense virtual arrays.
Takeaways & Limitations
The method is constrained by mesh quality and LiDAR-radar alignment, assumes static scenes, and is evaluated only on clear-weather ColoRadar captures.
Abstract
from arXiv · showhide
High-resolution 3D radar data is scarce. Commodity mmWave sensors use small antenna arrays that limit angular resolution to several degrees, and existing datasets provide only 2D range-azimuth maps or sparse point clouds rather than raw analog-to-digital converter (ADC) signals. Hardware scaling is expensive, synthetic-aperture scanning is impractical at fleet scale, and learned synthesis methods are bottlenecked by the very data shortage they aim to address. We present mmIR, an open-source differentiable frequency-modulated continuous-wave (FMCW) radar inverse renderer that fits a physics-based forward model to real captures and re-renders from dense virtual apertures to synthesize high-resolution 3D radar data. Because radar resolution is too coarse to recover geometry directly, mmIR performs LiDAR-assisted inverse rendering: using LiDAR-derived meshes as a geometric scaffold, mmIR optimizes per-vertex International Telecommunication Union (ITU) physics materials, vertex normals, and antenna beam patterns through end-to-end automatic differentiation of a phase-coherent multiple-input multiple-output (MIMO) forward model with multi-bounce propagation, polarization, and free-space diffraction. On seven outdoor and six indoor ColoRadar scenes, mmIR achieves 0.914 mean Pearson correlation on range-azimuth maps versus 0.307 for Sionna-RT. Scenes trained on a cascaded imaging radar transfer to a co-located single-chip radar without re-training (0.554 correlation), and dense virtual arrays (100x100 elements) produce single-frame 3D occupancy validated against LiDAR. Project page: https://mmwave-inverse-rendering.github.io/
1 Introduction
mmIR addresses the shortage of high-resolution 3D raw FMCW ADC data with LiDAR-assisted inverse rendering that fits a physics-based radar model to real captures and re-renders dense virtual apertures. It reports substantially higher RA correlation than Sionna-RT, cross-sensor transfer without retraining, and LiDAR-validated single-frame 3D occupancy.
- Motivation: Commodity radars and public datasets provide limited angular resolution and mostly post-processed 2D RA maps or sparse point clouds rather than raw ADC.These constraints contribute to a shortage of high-resolution, 3D, raw FMCW ADC data for radar perception.
- Motivation: Hardware scaling, synthetic-aperture scanning, and learned synthesis each remain impractical or constrained by radar data scarcity.Existing alternatives do not simultaneously provide affordable, diverse, high-resolution 3D radar data at fleet scale.
- Method: mmIR fits per-vertex ITU materials, vertex normals, and antenna beam patterns using end-to-end differentiation of a phase-coherent MIMO FMCW forward model on LiDAR-derived meshes.The model includes multi-bounce propagation, polarization-aware BSDFs, specular manifold sampling, and free-space diffraction; geometry and pose gradients are supported but fixed in the experiments.
- Results: 0.914 mean Pearson RA correlation is achieved by mmIR versus 0.307 for Sionna-RT on seven outdoor and six indoor ColoRadar scenes.The comparison is reported on rendered range-azimuth images.
- Results: 0.554 correlation is obtained when scenes trained on a cascaded imaging radar transfer to a co-located single-chip radar without retraining.The fitted scenes are also re-rendered from dense virtual arrays for single-frame 3D occupancy validated against LiDAR.
- Implications: mmIR is intended to support large-scale datasets for downstream detection, segmentation, and occupancy-prediction training without additional data collection.The paper demonstrates cross-sensor transfer and dense virtual-aperture rendering as part of this intended use.
2 Related Works
Related work spans differentiable rendering, physics-based FMCW ray tracing, and neural or Gaussian radar synthesis. mmIR distinguishes itself by rendering raw ADC with explicit scene physics, enabling joint calibration and arbitrary virtual-aperture re-rendering.
- Differentiable Rendering: Differentiable rendering recovers geometry, materials, and sensor parameters through gradient-based optimization, but has operated almost exclusively in optics.RF systems such as Sionna-RT propagate gradients through field coefficients and CIR computation but not path topology.
- FMCW Signal Modeling: Physics-based FMCW simulation commonly uses shooting-and-bouncing rays to model path geometry and attenuation for large scenes and MIMO arrays.Prior pipelines are forward-only, so scene and sensor parameters are fixed rather than calibrated against real captures.
- Neural and Gaussian Methods: Neural fields and Gaussian splatting synthesize radar across modalities including FMCW occupancy, Doppler, frequency-space responses, and range-azimuth rendering.These methods learn opaque, sensor-specific representations, whereas mmIR retains explicit scene physics.
- mmIR: mmIR renders raw FMCW ADC so measured and rendered data undergo identical FFT processing, separating scene physics from signal processing.Its explicit mesh uses per-vertex ITU materials and polarization-aware BSDFs for joint scene-sensor calibration.
3 Methods
mmIR models radar sensing as differentiable, phase-coherent FMCW forward rendering, then optimizes physically based scene parameters against real range–azimuth measurements. It uses LiDAR-derived geometry, multi-bounce propagation, and specialized sampling to render coherent ADC signals and synthesize dense-aperture radar data.
- 3.1 Forward Model: mmIR traces multi-bounce rays, evaluates physically based mmWave surface interactions, and coherently accumulates paths into raw FMCW ADC samples.The forward model derives from the bistatic radar equation and supports direct comparison with real captures.
- 3.1 Forward Model: The bistatic formulation replaces point-target radar cross-section with a spatially varying BSDF integrated over the receiver hemisphere with antenna gains, visibility, and geometric cosine factors.Rx-centric sampling follows from this solid-angle formulation.
- 3.1 Forward Model: For multi-bounce paths, an unbiased Monte Carlo estimator combines BSDF-over-PDF ratios, geometric coupling, antenna gains, visibility, and deterministic transmitter connections.Geometric coupling uses cosine factors and inverse-square distances between consecutive bounces.
- 3.1 Forward Model: Each traced path contributes a phase-delayed FMCW chirp whose delay is determined by total path length, and path contributions are summed coherently across Tx–Rx pairs.Shared path topology preserves the relative phase required for coherent MIMO FFT imaging.
- 3.2 Physically-Based mmWave BSDF: The BSDF uses ITU-based material parameters, polarization-aware Fresnel energy gating, and coherent–incoherent roughness mixtures with learnable KA/SPM and directional-lobe components.Material parameters are barycentrically interpolated from mesh vertices, while slab modeling captures multiple internal reflections.
- 3.3 Ray Generation: Specular Manifold Sampling solves for exact reflection points with Newton iteration and mesh reprojection, avoiding failures when specular points fall outside small LiDAR-derived triangles.The method is motivated by triangles smaller than the mmWave wavelength.
- 3.4 Differentiable Optimization: mmIR minimizes squared error between min–max normalized rendered and ground-truth range–azimuth magnitudes, then freezes fitted parameters for dense-aperture 3D rendering.Range–azimuth magnitude validates phase coherence through the TDM-MIMO FFT while avoiding unreliable absolute phase supervision.
4 Experiments
Across outdoor and indoor ColoRadar scenes, mmIR is evaluated for RA/ADC rendering, cross-sensor transfer, and dense-array 3D occupancy. It consistently outperforms Sionna-RT and demonstrates transfer and geometric reconstruction capabilities.
- 4.1 Training Evaluation: ADC Rendering and Range–Azimuth: 0.914 mean Pearson correlation across seven outdoor and six indoor scenes substantially exceeds Sionna-RT’s 0.307 overall mean.Outdoor and indoor means are 0.919 and 0.909 for mmIR, versus 0.324 and 0.288 for Sionna-RT.
- 4.1 Training Evaluation: ADC Rendering and Range–Azimuth: mmIR reproduces dominant scatterer locations and diffuse multipath, while Sionna-RT produces attenuated and spatially misaligned responses.The qualitative comparison covers both outdoor and indoor scenes; indoors, Sionna-RT over-responds in dense multipath.
- 4.1 Training Evaluation: ADC Rendering and Range–Azimuth: Three architectural differences support the performance gap: spatially varying per-vertex BSDF materials, joint normal and beam-pattern optimization, and deeper end-to-end gradient flow.These additions address mixed-material reflectance, noisy LiDAR tessellation, antenna-pattern errors, and frozen path topology in the baseline.
- 4.2 Radar Transfer Evaluation: 0.554 mean transfer correlation is achieved when scenes trained on the cascaded radar render a co-located single-chip radar without retraining, versus 0.114 for Sionna-RT.The trained scene parameters are frozen and rendered forward-only for the single-chip configuration.
- 4.3 Inference: 3D Occupancy via Dense Virtual Array: 100×100-element dense virtual arrays resolve true elevation for RAE-based single-frame 3D occupancy, which is evaluated against LiDAR using radar-guided filtering.The evaluation uses a 0.5 m KD-tree matching threshold and compares mmIR’s radar-physics-gated output rather than the input mesh.
- 4.3 Inference: 3D Occupancy via Dense Virtual Array: Occupancy quality varies with radar visibility: S1-F185 reaches Precision=0.997 and R-CD=0.067, while open scenes can lose recall when specular energy avoids receivers.Ablations report −3.7% correlation without antenna optimization and −1.0% without diffraction.
5 Conclusion
mmIR fits differentiable radar inverse-rendering scenes and re-renders dense virtual-aperture ADC for high-resolution 3D sensing. It achieves substantially higher rendered RA-image correlation than Sionna-RT, transfers across radar configurations, and remains bounded by mesh, alignment, and physical-model limitations.
- 0.914 mean Pearson correlation versus 0.307 for Sionna-RT on rendered RA images across seven outdoor and six indoor scenes.
- mmIR transfers trained scenes to a different radar configuration and supports dense virtual-aperture ADC for 3D occupancy detection and scene reconstruction validated against LiDAR.
- The rendered ADC is bottlenecked by mesh quality, while inverse rendering is sensitive to LiDAR-radar alignment.
- The evaluation uses clear-weather ColoRadar captures, and material validation against ground-truth dielectric properties remains future work.