3D Point Splatting for mmWave Radar Novel View Synthesis

Adnan Armouti; Yixuan Gao; Rajalakshmi Nandakumar

Source-linked AI summary

3D Point Splatting for mmWave Radar Novel View Synthesis

Adnan Armouti, Yixuan Gao, Rajalakshmi Nandakumar

arXiv:2609.11894v1Sep 10, 2026cs.CVcs.GRcs.LGeess.SP

TL;DR

Radar novel view synthesis lacks a renderer that is simultaneously physically faithful, complex-valued, and tractable across viewpoints. 3DPS uses LiDAR-initialized oriented points, closed-form ITU-R material modeling, and complex PSF splatting to generate multiple radar products, achieving strong held-out RA correlation against optical-NVS baselines. Its scope is outdoor single-bounce rendering with LiDAR initialization.

Problem
Radar NVS needs a physically faithful, complex-valued, multi-viewpoint-tractable renderer, but prior methods do not provide all three properties simultaneously.
Method
3DPS renders complex radar profiles from LiDAR-initialized oriented 3D points with closed-form ITU-R P.2040 materials and PSF splatting, producing ADC, CRP, and RA through FFT pipelines.
Results
3DPS reaches 0.587 mean Pearson correlation on held-out novel-view |RA| across six outdoor ColoRadar scenes, between 1.7× and 5.2× the three optical-NVS baselines.
Takeaways & Limitations
The same optimized scene supports product-agnostic ADC, CRP, and RA rendering without retraining for each format.
Takeaways & Limitations
The method requires LiDAR initialization and assumes single-bounce paths, leaving LiDAR-free recovery and multipath-dominated indoor scenes outside scope.

Abstract

from arXiv · show

Solving novel view synthesis (NVS) for millimeter-wave (mmWave) radar requires a renderer that is physically faithful, complex-valued, and multi-viewpoint-tractable. No prior method achieves these three properties simultaneously. Differentiable Monte Carlo (MC) ray tracers implement the radar forward model directly with explicit material modeling and complex outputs, but do not scale to the multi-view optimization NVS demands. Optical-NVS ports of NeRF, hash grids, and 3D Gaussians train fast but discard phase and replace explicit material modeling with opaque learned features, restricting them to power-only range-azimuth (RA) magnitudes. We propose 3D Point Splatting (3DPS), the first differentiable point renderer for radar, derived directly from the standard solid-angle form of the radar equation. Each oriented 3D point carries an ITU-R P.2040 material model, evaluated in closed form, with the resulting complex phasor splatted into range bins through a precomputed point spread function (PSF). The complex-valued output makes the renderer product-agnostic. The same optimized scene yields analog-to-digital converter (ADC), complex range profile (CRP), and RA outputs through standard fast Fourier transform (FFT) pipelines without retraining for each format. On six outdoor ColoRadar scenes, 3DPS reaches 0.587 mean Pearson correlation on held-out RA images. This is between 1.7x and 5.2x the three optical-NVS baselines (RadarSplat, Radar Fields, DART). Training takes approximately 3 minutes per scene on a single RTX 4090.

1 Introduction

Radar NVS needs a renderer that is physically faithful, complex-valued, and tractable across multiple viewpoints, but prior renderer families each miss at least one property. 3DPS addresses this gap with a radar-native point renderer and reports strong sparse-view performance.

1 Introduction: Radar NVS is difficult because supervision is sparse, largely two-dimensional, dominated by quasi-mirror reflections, and available from only a few views.Commercial radar arrays commonly integrate out elevation, while sparse-view NVS provides limited training viewpoints.
1 Introduction: MC ray tracers model radar physics and complex outputs but are slow, whereas optical-NVS adaptations are fast but discard phase and use opaque learned features.These trade-offs prevent prior approaches from satisfying all three requirements simultaneously.
1 Introduction: 3DPS preserves complex range profiles so one optimized scene produces ADC, CRP, and RA outputs without retraining for each format.Standard FFT pipelines convert the shared scene representation into the downstream products.
1 Introduction: 3DPS reaches 0.587 mean Pearson correlation on held-out novel-view |RA| across six outdoor ColoRadar scenes, at approximately 3 minutes per scene on one RTX 4090.The reported result is between 1.7× and 5.2× the three optical-NVS baselines.
1 Introduction: 3DPS is the first differentiable radar point renderer designed to combine physical fidelity, complex outputs, and multi-viewpoint tractability.It uses oriented 3D points with ITU-R P.2040 materials and PSF splatting of complex phasors.
1 Introduction: The method assumes LiDAR initialization because sparse radar supervision cannot recover full 3D structure in the target sparse-view setting, and it models only single-bounce paths.LiDAR-free shape-from-radar is outside scope.

2 Related Works

Prior radar renderers fall into physics-based MC ray tracing and optical-NVS adaptations. These families respectively emphasize physical modeling or speed, but existing approaches do not provide the full combination of radar-native complex rendering and tractable NVS.

2 Related Works: Differentiable MC ray tracers extend differentiable rendering to electromagnetic propagation and explicitly model radar scattering and propagation.Examples include Sionna RT, Hofmann et al., InverTwin, and deterministic MIMO ray tracers.
2 Related Works: Optical-NVS adaptations apply NeRF, hash-grid, and Gaussian-splatting machinery to radar, typically rendering range-Doppler or power-only RA magnitudes.DART, Radar Fields, and RadarSplat represent this family.
2 Related Works: Other neural-field methods extend radar-like rendering to dynamic scenes or additional coherent sensors, but they do not provide complex radar outputs.Related applications include SAR, ISAR, LiDAR, and sonar rendering.

3 Method

3DPS derives a deterministic radar renderer from the radar equation using LiDAR-initialized oriented points, closed-form material scattering, and complex PSF splatting. The resulting range profiles support ADC, CRP, and RA products through FFT operations.

3.1 Radar forward model: The radar forward model uses geometric path lengths, antenna gains, bistatic RCS, and a single-bounce TX→scatterer→RX path to produce complex ADC samples.The receiver-centered hemisphere integral is discretized by oriented points.
3.1 Radar forward model: A Hann-windowed PSF of length L=15 replaces per-sample chirp synthesis with scatter-add operations into adjacent range bins, reducing per-pair cost from O(N·Ns) to O(N·L).The PSF is centered at each point’s fractional range bin.
3.1 Radar forward model: For each TX–RX pair, the renderer computes geometry and visibility, evaluates a closed-form BSDF, forms a complex amplitude, and splats it into range bins.Each point contributes once per pair without Monte Carlo sampling or path enumeration.
3.2 Rendering pipeline: Stacked complex range profiles are transformed with an azimuth DFT to produce RA, while inverse range FFT recovers ADC and the range FFT provides CRP.The same optimized scene yields all three products without retraining.
3.2 Rendering pipeline: Oriented points remain appropriate at radar resolution because each point uses a continuous 15-tap PSF over approximately 89 cm of range, avoiding optical-resolution concerns about point holes and aliasing.Radar range resolution is 5.93 cm with approximately 1.4° azimuth resolution.

F | vs. |RAGT

The implementation uses fused CUDA kernels and optimization procedures adapted to the radar renderer. These choices support efficient training while keeping positions anchored to the LiDAR scaffold.

F | vs. |RAGT: The fused forward, backward, and optimizer step renders 20,000 points across 192 TX–RX pairs in approximately 10 ms on an RTX 4090.The implementation fuses BSDF evaluation, scatter-splatting, and analytical backward computation.
F | vs. |RAGT: Position gradients are tracked but excluded from optimizer updates, keeping positions on the LiDAR scaffold while material and rotation parameters are optimized.The carrier phase is detached from position gradients because range-bin indexing makes the position dependence piecewise constant.

4 Experiments

Experiments evaluate 3DPS on six outdoor ColoRadar scenes against three optical-NVS baselines, using held-out |RA| performance and complex-output metrics. 3DPS leads the quantitative and qualitative comparisons but shows a measurable train-test gap tied to scene and sensing conditions.

Experimental setup: Training takes approximately 3 minutes per scene for joint optimization over eight viewpoints on one RTX 4090.The evaluation uses eight training views per scene and a held-out middle frame.
Quantitative results: 0.587 mean held-out Pearson correlation makes 3DPS 1.7× better than RadarSplat, the strongest optical-NVS baseline, on six ColoRadar scenes.The comparison uses held-out novel-view |RA| images averaged across scenes.
Magnitude on |RA|: 3DPS reaches 0.812 training and 0.587 held-out Pearson correlation on Cartesian |RA|, ranking first across mean and per-scene PSNR, SSIM, and RMSE.RadarSplat is next best at 0.365 train and 0.339 test correlation.
Limitations and scene variation: A 0.225 absolute train-test correlation gap reflects limited cross-view consistency, 5 Hz viewpoint spacing, and scene-dependent high-frequency structure.Performance ranges from 0.508 on S2 F300 to 0.657 on S2 F160.
Complex-output results: 3DPS also produces native complex outputs, reaching held-out CRP phase coherence of 0.371 and 0.434 across range and virtual-antenna axes, plus ADC coherence of 0.414.The optical-NVS baselines emit power-only RA and therefore lack corresponding complex-output measurements.
Qualitative comparison: 3DPS preserves dominant walls, vehicles, and support columns, whereas RadarSplat blurs, Radar Fields under-fits, and DART produces noisy low-amplitude renders.The visual ranking matches the quantitative order in held-out novel-view comparisons.

5 Conclusion

The paper concludes that 3DPS is a physically based point renderer that synthesizes complex radar products from novel poses through closed-form material evaluation and PSF splatting. Its scope remains bounded by outdoor single-bounce assumptions, LiDAR initialization, and the 2D limitations of planar radar supervision.

Conclusion: 3DPS synthesizes complex FMCW radar range profiles from novel sensor poses using oriented points, ITU-R P.2040 materials, closed-form BSDFs, and precomputed PSFs.The renderer removes Monte Carlo sampling and post-FFT supervision from the described pipeline.
Product-agnostic rendering: The same optimized scene produces ADC, CRP, and RA through standard FFT pipelines without retraining.The splat outputs complex range profiles that are stacked across virtual elements and transformed into RA; ADC is recovered by inverse range FFT.
Future work: Future work includes LiDAR-free radar NVS, phase-aware losses, dynamic scenes, indoor multipath, downstream perception, and broader radar geometries.The current conclusion states that single-bounce evaluation is sufficient outdoors but not for indoor multipath.
Radar signal model: The radar signal chain maps complex dechirped ADC samples into CRP by a windowed range FFT and into RA by a virtual-array azimuth transform.A scatterer’s delay determines its beat frequency and range-bin location, while inter-element phase progression determines azimuth.
Scope and limitations: A planar virtual array with zero elevation aperture cannot resolve height, so RA supervision integrates scatterers sharing a range-azimuth cell.Combined with sparse views and sparse returns, this makes full 3D recovery unrecoverable from RA alone.

A.5 Polar-to-Cartesian conversion for evaluation

Evaluation converts native polar RA images into Cartesian radar-frame grids so image-domain metrics and visual inspection use physical coordinates consistently. The conversion interpolates polar bins using the radar-frame angle and applies the same harness across methods.

Coordinate conversion: Native RA uses rows for range bins and columns for azimuth bins, while evaluation resamples it onto a Cartesian radar-frame grid.The polar representation is the benchmark form for Pearson correlation, PSNR, SSIM, and RMSE.
Coordinate conversion: The conversion computes θ = arctan(x/y), maps Cartesian coordinates to polar range and azimuth, and uses bilinear interpolation.The inverse azimuth mapping is arcsin-spaced to match the radar bin grid.
Evaluation protocol: All methods use a shared conversion harness on a 399×399 forward-cone grid retaining range bins 15–110.The retained range excludes near-origin TX–RX coupling and far-range DFT wraparound.
Material context: ITU-R P.2040 supplies closed-form surface-scattering parameters, including permittivity, roughness, correlation length, thickness, and a Kirchhoff/SPM blend.These parameters belong to the renderer’s material model rather than the polar-to-Cartesian evaluation transform.

B Forward model: surface-to-hemisphere derivation

The forward model converts the visible-surface ADC integral into a hemisphere integral, then discretizes it over oriented points and evaluates it efficiently with PSF splatting.

Surface-to-hemisphere change of variables: The surface integral is reparameterized over the receiver-centered upper hemisphere, where first-hit ray casting implicitly enforces receiver visibility.The solid-angle Jacobian relates visible surface area to direction-space integration.
Point discretization: The hemisphere integral is discretized with oriented points, and the Riemann-sum Jacobian cancels the range-and-angle factor exactly.This yields the point-based form used by 3DPS.
Visibility handling: Visibility is restored for the non-hemisphere-conformal LiDAR point set with a front-facing indicator and upfront ray-casting removal of self-occluded points.The point representation does not automatically satisfy the first-hit assumption of the continuous integral.
PSF splatting: A 15-tap Hann PSF approximates the range-bin scatter without measurable energy loss, while fractional-bin offsets preserve the unrounded carrier amplitude and phase.The Hann sidelobe envelope falls below −50 dB beyond 15 taps, and rounding changes only splat support.
Numerical validation: Direct point evaluation followed by range FFT and PSF splatting agree within 1.19×10−7 across 90,000 path samples.The result is essentially fp32 machine precision, establishing numerical equivalence on identical inputs.
Material model: The material model uses closed-form ITU-R P.2040 terms for multilayer Fresnel reflection, coherent specular scattering, and mixed directional-Lambertian diffuse scattering.Per-point parameters include permittivity, roughness, correlation length, slab thickness, and specular-lobe blending.

G CRP and ADC evaluation

3DPS evaluates CRP and ADC from the same complex range-profile output using standard FFT operations and matched trainer-domain conventions.

Evaluation pipeline: The same optimized scene produces CRP, ADC, and RA outputs without retraining by applying azimuth FFT and inverse range FFT to the complex range-profile output.This makes evaluation product-agnostic across the three radar representations.
Evaluation conventions: CRP and ADC are evaluated after matched Hann windows and zeroing of range bins [0:15] to reject TX–RX coupling.Ground truth and predictions use the same range, azimuth, and coupling-rejection conventions.
Aggregation and metrics: Per-scene CRP/ADC tables average either eight training frames or one held-out test frame across six scenes, with means matching Table 2.The tables report 3DPS fidelity using the same metric layout as the main comparison.

H.1 Per-scene |RA| comparison across methods

Across the per-scene |RA| comparisons, 3DPS is shown against three magnitude-only baselines on training and held-out frames, with correlation values overlaid.

Per-scene |RA| renders: Figures 4–9 compare GT, 3DPS, RadarSplat, Radar Fields, and DART across eight training frames plus one held-out frame for each scene.The visual comparisons overlay per-frame train/test Pearson correlation and assess preservation of wall and vehicle structure.
Cross-product outputs: 3DPS is the only compared method shown with CRP and ADC outputs because the three optical-NVS baselines emit magnitude-only |RA|.Figures 10 and 11 therefore compare those products only between GT and 3DPS.
Visualization layout: The |RA| panels use side-by-side scene and frame comparisons with GT, predicted renders, and per-cell correlation annotations.The supplied figures cover scenes S0–S2 and multiple selected held-out frames.

I Wall-clock and peak GPU memory

The runtime comparison reports measured 3DPS and optical-NVS costs on one RTX 4090, while the MC reference is a single-viewpoint extrapolation rather than a benchmark baseline.

Measurement setup: Table 7 reports per-scene wall-clock and peak GPU memory on the six-scene benchmark using a single RTX 4090.Measured methods use the same 9-frame split and include forward, backward, optimizer, and I/O costs.
MC reference caveat: The Sionna RT value is a single-viewpoint reference linearly extrapolated from 8.2 minutes at 150 iterations to the 500-iteration budget.It is not a measured baseline on the eight-viewpoint benchmark.

K Limitations and future work

3DPS is limited by its single-bounce assumption, magnitude-only supervision, LiDAR-dependent initialization, and unaddressed dynamic-scene and downstream-perception settings.

Single-bounce path tracing: Single-bounce rendering is sufficient for static outdoor scenes with direct returns but remains unsuitable for indoor multipath-dominated environments.The authors identify multi-bounce path tracing as a planned extension for settings where higher-order reflections carry comparable energy.
Phase supervision: Magnitude-only |RA| supervision leaves approximately 22,000 per-(VA, range) absolute-phase degrees of freedom unconstrained.Differential-phase metrics avoid nuisance phase, while tighter absolute-phase fitting would require a complex loss term.
LiDAR dependence: LiDAR-conditioned initialization simplifies convergence but leaves pure radar-driven NVS as an open challenge for radar-only deployments.The scaffold is necessary under the paper’s cascading-sparsity conditions, which make 3D structure unrecoverable from sparse RA-only supervision.
Future work: Extending coherent rendering to moving targets and validating NVS-augmented data for radar perception remain future directions rather than addressed capabilities.The proposed complex-output property enables these directions but does not directly solve them.

L Ablations

Ablations show that adaptive density control, initialization choices, MIMO factorization, PSF width, and carrier-phase detachment materially affect 3DPS performance, while some settings are nearly neutral.

Point count: 0.586 at N=10k and 0.587 at N=20k peak before performance regresses to 0.538 at N=50k.The authors identify 20k as near-optimal and 5k–10k as viable for memory-constrained deployments.
Training-view budget: 0.573 with two views and 0.587 with eight views demonstrate near-constant quality across the tested view budgets, while training time rises from 0.9 to 3.0 minutes.The reported correlations for 2, 4, 6, and 8 views are 0.573, 0.575, 0.586, and 0.587.
Adaptive density control: 0.553 versus 0.587 results when adaptive density control is disabled, while CRP and ADC envelope correlations remain close to default.This indicates the mechanism mainly benefits magnitude-domain fitting in the reported benchmark.
MIMO factorization: 0.555 versus 0.587 and 5.3 versus 3.0 minutes show that MIMO factorization is both faster and more accurate than the expanded PyTorch fallback.The fused implementation avoids materializing the full BSDF and splat tensors and uses a hand-derived backward gradient.
Carrier-phase detach: 0.537 versus 0.587 shows that allowing position gradients through carrier phase causes the largest single ablation drop.At 77 GHz, sub-millimetre updates can create order-one phase rotations, making the position-loss landscape locally periodic.
Overall findings: 0.587 default test |RA| correlation depends most strongly on carrier-phase detachment, adaptive density control, FPS initialization, and MIMO factorization.Across nine axes, these four are load-bearing with changes of at least 0.03.