Source-linked AI summary
DyRAD: Radar Novel View Synthesis for Dynamic Driving Scenes
Merav Keidar, Tomer Borreda, Rajalakshmi Nandakumar, Or Litany
TL;DR
Prior radar NVS did not jointly model dynamic-scene Doppler and displaced-view synthesis. DyRAD renders full RAD with tracked point reflectors and a fixed sensor response; on RADIal, foreground hit rate increases from 26.9% to 90.7% over the strongest respective baselines.
Problem
Prior radar NVS did not render Doppler for dynamic driving scenes, and evaluation stayed on recorded trajectories rather than testing displaced views.
Source: §1 · pp. 2–3Method
DyRAD fits static and track-following point reflectors to full RAD measurements, projecting track-derived relative motion onto the line of sight and rendering through a fixed sensor-specific PSF.
Source: §1 · p. 3, §3.2 · p. 5Results
90.7% foreground hit rate on RADIal versus 26.9% over the strongest respective baselines; full-RAD correlation rises from 0.068 to 0.272.
Source: §1 · p. 3Takeaways & Limitations
Abstract
from arXiv · showhide
Reconstructing dynamic driving scenes from recorded sensor data supports closed-loop evaluation of autonomous driving systems by synthesizing observations beyond the original trajectory. Unlike cameras and LiDAR, radar measures radial velocity directly through Doppler. Yet existing radar novel-view synthesis fails to exploit this capability: methods addressing dynamic scenes reconstruct only range-azimuth tensors, while methods that render Doppler assume static scenes. Moreover, because radar processing spreads each reflection across multiple bins, existing representations absorb this spread into scene geometry, causing it to render incorrectly when the viewpoint moves. We present DyRAD, which models dynamic driving scenes using static background reflectors and motion-tracked dynamic point reflectors to render complete range-azimuth-Doppler (RAD) tensors. Reflector velocities are derived from object tracks and projected onto the line of sight, making Doppler both a rendered output and supervision for those tracks. Crucially, we render reflectors through a fixed analytic point-spread function (PSF) derived from the radar's signal-processing chain, preventing sensor-induced spread from being baked into the scene representation. Beyond improving scene reconstruction, this separation also enables zero-shot sensor-configuration transfer, allowing the same reconstructed scene to be rendered under different radar specifications without refitting. We evaluate DyRAD on RADIal, Boreas, and a synthetic benchmark across both on-path poses and displaced viewpoints untested by prior work. On RADIal, DyRAD recovers radar detections in 90.7% of reference-detected objects, compared with 26.9% for the strongest baseline.
1 INTRODUCTION
Radar’s direct Doppler capability makes it valuable for automotive simulation, but existing radar NVS leaves dynamic Doppler, sensor-response separation, and displaced-view testing unresolved. DyRAD addresses these gaps with track-based reflectors and a fixed sensor-specific PSF, reporting a 90.7% foreground hit rate on RADIal versus 26.9% for the strongest respective baselines.
1 INTRODUCTION
- Radar provides direct radial-velocity measurements through Doppler and robust operation in adverse weather, but sparse, low-resolution returns make reconstruction challenging.Power varies with range and viewing aspect, while Doppler depends on relative sensor–object motion.Source: §1 · p. 2
- Prior radar NVS leaves unresolved dynamic-scene Doppler, separation of scene structure from sensor spread, and evaluation at displaced viewpoints.Existing dynamic-scene methods omit Doppler, while Doppler-rendering methods assume static environments; on-path testing can reward interpolation between near-identical poses.Source: §1 · p. 2
- DyRAD jointly fits static and track-following point reflectors to full RAD, renders track-derived Doppler through a fixed sensor-specific PSF, and supports alternative radar configurations without refitting.Reflector velocities are projected onto the line of sight, making Doppler both a rendered output and a constraint on object tracks.Source: §1 · pp. 2–3
- 26.9% to 90.7% foreground hit rate on RADIal over the strongest respective baselines; full-RAD correlation rises from 0.068 to 0.272.The same reconstructed scene also supports sensor-configuration transfer without refitting, improving object-region RAD PSNR and nearly doubling detection F1 over linear upsampling.Source: §1 · p. 3
- DyRAD introduces displaced ground-truth views in a synthetic benchmark and a cycle-consistency protocol on real recordings to test off-path synthesis.Prior radar NVS evaluated only along the recorded trajectory.Source: §1 · pp. 2–3
2 RELATED WORK
Dynamic novel-view synthesis has advanced for camera and LiDAR, while radar methods have explored RA reconstruction, Doppler, and sensor modeling. DyRAD’s related-work context centers on bringing dynamic full-RAD synthesis and explicit sensor response modeling to radar.
2.1 NOVEL-VIEW SYNTHESIS FOR DYNAMIC DRIVING SCENES
- Dynamic-scene novel-view synthesis now supports viewpoints never driven for camera and LiDAR, a capability this work extends to radar.Reconstruction-based simulation enables rendering along new ego trajectories for training and closed-loop evaluation.Source: §2.1 · p. 3
2.2 RADAR RECONSTRUCTION AND DOPPLER
- Radar Fields and RadarSplat reconstruct static-scene RA, while RF4D handles dynamic scenes but still reconstructs RA alone; NeuRadar renders sparse radar detections without Doppler.These approaches differ in representation, but none described here provides dense dynamic RAD reconstruction.Source: §2.2 · p. 3
- Prior Doppler methods either use radar velocity as a visual-reconstruction cue without rendering radar measurements or render Doppler under static-scene assumptions.DyRAD instead reconstructs dynamic scenes from full RAD and uses Doppler to refine the object motion that produced it.Source: §2.2 · p. 3
2.3 SENSOR MODELING IN NEURAL RENDERING
- Neural renderers model sensor effects such as sampling, blur, and radar propagation, while DyRAD couples point reflectors to a fixed sensor-specific RAD PSF.This explicitly separates sensor-induced measurement spread from the learned scene representation.Source: §2.3 · p. 4
3 METHOD
DyRAD reconstructs dynamic driving scenes from radar measurements and renders complete RAD tensors at novel sensor poses using static and track-based dynamic reflectors. Its method combines a point-reflector scene representation, physics-grounded rendering, and joint initialization and optimization.
3 METHOD
- DyRAD jointly reconstructs static and dynamic reflectors and object tracks from radar measurements to synthesize observations at query times and sensor poses.Dynamic-object bounding-box annotations initialize tracks; geometry and motion are modeled in the ground plane.Source: §3 · p. 4
3.1 SCENE REPRESENTATION
- Dynamic objects use rigid tracks: each reflector’s world position combines the object’s interpolated pose with its learned object-frame position.Reflectors have no spatial extent; the renderer models sensor-induced spreading through a fixed PSF.Source: §3.1 · pp. 4–5
3.2 PHYSICS-GROUNDED RAD RENDERING
- Reflector Doppler is derived from track motion and relative sensor velocity projected along the line of sight, including translational and rotational motion.Translational velocity is estimated from four neighboring object-track points.Source: §3.2 · p. 5
- A fixed sensor-specific PSF spreads each reflector response across range, azimuth, and Doppler before reflector contributions are summed into RAD tensors.The kernels are computed before scene optimization and derived from the signal-processing chain when available.Source: §3.2 · pp. 5–6
3.3 SCENE INITIALIZATION AND OPTIMIZATION
- Reflectors are initialized from radar peaks associated with object boxes or the background, then jointly optimized with track points using reconstruction and interpolation-consistency losses.The interpolation loss constrains smoothed RA predictions between training frames; positive residual peaks can add background reflectors during optimization.Source: §3.3 · p. 6
4 EXPERIMENTS
DyRAD is evaluated on recorded-path and displaced-viewpoint reconstruction, sensor transfer, and ablations across RADIal, Boreas, and synthetic scenes. Results show improved object reconstruction and detection, with trade-offs across evaluation protocols and metrics.
4.1 EXPERIMENTAL SETUP
- Evaluation covers RADIal, Boreas, and synthetic scenes, using static-scene baselines and reconstruction and detection metrics.RADIal and Boreas provide real recordings; synthetic scenes provide displaced-view ground truth. Reconstruction metrics include correlation, PSNR, SSIM, and LPIPS, while detection uses Chamfer distance, joint F1, and foreground hit rate.Source: §4.1 · p. 7
4.2 NOVEL VIEW SYNTHESIS
- On RADIal, full DyRAD raises full-RAD correlation from 0.213 to 0.272, object-region RAD correlation from −0.013 to 0.658, and foreground hit rate from 7.0% to 90.7% over DyRAD-static.On Boreas, its full-measurement correlation is 0.752 versus 0.666 for RadarSplat, while object-region correlation is 0.622 versus 0.178.Source: §4.2 · p. 7, Table 1 · p. 7, Figure 4 · p. 8
- In real-data off-path cycle testing, DyRAD recovers 91.7% of reference-detected objects on RADIal, versus 18.1% for the strongest prior baseline and 20.9% for the static model.These scores compare predictions at original poses after a 2m lateral-shift render-and-refit cycle, not direct off-path accuracy.Source: §4.2 · p. 8, Table 2 · p. 8
- Across 25 synthetic scene–offset pairs, object-region RAD correlation is 0.356 for DyRAD, versus 0.161 for DyRAD-static and 0.086 for RadarFields.Foreground hit rate is 74.6%, versus 22.4% for DyRAD-static and 36.0% for RadarSplat; DyRAD also has more reference-empty annotated boxes containing predicted detections: 4.9 versus 0.7 and 3.4, respectively.Source: §4.2 · p. 8
4.3 ADDITIONAL CAPABILITIES
- Direct rendering with a finer sensor configuration reduces CD from 7.27 to 3.20m and raises F1 from 0.101 to 0.195 over linear upsampling, without refitting.Upsampling scores higher on most correlation and SSIM metrics, although direct rendering improves object-region RAD PSNR.Source: §4.3 · p. 9
4.4 ABLATION STUDIES
- Doppler supervision raises joint detection F1 from 0.144 to 0.179 on-path and from 0.120 to 0.236 off-path, while reducing object Doppler peak error in both protocols.Foreground hit rate remains nearly unchanged; object-region RAD correlation improves while full-measurement RA correlation declines.Source: §4.4 · p. 9
- Interpolation consistency improves on-path RA, RD, and RAD correlation by 0.219, 0.143, and 0.051, but lowers off-path RAD correlation from 0.447 to 0.363 and F1 from 0.296 to 0.236.The loss regularizes intermediate timestamps evaluated on-path, whereas off-path evaluation uses training timestamps; it is retained to balance temporal interpolation and RA fidelity against off-path RAD fidelity.Source: §4.4 · p. 9
- With fixed sensor-derived PSF modeling, off-path F1 reaches 0.236, versus 0.074 for learned extent and 0.124 for learned bandwidths.The fixed PSF improves object-region correlation across RA, RD, and RAD in both protocols, despite lower full-RAD PSNR and SSIM.Source: §4.4 · p. 9
5 CONCLUSION, LIMITATIONS, AND FUTURE WORK
DyRAD synthesizes complete RAD measurements of dynamic driving scenes from novel viewpoints using static and motion-tracked point reflectors with a fixed sensor response. The formulation improves object reconstruction and detection in on-path and off-path evaluations, while several limitations remain.
5 CONCLUSION, LIMITATIONS, AND FUTURE WORK
- DyRAD combines static and motion-tracked point reflectors with a fixed sensor response to synthesize complete RAD measurements from novel viewpoints.The authors describe it as the first radar NVS method to reconstruct dynamic driving scenes with Doppler measurements.Source: §5 · p. 9
- The formulation improves object reconstruction and detection in both on-path and off-path evaluations and supports sensor-configuration changes without refitting.Source: §5 · p. 9
- Reflector-to-object assignments rely on annotations and remain fixed during optimization, while the evaluated implementation uses a ground plane because its measurements lack elevation.Future work could refine assignments, model multipath and speckle noise, evaluate elevation-resolving radar, and test transfer across physical sensors.Source: §5 · p. 9
AI USE STATEMENT
The authors used generative AI tools to assist with writing, code, and literature review, and reviewed and revised AI-assisted outputs.
AI USE STATEMENT
- Generative AI assisted with writing and editing code, manuscript drafting and revision, and related-literature review; the authors reviewed and revised the outputs and take responsibility for the final content.Source: AI USE STATEMENT · p. 10
REPRODUCIBILITY STATEMENT
The appendix documents implementation, evaluation, and reproducibility details, including code and a deterministic synthetic-data generator. It describes sensor-specific fixed PSFs and a scene initialization procedure based on training-frame measurements and object annotations.
REPRODUCIBILITY STATEMENT
- The code includes usage examples, experiment configurations, and a deterministic generator for the synthetic off-path radar NVS benchmark; videos are available on the project page.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1 SENSOR-SPECIFIC PSF CONSTRUCTION
- Sensor-specific PSFs are constructed before optimization and held fixed; RADIal kernels use its processing chain, while Boreas uses approximate fixed responses because its processing chain is unavailable.For RADIal, range and Doppler responses approximate Hamming-windowed FFTs, with transmitter shifts gathered onto a reduced Doppler grid; azimuth uses its calibrated beamformer response.Source: §A.1 · p. 13
- For Boreas, tapered sinc-squared kernels approximate range and azimuth responses; the range width is selected by validation because its specifications omit that width, and its Doppler response is set to hD = 1.Source: §A.1 · pp. 13–14
B.1 SCENE INITIALIZATION DETAILS
- Scene initialization uses training measurements and bounding boxes to separate background and object regions, select returns, project them into world coordinates, and accumulate them across frames.Dynamic returns are aligned by track translation into object coordinates; this approximates alignment for turning objects, and deduplicated returns initialize static and dynamic reflectors.Source: §B.1 · p. 14
B.2 HYPERPARAMETER SELECTION
Hyperparameters are selected using validation data and conditional, independent sweeps, while densification adds static reflectors from residual peaks during training. The reported sensitivity sweep found performance stable across 36 configurations.
B.2 HYPERPARAMETER SELECTION
- Hyperparameter choices use held-out validation data; RADIal and Boreas use different validation-window and metric setups, with Boreas omitting RAD correlation because it has no Doppler dimension.RADIal candidates whose object recall falls more than 0.05 below the best arm are disqualified.Source: §B.2 · pp. 14–15
- Five optimization hyperparameters are swept independently over ten logarithmic values each, so selections are conditional rather than jointly optimal.Source: §B.2 · p. 15
- Per-window rank agreement confirms the interpolation-weight selection on RADIal and all five selections on Boreas, but disagrees or ties for four RADIal learning rates.The authors adopt the composite-score winners for those RADIal learning rates; Table 3 lists search ranges and selected values.Source: §B.2 · p. 15, Table 3 · p. 15
B.3 DENSIFICATION
- Residual-guided densification adds 1,000 strong static-reflector candidates every 200 training steps from step 1,000, subject to duplicate, distance, residual-threshold, and representation-size filters.Candidates come from positive-residual peaks in the ego-motion-compensated static-Doppler band; the complete representation is limited to 800,000 reflectors.Source: §B.3 · p. 15
- A 36-configuration densification sensitivity sweep found performance stable, and the authors retain its central configuration for every setting.Source: §B.3 · p. 15
C.1 EVALUATION PROTOCOL
The protocol combines reconstruction scores with detection tests, while diagnostic controls expose limits of global image metrics. It reports on-path results and real-data round-trip tests after a 2 m lateral shift.
C.1 EVALUATION PROTOCOL
- Reconstruction is assessed with PSNR, SSIM, LPIPS, and Pearson correlation, while detection tests whether synthesized returns produce usable point clouds and preserve returns in annotated boxes.Detection uses unchanged native CFAR settings; foreground hit rate and density similarity penalize missed returns in boxes containing reference detections.Source: §C.1 · pp. 15–16
- A structureless constant prediction scores higher in SSIM than every reconstruction method and higher in PSNR than RadarSplat and RadarFields, showing those metrics alone do not establish preserved radar structure.The protocol therefore interprets PSNR and SSIM alongside correlation, LPIPS, object-region scores, and detection.Source: §C.1 · p. 16, Table 4 · p. 16
C.2 ON-PATH EVALUATION
- Tables 5 and 6 report full on-path reconstruction and detection results, while Figures 6 and 7 show qualitative RADIal and Boreas comparisons.The supplied passages identify the results’ scope but do not give table values.Source: §C.2 · p. 16, Table 5 · p. 18, Table 6 · p. 18
C.3 REAL-DATA OFF-PATH EVALUATION
- The real-data off-path protocol reports full reconstruction and detection results after a 2 m lateral-shift round trip, with RadarFields’ near-constant and all-zero RADIal reconstructions included in averages.Figures 8 and 9 show the displaced renders and reconstructions at the original poses.Source: §C.3 · pp. 16, 18
C.4 SYNTHETIC OFF-PATH EVALUATION
The synthetic benchmark evaluates reconstruction at displaced views against ground truth, and the section also reports DyRAD’s computational cost. Its synthetic sensor model omits real DDMA demultiplexing artifacts.
C.4 SYNTHETIC OFF-PATH EVALUATION
- Five synthetic scenes provide ground-truth measurements from an idealized sensor model, with 40 frames per sequence and RADIal-format tensors.The model does not reproduce real DDMA demultiplexing artifacts.Source: §C.4 · pp. 18–19
- Methods are fitted on the base trajectory and evaluated without refitting at five displaced trajectories spanning lateral and yaw offsets, pooled across 25 scene–offset pairs.The reported standard deviation reflects variation across scenes and views, not training seeds.Source: §C.4 · p. 19, Table 9 · p. 21
- Tables 9 and 10 provide the synthetic off-path reconstruction and detection results; object-region RD metrics include all Doppler bins at annotated-vehicle ranges.The supplied passages describe table scope and masking but do not provide result values.Source: §C.4 · p. 19, Table 9 · p. 21
C.5 RUNTIME
- 7.2 minutes median training time, 5.65 ms full-RAD rendering latency, and 1055 MiB peak GPU memory are reported for DyRAD on an RTX 4090.The baselines render RA maps, so their rendering latency measures a different output; DyRAD’s modest overhead over splatting is described as consistent with motion evaluation and full-RAD synthesis.Source: §C.5 · pp. 20–21
D ABLATIONS
The ablations use RADIal on-path and off-path protocols, with the learned-extent variant capped at 300,000 primitives.
D ABLATIONS
- Ablations are evaluated on-path and through both fitting stages of the off-path cycle; the learned-extent variant is limited to 300,000 primitives.Tables report means and sample standard deviations across ten sequences.Source: D ABLATIONS · p. 21
E SENSOR CONFIGURATION TRANSFER
The transfer experiment fits a scene to coarse measurements and renders it under the native configuration without changing scene parameters, comparing this route with linear upsampling. The supplied tables also cover on- and off-path RADIal ablations.
E SENSOR CONFIGURATION TRANSFER
- The coarse configuration has fourfold coarser range resolution and twofold coarser Doppler resolution than native, while azimuth and Doppler wrap period remain unchanged.After range cropping, native and coarse RAD tensors contain 16 × 447 × 751 and 8 × 112 × 751 bins, respectively.Source: E SENSOR CONFIGURATION TRANSFER · p. 22
- Direct rendering replaces the coarse PSF and sampling grid with the native configuration’s while keeping reflector and motion parameters fixed; linear upsampling serves as the comparison.Both outputs are evaluated against the same held-out native measurements from the same RADIal recordings.Source: E SENSOR CONFIGURATION TRANSFER · pp. 21–22, Table 16 · p. 25
D ABLATIONS
- The supplied RADIal ablation tables cover on- and off-path reconstruction, detection, and Doppler; Doppler MAE is reported only for the Doppler-supervision comparison.The tables report means and sample standard deviations across ten sequences.Source: Table 12 · p. 23, Table 13 · p. 23, Table 14 · p. 24