PaperlayerView exact revision on arXiv

Source-linked AI summary

GRADE: Single-Frame Generative Radar Depth Estimation Under Visual Degradation

Bin Zhao, Patrick Chiou, Nakul Garg

arXiv:2609.10756v1cs.CVcs.RO

TL;DR

Optical depth sensing fails in smoke, fog, and darkness, while single-frame radar provides robust range but limited angular resolution. GRADE grounds latent diffusion refinement in radar-derived metric depth and uses residual camera cues when available. Across 12 buildings with real smoke, it achieves the best result on every reported metric in clear and smoke conditions, including 0.303 m clear-scene MAE and 0.313 m smoke-scene MAE.

  • Problem

    Optical sensors lose depth information under smoke, fog, and darkness, while single-frame radar’s small aperture limits angular resolution.

  • Method

    GRADE maps 4D radar spectra to coarse metric depth, conditions a latent diffusion backbone on it, and adds pixel-space RGB residual guidance that weakens with visibility.

  • Results

    GRADE achieves the best result on every reported metric across 12 buildings with real smoke, with MAE of 0.303 m in clear scenes and 0.313 m under smoke.

  • Takeaways & Limitations

    GRADE supports dense metric depth estimation in degraded-vision settings while retaining radar-conditioned predictions as visual evidence weakens.

  • Takeaways & Limitations

    The evaluation does not cover outdoor geometry, unseen radar interference, different sensor configurations, or materials absent from training.

Abstract

from arXiv · show

Dense 3D depth perception fails under smoke, fog, and darkness because optical sensors cannot penetrate airborne particulates. mmWave radar remains usable and measures range accurately under these conditions, but its small aperture limits angular resolution. We present GRADE, which grounds a pretrained generative prior in single-frame radar geometry to estimate high-fidelity metric depth. GRADE first maps raw 4D radar spectra to coarse metric depth. A latent diffusion backbone then recovers structural detail while conditioning every denoising step on this estimate. A pixel-space adapter uses residual camera cues when available and is trained across clear, smoke-degraded, and occluded inputs so the full output approaches the radar-conditioned path as visibility degrades. Trained and evaluated on ~95K frames across 12 buildings with real smoke, GRADE achieves an MAE of 0.303 m in clear scenes and 0.313 m under smoke, outperforming existing baselines. Code and datasets are available at https://phi-lab-rice.github.io/GRADE.

1 Introduction

GRADE targets single-frame metric depth estimation when smoke, fog, or darkness undermine optical sensing, using radar geometry and generative refinement to preserve detail. Across real-smoke evaluations, it maintains nearly unchanged MAE from clear to smoke conditions and outperforms the evaluated baselines.

  • Motivation: GRADE addresses the gap between radar’s robust range sensing and its limited angular resolution without requiring controlled motion or reliable odometry.Prior single-frame methods avoid motion requirements but remain limited in resolution, while SAR depends on displacement and accurate positioning.
  • Our approach: The visual branch contributes spatial precision when camera evidence remains useful and approaches the radar-conditioned path as visibility worsens.This asymmetric guidance is designed to avoid dependence on optical input during smoke or occlusion.
  • Summary of results: 0.303 m clear-scene MAE and 0.313 m smoke-scene MAE show GRADE’s depth accuracy changes little under visual degradation.The evaluation covers approximately 95K frames across 12 buildings, including more than 40K real-smoke frames.
  • Summary of results: Under heavy smoke, GRADE reaches 0.104 m2 Chamfer Distance, 40% below GRT and 24% below GRT+Image.The corresponding baseline values are 0.173 m2 for GRT and 0.137 m2 for GRT+Image.
  • Our approach: GRADE combines radar-to-depth translation, diffusion refinement, and residual RGB guidance to produce dense metric depth from a single radar frame.The radar module establishes metric geometry, while the diffusion and visual-guidance modules refine structure and use camera cues when reliable.

2 Related Work

Related work extends depth perception through optical fusion, synthetic aperture sensing, and single-frame radar, but each approach retains important sensing or motion assumptions. GRADE instead combines radar geometry with generative refinement and auxiliary visual guidance.

  • Depth Under Visual Degradation: Prior optical restoration and diffusion-inpainting methods remain coupled to residual visual information and fail when the optical path is blocked.This is the failure regime that GRADE targets with radar as its geometric backbone.
  • Multi-Modal Radar Fusion: GRADE reverses camera-radar dependence by using radar for metric depth and RGB only as an auxiliary correction when usable.This contrasts with fusion methods that depend on dense camera input and can fail when the optical path is blocked.
  • Synthetic Aperture Radar: SAR methods achieve high resolution through physical displacement and temporal accumulation, precluding per-frame depth when motion or pose estimates are unavailable.At 79 GHz, the cited half-wavelength positioning requirement is 1.9 mm.
  • Generative refinement: GRADE’s architecture uses a radar depth module, a diffusion refinement module, and an RGB visual guidance module across two processing stages.The diffusion module recovers fine structure from learned priors, while RGB injects pixel-space residual cues during denoising.

3 System Design

GRADE translates single-frame radar spectra into coarse metric depth, then uses radar-conditioned diffusion and residual RGB guidance to recover structural detail while retaining geometric grounding under degraded visibility.

  • 3 System Design: GRADE maps 4D radar spectra to coarse metric depth before a latent diffusion module refines structure under persistent radar conditioning.The radar depth module establishes the geometric foundation, while diffusion denoises from Gaussian noise into a structurally refined estimate.
  • 3 System Design: The radar module provides metrically grounded, spatially coherent depth because raw radar spectra lie outside pretrained vision models’ training distribution.Its role is modality translation rather than complete radar-only reconstruction.
  • 3 System Design: Removing the gradient loss raises MAE by 19–21% and CD by 45–63% across clear and smoke conditions.The loss sharpens depth discontinuities needed to anchor downstream diffusion refinement.
  • 3 System Design: Radar depth preserves global layout but loses clutter, small objects, and fine structure in complex scenes, motivating generative refinement.The shortfall is attributed to radar’s physical angular resolution rather than model capacity alone.
  • 3 System Design: Diffusion can add missing structures when radar geometry is reliable, but geometric ambiguity can place plausible structures at incorrect locations.The staircase example motivates adding RGB visual guidance.
  • 3 System Design: RGB guidance uses surviving edges and surface transitions when visible, while fully occluded inputs cause the output to follow the radar-conditioned prediction.This design supports detail correction under partial visibility and graceful fallback at zero visibility.

4 Implementation

GRADE is implemented with synchronized radar, camera, and depth data from 12 buildings, then trained in three sequential stages to translate radar spectra into metric depth and refine it with diffusion.

  • Data Collection: Approximately 95K synchronized frames, including more than 40K real-smoke frames, span 12 buildings with building-disjoint evaluation sequences.Frames are synchronized across radar, camera, and reference depth modalities, and test buildings are entirely withheld from training.
  • Ground Truth: Reference depth remains independent of smoke density because smoke is confined around the camera while the stereo system observes through a clear optical path.The evaluation setup uses an acrylic enclosure for the smoke-degraded camera and an external ZED reference camera.
  • Radar Processing: Raw I/Q measurements are transformed into a normalized real-valued 4D spectrum spanning range, Doppler, azimuth, and elevation.Magnitude and phase become separate channels after FFT processing across fast-time, slow-time, azimuth, and elevation.
  • Training Procedure: GRADE trains sequentially by learning the radar depth predictor, freezing radar and autoencoder modules during diffusion training, then training only visual-guidance components.The third stage uses real-smoke data and degraded clear images, including synthetic fog and full occlusion.
  • Inference: Inference replaces DDPM training-time diffusion with 8-step DDIM sampling.The inference sampler is initialized from the same noise schedule used during training.

5 Evaluation

GRADE is evaluated on unseen buildings using more than 25,000 frames, with metrics covering pixel accuracy and perceptual or structural similarity across clear and smoke conditions.

  • Evaluation Setup: More than 25,000 test frames come from buildings excluded entirely from training, enabling evaluation on unseen indoor environments.The study compares GRADE with six camera-only, radar-only, and radar-camera baselines on a common 288×512 depth grid.
  • Metrics: MAE measures average per-pixel depth error, while SSIM and LPIPS assess structural consistency and learned perceptual distance.MAE and LPIPS are lower-is-better metrics, whereas SSIM is higher-is-better.

5.2 Depth Prediction Quality

GRADE provides the strongest depth quality across clear and smoke conditions, combining radar-stable metric accuracy with improved structural and perceptual detail.

  • Quantitative Results: 0.303 m clear-scene MAE improves on 0.415 m, while smoke-scene MAE is 0.313 m and CD is 0.114 m2.In clear scenes, CD improves from 0.153 to 0.120 m2 relative to the strongest baseline on each metric.
  • Quantitative Results: GRADE achieves the best performance on all five metrics in both clear and smoke conditions.Table 2 reports median metrics on the building-disjoint test set.
  • Distributional Results: GRADE remains shifted toward lower error and higher SSIM across empirical distributions, while DA3 and CaFNet develop long high-error tails under smoke.The distributional result indicates that the median comparison is not driven by only a small subset of frames.
  • Baseline Comparison: Under smoke, GRT+Image lowers CD from 0.208 to 0.146 m2, but GRADE combines radar-stable metric accuracy with lower LPIPS than every baseline.GRT remains nearly unchanged in MAE across visibility conditions, while its LPIPS stays above 0.44.
  • Qualitative Results: GRADE most consistently preserves major surfaces, people, stairs, and furniture boundaries from clear through heavy-smoke scenes.The qualitative comparison shows finer boundaries than radar baselines and avoids the severe smoke-induced structural loss of camera-dependent baselines.

5.3 Stability Under Varying Smoke Density

Across light, medium, and heavy smoke strata, GRADE maintains stable performance while camera-dependent baselines degrade as sensor readings increase.

  • Stratified Results: GRADE is best on every metric in each smoke stratum and changes little from light to heavy smoke.Table 3 separates results using the defined sensor thresholds.
  • Camera Baselines: DA3’s MAE rises from 0.548 m in light smoke to 1.751 m in heavy smoke, while its CD rises from 0.290 to 7.796 m2.CaFNet shows the same directional degradation, with CD increasing from 0.278 to 3.503 m2.
  • Distributional Stability: GRADE distributions remain similar across smoke levels, whereas DA3 and CaFNet medians and interquartile intervals increase with sensor reading.The strata use light, medium, and heavy smoke groups defined by the infrared sensor.

5.4 Ablation Studies

Ablations show that GRADE’s visual guidance, radar-to-depth stage, edge-aware and 3D supervision, Doppler input, and sampling schedule each materially affect performance. The full system is strongest overall, while diffusion alone does not improve the radar estimate.

  • Stage-wise ablation: Ours_full is best on all six pooled-test metrics, including MAE of 0.308 m, LPIPS of 0.133, and CD of 0.116 m2.Without visual guidance, diffusion improves GE but worsens the other five metrics.
  • Visual guidance: GRADE benefits from informative visual cues and approaches the radar-only estimate as smoke increases, with ΔMAE remaining near zero rather than worsening steadily.Negative ΔMAE through light and medium smoke favors GRADE over the radar-only module.
  • Radar prediction stage: Replacing Stage 1 with GRT yields 0.433 m pooled MAE after retraining, still worse than Ours_full on every reported metric.Freezing the refinement stage gives 0.450 m MAE, so the Stage 1 advantage persists after downstream adaptation.
  • Doppler input: Removing Doppler raises pooled MAE by 47% for Ours_radar and 63% for GRT, while CD more than doubles for both architectures.The larger 3D degradation suggests Doppler helps separate scatterers sharing a radar cell.
  • Supervision losses: Removing Lgrad raises MAE from 0.320 to 0.385 m and CD from 0.132 to 0.203 m2, showing the value of edge-aware depth supervision.Adding L3D reduces MAE from 0.376 to 0.308 m and CD from 0.169 to 0.116 m2 through geometric regularization.
  • Sampling steps: Performance peaks at 8 DDIM steps and saturates beyond 10, so experiments use 8-step sampling.A single step is insufficient, while increasing beyond 10 provides no further improvement.

5.5 Performance Across Radar Sparsity and Range

GRADE remains effective across radar-return sparsity and depth ranges, although errors increase in denser-return scenes and generally with distance. The analysis evaluates MAE by return-count strata and AbsRel by 2-meter range bins.

  • Radar sparsity: GRADE’s MAE is 0.288 m in sparse scenes, 0.296 m in medium scenes, and 0.369 m in dense scenes.Dense-return scenes contain more radar-visible surfaces and depth transitions, increasing reconstruction complexity.
  • Range: GRADE’s AbsRel upper quartile rises from about 0.13 within 4 m to 0.37 in the farthest bin, while the median reaches about 0.18 at 8–10 m.Across the analyses, error increases with return density and range.

5.6 Performance Across Scene Complexity

All methods degrade as scene complexity increases, but GRADE remains the strongest across the evaluated complexity groups. Its advantage narrows in complex scenes with fine-scale structure unresolved by a single radar frame.

  • Complexity-stratified results: GRADE has the lowest LPIPS and CD in all three scene-complexity groups.RadarCam-Depth is second-best in LPIPS, while GRT+Image is second-best in CD.
  • Complexity-stratified results: LPIPS and CD increase with complexity for every method, and GRADE’s margin narrows in complex scenes.Complex scenes contain more fine-scale structure than a single radar frame can resolve.

6 Discussion and Limitations

The discussion identifies computational cost, missing temporal consistency, limited out-of-distribution coverage, and hallucination risk as important boundaries. These limitations motivate faster inference, sequence-aware modeling, broader testing, and careful interpretation of inferred detail.

  • Computational cost: Iterative denoising is the main computational bottleneck, and the prototype is not designed for real-time wearable operation.The best performance uses 8 DDIM steps; distillation, quantization, and smaller backbones are proposed to reduce cost.
  • Temporal consistency and stability: GRADE estimates frames independently, so it provides immediate depth without enforcing consistency across a sequence.Pose-aware constraints or short-window radar aggregation are suggested for continuous mapping and overlays.
  • Extreme and out-of-distribution conditions: Evaluations cover unseen indoor buildings and real smoke but not outdoor geometry, unseen radar interference, different sensor configurations, or absent training materials.These settings require matched radar-depth data and explicit out-of-distribution testing.
  • Generative hallucination risks: Some recovered details are inferred rather than directly measured, and GRADE can slightly trail the radar-only prediction on some subsets as visual features weaken.Plausible detail should therefore not automatically be treated as measured geometry.

7 Conclusion

GRADE grounds a generative vision prior in single-frame radar geometry for dense metric depth under visual degradation, achieving the best reported results across clear and smoke conditions.

  • GRADE achieves the best result on every reported metric in both clear and smoke conditions across 12 buildings with real smoke.
  • The method grounds a generative vision prior in single-frame radar geometry to estimate dense metric depth under visual degradation.
  • GRADE evaluates dense metric depth across 12 buildings containing real smoke.
Loading 2609.10756v1…