Semantic Reconstruction and 3-D Detection via Learned Multi-Pair Fusion in RF Imaging

Amir Rezaei; Wen-Xin Pan; Giuseppe Caire

Source-linked AI summary

Semantic Reconstruction and 3-D Detection via Learned Multi-Pair Fusion in RF Imaging

Amir Rezaei, Wen-Xin Pan, Giuseppe Caire

arXiv:2608.23249v1Aug 24, 2026cs.CVeess.SP

TL;DR

The paper asks whether semantic perception and 3-D detection can remain useful when anisotropic, under-determined multistatic RF reconstructions deteriorate in noise. It feeds standard per-pair reconstructions into a 3-D U-Net for voxel classification and then applies geometric post-processing for instances and oriented boxes. The resulting semantic reconstruction and detection remain coherent deeper into noise than classical intensity fusion, while an outlier-exposed unknown class rejects held-out novel objects.

Problem
Under-determined multistatic RF imaging must support semantic labeling and object detection despite anisotropic coupling and reconstruction degradation with noise.
Method
Per-pair BP, LASSO, or group-LASSO reconstructions are fused by a 3-D U-Net, followed by DBSCAN instance clustering, PCA-oriented boxes, and an outlier-exposed unknown class.
Results
Semantic reconstruction and detection remain coherent well into noise levels where classical intensity reconstruction dissolves; fading inputs retain near-perfect known-object recall to about −50 dB and 18% of held-out novel objects are mislabeled as known.
Takeaways & Limitations
Learned multi-pair fusion can preserve usable semantic reconstruction, detection, and open-set rejection beyond the operating range of classical intensity reconstruction.
Takeaways & Limitations
The study is a controlled simulation with idealized forward and fading models, no measured data, and one training run per input, so the ordering lacks error bars.

Abstract

from arXiv · show

We consider a multistatic radio-frequency imaging problem with anisotropy, in which the reflection from a point depends on the positions of the transmit (Tx) and receive (Rx) arrays. The goal is to label the voxels of a field of view by a finite set of semantic classes and to group them into object instances. For the image formation of each Tx--Rx pair we apply a standard inverse-problem solver, and we feed the resulting per-pair reconstructions into a trained three-dimensional (3-D) U-Net that performs the fusion implicitly and the per-voxel classification explicitly. On a controlled, under-determined multistatic setup, we consider the following image formation methods: back-projection (BP) and the least absolute shrinkage and selection operator (LASSO) from a single deterministic snapshot, and incoherent BP and group-LASSO from multiple fading snapshots. For each imaging method we train a separate U-Net that fuses the six Tx--Rx pairs (its input channels) and assigns each voxel a probability vector over the classes. Taking the most probable class gives a labeled volume---the semantic reconstruction. Object instances and their oriented bounding boxes then follow by geometric post-processing (clustering and principal-component analysis). Across a wide range of signal-to-noise ratio, the semantic reconstruction (scored against ground truth by segmentation intersection-over-union) and the resulting 3-D detection degrade far more gracefully than the classical intensity reconstruction: the detection in particular stays reliable well into noise levels at which that reconstruction has dissolved. Because real scenes contain objects of classes the network was not trained on, we add an explicit unknown class trained by outlier exposure, which labels held-out novel objects as unknown instead of mislabeling them as a known class by reconstructed shape.

I. INTRODUCTION

The paper studies whether learned multi-pair fusion can preserve semantic perception when classical RF reflectivity reconstruction becomes unreliable. It replaces image-then-threshold processing with a U-Net pipeline that supports semantic reconstruction, instance grouping, oriented boxes, and unknown-object rejection.

Motivation: Multistatic RF reconstruction is ill-posed: sparse priors sharpen images at high SNR but degrade steeply with noise, while BP is robust but blurry.The paper focuses on the gap between reconstruction quality and downstream perception robustness.
Approach: Per-pair BP, LASSO, or group-LASSO reconstructions become input channels to a 3-D U-Net that outputs per-voxel semantic-class probabilities.The most probable class forms the semantic reconstruction; DBSCAN and PCA then produce instances and oriented bounding boxes.
Approach: An explicit unknown class trained by outlier exposure lets the network reject objects from classes absent during training instead of assigning them to known classes by shape.This addresses open-set scenes in which novel objects must not be absorbed into the known taxonomy.
Evaluation: The study compares BP and sparse reconstruction over a −70 to 0 dB per-antenna-element reference-SNR sweep and evaluates semantic reconstruction, detection, false positives, and open-set behavior.The evaluation uses a controlled, fully specified multistatic RF imaging setup.
Positioning: Unlike prior radar semantic-perception work based on automotive tensors or point clouds, this paper starts from a multistatic RF inverse-imaging problem.The comparison is framed around semantic reconstruction and downstream robustness rather than subjective image quality.

A. Geometry and forward model

The system uses three UPA terminals and six Tx–Rx views to form an under-determined multistatic RF image, with anisotropic coupling retained in per-pair reflectivities. A learned branch produces voxel labels, while geometric post-processing converts foreground labels into detected boxes.

Geometry: Three UPA terminals outside the FoV create six Tx–Rx pairs whose coloured links provide the multistatic views.The terminals are positioned on an equilateral triangle around the FoV centroid, with heights of 10, 15, and 20 m.
Fusion pipeline: The conventional branch averages the six per-pair reconstructions into a naive intensity fusion that is blurry and has no class labels.The learned branch instead produces semantic labels and detected boxes from the same scene data.
Fusion pipeline: The learned branch outputs per-voxel class probabilities, assigns each voxel its most probable class, and clusters foreground voxels for instance detection.The pipeline represents probabilities as voxel intensity and uses the resulting foreground labels for downstream geometry.
Geometric detection: The six learned input channels are fused by a 3-D U-Net, whose predicted class volumes are converted into oriented boxes using DBSCAN clustering and PCA.The post-processing adds no learning and derives box orientation from reconstructed object shape.
Anisotropic forward model: Each voxel’s reflectivity is coupled to Tx–Rx geometry through visibility and incidence-angle cosines, but the target-dependent directivity is unknown to the estimator.The estimator uses geometry embedded in its dictionary while recovering directivity through the per-pair reflectivity.

B. Operating point, SNR, and link budget

The setup uses a strongly under-determined sensing configuration and calibrates noise by a conservative per-element reference SNR before processing gain.

Operating point: Each view is 4× under-determined, with M=16384 measurements and Q=65536 voxels.The sensing panels use 16×8 elements and T=128 pilot slots.
SNR calibration: Noise is calibrated to a per-element reference SNR ρ with reflectivity normalized to unit peak and channel gain included in A.The noise variance uses the largest two-way channel gain in the field of view.
Link budget: A per-element ρ=−40 dB remains a hard operating point because matched filtering alone contributes approximately 42 dB of processing gain.Six-pair and, in fading, multi-snapshot fusion provide additional gain.

A. Per-pair imaging methods

Each Tx–Rx pair is reconstructed independently by a standard solver, and the resulting six views are either fused naively or supplied to learned fusion. The methods differ by snapshot regime and sparsity treatment.

Per-pair solvers: Four inverse-problem solvers are applied separately to every Tx–Rx pair, producing six input channels for the U-Net.The estimators include BP, LASSO, fading BP, and group-LASSO.
Deterministic regime: In the deterministic regime, BP is a column-normalized matched filter applied to a single snapshot.The reflectivity is treated as fixed when g≡1 and Ns=1.
Deterministic regime: LASSO uses FISTA to solve the sparse reconstruction problem with λ=η max_q |(A^H y)_q| and η=0.01.The regularization is set from the maximum matched-filter response.
Fading regime: Under fading, incoherent BP estimates reflectivity power by averaging per-snapshot intensities, while group-LASSO exploits common support across snapshots.The fading regime uses Ns=8 snapshots and fluctuating per-pair gains.
Fusion baseline: Naive intensity fusion averages the six per-pair intensities and serves as the baseline replaced by learned U-Net fusion.This baseline is used for the non-U-Net reconstruction figures.
Observed behavior: The point-spread function has a localized main lobe without aliased grating lobes, while group-LASSO is sharpest when clean and BP is most robust at −45 dB.The sparse group-LASSO dissolves into noise at −45 dB, whereas BP remains comparatively stable.

B. Learned fusion, segmentation, and geometric boxes

A 3-D U-Net implicitly fuses six per-pair reconstructions into voxelwise semantic probabilities, after which argmax labels support clustering and PCA-based oriented boxes. Explicit unknown-class training addresses held-out object classes.

Learned fusion: The U-Net outputs six-class voxel probabilities covering background, four known object classes, and an unknown class.The six classes are background, car, human, tree, bench, and unknown.
Semantic reconstruction: The most-probable voxel class forms the semantic reconstruction, with background and unknown assigned when their probabilities are largest.The semantic reconstruction is a labeled volume.
Figure guide: Fig. 3 compares naive intensity-fusion point-spread functions across four methods and two noise levels, using dB relative to peak in an xy cut.The white circle marks the sphere footprint; the panels are not U-Net outputs.
Open-set recognition: Held-out novel objects are explicitly assigned an unknown class through outlier exposure instead of being forced into a known class by shape resemblance.The closed-set alternative sends 68% of novel-object voxels to known classes, with posthoc AUROC only 0.66–0.69.
Training: The network uses log-compressed six-channel intensities, joint per-scene standardization, and a soft-Dice plus class-weighted cross-entropy loss.A separate identically configured U-Net is trained for each imaging method across all SNR levels.
Geometric boxes: DBSCAN separates foreground voxel clouds into instances, and PCA fits one oriented bounding box per instance without additional learning.Detection reuses the same label map as the semantic reconstruction.
Evaluation: Semantic quality is measured by mean per-class IoU, while detection uses known-class box recall, precision, and false positives per scene.The reconstruction baseline is evaluated with threshold-free Esdf and Chamfer distance.

A. The classical reconstruction breaks at low SNR

Naive intensity reconstruction exhibits a sharpness–robustness trade-off: sparse methods are best when clean, but BP overtakes them as noise increases.

Sparse methods: Esdf≈0.43 m for LASSO and 0.50 m for group-LASSO when clean, but both degrade to 1.3–1.4 m by −60 dB.The sparse methods degrade steeply below ρ≈−40 dB.
BP robustness: BP changes from 0.59 to 0.97 m across the deterministic SNR range and overtakes sparse methods at low ρ.BP is blurrier but nearly flat over the full range.
Cross-metric behavior: The Chamfer distance shows the same low-SNR crossover as Esdf.Naive intensity fusions remain legible at −25 dB but degrade by −45 dB.

B. The learned reconstruction is robust and rejects novel objects

The learned semantic reconstruction remains coherent at substantially lower SNR than classical intensity reconstruction, while fading inputs support robust detection and open-set rejection of novel objects.

Group-LASSO reaches the highest clean semantic mIoU at 0.53, while robust BP-fading degrades most gently.The reported absolute mIoU is moderated by the narrowband single-subcarrier regime.
Semantic reconstruction stays coherent down to ρ≈−40 dB, where classical intensity reconstruction has already dissolved into noise.The label map preserves correct shapes and classes despite the intensity reconstruction becoming unusable.
Fading inputs retain near-perfect known-object recall down to about −50 dB, with group-LASSO achieving about 0.3 false positives per scene when clean.Deterministic inputs are weaker and degrade faster.
The explicit unknown class limits novel-object mislabeling to 18% for fading inputs, compared with 68% of held-out voxels assigned known classes by a closed-set detector.Deterministic inputs mislabel roughly one-third to one-half of held-out objects.

V. CONCLUSION

The pipeline combines standard per-pair imaging, learned semantic fusion, and geometric post-processing to maintain coherent reconstruction and detection under noise. The study also identifies simulation and validation boundaries and outlines open-set, measured-data, solver, and statistical extensions.

Pipeline: A 3-D U-Net fuses six per-pair reconstructions, labels every voxel semantically, and supports instance extraction with PCA-oriented bounding boxes.The pipeline uses standard solvers before learned fusion and geometric post-processing after voxel labeling.
Robustness: Semantic reconstruction and detection remain coherent at noise levels where classical intensity reconstruction has dissolved, while maintaining controlled false positives.The conclusion attributes box orientation to PCA applied to reconstructed shape.
Limitations: The study is a controlled simulation using an idealized forward and fading model, AWGN, analytic primitives, and no measured data or material scattering.Its consistent method ordering is not yet established with error bars because there was a single training run per input.
Future work: Future work targets out-of-field-of-view scatterers, measured implementation, stronger per-pair solvers, and a calibrated open-set head.These extensions address structured interference, omitted physical effects, aspect dependence, specular returns, and known-recall trade-offs.
Validation: Statistical and ablation validation should include multiple seeds with error bars, single- and dropped-pair fusions, and a matched classical detection baseline.The proposed comparisons are intended to separate fusion and detection effects more rigorously.