RadioSight: Predictive mmWave XR Network Optimization from Dynamic Neural Radio Fields

• RadioSight targets the gap between static, offline radio-field reconstruction and actionable control, since prior approaches lack real-time synchronization or transmitter-side spatial spectra for downlink beamforming.Source: §1 · p. 2
• RadioSight fuses camera, IMU, and radio observations in an efficient 3D Gaussian Splatting field, using semantic information to represent movable objects and support lightweight edits instead of full retraining.The system is implemented as an edge-executable pipeline integrated with a commercial 28 GHz testbed and Meta Aria glasses.Source: §1 · pp. 2–3
• Backward beam-tracing uses accurate scene geometry to trace multipath arrivals from receiver to transmitter, identifying optimal transmitter AoD without exhaustive channel sounding.Source: §1 · pp. 2–3
• For transient blockages, RadioSight tracks objects with lightweight 3D bounding boxes and co-renders their effects; for long-term changes, it incrementally updates only displaced objects and their radio appearance.Source: §1 · p. 2
• RadioSight predicts future user motion from client-side SLAM and kinematic models, then pre-queries the synchronized field to schedule beam switches and rate adaptation before link degradation.Beam management for an upcoming scheduling horizon runs during the preceding one to mask computation latency; reactive switching can exceed mmWave channel coherence time.Source: §1 · pp. 2–3
• Beam alignment errors fall by up to ∼50%, median throughput improves by up to 2×, and RadioSight eliminates rescans and disconnections inherent to reactive schemes.Source: §1 · p. 3

Neural Radio Fields

• Prior radio fields—ray tracing, NeRF [24], and 3DGS —predict receiver-side spectra, not transmit AoD for NLoS MU-MIMO, and static models need costly retraining.Ray-tracing simulations [4] [14] remain offline planning tools; the paper states that RadioSight addresses all three limitations.Source: §2 · p. 3

XR mmWave Network System

• XR predictors such as Lumos5G, Habitus, and MP2, plus viewport and head-pose systems [30] [38], work above physics and react to degradation.RadioSight replaces this black-box approach with a physics-guided real-time radio field modeling geometry, scene dynamics, and multipath propagation to steer beams before outages.Source: §2 · p. 3

mmWave Beam Management

• 802.11ad/ay and 5G NR [10] use costly exhaustive sweeps; scan-history methods falter under mobility, IMUs miss blockages, and infrastructure sensors [15] [6] require LoS.RadioSight uses client-side egocentric perception with the base station to maintain a real-time radio twin and predict beams across LoS and NLoS conditions under dynamic blockages.Source: §2 · pp. 3–4

System Overview

• RadioSight resolves future valid radio paths by tracking UE pose, transient blockers, and registered movable objects, enabling beam decisions for each scheduling window during the preceding one.The paths are evaluated at future digital-twin instances rather than the present instance.Source: §3 · p. 4
• The XR Client reports its 6DoF pose, detected blockers’ 2.5D bounding boxes, and optional sparse spectrum scans over an omnidirectional 2.4 GHz control channel while receiving directional mmWave XR video.Source: §3 · p. 4
• The Edge Server fuses client reports into a shared dynamic scene, synchronizes tracked-object state to UEs, and derives beamforming and rate-adaptation commands from predicted-state spatial spectra.It extracts beam candidates before dispatching commands to the transmitter array and UEs.Source: §3 · p. 4

Scene-Update Thread

• Scene updates require ∼0.18 s per scan, ∼30–60 s of walking, and ∼1–2 min for one-time reconstruction, while recurring client-and-server updates take ∼2.3 ms.The client triggers updates on substantial UE-pose shifts or registered-object movement during mmWave idle periods; beam management reads the latest scene without blocking on updates.Source: §3 · p. 4, Figure 3 · p. 4

Beam-Management Thread

• Beam management costs ∼10.5 ms per window to predict five 10 ms frames over the coming 50 ms for two multiplexed UEs, while the motion-to-photon thread costs ∼29 ms.Beam processing renders spectra, back-traces them into transmitter-side candidates, then groups MU-MIMO users and synthesizes beamforming vectors.Source: Figure 3 · p. 4

4.1 Multi-modal Radio Field Construction

• A shared Gaussian representation jointly encodes RGB, radio, and semantic features, using visual geometry to anchor radio predictions and language-aligned semantics to identify movable-object clusters.The field renders RGB, radio spatial spectrum, semantic, and depth maps; the spectrum supports beam prediction, while object-level segmentation enables targeted dynamic updates.Source: §4.1 · p. 5

4.2 Field Training and Rendering

• Differentiable modality-specific supervision trains shared geometry from RGB, radio features from mmWave measurements, and semantic features from contrastive alignment.Surface-shape regularization reduces geometric ambiguity; at inference, rendering only radio and depth at array-matched resolution cuts one spectrum query to about 1.0 ms.Source: §4.2 · p. 6

4.3 Backward Beam-Tracing

• Backward beam-tracing converts peaks in the predicted receiver spectrum into transmitter departure directions, enabling proactive beam scheduling without exhaustive channel-sounding scans.It renders the spectrum and depth map together, uses depth to locate each peak’s surface intersection in constant time, then traces that point toward the transmitter; the method assumes single-bounce propagation.Source: §4.3 · p. 7

4.4 Collaborative Dynamic Radio Field

• RadioSight updates transient blockers as tracked 3D boxes and edits relocated objects’ Gaussian clusters in place, avoiding full-field retraining.Clients detect changes by comparing local geometry with sensor depth; object moves can trigger an optional scan when the mmWave channel is idle or has abundant resources. The system synchronizes updated geometry to UEs and incorporates predicted object poses and opaque blocker boxes into subsequent beam queries; blocker detection and fusion take about 2.3 ms, or 3.8 ms end-to-end including uplink.Source: §4.4 · pp. 7–8

4.5 Proactive Operation and Multi-User AP

• Predicted user and blocker trajectories drive batched radio-field queries and joint MU-MIMO beam selection that seeks high-energy paths with low inter-user interference.The system can also adapt bitrate before predicted SINR falls below the QoE threshold and coordinate APs to prepare handovers before a signal drop.Source: §4.5 · pp. 8–9

4.6 Discussion

• RadioSight supplements wearer vision with multi-user and infrastructure sensing, while its predictive radio spectra support proactive application-layer adaptation.The system is physics-guided rather than a full electromagnetic solver: it models object boundaries and occlusion relationships without phase, relying on single-beam transmission for XR streaming.Source: §4.6 · pp. 9–10

Implementation

• The prototype integrates a Radio Field engine with a real-world 28 GHz mmWave testbed, using an on-glass client and an mmWave edge node.The hardware deployment is shown in Fig. 12.Source: §5 · p. 10, Figure 12 · p. 10

Client-Side Architecture

• The client uses Meta Project Aria glasses to stream RGB frames and 6DoF poses, while a headset-mounted 4 × 4 array captures RF samples for processing on an Intel NUC11 i5.A down-converter and Ettus USRP B210 digitize the 28 GHz signals at 10 MHz instantaneous bandwidth; shifting the intermediate frequency synthesizes an effective 320 MHz channel.Source: §5 · p. 10

Edge-Side Architecture

• An edge server with an RTX 4090 updates the neural radio field from client control messages and dispatches derived downlink beamforming decisions to its local baseband SDR and UE.The generated signal is up-converted and transmitted through a 4 × 4 phased array; the testbed uses a fast beamforming algorithm (Appendix B.1).Source: §5 · p. 10

Experimental Scenarios and Datasets

• Measurements span six indoor environments, including direct-LoS, two NLoS layouts, a corner-bending corridor, a large lobby, and a lecture room, alongside mobility traces from XR tasks.The study also uses 28 GHz channel measurements from an 8 × 8 transmit and 8 × 8 receive phased-array sounder.Source: §5 · p. 10

6.1 Experimental Setup

• Ground-truth beams came from exhaustive beam-space sweeps, and comparisons covered 802.11ay, neural-network, vision-aided, NeRF2, and WRF-GS baselines.The vision-aided baseline used router-side camera tracking and rescanned locally when tracking failed; NeRF2 was RF-only, while WRF-GS used SLAM point-cloud initialization.Source: §6.1 · p. 10

6.2 Beam Prediction Accuracy

• 12.7–15.5° median beam-AoD error across four XR mobility types was lower than 802.11ay’s 14.9–21.7° and NN prediction’s 22.8–25.3°.At 100 ms, pose prediction error was 0.5–1.7 cm translation and 1.9–3.8° rotation; the two-UE full pipeline completed in ∼10.5 ms per scheduling window, versus over 1 s for NeRF2.Source: Figure 14 · p. 11, §6.2 · pp. 11–12
• 1.9° LoS and 2.8° NLoS median beam-AoD errors for RadioSight were below WRF-GS (9.0°/9.4°) and NeRF2 (7.9°/11.9°).The visually trained geometry and backward beam-tracing locate optimal transmitter AoD from reflection paths, which RF-supervised fields cannot recover.Source: Figure 15 · p. 11, §6.2 · p. 11

6.3 End-to-End Throughput Performance

• 3.84 Gbps median throughput for RadioSight exceeded vision-aided (3.55 Gbps), 802.11ay (3.21 Gbps), and NN-based (2.00 Gbps) baselines, a ∼1.9× gain over NN.At the 25th percentile, RadioSight sustained 3.39 Gbps versus 1.97, 1.60, and 1.37 Gbps for those baselines, respectively. Across sites, its median throughput was 3.7 Gbps in LoS and the hallway, 3.6 Gbps in the lobby and lecture room, and 2.3/1.7 Gbps in NLoS Tx1/Tx2.Source: Figure 16 · p. 12, §6.3 · p. 12

6.4 Dynamics and Handover

• RadioSight anticipated pedestrian blockage and reflector relocation to switch to an alternative NLoS path, while 802.11ay throughput repeatedly collapsed nearly to zero.During multi-AP handover, predicted trajectories enabled prescheduled transitions and smooth connectivity; 802.11ay waited for link degradation and produced throughput gaps.Source: Figure 17 · p. 12, §6.4 · p. 12

6.5 Impact of Antenna Array

• 76.8/87.7/90.9% multiplexing opportunity with LoS+NLoS paths exceeded LoS-only 14.5/38.6/50.9%, raising mean sum-rate to 6.0/6.9/7.3 from 4.4/5.3/5.8 Gbps.Across 4 × 4/6 × 6/8 × 8 arrays, 4 × 4-trained predictions had 1.2° median AoD error; 78/74/70% of positions lost under 1 dB of array gain.Source: Figure 18 · p. 13, §6.5 · p. 13

Evaluation setup

• Mobile XR sessions replay users walking through the environment while requesting cloud-rendered content, measuring throughput, stalls, and motion-to-photon latency for each method.QoE is computed as a weighted combination of throughput, stall rate, and motion-to-photon latency.Source: §7 · p. 13

Achieved XR QoE

• 82.3% of the oracle optimum: RadioSight improves QoE by 55% over the NN-based method, 25% over 802.11ay, and 10% over the vision-aided baseline, while reducing stalls by 3.6× versus the NN-based method.The gains are attributed to more accurate short-horizon throughput prediction and low-latency beam management during user motion.Source: §7 · p. 13, Figure 19 · p. 13

Throughput Prediction

• RadioSight yields the tightest throughput-prediction error distribution among the practical baselines, supporting more appropriate transmission settings and stable delivery under mobility.The error CDF is shown in Fig. 19(a).Source: §7 · p. 13, Figure 19 · p. 13

8 Conclusion

• RadioSight replaces reactive beam scanning with proactive beam optimization under user motion and blockages by synchronizing its Radio Field with XR dynamics and using backward beam-tracing.Source: §8 · p. 13

A.1 Segmentation on Multi-modal 3DGS

• Open-vocabulary 3D scene decomposition separates static backgrounds from movable entities using VLM-extracted keywords and a frozen CLIP text encoder with positive and generic negative vocabulary queries.Source: §A.1 · p. 15
• Each 3D Gaussian receives a view-independent semantic feature vector compressed to d=16, reflecting that an object's intrinsic semantics remain consistent across viewing directions.Source: §A.1 · pp. 15–16
• A shallow MLP maps the rendered compact features to final CLIP-aligned feature maps, bridging the dimensional gap to CLIP space.Source: §A.1 · p. 16
• For each Gaussian, RadioSight compares projected semantic features with encoded text embeddings by cosine similarity and applies a temperature-scaled softmax to produce vocabulary probabilities.Source: §A.1 · p. 16
• Gaussians exceeding the positive-query confidence threshold, such as τ=0.6, are segmented from the background and registered and tracked as independent physical obstacles in the digital twin.Source: §A.1 · p. 16

A.2 Multi-modal Training

• Sparse, discrete LiDAR geometry and noisy radio measurements make radio-only optimization ill-posed, producing cloud-like Gaussians that undermine physical ray back-tracing.Reliable ray back-tracing requires solid surfaces to estimate incidence, reflection, and diffraction.Source: §A.2 · p. 16
• RadioSight combines a scale penalty and normal-alignment loss to regularize the Gaussian representation toward continuous, rigid surfaces.The regularization objective is L_reg = λ_s L_scale + λ_n L_normal.Source: §A.2 · p. 16
• The scale loss penalizes each Gaussian’s shortest axis to form surfel-like disks, whose minimized-scale rotation axis defines the local microscopic normal.Rendered per-Gaussian normals are aligned with macroscopic normals derived from rendered-depth gradients.Source: §A.2 · p. 16
• RadioSight’s scale and normal constraints suppress volumetric ambiguity and outperform WRF-GS in high-fidelity radio-field reconstruction.The constraints flatten Gaussians into distinct surfels and align their orientations to support geometrically accurate physical boundaries.Source: §A.2 · p. 16

A.3 Single-Bounce Energy Fraction

• Validation combines measurements in an open lobby and furnished lecture room with seven indoor simulations in Sionna RT at 28 GHz, resolving geometrical paths with up to four interactions.The simulations include a digital twin of the measured lobby and six furnished WiSegRT rooms, including a storage hall whose surfaces are 64% metal.Source: §A.3 · pp. 16–17
• Multi-bounce paths contribute 1.8% in the measured lobby, 2.4% in its digital twin, 7.3% in the lecture room, and 9.8%–16.4% in furnished rooms; P90 reaches 25.5%–46.4%.The single-bounce assumption is well justified in open indoor environments and remains a reasonable approximation in furnished rooms, though modeling multi-bounce propagation could improve fidelity, especially at positions where higher-order interactions dominate.Source: Table 2 · p. 17, §A.3 · p. 17

A.4 Object Dynamic Modeling Details

• Clients detect depth discrepancies and send sparse 2.5D proposals for server-side 3D fusion, EKF tracking, obstacle registration, and scene updates.Each client maintains local geometry; the server associates proposals with tracked boxes, while newly confirmed obstacles receive conservative high RF opacity.Source: §A.4 · pp. 17–18
• ∼2.3 ms client processing latency enables RadioSight’s tiered dynamic-blocker tracking while the server maintains a radio field that evolves with the environment.Client pose prediction spans a 100 ms horizon with roughly 1–2 cm position error and a few degrees of orientation error.Source: §A.4 · pp. 17–18

A.5 MU-MIMO Beamforming

• The NP-hard MINLP is approximated by AoD-based FDMA isolation, RSSI-ranked sector scheduling, and LCMV null steering against co-scheduled users.The formulation jointly considers user scheduling, frequency allocation, and beamforming under rate, RF-chain, and transmit-power constraints.Source: §A.5 · pp. 18–19
• 0.24 ms suffices for MU-MIMO beam synthesis with K = 2 co-scheduled UEs using a closed-form LCMV solution.The small constraint matrix follows from a strictly bounded number of spatial streams per group.Source: §A.5 · p. 19

B.1 Beam Scanning Codebook Construction

• The static scanning codebook contains 17 Tx and 55 Rx beam combinations over a 90° field of view, with 45° elevation range and 360° azimuth range.Tx spacing is 22.5° in elevation and 45° in azimuth; Rx spacing is 15° and 20°, respectively.Source: §B.1 · p. 19

B.2 Training and Rendering Details

• Fields use degree-3 spherical harmonics and are optimized for up to 30k iterations on a single RTX 4090 with 24 GB of memory.Cold-start training uses 7k iterations and takes ∼1–2 min; inference queries the receiver pixel grid at resolution matched to array beamwidth.Source: §B.2 · p. 19

B Evaluation / B.3 Spectrum Prediction Similarity

• The appendices provide complementary evaluation results for §6.The spectrum-fidelity analysis reports PSNR 16.7, SSIM 0.302, and LPIPS 0.287 against ground truth; despite noisy, interference-affected spectra, the rendering captures dominant radio-path locations.Source: §B.3 · p. 19, B Evaluation · p. 19, Figure 20 · p. 20

B.4 Latency Analysis

• Beam management runs before frame transmission using predicted poses, and RadioSight dispatches optimized beam commands within a scheduling horizon that strictly exceeds T_bm.The latency analysis distinguishes beam management, scene update, and motion-to-photon as concurrent threads with non-additive totals.Source: §B.4 · p. 20
• T_bm ≈10.5 ms for two multiplexed UEs, dominated by rendering 10 × 1.0 ms; the adopted 50 ms horizon balances scheduling pressure against prediction accuracy.The horizon must exceed beam-management latency; longer horizons allow more multi-user scheduling but push predictions farther into the future.Source: §B.4 · pp. 20–21
• 34.0 ms live MTP latency (p99: 40.3 ms) was measured on the complete chain; the deployment budget is ≈29 ms at 4K 120 Hz.The prototype’s single-threaded software decoding costs 13 ms per frame, versus 2.5 ms for commercial hardware decoding.Source: §B.4 · p. 21
• T_update ≈3.8 ms in the deployment budget (2.3 ms processing plus the 5G uplink hop); prototype bench T_update is 2.3 ms with a host socket replacing that hop.Scene updates are asynchronous, so T_update adds to neither T_bm nor T_MTP.Source: §B.4 · p. 21

B.5 Per-Element Array Processing and Cross-Aperture Evaluation

• The public 28 GHz dataset provides per-element scans of a real 8 × 8 (64-element) array, recording each element’s complex channel response at every Rx position with responses phase-aligned across elements.Any centered n × n sub-array (n ≤ 8) extracted from these responses represents a real physical aperture of that size, obtained without separate captures.Source: §B.5 · p. 21
• The same 4 × 4-trained beam direction loses more gain to angular offset on 8 × 8 than on 4 × 4 (Fig. 18(a)).RadioSight predicts an aperture-independent beam direction, which is applied to the larger sub-arrays’ ground-truth spectra to measure gain lost relative to the optimum.Source: §B.5 · p. 21
• Two-user MU-MIMO scenarios are composed offline from separate single-UE traces, with scheduling and null-steering applied in post-processing.Phase-coherent dual-UE over-the-air transmission is left to future work.Source: §B.5 · p. 21