PaperlayerView exact revision on arXiv

Source-linked AI summary

RadioSight: Predictive mmWave XR Network Optimization from Dynamic Neural Radio Fields

Lihao Zhang, Paul Kudyba, Zhenlin An, Haijian Sun

arXiv:2608.29504v1cs.NI

TL;DR

mmWave XR links are highly directional and fragile under mobility and blockage, while prior neural radio-field work largely focuses on static, offline reconstruction. RadioSight builds a real-time multimodal radio twin with backward beam-tracing and lightweight semantic synchronization to predict future beams. In real-world evaluations, it reduces beam alignment errors by up to ∼50% and improves median throughput by up to 2×.

  • Problem

    Existing neural radio fields mainly support static, site-specific offline channel reconstruction, while dynamic mmWave XR links require predictive beam control under mobility and blockages.

  • Method

    RadioSight combines a continuously updated multimodal 3DGS radio twin, backward beam-tracing, and semantic object synchronization to predict future transmitter beams without full retraining.

  • Results

    Up to ∼50% lower beam alignment errors and up to 2× higher median throughput are reported in diverse real-world XR mobility scenarios.

  • Takeaways & Limitations

    RadioSight replaces reactive beam scanning with proactive beam optimization under user motion and blockages.

  • Takeaways & Limitations

    The backward beam-tracing method assumes single-bounce propagation, although the experiments suggest single-bounce reflections generally dominate typical indoor mmWave NLoS propagation.

Abstract

from arXiv · show

Next-generation extended reality (XR) networks rely on mmWave communication for multi-gigabit throughput, yet highly directional links are vulnerable to user mobility and blockages, causing frequent outages under reactive beam management. Emerging neural radio fields can predict radio propagation, but prior work remains limited to offline channel reconstruction. We introduce RadioSight, a real-time multi-modal radio field system for predictive mmWave optimization and proactive Multi-User MIMO beamforming. RadioSight combines backward beam-tracing with real-time semantic object synchronization to anticipate RF geometry changes without full model retraining. Implemented as an edge-executable pipeline for commercial 28 GHz arrays, RadioSight determines each scheduling window's beams during the preceding window without exhaustive beam sweeps. Experiments show that RadioSight reduces beam-search error by up to ~50%, improves median throughput by 2x, and enhances link stability.

1 Introduction

RadioSight addresses mmWave XR fragility by maintaining a dynamic, multimodal radio twin that predicts future channel states and proactively selects beams before mobility or blockages cause degradation. Its edge-executable pipeline combines geometric backward beam-tracing, semantic synchronization, and predictive cross-layer control, improving beam alignment and throughput in real-world evaluations.

  • Motivation: mmWave XR links support multi-gigabit applications but are vulnerable to motion, head rotation, and environmental dynamics that can invalidate optimal beams.Reactive systems respond with exhaustive scanning after disconnection, introducing latency that degrades immersive experiences.
  • Limitations of Prior Work: Existing neural radio fields mainly reconstruct static sites offline, while dynamic systems face costly retraining and often recover only receiver-side spectra.These limitations prevent direct inference of transmitter-side beams needed for proactive MU-MIMO optimization.
  • RadioSight Approach: RadioSight builds a continuously updated multimodal radio twin using visual and radio observations with an efficient 3DGS representation and lightweight edits instead of full retraining.Semantic features segment movable objects, while explicit geometry supports downstream path reasoning.
  • RadioSight Approach: Backward beam-tracing follows multipath arrivals from receiver to transmitter to infer dynamics-aware optimal AoDs without exhaustive channel sounding.The method uses accurate scene geometry to identify transmitter-side beam candidates, including NLoS reflection paths.
  • Proactive Optimization: Predictive beam management uses future trajectories and synchronized radio-field states to switch paths and adapt rates before blockages or LoS-to-NLoS transitions.Beam decisions for the upcoming horizon are computed during the preceding scheduling window.
  • Results: Up to ∼50% lower beam alignment errors and up to 2× higher median throughput are reported in diverse real-world XR mobility scenarios.The evaluation also reports eliminated rescans and disconnections inherent to reactive schemes.

2 Related Work

Prior work spans neural radio-field reconstruction, XR link prediction, and beam management, but the surveyed approaches generally lack dynamic adaptation, transmitter-side beam inference, or robust blockage handling. RadioSight is positioned against these gaps through predictive beam optimization under dynamic LoS and NLoS conditions.

  • Neural Radio Fields: Neural radio-field methods synthesize radio signals across 3D space, but existing approaches target receiver-side spectra and static environments.They cannot directly derive transmit-side AoDs for MU-MIMO under NLoS reflections and may require costly retraining for dynamic occlusions.
  • Positioning: RadioSight is compared with state-of-the-art systems as a real-time predictive optimization engine based on multimodal radio fields.Its stated scope combines synchronized environmental dynamics with backward beam-tracing for proactive beam optimization.
  • XR mmWave Network Systems: XR network systems predict throughput, body-induced blockage, or mobility-related signal drops using historical link data, pose tracking, or multipath networking.These methods address different aspects of XR connectivity rather than constructing a synchronized physical radio environment.
  • mmWave Beam Management: Standard mmWave protocols rely on periodic exhaustive beam sweeps, while prediction- and IMU-based methods degrade under mobility or remain blind to newly appearing environmental blockers.The related-work comparison motivates methods that incorporate both user movement and scene changes.

3 System Overview

RadioSight uses a client–edge architecture in which reported XR dynamics update a shared radio-field digital twin, while the edge server predicts future states and derives beams. Parallel threads separate scene synchronization, beam management, and XR content delivery so beam computation uses the latest available scene without blocking content processing.

  • Radio Field Digital Twin: The Radio Field Digital Twin resolves valid transmitter-to-receiver propagation paths at future UE poses while tracking UE pose, transient blockers, and registered movable objects.Future-state path resolution supports beam decisions for the next scheduling window rather than the current state.
  • Client–Edge Architecture: The XR Client reports 6DoF pose, detected-blocker bounding boxes, and optional sparse spectrum scans to the Edge Server over a 2.4 GHz control channel.The directional mmWave link simultaneously carries the scheduled XR video stream.
  • Client–Edge Architecture: The Edge Server fuses client reports into a shared dynamic scene, updates tracked-object state, synchronizes it to UEs, and derives beam and rate-adaptation commands.It renders predicted spatial spectra, extracts beam candidates, and computes beamforming decisions.
  • Parallel Threads: Parallel scene-update, beam-management, and motion-to-photon threads keep the twin synchronized, compute upcoming beams, and deliver XR content independently.The architecture lets beam management read the most recently updated scene without blocking on scene updates or content delivery.
  • Beam Management: The beam-management thread predicts five 10 ms frames across the next 50 ms window for two multiplexed UEs, with approximately 10.5 ms total beam-management latency.Its work includes spectrum rendering, transmitter-side back-tracing, and MU-MIMO beamforming synthesis.

4 System Design

RadioSight builds a multi-modal 3DGS radio field that jointly represents geometry, visual appearance, radio propagation, semantics, and depth for dynamic beam optimization. Its rendering and backward beam-tracing pipeline converts predicted receiver-side spectra into transmitter-side beam directions while supporting efficient updates for movable objects.

  • Multi-modal radio field: RadioSight extends neural radio fields into multi-modal 3DGS primitives that share geometry across RGB, radio, and semantic features.The shared representation augments each Gaussian with modality-specific channels while retaining center, covariance, and opacity.
  • Field rendering: The differentiable renderer synthesizes receiver-side radio spatial spectra over angle of arrival alongside RGB, semantic, and depth outputs.The radio spectrum captures multipath signal distribution and intensity across AoAs for beam prediction.
  • Multi-modal radio field: RGB supervision anchors Gaussian geometry to physical surfaces, while semantic embeddings cluster movable objects for targeted dynamic updates.The system renders RGB, radio, semantic, and depth maps from the same representation; semantic features identify movable entities.
  • Geometry regularization: Geometry regularization forces Gaussians toward thin surfel-like shapes, preventing volumetric ambiguity that would disrupt downstream ray back-tracing.Without regularization, primitives can degenerate into volumetric blobs that do not align with physical surfaces.
  • Backward beam-tracing: Backward beam-tracing identifies dominant receiver-side AoA peaks, estimates radio-surface intersections, and traces them to the transmitter to derive optimal AoDs.This bridges the missing transmitter-side information and enables proactive beam scheduling without exhaustive channel-sounding scans.
  • Dynamic radio field: RadioSight updates dynamic scenes by synchronizing visual-geometric changes and tracking transient blockers with lightweight 3D bounding boxes instead of retraining the full field.The continuously synchronized radio twin is then queried using predicted user and object positions for proactive optimization.

LoS NLoS

RadioSight converts predicted trajectories and dynamic radio-field queries into proactive beam scheduling, bitrate adaptation, handovers, and MU-MIMO control. Its architecture combines geometric beam tracing, semantic synchronization, and collaborative sensing for changing XR environments.

  • MU-MIMO: For MU-MIMO, the server jointly optimizes beam-steering vectors across users to maximize sum-rate while minimizing inter-user interference through null-steering.The batch-query results support joint scheduling and beamforming for users sharing time-frequency resources.
  • Beam Tracing: Backward beam-tracing uses multimodal geometry to identify transmitter-side paths needed for beam optimization.The system combines visual and semantic features with radio information to support geometric editing and path tracing.
  • Cross-Layer Control: The edge server can proactively reduce a UE’s video resolution when predicted SINR falls below the QoE threshold.It uses codebook-based synthesis or LCMV to form beam patterns before anticipated motion or blockage.
  • Proactive Handover: Candidate APs query local radio twins in parallel so target beams are prepared before handover while the source AP maintains the link.This replaces signal-drop-triggered searches with coordinated proactive scheduling.
  • Collaborative Sensing: Collaborative sensing fuses visual proposals from multiple XR users and infrastructure sensors to mitigate individual field-of-view blind spots.The resulting scene representation supports beam decisions under rapid motion and occlusions.

5 Implementation

RadioSight is implemented as an integrated XR client, radio interface, edge pipeline, and 28 GHz testbed. Its experiments span diverse indoor geometries and mobility traces representative of XR use.

  • Hardware: The prototype integrates a radio-field engine with a real-world 28 GHz mmWave testbed.The client uses Meta Project Aria glasses and a local Intel NUC11 i5 for RGB and 6DoF pose streaming.
  • Hardware: The hardware deployment consists of an on-glass mmWave client and an mmWave edge node.This deployment provides the physical client and edge components for the testbed.
  • Experimental Scenarios: Experiments cover direct-LoS, two NLoS layouts, a bent corridor, and mobility traces from walking, gaming, navigation, and interaction.The scenarios vary substantially in geometry and material and include 28 GHz channel measurements.

6 Evaluation

RadioSight is evaluated against reactive, neural, and vision-aided baselines for beam accuracy, throughput, dynamic events, array transfer, and MU-MIMO. Results show lower beam error, real-time execution, higher throughput, and stronger multiplexing under challenging conditions.

  • Static Beam Accuracy: 1.9° LoS and 2.8° NLoS median beam-AoD errors demonstrate RadioSight’s static accuracy.The system outperforms NeRF2 and WRF-GS in both LoS and NLoS regions.
  • Mobility: Up to ∼50% lower beam error than the NN-based predictor is achieved across four mobility traces.RadioSight’s median beam-AoD error ranges from 12.7–15.5°, while the NN baseline ranges from 22.8–25.3°.
  • Runtime: ∼10.5 ms per scheduling window enables real-time beam management for two multiplexed UEs, whereas NeRF2 requires over 1 s.The measured pipeline includes rendering, back-tracing, and MU-MIMO optimization.
  • Throughput: 3.84 Gbps is RadioSight’s highest median throughput, versus 2.00 Gbps for the NN-based baseline.The corresponding median gain over the NN predictor is ∼1.9×, with the largest separation in blocked or reflective regions.
  • Dynamics and Handover: RadioSight maintains continuous connectivity and stable throughput during dynamic blockage and multi-AP handovers instead of waiting for link failure.The evaluation includes semantic-driven movable-object and transient-blocker modeling.
  • MU-MIMO and Array Impact: 76.8/87.7/90.9% spatial-multiplexing opportunity with LoS+NLoS paths raises mean sum-rate to 6.0/6.9/7.3 Gbps for 4 × 4/6 × 6/8 × 8 arrays.The corresponding LoS-only multiplexing opportunities are 14.5/38.6/50.9%, with mean sum-rates of 4.4/5.3/5.8 Gbps.

7 Case Study: XR Streaming QoE

The XR streaming case study evaluates delivered throughput, stalls, and motion-to-photon latency through a weighted QoE score. RadioSight achieves higher QoE and fewer stalls by combining accurate short-horizon throughput prediction with low-latency beam management.

  • Evaluation Setup: The end-to-end study replays mobile XR sessions and scores performance using throughput, stall rate, and motion-to-photon latency.Users walk through the environment while requesting cloud-rendered content.
  • Achieved XR QoE: 82.3% of the oracle optimum is achieved by RadioSight, improving QoE by 55% over the NN-based method.It also improves QoE by 25% over 802.11ay and 10% over the vision-aided baseline.
  • Achieved XR QoE: 3.6× fewer stalls than the NN-based method are achieved during mobile XR streaming.The results are attributed to more accurate short-horizon throughput prediction and low-latency beam management.
  • Throughput Prediction: RadioSight produces the tightest throughput-prediction error distribution among the practical baselines.This supports more appropriate transmission-parameter selection and more stable delivery under mobility.

8 Conclusion

RadioSight replaces reactive beam scanning with proactive beam optimization by synchronizing a multi-modal radio field with XR-observed dynamics and using backward beam-tracing.

  • RadioSight synchronizes the Radio Field with XR-observed dynamics to support predictive mmWave optimization under user motion and blockages.
  • Open-vocabulary 3D scene decomposition isolates static backgrounds from movable entities using visual-language features and vocabulary queries.
  • Semantic features are compressed to d = 16 and rendered view-independently before an MLP produces CLIP-aligned feature maps.
  • Gaussians exceeding the confidence threshold τ = 0.6 for a positive query are segmented and registered as distinct entities.

A.2 Multi-modal Training

RadioSight adds geometric regularization to prevent radio-only Gaussian representations from becoming volumetric and physically ambiguous, enabling accurate surface-aligned radio modeling.

  • Sparse LiDAR initialization plus radio-spectrum-only training creates an ill-posed problem in which Gaussians can expand into volumetric clusters instead of physical boundaries.
  • RadioSight combines a scale penalty and normal alignment loss as L_reg = λ_sL_scale + λ_nL_normal to enforce continuous rigid surfaces.
  • Scale regularization penalizes each Gaussian’s shortest axis, encouraging flat surfel-like disks that represent physical boundaries.
  • The shortest-axis direction defines each Gaussian’s microscopic normal from the corresponding column of its rotation matrix.
  • Normal alignment matches per-Gaussian normals with macroscopic geometry derived from rendered depth maps.
  • With these constraints, RadioSight suppresses volumetric ambiguity and significantly outperforms WRF-GS in high-fidelity radio-field reconstruction.

A.3 Single-Bounce Energy Fraction

RadioSight combines dynamic object perception, server-side state fusion, radio-field querying, and MU-MIMO optimization for predictive networking; its single-bounce assumption is supported across tested indoor environments.

  • A.3 Single-Bounce Energy Fraction: The evaluation spans measured lobby and lecture-room environments plus six WiSegRT indoor scenes simulated at 28 GHz with up to four path interactions.
  • A.3 Single-Bounce Energy Fraction: Single-bounce paths account for almost all NLoS energy in open scenes, while multi-bounce contributions reach 7.3% in a furnished lecture room and 9.8%–16.4% in six furnished rooms.
  • A.4 Object Dynamic Modeling Details: The client predicts UE pose over a 100 ms horizon using a kinematic model, maintaining roughly 1–2 cm position error and a few degrees of orientation error.
  • A.4 Object Dynamic Modeling Details: Client depth discrepancies produce sparse 2.5D proposals, which the server back-projects and fuses into globally consistent 3D tracked states.
  • A.4 Object Dynamic Modeling Details: The tiered architecture tracks dynamic blockers with ∼2.3 ms client processing latency while maintaining a high-fidelity evolving server radio field.
  • A.5 MU-MIMO Beamforming: The edge server formulates joint MU-MIMO scheduling, frequency allocation, and beamforming as a constrained MINLP with rate, RF-chain, power, and assignment limits.
  • A.5 MU-MIMO Beamforming: The MU-MIMO beam-synthesis pipeline completes in 0.24 ms for K = 2 co-scheduled UEs.
  • Rendered spectrum prediction achieves PSNR 16.7, SSIM 0.302, and LPIPS 0.287 against ground truth.

B.4 Latency Analysis

RadioSight separates concurrent beam-management, scene-update, and motion-to-photon pipelines rather than adding their latencies, enabling predictive beam decisions within a scheduling horizon.

  • Beam management renders predicted future poses, extracts beam candidates, optimizes global beamforming, and dispatches commands before transmission.
  • T_bm ≈ 10.5 ms for two multiplexed UEs, with rendering dominating beam-candidate extraction and MU-MIMO optimization.
  • The 50 ms horizon trades greater multi-user scheduling room against reduced dynamic-prediction accuracy farther into the future.
  • T_update ≈ 3.8 ms including the 5G uplink hop, while the bench prototype measures 2.3 ms processing latency.
  • Scene updates are asynchronous, so T_update adds to neither T_bm nor T_MTP.

B.5 Per-Element Array Processing and Cross-Aperture Evaluation

RadioSight uses phase-aligned per-element measurements from an 8 × 8 array to evaluate centered sub-arrays and cross-aperture beam-direction generalization. The radio field is trained on 4 × 4 data, while larger apertures provide narrower beams and higher gain but incur greater loss from angular offsets.

  • Per-element array processing: The public 28 GHz dataset contains phase-aligned complex responses for every element of a real 8 × 8 array at each receiver position.Centered n × n sub-arrays with n≤8 can therefore represent real physical apertures without separate captures.
  • Per-element array processing: Larger apertures yield narrower main lobes and higher gain.
  • Cross-aperture evaluation: RadioSight’s radio field is trained only on the 4 × 4 sub-array and predicts an aperture-independent beam direction.The same predicted direction can steer any aperture.
  • Cross-aperture evaluation: Cross-aperture evaluation applies the 4 × 4-trained direction to ground-truth spectra from larger sub-arrays and records gain lost from angular offset.
  • Cross-aperture evaluation: The same angular offset costs little on 4 × 4 but more on 8 × 8.
Loading 2608.29504v1…