Source-linked AI summary
Heterogeneous Mixture-of-Experts for Energy-Efficient Multimodal ISAC in Highly Mobile Networks
Wenqi Fan, Ning Wei, Rongyan Xi, Ahmad Bazzi, Yue Xiu, Chadi Assi, Jing Dong, Jing Jin
TL;DR
The paper addresses the energy cost of continuous visual sensing and the reliability loss caused by delayed sensing in highly mobile V2I networks. It formulates a physics-aware M-ISAC problem using semantic AoI and solves the coupled temporal scheduling and spatial beamforming problem with RL-H-MoE. Simulations show lower system cost, ultra-low sensing error, and reliable physical-layer connectivity.
Problem
Continuous visual sensing consumes prohibitive computational energy, while delayed multimodal sensing worsens beam misalignment and couples temporal AoI evolution with non-convex spatial constraints.
Method
The physics-aware RL-H-MoE framework decouples temporal sensor scheduling from instantaneous spatial beamforming while modeling semantic AoI and constant modulus constraints.
Results
RL-H-MoE minimizes long-term system energy while maintaining ultra-low sensing error and reliable physical-layer link connectivity.
Takeaways & Limitations
Event-triggered multimodal sensing can jointly reduce energy cost and preserve tracking and communication reliability in highly mobile V2I networks.
Abstract
from arXiv · showhide
The integration of multimodal sensing and millimeter-wave (mmWave) communications is a key enabler for highly mobile vehicle-to-infrastructure (V2I) networks. However, continuous high-resolution visual sensing incurs prohibitive computational energy, while delayed sensing information worsens beam misalignment. In this paper, we establish a physics-aware multimodel integrated sensing and communication (M-ISAC) framework that quantifies the mathematical trade-off between sensing energy and communication reliability using the semantic age of information (AoI). To address the coupled challenges of temporal AoI evolution and instantaneous non-convex constant modulus constraints, we propose a novel reinforcement learning approach empowered by a heterogeneous mixture-of-experts (RL-H-MoE) architecture. By strictly decoupling the temporal scheduling and spatial phase mapping, the RL-H-MoE avoids prevalent gradient conflicts in multi-task learning. Extensive simulations demonstrate that the proposed architecture achieves an optimal event-triggered sensing policy, significantly minimizing the long-term system cost while guaranteeing ultra-low sensing errors and reliable physical-layer link connectivity.
I. INTRODUCTION
Highly mobile V2I networks need multimodal sensing for reliable mmWave beam alignment, but visual processing is energy-intensive and stale sensing worsens misalignment. The paper addresses these coupled temporal and spatial challenges with a physics-aware RL-H-MoE framework.
- I. INTRODUCTION: Continuous high-resolution visual processing incurs exorbitant computational energy, while delayed sensing information increases beam misalignment in resource-constrained V2I networks.
- I. INTRODUCTION: The physics-aware M-ISAC model integrates cross-layer semantic AoI evolution with physical-layer constant modulus constraints.
- I. INTRODUCTION: RL-H-MoE jointly optimizes beamforming and sensor scheduling for energy-efficient M-ISAC in high-mobility V2I networks.
- I. INTRODUCTION: An Actor-Critic training algorithm uses expected signal gain as physical prior knowledge to address the non-differentiable environment.
- I. INTRODUCTION: Extensive simulations show that H-MoE minimizes time-averaged system energy while guaranteeing physical-layer link reliability.
A. Communication and Channel Model
The model describes mmWave downlink communication from a base station to vehicles in a V2I multimodal ISAC scenario. It specifies received-signal effects and a geometric multipath channel based on physical propagation characteristics.
- A. Communication and Channel Model: The base station uses mmWave downlink communication to serve vehicles with time-varying beamforming vectors.The beamforming vector v_k(n) is defined for vehicle k at time slot n.
- A. Communication and Channel Model: The received signal includes the intended data symbol, additive white Gaussian noise, and multi-user interference from spatial beam leakage.The noise is modeled as z_k(n) ~ CN(0, σ^2), while the second signal term represents multi-user interference.
- A. Communication and Channel Model: The overall system is illustrated as a multimodal ISAC configuration in a vehicle-to-infrastructure scenario.The figure provides a system-level illustration of the multimodal ISAC setting.
- A. Communication and Channel Model: The geometric multipath channel models mmWave propagation through multiple paths, path gains, propagation delays, and angle-dependent array steering vectors.Each path is characterized by α_l,k(n), τ_l,k(n), and a(θ_l,k(n)), with θ_l,k(n) denoting the angle of departure.
B. Multimodal Sensing and System Energy Model
The M-ISAC model combines low-power radar with energy-intensive visual sensing, linking event-triggered visual activation to semantic AoI, beam misalignment, channel uncertainty, and total energy consumption.
- Multimodal sensing: Visual sensing provides precise semantic context, while radar supplies low-power but noisy real-time angle and distance estimates for each vehicle.The binary decision πk(n) activates the visual expert for spatial calibration when needed.
- System energy: System energy combines deterministic visual-processing energy with expected RF recovery energy incurred after beam tracking failure.The visual cost depends on the activation decision and the recovery cost depends on beam misalignment probability.
- Semantic AoI: Semantic AoI increases by one when visual sensing is skipped and resets to the processing delay Tproc when visual sensing is activated.This models the staleness introduced by intermittent visual sensing.
- Beam misalignment: Beam misalignment probability increases monotonically with semantic AoI through Pmisa,k(n) = 1 − exp(−βk · Ak(n)), where βk is the vehicle’s kinematic uncertainty rate.Semantic AoI acts as a proxy for spatial uncertainty in highly dynamic V2I networks.
- Channel uncertainty: The model couples delayed sensing to beamforming by making channel-error variance E[∥∆hk(n)∥2] = ρ · Pmisa,k(n), so skipped scheduling degrades effective beamforming gain.The estimated channel is based on delayed multimodal sensing, while ∆hk(n) captures misalignment-induced estimation error.
D. Problem Formulation
The optimization minimizes long-term average system energy while enforcing communication reliability, analog beamforming hardware constraints, transmit-power limits, safety margins, and binary sensing decisions.
- Objective: The objective minimizes the long-term time-averaged combined energy of camera activation and anticipated RF recovery.The formulation focuses solely on the system’s energetic footprint in Joules.
- Reliability constraints: Reliability is enforced through a maximum beam-misalignment threshold and a safety margin that proactively prevents catastrophic tracking loss.These requirements constrain the tracking accuracy needed for proper beam alignment.
- Hardware constraints: Analog phase shifters impose a constant modulus constraint, making beamforming non-convex, while aggregate transmit power is also restricted.The resulting formulation combines hardware limitations with stringent reliability requirements.
- Scheduling constraint: Binary sensor activation creates a discrete scheduling constraint, producing a highly complex hybrid action space.This constraint is coupled with the continuous beamforming design.
III. PROPOSED HETEROGENEOUS MOE ARCHITECTURE
The RL-H-MoE framework decouples long-term temporal scheduling from instantaneous non-convex spatial mapping by assigning each process to a specialized expert.
- Architecture motivation: RL-H-MoE addresses the intractability caused by coupling semantic-AoI evolution with instantaneous non-convex spatial mapping.The framework is designed to decouple these temporal and spatial processes.
- Temporal Constraint Expert: The LSTM-based Temporal Constraint Expert analyzes historical sequences and outputs discrete, event-triggered sensor activation decisions.It infers kinematic inertia from temporal information.
- Non-Convex Solver Expert: The MLP-based Non-Convex Solver Expert operates in the spatial domain and maps features onto the feasible analog-phase-shifter manifold.This specialization targets the constant modulus constraint directly.
A. Temporal Constraint Expert (LSTM-based)
The temporal expert tracks evolving spatial uncertainty and proactively schedules visual calibration, while the spatial expert independently generates feasible phase mappings optimized for signal gain.
- A. Temporal Constraint Expert (LSTM-based): The LSTM expert monitors AoI and accumulated kinematic errors to anticipate when reliability safety margins may be violated.Its hidden state learns boundary conditions and triggers visual calibration before AoI exceeds the allowable threshold.
- Spatial phase mapping: The spatial MLP expert is isolated from temporal scheduling to prevent gradient conflicts and outputs phase angles that satisfy the constant modulus constraint.Phase-angle outputs avoid unconstrained complex values that could violate analog hardware limits.
- Beamforming reconstruction: The final layer produces the phase-shift matrix Φn, from which each continuous beamforming vector is deterministically reconstructed.The reconstruction uses an exponential mapping.
- Training procedure: The MLP is optimized using a physical-channel-gain-driven spatial loss and updated episodically at trajectory ends rather than every time step.This design projects the generated beamforming matrix onto the feasible hardware manifold.
C. Markov Decision Process (MDP) Formulation and Gradient Decoupling Proof
The MDP jointly represents discrete sensor scheduling and continuous beamforming, while RL-H-MoE separates temporal scheduling gradients from spatial optimization to avoid multi-task interference.
- MDP Formulation: The action combines discrete sensor scheduling π(n) with continuous beamforming v(n), capturing the joint control problem.The MDP is defined over state, action, reward, and transition components.
- Gradient Decoupling Proof: The architecture physically separates temporal constraints and non-convex spatial mapping, with isolated backpropagation paths that circumvent negative transfer.The upper and lower branches correspond to temporal and spatial processing, respectively.
- MDP Formulation: The reward penalizes total energy and violations of the physical-layer reliability threshold.The reliability penalty uses a positive coefficient λ_misa.
- Gradient Decoupling Proof: RL-H-MoE decouples temporal LSTM scheduling from spatial MLP mapping, preventing high-frequency spatial gradients from disrupting long-term temporal memory.The total objective uses separate parameter vectors for the temporal and spatial experts.
- Gradient Decoupling Proof: Although scheduling and beamforming remain physically coupled through AoI-induced channel error, gradient isolation is stated to guarantee stable convergence.The spatial expert is driven by physical channel gain without passing its gradient through the temporal network.
IV. SIMULATION RESULTS AND ANALYSIS
The evaluation uses Monte Carlo simulations against five baselines and trains RL-H-MoE with separate temporal and spatial updates under specified learning and physical-energy settings.
- Simulation Setup: The experiments compare RL-H-MoE with Vision-Only, Radar-Only, Standard PPO, Homogeneous MoE, and an RL-H-MoE without AoI ablation.The baselines cover sensing extremes, a monolithic RL model, a structurally homogeneous MoE, and an AoI ablation.
- Training Procedure: Training performs temporal scheduling inference with an LSTM and spatial beam-phase inference with an MLP before executing constrained actions.The algorithm observes rewards and channels, computes scheduling advantages, and then forms separate temporal and spatial losses.
- Training Procedure: RL-H-MoE updates temporal and spatial parameters independently using distinct learning rates and isolated gradients.The temporal parameters follow the scheduling loss, while spatial parameters follow the physics-informed spatial loss.
- Simulation Setup: The system uses a 64-element ULA at 28 GHz for four vehicles, with a 30 dBm transmit-power budget and heavily penalized visual processing energy.Continuous visual activation costs 40 Joules per slot in the stated setup.
A. Trade-off Between System Energy and Sensing Reliability
At SNR = 10 dB, the proposed RL-H-MoE achieves low sensing error while reducing time-averaged energy relative to sensing and learning baselines.
- Baseline Comparisons: Vision-Only reaches the lowest sensing MAE but incurs exactly 40 Joules because it continuously processes high-resolution images.This establishes the energy cost of maintaining the strongest sensing accuracy in the comparison.
- Overall Trade-off: Over 40% energy savings versus Vision-Only accompany sensing MAE nearly matching Vision-Only, with energy reduced to the 20–25 Joules region.The result is reported for steady-state tracking at SNR = 10 dB.
- Baseline Comparisons: Radar-Only develops diverging kinematic errors and exceeds 60 Joules on average after beam failures trigger exhaustive RF beam sweeping.The absence of semantic visual calibration increases spatial uncertainty and beam-recovery cost.
- Baseline Comparisons: Standard PPO and Homogeneous MoE stagnate near 30 Joules because gradient conflicts impair joint spatial mapping and temporal scheduling.The reported conflict is between high-frequency spatial optimization and long-term temporal scheduling.
- Ablation: Removing AoI causes the ablation to fail, indicating that AoI information is required for inferring kinematic inertia in the evaluated policy.The claim is specific to the reported ablation study.
B. Robustness Evaluation Across Varying RF Environments
Across RF SNRs from 0 to 20 dB, RL-H-MoE adapts sensing effort to reduce long-term energy while maintaining robust tracking and link reliability.
- At 20 dB, long-term system energy reaches as low as 20 Joules as RL-H-MoE curtails unnecessary visual computations.The agent relaxes its AoI tolerance as radar tracking becomes more reliable at higher SNRs.
- Across varying RF SNRs, the proposed architecture minimizes time-averaged system energy while guaranteeing physical-layer link reliability.
- The RL-H-MoE agent demonstrates environmental adaptability across RF SNRs ranging from 0 to 20 dB.
- Simulations report superior robustness and reduced total system cost while strictly guaranteeing tracking accuracy for 6G ISAC networks.