PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Beam Prediction for V2X Networks with Multi-Modal Sensing

Chen Shang, Dinh Thai Hoang, Diep N. Nguyen, Jiadong Yu

arXiv:2609.10200v1cs.LG

TL;DR

Reliable V2X beam prediction is difficult when RF sensing degrades and sensing modalities are incomplete. The paper proposes BeamTransFuser, which progressively fuses camera, LiDAR, radar, and GPS features and generates missing modality features; experiments show stronger overall performance and improved incomplete-sensing robustness.

  • Problem

    Existing sensing-assisted beam prediction relies heavily on RF sensing, which can become unreliable in complex vehicular environments, while incomplete sensing remains a practical deployment condition.

  • Method

    BeamTransFuser uses hierarchical Transformer-based fusion of camera, LiDAR, radar, and GPS observations, with a generative module reconstructing missing modality features.

  • Results

    BeamTransFuser ranks first overall with a DBA-score of 0.9129, while CVAE-generated features improve Top-1 accuracy from 6.67% to 45.51% with missing radar and to 56.18% with missing LiDAR.

  • Takeaways & Limitations

    The framework provides more stable beam prediction across evaluated conditions and improves robustness when radar or LiDAR observations are unavailable.

Abstract

from arXiv · show

Integrated sensing and communication (ISAC) provides a promising foundation for beam prediction in future vehicle-to-everything (V2X) networks. However, existing sensing-assisted beamforming methods still rely heavily on radio-frequency sensing, which may become unreliable in complex vehicular environments. Meanwhile, the growing availability of heterogeneous sensors, such as cameras and LiDAR, offers new opportunities to improve beam prediction through richer environmental perception. Motivated by this, this paper proposes a multi-modal beam prediction framework for V2X networks. Specifically, we develop BeamTransFuser, a hierarchical Transformer-based architecture that progressively fuses camera, LiDAR, radar, and GPS observations for robust beam prediction. In addition, to handle possible missing modalities in practical deployment, we introduce a generative module that reconstructs missing modality features from the available observations. Experimental results on a real-world multi-modal V2X dataset show that the proposed framework consistently outperforms representative baselines, while the generative module further improves robustness under incomplete sensing conditions.

I. INTRODUCTION

Practical V2X beam prediction is challenged by RF-sensing degradation and trade-offs among sensing accuracy, communication efficiency, and implementation complexity. The paper addresses this with BeamTransFuser and modality generation, using heterogeneous sensing to improve robustness under complete and incomplete observations.

  • Motivation: Existing ISAC designs face trade-offs among sensing accuracy, communication efficiency, and implementation complexity.Joint-design approaches additionally require sophisticated processing, tighter synchronization, and higher hardware overhead.
  • Motivation: RF sensing can degrade under blockage, multipath, and NLoS conditions, making RF-only beam prediction insufficient for realistic V2X scenarios.These impairments reduce sensing fidelity and beam-alignment reliability under high mobility and changing road conditions.
  • Proposed framework: BeamTransFuser jointly leverages camera, LiDAR, radar, and GPS through hierarchical Transformer-based cross-modal fusion.The architecture extracts modality-specific features and progressively models complementary information across sensing sources.
  • Proposed framework: A modality generation mechanism reconstructs missing modality features from available sensing inputs for incomplete-sensing conditions.The mechanism is intended to support beam prediction when part of the sensing input is unavailable.
  • Evaluation: Experiments on a real-world multimodal V2X dataset show consistent gains over representative baselines, including under incomplete sensing.The generative module further improves robustness by reconstructing missing modality features.

II. SYSTEM OVERVIEW AND PROBLEM FORMULATION

The system uses an RSU to collect heterogeneous observations and predict a beam index instead of exhaustively searching the beam codebook online. The formulation maps multimodal sensing samples to beam-index labels through a learned prediction model.

  • System overview: The RSU integrates GPS, camera, LiDAR, and radar observations to obtain real-time environmental awareness for beam prediction.These modalities provide complementary information under diverse propagation and mobility conditions.
  • Problem formulation: The system uses a predefined codebook of M candidate beamforming vectors, with each vector applied to the downlink symbol.The received signal is modeled using the selected beamformer, channel vector, transmitted symbol, and complex Gaussian noise.
  • System overview: The RSU processes real-time observations with a pretrained model before determining the beamforming decision.This processing-and-decision flow is summarized in Fig. 1.
  • Problem formulation: Multi-modal sensing observations are directly used to infer the target beam index rather than performing exhaustive online beam search.The learning task uses sensing samples X, beam-index labels Y, and a parameterized prediction model M_Θ.
  • Problem formulation: The core formulation challenge is extracting useful representations from heterogeneous modalities and fusing them into a unified feature space for accurate beam prediction.This motivates the BeamTransFuser framework introduced next.

III. THE PROPOSED MULTI-MODAL FRAMEWORK

BeamTransFuser combines four modality-specific branches with a four-stage hierarchical Transformer fusion backbone. Progressive fusion and residual connections preserve modality-specific information while modeling cross-modal dependencies across representation levels.

  • Architecture: BeamTransFuser contains dedicated branches for camera, LiDAR, radar, and GPS inputs followed by hierarchical Transformer-based fusion.Each branch first learns modality-aware features before cross-modal interaction begins.
  • Architecture: Fig. 2 depicts the BeamTransFuser architecture as modality-specific branches connected by hierarchical cross-modal fusion.The architecture progressively integrates the sensing streams rather than fusing them only once.
  • Architecture: The four-stage backbone interleaves multimodal fusion blocks between adjacent stages to model cross-modal dependencies at multiple representation levels.Residual connections retain modality-specific information and improve training stability.

A. Multi-Modal Branches

The framework uses modality-specific encoding before progressively aligning and fusing representations. Spatial sensors are encoded into feature maps, while GPS is projected into the shared feature space through an MLP.

  • Modality-specific encoding: Camera, LiDAR, and radar use dedicated convolutional encoders, whereas GPS is mapped into the common feature space through an MLP.This reflects the spatial structure of the first three modalities and the low dimensionality of GPS.
  • Modality-specific encoding: Encoded modality representations are aligned to a common scale and converted into token sequences for progressive Transformer-based fusion.The design retains modality-specific characteristics while integrating complementary information across sensing sources.

B. Multi-Modal Fusion Block

The multi-modal fusion block converts heterogeneous sensing features into a shared token representation and uses Transformer attention to model cross-modal dependencies across progressive stages.

  • Modality-specific features are pooled and projected into common tokens before concatenation into a unified multimodal sequence.
  • Learnable projections form query, key, and value matrices for scaled dot-product attention.
  • Attention updates each token by selectively aggregating informative features from other modalities in the shared sequence.
  • After fusion, camera, LiDAR, and radar tokens become feature maps, whereas GPS tokens remain in token form for subsequent processing.

C. Beam Generator and Model Training

After the final fusion stage, modality features are adaptively combined and passed to an MLP beam generator that produces scores for codebook beam selection.

  • Softmax-normalized weights adaptively adjust each modality's contribution when globally pooled features are integrated.
  • The final fused representation is passed to a compact MLP-based beam generator to produce a beam score vector.
  • The predicted beam index selects the codebook beamforming vector with the largest score.
  • More accurate beam prediction directly translates into better beam alignment and improved communication performance.

IV. ENHANCING BEAMTRANSFUSER ROBUSTNESS VIA MODALITY GENERATION

A generative module improves BeamTransFuser operation when sensing modalities are missing by conditionally reconstructing unavailable modality representations from available observations.

  • The missing-modality module infers an unavailable sensing representation from remaining observations so beam prediction can continue with incomplete inputs.
  • The generation mechanism uses a VAE-based conditional model to generate a missing modality from available modalities and latent variables.
  • The conditional model represents the missing modality distribution using available observations as context rather than an unconditional prior.
  • Task-aware fine-tuning jointly optimizes reconstruction and downstream beam prediction losses after initial CVAE training.
  • A fixed isotropic Gaussian prior keeps the generative module lightweight for delay-sensitive V2X deployment, while simplifying the latent-prior modeling.
  • At inference, the retained decoder reconstructs the missing modality from available inputs and a sampled latent variable before feature-level injection into BeamTransFuser.

V. PERFORMANCE EVALUATION

The evaluation uses a real-world multi-modal V2X dataset and examines both training behavior and beam-prediction performance under the stated experimental setup.

  • The DeepSense 6G dataset provides synchronized camera, LiDAR, radar, GPS, and beam-label data for real-world multi-modal V2X evaluation.
  • Training and validation losses decrease steadily and then stabilize without noticeable later validation-loss increase.
  • 11,243 samples are obtained after combining development and adaptation subsets, with 90% used for training and 10% for validation.
  • The evaluation uses DBA-score and Top-k accuracy as the main beam-prediction metrics.

B. Experimental Results

BeamTransFuser achieves the highest overall DBA-score and remains stable across diverse urban scenarios. Its performance reflects complementary multimodal sensing and hierarchical fusion, while training converges without evident overfitting.

  • Convergence Behavior and Training Time: Training and validation losses steadily decrease and stabilize, with optimization largely converged after approximately one hour.Further training yields only marginal improvement, and validation loss shows no noticeable late increase.
  • Beam Prediction Performance: 1.0000, 0.9038, 0.8988, and 0.8945 are BeamTransFuser’s DBA-scores for Scenarios 31, 32, 33, and 34, respectively.The method achieves the highest overall DBA-score among the compared schemes.
  • Beam Prediction Performance: BeamTransFuser maintains reliable beam prediction across illumination, line-of-sight, and non-line-of-sight variations in the four urban scenarios.The evaluated settings include daytime and nighttime conditions as well as LoS and NLoS propagation.
  • Beam Prediction Performance: BeamTransFuser remains more stable from Scenarios 32 to 34, although QTNs and the method in perform slightly better in Scenario 34.Its overall DBA-score is 0.9129, ranking first across the evaluated conditions; the Scenario 34 margins are 1.99% and 1.42%.
  • Beam Prediction Performance: Camera, LiDAR, radar, and GPS provide complementary information that supports beam prediction when individual modalities become unreliable.The modalities contribute geometric structure, visual semantics, motion cues, and location information in urban V2X settings.

3) Performance with Modality Completion:

The modality-completion experiment evaluates missing Radar or LiDAR by comparing degraded replacements with CVAE-generated features. Generated features substantially restore beam-prediction accuracy under incomplete sensing.

  • Performance with Modality Completion: Missing Radar or LiDAR sharply reduces Top-1 accuracy when the absent input is replaced with zero-filled data or Gaussian noise.These replacements emulate sensor failure or incomplete deployment.
  • Performance with Modality Completion: 45.51% is the Top-1 accuracy with generated features versus 6.67% for the missing-Radar case.The CVAE-generated features substantially restore prediction accuracy.
  • Performance with Modality Completion: 56.18% is the Top-1 accuracy with generated features versus 6.67% for the missing-LiDAR case.The result indicates meaningful compensation from reconstructing the unavailable modality.
  • Performance with Modality Completion: The experiment does not report the generated-camera case because reconstructing high-dimensional RGB features is more difficult and complex.The reported recovery results therefore concern missing Radar and missing LiDAR.

VI. CONCLUSION

The paper concludes that hierarchical multimodal fusion and modality generation improve robust beam prediction in V2X networks. BeamTransFuser uses camera, LiDAR, radar, and GPS, while its generative module supports prediction when sensing inputs are incomplete.

  • VI. CONCLUSION: BeamTransFuser progressively fuses camera, LiDAR, radar, and GPS information through a hierarchical Transformer-based framework.The framework is designed for accurate and robust beam prediction.
  • VI. CONCLUSION: The generative module reconstructs missing modality features from available observations without requiring retraining.Experiments on a real-world multimodal V2X dataset show stable beam prediction when part of the sensing input is unavailable.
  • VI. CONCLUSION: Future work will extend the framework to more challenging urban scenarios and multi-vehicle settings.These settings define the stated scope of future evaluation.
Loading 2609.10200v1…