PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrated Sensing and Communications Towards Proactive Beamforming in mmWave V2I via Multi-Modal Feature Fusion (MMFF)

Haotian Zhang, Shijian Gao, Xiang Cheng, Liuqing Yang

arXiv:2310.02561v3eess.SP

TL;DR

Narrow mmWave beams make reliable V2I alignment challenging, and conventional beam training or tracking entails signaling overhead. This paper proposes MMFF-Net to combine multi-modal sensing and communication data for proactive angle prediction; it reports improved angle prediction, achievable rate, and outage performance over benchmark methods, including under drifting and environmental interference.

  • Problem

    Conventional beam training and tracking impose overhead or latency, while existing proactive approaches can be constrained by motion-model assumptions or limited environmental features.

    Source: §I · pp. 1–2

  • Method

    MMFF-Net combines multi-modal feature extraction and fusion with adaptive weight learning to predict vehicle position and beamforming angle from sensing and communication data.

    Source: §I · p. 2, §III · p. 3

  • Results

    MMFF-Net reduces angle prediction error by 31.52 percent overall and 11.76 percent during approach, while yielding at least 25.30 percent achievable-rate gain and 15.69 percent outage-probability reduction against benchmark schemes with a large antenna array.

    Source: §I · p. 2

  • Takeaways & Limitations

    In the tested complex dynamic scenarios, the scheme outperforms eight representative methods across angle prediction accuracy, achievable rate, and outage performance, with reported robustness to drifting and environmental interference.

    Source: §VII · p. 13

    The evaluated drifting behavior is abrupt and unsignaled, so none of the schemes can predict the drift event and all experience a rate decrease from misalignment.

    Source: B. Achievable Rate · p. 12

Abstract

from arXiv · show

The future of vehicular communication networks relies on mmWave massive multi-input-multi-output antenna arrays for intensive data transfer and massive vehicle access. However, reliable vehicle-to-infrastructure links require exact alignment between the narrow beams, which traditionally involves excessive signaling overhead. To address this issue, we propose a novel proactive beamforming scheme that integrates multi-modal sensing and communications via Multi-Modal Feature Fusion Network (MMFF-Net), which is composed of multiple neural network components with distinct functions. Unlike existing methods that rely solely on communication processing, our approach obtains comprehensive environmental features to improve beam alignment accuracy. We verify our scheme on the Vision-Wireless (ViWi) dataset, which we enriched with realistic vehicle drifting behavior. Our proposed MMFF-Net achieves more accurate and stable angle prediction, which in turn increases the achievable rates and reduces the communication system outage probability. Even in complex dynamic scenarios with adverse environment conditions, robust prediction results can be guaranteed, demonstrating the feasibility and practicality of the proposed proactive beamforming approach.

I. INTRODUCTION

Narrow mmWave beams make reliable V2I alignment difficult, while conventional training and tracking incur overhead or rely on motion assumptions that limit prediction. The paper proposes MMFF-Net, which combines sensing and communication data to predict beam angles, and evaluates it in a ViWi-based dataset enriched with vehicle drifting.

I. INTRODUCTION

  • Narrow mmWave beams require precise transmit–receive alignment, while beam training and tracking incur overhead or latency and conventional proactive methods rely on motion models or feedback that limit applicability.The introduction notes that state-evolution methods are limited to ideal straight paths and that nonlinear models constrain prediction accuracy.Source: §I · p. 1
  • MMFF-Net integrates multi-modal feature extraction and fusion with adaptive weight learning, using two regression models to predict vehicle position despite random drifting behavior.The proposed scheme targets V2I proactive beamforming using vehicle motion characteristics.Source: §I · p. 2
  • The evaluation dataset builds on the ViWi framework and adds random vehicle drifting to represent more complex user behavior.The dataset trajectory points include synchronized RGB images, depth maps, and CSI.Source: §I · p. 2
  • MMFF-Net reduces angle prediction error by 31.52 percent overall and 11.76 percent when vehicles approach the RSU, with at least 25.30 percent achievable-rate gain and 15.69 percent outage-probability reduction.The comparisons are against benchmark schemes when a large antenna array is adopted.Source: §I · p. 2

B. Problem Description

  • The problem setup has the RSU predict the vehicle’s next-slot relative azimuth from sequences of RGB images, depth maps, and sub-6 GHz CSI, assuming the vehicle drives parallel to the RSU array.The paper notes this parallel-driving assumption may not hold on some roads; vehicle-side AoA can then be estimated separately using array processing.Source: B. Problem Description · p. 3

III. MULTI-MODAL FEATURE EXTRACTION

The feature-extraction pipeline preprocesses heterogeneous sensory and channel data into motion-relevant features. Its angular-feature module derives a low-dimensional CSI representation and uses major-beam detection to retain the strongest directional components.

III. MULTI-MODAL FEATURE EXTRACTION

  • MMFF-Net uses three distinct feature-extraction modules with modality-specific preprocessing and neural-network architectures to derive vehicle-motion-related features from heterogeneous data.The RSU processes multi-modal sensory and wireless channel data while continuously serving the vehicle.Source: §III · p. 3

A. Angular Feature Extraction Module

  • Sub-6 GHz CSI is preprocessed to expose rough angular information, avoiding the excessive overhead and latency associated with obtaining sufficient mmWave training symbols.The paper treats CSI as compressed electromagnetic-environment information and defines an angular feature related to the LoS channel azimuth.Source: A. Angular Feature Extraction Module · pp. 3–4

1) Pre-Processing of Raw CSI:

  • The CSI preprocessing calculates beam energy using a super-resolution DFT codebook, averages across subcarriers, and applies OMP to detect major beams.Averaging achievable rates across subcarriers is described as improving reliability in identifying effective beams; OMP addresses beam-energy dispersion.Source: 1) Pre-Processing of Raw CSI · p. 4
  • The preprocessing retains the selected major-beam elements in the CSI representation and sets the remaining elements to zero.The paper presents this exclusion of negligible beams as a way to emphasize angular features and support the network’s learning.Source: 1) Pre-Processing of Raw CSI · p. 4

2) Angular Feature Extraction:

After CSI preprocessing, the AFE module maps the resulting representation into an angular feature using an MLP. The output feature size is adjustable.

2) Angular Feature Extraction:

  • The AFE module feeds preprocessed CSI into an MLP to extract the angular feature used as the wireless part of the multi-modal input.The MLP has three hidden layers and one output layer, whose neuron count sets the adjustable angular-feature size.Source: 2) Angular Feature Extraction · p. 4

3) Top-K Versus All:

The top-k analysis supports retaining only major beams in the CSI representation, while visual features complement the rough location information available from CSI. Together, these features provide different environmental cues for vehicle-position prediction.

3) Top-K Versus All:

  • The top two major beams account for 97.9% of total beam energy, and the curve saturates beyond two, supporting their use as latent features with minimal performance loss.The passage attributes the concentration to a few main propagation paths and says remaining beams contribute negligible power.Source: 3) Top-K Versus All · p. 5

B. 2D-Visual Feature Extraction Module

  • CSI’s angular feature gives relatively rough vehicle-location information, while images provide global visual context and can reveal surrounding scatterers to support localization and future-position prediction.The paper incorporates a 2D-visual feature extraction module to obtain these image features.Source: B. 2D-Visual Feature Extraction Module · p. 5

1) Pre-processing of RGB Image:

The RGB pipeline detects vehicles, embeds their relative position, and extracts a compact visual feature; the distance module adds ranging information that RGB-based features do not provide precisely.

1) Pre-processing of RGB Image:

C. Distance Feature Extraction Module

  • Radar ranging complements RGB semantics with distance information and 3D geometric cues because the AFE and VFE modules do not provide precise distance information.The ViWi dataset lacks radar data, so radar sensing is simulated by adding Gaussian noise to depth maps.Source: C. Distance Feature Extraction Module · p. 6

2) 2D-Visual Feature Extraction:

1) Pre-processing of Depth Map:

The depth pipeline treats depth maps as an independent modality, converts them into distance features, and fuses them with angular and visual features using learned modality weights.

1) Pre-processing of Depth Map:

  • Depth maps are kept separate from RGB because practical radar distances may be inaccurate or misaligned with image pixels, and the method is intended to generalize across radar frequencies.Source: 1) Pre-processing of Depth Map · p. 6
  • Depth-map inputs are min-max normalized after zero-mean Gaussian noise is added, with noise variance N0 set to 0.1.Source: 1) Pre-processing of Depth Map · p. 6

2) Distance Feature Extraction:

  • Radar ranging results are arranged as images to represent the environment’s overall spatial structure, and customized ResNet-18 extracts a distance feature of length LD.Source: 2) Distance Feature Extraction · pp. 6–7

A. Adaptive Weighted Fusion of Multi-Modal Feature

  • The angular, visual, and distance features are concatenated to preserve their distinct semantic information, then an adaptive weight learning network assigns feature weights.Source: A. Adaptive Weighted Fusion of… · p. 7
  • The network averages weights within each modality and applies a modality-wise product to form weighted multimodal features for the X-axis and Y-axis predictors.Source: A. Adaptive Weighted Fusion of… · p. 7

B. Beamforming Angle Prediction

The prediction approach accounts for lateral vehicle drift by using temporal multimodal features for recurrent Y-coordinate prediction, while modeling the assumed straight-lane X motion separately.

B. Beamforming Angle Prediction

1) Temporal Feature Extraction and Recurrent Prediction for Y-Coordinates Prediction:

2) MLP Prediction for X-Coordinates Prediction:

The X-coordinate is predicted with an MLP and converted with the predicted Y-coordinate into a beamforming angle; evaluation uses a ViWi-based dataset with randomized vehicle drifting.

2) MLP Prediction for X-Coordinates Prediction:

  • The X-coordinate prediction uses a four-layer MLP, and the predicted 2D position determines the next beamforming angle.The MLP is used because X-axis motion is described as easier for a neural network to learn, reducing computational load and accelerating convergence.Source: 2) MLP Prediction for X-Coordinates… · p. 8
  • MMFF-Net can extend to multi-user tracking by associating vehicles with subcarriers and applying detection and ranging information to each vehicle.The extension assumes user association has already been performed using existing schemes,,.Source: 2) MLP Prediction for X-Coordinates… · p. 8

A. Dataset Overview

  • The test dataset builds on ViWi’s single-car city-street scenario and adds random adjacent-lane movements to represent vehicle drifting.Source: A. Dataset Overview · p. 8
  • Training and testing use two entirely different randomly generated trajectories with distinct drifting behaviors.Source: A. Dataset Overview · p. 8

B. Signal Model

The signal model uses the predicted angle to form transmit and receive beams for a downlink V2I link. It evaluates link quality through received SNR and achievable rate.

B. Signal Model

  • The RSU sends a directional downlink stream, and the vehicle forms its receive beamformer using the predicted angle.The transmit beamformer is likewise formed from that predicted direction.Source: B. Signal Model · p. 8
  • The RSU and vehicle use transmit and receive steering vectors, respectively, with the RSU array assumed to have half-wavelength antenna spacing.The receive steering vector is denoted by b(θ).Source: B. Signal Model · p. 8
  • MMFF-Net’s block diagram comprises multi-modal feature extraction and fusion, Y-axis recurrent prediction, and X-axis prediction MLP modules.A sample vehicle trajectory with random drifting is shown for the “dist cam” scenario.Source: Figure 7 · p. 9, Figure 8 · p. 9
  • The received signal includes the LoS channel coefficient, Doppler frequency, reflection coefficient, and zero-mean Gaussian noise.Doppler frequency depends on vehicle velocity, relative angle, and carrier frequency.Source: B. Signal Model · p. 9
  • The achievable rate is Rn = log2(1 + SNRn), and it reaches its upper bound when the predicted and actual angles match.The SNR expression depends on transmit and receive beamformers formed from the predicted angle.Source: B. Signal Model · p. 9

C. Network Configuration

The network configuration specifies feature and model architectures, while the benchmark section describes comparison schemes and their evaluation settings. The benchmarks include tracking, vision-based, and single-modality approaches.

C. Network Configuration

  • The network design uses a smaller angular-feature size than visual and distance features because angular position information may be less intuitive and reliable in complex environments.The architecture was selected after evaluating neural networks of varying depths and widths with performance and computational complexity in view.Source: C. Network Configuration · p. 9, Table II · p. 9
  • MMFF-Net uses two GRU layers for Y-axis prediction and a three-hidden-layer MLP for X-axis prediction.The feature-extraction MLP has two hidden layers; the model is trained in PyTorch with ADAM.Source: C. Network Configuration · p. 9, Table II · p. 9

D. Benchmarks

  • The evaluation includes comparisons with several existing benchmarks and three unimodal data-based schemes designed and fine-tuned for the same proactive beamforming task.Benchmark experimental parameter settings are consistent with those in,,, and, and KF-T measurements are set to ground-truth values to ensure fair comparisons.Source: D. Benchmarks · p. 10
  • The benchmark set includes KF-T, echo-based EKF-T and DFRC-T, DL-PB, and image-classification-based V-A-BP.EKF-T and DFRC-T assume ideal straight-line or straight-lane motion; V-A-BP maps similar vehicle positions to the same codebook beam.Source: D. Benchmarks · p. 10
  • I-PB, D-PB, and C-PB isolate image, depth, and CSI inputs, respectively, with C-PB predicting adjacent-slot vehicle displacement.I-PB otherwise uses MMFF-Net’s image parameters, processing, and architecture; D-PB omits CSI and images.Source: D. Benchmarks · p. 10
  • V-A-BP and MMFF-Net do not solve the same problem, and adapting V-A-BP to the evaluation metrics may create fairness issues.Source: D. Benchmarks · p. 10

VI. PERFORMANCE EVALUATION

MMFF-Net is evaluated against tracking, predictive-beamforming, and unimodal benchmarks. It achieves more accurate, stable angle prediction and higher rates in reported array settings, while abrupt drifting remains difficult for every scheme.

VI. PERFORMANCE EVALUATION

  • The reported simulations average results over 300 independent realizations.Source: §VI · p. 10

A. Angle Prediction

  • 65.93% higher than MMFF-Net was DFRC-T’s minimum angle prediction error; MMFF-Net maintained stable, accurate predictions throughout tracking.The comparison is reported for N_HF = 8.Source: A. Angle Prediction · p. 10, Figure 9 · p. 11
  • MMFF-Net’s average angle error and standard deviation were lower than D-PB’s by comparisons of 46.02% and 45.70%, respectively.D-PB was the closest-performing unimodal scheme; the evaluation also reports the effect of AWLN in Table IV.Source: A. Angle Prediction · p. 11, Table IV · p. 11

B. Achievable Rate

  • In the 16-antenna case, MMFF-Net’s average achievable rate was 7.97 bps/Hz versus 7.48 bps/Hz for DFRC-T; in the 32-antenna case, it was 8.32 bps/Hz versus 6.30 bps/Hz.The figure compares achievable rates for MMFF-Net, DFRC-T, V-A-BP, and I-PB.Source: B. Achievable Rate · pp. 11–12, Figure 11 · p. 12
  • During the latter half of tracking, MMFF-Net averaged 8.05 bps/Hz versus DFRC-T’s 6.80 bps/Hz with 16 antennas, and 8.52 versus 3.52 bps/Hz with 32 antennas.Source: B. Achievable Rate · p. 12
  • When vehicles drift abruptly between lanes, neither MMFF-Net nor the unimodal schemes predict the event, and all schemes experience a notable rate decrease from misalignment.Source: B. Achievable Rate · p. 12
  • With drifting, the 32-antenna array can underperform the 16-antenna case because its narrower beam is more likely to miss the vehicle when prediction accuracy is insufficient.Source: B. Achievable Rate · p. 12

C. Outage Probability

MMFF-Net minimizes outage across the overall tracking process, despite exceptions at high rate thresholds and with larger arrays. Tests with noisy multimodal inputs show performance degradation but retain advantages over DFRC-T and I-PB; the paper concludes that the approach performs across complex, dynamic scenarios.

C. Outage Probability

  • The advantage is not uniform: DFRC-T has lower outage at higher rate requirements near the RSU, while MMFF-Net's 32-antenna case has more outage than its 16-antenna case at 3.80–7.50 bps/Hz.The text attributes the array comparison to narrower beams being more likely to miss the target vehicle.Source: C. Outage Probability · p. 12
  • Fig. 12 shows MMFF-Net minimizes outage over the overall tracking process, with reductions versus DFRC-T, I-PB, and D-PB across 8-, 16-, and 32-antenna cases.Against DFRC-T, reductions are 1.14%, 5.14%, and 17.39%; against I-PB, 5.46%, 20.70%, and 15.69%; and against D-PB, 5.48%, 12.55%, and 19.82%, respectively.Source: C. Outage Probability · p. 12, Figure 12 · p. 13

D. Robustness Testing Against Adverse Environmental Conditions

  • Under adverse conditions, MMFF-Net degrades but its average achievable rate remains 9.84% above DFRC-T and 4.16% above I-PB in the 32-antenna case.Testing simulates poor lighting, CSI estimation error at −15 dB NMSE, and depth-map noise variance of 1; Table V reports achievable-rate performance under these conditions.Source: Table V · p. 13, D. Robustness Testing Against Adverse… · p. 13

VII. CONCLUSIONS

  • On the ViWi-derived dataset enriched with vehicle drifting, the paper reports that MMFF-Net outperforms eight representative methods in angle prediction accuracy, achievable rate, and outage performance.The conclusion identifies lower-frequency CSI preprocessing and extension to multi-vehicle scenarios as future work.Source: §VII · p. 13
Loading 2310.02561v3…