PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Multimodal Beam Prediction for Real-World 6G UAV Communications via Image-Position-Synthetic Fusion

Rifat Bin Reza, Md Ahadul Haque Suvo, Jia Uddin, Saifur Rahman Sabuj

arXiv:2610.14593v1eess.SPeess.IV

TL;DR

Frequent beam updates make exhaustive measurement costly for mobile UAV mmWave links, and the complementary roles of visual and position-derived representations remain to be examined. This paper evaluates a staged multimodal predictor on real UAV measurements, finding that image–synthetic fusion improves over image-only inference and that full fusion gives the highest measured performance.

  • Problem

    Highly mobile mmWave UAV links require frequent beam updates, while exhaustive codebook sweeps consume radio resources and increase access latency.

    Source: §1 · p. 1

  • Method

    A staged predictor maps synchronized images and position-derived inputs to measured retained receiver-beam labels, fusing image and synthetic representations before explicit geometry.

    Source: §1 · p. 3, §2.2 · p. 5

  • Results

    87.14% Top-1, 99.12% Top-3, and 99.60% Top-5 accuracy were achieved by full fusion, the highest measured performance in Scenario 23.

    Source: §5 · p. 17

  • Takeaways & Limitations

    Image–synthetic fusion improves Top-1 accuracy by 2.64 percentage points over image-only inference, while controlled interventions show position-derived features remain useful under visual degradation.

    Source: §5 · p. 17

    Top-k accuracy indicates candidate-set compression in offline evaluation, not measured reductions in over-the-air training trials, confirmation time, or protocol overhead.

    Source: §2.6 · p. 11

Abstract

from arXiv · show

Camera-based beam selection can reduce the need for exhaustive channel measurement in highly mobile millimeter-wave UAV links, but the complementary value of visual and position-derived representations and their behavior under modality degradation require further investigation. This paper develops a staged multimodal framework for current-beam prediction using real-world UAV measurements. The proposed 32-beam classifier combines an ImageNet-pretrained EfficientNet-B3 visual encoder with two deterministic position-derived representations: an explicit Geometric-Kinematic vector and a nonlinear synthetic feature basis constructed from position. The image and synthetic representations are fused first, followed by late fusion with the geometry representation. Experiments on DeepSense 6G Scenario 23 show that Image-Synthetic fusion achieves 86.92% Top-1, 99.03% Top-3, and 99.56% Top-5 accuracy, improving Top-1 performance by 2.64 percentage points over image-only inference and outperforming the 60.71% Top-1 accuracy of synthetic-only inference. Adding the Geometric-Kinematic branch yields the highest measured performance, reaching 87.14% Top-1, 99.12% Top-3, and 99.60% Top-5 accuracy. Controlled input interventions further demonstrate complementary behavior under visual degradation and strong sensitivity to severe synthetic-input corruption. Position-derived nonlinear features provide complementary information to visual context and improve current-beam prediction in the evaluated real-world UAV setting.

1 Introduction

Highly mobile UAV mmWave links need frequent narrow-beam updates, while exhaustive sweeps consume radio resources and increase access latency. The paper investigates whether deterministic position-derived features complement vision and whether the fused predictor remains reliable under modality degradation.

1 Introduction

  • Exhaustive codebook sweeps consume radio resources, increase access latency, and shorten payload-transmission time as UAV motion requires frequent beam updates.The high propagation loss of mmWave links necessitates large antenna arrays and narrow directional beams.Source: §1 · p. 1
  • DeepSense 6G Scenario 23 aligns real-world UAV camera observations, practical position information, and beam-sweep measurements for supervised beam selection.The measurements include effects such as sensor noise, calibration error, background clutter, and irregular flight trajectories.Source: §1 · p. 2
  • Position is inexpensive and already available but omits blockage and scene appearance, whereas RGB provides that context through an indirect data-driven mapping.The nonlinear synthetic basis re-expresses position with distance- and phase-like coordinates; it adds no new physical information or active sensor.Source: §1 · p. 2
  • The unresolved questions are whether nonlinear reparameterization of available position adds information beyond a strong visual encoder, whether explicit geometry remains useful, and how fusion behaves under modality degradation.The study compares image-only, synthetic-only, image–synthetic, and full-fusion hypotheses using controlled evaluation.Source: §1 · p. 3
  • The study evaluates staged visual and position-derived representations, characterizes the deterministic nonlinear position transformation, and tests reliability through controlled input interventions.Its contributions include a common fixed evaluation partition and performance-based group attribution.Source: §1 · p. 3

2.1 Wireless Link and Beam-Selection Model

The link model uses a retained receiver-beam codebook and defines beam utility over OFDM subcarriers. In the measured configuration, the retained task has 32 beams.

2.1 Wireless Link and Beam-Selection Model

  • The measurement configuration retains Q = 32 beams from an original Q0 = 64-beam codebook by keeping every second codeword.The preprocessing requires Q0 to be even, which holds for the retained configuration.Source: §2.1 · p. 4
  • Beam utility is averaged over OFDM subcarriers, with the stated arithmetic-mean aggregation described as one choice among alternatives.The passage names worst-case subcarrier SNR and a rate-weighted average as other possible aggregations.Source: §2.1 · p. 5

2.2 Problem Formulation and Fusion Overview

The task predicts retained receiver-beam labels from synchronized image and position-derived inputs rather than explicit CSI. The staged architecture fuses learned image and synthetic representations first, then uses explicit geometry for late refinement.

2.2 Problem Formulation and Fusion Overview

  • The synthetic input is a deterministic handcrafted transformation of position telemetry, not measured CSI or a channel-estimation output.The observation tuple also includes an RGB image and geometric–kinematic telemetry.Source: §2.2 · p. 5
  • The supervised target is the measured Scenario 23 receiver beam with maximum recorded received power after preprocessing, predicted from synchronized image, geometry, and synthetic inputs.The model produces a posterior ranking over retained beam classes for Top-k evaluation.Source: §2.2 · p. 5
  • The model first fuses learned image and synthetic representations, then late-fuses that embedding with a separate geometric–kinematic representation.The synthetic branch lifts position nonlinearly, while the geometry branch retains explicit spatial and motion descriptors for refinement.Source: §2.2 · p. 5

2.3 Beam-Label Construction and Learning Problem

The section defines beam prediction from measured powers on retained codebook beams and constructs synchronized RGB, geometric–kinematic, and synthetic position-derived inputs.

2.3 Beam-Label Construction and Learning Problem

  • The target is the retained beam with the highest measured received power, used for supervised learning under stated conditions linking power maximization to SINR maximization.The equivalence assumes approximately invariant effective noise and residual interference, consistent transmit power, consistent subcarrier aggregation, and linear-power processing.Source: §2.3 · pp. 5–6
  • Sequential trajectories and adjacent-sample geometry features mean temporally neighboring records can share position and visual context across the sample-level partition.The section says the effect on Top-1 accuracy is quantified in Section 3.1.Source: §2.3 · p. 6

2.3.1 Geometric–Kinematic Feature Construction

  • The 15-dimensional Geometric–Kinematic vector represents planar position and adjacent-sample motion, with position increments treated as displacement rather than physical velocity.The features are standardized using statistics estimated from the training partition.Source: §2.3.1 · p. 6, §2.3 · p. 6

2.3.2 Synthetic Position-Derived Features

  • The 11-dimensional synthetic vector applies deterministic nonlinear transformations to planar position and is explicitly distinguished from measured complex CSI.Its range basis is constructed using a training-set range scale; the periodic range transformation is not a physical propagation phase, and signed feature values are preserved during min–max normalization.Source: §2.3.2 · pp. 6–7

2.3.3 RGB Image Feature Construction

  • RGB frames are augmented stochastically during training and undergo a deterministic resize, center crop, and intensity normalization for evaluation.The evaluation transform is applied identically to all test observations.Source: §2.3.3 · p. 7

2.4 Multimodal Architecture

The architecture learns image, synthetic position-derived, and Geometric–Kinematic representations in stages, fusing imagery with synthetic features before late geometry fusion.

2.4 Multimodal Architecture

  • The model has image, synthetic position-derived, and Geometric–Kinematic branches, combined through image–synthetic fusion followed by geometry-guided final fusion.This staged sequence is used rather than directly concatenating all raw modalities.Source: §2.4 · p. 7

2.4.1 Design Overview

  • The sequential fusion design freezes earlier stages while fitting later ones, making each fusion block’s incremental contribution experimentally interpretable.The image and synthetic representations are fused first, then refined with the geometry branch.Source: §2.4.1 · p. 7

2.4.2 Image Encoder

  • The image branch processes synchronized RGB imagery with an ImageNet-pretrained EfficientNet-B3 backbone and a projection layer that produces the visual embedding.An auxiliary image-only classification head is discarded before multimodal fusion.Source: §2.4.2 · pp. 7–8

2.4.3 Synthetic-Feature Encoder

  • The synthetic-feature encoder transforms the position-derived vector through fully connected layers with normalization, GELU activation, and dropout.Its temporary auxiliary classification head is removed after pretraining.Source: §2.4.3 · p. 8

2.4.4 Geometric–Kinematic Encoder

  • A separate encoder maps the standardized Geometric–Kinematic representation into its learned branch representation.Source: §2.4.4 · p. 8

2.4.5 Image–Synthetic Fusion

  • Image and synthetic embeddings are concatenated and processed by two fusion layers to form an intermediate representation.The temporary image–synthetic classification head is removed before the geometry-guided stage.Source: §2.4.5 · p. 8

2.4.6 Final Geometry-Guided Fusion

  • The final stage fuses the intermediate image–synthetic representation with the geometric–kinematic representation and produces Q beam-selection logits.Nonlinear GELU transformations occur in hidden representations, while the final logits are linear and passed to softmax.Source: §2.4.6 · p. 9

2.5 Staged Training Objective

Training proceeds through branch pretraining, image–synthetic fusion, and final geometry-guided fusion, with previously trained stages frozen as later stages are optimized.

2.5 Staged Training Objective

  • The four-stage procedure pretrains the synthetic and image branches, trains their fusion with both branch encoders frozen, then trains geometry-guided fusion while earlier stages remain frozen.The final optimization freezes the image–synthetic network and trains only the geometry encoder and final classifier.Source: §2.5 · pp. 9–10
  • Final multimodal training applies label smoothing and uses AdamW with a cosine-annealing schedule, weight decay, and gradient clipping.The initial learning rate is 4 × 10^-4, weight decay is 10^-4, the schedule runs for 32 epochs, and the maximum gradient norm is 1.0.Source: §2.5 · p. 9
  • Top-k beam-selection accuracy measures whether the ground-truth beam appears among the k highest-probability predictions.The test-set observation count is denoted by N_te.Source: §2.5 · p. 10

2.6 Inference-Time Beam Recommendation

At inference, synchronized imagery and current and previous positions pass through the trained staged model to produce a ranked beam shortlist with confidence scores.

2.6 Inference-Time Beam Recommendation

  • Inference applies the training and evaluation preprocessing, extracts all three representations, performs staged fusion, and returns the highest-ranked k beams.The procedure uses fitted normalization statistics and does not require measured instantaneous CSI.Source: §2.6 · pp. 10–11
  • The output logits and softmax values are beam-ranking scores, not calibrated estimates of link quality.Source: §2.6 · p. 11
  • Top-k accuracy indicates whether the correct beam remains in a short candidate list, but the study does not measure reductions in over-the-air trials, confirmation time, or protocol overhead.The described fallback probes the top-ranked beam first and uses the remaining shortlist if link quality is below threshold.Source: §2.6 · p. 11

3 Experimental Setup

The experiments use a reconstructed, consistently partitioned Scenario 23 dataset and a fixed staged optimization protocol to compare model configurations.

3.1 Reproducibility and Data Protocol

  • 11,387 aligned observations were split into 9,109 training and 2,278 evaluation samples, with the same partition used across configurations.The data subsets were combined to recover synchronized image, position, and beam-sweep records; the split used stratification and seed 42.Source: §3.1 · p. 11
  • Preprocessing parameters were fitted on training data only, and evaluation used unchanged normalization parameters and deterministic image preprocessing.Stochastic image augmentation was restricted to training.Source: §3.1 · p. 11
  • 88.7% of evaluation samples had a training sample within 150 ms; Top-1 accuracy was 87.56% within 150 ms and 85.69% farther away.The 1.87-point difference was described as modest, indicating temporal proximity did not produce a large accuracy shift within this partition.Source: §3.1 · p. 11
  • Table 3 brackets give 95% Wilson score confidence intervals over 2,278 test samples, and reported drops use Full fusion’s 87.14% Top-1 as the reference.Table 3 compares trained component ablations on the fixed test partition.Source: §3.1 · p. 12, Table 3 · p. 12

3.2 Optimization Protocol

  • Optimization introduced synthetic, image, image–synthetic, and geometry-guided stages, retaining pretrained representations while optimizing newly introduced components.Fixed epoch budgets and checkpoints were used; the test partition was not accessed for training or checkpoint selection and was evaluated once after each stage.Source: §3.2 · p. 12

4 Results and Discussion

Image–synthetic fusion improves on image-only beam prediction, and full fusion with geometric–kinematic features gives the highest measured ranking accuracy. Controlled interventions show complementary behavior under visual degradation but strong sensitivity to severe synthetic-input corruption.

4.1 Ablation of the Input Representations

  • Image-only inference reaches 84.28% Top-1 accuracy, exceeding Synthetic-only inference at 60.71%, although the position-derived basis remains predictive on the fixed test partition.The results distinguish useful position-derived information from the stronger standalone discriminative power of visual context.Source: §4.1 · p. 12
  • 87.14% Top-1, 99.12% Top-3, and 99.60% Top-5 accuracy make Full fusion the highest measured configuration, ahead of Image + Synthetic on all three metrics.Image + Synthetic improves Top-1 by 2.64 percentage points over image-only inference.Source: §4.1 · p. 12, Figure 3 · p. 13
  • Post-hoc test-time interventions probe the selected fixed image–synthetic predictor, unlike ablation, which compares separately optimized input configurations.The evaluated perturbations are controlled diagnostics, not empirical models of naturally occurring blockage, NLOS propagation, or sensor failure.Source: §4.1 · p. 12, §4.2 · p. 14
  • Zeroing all 11 synthetic coordinates causes a 78.05-point Top-1 loss, versus 41.97 points for a black frame; the prescribed synthetic-feature shift reduces accuracy by 2.46 points.The synthetic coordinates are correlated deterministic transformations of a common position record, so intervention is assessed at the group level.Source: §4.1 · p. 13

4.2 Controlled Input-Stress Evaluation

  • Under visual degradation, image–synthetic fusion retains 44.60% accuracy with a black frame and 54.08% at 25% brightness, outperforming image-only results of 23.18% and 25.37%.Static fusion nevertheless does not recover the full Synthetic-only baseline when the image becomes uninformative.Source: §4.2 · p. 14
  • Suppressing the synthetic vector reduces fusion accuracy from 86.92% to 8.52%, while the independently trained image branch retains 84.28%; static fusion does not reliably favor the healthier branch under perturbation.The clean-input fusion also falls to 64.53% under Gaussian synthetic noise, but the prescribed deterministic shift causes only a 2.46-point decrease.Source: §4.2 · p. 15

4.3 Explainable Multimodal Attribution Analysis

  • Group-level attribution assigns 53.68 percentage points above chance to image and 30.11 to synthetic data, consistent with ablation’s finding that position-derived features complement rather than supersede vision.Under corruption, attribution shifts away from the corrupted modality; a suppressed synthetic vector receives a strongly negative contribution.Source: §4.3 · p. 16, Figure 6 · p. 16, Figure 7 · p. 17

5 Conclusion

The study evaluates staged multimodal current-beam prediction using RGB imagery and position-derived representations on DeepSense 6G Scenario 23. Image–synthetic fusion improves on the image-only branch, while adding geometric–kinematic features gives the highest measured performance; input interventions also reveal sensitivity to severe synthetic-input corruption.

5 Conclusion

  • 86.92% Top-1, 99.03% Top-3, and 99.56% Top-5 accuracy for image–synthetic fusion, improving Top-1 by 2.64 percentage points over image-only inference.Adding the explicit Geometric–Kinematic representation reaches 87.14% Top-1, 99.12% Top-3, and 99.60% Top-5 accuracy, the highest measured performance.Source: §5 · p. 17
  • Controlled input interventions show that position-derived representations remain useful under visual degradation, while severe synthetic-input corruption can substantially degrade the fused predictor.The standalone image branch remains strong, but the fused model's sensitivity motivates investigation of reliability-aware modality gating.Source: §5 · p. 17
  • The evaluated Scenario 23 results support position-assisted multimodal beam prediction; future work will examine broader trajectories and deployment latency and resource measurements.The paper also identifies reliability-aware modality gating as future work.Source: §5 · p. 17

Related papers

Loading 2610.14593v1…