PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual-Calibrated Multimodal Spatiotemporal Learning for Future Beam Prediction

Md Ahadul Haque Suvo, Rifat Bin Reza, Saifur Rahman Sabuj

arXiv:2610.33892v1eess.SP

TL;DR

Reliable future-beam prediction must account for changing propagation conditions, and this study asks whether explicit beam-history embeddings can be combined with visual and location sequences. It evaluates a multimodal temporal predictor with a separately trained residual adjustment on held-out capture runs; the supervised model leads at Top-1, while residual adjustment gives a small Top-2 advantage.

  • Problem

    The study asks whether learned historical beam-label embeddings and a separately trained residual head can support prediction of the next dataset record alongside visual and location sequences.

    Source: §1 · p. 2

  • Method

    A multimodal temporal predictor fuses four RGB frames and location features with historical beam-label embeddings, then an offline residual actor–critic head adjusts frozen supervised logits.

    Source: §1 · pp. 1–2, §3.1 · p. 5, §4.5 · p. 8

  • Results

    On held-out runs, the supervised predictor leads at Top-1, Top-3, and Top-5, while ORCC leads Top-2 by 0.27 percentage points and the validation-selected mixture matches ORCC.

    Source: §6.1 · p. 10, §7 · p. 12

  • Takeaways & Limitations

    The initial held-out-run evaluation favors the supervised predictor for Top-1 selection and characterizes ranking over encoded labels.

    Source: §6.1 · p. 10, §7 · p. 12

    The evaluation uses one run-disjoint split without controlled ablations or repeated runs, and lacks a documented physical beam mapping and communication metrics.

    Source: §6.4 · p. 12, §7 · p. 12

Abstract

from arXiv · show

We propose a residual calibrated multimodal spatiotemporal framework for future beam prediction on DeepSense 6G Scenario 23. The framework integrates visual observations, location features, and historical beam labels to predict the beam label of the next dataset record. A pretrained Swin-Tiny encoder extracts features from four consecutive RGB frames. Bidirectional cross-attention integrates visual and two dimensional Unit 1 location representations, while a temporal Transformer incorporates historical beam label embeddings to model dependencies across the observation window. A supplementary offline residual head adjusts the frozen supervised logits using immediate correctness rewards and an auxiliary classification loss. A fixed probability mixture of the supervised and residual outputs is selected using validation accuracy. From 11,387 aligned records, we construct 11,255 temporal windows and partition them by capture run into 7,854 training, 1,905 validation, and 1,496 test samples. The supervised predictor achieves Top-1, Top-2, Top-3, and Top-5 test accuracies of 62.83%, 74.80%, 79.75%, and 81.08%, respectively. The residual predictor achieves corresponding accuracies of 61.76%, 75.07%, 79.41%, and 81.02%. Residual adjustment increases Top-2 accuracy by 0.27 percentage points, while the supervised predictor retains higher Top-1, Top-3, and Top-5 accuracies. The validation-selected mixture coincides with the residual predictor. The proposed framework combines cross-modal sensing features with explicit beam-history conditioning and a separately trained residual adjustment stage for one-record-ahead beam prediction. Initial evaluation on held-out capture runs characterizes the candidate-ranking performance of both prediction stages and favors the supervised variant for Top-1 selection. These findings motivate further study of historical conditioning and residual refinement.

1 Introduction

This study asks whether historical beam-label embeddings and a separately trained residual head can support next-record beam prediction alongside visual and location sequences. Its run-disjoint evaluation finds the supervised model leads on most reported Top-K metrics, while the residual head leads only at Top-2.

1 Introduction

  • RGB imagery, location features, and recent beam labels provide scene, spatial, and link-history information for conditioning future-beam estimates as propagation geometry, blockage, and motion evolve.Prior work has explored sensing-assisted prediction, multimodal fusion, semantic scene representations, and temporal tracking [1] [2].Source: §1 · pp. 1–2
  • The research question is whether learned embeddings of four recorded beam labels and a separately trained residual actor–critic head can support prediction of the next dataset record.The input history ends at index r and the target is the recorded beam label at index r + 1.Source: §1 · p. 2
  • The predictor combines Swin-Tiny visual features, bidirectional visual–location cross-attention, historical beam-label embeddings, and temporal Transformer processing to estimate the next-record label.ORCC is a separately trained, offline residual actor–critic head that adjusts supervised outputs.Source: §1 · p. 2, Figure 1 · p. 3
  • Unlike the current-beam formulation of Reza et al., this study predicts the next record, embeds four historical labels, and evaluates a separate residual-adjustment stage.The studies are not numerical baselines because their prediction targets, label spaces, and input histories differ.Source: §1 · p. 2
  • On held-out runs, the supervised predictor leads at Top-1, Top-3, and Top-5, while ORCC leads Top-2 by 0.27 percentage points; the validation-selected mixture matches ORCC.The evaluation uses 7,854 training, 1,905 validation, and 1,496 test windows assigned by capture run.Source: §1 · p. 2

2 Related Work

Related work spans sensing-assisted beam prediction, multimodal fusion, and reward-based beam management. This study distinguishes its next-record target and explicit beam-history conditioning from related designs, while treating differing evaluation protocols as unsuitable for matched accuracy comparisons.

2.1 Sensing and beam prediction

  • Sensing-assisted beam studies use imagery, location, LiDAR, radar, and semantic scene representations, while GPS utility depends on localization quality [1].Other work includes multimodal Transformer prediction at the terminal sample [2] and beam-group prediction followed by Q-learning in simulation [8].Source: §2.1 · p. 3

2.2 Multimodal fusion and mobile scenarios

  • Prior mobile-scenario methods include four-step image–GPS prediction at t + 1 and image plus GPS/IMU fusion for 64-beam optimal-beam prediction.The latter does not report a temporal sequence model; Reza et al. instead study current-beam prediction with position-derived features.Source: §2.2 · pp. 3–4
  • The present formulation predicts the next dataset record, explicitly conditions on historical beam labels, and uses bidirectional attention before temporal Transformer processing; published accuracies are not matched baselines across differing protocols.The separate ORCC stage fits an additive logit adjustment from frozen representations and supervised logits.Source: §2.2 · p. 4

2.3 Learning objectives and evaluation

  • Supervised beam prediction targets agreement with recorded labels, whereas reward-based management can optimize system objectives through sequential decisions; ORCC instead trains offline on fixed contexts.Its sampled action does not alter subsequent recorded observations, and evaluation reports Top-K encoded-label accuracy rather than throughput or physical beamforming gain.Source: §2.3 · pp. 4–5

3 Problem Formulation

The formulation uses four consecutive records within one capture run to predict the next record’s encoded beam label. Although generic communication quantities provide context, the experiments evaluate label ranking rather than physical-link performance.

3.1 Communication model and assumptions

  • The generic communication context assumes a narrowband link with a uniform linear array and single-antenna receiver; its channel model uses path gains and angles of departure.The physical codebook is illustrative, and experiments do not evaluate a physical array, codebook, or channel.Source: §3.1 · p. 5
  • The target in the communication context is the beam maximizing per-beam SNR, but evaluated labels are not mapped to those physical entries.Normalized beamforming gain and achievable rate are defined as possible metrics, not reported experimental outcomes.Source: §3.1 · p. 5
  • The experiments compute no communication metrics because channel or codebook measurements are absent; Top-K accuracy is not a substitute for beamforming gain or achievable rate.The reported task measures ranking over encoded labels.Source: §3.1 · p. 5
  • The input is four records ending at r, with image I_r, two-dimensional location p_r, and encoded beam label b_r; the target y_r is b_(r+1).Only windows with five consecutive records from the same capture run are retained, and timestamps are not included to characterize elapsed time.Source: §3.1 · p. 5
  • The location input is the parser-selected two-dimensional unit1_loc field, treated as a location feature rather than UAV telemetry.Targets are read from unit1_beam_index.Source: §3.1 · p. 5
  • The model predicts among 56 observed encoded classes, which are categorical labels rather than physical beam indices.Their relationship to the illustrative physical codebook is unspecified.Source: §3.1 · p. 5

4 Proposed Architecture and Training

The architecture encodes four-frame visual and Unit 1 location histories, fuses them bidirectionally, and conditions temporal prediction on recorded beam labels. A supervised classifier is trained first; a separately trained ORCC head adjusts its frozen logits, with a global probability mixture selected on validation data.

4.1 Input processing

  • The position branch uses normalized two-dimensional Unit 1 coordinates, while RGB inputs are resized and normalized for the visual encoder.The position branch excludes speed, altitude, distance, and orientation.Source: §4.1 · p. 6

4.2 Visual and position encoding

  • A pretrained Swin-Tiny backbone produces pooled frame-level features, while a shared multilayer projection encodes the corresponding location vectors.Only the final Swin stage and normalization parameters are fine-tuned; global average pooling discards patch-level spatial detail.Source: §4.2 · p. 6

4.3 Bidirectional fusion and temporal encoding

  • Bidirectional cross-attention fuses aligned visual and location sequences, then beam-label and temporal embeddings enter a Transformer that predicts the next-record label.The encoder uses only records at or before the input index.Source: §4.3 · pp. 6–7, Figure 2 · p. 7

4.4 Supervised Learning Objective

  • The supervised classifier uses clipped inverse-frequency class weights and label-smoothed weighted cross-entropy, with its checkpoint selected by validation Top-1 accuracy.Training runs for 80 epochs without early stopping.Source: §4.4 · pp. 7–8

4.5 A3C-inspired residual actor–critic refinement (ORCC)

  • ORCC trains separately on frozen supervised representations and logits, learning additive logit adjustments from immediate correctness rewards and an auxiliary classification loss.A fixed global probability-mixture coefficient is selected using validation Top-1 accuracy; the selected blend and residual head have identical test metrics at reported precision.Source: §4.5 · p. 8

4.6 Training and prediction algorithms

  • The procedure selects a supervised checkpoint, fits ORCC on frozen features, then selects a global blend; held-out histories receive deterministic candidate rankings.The target label is used only to calculate final Top-K metrics.Source: §4.6 · pp. 8–9

5 Dataset and Experimental Protocol

The experimental protocol constructs temporal windows and assigns complete capture runs to separate training, validation, and test partitions. Evaluation ranks encoded labels on held-out histories, with limits on class coverage and reproducibility documented.

5 Dataset and Experimental Protocol

  • 11,255 valid windows from 11,387 manifest rows were split by complete capture run into 7,854 training, 1,905 validation, and 1,496 test examples.Table 2 summarizes run-disjoint partition sizes and distinct encoded-label coverage.Source: §5 · p. 9, Table 2 · p. 9
  • The 56-class vocabulary contains 53, 38, and 31 distinct labels in training, validation, and test, respectively.Some test labels are absent from training, limiting class-wise interpretation.Source: §5 · pp. 9–10
  • The evaluation algorithm ranks candidate labels and computes Top-K scores for a held-out history when its target label is supplied.The test partition is used only for final evaluation.Source: §5 · p. 10
  • Bitwise reproducibility is not assured, and inference latency, computational or energy cost, and variability across random seeds are unmeasured.The execution setup enables cuDNN benchmarking and omits software-environment details.Source: §5 · p. 10

6 Results and Discussion

On the test set, the supervised classifier leads at Top-1, Top-3, and Top-5, while the residual head leads at Top-2. Checkpoint, class-level, and single-split analyses qualify these results.

6.1 Top-K accuracy

  • The supervised classifier leads at Top-1, Top-3, and Top-5, while the residual head has a 0.27-percentage-point advantage at Top-2.The validation-selected blend matches the residual-head scores at reported precision.Source: Table 3 · p. 10, §6.1 · p. 10

6.2 Checkpoint selection and validation–test comparison

  • At the selected checkpoints, the supervised model has 69.03% validation and 62.83% test Top-1 accuracy; ORCC has 69.13% and 61.76%, respectively.The validation-to-test differences are 6.20 and 7.37 percentage points, respectively, on this single run-disjoint split.Source: §6.2 · p. 10, Table 4 · p. 11
  • ORCC validation argmax accuracy falls from 69.13% at epoch 1 to 67.14% at epoch 30 as sampled training reward rises from 0.5557 to 0.6387.This divergence is compatible with overfitting but does not establish its cause; reward and validation accuracy are distinct measures.Source: §6.2 · pp. 10–11, Figure 3 · p. 11

6.3 Class-level performance

  • The supervised model’s macro-F1 is 0.39 and weighted-F1 is 0.58; class 31 has 503 test examples and F1 0.93, while class 1 has 241 and F1 0.00.Test labels absent from training also caution against treating aggregate accuracy as representative across the full codebook; encoded identifiers are not verified physical beam indices.Source: §6.3 · pp. 11–12

6.4 Discussion

  • The run-disjoint results characterize the full implementation on one split, but controlled ablations were not conducted to isolate individual inputs or modules.The literature comparison is qualitative rather than a matched ranking because tasks, label spaces, inputs, partitions, and metrics differ.Source: §6.4 · p. 12

7 Conclusion

On a run-disjoint test split, the supervised classifier achieved 62.83% Top-1 accuracy, while residual adjustment did not yield consistent gains. The evaluation covers ranking over 56 observed encoded labels, not communication-level performance.

7 Conclusion

  • 62.83% Top-1 accuracy was achieved by the supervised Swin-Tiny classifier on the run-disjoint test split.The residual actor–critic component exceeded the supervised model only at Top-2, and the validation-selected mixture matched ORCC at the reported precision.Source: §7 · p. 12
  • The evaluation characterizes candidate ranking over 56 observed encoded labels rather than communication-level performance.Source: §7 · p. 12
  • Controlled ablations, repeated runs, documented physical beam mapping, and communication metrics are needed to establish broader benefits.Source: §7 · p. 12

Related papers

Loading 2610.33892v1…