PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Channel Data: Configuration-Enriched Radio Foundation Models

Shrihari Vasudevan, H. G. Ranjani, Raaghul Ranganathan, Satwik Sahu, Thirumaran Ekambaram

arXiv:2610.32542v1cs.AIeess.SP

TL;DR

Radio foundation models learn from channel data, but prior work rarely encodes the channel-generation configuration in pretraining. This paper adds CFG tokens to the RFM input and evaluates frozen embeddings across downstream tasks and data regimes. CFG-enriched embeddings are best or competitive with baselines across tasks and seen and OOD settings, with especially consistent advantages over raw channel tokens in few-shot and OOD regimes.

  • Problem

    Most existing work does not leverage channel-generation configuration during RFM pretraining, despite RFMs being evaluated for downstream tasks with limited task-specific labels.

    Source: B. Research Questions · p. 1, §I · p. 1

  • Method

    The paper adds CFG tokens alongside channel tokens during RFM pretraining and evaluates frozen embeddings across downstream tasks, training-data sizes, and seen and OOD settings.

    Source: §I · p. 1, C. Contributions · p. 2, A. Architecture · p. 2

  • Results

    CFG-containing embeddings achieve the best or competitive performance across downstream tasks, training regimes, and seen and OOD scenarios, and compact 64-dimensional CFG-enriched embeddings outperform raw channel tokens across all three tasks and regimes, particularly in few-shot and OOD settings.

    Source: D. Summary · p. 6, §IV · p. 6

  • Takeaways & Limitations

    CFG-enriched embeddings can provide a low-dimensional inductive prior for downstream generalization, with benefits varying by task and H remaining dominant for TA prediction.

    Source: §IV · p. 6, D. Summary · p. 6

Abstract

from arXiv · show

This paper proposes incorporating channel configuration (CFG) information into Radio Foundation Model (RFM) pre-training and demonstrates that the resulting CFG-enriched embeddings serve as effective inductive priors for downstream tasks under label scarcity. We compare the proposed embeddings with raw channel tokens and multiple baseline models, under three different data regimes, for multiple downstream tasks --- Line-of-Sight/Non-Line-of-Sight (LoS/NLoS) classification, timing advance (TA) estimation, and beam selection. Evaluation is done in both in-distribution and out-of-distribution configuration settings. Experimental outcomes show that the proposed embeddings consistently outperform and/or are competitive to the baselines, across experiments in both seen and unseen configuration scenarios.

I. INTRODUCTION

Radio foundation models learn embeddings from unlabeled channel data and have shown downstream generalization, but existing work rarely encodes the channel-generation configuration explicitly. This paper proposes adding CFG to the RFM token sequence and evaluates its embeddings across tasks, training-data sizes, and seen and out-of-distribution settings.

I. INTRODUCTION

  • RFMs typically learn embeddings from unlabeled channel frequency responses or received signals, and prior work reports downstream generalization with limited task-specific labels.The paper focuses on the input data used for RFM pretraining.Source: §I · p. 1

A. Related work

  • Unlike most channel-RFM work, this paper explicitly incorporates channel-generation configuration into pretraining; prior configuration diversity approaches use variable-length tokenization or MoE routing.The cited exception incorporates base-station position for positioning, rather than explicitly encoding channel-generation CFG.Source: A. Related work · p. 1
  • The study tests whether CFG-enriched embeddings help generalization, especially in OOD settings, using frozen embeddings across downstream tasks and training-data sizes.The probing evaluation uses CFR data from DeepMIMO with the CFG used to generate it.Source: A. Related work · p. 1

B. Research Questions

  • The research questions concern CFG enrichment’s impact on existing embeddings and the efficacy of new CFG-enriched embeddings across data sizes and seen and OOD scenarios.These questions address an area where prior work had not leveraged CFG during RFM pretraining.Source: B. Research Questions · pp. 1–2

C. Contributions

The paper evaluates CFG-enriched RFM embeddings as compact alternatives to raw channel inputs and as a way to incorporate configuration into pretraining. Its architecture adds CFG tokens to channel tokens while retaining the baseline’s general transformer design.

C. Contributions

  • RFM embeddings outperform raw data with or without raw CFG augmentation, and CFG embeddings offer a low-dimensional alternative to lighter CLS or heavier channel embeddings.The work assesses enrichment of existing embeddings as well as the viability of CFG embeddings.Source: C. Contributions · p. 2
  • CFG encoding during RFM pretraining yields competitive-to-better downstream performance, with its positive impact more often visible in OOD scenarios.The paper presents CFG as an inductive prior for downstream generalization.Source: C. Contributions · p. 2

A. Architecture

  • The baseline CFR-based RFM uses a multi-layer, multi-head attention Transformer encoder and a simple fully connected decoder.The paper uses this architecture to demonstrate an idea it says is applicable to any RFM architecture.Source: A. Architecture · p. 2, Figure 1 · p. 2
  • The baseline tokenizes CFR into nonoverlapping real- and imaginary-component patches, then derives CLS, mean-pooled, or max-pooled channel embeddings.The CLS embedding is the output at the CLS position after joint Transformer processing; pooled channel embeddings aggregate over channel tokens.Source: A. Architecture · p. 2
  • The proposed input inserts CFG tokens between CLS and channel tokens, then forms embeddings from CLS output or mean/max pooling over CFG and CFG-enriched channel tokens.The paper also evaluates pooling across the entire sequence; the rest of the architecture is similar to the baseline.Source: A. Architecture · p. 2

B. Hypothesis

Channel frequency response depends on transmitter, receiver, and medium properties summarized by CFG. The paper hypothesizes that CFG tokens provide a low-dimensional inductive prior for downstream generalization, particularly with few labels or unseen configurations.

B. Hypothesis

  • A channel frequency response is governed by transmitter, receiver, and medium properties, including antenna geometry, carrier frequency, bandwidth, and subcarrier spacing.The paper groups these properties into CFG tokens.Source: B. Hypothesis · pp. 2–3
  • The MCM objective predicts masked H tokens from visible ones; adding CFG tokens extends the learned correlations to H–CFG patterns.CFG embeddings are initialized through a learnable linear projection before pretraining.Source: B. Hypothesis · p. 3
  • The paper hypothesizes that pooled CFG embeddings provide a low-dimensional inductive prior, especially with limited labeled data or unseen configurations.It compares CFG and H,CFG pooled embeddings with baseline channel and CLS embeddings, and with fully connected models using channel inputs with or without CFG augmentation.Source: B. Hypothesis · p. 3

III. EXPERIMENTS AND RESULTS

The experiments pretrain baseline and CFG-enriched RFMs on diverse DeepMIMO configurations, then evaluate three downstream tasks on a new scenario. The setup includes multiple configuration variants and derives TA labels from path delays.

A. Pretraining configurations

  • The baseline RFM is pretrained without CFG tokens, while the proposed RFM includes them; pretraining spans eight DeepMIMO scenarios with diverse configurations.Table I summarizes configurations yielding approximately 126K channel samples from 30 unique CFG variants.Source: A. Pretraining configurations · p. 3, Table I · p. 3

1) Tasks:

  • Evaluation on a new scenario covers LoS/NLoS classification, TA estimation, and beam selection with codebook sizes 8, 16, and 32.Labels come from DeepMIMO except for TA, which is computed from path delays.Source: 1) Tasks · p. 3

A. Pretraining configurations

  • Table II lists the architecture and pretraining hyperparameters used in the experiments.Source: Table II · p. 3

2) Scenarios:

The evaluation distinguishes configurations sampled during pretraining from new CFG combinations. It compares raw channel-token baselines with embeddings from RFMs trained with and without CFG information.

2) Scenarios:

  • Seen testing uses a CFG combination sampled from pretraining, whereas unseen testing uses new combinations not observed during pretraining.Source: 2) Scenarios · p. 3
  • The baselines use raw H tokens alone or with CFG tokens, while the baseline RFM uses embeddings pretrained without CFG.Source: 2) Scenarios · p. 3
  • The proposed RFM supplies CFG-enriched embeddings and CFG-augmented channel embeddings for comparison with the baselines.Source: 2) Scenarios · p. 3

3) Downstream Model Architecture:

A fixed FCN head is applied across embeddings, and evaluation spans multiple downstream training-data regimes. The data-rich threshold is motivated by the low-dimensional structure of the channel data and embeddings.

3) Downstream Model Architecture:

  • A fixed FCN head is used uniformly across embeddings, with 50 epochs, learning rate 10^-3, batch size 8, dropout 0.3, and early stopping patience 5.Downstream models train on a subset of seen CFG conditions and are tested on seen and unseen conditions.Source: 3) Downstream Model Architecture · p. 3

4) Downstream Training-Data Regimes:

C. Results and Analysis

Across the reported results, seen-scenario performance generally improves with more data, but that trend does not generally hold for unseen configurations. For LoS/NLoS classification, RFM embeddings substantially exceed raw-token baselines, and the joint H-CFG representation is strongest in unseen settings.

C. Results and Analysis

  • For seen configurations, performance generally improves with training-data size; for unseen configurations, this trend does not generally hold and depends on downstream generalization.Tables IV and VI cover seen configurations, while Tables V and VII aggregate unseen-configuration performance.Source: C. Results and Analysis · p. 4

1) LoS/NLoS classification:

  • Raw H-token baselines perform well below all 64-dimensional embeddings across regimes, with a wider gap in unseen classification; adding CFG tokens gives only modest seen data-rich gains and negligible unseen improvement.Source: 1) LoS/NLoS classification · p. 4
  • RFM embeddings saturate rapidly in seen classification, with B3 and proposed embeddings differing by at most 0.001 F1 across regimes; performance variance is low across conditions.Source: 1) LoS/NLoS classification · p. 4
  • In seen classification, CFG-pooled embeddings are indistinguishable from E† CLS.Source: 1) LoS/NLoS classification · p. 4
  • In unseen LoS/NLoS classification, E† CLS consistently outperforms all embeddings across data regimes, demonstrating stronger OOD robustness for the joint H-CFG representation.Source: 1) LoS/NLoS classification · p. 4

2) TA Estimation:

For TA estimation, CFG-enriched embeddings improve on raw-token baselines, with the clearest gains under label scarcity; the value of CFG and pooling choices varies by task and setting. Beam selection results also show that OOD behavior changes as beam discrimination becomes finer.

2) TA Estimation:

  • Raw channel-token baselines remain substantially inferior to 64-dimensional embeddings across TA regimes, especially in unseen configurations.Adding TCFG tokens alone yields no consistent improvement over B1.Source: 2) TA Estimation · p. 4, 3) Beam Selection: Beam-8 · p. 4
  • In seen configurations, most proposed embeddings outperform the best B3 embedding in few-shot TA estimation, while remaining competitive as data increases.In unseen configurations, B3 and proposed embeddings are comparable in few-shot and moderate regimes, and proposed embeddings match or outperform B3 in the data-rich regime.Source: 2) TA Estimation · p. 4
  • Channel-augmented aggregate embeddings outperform CFG-only pooled embeddings across seen TA regimes; CFG adds value but cannot substitute for H.The passage attributes this to TA being a function of time of arrival extractable from the channel impulse response.Source: 2) TA Estimation · p. 4
  • Few-shot TA estimates show high variance for nearly all embeddings on seen and unseen configurations, reflecting instability with very few labeled samples.The passage describes the few-shot regime as inherently unstable for TA regression.Source: 3) Beam Selection: Beam-8 · p. 4
  • E† CLS leads among seen-scenario RFM embeddings in few-shot and moderate TA regimes, while B3 ECLS leads in the data-rich regime.In unseen settings, E† CLS and ECFG,max lead in few-shot and data-rich regimes, though B1 occasionally edges ahead in the moderate regime.Source: 3) Beam Selection: Beam-8 · p. 4

Beam Selection: Beam-8

  • For Beam-8, the seen-to-unseen gap grows with training size, while RFM embeddings have low variance across unseen training sizes.The passage links the growing gap to greater overfitting at larger sizes and describes few-shot performance as relatively more stable under distribution shift.Source: 3) Beam Selection: Beam-8 · p. 5

Beam Selection: Beam-16 and Beam-32

Beam Selection

  • In the unseen scenario, EHmax (B3) dominates across all regimes for both tasks, while proposed embeddings fail to match B3; Beam-16 and Beam-32 overall F1 scores are substantially lower than Beam-8.Source: 4) Beam Selection: Beam-16 and Beam-32 · p. 6

D. Summary

Across tasks, regimes, and seen and OOD settings, CFG-containing embeddings are best or competitive against baselines, but their utility depends on task and data regime.

D. Summary

  • Across downstream tasks, training regimes, and seen and OOD scenarios, CFG-containing embeddings achieve the best or competitive performance against baselines.The paper reports more pronounced gains in OOD settings.Source: D. Summary · p. 6
  • CFG benefits LoS/NLoS classification and beam selection more consistently than TA prediction, where H remains the dominant factor..Source: D. Summary · p. 6
  • CFG-enriched embeddings tend to outperform CFG-augmented embeddings in few-shot and moderate regimes, while augmented embeddings are more competitive with abundant data.The passage suggests effective CFG augmentation requires more training data, whereas CFG-enriched embeddings provide more value under label scarcity.Source: D. Summary · p. 6

E. T-SNE visualization of embeddings

The t-SNE visualizations show clearer cluster separation for CFG-enriched embeddings, which the passage interprets as propagation-geometry conditioning organizing the latent space along meaningful axes.

E. T-SNE visualization of embeddings

  • CFG-enriched embeddings show clearer t-SNE cluster separation; the passage says CFG tokens pull same-class representations closer and separate propagation regimes.The passage connects this structure to more generalizable downstream decision boundaries.Source: E. T-SNE visualization of embeddings · p. 6

IV. CONCLUSION

Compact 64-dimensional CFG-enriched embeddings outperform raw channel tokens across tasks and regimes, especially in few-shot and OOD settings. Their limits emerge for fine-grained beam discrimination, where the CFG tokens lack sufficient spatial resolution.

IV. CONCLUSION

  • Compact 64-dimensional CFG-enriched embeddings consistently outperform high-dimensional raw channel tokens across all three tasks and training regimes, particularly in few-shot and OOD regimes.E† CLS demonstrates robust OOD generalization for classification; gains are less consistent for moderate-OOD TA, and B3 ECLS remains competitive in data-rich beam selection.Source: §IV · p. 6
  • CFG tokens lack the fine spatial resolution needed when adjacent beams differ by only a few degrees, as in Beam-16 and Beam-32.The passage identifies richer CFG descriptors as a future-work direction.Source: §IV · p. 6

Related papers

Loading 2610.32542v1…