PaperlayerView exact revision on arXiv

Source-linked AI summary

LWM-TF: A Wireless Foundation Model on 5G NR Time-Frequency Resource Grids

Namhyun Kim, Sadjad Alikhani, Ahmed Alkhateeb

arXiv:2610.21013v1cs.ITeess.SPmath.IT

TL;DR

Wireless foundation models have rarely pre-trained on received OFDM grids, which preserve complex pilot and payload observations alongside channel and noise effects. LWM-TF pre-trains a masked autoencoder on these grids and evaluates its frozen encoder across receiver tasks. Its probes perform strongly across tasks, especially with scarce labels, on a separately generated test set.

  • Problem

    Existing wireless foundation-model inputs omit some combination of payload observations, phase, or the resource-element layout needed for a shared receiver representation.

    Source: §I · p. 1

  • Method

    LWM-TF pre-trains a masked autoencoder on received PUSCH grids, reconstructing masked patch-normalized log-power and predicting patch scale.

    Source: §I · p. 1

  • Results

    With 10,000 labels, frozen-encoder probes reach 0.939 modulation accuracy, 0.964 CRC AUROC, and 0.92 mobility AUROC on a separately generated test set, well above engineered receiver statistics.

    Source: §VII · p. 15

  • Takeaways & Limitations

    One frozen encoder provides features that transfer across modulation, CRC, and mobility tasks, with the strongest advantage reported when labels are scarce.

    Source: §I · p. 1

Abstract

from arXiv · show

Received OFDM grids preserve both pilot and payload observations, yet wireless foundation models are rarely pre-trained on them. We introduce LWM-TF, a 9.58M-parameter factorized masked autoencoder pre-trained on complex 5G NR uplink grids taken after the FFT and before channel estimation. Each block attends first across OFDM symbols and then across resource blocks and receive antennas, and the mask follows the same two axes. Because the payload phase is unpredictable, the model reconstructs the normalized log-power of each masked token and predicts its scale separately; ablations show that the two terms are complementary and that this target outperforms complex-signal reconstruction. The encoder is pre-trained on 850,496 physical uplink shared channel (PUSCH) slots from a 10^6-slot dataset simulated with the NR transmit chain under various channel conditions. With the encoder frozen, linear probes classify modulation, predict CRC success, and detect UE mobility. On a separately generated test set, 16 labeled slots per class give 0.85 modulation accuracy, 0.82 CRC area under the ROC curve (AUROC), and 0.73 mobility AUROC, ahead of a supervised CNN on every paired draw. With 10,000 labels the probes reach 0.94, 0.96, and 0.92, well above engineered receiver statistics, and detect mobility where the CNN stays at chance. Without any impairment labels, distances in the frozen feature space flag unseen waveform impairments with 0.78 macro AUROC, against 0.68 for receiver statistics. A single pre-trained encoder thus provides receiver features that transfer across tasks, most strongly when labels are scarce.

I. INTRODUCTION

Wireless PHY models are usually task-specific, while existing wireless foundation models use inputs that omit some combination of payload, phase, or resource-grid structure. LWM-TF pre-trains on received OFDM grids and evaluates whether one frozen representation supports multiple receiver tasks.

I. INTRODUCTION

  • Wireless PHY models are often developed separately for each task, requiring new labeled data and training cycles rather than reusing a common representation.Foundation models offer an alternative: pre-train an encoder on unlabeled data, then adapt it to multiple tasks with lightweight heads.Source: §I · p. 1

A. Related Work

  • Existing wireless foundation models use channel data, spectrograms, I/Q streams, or combinations, each omitting some received-grid information needed for a shared receiver representation.LWM-TF instead uses complex pilot and payload observations indexed by symbol, subcarrier, and antenna, before channel estimation and equalization.Source: §I · p. 1, A. Related Work · p. 2

B. Contributions

  • LWM-TF factorizes attention and masking across the grid’s symbol and antenna–PRB axes, reconstructing patch-normalized log-power with a per-patch scale target.Its log-power target with scale supervision outperforms complex-grid reconstruction in controlled ablations.Source: B. Contributions · p. 2, A. Related Work · p. 2
  • A frozen LWM-TF encoder supports modulation, CRC, and mobility probes; with 10,000 labels, they reach 0.939 accuracy, 0.964 AUROC, and 0.92 AUROC, respectively.The dataset uses a 3GPP-compliant PUSCH transmit chain and TR 38.901 channel models, with a separately generated test set.Source: B. Contributions · pp. 2–3

II. LINK-LEVEL SIGNAL GENERATION

The study simulates a fixed 5G NR uplink and forms received grids directly on a single UE’s allocation. Channel and noise calibration places stochastic scenarios on a common SNR distribution while retaining propagation structure.

A. Link Configuration and Transmit Chain

  • Each simulated slot is a rank-1 SIMO uplink with one UE allocated 24 PRBs, 14 OFDM symbols, and four receive ports.The allocation represents one UE’s scheduled resources, not a full carrier.Source: A. Link Configuration and Transmit Chain · p. 3
  • The transmit chain follows 3GPP TS 38.211/38.212, including CRC, LDPC coding, scrambling, QAM mapping, and DMRS resource-grid mapping.DMRS additional-position settings 0–3 use symbol sets {2}, {2, 11}, {2, 7, 11}, and {2, 5, 8, 11}.Source: A. Link Configuration and Transmit Chain · p. 3

B. Channel and Noise Calibration

  • The dataset also includes ray-traced DeepMIMO links, which complement the stochastic channel scenarios with site-specific propagation.Across the compared links, the reported median RMS delay spreads and in-band |H| ranges differ between UMa and ray-traced channels.Source: B. Channel and Noise Calibration · p. 4
  • The received grid is formed per resource element from the transmitted grid, channel response, and noise; it represents the post-FFT observation before channel estimation.The setup omits OFDM modulation and demodulation, and therefore neglects inter-symbol and inter-carrier interference.Source: B. Channel and Noise Calibration · p. 4
  • Target-SNR calibration makes the three stochastic scenarios share an identical SNR distribution, leaving their propagation structure as the difference between them.The study uses γ = 0 to remove the SNR shortcut for identifying LoS links; LoS channels remain flatter across the band, and the frozen encoder reads the LoS label at AUROC 0.82.Source: B. Channel and Noise Calibration · pp. 4–5

C. Link Adaptation and Modulation-Balanced Sampling

The section sets a static SINR margin for link adaptation and balances modulation classes independently of SINR to reduce shortcuts in modulation classification.

C. Link Adaptation and Modulation-Balanced Sampling

  • A 3 dB SINR margin raises the ILLA proposal’s CRC pass rate from 57% to 86%, at a cost of 10% in mean spectral efficiency.The receiver decodes from a least-squares channel estimate, while ILLA uses the true channel.Source: C. Link Adaptation and… · p. 5
  • QPSK, 16-QAM, and 64-QAM are drawn uniformly and independently of SINR, then mapped to the highest supported MCS in the selected class.If no lower valid index exists, the class’s lowest valid index is used.Source: C. Link Adaptation and… · p. 5
  • Modulation-balanced sampling changes the modulation–SINR rank correlation from +0.84 to −0.00 over 106 slots, with class frequencies changing from 14/20/66% to 33/33/33%.This prevents the modulation probe from using noise level instead of constellation structure; the class-constrained final MCS need not meet the 0.1 BLER criterion.Source: C. Link Adaptation and… · p. 5

D. Receiver Chain and Labels

The receiver chain supplies CRC decodability labels while LWM-TF takes its input before receiver-specific channel estimation. The model tokenizes and masks the received grid, with factorized-uniform masking selected after comparison with DMRS-stratified masking.

D. Receiver Chain and Labels

  • The 3GPP receiver produces a CRC decodability flag for every slot, and 20.5% of the 106 slots fail CRC; labels are reserved for downstream probes, not pretraining.The stored input is per-sample RMS-normalized because receive levels span approximately 10^-10 to 10^-4.Source: D. Receiver Chain and Labels · p. 5
  • The encoder uses the received grid before channel estimation because channel estimates discard data REs and depend on estimator choices, while the channel remains observable in the grid.Equalization would divide the channel out of data REs.Source: D. Receiver Chain and Labels · p. 5

A. Input Tensor and Patch Embedding

  • Each token represents one PRB on one receive antenna in one OFDM symbol; a linear layer maps its 24 real I/Q values to width d.The input has shape (8, 14, 288), yielding 1,344 tokens of width 256, with learned time and antenna–PRB embeddings.Source: A. Input Tensor and Patch Embedding · pp. 5–6

III. MODEL ARCHITECTURE: LWM-TF

  • LWM-TF is a factorized masked autoencoder that reconstructs masked PRB-token log-power with patch-scale heads, while downstream tasks probe the mean-pooled frozen encoder.The architecture overview presents pretraining and downstream probing as distinct stages.Source: §III · p. 5, Figure 5 · p. 6

B. Mask Schedule

  • Factorized-uniform and DMRS-stratified masking differ by at most 0.001 on either primary task across three-seed means, so the main model uses factorized-uniform masking.The seed-0 paired bootstrap intervals include zero; the default keeps 7 of 14 symbols and 48 of 96 spatial positions, leaving 336 of 1,344 tokens visible.Source: B. Mask Schedule · p. 6

C. Asymmetric Encoder–Decoder

LWM-TF uses an asymmetric factorized encoder–decoder and a label-free objective combining masked log-power reconstruction with per-patch scale targets.

C. Asymmetric Encoder–Decoder

  • Each encoder block attends across OFDM symbols at each antenna–PRB position, then across antenna–PRB positions within each symbol, before applying an MLP.The encoder processes visible tokens only; its attention uses 148k pair terms per layer rather than 1.8M for joint attention at T = 14 and S = 96.Source: C. Asymmetric Encoder–Decoder · p. 6
  • The lightweight decoder restores the full 14 × 96 grid with mask tokens and predicts each patch’s log-power target.It has four factorized blocks of width 128 and maps each patch to 12 values.Source: C. Asymmetric Encoder–Decoder · p. 6

D. Per-Patch Auxiliary Heads

  • Two scale heads regress the same per-patch statistic on visible encoder tokens and masked decoder tokens, each at weight 0.1; relative-phase loss is used only in P5.The per-patch scale target restores the statistic removed by patch normalization at the same granularity.Source: D. Per-Patch Auxiliary Heads · pp. 6–7

E. Model Configuration and Scaling

  • The model has 9.58 M parameters, including an 8.46 M encoder and a 1.12 M decoder; pretraining runs for 5,000 steps before the main model continues to 40,000.The encoder’s eight factorized blocks account for 8.43 M parameters.Source: E. Model Configuration and Scaling · p. 7

A. Multi-Loss Objective

  • The label-free objective combines masked log-power reconstruction with two per-patch scale terms; all targets are computed from the input grid without downstream labels, pilot layout, or simulator quantities.The main model sets βenc = βdec = 0.1 and βφ = 0; βφ = 1 only in P5.Source: A. Multi-Loss Objective · p. 7

B. Masked Log-Power Reconstruction

Masked reconstruction targets patch-normalized log-power rather than complex samples because scrambled payload phase makes complex prediction degenerate on many masked data elements. The target reveals channel-envelope structure, but random symbol amplitudes and noise limit exact per-element recovery.

B. Masked Log-Power Reconstruction

  • Patch-normalized log-power is reconstructed over masked patches, while patch scale is supervised separately; normalization prevents high-power patches from dominating loss.The reconstruction target uses patch statistics, and each masked-patch resource element contributes, including DMRS and empty pilot-comb elements.Source: B. Masked Log-Power Reconstruction · p. 7
  • At 18–22 dB, adjacent-subcarrier correlation is 0.00 for y versus 0.49 for |y|2, motivating power rather than complex-signal reconstruction.The transmit chain scrambles the codeword before modulation, so the paper models payload symbols as independent and zero-mean.Source: B. Masked Log-Power Reconstruction · pp. 7–8
  • The channel-power proxy has squared correlation 0.45 with log |y|2 overall and 0.62 at 18–22 dB, while normalized within-patch correlation is 0.09.The token means correlate at 0.87; for QPSK, normalized within-token correlation is 0.19.Source: B. Masked Log-Power Reconstruction · p. 8
  • Random payload amplitudes and receiver noise leave an error floor, so reconstruction loss is a pre-training signal rather than a measure of representation quality.For QPSK, fixed symbol amplitude lets the channel envelope explain nearly all signal variation; 64-QAM retains random symbol amplitudes.Source: B. Masked Log-Power Reconstruction · p. 8
  • R2 reaches 0.62 per RE and 0.85 for the patch mean when a whole OFDM symbol is hidden.Across 256 test slots, predictions restore the fading pattern in fully hidden symbols and in masked regions of visible symbols.Source: B. Masked Log-Power Reconstruction · p. 8

C. Auxiliary Targets

The auxiliary targets restore information removed by patch normalization or omitted from the log-power reconstruction. Patch-scale supervision is used in the main configuration, while relative phase provides an additional phase-bearing target.

C. Auxiliary Targets

  • The relative-phase target is optional, with zero loss weight in the main configuration.Source: C. Auxiliary Targets · p. 8
  • Patch-scale supervision restores mean and variance removed by normalization, using received log-power statistics from all resource elements in each patch.The statistics include non-pilot resource elements in DMRS-bearing symbols and describe received log-power, not SINR or channel power.Source: C. Auxiliary Targets · pp. 8–9
  • Relative phase uses the second moment across receive ports, retaining phase information without the degeneracy of complex first-moment reconstruction.The normalized off-diagonal covariance carries relative phase; its magnitude is already represented by the diagonal.Source: C. Auxiliary Targets · p. 9
  • At 18–22 dB, normalized relative phase tracks channel phase with 5.3° median error and 0.96 complex correlation.Across the full SNR range, median error is 12°; adjacent-subcarrier correlation is 0.83 for normalized relative phase versus 0.60 for the raw product.Source: C. Auxiliary Targets · p. 9
  • Both auxiliary losses are computed per patch because normalization removes scale and power reconstruction omits phase at that same granularity.A per-sample target on the mean-pooled readout could recover only one scalar per slot, insufficient for a frequency-selective channel.Source: C. Auxiliary Targets · p. 9

D. Training Configuration

The study pre-trains on a large simulated PUSCH pool, compares objective and data-pool variants, and continues its main model beyond the initial matched-budget runs. The 40,000-step continuation improves all reported test tasks over 5,000 steps.

D. Training Configuration

  • The 850,496-slot pre-training pool is distinct from 149,504 development/validation slots; D0 uses 423,936 source-only slots.Table IV summarizes cumulative per-run exposure, counting repeated presentations rather than distinct slots.Source: D. Training Configuration · pp. 9–10, Table IV · p. 10
  • The main model continues three seeded runs to 40,000 total steps; the horizon was selected using seed-0 development validation.P0–P4 and D0 begin with 5,000 steps, batch size 512, and seeds 0, 2, and 3; P5 adds relative-phase loss to P0.Source: D. Training Configuration · p. 9
  • At 80,000 steps, seed 0 still improves modulation accuracy from 0.935 to 0.945 and speed R2 from 0.49 to 0.60, but not CRC AUROC.The reported three-seed results use the shared 40,000-step checkpoint, making them conservative relative to that exploratory continuation.Source: D. Training Configuration · p. 10
  • From 5,000 to 40,000 steps, modulation accuracy rises from 0.906 to 0.938, CRC AUROC from 0.953 to 0.962, speed R2 from 0.275 to 0.529, and mobility AUROC with 1,000 labels from 0.75 to 0.89.The continuation results show the representation was not saturated at 5,000 steps.Source: D. Training Configuration · p. 10

V. DATA REPRESENTATION

The encoder receives a normalized four-antenna NR resource grid and downstream probes use its pooled embedding. The evaluation reads out task labels with the encoder frozen, spanning classification, receiver-specific CRC prediction, mobility detection, and controls or stress tests.

V. DATA REPRESENTATION

  • The encoder input is a normalized 8 × 14 × 288 real I/Q tensor formed from the four-antenna complex received grid.The grid is standard-aligned to NR resource elements, which defines PRB tokens and exposes empty DMRS-comb elements for receiver-statistics noise measurement.Source: §V · p. 10
  • At inference, probes receive only the mean of full-grid encoder features, with no decoder output, per-token features, or metadata.CRC and other physical labels are stored separately and never enter pre-training; targets are deterministic functions of the input grid.Source: §V · p. 10

A. Tasks

  • Frozen linear probes evaluate modulation classification and receiver-specific CRC decodability as primary tasks, with modulation labels balanced and uncorrelated with SNR.CRC decodability predicts the pass/fail outcome of the specified LDPC, LMMSE, and LS receiver; it does not itself decode.Source: A. Tasks · pp. 10–11
  • Mobility detection separates links at UE speed ≥20 km/h, while SNR estimation is a control and unseen waveform impairments form a stress test.Mobility methods and contrasts were fixed before scoring; anomaly detectors see clean grids only, making all three impairment families unseen.Source: A. Tasks · p. 11
  • Five secondary diagnostics use the same frozen probes: DMRS position, LoS/NLoS, scenario, distance, and UE speed regression.Source: A. Tasks · p. 11

B. Probe and Evaluation Protocol

The evaluation uses frozen linear probes and matched controls across two separately reported test sets. At the full label budget, the frozen encoder outperforms the receiver-statistic MLP on CRC and modulation, while its modulation margin over the evaluated CNN is not statistically resolved.

B. Probe and Evaluation Protocol

  • The frozen probes use separately generated test sets, matched label indices, fixed probe hyperparameters, and fit-split statistics for embedding standardization.The main evaluation uses new samples from the same scenarios and simulator; a separate analysis tests unseen scenarios and DMRS configurations within the TR 38.901 simulator.Source: B. Probe and Evaluation Protocol · pp. 11–12

C. Main Results

  • SNR estimation reaches R2 = 0.9992 with the parameter-free estimator and 0.9992–0.9993 with receiver-statistic probes, so it is not evidence for the learned representation.The paper says SNR is solved in closed form on this input.Source: C. Main Results · p. 12
  • Modulation accuracy exceeds the receiver-statistic MLP by 0.39 on both test sets, while the 0.032–0.033 margin over the evaluated CNN has crossed intervals that include zero.At the full budget, the paper characterizes the CNN and frozen encoder as on par for modulation.Source: C. Main Results · p. 12
  • The randomly initialized encoder is near chance on modulation at 0.3717 but reaches 0.8479 on CRC, consistent with random features retaining some link-quality information.The scratch transformer reaches 0.6416 on modulation and 0.8723 on CRC; these comparisons are descriptive, not predeclared.Source: C. Main Results · p. 12
  • The protocol also probes DMRS position as a positive control and reports speed and scenario diagnostics, while pooled distance prediction does not indicate ranging.Within each scenario, distance prediction has negative R2; refitting within each scenario reaches only R2 ≈−0.06.Source: C. Main Results · p. 12
  • CRC AUROC exceeds the receiver-statistic MLP by 0.075–0.082 and the evaluated CNN by 0.019–0.020 across the two test sets; all four crossed intervals exclude zero.The CNN was trained without data augmentation, so the paper treats this margin as specific to that recipe.Source: Table V · p. 12, Figure 9 · p. 12, C. Main Results · p. 12
  • Single-slot inference takes 5.18 ms, while batch-32 inference reaches 1,431 slots/s; the encoder is therefore suited to batched, non-real-time analysis rather than tight per-slot latency.The paper notes that a smaller task-specific model favors latency-critical processing.Source: C. Main Results · pp. 12–13
  • The frozen embedding separates modulation orders qualitatively, and its 20 nearest neighbors share modulation 85% of the time, versus 33% for random initialization.The t-SNE projection is qualitative; the quantitative claims rely on probe results.Source: Figure 10 · p. 12, C. Main Results · p. 13

D. Channel Time-Variation Detection

The frozen encoder detects channel time variation with few labels, including where supervised controls remain near chance. Its label-efficient probes also outperform the evaluated CNN on every paired draw with 16 labeled slots per class, while practical suitability depends on latency and task count.

D. Channel Time-Variation Detection

  • The mobility task tests whether pretraining retains channel variation across the OFDM symbols of a slot, which the grid CNN does not discover from labels.The lower block of Table VI checks performance within scenario–DMRS settings and without stationary slots; a setting-only predictor is at chance.Source: D. Channel Time-Variation Detection · p. 13
  • With 10,000 labels, the frozen mobility probe reaches 0.92 AUROC and exceeds Doppler features by 0.099 [0.088, 0.111]; the CNN remains at chance.At 100 and 1,000 labels, the frozen probe reaches 0.79 and 0.89, respectively; results hold within scenario–DMRS settings and without stationary slots.Source: Table VI · p. 13, D. Channel Time-Variation Detection · p. 13

E. Label Efficiency

  • With 100 labels, the frozen probe reaches 0.87 modulation accuracy, 0.89 CRC AUROC, and 0.79 mobility AUROC, leading at every budget and on every seed.The receiver-statistic probes reach 0.45 modulation accuracy and 0.86 CRC AUROC with 100 labels, and gain little with more labels.Source: Figure 11 · p. 13, E. Label Efficiency · p. 13
  • With 16 labeled slots per class, the frozen probe averages 0.86 modulation accuracy, 0.86 CRC AUROC, and 0.73 mobility AUROC across five shared draws, beating the CNN on every draw.The CNN averages 0.40, 0.65, and 0.51 on those tasks; the 80,000-step continuation reaches 0.87, 0.83, and 0.84, respectively, at K = 16.Source: E. Label Efficiency · p. 13, Figure 12 · p. 14
  • The CRC probe varies most with its support set, but at K = 16 it exceeds receiver statistics by 0.03 [0.02, 0.04] and at K = 64 by 0.06 [0.05, 0.06].At K = 64 it wins all 100 draws; the support-draw SD is 0.06 at K = 16 versus 0.17 at K = 1.Source: E. Label Efficiency · p. 13
  • One encoder pass can serve several tasks and runs efficiently on batched slots, but latency-critical single-task use favors the cheaper CNN, and SNR favors closed-form estimation.The label-efficiency and few-label analyses were designed after the primary results, with protocols fixed before scoring.Source: E. Label Efficiency · pp. 13–14

F. Objective and Masking Ablations

Objective ablations favor log-power reconstruction with scale supervision, while relative-phase supervision trades payload-task performance for spatial readout. Transfer is strong in the tested held-out settings, but detection does not transfer to held-out impairment families.

F. Objective and Masking Ablations

  • Removing scale target costs 0.024 modulation accuracy and 0.005 CRC AUROC; removing reconstruction costs 0.012 and 0.004, while complex-I/Q trails log-power by 0.014 and 0.004.Source: F. Objective and Masking Ablations · p. 14
  • At 40,000 steps, relative-phase supervision raises LoS AUROC from 0.820 to 0.872 and scenario accuracy from 0.575 to 0.618, but lowers modulation, CRC AUROC, and speed R2.Modulation accuracy falls from 0.938 to 0.911, CRC AUROC from 0.962 to 0.953, and speed R2 from 0.529 to 0.395.Source: F. Objective and Masking Ablations · p. 14

G. Held-Out Scenario and DMRS Configuration

H. Waveform Anomaly Detection

F. Objective and Masking Ablations

  • P5 explains 59% of the label-free port-covariance index variance versus 47% for P0, supporting added inter-port structure; its modulation cost may reflect untuned loss weight or target competition.Source: F. Objective and Masking Ablations · p. 15

H. Waveform Anomaly Detection

  • In held-family detection, P0 has no advantage (macro AUROC 0.613 versus 0.637 for the random encoder); measured interference and converter effects remain a natural extension.Source: H. Waveform Anomaly Detection · p. 15

VII. CONCLUSION

  • With 10,000 labels, probes reach 0.939 modulation accuracy, 0.964 CRC AUROC, and 0.92 mobility AUROC; with 16 labels per class, they lead the CNN on every paired draw.Source: §VII · p. 15

APPENDIX A WAVEFORM-DOMAIN ANOMALY INJECTION

The appendix describes waveform-domain anomaly generation and the receiver-statistic features used as a control.

APPENDIX A WAVEFORM-DOMAIN ANOMALY INJECTION

  • Anomaly links are generated as time-domain PUSCH waveforms, passed through TR 38.901 channels and noise, then given at most one injected impairment before FFT and grid normalization.The three families are NBI, bursts, and ADC-like clipping/quantization; each parent is stored with a clean grid and six impaired variants, with parent links kept within one split.Source: Appendix A · pp. 15–16

APPENDIX B RECEIVER STATISTICS

  • The seven receiver statistics summarize grid log-power, noise/data/pilot power relative to total power, an SNR estimate, and data-power variation across antennas.Source: Appendix B · p. 16

Related papers

Loading 2610.21013v1…