PaperlayerView exact revision on arXiv

Source-linked AI summary

RF-GPT: Teaching AI to See the Wireless World

Hang Zou, Yu Tian, Bohao Wang, Lina Bariah, Samson Lasaulce, Chongwen Huang, Mérouane Debbah

arXiv:2602.14833v1eess.SPcs.LG

TL;DR

RF foundation models lack native access to RF signals, while telecom LLMs are largely text-centric and conventional RF models remain task-specific. RF-GPT converts IQ waveforms into spectrograms, uses a pretrained vision encoder to produce RF tokens for a multimodal LLM, and trains on synthetic instruction data. Across multiple RF benchmarks, RF-GPT performs strongly while general-purpose VLMs without RF grounding largely fail.

  • Problem

    RF signals are not natively supported by foundation models, and existing telecom LLMs and RF models remain separated by a modality and task-specificity gap.

  • Method

    RF-GPT maps complex IQ waveforms to spectrogram images, encodes them with a pretrained vision encoder, and fine-tunes a multimodal LLM on synthetic RF instructions.

  • Results

    RF-GPT achieves strong multi-task performance across RF benchmarks, while general-purpose VLMs without RF grounding largely fail.

  • Takeaways & Limitations

    A single RF-grounded multimodal model can provide RF understanding together with natural-language answers, structured outputs, and explanations.

Abstract

from arXiv · show

Large language models (LLMs) and multimodal models have become powerful general-purpose reasoning systems. However, radio-frequency (RF) signals, which underpin wireless systems, are still not natively supported by these models. Existing LLM-based approaches for telecom focus mainly on text and structured data, while conventional RF deep-learning models are built separately for specific signal-processing tasks, highlighting a clear gap between RF perception and high-level reasoning. To bridge this gap, we introduce RF-GPT, a radio-frequency language model (RFLM) that utilizes the visual encoders of multimodal LLMs to process and understand RF spectrograms. In this framework, complex in-phase/quadrature (IQ) waveforms are mapped to time-frequency spectrograms and then passed to pretrained visual encoders. The resulting representations are injected as RF tokens into a decoder-only LLM, which generates RF-grounded answers, explanations, and structured outputs. To train RF-GPT, we perform supervised instruction fine-tuning of a pretrained multimodal LLM using a fully synthetic RF corpus. Standards-compliant waveform generators produce wideband scenes for six wireless technologies, from which we derive time-frequency spectrograms, exact configuration metadata, and dense captions. A text-only LLM then converts these captions into RF-grounded instruction-answer pairs, yielding roughly 12,000 RF scenes and 0.625 million instruction examples without any manual labeling. Across benchmarks for wideband modulation classification, overlap analysis, wireless-technology recognition, WLAN user counting, and 5G NR information extraction, RF-GPT achieves strong multi-task performance, whereas general-purpose VLMs with no RF grounding largely fail.

I. INTRODUCTION

RF-GPT addresses the gap between text-centric telecom models and task-specific RF systems by grounding a multimodal LLM in RF spectrograms. It uses synthetic, standards-compliant data and instruction synthesis to support unified RF understanding and natural-language interaction.

  • Motivation: RF signals remain outside mainstream foundation models, while existing RF intelligence relies mainly on narrow, task-specific architectures and datasets.Telecom LLMs primarily process text and structured data rather than RF waveforms, spectrograms, or channel estimates.
  • Challenges: Realizing an RFLM remains difficult because RF data are complex-valued, high-rate time series requiring time–frequency, protocol, and propagation understanding, while expert-annotated real-world datasets are unavailable.Over-the-air collection also raises privacy concerns and may miss rare but critical interference or coexistence scenarios.
  • Approach: RF-GPT converts complex IQ samples into time–frequency spectrograms, encodes them with a pretrained vision encoder, and injects the resulting RF tokens into a language model.The framework treats spectrograms as visual inputs and uses a lightweight modality adapter to connect visual representations with the LLM.
  • Data and training: Standards-compliant synthesis covers six wireless technologies, while deterministic captions and LLM-generated instruction–answer pairs support explanations, quantitative queries, and structured outputs.The technologies include 5G NR, LTE, UMTS, WLAN, DVB-S2, and Bluetooth.
  • Results and scope: RF-GPT supports modulation classification, technology recognition, overlap analysis, and free-form RF question answering, whereas generic VLMs without RF grounding fail on the same benchmarks.The design targets a unified instruction-following interface for RF engineers, operators, and higher-level agents.

II. PROBLEM FORMULATION AND RF-GPT ARCHITECTURE

RF-GPT converts IQ waveforms into spectrogram-derived RF tokens, processes them with a vision-based encoder, and uses an adapter to condition a decoder-only language model. Its magnitude-spectrogram representation preserves key time–frequency structure while remaining lossy for phase-dependent tasks.

  • II. PROBLEM FORMULATION AND RF-GPT ARCHITECTURE: RF-GPT combines a Transformer-based RF encoder, a linear RF adapter, and a decoder-only LLM for RF-grounded text generation.The adapter projects RF embeddings into the LLM dimension before language-model conditioning.
  • A. RF Encoding via Spectrograms and Vision Encoder: RF-GPT maps complex IQ sequences to time–frequency spectrograms using STFT before visual encoding.The pipeline uses a sampling rate, window, FFT size, and hop size to construct the STFT.
  • A. RF Encoding via Spectrograms and Vision Encoder: The complex spectrogram is converted into a normalized magnitude or power image through dB log compression and grayscale or pseudo-RGB mapping.The resulting image is processed by a standard vision encoder as the complete RF encoding pipeline.
  • A. RF Encoding via Spectrograms and Vision Encoder: The image is partitioned into non-overlapping patches, linearly projected, and combined with positional embeddings to form M RF tokens.Patchification acts as the RF tokenizer and preserves each patch’s time–frequency location.
  • A. RF Encoding via Spectrograms and Vision Encoder: All patch tokens, rather than a special classification token, are passed through Transformer encoder layers to produce RF-grounded latent vectors.The encoder uses self-attention, positionwise MLPs, residual connections, and layer normalization.
  • A. RF Encoding via Spectrograms and Vision Encoder: The vision encoder serves as the RF encoder by representing modulation, resource allocation, interference, and coexistence as latent time–frequency patterns.This design accommodates RF technologies with different carrier frequencies, bandwidths, and sampling rates.
  • A. RF Encoding via Spectrograms and Vision Encoder: Magnitude spectrograms retain attributes such as bandwidth, center frequency, Doppler shifts, burst timing, sparsity, and modulation structure.They provide a rich representation for perception-oriented RF tasks without preserving all waveform information.
  • A. RF Encoding via Spectrograms and Vision Encoder: Magnitude-only spectrograms are a lossy front end that cannot support every task requiring absolute phase or precise multi-antenna phase relationships.The paper reports that the representation nevertheless supports the perception tasks evaluated in its benchmarks.

B. Language Model Architecture and RF Conditioning

RF-GPT conditions a decoder-only Transformer on RF tokens by projecting vision features into the language-model space and concatenating them with text embeddings. Standard decoder components then autoregressively generate the RF-grounded output.

  • RF Conditioning: The RF encoder produces visual tokens that a trainable linear adapter projects into the LLM embedding dimension.The projected RF tokens function as a visual–language interface before concatenation with text tokens.
  • RF Conditioning: RF and text embeddings are concatenated into one sequence, with RF tokens serving as the conditioning prefix.The decoder processes the combined sequence using standard positional embeddings.
  • Language Model Architecture: The decoder-only backbone uses RMSNorm, grouped-query attention, a causal mask, and a gated Up/Down MLP.These components form the modern pre-normalized Transformer layers used after RF conditioning.
  • Language Model Architecture: Grouped-query attention uses Hq query heads and Hk key–value heads, with queries mapped to shared key–value groups.The causal mask restricts each position to attending only to earlier or current positions.
  • RF Conditioning: The final N text positions are transformed by an output head and softmax into the conditional distribution over RF-grounded text.RF tokens are the primary architectural addition; the remaining decoder follows a modern LLM design.

C. RF-Grounded Supervised Fine-Tuning

RF-GPT is trained with RF-grounded supervised fine-tuning on waveform, instruction, and answer triplets. The RF prefix conditions answer generation, and every target answer is constructed to remain consistent with its underlying RF scene.

  • RF-Grounded SFT: RF-grounded SFT adapts a generic VLM using a synthetic instruction dataset produced by standards-compliant waveform generators.The dataset supplies RF waveforms, natural-language questions, and desired answers.
  • RF-Grounded SFT: Each training triplet pairs IQ samples with a spectrogram-related instruction and an answer such as a caption, explanation, or JSON summary.The triplets are constructed through RF spectrogram captioning and RF instruction synthesis.
  • RF-Grounded SFT: The model minimizes autoregressive cross-entropy on answer tokens while conditioning on both the RF-token prefix and the instruction.Role markers combine the instruction and answer into the instruction-tuning sequence, while gradients apply only to answer tokens.
  • RF-Grounded SFT: Because every target token is RF-grounded by construction, training encourages alignment between spectrogram patterns and RF concepts expressed in language.The stated supervision constraint requires each answer to be consistent with its underlying RF scene.

III. RF GROUNDING AND BENCHMARKING

RF-GPT’s grounding data strategy addresses the difficulty of obtaining accurate RF captions and labels from real measurements. Standards-compliant synthesis provides controlled scenes with exact latent signal variables for broad dataset coverage.

  • Dataset Construction: High-quality RF spectrogram captions require expertise in modulation, coexistence, scheduling, interference, and impairments, making manual labeling expensive and unreliable.Generic VLMs are described as insufficiently RF-aware for trustworthy caption generation.
  • Dataset Construction: Real over-the-air datasets are difficult to build because they require specialized calibrated hardware, broad measurement campaigns, and ground truth often hidden in proprietary systems.These constraints complicate accurate RF–text pair construction from real observations.
  • Dataset Construction: Standards-compliant software synthesis provides exact modulation, SNR, bandwidth, timing, resource-allocation, and other latent variables.The simulation pipeline can systematically vary technologies, numerologies, bandwidths, traffic, coexistence, and impairments.

A. Wireless Technology Waveform Generation

The paper builds a unified synthetic-waveform workflow for six wireless technologies, using standards-compliant generators, randomized configurations, channel modeling, spectrogram conversion, and complete metadata logging.

  • Technology coverage: Six technologies—5G NR, LTE, UMTS, WLAN, DVB-S2, and Bluetooth—are generated through a common workflow using technology-specific APIs and validation checks.The generators sample technology configurations and reject invalid combinations that violate waveform-toolbox or standard constraints.
  • Configuration sampling: Randomized configuration spaces cover parameters such as bandwidth, numerology, modulation and coding, frame structure, resource allocation, and reference signals.
  • Channel modeling: Channel models and impairments, including fading, frequency-selective channels, and noise, can be applied while recording exact channel and SNR parameters.
  • Spectrogram construction: Complex baseband IQ waveforms are converted with an STFT into fixed-resolution grayscale or pseudo-RGB magnitude or power spectrograms for RF encoding.
  • Metadata logging: Each accepted sample stores the waveform, spectrogram, full configuration, channel and noise parameters, and derived attributes such as users, modulation, occupancy, and overlaps.Because configurations are retained, the resulting dataset provides machine-readable ground truth for generated scenes.

B. Spectrogram Instruction Synthesis

The instruction-synthesis pipeline converts metadata-rich RF spectrogram captions into targeted, validated instruction–answer examples by selectively exposing information levels and task formats.

  • Caption design: Each spectrogram caption combines structured metadata with a dense natural-language scene description, but its fixed, over-detailed format is unsuitable for direct instruction training.
  • Caption information levels: Caption information is organized into five levels: summary, global visual, global context, signal visual, and signal context.These levels separate visually observable content from metadata-derived context at scene and signal granularity.
  • Selective composition: Instructions selectively combine information levels so each training example focuses on a limited set of related concepts.
  • Synthesis pipeline: A deterministic captioner and a text-only LLM transform caption records into diverse RF-grounded instruction–answer pairs for RF-GPT supervised fine-tuning.
  • Scope boundary: The current work mainly targets summary and visual information, leaving fine-grained signal context such as DM-RS, PTRS, and CORESET configurations for future work.
  • Task generation: The pipeline samples task templates covering counting, modulation or technology recognition, information extraction, overlap analysis, consistency checks, and open-ended description.
  • Output validation: Generated answers use task-specific formats such as labels, JSON snippets, or short explanatory paragraphs, and successful pairs retain task and difficulty metadata.

C. Benchmark Constructions

The benchmark suite evaluates RF-GPT on component recognition, counting, and time–frequency reasoning, including wide-band modulation classification with increasing task difficulty.

  • Benchmark scope: The benchmark suite tests component recognition, quantity counting, and time/frequency relationship reasoning for RF understanding.
  • WBMC setting: WBMC extends narrowband modulation classification to wideband spectrograms containing 2–5 potentially overlapping signals.
  • WBMC tasks: WBMC difficulty increases from identifying modulation families, to selecting exact classes from candidate lists, to selecting exact classes from the full modulation list.
  • WBMC scoring: The hard WBMC score requires strict temporal ordering and signal-count matching, assigning zero when the predicted and ground-truth signal counts differ.
  • Dataset construction: Each WBMC difficulty level uses 2000 VQA pairs generated with co-channel overlap probability set to zero, while overlap reasoning is evaluated separately.

2) Wide-Band Overlap Detection (WBOD):

WBOD evaluates RF-GPT’s ability to classify and quantify signal overlap in time and frequency through global, pairwise, and overlap-strength tasks.

  • Overlap representation: WBOD derives pairwise overlap from signals’ time and frequency supports, categorizing relationships as neither, time-only, frequency-only, or both.
  • Caveat: The overlap benchmark’s nominal zero-overlap setting does not completely eliminate overlapping signals.
  • Easy task: WBOD-Easy aggregates pairwise relationships into one global scene label using precedence rules for both, time-only, frequency-only, and neither.
  • Medium task: WBOD-Medium asks about specified signal pairs and measures accuracy on the predicted overlap type.
  • Hard task: WBOD-Hard estimates overlap strength separately along time and frequency rather than only determining whether overlap exists.The task quantizes each axis’s overlap ratio into levels and requires both components to match.
  • Evaluation conditions: The WBOD data use co-channel overlap probability 0.6 and SNR values from 10–50 dB.

3) Wireless Technology Recognition (WTR):

The WTR benchmark evaluates whether an RFLM can identify wireless technologies and, when applicable, link direction from spectrograms. It uses technology–direction labels generated from waveform configurations and reports top-1 accuracy.

  • Task definition: WTR measures top-1 accuracy for identifying wireless technology–direction pairs from spectrograms.The label space is derived directly from waveform-generator configurations, including NR, LTE, and WLAN.
  • Task definition: The benchmark includes separate technology–direction labels for multiple wireless standards, including DVB-S2, Bluetooth, UMTS, LTE, NR, and WLAN.The supplied passage lists these technologies as part of the test-set construction.
  • Related benchmark setup: WNUC estimates simultaneous WLAN users from wideband spectrograms using ground-truth counts obtained from WLAN configuration metadata.The benchmark groups counts into fixed-size intervals rather than always requiring exact numeric prediction.
  • Related benchmark setup: WNUC evaluates easy 15-user buckets, medium 10-user buckets, and a hard rounded numeric count.For the hard setting, counts below 10 are exact, while larger counts are rounded to the nearest multiple of 10.
  • Related benchmark setup: The WNUC test set contains 1000 WLAN-AX samples and 1000 WLAN-BE samples, with bucket-level or rounded-count accuracy reported by difficulty.Easy and medium scores require matching the true interval; hard scores require matching the rounded target.

5) New Radio Information Extraction (NRIE):

NRIE tests whether models can extract protocol-specific 5G NR attributes from spectrograms, while the broader benchmarks compare RF-grounded models with general-purpose VLMs. RF-GPT shows strong performance across NR extraction and other RF tasks, whereas unguided models perform poorly or near chance.

  • NRIE task: NRIE asks models to extract specific 5G NR attributes from individual spectrograms, including UE count, SCS, SSB pattern, CSI-RS count, and SRS count.Some questions are conditioned on downlink or uplink direction, and evaluation uses exact-match accuracy.
  • Cross-task benchmark results: RF-GPT-7B reaches 82.4% / 74.2% / 47.8% on WBMC Easy / Medium / Hard, while RF-GPT correctly identifies the number of signals in around 98% of cases.General-purpose models average below 2% on WBMC and almost never identify the correct signal count.
  • Cross-task benchmark results: RF-GPT-7B reaches 71.7% on the hard joint time–frequency WBOD task, while RF-GPT-3B reaches 91.2% / 85.2% / about 65.0% on Easy / Medium / Hard.Generic VLMs produce essentially random overlap labels, partly because they over-select the “none” category.
  • Cross-task benchmark results: RF-GPT-7B achieves 99.64% joint WTR accuracy, compared with 5.01% for Qwen2.5-VL-3B and 4.98% for Qwen2.5-VL-7B.RF-GPT also approaches 100% on most individual technologies and link directions.
  • NRIE results: RF-GPT-3B reaches 72% average NRIE accuracy, with near-perfect SCS and SSB-pattern recognition and moderate performance on UE, CSI-RS, and SRS estimation.The result demonstrates extraction of both categorical NR parameters and resource counts from spectrograms.
  • NRIE results: RF-GPT-7B achieves similar or improved performance on most NRIE tasks, notably for SRS and UE estimation.The passage attributes the broader capability to extracting richer NR structure from spectrograms.
  • NRIE results: General-purpose VLMs perform poorly on NR-specific attributes, averaging around 20% accuracy with substantial variation across tasks.They perform somewhat better on UE count but remain weak on SSB pattern and SCS.
  • Overall result: Across modulation, overlap, technology recognition, WLAN user counting, and NR attribute extraction, RF-GPT consistently outperforms general-purpose VLMs.The reported pattern supports RF grounding as the distinguishing factor across the evaluated benchmark suite.

D. Ablation Studies

The ablations examine RF-GPT’s robustness, efficiency, and design choices across NRIE, WLAN user counting, and WBMC. RF-GPT-7B outperforms conventional baselines on NRIE, while higher resolution improves WBMC accuracy at added computational cost.

  • Robustness against impairments: IQ imbalance reduces NRIE accuracy from 75.29% to 70.21%, whereas CFO and PA cause minor degradation and TDL causes a moderate drop.The impairment ranking is attributed to how each condition alters spectrogram structure, with IQ imbalance creating mirror-like or duplicated patterns.
  • CNN/Transformer comparison: RF-GPT-7B reaches 76.96% on NRIE by epoch 3, slightly exceeding ViT-H’s 76.39% after 30 epochs.RF-GPT-3B reaches 71.83% by epoch 3, close to ViT-B’s 72.34% and above EfficientNet’s 70.85% at epoch 30.
  • CNN/Transformer comparison: RF-GPT-7B’s NRIE advantage reflects unified instruction-conditioned inference, replacing separate task-specific heads with one token-level cross-entropy objective.Language instructions handle heterogeneous NR attributes without modifying the architecture, while conventional baselines require separate heads and supervision.
  • Image resolution: WBMC accuracy rises with image resolution: from 72.94% to 82.41% on Easy, 64.52% to 74.24% on Medium, and 39.64% to 47.94% on Hard from 224 to 512.Higher resolution provides finer RF tokens but increases memory, computation, and inference latency.

V. CONCLUSION

The paper presents RF-GPT as the first radio-frequency language model, integrating RF spectrograms with a multimodal LLM through synthetic instruction tuning. It outperforms general-purpose VLMs without RF priors and remains competitive with CNN and Transformer baselines, while current grounding is limited mainly to synthetic, single-input spectrograms.

  • RF-GPT is introduced as an RFLM that provides a unified interface between RF signals and natural language.
  • Synthetic waveform generation, deterministic captioning, and LLM-based instruction synthesis create a multi-technology, multi-task RF instruction-tuning dataset.
  • RF-GPT substantially outperforms general-purpose VLMs without RF priors and remains competitive with strong CNN and Transformer baselines.
  • A single model additionally produces natural-language answers, structured outputs, and explanations.
  • Current RF grounding is mainly synthetic and limited to single-input spectrograms, motivating future work on real over-the-air and multi-antenna data.
Loading 2602.14833v1…