PaperlayerView exact revision on arXiv

Source-linked AI summary

PReD: An LLM-based Foundation Multimodal Model for Electromagnetic Perception, Recognition, and Decision

Zehua Han, Jing Xiao, Yiqi Duan, Mengyu Xiang, Yuheng Ji, Xiaolong Zheng, Chenghanyu Zhang, Zhendong She, Junyu Shen, Dingwei Tan, Shichu Sun, Zhou Cong, Mingxuan Liu, Fengxiang Wang, Jinping Sun, Yangang Sun

arXiv:2603.28183v2cs.AI

TL;DR

General-purpose multimodal models lack intrinsic EM signal priors, and existing datasets and systems do not unify the domain’s tasks in an instruction-aligned framework. PReD combines multi-view EM data with multi-stage training for perception, recognition, and decision-making. It outperforms general-purpose MLLMs on PReD-Bench while retaining competitive general multimodal performance.

  • Problem

    General-purpose MLLMs lack intrinsic EM signal priors, while public datasets and existing approaches provide limited diversity and do not unify the domain’s tasks in an instruction-aligned multimodal framework.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    PReD combines complementary visual representations of EM signals, multitask instruction supervision, and a multi-stage training pipeline in a unified foundation model.

    Source: §1 · p. 3, §4.1 · p. 6

  • Results

    PReD significantly outperforms current state-of-the-art general-purpose MLLMs on PReD-Bench; on AJSD, PReD achieves BLEU4=0.253 while generalist models score near zero.

    Source: §6 · p. 8, §5.1 · p. 8

  • Takeaways & Limitations

    PReD demonstrates specialized EM proficiency while maintaining strong, competitive performance on standard multimodal benchmarks.

    Source: §6 · p. 8, §5.2 · p. 8

Abstract

from arXiv · show

Multimodal Large Language Models have demonstrated powerful cross-modal understanding and reasoning capabilities in general domains. However, in the electromagnetic (EM) domain, they still face challenges such as data scarcity and insufficient integration of domain knowledge. This paper proposes PReD, the first foundation model for the EM domain that covers the intelligent closed-loop of "perception, recognition, decision-making." We constructed a high-quality multitask EM dataset, PReD-1.3M, and an evaluation benchmark, PReD-Bench. The dataset encompasses multi-perspective representations such as raw time-domain waveform, frequency-domain spectrograms, and constellation diagrams, covering typical features of communication and radar signals. It supports a range of core tasks, including signal detection, modulation recognition, parameter estimation, protocol recognition, radio frequency fingerprint recognition, and anti-jamming decision-making. PReD adopts a multi-stage training strategy that unifies multiple tasks for EM signals. It achieves closed-loop optimization from end-to-end signal understanding to language-driven reasoning and decision-making, significantly enhancing EM domain expertise while maintaining general multimodal capabilities. Experimental results show that PReD achieves state-of-the-art performance on PReD-Bench constructed from both open-source and self-collected signal datasets. These results collectively validate the feasibility and potential of vision-aligned foundation models in advancing the understanding and reasoning of EM signals.

1. Introduction

General-purpose MLLMs lack intrinsic representations of EM signal structure, while existing EM-oriented systems and datasets do not provide a unified, instruction-aligned framework across perception, recognition, and decision-making. PReD addresses this gap with a multimodal foundation model and cross-view, multitask supervision.

Research gap

  • General-purpose MLLMs lack intrinsic EM priors such as I/Q waveforms, spectrogram structure, constellation geometry, and micro-Doppler signatures, limiting their use where expert, auditable signal reasoning is required.The paper identifies spectrum management, wireless security, and radar semantic understanding as demanding applications.Source: §1 · p. 2
  • Public signal repositories are scarce or narrow, EM tasks are usually modeled separately, and general MLLMs often fail to internalize EM priors under difficult signal conditions.The cited conditions include low SNR, strong interference, and complex waveforms.Source: §1 · p. 2
  • Prior EM-oriented LLMs emphasize knowledge or network planning, while classical RF models are typically task-specific and lack instruction alignment and standardized evaluation [5–10, 15–17].The paper describes these limitations as obstacles to closed-loop perception, recognition, and decision-making generalization and verifiability.Source: §1 · pp. 2–3

Contributions

  • PReD combines cross-view EM signal representations, reasoning-style supervision, and multi-stage multitask training to unify signal perception, recognition, and anti-jamming decisions.The paper presents PReD as the first integrated EM foundation model and reports competitive performance across diverse EM tasks.Source: §1 · p. 3

2. Related Work

The related work traces general multimodal models’ language-centered interfaces and the challenges of transferring them to EM signals. Existing EM-oriented efforts remain limited in raw-signal coverage, task integration, and evaluation.

2. Related Work

  • Image-text, cross-sensor, and video-language models use transferable interfaces and language-centered multitask paradigms for visual, audio, and video understanding [1] [2] [4] [18] [19] [20] [22] [23] [24] [25].Examples include BLIP-2, LLaVA, MiniGPT-4, Qwen-VL, ImageBind, Flamingo, Kosmos-2, and GPT-4Vis.Source: §2 · p. 3

2.1. General-Purpose Multimodal Large Models

  • General multimodal models commonly align modalities through encoders and projection or query modules, but EM transfer faces missing signal priors, interface mismatch, and supervision or metric mismatch [26] [28] [29].The paper contrasts EM needs such as regression and closed-loop control with generic QA-style generation.Source: §2.1 · p. 3

2.2. Large-Model Directions in the EM Domain

  • WirelessLLM, RadioLLM, and RFF-LLM focus on knowledge QA, cognitive-radio workflows, or device interpretation, with limits in raw-signal modeling and task breadth [5] [6] [7].The paper argues that existing approaches rarely unify detection, recognition, estimation, emitter identification, and anti-jamming decisions.Source: §2.2 · p. 3
  • The paper motivates EM-specific foundation models to integrate data, modeling, and evaluation; Figure 2 traces signal visualizations into QA pairs and a benchmark kept separate from training.The stated motivation includes cross-view consistency among I/Q, spectrum, spectrogram, and constellation representations.Source: §2.2 · p. 4, Figure 2 · p. 4

3. The PReD Dataset and Benchmark

The paper builds an EM instruction corpus from multiple signal views and task families, then forms a separate benchmark for evaluating perception, recognition, and decision-making. Its supervision spans open-ended and multiple-choice formats.

3.1. Signal Representation and Task Formulation

  • The instruction corpus covers signal segment detection, signal parameter estimation, modulation recognition, protocol recognition, emitter identification, and anti-jamming strategy decision-making.The task instructions pair signal views with multi-format question-answer templates.Source: §3.1 · p. 4
  • Each signal is represented through complementary I/Q constellation, FFT magnitude spectrum, STFT spectrogram, and raw I/Q waveform views, with OpenQA and MCQA instruction formats.MCQA includes an “Unable to answer” option; OpenQA uses structured output tags to facilitate automated scoring and downstream integration.Source: §3.1 · pp. 4–5

3.2. The PReD-1.3M Corpus

  • PReD-1.3M contains 1.33 × 10^6 question-answer pairs, approximately 90.3% OpenQA and 9.7% MCQA, with task scale and composition summarized in Figure 3.The corpus draws on public and controllable synthetic sources.Source: §3.2 · p. 5, Figure 3 · p. 5
  • Signal detection uses radar, communication, and noise segments, while parameter estimation requests pulse width, period, count, or delay from amplitude traces.The detection set contains approximately 1.98 × 10^5 labeled segments; parameter estimation contains 269K samples and uses MCQA values within ±1 µs of the correct value.Source: §3.2 · p. 5
  • Modulation recognition uses RadioML2016.10a across varied modulation classes, while protocol recognition spans avionics, short-range, Bluetooth, IEEE 802.11, and IEEE 802.15.4 protocols.The modulation and protocol datasets contain 198K and 263K samples, respectively.Source: §3.2 · p. 5
  • Emitter identification combines real and simulated receiver-chain variations, and anti-jamming examples span noise, multiple-interferer regimes, and diverse radar waveforms.The emitter-identification set contains 161K samples; anti-jamming includes causal OpenQA pairs for strategy decision-making.Source: §3.2 · pp. 5–6

3.3. The PReD-Bench Benchmark

  • PReD-Bench stratifies sampling by SNR, remains isolated from training, and evaluates six tasks using standardized OpenQA and MCQA inputs, scoring, and quality auditing.It combines multi-origin public data and controllable simulation, with anti-jamming included to test reasoning under adversarial EM conditions.Source: §3.3 · p. 6, Table 1 · p. 6

4. PReD Model

PReD combines a vision encoder, projector, and language-model decoder to process multi-view EM renderings alongside instructions. A four-stage curriculum aligns visual features, expands multimodal competence, and adapts the model to EM tasks.

4.1. Model Architecture

  • SigLIP encodes each EM view, a two-layer MLP maps visual tokens into the language-model embedding space, and a pretrained Qwen3-8B decoder processes them with instructions.Visual tokens are concatenated with boundary tokens and view-index embeddings before being prepended to the instruction prompt.Source: §4.1 · p. 6, Figure 4 · p. 7

4.2. Multi-Stage Training

  • Training progresses from projector warm-up through general single-view and multi-view learning to EM adaptation on PReD-1.3M across all six tasks.The curriculum mixes 40% of Stage-2 data into Stage 3, then 25% of Stage-2 and 50% of Stage-3 data into Stage 4 to retain earlier capabilities.Source: §4.2 · pp. 6–7, Table 2 · p. 7

5. Experiment and Results

PReD outperforms general-purpose baselines across the EM perception–recognition–decision pipeline, with its clearest advantage on anti-jamming strategy generation. It also retains competitive general multimodal performance, including a higher HallusionBench score than GPT-4V.

5.1. Performance Evaluation of EM Tasks

  • PReD scores BLEU4=0.253 on AJSD, while generalist models score near zero and fail at complex recognition and decision-making.The paper reports that PReD consistently outperforms general-purpose baselines across the pipeline; the Table 3 caption notes that general MLLMs are omitted for EI because it requires device-specific RF fingerprint learning.Source: §5.1 · pp. 7–8, Table 3 · p. 8

5.2. Evaluation on General Multimodal Benchmarks

  • 65.4 is PReD’s average general-multimodal score, versus 64.0 for LLaVA-Next and 65.2 for Qwen-VL-Max.The authors describe PReD as maintaining competitive general multimodal understanding while benefiting from domain-oriented enhancements.Source: §5.2 · p. 8
  • 54.5 vs. 46.5 on HallusionBench favors PReD over GPT-4V, while their overall scores are 65.4 vs. 66.9.The authors suggest the training paradigm may contribute to improved reliability in hallucination-prone scenarios.Source: §5.2 · p. 8

6. Conclusions

The conclusion presents PReD as a foundation model unifying EM perception, recognition, and decision-making, combining a large EM corpus and multi-stage training with strong specialized and competitive general multimodal performance. The appendix documents the training and evaluation setup and adds complementary analyses.

6. Conclusions

  • PReD unifies the full EM perception, recognition, and decision-making pipeline and outperforms general-purpose MLLMs on PReD-Bench while remaining competitive on standard multimodal benchmarks.The authors frame vision-aligned foundation models as having potential for adaptation to other complex scientific domains.Source: §6 · p. 8

Appendix

  • The appendix details the four-stage curriculum, data composition, progressive tuning, hyperparameters, and automatic metrics for free-form OpenQA# answers.It also includes extensive experimental analysis and visual and qualitative material.Source: Appendix · pp. 8–9
  • Supplementary experiments examine domain-specific expertise, input-modality contributions, out-of-domain generalization, and robustness across SNRs.The appendix also presents benchmark-composition visualizations, comparative performance plots, and representative EM-task case studies.Source: Appendix · p. 9

A. Details of Training Pipeline for PReD

PReD’s four-stage curriculum progresses from projector alignment through general single- and multi-view training to EM specialization. Data mixing and increasing visual capacity support this progression while the final stage retains general-image data.

A. Details of Training Pipeline for PReD

  • Stage 1 warms up only the projector with the vision encoder and language model frozen, using 384 resolution and 729 vision tokens for initial vision–language alignment.It uses the blip_laion_cc_sbu_558k dataset and a learning rate of 1 × 10^-3.Source: A. Details of Training Pipeline for PReD · p. 9
  • Stage 2 unfreezes the full model for single-image instruction tuning, while Stage 3 adds static multi-view inputs and mixes in 40% of Stage 2’s general single-image corpus.Stage 2 uses 858,231 samples; Stage 3 uses 489,974 samples, expands the token budget to 729 × 10, and doubles maximum sequence length to 8,192.Source: A. Details of Training Pipeline for PReD · p. 9
  • Stage 4 uses 1,332,759 PReD-1.3M samples for EM-task tuning, mixed with 25% Stage 2 single-image data and 50% Stage 3 multi-image data to prevent catastrophic forgetting.It retains the maximum spatial coverage and token capacity established in Stage 3.Source: A. Details of Training Pipeline for PReD · p. 9
  • Across the four stages, AdamW, DeepSpeed-Zero3, cosine learning-rate scheduling, zero weight decay, and a 0.03 warmup ratio are used.The pipeline moves from general visual perception to multi-view processing and then EM expertise; each stage trains for one epoch on a 2 × 8 A100 GPU setup.Source: A. Details of Training Pipeline for PReD · pp. 9–10

B. Details of Evaluation Metrics

The appendix defines four automatic metrics for evaluating free-form OpenQA# answers, spanning n-gram precision and recall, alignment-based matching, and consensus against multiple references.

B. Details of Evaluation Metrics

  • BLEU-4 measures local 1–4-gram precision with a brevity penalty, while ROUGE assesses n-gram recall and LCS-based F-score.METEOR uses alignment-based precision and recall with a fragmentation penalty and stem and synonym matching; CIDEr uses TF–IDF-weighted n-grams and cosine similarity across references [51].Source: B. Details of Evaluation Metrics · p. 10

C.1. Ablation Study on Domain-Specific EM Expertise Stage (Stage-4)

The Stage-4 ablation compares fine-tuning on EM data alone with mixing EM and general visual corpora. EM-only fine-tuning slightly improves specialized-task performance but sharply reduces general multimodal performance.

C.1. Ablation Study on Domain-Specific EM Expertise Stage (Stage-4)

  • The ablation compares Stage-4 fine-tuning on PReD-1.3M alone with fine-tuning on PReD-1.3M mixed with general visual corpora from the earlier stages.Both variants are evaluated on specialized EM tasks and general multimodal capabilities.Source: §C.1 · p. 10
  • The study reports standard accuracy for SSD, SPE, MR, PR, and EI, and a four-metric composite score for AJSD.The AJSD score averages BLEU-4, ROUGE-L, METEOR, and CIDEr, then multiplies by 100.Source: §C.1 · p. 10
  • PReD− averaged 73.10% on specialized EM tasks versus 71.85% for PReD, while general-benchmark averages were 35.92% versus 65.37%, respectively.The authors conclude that the slight specialized-task advantage comes with a substantial loss in general multimodal capability.Source: §C.1 · p. 10

C.2. Ablation Study on Input Modalities

The modality ablations favor a three-view input as an efficiency–accuracy trade-off, while zero-shot and varying-SNR analyses report recognition robustness patterns. The section also visualizes benchmark composition across tasks and question formats.

C.2. Ablation Study on Input Modalities

  • IQ alone is the strongest single modality, with 50.8% modulation-recognition accuracy and a 0.363 AJSD average score.The authors attribute its value to retaining the signal’s raw amplitude and phase variations.Source: §C.2 · p. 11
  • 64.2% modulation-recognition accuracy for Constellation+IQ exceeds the sum of its parts, whereas FFT+STFT reaches 37.8%, which the authors associate with redundancy.The paper relates the gain to IQ’s temporal details complementing constellation geometry, while STFT already captures spectral information over time.Source: §C.2 · p. 11
  • 70.0% MR accuracy and 0.414 AJSD average score from Constellation+STFT+IQ reach 97.2% and 85.1% of the four-modality results, respectively.All four modalities yield 72.0% MR accuracy and a 0.4860 AJSD average score; the authors recommend the trio when resources are constrained.Source: §C.2 · p. 12, Table 8 · p. 12, Table 9 · p. 13

C.3. Out-of-Domain Generalization for Modulation Recognition

  • PReD achieved 66.0% zero-shot modulation-recognition accuracy (1452/2200) on the OOD RadioML2016.04c benchmark, demonstrating generalization to unseen signal environments without any fine-tuning.Performance is modest below −10 dB, improves from −10 dB to 10 dB, and stabilizes above 10 dB.Source: §C.3 · p. 12, Figure 6 · p. 13

C.4. Performance Analysis under Varying SNR

  • MR and PR accuracy degrades gracefully as SNR falls, with the largest drop in negative SNR and non-trivial recognition still retained.EI is excluded because its real-world captured signals lack ground-truth SNR annotations.Source: §C.4 · p. 13, Figure 7 · p. 13

D. Visual Analysis of PReD-Bench and Benchmark Results

D.1. Hierarchical Composition of PReD-Bench

  • Figure 8 distributes benchmark items across three dimensions, six tasks, and question types; perception mixes OpenQA and MCQA, most recognition tasks use MCQA, and AJSD is OpenQA.Perception, recognition, and decision-making account for 59.1%, 17.2%, and 23.6%, respectively.Source: §D.1 · p. 14, Figure 8 · p. 14

D.2. Comparative Performance Visualization on EM Tasks

PReD forms the outer envelope in the five-task performance radar visualization, while general-purpose MLLMs contract on open-ended decision-making. Qualitative examples illustrate responses across all six benchmark tasks, with selected open-ended tasks standardized for evaluation.

D.2. Comparative Performance Visualization on EM Tasks

  • PReD forms the outer envelope and largest enclosed area in the five-task radar plot, while general-purpose MLLMs contract sharply on AJSD.GPT-5, Gemini-2.5-Pro, and Claude-Sonnet-4 show severe contraction toward the AJSD axis; their capabilities are described as moderate on foundational perception and recognition tasks.Source: §D.2 · p. 14

D.3. Qualitative Examples on PReD-Bench Tasks

  • Figure 10 presents representative user queries and PReD responses across all six core tasks, from perception and recognition to decision-making.For MR, PR, and EI, open-ended responses were removed from the test set to standardize evaluation because different expressions can convey the same meaning.Source: §D.3 · pp. 14–15, Figure 10 · p. 16

Related papers

Loading 2603.28183v2…