PaperlayerView exact revision on arXiv

Source-linked AI summary

MERLIN: Building Low-SNR Robust Multimodal LLMs for Electromagnetic Signals

Junyu Shen, Zhendong She, Chenghanyu Zhang, Yuchuang Sun, Luqing Luo, Dingwei Tan, Zonghao Guo, Bo Guo, Zehua Han, Wupeng Xie, Yaxin Mu, Peng Zhang, Peipei Li, Fengxiang Wang, Yangang Sun, Maosong Sun

arXiv:2603.08174v2cs.CV

TL;DR

EM-domain MLLMs face gaps in generalizable perception and reasoning, paired signal-text data, standardized evaluation, and low-SNR robustness. The paper introduces EM-134K, EM-Bench, and MERLIN, and reports state-of-the-art benchmark performance with robustness in noisy conditions.

  • Problem

    Task-specific EM methods lack generalization across diverse perception and reasoning challenges, alongside shortages of paired data and standardized evaluation and fragility in low-SNR conditions.

    Source: §1 · p. 2

  • Method

    The paper introduces EM-134K and EM-Bench and trains MERLIN through multi-task pre-training followed by distillation from a frozen high-SNR teacher to a low-SNR student.

    Source: §1 · p. 2, §4.3.2 · p. 7

  • Results

    MERLIN achieves state-of-the-art performance on EM-Bench and exceptional robustness in noisy conditions where other methods falter.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The authors present these resources and framework as a robust, validated pathway for future EM-domain MLLM research.

    Source: §6 · p. 9

Abstract

from arXiv · show

The paradigm of Multimodal Large Language Models (MLLMs) offers a promising blueprint for advancing the electromagnetic (EM) domain. However, prevailing approaches often deviate from the native MLLM paradigm, instead using task-specific or pipelined architectures that lead to fundamental limitations in model performance and generalization. Fully realizing the MLLM potential in EM domain requires overcoming three main challenges: (1) Data. The scarcity of high-quality datasets with paired EM signals and descriptive text annotations used for MLLMs pre-training; (2) Benchmark. The absence of comprehensive benchmarks to systematically evaluate and compare the performance of models on EM signal-to-text tasks; (3) Model. A critical fragility in low Signal-to-Noise Ratio (SNR) environments, where critical signal features can be obscured, leading to significant performance degradation. To address these challenges, we introduce a tripartite contribution to establish a foundation for MLLMs in the EM domain. First, to overcome data scarcity, we construct and release EM-100k, a large-scale dataset comprising over 100,000 EM signal-text pairs. Second, to enable rigorous and standardized evaluation, we propose EM-Bench, the most comprehensive benchmark featuring diverse downstream tasks spanning from perception to reasoning. Finally, to tackle the core modeling challenge, we present MERLIN, a novel training framework designed not only to align low-level signal representations with high-level semantic text, but also to explicitly enhance model robustness and performance in challenging low-SNR environments. Comprehensive experiments validate our method, showing that MERLIN is state-of-the-art in the EM-Bench and exhibits remarkable robustness in low-SNR settings.

1. Introduction

The introduction identifies limits in EM models, paired-signal data, standardized evaluation, and low-SNR robustness. It presents EM-134K, EM-Bench, and MERLIN as resources and a training framework addressing these challenges.

1. Introduction

  • Existing task-specific EM methods lack generalization across diverse perception and reasoning challenges, while native MLLM approaches remain underexplored.The introduction identifies pipelined or task-specific architectures as a departure from the standard end-to-end MLLM paradigm.Source: §1 · p. 2
  • EM-134K addresses paired-data scarcity with over 134,000 signal-text pairs drawn from open-source signals, simulations, and real-world samples.The dataset is intended to provide a foundation for model pre-training.Source: §1 · p. 2
  • EM-Bench provides over 4,200 high-quality question-answer pairs for standardized evaluation from fundamental perception to complex reasoning.The introduction presents it as a comprehensive benchmark for EM-domain MLLMs.Source: §1 · p. 2
  • The paper defines low-SNR as below 0 dB and attributes performance collapse to noise-corrupted features and homogenized representations.It reports that linear interpolation from noisy features toward clean-signal features systematically improves performance, motivating feature-level noise handling.Source: §1 · p. 2
  • MERLIN adds a knowledge-distillation stage that trains a student on low-SNR signals to align with a frozen teacher processing corresponding high-SNR signals.The framework follows an initial multi-task pre-training stage and is designed to retain general multi-task capabilities.Source: §1 · p. 2
  • MERLIN achieves state-of-the-art EM-Bench performance and exceptional low-SNR robustness while other methods falter.The authors also report that it substantially outperforms baselines in noisy settings while retaining foundational multi-task abilities.Source: §1 · p. 3

2. Related Work

Related work traces MLLMs from vision, video, and audio to early EM models, alongside datasets for wireless perception and methods for low-SNR robustness.

Multimodal Large Language Models

  • EM MLLM efforts include RadioLLM’s Q-Former mapping of I/Q features into an LLM semantic space and encoding strategies from Spectrum-LLM and WirelessLLM.The section situates these approaches within the broader use of MLLMs for visual dialogue, video, and raw-audio interaction.Source: §2 · p. 3

Electromagnetic Datasets

  • Wireless datasets progressed from RadioML’s modulation-classification benchmark to broader Hisar-Mod and RadarCommDataset collections and the 800,000-sample EM corpus.The cited datasets expand signal, modulation, channel, or protocol coverage, while the EM corpus supports pre-training.Source: §2 · p. 3

Low-SNR Robust Perception

  • Low-SNR robustness work in images includes physics-guided enhancement, adaptive multi-scale attention, and contrastive feature-recovery methods.The section describes these approaches as improving visual perception under low-light and high-noise conditions.Source: §2 · p. 3

3. EM-134K & EM-Bench

EM-Bench organizes EM evaluation across perception and reasoning, while a shared signal corpus and two generation pathways produce distinct evaluation and training resources.

3.1. Evaluation Dimensions

  • EM-Bench divides evaluation into perception and reasoning across three levels and 14 subtasks, including signal characterization, jamming identification, fragment detection, and strategy generation.Perception uses mainly single-choice questions; reasoning uses open-ended answers. The subtasks include eight signal-characterization tasks, two jamming tasks, and four strategy-generation tasks.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Data Collection and QA Pairs Construction

  • The construction process draws on 35,869,745 IQ signals across 10 subtasks, then channels shared LLM-generated outputs into expert-refined benchmark pairs and automatically formatted fine-tuning data.The corpus uses synthetic and real-world signals, a uniform 20 MHz sampling rate, SNRs from -20 to 20 dB, and 1024-sample length; experts review benchmark pairs for accuracy, clarity, and logic.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3. Analysis

  • EM-Bench comprises 4,200 expert-validated QA pairs, with 300 pairs for each of 14 subtasks, while EM-134K contains 134,107 single-turn instruction-tuning samples.The benchmark uses single-choice perception and constrained open-ended reasoning formats; together, the benchmark and training dataset serve as evaluation and specialized training resources.Source: §3.3 · p. 5

4. MERLIN Framework

MERLIN targets low-SNR fragility with a two-stage EM MLLM framework, motivated by evidence that degraded signal features—not simply the amount of low-SNR training data—are central to the problem. It combines foundational signal-text training with teacher–student distillation and denoising feature projection.

4. MERLIN Framework

  • MERLIN uses foundational cross-modal pretraining followed by a low-SNR robustness-enhancement stage.The two-stage framework is designed to establish a signal–text link and address fragility in noisy conditions.Source: §4 · p. 5

4.1. Motivation.

  • Low-SNR data mixing gave only marginal benefit; class-overlapped embeddings mark feature collapse, while interpolation toward high-SNR embeddings recovered generative performance.These findings motivate targeting the signal feature space rather than relying only on more low-SNR training data.Source: §4.1 · p. 5, §4.2 · p. 6

4.2. Model Architecture.

  • The baseline maps signals through an encoder and two-layer MLP projector into an LLM, which processes signal and text embeddings to generate language.The projector uses GELU activation.Source: §4.2 · pp. 5–6

4.3.1 Stage1: Foundational Pretraining

  • Stage 1 trains the model as an instruction-following generator, predicting answer tokens from the signal and question while jointly optimizing encoder, projector, and LLM parameters.The objective is standard next-token prediction loss.Source: §4.3.1 · pp. 6–7

4.3.2 Stage2: Low-SNR Robustness Enhancement

  • Stage 2 trains a low-SNR student against a frozen teacher processing the corresponding high-SNR signal, combining task, feature-level, and logits-level distillation losses.The Denoising Subspace Module projects noisy student features into a signal subspace before feature distillation, filtering noise-dominated components; logits distillation aligns output distributions.Source: §4.3.2 · p. 7

5. Experiment

MERLIN is evaluated on EM-Bench against proprietary and open-source LLMs that process signals as text, and achieves state-of-the-art performance across perception and reasoning. Ablations report gains from specialized second-stage adaptation and the framework’s distillation components.

5.1. Experimental Setup

  • Both stages use AdamW, cosine scheduling, up to 8 epochs, a global batch size of 256, and an initial learning rate of 5e-5.Checkpoints are selected by early stopping on validation loss using a 10% held-out portion of training data.Source: §5.1 · p. 7
  • Stage 1 uses the full EM-134K dataset; Stage 2 adds Gaussian-noised low-SNR counterparts and replay from the pre-training set, with final evaluation on EM-Bench.The noise-injected pairs are generated from high-SNR signals.Source: §5.1 · p. 7
  • The primary baseline feeds normalized IQ sequences as numerical strings in task-specific prompts to a state-of-the-art LLM in a zero-shot setting.This baseline is intended to isolate gains from the multimodal architecture.Source: §5.1 · p. 7

5.2. Main Results

  • MERLIN achieves state-of-the-art performance across EM-Bench perception and reasoning tasks, outperforming proprietary and open-source textualized-input LLM baselines.The baselines process EM signals only as text, without a dedicated signal encoder.Source: Table 2 · p. 8, §5.2 · p. 8

5.3. Ablation Study

  • Stage-2 fine-tuning improves on the Stage-1 baseline; adding feature distillation, DSM, and logits loss culminates in the highest-performing full framework.The ablation reports accuracy on EM-Bench multiple-choice tasks.Source: §5.3 · p. 8, Table 3 · p. 8

6. Conclusion

The conclusion presents EM-134K and EM-Bench as resources addressing data scarcity and standardized evaluation, alongside MERLIN for low-SNR robustness. Experiments support state-of-the-art EM-Bench performance and robustness in noisy conditions.

6. Conclusion

  • EM-134K and EM-Bench address EM MLLM data scarcity and the lack of standardized evaluation, respectively.The conclusion describes them as foundational resources for the field.Source: §6 · pp. 8–9
  • MERLIN achieves state-of-the-art performance on EM-Bench and exhibits robustness in noisy conditions where other methods falter.The conclusion presents these results as validation of the integrated approach.Source: §6 · p. 9
  • Together, the dataset, benchmark, and MERLIN provide a validated pathway for future research into EM-domain MLLMs.Source: §6 · p. 9

A.1. Overview of Appendix

The appendix supplies supplementary details for the pre-trained baseline, dataset, and benchmark. Its overview lists implementation details, benchmark datasheets, sample visualizations, SNR robustness, and comparative-experiment prompts.

A.1. Overview of Appendix

  • The appendix adds technical specifics for the pre-trained baseline, dataset, and benchmark that could not fit in the main paper.Source: §A.1 · p. 12
  • Listed appendix materials cover baseline implementation, benchmark datasheets, sample visualizations and challenging cases, SNR robustness, and prompts for comparative experiments.Source: §A.1 · p. 12

A.2. Implementation Details of Baseline Model

The baseline uses Qwen3-8B with an integrated signal encoder, SIT, to embed IQ data as a second modality. Training first aligns signal and text semantics, then applies low-SNR knowledge distillation while freezing the LLM.

A.2. Implementation Details of Baseline Model

  • The baseline derives from Qwen3-8B and integrates a Signal Integration Transformer encoder to turn IQ data into feature vectors supplied alongside text.Source: §A.2 · p. 12
  • Stage 1 fully fine-tunes the model to align signal and text semantics; Stage 2 uses knowledge-distillation fine-tuning for low-SNR robustness.Source: §A.2 · p. 12
  • During Stage 2, the LLM parameters are frozen and only the signal encoder and projection layers are trained.Source: §A.2 · p. 12
  • Table 4 presents the two-stage training configuration and hyperparameters; SIT denotes Signal Integration Transformer encoder.Source: Table 4 · p. 12

A.3. Datasheets for benchmark

EM-Bench organizes evaluation across perception and reasoning, with 14 tasks and 4,200 VQA pairs. Figure 6 compares MERLIN with the Stage-1 baseline across 10 subtasks and varying SNR conditions.

A.3. Datasheets for benchmark

  • EM-Bench spans three hierarchy levels across Perception and Reasoning, with 14 distinct L3 tasks.Each subtask contains 300 samples, for 4,200 VQA pairs total.Source: Table 5 · p. 12
  • The benchmark table lists task abbreviations, data quantities, and answer types: Single Choice or Open-ended Strategy.Source: Table 5 · p. 12
  • Figure 6 compares MERLIN with the Stage-1 baseline across 10 subtasks under varying SNR conditions.Source: Figure 6 · p. 13

A.4. Detailed Performance Analysis across SNR Levels

Across 10 subtasks and varying SNR conditions, MERLIN leads the Stage-1 baseline on complex parameter estimation, remains resilient in modulation and detection, and matches or slightly exceeds it on saturated tasks. Additional visualizations show task-specific estimation, classification, and recognition details.

A.4. Detailed Performance Analysis across SNR Levels

  • MERLIN leads the Stage-1 baseline by often > 10% on Duty Cycle, Pulse Width, and PRF estimation across the SNR spectrum.For Modulation Recognition and Signal Detection, it maintains a clearer operational margin at −10 dB to 0 dB; on Protocol Identification and Jamming Recognition, it matches or slightly exceeds the baseline.Source: §A.4 · p. 13

A.5. Visualizations of samples and challenging cases

Related papers

Loading 2603.08174v2…